mobile image processing part2

Mobile image processing – Part 2
 Mobile application development on Android

“
Hello World
”
project

Hello

World

project
 “Viewfinder EE368” project
 Ima
g
e processin
g
librar
y
ggy
 Mobile application examples
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 1
Google Goggles
Product recognition
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 2
/>How to download Google Goggles
 Open the Android Market application

Search for
“

Google Goggles
”

Search

for

Google

Goggles
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 3
Recognizing video at a glance
Visual bookmarks for movies and TV shows
(
1
)
User
snaps
a
photo
of
screen
(3) User resumes video on the phone
(
1
)
User
snaps
a
photo

of
screen
.
(3)

User

resumes

video

on

the

phone
.
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 4
(2) Ou
r
system identifies video and
frame within the video.
Recognizing video at a glance
Visual bookmarks for movies and TV shows
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 5
/>Mobile devices versus personal computers
Feature Motorola DROID Typical PC
Screen size 4.6 in x 2.4 in 25 in x 16 in
Processor speed 550 MHz 3.0 GHz
RAM

256
MB
4GB
RAM
256
MB
4

GB
Disk space
133 MB internal
8 GB external flash
500 GB disk
Internet access
Typically 3G
WiFi at hotspots
WiFi or wired Ethernet
Tele
p
hon
y
Core feature Su
pp
lementar
y
feature
py
pp y
Camera
5 MP camera

embedded
External webcam
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 6
“Viewfinder EE368” project
 Goals of this project
z Learn how to access frames from the viewfinde
r
z Learn how to modify camera parameters
z Learn how to augment the viewfinder frames
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 7
Beginning the project
 Download the tutorial from class page
http://ee368 stanford edu/Android
http://ee368
.
stanford
.
edu/Android
 Create a new Android project
 Recommended settin
g
s
g
z Project Name: ViewfinderEE368
z Android 2.0.1
z
Application Name: Viewfinder EE368
z
Application

Name:

Viewfinder

EE368
z Package Name: com.example.viewfinderee368
z Create Activity: ViewfinderEE368
z Min SDK Version: 6
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 8
Android manifest file (AndroidManifest.xml)
Landscape mode
…
Landscape

mode
Main Activit
y
y
…
Set the proper
hardware permissions
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 9
Class hierarchy
Viewfinder
Manage two views and
Viewfinder

EE368
(Activity)
Manage

two

views

and

manage program
open/close
Draw on
Preview
(View)
Draw

on

Top
(View)
Handle incoming
viewfinder frames
and ad
j
ust camera
Augment viewfinder
frames with a new
la
y
er of information
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 10
j

parameters
y
on top
Viewfinder class: full screen mode
Icon and title bars visible
Icon and title bars hidden
Viewfinder EE368
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 11
Class hierarchy
Viewfinder
Manage two views and
Viewfinder

EE368
(Activity)
Manage

two

views

and

manager program
open/close
Draw on
Preview
(View)
Draw

on

Top
(View)
Handle incoming
viewfinder frames
and ad
j
ust camera
Augment viewfinder
frames with a new
la
y
er of information
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 12
j
parameters
y
on top
Timeline of events on the mobile device
Nf
Nf
N
ew
f
rame
N
ew
f
rame

Capture
Preview
Capture
Capture

Augment
Preview
DrawOnTop
Capture

Augment
Time
Frame Interval
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 13
Preview class: frames go down two paths
Preview frames Display on
phone screen
Preview
(View)
…
phone

screen

Forward to
callback function
callback

function
…

Data in YCbCr 4:2:0 format
width w
h
eight h
w/2
w/2
h/2
h/2
Cb Cr
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 14
h
Y
Preview class: set camera parameters
Preview
(View)
(View)
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 15
Class hierarchy
Viewfinder
Manage two views and
Viewfinder

EE368
(Activity)
Manage

two

views

and

manage program
open/close
Draw on
Preview
(View)
Draw

on

Top
(View)
Handle incoming
viewfinder frames
and ad
j
ust camera
Augment viewfinder
frames with a new
la
y
er of information
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 16
j
parameters
y
on top
DrawOnTop class: create a paint brush
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 17

DrawOnTop class: YCbCr to RGB
width w
t
h
w/2 w/2
2
2
w
heigh
t
h/
2
h/
2
Y
Cb Cr
h
RGB
921.222
256
58.408
256
08.298
−
⋅
+
⋅
=
CrY
R

58.135
256
12.208
256
29.100
256
08.298
256
256
+
⋅
−
⋅
−
⋅
=
CrCbY
G
84.276
256
41.516
256
08.298
−
⋅
+
⋅
=
Cb
Y

B
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 18
DrawOnTop class: draw histogram bars
Bin 0 Bin 1 Bin 2 Bin 3 Bin 4
Rect rect = new Rect( left x, top y, right x, bottom y );
Canvas.drawRect( rect );
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 19
Viewfinder EE368 demo
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 20
/>Real-time debugging using DDMS (1)
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 21
Real-time debugging using DDMS (2)
03EH368929E3K
Device
File system
on device
Device
Messages on
the device
the

device
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 22
Danger of unresponsiveness
 Users notice delays longer than ~0.5 sec

Android only lets your program
“
hang
”

or
“
freeze
”
for
~
5sec

Android

only

lets

your

program

hang

or

freeze

for

5

sec

before bringing up warning dialog
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 23
Activity life cycle
[ Professional Android Application Development ]
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 24
[

Professional

Android

Application

Development

]