mobile image processing part2

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 1
Mobile image processing – Part 2
 Mobile application development on Android
 Project 1: “Hello Image”, reading/writing/modifying images
 Project 2: “Viewfinder EE368”, processing viewfinder frames
 Project 3: “CVCamera”, utilizing OpenCV functions
 Mobile application examples
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 2
Recognizing video at a glance
(1) User snaps a photo of screen.
(2) Our system identifies video and
frame within the video.
(3) User resumes video on the phone.
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 3
/>Recognizing video at a glance
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 4
Mobile device versus personal computer
Feature Motorola DROID Typical PC
Screen size
4.6 in × 2.4 in 25 in × 16 in
Processor speed 550 MHz 3.0 GHz
RAM 256 MB 4 GB
Permanent storage
133 MB internal
8 GB external flash
500 GB disk
Internet access
Typically 3G or
WiFi at hotspots
WiFi or wired Ethernet
Telephony Core feature Supplementary feature

Camera
5 MP camera
embedded
External webcam
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 5
“Viewfinder EE368” project
 Goals of this project
z Learn how to access frames from the viewfinder
z Learn how to modify camera parameters
z Learn how to augment the viewfinder frames
 Step-by-step instructions available in Tutorial #1:
z />Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 6
Android manifest file
…
…
<application android:icon="@drawable/icon"
android:label="@string/app_name">
<activity android:name=".ViewfinderEE368"
android:label="@string/app_name"
android:screenOrientation="landscape“
android:theme="@android:style/Theme.NoTitleBar">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
<uses-sdk android:minSdkVersion=“7" />
<uses-permission android:name="android.permission.CAMERA"/>
Set the proper

hardware permissions
Landscape mode
without title bar
Main
Activity
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 7
Class hierarchy for “Viewfinder EE368”
Viewfinder
EE368
(Activity)
Preview
(View)
Draw on
Top
(View)
Manage two views and
manage program
open/close
Handle incoming
viewfinder frames
and adjust camera
parameters
Augment viewfinder
frames with a new
layer of information
on top
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 8
Viewfinder class: full screen mode
Viewfinder EE368
Icon and title bars visible Icon and title bars hidden

public void onCreate(Bundle savedInstanceState) {

getWindow().setFlags(
WindowManager.LayoutParams.FLAG_FULLSCREEN,
WindowManager.LayoutParams.FLAG_FULLSCREEN
);
requestWindowFeature(Window.FEATURE_NO_TITLE);

}
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 9
Class hierarchy for “Viewfinder EE368”
Viewfinder
EE368
(Activity)
Preview
(View)
Draw on
Top
(View)
Manage two views and
supervise program
open/close
Handle incoming
viewfinder frames
and adjust camera
parameters
Augment viewfinder
frames with a new
layer of information
on top

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 10
Timeline of events on the mobile device
Time
New frame
New frame
Capture
Augment
Preview
DrawOnTop
Capture
Augment
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 11
Preview class: frames go down two paths
Preview
(View)
Viewfinder frames
…
Display on
phone screen
Forward to custom
callback function
myCamera.setPreviewCallback(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera)
{ Pass data to DrawOnTop class }
});
Data in YCbCr 4:2:0 format
w
h
w/2
w/2

h/2
h/2
Y
Cb Cr
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 12
Preview class: set camera parameters
Preview
(View)
Camera.Parameters parameters = myCamera.getParameters();
parameters.setPreviewSize(640, 480);
parameters.setPreviewFrameRate(15);
parameters.setSceneMode(Camera.Parameters.SCENE_MODE_NIGHT);
parameters.setFocusMode(Camera.Parameters.FOCUS_MODE_AUTO);
myCamera.setParameters(parameters);
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 13
Class hierarchy for “Viewfinder EE368”
Viewfinder
EE368
(Activity)
Preview
(View)
Draw on
Top
(View)
Manage two views and
manage program
open/close
Handle incoming
viewfinder frames
and adjust camera

parameters
Augment viewfinder
frames with a new
layer of information
on top
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 14
DrawOnTop class: YCbCr to RGB conversion
84.276
256
41.516
256
08.298
58.135
256
12.208
256
29.100
256
08.298
921.222
256
58.408
256
08.298
−
⋅
+
⋅
=
+

⋅
−
⋅
−
⋅
=
−
⋅
+
⋅
=
CbY
B
CrCbY
G
CrY
R
w
h
w/2 w/2
h/2
h/2
Y
Cb Cr
w
h
RGB
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 15
DrawOnTop class: create some paint brushes
myPaintRed = new Paint();

myPaintRed.setStyle(Paint.Style.FILL);
myPaintRed.setColor(Color.RED);
myPaintRed.setTextSize(25);
myPaintGreen = new Paint();
myPaintGreen.setStyle(Paint.Style.FILL);
myPaintGreen.setColor(Color.GREEN);
myPaintGreen.setTextSize(25);
myPaintBlue = new Paint();
myPaintBlue.setStyle(Paint.Style.FILL);
myPaintBlue.setColor(Color.BLUE);
myPaintBlue.setTextSize(25);
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 16
DrawOnTop class: draw histogram bars
Bin 0 Bin 1 Bin 2 Bin 3 Bin 4
// Draw red rectangle
Rect rect = new Rect(left x, top y,
right x, bottom y);
myCanvas.drawRect(rect, myPaintRed);
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 17
Viewfinder EE368 demo
/>Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 18
Real-time debugging using DDMS (1)
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 19
Real-time debugging using DDMS (2)
File system
on device
Messages on
the device
0123456789ABC
Device

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 20
Taking a screenshot of the phone (1)
0123456789ABC
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 21
Taking a screenshot of the phone (2)
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 22
Class Project: Mobile Image Completion
H. Wang and A. Lin, Spring 2010
/>Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 23
Class Project: Mobile HDR Imaging
J. Mathe and T. Wong, Spring 2010
/>Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 24
Mobile image processing – Part 2
 Mobile application development on Android
 Project 1: “Hello Image”, reading/writing/modifying images
 Project 2: “Viewfinder EE368”, processing viewfinder frames
 Project 3: “CVCamera”, utilizing OpenCV functions
 Mobile application examples