mobile image processing part 3

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 1
Mobile image processing – Part 3
 Mobile application development on Android
 Project 1: “Hello Image”, reading/writing/modifying images
 Project 2: “Viewfinder EE368”, processing viewfinder frames
 Project 3: “CVCamera”, utilizing OpenCV functions
 Mobile application examples
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 2
/>Mobile feature tracking
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 3
/>Mobile book spine recognition
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 4
Mobile book spine recognition
Server
Mobile Device
Track
Camera Pose
…
Low Motion
High Motion
Send Query
Frame
Query
Vocab. Tree
Check
Geometry
Send
Response
Time
Wireless
Network

Viewfinder
Frames
Signal Processing and Linear Systems
Extract
Features
Segment
Spines
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 5
Jon’s Java Imaging Library (JJIL)
 Publicly available source code
/> Collection of common image processing functions
 Images are processed through Sequences
Rgb
3x3Average
Rgb
SubSample
Sequence
RgbImage
RgbImage
Sequence.Add
(new Rgb3x3Average)
Sequence.Add
(new RgbSubSample)
Sequence.Push
(RgbImage)
Sequence.GetFront()
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 6
JJIL: 3x3 averaging filter
Original Image
Processed Image

Rgb3x3Average
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 7
JJIL: Horizontal differences
Original Image
Processed Image
RgbAvgGray Gray8HorizSimpleEdge
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 8
JJIL functions
ApplyMaskRgb
BinaryHeap
Complex32Gray32
Complex32IFFT
Fft1d
Gray16Gray8
Gray16LinComb
Gray16Threshold
Gray32Div
Gray32Gray8
Gray32Scale2Gray8
Gray32Threshold
Gray3Bands2Rgb
Gray82Gray32
Gray8Abs
Gray8Add
Gray8And
Gray8CannyHoriz
Gray8ConnComp
Gray8Crop
Gray8DeblurHorizHalftone
Gray8DetectHaarMultiScale

Gray8Fft
Gray8GaussDeblurHoriz
Gray8GaussHoriz
Gray8GaussSmoothVert
Gray8Gray32
Gray8Hist
Gray8HistEq
Gray8HistMatch
Gray8HorizSimpleEdge
Gray8HorizSum
Gray8HorizVar
Gray8HorizVertContrast
Gray8InverseFilter
Gray8LinComb
Gray8Peak3x3
Gray8Reduce
Gray8Shrink
LinefitHough
Rgb3x3Average
RgbAvgGray
RgbClip
RgbCrop
RgbDimMask
RgbHorizGaussSmooth
RgbHsv
RgbMaskedAbsDiff
RgbMaxContrast2Gray
RgbMaxDiff
RgbMeanStdDev
RgbMinDiff

RgbSelectGray
RgbSubSample
…
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 9
“CVCamera” project
 Goals of this project
z Learn how to incorporate C/C++ code into an Android program
z Learn how to utilize OpenCV library functions
z Learn how to draw feature keypoints on viewfinder frames
 Step-by-step instructions available in Tutorial #2:
z />Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 10
OpenCV: Open Source Computer Vision
/>Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 11
OpenCV for Android
 Port of OpenCV to Android framework
z Over 2K optimized algorithms written in C/C++
z Compiled with customized Android NDK (Crystax r4)
z Enables popular CV functions to be used on mobile images/videos
 Differences from regular Android programming
z Requires writing C/C++ code
z Requires writing Java Native Interface (JNI) wrappers
z Requires using Android NDK in addition to Android SDK
 Class tutorial #2 helps you make the transition
z How to set up OpenCV programming environment
z How to write Android apps that call OpenCV functions
z How to integrate NDK compilation into Eclipse IDE
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 12
Project structure for “CVCamera”
CVCamera
Activity

(Java)
Processor
(C++)
CVCamera
JNI
(Java)
Manages user interface and
program execution
Processes a “pool”
of images
Interfaces with
C/C++ code
OpenCV Library
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 13
CVCamera class: menu options
public boolean onCreateOptionsMenu(Menu menu) {
menu.add("FAST");
menu.add("STAR");
menu.add("SURF");
menu.add("MSER");
menu.add("Chess");
menu.add("Settings");
return true;
}
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 14
CVCamera class: calling feature extractors
public boolean onOptionsItemSelected(MenuItem item) {
LinkedList<PoolCallback> defaultCallbackStack =
new LinkedList<PoolCallback>();
defaultCallbackStack.addFirst(myGLView.getDrawCallback());

if (item.getTitle().equals("FAST")) {
defaultCallbackStack.addFirst(new FastProcessor());
}
else if (item.getTitle().equals("STAR")) {
defaultCallbackStack.addFirst(new STARProcessor());
}
else if (item.getTitle().equals("SURF")) {
defaultCallbackStack.addFirst(new SURFProcessor());
}
else if (item.getTitle().equals("MSER")) {
defaultCallbackStack.addFirst(new MSERProcessor());
}
myPreview.addCallbackStack(defaultCallbackStack);
return true;
}
Stack of callback
functions
Draw callback
Feature
extraction
callbacks
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 15
Project structure for “CVCamera”
CVCamera
Activity
(Java)
Processor
(C++)
CVCamera
JNI

(Java)
Manages user interface and
program execution
Processes a “pool”
of images
Interfaces with
C/C++ code
OpenCV Library
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 16
CVCamera JNI class: interface to C/C++ code
static {
try {
System.loadLibrary("android-opencv");
System.loadLibrary("cvcamera");
} catch (UnsatisfiedLinkError e) {
throw e;
}
}
public final static native int DETECT_FAST_get();
public final static native int DETECT_STAR_get();
public final static native int DETECT_SURF_get();
public final static native int DETECT_MSER_get();
public final static native long new_Processor();
public final static native void delete_Processor(long jarg1);
Each function
prototype
corresponds to an
external C/C++
function
Load OpenCV and

CVCamera
libraries built by
the NDK
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 17
Project structure for “CVCamera”
CVCamera
Activity
(Java)
Processor
(C++)
CVCamera
JNI
(Java)
Manages user interface and
program execution
Processes a “pool”
of images
Interfaces with
C/C++ code
OpenCV Library
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 18
Processor class: initialize feature detectors
class Processor {
private:
cv::StarFeatureDetector my_stard;
cv::FastFeatureDetector my_fastd;
cv::SurfFeatureDetector my_surfd;
cv::MserFeatureDetector my_mserd;
std::vector<cv::KeyPoint> my_keypoints;
public:

Processor():
my_stard(STAR detector parameters),
my_fastd(FAST detector parameters),
my_surfd(SURF detector parameters),
my_mserd(MSER detector parameters)
{}
void detectAndDrawFeatures
(int idx, image_pool* pool, int feature_type);
};
Instantiation list
Different feature
extractors stored
as member objects
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 19
Processor class: detect and draw keypoints
// Detect feature keypoints
Mat grey_im = pool->getGrey(idx);
Mat color_im = pool->getImage(idx);
my_keypoints.clear();
my_mserd->detect(grey_im,
my_keypoints);
// Draw feature keypoints
vector<KeyPoint>::const_iterator it;
for (it = my_keypoints.begin();
it != my_keypoints.end(); ++it)
{
// Draw black circle
Point2f pt = it->pt;
circle(color_im, pt, it->size,
cvScalar(0,0,0,0), 2);

// Draw yellow circle
pt.x += 1;
pt.y += 1;
circle(color_im, pt, it->size,
cvScalar(255,255,0,0), 2);
}
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 20
CVCamera demo
/>Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 21
Communications between phone and server
Phone
Server
HTTP Post (e.g., Image, Features)
HTTP Get (e.g., Identification)
Wireless
Network
Apache
PHP
MATLAB / C
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 22
HTTP file upload
URL connectURL = new URL(web address);
HttpURLConnection conn = connectURL.openConnection();
conn.setRequestMethod(“POST”);
DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
FileInputStream fis = new FileInputStream(file location);
int bufferSize = 1024;
byte[] buffer = new byte[bufferSize];
int bytesRead = fileInputStream.read(buffer, 0, bufferSize);
while (bytesRead > 0) {

dos.write(buffer, 0, bufferSize);
int bytesAvailable = fis.available();
bufferSize = Math.min(bytesAvailable, bufferSize);
bytesRead = fis.read(buffer, 0, bufferSize);
}
dos.flush();
dos.close();
fis.close();
Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 23
Class Project: Mobile Panorama Creation
T. Chu, B. Meng, and Z. Wang, Spring 2010
/>Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 24
Class Project: Mobile Face Recognition
G. Davo. K. Sriadibhatla, and X. Chao, Spring 2010
/>Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 25
Android resources
 Developer website

 Class tutorials
/> Android-related office hours
Wednesday, 3:30pm – 5:00pm, SCIEN Lab
 Reference book
Meier, Professional Android 2 Application Development