A high dynamic range imaging algorithm and its implementation for a 4 by 1 camera array

Nghiên cứu khoa học công nghệ

A HIGH DYNAMIC RANGE IMAGING ALGORITHM AND ITS
IMPLEMENTATION FOR A 4 BY 1 CAMERA ARRAY
Vu Hong Son*, Pham Ngoc Thang
Abstract: Camera specification becomes smaller and smaller accompanied with
great strides in technology and thinner product demands, which leads to some
challenges and problems. One of those problems is that micro lens captures less
light than normal lens, which makes low quality noise-image. Moreover, current
image sensor cannot preserve whole dynamic range in real world. High Dynamic
Range (HDR) image with multi-exposure images overcomes the problems
mentioned above. Choosing good exposure time is a seldom-discussed but important
issue in HDR imaging technology. This paper proposes a Histogram Based
Exposure Time Selection (HBETS) method to automatically adjust proper exposure
time of each lens for different scenes. It guarantees at least two valid reference
values for HDR image processing. Adopting the proposed weighting function
restrains random distributed noise caused by micro-lens and produces a high
quality HDR image. An integrated tone mapping methodology, which keeps all
details in bright and dark parts when compressing the HDR image to Low Dynamic
Range (LDR) image for being displayed on monitors is also proposed. We first align
these images to the same plane, and then adopt the proposed methods. The result
image has extended the dynamic range, i.e. comprehensive information is provided.
Eventually, we implement the entire system on Adlink MXC-6300 platform that can
reach 10fps to demonstrate the feasibility of the proposed technology.
Keywords: Auto-exposure; HDR image; Tone mapping.

1. INTRODUCTION
In recent years, HDR imaging technology becomes more and more popular. An HDR
imaging system for micro camera array is composed of many stages such as auto-exposure
control, HDR generation, and tone mapping. HDR imaging requires multiple exposed
photographs to reproduce higher quality and clearer images. One kind of these methods is

using bracketing [1]–[5], which captures the different-exposure image sequence by
adjusting Exposure Value (EV). The other kind of methods is brute force, which
photographs lots of different-exposure images with no pixels over-exposed and underexposed. Benjamin Guthier et al. [6] exploited pre-established HDR radiance histogram to
derive the exposure time, which satisfies the user-defined shape of LDR histogram. O.
Gallo et al. [7] also proposed an approach to estimate HDR histogram of the scene, and
selected the appropriate exposure times for LDR images.
HDR image processing technology can mainly be classified into two methods, i.e.
exposure fusion and recovering high dynamic range radiance maps. Both approaches
require multiple exposed photographs to reproduce higher quality and clearer images.
Image fusion technologies [8], [14]–[16] have been developed for several years, which
include depth-of-field extension [10], image enhancement [11], and multi-resolution
image fusion [12]. Fusion of multi-exposure images [15] proposed by Goshtasby is a
famous approach to reproduce high quality image, but it cannot handle the boundary of
objects perfectly. Exposure fusion technology proposed by Mertens et al. [8] generates an
ideal image by preserving the perfect portion of the multiple different exposure images.
Fusion process technique described in [8] inspired by Burt and Adelson [13] transforms
the domain of image, and adopts multiple resolutions generated by pyramidal image
decomposition. Debevec et al. [9] proposed a method, which uses differently exposed

Tạp chí Nghiên cứu KH&CN quân sự, Số Đặc san ACMEC, 07 - 2017

191


Điều khiển – Cơ điện tử - Truyền thông

photographs to recover the camera response function and blends multiple exposed images
into a single high dynamic range radiance map. The final stage of the HDR imaging
system is the tone mapping that is required to compress HDR image to LDR one. The tone
mapping approaches can be classified into two categories, i.e. local tone mapping and

global tone mapping. Fattal et al. [17] proposed a local tone mapping method, called
gradient domain HDR compression. This method is based on the changes of luminance in
high dynamic range image. It uses diffident levels of attenuation to compress high
dynamic range according to the magnitude of the gradient. Reinhard et al. [18] proposed a
global tone mapping method, called linear mapping approach. In this paper, we develop a
high dynamic range imaging algorithm and its implementation for a 4 by 1 camera array
with more implementation details and additional experimental results than our previous
work [20].
The rest of this paper is organized as follows: Section 2 describes the proposed
algorithm that combines a histogram based exposure time selection, new weighting
function and integrated tone mapping. Section 3 presents experimental results and
performance analysis. Finally, the conclusion is given in Section 4.
2. PROPOSED ALGORITHM
In order to achieve a high quality and high dynamic range imaging system, we propose
a system that can deal with higher noises of the low dynamic range images captured by
using micro camera arrays. In addition, by using the proposed algorithm, all details in the
extreme scene can be completely preserved. The design flow of the overall system is
shown in Figure 1.

Figure 1. The design flow of the proposed HDR system.
As shown in Figure 1, the proposed algorithm is composed of many stages for different
purposes. The upper part represents the initialization of system, and the others indicate
multi-exposure high dynamic range imaging generation and tone mapping. In the
histogram based exposure time selection stage, appropriate images are chosen for
generating high quality HDR images. Then, the new weighting function is used in HDR
generation stage. Eventually, those pixel values of high dynamic range image over 255
must be compressed through the tone mapping stage for display. The details of each stage
of the proposed work are presented in the following paragraphs.
A. Image Alignment
An image alignment consists of the mathematical relationships that map pixel

coordinates from source images to target image, is used due to each camera in camera
array has its own viewpoint. A feature-based method is adopted to accomplish image
alignment, which is described in the following. The feature point, which has information

192 V. H. Son, P. N. Thang, “A high dynamic range imaging algorithm… 4 by 1 camera array.”


Nghiên cứu khoa học công nghệ

about the position and its descriptor, is extracted from images. We can recognize the
similarity among these features in different images by the feature descriptors. In order to
calibrate images to the same coordinate system, the homography matrix, which is a three
by three coordinate transformation matrix, is adopted. Only eight elements are needed in
light of a two-dimensional image, as shown in (1). The relationship between the original
coordinate and the objective coordinate is represented by (2) and (3).

 x'  h11 h12
 y'  h
   21 h 22
 z'  h 31 h 32
x' 

h13   x 
h 23   y 
1   1 

h11 x  h12 y  h13 ,
h x  h22 y  h23
y '  21
h31 x  h32 y  1

h31 x  h32 y  1

(1)

(2)

x'
, Y  y' , Z  1
(3)
z'
z'
B. Histogram Based Exposure Time Selection (HBETS)
We propose a method called Histogram Based Exposure Time Selection (HBETS) to
choose suitable source images to generate the HDR images. The flow of HBETS is shown
in Figure 2 and Figure 3 in [20]. Figure 3 shows the generated HDR image by using the
source images captured by the proposed HBETS method. Comparing Figure 3 with Figure
2, we can see that the red box region in Figure 3 has higher performance than that in
Figure 2 after adopting HBETS to guarantee two effective pixel values, one of which is a
redundant pixel value to reckon as a remedy to suppress the noise effect, and construct a
higher quality HDR image.
X

Figure 2. The HDR result image
with distortion.

Figure 3. The HDR resulting image
generated by using the source images
chosen by the proposed HBETS.

C. HDR Generation for Image Continuity

The camera response function curve g(x) has intense slope near the maximum and
minimum pixel values, so g(x) is considered to be less smooth and more inaccurate near
these two sides. To overcome this, Debevec et al. [10] proposed the triangle weighting
function that highlighted the importance in the middle of pixel values. In the case of
different exposure, short exposure images generally have larger noise than long exposure
images. The micro camera array composed of small lens receives less amount of light than
common cameras. The ISO value of micro camera should be increased for enhancement.
However, noise is also amplified. After applying the process of the Debevec’s weighting
function, the noise dominates the pixel value. Hence, this resulting pixel value is not the
realistic luminance. In order to overcome the drawbacks mentioned above, we propose a
new weighting function to enhance HDR image quality as shown in [20]. This weighting

Tạp chí Nghiên cứu KH&CN quân sự, Số Đặc san ACMEC, 07 - 2017

193


Điều khiển – Cơ điện tử - Truyền thông

function can suppress more noises than Debevec’s weighting function does in the result
image as shown in Figure 4, where Figure 4(a) uses the weighting function proposed by
Debevec, and Figure 4(b) uses the proposed weighting function.

(b)
(a)
Figure 4. Comparison of adopting two different weighting functions. (a) HDR image using
Debevec’s weighting function, (b) HDR image using the proposed weighting function.
Moreover, we utilize Gaussian filter and Laplacian filter to denoise and enhance image
for further improving the image quality. The method of applying Gaussian filter is to
obtain a smoother image through a convolution of image with a normal Gaussian

distribution model. A three by three Gaussian kernel is used to achieve this target as
shown in Table I(a). Then, we adopt Laplacian filter to further enhance the image quality
by strengthening the region changing rapidly such as edges and making image clearer, as
shown in Table I(b).
Table I. Two Different Enhancement Kernels Adopted in The Proposed Algorithm.
(a) Gaussian Kernel. (b) Laplacian Kernel
(a) Gaussian kernel
(b) Laplacian kernel
0.0751
0.1238
0.0751
0
-1
0
0.1238
0.2042
0.1238
-1
5
-1
0.0751
0.1238
0.0751
0
-1
0
In addition, we consider that the pixel having large value in short exposure than the one
in corresponding long exposure has a higher chance of noise by reason of noise
characteristics. Consequently, there is a correction on the problematic pixel value. The
average of eight pixels which are the neighborhood of the problematic pixel in short

exposure image is calculated, and used to replace the problematic pixel. As shown in
Figure 5, the noise (i.e. red dot in the Figure 5(a)) is eliminated by the proposed method of
pixel correction.

(a) Before denoising
(b) After denoising
Figure 5. Denoised image by applying the proposed method.
D. Integrated Tone Mapping
There are two major kinds of tone mapping techniques, i.e. global tome mapping and
local tone mapping. The global tone mapping technique such as photographic compression
uses a fixed formula for each pixel in compressing HDR image into LDR image. This
approach is relatively fast, but it loses details in high luminance regions. On the other

194 V. H. Son, P. N. Thang, “A high dynamic range imaging algorithm… 4 by 1 camera array.”


Nghiên cứu khoa học công nghệ

hand, the local tone mapping technique such as gradient domain compression refers to
nearby pixel values before compression. As a result, all details can be retained, but it takes
a lot of computation time. Since both kinds of tone mapping methods have pros and cons,
this motivate us to propose a new tone mapping approach that can preserve details in
bright regions accompanying with lower computation time.
Figure 6(a) to Figure 6(d) show four input images captured respectively with exposure
time 0.33 ms, 2.10 ms, 10.49ms, and 66.23 ms, selected by the proposed HBETS method.
Meanwhile, Figure 6(e) demonstrates photographic tone mapping, and Figure 6(f) to
Figure 6(h) are images used in the proposed algorithm with the scaling parameters 0.8, 0.5,
and 0.2, respectively. Photographic tone mapping lost details in bright regions (e.g. the
shape of lamp and the word near the lamp). In the proposed tone mapping method, large
scaling parameter leads to discontinuity and small scaling parameter causes the unclear

details. Hence, some corrections are put on (4).

(a)

(b)

(c)

(d)

(e)

(i)
(g)
(h)
(f)
Figure 6. (a) to (d) show four input images captured respectively with exposure time
0.33 ms, 2.10 ms, 10.49ms, and 66.23 ms, selected by the proposed HBETS method. (e)
demonstrates the result by using photographic tone mapping. (f) to (h) are images used
in the proposed algorithm with the scaling parameters 0.8, 0.5, and 0.2, respectively. (i)
is the final result of the proposed tone mapping.
Our idea is to blend two lower exposure source images first, which preserves details and
also adjusts the brightness for image continuity, and then use the same equation to gain the
result image, as shown in (5).
I result (x, y)  (1 -  ) * I photograph (x, y)   * I ' source (x, y)

(4)

I ' source (x, y)  (1   ) * I source1 ( x, y )   * I source 2 ( x, y )

(5)

where α is shown in (6), and β is also a Gaussian-like function as illustrated in (7).


   * exp   4

I

photograph ic






  u  v * exp  4


( x , y )  255

255  I threshold 

2



2







I source 2 ( x, y )  2552 
255 2




Tạp chí Nghiên cứu KH&CN quân sự, Số Đặc san ACMEC, 07 - 2017

(6)

(7)

195


Điều khiển – Cơ điện tử - Truyền thông

where u is a constant value which dominates the weighting to the two image’s pixel
values, v is a scaling parameter and Isource2 is the second low exposure image’s luminance.
By using the proposed tone mapping method as shown in (5), the result image, which
retains details in the brightness regions and keeps color continuity, can be acquired as
shown in Figure 6(i). In our experiment, the setting of γ as 0.5, u as 0.1, and v as 0.4
obtains a better result. Comparing Figure 6(i) with Figure 6(e) to Figure 6(h), the word and
the texture of the lamp in the scene by using the proposed algorithm can be preserved
comprehensively.
3. EXPERIMENTAL RESULTS AND PERFORMANCE ANALYSIS

We have implemented the proposed 4-CAM HDR system on Adlink MXC-6300
platform that can reach VGA video @10 fps. The camera array consists of four Logitech
webcams is shown in Figure 7.

Figure 7. Four Logitech webcams
to form a 4x1 camera array.
(a)

(b)

Table II. Computation Time Analysis of The
Proposed Algorithm.
Execution time (ms)
Functions
Before
After
optimization optimization
Alignment
31.01
19.10
HDR generation
61.48
28.51
Tone Mapping
44.99
39.27
HBETS and data
23.03
14.62
type conversions

Total
181.47
101.50
(c)

(d)

(e)
(f)
Figure 8. Experimental results of the proposed HDR algorithm preserving more details in
dark regions. (a) to (d) are input images, (e) is the result image of easyHDR, and (f) is the
result image of the proposed method.
Table II shows a detailed computational complexity analysis of code optimization for
the proposed algorithm. From Table II, we can see that the proposed design achieves 1.79
times faster processing speed after code optimization. Besides, Table II also shows that
tone mapping and HDR generation are the two most computation intensive computations
which occupy near 70% computation time of the proposed HDR algorithm. To further

196 V. H. Son, P. N. Thang, “A high dynamic range imaging algorithm… 4 by 1 camera array.”


Nghiên cứu khoa học công nghệ

illustrate the validity of the proposed algorithm, we show the high dynamic range imaging
results processed by the proposed algorithm, and compare those to a well-known
commercial software tool called easyHDR [19] in the following.
(a)
(b)
(c)
(d)

(f)
(e)
Figure 9. Experimental results of the proposed HDR algorithm achieving more saturated
scene. (a) to (d) are input images, (e) is the result image of easyHDR, and (f) is the result
image of the proposed method.
The exposure times of input images are chosen by the proposed HBETS approach with
a four by 1 camera array, which means four source images are used to generate an HDR
image. Figure 8 demonstrates the proposed HDR result image, where Figure 8(a) to Figure
8(d) are input images; Figure 8(e) is the result image of easyHDR; and Figure 8(f) is the
result image of the proposed algorithm. Experimental results shown in Figure 8 indicate
that the proposed algorithm preserves more details in dark region than easyHDR. The
texture and the words inside the box are able to be viewed clearly, and vibrant scene is
given in the result image by applying the proposed method. Comparing easyHDR result
image shown in Figure 9(e) and the result image adopting the proposed method shown in
Figure 9(f), we can observe that using the proposed algorithm generates more saturated
image in HDR.
4. CONCLUSION
The proposed HDR system has been implemented on a four-by-one micro-camera array
so that the four source images can be used to generate HDR image. The proposed
histogram-based adaptive exposure time selection, which conquers the problem of extreme
environment that auto-exposure system cannot afford, not only enhances image contrast
but also keeps the image details in light-regions, and it also reduces the noise effect. HDR
system in camera array records comprehensive details at extremely low-light scene, which
could be applied on car event recorders, surveillance systems, HDR movies, smart phones,
and etc. This work solves severe conditions such as night vision, and provides better
visibility of the video at night. Moreover, for entertainment applications, movie filmed
with this technique will produce realistic videos for human visual perception. Higher
quality in back-lighted scene can also be achieved by the proposed design. The proposed
HDR system makes cameras achieving high dynamic range close to that of human eyes.

REFERENCES
[1]. Reinhard, E., Pouli, T., and Cunningham, D., “Image Statistics: From Data
Collection to Applications in Graphics,” Proc. ACM SIGGRAPH, no. 6, 2010.

Tạp chí Nghiên cứu KH&CN quân sự, Số Đặc san ACMEC, 07 - 2017

197


Điều khiển – Cơ điện tử - Truyền thông

[2]. M. A. Robertson, S. Borman, and R. L. Stevenson, “Estimation-theoretic approach
to dynamic range enhancement using multiple exposures,” J. Electronic Imaging, pp.
219–285, Apr. 2003.
[3]. R.C. Bilcu,A. Burian, A. Knuutila, M. Vehvilainen, “High Dynamic Range Imaging
on Mobile Devices,” Proc. IEEE International Conference on Electronics, Circuits
and Systems, pp. 1312–1315, 2008.
[4]. M. A. Robertson, S. Borman, and R. L. Stevenson, “Dynamic Range Improvement
through Multiple Exposures,” Proc. International Conference Image Processing, pp.
159–163, 1999.
[5]. W. C. Kao, C. C. Hsu, C. C. Kao, and S. H. Chen, “Adaptive Exposure Control and
Real-time Image Fusion for Surveillance Systems,” Proc. IEEE International
Symposium on Circuits and Systems, pp. 935–938, 2006.
[6]. B. Guthier, K. Ho, S. Kopf, and W. Effelsberg, “Determining exposure values from
HDR histograms for smartphone photography,” Proc. International Conference on
Multimedia, pp. 425–426, 2013.
[7]. Gallo O., Tico, M., Manduchi, R., Gelfand, N., and Pulli, K., “Metering for Exposure
Stacks,” Proc. Computer Graphics Forum, pp. 479–488, 2012.
[8]. T. Mertens, J. Kautz, and F. V. Reeth, “Exposure fusion,” Proc. Pacific Graphics, pp.
382–390, 2007.

[9]. P. Debevec and J. Malik, “Recovering High Dynamic Range Radiance Maps from
Photographs,” Proc. ACM SIGGRAPH, pp. 369–378, 1997.
[10]. E. Eisemann and F. Durand, “Flash photography enhancement via intrinsic
relighting,” Proc. ACM SIGGRAPH, pp. 673–678, 2004.
[11]. R. Raskar, A. Ilie and J. Yu, “Image fusion for context enhancement and video
surrealism,” Proc. International Symposium Non-Photorealistic Animation and
Rendering, pp. 85–94, 2004.
[12]. T. Stathaki, “Image Fusion: Algorithms and Applications,” Academic 2008.
[13]. P. J. Burt and E. H, Adelson, “The Laplacian Pyramid as a Compact Image Code,”
IEEE Trans. Commun., vol. Com-31, no. 4, pp. 532–540, 1983.
[14]. M. Tico, N. Gelfand, and K. Pulli, “Motion blur free exposure fusion,” Proc. IEEE
International Conference on Image Processing, pp. 26–29, 2010.
[15]. A. A. Goshtasby, “Fusion of multi-exposure images,” Image and Vision Computing,
vol. 23, no. 6, pp. 611–618, Jun. 2005.
[16]. Kong, J. et al., “A Novel Fusion Approach of Multi-exposure Image,” Proc.
International. Conference on “Computer as a Tool”, pp. 163–169, 2007.
[17]. R. Fattal, D. Lischinski and M. Werman, “Gradient domain high dynamic range
compression,” Proc. ACM SIGGRAPH, pp. 249–256, 2002.
[18]. E. Reinhard, M. Stark, P. Shirley and J. Ferwerda, “Photographic tone reproduction
for digital images,” Proc. ACM SIGGRAPH, pp. 267–276, 2002.
[19]. easyHDR,
“High
Dynamic
Range
photography
made
easy”,
/>[20]. P. H. Huang, Y. H. Maio, and J. I. Guo, “High Dynamic Range Imaging Technology
for Micro Camera Array,” Proc. 2014 Asia-Pacific Signal and Information
Processing Association Annual Summit and Conference, pp.1–4, 2014.

198 V. H. Son, P. N. Thang, “A high dynamic range imaging algorithm… 4 by 1 camera array.”


Nghiên cứu khoa học công nghệ

TÓM TẮT
MỘT THUẬT TOÁN TẠO ẢNH DẢI ĐỘNG CAO VÀ SỰ THỰC HIỆN CỦA NÓ
CHO MỘT MẢNG CAMERA 4x1
Đặc điểm camera trở lên nhỏ hơn kèm theo những tiến bộ vượt bậc trong công
nghệ và nhu cầu sản phẩm mỏng hơn dẫn tới một số thách thức và vấn đề. Một
trong những vấn đề đó là thấu kính nhỏ thu ánh sáng yếu hơn so với thấu kính
thông thường, cái tạo ra ảnh - nhiễu chất lượng thấp. Ngoài ra, cảm biến ảnh hiện
tại không thể bảo toàn toàn bộ dải động ở thế giới thực. Ảnh HDR với nhiều hình
ảnh phơi sáng có khả năng khắc phục những vấn đề đã được đề cập ở trên. Lựa
chọn một thời gian phơi sáng tốt là một vấn đề ít được thảo luận nhưng lại là vấn
đề quan trọng ở kỹ thuật tạo ảnh HDR. Bài báo này đề xuất một phương pháp lựa
chọn thời gian phơi sáng căn cứ vào biểu đồ để tự động điều chỉnh thời gian phơi
sáng phù hợp mỗi thấu kính cho các ngữ cảnh khác nhau. Nó đảm bảo ít nhất hai
giá trị tham chiếu hợp lệ cho xử lý ảnh HDR. Thông qua hàm trọng số đã đề xuất để
hạn chế nhiễu phân phối ngẫu nhiên được sinh ra bởi thấu kính nhỏ và tạo ra một
ảnh HDR chất lượng cao. Một phương pháp được tích hợp ánh xạ sắc để giữ tất cả
các chi tiết ở các phần tối và sáng khi nén ảnh HDR cho ảnh dải động thấp để hiển
thị trên các màn hình là cũng được đề xuất. Đầu tiên chúng tôi sắp các ảnh này trên
cùng một mặt phẳng, rồi sau đó thông qua các phương pháp đã đề xuất. Ảnh kết
quả đã được mở rộng dải động, tức là thông tin toàn diện là được cung cấp. Cuối
cùng chúng tôi thực hiện toàn bộ hệ thống trên nền Adlink MXC-6300, cái có thể
đạt 10 khung hình trên giây để chứng minh cho tính khả thi của kỹ thuật đã đề xuất.
Từ khóa: Phơi sáng tự động; Hình ảnh HDR; Ánh xạ sắc.

Received date, 02nd May, 2017
Revised manuscript, 10th June, 2017
Published, 20th July, 2017
Author affiliations:
Hung Yen University of Technology and Education.
*Corresponding author:

Tạp chí Nghiên cứu KH&CN quân sự, Số Đặc san ACMEC, 07 - 2017

199