Assessing the feasibility of increasing spatial resolution of remotely sensed image using HNN super-resolution mapping combined with a forward model

Bạn đang xem bản rút gọn của tài liệu. Xem và tải ngay bản đầy đủ của tài liệu tại đây (2.9 MB, 12 trang )

(1)<div class='page_container' data-page=1>

VNƯ Jo u rn al o f Science, E a rth Sciences 28 (2012) 264-275

Assessing the feasibility o f increasing spatial resolution o f

remotely sensed image using HNN super-resolution mapping

combined with a forward model

Nguyen Quang Minh*

Faculty o f Surveying and Mapping, Hanoi University o f M ining and Geology

Received 03 September 2012

Revised 24 September 2012; accepted 15 October 2012

A bstract. Spatial resolution o f land covers from remotely sensed images can be increased using
super-resolution m apping techniques for soft-classified land cover proportions. A further
development o f super-resolution mapping techniques is downscaling the original remotely sensed
image usmg super-resolution mapping techniques with a forward model. In this paper, the model
for increasing spatial resolution o f remote sensing multispectral image is tested with real SPO T 5
imagery for an area in Bac Giang Province, Vietnam in order to evaluate the feasibility o f
application o f this model to the real imagery. The soft-classified land cover proportions obtained
using a fuzzy c-means classification are then used as input data for a Hopfield neural netw ork
(HNN) to predict the m ultispecừal images at sub-pixel spatial resolution. Visually, the resulted
image is compared with a SPOT 5 panchromatic image acquired at the same time with the
multispecừal data. The predicted image is apparently sharper than the original coarse spatial
resolution image.

Keywords: Hopfield neural network optimisation, soft classification, image downscaling, forward

model.

1. Introduction m apping such as E lad a n d F e u e r [1], T ipping

and B ishop [2]. A lthough w idely applied in
Spatial resolution o f im age and photos can im age processing, these approaches are hardly

be increased by the super-resolution algorithm s. applicable for super-resolution o f rem otely

In the im age processing context, im age super- sensed m ultispectral (M S) im agery because o f

resolution com m only refers to the process o f the lack o f a sequence o f im ages o f the scene at

using a set o f cross-correlated coarse spatial the same or sim ilar tim es. The only feasible

resolution im ages o f the sam e scene to obtain a application o f the super-resolution approaches

single higher spatial resolution im age. T here are using im age sequences is for hyperspectral

num erous studies on such super-resolution im agery [3]. F or o ther com m on m ultispectral

rem otely sensed im agery, only few m ethods for

* Tel-84-982721243 in cre asin g th e sp a tia l re so lu tio n to sub-pixel
E-mail:

</div>
(2)<div class='page_container' data-page=2>

N.Q. Mirth / V N U Journal of Science, Earth Sciences 28 (2012) 264-275 265

level have been proposed such as a Point

Spread Function-derived convolution filter [4],

segm entation technique [5], and geostatistical

m ethod [6],

Sub-pixel spatial resolution land cover

m aps can be predicted using super-resolution

m apping techniques. The input data for super

resolution m apping are com m only the land

cover proportions estim ated by soft-

classification [7], T here is a list o f super

resolution m apping techniques have been

inừoduced including spatial dependence

m axim isation [8], linear optim isation

techniques [9], H opfield neural netw ork (H NN )

optim isation [10], tw o-point histogram

optim isation [1 1], genetic algorithm s [12] and

feed-forw ard neural netw orks [13], The

supplem entary data are also supplied to H N N to

produce m ore accurate sub-pixel land cover

m aps such as m ultiple sub-pixel shifted im age

[14], fused and panchrom atic (PA N ) im agery

[15,16]. These latter approaches produce a

synthetic M S or PA N im age as an interm ediate

step for super-resolution m apping based on a

forw ard m odel and then these im ages are

com pared w ith the predicted and observed MS

or PAN im ages to produce an accurate sub

pixel image classification.

The creation o f the predicted M S an d then

PA N im age b y a forw ard m odel suggested a

possibility to im plem ent a super-resolution for

the M S image. A m ethod for increasing the

spatial resolution o f the original M S image IS

inừoduced by N guyen Q uang M inh et al [17].

T he new m odel is based on the H NN super

resolution m apping technique from

unsupervised soft-classification com bined w ith

a forw ard m odel using local end-m em ber

spectra [15,16]. T he m ethod is exam ined w ith a

degraded rem ote sensing im age and both visual

and statistical evaluations show n a good result.

H ow ever, there still exist som e concerns about

the feasibility o f the m odel because it is not

tested in a m ore com plicated landscape w ith

different kinds o f land cover features w hich are

varying in sizes and shapes as well as specfral

characteristics. This paper, therefore, is to

im plem ent the test o f the algorithm in a

com plicated landscape.

2. G eneral m odel

T he proposed m odel is an extension o f the

super-resolution m apping approach based on

H N N optim isation. T he prediction o f a MS

im age at the sub-pixel spatial resolution is

based on a forw ard m odel w ith local specfra as

w as used in N guyen et al., 2006 [15], In

addition to the goal functions and the

proportion con sừ ain t o f the H N N for super-

resolution m apping, a reflectance constraint is

used to retain the brighừiess values o f the

</div>
(3)<div class='page_container' data-page=3>

266 N.Q. M in h / V N U Journal of Science, Earth Sciences 28 (2012) 264-275

Spatial SR MS image (2 0 m)

convolution

Figure 1. General model for super-resolution MS imagery prediction.

Synthetic MS 
image (4 0 m )

Figure 1 presents the H N N sub-pixel MS

im age prediction algorithm . T he procedure is as

follow s: From the M S im ages at the original

M S spatial resolution, land cover area

proportion images are predicted using a soft-
classifier. A set o f local end-m em ber specừ a

v a lu e s is calc u la te d b a se d o n th e e stim a te d land

cover proportions and the original M S image.

L and cover proportions are then used to

constrain the H N N for super-resolution

m apping w ith a zoom factor z to produce the

land cover map at the sub-pixel spatial

resolution. From the super-resolution land

cover m ap at the first iteration, an estim ated MS

im age (at the sub-pixel spatial resolution) is

then produced using a forw ard m odel and the

estim ated local end-m em ber spectra. The

estim ated M S image is then convolved spatially

to create a synthetic M S im age at the coarse

spatial resolution o f the original image.

Follow ing a com parison o f the observed and

synthetic M S im ages, an error value is

produced to retain the brightness value o f the

pixels o f the original M S image. The process is

repeated until the energy function o f the HNN

is m inim ised and the synthetic M S im age is

generated.

A dem onsfration o f the algorithm for an

image o f 2x2 pixels can be d escn bed in Figure

2. Firstly, the soft-classification predicts land

cover proportion as in Figure lb from the MS

specừal im age as in Figure la. There are two

land covers in this im age called Class A and

Class B. From the land cover proportions in

Figure lb , the land cover classes at sub-pixel

level are predicted as in Figure Ic w here a pixel

is divided into 4 x 4 sub-pixels and the 2x2

</div>
(4)<div class='page_container' data-page=4>

N.Q. M in h Ị V N U Journal of Science, Earth Sciences 28 (20Ĩ2) 264-275 267

land cover image o f Class A and Class B. The

brightness o f the new 16x16 pixels image is

predicted using end-m em ber spectra (standard

brightness for the C lass A and B in this area o f

the image). For exam ple, the brightness o f the

pure pixel o f Class A is 35 and Class B is 50

and it is possible to produce a new spectra!

image by assigning all the sub-pixels belonging

to Class A the brightness value o f 35 and the

sub-pixels o f C lass B the brighừiess value o f 50

as in Figure Id.

100% land
coverA

50% land
c o v erA
50% land
c o v er B

62.5% land
cover A
37.5% land
cover B
100% land
coverB
(b)

Figure 1. Creation o f !6><16 pixels image from 2x2 pixels image.

2.Ỉ. Soft-classification f o r super-resolution 
m apping o f M S im agery

Soft-classification is an intennediate step in

the sub-pixel M S im age predictio n process. The

prediction o f the M S im age based on super

resolution m apping requires land cover

proportions w hich are obtained from soft-

classi fication as input data. C onventionally,

there m ust be a set o f training data for m ost o f

the soft-classifiers. A ccordingly, it is necessary

to have som e prior infom iation about the

specừal distribution o f land cover classes in the

M S bands, although training data are not

alw ays available for the im age. A nd som etim es,

there is a requirem ent o f increasing the spatial

resolution o f the im age w itho ut concerning the

lan d cover classes in the im age scene. In these

cases, the algorithm can be im plem ented w ith

unsupervised soft-classified land cover

proportions such as fuzzy c-means classifier [18].

Supervised soft-classifiers could also be

used, such as B ayesian, neural netw ork or k 'N N

classifiers. H ow ever, the training data for these

soft-classification techniques should be

obtained from the unsupervised classifications.

In the research im plem ented by N guyen Q uang

M inh et al [17], a test for algorithm w as

im plem ented w ith a set o f degraded MS im age

and soft-classified land cover proportions w as

obtained using k-N N classification using a

fraining data set ex ừ acted from unsupervised

Interactive Self-O rganising D ata (ISO D A TA )

classifier. In this case, training data w ere

clustered in the reference image. In this

experim ent, a fuzzy c-m eans classification is

used to predict land cover proportions o f a real

SPO T im age to produce super-resolved specừ al

image o f different spatial resolutions to evaluate

the algorithm .

2.2. F orw ard m odel a nd end-m em ber spectra

A fter the first iteration o f the H N N

algorithm , once the sub-pixel classification is

obtained, a forw ard m odel is used to p ro duce a

sub-pixel M S im age from the sub-pixel land

cover classes. The brighừiess value (e.g.,

reflectance, radiance, digital num ber) o f a sub

</div>
(5)<div class='page_container' data-page=5>

268 N.Q . M in h / V N U journal o f Science, Earth Sciences 28 (2012) 264-275

+

(

)

w here Ve is the output neuron o f the class e and

Ss e is th e end-m em ber spectra o f the land cover
cla ss e for a spectral band s. A s presented in

N guyen et al., 2006 [15], the end-m em ber

specfra vector Ss (Ss = [5^ o f the

original pixel (x,y) o f the specừ al band A’ can be

e s tim a te d lo c a lly u s in g th e p re d ic te d la n d c o v e r
class proportions and the M S im age at the

original coarse spatial resolution as

S ,= iP ^ W P )~ 'w P ^ R ,, (2)

where p is a m aừix o f land cover proportions wiửi

r ^ x -\) { y - \)

p = PT / T

and w is the m atrix o f w eights w ith

w =

3. E xp erim en t condition

5.7. Data

T he experim ent in N guyen Q uang M inh et

al [17] is conducted in an area having many

large objects w ith linear boundaries. It m ay lead

to a concern that th e algorithm proposed in this

p aper is able to w o rk w ell only w ith some

specific landscapes. Therefore, a second data

set is used for testin g the algorithm in a m ore

com plicated landscape. This im age was

obtained in B ac G iang Province, Vietnam.

T he SPO T 5 im age used in this test was

acquired in A ug ust 2011 w ith the spatial

resolution o f 10m and four spectral bands

(Figure 2). T he test im age is registered to

W G S-1984 U TM m ap projection in Zone 48N

and the location is at 21°17’53.65"N ,
1 0 6 °H 7 .6 4 " E . T he test im age covers 1 square

kilom etre area o f 102x102 pixels. To evaluate

the results o f increasing the spatial resolution

algorithm , a 2.5m spatial resolution

panchrom atic im age acquired at the sam e time

</div>
(6)<div class='page_container' data-page=6>

N.Q. M in h / V N U journal of Science, Earth Sciences 28 (2012) 264-275 269

_(c) „

Figure 2. SPOT 5 image in Bac Giang Province, Viettiam: (a) Band 1, (b) Band 2, (c) Band 3 and (d) Band 4.

3.2. Soft-classification

The land cover proportions are estim ated

from 10m spatial resolution SPO T 5 im age

using fuzzy c-m eans classifiers so it is not

necessary to have prior understanding about

land cover classes in the area. The soft-

classified land cover proportions o f five land

cover classes and six land cover classes are

obtained as in Figure 3c and Figure 3d,

respectively.

</div>
(7)<div class='page_container' data-page=7>

270 N.Q. M in h / V N U Journal o f Science, Earth Sciences 28 (2012) 264-275

4. R esults and discussions

4.1. R esuits

The m eth o d o f increasing the M S image

using H N N an d forw ard m odel w as applied to

super-resolve the 10m SPO T 5 M S im ages to

p red ict M S im age at spatial resolutions o f 5m

(zoom factor o f 2), 3.3m (zoom factor o f 3) and

2.5m (zoom factor o f 4). The predicted soft-

classified proportions w ere used to consừain

the H N N w ith w eighting factors o f kị = 100, k:

=1 0 0, k i = 100 and 100 to predict the sub

pixel land cover and then the M S image. The

false colours com positions using B and 1, Band

2 and B and 4 as R ed, G reen and Blue are

show n in Figure 4.

(e) ^ _ (f)

Figure 4. Super-resolution o f 10m SPOT 5 multispectral image: (a) 5m super-resolved image using 5 land cover
classes, (b) 3.3m super-resolution image using 5 land cover classes, (c) 2.5m super-resolution image using 5 land

</div>
(8)<div class='page_container' data-page=8>

N.Q. M in h / V N l ỉ Journal o f Science, Earth Sciences 28 (2012) 264-275 271

4.2. Evaluation

In the case o f degraded image as in N guyen

Q uang Minh et al. [17], the dow nscaled

imagery was com pared with the reference

m ultispectral image at finer spatial resolution to

obtain visual and qualitative evaluations. In this

experim ent, the finer m ultispectral im agery is

not available for the full assessm ent. Therefore,

the predicted m ultispectral im age at finer

resolution can be only com pared w ith 2.5m

panchrom atic im age for a visual evaluation.

The visual com parison o f the super

resolved image from the real SPOT 5 data with

the panchrom atic im age (Figure 3b) also shows

an im provem ent in sharpness o f the results. The

objects in Figure 4(a-f) are sharpened and look

clearer that o f original im age ( r ig u r e 3a).

A lthough the landscape o f im age area is

com plicated w ith sm all and linear features such

as houses and roads, the im provem ent o f the

algorithm can be seen in the b ou ndaries o f

between the objects. Figure 5 show s the

im provem ent o f the algorithm for increasing the

spatial resolution o f M S im age using H N N with

a forw ard m odel to the original im age. The

boundaries o f ponds in the centre o f the original

M S image (Figure 5a) are blurred and

fragm ented because o f the m ixing o f the land

categories in these boundary pixels. In the

predicted M S im age using HNN and a forw ard

m odel (Figure 5b), these boundaries are clear

and look m ore sim ilar to the real ponds in

panchrom atic im age (Figure 5c).

(a) (b) (c)

Figure 5. Some land cover features in (a) original MS image (false colour composite), (b) increased resolution
MS image to 2.5m spatial resolution from 5 land cover class proportions (false colour composite)

and (c) panchromatic image.

F or sm all objects such as houses and roads,

there are few objects w hich is not clearly seen

in the original im age can be recognised in the

super-resolved image. In Figure 6a (com posite

image using Band 1, Band 2 and Band 4 o f the

original MS im age), the road is difficult to be

recognised because it is fragm ented due to the

m ixed pixels effect. U sing the H NN super

resolution m apping and then the forw ard m odel

as in Figure 6b (com posite im age u sing B and 1,

Band 2 and B and 4 o f the 2.5m spatial

resolution increased im age), it is possible to

recreate the road sim ilar to the shape o f the real

road shown in the panchrom atic im age Figure

6c.

For the sm all features such as a group o f

houses in Figure 6c, the perform ance o f the

</div>
(9)<div class='page_container' data-page=9>

272 N.Q. M inh / V N U Journal of Science, Earth Sciences 28 ( 2 0 W 264-275

However, the new ly proposed algorithm still

show s some im provem ent in defining clear

boundaries o f these features. T his m ay be

because o f the soft-classifier cannot define the

houses as a separate class. T his problem m ay be

resolved by increasing the num ber o f classes for

fuzzy c-m eans classifier or using prior

inform ation on these classes in supervised soft-

classifiers.

Figure 6. Some land cover features such as roads and houses in (a) original image, (b) spatial resolution
increased image (false colour composite) and (c) panchrom atic image.

4.3. D iscussions

The effect o f zoom factor to spatial

resolution increasing algorithm can be seen in

Figure 8. . C om paring the im age created by

H N N using zoom factor o f 2 (Figure 4a), with

the image created w ith zoom factor o f 3 (Figure

4b) and 4 (Figure 4c), it is possible to see that

w hen the zoom factor increases, the boundaries

betw een the features are sm oother. The

boundaries betw een the ponds and the

surrounding features are fragm ented in Figure

8. a and F igure 8. b and look sm oother and

clearer in F igure 8. c.

(a) (b) (c)

Figure 8. Effect o f zoom factor to spatial resolution increasing algorithm: (a) zoom factor of 2, (b) zoom factor
o f 3 and (c) zoom factor o f 4.

In spite o f increasing the spatial resolution

o f the rem otely sensed M S im ages, the

proposed m ethod has a problem w ith pixels that

belong to the sam e class (referred to as pure

pixels in this paper). A lthough the problem can

</div>
(10)<div class='page_container' data-page=10>

N.Q. M in h / V N U Journal of Science, Earth Sciences 28 (2012) 264-275 273

classes so there m ore m ixed pixels or dividing a

class into sub-classes so several pure pixels are

defined as m ixed pixels, there w ill still exist

pure pixels. The effect o f num ber o f land cover

classes can be seen in Figure 9.. The ponds in

Figure 9 .a apparently sm aller than those in

Figure 9.b due to som e “pure pixels” in the

boundaries w ere re-classified as m ixed pixels

w hen the num ber o f classes increased from 5 to

6. These m ixed pixels are then super-resolved

to produce different boundaries betw een the

same features in the tw o im ages.

Figure 9. (a) Result from HNN using 5 land cover classes, and (b) result from HNN using 6 land cover classes.

B ecause the H N N super-resolution m ethod

w orks only on m ixed pixels, w hich are usually

located across the border betw een different

classes, it is suggested that the m ethod is

suitable for the super-resolution o f im ages o f

large objects, for exam ple the agricultural

scenes. In these im ages, spatial variation is

hom ogeneous w ithin the land p arcels and

super-resolution based on the spatial clustering

goal functions o f the H N N can the agricultural

field boundaries or increasing the sharpness o f

linear features such as roads or canals.

As m entioned above, the use o f

unsupervised classification can reduce the

eư o rs in land cover class p roportion prediction.

Furtherm ore, the use o f unsupervised

classification facilitates the autom ation o f the

spatial resolution increasing process because

the class p roportions can b e obtained w ithout

fraining data and w ithout a fraining step.

Through the choice o f the num ber o f classes,

the user can control the effect o f the super

resolution algorithm on the resulting sub-pixel

M S im ages. W hen the num ber o f classes is

changed, the num ber o f m ixed pixels m ay be

changed as a result.

A draw back o f the H N N super-resolution

procedure is the subjective choice o f the

param eters for the goal functions, the

proportion co n sừ ain t and the m ulti-class

constraint [15]. B y em pirical investigation, the

values o f the param eters should retain an equal

effect betw een the con sừ ain ts and the goal

functions in the optim isation process. For

exam ple, the em pirical investigation in this

paper show s th at the values o f these param eters

in this paper w ere sim ilar and around the value

o f 100. T he finding is also obtained from the

</div>
(11)<div class='page_container' data-page=11>

27 4 N.Q. M in h / V N U journal of Science, Earth Sciences 28 (2012) 264-275

5. C onclusions

The approach for increasing the spatial

resolution M S im agery utilised the H N N super

resolution m apping technique com bined w ith a

forw ard m odel is tested w ith 10 SPO T 5

rem otely sensed data. In this research, the soft-

classified land cover proportions were

estim ated using a fuzzy c-m eans classifier. The

feasibility o f the m ethod w as evaluated based

on visual com parison o f the resulted im age w ith

panchrom atic im age acquired at the sam e tim e

w ith original image. T he com parison show ed

that the proposed m ethod can generate MS

im ages w ith m ore detail features. The super

resolved im age w as apparently sharper than the

original coarse spatial resolution im age. In

addition, the evaluation also dem onsừ ated that

w hen the zoom factor increased, the resulting

sub-pixel im ages w ere closer to the reference

image.

R eferen ces

[1] M. Elad and A. Feuer, Super-resolution
reconstruction of image sequences, ĨEEE

Transactions on Pattern Analysis and Machine 
Intelligence, vol. 21, pp. 817-834, 1999.

[2] M. E. Tipping and

c.

M. Bishop, Bayesian
image super-resolution in Advances in Neural
Information Processing Systems, Boston, 2003.
[3] T. Akgun, Altunbasak Y., and R.M. Mersereau,
Super-resolution reconstruction of hyperspectral
images, ỈEEE Transactions on Image 
Processing, vol. 14, pp. 1860-1875,2005.

[4]

c.

Pinilla Ruiz and J. Ariza Popez F., Restoring
SPOT images using PSF-derived deconvolution
filters, International Journal o f Remote Sensing, 
vol. 23, pp. 2379-2391,2002/

[5]

w.

Schneider and J. Steinwendner, Land cover
mapping by interrelated segmentation and
classification of satellite images. International

Archives o f Photogrammetry and Remote 
Sensing, vol. 32, pp. part 7-4-3, 1999.

[6] P. c . Kyriakidis and E. H. Yoo, Gcoaatiitica
prediction and simulation of point values Tom
the areal data, Geographical Analysis^ /o! 37, 
pp. 124-152, 2005.

[7] G. M. Foody, Hard and soft classificatuns by a
neural network with a nonexhaustively defined
set of classes, International Journal o f Remote

Sensing, vol. 23, pp. 3853-3864, 2002.

[8] P. M. Atkinson, Mapping sub-pixel boindarics
from remotely sensed images, in Ịnnoviỉion in

GỈS 4, z. Kemp (Ed.), Ed. London: Tiyicr &

Francis, 1997, pp. 166-180.

[9] J. Verhoeye and R. De Wulf, Lane covcr
mapping at sub-pixel scales using linear
optimisation tcchniqucs, Remote Serving o f

Environment, pp. 96-104, 2002.

[10] A. J. Tatcm, H. G. Lewis, p. M. Atkins)n, and
M. S. Nixon, Supcr-resoiution land cover
pattern prediction using a Hopfield neural
network, in Uncertainty in Remote Sensng and

GỈSy Foody G. M. and Atkinson p. V., Eds.

London; John Wiley & Sons, 2002, pp. 5Í-76.
[11] P.M. Atkinson, E. Pardo-Iguzquiza, ind M.

Chica-Oỉmo, Downscaling Cokriging forSupcr-
Resolution Mapping of Continua in Remotely
Sensed Images, ĨEEE Transacíioỉs on
Geoscience and Remote Sensing, vol. 46 no. 2,

pp. 573 - 580, 2008.

[12] K. c . Mcrtems, L., Duchcyiic, E. I. Virbekc,
and R. De Wulf. Using gcnctic algoritims in
sub-pixcỉ mapping, International Jouvial o f

Remote Sensing, vol. 24, pp. 4241-4247,1003.

[13] K. c . Mertems, L. Verbeke, T. Westra, ind R.
De Wulf, Sub-pixel mapping and SLO-pixcl
shaq^ening using neural network pridictcd
wavelet coefficients, Remote Sensiĩ^ o f 
Environment, vol. 91, no. 2, pp. 225-236.2004.

[14] Feng Ling, Yun Du, Fci Xiao, Huapíriị Xue,
and Sheng Jun Wu, Super-rcsolution lanc-covcr
mapping using multiple sub-pixel ;hiftcd
rem otely sensed im ages, ỉn tern a tio n a ỉ J)urnaỉ

o f Remote Sensing, vol. 31, no. 19, pp. 5023

5040, 2010.

[15] Q. M. Nguyen, p. M. Atkinson, and H. c 2006
Lewis, Super-resolution mapping using
Hopfleld neural netwok with fused inages,

IEEE Transactions on GeoscAence and ỉemoĩe 
Sensing, vol. 44, no. 3, pp. 736-749, 2006

</div>
(12)<div class='page_container' data-page=12>

N.Q. M in h / V N U journal of Science, Earth Sciences 28 (2012) 264-275 275

Remote Sensing, vol. 32, no. 21, pp. 6149-6176,

2011.

7] Quang Minh NGUYEN, Van Duong DO, P.M.
Atkinson, and H.G.Lewis, Downscaling
Multispectral Imagery Based on the HNN Using
Forward Model, in 7'^‘ FIG Regional 
Conference on Spatial Data Serving People: 
Land Governance and the Environment- 
Building the Capacity, Hanoi, 2009.

[18] J.