Ch07 seam carving

Chapter 7 – Seam Carving

7-1

Chapter 7
Seam Carving

Michael Rubinstein, MIT
Shai Avidan, Mitsubishi Electric Research Labs
Ariel Shamir, The Interdisciplinary Center & MERL
Department of Mechatronics


Chapter 7 – Seam Carving

7-2

Contents
Image resizing
• Seam Carving algorithm

Department of Mechatronics


Chapter 7 – Seam Carving

7-3

Display Devices

Department of Mechatronics

Chapter 7 – Seam Carving

7-4

Content Retargeting

PC
Department of Mechatronics

iPhone


Chapter 7 – Seam Carving

7-5

Page Layout

Department of Mechatronics


Chapter 7 – Seam Carving

7-6

Simple Media Retargeting Operators

Letterboxing

Scaling

?
Department of Mechatronics

?


Chapter 7 – Seam Carving

7-7

Content-aware Retargeting Operators
“Important”
content

Contentaware

Contentoblivious

Department of Mechatronics


Chapter 7 – Seam Carving

7-8

Content-aware Retargeting

Scale

Input

Department of Mechatronics

“less-Important”
content

Crop

Content-aware


Chapter 7 – Seam Carving

7-9

Image Retargeting
• Problem statement:
• Input Image I with size n × m, and new size n’ × m’
• Output Image 𝑰𝑰𝑰 of size n’ × m’ which will be “good
representative” of the original image I
• To date, no agreed definition, or measure, as to what a
good representative is in this context!

Department of Mechatronics


Chapter 7 – Seam Carving

7-10

Image/Video Retargeting
• In large, we would expect:
1. Adhere to the geometric constraints (display/aspect ratio)
2. Preserve the important content and structures
3. Limit artifacts
4. Perhaps a new representation that will support different
sizes?
• Very ill-posed!
o How do we define important? Is there a universal ground
truth?
o Would different viewers think the same about a retargeted
image?
o What about artistic impression in the original content?
Department of Mechatronics


Chapter 7 – Seam Carving

7-11

Importance (Saliency) Measures
• A function S: p  [0,1]

Wang et al. 2008

• More sophisticated: attention models, eye tracking (gazing
studies), face detectors, …

Judd et al. ICCV09 Learning to predict where people look
Department of Mechatronics


Chapter 7 – Seam Carving

7-12

General Retargeting Framework
1. Define an energy function E(I)
(interest, importance, saliency)

2. Use some operator(s)
to change the image I

Recompose

Crop

Setlur et al.
[2005]

Santella
et al.
[2005]
Gal et al.
[2006]

Department of Mechatronics

Warp


Chapter 7 – Seam Carving

7-13

Previous Retargeting Approaches
• Optimal Cropping Window

• For videos: “Pan and scan”
Still done manually in the movie industry

Liu and Gleicher, Video Retargeting: Automating Pan and Scan (2006)
Department of Mechatronics


Chapter 7 – Seam Carving

7-14

Cropping

Department of Mechatronics


Chapter 7 Seam Carving

7-15

Seam Carving
ã Assume m ì n m ì n, n < n
ã Basic Idea: remove unimportant pixels from the image
o Unimportant = pixels with less “energy”

• Intuition for gradient-based energy:
o Preserve strong contours
o Human vision more sensitive to edges – so try remove content
from smoother areas
o Simple, enough for producing some nice results
o See their paper for more measures they have used

Department of Mechatronics


Chapter 7 – Seam Carving

7-16

Pixel Removal

Optimal
Department of Mechatronics

Least-energy pixels Least-energy columns
(per row)


Chapter 7 – Seam Carving

7-17

A Seam
• A connected path of pixels from top to bottom (or left to right).
Exactly one in each row

Department of Mechatronics


Chapter 7 – Seam Carving

7-18

Seams in Images








Efros & Freeman [2001] – Texture synthesis
Kwatra et al. [2003] – Image and video synthesis
Agarwala et al. [2004] – Digital Photomontage
Mostly
used Image
for composition
of
Perez et al. [2003]

– Poisson
Editing
Jia et al. [2006]
– Drag-and-Drop
Pasting
two
(or more) images
or patches…
Rother et al. [2006] – Auto-Collage
Wang and Cohen [2006] – simultaneous matting and
compositing

Agarwala et al., Interactive Digital Photomontage
Department of Mechatronics


Chapter 7 – Seam Carving

7-19

Finding the Seam?

Department of Mechatronics


Chapter 7 – Seam Carving

7-20

The Optimal Seam

E (I ) =| ∂ I | + | ∂ I |
∂y
∂x
Department of Mechatronics

s * = arg min E ( s )
S


Chapter 7 – Seam Carving

7-21

The Optimal Seam
• The recursion relation

M(i, j )= E (i, j ) + min(M(i − 1, j − 1), M(i − 1, j ), M(i − 1, j + 1) )
• Can be solved efficiently using dynamic programming
in O ( s ⋅ n ⋅ m)
(s=3 in the original algorithm)

Department of Mechatronics


Chapter 7 – Seam Carving

7-22

Dynamic Programming

• Invariant property:
o M(i,j) = minimal cost of a seam going through (i,j) (satisfying
the seam properties)

Department of Mechatronics

5

8

12

3

9

2

3

9

7

3

4

2

4

5

7

8


Chapter 7 – Seam Carving

7-23

Dynamic Programming
M(i, j )= E (i, j ) + min(M(i − 1, j − 1), M(i − 1, j ), M(i − 1, j + 1) )

Department of Mechatronics

5

8

12

3

9

2+5

3

9

7

3

4

2

4

5

7

8


Chapter 7 – Seam Carving

7-24

Dynamic Programming
M(i, j )= E (i, j ) + min(M(i − 1, j − 1), M(i − 1, j ), M(i − 1, j + 1) )

Department of Mechatronics

5

8

12

3

9

7

3+3

9

7

3

4

2

4

5

7

8


Chapter 7 – Seam Carving

7-25

Dynamic Programming
M(i, j )= E (i, j ) + min(M(i − 1, j − 1), M(i − 1, j ), M(i − 1, j + 1) )

Department of Mechatronics

5

8

12

3

9

7

6

12

14

9

10

8

14

14

15 8+8