Scene Completion Using Millions of Photographs

Hays and Efros

computational photography scene stitching image search match seams

@article{hays:cacm-2008,
  author={James Hays and Alexei A. Efros},
  title={Scene Completion Using Millions of Photographs},
  journal={Communications of the {ACM}},
  volume={51},
  number={10},
  year={2008}
}

Takes very large repositories of photographs and uses them to fill-in data from photographs

• I.e., replacing people and other objects, corrupted areas, etc

Photos in repository are first processed to produce gist summaries

• Block based descriptors of photo

Input photo gist is then quickly matched against those gists to cull most of the repository

• Comparison requires arbitrary dimensionality as parts of input are missing, limiting clustering techniques that may be used in practice

Resulting photo set is then matched in more detail against boundaries of missing areas in input

Selected photos are merged together by melding along seam lines, deleting and interpolating low energy seams on both the input and contributor to blend them together

Largely very effective results

Well done use study, experimental setup in paper

Has problems w/ scenes including many objects as there is no object recognition, so their boundaries are not always respected

• People cut in half, etc

Interesting notes on applying the same matching technique to other applications, e.g., determining the location of a photo by matching against large database of geo-referenced photos

• Can work well if database is large enough, which may be the case in this day and age