Abstract

We propose an approach that utilizes large collectionsof photo streams and blog posts, two of the most preva-lent sources of data on the Web, for joint story-based sum-marization and exploration. Blogs consist of sequences ofimages and associated text; they portray events and expe-riences with concise sentences and representative images.We leverage blogs to help achieve story-based semanticsummarization of collections of photo streams. In the op-posite direction, blog posts can be enhanced with sets ofphoto streams by showing interpolations between consecu-tive images in the blogs. We formulate the problem of jointalignment from blogs to photo streams and photo streamsummarization in a unified latent ranking SVM framework.We alternate between solving the two coupled latent SVMproblems, by first fixing the summarization and solvingfor the alignment from blog images to photo streams andvice versa. On a newly collected large-scale Disneylanddataset of 10K blogs (120K associated images) and 6Kphoto streams (540K images), we demonstrate that blogposts and photo streams are mutually beneficial for sum-marization, exploration, semantic knowledge transfer, andphoto interpolation.