<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Storyo - Medium]]></title>
        <description><![CDATA[AI-powered photo selection and storytelling for automatic photo book creation - Medium]]></description>
        <link>https://medium.com/storyo?source=rss----dfcba6f95ead---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Storyo - Medium</title>
            <link>https://medium.com/storyo?source=rss----dfcba6f95ead---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:28:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/storyo" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Automatic Curation of Photo Memories through Unsupervised Learning]]></title>
            <link>https://medium.com/storyo/automatic-curation-of-photo-memories-through-unsupervised-learning-53cc985aa7ac?source=rss----dfcba6f95ead---4</link>
            <guid isPermaLink="false">https://medium.com/p/53cc985aa7ac</guid>
            <category><![CDATA[curation]]></category>
            <category><![CDATA[photography]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[memories]]></category>
            <category><![CDATA[storytelling]]></category>
            <dc:creator><![CDATA[Manuel Costa]]></dc:creator>
            <pubDate>Fri, 01 Mar 2019 15:42:05 GMT</pubDate>
            <atom:updated>2019-03-01T15:42:05.684Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1EiLsC7ftxhqeJabP5_CNA.png" /></figure><h3>Introduction</h3><p>Automatic discover of stories based on a device camera roll or a cloud backup system became an important strategy to increase return rate of apps dealing with photos and photo memories in particular. The discovery of these stories can follow different strategies and probably the most common way to do it today is through deep learning to classify photos based on their content or recaps like “best of 2018”. You can then create stories for photos classified as beach, sunset, flowers, with an endless set of categories or simply select the “best ones” according to some criteria and constraint it to a specific date span.</p><p>Another possible approach to search for stories is by trying to get closer to how our brain tends to organize memories. Stories are strongly related to space and time where the event occurred and temporal scale of our memory tends to increase in special occurrences like travel scenarios. On the other end, home events or moments happening quite frequently like going to an office of one’s company in a different city or even in a different country tend to be all messed up and are remembered as less relevant moments.</p><p>An additional aspect of automatic stories that is worth emphasizing is that they can be used to communicate naturally with users even when the app is not explicitly running. For example, if a relevant story is found, a user can be send a push notification and, thus, promote return rate and increase conversion.</p><p>When searching for stories on a mobile device one should also look at the degrees of freedom to execute them. Deep learning execution to be viable has to run on GPU or dedicated HW and it should be noted that iOS does not allow access to the GPU while the app is in background to avoid a possible impact on an app running in foreground and to avoid unexpected battery drain.</p><p>In the <a href="http://storyoapp.com/"><strong>Storyo app</strong></a> and now in <a href="http://www.storyoapp.com/sdk/"><strong>Storyo SDK</strong></a> we have been always focused on giving the best experience for final users by designing a story discovery strategy that best matches how people tend to remember moments and having in mind how the mobile operating systems work. In this article we will disclose the work that we have been doing to discover stories tuned based on the statistics that we collect and feedback received.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Ihjc5VC445jkO2c2Tunv6g.png" /></figure><p>This article will start by describing the strategy used to create stories optimized for a mobile environment followed by the steps used in the recipe to produce a set of Smart Albums that can be transformed into stories. We then disclose some statistics collected through Storyo app 2.x for iOS and 1.x for Android and include some final considerations.</p><h3>Finding Stories</h3><p>At Storyo we have been always concerned with speed and designed our algorithms to provide a seamless experience for our users. Because they can represent an important entry point for customers and also an important path to increase loyalty, this search has been optimized throughout the years.</p><p>Before going into detail, we have to remember that photos include meta-information time and space if the user configured these data points to be active.</p><p>As we presented in <a href="https://medium.com/storyo/automatic-photo-curation-through-storytelling-and-deep-learning-fcb88a0ea685"><strong>“Automatic Photo Curation through Storytelling and Deep Learning</strong></a><strong>”</strong>, this information can be used to create the skeleton of a narrative for a preset of photos to be used in a curation flow of photos. Now, remember the way we organize memories in our brain is strongly related to where and when a moment happened and scale of memory changes if an event is part of daily life or if it is something happening in a new or unusual geography.</p><p>Again space and time can play an important role not only for narrative foundation but also in the curation of memories. We just need a way to organize data to follow people’s brain that is computationally efficient and is designed for the degrees of freedom available in a mobile world.</p><p>It is beneficial to emphasize once more that a memory can be a key gateway for your marketing communications. Execution in background can be quite interesting to identify a new set of photos, upload this fact to your server and later on push notify users promoting the visualization of a memory that can be converted into a sale.</p><p>It is relevant to note that iOS is the most restrictive operating system and you can request your app to run once in a while in background if you follow Apple’s rules. Remember again, though, that in iOS it is not possible to use GPU and / or specialized HW in background and that content analysis through deep learning needs it to do things in a reasonable amount of time.</p><p>For all this, we created a curation flow based in two fundamental and independent steps. First, we organize memories just based on meta-information space and time by using only CPU and later, when the app is in foreground, we leverage all of the HW power to create a story and do the curation flow as explained in <a href="https://medium.com/storyo/automatic-photo-curation-through-storytelling-and-deep-learning-fcb88a0ea685"><strong>“Automatic Photo Curation through Storytelling and Deep Learning”</strong></a> and, eventually, enriched with contextual data as illustrated in <a href="https://medium.com/storyo/data-driven-storytelling-based-on-photo-metadata-116896a3cf8c"><strong>“Data-driven storytelling based on photo metadata”</strong></a>.</p><p>Now that we do have a strategy identified, we will explain our algorithmic cooking recipe that is designed to run 100% on the device and we will start by defining the concept of “frequent location” which plays a key rule in the memory curation flow.</p><h3>Frequent location</h3><h4>Overview</h4><p>Because they will be used as key reference regions, the concept of frequent location is crucial in our strategy to discover candidates for stories. It is important to note that a Frequent Location does not have to happen in user’s home. It can happen abroad if your user has offices in more than one country, for example, it can happen in a weekend house or perhaps it is simply a place that the user likes to go and take pictures frequently.</p><p>The major issue about these regions is that our brain starts messing up moments in these areas with a lot of occurrences and the way people overcome it is by reducing temporal memory and confine it to short moments. On the other end, if you are not in a frequent location like it happens while traveling on vacations to a new country during several days, you tend to remember it as a all.</p><h4>Formal Definition</h4><p>Now let’s formalize what a Frequent Location is in <a href="http://storyoapp.com/"><strong>Storyo app</strong></a> and <a href="http://www.storyoapp.com/sdk/"><strong>Storyo</strong> <strong>SDK</strong></a>. A Frequent Location is</p><ul><li>a <strong>Space Cluster</strong> confined to a relatively small area computed based on the entire camera roll set;</li><li>with a <strong>Temporal Expression</strong> measured in different number of months above some threshold with at least one picture taken.</li></ul><p>A few clustering methods can be used to segment photos in space clusters. Several methods use random factors in their flow, which can end up having different sets of clusters if executed twice on the same set of photos. To avoid it, we use same method applied to the narrative creation that produces always the same result (Mean Shift).</p><p>Because you can have very different scales like pictures in US, Europe and Asia and also neighborhood scale of user’s home, we do the clustering based on the two space variables hierarchically until the bounding box of a cluster or a sub-cluster is bellow some threshold. In the end we use leaf nodes as candidates to be a Frequent Location.</p><p>After finishing this clustering process, we have to define if candidate clusters respect time expression rules. In fact we do have two rules for T<strong>emporal Expression</strong>:</p><ul><li><strong>Time representation</strong> — It must have at least x different months (ex: jan 2017, Mar 2017, Apr 2107, Jun 2017 and Jan 2018)</li><li><strong>Cyclic behaviour prevention</strong> — It must have at least z different months independently of the year (ex: Jan, Mar, Apr, Jun)</li></ul><p>Of course that these rules can be always fine tuned or changed. The key point is that it uses a component to give a notion of time representation and a second one to prevent classifying travel destinations showing a cyclic behaviour like someone traveling on vacations to the same snowboard resort.</p><h3>Finding Countries</h3><p>To have a notion of where home country is to help us understanding our user and to help us predicting what is “abroad”, we will now find the country for each photo.</p><p>The most common way in mobile world to associate a geographic coordinate to a geographic label is by using reverse geolocation services offered by Google or Apple, respectively, or by consuming some third party service online. Independently of the service used, it implies that you will be dependent on a network access to do the task and that is something we want to avoid.</p><p>To steer clear of this network shortcoming, we do store the boundaries of all countries in the app bundle /<strong> </strong><a href="http://www.storyoapp.com/sdk/"><strong>Storyo SDK</strong></a> and just check if a geographic point is inside one of the polygons that compose a country (i.e., a polygon for the continental area, several for islands and, eventually, an enclave). Having these coordinates with a reasonable resolution can be carried out with approximately 2 MB of compressed data, which seems quite balanced for the advantages it brings to discover travel memories as it will be soon explained.</p><h3>Home information</h3><p>After computing Frequent Locations and associating a country to each photo, one can infer home information. Although it can be inaccurate, we can assume that user’s home is located in the Frequent Location with more expressive time representation.</p><p>Home country is also important as it will be used to infer travels abroad. We can assume again that home country is the one with more number of different months independently of the number of photos taken.</p><h3>Travel</h3><p>By their nature often associated with unusual and important events of user’s life, travel scenarios deserve a special attention as memories. Travel can happen in different scenarios and our search will use the following references:</p><ol><li><strong>Travel, abroad</strong> — it can happen if the user lives in New York and travels to Europe where one or more countries are visited during that journey;</li><li><strong>Travel, domestic</strong> — living in New York and visit Yosemite park;</li><li><strong>Travel, domestic and abroad</strong> — living in New York and visit Montreal in Canada by car.</li></ol><p>It is also important to emphasise that travel discovery uses both geographic and time references to split data and isolate travels. We assume that if there are pictures taken in a frequent place, pictures taken before and after do not belong to the same travel and we also assume that if a certain date span occurred between consecutive pictures, travel story is not the same (Fig. 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PszI0S5D-M19vydR" /><figcaption><strong>Figure 1 </strong>— Illustrative example of the criteria used to split photos to isolate travel candidates (frequent place and date span)</figcaption></figure><h4>Travel Abroad — based on inferred Home Country</h4><p>To find travels happening abroad we use home country as a first geographic reference and also Frequent Locations to prevent finding travels that are part of user’s daylife like having an office in a different country. The key steps to find this type of travels will now be specified.</p><p><strong>Split photos by home country</strong></p><p>1. Remove photos with geographic data<strong> </strong>— Filter out photos with no geographic data;</p><p>2. Find first travel candidates by splitting photos based on home country — Organize photos chronologically, go from the oldest to the newest and every time a photo or a sequential group of photos are flagged as home, use that gap between abroad photos to do a split and throwout home country photos;</p><p><strong>Split photos by Frequent Location</strong></p><p>3. For each group found in 2) do the same thing, but use frequent location as reference. If a photo or set of photos is found belonging to a Frequent Location you remove them and split the remaining by the gap created.</p><p><strong>Split photos by date span</strong></p><p>4. Finally, for each group found in 3) do again the same thing, but use now date span between consecutive photos and if it is above some threshold you split them.</p><p>Clusters of photos produced in step 4) represent independent unusual travel stories that are classified as Travel Abroad.</p><h4>Relative to Frequent Locations</h4><p>We will now follow a similar approach for travel stories based on Frequent Locations independently of the country. The key steps are as follows.</p><p><strong>Split photos by Frequent Location</strong></p><p>1. Remove photos with geographic data — Filter out photos with no geographic data;</p><p>2. Find first travel candidates by splitting photos based on Frequent Locations — Organize photos chronologically, go from the oldest to the newest and every time a photo or a sequential group of photos are flagged as Frequent Location, use that gap between non Frequent Location photos to do a split and throwout Frequent Location photos;</p><p><strong>Split photos by date span</strong></p><p>3. For each group found in 2) you will do the same thing but will use date span as reference. To prevent splitting too much you can weight this span based on the distance to the closest Frequent Location. If it is far away from a Frequent Location you use a larger span before doing the split and if it is closer the likeliness of being independent events increases and thus a smaller span can be used.</p><p><strong>Filter out Travel candidates</strong></p><p>4. Finally, we will filter out trial travel sets by a distance to the closest Frequent Location and also based on a time criteria.</p><p>Independent groups of photos are now independent travels obtained by using Frequent Locations as reference points.</p><h3>Small moments</h3><p>A simple and quite efficient way to organize photos in moments is by simply looking to the time span between consecutive photos and to the geographic distance between consecutive photos. If either time or geographic spans are above some threshold, you divide them in different groups.</p><h3>Photo Classification</h3><p>Now that we do have several organization perspectives, we will classify each photo based on previous groups where we will try to fill some blank spaces in photos with no geographic data. This classification can be used for different things, but we will just focus on how to use them to produce Smart Albums.</p><p>It is important to note that all previous organizations were based on photos stamped with geographic information, but “Small Moments”. This means that there will be no Frequent Locations nor Travel groups if all photos do not have geographic data.</p><p>It is also important to highlight that sometimes you can have several photos stamped with geographic data and others without it even if the user always used same Camera app. This is specially relevant in Android ecosystem as it will be later emphasised.</p><p>Note also that we used two different approaches to find Travel events and we did it to minimize inference errors by just considering a travel category that merges both grouping strategies.</p><p>If we organize camera roll photos chronologically in a spreadsheet with photos as rows and associated group as column for the scenario of a New Yorker going by car from New York to Montreal in Canada, you can have something like Table 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qh1Y99N9vgZnmrjzr4XS8g.png" /></figure><p>This example illustrates two possible situations that can happen often. The first is to have both travel classifications not matching in all records and second is to have photos without geographic data in the middle of photos with geographic information. To improve our inference in both situations we will start by adding another column to our table and do merge both travel classifiers. If a photo was classified as travel in at least one column, we do classify it as travel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*-q-lL7Lfrk7cedw2VshdKw.png" /></figure><p>Our classification of individual photos ends up by inferring travel classification for photos without geographic data like photos 3 and 4, respectively. To do so, we will look to the start and end date of Travel Merged column and will flag all photos without geographic data taken within that date range with same travel classification id.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kQ8CtxlVvUevOq2_P2MQQw.png" /></figure><h3>Smart Albums</h3><h4>Creation</h4><p>After having the classification process finished we can create Smart Albums. To do so, we will look exclusively to Travel Merged column and Moments. First albums are identified by Travel Merged and later we look to the Moments Id column in the remaining photos. The creation process ends by organizing all albums chronologically.</p><p>The number of columns used in the classification can be extended with information like calendar events (ie, Christmas, Easter, etc) to enrich your UI or leverage the use of filters to segment albums based on a set of criteria that you can define according to the requirements of your app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*W0--fKjcmVh8UPa4jVjjbA.png" /><figcaption><strong>Figure 2</strong> — Example of an UI showing Smart Albums where events<br>happening in a Travel context are slightly emphasized</figcaption></figure><h4>Filtering out photos in Albums</h4><p>Another important functionality that you can implement is the use of filters to filter out album photos. For example, Whatsapp App injects photos shared in chat channels into user’s camera roll that can just represent pollution in a Smart Album.</p><h4>From Smart Album to a Story</h4><p>Note that a Smart Album is just a set of photos that does not have any kind of computer vision or content analysis through deep learning yet. All the work so far is entirely based on photo meta-information.</p><p>To create a story in a video or a photo book formats you can trigger the curation flow explained in <a href="https://medium.com/storyo/automatic-photo-curation-through-storytelling-and-deep-learning-fcb88a0ea685"><strong>“Automatic Photo Curation through Storytelling and Deep Learning”</strong></a><strong> </strong>and filter out undesired images, group similar and select the best ones. You can also use the elastic capacity explained in <a href="https://medium.com/storyo/data-driven-storytelling-based-on-photo-metadata-116896a3cf8c"><strong>“Data-driven storytelling based on photo metadata”</strong></a> to transform an original set of 500 travel photos in a sharable one minute video format or in an affordable 50 page photo book all happening with an automatic flow.</p><h3>Apple and Android geolocation in photos: a very different reality.</h3><p>We will now share some insights based on data collected through Storyo Apps for Android and iOS that can help you in a decision process on how to use automatic discovery of stories.</p><p>Apple soon realised the importance of location stamped in pictures in the long term and have been promoting its use in the flow of installation of new devices and seems to do a quite good balance in GPS error and battery consumption. As expected, error tend to happen mostly indoor when no wifi is present.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*cSr5YoTp3I8ncy4fQyIoug.png" /><figcaption><strong>Figure 3</strong> — % of camera rolls with at least<br>one geo-tagged photo</figcaption></figure><p>On the other end, Android penetration of geo-tagged photos continue to be far more modest compared to its iOS counterpart. Storyo user base shows a penetration of photos stamped with geographic data above 75% of the camera rolls in iOS and in Android we do have near a 25 % penetration.</p><p>Even when camera rolls do have photos with geographic data stamped, they tend to have 50 % of photos with no geographic data in Android devices. This seems to be caused mostly by GPS / battery consumption optimizations and / or the error that Google allows photos to be stamped with latitude and longitude. For example, if you put Google Maps running in background on an Android device and start taking pictures, they will all end up having geographic data stamped unless you are indoor and no wifi is present.</p><p>Independently of the cause creating this disparity, we consider quite relevant for you to take decisions designing your solution with this reality in mind.</p><h3>Conclusion</h3><p>In this white paper we disclosed an algorithmic recipe used in <a href="http://www.storyoapp.com/sdk/"><strong>Storyo SDK</strong></a><strong> </strong>to curate photo memories based on space and time variability of photos present in user’s camera roll. The recipe is optimized for mobile environments and designed to be implemented CPU based to avoid background execution restrictions.</p><p>You can use automatically curated Albums as a gateway to communicate with your customers to foster loyalty and promote conversion where photos are mostly produced.</p><p>It’s important to note that the description in this and our previous articles <a href="https://medium.com/storyo/automatic-photo-curation-through-storytelling-and-deep-learning-fcb88a0ea685"><strong>“Automatic Photo Curation through Storytelling and Deep Learning”</strong></a><strong> </strong>and <a href="https://medium.com/storyo/data-driven-storytelling-based-on-photo-metadata-116896a3cf8c"><strong>“Data-driven storytelling based on photo metadata”</strong></a><strong> </strong>represent an implementation based on the patent “Systems and methods for automatic narrative creation” Issued by USPTO Jun 12, 2018 (<a href="http://patft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&amp;Sect2=HITOFF&amp;d=PALL&amp;p=1&amp;u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&amp;r=1&amp;f=G&amp;l=50&amp;s1=9996537.PN.&amp;OS=PN/9996537&amp;RS=PN/9996537">9,996,537</a>) for automatic video creation.</p><p>Our company believes that data is a powerful ally in the creation of long-lasting memories and that technology can power products and services that better engage users in the re-discovery and celebration of their experiences through our SDK geared for photo printing partners.</p><p>For more information, please contact us at <a href="/service/mailto: hello@storyoapp.com">hello@storyoapp.com</a>.</p><p><em>Next article: Deep learning strategies for content analysis photo curation.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53cc985aa7ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/storyo/automatic-curation-of-photo-memories-through-unsupervised-learning-53cc985aa7ac">Automatic Curation of Photo Memories through Unsupervised Learning</a> was originally published in <a href="https://medium.com/storyo">Storyo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automatic Photo Curation through Storytelling and Deep Learning]]></title>
            <link>https://medium.com/storyo/automatic-photo-curation-through-storytelling-and-deep-learning-fcb88a0ea685?source=rss----dfcba6f95ead---4</link>
            <guid isPermaLink="false">https://medium.com/p/fcb88a0ea685</guid>
            <category><![CDATA[photography]]></category>
            <category><![CDATA[storytelling]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[mobile-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Manuel Costa]]></dc:creator>
            <pubDate>Thu, 21 Feb 2019 15:33:35 GMT</pubDate>
            <atom:updated>2019-02-21T15:56:53.165Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Every company needs to be an AI company, and photo book companies are no exception. We’re inspired by “great, unsolved questions” around photography and storytelling, so here’s our take on how on-device AI, deep learning, and neural networks can be leveraged to create the next generation of photo printing products.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6qsMGcOiUKb5_fIQFtvGCg.png" /></figure><h3>Introduction</h3><p>The amount of pictures that we all take on our mobile phones is simply incredible, and that number is growing as cameras improve, more space is available on mobile devices, and cloud solutions like iCloud or Google Photos synchronize them with server infrastructure.</p><p>From <a href="http://storyoapp.com/">Storyo’s</a> user base, we know that on average, users have around 1,000 to 1,500 pictures on their devices (iOS and Android), and if we jump to the 98th percentile, we see that 2% of our user base has more that 20,000 pictures.</p><p>Photo organization is a hot topic that we’ve been looking at since the creation of the initial version of Storyo back in July 2014, which we’ve now also made available in our native <a href="http://storyoapp.com/sdk/">SDK</a> for iOS and Android.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o7BiOnq5GK2ZirZWl2D8ug.png" /></figure><p>We’ve always viewed photo organization as consisting of two big, relatively independent areas:</p><ul><li><strong>Automatic discovery of stories -</strong> discovery of relevant stories or sets of photos for users, and;</li><li><strong>Automatic photo selection -</strong> direct curation process for a given set of photos, either selected automatically or by a manual flow.</li></ul><p>Stories are composed by groups of photos and can be shown with very different organizations. We’ve been always in favor of organizing stories chronologically in a story perspective to represent a timeline of users’ camera rolls. Our users have validated this approach, with nearly 40% of the stories played in Storyo starting with a click on an automatic story.</p><p>Next, it’s crucial to decide how to select individuals photos from that initial set. For example, a travel story can easily have more than 200 photos, and if one wants to create a short video or a small photo book, one has to decide how to select a set of photos for the degrees of freedom available—like a 25 second video or a 25 page photo book.</p><p>Should we select the 25 best photos according to computer-vision based criteria, or should we keep the story as relevant as possible by identifying chapters and then select the best photos given a chapter’s context? With Storyo, we strongly believe in the second approach—organizing a story in chapters, and deciding later on which photos are the right ones to be added.</p><p>Another key aspect is centered on the option of local versus server-side execution. Deep learning models can occupy a lot of space and require strong computational power. Theoretical knowledge is evolving fast, and all the big players are investing a lot in tools and specialized hardware to run neural networks with unbelievable speed, directly on mobile devices. This is now a reality.</p><p>At Storyo, we’re focused on bringing the best experiences to users, and with the power of mobile devices today, local execution is definitely the best way to go if the space required is not a critical factor.</p><h4>This article will focus on how to create a photo curation flow that leverages storytelling and computer vision / deep learning, based on what we’ve built within the Storyo app and Storyo’s SDK for photo book and video timeline creation.</h4><h4>In an upcoming article of the series we’ll focus on automatic story discovery, and then we’ll continue with a set of articles detailing how deep learning models can be used as part of the curation flow.</h4><h3>Filtering out undesired images</h3><p>We know well that our mobile phone camera rolls are full of garbage — pictures of whiteboards in a meeting to keep information before someone clears it all, pictures of a flyer to keep the details of the hotel where we’ll stay for a few vacation days, and so on and so on.</p><p>Beyond these scenarios, we tend to forget to delete pictures that look horrible because of some kind of involuntary motion of our target, or because it was indoor and lighting conditions were poor.</p><p>Although these pictures can be meaningful in meeting some particular goal, they’re usually outliers in the context of “pictures as memories.” So, how do we get rid of them?</p><p>The answer to this question is mixing all the new possibilities with content analysis through deep learning and core computer vision methods to assess things like sharpness in an image. Although it seems quite a bit like science fiction for many people and even more difficult to understand, deep learning methods can be used by non-experts, and filtering out is a very good starting point.</p><p>Deep learning is evolving at an unbelievable speed, and the amount of tools offered by big players like Google, Apple, and Microsoft to developers and non-developers is growing at a high rate.</p><p>Independent of the service used, the trend now to tag images with classifiers like “Document” or “Whiteboard” is all about using a pre-trained convolutional neural network for classification within many categories. These models will learn to identify very generic features, and with a small add-on, can be used to extrapolate well to other categories of content for which they were not trained.</p><p>If you want to train your own set of categories by using a model like this, you just have to train a final layer (or layers) mapping those generic features to your content with a very small set of images. Microsoft offers several services through its Azure infrastructure and Cognitive service in particular, Google has its powerful Firebase services and recently launched Cloud AutoML to train custom models, and Apple offers Create ML, built to run on a Mac and train models for its ecosystem.</p><p>As with all deep learning models, size matters, and if you want to run models locally, this can be a shortcoming. With iOS 12, though, Apple implemented a small step that will allow apps to make a giant leap forward with deep learning execution on mobile devices—they added a base model for classification to the operating system bundle.</p><p>Using Apple’s Create ML to drag &amp; drop a few undesired images—like pictures of documents—we can create a new model with a small set of bytes to be assigned to an app, instead of the usual scale of several MBs.</p><p>Independent of the tool used, and with not many images required for training, it’s important to be careful. A good strategy is to create a category like “whiteboard” and several others as a control. For example, this control category could have pictures of people in front of a whiteboard in the background to mitigate misclassification and, consequently, filtering out good images.</p><p>In the current series of articles, we’ll dedicate one to image quality assessment through deep learning, which will include more insights about this subject and other things not so obvious, like sharpness classification through content analysis.</p><h3>Group Similar photos</h3><p>Grouping similar images can be something quite objective with no direct understanding of the content, or it can be something more relative and closer to human interpretation of the content.</p><p>Let’s now take two examples to better illustrate both situations. We’ll start with two images of a fish that can easily be matched by identifying a few key discontinuity points, and compare them at “pixel” level, even if the color doesn’t match and the images aren’t exactly the same (Fig. 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rOo67ELFck4uEVtwRm2RzQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gP-jdIYcZk06PDFY1MD9iQ.jpeg" /><figcaption><strong>Figure 1 </strong>— photos with same features at “pixel level”</figcaption></figure><p>Now we’ll look at two images with a school of fish, where the fish are constantly moving, expecting food to be dropped into the water (Fig. 2). Your brain will tell you immediately that those photos represent similar images. However, if you want to search for discontinuity points and match pixels, you won’t succeed. In fact, these two images are not the same or even partially the same.</p><p>The only way to do this type of match is through content analysis, where deep learning methods will eagerly answer, like children sitting in the front row of a classroom, if the images are similar or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*yIFJIGIdexgM7UvxT7dsrQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*DL1x8AAEhZ9FZUnmJQVMGQ.png" /><figcaption><strong>Figure 2</strong> — images with similar content, but not similar features</figcaption></figure><p>Because these methods require a bit more technical knowledge, we’ll leave their explanations for another dedicated article on these topics. It’s important to remember, however, that both approaches are valid and that they can complement each other.</p><h3>Rank Images</h3><p>After grouping similar images, you have to select the best candidate to represent each group. Several factors can be used in this process, and we’ll list just a few that we consider relevant in a practical perspective.</p><p>Let’s start with a simple and usually agreed-upon one: the presence of faces. You can detect if a face is present in an image by using off-the-shelf SDKs offered by Android and iOS. If a face is present, you can use that as well as its size to affect relative ranking. <a href="https://heartbeat.fritz.ai/building-a-real-time-face-detector-in-android-with-ml-kit-f930eb7b36d9">Face detection</a> is now moving also to deep learning and, thus, is quite powerful, which means that unsharp faces can be easily detected! We’ll get to this later.</p><p>Face landmark detection, which is used by funny apps like <a href="http://msqrd.me/">MSQRD</a>, can also represent an important piece of information. A landmark represents a specific point on a face, and its relative position can be used to do things like detect whether eyes open or closed or even smiles. Again, more complex assessments can be done through specific deep learning methods to infer a face’s (good) mood with a bit more bytes added to the app.</p><p>In face landmark detection, it’s important not to forget that in the iOS ecosystem, you can have iCloud involved with an Internet layer in the middle. If you want to increase accuracy of these marks, you might end up having an undesired network dependency that can drastically affect the experience offered to your users.</p><p>Sharpness is also an important measure in the relative comparison of images, and that can be done without deep learning involved to compare similar images with relatively good accuracy. If possible, it’s always good to avoid deep learning and the associated size added to your app. Note that we previously did propose using sharpness based on content to filter out images that can run simultaneously with other types of classifications.</p><p>Old methods looking to neighboring pixels like <a href="http://hyperphysics.phy-astr.gsu.edu/hbase/lapl.html">Laplacian</a> or a bit more complex ones involving <a href="http://mathworld.wolfram.com/HaarFunction.html">Haar Wavelet</a> can do the job. Because there are multiple websites talking about these methods, we suggest that you investigate them a bit more online, or buy a book about computer vision.</p><p>Independent of the method used, we have to remember that faces will be detected with powerful deep learning methods, and they can be unsharp. It’s important, therefore, to isolate the sharpness assessment to face bounding boxes if at least one is present. Then you can do things like assign a sharpness value of the picture based on the measure carried out on the biggest face.</p><p>Another good candidate to help us in the ranking process is to have a metric of aesthetics. Since aesthetics are inherently subjective , deep learning again is a good candidate to give us some help. We did test several models, and the one that left us happy was <a href="https://arxiv.org/pdf/1709.05424.pdf">NIMA: Neural Image Assessment</a>, which predicts what the histogram of classifications between 1 and 10 that a group of people would give to a certain image.</p><p>By predicting the classification range, it’s possible to infer not only the average ranking, but also metrics of variability like standard deviation. If variability is low, the result represents unanimity independent of the average value. On the other end, if it’s high, it reflects that the picture is not clearly good or bad.</p><p>For example, in Fig. 3, it’s possible to check its results for quite similar images taken with an iPhone X with no filters on the left, and masked by a depth filter on the right. The average ranking increases with this special filter, and the standard deviation of the classification decreases, which means that the model predicts that people will tend to like the picture on the right more, and the opinion will likely be unanimous.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tyNSO5nsin9yWFniLgxEAw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oru51jPyc0nkhRj3CST7Hg.jpeg" /><figcaption><strong>Figure 3 —</strong> aesthetics average classification and standard deviation for each image<strong> ( Left image: 5.46 +/- 1.77; Right image: 5.61 +/- 1.26 )</strong></figcaption></figure><p>Nevertheless, don’t get too confident with these results. NIMA was trained in public contests around photography, which creates some bias towards the types of pictures on which they were trained. For example, an image with motion blur caused by a restless dog can look very bad, but NIMA will produce a very good classification for it!</p><p>Ultimately, we think a good strategy to rank images is to use an index that uses all variables previously described. That is, though, out of the scope of this article. We might disclose how we do it in Storyo SDK in another article dedicated to this specific subject in the near future.</p><h3>Storytelling based on time and space</h3><p>Now that we have bad images filtered out, similarly grouped, and the best images selected within each group, we might end up having more pictures than desired.</p><p>For example, if you end up with 100 different photos (which can easily happen in a travel context) and want to show your photos as a video slideshow with exactly 60 seconds to be shared on Instagram, how do you do this? If we assume that the video shouldn’t show more than a 1 picture at a time, and that each picture should be visible for 2 seconds, how do we select 30 photos from the already-curated 100?</p><p>This is where storytelling can give you a hand. Before jumping into it, though, let’s talk a bit about photo meta-information space and time first. Photos taken by a device camera will always have a timestamp associated with them, and they can also have a location through latitude and longitude variables if the user configured these data points to be active.</p><p>It’s possible to use these 3 variables and cluster them through a clustering method, using the dataset computed previously. If latitude and longitude aren’t present, we can leave them constant in all photos or simply compute clustering with time variables only.</p><p>The Storyo App and Storyo SDK use a density-based clustering method (<a href="https://spin.atomicobject.com/2015/05/26/mean-shift-clustering/">Mean Shift</a>) that guarantees no random factors. There are a lot of clustering methods that require a random initialization and that do not always guarantee the exact same clusters when executed several times. A random factor that will end up changing your selection can easily be perceived as a bug from your user.</p><p>The clustering process then moves on by applying the exact same method to each cluster recursively until some stop criteria is matched. For example, a bounding box is below x km or the time range is below some specific span.</p><p>Imagine now that someone is traveling from New York to Europe and has 3 different pictures taken at JFK airport before their journey began. And then this person visits Rome, where 30 different pictures are taken, followed by Venice with 50 different photos. Then the user catches a plane to Paris and the journey ends up with 17 dissimilar pictures.</p><p>Because there is a huge geographic gap between the US, Italy, and France, the first clustering segmentation happens by country, with the first node representing photos taken in JFK, the second group of photos taken in Italy, and the third in Paris.</p><p>JFK photos are very confined in space and time and, thus, the clustering process ends there. Imagine now that the user spent one day in Rome, two days in Venice, and another day in Paris.</p><p>The Italy node is divided by the dominant geographies, with Rome in the first node and Venice in the second. In the day spent in Rome, the user took 10 different photos in the early morning, 5 in the afternoon, and 15 during dinner with a few local friends. Because the time there was short, the user didn’t move much in town, and the clustering division ended, dominated by the temporal gap between the three sets of photos (morning, afternoon, and dinner).</p><p>Although they spent two days in Venice, the user didn’t spend much time taking pictures, only taking 30 photos on day one and 20 on day two. In this scenario, the clustering ended up in a leaf node for day 1 and another leaf node for day 2.</p><p>Finally, during the day spent in Paris, the user took 7 pictures during lunchtime and 10 more at night, ending up with 2 children nodes divided by the time gap between the two photographic moments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gMaOxvgg-Wl_FSevbliArw.png" /><figcaption><strong>Figure 4</strong> — clustering tree based on space and time for a New Yorker visiting Rome, Venice and Paris</figcaption></figure><p>Now that we have a skeleton, we have to use it! To do so, we’ll grab our pre-defined 60 seconds and drop them at the root of the tree in a delegation flow of representation time.</p><p>We do know from our initial assumption that a photo should have 2 seconds, and that multiples of 2 should become our unit to split time. Another thing we have to do is to decide how to divide time between children nodes. A good way to do this is to use the number of photos in each child node. Don’t forget that repeated photos have already been removed and, thus, the number of different photos represents a very good candidate to weight the divisions.</p><p>Let’s see how a 60-second delegation flow can be carried out in a root node by tentatively sending time to JFK, Italy, and Paris nodes respectively. Remember that we have the following proportion 3 : 80 : 17 of different photos with 2 seconds as our unit. If we assign 2 seconds to JFK, 48 seconds to Italy, and 10 seconds to Paris, we’ll keep the proportions quite reasonable and assume that the first delegation did succeed.</p><p>We have to now tentatively delegate representation time in the Italy and France nodes independently. We start with Italy, where we drop 48 seconds to Rome and Venice in a proportion of 30 : 50 and can keep it successful by delegating 18 seconds to Rome and 30 seconds to Venice.</p><p>Continuing in Italy, we’ll try to delegate 18 seconds in Rome to morning, afternoon, and dinner in a proportion of 10 : 5 : 15. This gives a direct division of 6, 3, and 9 seconds, respectively. 3 and 9 are not multiples of 2 and, thus, we have to find a balance. A good possibility is to remove 1 second from dinner and give it to the afternoon, ending up with a division of 6, 4, and 8 seconds, respectively. Because there are no more children, delegation in the Rome branch ends here.</p><p>Venice is the next city to manage delegation. 30 seconds will be divided by day 1 and day 2 with a proportion of 30 : 20, which produces a successful delegation of 18 seconds for day 1 and 12 seconds for day 2. Again, no children are present and delegation stops.</p><p>We still have to manage Paris’s delegation, which was left with 10 seconds by dividing them with a proportion of 7 : 10 and ending up with an approximation of 4 seconds to lunch and 6 to night/dinner.</p><p>After establishing this delegation flow, we have to select photos. To simplify our final step, we’ll create a summary table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FI7eD9JXVoQpW3ww2eJwPg.png" /></figure><p>Remember that we already have the ranking computed for all photos that were used to select a photo within a group of similar photos. We can now use this same ranking to select in each group. Because they are much more segmented, we can simply pick those with the higher ranking now.</p><p>If you want to use this exact same approach for photo books, you just have to change the unit, and instead of seconds, you’ll work with pages. You’ll have a target number of pages, and you’ll tentatively delegate them throughout the clustering tree toward leaf nodes.</p><p>Be on the lookout for future articles this series, were we’ll explore automatic story discovery, and then we’ll continue with a set of articles detailing how deep learning models can be used as part of the curation flow. Stay tuned, and until then, give <a href="http://storyoapp.com/">Storyo</a> a try and let us know what you think!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcb88a0ea685" width="1" height="1" alt=""><hr><p><a href="https://medium.com/storyo/automatic-photo-curation-through-storytelling-and-deep-learning-fcb88a0ea685">Automatic Photo Curation through Storytelling and Deep Learning</a> was originally published in <a href="https://medium.com/storyo">Storyo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data-driven storytelling based on photo metadata.]]></title>
            <link>https://medium.com/storyo/data-driven-storytelling-based-on-photo-metadata-116896a3cf8c?source=rss----dfcba6f95ead---4</link>
            <guid isPermaLink="false">https://medium.com/p/116896a3cf8c</guid>
            <category><![CDATA[data-driven-design]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[photography]]></category>
            <category><![CDATA[photobook]]></category>
            <category><![CDATA[data-driven-stories]]></category>
            <dc:creator><![CDATA[Leonel Duarte]]></dc:creator>
            <pubDate>Thu, 21 Feb 2019 14:58:46 GMT</pubDate>
            <atom:updated>2019-03-01T15:45:06.096Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Storyo is using our digital footprint and AI to create stories using photos and contextual data to power photo products.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eXt8E-_HOEdmrWDxC2fk2A.png" /></figure><h4>The camera rolls on our devices are uncurated visual timelines of our lives. And they grow everyday.</h4><p>As a result the records of our most precious moments are many times lost in endless scrolls or entangled with disposable or unrelated imagery and chunks of similar photos proof of our determination to capture the perfect shot. And more importantly the life stories that get trapped within the data these lost memories contain.</p><p>Along with the impact in photography, smartphones today are processing powerhouses allowing complex algorithms to run locally and very fast. They are also at the center of many other aspects of our lives, shaping the way we connect to others, the world, our bodies and assuming a pivotal role in our data trail.</p><p>The essence behind Storyo is about leveraging this data trail and the power of modern devices to develop tools that uncover the stories in our photos, find the shots, give them context and add other bits and pieces of information, such as maps and real world information, to deliver it all in a narrative that can be cherished, celebrated and shared.</p><h3>A data-driven storytelling engine.</h3><p>When you take a photo with location data you create a bookmark. You pin yourself to a moment in space and time. Add a few of those together and you can have a detail account of what you have been up to in a specific timeframe. Add a few more and you can detect patterns, your favorite locations, your trips, the faces that make up your world and start to build a profile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_pay9UsEwyuhGW6ZnYhvwA.jpeg" /></figure><p>In our native <a href="http://storyoapp.com/sdk/">SDK</a> for iOS and Android for Photo Books, auto detection of stories in any given device begins with a scan of the entire photoset. Unrelated or otherwise ineligible photos are discarded and individual stories are isolated. As a film strip being cut in individual scenes, the camera roll is sliced in unique stories.</p><h4><strong>As an example Storyo SDK takes less than one second on an iPhone X with 20.000 photos, to detect all the stories in the camera roll.</strong></h4><p>This process is detailed by <strong>Manuel Costa </strong>(Storyo’s CTO) in his <a href="https://medium.com/@manuel.costa/automatic-photo-curation-through-storytelling-and-deep-learning-fcb88a0ea685"><strong>article</strong></a>: <em>“Automatic Photo Curation through Storytelling and Deep Learning”</em>and as he points out:</p><blockquote><strong>“We have been always in favor of having stories organized chronologically to represent a timeline of the users’ camera rolls in a story perspective. This idea has been validated by our users, with near 40% of the stories played in our video </strong><a href="http://www.storyoapp.com/"><strong>app</strong></a><strong> starting with a tap on an automatic story.”</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o7BiOnq5GK2ZirZWl2D8ug.png" /><figcaption>Storyo SDK main features.</figcaption></figure><p>All these processes run locally, on the fly and use all the already available ML dedicated hardware, ensuring performance, privacy for the user and no server costs. Three of the main outputs that are key to automate the narrative creation are:</p><ul><li><strong>User profile:</strong> The system compiles a set of statistics about a user based on his/her set of photos;</li><li><strong>List of Stories:</strong> Each story is categorized by a set of criteria like Frequent Location, Calendar events, Travel, etc;</li><li><strong>Photo ranking and profile:</strong> Each photo is similarly categorized and is also analysed and ranked for technical quality, aesthetics interest, face detection, similarity and storytelling relevance.</li></ul><h3>Space-time narratives</h3><p>In Storio’s automatic story discovery feature, every photo is clustered in a space-time structure we call the <strong>Tree of Events</strong>. This is created for each individual story and preserves the chronological order in wish the photos were shot. This clustering structure provides the backbone for Storyo’s storytelling capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*egtUB22eeAGwj0jtI-V_zQ.png" /><figcaption>Example of a Tree of Events</figcaption></figure><p>Let’s take as an example a set of 100 photos from a trip to Italy and France. From the individual profile, created in the initial analysis of the entire camera roll, the system determines that all the photos selected belong to the same travel journey and groups separate photo moments from different countries in one single story including the first photo taken at the airport of departure.</p><p>The nodes in the Tree of Events can be space or time based and each node corresponds to a possible chapter in the storyline. In our example, the top 3 chapters are space-based (JFK, Italy and France). As we go down the tree, sub-chapter segmentation can be dominated by space based organizations (ex: Rome and Venice) and others can be organized by time (ex: Venice Day 1, Venice Day 2).</p><p>From the Tree of Events the system can output more or less detailed storylines by exposing or not, the nature of each node in the final layout.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TbIfZMK-v6OxSuljnpruZw.png" /><figcaption>Different storylines from the same Tree of Events with increasing detail levels</figcaption></figure><h3>Elastic storytelling</h3><p>The numbers of possible chapters in a story is directly dependent on the space-time nature of the photos in each photoset. But whether they are shown or not in the final storyline is algorithmically calculated taking in consideration parameters such as the total number of pages available in the photo book or the desired number of photos per page.</p><p>This flexibility allows the same 100 photos in our example to be turned into a brief 20 page book with 3 chapters or up to an epic 100 page, 8-chapter photo book, but both depict the same storyline or experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nX074HQARnQEDBWbHM9amQ.jpeg" /></figure><h3>Telling the story</h3><p>To understand the way the storyline is exposed in the final layout we need to get familiar with the concept of <strong>Storytelling Units. </strong>They are the design and content elements that give context to the sequence of photos and convey the underlined narrative. They can take up a full page in the book layout, like a map introducing a chapter, or can be placed along the photos such as a caption next to an image.</p><p>They can be text based and derive directly from the metadata in the photos like a location name (ex: street name or the time a photo was shot). But can also result from the clustering in the Tree of Events such as location names or timeframes common to all photos in a chapter (ex: Paris or Day 1). Others can be visual elements and also derive directly from the metadata, like a map with a point representing a photo. But again can be generated by the Tree of Events such a map with a line connecting two or more points depicting a movement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LWw1trjESaszM5f-nwyFaw.png" /><figcaption>Example of a map depicting a group of photos from a short timeframe</figcaption></figure><p>These Storytelling Units can come from external sources by cross referencing space and time data. It can be a map service, as we already covered, or weather, Facebook activity and others. Pre-defined conditions determine when an external source is added. For instance, the system can cross reference weather conditions of the travel journey with those verified at home and add information if a relevant difference is found as weather will then be an important part of the experience. Similarly if the user’s Facebook account is connected, not only username and profile photo can be added to the layout but the system can also cross reference activity in the social network with the time and location of the photos and determine if a relevant post is available to be added to the story.</p><h3>Designing Photo Book Stories</h3><p>There are 3 categories of Storytelling Units: Full story level, chapter level and photo level. The way that these different units can be used is fully customizable. They can be in fixed positions in the layout or can be parametric and can or not show up depending on the specific photoset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*QVcqCRjH1UWOvMVCyWQq_w.png" /></figure><p>The easiest way to understand how the system can be used to create a photo book travel story is to think of if as a storyboard. Let’s again refer to our example and for the sake of clarity go with the 3 chapter version of the story.</p><p>For the sake of this explanation let’s also introduce the concept of <strong>Entry</strong>. In the case of photo books, Entries are the individual pages and each one has a role in the storyline. Some serve the purpose of displaying photos, others may be intros, story summaries, chapters breaks, external data sources, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*rU-zrnUBB_ND4prhzHT0rg.png" /></figure><p>In our example we start with the cover that displays the top ranked photo of the pack and a title that starts with “Trip to” because the system profiled the story as a trip. “Europe” is the top level location name for all the locations outside the “home” country. The title is complemented with a reference to the time frame of the photos and because this specific trip lasted a couple of month the generated “time” sub-title is “Summer of 2017”.</p><p>The cover entry is followed by the intro page that displays a visual summary of the trip in the form of a map with the path of the trip. Given that the user is logged in with their Facebook account, profile pictures can be automatically added to the layout.</p><p>Due to the nature of the first chapter (which represents home and has only one photo) there’s no chapter break page so the correspondent map is displayed on the photos page of the chapter. Next come the second and third chapters, and both have chapter break pages that display a map generated from the bounding box of the following photos with a label that is the common location name for the images in each chapter. In these cases is the country name, Italy and France. Then follows the photos pages that are configured to display up to 2 pictures per page and a map displaying the locations of the photos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qSKI8VCCbT4Mylz9hSADnA.png" /></figure><p>By design this layout also includes the introduction of a Facebook post if available. By cross referencing timeframe and locations of the photos with the user’s posts, it is possible to add the most popular one to the story representation. This is just an example of a possible layout for a travel photo book — other options may give less emphasis to maps, or replacing chapters brake pages with text labels along photo pages. Other external sources of contextual data can also be added like weather or stock imagery cross-referenced with the photos content.</p><h3>More than photobooks</h3><p>The Storyo <a href="http://storyoapp.com/">app</a> uses the same storytelling concept in the form of video. Users have access to automatic stories created from their camera rolls but they can also create stories by selecting a timeframe of photos or picking them one by one. The app then creates a video on the fly automatically. The user can edit every aspect of the video, add stock imagery and share it directly from the app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lRw26wjSMTF1lXxFvbwPfQ.png" /><figcaption>Storyo app available for iOs</figcaption></figure><p>Another breakthrough that the app offers is the ability to create group videos inviting others to add their photos with the same automation and easiness. In the case of group stories each users gets a story that can be different from the other users. As an example, if I visit a friend in another country, my version of the story will reflect the fact that I’m traveling abroad by displaying a 3D globe showing origin and destination. My friend’s version will be a home story and the video might not even show maps at all.</p><h3><strong>Creating the next generation photo book products</strong></h3><p>As smartphones became the main device we use to take photos, our habit of taking photos has grown exponentially. At the same time photography itself has become a widely used tool to take notes, remember things or otherwise capture something for future reference. Add screenshots, memes, messaging services and an unlimited storage cloud service and there is a massive amount of images and data at the tip of your fingers.</p><p>Our technology is here to leverage all that data to power the creation of memorable stories for people everywhere in a simple and efficient way by using AI, neural networks and every means at our disposal to make the process of memory creation as seamless as our own experience of living those memories in the first place.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=116896a3cf8c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/storyo/data-driven-storytelling-based-on-photo-metadata-116896a3cf8c">Data-driven storytelling based on photo metadata.</a> was originally published in <a href="https://medium.com/storyo">Storyo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>