<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shourabh Rawat on Medium]]></title>
        <description><![CDATA[Stories by Shourabh Rawat on Medium]]></description>
        <link>https://medium.com/@shrawat?source=rss-c9fe594faafc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*B_gHv3dXCn5oYAEO3NtGvg.jpeg</url>
            <title>Stories by Shourabh Rawat on Medium</title>
            <link>https://medium.com/@shrawat?source=rss-c9fe594faafc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:20:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shrawat/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Extracting Salient Thumbnails for Real Estate Panoramas]]></title>
            <link>https://medium.com/zillow-tech-hub/extracting-salient-thumbnails-for-real-estate-panoramas-b2b749890eda?source=rss-c9fe594faafc------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2b749890eda</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[360-degree-video]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[360-panorama]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Shourabh Rawat]]></dc:creator>
            <pubDate>Mon, 03 Dec 2018 23:22:43 GMT</pubDate>
            <atom:updated>2019-04-29T20:37:46.600Z</atom:updated>
            <content:encoded><![CDATA[<p>Co-Authored by Alex Colburn, Principal Applied Scientist, Zillow Group</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*kfbVv4uk0th9W1wJtoo9oA.png" /></figure><p>In a <a href="https://www.trulia.com/blog/tech/selecting-hero-images-with-deep-learning/">previous post</a>, we shared how Trulia leverages computer vision and deep learning to select the hero image for listings on our site. Today we’d like to share how we’ve extended this technique to extract salient thumbnails from 360-degree panoramas of homes.</p><p>The development of automatic panorama generation in modern cameras has made 360-degree images ubiquitous. They are widely used in multiple industries, in everything from Google Street View, to virtual reality (VR) films. Last year, Trulia’s parent company Zillow Group, released <a href="https://www.zillow.com/marketing/3d-home/">3D Home</a>, enabling users to upload and share 360-degree panoramas of their listings. Compared to traditional photography, 360-degree panoramas create an immersive experience by providing a complete view of the room.</p><p>This capability has added value for home-buyers who can see a room and home from all angles, not just what the photographer chose to highlight. However, given the wide field of view, it is easy for the viewer to look in the wrong direction. This creates a need for a mechanism to direct viewers attention to the important parts of the scene. This is particularly true in the context where the 360-degree panoramas need to be represented as a static 2D image as shown below in Figure 1. Here we ensure that these 360-degree panoramas are discoverable and engaging to the user by identifying the most salient viewpoints within the panorama.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*z9AskzNcmjvLcPGJ-nw5Wg.png" /><figcaption><em>Figure 1. A screenshot of the 3D home user experience. The thumbnails in the bottom row help viewers get a sense of the important aspects of each panorama.</em></figcaption></figure><h3>Zillow Group 3D Home</h3><p>With the release of the <a href="https://www.zillow.com/marketing/3d-home/">Zillow 3D Home</a> last year, agents and sellers can capture 360-degree panoramas of a home and add them to their listing on Trulia or Zillow. The app guides the users through the capture process and then uploads these panoramas to ZG servers. If you are interested to know more on how we use computer vision techniques to create panoramas and compose a rich 360-degree experience we encourage you to read our <a href="https://www.zillow.com/engineering/behind-zillow-3d-home-backend-algorithms/">blog</a>.</p><p>These panoramas help provide an immersive home viewing experience that isn’t possible with the standard, limited field of view 2D photos. Panoramas like these help agents and sellers attract more views to a listing.</p><p>In Figure 1 we show an example of how these panoramas look on our site. The entry point to each panorama is marked with a salient thumbnail designed to provide an informative view of each panorama. Such a thumbnail can help increase the chances of a user to engage with the content and enter the 360-degree experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*10bnw69PJOn5EXvihrPPkQ.png" /><figcaption><em>Figure 2. A 360-degree real estate panorama with 60-degree vertical field of view.</em></figcaption></figure><h3>Characteristics of a Salient Thumbnail</h3><p>Salient thumbnails are visually informative and aesthetically pleasing viewpoints of a spherical 360-degree panoramic image. We define salient thumbnails as having three key characteristics, they must be representative, attractive, and diverse.</p><ol><li><strong>Representative</strong>: A salient thumbnail should be informative and representative of the visual scene being captured by the panorama.</li><li><strong>Attractive</strong>: A salient thumbnail should capture aesthetically pleasing viewpoints within a scene.</li><li><strong>Diverse</strong>: If multiple salient thumbnails exist, they should capture different viewpoints within the scene.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*a9MrSiL9ZyFuGAgiazt6MQ.png" /><figcaption><em>Figure 3. Extracted 2D viewpoints from the above panorama. Viewpoint C is selected by our algorithm as the salient viewpoint while Viewpoint D represents the thumbnail generated by the baseline approach. Viewpoint C is able to provide a representative view of the scene while being more aesthetically pleasing and relevant than other viewpoints.</em></figcaption></figure><h3>Generating Thumbnails</h3><p>An initial approach to generating a thumbnail would be to simply extract a single viewpoint in the hope that it will result in a salient thumbnail. However, this presents certain challenges.</p><p>Panorama images represent 360 degrees of horizontal view and 180 degrees vertical view resulting in a 2:1 aspect ratio. In some instances like in Figure 2, an image may have a limited vertical field of view and require padding with zero values, or black pixels to maintain the aspect ratio.</p><p>We could extract the central viewpoint in the panorama, which often corresponds to the direction in which the camera was facing when the capture process began. However, this approach is reliant on users starting with a representative viewpoint in order for it to be salient.</p><p>Furthermore, we cannot simply crop a rectangular section from a panorama image. Unlike 2D images, panoramas are spherical, simply cropping the central viewpoint leads to thumbnails which have spherical artifacts, e.g. straight lines look curved.</p><p>To extract a viewpoint from a spherical panoramic image, we project the panorama into a 2D image with a specific Field of View, 3D orientation, aspect ratio, and viewport size as shown in Figure 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*90-nPgT2xZrO9aWQYsbTrg.png" /><figcaption><em>Figure 4. Extracting perspective images from 360 degree spherical panoramas</em></figcaption></figure><p>While this addresses the spherical artifacts, thumbnails extracted from the central viewpoint can often be unrepresentative of the scene the panorama is trying to capture, as shown in Figure 3 above.</p><h3>Generating Salient Thumbnails</h3><p>To resolve this we needed an algorithm that ranks all the potential perspective thumbnails extracted from the panorama based on our defined saliency criteria. Below we list out some of the key components of our approach at extracting salient thumbnails.</p><h3>Optimal Field Of View (FOV) Estimation:</h3><p>Dedicated 360 cameras often offer full 180-degrees vertical Field of View (FOV). However, capture devices like smartphones often have a limited field of view. For example, a standard iPhone camera has a 60-degree vertical FOV. In such cases where the whole 180-degree vertical FOV is not present, it’s common practice to pad panorama images with zero value pixels as shown in Figure 2.</p><p>Due to these variations in vertical FOV, correctly estimating the vertical FOV before extracting the thumbnail images is paramount. Otherwise, we may be left with uncaptured regions surrounding a thumbnail image when estimated FOV is too large, or a significant loss of views in the vertical direction if the estimated FOV is too small.</p><p>To address this issue, for each panorama we identify the exact location of horizontal border lines that separate black regions from image regions and then use this location to obtain optimal FOV.</p><h3>Computing Saliency Scores</h3><p>Once the optimal FOV has been estimated, the system extracts 90 thumbnail images sampled uniformly at a horizontal view interval of 4 degrees from the panorama. The 4-degree intervals were empirically chosen to achieve a tradeoff between computational overhead and the ability to capture all diverse salient viewpoints within the panorama. These images or viewpoints are then ranked based on a saliency score. The viewpoint with the highest score is then chosen as the salient thumbnail.</p><h4>Saliency Models</h4><p>In order to compute a saliency score, we rely on three different Deep Convolutional Neural Networks (CNN) models that we briefly covered in a <a href="https://www.trulia.com/blog/tech/selecting-hero-images-with-deep-learning/">previous post</a>. Each of these models are trained on a different dataset and help capture the various characteristics of a salient thumbnail.</p><p><strong>Scene Model: </strong>This model helps capture the representativeness of a viewpoint to the panorama. Since we are capturing real estate panoramas, viewpoints that are categorized as relevant scenes like the kitchen or living room are considered more representative than viewpoints that focus on objects or non-scene related views like blank walls, windows, or chandeliers. This model is trained to categorize over 60 different scene classes commonly found in real estate photos.</p><p><strong>Attractiveness Model: </strong>This model captures the visual attractiveness of a viewpoint. Viewpoints that are pointing to un-attractive regions of the panoramas or have low visual quality (blurry, dark, etc) are penalized, while viewpoints that are attractive and aesthetically pleasing earn a higher score. Training such deep learning models often require large amounts of data. In general, in our datasets we have found that luxury homes often have high quality, professionally staged, and aesthetically pleasing photos, while fixer-uppers sometimes have low quality, poorly captured images. Hence we use the predictions from another ML classification model that uses property features like price, location, property description, etc to categorize properties as <em>Luxury</em> or <em>Fixer</em> and then use it to label images from those properties as either <em>Attractive</em> (if it belongs to a Luxury property) or <em>Not-Attractive</em> (if it belongs to Fixer Upper). This allows us to train a deep learning model for image attractiveness on millions of images without the need of expensive data collection.</p><p><strong>Appropriateness Model</strong>: This binary classification model helps to differentiate between relevant viewpoints like views of a bedroom from irrelevant views like walls, macro objects, pets, humans, text, etc. Along with the scene model, this model is used to quantify the representativeness and relevance of a viewpoint to the panorama.</p><p>Each of the candidate thumbnails/viewpoints are processed by the above deep learning models to generate three distinct scores. The saliency score is then defined as the weighted sum of three scores obtained from the scene, attractiveness, and appropriateness models.</p><h4>Smoothing Model Outputs</h4><p>Since our deep CNN models are trained on 2D images that are sampled from real estate listings, they can be susceptible to errors when applied to 2D image frames extracted from continuous streams like video or 360-degree panoramas. One potential reason for this is the bias introduced by training data where 2D images do not fully represent the distribution of all potential viewpoints that can be extracted from a 360-degree panorama.</p><p>This sampling bias during training and testing can cause the above deep CNN models to generate non-smooth outputs across adjacent frames. This is a general problem with deep networks and we plan to address it in our future versions by tuning these networks to produce smoother outputs across adjacent viewpoints.</p><p>To make our algorithm more robust and accurate to the aforementioned artifacts, we apply a smoothing function, in this case, Gaussian Kernels, to the probability outputs of individual classes for these deep models along horizontal sampled viewpoints. This smoothing ensures that the model predictions do not change drastically across adjacent viewpoints.</p><h4>Scene type dependent weighting</h4><p>As mentioned earlier, the scene model helps us to quantify the representativeness of a viewpoint. For example, a thumbnail image with a scene type of living room should be preferable to a thumbnail with a scene type of wall. To achieve this we associate a preference weight for each scene type. In general, scene types such as living room, bedroom, dining room and kitchen should have higher weights than scene types such as a window, door, or wall. The representativeness score of a thumbnail is then computed as the product of the predicted probability of that scene type and the preference weight of that scene class.</p><h4>Introducing Diversity</h4><p>While the use case we showed above (Figure 1) requires one salient thumbnail per panorama, the algorithm supports returning multiple salient thumbnails. This is relevant in scenarios where the application might want to provide multiple salient viewpoints like with a panorama summary that uses an image collage, or where more than one salient viewpoint exists, for example a panorama capturing an open floor plan with viewpoints of both the living room and kitchen.</p><p>When choosing multiple salient thumbnails, it’s important to consider diversity in the top N salient thumbnails that are returned to the user. In this case, we introduce diversity by ensuring that the selected horizontal viewpoints are well separated from each other. To achieve this, we first score all the viewpoints using the algorithm described above and then rank them by their saliency scores. After selecting the top salient thumbnail, the next salient thumbnail is chosen among thumbnails that have at least 30 degrees of separation in horizontal views compared to the already selected thumbnails. The same principle continues until either top N salient thumbnails have been selected or no more thumbnails are left that satisfy the above condition.</p><h3>Evaluation through Human Judgement</h3><p>Now that we have a model, how do know if it actually works? It is one thing to test a model against a labeled ground truth test set, and an entirely different thing to learn how humans perceive the results. In order to test with humans, we performed a series of qualitative and quantitative experiments.</p><p>We used a total of 137 panoramas to evaluate the performance of the new algorithm compared to the baseline algorithm which simply extracts the central viewpoint. A total of 50 Amazon MTurk workers participated in the evaluation.</p><p>For each panorama, we show to the users three images: the original panorama, a thumbnail image using the baseline algorithm, and a thumbnail image using the new algorithm. Each worker is then asked to choose which thumbnail image from two candidates he/she prefers, as well as whether the worker <em>strongly</em> prefers or <em>slightly</em> prefers their selected thumbnail image</p><p>To ensure order doesn’t bias results we randomly shuffle the order in which the thumbnails from the two algorithms are displayed for a panorama. To ensure crowdsourcing quality, we also randomly inject benchmark or gold standard images into the experiments to track and monitor worker quality.</p><p>Overall results of the experiments showed a statistically significant improvement for our new algorithm when compared to the baseline algorithm. (Table 1)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*6LDvN1NQk72mZCQY7xYjvA.png" /><figcaption><em>Table 1: Results from the human evaluation. Each panorama was shown to 50 users. For ~70% of the panoramas, the majority of the users preferred salient thumbnail over the default one. In terms of overall votes, ~66% preferred the salient thumbnail over default thumbnail, out of which ~66% were strong preferences.</em></figcaption></figure><h3>Deployment to Production</h3><p>To deploy these models we leverage our ML framework for deploying deep learning models at Trulia that we described in our <a href="https://www.trulia.com/blog/tech/deploying-deep-learning-trulia/">previous post</a>.</p><p>We expose the thumbnail generation service as a Rest API on top of our ML platform. The API allows the client applications to query the service with a thumbnail image and optional parameters like desired aspect ratio, height, width, number of diverse salient thumbnails to be extracted, etc. The service then asynchronously computes the desired number of diverse salient thumbnails and returns them to the client via s3 or as a base64 encoded image via sqs.</p><p>Note that each panorama can have a large number of potential viewpoints. For example, sampling per degree of horizontal view, while keeping other parameters constant, can lead to 360 distinct viewpoints to score. Even though the service is deployed on GPU instances and our CNN models can compute saliency scores per viewpoint within milliseconds, sampling and scoring all potential viewpoints can soon become computationally expensive given our low latency requirement for the thumbnail generation service. Furthermore, many of these viewpoints will have significant overlap and do not provide diverse enough scores to justify the additional overhead. Through our experiments we have found extracting 90 thumbnail images, uniformly at a horizontal view interval of four degrees each, to be a good balance between recall (capturing a salient viewpoint) of the algorithm and overall latency of the service.</p><h3>Going Forward</h3><p>The above results demonstrate the value of leveraging deep learning to extract salient thumbnails from 360-degree panoramas. Going forward we will improve upon this approach. First, we’d like to develop a better understanding of saliency based on user feedback. We can capture user preferences to candidate thumbnails suggested by the model, and use that as a training signal for our models. Second, instead of training 3 different models on disjoint datasets to define saliency, we will like to move toward a single end-to-end CNN architecture to model saliency of an extracted thumbnail.</p><p>Stay tuned for future posts on how Trulia is leveraging AI and ML to help consumer discover a place they’ll love to live.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2b749890eda" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zillow-tech-hub/extracting-salient-thumbnails-for-real-estate-panoramas-b2b749890eda">Extracting Salient Thumbnails for Real Estate Panoramas</a> was originally published in <a href="https://medium.com/zillow-tech-hub">Zillow Tech Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Deep Learning]]></title>
            <link>https://medium.com/zillow-tech-hub/deploying-deep-learning-at-trulia-f584a628a6ae?source=rss-c9fe594faafc------2</link>
            <guid isPermaLink="false">https://medium.com/p/f584a628a6ae</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Shourabh Rawat]]></dc:creator>
            <pubDate>Fri, 16 Feb 2018 16:00:00 GMT</pubDate>
            <atom:updated>2019-09-02T18:45:45.529Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deploying Deep Learning in Production</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*7slchLO86EU-xckWz7g4XA.png" /></figure><p>In previous posts, we shared how <a href="https://www.trulia.com/">Trulia</a> uses deep learning and image recognition to drive consumer engagement through applications like our homepage <a href="https://www.trulia.com/blog/tech/organizing-photo-collections-for-visual-browsing/">collections of homes</a> and listing <a href="https://www.trulia.com/blog/tech/selecting-hero-images-with-deep-learning/">hero images</a>. In this post, we will share how we deploy these models to production, and some of the challenges and design decisions involved in the process.</p><p>The lifecycle of any machine-learning or deep-learning model can be broadly broken down into three phases: 1) data gathering, 2) training, and 3) deployment. We’ve talked about how we train some of these models, but training models is only part of the challenge, serving these models in production has its own set of challenges.</p><p>In our case at Trulia, our inference engine and the underlying hardware and software infrastructure needed to support a large volume of traffic while being extensible, scalable and fault tolerant.</p><p>While building our system we had the following key features in mind:</p><ul><li>Machine learning as a service</li><li>Decoupled API and prediction layer</li><li>Scalable and fault tolerant infrastructure</li><li>Extensibility: Support for multiple frameworks</li><li>Optimizing performance for high throughput and low latency</li></ul><p>Let’s look at each of these features in more detail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*cOonU3B_qEbcIcBXz_2e4A.png" /></figure><h3>Machine Learning as a Service (Service Oriented Architecture)</h3><p>At Trulia, we deploy our deep-learning models as a RESTful API service. Serving machine-learning models as a service has several key advantages. First, deep-learning models often require dedicated hardware like Graphics Processing Units (GPUs) to accelerate processing. Abstracting the hardware and software dependencies of the models behind the service makes integration with downstream applications easy. Second, it allows our models to be easily usable across different teams and applications while sharing the underlying infrastructure.</p><p>The service supports both synchronous and asynchronous requests, making it suitable for both real-time and batch processing workflows.</p><p>Asynchronous mode is useful for near real-time or batch processing workflows that can consume the predictions generated by the system asynchronously through pluggable callbacks like <a href="https://kafka.apache.org/">Kafka</a>, <a href="https://aws.amazon.com/sqs/">Amazon SQS</a>, <a href="https://aws.amazon.com/s3/">Amazon S3</a> or a custom callback service, etc. For example, we use asynchronous requests with Kafka callbacks to stream near real-time updates of image tags to our image indexes that power applications like Trulia’s homepage or hero image project.</p><p>On the other hand, synchronous mode is ideal for low-latency applications that want real-time predictions for their inputs.</p><h3>Decoupled API Layer and Prediction Layer</h3><p>Trulia’s inference engine consists of two core components: The API layer and the prediction layer.</p><p>The API layer is responsible for handling client requests and directing them to the backend prediction layer that hosts machine-learning models to make predictions. The API layer and prediction layer communicate with each other using <a href="http://www.celeryproject.org/">Celery</a>, a distributed task queue that enables distributed processing of tasks across single or multiple worker servers. For our setup, we use <a href="https://www.rabbitmq.com/">RabbitMQ</a> as the message broker and <a href="https://redis.io/">Redis</a> as the backend for storing system state and task results.</p><p>The API layer is responsible for composing and pushing tasks to the job queue, while the prediction layer is comprised of workers that consume and execute tasks from the job queue.</p><p>The decoupling of the API layer from the prediction layer has several key advantages:</p><p><strong>Abstracting Prediction Layer</strong>: The API layer helps to abstract away the prediction layer from the clients. This is essential to regulate the traffic to the prediction layer to ensure it doesn’t get overwhelmed and that the prediction resources (e.g. GPUs) are properly utilized. The API layer is responsible for parsing user requests and composing tasks for the prediction layer. This includes chaining and routing the multiple CPU/GPU bound tasks needed to complete the request.</p><p><strong>Decoupling Responsibility</strong>: The API layer is built for high availability and low latency service to the clients, while the prediction layer focuses on managing machine-learning models and providing high prediction throughput.</p><p><strong>Decoupling Infrastructure</strong>: The design enables the API layer to be hosted in a separate set of nodes with different software and hardware dependencies than the prediction layer. The API layer requires CPU nodes to handle concurrent requests while the prediction layer leverages GPU nodes to provide high throughput prediction. This enables us to horizontally scale each of these services independently as required.</p><p>Trulia receives millions of images every day, as new listings hit the market. Each of these images needs to be annotated by several deep-learning models before their attributes can be leveraged by downstream applications.</p><p>GPUs have been widely used for deep learning, as they provide significant performance improvements over CPU. To be able to process this scale of data and provide near real-time predictions, we have set up an in-house cluster of GPU machines for deploying these models.</p><p>We have found <a href="https://www.nvidia.com/en-us/geforce/products/10series/titan-x-pascal/">Nvidia Titan X</a> (Pascal/Maxwell architectures) GPU cards ideal for our setup, given their large GPU memory and compute power. At full capacity, our system can process more than 20 million images per day, even with a reasonably sized cluster.</p><p>As mentioned earlier, our setup helps us to horizontally scale our compute capacity by simply adding new GPU/CPU nodes to our prediction and API layers, while providing fault tolerance against node failures.</p><p>While there are a lot of cloud solutions providing support for GPUs, like <a href="https://aws.amazon.com/">Amazon Web Services</a> and <a href="https://cloud.google.com/">Google Cloud Services</a>, we have found the in-house cluster to be more cost effective for our use case, as our services are running 24/7.</p><p>However, in certain scenarios, like in the case of a sudden spike in traffic or in re-tagging images in our database when a new model is released (in tens of millions), we would like our system to seamlessly scale without impacting the performance of our existing applications. To achieve this, we have also added support for on-demand scaling with <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html">AWS Auto Scaling Groups</a>, which allows us to spin up new AWS GPU/CPU instances to scale our infrastructure to meet the required capacity.</p><p>This hybrid architecture helps us keep the day-to-day running costs down with the use of an in-house cluster, while at the same time providing us the ability to efficiently handle high-load use cases when needed.</p><h3>Extensibility: Support for Machine Learning Frameworks</h3><p>At Trulia, we use several different kinds of machine-learning models and frameworks, both custom as well as open source. While building our inference engine, we wanted to create a generic core platform where different kinds of models can be easily deployed, thereby leveraging the common infrastructure and functionalities.</p><p>To enable this, we have created an abstraction that makes it easy for us to plug in support for new machine-learning frameworks and models while reusing common components. Currently, the system supports deployment of deep-learning models trained with <a href="http://caffe.berkeleyvision.org/">Caffe</a> as well as Gradient Boosted Tree models trained with <a href="https://github.com/dmlc/xgboost">XGBoost</a>, with support for <a href="https://www.tensorflow.org/">Tensorflow</a> and others coming soon.</p><p>Models deployed on this system can be run in either CPU or GPU mode. The system supports multiple versions of the models to be deployed as well the ability to combine multiple models to form composite models or pipelines.</p><h3>Optimizing Performance for Throughput and Latency</h3><p>One of the key challenges while deploying deep-learning models is to ensure effective usage of hardware resources, GPU in particular, to maximize overall throughput of the system. This includes both proper utilization of the GPU memory, as well as making sure the GPU cores are kept saturated most of the time.</p><ul><li><strong>Sharing GPU resources across Clients and Models<br> </strong>The system is designed to serve multiple models across different applications and clients at any given time. The prediction layer comprises several worker nodes each running one or more copies of the prediction layer, each capable of performing predictions for any of the supported models. Models share GPU/CPU memory and are loaded lazily, which means that GPU/CPU resources are only allocated when a model related task is first assigned to the worker process. Since multiple clients can be using our service at any given time, we wanted to make sure that compute resources are fairly allocated based on expected traffic and latency requirements of the clients. This is achieved by creating dedicated queues by either task, model or client type. For example, for lower latency synchronous tasks, we create queues with dedicated workers per task; while for high-throughput tasks, we define general purpose queues that are assigned a larger number of workers shared across multiple clients.</li><li><strong>Decoupling CPU and GPU Processing<br> </strong>Every model prediction pipeline comes with its own set of operations. Each of these operations are different: some are I/O bound, others are compute intensive; some run on CPU, while others on GPU. For example, in case of our image recognition pipelines, the inputs are raw URLs which need to be first downloaded, preprocessed (resized, mean subtraction, etc.), make model predictions and then push the prediction results to a callback. While the preprocessing (download and resizing) and the post-processing (pushing results to callback) operations are performed on CPU, the model predictions are computed by performing a forward pass on the GPU. Preprocessing operations like resizing can be significantly expensive when compared to time taken to generate model predictions on a GPU. Furthermore, I/O bound operations like downloading URLs and pushing to callback can be an order of magnitude slower depending on network latency. Executing these operations sequentially would severely impact throughput as the GPU would be sitting idle most of the time waiting for CPU operations to complete. To handle this issue, we consider operations in a pipeline as nodes in a computational graph (Directed Acyclic Graph), where each node (i.e. operation) could be individually assigned to execute in either CPU or GPU mode. This decoupling allows GPU worker threads to only process GPU bound operations without waiting on CPU operations, thereby maximizing GPU usage. By implementing model pipelines as disjoint linked tasks and using CPU and GPU queues each with dedicated worker instances, we see significant improvements in throughput.</li></ul><p>Once the predictions are generated by the system, the data is streamed to the callbacks specified in the request. Dedicated consumers listen to these callbacks and publish them to downstream data stores like HDFS, S3, Solr, Redis, etc., in real-time.</p><p>This data is also joined with other sources, like property metadata, either as part of a batch <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a>/<a href="https://spark.apache.org/">Spark</a> job or in real-time and indexed into our data stores (e.g. Solr and Redis), which power our production APIs that surface this content to the front-end for rich consumer experiences. Note that the production APIs are different from our prediction APIs as they are serving live consumer traffic with static data, and have much lower latency requirements than the prediction service.</p><p>Deploying deep-learning models comes with its own sets of challenges some of which we have covered above. The system is constantly evolving as we continue extending it to broader applications and prioritizing features to add based on immediate business needs. Stay tuned for more from us on this throughout the year.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f584a628a6ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zillow-tech-hub/deploying-deep-learning-at-trulia-f584a628a6ae">Deploying Deep Learning</a> was originally published in <a href="https://medium.com/zillow-tech-hub">Zillow Tech Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Makes a Photo Click: Selecting Hero Images with Deep Learning]]></title>
            <link>https://medium.com/zillow-tech-hub/what-makes-a-photo-click-selecting-hero-images-with-deep-learning-24d213e92400?source=rss-c9fe594faafc------2</link>
            <guid isPermaLink="false">https://medium.com/p/24d213e92400</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[convolution-neural-net]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Shourabh Rawat]]></dc:creator>
            <pubDate>Fri, 08 Jul 2016 12:15:00 GMT</pubDate>
            <atom:updated>2018-12-20T21:28:25.072Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*hL1Y21i39aE-GX2XamqJDQ.jpeg" /></figure><p>Real estate photos come in all shapes and sizes. By this we mean that there is a lot of diversity in terms of both image quality and content. At <a href="http://www.trulia.com/">Trulia</a>, we know the first property photo we show a consumer is the most important one. It’s what we call the hero image and it’s the photo that represents a listing across Trulia on all platforms, and it needs to be aesthetically pleasing in order for people to want to click through for more information on the listing.</p><p><strong>How Trulia Traditionally Chose Hero Images<br> </strong>Each new property listing we receive at Trulia comes with a collection of photos. Historically, the first photo in this collection is selected as the hero image. But, we have found this is not always ideal. While some property listings come with high resolution and professionally staged photos, others contain low resolution and poorly taken photos, with artifacts like low brightness, blurriness, watermarks, etc.</p><p>We often find cases where the first photo is lower quality or has irrelevant content, and a better, more representative photo is available in the collection. In this case, surfacing that first photo would not only leave a sub-optimal first impression for the consumer, it would also likely reduce the chances of a consumer clicking through to get more information and eventually submit a lead. So, we decided to test a new method, leveraging image recognition and deep learning to help determine which photo in a listing’s collection should be selected as the hero image.</p><p><strong>Looking for a Change: Our Hypothesis and Methodology<br> </strong>We hypothesized that by selecting a more attractive and relevant photo from the collection to be the hero image, we would be able to improve the user experience while increasing the likelihood of properties getting clicked on. To test our hypothesis, we focused on Trulia Rentals and scored each photo in the property collection across three parameters:</p><ol><li><strong>Image Appropriateness</strong><br> Unlike image quality, image appropriateness can be easily defined in terms of artifacts that violate Trulia photo upload policies. This includes photos with prominent text and watermarks, advertisements, humans, animals, non-real estate content, etc. Like before, we trained another CNN to differentiate between appropriate and inappropriate photos using the curated dataset maintained by our quality control team.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*Bgp-KGZY57loDH7vP4jgvg.jpeg" /><figcaption><em>Examples of different types of real estate images</em></figcaption></figure><p><strong>2. Image Quality</strong><br> Image quality is subjective, but for the purpose of selecting our hero images, we want photos that are staged and professionally captured, high resolution, and contain luxury elements, like chandeliers, fireplaces, etc., to score higher, while images that are low resolution, clutter, etc., score lower. One of the challenges in training such a visual model for image quality is collecting a large enough labeled dataset of high and low-quality images. To overcome this we resorted to using predictions from another machine learned attribute-based model that categorizes properties as either “Luxury” or “Fixer,” based on features like price, keywords, etc. When we did this, we found the homes scored as “Luxury” often had high resolution and professionally taken photos of attractive spaces, while “Fixer” homes usually had low resolution and poorly captured photos. This matched our above definition for image quality and we trained a Deep Convolutional Neural Network (CNN) to learn to differentiate between photos from luxury homes (positive class) from photos from fixer homes (negative class).</p><p><strong>3. Image Relevance<br> </strong>Another key aspect in determining the hero image is understanding which scene or room type the photo belongs to, ensuring the content is relevant and representative of the property. For example, it might make more sense to prefer an indoor photo of the property to an outdoor view of the neighborhood. Similarly, a kitchen photo might appeal more than a bathroom photo. By using our scene classification model (again a CNN), we enable our product team to weigh certain scenes more than the others based on their criteria of relevance. For each of the above models, we used CNN architecture very similar to the GoogleNet model that won the ILSVRC 2014 competition. All the above deep networks were trained using <a href="http://caffe.berkeleyvision.org/">caffe deep learning framework</a> on a Titan X GPU card. Finally, each photo in the property is scored using the above three models and a final score is computed as a weighted sum. The photo with the highest score is selected as the hero image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*V3QGugARWXmojy_DNzzJcA.jpeg" /><figcaption><em>Trulia hero image selection pipeline</em></figcaption></figure><p><strong>The Results<br> </strong>We conducted thorough AB tests and analysis to compare our hero image selection algorithms against the existing method of selecting the first photo as the hero image. We found significant improvements in consumer engagement as well as consumer and landlord contact rates on all Trulia Rentals platforms, thereby confirming our hypothesis. Not surprisingly, the improvements were most pronounced on the mobile app and web platforms, which incorporate more of the image photo when displayed in the UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*hL1Y21i39aE-GX2XamqJDQ.jpeg" /><figcaption>Trulia hero images, before (left) and after (right) implementing a hero image selection algorithm.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24d213e92400" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zillow-tech-hub/what-makes-a-photo-click-selecting-hero-images-with-deep-learning-24d213e92400">What Makes a Photo Click: Selecting Hero Images with Deep Learning</a> was originally published in <a href="https://medium.com/zillow-tech-hub">Zillow Tech Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>