<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zachary Jablons on Medium]]></title>
        <description><![CDATA[Stories by Zachary Jablons on Medium]]></description>
        <link>https://medium.com/@zmjjmz?source=rss-4e8c0fa5592------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*r8EKIrM63IMbFfLd.</url>
            <title>Stories by Zachary Jablons on Medium</title>
            <link>https://medium.com/@zmjjmz?source=rss-4e8c0fa5592------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 20:38:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zmjjmz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Large-scale collaborative filtering to predict who on OkCupid will like you*, with JAX**]]></title>
            <link>https://tech.okcupid.com/large-scale-collaborative-filtering-to-predict-who-on-okcupid-will-like-you-with-jax-88ac8a934044?source=rss-4e8c0fa5592------2</link>
            <guid isPermaLink="false">https://medium.com/p/88ac8a934044</guid>
            <category><![CDATA[jax]]></category>
            <category><![CDATA[collaborative-filtering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Zachary Jablons]]></dc:creator>
            <pubDate>Tue, 02 Feb 2021 16:49:29 GMT</pubDate>
            <atom:updated>2021-03-29T22:34:44.626Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VPpMiL342AhO0eRT.jpg" /><figcaption>A picture of Jacksonville, FL. No relation.</figcaption></figure><p><em>* your likeability not guaranteed, but we will try</em></p><p><em>** JAX is experimental and not an officially supported Google product. Version lock your code with containerization where possible.</em></p><h3>Brief note on terminology</h3><p>On OkCupid, when you tell us who you like and who you, well, don’t like, we record that on the backend as ‘votes’. For the rest of this, we’ll be talking about ‘voters’ and ‘votees’ to refer to you and the people you’re (not) liking, respectively. Each like or pass is a ‘vote’.</p><p>We record a lot of votes!</p><h3>Motivation</h3><p>(If you already know about collaborative filtering and SVD approximations, feel free to skip to the Implementation section)</p><p>We want to be able to tell, given your history of passing and liking, who might you like that you haven’t seen yet? If we have this, we can leverage it to show you people who you are more likely to like, meaning that you like them, they get a like, maybe they like you, and hopefully it all goes great from there!</p><p>The first step to understanding how we’re going to handle this is to put all these votes into an ‘interaction matrix’. Essentially, the rows of this matrix are the ‘voters’ and the columns are the ‘votees’. When a voter likes a votee, we put a 1 at the coordinate corresponding to that (voter, votee) pair and we put a 0 if the voter passes on a votee.</p><p>So let’s say we have voters A, B, and C, and we’ve recorded some votes from A on B and C, a vote from B on A, and a vote from C on B.</p><p>Our interaction matrix now looks like this, with voters as the rows and votees as the columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*1IBpCP_LZZ7J1nEjtplaQQ.png" /><figcaption>Ignore the weird formatting</figcaption></figure><p>Simple enough, and now our task is to try and figure out how to ‘fill in’ the question marks here, e.g. how would C vote on A?</p><p>So, how do we do this? Well we could try looking for a similar voter to C based on who they’re liking and passing on. Once we’ve done this, we can take those similar voter’s votes and extrapolate them to the missing votees for our voter.</p><p>This is known as memory-based collaborative filtering, and while good in theory, it is computationally not very scalable. When we have millions of voters and votees making hundreds of millions of votes each week, it’s very difficult to do this when a user needs a list of recommendations pronto!</p><h3>Learning Representations instead</h3><p>So, let’s take an alternative approach, and approximate these similarities.</p><p>To do this, we’ll represent each voter and votee with a vector. A vector, for our purposes, is just a fixed-length array of numbers. What we’ll want from these vectors is that when we take the dot product of them (i.e., multiply each number with the corresponding number in the other vector, take the sum) it corresponds to the outcome we observed (or, would observe) when the voter interacts with the votee.</p><p>So, we would want e.g. dot(A_vector, B_vector) = 0 and dot(A_vector, C_vector) = 1. An important note here is that, as you&#39;ll notice, we want dot(A_vector, B_vector) != dot(B_vector, A_vector). This doesn&#39;t actually work (the dot product is commutative), so we&#39;re going to record two vectors for each user - a votee vector (A_votee_vector) and a voter vector (A_voter_vector).</p><p>With this, we’ll have dot(A_voter_vector, B_votee_vector) = 0 and dot(B_voter_vector, A_votee_vector) = 1.</p><p>Sorry B, A’s just not that into you.</p><p>Ok, so now we’ve established that we want vectors. Vectors have uh, numbers in them. But, how do we figure out which numbers, exactly? There are so many numbers we could fill them with!</p><p>Well, we can learn the numbers from the votes we’ve observed! We’ll do this using Gradient Descent, which can be described as follows:</p><ul><li>Randomly initialize everyone’s voter and votee vector</li><li>For some number of ‘epochs’ (passes over the observed votes), we’ll go through each vote we’ve observed, and</li><li>* Compute the dot product between the voter’s voter vector and the votee’s votee vector</li><li>* Compute the difference (error) between the dot product and the actual outcome</li><li>* Take the <strong>gradient</strong> of the error</li><li>** This tells us how much each number in the vector contributed to the error</li><li>** We can use this as a ‘direction’ to move the vector</li><li>* Move opposite that direction by subtracting the gradient from the vector</li><li>** This will reduce (<strong>descend</strong>) the error</li><li>** Notably, we don’t have to do this vote-by-vote, and instead can do a bunch at a time by <strong>batching</strong> these computations.</li><li>Return the vectors we’ve learned for each voter and votee.</li></ul><p>(Author’s note: Medium doesn’t have nested lists, the original platform does, I apologize)</p><p>So, once this is done, what can we do with these vectors? Well now, when we want to know if a voter will like a votee, we can simply take the dot product of their respective vectors and find out what the model thinks is going to happen!</p><p>This is a very fast operation that we can offload to <a href="https://tech.okcupid.com/vespa-vs-elasticsearch/">Vespa</a> to provide the recommendations, and it doesn’t get slower the more users we have!</p><p>But why did we do this process in particular? I’ve skipped a lot of the process, but essentially this whole process is derived from approximating a Singular Value Decomposition (SVD) to reconstruct the voter-votee interaction matrix. If you’re interested in reading more into that part, I’d recommend you read the seminal <a href="https://sifter.org/~simon/journal/20061211.html">Netflix prize post</a> that a lot of this work is derived from.</p><p>For simplicity, we’re omitting things like biases, regularization, shuffling, and so on that we do use in practice.</p><h3>Scale issues</h3><p>While this is a lot faster when we want to figure out if a user will like another user, the training process still requires us to loop through and do a gradient update for each vote. While these gradient updates are relatively cheap, they add up with hundreds of millions of votes to train on!</p><h3>Implementation</h3><p>So, we need to make this happen quickly. In order to keep everyone’s recommendations fresh, we want to compute (and recompute) these vectors every day. Additionally, we don’t want to be stuck with just this simple operation — we want to be able to try out and research different ways to update the vectors using different error functions, training methods, optimizers, and so on.</p><p>We’ve encountered quite a few libraries out there that implement this basic algorithm. Below, we’ll introduce our old favorite, and then talk about why we built our own approach using JAX.</p><h3>The Baseline — Surprise</h3><p>When we first started this project, we used the popular <a href="https://github.com/NicolasHug/Surprise">Surprise</a> library by Nicolas Hug. While Surprise provides many recommendation algorithms, we specifically are interested in its implementation of <a href="https://surprise.readthedocs.io/en/stable/matrix_factorization.html#surprise.prediction_algorithms.matrix_factorization.SVD">SVD</a>. Despite being a Python library, it implements all of the critical parts using fancy C-level Python extensions. This makes it a lot faster than pure Python.</p><p>However, if we come up with some complicated new error function, we would have to manually compute the corresponding gradient function. Aside from sheer laziness, we want to avoid this is as it’s a process that can introduce a lot of difficult to diagnose bugs and hard to maintain code.</p><p>Plenty of frameworks out there are built to automatically compute gradients (e.g. TensorFlow, PyTorch), but can we attain this flexibility while maintaining the efficiency of Surprise?</p><h3>What is JAX? Isn’t that in Florida?</h3><p><a href="https://github.com/google/jax">JAX</a> provides a simple Python interface for expressing (with a numpy-like API) the operations we want, while also allowing us to convert these functions into their gradient-returning versions!</p><p>It’s much more than that as well, as it provides a number of ways to optimize our code using Just-In-Time-compilation and vectorization, among other neat features. If you’re interested, I recommend reading up on their <a href="https://github.com/google/jax/tree/543dcb37e357783b71dd0b5dc69de5d68d03f591#what-is-jax">documentation</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb4a5d68a24a56da37a1f07751182afd/href">https://medium.com/media/bb4a5d68a24a56da37a1f07751182afd/href</a></iframe><p>So let’s define the function that does our dot product. Simple enough.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5511e73e50c0677b3016ec0666e8417/href">https://medium.com/media/d5511e73e50c0677b3016ec0666e8417/href</a></iframe><p>And now let’s define our error function. For now, we’ll be using the squared <a href="https://en.wikipedia.org/wiki/Euclidean_distance">Euclidean distance</a> between the real outcome and the predicted outcome. This is commonly known as L2 loss.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2f23ac22926ab38a9e02325cbfaadaf/href">https://medium.com/media/e2f23ac22926ab38a9e02325cbfaadaf/href</a></iframe><p>Now, we’ll average the loss over our batch of votes that we’re training on using the function transformation to get our final loss function for the batch. jax.vmap is used here to allow us to efficiently run the function on every row of the inputs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9790e0a096a1dccd42686b63ef07633e/href">https://medium.com/media/9790e0a096a1dccd42686b63ef07633e/href</a></iframe><p>This allows us to give two matrices, each of shape (batch_size, vector_size) (for the voter and votee vectors), as well as a vector of shape (batch_size,) (for the real outcomes) to the loss function. This function will then return a scalar value that represents, on average, how &#39;wrong&#39; the vectors currently are when predicting the outcomes.</p><p>So, now how do we get the gradient? We’ll use the jax.grad function as so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2f7827a8d6fc99b02ca443e5503f529/href">https://medium.com/media/f2f7827a8d6fc99b02ca443e5503f529/href</a></iframe><p>This will produce a function that, when given the same arguments as the loss function, returns two matrices of gradients corresponding to the voter_vectors ( argnum 0) and votee_vectors ( argnum 1). Additionally, we&#39;re running all of these functions through jax.jit so that we can take advantage of faster executions once they&#39;re compiled!</p><p>So, now that we have all of these, let’s put together our training loop.</p><p>For our data, we’ll have votes in the form of a tuple as so:</p><p>(voter_index, votee_index, real_outcome)</p><p>These indices are mapped (beforehand) to correspond voters and votees to rows in the associated voter_matrix and votee_matrix.</p><p>So our training loop looks something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/da37942519bfa0f50db991f67fb58597/href">https://medium.com/media/da37942519bfa0f50db991f67fb58597/href</a></iframe><p>Let’s call this approach jax_naive. So, how does this do?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*RXzWayS9vGEn4HdC.png" /><figcaption>ignore the shuffle stuff, that didn’t make the cut</figcaption></figure><p>Yikes! Even for a small amount of votes we’re spending way more time than Surprise! Can we do better?</p><h3>Gotta go fast(er)</h3><p>Ok, well, that clearly did not go well. If we think back to it, can we try and use JAX’s numpy-like API to JIT-compile a lot of the indexing and update parts? Would that be faster?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6edccfcf0598aa3941510340a735cdc2/href">https://medium.com/media/6edccfcf0598aa3941510340a735cdc2/href</a></iframe><p>Ok, cool. So, a few notes — what’s up with the jax.ops.index_add? Why are we assigning the result of that to new_voter_matrix?</p><p>One of the key limitations of JIT-compiling a JAX function is that we cannot have any <em>in-place operations</em>. This means that any function that modifies its input will not work. For more on why, see their <a href="https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#%F0%9F%94%AA-In-Place-Updates">documentation</a>.</p><p>How does this version, which we’ll call the amateur_jax_model, measure up?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*yEOYy8aZzaYMXfYN.png" /></figure><p>Welp, we’re still going way too slow. Surprise is beating us handily!</p><p>What else can we stuff into a JIT-compiled function?</p><p>I should note that we’re using a batch_size of 1 for these comparisons. This isn&#39;t because that&#39;s necessarily a good batch size, but it&#39;s because that&#39;s what Surprise allows for - so we want to make sure our comparison is fair.</p><h3>Gotta go fast(est)</h3><p>Right now we’re copying (remember, no in-place changes) the entire voter matrix and votee matrix into the JIT-compiled function every batch, even though we’re only updating (at most) batch_size rows! When we have millions of voters and votees, this can become expensive.</p><p>But what if we could stuff the entire train_epoch into one JIT-compiled function?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/953fb7af244ea9fdf17cae2997fe6d06/href">https://medium.com/media/953fb7af244ea9fdf17cae2997fe6d06/href</a></iframe><p>Wow, so, this looks quite different but at the same time quite similar! This is thanks to the <strong>experimental</strong> <a href="https://jax.readthedocs.io/en/latest/jax.experimental.loops.html">JAX </a><a href="https://jax.readthedocs.io/en/latest/jax.experimental.loops.html">loops API</a>, which allows us to write a stateful-looking loop that gets turned into an in-place (or, pure) function by JAX. All of our mutable state in that loop has to be placed in the loops.Scope object.</p><p>But once that’s done, we can compile it using jax.jit.</p><p>A few things get complicated here however, and the root of the problem is that batch_start keeps changing. This means that we can&#39;t use a typical slice operator like voter_indices[batch_start:batch_end] - we have to use lax.dynamic_slice instead. Additionally, we have to specify that batch_size and batched_dataset_size are static values, which means that JAX will recompile train_epoch if they change. So, when calling train_epoch we now have to compute how many batches will be in our training run. Not a big deal for simple gradient descent!</p><p>Ok, so, this is very complicated and took me like somewhere around a month of banging my head against the keyboard to get right, but how does this approach (which we’ll call the full_jax_model) compare?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*d4LNDcP7-f1ywC53.png" /><figcaption>the others aren’t even worth looking at at this point</figcaption></figure><p>Now that’s what I call speed! In fact, we’re seeing that as we hit tens of millions of votes, we’re not slowing down nearly as much as Surprise does! We couldn’t have even dreamed of getting to this level of comparison with the earlier versions of the model!</p><h3>Taking advantage of JIT compilation?</h3><p>While it’s an impressive speedup even with just one epoch, when we start to train for more than one epoch we should see an even more dramatic difference as train_epoch is compiled, and subsequent invocations after the first one should use the already-compiled version.</p><p>So, compared to Surprise, we should see JAX take significantly less extra time per epoch, right? Let’s look at how many times slower each one is after a few epochs compared to just one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/0*4qp13nR9sbikj9NU.png" /></figure><p>Well, not quite. As it turns out, relative to one epoch of training, the JAX version here slows down at a similar rate to Surprise, but is doing a bit better. In absolute terms however, we’re adding orders of magnitude less seconds per epoch!</p><h3>JAX land vs. numpy land</h3><p>One thing that hasn’t been made super clear throughout this is that train_epoch isn&#39;t returning numpy.ndarray objects, but instead jax.interpreters.xla.DeviceArray objects. While these have largely similar APIs, when profiling JAX models and their JIT-compiled methods, one has to be careful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/0*WW03J1poRsmZondO.png" /></figure><p>Fundamentally, while it may look like the computation is done when train_epoch (the inner portion) is called, the results you get back are actually futures. So, if we want to work with them in numpy land or print them or anything useful like that, we have to &#39;extract&#39; them. We can see that if we measure the time it takes to run train_epoch a few times on the same matrices, it looks like the amount of time taken doesn&#39;t increase with the amount of epochs!</p><p>However once we extract these, we can see the amount of time taken jump up. So, be careful when profiling your JAX code! You can read more on this async dispatch stuff <a href="https://jax.readthedocs.io/en/latest/async_dispatch.html">here</a>.</p><h3>Why not use TensorFlow or PyTorch?</h3><p>While TensorFlow and PyTorch are great for a lot of gradient descent driven problems, there are some key parts of this one that they do not handle well.</p><ul><li>Since we need to train a vector for every voter and every votee, we can’t downsample our training set, and thus need a fast way to iterate through it</li><li>Neither TensorFlow nor PyTorch offer optimizations for this</li><li>We’re also interested here in optimizing a lot of disparate parameters</li><li>Neural networks tend to optimize the same parameters every time</li><li>We’re dealing here with a small number of parameters updated at each step, but from a large matrix of parameters</li></ul><p>After considerable research and failed attempts on our end, we determined that there wasn’t a viable route for us to achieve this kind of performance in TensorFlow or PyTorch. However it’s been a year since we’ve looked at that, and if anyone knows otherwise feel free to let me know how wrong I am!</p><h3>Don’t try this at home</h3><p>Back in December 2019 when COVID-19 was ‘anomalous pneumonia’ and the JAX version was 1.55, I’d built out this method and gotten these great results.</p><p>This past week, when I was writing this post and realizing I had a few results I wanted to re-run and explore further, I had an awful finding. My timing results could no longer be reproduced! As it turns out, something changed in the JAX library (I think) that slowed down this method significantly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*RVZoXGJepuGoMsCA.png" /><figcaption>that doesn’t look good</figcaption></figure><p>We’ve <a href="https://github.com/google/jax/issues/5496">filed a ticket</a> for this, but in the meantime if you want to reproduce these results you can do so with the following versions of jax and numpy.</p><pre>jax==0.1.55<br>jaxlib==0.1.37 <br>numpy==1.17.5</pre><h3>Conclusion &amp; Further Work</h3><p>While this was definitely an improvement on our baseline, it is still a rather basic form of collaborative filtering for recommendations — but it’s worked out well for us so far, allowing us to train these vectors for every voter and votee seen in the past week over the entire OkCupid site within three hours or so on average!</p><p>We’ve leveraged this to provide significant improvements in recommendations for all of our users this year.</p><p>But, where do we go from here?</p><h3>Alternative Training Loops and loss functions</h3><p>There’s all sorts of interesting ways to order the training set other than just one after the other or random. Commonly, methods like BPR and WARP use the model’s current performance to select ‘hard’ examples that it’s not getting right. These methods may be helpful, and are a direction of future work for us.</p><h3>Neural Collaborative Filtering</h3><p>NCF is a promising approach as well, allowing us to take advantage of the oh so lauded <em>deep learning</em> to do a non-linear version of all of this. We were awaiting mainlined <a href="https://github.com/google/jax/issues/4600#issuecomment-765875459">pytrees support in </a><a href="https://github.com/google/jax/issues/4600#issuecomment-765875459">loops.Scope</a> however, which has only very recently been added.</p><p>Without that support, managing all of the disparate state that NCF requires will be nightmarish.</p><h3>Better optimization methods</h3><p>The method we have for turning the gradient into new vectors is fairly crude, and there are plenty of alternatives that show promise — but again, ultimately rely on or are made significantly easier by the aforementioned pytrees support.</p><p><em>Originally published at </em><a href="https://tech.okcupid.com/large-scale-collaborative-filtering-to-predict-who-on-okcupid-will-like-you-with-jax/"><em>https://tech.okcupid.com</em></a><em> on February 2, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88ac8a934044" width="1" height="1" alt=""><hr><p><a href="https://tech.okcupid.com/large-scale-collaborative-filtering-to-predict-who-on-okcupid-will-like-you-with-jax-88ac8a934044">Large-scale collaborative filtering to predict who on OkCupid will like you*, with JAX**</a> was originally published in <a href="https://tech.okcupid.com/">OkCupid Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The state of SageMaker: It’s a work in progress.]]></title>
            <link>https://tech.okcupid.com/the-state-of-sagemaker-its-a-work-in-progress-eb7c2849444e?source=rss-4e8c0fa5592------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb7c2849444e</guid>
            <category><![CDATA[complaining]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[sagemaker]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Zachary Jablons]]></dc:creator>
            <pubDate>Thu, 19 Apr 2018 14:59:09 GMT</pubDate>
            <atom:updated>2021-03-30T15:27:02.824Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*LvMxCIPagzOx68oo.png" /></figure><p>I have been working on evaluating AWS <a href="https://aws.amazon.com/sagemaker/">SageMaker</a> with TensorFlow, and integrating it into our Python heavy ML stack to provide the model training step. The idea is to avoid buying GPU heavy servers to train and deploy models, and to take advantage of co-located S3 buckets for training and data storage. What is below is a review from our experiences over the past quarter.</p><p>I want to preface it by saying that SageMaker was released 5 months ago. It is early in its lifecycle, and we expect it to continue to quickly improve. We are still planning on using SageMaker and building our ML infrastructure around it despite all of the below. A lot of the issues are minor or easily worked around.</p><p>Some of the following are pitfalls that might not be super obvious based on their documentation, and some are legitimate bugs, many of which are under their consideration. However, for anyone considering using SageMaker, take note of what we’ve run into:</p><h3>Performance</h3><p>It takes ~5–6 minutes to start a training job and an endpoint. The only ways to get predictions out of a model trained on SageMaker are to either download and recreate it locally or start up the endpoint (which is a TF serving instance afaict). I try not to keep endpoints running since they cost money (even if you’re not making predictions against them), and even if they’re cheap managing that is a bit more overhead than I’d like.</p><p>They’re not the best about keeping Tensorflow up to date — currently you can use 1.6. You can always specify newer versions in your requirements.txt, but that adds to the startup time.</p><h3>Inconveniences</h3><p>When you create a TensorFlow training job, the idea is that you give Sagemaker the code that you need to define a tf.Estimator (i.e. a model_fn, train_input_fn, etc.), and then it goes and does the training thing with the args you provide. If you have dependencies you can either specify them in requirements.txt (if possible) or include them in the source_dir, which is tar&#39;d up and sent over. One note about the train_input_fn: SageMaker seems to expect the signature of train_input_fn to be ()-&gt;(dict of features, targets), whereas the tf.Estimator constructor just wants the signature of something like <a href="https://www.tensorflow.org/api_docs/python/tf/estimator/inputs/numpy_input_fn">tf.estimator.inputs.numpy_input_fn</a>. This is a little annoying if you want to have a way to test these functions without hitting SageMaker.</p><p>Their TensorBoard wrapper (i.e. setting run_tensorboard_locally=True in the fit call) is hilariously bad. I&#39;d recommend instead just running TensorBoard yourself and pointing it at the S3 bucket where you store your checkpoints (it can do this by default, just give it the URI), however TensorBoard is a bit slow in doing this -- and you don&#39;t really know when it&#39;s done loading new summaries (that&#39;s not Amazon&#39;s fault).</p><p>Their &#39;local&#39; mode works great once you have the docker setup, but it&#39;s not obvious how to do so. Turns out it&#39;s tucked away in some <a href="https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/tensorflow_distributed_mnist/setup.sh">example somewhere</a>. Once that&#39;s all taken care of, you can just specify instance_type=&#39;local&#39;.</p><p>And, for the frustrated practitioner like me who just wants to know what the hell the TF container is doing: if you have the local mode container you can pull out that code from the container at /usr/local/lib/python2.7/dist-packages/tf_container. You&#39;re welcome.</p><p>It does however leave uncleanable root-owned files around after training. This is less than ideal, as one might imagine. Additionally it stores everything in /tmp unless you specify &#39;local&#39;:{&#39;container_root&#39;:&#39;/not/tmp&#39;} in ~/.sagemaker/config.yaml.</p><p>One of my favorite things about AWS is its detailed permissions system. Also, it’s one of the most annoying things, and it’s not easier in SageMaker.</p><p>Their Python SDK is weirdly incomplete in places. Like, you can’t delete a model through it, you’d have to grab your boto3 session and then call the API directly through it.</p><h3>Bugs</h3><p>Don’t update tensorflow-serving-api to version 1.7. It <a href="https://github.com/aws/sagemaker-python-sdk/issues/139">breaks</a> the sagemaker-python-sdk.</p><p>You know that thing about requirements.txt? Nope. Sorry. It&#39;s been in the documentation for like a month now. Doesn&#39;t seem to do <a href="https://github.com/aws/sagemaker-python-sdk/issues/154">anything</a> ¯_(ツ)_/¯.</p><h3>Limitations</h3><p>Ultimately there’s no way to ‘extend’ the Tensorflow container and still use it (afaict) with the Tensorflow SDK they provide since it will call upon the default Tensorflow image. You can of course write your own SageMaker containers but that’s a lot more work…</p><p>Apparently, there’s a tiny size limit on the hyperparameters you send to SageMaker to start a training job (256 characters). Seems a little insane to me, but here we are. Working with AWS on getting this fixed.</p><p>There’s no VPC support for training jobs. What this means for us is that we can’t have training jobs pull data from our database since we’d need to setup a VPC pairing to do so. Instead I currently have a job that pulls and processes the data elsewhere then uploads it to the appropriate S3 bucket. The fix according to AWS Support is to store the hyperparameters in a channel much like how you’d store model initialization stuff. This is awkward but works.</p><p>Passing training / evaluation hooks is basically impossible, since you need to serialize anything you’re sending over. You can have them put into the EstimatorSpec that your model_fn returns, but since that&#39;s called like... everytime you restore checkpoints, if you need the hook to preserve state it gets super awkward real quick.</p><h3>What does work</h3><p>It’s really easy to stand up an endpoint from the SDK given just the information needed to create the model.</p><p>GPU training is easy-peasy — just tell it to use a GPU instance. You have to get a service limit increase to use more than one however.</p><p>Distributed training is also easy: just tell it to use more instances!</p><h3>Other notes</h3><p>It seems to store the tar’d source dirs on a per-run basis in an S3 bucket I never told it to use (and keeps telling me it’s creating that bucket when it runs). Actually looking at the code this is a pretty minor bug: in some cases it just asks for the default bucket rather than use the provided one.</p><p>The tf.Estimator framework with train_and_evaluate (not <em>really</em> a SageMaker problem, but that&#39;s what they use for the TF container as of 1.6) is in my opinion awkward to use, and has some large holes (like early stopping is kind of impossible).</p><p>The README.txt on the sagemaker-python-sdk repo is much better documentation than like, the actual AWS documentation for SageMaker. I&#39;d go to that first.</p><p>One last thing — the order of responsiveness I’ve found for support channels has been:</p><ol><li>Our actual customer support line.</li><li>Filing an issue against their SDK’s Github repo.</li><li>The SageMaker developer forum. Don’t even bother with this one.</li></ol><p>Hope this helps anyone out there looking to start doing their ML training in <a href="https://youtu.be/t8sl2uLmT5o?t=71">the cloud</a>!</p><p><em>Originally published at </em><a href="https://tech.okcupid.com/the-state-of-sagemaker/"><em>https://tech.okcupid.com</em></a><em> on April 19, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb7c2849444e" width="1" height="1" alt=""><hr><p><a href="https://tech.okcupid.com/the-state-of-sagemaker-its-a-work-in-progress-eb7c2849444e">The state of SageMaker: It’s a work in progress.</a> was originally published in <a href="https://tech.okcupid.com/">OkCupid Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating Perceptual Image Hashes at OkCupid]]></title>
            <link>https://tech.okcupid.com/evaluating-perceptual-image-hashes-at-okcupid-e98a3e74aa3a?source=rss-4e8c0fa5592------2</link>
            <guid isPermaLink="false">https://medium.com/p/e98a3e74aa3a</guid>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[hashing]]></category>
            <category><![CDATA[perceptual-hashing]]></category>
            <category><![CDATA[spam]]></category>
            <dc:creator><![CDATA[Zachary Jablons]]></dc:creator>
            <pubDate>Wed, 31 May 2017 15:26:17 GMT</pubDate>
            <atom:updated>2021-03-29T21:51:00.495Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AWeMvkbs8D_vDEmT.png" /></figure><p>At OkCupid, we sometimes run into users and bots that re-use the same photos we’ve seen before. While sometimes we can simply check if a photo has been seen (and banned) before, we often run into the case where a banned photo has been manipulated slightly — so an exact comparison won’t work. Even a slight shift in the image would completely ruin an exact match. This is a problem we identified years ago, and we had a partial solution using a simple perceptual hash function. However over the years we’ve become familiar with the failings of the current system. The need for a more robust approach magnified once we made photos a requirement for new users.</p><p>There are a few ways to approach this problem, however the most common way is using a perceptual hash. For the uninitiated, a hash in general is (according to the almighty <a href="https://en.wikipedia.org/wiki/Hash_function">Wikipedia</a>) a “function that can be used to map data of arbitrary size to data of fixed size”, e.g. turning an image into a bitstring that represents that image.</p><p>Traditionally this means a cryptographic hash (e.g. MD5, SHA-256), in which the slightest change in the input will produce a drastic change in the output. However, for our purposes, we want nearly the opposite — a slight change in the input (like, shifting or rotating the image a bit) should produce little to no change in the output. This is where the idea of a perceptual image hash comes in.</p><p>Ultimately a perceptual image hash seeks to stay largely <a href="https://en.wiktionary.org/wiki/invariant"><em>invariant</em></a> in its output to slight changes in the input, while producing a small (i.e., manageable) output. Armed with these sorts of functions, we should be able to check if a photo uploaded to the site is the same one that’s just been slightly changed in an attempt to get around exact image checks.</p><p>The rest of this post details an exploration of four different perceptual hash algorithms that fit this format, under what transformations they fail to stay invariant and which ones perform better overall. While we already had a perceptual hash system in place, it’s time for an upgrade!</p><h3>Evaluation</h3><p>For the rest of this blog post we’ll be discussing perceptual image hash functions that take an arbitrary image and produce a fixed size bitstring. This allows us to just use the <a href="https://en.wikipedia.org/wiki/Hamming_distance">Hamming distance</a> between bitstrings in our evaluations. Theoretically a perceptual hash could produce a dense representation of an image (i.e. a vector of floats), but that makes comparisons more complicated.</p><p>There’s two main things we evaluate here:</p><ol><li>Image Transforms (or attacks), i.e. what transformations would attackers apply to the images in order to try and fool a detection system</li><li>Perceptual Hash function accuracy</li></ol><p>The goal of this evaluation is to determine how well perceptual hashes differentiate between entirely distinct image pairs and image pairs which are just an image and its transformed version.</p><h3>Evaluation Overview</h3><p>We set this up as measuring distances in two cases:</p><ol><li>Positive case — the distances between images and the transformed versions of those images</li><li>Negative case — the distances between distinct original images</li></ol><p>We work with a sample of about 10k images on the site to balance getting this done in a reasonable amount of time and statistical significance, and from this sample we run 10 trials of each case on these images. For the negative case we actually sample 1k image pairs (and get their hash distances), whereas for the positive case we just evaluate a ton of transformations and their distances and then sample them down to 1k examples. This gives us a sense of how reliable each metric is — although generally we didn’t observe much significant variation.</p><p>One important thing to add here is that these images are deliberately filtered to make sure none of them came from spammers to prevent re-used spam photos from being treated as distinct image pairs. Additionally, we take the extra step of removing any non-unique photos (by comparing their exact hashes).</p><h3>Transformations</h3><p>We identified five typical image transformations that our hardworking, tireless moderation team sees most commonly when dealing with spammers.</p><ul><li>Rotation (about the origin)</li><li>Crop</li><li><a href="https://en.wikipedia.org/wiki/Gamma_correction">Gamma correction</a></li><li>Adding a border (or reverse crop)</li><li>Salt &amp; Pepper noise</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*pLrQal91Ao0E7S5OIFKo3g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*l9QKlI4KOGcvbo2b.jpg" /></figure><p>Each of these transformations have a parameter that controls the intensity of the operation — e.g. how many degrees to rotate. For simplicity of analysis, we specified three buckets of each of these parameters: mild, medium, and 🔥 <strong>extreme</strong> 🔥. Whenever a transformation is applied, the bucket is specified and the parameter is uniformly sampled from that bucket (see the table above for those buckets). In most of these analyses, each transformation / bucket is computed three separate times because more data is always better.</p><p>But wait there’s mirr..ore! For each transformation application we flip a metaphorical coin and mirror the transformed image across its vertical axis. We skip the horizontal axis since it’s just not seen as much in the wild — one might imagine that upside down pictures are a pretty obvious spam giveaway.</p><h3>Perceptual Hash functions</h3><p>We won’t go too much into the specifics of each perceptual hash function in this post since they’re well detailed in the references provided. For simplicity we limited ourselves to evaluating four common perceptual hash functions:</p><ul><li><a href="http://www.hackerfactor.com/blog/index.php?/archives/529-Kind-of-Like-That.html">Difference Hash</a> (denoted diff_hash or ih_dhash)</li><li><a href="http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html">Average Hash</a> (denoted average_hash or ih_avghash)</li><li><a href="http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html">Perceptual (DCT) Hash</a> (denoted dct_hash or ih_phash)</li><li><a href="https://fullstackml.com/wavelet-image-hash-in-python-3504fdd282b5">Wavelet Hash</a> (denoted wavelet_hash or ih_whash)</li></ul><p>The implementations used here are all provided by Johannes Buchner’s excellent and convenient Python <a href="https://github.com/JohannesBuchner/imagehash">ImageHash library</a>.</p><h3>Normalized Hamming distance</h3><p>Since each hash function is reducing the image to a bitstring, we use the Hamming distance to compare them. This is pretty much a fancy way of saying that we just measure how many bits are different between each hash.</p><p>In order to compare hash functions that produce different size bitstrings, we also normalize against the length of the bitstrings produced (given that we’re not comparing bitstrings between hash functions, which wouldn’t make a lot of sense anyways). So for example, when you see the histograms further down the line take note that the distances (i.e. the x-axis) are from zero to one. Effectively this is the ratio of bits that are different between the two bitstrings to the length of those bitstrings.</p><h3>Metrics</h3><p>How do we measure the quality of a perceptual hash function? The primary paradigm that we’ll evaluate these functions against is by looking at the distributions of distances in the positive and negative cases for specific scenarios. Essentially, what we want is for the distribution (or histogram) of distances in each case to be fairly separate. There are a few ways to approach this, but we’ll just consider three metrics:</p><ul><li><a href="https://en.wikipedia.org/wiki/Earth%5Fmover&#39;s%5Fdistance">EMD distance</a> (or Wasserstein distance as all the deep learning people mysteriously started calling it). We want this to be large.</li><li>Intersection-over-Union (IoU) of the two histograms. We want this to be pretty small.</li><li>Maximal <a href="https://en.wikipedia.org/wiki/F1_score">F1-score</a> at an optimal distance threshold. We want this to be real close to 1.</li></ul><p>EMD may be the most consistent measure here since, say you have two histograms that are separate and want to compare them to two histograms which are also separate but further apart. Under the optimal F1 score and the IoU, both cases would already be optimal — however EMD would favor the latter case.</p><p>We bring in the F1-score to give us a threshold, which is discussed a bit more later. The easiest way to think of this is sliding that threshold along the x-axis, and finding that which separates the histograms optimally.</p><p>The histogram IoU is provided as an extra comparison, and basically measures directly how much each histogram overlaps with the other.</p><h3>Results</h3><h4>General Comparison</h4><p>Let’s say we set it up so that we’re evaluating all of the parameter buckets, all of the hashes, and all of the transformations. In this scenario — if you’re looking for a quick answer here — the best options <em>according to the metrics</em> appear to be either the average or wavelet hashes, each performing about the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*xR7VPFSjT-Q_zZctxT8zmA.png" /></figure><p>However, this can be deceiving — and the histograms for each hash function evaluated illustrate why (less overlap is better).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*UfH8cfGAmdobnPoZ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*0_KjqHQ--mhOsl2W.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*D_1tSHujjGrLN3kg.png" /></figure><p>As a baseline, look at the distribution of hashes when you use a cryptographic hash (which was very helpful to look at when debugging this evaluation).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*Hwfm9juA-Ftvvr3Z.png" /></figure><p>We can see in this case that a cryptographic hash produces completely overlapping normal distributions centered around 0.5, which is as expected. I would then advise interpreting all of the other histograms in this light — the negative class <em>should</em> look like that distribution, and the positive class would ideally be entirely shifted to the left.</p><p>So at first glance you can see why the metrics favor the wavelet and average hashes — it’s pretty clear that both the perceptual and difference hashes have a more significant overlap of negative and positive distances. However, we have to consider the actual application — if we want to find the closest hash to a given picture’s hash, we want to be as insulated from false positives as possible. This means that the spread of negative distances seen in the wavelet and average hashes could produce more false positives in a case where we’re trying to find the closest match to a given image!</p><p>So to take away some main findings from this evaluation with everything turned on (i.e. the hardest case), we see that for the positive case:</p><ul><li>dct_hash and diff_hash have bimodal positive distance distributions, and are clearly fooled by something we&#39;re doing, since one of the modes is near the expected random distance.</li><li>wavelet_hash and average_hash have what looks like sort of long-tail distance distributions that have less mass as distance increases, implying that less transformations &#39;confuse&#39; them.</li></ul><p>And for the negative case:</p><ul><li>dct_hash and diff_hash both have tight distributions around the 0.5 mark, behaving similarly to the cryptographic hash case.</li><li>wavelet_hash and average_hash both have distributions that, while normal and centered around the expected random distance, spread pretty far in both directions. I don&#39;t have a great answer for why something like this happens, but my best guess is that by capturing the information that they do, they&#39;re somehow actually doing a better job of checking a higher level of image similarity (e.g. similar color palettes). This is probably worth investigating further.</li></ul><p>Based on the above, I think it’s fair to categorize dct_hash and diff_hash as (relatively) high precision, low recall perceptual hashes, and wavelet_hash and average_hash as high recall, low precision hashes.</p><h4>What transforms do certain hashes fail at?</h4><p>So let’s say we’re incredibly concerned with that spread of negative case distances, and want to know what sacrifices we’re going to have to make to use dct_hash or diff_hash instead.</p><h4>Invariance to Mirroring</h4><p>The first thing we might want to accept that we can’t really handle is the mirror operation, as shown by comparing the median distance in the positive case when the transformed image is mirrored to when it is not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*ZtQgTuKn1IolTRZWN4UYQQ.png" /></figure><p>It appears that, while every hash does significantly better when the transformed image is unmirrored, diff_hash and dct_hash appear to be entirely fooled with their median distance approaching that of the expected random distance.</p><p>So with this, and the fact that you can only really mirror an image once, we can accept a lack of mirroring. Without mirroring, we get histograms that look more reasonable for those two low recall hashes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*uJa30uXOo0Z1ZBdm.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*4dx4hhpX2fmhomzs.png" /></figure><p>With the mirroring removed however, the table below shows that the other hashes also get a significant metrics boost — but of course, the problem with the negative case distributions remains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*_hUJPs34jj78BK_FY4SiGw.png" /></figure><h4>Bringing down the spice level</h4><p>One other thing to consider here is that these evaluations are all run on all of the transform parameter buckets. What if we assume that none of our attackers want to rotate images beyond like 5 degrees or crop less than 95% of the image? And are allergic to mirrors?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*FxM2Nm7qPYy55qshXLnAnw.png" /></figure><p>Check out those metrics above! It looks like some of them have achieved <em>full separability</em>!</p><p>In this case, we can clearly see that diff_hash and dct_hash get a significant boost in their metrics. Since this (of course) doesn&#39;t change the spread of distances for the negative case, if we&#39;re only concerned with the mild transform case then we might have a decent threshold to work with that doesn&#39;t produce too many false positives!</p><p>For brevity we’ll focus on diff_hash for discussing this, since I&#39;ve found that dct_hash behaves similarly. Below is the distance histogram comparison for the mild case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*UbacKbdq1_A0M-dV.png" /></figure><h4>Thresholding</h4><p>We can use the distance threshold given to us by optimizing the F1-score to give us a cutoff (for those applications where we need one) such that any image pair whose distance lies below the threshold is considered the same, and any above is a distinct image.</p><p>If we look at the mild case, we notice that there are hashes that get nearly perfect F-1 score. This obviously only really applies to the mild case, but what transformation sacrifices do we have to make to use this threshold? Is it all transforms? Or do we just lose invariance to some of the more extreme transformations and not all?</p><p>So let’s look at this for diff_hash: when we take the threshold we get from only looking at the mild transformation parameters, what&#39;s the percentage of positive cases we end up treating as distinct image pairs for the rest of the transformations?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*KDAlkN954-O6YXxOVYIQ4Q.png" /></figure><p>Apparently most of the time we rotate beyond 15 degrees it completely falls apart (example below) — and the same can be said for cropping down to less than 85% of the image. It doesn’t do great at the medium case either, which for rotation is from 6 to 15 degrees!</p><p>We took a look at that failure chance for different hashes — average_hash and wavelet_hash fail in the same places, however to a lesser extent -- think 80% not 96% of the time.</p><p>Below is a quick example to illustrate what a 30 degree rotation looks like, using a picture of Alice — our director of support.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/0*7-VW9ZUuEVLcuZ1K.png" /></figure><p>Alice is to OkCupid like Tom is to MySpace.</p><h3>How long does this take anyways?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*aMInNP5gSx__Ke1uQVAcOA.png" /></figure><p>The slowest overall is the wavelet hash, with the rest of them pretty much clocking in at the same amount of time taken.</p><p>Just as a quick note — generally one of the slowest steps here is going to be resizing the image to the tiny hash size that most of these perceptual hashes use. This may seem weird at first, but it makes sense — every other operation is generally working on a tiny (by modern computer standards) 16x16 array, so even those transformations with a high time complexity can take next to no time, whereas even a well optimized resize will have to work with a much larger image. This can also explain the variance in time each hash takes, since we don’t resize images to a fixed size beforehand.</p><p>Nonetheless, each hash is pretty fast so it’s not too much of a concern, especially when it can be done asynchronously from any sort of image uploading.</p><h3>Other notes</h3><p>We investigated some other stuff that didn’t produce much worthy of its own section here, so here’s some quick notes instead:</p><ul><li>Increasing the bitstring length (or hash size) led to significantly worse results for most of the perceptual hashes, likely because it’s simply capturing too much detail.</li><li>For the wavelet hash I tried a bunch of different image scales and wavelet strategies, none of them really made any significant difference metrics-wise except for using Daubechies wavelets instead of Haar — which performed significantly worse.</li></ul><h3>Conclusions &amp; Future Work</h3><p>The main draw that we’ve gotten from this is that there really isn’t just one hashing algorithm that handles everything we need. It is likely that we’ll have to judiciously combine multiple perceptual hashes to get a workable solution. However, each of these hashes is rather cheap to compute (and I would argue could be further optimized), which means that computing multiple hashes per image is probably a viable basis to build an image similarity engine on.</p><p>One might, for example, use a low precision hash as an initial check followed by a high precision hash to separate out cases that might need review from those that are definitely the same image.</p><p>Some thoughts for future work:</p><ul><li>There are a ton of <a href="http://arxiv.org/abs/1306.4079">newer</a> <a href="http://dl.acm.org/citation.cfm?id=1194040">interesting</a> hashes that are worth consideration.</li><li>There’s quite a few more ways to transform images that we see in the wild (e.g. adding captions, superimposing over other images with transparency, compression artifacts) that should be evaluated.</li><li>It’s definitely a bit weird which hashes behave similarly to each other. There’s probably something meaningful going on there that’s worth looking into.</li><li>Evaluate / design a gating architecture as briefly described above</li><li>We could investigate what characteristics of given image pairs cause the low precision hashes to be unreliable / give them a low distance, and use that to determine which algorithm’s hash distances can be trusted for a given image pair.</li><li>The end-to-end approach of training a Convolutional Net to embed images such that the positive / negative case distances separate nicely is probably the cool new hypebeast way forward. This would have the added advantage of being able to handle whatever transformations you can think of at training time.</li></ul><p><strong>OkCupid is hiring</strong> <a href="https://www.okcupid.com/careers"><strong>Click here to learn more</strong></a></p><p><em>Originally published at </em><a href="https://tech.okcupid.com/evaluating-perceptual-image-hashes-okcupid/"><em>https://tech.okcupid.com</em></a><em> on May 31, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e98a3e74aa3a" width="1" height="1" alt=""><hr><p><a href="https://tech.okcupid.com/evaluating-perceptual-image-hashes-at-okcupid-e98a3e74aa3a">Evaluating Perceptual Image Hashes at OkCupid</a> was originally published in <a href="https://tech.okcupid.com/">OkCupid Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>