<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Drop Engineering - Medium]]></title>
        <description><![CDATA[Drop Engineering Blog - Medium]]></description>
        <link>https://medium.com/drop-engineering?source=rss----11c02d7aac33---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Drop Engineering - Medium</title>
            <link>https://medium.com/drop-engineering?source=rss----11c02d7aac33---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:38:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/drop-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[dbt integration with Airflow]]></title>
            <link>https://medium.com/drop-engineering/dbt-integration-with-airflow-33e070c20e20?source=rss----11c02d7aac33---4</link>
            <guid isPermaLink="false">https://medium.com/p/33e070c20e20</guid>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Dilmeet Malhi]]></dc:creator>
            <pubDate>Thu, 22 Dec 2022 17:51:06 GMT</pubDate>
            <atom:updated>2022-12-22T17:39:43.220Z</atom:updated>
            <content:encoded><![CDATA[<p>As <a href="https://www.joindrop.com/">Drop</a> continues to grow, it becomes increasingly difficult to deal with enormous amounts of data. Transforming, removing duplicates, organizing and filtering data are some of the challenges that the data engineers face on a daily basis, so how do we overcome them with relative ease and speed?</p><p>In todays fast-paced market, we constantly see new tools emerging that help accelerate data transformation processes but <a href="https://docs.getdbt.com/">dbt</a> is one such tool that has caught everyones eye.</p><h4>So, what is dbt?</h4><p><a href="https://github.com/dbt-labs/dbt-core">dbt</a> or data build tool is an open source tool written in python that is used as a part of the transformation layer in Extract-Load-Transform (ELT) pipelines. dbt helps improve and fasten all the data transformation processes by creating and modularizing data models which are written in SQL.</p><p>At Drop, dbt will be used to build dimension and fact tables, and write tests to ensure and preserve <strong>data quality</strong> in the process of building a more <strong>reliable </strong>data<strong> </strong>warehouse in Redshift.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6I8iEBTwipaZlvJq" /><figcaption><a href="https://www.startdataengineering.com/post/advantages-of-using-dbt-data-build-tool/">Image source</a></figcaption></figure><h4>Why do we want dbt?</h4><p>dbt has become a key component of the modern data engineering workflow and here’s why.</p><ul><li>dbt comes with an automated testing framework. There are multiple packages you can find that help with complex tests. You can also develop custom tests for your data using Jinja or SQL.</li><li>dbt has various materializations like views, tables and incremental tables. It optimizes the workflow for incremental models as we no longer need to code incremental logic in long running SQL scripts.</li><li>dbt automatically creates data documentation and definitions. It centralizes documentation into a consistent and consumable repository that can be accessed by anyone.</li><li>dbt has a surprisingly low learning curve. Anyone who can write SQL queries and has basic knowledge of python can write dbt models!</li></ul><h4>How to set up dbt with Airflow?</h4><p>At Drop, Airflow and dbt will be used together to achieve the goal of providing trustworthy data to internal teams while utilizing a common interface.</p><p>In this blogpost, we will walk through the integration of dbt with Airflow in detail. First, we will install dbt and run some basic dbt commands to get it to work. Then, we will walk through setting up dbt with Airflow and finally, write a basic DAG which would invoke dbt commands to run and test all of your models in the data warehouse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EU8L9pTGZilFpr4Z9kLiOA.png" /><figcaption><a href="https://www.datafold.com/blog/running-dbt-with-airflow">Image source</a></figcaption></figure><p><a href="https://airflow.apache.org/docs/apache-airflow/stable/index.html">Airflow</a> is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. Airflow’s Python framework allows you to develop dbt workflows. It provides a scalable environment to run your SQL dbt models. One of the key features of Airflow that we will be using is that it enables you to trigger models only after the necessary tasks have run successfully. It also has an interactive web interface that helps you monitor and debug your pipelines in case of failure.</p><h4>dbt set up in an existing airflow project</h4><ul><li>Create dbt project</li></ul><p>First step is to create your dbt repository which would include all of the data models, tests, sources and configurations. dbt has great tutorials on how to write data models and tests and you can find them <a href="https://docs.getdbt.com/docs/get-started/getting-started/overview">here</a>. The best practice would be to include requirements.txt file which would include all the packages to be installed to run dbt jobs like dbt-core and dbt-redshift.</p><p>You will need to set credentials to database in redshift in profiles.yml file to run dbt jobs successfully. Storing your credentials as environment variables is recommended.</p><p>Credentials can be set by using iam roles or by using username and password method. For the purpose of this demo, we will use username-password method and your profiles.yml should look like the below.</p><pre>Drop:<br>  target: dev<br>  outputs:<br>    dev:<br>      type: redshift<br>      host: hostname.region.redshift.amazonaws.com<br>      user: username<br>      password: password<br>      port: 5439<br>      dbname: dev<br>      schema: personal_schema<br>      threads: 4<br>      keepalives_idle: 240 # default 240 seconds<br>      connect_timeout: 10 # default 10 seconds<br>      # search_path: public # optional, not recommended<br>      ra3_node: true # enables cross-database sources</pre><ul><li>Create virtual environment</li></ul><p>Run the following command to create a virtual environment and install all the required packages.</p><pre>python -m venv .venv  <br>source .venv/bin/activate<br>pip install -r requirements.txt</pre><ul><li>At this point, you should be able to run dbt commands. Run the following command to test if the connection to redshift is successful.</li></ul><pre>dbt debug</pre><ul><li>The following commands run and test data models and create automated documentation. It is important to note that when you run these commands dbt creates a file called manifest.json. This file is very handy, as it has the name of every model, every test or assumptions, and the dependency relationships between the models!</li></ul><pre>dbt run<br>dbt test<br>dbt docs generate</pre><h4>How to orchestrate dbt jobs using Airflow?</h4><p>In order to build an analytics workflow in Airflow, we need to understand how it works. A workflow is represented as a DAG (a Directed Acyclic Graph) written in Python. It consists of tasks which behave as individual units, organized by considering all the dependencies. Here, each task will represent a dbt command to be run and Airflow will trigger each task when it is ready to run by taking the data flow into account. The workflow would ideally consist of updating staging tables and performing the transformation to create dimension tables followed by fact tables.</p><p>First step would be to move the dbt repository to the folder where Airflow is set up. At this point, the Airflow repository should include all of your DAGs and the dbt project.</p><ul><li>Initial set up</li></ul><p>If you have an existing Airflow docker container running, the requirements.txt file of the Airflow docker container needs to be updated to include dbt-redshift, dbt-core and airflow-dbt in order to run dbt commands. dbt-redshift package includes base models for Redshift and airflow-dbt package includes Airflow operators to provide easy integration with dbt. Once that is completed locally, rebuild the docker image.</p><ul><li>Building the DAG</li></ul><p>For the purpose of writing tasks in the DAG, I used a PYPI package <a href="https://pypi.org/project/airflow-dbt/">airflow-dbt</a>. It has the following <a href="https://github.com/gocardless/airflow-dbt">operators</a> which provide a predefined template to run your dbt commands with ease:</p><ol><li>DbtDocsGenerateOperator: Calls dbt docs generate</li><li>DbtDepsOperator: Calls dbt deps</li><li>DbtSeedOperator: Calls dbt seed</li><li>DbtSnapshotOperator: Calls dbt snapshot</li><li>DbtRunOperator: Calls dbt run</li><li>DbtTestOperator: Calls dbt test</li></ol><p>Each of the above operators accept the following arguments:</p><ul><li>profiles_dir: If set, passed as the — profiles-dir argument to the dbt command</li><li>target: If set, passed as the — target argument to the dbt command</li><li>dir: The directory to run the dbt command in</li><li>full_refresh: If set to True, passes — full-refresh</li><li>models: If set, passed as the — models argument to the dbt command</li><li>exclude: If set, passed as the — exclude argument to the dbt command</li><li>select: If set, passed as the — select argument to the dbt command</li></ul><p>You can run Airflow tasks by using the above dbt operators which provide a predefined task template to run various data models. A basic DAG should look something like this</p><pre>from airflow_dbt.operators.dbt_operator import (<br>    DbtRunOperator,<br>    DbtTestOperator<br>)<br><br>with DAG(<br>    &quot;dbt_basic_dag&quot;,<br>    start_date=datetime(2022, 11, 23),<br>    description=&quot;An Airflow DAG to invoke dbt runs to update and test tables&quot;,<br>    schedule_interval=&quot;@once&quot;,<br>    catchup=False,<br>    default_args=args<br><br>) as dag:<br>    dbt_run = DbtRunOperator(<br>            task_id=&#39;dbt_run&#39;,<br>            target = target_env,<br>            profiles_dir =profiles_dir,<br>            dir = project_dir,<br>            full_refresh = False,<br>            select = &#39;example_model&#39;, # Allows you to specify models to run<br>      )<br>    dbt_test = DbtTestOperator(<br>            task_id=&#39;dbt_test&#39;,<br>            target = target_env,<br>            profiles_dir =profiles_dir,<br>            dir = project_dir,<br>            select = &#39;example_model&#39;, # Allows you to specify models to run<br>      )<br><br>dbt_run &gt;&gt; dbt_test</pre><p>You can also run dbt commands by using bash operators. The following task will test your database connection using bash operator which ideally would be the first task you run in your DAG.</p><pre>dbt_debug = BashOperator(<br>        task_id=&quot;dbt_debug&quot;,<br>        bash_command=f&quot;dbt debug --profiles-dir {profiles_dir} --project-dir {project_dir} --target {target_env}&quot;,<br>    )</pre><p>Once the DAG is built, you can trigger the DAG manually on your local Airflow or trigger it by executing commands on the command line once you run the Airflow container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ClEIEd_B5Orqu4fkPTmc3A.png" /><figcaption>Airflow interface</figcaption></figure><p>Once you open the Airflow interface locally, you will be to check if the DAG has run successfully and your tasks should turn green! You can also verify if the data has been updated by checking your database.</p><p>In case of failure, Airflow has a mechanism to send alerts using automated emails or slack notifications. You will be able to see the logs on the interface which are very helpful to troubleshoot.</p><p>One of the challenges I faced was backfilling an incremental fact table. It contained billions of rows and a complex SQL logic so running a simple dbt run command took a lot of time and CPU usage. Luckily, dbt has another materialization called <a href="https://github.com/dbt-labs/dbt-utils#insert_by_period">insert_by_period</a> model which allows you to insert records into a table one period a time. We were able to backfill the table using yearly periods and once that was done, we proceeded with our incremental runs.</p><h4>Future Ideas</h4><p>With the help of manifest.json file which contains information about all of the models, we should be able to automate writing DAG files for all dbt projects and make the pipelines dynamic!</p><h4>Impact</h4><p>With dbt, we were able to produce a more reliable warehouse without needing to write complex queries to load incremental tables. With dbt’s additional functionality, we are able to test our tables for duplicates actively which we had no way of doing before. It posed as a big challenge while dealing with some large fact tables. With improved data lineage and traceability throughout the lifecycle all the way out to reporting, we now know which models connect to which which further allows us to refresh data upstream/downstream as needed with simple commands. We are also able to provide transparency and visibility into what the data is for and how it is produced in the age of data-driven decision making.</p><p>With it’s compatibility with Airflow, it is easy to orchestrate and schedule dbt jobs to provide usable data on a daily basis in an optimal fashion!</p><h4>Helpful Sources</h4><p><a href="https://www.datafold.com/dbt">Datafold</a> , <a href="https://www.getdbt.com/blog/">dbt labs</a> and <a href="https://docs.astronomer.io/learn/airflow-dbt">Astronomer</a> have multiple blogs explaining different ways on how to run dbt and Airflow together. I particularly found one such blog very useful and it is attached <a href="https://www.datafold.com/blog/running-dbt-with-airflow">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33e070c20e20" width="1" height="1" alt=""><hr><p><a href="https://medium.com/drop-engineering/dbt-integration-with-airflow-33e070c20e20">dbt integration with Airflow</a> was originally published in <a href="https://medium.com/drop-engineering">Drop Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running a Migration Making 800k API Calls Without Crashing Prod]]></title>
            <link>https://medium.com/drop-engineering/running-a-migration-making-800k-api-calls-without-crashing-prod-2757917e6e9?source=rss----11c02d7aac33---4</link>
            <guid isPermaLink="false">https://medium.com/p/2757917e6e9</guid>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[George Shao]]></dc:creator>
            <pubDate>Wed, 29 Jun 2022 14:34:32 GMT</pubDate>
            <atom:updated>2022-06-29T14:34:32.648Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U6Ac_MpbDuYg44bZV10qNA.jpeg" /></figure><p>Executing long-running scripts in production can be quite a time-consuming, inelegant task. At Drop, we previously had a repository of scripts that we would use by manually accessing the Kubernetes cluster through the terminal and running the script.</p><p>This was tedious and worked alright for a while, but soon we found ourselves with an issue that was near-impossible to solve with this method:</p><p>The main server ingests many requests per second, with 2/3 of them coming in from our third-party APIs, potentially causing latency spikes among other performance issues. These numbers would only continue to grow as Drop gained more users.</p><p>Currently, we have third-party APIs sending requests to the main server, which we then forward to the secondary server, which handles all of the request processing. An obvious improvement would be to set the third-party APIs to send requests directly to the secondary server, reducing the request load on the main server by 67%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Z_4kvpm-AGWkVxyhoXyYJA.png" /><figcaption>Before/After General System Diagram</figcaption></figure><p>The problem is that the webhook endpoint that we tell our third party APIs to send requests to, is configured on a per-user basis. In other words, we will need to make a massive number of requests to our third party APIs to update the webhook endpoint, one-by-one.</p><p>If we used our usual method to perform this task, <strong>it would take 10 days running 24/7</strong> in order to complete. As a result, we decided to switch to using the <a href="https://github.com/Shopify/maintenance_tasks">Maintenance Task gem</a> to streamline the process.</p><h3>What Are Maintenance Tasks?</h3><p>Shopify’s <a href="https://github.com/Shopify/maintenance_tasks">Maintenance Tasks gem</a> provides a Rails engine for executing long-running tasks, with many extraordinarily useful features. It also provides a nice dashboard to manage and monitor the progress and status of tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TbgX5koS67v8OvgR7iY3ug.png" /><figcaption>Shopify’s Maintenance Task Gem Dashboard</figcaption></figure><h3>Persistence Across Deployments</h3><p>Using the previous method of manually accessing the Kubernetes cluster and running the script here would block us from deploying any new back-end code to production for 10 full days, undermining our CI/CD efforts and drastically slowing development at Drop, wasting dev time.</p><p>By using the Maintenance Task gem, we were able to keep our tasks running across deployments without adding any additional logic. The following task ran for about 22 hours (during which we probably deployed to production quite a few times), but tasks can theoretically be run indefinitely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCnBOGmDfj6XKE3C3gXWmQ.png" /><figcaption>Maintenance Task Dashboard Showing Completed 22 Hour Task</figcaption></figure><h3>Working With CSV Files</h3><p>With our previous method of running scripts, if we wanted a task to loop over a CSV file, we would have to add the CSV file to the code repository itself (and merge and deploy it to QA, canary, then production Kubernetes clusters every single time the CSV file changes).</p><p>The Maintenance Task gem allowed us to upload a CSV file in the web dashboard, and have the process run for each item in the file. With the vast majority of our current scripts using CSV files, this will save a lot of time in the long run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*JBrFR18OhBbuVdfmCnzC6g.png" /><figcaption>“Upload CSV” Button on Maintenance Task Dashboard</figcaption></figure><h3>Customizing Runtime Variables</h3><p>Before we added error handling to our tasks, they would often crash every few hours. The Maintenance Task gem allows any user of the web dashboard to pass in custom arguments for each run of a given task.</p><p>We added the arguments processing_window_start and processing_window_end to define a processing window, especially useful to test the task on a small subset of items first, or to tell the task to continue processing from where it left off.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VXWkNFml3XJ1BM_q_3t_7Q.png" /><figcaption>Maintenance Task Dashboard Arguments</figcaption></figure><h3>Handling Errors</h3><p>While running our tasks, we encountered numerous errors including vendor-side issues with our third party APIs, networking issues when restarting Kubernetes pods, and database contention.</p><p>Unfortunately, there was no built-in or default behaviour to retry the task or resolve the error in another way. There were some built-in features that could be used to make this functionality ourselves such as the after_error callback and the throttling mechanism. However, having default error handling behaviour that can be overridden would be welcome.</p><p>To resolve these issues, we implemented our own custom error handling logic: we rescued the errors, <a href="https://nathanmlong.com/2015/02/exponential-backoff-in-ruby/">retried the process with linear backoff</a>, and logged the error if it persisted. This helped ensure that we could continue running the maintenance task theoretically indefinitely, avoiding temporary/rare issues from halting task progress, while still keeping track of what was causing errors that required additional developer intervention to fix.</p><h3>Speeding Up The Process</h3><p>After implementing error handling, the task was able to continue running across deploys for 24+ hours. However, the problem remained that it would still take <strong>10 days running 24/7</strong> to finish the task.</p><p>Unfortunately, the Maintenance Task gem <a href="https://github.com/Shopify/maintenance_tasks/issues/578">does not support instantiating multiple instances</a> of the same task to run in parallel, so we duplicated the task to create TaskA, TaskB, and TaskC, which we could run in parallel.</p><p>We divided the items that still needed to be run into 3 ranges of equal size and assigned each range to a task using the processing_window_start and processing_window_end arguments. This allowed us to run 3 of the same task in parallel, and <strong>we were able to finish 3x as fast</strong> (3-4 days instead of 10 days).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VmP6dqYM2GhFpVOgXovqIA.png" /></figure><h3>Logging and Metrics Collection</h3><p>We wanted to track the progress of our maintenance tasks, but there was no built-in method to collect &amp; display custom logs and metrics live.</p><p>As a result, we used a logging provider for logging as well as DataDog and StatsD for metrics collection, allowing us to figure out how many users had data being sent to legacy versus current webhook endpoints and ensure that our tasks were working correctly in real-time. We also created the unique_runtime_id argument in order to help differentiate between different runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VL8QiX_IsDJ7YFz9qqvxtg.png" /><figcaption>Incoming Third Party API Request Destination Count (from DataDog)</figcaption></figure><p>Above is a screenshot from DataDog showing the number of requests going to the main server (light blue) and the secondary server (dark blue) stacked on top of each other. Over time, you can clearly see a visible change as incoming requests change from going through both servers (both light blue and dark blue) to directly to the secondary server (only dark blue).</p><h3>Impact</h3><p>In the end, we <strong>reduced the traffic load</strong> <strong>coming into the main server</strong> <strong>by 67%</strong>, opening up more resources to handle more customer traffic on the app, and <strong>reducing request latency by 80%</strong>. We also created a <strong>cleaner separation of logic</strong> between the main server and the secondary server. Without the Maintenance Tasks gem, this process would have been far more complicated and inefficient.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2757917e6e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/drop-engineering/running-a-migration-making-800k-api-calls-without-crashing-prod-2757917e6e9">Running a Migration Making 800k API Calls Without Crashing Prod</a> was originally published in <a href="https://medium.com/drop-engineering">Drop Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we built an event-driven architecture at Drop]]></title>
            <link>https://medium.com/drop-engineering/how-we-built-an-event-driven-architecture-at-drop-77613f38555b?source=rss----11c02d7aac33---4</link>
            <guid isPermaLink="false">https://medium.com/p/77613f38555b</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[sidekiq]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[debezium]]></category>
            <dc:creator><![CDATA[Eric M Payne]]></dc:creator>
            <pubDate>Tue, 22 Jun 2021 20:24:46 GMT</pubDate>
            <atom:updated>2021-06-22T20:24:45.998Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*21fFEokpuWByGRhzPgdrdQ.png" /><figcaption>Drop aims to be the most personalized rewards platform in the world.</figcaption></figure><p><a href="http://paulgraham.com/ds.html">“Do things that don’t scale”</a> is a common adage heard at early-stage startups, and building software is no exception to this rule. As a startup grows, though, unscalable software solutions need to be revisited and rearchitected, as software architecture that worked for thousands of users may not work for a user base in the millions! Here at Drop, we’re moving towards an event-driven architecture to support our scaling needs. In this blog post, I’ll cover the following topics:</p><ul><li>what is an event-driven architecture?</li><li>how did Drop build an event-driven architecture?</li><li>what are Drop’s first use cases for an event-driven architecture?</li></ul><h3>What is an event-driven architecture?</h3><p>Firstly, what is an event? An <em>event</em> refers to some change of state in a software system; computations that cause this state change are referred to as <em>event producers</em>, while computations that occur as a result of this state change are called <em>event consumers</em>. The goal of an <em>event-driven architecture</em> is to decouple our producer logic from our consumer logic.</p><p>Consider the following Ruby on Rails code snippet:</p><pre>class Foo &lt; ApplicationRecord<br>  after_create :call_bar_service</pre><pre>  private</pre><pre>  def call_bar_service<br>    BarService.new.perform_async(self.id)<br>  end<br>end</pre><p>In this example:</p><ul><li>our <em>event</em> is an INSERT to the foos table in a SQL database</li><li>our <em>event producer</em> is a call to .create on Foo</li><li>our <em>event consumer</em> is our BarService Sidekiq job, which performs asynchronously using the new Foo instance’s ID as a parameter</li></ul><p>The implementation of Foo has knowledge of BarService, meaning our producer and consumer are tightly coupled. We want to avoid this type of design; the above example is simple, but this type of tight coupling in a large codebase makes a software application difficult and expensive to maintain. An <em>event driven architecture</em> should enable a BarService to run without Foo having any knowledge of that service; in other words, these software components would be decoupled.</p><h3>Kafka and the Kafka consumer</h3><p>To build this decoupled event-driven architecture, we selected <a href="https://kafka.apache.org/">Apache Kafka</a> to enable fault-tolerant, real time streaming of events. Kafka is a software bus that stores event data written by an event producer (also known as a <em>record</em>). This record is eventually read by an event consumer. This data is divided into different <em>topics </em>based on their source.</p><p>A <em>Kafka consumer</em> is a program that executes in response to new data being added into Kafka, and is configured to monitor specific topics that it will read event data from. Drop uses <a href="https://github.com/zendesk/racecar">Racecar</a>, a Kafka consumer framework written in Ruby, to consume these events and execute useful business logic upon consumption.</p><h3>Debezium and data capture</h3><p>If Racecar is the event consumer in this architecture, then what is the event producer? We use<a href="https://debezium.io/"> <em>Debezium</em>, an open-source change data capture (CDC)</a> Kafka connector on top of our PostgreSQL database.</p><p>When an INSERT, UPDATE, or DELETE occurs on a database table we’re interested in, PostgreSQL captures the complete contents of this change with column-level granularity in its write-ahead log. Debezium then converts these changes into JSON payloads, which are then placed onto Kafka in a topic corresponding to the table name. This JSON payload specifies the type of write that occurred (INSERT vs. UPDATE vs. DELETE), as well as all of the values contained in the columns of the new row.</p><h3>Putting it all together</h3><p>We use Debezium to produce an event for every write to a table of interest in PostgreSQL; the event is then persisted to a topic on Kafka. Racecar, our Kafka consumer, will read this event data off of Kafka topics of interest, which then runs application-specific logic based on the contents of that event.</p><p>Returning to our original example, we can redefine Foo to be significantly simpler:</p><pre>class Foo &lt; ApplicationRecord<br>  # We no longer need an after_create callback,<br>  # or any reference to BarService!<br>end</pre><p>We configure Debezium to:</p><ul><li>listen to writes to the foos table in PostgreSQL</li><li>create corresponding event data on Kafka’s foos topic</li></ul><p>We then configure Racecar to listen to the foos topic on Kafka, and define the consumer like so:¹</p><pre>class FooEventConsumer &lt; Racecar::Consumer<br>  subscribes_to &#39;foos&#39;</pre><pre>  def process(message)<br>    event_data = JSON.parse(message.value)</pre><pre>    # c for &#39;create&#39;, ie. row insertion<br>    if event_data[&#39;op&#39;] == &#39;c&#39;<br>      foo_id = event_data[&#39;payload&#39;][&#39;id&#39;]<br>      BarService.new.perform_async(foo_id)<br>    end<br>  end<br>end</pre><p>The steps in the data flow look like so:</p><ol><li>A new row is inserted into the foos table on PostgreSQL (ie. a call to Foo.create in our Rails application code was successful).</li><li>Debezium consumes this data from PostgreSQL’s write-ahead log and generates a JSON payload from it.</li><li>Debezium writes this JSON payload to the foos topic on Kafka.</li><li>A Racecar consumer, which has been listening to the foos topic on Kafka, reads and parses the JSON payload (which contains complete information about the new row in the foos table).</li><li>The Racecar service calls BarService with the ID of the newly created Foo.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sChNsAHed0HfP-cAUmYfiw.png" /><figcaption>Data flow in our event-driven architecture</figcaption></figure><h3>Our first use case: Audience membership calculation</h3><p>So, how do we actually use this event-driven architecture at Drop? Our first use case is calculating which<em> Audiences </em>our members belong to.</p><p>Drop aims to be the most personalized rewards platform in the world, meaning that there’s a high degree of personalization in determining what content we show to our members in the Drop app. An <em>Audience</em> is a domain model that enables this personalization, and determines what our members see when they open the Drop app. A user belongs to many Audiences, which can be:</p><ul><li>based on demographics (eg. age, gender)</li><li>based on behaviour (eg. having previously purchased from one of our partner brands, having recently onboarded onto the Drop app)</li></ul><p>In our previous implementation, we calculated which Audiences a user belonged to whenever they opened the Drop app on their mobile device. However, there were two problems with this approach:</p><ul><li>Calculating a user’s Audience memberships is computationally intensive. It requires a number of database queries, which slows down the app load time when a user opens Drop on their mobile device. To minimize this slow load time, we cached users’ Audience memberships. However, there was a tradeoff in doing so — a user wouldn’t necessarily see the most up to date content that was relevant to them due to this caching.</li><li>We didn’t persist this Audience membership data to PostgreSQL for analytics purposes. Any persisted values would be inherently out-of-date and inaccurate, as recalculation was dependent on a user opening their app on their phone. This means that from an analytics perspective, our previous implementation limited any possible insight into which users belonged to which Audiences.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ufxdWFMxcS_UC5wQVLTSnw.png" /><figcaption>Drop’s previous implementation of Audience membership calculation</figcaption></figure><p>Our new event-driven architecture allows us to precompute these values in real-time for an active user with the following steps:</p><ol><li>Configure Debezium to produce events when our Rails application writes to a table that may have changed a user’s Audience memberships. For example, our “device type” audience (ie. an Android vs. iOS user) may change upon a write to the devices<em> </em>PostgreSQL table (which stores data about what device type a user opened the Drop app with). So, a write to this devices table will cause Debezium to generate an event, which is written to Kafka.</li><li>Run Racecar (our Kafka consumer) to process these events that may cause a change in a user’s Audience memberships. The Racecar consumer will:<br>- read a user ID from the event<br>- determine the Audience type from the topic (eg. “device type” from the devices topic)<br>- enqueue an asynchronous Sidekiq job to recompute Audience memberships of that Audience type (eg. “device type”) for that user.</li><li>In the asynchronous Sidekiq job from 2), compute the user’s Audience memberships (ie. the intensive computation we used to do at app load time), then persist those values to PostgreSQL.</li><li>Query PostgreSQL for these persisted values when the user opens Drop on their mobile device (rather than do an expensive recomputation).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X5yQfmF3m4SRLaNlfx1Rwg.png" /><figcaption>Recalculating Audience memberships in our event-driven architecture</figcaption></figure><h3>Non-obvious implementation details</h3><p>Some interesting challenges we faced while implementing the above architecture included:</p><ol><li><em>Debouncing</em>: in practice, our Rails application does many writes in short succession for the same user and Audience type. For example, when Drop syncs credit card transaction data for a user, we don’t need to recalculate that user’s Audience memberships for <em>every single</em> credit card transaction (ie. each database write). We would get the correct result if we did, but this would be an inefficient use of our compute resources. To avoid wasteful duplicate reprocessing, we added debouncing logic into our Kafka consumer.</li><li><em>Transaction log disk usage: </em>our initial configuration for our Kafka brokers didn’t provision enough disk space, leading to unexpected interactions between Debezium and Kafka, ultimately causing downtime in our Kafka brokers. A quick re-provisioning with more disk space for the Kafka brokers mitigated this issue.</li><li><em>Adding new tables:</em> Debezium’s configuration is administered through the <a href="https://kafka.apache.org/documentation.html#connect_rest">Kafka Connect REST API</a>. Typically, we avoid making manual REST requests by using <a href="https://github.com/Mongey/terraform-provider-kafka-connect">a Terraform provider</a>. However, due to some Debezium-specific quirks, a manual process of making REST requests is required when we want to change the set of tables we produce events for.</li><li><em>Numerous edge cases in our Audiences logic: </em>not all Audience membership changes are caused by database writes. For example, a change in an age-based Audience isn’t altered by any writes to the database; it’s actually determined by the passage of time, requiring a periodic job to recompute age-based memberships for some users. There are many such edge cases for precomputing Audience memberships.</li></ol><h3>Impact in production</h3><p>To verify and quantify the correctness and impact of our new architecture, we used <a href="https://github.com/github/scientist">Scientist, a “Ruby library for carefully refactoring critical paths.”</a> In the area of our codebase where we currently compute our Audience memberships at app open time, we define an <em>experiment</em>. In this experiment, we:</p><ul><li>define a <em>control</em> block (ie. our current, “known good” implementation for computing Audience memberships)</li><li>define a <em>candidate</em> block (ie. using our stored Audience memberships that were precomputed upon consuming a Debezium event)</li></ul><p>Scientist will always return the value from the control block, but will report whether or not our candidate block returned the same value as our control block, as well as provide timing information. Using Scientist, we verified that our stored Audience memberships:</p><ul><li>had an average accuracy rate of 99.98% when compared to the control value²</li><li>had an average latency that was 1.15% higher than the control value. This control value includes our stale cached values, meaning that returning our up-to-date precomputed Audience memberships is comparable in performance to returning stale cached values. In absolute terms, this adds under 1ms of latency to a computation that takes 20–30ms, which is well within the parameters of this project’s success metrics.</li></ul><p>So, our new implementation:</p><ul><li>provides more up-to-date Audience membership data than our previous caching implementation (with no meaningful hit to performance and accuracy)</li><li>stores this data so that it can be queried for analytics purposes</li></ul><h3>Generalized use case: the Outbox pattern</h3><p>Our next use case for Debezium at Drop is the <a href="https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/"><em>outbox</em> communication pattern</a>. By writing to the outbox table in PostgreSQL, we atomically update our database and publish an event to Kafka via Debezium. A Kafka consumer can subscribe to the outbox topic on Kafka, which may contain messages created by other services. This enables interservice communication that decouples our publisher and subscriber.</p><p>Drop’s current pub-sub model is loosely based on the outbox pattern; however, it:</p><ul><li>uses Sidekiq across service boundaries, which works for our relatively monolithic architecture, but won’t be scalable as we move to a more service-based architecture</li><li>uses publishers that are loosely coupled with their subscribers (rather than completely decoupled)</li></ul><p>This new architecture allows us to decouple our services in a scalable manner.</p><h3>Conclusion</h3><p>Using Drop’s new event-based architecture, our first use case allows us to:</p><ul><li>precompute a user’s Audience memberships before they open the Drop app, allowing users to see up-to-date and personalized offers in a performant manner</li><li>accurately query which users belong to which Audiences from an analytics perspective</li></ul><p>It’s worth noting that Kafka can also be used for more than just enabling an event-driven architecture; more generally, Kafka can act as a message bus for one or more services. Looking forward to the future, this will allow Drop to move towards a less monolithic and more service-based architecture as our company scales and grows.</p><h3>Footnotes</h3><p>[1]: The JSON format that this example could successfully parse doesn’t <em>quite</em> map to a real Debezium payload. For brevity and simplicity, this blog post skips over some of the complexities of Debezium (such as schema metadata and before/after values), and so this example is simpler than our actual production code.</p><p>[2]: Why not 100% accuracy? We compute our Audience memberships asynchronously in a Sidekiq queue, meaning that there’s some lag between when an action occurs that should update a user’s Audience memberships and when that computation actually occurs.</p><p>This may sound strange, given that one of the goals of this project was to remove the caching that caused our users to not have up-to-date Audience memberships. However, the delay caused by this caching was measured in <em>hours</em>, while the delay caused by asynchronous processing is measured in <em>seconds</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77613f38555b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/drop-engineering/how-we-built-an-event-driven-architecture-at-drop-77613f38555b">How we built an event-driven architecture at Drop</a> was originally published in <a href="https://medium.com/drop-engineering">Drop Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying React Native’s Animated API: Part 2]]></title>
            <link>https://medium.com/drop-engineering/demystifying-react-natives-animated-api-part-2-15f655a84f81?source=rss----11c02d7aac33---4</link>
            <guid isPermaLink="false">https://medium.com/p/15f655a84f81</guid>
            <category><![CDATA[animation]]></category>
            <category><![CDATA[react-native]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Simon Steer]]></dc:creator>
            <pubDate>Tue, 09 Feb 2021 21:24:07 GMT</pubDate>
            <atom:updated>2021-02-09T21:24:07.212Z</atom:updated>
            <content:encoded><![CDATA[<p>Please note that this is a follow up to <a href="https://drop.engineering/demystifying-react-natives-animated-api-part-1-681da7ca4661">part one of this series</a>, and we will be building on top of previously discussed concepts such as the interpolation model, when to use certain React hooks with the Animated API, etc. If you did read it, already have a decent understanding of the Animated API, or don’t care either way and just want to see some stuff animate, let’s get to it!</p><p>Throughout this blog post, we are going to build this confetti animation step-by-step from start to finish using the Animated API 🎊</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*xJPa7UqVSe4QGny_f5hkLg.gif" /></figure><p>Hopefully by the time you are done reading, you not only will have a more advanced understanding of the Animated API, but will also be better equipped to break down animations into small, understandable steps so you can avoid <a href="https://www.bmc.com/blogs/spaghetti-code/">spaghetti code</a> 🍝</p><p>To simplify our mental model of what confetti <em>is</em>, we can think of confetti as a bunch of little rectangles that float down from the top of the screen to the bottom. We can start by rendering these rectangles at the top of the screen, without worrying about how to animate them for the time being:</p><pre>import React from &quot;react&quot;<br>import { Animated, StyleSheet, View } from &quot;react-native&quot;</pre><pre><em>const</em> CONFETTI_COLORS = [&quot;#623cea&quot;, &quot;#ffe66d&quot;, &quot;#06d6a0&quot;, &quot;#cc3363&quot;, &quot;#00bbf9&quot;]</pre><pre>export default <em>function</em> Confetti({<br> <em>amount</em> = 60<br>}: {<br> amount?: <em>number<br></em>}) {<br>  return (<br>    &lt;<em>View pointerEvents=&quot;none&quot;</em> style={styles.confettiContainer}&gt;<br>      {[...<em>Array</em>(<em>amount</em>)].map((<em>_</em>, <em>index</em>) <em>=&gt;</em> {<br>        <em>const</em> backgroundColor =<br>          CONFETTI_COLORS[<em>index</em> % CONFETTI_COLORS.length]</pre><pre>        return (<br>          &lt;<em>Animated.View<br>            key={index}<br>            </em>style={[<br>              styles.confettiBit,<br>              { backgroundColor }<br>            ]}<br>          /&gt;<br>        )<br>      })}<br>    &lt;/<em>View</em>&gt;<br>  )<br>}</pre><pre><em>const</em> styles = StyleSheet.create({<br>  confettiContainer {<br>    ...StyleSheet.absoluteFillObject,<br>    justifyContent: &quot;center&quot;,<br>    alignItems: &quot;center&quot;<br>  },<br>  confettiBit: {<br>    position: &quot;absolute&quot;,<br>    top: 0,<br>    left: 0,<br>    width: 25,<br>    height: 10,<br>    borderRadius: 7<br>  }<br>})</pre><p>Rendering our component should result in the confetti all packed into the top-left corner, barely visible on the simulator:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*cZX-MaR7D8CzbtLsL-VjNA.png" /></figure><p>We should reposition each confetti bit so they are distributed across the width of the screen instead of in the same area. Let’s translate our ConfettiBit components across the x-axis.</p><pre>export default <em>function</em> Confetti({<br> <em>amount</em> = 60<br>}: {<br> amount?: <em>number<br></em>}) {<br>  <em>const</em> dimensions = Dimensions.get(&quot;screen&quot;)</pre><pre>  return (<br>    &lt;<em>View pointerEvents=&quot;none&quot;</em> style={styles.confettiContainer}&gt;<br>      {[...<em>Array</em>(<em>amount</em>)].map((<em>_</em>, <em>index</em>) <em>=&gt;</em> {<br>        <em>const</em> backgroundColor =<br>          CONFETTI_COLORS[<em>index</em> % CONFETTI_COLORS.length]</pre><pre><em>        const</em> startX = Math.random() * dimensions.width<br>        <br>        return (<br>          &lt;<em>Animated.View<br>            key={index}<br>            </em>style={[<br>              styles.confettiBit,<br>              {<br>                backgroundColor,<br>                transform: [{ translateX: startX }]<br>              }<br>            ]}<br>          /&gt;<br>        )<br>      })}<br>    &lt;/<em>View</em>&gt;<br>  )<br>}</pre><p>Our confetti will now render across the top of the device’s screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*SkAR0qtHwQAI6p4ALrDfEA.png" /></figure><p>Now that our confetti is evenly distributed, we should animate it floating to the bottom of the screen. Let’s create an Animated.Value to represent the timeline of our animation, then interpolate that value to represent the y-translation we would like each confetti to undergo. We can do the same with the x-translation so the confetti doesn’t move in a perfectly vertical line. Let’s also add a function to animate our value from 1 → 2, and a button so we can test the animation:</p><pre>export default <em>function</em> Confetti({<br> <em>amount</em> = 60<br>}: {<br> amount?: <em>number<br></em>}) {<br>  <em>const</em> dimensions = Dimensions.get(&quot;screen&quot;)</pre><pre>  const animate = () <em>=&gt; </em>{<br>    animatedValue.setValue(0)<br>    Animated.timing(animatedValue, {<br>      toValue: 1,<br>      useNativeDriver: true,<br>      easing: <em>Easing</em>.linear,<br>      duration: 3000<br>    }).start()<br>  }</pre><pre>  return (<br>    &lt;<em>View pointerEvents=&quot;none&quot;</em> style={styles.confettiContainer}&gt;<br>      {[...<em>Array</em>(<em>amount</em>)].map((<em>_</em>, <em>index</em>) <em>=&gt;</em> {<br>        <em>const</em> backgroundColor =<br>          CONFETTI_COLORS[<em>index</em> % CONFETTI_COLORS.length]</pre><pre><em>        const</em> startX = Math.random() * dimensions.width<br>        const endX = Math.random() * dimensions.width<br>        const translateX = animatedValue.interpolate({<br>          inputRange: [0, 1],<br>          outputRange: [startX, endX]<br>        })</pre><pre>        const startY = 0<br>        const endY = dimensions.height<br>        const translateY = animatedValue.interpolate({<br>          inputRange: [0, 1],<br>          outputRange: [startY, endY]<br>        })<br>      <br>        return (<br>          &lt;<em>Animated.View<br>            key={index}<br>            </em>style={[<br>              styles.confettiBit,<br>              {<br>                backgroundColor,<br>                transform: [{ translateX }, { translateY }]<br>              }<br>             ]}<br>          /&gt;<br>        )<br>      })}<br>      &lt;<em>Button</em> title=&quot;test animation&quot; onPress={animate} /&gt;<br>    &lt;/<em>View</em>&gt;<br>  )<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*s-yAsb-ZViwHSG9dw6wFsg.gif" /></figure><p>So far so good. Obviously this doesn’t look like confetti just yet, but we have the basic timeline and movement of our animation down. Now comes the fun part where we add more details to our animation to breathe some life into it. We can start by having each confetti bit undergo a random number of rotations throughout the animation. We can also do some rotations along the x-axis to give the illusion that each confetti bit is flipping mid-air.</p><pre>// inside our .map() callback</pre><pre>...</pre><pre>// start at a random angle between 0°-360°<br>// end after 2-10 rotations<br>const rotateStart = Math.random() * 360<br>const rotateEnd =<br>  rotateStart + 720 + Math.random() * 8 * 360</pre><pre>const rotate = animatedValue.interpolate({<br>  inputRange: [0, 1],<br>  outputRange: [rotateStart + &#39;deg&#39;, rotateEnd + &#39;deg&#39;]<br>})</pre><pre>// start at a random angle between 0°-360°<br>// end after 1-20 rotations along the x-axis<br>const rotateXStart = Math.random() * 360<br>const rotateXEnd =<br>  rotateStart + 360 + Math.random() * 19 * 360</pre><pre>const rotateX = animatedValue.interpolate({<br>  inputRange: [0, 1],<br>  outputRange: [rotateXStart + &#39;deg&#39;, rotateXEnd + &#39;deg&#39;]<br>})</pre><pre>return (<br>  &lt;<em>Animated.View<br>    key={index}<br>    </em>style={[<br>      styles.confettiBit,<br>      {<br>        backgroundColor,<br>        transform: [<br>          { translateX },<br>          { translateY },<br>          { rotate },<br>          { rotateX }<br>        ]<br>      }<br>     ]}<br>  /&gt;<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*LgR4UzvfMxlrIQWab4MwIA.gif" /></figure><p>Now we have each of our confetti bits distributed across the x-axis at the start of our animation, and moving down the y-axis, and flipping a random number of times throughout the animation. We do have one problem, though — all the confetti floats down the screen at the same rate! Real confetti doesn’t move like this; each piece should float down at differing speeds. We can achieve this by adjusting each confetti’s startTranslateY and endTranslateY values by a random amount between the value of the longest edge of each confetti bit (to ensure it starts and ends offscreen) and the height of the device. This will result in each confetti bit traveling across a different distance as it travels down the screen, but over the same amount of time. This should result in each confetti bit moving at variable speeds.</p><pre>// start at 0<br>// subtract confetti&#39;s longest edge (width)<br>// offset again by a random amount between 0 - device height<br>let startY = 0<br>startY -= styles.confettiBit.width<br>startY -= Math.random() * dimensions.height</pre><pre>// start with screen height<br>// add confetti&#39;s longest edge (width)<br>// add random amount between 0 - screen height<br>let endY = dimensions.height<br>endY += styles.confettiBit.width<br>endY += Math.random() * dimensions.height</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*eULGRUaAum046D7pvubMzg.gif" /></figure><p>Now that we’ve we increased the average distance our confetti travels, 3000ms doesn’t feel quite long enough for our animation. Let’s increase the duration to 8000ms, and change up the easing a bit.</p><pre><em>// update our animate function<br>// increase timing<br>// change easing function</em></pre><pre><em>const</em> animate = () <em>=&gt;</em> {<br>  animatedValue.setValue(0)<br>  Animated.timing(animatedValue, {<br>    toValue: 1,<br>    useNativeDriver: true,<br>    easing: <em>Easing</em>.out(<em>Easing</em>.quad),<br>    duration: 8000<br>  }).start()<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*F9H7INH_4LJ_V2Y4mtITrQ.gif" /></figure><p>Now that we have everything looking the way we want, let’s clean up our code a bit. To start, we can abstract each confetti bit we render into a single component:</p><pre><em>const</em> CONFETTI_COLORS = [&quot;#623cea&quot;, &quot;#ffe66d&quot;, &quot;#06d6a0&quot;, &quot;#cc3363&quot;, &quot;#00bbf9&quot;]</pre><pre>function ConfettiBit({<br>  index,<br>  animatedValue<br>}: {<br>  index: number;<br>  animatedValue: Animated.Value<br>}) {<br>  // store style in state so subsequent renders<br>  // don&#39;t result in new random values from Math.random()</pre><pre>  const [style] = useState(() =&gt; {<br>    <em>const</em> dimensions = Dimensions.get(&quot;screen&quot;)</pre><pre>    <em>const</em> backgroundColor =<br>      CONFETTI_COLORS[<em>index</em> % CONFETTI_COLORS.length]</pre><pre><em>    const</em> startX = Math.random() * dimensions.width<br>    const endX = Math.random() * dimensions.width<br>    const translateX = animatedValue.interpolate({<br>      inputRange: [0, 1],<br>      outputRange: [startX, endX]<br>    })</pre><pre>    let startY = 0<br>    startY -= styles.confettiBit.width<br>    startY -= Math.random() * dimensions.height<br>  <br>    let endY = dimensions.height<br>    endY += styles.confettiBit.width<br>    endY += Math.random() * dimensions.height<br>  <br>    const translateY = animatedValue.interpolate({<br>      inputRange: [0, 1],<br>      outputRange: [startX, endX]<br>    })</pre><pre>    const rotateStart = Math.random() * 360<br>    const rotateEnd =<br>      rotateStart + 720 + Math.random() * 8 * 360</pre><pre>    const rotate = animatedValue.interpolate({<br>      inputRange: [0, 1],<br>      outputRange: [rotateStart + &#39;deg&#39;, rotateEnd + &#39;deg&#39;]<br>    })</pre><pre>    const rotateXStart = Math.random() * 360<br>    const rotateXEnd =<br>      rotateStart + 360 + Math.random() * 19 * 360</pre><pre>    const rotateX = animatedValue.interpolate({<br>      inputRange: [0, 1],<br>      outputRange: [rotateXStart + &#39;deg&#39;, rotateXEnd + &#39;deg&#39;]<br>    })</pre><pre>    return [<br>      styles.confettiBit,<br>      {<br>        backgroundColor,<br>        transform: [<br>          { translateX },<br>          { translateY },<br>          { rotate },<br>          { rotateX }<br>        ]<br>      }<br>    ]<br>  })</pre><pre>  return &lt;<em>Animated.View </em>style={style}/&gt;<br>}</pre><p>This lets us update our Confetti component to be a bit more compact. We can also remove the button we were using to test the animation.</p><pre><em>function</em> Confetti({<br>  <em>numConfetti</em> = 60<br>}: {<br>  numConfetti?: <em>number<br></em>}) {<br>  <em>const</em> animatedValue = useRef(new Animated.<em>Value</em>(0)).current<em><br>  const</em> animate = () <em>=&gt;<br>    </em>Animated.timing(animatedValue, {<br>      toValue: 1,<br>      easing: <em>Easing</em>.out(<em>Easing</em>.quad),<br>      duration: 8000,<br>      useNativeDriver: false<br>    }).start(() <em>=&gt;</em> {<br>      animatedValue.setValue(0)<br>    })</pre><pre>  return (<br>    &lt;<em>View pointerEvents=&quot;none&quot;</em> style={styles.confettiContainer}&gt;<br>      {[...<em>Array</em>(<em>numConfetti</em>)].map((<em>_</em>, <em>index</em>) <em>=&gt;</em> (<br>        &lt;<em>ConfettiBit<br>          </em>key={<em>index</em>}<br>          index={<em>index</em>}<br>          animatedValue={animatedValue}<br>        /&gt;<br>      ))}<br>    &lt;/<em>View</em>&gt;<br>  )<br>}</pre><p>As neat and tidy as this code is, we still have one thing left to consider; how would a developer use this animated confetti component? Most likely, we would associate the animation with an achievement — maybe a user just completed onboarding, or opened the app on their birthday, for example. Currently, we only have access to the animate function inside of our Confetti component, which makes it difficult for us to interact with and trigger animations arbitrarily from outside of it. We can solve this elegantly by turning our confetti animation into a hook:</p><pre><em>function</em> useConfetti(numConfetti = 60) {<br>  <em>const</em> animatedValue = useRef(new Animated.<em>Value</em>(0)).current<em><br>  const</em> animate = () <em>=&gt;<br>    </em>Animated.timing(animatedValue, {<br>      toValue: 1,<br>      easing: <em>Easing</em>.out(<em>Easing</em>.quad),<br>      duration: 8000,<br>      useNativeDriver: false<br>    }).start(() <em>=&gt;</em> {<br>      animatedValue.setValue(0)<br>    })</pre><pre>  const confetti = (<br>    &lt;<em>View pointerEvents=&quot;none&quot;</em> style={styles.confettiContainer}&gt;<br>      {[...<em>Array</em>(<em>numConfetti</em>)].map((<em>_</em>, <em>index</em>) <em>=&gt;</em> (<br>        &lt;<em>ConfettiBit<br>          </em>key={<em>index</em>}<br>          index={<em>index</em>}<br>          animatedValue={animatedValue}<br>        /&gt;<br>      ))}<br>    &lt;/<em>View</em>&gt;<br>  )<br>  <br>  return [confetti, animate] as const<br>}</pre><p>Now, instead of being locked into fiddling around with the internals of our Confetti component, we can just use our hook 🤩</p><pre><em>function</em> App() {<br>  <em>const</em> [confetti, animateConfetti] = useConfetti()</pre><pre>  return (<br>    &lt;<em>View</em> style={{ flex: 1 }}&gt;<br>      &lt;<em>ScrollView</em>&gt;<br>        &lt;<em>Text</em> style={{<br>          fontSize: 48,<br>          marginVertical: 64,<br>          textAlign: &quot;center&quot; <br>        }}&gt;<br>          Happy Birthday<br>       &lt;/<em>Text</em>&gt;<br>       &lt;<em>Button</em> title=&quot;let&#39;s party&quot; onPress={animateConfetti} /&gt;<br>      &lt;/<em>ScrollView</em>&gt;<br>      {confetti}<br>    &lt;/<em>View</em>&gt;<br>  )<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*xJPa7UqVSe4QGny_f5hkLg.gif" /></figure><p>And there you have it! We could go even further and have some of the “magic numbers” (number of flips, etc.) in our style code be configurable via arguments to our hook function, but I’ll leave it there for the purposes of this blog post before this gets even longer.</p><p>Thanks for reading! will there be a part 3 to this series? Possibly 😉 stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15f655a84f81" width="1" height="1" alt=""><hr><p><a href="https://medium.com/drop-engineering/demystifying-react-natives-animated-api-part-2-15f655a84f81">Demystifying React Native’s Animated API: Part 2</a> was originally published in <a href="https://medium.com/drop-engineering">Drop Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying React Native’s Animated API: Part 1]]></title>
            <link>https://medium.com/drop-engineering/demystifying-react-natives-animated-api-part-1-681da7ca4661?source=rss----11c02d7aac33---4</link>
            <guid isPermaLink="false">https://medium.com/p/681da7ca4661</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[animation]]></category>
            <category><![CDATA[react-native]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Simon Steer]]></dc:creator>
            <pubDate>Wed, 30 Sep 2020 15:19:54 GMT</pubDate>
            <atom:updated>2020-09-30T15:19:54.740Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://www.joindrop.com/">Drop</a>, we use React Native and TypeScript to develop our mobile app. Being able to write platform-agnostic code in a widely-adopted framework makes the switch from web development in React to mobile development in React Native relatively easy. As someone who loves the UX side of things, I found the biggest challenge when learning React Native was wrapping my head around the difference in mental models for animations.</p><p>By the time you are done reading this article, you should have a decent understanding of React Native’s Animated API’s mental model. This blog post is the first of a two-part series; in this post we will cover the basics of the Animated API by rebuilding an HTML/CSS button element and its hover states in React Native. In a follow-up blog post we’ll cover some more advanced concepts by building out a performant confetti animation step-by-step.</p><h3><strong>A brief overview of the differences in styling between React Native and React for web</strong></h3><p>On the web, you might animate a button using the CSS transition property like so:</p><pre>.my-button {<br>  padding: 8px 16px;<br>  border-radius: 4px;<br>  border: 2px solid black;<br>  background: white;<br>  outline: none;<br>  color: black;<br>  font-family: sans-serif;<br>  font-weight: bold;<br>  transition: color 0.2s, background 0.2s;<br>}</pre><pre>.my-button:hover {<br>  background: black;<br>  color: white;<br>}</pre><p>Then, in your html:</p><pre>&lt;button class=&quot;my-button&quot;&gt;<br>  hover over me!<br>&lt;/button&gt;</pre><p>You may also use CSS keyframe animations to give it a bit more life:</p><pre>@keyframes shake {<br>  25% {<br>    transform: rotate(3deg);<br>  }<br>  75% {<br>    transform: rotate(-3deg);<br>  }<br>}</pre><pre>.my-button:hover {<br>  background: black;<br>  color: white;<br>  animation: shake 0.2s infinite linear;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*9QN49UDCGbnXrSNe" /></figure><p>In React Native, applying styles and animations to your elements doesn’t quite work the same way. Most notably:</p><ol><li><strong>Styles in React Native are not applied via </strong><a href="https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors"><strong>selectors</strong></a><strong>, they are represented with objects.</strong> React Native provides a module, StyleSheet, which we can use to create a stylesheet via StyleSheet.create. Under the hood, StyleSheet.create registers the passed in object styles in an internally maintained list to refer back to when applying them to native views. As you may have guessed, no access to traditional CSS selectors also means no usage of :hover, :hover, :focus, etc. If you wish you apply your own custom states, you must animate them yourself.</li><li><strong>React Native styles do not </strong><a href="https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Cascade_and_inheritance#Understanding_the_cascade"><strong>cascade</strong></a><strong>, and they do not support inheritance. </strong>Upper level styles will not “trickle down” to child element’s styles. For example, if you apply a font color to a parent view, its child views will not inherit the font color.</li><li><strong>React Native elements exclusively use </strong><a href="https://developer.mozilla.org/en-US/docs/Glossary/Flexbox"><strong>Flexbox</strong></a><strong>.</strong> Every element in RN follows the Flexbox model, and as such you cannot use <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Grid_Layout">CSS Grid</a>, floats, or block display types.</li></ol><p>This is just a brief list of what I consider to be the most important differences when it comes to styling, but you can <a href="https://reactnative.dev/docs/style">take a look at the docs</a> to learn more.</p><h3><strong>Animations in React Native</strong></h3><p>Here is how you might recreate the button from before without the shake animation or hover states in React Native:</p><pre>import React from &#39;react&#39;<br>import { TouchableWithoutFeedback, StyleSheet, Text } from &#39;react-native&#39;</pre><pre>const styles = StyleSheet.create({<br>  button: {<br>    paddingVertical: 8,<br>    paddingHorizontal: 16,<br>    borderRadius: 4,<br>    borderWidth: 2,<br>    borderColor: &#39;black&#39;,<br>    backgroundColor: &#39;white&#39;,<br>    flex: 0,<br>  },<br>  text: {<br>    color: &#39;#1e90ff&#39;,<br>    fontFamily: &#39;sans-serif&#39;,<br>    fontWeight: &#39;bold&#39;,<br>  },<br>})</pre><pre>const Button = () =&gt; (<br>  &lt;TouchableWithoutFeedback<br>    accessibilityRole=&quot;button&quot;<br>    accessibilityComponentType=&quot;button&quot;<br>    onPress={() =&gt; console.log(&#39;you pressed the button&#39;)}<br>    style={styles.button}<br>  &gt;<br>    &lt;Text style={styles.text}&gt;Press me!&lt;/Text&gt;<br>  &lt;/TouchableWithoutFeedback&gt;<br>)</pre><p>We create a stylesheet with the styles for both the button and the text. We do this because styles in React Native do not cascade/inherit. Since we can’t create stylesheet declarations for hover states in React Native, how do we go about animating the color change and shake animations? Instead of specifying fixed transitions for style properties when animating like we would on the web, we can interpolate a value over time to achieve natural, fluid motion, with greater control.</p><p><strong>What the heck is interpolation?</strong></p><p>Interpolation can be thought of as mapping one value to another. Let’s say we have a single value, that starts at 0, and increases to 1, over the duration of one second:</p><p>0 → 1</p><p>We can interpolate this value by creating a second value, which changes as our first value does, but with different upper and lower bounds. So we could, for example, interpolate our original value so as it moves from 0 to 1, our interpolated value moves from 2 to 7:</p><p>0 → 1</p><p>2 → 7</p><p>The Animated API lets us create values to animate and interpolate this way, and pass them as values to our element’s style objects. This allows us to think of the style properties we are animating as dynamic values, instead of fixed, time-based transitions.</p><p><strong>Back to the Button</strong></p><p>To put the concept of interpolation into practice here, let’s start by animating the button’s color. Since there is no cursor on mobile applications, the concept of a “hover” state doesn’t <em>really</em> exist, so we will make the button’s color change when the user has their finger pressed down.</p><p>We will start by creating an animated value and storing a reference to it in our React component. This is the value we will interpolate over time.</p><pre>// import necessary modules<br>import { useRef } from &#39;react&#39;<br>import { Animated } from &#39;react-native&#39;</pre><pre>...</pre><pre>// store a reference to an animated value in your component. We use useRef here otherwise we will create a new value on every render.</pre><pre>const animatedValue = useRef(new Animated.Value(0)).current</pre><p>Now that we have an Animated.Value to work with, let’s interpolate it! Animated.Values have an interpolate method which we can use to describe the mappings we want to apply. React Native does some magic under the hood which allows us to interpolate CSS color strings, as well as rgb/rgba values:</p><pre>const animatedValue = useRef(new Animated.Value(0)).current</pre><pre>/* inputRange represents the bounds of your value. Numbers provided to inputRange must monotonically increase. */</pre><pre>/* outputRange length must equal inputRange length, represents what values the bounds map to */</pre><pre>const backgroundColor = animatedValue.interpolate({<br>  inputRange: [0, 1],<br>  outputRange: [&#39;white&#39;, &#39;black&#39;],<br>})</pre><p>We now have our interpolation for the background color mapped out: When our animated value is at 0, our backgroundColor variable is white, and as our animated value approaches 1, our backgroundColor variable darkens to black.</p><p>Now we can use our interpolated value as the background color of the button; when we want the color of the button to animate to black, we increase our animated value to 1. We’ll also move our button’s styles to an Animated.View, since the Animated API requires that we use its special components if we want to animate an element’s style props in this way.</p><pre>const Button = () =&gt; {<br>  const animatedValue = useRef(new Animated.Value(0)).current</pre><pre>  const backgroundColor = animatedValue.interpolate({<br>    inputRange: [0, 1],<br>    outputRange: [&#39;white&#39;, &#39;black&#39;],<br>  })</pre><pre>  return (<br>    &lt;TouchableWithoutFeedback<br>      accessibilityRole=&quot;button&quot;<br>      accessibilityComponentType=&quot;button&quot;<br>      onPress={() =&gt; console.log(&#39;you pressed the button&#39;)}<br>    &gt;<br>      &lt;Animated.View style={[styles.button, { backgroundColor }]}&gt;<br>        &lt;Text style={styles.text}&gt;Press me!&lt;/Text&gt;<br>      &lt;/Animated.View&gt;<br>    &lt;/TouchableWithoutFeedback&gt;<br>  )<br>}</pre><p>The last step is to animate our value to 1 when the button is pressed down, and animate it back to 0 when the button is released. The Animated API provides some helpful methods specifically for animating Animated.Values. Most commonly, you’ll end up using Animated.timing and Animated.spring. For the purposes of this example, we will use the method with the easiest mental model, Animated.timing.</p><pre>const handlePressIn = () =&gt;<br>  Animated.timing(animatedValue, {<br>    toValue: 1,<br>    duration: 300,<br>    useNativeDriver: false, // &lt;-- more on this in part 2 ;)<br>  }).start()</pre><pre>const handlePressOut = () =&gt;<br>  Animated.timing(animatedValue, {<br>    toValue: 0,<br>    duration: 100,<br>    useNativeDriver: false,<br>  }).start()</pre><p>Animated.timing takes two arguments; an Animated.Value instance and a configuration object which describes the timing of our animation. For more information on the available configuration options, you can <a href="https://reactnative.dev/docs/animated#timing">check out the docs</a>.</p><p><strong>Putting It All Together</strong></p><pre>const Button = () =&gt; {<br>  const animatedValue = useRef(new Animated.Value(0)).current<br>  <br>  const backgroundColor = animatedValue.interpolate({<br>    inputRange: [0, 1],<br>    outputRange: [&#39;white&#39;, &#39;black&#39;],<br>  })</pre><pre>  const handlePressIn = () =&gt;<br>    Animated.timing(animatedValue, {toValue: 1,<br>      duration: 100,<br>      useNativeDriver: false,<br>    }).start()</pre><pre>  const handlePressOut = () =&gt;<br>    Animated.timing(animatedValue, {<br>      toValue: 0,<br>      duration: 100,<br>      useNativeDriver: false,<br>    }).start()</pre><pre>  return (<br>    &lt;TouchableWithoutFeedback<br>      accessibilityRole=&quot;button&quot;<br>      accessibilityComponentType=&quot;button&quot;<br>      onPress={() =&gt; console.log(&#39;you pressed the button&#39;)}<br>      onPressIn={handlePressIn}<br>      onPressOut={handlePressOut}<br>    &gt;<br>      &lt;Animated.View style={[styles.button, { backgroundColor }]}&gt;<br>        &lt;Text style={styles.text}&gt;Press me!&lt;/Text&gt;<br>      &lt;/Animated.View&gt;<br>    &lt;/TouchableWithoutFeedback&gt;<br>  )<br>}</pre><p>Now when a user presses down on our custom button component, it will animate to black, and when the button is released, will animate back to white. To recap how we got here:</p><ol><li>We stored a reference to an Animated.Value instance in our component.</li><li>We created an interpolated value such that when our Animated.Value is at 0, our interpolated value is the color white, and when our Animated.Value is at 1, our interpolated value is the color black.</li><li>We made some functions using the Animated.timing method to animate our value from 0 to 1 and back again.</li><li>We mapped these functions to some press events on our custom button.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*OzDsmHgY4blW3MUE" /></figure><p>The only part of the animation we are lacking is the shake that we made our browser-rendered button do. We have two options here:</p><ol><li>Create a new Animated.Value to map to the rotation of the button. This would probably be the easiest way to achieve what we want.</li><li>Reuse our Animated.Value from before so everything runs on a single value. This is a more complicated solution, but results in fewer values to keep track of, and is more performant.</li></ol><p>Since we already pursued option (1) when animating the color of our button, let’s use option (2) to make our button undergo the shaking animation when pressed down. Let’s start by creating another interpolated value for our button’s rotate transform, and applying it to our button’s styles:</p><pre>const rotate = animatedValue.interpolate({<br>  inputRange: [0, 1],<br>  outputRange: [&#39;0deg&#39;, &#39;3deg&#39;],<br>})</pre><pre>...</pre><pre>return (<br>  &lt;TouchableWithoutFeedback<br>    accessibilityRole=&quot;button&quot;<br>    accessibilityComponentType=&quot;button&quot;<br>    onPress={() =&gt; console.log(&#39;you pressed the button&#39;)}<br>    onPressIn={handlePressIn}<br>    onPressOut={handlePressOut}<br>  &gt;<br>    &lt;Animated.View<br>      style={[<br>        styles.button,<br>        { backgroundColor, transform: [{ rotate }] }, // &lt;-- here<br>      ]}<br>    &gt;<br>      &lt;Text style={styles.text}&gt;Press me!&lt;/Text&gt;<br>    &lt;/Animated.View&gt;<br>  &lt;/TouchableWithoutFeedback&gt;<br>)</pre><p>We now have our button’s color shifting to black, and rotating 3 degrees when pressed down. But we need the button to not just rotate and sit at 3 degrees of rotation; we need it to <a href="https://www.youtube.com/watch?v=_wjFahULCK8">shake shake, shake shake, shake</a>, and oscillate between 3 and -3 degrees of rotation.</p><p>First, let’s store a reference to a boolean value in our component that will represent whether our button is pressed or not:</p><p>const isPressed = useRef(false)</p><p>We’re using useRef here, and not useState, for two reasons.</p><ol><li>Animated methods do not respect the order of execution of React hooks because they do not run on the React thread. This means that if we rely on the return value from useState to determine when to animate our button’s styles, we would not get the result we want because Animated methods won’t wait for our hooks to finish running.</li><li>We <em>could</em> enforce usage of useState by wrapping our animation methods with useCallback hooks, for example, but we would be causing unnecessary renders, since the animations happen independent of React’s hook cycle anyway.</li></ol><p>Next, let’s modify our methods to update this value when pressed and released.</p><pre>const handlePressIn = () =&gt; {<br>  isPressed.current = true<br>  Animated.timing(animatedValue, {<br>    toValue: 1,<br>    duration: 100,<br>    useNativeDriver: false,<br>  }).start()<br>}</pre><pre>const handlePressOut = () =&gt; {<br>  isPressed.current = false<br>  Animated.timing(animatedValue, {<br>    toValue: 0,<br>    duration: 100,<br>    useNativeDriver: false,<br>  }).start()<br>}</pre><p>We can then extend our rotation interpolation to support our animated value going from 0 → 1 → 2 instead of just from 0 → 1. When we animate our value from 0 to 1, the button will shift in color from white to black and rotate 3 degrees. When we animate our value to 2, our button will remain black, but will rotate to -3 degrees.</p><pre>const rotate = animatedValue.interpolate({<br>  inputRange: [0, 1, 2],<br>  outputRange: [&#39;0deg&#39;, &#39;3deg&#39;, &#39;-3deg&#39;],<br>})</pre><p>Now we can write a method that will animate our value from 1 → 2 and back again repeatedly to simulate the shake animation. We can use the Animated.sequence method to queue up multiple animations and have them run in order, then pass our shakeButton function as the callback to .start() so it re-executes on completion.</p><pre>const shakeButton = () =&gt; {<br>  // don’t run the animation while the button is not pressed<br>  if (!isPressed.current) {<br>    return<br>  }</pre><pre>  Animated.sequence([<br>    Animated.timing(animatedValue, {<br>      toValue: 2,<br>      duration: 80,<br>      useNativeDriver: false,<br>    }),<br>    Animated.timing(animatedValue, {<br>      toValue: 1,<br>      duration: 80,<br>      useNativeDriver: false,<br>    }),<br>  ]).start(shakeButton) // &lt;-- reruns on completion<br>}</pre><p>Finally, let’s have the shakeButton animation start once we have pressed down on the button and it has animated to black:</p><pre>const handlePressIn = () =&gt; {<br>  isPressed.current = true<br>  Animated.timing(animatedValue, {<br>    toValue: 1,<br>    duration: 100,<br>    useNativeDriver: false,<br>  }).start(shakeButton) // &lt;-- add this<br>}</pre><p>And here is the full component code:</p><pre>import React, { useRef } from &#39;react&#39;<br>import {<br>  Animated,<br>  StyleSheet,<br>  Text,<br>  TouchableWithoutFeedback<br>} from &#39;react-native&#39;</pre><pre>const styles = StyleSheet.create({<br>  button: {<br>    paddingVertical: 8,<br>    paddingHorizontal: 16,<br>    borderRadius: 4,<br>    borderWidth: 2,<br>    borderColor: &#39;black&#39;,<br>    backgroundColor: &#39;white&#39;,<br>    flex: 0,<br>  },<br>  text: {<br>    color: &#39;#1e90ff&#39;,<br>    fontFamily: &#39;sans-serif&#39;,<br>    fontWeight: &#39;bold&#39;,<br>  },<br>})</pre><pre>export const Button = () =&gt; {<br>  const animatedValue = useRef(new Animated.Value(0)).current<br>  const isPressed = useRef(false)<br> <br>  const backgroundColor = animatedValue.interpolate({<br>    inputRange: [0, 1],<br>    outputRange: [&#39;white&#39;, &#39;black&#39;],<br>  })<br>  <br>  const rotate = animatedValue.interpolate({<br>    inputRange: [0, 1, 2],<br>    outputRange: [&#39;0deg&#39;, &#39;3deg&#39;, &#39;-3deg&#39;],<br>  })</pre><pre>  const shakeButton = () =&gt; {<br>    // don’t run the animation while the button is not pressed<br>    if (!isPressed.current) {<br>      return<br>    }</pre><pre>  Animated.sequence([<br>      Animated.timing(animatedValue, {<br>        toValue: 2,<br>        duration: 80,<br>        useNativeDriver: false,<br>      }),<br>      Animated.timing(animatedValue, {<br>        toValue: 1,<br>        duration: 80,<br>        useNativeDriver: false,<br>      }),<br>    ]).start(shakeButton)<br>  }</pre><pre>  const handlePressIn = () =&gt; {<br>    isPressed.current = true<br>    Animated.timing(animatedValue, {<br>      toValue: 1,<br>      duration: 100,<br>      useNativeDriver: false,<br>    }).start(shakeButton)<br>  }</pre><pre>  const handlePressOut = () =&gt; {<br>    isPressed.current = false<br>    Animated.timing(animatedValue, {<br>      toValue: 0,<br>      duration: 100,<br>      useNativeDriver: false,<br>    }).start()<br>  }</pre><pre>  return (<br>    &lt;TouchableWithoutFeedback<br>      accessibilityRole=&quot;button&quot;<br>      accessibilityComponentType=&quot;button&quot;<br>      onPress={() =&gt; console.log(&#39;you pressed the button&#39;)}<br>      onPressIn={handlePressIn}<br>      onPressOut={handlePressOut}<br>    &gt;<br>      &lt;Animated.View<br>        style={[<br>          styles.button,<br>          { backgroundColor, transform: [{ rotate }] },<br>        ]}<br>      &gt;<br>        &lt;Text style={styles.text}&gt;Press me!&lt;/Text&gt;<br>      &lt;/Animated.View&gt;<br>    &lt;/TouchableWithoutFeedback&gt;<br>  )<br>}</pre><p>Here is a comparison of the button we animated for the web, versus the button we animated in React Native:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*1EYt6JkBOoj9NIOL" /><figcaption>Button animated on the web using HTML/CSS</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*sXtHGWt8OGg1TCXq" /><figcaption>Button animated in React Native using the Animated API</figcaption></figure><p>That’s all for part one of this series! Hopefully you now have a decent understanding of the mental model of the Animated API, and have enough knowledge to begin making your own animations in React Native. Stay tuned for part two, where we take the concepts presented here and build a complex, performant confetti animation together!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=681da7ca4661" width="1" height="1" alt=""><hr><p><a href="https://medium.com/drop-engineering/demystifying-react-natives-animated-api-part-1-681da7ca4661">Demystifying React Native’s Animated API: Part 1</a> was originally published in <a href="https://medium.com/drop-engineering">Drop Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Drop Eng Internship]]></title>
            <link>https://medium.com/drop-engineering/my-drop-eng-internship-7bb97f1e8d41?source=rss----11c02d7aac33---4</link>
            <guid isPermaLink="false">https://medium.com/p/7bb97f1e8d41</guid>
            <category><![CDATA[internships]]></category>
            <category><![CDATA[life]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[David Kang]]></dc:creator>
            <pubDate>Thu, 23 Jul 2020 21:14:09 GMT</pubDate>
            <atom:updated>2020-07-28T13:34:56.664Z</atom:updated>
            <content:encoded><![CDATA[<h3>My Experience as an Eng Intern at Drop</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mUYlnrxJltBIQVm37DA06A.png" /></figure><p>Hey everyone, my name is David, and I’m a 4th year Software Engineering student from the University of Waterloo. I’m writing this post as I finish up my internship at Drop because I hope to share how my time here has helped me grow in a ton of ways. In fact, thanks to the sense of self-fulfillment I was able to find throughout the experience, when I was asked if I wanted to extend my contract on a part-time basis past the university-prescribed 4 months, I took the opportunity enthusiastically despite the current pandemic pushing everyone to work remotely. It’s been an amazing experience, and maybe this will serve as a great “semester-in-review” for anyone interested in pursuing internships or full-time positions at a high-velocity startup company!</p><h3>What I did</h3><p>I joined Drop as a Software Engineering Intern on the Core Experience Team, focusing on backend development. I only say “focusing on” because I was never constrained by anybody to just work on the backend of the platform. Members are always encouraged to take ownership over all areas of the stack, and I was no exception. From the very beginning, my mentor and manager made sure that I was able to decide on what kind of work I may find valuable or interesting. My choices and passions would be able to drive the kind of experience I had, with regular check-ins to allow me to bring up any questions and review that I was having fun and learning as much as I could. This was an immediate plus, and here are some of the cool things™ I was able to work on based on that.</p><h4>Referrals</h4><p>After getting used to the ropes from some smaller tasks and issues, the first big project I worked on was revamping the existing referral system. Previously, when using referral codes from influencers or other users, everyone would have to complete the same set of requirements and only the amount of points received could be adjusted per code in any way. The purpose of my changes were to allow flexible, plug-and-playable conditions that could be different for groups of users. This would in turn allow us to run promotions and experimental campaigns for our rapidly growing user base more easily and have a robust framework for adding, removing, and toggling different conditions on the fly.</p><p>I learned a ton from this undertaking, not just in terms of technical considerations like architecture and design but also in soft skills such as managing scope and communicating clear intentions and questions. These are arguably more important to learn during an internship in my opinion — I’m learning more and more constantly that being a good software engineer is much more than just writing quality code, it’s also about why, when, and how you choose to write that code, based on coordination with other stakeholders.</p><blockquote>Being a good software engineer is much more than just writing quality code, it’s also about why, when, and how you choose to write that code, based on coordination with other stakeholders.</blockquote><p>Near the end of the project, I was able to give demos of the new changes at the company’s all-hands and our Drop Tech Tingz (more on this later!) meetings and receive some great feedback. Several people commented how they’ve been hoping for these changes for a long time now and the different ways we could leverage it to fuel our organic growth and revenue. The system still has room for improvement and even now there are efforts to make the process more user-friendly on the frontend and add more security measures, so I’m glad my work resulted in something that’s scalable and maintainable.</p><h4>App Performance</h4><p>A major initiative that I participated in for the final portion of my internship was “improving app performance”. This was a cross-stack effort across the frontend, backend, and the infrastructure engineers to investigate, experiment with, and implement various ways to improve the application and the platform based on several different dimensions such as API response times, server load, and rendering speed. This collective endeavor allowed us to shave thousands of milliseconds across our API endpoints and add better monitoring around heavy workloads, leaving open even more future opportunities to make significant changes with confidence.</p><p>The biggest contribution I made for this project was using a multi-tiered caching strategy to dramatically reduce a core endpoint’s response time and load on our server. The general idea behind what I implemented is that if users are fetching resources that are filtered to only contain items relevant for them, we can split this up into 2 cheaper calls — one that fetches ALL of the available items, relevant or not, and one that fetches only identifiers for the relevant items. The first call returns the same set of resources for every user, so we can cache it easily at the frontend and the backend. My hope is that this concept will be available for extension on various other types of resources too, and that these cached results can also be leveraged by other potentially expensive services.</p><p>I found myself diving into this project really passionately on both the backend and the frontend because of how fulfilling and measurable the returns were, along with how it required a lot of technical thinking, architecting, and prototyping. I discovered a problem to solve and was given not just the autonomy but also the support required to pursue a solution. There were many sessions where everyone updated each other on their progress, bounced ideas, and celebrated big wins, and I think this kind of hybrid collaborative-independent work really appealed to me.</p><h3>What I Loved</h3><p>Apart from the type of work I got to do, there were a lot of other aspects that I really enjoyed about working at Drop and the people here. The following are just a few of them.</p><h4>Transparency</h4><p>One thing that made a big impact on me while working at Drop was the sheer volume of information I was hit with and was available — to everyone. People make an effort to relay and chase after information here (“Is there a dashboard for that?”), and there are always all sorts of questions and answers popping up in various channels on Slack, the weekly all-hands meetings, team office hours, and more. I’m glad that I was not just boxed into an “intern project” — I was welcomed into discussions, and was able to feel like I was ramping up as a proper full-time employee, learning to care about the product.</p><p>The transparency is just as prevalent on the engineering side of things. Everyone is encouraged to learn from each other and chase down more information. Pair programming or brainstorming is completely optional but commonplace. I was encouraged to participate in code reviews from day 2, and that was the way I learned my way around the codebase the fastest! After working on the referrals revamp, I essentially became the most knowledgeable person on the team related to referrals and began fielding both technical and business-logic related questions. The amount of knowledge sharing that occurs, and everyone’s openness to being tapped on the shoulder for questions and feedback make it clear that there are always different ways you can contribute to the team and any ongoing major efforts.</p><h4>Driving Fast, but Responsibly</h4><p>The amount of changes that we churn out consistently is really amazing. From things like UI tweaks or bug fixes to major feature releases and library updates, the changes that get deployed never seem to stop. One part of this, of course, is that everyone is working hard on different product or engineering tasks and so they collect into a stream of continuous improvements. Another is that everyone is allowed to take the driver’s seat in delivering their changes and deploy them at their discretion. This allowed me to learn slowly but surely how different areas of the stack come together and what principles and ideas drive the growth of our codebase.</p><p>However, we still take time to review everything thoroughly without sacrificing velocity. Retrospectives and frontend/backend/infrastructure team meetings are regularly scheduled to evaluate ways in which we can improve or what we need to keep in mind when introducing future changes. There is always a careful balancing act between making the changes we need now versus planning for the future. I loved that because of this philosophy, and the different opinions that everyone is allowed to bring to the table, I constantly felt challenged in a way that helped me grow as a software engineer. I had to carefully think about and plan for cases that may be missed or more important and found myself following other feature flows and gathering data and benchmarks to be able to present or explain my reasoning. Looking back, I’m really glad that the team’s accountability was able to rub off on me over the months I’ve been here.</p><h4>Cool Practices</h4><p>In addition to general philosophies and practices, there are some other specific cool things at Drop that also helped me enjoy working here. One of them is the above mentioned Drop Tech Tingz — a bi-weekly meeting where the anyone from the data and engineering teams are invited to give a talk on any topic they want, whether it be related to a new feature, an informative overview of a service or library, or going over areas of the stack that other engineers might not be familiar with. I always looked forward to Fridays with DTTs because I would come out learning something new or interesting, and even gave a talk myself on the changes I made to referrals discussed above! Some of my favorite short talks were discussions related to how to organize commits and pull requests, a dive into our pub-sub architecture, and a demo of Terraform.</p><p>In addition to these technical showcases, there are general demo sessions that are (in my opinion, at least!) the highlights of the weekly all-hands meetings. These are also timeslots that anyone can sign up for to present anything they want to share, but the topics are way more diverse. UX research results, design overhauls, and fascinating data science findings on user behavior — these were just a sample of all of the interesting topics I can recall, and the demos were a huge part of how I came to learn about what different teams work on as well as the vision and central themes of the product development process driving everyone forward.</p><p>Related to the above, a really wholesome aspect of the culture at Drop is how strongly people celebrate successes. For all of the daily wins, big releases, securing partnerships, there is a dedicated Slack channel for giving “kudos” to other team members in recognition of their achievements or in appreciation for their help. In fact, every DTT session even begins with some reserved time to give team appreciations! I loved seeing how everyone is always encouraged to lift each other up and how this naturally leads to a more fearless, collaborative environment, some aspects of which even seemed to be strengthened after transitioning to remote work. Mistakes or points of improvement are reflected on as a team, and successes are celebrated and recognized — tying into the theme of transparency I talked about previously, it really helps making everything feel so <em>open</em>.</p><h3>Looking Back</h3><p>I hope I hit all of the key points that stood out the most for me over the course of my time here. It honestly feels like time has passed unnaturally quickly since my first day when I entered the lobby, unsure of who to talk to, and soon, I’ll be returning all of the hardware I borrowed and going back to my full-time studies at school. I truly enjoyed these past 5 months at Drop, and I feel that I was able to level up a lot, both as a software engineer and as a collaborative professional thanks to my experiences here. I’m sure that anyone interning here in the future will be able to say the same by the time they’re done.</p><p>I want to take this opportunity to thank <a href="https://www.linkedin.com/in/sheethala/">Sheethala</a>, my mentor, who was always looking out for me to make sure I was having a valuable experience and that I had the help I needed. She was the one who pushed me to explore all of the different things I could work on and connected me to other people who were able to give me valuable insights in various domains. I also want to thank everyone on the Core team as well as everyone I worked with during my time at Drop for showing me what great teamwork looks like and allowing me to feel like a full-fledged member. I have you all to thank for building this environment that I felt like a comfortable, productive part of — not as “the intern”, but just David, a Drop employee just like everyone else.</p><p>I’ll soon be logging off for the last time, but I’m excited to see how Drop will grow as a company in the near future. Stay classy, everyone!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7bb97f1e8d41" width="1" height="1" alt=""><hr><p><a href="https://medium.com/drop-engineering/my-drop-eng-internship-7bb97f1e8d41">My Drop Eng Internship</a> was originally published in <a href="https://medium.com/drop-engineering">Drop Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Startup’s Research Experience during a Pandemic]]></title>
            <link>https://medium.com/drop-engineering/designing-a-startups-research-experience-during-a-pandemic-6c7521b2b091?source=rss----11c02d7aac33---4</link>
            <guid isPermaLink="false">https://medium.com/p/6c7521b2b091</guid>
            <category><![CDATA[startup-research]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[design-thinking]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[ux-research]]></category>
            <dc:creator><![CDATA[Alya Naumova]]></dc:creator>
            <pubDate>Tue, 02 Jun 2020 15:11:47 GMT</pubDate>
            <atom:updated>2020-06-02T15:11:47.233Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The UX Research team at Drop is exploring shorter, more engaging formats for communicating user insights, while seeking out new ways to include user viewpoints in everyday work for the whole team. Our call is to encourage the team to lean in into a deeper understanding of our users, as Covid-19 is making human connections more challenging and solid UX — vital.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GqpINLs4yuAzFUfRoeBEvg.jpeg" /></figure><p>During the time, when we collaborate through little Zoom windows, feeling close to the user — and acting on it — is far more challenging. Coming closer to a user may feel awkward. The pandemic has thrown many into emotional and financial turmoil. Observing people going through something disturbing and messy — as we go through it ourselves — is not an exciting task. From what we have seen as researchers — connecting with users becomes something non-researchers avoid, not throw themselves into.</p><p>And yet, the pandemic also makes it far more important than ever to deeper our understanding of what our users are after. Smaller tech companies are struggling to survive. Designing better user experiences is vital to that survival.</p><p><strong>To encourage the move <em>towards</em> our users the UXR team at Drop decided to make June the month, when we work to shrink the distance between our team and the users</strong>. Here is what we have set out to do and how we hope to accomplish that.</p><h3>Get the team re-engaged in the ongoing discussion on design and user experience</h3><p><strong>We will work to transform the existing platform for socializing UX research insights into a place, where people talk about user experience</strong></p><p>Many startups have a Slack channel called “Research,” so did we. In its heydays, we had most of the company included as members, with each post leading to healthy discussions threads. Since the beginning of the pandemic, the team has been engaging far less with our attempts to encourage discussion. The natural reaction to the lack of engagement from colleagues is to disengage. <strong>Instead, we are going to lean in.</strong></p><p>We renamed the channel to “Users +UX.” This was a call to action. We wanted to announce to the team that the up-date it as a space for discussions on user experience, not just a place for posting ongoing research highlights.</p><p><em>Instead of posting frequently like we used to — mostly snippets from user interviews and research highlights — we are switching to short once-a-day posts that capture smaller nuggets with a bigger bang — insights that speak directly to the impact of recent and ongoing design decisions. Each one will be accompanied with 1–2 questions to get the discussion going.</em></p><blockquote>We want our team to re-engage, and it is easier to engage with a snippet that looks like it could be swallowed whole and absorbed into the decision-making that same day.</blockquote><h3><strong>Make user-experience-focused thinking ubiquitous — on par with tracking data points and what to use for your zoom background</strong></h3><p><strong>We will post short clips of user interviews across high-traffic channels on slack.</strong></p><p>We are going to introduce these snippets with a well-crafted interpretive sentence, so that our colleagues were not left wondering: “What am I supposed to get out of this?” The idea is to cultivate consistent exposure to user experiences.</p><h3><strong>Democratize user-focused thinking</strong></h3><p><strong>We will bring back company lunches with users and make them frequent and easy to attend</strong></p><p>Back in pre-Covid times, these events used to be an hour long. They were hosted in our large communal kitchen and came with a catered lunch. In June, we are making them weekly and 20 minutes long. The objective here is to make them the easiest thing to attend, to remove all friction. Team members will be able to mute themselves and turn off their cameras, as they watch the two researchers interview a user. They can anonymously submit their questions to one of the facilitators. Following the interview, we post 2 questions on Users + UX channel to get the processing going.</p><h3><strong>Give the team a sense of ownership over their connection with users</strong></h3><p><strong>We will make ongoing user insights more accessible and looking for them more fun</strong></p><p>Including cross-functional team-members in research sessions as note-takers is an established best practice in UX Research. So is bringing your team members in at the synthesis and analysis stages. All that requires a researcher to act as a facilitator.</p><p>Given the size of our company, we always encouraged a cross-functional research practice, with multiple teams running their own studies with our support.</p><blockquote>So, we thought: “What if we democratized access to research even further and made it possible for designers to look at ongoing research work as it was enveloping?” For instance, why not make it possible for a designer working on a feature to go into <em>a</em> platform that held interview transcripts and clipped videos and look up what we have pertaining to a tag “Content discovery,” “Search,” or “Depth of Scroll?”</blockquote><p>After careful search, we picked Condens as the most cost-efficient way to help us accomplish that. We paid for an additional admin account to make it available to the team. We brought in a designer to help us demo it during an all-company meeting, because we wanted to add a perspective of a research user. We asked the designer how they would use it and asked them to demonstrate a specific use-case to everyone.</p><h3><strong>Make our reports more useful, usable, and enjoyable</strong></h3><p><strong>We reconsidered the way we created our reports from the point of view of our research users — our team. </strong>Our reporting process prior consisted of posting ongoing research insights daily and then walking the team through a deck. This process — although standard in the UXR world — is designed to support reporting, it is not made for use by designers and Product.</p><p><strong>We did away with the decks and presentations. Instead, we are crafting reports that look more like Medium articles — but with more white space, pictures, and clipped videos.</strong> They are short, to the point, and focus on a specific issue at a time. This new format also allows us to play with new ways of communicating our insights. This far, we were able to easily put together user journeys (something that used to require involving a designer!), but we are also looking forward to experimenting with including hand-drawn sketches that could take stakeholders through a user’s decision-making process.</p><p><em>*The article includes references to the platform Condens. Condens did not pay for this promotion. We genuinely like their product and it helped us solve a number of issues.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c7521b2b091" width="1" height="1" alt=""><hr><p><a href="https://medium.com/drop-engineering/designing-a-startups-research-experience-during-a-pandemic-6c7521b2b091">Designing a Startup’s Research Experience during a Pandemic</a> was originally published in <a href="https://medium.com/drop-engineering">Drop Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Drop’s Data Lake]]></title>
            <link>https://medium.com/drop-engineering/drops-data-lake-49aeb80e12d9?source=rss----11c02d7aac33---4</link>
            <guid isPermaLink="false">https://medium.com/p/49aeb80e12d9</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-infrastructure]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Michael Chau]]></dc:creator>
            <pubDate>Mon, 09 Mar 2020 11:01:01 GMT</pubDate>
            <atom:updated>2020-03-09T11:01:01.369Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RWTqxbRbzuIOGVIQHAqq4g.jpeg" /></figure><p>Drop is an intelligent rewards platform, aimed at levelling up the lives of our members by rewarding them with Drop points for shopping at brands they already love, as well as helping them discover new brands that resonate with their lifestyle. Core to enabling this experience is Drop’s commitment to promoting a data informed culture throughout the organization. Data at Drop is critically depended upon by every function of the business and takes many forms including but not limited to business intelligence, measuring experiments, and building machine learning models.</p><p>To ensure that we are effectively leveraging data at Drop our engineering team is constantly looking for ways to improve our data infrastructure to scale with our present and future needs. Similar to the experiences of many other high-growth startups, the scale of our data needs were exceeding the capabilities of our infrastructure. We needed to evolve our business intelligence centric data infrastructure to one that could unlock our big data requirements and ambitions.</p><h3>Motivation</h3><p>Early as a company, Drop invested in data infrastructure in an effort to provide visibility for key business metrics using automated reports and dashboards. The first generation of our data infrastructure started with a data warehouse centric architecture using <a href="https://aws.amazon.com/redshift/">AWS Redshift</a> as our data warehouse appliance, <a href="https://airflow.apache.org/">Apache Airflow</a> for scheduling our custom batch ETL jobs, and <a href="https://looker.com/">Looker</a> for automated dashboards and reporting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dDzqNJwsU7Zv8eeM8-Di2A.png" /><figcaption>First Generation Data Infrastructure: 2017–2019</figcaption></figure><p>While this architecture was very successful in accomplishing our early goals, the appetite to leverage data was extending past that of business intelligence. To provide our app members with a more relevant and personalized experience, we needed a data platform that could support advanced capabilities such as deeper analytics, comprehensive measurement of experiments, and development of machine learning models. We immediately recognized the technical limitations associated with the original architecture, which will likely prevent us from unlocking those desired capabilities. Some of the most prominent limitations included:</p><ul><li>Run-time of critical batch ETL jobs were skyrocketing as the size of our data scaled, and some jobs could no longer be run under 24 hours.</li><li>AWS Redshift couples compute with storage, limiting our abilities to scale the data we ingested into the warehouse. We became very selective of which data was made available in Redshift, which deviated its position as “center of truth”.</li><li>Building our machine learning infrastructure on top of our Redshift cluster was not ideal because of Redshift’s requirement to work only with structured data, and high latency associated with the batch ETL jobs needed to populate the warehouse.</li></ul><p>To address these critical short-comings, and put us in a better position to scale for the long-term, we had to evolve our data infrastructure. A <a href="https://en.wikipedia.org/wiki/Data_lake">data lake</a> is a centralized repository that is capable of storing data in both unstructured and structured formats at any scale. Building a data lake at Drop would enable us to integrate new technologies and services that address the limitations experienced by our first generation data infrastructure, while allowing us to preserve critical components of our original infrastructure that still fit into our long-term data infrastructure plans.</p><h3>Building the Lake</h3><p>When we set out to build Drop’s data lake, we followed these guiding principles:</p><ol><li><strong>Keep our technology stack simple: </strong>As one of Drop Engineering’s core <a href="https://drop.engineering/3-decisions-weve-made-while-growing-drop-s-tech-c87e0e4104de">philosophies</a>, we aimed to keep our tech stack simple by leveraging existing and proven AWS technologies, and only adopting services that would drive a substantial impact. Drop is primarily an AWS shop, so continuing to leverage AWS technologies made sense because of our existing AWS experience, the ability to prototype new features quickly, and the inherent integration benefits of leveraging other services within the AWS ecosystem. We also made a commitment to stay/remain a Python and SQL shop across the data team. Therefore, rather than learning new languages that could be leveraged for data ingestion or processing, all of our <a href="https://spark.apache.org/">Apache Spark</a> jobs are instead written in PySpark.</li><li><strong>Don’t reinvent the wheel: </strong>We<strong> </strong>looked to the open-source and tech community for successful data lake implementation stories, and key learnings. We turned to <a href="https://eng.uber.com/uber-big-data-platform/">Uber</a> and <a href="https://medium.com/airbnb-engineering/data-infrastructure-at-airbnb-8adfb34f169c">Airbnb</a> for inspiration in regards to high-level architecture. For more technology specific knowledge, we relied on posted AWS re:Invent <a href="https://www.youtube.com/channel/UCd6MoB9NC6uYN2grvUNT-Zg">talks</a>. Where it all came together was by attending AWS re:Invent in 2018, and hearing <a href="https://youtu.be/rlobQYMb7zY">Robinhood’s story</a> on how they implemented their data lake. Robinhood’s story resonated deeply with ours because of the similarities of our technology stack, problems we were looking to solve, and shared philosophies for how to tackle the problem. The culmination of various components from all of these stories and resources served as the initial blueprint for our data lake.</li><li><strong>Make sure it can scale: </strong>The scale of our data, and the complexity to leverage it grew rapidly as our organization transitioned from an early-stage startup to a high-growth organization. This meant identifying technologies that could perform effectively in exponential growth scenarios while minimizing the associated technical complexity overhead.</li></ol><h3>Architecture Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zvtBia6FE17xfuI1nevBOg.png" /><figcaption>Second Generation Data Infrastructure: 2019 — Present</figcaption></figure><h3>Data Lake Layers</h3><h4>Scheduling</h4><p>This scheduling layer acts like an air traffic controller responsible for managing and executing all workflows throughout the data lake. A majority of all data movement throughout the data lake is coordinated by the scheduler. We made the decision to double-down on our commitment to continue using Apache Airflow as our scheduler for numerous strategic reasons. Utilizing Airflow enables us to build all of our work flows and infrastructure as code via Airflow Directed Acyclic Graphs (DAGs). This key decision also simplified our engineering development and deployment processes, while providing version control for all aspects of our data infrastructure. The Apache Airflow project also contains a plethora of well supported AWS integration libraries and example DAGs which allows us to integrate and evaluate new technologies quickly.</p><h4>Ingestion</h4><p>While we pull data from multiple disparate data sources such as server logs, growth marketing platforms, and business operations services, data is primarily ingested into the data lake from our RDS Postgres database and enters in two forms: batch or stream. To enable these ingestion forms, we chose <a href="https://aws.amazon.com/dms/">AWS Database Migration Services (DMS)</a> because of its native integration with RDS Postgres and its ability to ingest data into S3 in both batch and stream formats. Stream data enters the lake using DMS’s “Change Data Capture” (CDC) tasks, in which DMS replicates data from the Postgres “Write Ahead Logs (WAL)” and generates <a href="https://en.wikipedia.org/wiki/Apache_Parquet">Apache Parquet</a> files in S3 in one-minute partitions. Batch data enters the lake using DMS’s “Full-Migration” tasks in the form of snapshots of the entire table from Postgres into S3. The batch pipeline is managed by the Scheduling layer via Airflow in the form of custom DAGs. This allows us to control the schedule of batch snapshots into S3, and reduces service costs by spinning down unused DMS resources at the end of the batch run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QARaIpo7_vyiDsKi" /><figcaption>Drop’s DMS “Full-Migration” Airflow DAG</figcaption></figure><h4>Processing</h4><p>The Processing layer is responsible for transforming data from the “Raw” section of the storage layer, to a standardized columnar format and partition structure in the “Columnarized” section of the lake. We use the <a href="https://en.wikipedia.org/wiki/Lambda_architecture">Lambda Architecture</a> for reconciling the batch and stream data of a given data source. This data processing model allows us to combine high-quality batch snapshots that we treat as the “gold-standard” data, with the speed of our one-minute latency stream files to produce the most up to date columnar version of a given data set. We use <a href="https://aws.amazon.com/glue/">AWS Glue</a> and its Data Catalog as our data lake’s central metastore management service. This metastore contains metadata on each data set, such as location within S3, structure definition, and overall size. This metadata can also be captured and updated using <a href="https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html">AWS Glue Crawlers</a>. The overall process is as follows:</p><ol><li>The Scheduling layer kicks off a scheduled Lambda Architecture processing DAG of a specific data set.</li><li>The DAG uploads the PySpark application to S3, spins up an <a href="https://aws.amazon.com/emr/">AWS EMR </a>cluster, and runs the PySpark application as an EMR step.</li><li>Both batch and stream data from the “Raw” section of the storage layer are sourced as inputs to the EMR Spark Application, and the final output is a Parquet dataset reconciled using the Lambda Architecture outputted in the “Columnarized” section of the Storage Layer</li><li>Upon Completion of the EMR step, the EMR cluster is terminated, which ultimately reduces EMR costs.</li><li>An AWS Glue Crawler is run to update the table metadata in the AWS Glue catalog, which acts as the central metastore for the entire lake.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nTSTqmns_9Rr0y5GP9Q8Ow.png" /><figcaption>Drop’s Lambda Architecture Overview</figcaption></figure><h4>Storage</h4><p><a href="https://aws.amazon.com/s3/">S3</a> is our primary storage platform for three reasons: S3’s ease of use, high-reliability, and relatively low-cost. Data is stored in the storage layer in three sections: “Raw”, and “Columnarized” which are both based in S3, and “Warehouse” in which data resides in our Redshift cluster. All data in the lake is first landed in the “Raw” section in various formats via the Ingestion layer. For data to reside in the “Columnarized” section, it is transformed to a columnar format sourced from the “Raw” section via the Processing layer and adheres to our partitioning standards. Apache Parquet is our standard columnar file format because of its widespread adoption across many data processing services, as well as its performance and storage benefits. The “Warehouse” section of the Storage layer is comprised of our Redshift data warehouse in which data is sourced originally from the “Raw” and “Columnarized” sections. Each section of the Storage layer can also be classified by the “temperature” of the data, a metaphor for the data’s frequency of access. “Raw” data is the coldest, “Columnarized” is warmer than “Raw”, and “Warehouse” data is the hottest because of its frequent access from automated dashboards and reports. Another advantage of S3 is our ability to control our costs as the amount of our data scales. For colder data, we can update the S3 storage class type to a more economical option, or migrate the data to <a href="https://aws.amazon.com/glacier/">AWS Glacier</a> in an effort to reduce costs.</p><h4>Accessing Data from the Lake</h4><p>The data lake was built to be leveraged by both technical and non-technical teams in an effort to use data to better inform our decisions and enhance the overall product experience. Our non-technical teams commonly access data through our business intelligence platform, Looker, by monitoring metrics in the form of automated dashboards and reports.</p><p>Our technical teams typically have a broader scale of use-cases for accessing data from the lake ranging from ad hoc queries as part of an exploratory analysis, to developing machine learning models and pipelines. Data within the lake can be queried directly by <a href="https://aws.amazon.com/athena/">AWS Athena</a>, a managed <a href="https://prestodb.io/">Presto</a> service, or through our Redshift data warehouse. AWS Athena is also integrated with AWS Glue’s Data Catalog, which enables Athena to have full access to our central metastore. This native integration allows Athena to act as the primary querying service for data stored in S3. We are also able to leverage <a href="https://docs.aws.amazon.com/redshift/latest/dg/c-getting-started-using-spectrum.html">Redshift Spectrum</a> to query data from S3, which is particularly useful in ad hoc situations in which we would like to enrich data within our Redshift cluster with data that resides in S3. For more formal exploratory analysis and development of machine learning models, teams typically work out of Jupyter Notebooks and use a combination of Spark and our Glue catalog to query and interact directly with the objects in S3.</p><h4>Validation &amp; Monitoring</h4><p>Ensuring that consumers of the lake have complete trust in the accuracy of the data is of paramount importance to us. We have built a comprehensive set of checks and monitors to alert us of any unexpected issues. With a vast majority of our data lake processes facilitated by Airflow DAGs, we are able to monitor Airflow job failures using Datadog, and forward critical issues to our on-call engineers via <a href="https://www.pagerduty.com/platform/">PagerDuty</a>. To establish data quality checks, we have also built a series of Airflow DAGs to generate custom data validation metrics specific to a data set. A standard set of validation checks and metrics that we run on a given data set include, but are not limited to, null checks, row count checks, and data latency metrics. Each of these metrics are pushed to Datadog where we can monitor anomalies and alert the necessary parties when necessary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nNVhcBIVlC5MGEBVskjR5Q.png" /><figcaption>Drop’s Airflow DAG for pushing custom validation metrics to Datadog</figcaption></figure><h3>Current State</h3><p>Drop’s data lake is now ingesting over a billion records from our Postgres database daily, and processing terabyte sized jobs. The lake has now been adopted by numerous teams and directly contributes to our enhanced data analytics capabilities, as well as the ability to develop and ship machine learning models. Overall we are happy with the early results, and we are confident that this evolution of our data infrastructure puts us in a much better position to scale with our present and future needs.</p><h3>Key Learnings</h3><p>Our implementation journey had its share of ups and downs, and we thought it would be valuable to share some of our biggest take-aways:</p><ul><li><strong>The benefits of ingesting data in columnar formats: </strong>Although, the “Raw” section of the storage layer is file format agnostic, there are downstream benefits to ingesting data in columnar formats. When ingesting data from our Postgres tables that contained columns with verbose and complex text blobs, such as jsonb columns, parsing data downstream became a nightmare, and further relying on <a href="https://cwiki.apache.org/confluence/display/Hive/SerDe">serDe</a> files to assist with parsing data only increased the overall complexity. Configuring DMS to output the files in Parquet format maintained the schema integrity of the columns because of parquet’s columnar nature, and our downstream processing layer jobs saw tremendous performance gains in terms of speed and data quality. This trade-off comes at the cost of increased memory resources needed for DMS to generate the parquet files.</li><li><strong>Adopting Spark into our stack: </strong>Our engineering team had limited prior in-house <a href="https://hadoop.apache.org/">Hadoop</a> experience, therefore adopting Spark came with its share of struggles. While there is definitely a plethora of resources available for Spark, our biggest pain points came in regards to resource management and trouble-shooting errors. We opted to use EMR over AWS Glue for ETL jobs because of the increased level of control over utilized resources, which ultimately led to both performance and cost-saving gains, but came at the cost of complexity. The learning curve for configuring the optimal amount of EMR resources for each of our Spark jobs was steep, and we quickly learned that these optimization efforts quickly became a costly time sink. When EMR jobs failed, we also had more difficulty than expected trouble-shooting the root-cause. In a lot of our cases, the error logs generated by EMR clusters were not granular enough, and inadvertently masked important details. Through improving our logging processes, and directly referencing the Spark executor logs, we were able to surface more detailed information that better highlighted the root-causes.</li><li><strong>Turn-off Idle Resources:</strong> We can credit a majority of our cost-savings to our experience with working with transient workloads. With almost all of our data lake operations composed as Airflow DAGs, we can automate when to remove unused resources. Our ability to spin-up DMS replication instances only when needed for our batch ingestion jobs via Airflow has allowed us to save over 90% of the equivalent costs of keeping the instances available at all times. Similarly we only spin-up EMR clusters using spot instances when our Airflow schedules a Processing Layer Job, and terminates the cluster upon completion, which for us equates to an average savings of over 70% in comparison to having the nodes available at all times.</li></ul><h3>Next Steps</h3><p>We are constantly looking for ways to improve our data infrastructure, and we have already started to investigate a number of initiatives including:</p><ul><li>Improving data availability, and version control management within our Storage layer through a technology like <a href="https://hudi.apache.org/">Apache Hudi</a> or <a href="https://delta.io/">Delta Lake</a>.</li><li>Tweaking our ingestion layer abilities by using tools currently used by the product engineering teams for event driven development via <a href="https://kafka.apache.org/">Apache Kafka</a> and <a href="https://debezium.io/">Debezium</a>.</li><li>Improving data access governance with tools like <a href="https://aws.amazon.com/lake-formation/">AWS Lake Formation</a>, where we can impose granular controls on what data can be accessed by our teams.</li></ul><p>Lastly, kudos to the entire Drop Data Engineering team for tackling this initiative in such a short period of time, and to Darren Fung for his unwavering support and commitment to this project since day one. Feel free to drop us a line if these kinds of projects resonate with you as we are always hiring!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49aeb80e12d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/drop-engineering/drops-data-lake-49aeb80e12d9">Drop’s Data Lake</a> was originally published in <a href="https://medium.com/drop-engineering">Drop Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveling Up Transactions @ Drop — INT to BIG INT!]]></title>
            <link>https://medium.com/drop-engineering/leveling-up-transactions-drop-int-to-big-int-41a8fecc9ca1?source=rss----11c02d7aac33---4</link>
            <guid isPermaLink="false">https://medium.com/p/41a8fecc9ca1</guid>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Harsh Shah]]></dc:creator>
            <pubDate>Fri, 04 Oct 2019 17:12:15 GMT</pubDate>
            <atom:updated>2019-10-04T17:17:03.864Z</atom:updated>
            <content:encoded><![CDATA[<h3>Leveling Up Transactions @ Drop — INT to BIG INT!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OGMcNh3jrzd-ZLnM15u4dQ.png" /></figure><h3>The Importance of Transactions at Drop</h3><p>Transaction data is core to Drop’s ability to reward our members and personalize their shopping experience. Using machine learning models and transaction data, we have surfaced highly relevant brands to members they would not otherwise be aware of.</p><p>Before diving deeper, let’s take a quick tour of what happens behind the scenes when our member, Jamie, is awarded points.</p><ul><li>Jamie links a credit/debit card to the Drop account</li><li>Jamie makes a purchase at Walmart online/in-store</li><li>Drop securely fetches transactions and maps it to an offer activated by Jamie</li><li>Jamie is awarded Drop points for the purchase! 🎉</li></ul><p>In this post, I’m going to share a scaling challenge that we faced as we experienced significant member growth and an explosive amount of transaction data.</p><h3>How scale broke the system</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/0*gerd-W4S_98vxf5m" /></figure><p>We use PostgreSQL as our database to store transactions. The primary key for the transactions table was created with the SERIAL data type (i.e. an auto-incrementing column of integers). This data type served our needs until one fine day we exhausted the number of transaction IDs we could generate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-62c5lcpLu9VSi53" /></figure><p>Reaching the maximum positive signed integer limit (2³¹ — 1) in PostgreSQL meant that all subsequent inserts would fail since it is not able to generate the next primary key value. New transactions weren’t able to be inserted, we weren’t able to award points to our members, resulting in a degraded production system.</p><h3>Building a scalable solution</h3><p>With a member first philosophy, our primary concern was to award points to our members as soon as possible! We were brainstorming ideas that would satisfy the following, in order of importance:</p><ol><li>Get members their points ASAP</li><li>Guarantee data integrity</li><li>Smaller and parallel iterations that could be verified at every stage</li><li>Minimize application code change</li></ol><blockquote>“Quality tech doesn’t happen in silos”</blockquote><p>We came up with a 5 step solution that would allow us to satisfy the constraints and minimize downtime. We’ll first go over the strategies behind the solution, then dig deeper into how we executed the steps.</p><h3>Strategy</h3><h4>Stop-gap Fix</h4><p>Seeing the urgency for a solution and the complexity of the problem, we used the negative integer space as a temporary stop-gap fix. Since an integer is a signed data type in PostgreSQL, we leveraged its signed nature to our advantage. We allowed the primary key to have negative values starting from -1. In theory, this gave us room for an additional 2³¹ IDs, but in the negative space up to -2³¹. Although negative IDs are not ideal and a bit weird, this allowed us to immediately restore the processing of transactions and resume awarding points to our members.</p><p>Now that our system was restored and resumed processing transactions, the next step for us was to come up with a long term solution by migrating over to the BIG INT column type.</p><h4>Create new column of type BIG INT</h4><p>The range of a signed BIG INT (or BIG SERIAL) is from -2⁶³ to 2⁶³ -1, which is a little over 9.2 Quintilian values in the positive space. By migrating the primary key column to the BIG INT type, we would have a solution that would allow us to grow our transactions data set by 2³², which would last us a long time. Thus, we added a column of type BIG INT to the transactions table — <em>id_bigint</em>. This column would be populated in tandem with the original ID column that is currently counting backward until we have switched over to the new column.</p><h4>Start populating in the new ID column</h4><p>Once we added the new column, we updated our application code so that every new insert into the transactions table would also be inserting into the new BIG INT column if the column was present. This conditional insert is important to call out because later we would be replacing the old column with this new column and we wanted to avoid synchronizing code changes and deployments when this happened.</p><p>At the same time, we started back-filling historical data into the new column, a very slow and resource-intensive process. Naively trying, <em>“UPDATE transactions SET id_bigint=id”</em> was not an option for us since it would have to update more than 2³¹ records in one go, and it would have completely blown up the database in terms of CPU and IOPS. Updating in batches was the only option for us, as we wanted to control the impact on CPU and IOPS to not degrade other services. Based on our staging environment (which is identical to our production environment), this update would have taken us 4–5 days.</p><h4>Prepare migration to the new ID column</h4><p>Before migrating to the new column, we had to ensure that it had the same indices and foreign keys as the existing column. We created a new unique index on the new column, as it would serve as a primary key constraint. The unique index was created using PostgreSQL’s <em>CREATE INDEX CONCURRENTLY</em><strong> </strong>option. By using the concurrently option, PostgreSQL built the index without acquiring any locks that otherwise would have blocked inserts, updates, or deletes from happening at the same time. The caveat is that index creation takes longer to finish, but this allowed us to continue to use the table as it was being built.</p><p>This prep work sped up the migration to the new column and minimized the downtime.</p><h4>Switch over to the new primary key column</h4><p>At this point, we were fully ready to migrate to the new column. To do this, we dropped the existing primary key constraint on the transactions table, created a new primary key constraint using the unique index we created earlier, and then we renamed old and new ID columns so that they replaced each other. Finally, we dropped the old sequence that was decrementing into the negative space and created a new sequence for the new column to start from where we had left off in the positive space (2³¹).</p><h3>Execution</h3><p>With any engineering challenge, we need to have a step by step plan to execute safely, and with minimal disruption. We started by writing all the SQL queries required to run as part of the migration. Then we ran the queries in our staging environment, to get a sense of the time it would take to execute.</p><p>For long-running queries or queries that needed locks at the table level, we broke them down to smaller queries, then re-ran it on our staging environment just to make sure we didn’t miss anything during our query plan optimization. We were prepared for the worst — what-if rolling back goes sideways as well? To mitigate failure effects, we decided to create a manual database snapshots after each step. We notified our internal stakeholders about the plan and the timeline, and our members were notified with an in-app banner about the limited functionality during the final steps of the migration.</p><h4>Stop-gap Fix</h4><p>To start using negative integers as IDs, we simply modified the sequence associated with the existing ID column. We specified a new minimum value for the sequence, the new value to restart the sequence from, and set the increment value as -1 to have it change to a descending sequence.</p><pre>ALTER SEQUENCE transactions_id_seq INCREMENT BY -1 RESTART -1 MINVALUE -2147483648;</pre><h4>Create new column of type BIG INT</h4><p>This change was a very simple addition of a new column:</p><pre>ALTER TABLE transactions ADD COLUMN ‘id_bigint’ bigint;</pre><p>Our rollback strategy here was simply to drop the new column.</p><pre>ALTER TABLE transactions DROP COLUMN ‘id_bigint’;</pre><h4>Start populating in the new ID column</h4><p>Updating application code to start inserting into the new column was relatively trivial. However, back-filling historical data into the column in a controlled and efficient way was challenging. For this, we used asynchronous workers using our background job processing framework, Sidekiq. We were running the workers to batch update 10,000 records at a time. We came up with a batch size of 10,000 based on our experience running it on our staging environment and making sure it didn’t blow up the database. It took almost 5 days to back-fill all the historical data.</p><h4>Prepare migration to the new ID column</h4><p>We consciously split this step and the next step into two days to minimize downtime and degraded performance. As well, we ran these steps overnight when our database had relatively less load.</p><p>Based on our estimates from running it in a staging environment, creating the unique index on the new ID column would complete in about 6 hours. In reality, this took a little over 10 hours. We had expected the number to be higher since our staging environment does not have the same consistent load as on production. Since this was not a blocking operation, we weren’t too stressed about it as long as it successfully created a unique index.</p><pre>CREATE UNIQUE INDEX CONCURRENTLY transactions_pkey_new ON transactions(id_bigint);</pre><p>Rolling back this step simply meant dropping the index in case index creation was not successful or if the index became corrupted.</p><pre>DROP INDEX CONCURRENTLY IF EXISTS transactions_pkey_new;</pre><p>Next, we had to create foreign keys. Creating foreign key constraints requires a <em>SHARE UPDATE EXCLUSIVE</em> lock. This lock would prevent concurrent schema changes and VACUUM runs but <strong>would not </strong>impact inserts, updates, or deletes. This took us about 4 hours but since inserts and updates were not impacted, there was no downtime.</p><pre>BEGIN;<br>LOCK TABLE transactions IN SHARE UPDATE EXCLUSIVE MODE;<br>ALTER TABLE transactions ADD CONSTRAINT fk_rails_7b42c113e2_2 FOREIGN KEY (someother_id_bigint) REFERENCES transactions(id_bigint) NOT VALID;<br>ALTER TABLE transactions VALIDATE CONSTRAINT fk_rails_7b42c113e2_2;<br>COMMIT;</pre><p>If for any reason this step didn’t go as planned, then we would simply conditionally drop the constraint</p><pre>ALTER TABLE transactions DROP CONSTRAINT IF EXISTS fk_rails_7b42c113e2_2;</pre><p>The replacement of the old foreign key constraints with the new foreign key constraints was relatively trivial and ran almost instantaneously</p><pre>BEGIN;<br>LOCK TABLE transactions IN ACCESS EXCLUSIVE MODE;<br>ALTER TABLE transactions RENAME CONSTRAINT fk_rails_7b42c113e2 TO fk_rails_7b42c113e2_defunct;<br>ALTER TABLE transactions RENAME CONSTRAINT fk_rails_7b42c113e2_2 TO fk_rails_7b42c113e2;<br>COMMIT;</pre><h4>Switch over to the new primary key column</h4><p>This step was the most risky step of them all. Switching over to the new columns required an <em>ACCESS EXCLUSIVE</em> lock, so inserts and updates would be blocked but reads would go through. It took approximately 1.5 hours to add the primary key constraint in this step. All the other statements ran almost instantaneously. This was the only step where we had a degraded system, as for 1.5 hours, no new transactions (and thus, points) were being created.</p><pre>BEGIN;<br>LOCK TABLE transactions IN ACCESS EXCLUSIVE MODE;<br>ALTER TABLE transactions DROP CONSTRAINT IF EXISTS transactions_pkey CASCADE;<br>ALTER TABLE transactions ADD CONSTRAINT transactions_pkey PRIMARY KEY USING INDEX transactions_pkey_new;</pre><pre>ALTER TABLE transactions RENAME COLUMN id TO id_defunct;<br>ALTER TABLE transactions RENAME COLUMN id_bigint TO id;</pre><pre>DROP SEQUENCE transactions_id_seq CASCADE;<br>CREATE SEQUENCE transactions_id_seq START WITH 2147483648;<br>ALTER TABLE transactions ALTER COLUMN id SET DEFAULT NEXTVAL(‘transactions_id_seq’);<br>COMMIT;</pre><p>The rollback procedure for this step was a little more complicated of all the steps, as it involves undoing all the steps we had done but in reverse order.</p><pre>BEGIN;<br>LOCK TABLE transactions IN ACCESS EXCLUSIVE MODE;<br>ALTER TABLE transactions RENAME COLUMN id TO id_bigint;<br>ALTER TABLE transactions RENAME COLUMN id_defunct TO id;</pre><pre>ALTER TABLE transactions DROP CONSTRAINT IF EXISTS transactions_pkey;<br>CREATE UNIQUE INDEX transactions_pkey_id ON transactions (id);<br>ALTER TABLE transactions ADD CONSTRAINT transactions_pkey PRIMARY KEY USING INDEX transactions_pkey_id;</pre><pre>SELECT MIN(id) — 1 FROM transactions;<br>DROP SEQUENCE transactions_id_seq CASCADE;<br>CREATE SEQUENCE transactions_id_seq START WITH &lt;value from before&gt;; <br>ALTER TABLE transactions ALTER COLUMN id SET DEFAULT NEXTVAL(‘transactions_id_seq’);<br>COMMIT;</pre><h3>Learnings</h3><p>During this event, our error tracking tool, Sentry, was noisy in terms of the number of errors that weren’t actionable and we could have missed out on this error. Fortunately, that didn’t happen with us. Since then, we’ve actively worked on cleaning up the majority of non-actionable errors in Sentry to ensure most errors are actionable.</p><p>This incident also showed us the need to invest in our transaction ingestion infrastructure. As a result, we’ve invested significantly in re-designing our transaction services for ingestion performance and isolating it from our other production workloads.</p><p>Sh*t happens, and as long as we have actionable learnings and have measures in place so that we don’t repeat it, we treat this as a valuable learning experience. What we learned, albeit the hard way, is to be proactive rather than reactive. Storage is cheap, so new tables have a BIG INT primary key column by default. Particularly in this case, if we had necessary monitoring and alerting in place for high growth tables like transactions, we could have avoided this mayhem.</p><h3>Conclusion</h3><p>We believe in hiring problem solvers, not coders. Very few of us on the Drop engineering team had advanced database administration experience when this event happened, so it was a learning experience for many of us. Coming up with creative solutions to problems is what we strive for. For example, leveraging negative numbers as transaction IDs was not ideal but it was a creative fix to ensure our members’ experience was not affected while we agreed on the long term solution and fixing it the right way.</p><p>“Quality tech doesn’t happen in silos” is one of our Drop Tech values we live and breathe by. It might seem like a trivial process to migrate a primary key column to another primary key column, but it was very involved, challenging us to think out of the box while under pressure to get production up and running ASAP. It wasn’t a one-person army, so I would like to thank <a href="https://medium.com/u/b1b00664b871">Darren Fung</a>, <a href="https://medium.com/u/707ffcabc7b9">Fahad Fayyaz</a> and Ahmed Abu Lawi in helping at various points to get it to the finish line.</p><p>Does the way we build and firefight resonate with you? Well, you’ll be happy to know that we’re hiring! Take a look at our open roles <a href="https://jobs.lever.co/joindrop/">here.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41a8fecc9ca1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/drop-engineering/leveling-up-transactions-drop-int-to-big-int-41a8fecc9ca1">Leveling Up Transactions @ Drop — INT to BIG INT!</a> was originally published in <a href="https://medium.com/drop-engineering">Drop Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Recommender System Using Embeddings]]></title>
            <link>https://medium.com/drop-engineering/building-a-recommender-system-using-embeddings-de5a30e655aa?source=rss----11c02d7aac33---4</link>
            <guid isPermaLink="false">https://medium.com/p/de5a30e655aa</guid>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Grace Tenorio]]></dc:creator>
            <pubDate>Wed, 31 Jul 2019 15:46:38 GMT</pubDate>
            <atom:updated>2019-08-01T16:11:11.612Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WZ_DR3vu6eFsXE7KE-cy2g.png" /></figure><h3>Motivation</h3><p>At the core of the Drop ecosystem are two key entities: our members and our partner brands. Our focus at Drop is to provide both with exceptional value and consistently win their trust. What allows us to identify relationships between our members and our partners, and simultaneously deliver value? Data!</p><p>The Data Science and Machine Learning team at Drop recently developed a recommender model to power the “Recommended For You” feature in the app. With our members entrusting us with their transaction data (the protection of which we take very seriously as explained <a href="https://www.earnwithdrop.com/pages/security">here</a>), we are able to help them earn points faster by surfacing offers for brands they are likely to shop at. In addition, we are able to enrich their shopping experience by helping them discover brands they would not otherwise interact with. The impact for our partners: revenue driven by engaged and loyal customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EbD8hYDSM7LaRUqRWTzJ0g.png" /><figcaption>Figure 1: “Recommended for you” app feature</figcaption></figure><h3>Basic Components of Our Recommender Model</h3><p>The input to the model is a “member journey” data set: for each member, we have a list of all their transactions within a specified timeframe. Each transaction has metadata, which includes the brand name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RM8H41vWu0OH-xXxAebemA.png" /><figcaption>Figure 2: Two member journeys based on transaction data</figcaption></figure><p>The output of the model is a “brand recommendations” data set: for each member, we have a ranked list of <em>n</em> brands. The brand at the top of the list represents the retailer where the member is most likely to shop next:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ApH_4Zu9VlS_qI_Mc-i4Yg.png" /><figcaption>Figure 3: Sample recommendations for two members</figcaption></figure><p>The team experimented with a couple of models, both of which involved learning embeddings to represent the members and brands on our platform. In the end, we decided to deploy a sequence model to production as it achieved the best performance across key metrics. Before we start diving into the details of the model, let’s first try to gain intuition on what embeddings are and why they make sense in the context of recommender systems.</p><h3>Embeddings</h3><h4>What are Embeddings?</h4><p>Embeddings are a way to represent entities using learned vectors. Simply put, it is a mapping of discrete objects to a sequence of continuous numbers. Given enough data, we can train an algorithm to understand relationships between entities and automatically learn features to represent them. This automated approach is powerful as it allows us to scale more easily. For a platform like Drop, being able to scale is top of mind as new entities are constantly being added to our system.</p><p>To help understand embeddings, let’s take a look at how we might represent the brands on our platform.</p><h4>Brand Embeddings</h4><p>A brand embedding is a mapping of a brand to a vector of continuous numbers. Below are examples of brands, each mapped to an 8-dimensional vector:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x--s20pQZpoFGJUctglraw.png" /><figcaption>Figure 4: Mapping from brand to vector</figcaption></figure><p>Each vector component corresponds to a brand feature or attribute that an algorithm has learned. For example, it may have learned the concept of “food” and encoded this information in the first dimension of the vector. A large numeric value in a given dimension indicates that the brand strongly exhibits the feature. We see that this is the case for Uber Eats (0.96), Foodora (0.92) and Costco (0.86), brands that we know are related to food. Another example of a learned feature may be the concept of “service”, which the algorithm may have encoded in the second dimension of the vector. Note the high values for Uber Eats (0.98) and Foodora (0.94), which are both food delivery services, as well as Lyft (0.96), a ride sharing service.</p><p>The concepts of “food” and “service” seem reasonable choices for describing the first two dimensions of the vectors. However, in practice, it is difficult to interpret what these learned features actually represent. For our specific use case, this is not a concern as the benefits of the learned embeddings far outweigh the loss in interpretability. In addition to scalability, one major advantage of the brand embeddings is that they can be used to compare similarity between brands. This is useful in the context of recommender systems, as it allows us to give recommendations to our members simply by identifying brands similar to those they have previously interacted with.</p><p>Concretely, given that a member has ordered from Uber Eats in the past, we can make recommendations by finding brands similar to Uber Eats. Intuitively, this translates to identifying brands with embeddings close to that of Uber Eats. Plotting the first two dimensions of our brand embeddings, we’ll see that brands that are similar to each other are closer together, while brands that are dissimilar are further apart. In this example, the brand closest to Uber Eats and therefore most similar, is Foodora.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DNxJ9IJckDqZ_zb5JC1-jA.png" /><figcaption>Figure 5: Similar brands are closer together</figcaption></figure><h4>Member Embeddings</h4><p>We can leverage the learned brand embeddings in order to find representations for our members. While brand embeddings encode the extent to which a brand exhibits certain attributes, member embeddings can be constructed to reflect member preference for these attributes[1]. Given a member with tendency to order in food, we can expect their representation to encode strong preference for features like “food” and “service”. In other words, we would expect this member to score high, on average, on these two features.</p><p>To illustrate this idea, let’s assume this member’s journey is depicted below and that the first two dimensions of the brand embeddings correspond to “food” and “service”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZTpphVGG0Vc2naN65afU-g.png" /><figcaption>Figure 6: A sample member journey</figcaption></figure><p>One way to represent this member is to take an average of the brand embeddings corresponding to their member journey. For example, to quantify their preference for “food”, we can take the first dimension of the embeddings and compute an average as follows: (0.96 + 0.92 + 0.86 + 0.96) / 4 = 0.93. Applying the same methodology to the “service” dimension, the resulting representation for this member is the vector (0.93 0.62). Given the member journey above, this representation makes sense as all four brands are “food” brands, and all but Costco are “service” brands.</p><h4>Measuring Similarity with Dot Product</h4><p>We now have two sets of embeddings where the corresponding dimensions represent the same features. To understand how much a member will like a given brand, we simply need to quantify similarity between their embeddings. One approach is to compute their dot product. Recall that given two n-dimensional vectors, a and b, where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Upx_jAJ2zJZQDU3x6Hwwug.png" /></figure><p>their dot product is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q-tRCZB-fdiMfofJ5NEVtw.png" /></figure><p>Intuitively, the higher the values for the corresponding features, the higher the likelihood that the member will identify with the brand. Below are concrete examples showing the dot products between our previously computed member embedding and the Uber Eats and Costco brand embeddings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sYWtmoCIurkkVy6MPw5wYA.png" /><figcaption>Figure 7: Computing the dot product</figcaption></figure><p>Based on the resulting dot products, we can conclude that the member is more likely to prefer Uber Eats than Costco. Overlaying the member embedding to our plot in <em>Figure 5</em>, we can see that it is indeed closer to Uber Eats than it is to Costco.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s-igZz9CtHYyB6lWiKidmQ.png" /><figcaption>Figure 8: Brand and member embeddings mapped in the same space</figcaption></figure><h3>Sequence Model</h3><h4>Overview</h4><p>As previously mentioned, we deployed a sequence model in production in order to provide brand recommendations to our members. In a nutshell, our model exploits the sequential nature of transaction data and treats generating recommendations as a sequence prediction task. Concretely, given a sequence of brands where a member has previously shopped, the model predicts the next brand that the member will most likely interact with. Under the hood, our model is implemented using a neural network whose objective is to learn embeddings to represent the brands on our platform.</p><h4>Training</h4><p>The goal of the training process is to learn a set of brand embeddings using our member journey dataset as input. Recall that this dataset contains, for each member, a sequence representing the brands where they have previously shopped. The algorithm’s task is to predict the next brand in each member’s sequence. Explicitly, for the following sequence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wjsbglQXsbk4kxIkDTuS1Q.png" /></figure><p>the algorithm makes predictions for each of the following sub-sequences:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3fa7KYzGYyYklJroJMYE-Q.png" /></figure><p>In order for the algorithm to learn, we need a way for it to know how well it’s performing on its task. Specifically, it needs to know when it gets the predictions wrong, so it can make proper adjustments to the brand embeddings that it has learned so far. This leads us to the concept of a loss function, which is a way to quantify how well or poorly an algorithm is performing given its learning task.</p><p>Conceptually, for our sequence prediction task, all we want is for the algorithm to compute a higher score for the brand the member actually interacted with next, than one it did not. The former is referred to as a “positive” example (ie: what actually happened), and the latter is called a “negative” example. Ideally, we would like the algorithm to consistently rank the positive example higher than its negative counterpart. To promote this type of ranking, we chose the Bayesian Personalized Ranking loss function[2] to train our model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EViXrdxdi-1AsYVSp-HIMA.png" /></figure><p>Recall that the <a href="https://en.wikipedia.org/wiki/Sigmoid_function">sigmoid</a> function, sigmoid(x) approaches 1 when x is large and 0 when x is small. Therefore loss is minimized (ie: algorithm is doing well) when the score for the positive example is greater than that of the negative example. When loss is high, we need the algorithm to correct its mistake by adjusting the values of the brand embeddings.</p><h4>Inference</h4><p>Using our sequence model to make predictions is easy. Given the following member journey, the goal is to predict the next brand that the member will most likely interact with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*boDUA-UobRxGnq8LicfMKQ.png" /></figure><p>Once the brand embeddings are learned, we simply represent our member as the mean of the embeddings corresponding to the brands they have previously interacted with (recall <em>Figure 6 </em>in “Member Embeddings”). We then take the learned embeddings for each brand on our platform and compute their dot product with the derived member embedding (recall <em>Figure 7 </em>in “Measuring Similarity with Dot Product”<em>)</em>. From this point on, it becomes a ranking exercise where the brand with the highest score represents the next brand that the member will most likely interact with. Note that we perform these computations for each member on our platform.</p><h4>Visualizing Our Brand Embeddings</h4><p>In order to visualize the brand embeddings that our sequence model has learned, we need to map our 8-dimensional vectors to a lower dimensional space so they can be plotted. One way to achieve this is by using a dimensionality reduction technique such as <a href="https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding">t-SNE</a>. Below is a video showing some of the clusters that emerged using our brand embeddings, reduced to 2-dimensional space.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FMi2Z0hxuH5k%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DMi2Z0hxuH5k&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FMi2Z0hxuH5k%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2908508706dc772564918267c1fb15c/href">https://medium.com/media/d2908508706dc772564918267c1fb15c/href</a></iframe><p>For a closer look, we highlight the following clusters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xkqsScVJBKzKHcRgB5Ilgw.png" /><figcaption>Figure 9: Canadian, Travel and Pet clusters</figcaption></figure><h3>Impact</h3><p>With our members entrusting us with their data, we are able to provide value by optimizing their experience on our platform. By surfacing relevant brands, we are able to help them earn points faster, thus getting them closer to redeeming their next reward. For our partners, the value-add is a growing consumer base with strong brand affinity.</p><p>Internally, the learned brand embeddings have become an integral part of various initiatives at Drop. Some of the areas where they are being leveraged include our product on-boarding flow, targeted marketing communications, as well as lead generation for our business development team.</p><h3>Future Direction</h3><p>While we have achieved remarkable performance with our existing model, the team is excited to explore ideas to further enhance our recommender. Some initial thoughts revolve around using temporal information as input into the model, as well as leveraging explicit feedback we receive from our members through the Drop app (eg: clicks on offers the recommender has surfaced). There is also the possibility of leveraging text and image data related to brands and products in order to build hybrid recommender models. If these future initiatives sound exciting to you, drop us a line! We’re always on the lookout for great people to join our growing team. :)</p><h3>References</h3><p>[1] Kula, M. (2015). <a href="https://arxiv.org/pdf/1507.08439.pdf">Metadata embeddings for user and item cold-start recommendations</a>. arXiv:1507.08439.</p><p>[2] Rendle, S. et al (2009). <a href="https://arxiv.org/pdf/1205.2618.pdf">BPR: Bayesian Personalized Ranking from Implicit Feedback</a>. arXiv:1205.2618.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de5a30e655aa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/drop-engineering/building-a-recommender-system-using-embeddings-de5a30e655aa">Building a Recommender System Using Embeddings</a> was originally published in <a href="https://medium.com/drop-engineering">Drop Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>