<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vineet Goel on Medium]]></title>
        <description><![CDATA[Stories by Vineet Goel on Medium]]></description>
        <link>https://medium.com/@vineetgoel?source=rss-a319012c3b3d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SmqlfiUP5uW5VeNooOpGvQ.jpeg</url>
            <title>Stories by Vineet Goel on Medium</title>
            <link>https://medium.com/@vineetgoel?source=rss-a319012c3b3d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 16:43:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vineetgoel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Adding Faust to your Existing Architecture]]></title>
            <link>https://medium.com/robinhood-engineering/adding-faust-to-your-existing-architecture-39b0ebd1f7c9?source=rss-a319012c3b3d------2</link>
            <guid isPermaLink="false">https://medium.com/p/39b0ebd1f7c9</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[stream-processing]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Vineet Goel]]></dc:creator>
            <pubDate>Wed, 22 Aug 2018 20:16:37 GMT</pubDate>
            <atom:updated>2018-08-22T20:24:35.918Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Quick Links: </strong><a href="https://github.com/robinhood/faust"><strong>GitHub</strong></a><strong> | </strong><a href="http://faust.readthedocs.io/"><strong>Documentation</strong></a></p><p>A few weeks ago we <a href="https://robinhood.engineering/faust-stream-processing-for-python-a66d3a51212d">open sourced</a> <a href="https://github.com/robinhood/faust">Faust</a>, a Python stream processing library that we built at <a href="https://robinhood.com/">Robinhood</a> to make it extremely easy to build and deploy traditionally complex streaming architectures.</p><p>As Robinhood has grown and we have added more and more functionality to our product, our infrastructure has also evolved. We have added numerous internal services and technologies to help us solve different problems. This has resulted in a typical application often needing to interact with one or many different services.</p><p>Typical streaming frameworks such as <a href="https://spark.apache.org/streaming/">Spark</a> require external dependencies to be packaged with the app in specific ways, and submitted into the <a href="https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html">Yarn</a>/<a href="http://mesos.apache.org/">Mesos</a> cluster that is running the application. This is usually a detour from how Python applications typically manage dependencies — <a href="https://docs.python-guide.org/dev/virtualenvs/">virtualenv and pip</a>.</p><p>We built Faust as a library to allow for it to be used with any existing tools you may be using. Simply install Faust, and use it to develop Python applications as you typically would. We use <a href="https://medium.freecodecamp.org/a-guide-to-asynchronous-programming-in-python-with-asyncio-232e2afa44f6">Python Asyncio</a> to achieve high performance I/O.</p><p>In this blog post we will walk through some examples of how we use Faust to interact with various different services using off-the-shelf libraries.</p><h3>Faust + Redis</h3><p><a href="https://redis.io/">Redis</a> has established itself as an in-memory data store of choice owing to its <a href="https://redis.io/topics/data-types">data structures</a>, amazing query speeds and simplicity. We use Redis on Robinhood’s Data team across a variety of use cases. Following is an example, showing how we use Redis to cache messages on the Robinhood Feed.</p><p>We can install <a href="https://github.com/NoneGG/aredis">aredis</a> and Faust using pip:</p><pre>pip install aredis<br>pip install faust</pre><p>Upon installing the dependencies, let’s first define our Faust application, Kafka topic and models:</p><pre>import datetime<br>import faust</pre><pre>class Activity(faust.Record, isodates=True):<br>    user: str<br>    message: str<br>    timestamp: datetime.datetime</pre><pre>app = faust.App(&quot;redis_example&quot;, broker=&quot;kafka://localhost:9092&quot;)<br>activities_topic = app.topic(&quot;feed_activities&quot;, value_type=Activity)</pre><p>We can now create an <a href="http://faust.readthedocs.io/en/latest/userguide/agents.html">agent</a> which reads feed activities coming in through this topic, and adds the messages to the user’s Redis sorted set as follows:</p><pre>import aredis</pre><pre>@app.agent(activities_topic)<br>async def save_activities(activities):<br>    async for activity in activities:<br>        client = aredis.StrictRedis(host=&quot;localhost&quot;, port=6379)<br>        await client.zadd(activity.user, activity.timestamp, activity.message)</pre><p>As shown above, we use Redis as you would use it in any app. Faust doesn’t require any special drivers or modes for using Redis. All it needs is a Redis library that’s compatible with Python Asyncio.</p><h3>Faust + HTTP</h3><p>We often use streaming apps that need to talk to other services over HTTP. Below is an example of how we use the Python <a href="https://aiohttp.readthedocs.io/en/stable/index.html">aiohttp</a> library from a Faust streaming app for one of our use cases at Robinhood.</p><p>First, let us install the Python library we will use for HTTP requests:</p><pre>pip install aiohttp</pre><p>We skip the app and model definition which is similar to the previous, and straightaway look at how we would design our agent. We create an agent which processes orders and uses a third part HTTP API to send order confirmation emails to our customers:</p><pre>import aiohttp</pre><pre>async def send_confirmation(order):<br>    url = f&quot;https://emailer.robinhood.com/&quot;<br>    data = {<br>        &quot;user&quot;: order.user_id,<br>        &quot;subject&quot;: &quot;Order Confirmation&quot;,<br>        &quot;body&quot; f&quot;Order: {order.quantity} shares of {order.symbol}&quot;,<br>    }<br>    async with aiohttp.ClientSession() as session:<br>        await session.post(url, json=data)</pre><pre>@app.agent(orders_topic)<br>async def add_symbol(orders):<br>    async for order in orders:<br>        await send_confirmation(order)</pre><p>A lot of our internal services export REST APIs. The ability to easily integrate aiohttp with Faust apps allows us to break down a variety of our backend systems into simple and isolated streaming apps.</p><h3>Faust + InfluxDB</h3><p>Robinhood operates on large amounts of time series data such as tick by tick price data for each stock symbol. We use <a href="https://www.influxdata.com/">InfluxDB</a> to store some of these time series. Below is an example of how we query InfluxDB from a Faust application.</p><p>Again, as before, let us install the Python library we will use to query InfluxDB:</p><pre>pip install aioinflux</pre><p>We now create an agent which looks at the orders topic from above and looks at the time series in InfluxDB for the particular stock to get the price at which the order executed was the price in the market at the time. We do this to ensure that we are giving the best quality of executions to our customers.</p><pre>import aioinflux</pre><pre>@app.agent(orders_topic)<br>async def add_symbol(orders):<br>    async for order in orders:<br>        client = aioinflux.InfluxDBClient()<br>        query = f&quot;SELECT price FROM marketdata WHERE symbol = {order.symbol} AND timestamp &lt;= {order.timestamp} ORDER BY DESC LIMIT 1&quot;<br>        order.market_price = await client.query(query)<br>        await quality_topic.send(key=order.id, value=order)</pre><p>The ability to easily integrate streaming apps with InfluxDB lets us solve problems where we need to work with multiple time series.</p><h3>Faust + Elasticsearch</h3><p>The data team at Robinhood uses the <a href="https://www.elastic.co/elk-stack">ELK (Elasticsearch-Logstash-Kibana)</a> stack quite extensively. Last year, we published a <a href="https://robinhood.engineering/taming-elk-4e1349f077c3">post</a> about how we use the ELK stack along with Kafka. Below is an example of how you can query Elasticsearch from your Faust app.</p><p>We first install the <a href="https://github.com/elastic/elasticsearch-py-async">elasticsearcy-py-async</a> library that we will use in this example:</p><pre>pip install elasticsearch-async</pre><p>We now create an agent which looks at incoming search queries and adds the top search result from our search service running on top of Elasticsearch.</p><pre>import elasticsearch_async</pre><pre>async def get_top_result(query_string):<br>    client = elasticsearch_async.AsyncElasticsearch()<br>    query = {&quot;query&quot;: query_string}<br>    resp = await client.search(index=&quot;search_index&quot;, body=query)<br>    return resp[&quot;hits&quot;][&quot;hits&quot;]</pre><pre>@app.agent(search_queries_topic)<br>async def add_top_search_result(search_queries):<br>    async for query in search_queries:<br>        query.top_result = await get_top_result(query.query_string)<br>        await top_results_topic.send(key=query.top_result, value=query)</pre><p>Faust makes it easy for us to build architectures where we use Elasticsearch as our data store of choice.</p><h3>Concluding Thoughts</h3><p>The above examples show how we use Faust with external dependencies, across several use cases. In the coming weeks we will go deeper into some of the specific ways we use Faust.</p><p>Faust works well with most common Python libraries making it very easy to use with any existing applications. We’re very excited to see how developers will use Faust, and what use cases they will come up with.</p><p>Please feel free to leave a comment here or a note on <a href="https://github.com/robinhood/faust">Github</a> if you have any questions or thoughts. We’re also <a href="https://careers.robinhood.com/">hiring</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39b0ebd1f7c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/robinhood-engineering/adding-faust-to-your-existing-architecture-39b0ebd1f7c9">Adding Faust to your Existing Architecture</a> was originally published in <a href="https://medium.com/robinhood-engineering">Robinhood</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Robinhood uses Airflow]]></title>
            <link>https://medium.com/robinhood-engineering/why-robinhood-uses-airflow-aed13a9a90c8?source=rss-a319012c3b3d------2</link>
            <guid isPermaLink="false">https://medium.com/p/aed13a9a90c8</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[cron]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Vineet Goel]]></dc:creator>
            <pubDate>Tue, 09 May 2017 23:07:37 GMT</pubDate>
            <atom:updated>2017-05-09T23:07:37.426Z</atom:updated>
            <content:encoded><![CDATA[<p>Robinhood relies on batch processing jobs at set schedules for a large number of tasks. These jobs range from data analysis and metric aggregations, to brokerage operations such as dividend payouts. We started off with using cron to schedule these jobs but with their growing number and complexity, it became increasingly challenging for us to manage them using cron:</p><ul><li><strong>Managing dependencies </strong>between jobs was difficult. With cron we would use worst-case expected durations for upstream jobs to schedule downstream jobs. This was getting increasingly harder with scale as the complexity of these jobs and their dependency graphs increased.</li><li><strong>Failure handling</strong> and alerting had to be managed by the job. We would have to rely on the job, or the on-call engineer to handle retries and upstream failures in the case of dependent jobs.</li><li><strong>Retrospection </strong>was difficult. We would need to sift through logs or alerts to check how a job may have performed on a certain day in the past.</li></ul><p>We decided to move away from cron for our scheduling needs and replace it with something that would solve the above problems. We explored a few open source alternatives like <a href="https://github.com/pinterest/pinball">Pinball</a>, <a href="https://azkaban.github.io/">Azkaban</a> and <a href="https://github.com/spotify/luigi">Luigi</a>, before finally deciding on <a href="http://pythonhosted.org/airflow/index.html">Airflow</a>.</p><h4>Pinball</h4><p>Pinball, developed at Pinterest, has a lot of features of a distributed, horizontally scalable, workflow management and scheduling system. It solves a lot of the problems mentioned above but the documentation was sparse and the relative size of the community was small.</p><h4>Azkaban</h4><p>Azkaban, developed at LinkedIn, is probably the oldest among the alternatives we considered. It uses properties files to define workflows while most of the newer alternatives use code. This makes it harder to define complex workflows.</p><h4>Luigi</h4><p>Luigi, developed at Spotify, has an active community and probably came the closest to Airflow during our exploration. It uses Python for defining workflows and comes with a simple UI. However, Luigi doesn’t have a scheduler and users still have to rely on cron for scheduling jobs.</p><h3>Hello Airflow!</h3><p>Airflow, developed at Airbnb has a growing community and seemed to be the best suited for our purposes. It is a horizontally scalable, distributed workflow management system which allows us to specify complex workflows using Python code.</p><h4>Dependency Management</h4><p>Airflow uses <a href="https://airflow.incubator.apache.org/concepts.html#operators">Operators</a> as the fundamental unit of abstraction to define tasks, and uses a <a href="https://airflow.incubator.apache.org/concepts.html#dags">DAG</a> (Directed Acyclic Graph) to define workflows using a set of operators. Operators are extensible which makes customizing workflows easy. Operators are divided into 3 types:</p><ul><li><strong>Action</strong> operators that perform some action such as executing a Python function or submitting a Spark Job.</li><li><strong>Transfer</strong> operators that move data between systems such as from Hive to Mysql or from S3 to Hive.</li><li><strong>Sensors</strong> which trigger downstream tasks in the dependency graph when a certain criteria is met, for example checking for a certain file becoming available on S3 before using it downstream. Sensors are a powerful feature of Airflow allowing us to create complex workflows and easily manage their preconditions.</li></ul><p>Below is an example of how the different type of sensors can be used for a typical ETL (<a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">Extract Transform Load</a>) workflow. The example uses Sensor operators to wait until data is available and uses a Transfer operator to move the data to the required location. An Action operator is then used for the Transform stage followed by using the Transfer operator to load the results. Finally, we use Sensor operators to verify that the result was stored appropriately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CcxrRbffqn45YwGglCyexw.png" /><figcaption>An ETL workflow using different types of Airflow Operators</figcaption></figure><h4>Failure Handling and Monitoring</h4><p>Airflow allows us to configure retry policies into individual tasks and also allows us to set up alerting in the case of failures, retries, as well as tasks running <a href="https://airflow.incubator.apache.org/concepts.html#slas">longer than expected</a>. Airflow comes with an intuitive UI with some powerful tools for monitoring and managing jobs. It provides historical views of the jobs and tools to control the state of jobs — such as kill a running job or manually re-running a job. One of the unique features of Airflow is the ability to create charts using job data. This allows us to build custom visualizations to monitor the jobs closely and also acts as a great debugging tool while triaging issues with jobs and scheduling.</p><h4>Extensible</h4><p>Airflow Operators are defined using Python classes. This makes it very easy to define custom, reusable workflows by extending existing operators. We have built a large suite of custom operators in-house, a few notable examples of which are the OpsGenieOperator, DjangoCommandOperator and KafkaLagSensor.</p><h4>Smarter Cron</h4><p>Airflow DAGs are defined using Python code. This allows us to define more complex schedules beyond what cron offers. For example, some of our DAGs need to run only on market open days. With simple cron, we would schedule these to run on all weekdays and handle the market holiday case in the application.</p><p>We also use Airflow sensors to run jobs right after market close, while handling market half-days. The following example uses custom operators built for running workflows on complex schedules that dynamically update according to the market hours for a given day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*avVioxXl1jTrnC0rj0oEYA.png" /><figcaption>Workflow with a dynamic schedule depending on market hours for a given day</figcaption></figure><h4>Backfills</h4><p>We use Airflow for metrics aggregations and batch processing of data. With evolving needs, we sometimes need to go back and change how we aggregate certain metrics or add new metrics. This involves running backfills across arbitrary spans of time in the past. Airflow provides a CLI which allows us to run backfills across arbitrary spans of time with a single command, and also allows us to trigger backfills from the UI. We use Celery (built by our very own <a href="https://medium.com/u/ecdd5b4bb13">Ask Solem</a>) to distribute these tasks across worker boxes. The distribution capabilities of Celery make backfills quick and easy by allowing us to spin up more worker boxes while running backfills.</p><h4>Common Pitfalls and Weaknesses</h4><p>We are currently on Airflow 1.7.1.3 which works well in production but comes with its own set of <a href="https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls">weaknesses and pitfalls</a>.</p><ul><li><strong>Time zone issue</strong> — Airflow relies on the system time zone (instead of UTC) for scheduling. This requires the entire airflow setup to be run in the same time zone.</li><li>The <strong>Scheduler</strong> works separately for scheduled jobs and backfill jobs. This can result in weird outcomes such as backfills not respecting a DAG’s max_active_runs configuration.</li><li>Airflow was built primarily for data batch processing due to which the Airflow designers made a decision to always <strong>schedule jobs for the previous interval</strong>. Hence, a job scheduled to run daily at midnight will pass in the execution date “2016–12–31 00:00:00” to the job’s context when run on “2017–01–01 00:00:00”. This can get confusing especially in the case of jobs running at irregular intervals.</li><li><strong>Unexpected backfills</strong> — Airflow by default tries to backfill missed runs when resuming a paused DAG or adding a new DAG with a start_date in the past. While this behavior is expected, there is no way to get around this, and can result in issues if a job shouldn’t run out of schedule. Airflow 1.8 introduces the <a href="https://github.com/apache/incubator-airflow/blob/master/airflow/operators/latest_only_operator.py">LatestOnlyOperator</a> to get around this issue.</li></ul><h3>Conclusion</h3><p>Airflow has quickly grown to become an important component of our infrastructure at Robinhood. The ability to define DAGs with Python code and the extensible API makes Airflow a configurable and powerful tool. Hopefully this post is useful for anyone exploring scheduling and workflow management tools for their own needs. We are happy to answer any questions. If this kind of stuff is interesting to you, we are <a href="https://boards.greenhouse.io/robinhood#.WQqFh1PyvUI">hiring</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aed13a9a90c8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/robinhood-engineering/why-robinhood-uses-airflow-aed13a9a90c8">Why Robinhood uses Airflow</a> was originally published in <a href="https://medium.com/robinhood-engineering">Robinhood</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>