<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ace Haidrey on Medium]]></title>
        <description><![CDATA[Stories by Ace Haidrey on Medium]]></description>
        <link>https://medium.com/@acehaidrey?source=rss-2b12a5e48400------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fN1wPL0rwFFMfA9PQtKO6w.jpeg</url>
            <title>Stories by Ace Haidrey on Medium</title>
            <link>https://medium.com/@acehaidrey?source=rss-2b12a5e48400------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 05:56:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@acehaidrey/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Over-Optimizing: The Peril of Chasing Perfection in Tech and Life]]></title>
            <link>https://medium.com/@acehaidrey/over-optimizing-the-peril-of-chasing-perfection-in-tech-and-life-254d83b4830c?source=rss-2b12a5e48400------2</link>
            <guid isPermaLink="false">https://medium.com/p/254d83b4830c</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[personal-growth]]></category>
            <category><![CDATA[family]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[personal-development]]></category>
            <dc:creator><![CDATA[Ace Haidrey]]></dc:creator>
            <pubDate>Fri, 22 Sep 2023 12:54:46 GMT</pubDate>
            <atom:updated>2023-09-22T13:04:43.672Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="https://unsplash.com/photos/gb0BZGae1Nk?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditShareLink" src="https://cdn-images-1.medium.com/max/1024/1*y2e2sw63V-RFN4OYEeIaOw.jpeg" /><figcaption>Diamonds Are Forever?</figcaption></figure><p>As a software engineer, I’m well-acquainted with the practice of optimization. It’s an essential aspect of our profession, ensuring that systems run efficiently, productively, and as intended. However, like many things in life, there exists a threshold where striving for perfection can cross into the realm of over-optimization. It’s a line that I’ve crossed more times than I care to admit, not only in my professional endeavors but also in my personal life.</p><p>In the software realm, over-optimization can manifest as spending days or weeks refining code to handle improbable edge cases, delaying product releases to achieve 1000% test coverage across unit, integration, and E2E tests, or architecting a solution for scalability even when the present demands are modest. This meticulous attention to detail, while commendable, often leads to diminishing returns.</p><p>But this propensity doesn’t end when I clock out. As someone with a type-A personality, my penchant for optimization pervades every corner of my life. Be it maximizing credit card points, ensuring optimal space utilization when stocking groceries, or even navigating the intricate world of diamond shopping (a task that took me to an exhaustive number of jewelers to evaluate every minute aspect of the 4 C’s), I often find myself overthinking, over-analyzing, and over-optimizing.</p><p>This relentless pursuit of perfection, though born out of a genuine desire for excellence, has its drawbacks. There’ve been moments where I’ve realized that by the time I’m ready to release a project or make a decision, the world has moved on. Alternatives have sprung up, the urgency has diminished, or the landscape has evolved in ways I hadn’t foreseen. It’s a race against time, and the clock is unyielding.</p><p>My wife often recalls our diamond shopping experience with a mix of amusement and exasperation. While I pored over every detail, she saw the bigger picture: the joy of the moment, the symbol of our commitment. This encapsulates the heart of the matter. In seeking optimization, it’s easy to lose sight of the end goal, the core purpose. Whether it’s delivering value to users or cherishing life’s precious moments, it’s essential to strike a balance.</p><p>In reflecting on this, I’ve come to understand that life, much like software development, is iterative. Not every release needs to be flawless. Not every decision requires exhaustive analysis. Sometimes, good enough on time is better than perfect too late.</p><p>To my fellow professionals who grapple with this, I offer this perspective: It’s okay to strive for excellence, but it’s equally okay to release, to decide, to move forward, even if things aren’t ‘perfect’. After all, in both tech and life, there’s always another iteration, another version, another day.</p><p>In sharing this, I hope to spark a conversation about the beauty of balance and the importance of understanding when optimization enhances our lives and when it detracts from it. I invite you to join me on this journey of reflection and growth, and expend advice on how you force yourself out of this mindset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=254d83b4830c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yelp: Step on Small Businesses While They’re Down]]></title>
            <link>https://medium.com/@acehaidrey/yelp-step-on-small-businesses-while-theyre-down-7327d285a0cc?source=rss-2b12a5e48400------2</link>
            <guid isPermaLink="false">https://medium.com/p/7327d285a0cc</guid>
            <category><![CDATA[small-business]]></category>
            <category><![CDATA[injustice]]></category>
            <category><![CDATA[small-business-owner]]></category>
            <category><![CDATA[yelp]]></category>
            <category><![CDATA[racism]]></category>
            <dc:creator><![CDATA[Ace Haidrey]]></dc:creator>
            <pubDate>Wed, 11 Nov 2020 16:01:05 GMT</pubDate>
            <atom:updated>2020-11-11T16:01:05.416Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Yelp: Step on Small Business While They’re Down</strong></h3><p>My name is Ace Haidrey. I’m a software engineer in the data org at Pinterest but I’m also a part time small business owner — my family and I run two mom and pop pizza shops in Lake Forest, California. It’s an interesting crossroads to have the perspective of both tech and being a small guy owning a business.</p><p>Generally speaking, tech apps try to take the high road when it comes to moral grounds, helping minorities, enabling “equality”, and improving the qualities of life for the common person. Though they have internal disparities that spill over to the public arena, they try to generally give the right answer, unequivocal in their voices that they’d correct any wrongdoings. You can expect this from all big tech such as Google, Facebook, Microsoft, Apple, and even the smaller players like Lyft, Pinterest, Airbnb, and more. The one company you will not find on this list though is <strong>Yelp</strong>.</p><p>Yelp has a reputation within the small businesses realm as the anti small business crusader. They allow for bullying, spite, ill language, inaccurate statements, and more obscurities thrown toward business owners. They try to force businesses to pay for their services and advertisements, or they will not help them; even when you pay they still do not help. These are not one off cases. There are thousands of grievances filed against Yelp for defamation and extortion. This can be verified online by searching the Federal Trade Commission records. This has always been known that Yelp is not a business who cares about accuracy or helping those who involuntarily get forced onto its platform; they more so want to fight their battle with Google and try to win as many active users on its platform as possible.</p><p>My specific case that I’d like to share with you is based on an experience back in July. In our entire experience as pizza shop operators, we have never dealt with someone this hostile and aggressive. This customer, Ryan, had a mistake on one of his pizzas, where we missed two toppings. We own up to our mistakes always and told him we would correct it. We gave three options to remedy the issue — we can remake the pizza and send it out right away, we can remake the pizza for another day, and just give them that amount of money in store credit, or we can do a partial refund on that pizza. Ryan was so upset about all options, and then went on to harass me and curse me — and that wasn’t even the worst part of it. It’s the fact that he felt he was in the right to act this way and say these things, and it’s because some of these apps reinforce that if you complain, you are right and get whatever you want. There are screenshots of his texts attached, and there is some missing context, but it paints the type of mannerisms this person exhibits (for example he claims I hung up on him but that is after he was berating me and not letting me talk, so I told him Ryan this isn’t helpful and hung up).</p><p>His outburst and temper tantrum were so overwhelming that we became really concerned. In this environment, it’s hard to know to what lengths people will go to make a point. We were forced to file a police report and we had real concerns he would try to hurt us or break our store. He went on to write this help review: <a href="https://www.yelp.com/biz/aroma-pizza-and-pasta-lake-forest-2?hrid=MxRnsSt--ZDZG8-GyqSrVg&amp;utm_campaign=bizsite_review_share_popup&amp;utm_medium=copy_link&amp;utm_source=(direct)">Ryan K’s Yelp Review</a>. In it you can identify where he says the following:</p><p><em>ACE, Said, Mohammad or whatever your real name is..</em></p><p>That is where it crossed the line for us. He makes many false accusations, uses profanity tirades, and all caps in his communications just through text. I wish I had the phone recording.</p><p>While we know generally, Yelp will allow such behavior, we thought even they would try to prevent stereotyping people, who look like me, my father or otherwise, and use it in a negative manner.</p><p>We were in fact wrong. Yelp first used its auto labeler to let me know they were upholding the review. Then secondly, we wrote in again and the customer service representative Michael (no last name given), said the following:</p><p><em>We allow reviewers to describe the appearance, attitude, and behavior of staff or other patrons involved in their consumer experience in their own words, but ask that they do so without resorting to stereotypes or terms that are demeaning towards a person or group based on their identity. As a result, we will rarely remove a review due to word choice alone. As long as a review appears to reflect the reviewer’s personal experiences and opinions, we’ll generally allow them to stand behind their review.</em></p><p><em>With that in mind, we’ve taken another look and we stand by our original decision to leave the review up, as we did not find it to be in violation of our guidelines.</em></p><p>The irony is that they then issue this statement recently, as one of the last major companies to make a response to BLM and the minority plight: <a href="https://blog.yelp.com/2020/10/new-consumer-alert-on-yelp-takes-firm-stance-against-racism">Yelp’s Firm Stance Against Racism.</a></p><p><em>At Yelp, we value diversity, inclusion and belonging, both internally and on our platform, which means we have a zero tolerance policy to racism.</em></p><p>Again another lie to just try to cover it’s bases with the general public. It’s remarkable how far Yelp will take its lies and claims, making weightless statements that hold no merit. They are phonies who are the exact resemblance of those who perpetuate stereotypes. It is very clear how this statement can be offensive, and rather than ask the user to remove that single line, they decide to “stand their ground on policies they create”. I reread this policy, and it so clearly just opens the door for hate speech with no grounds to protect the individuals being attacked. I even recall having a phone call with an account manager from Yelp, and when I mentioned this, they agreed and told me “The customer can get away with saying anything as long as they don’t use too much profanity.” The fact that their own employees say this and find it comical is baffling and morally wrong.</p><p>Yelp needs to take accountability and a lending ear to those it forces onto its platform. The lack of competition in this area is what enables them to ignore our frustrations and cries. I know writing this may not move the needle but I hope it can add awareness to what is happening. Especially in times like covid where more are turning to their phones to order online, and check online reviews, it’s critical that the business owners have a voice in all of this. They struggle to survive, work long and hard hours, go above and beyond to make each order as satisfactory as possible, but they’ve become silenced. Try to stand up, and you get shut down. Keep silent and the status quo, and you remain a punching bag to companies such as this.</p><p>This is imperfectly written but I hope the gist has been understood here.</p><p>Thank you for reading,</p><p>Your local restaurant owner,</p><p>Ace</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xFhLTQP5PyG3oomP" /><figcaption>First Page of Texts with Ryan</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2GmBOaGmiG8GDzGo" /><figcaption>Second Page of Texts with Ryan</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7327d285a0cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Airflow at Pandora]]></title>
            <link>https://medium.com/algorithm-and-blues/apache-airflow-at-pandora-1d7a844d68ee?source=rss-2b12a5e48400------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d7a844d68ee</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[pandora]]></category>
            <dc:creator><![CDATA[Ace Haidrey]]></dc:creator>
            <pubDate>Thu, 15 Mar 2018 21:55:53 GMT</pubDate>
            <atom:updated>2018-03-16T18:10:39.059Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8uKNeHDEqbmISInS44_OwQ.jpeg" /><figcaption>Moment | <a href="https://www.gettyimages.com/search/photographer?family=creative&amp;photographer=photography+by+Maico+Presente">Photography by Maico Presente</a> | Getty Images</figcaption></figure><p>Big data pipelines don’t run themselves. Beyond the obvious need to scale pipelines up to large data volumes, there’s a lot of work that goes into managing dependencies, handling retries, alerting, etc. This need only becomes more pressing as the organization grows and new teams seek to analyze existing data in different ways. As new teams add new pipelines, they generally want to build off the output of existing pipelines and they want to copy well-defined patterns around retries, alerts, etc so they can focus on the logic of their particular pipeline.</p><p><a href="https://www.pandora.com/">Pandora</a> faced this issue long before the current plethora of open source schedulers were available. Faced with a problem without any obvious off-the-shelf solution, we built our own solution around the Quartz Scheduler (for the core scheduling logic) and Jetty (to serve up a web interface). While this solution suited our needs at the time, it’s limitations became more and more pronounced as the company grew:</p><ul><li>There was a <strong>high barrier to entry</strong>. We have a wide range of pipeline creators including Analysts, Scientists, and Engineers. Compiling and deploying Java code can be tough.</li><li>For many folks, it was easier to use <strong>cron</strong> even though cron has no built-in mechanisms for managing dependencies, triggering alerts, etc.</li><li>Understanding the structure of jobs was tricky. The team had added a lot of useful features to the web UI over the years, but there was <strong>little to no ability to visualize</strong> the structure of tasks within jobs or dependencies across jobs.</li><li>Many of our jobs are long running jobs consisting of many different tasks. Occasionally, a flaky task would fail in the middle of a particularly long-running job. Faced with this sort of failure, folks had to choose between manually re-running the failed task and any downstream tasks or re-running the entire job. Not surprisingly, many chose the easier path of re-running the entire job even if it meant redoing a substantial amount of work.</li><li>The team that owned our in-house scheduling framework was focused on other problems that were more critical to the business. When it comes time to decide if folks should work on new pipelines to support Pandora Premium or improve our in-house scheduler, the decision to push off any scheduler-related work was almost a foregone conclusion. When bugs were found or new features were requested, there were no <strong>dedicated resources</strong> to maintain and develop our in-house solution.</li></ul><p>With these limitations becoming increasingly apparent, it was time to take a step back and see what other solutions existed. We wanted something that was widely used with lots of examples and documentation to help lower the learning curve and we wanted it to be open source so we could pickup bug fixes and new releases as needed.</p><h4>Why Airflow?</h4><p>After looking into Spotify’s <a href="https://github.com/spotify/luigi">Luigi</a>, LinkedIn’s <a href="https://github.com/azkaban/azkaban">Azkaban</a>, and a few other options, we ultimately moved forward with Airbnb’s <a href="https://github.com/apache/incubator-airflow">Airflow</a> for the following reasons:</p><ul><li>DAGs (Directed Acyclic Graph) are written in <strong>Python</strong> — Python is more familiar than Java to most analysts and scientists. It’s also easier to get started and iterate quickly when you’re not waiting for builds, etc.</li><li>It’s <strong>easy to update individual DAGs</strong> — With our old system, pushing out updates was disruptive to all jobs running on that particular instance. We either had to wait for a window when nothing was running or kill currently running jobs and manually restart them afterwards.</li><li>Airflow’s <strong>visualizations</strong> are helpful in so many different ways. Having a graphical overview of the DAG makes it easy to spot-check the structure as you build it and makes it easier for other folks on the team to get a sense of how things are organized (whether it’s because they need to change/update it or because they are responding to an alert in the middle of the night).</li><li>With nearly <strong>400 contributors</strong> to date there’s no doubt that Airflow is a popular open source project with a strong and growing community.</li></ul><h4>What is Airflow?</h4><p>As the Airflow docs put it, <em>“Apache Airflow is a way to programmatically author, schedule, and monitor data pipelines.”</em></p><p>There are extensive explanations of <a href="https://medium.com/airbnb-engineering/airflow-a-workflow-management-platform-46318b977fd8">what Airflow does and how it works</a> available online. For the unfamiliar, the key concepts are as follows:</p><ul><li><strong>DAG</strong>: a directed acyclic graph object that ties together all the tasks in a cohesive workflow and dictates the execution frequency (i.e. <strong>schedule</strong>).</li><li><strong>Task</strong>: a unit of work to be executed that should be both atomic and idempotent. In Airflow there are two types of tasks: <strong>Operators </strong>and <strong>Sensors</strong>.</li><li><strong>Operator</strong>: a specific type of work to be executed. Airflow comes with many types out of the box such as the <strong>BashOperator</strong> which executes a bash command, the <strong>HiveOperator</strong> which executes a Hive command, the <strong>SqoopOperator</strong>, etc. It’s easy to create new ones for specific types of tasks. Some operators, like the <strong>BashOperator</strong> and the <strong>PythonOperator</strong>, are particularly flexible while others are more constrained.</li><li><strong>Sensor</strong>: a <strong>blocking</strong> task that runs until a condition is met or until it times out. Sensor tasks are used as <strong>upstream dependencies</strong> of Operators. They prevent work from starting until certain preconditions are met. For example, if you have a Hive script that depends on certain table partitions existing, you may use a <strong>HivePartitionSensor</strong> to block until they are created.</li></ul><p>Besides the basic building blocks, Airflow includes a <strong>powerful UI</strong> and an even more<strong> versatile CLI</strong>. In addition, Airflow allows us to add customizations via its support for plugins.</p><h3>Our Infrastructure &amp; Deployment</h3><h4>Set Up</h4><p>When a team is interested in using Airflow, they make a request to the Infrastructure team to setup a new instance for them. The Infrastructure team uses Ansible to deploy instances in a repeatable/scalable way (we are up to 26 production instances). The Infrastructure team is responsible for the overall health of the system (e.g. monitoring and alerting) and each team is responsible for building and maintaining their DAG definitions.</p><h4><a href="https://www.ansible.com/">Ansible</a></h4><blockquote>Ansible is software that automates software provisioning, configuration management, and application deployment.</blockquote><p>As mentioned above, we use Ansible to automate setting up and updating all of our Airflow instances. Our Ansible playbook creates a supervisor instance, installs the Airflow dependencies, creates all the necessary directories (such as the DAGs, logs, and plugin directories), installs our plugin, registers this instance in Consul (amongst other things, we use Consul to define basic health checks and define a scrape target for Prometheus; more on this later), creates the necessary databases in Postgres, adds some custom connections and more.</p><p>One of the drawbacks of this approach is that we must ensure the machines we are deploying to via Ansible have certain utilities the playbook is expecting. We have ambitions to “containerize” things so that we can deploy to our internal cloud infrastructure using Nomad and Docker. However, this is still a work in progress.</p><h4><a href="http://supervisord.org/">Supervisor</a></h4><blockquote>Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.</blockquote><p>We use supervisor to control all of our airflow processes: the webserver, the scheduler, and the workers. Having the supervisor UI to check the process logs and perform actions on the processes such as stop, restart, etc. have proven to be very valuable and it makes maintaining easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fr-f4ffZG1heYkOUaNgp4Q.png" /><figcaption>The supervisor UI set up to monitor the airflow processes</figcaption></figure><h4><a href="https://www.consul.io/">Consul</a> &amp; <a href="https://github.com/fabiolb/fabio">Fabio</a></h4><blockquote>Consul is a tool for discovering and configuring services in your infrastructure. It provides several key features: Service Discovery, Health Checking, KV Store, and Multi Datacenter.</blockquote><blockquote>Fabio is a fast, modern, zero-conf load balancing HTTP(S) and TCP router for deploying applications managed by <a href="https://consul.io/">consul</a>.</blockquote><p>We have a lot of different teams with their own Airflow instance. It would be a pain to keep track of where each instance was running if it had to be done manually. Thankfully, Consul makes it easy for us to register the instances as we create them. We then have a central system to turn to when folks ask “where is instance XYZ running?” Additionally, we use certain tags in the service definition to identify these instances as Prometheus scrape targets (more on this later).</p><p>Beyond service registration, we also define basic health checks in Consul. We have a simple HTTP check which gives us a basic gauge of the health of the webserver. We also wanted to have a health check that gave us some visibility into the health of Airflow’s scheduler and workers. We settled on using a TTL check which is updated by a simple DAG we schedule on all of our instances by default.</p><p>By registering our Airflow instances in Consul, we were also able to leverage <a href="https://github.com/hashicorp/consul-template">consul-template</a> to quickly build a dashboard that shows where all of our instances are deployed (i.e. hostname and port) and the status of the health checks for that instance. Although all of this information is available by navigating around the Consul UI, it was convenient to have a simplified and consolidated dashboard where we could check on instances “at a glance.”</p><p>Lastly, we use Fabio to provide more memorable, user-friendly URLs. We didn’t want our users to have to access their instances using URLs like machineXYZ:9001. Fabio gave us an easy way to give teams unique, team-specific names for their instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RnAbm04MoSmhY42c-hkOtA.png" /><figcaption>A simple admin portal built on top of the consul data</figcaption></figure><h4><a href="https://github.com/prometheus/prometheus">Prometheus</a> &amp; <a href="http://grafana.org/">Grafana</a></h4><blockquote>Prometheus is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.</blockquote><blockquote>Grafana is an open source metric analytics &amp; visualization suite. It is most commonly used for visualizing time series data for infrastructure analytics.</blockquote><p>We use Prometheus to monitor our Airflow instances and configure alerts around certain key metrics. Our internal plugin adds a ‘/metrics’ endpoint to each Airflow instance. More specifically, Airflow enables the addition of new web views via Flask Blueprints. Within the context of the plugin it’s easy to calculate instance specific metrics such as log size, number of database connections and DagBag processing time and others. We use Prometheus’ AlertManager to configure alerts for some of these metrics and for general instance health (e.g. are either of the Consul health checks failing?).</p><p>Once Prometheus starts scraping metrics, it’s easy to build dashboards in Grafana to visualize those metrics.This is really helpful for eyeballing if something seems off and allows us to be proactive in catching discrepancies before users notice.</p><p>Aside from metrics we collect via Prometheus, Airflow does have some support for sending stats to StatsD. We initially wanted to use <a href="https://github.com/prometheus/statsd_exporter">statsd_exporter</a> to forward these stats along to Prometheus, but ultimately decided not to after learning that some of these stats were incorrect/inaccurate (see <a href="https://issues.apache.org/jira/browse/AIRFLOW-774">AIRFLOW-774</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oMHQQ_voe2Yv2WvSBOln8Q.png" /><figcaption>Grafana dashboard built from the Prometheus metrics</figcaption></figure><h4>Authentication</h4><p>We use Airflow’s support for LDAP to authenticate users and limit access to the UI to members of the relevant team. The Infrastructure team is also granted access so that they can quickly debug issues if/when they arise. This can be accomplished by using the following <em>“trick”</em> for the user filter value in the airflow.cfg<strong>:</strong></p><pre><strong>[ldap]</strong><br>user_filter = |(memberOf=cn=teamXYZ,ou=groups,dc=...,dc=...)(memberOf=cn=infrastructure,ou=groups,dc=...,dc=...)</pre><p>In the above expression, the “|” operates as a logical OR allowing members of teamXYZ and the infrastructure team to login to this particular instance.</p><h4>Forking Airflow</h4><p>After some debate, we decided to maintain our own internal fork of Airflow. Airflow still has its fair share of bugs, but with such a large and active community we find that fixes are often available and committed to master by the time we run into them. We didn’t want to subject ourselves to the instability of master, but we did want to have the flexibility to cherry pick certain fixes and back port them to more stable release branches. Right now our internal release is based off of the <strong>1.8.1</strong> release branch and we plan to upgrade to <strong>1.9.1</strong> when that release is made available.</p><p>For the times when we run into a bug that hasn’t yet been addressed by the community, we also commit fixes to our internal fork. In order to prevent our fork from drifting too far from the publicly available versions we almost always contribute these changes back. So far we have submitted and merged small UI fixes, additions to the HiveOperator, corrections to the SqoopOperator, success/failure callbacks on the DAG object level, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cyv9JeZA5VnlzvZz8KKv1w.png" /><figcaption>Contribution tab on the Apache-Airflow project: <a href="https://github.com/apache/incubator-airflow/graphs/contributors">https://github.com/apache/incubator-airflow/graphs/contributors</a></figcaption></figure><h4><strong>Customizing Airflow</strong></h4><p>We have used Airflow’s plugin mechanism to add a number of customizations to our Airflow instances including adding Pandora-specific Operators, Sensors, Views, Blueprints, Connections and Menu Links. For example, we’ve added a view that allows users to add new DAGs or sync existing DAGs from their BitBucket repositories and another view that allows folks to backfill directly from the UI instead of needing to use the CLI.</p><p>We have also added our own preloaded DAGs to these instances to do simple tasks. As mentioned previously, we have a “canary” DAG that exists solely to satisfy the TTL health check we’ve setup in Consul. If there are any issues with the instance that prevent this DAG from running, the Consul health check will start failing and the infrastructure team will receive an alert so that any issues can be addressed as soon as possible. Some of the other preloaded DAGs are simple examples showcasing some of the custom operators we’ve created. Other DAGs assist with basic maintenance tasks like clearing out logs older than a certain limit. It’s interesting to use Airflow to monitor and maintain itself.</p><pre><strong>from</strong> pandora_airflow_plugin.operators.postgres_sensors <strong>import</strong> PostgresTableSensor<br>airflow_import_error_psql_sensor = \ <br>    PostgresTableSensor(<br>        <strong>task_id</strong>=&#39;postgres_table_sensor_example&#39;,<br>        <strong>table</strong>=&#39;import_error&#39;,<br>        <strong>conn_id</strong>=&#39;postgres_default&#39;,<br>        <strong>dag</strong>=dag,<br>    )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tW-QQHw3EkP5V1FvW3jEIA.png" /><figcaption>Example Operators and Sensors part of the plugin</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2o1FCEIXVZcYBo8PMs-uxw.png" /><figcaption>Additional Menu Link and Category added as part of the plugin</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZcH9r7mEKfGF_MVKuxPsZA.png" /><figcaption>A custom view added as part of the plugin. It takes advantage of some of the internals of airflow where a user can migrate a table from one user space to the user space owning this airflow instance. It creates a dagrun of the hive_migration_dag on demand to handle the steps involved of moving the table.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gfyYKuK0dA07oFAe19-OBg.png" /><figcaption>Custom success/failure callback email set on DAG object</figcaption></figure><h3>Conclusion</h3><h4>Drawbacks</h4><p>As mentioned before, we still run into little bugs a bit more often than we’d like. The good news is that scouring Airflow <a href="https://issues.apache.org/jira/projects/AIRFLOW/issues">JIRA</a> tickets often reveals a fix or a discussion that sheds light on what is happening.</p><p>Some of the issues we continue to face include:</p><ul><li>The UI has some issues and idiosyncrasies that can throw off newcomers. One example is when new DAGs are added or updated, the webserver processes <strong>don’t sync</strong> immediately, causing users to get confused. Another common pain point for us in the task view. Clicking on a specific execution date of a task doesn’t actually take you to the run for that execution date. Instead you are taken to the DAG page which defaults to showing the latest run. Luckily the fix is easy and just needs to be implemented. There are a handful of other bugs we have come across like stated above that just <strong>don’t match expectations</strong>.</li><li>Manual, Backfill, and Scheduled runs are all <strong>treated differently</strong>. Users often ask for help getting a manual run to actually run (it must have schedule turned on) and getting backfills to respect <strong>max_active_runs</strong>. There are a handful of other <strong>inconsistencies</strong> that repeatedly cause confusion.</li><li>It should be easier to <strong>kill</strong> DAGs in bulk. If you accidentally set off multiple DagRuns (and if you use Airflow, you’ve probably done this at some point or another) there’s no easy way to stop all the executions.</li><li>The scheduler will <strong>stall out</strong> sometimes on our more heavily used instances. Tweaking the <strong>num_runs</strong> and <strong>run-duration</strong> options helped, but it’s still a bit of mystery to us why this is happening. This particular issue is usually preceded by a sharp increase in DAG file processing times.</li><li>Airflow is a fast-growing open source project, which is awesome, but with so many contributors it can sometimes be difficult to look through the source code and <strong>understand what the intention</strong> was. Code styles and conventions are different, and sometimes “fixes” get through that could benefit from further discussion (for example <a href="https://issues.apache.org/jira/browse/AIRFLOW-1349">AIRFLOW-1349</a>).</li><li>Airflow currently does not have the capability to have a task that creates tasks. There are cases, for example, where you may want to set off a task per log file but you don’t know before the DAG starts just how many log files will exist. Currently you’d have to bundle the processing of all those log files into a single task.</li></ul><h4>Closing Remarks</h4><p>Just like Airflow has grown popular in the tech community, it is quickly gaining traction within Pandora. Even with some shortcomings, it’s a very powerful tool that continues to get better over time. We hope you find this post on how we deployed Airflow here at Pandora helpful. We would be happy to respond to any questions or comments. Thank you!</p><p><em>Special thanks to </em><a href="https://medium.com/@LetsGoCard"><em>Matthieu Martin</em></a><em> for managing and leading this project and to </em><a href="https://medium.com/@ama_72227"><em>Alan Ma</em></a><em> for his contribution to this project.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d7a844d68ee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/algorithm-and-blues/apache-airflow-at-pandora-1d7a844d68ee">Apache Airflow at Pandora</a> was originally published in <a href="https://medium.com/algorithm-and-blues">Algorithm and Blues</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>