<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by CDL Data Intelligence Team on Medium]]></title>
        <description><![CDATA[Stories by CDL Data Intelligence Team on Medium]]></description>
        <link>https://medium.com/@cdl.data.intelligence?source=rss-6879b5c15250------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BgwHWU11-yfAPtVSjvgkzw.jpeg</url>
            <title>Stories by CDL Data Intelligence Team on Medium</title>
            <link>https://medium.com/@cdl.data.intelligence?source=rss-6879b5c15250------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:42:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cdl.data.intelligence/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Flexible Workflow Orchestration at CDL]]></title>
            <link>https://medium.com/compass-digital-labs/flexible-workflow-orchestration-at-cdl-83f3d7b4f22d?source=rss-6879b5c15250------2</link>
            <guid isPermaLink="false">https://medium.com/p/83f3d7b4f22d</guid>
            <dc:creator><![CDATA[CDL Data Intelligence Team]]></dc:creator>
            <pubDate>Thu, 09 Jul 2020 20:04:07 GMT</pubDate>
            <atom:updated>2020-07-09T20:37:33.746Z</atom:updated>
            <content:encoded><![CDATA[<p>Improving the Airflow development experience using containers.</p><figure><img alt="An image of Apache Airflow’s Logo" src="https://cdn-images-1.medium.com/max/1024/1*iPF_70zBWvdhV_G5Ft41Wg.png" /><figcaption>Apache Airflow’s Logo (<a href="https://en.wikipedia.org/wiki/Apache_Airflow">source</a>)</figcaption></figure><p>As a team, we’re always searching for insights; it only made sense that we’d want to have a deeper look inside the activity of our workflows.</p><p>Earlier this year, our Data Intelligence department here at <a href="https://compassdigital.io/">Compass Digital Labs</a> (CDL) decided to deploy <a href="https://airflow.apache.org/">Apache Airflow</a> to orchestrate and automate several workflows. Before using Airflow, our team used an in-house developed scheduler tool. We chose to migrate some of our workloads to Airflow so that our team could have improved the observability of our workflows. We use Airflow in a variety of use cases:</p><ul><li>Data ingestion into our Data Lake and our Data Warehouse (ETL) from a multitude of source systems</li><li>Automated report generation</li><li>Training and deploying machine learning models</li></ul><h3><strong>Initial Platform</strong></h3><p>Our initial deployment was very minimalist. We had deployed Airflow on a single virtual machine (VM), using the Local Executor. We used Amazon’s Relational Database Service (RDS) as our metadata database. All of the workflows were executed on a single VM. The architecture looked something like the following:</p><figure><img alt="A diagram of single node Airflow Deployment running on Amazon Web Services" src="https://cdn-images-1.medium.com/max/972/1*vEOJ_ensp8XKHSdJkpB9fQ.png" /><figcaption>Single node Airflow Deployment running on AWS</figcaption></figure><p>We had all of our Airflow components (web server, scheduler, and worker) running on the same VM. With the adoption of Airflow in our team steadily growing, we inevitably ran into scaling problems.</p><p>Problems often arose when trying to run many tasks concurrently. Airflow quickly ate up all the resources on the VM and sometimes would bring down the deployment due to not being able to allocate any memory.</p><p>Another issue we ran into using Airflow is that it requires the workflows to be written using Python. Members on our team had existing workflows that were written in other languages (mainly R) which were not that easy to port into Airflow because of dependency issues.</p><h3><strong>Finding the Best Options</strong></h3><p>Using a larger VM was not an optimal solution due to underutilizing the resources, as during times when not many <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">DAGs</a> are being scheduled to run, the provisioned resources go to waste.</p><p>We also did not use the CeleryExecutor or the KubernetesExecutor. First off, we chose not to use the CeleryExecutor due to the increased overhead of maintaining a Celery + Redis/RabbitMQ deployment. In addition, using this executor would not have solved our dependency management problem as each Celery / Airflow worker would need all the required dependencies to be installed.</p><p>As mentioned above, we also did not choose to migrate to the KubernetesExecutor. The main reason for not deploying onto this executor was again due to language and dependency constraints. If we were to build a container to deploy to ECR (Elastic Container Registry) to run our deployment on the KubernetesExecutor, each build would take 15+ minutes to install all of the R packages and dependencies.</p><h3><strong>New Platform</strong></h3><p>At a high level, our current platform continues to runs the Airflow webserver and scheduler on an EC2 VM. The newest components to our platform are a private container registry hosted on ECR, and our Kubernetes cluster, running on AWS Elastic Kubernetes Service (EKS). Our cluster has an autoscaling Node Group and it will serve as the compute resources to run Airflow Tasks. Now, team members can write DAGs that can leverage the KubernetesPodOperator to execute their own custom Docker images, stored in ECR.</p><figure><img alt="A diagram of high-level architecture of our platform using Kubernetes" src="https://cdn-images-1.medium.com/max/972/1*gj9TFOo8LjkQAH0SsB12DQ.png" /><figcaption>High-level architecture of our platform using Kubernetes</figcaption></figure><p>Kubernetes is a container orchestration platform, which was open-sourced by Google, that allows users to deploy, schedule, and run containers. We wanted to migrate some workflows to using KubernetesPodOperators for the following reasons:</p><ul><li>Improve development lifecycle and process</li><li>Decouple the compute resource from the scheduling resource</li><li>Improve dependency management</li></ul><h3><strong>Development Process</strong></h3><p>Each “project” or “DAG” will now have its own Git repository. This may seem unnecessary but has helped us improve the sustainability and maintainability of different code bases. With each project being maintained in different repositories, teams are enabled to write DAGs in any language they choose, enforce style guides, and most importantly, have a CI/CD pipeline that will deploy their DAG implementation to a private container registry.</p><h3><strong>Removing Implementation Details</strong></h3><p>Now that the containers are deployed, the only new files that are contributed to our Airflow deployment repository are Airflow DAG definition. This leaves the repo a bit cleaner as data scientists and engineers no longer need to write DAGs for Airflow, but rather just let Airflow schedule and execute the DAG, and all the implementation details are abstracted away from Airflow. Our team now is enabled to develop robust, scalable automation workflows in any language (R, Golang, Rust, etc.) and deploy it without worrying about Airflow dependencies.</p><h3><strong>Dependency Management</strong></h3><p>Without the need to maintain any dependencies for your workflow directly on your Airflow deployment, it alleviates problems with conflicting package versions as well as the need to have support for specific packages for one-off DAGs. All the dependencies are now isolated to the individual Docker container.</p><h3><strong>Trivial Example of using KubernetesPodOperator</strong></h3><p>To conclude, here’s a simple DAG that uses just a single KubernetesPodOperator.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/78d7ca50cd7fb31a054f69ae4d0dc866/href">https://medium.com/media/78d7ca50cd7fb31a054f69ae4d0dc866/href</a></iframe><p>In this example, we have an existing container hosted in Amazon’s container registry, ECR, and we have a Kubernetes config file somewhere on our Airflow deployment. We use Airflow variables to store the container’s URL and also the path to our Kubernetes config file. Once these variables are integrated with the Airflow metadata database, we can then pass them as environment variables or command-line arguments to the Docker image that will execute on the Kubernetes Cluster. Airflow will take some time to create a new pod on the EKS cluster, and will then run, and terminate the pod once completed.</p><h3><strong>Feedback</strong></h3><p>Feedback is certainly most welcomed! If you have any questions or comments, feel free to leave them and I will do my best to answer everything I can.</p><p><em>Written by: Brad Bonitatibus</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83f3d7b4f22d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/compass-digital-labs/flexible-workflow-orchestration-at-cdl-83f3d7b4f22d">Flexible Workflow Orchestration at CDL</a> was originally published in <a href="https://medium.com/compass-digital-labs">Compass Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Changing States of Our Plates]]></title>
            <link>https://medium.com/compass-digital-labs/the-changing-states-of-our-plates-99db368450b5?source=rss-6879b5c15250------2</link>
            <guid isPermaLink="false">https://medium.com/p/99db368450b5</guid>
            <category><![CDATA[customer]]></category>
            <category><![CDATA[trends]]></category>
            <category><![CDATA[retail]]></category>
            <category><![CDATA[plant-based]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[CDL Data Intelligence Team]]></dc:creator>
            <pubDate>Wed, 27 May 2020 21:38:22 GMT</pubDate>
            <atom:updated>2020-07-09T20:37:03.106Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A plant-based hamburger patty on a stone background." src="https://cdn-images-1.medium.com/max/1024/1*hkbv6touM9QKzCaWfuegiA.jpeg" /><figcaption>Within Compass-Served Cafes*, 90% of fast-growing alternative meat brands are plant-based, whereas soy-based options are declining in popularity.</figcaption></figure><p><em>The “What do you want to order for dinner?” argument among loved ones. The bountiful-menu-induced choice paralysis. The hunger-fueled shopping trip that results in a bizarre grocery cart cornucopia. All of these are signals that we have more food choices than ever before — so, what exactly are we choosing to eat, then?</em></p><p><strong>The Customer is Always Right</strong></p><p>Compass Group has a large footprint in the food &amp; hospitality industry that covers all <a href="https://www.compass-usa.com/">North</a> <a href="https://www.compass-canada.com/">American</a> regions, market verticals, and B2C endpoints. This provides us (the <a href="https://compassdigital.io/Home">CDL</a> Data Intelligence team) with a unique advantage when understanding customer behavior and motivations to discover new opportunities.</p><p><strong>The numbers can provide curated insight into regional preferences, which include but are certainly not limited to: the proliferation of brands, menu items, ingredients, and broader-scale movements.</strong></p><p>By highlighting impactful trends and insights, the goal is to <strong>improve the customer experience as well as drive innovation in our industry</strong>. For example, we can leverage the data to forecast the spread of trends and ingredients, as well as provide menu engineering recommendations to chefs. But ultimately, it’s about keeping our consumers satisfied and making sure we’re well ahead of the curve.</p><p><strong>Where Trends Come From</strong></p><p>In the past, we’ve noticed some interesting patterns that revealed intriguing (and now relatively well-recognized) consumer trends. Typically stemming from California’s major cities, healthy ingredients such as kale, soy, and quinoa, will see a spread into metropolitan areas — such as NYC — within a few months, and finally become fairly ubiquitous throughout North America within the year. In contrast, non-health oriented trends, such as new soda and chip flavors, follow a different trajectory with their start usually coming from the mid-west.</p><p>With all of this in mind, we’re currently tracking the trajectory of another rising star: <strong>plant-based proteins.</strong></p><figure><img alt="Five bowls, each containing a different type of bean." src="https://cdn-images-1.medium.com/max/1024/1*BsEHU7Ek64pxls4_SNIr5w.jpeg" /></figure><p><strong>Some Possible Explanations</strong></p><p>As demonstrated through our own data analysis*, we’ve noticed that people are still ordering a fairly large amount of meat. However, while alternative meat itself currently has the smallest slice in the category’s hypothetical pot pie, it’s showing the fastest rate of growth. Why is this happening now, when meat substitutes have been the <a href="https://www.latimes.com/food/dailydish/la-dd-20-years-tofurky-20141114-story.html">butt of jokes on TV for over 20 years</a>? One possibility is that, presently, popular brands such as Beyond Meat and Impossible Foods have a target beyond vegans and vegetarians: meat-eaters. People who enjoy eating meat don’t want to stop doing so but are becoming increasingly interested in eating fewer animal products overall.</p><p><strong>It’s less about changing the consumer, and more about supporting them where they are.</strong> If someone has a craving for wings or a burger but is open to substituting the meat with a protein that tastes and feels like the real thing, they can eat what they want with a side of several reasons to feel good about themselves. Within Compass-Served Cafes, diners with a preference for alternative meats tend to gravitate toward fresh produce over canned or frozen, order salads as entrees and pick sparkling water and iced coffee as their beverages of choice. This suggests a few things: the crowds may skew younger, they may be health-conscious, and they likely enjoy keeping up with trends.</p><p><strong>Not All Alternative Meats Are Made Equal</strong></p><p>All that being said, our data shows that* <strong>90% of fast-growing alternative meat brands are plant-based, whereas soy-based options are declining in popularity.</strong></p><figure><img alt="A group of soy-based products." src="https://cdn-images-1.medium.com/max/1024/1*jqdazoWrBYH5Mq3MAmWFQw.jpeg" /></figure><p>We also can’t say, in terms of people’s diets, whether plant-based proteins are a replacement or an add-on. However, we’ve noted some possible reasons for these changes in consumer behavior — plant-based products come from more familiar sources such as legumes, they’re nutrient-dense while being low in fat and calories, and they’re sustainable. According to the <a href="https://www.mintel.com/global-food-and-drink-trends">Mintel</a> 2019 report on Global Food and Drink Trends: “53% of US consumers agree that plant-based foods are better for the environment than animal-based options.”</p><blockquote><strong>It’s less about changing the consumer, and more about supporting them where they are.</strong></blockquote><p><strong>Where Do We Go from Here?</strong></p><p>Alongside Compass, we’ll be continuing to keep our proverbial thumbs on the pulse to ensure that our customers are always getting the best experience possible. In this case, we’re taking a quantitative approach to do so in a very qualitative field that is hospitality.</p><p>It’s important to have a strategy that is specific to customer needs, and the nuances that go along with that — be it an influx of <a href="https://www.healthline.com/nutrition/flexitarian-diet-guide">flexitarian</a> diners, influential on-campus vegans, or simply a growing number of folks that want to give the Earth a hand.</p><p><em>Footnote: Trends provided with Compass data as of August 2019</em></p><p><em>Written by: Jasmine Jia &amp; Sravya Atluri</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99db368450b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/compass-digital-labs/the-changing-states-of-our-plates-99db368450b5">The Changing States of Our Plates</a> was originally published in <a href="https://medium.com/compass-digital-labs">Compass Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>