<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Endeavor: Data Blog on Medium]]></title>
        <description><![CDATA[Stories by Endeavor: Data Blog on Medium]]></description>
        <link>https://medium.com/@endeavordata?source=rss-618e193438f6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6YYgAEk955JbN-DTe_VItw.jpeg</url>
            <title>Stories by Endeavor: Data Blog on Medium</title>
            <link>https://medium.com/@endeavordata?source=rss-618e193438f6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:30:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@endeavordata/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using AWS DMS For Migrating Data Into Snowflake]]></title>
            <link>https://medium.com/@endeavordata/using-aws-dms-for-migrating-data-into-snowflake-590d0d7dbbc0?source=rss-618e193438f6------2</link>
            <guid isPermaLink="false">https://medium.com/p/590d0d7dbbc0</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[dms]]></category>
            <dc:creator><![CDATA[Endeavor: Data Blog]]></dc:creator>
            <pubDate>Tue, 26 Mar 2024 14:21:25 GMT</pubDate>
            <atom:updated>2024-03-28T22:58:08.250Z</atom:updated>
            <content:encoded><![CDATA[<p>By Joshua Gitter</p><h3><strong>Migrating Transactional Data</strong></h3><p>Transactional data is often at the heart of meaningful analysis. At Endeavor, transactional data helps us to predict the likelihood a fan will watch a pay-per-view UFC fight card or attend a match at the Miami Open. It helps us understand what car someone may be interested in seeing at a Barrett-Jackson car auction, or what event might be the hot ticket item at this year’s Olympic Summer Games in Paris. But before our data scientists are able to do impactful analysis, data needs to be made available in our data warehouse. The process of ingestion is the bedrock of a powerful data warehouse, and that’s where our data engineering team comes in.</p><p>One of the core responsibilities of data engineering at Endeavor is to ingest data from transactional systems into our data warehouse for analysis. This ingestion can come from a plethora of places including Amazon AWS, FTP servers, or SQL servers, hosted both on-prem and on the cloud.</p><p>To help better serve our fans, Endeavor continually integrates with new partners and vendors. Some of these partners have data in transactional databases that we need to migrate data over from. Database migrations can be technically challenging, expensive, and time consuming. The difficulty, cost, and time sink all go up when migrating data at scale and in real time. Amazon Database Migration Services (DMS) is a great option for this type of real time migration and this article will discuss how we are able to migrate data to Snowflake using DMS.</p><h3>Why We Choose DMS</h3><p>There are several database migration solutions out there such as AWS DMS, Fivetran, and Google Cloud Database Migration Service. Any of them could be a potential fit for a use case but for those already in the AWS ecosystem, DMS is an attractive choice. DMS can also be cheaper when compared to no-code solutions such as FiveTran, or Airbyte. No-code solutions can be excellent migration choices but quickly become expensive with increasing monthly active rows. DMS offers options to run replication services using either on-demand or serverless compute resources, let’s compare the two.</p><p><strong>AWS On-Demand</strong></p><p><strong>Pros:</strong></p><ul><li>Good choice for predictable migration workloads and for those who want granular control over resource allocation.</li><li>More documentation available as on-demand launched in 2016 and serverless in 2023.</li><li>More databases are supported for target and source endpoints. A full list can be found <a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html">here</a> and <a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Sources.html">here</a>.</li></ul><p><strong>Cons:</strong></p><ul><li>Generally more expensive — Users need to provision a replication instance. This means choosing the capacity (DCUs) of the instance and paying for it hourly.</li><li>More complex to setup — Users need to setup a migration task and create a source and target endpoint for the task to use.</li></ul><p><strong>AWS Serverless</strong></p><p><strong>Pros:</strong></p><ul><li>Cheaper — Users don’t need to provision replication instances as the capacity is automatically provisioned and managed. You only pay for what you use.</li><li>Simple setup — Migration tasks and endpoints are created as part of the serverless instance and don’t need to be created separately.</li></ul><p><strong>Cons:</strong></p><ul><li>Less databases are supported, a full list can be found <a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Serverless.Components.html#CHAP_Serverless.SupportedVersions">here</a>.</li><li>Documentation isn’t as readily available.</li></ul><p>Whichever you choose, DMS allows us to migrate data in near real time. Real time data has become critical for running a modern data-driven operation. It allows us to put information about transactions and inventory directly in front of analysts, enabling them to make timely decisions and get to value faster. This also means that both analytical and operational workloads can be supported using Snowflake.</p><h3>The Solution</h3><p>While DMS generally works as an excellent off-the-shelf solution, there are a limited number of supported destinations. Our data warehouse of choice is Snowflake, which is not one of the supported target databases for DMS. The good news is that DMS can write data to S3 buckets which we can leverage to our advantage.</p><p>Configuring DMS to migrate data to an S3 bucket is trivial as it already has built-in capability to do this. Once the data is inside of the bucket, Snowflake has its own set of objects which can aid us in continuing the migration.</p><p>A Snowflake stage is a temporary holding area for files being loaded into Snowflake from outside sources such as S3 or other storage locations. In order for stages to work we also need a storage integration. A storage integration is a Snowflake object that allows users to store an IAM policy to grant the stage access to files in storage locations.</p><p>Once data hits the stage, dynamic tables and tasks are the two main objects that aid in migration as they allow for the scheduling of queries. We can use these queries to move data from the staging area to our final destination.</p><p>Snowpipe and streams also offer viable options for migrating data into Snowflake, and we will discuss the differences in greater depth later in this article.</p><p>The solution we came up with needed to handle both full loads and change data capture (CDC). Full loads are a snapshot of a table at a certain time. CDC is a way of capturing changes made to a table over time. It allows for streaming inserts, updates, and deletes in real time as records are changed in the source database.</p><p>These two work well in tandem because the full load provides a starting point for the data migration and then CDC takes over for any changes made to that data once the full load is complete. Full loads on their own may be useful for datasets that are finalized and will not change. CDC on its own may be beneficial if a dataset is already migrated and ongoing changes need to be captured. Thankfully, Snowflake offers a range of tools that enable us to easily load both full datasets and capture data being written in real time to an S3 bucket.</p><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s1yylc5xAajdcPfWNwd7vQ.png" /></figure><p>We have found four possible architectures that work for this migration. The first difference between the architectures is whether to use a Snowflake task or Snowpipe with an Amazon Simple Notification Service (SNS) Topic for loading data into the staging table. The second is whether to use dynamic tables or streams and tasks to move data to its final destination. The path the data takes from the source database to the Snowflake stage is the same irrespective of choice as is the need for a staging table. All four architectures are viable and the choice to use which will depend on your use case. We will explain the differences more in depth below.</p><p><strong>Task vs Snowpipe with SNS Topic</strong></p><p><strong>Task<br></strong>A Snowflake task is a way to schedule SQL queries to run on a given schedule. Imagine we had the following query:</p><pre>CREATE TASK example_task<br>Warehouse = my_warehouse<br>Schedule = &#39;1 Minute&#39;<br>AS<br>COPY INTO my_table<br>FROM @my_stage<br>FILE_FORMAT = my_file_format<br>PATTERN = my_file_pattern;<br><br>ALTER TASK example_task RESUME;</pre><p>This SQL query will create a task that will continuously execute the COPY INTO statement every 1 minute. Tasks are created in a SUSPENDED state so we need to make sure to RESUME the task after creation. Tasks are very powerful and can help us perform complex data transformations at scale.</p><p>In our architecture, we use tasks to run automated COPY INTO statements every 1 minute. These queries copy files from the stage and load the data into a staging table. The difference between the stage and staging table is that the stage is where the files from our S3 bucket are loaded while the staging table is where we actually unload that data from those files. As new files are added to the S3 bucket, they are automatically picked up by the stage.</p><p>Pros:</p><ul><li>All COPY INTO parameters are supported</li><li>SELECT statements using a WHERE clause are supported. This is useful if you want to select certain rows from the stage instead of doing a COPY INTO.</li></ul><p>Cons:</p><ul><li>The task will run every minute whether or not a new file was actually dropped off. This is a waste of compute resources as tasks run on virtual warehouses and consume credits which will incur a cost.</li></ul><p><strong>Snowpipe with SNS Topic<br></strong>Snowpipe is another way to automate COPY INTO statements with the added benefit of being able to run only when new files are added to an S3 bucket. If data is being written as a file object to S3, then Snowpipe can automatically be notified of new files with the use of an Amazon SNS Topic.</p><p>An SNS topic acts as a communication channel that can broadcast messages about new files in an S3 bucket to its subscribers. Snowpipe can subscribe to this topic and will start the copy into job whenever it is notified about a new file in the S3 bucket. The following code shows how to create a Snowpipe that automates a copy into statement from an SNS topic:</p><pre>CREATE PIPE my_pipe <br>AUTO_INGEST = TRUE <br>AWS_SNS_TOPIC = &#39;&lt;sns_topic_arn&gt;&#39;<br>AS<br>  COPY INTO my_table<br>  FROM @my_stage<br>FILE_FORMAT = my_file_format;</pre><p>An SNS topic still needs to be configured on the AWS side with this approach.</p><p>Pros:</p><ul><li>Snowpipe will only run when there is a new file to process, so compute is not wasted.</li></ul><p>Cons:</p><ul><li>Snowpipe doesn’t allow the use of certain parameters in its copy into statements including: ON_ERROR, PURGE, FORCE, and SIZE_LIMIT.</li><li>Only simple SELECT statements are supported. Filtering using a WHERE clause is not supported.</li></ul><p><strong>Dynamic Tables vs Streams and Tasks</strong></p><p>Next, lets discuss the method of materializing the migrated data. When using CDC, the DMS migration files include ongoing changes made to a target table. Along with these changes will be a column called ‘Op’ that denotes what happened to a row. ‘D’ indicates that a row was deleted, ‘I’ indicates that a row was inserted, and ‘U’ indicates that a row was updated which tells us which transformation to apply to a given row.</p><p>Handling this transformation and materializing the data in a final table can be done using dynamic tables or streams with tasks.</p><p><strong>Dynamic Tables<br></strong>A dynamic table is a table that materializes from the results of a query. By default, data in the table is refreshed incrementally but it can be configured to refresh with full loads. The refresh works by using change tracking applied on base table(s) to track which rows were modified since the last refresh. Akin to tasks, this will run at a given cadence automatically. Here is an example of a dynamic table creation query:</p><pre>CREATE DYNAMIC TABLE my_table<br>TARGET_LAG = &#39;1 minute&#39;<br>WAREHOUSE = my_warehouse<br>AS<br>SELECT      t.* exclude (&quot;Op&quot;)<br>FROM        my_staging_table t<br>qualify &quot;datetime_column&quot; = max(&quot;datetime_column&quot;) over (partition by &quot;primary_key&quot;) and &quot;Op&quot; != &#39;D&#39;;</pre><p>This query will materialize the results of the SELECT query as a dynamic table. The data in the dynamic table will be no more than 1 minute older than the base table. 1 minute is the absolute minimum lag possible for a dynamic table or a task. Also akin to tasks, dynamic tables are very helpful for data transformation as the table creation and data transformation are done in the same step.</p><p>During our dynamic table creation we write the SELECT portion of the query to get the newest row for every primary key. In plain English, we are trying to find the most up to date row for every unique customer/transaction in the database. We also only select from the staging table where the ‘Op’ is ‘U’ or ‘I’ so that deletes are not included in the dynamic table. Because we are doing this, all tables migrated with this solution require a primary key and a datetime column.</p><p>Pros:</p><ul><li>Snowflake recommends dynamic tables as the preferred way to build out data pipelines</li><li>Transformation of data is done in the same step that loads it into the final table.</li></ul><p>Cons:</p><ul><li>Change tracking needs to be enabled on the base table(s) that the dynamic table selects from.</li></ul><p><strong>Streams and Tasks<br></strong>We have already discussed tasks and their effectiveness but they can’t replace dynamic tables on their own. Streams are Snowflake objects that track changes made to underlying objects. The following query will create a stream on table my_table:</p><p>CREATE STREAM table_stream ON TABLE my_table;</p><p>Next, a task will need to be created. When creating the task we can have it check every minute to see if the stream has any new data:</p><pre>CREATE TASK example_task<br>    WAREHOUSE = my_warehouse<br>    SCHEDULE = &#39;1 MINUTE&#39;<br>    WHEN SYSTEM$STREAM_HAS_DATA(&#39;table_stream&#39;)<br>    AS<br>MERGE INTO final_table f<br>USING ( <br>SELECT      t.*<br>FROM        table_stream t<br>QUALIFY &quot;datetime_column&quot; = max(&quot;datetime_column&quot;) over (partition by &quot;primary_key&quot;)<br>) s<br>ON f.&quot;primary_key&quot; = s.&quot;primary_key&quot;<br>when matched<br>    and s.&quot;Op&quot; = &#39;D&#39;<br>    then delete                   <br>when matched<br>    and s.&quot;Op&quot;= &#39;U&#39;      <br>    then update <br>    set f.&quot;primary_key&quot; = s.&quot;primary_key&quot;,<br>        ...<br>when not matched <br>    and s.&quot;Op&quot; =&#39;I&#39;<br>    then insert <br>    (&quot;primary_key&quot;,...)<br>    values<br>    (s.&quot;primary_key&quot;,...);<br>    <br>ALTER TASK example_task RESUME;</pre><p>Whenever the stream has data we merge from the staging table into the final table. Similar to the dynamic table query, we get the newest row for each primary key and use that rows ‘Op’ to perform the data transformation.</p><p>Pros:</p><ul><li>Can be used without enabling change tracking on base table(s)</li></ul><p>Cons:</p><ul><li>The merge query here is more complex then the query used for the dynamic table.</li></ul><h3>Closing Remarks</h3><p>Migrating data easily, at scale, in near real time, and without significant cost is truly a substantial challenge. Amazon is able to provide a practical solution for this with AWS DMS.</p><p>Though Snowflake is not a DMS supported database, built-in Snowflake objects such as dynamic tables and tasks help us bridge the gap. Completing migrations such as these help Endeavor succeed by aiding in making real time data available for applications such as operational reporting, machine learning, and business analytics.</p><p>We believe this post can work as a guide for those looking to migrate data from various databases into Snowflake and to do so in real time.</p><p>To learn more — reach out to the author <a href="https://www.linkedin.com/in/joshua-gitter-955723139/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=590d0d7dbbc0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Open-Source Streamlit Using AWS]]></title>
            <link>https://medium.com/@endeavordata/deploying-open-source-streamlit-using-aws-868530005833?source=rss-618e193438f6------2</link>
            <guid isPermaLink="false">https://medium.com/p/868530005833</guid>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Endeavor: Data Blog]]></dc:creator>
            <pubDate>Wed, 20 Dec 2023 09:01:41 GMT</pubDate>
            <atom:updated>2023-12-20T09:01:41.530Z</atom:updated>
            <content:encoded><![CDATA[<p>By Adam Ivansky</p><p>The data team at Endeavor empowers our business users and external clients through self-service reporting and visualization platforms. We enable our customers to make timely and well informed decisions through carefully designed KPIs and illustrative data visualizations. Some examples of these include product and marketing analytics, <a href="https://medium.com/@endeavordata/building-a-second-generation-machine-learning-platform-using-snowpark-555463da04f7">machine learning operations</a>, marketing channel performance, sales insights, and customer feedback on our products.</p><p>In today’s tech landscape, there are a number of business intelligence tools that promise analysts the ability to quickly create dashboards on top of prepared datasets. Moreover, feature-rich business intelligence offerings are now baked into software products such as marketing and ad-tech platforms and have accustomed data consumers to expect interactivity and quick response times even when visualizing terabyte-scale datasets.</p><p>Thanks to the power of <a href="https://streamlit.io/">Streamlit</a> — an open-source data visualization framework — our data scientists are able to create beautiful and highly-customized interactive web applications for visualizing and exploring data. Better yet, the flexibility of Streamlit allows us to create virtually any rich, purpose-built features into our data applications as long as they are express-able as Python code — an important differentiator with most stock business intelligence tools on the market today.</p><p>In this article, we will dive deeper into the architecture for deploying Streamlit on the Amazon Web Services (AWS) cloud and development patterns that our data scientists have found useful.</p><h3>What is <a href="https://docs.streamlit.io/">Streamlit</a> ?</h3><p>Streamlit is an open-source framework for building data applications and highly interactive charts using Python. These applications are exposed as web applications accessible via web browsers such as Chrome and Safari. Developers can run Streamlit in a local environment, access it using their localhost URL, or expose Streamlit as a service to multiple users on a wider network.</p><p>Streamlit removes the complexity behind front-end UI development as well as client-server interaction, and presents a simple API in the form of a Python package. Streamlit also comes with a number of features for handling of latency and data caching.</p><p>The example below shows a “Hello World” website created with the popular Python server framework <a href="https://flask.palletsprojects.com/en/2.3.x/">Flask</a>:</p><pre>from flask import Flask<br>app = Flask(__name__)<br><br>@app.route(&#39;/&#39;)<br>def hello_world():<br>    return &quot;&quot;&quot;<br>    &lt;!DOCTYPE html&gt;<br>    &lt;html&gt;<br>        &lt;head&gt;<br>            &lt;title&gt;Hello world example&lt;/title&gt;<br>        &lt;/head&gt;<br>        &lt;body&gt;<br>            &lt;p&gt;Hello world&lt;/p&gt;<br>        &lt;/body&gt;<br>    &lt;/html&gt;<br>    &quot;&quot;&quot;</pre><p>The example below shows the same “Hello World” website created in Streamlit:</p><pre>import streamlit as st<br><br>st.text(&quot;Hello world&quot;)</pre><p>Notice how the Streamlit implementation abstracts out client-server communication, HTML/javascript front-end, and HTTP request handling, allowing the developer to focus on what is actually visible on the screen.</p><p>While this post describes our deployment of Streamlit open-source, we should note that Streamlit is now also available as a <a href="https://www.snowflake.com/en/data-cloud/overview/streamlit-in-snowflake/">native integration in Snowflake</a>. Although feature parity between open-source and Streamlit-in-Snowflake is not yet identical, the latter could be an excellent option for teams looking to quickly deploy Streamlit on top of their data warehouse.</p><p>So why open-source?</p><p>A customized, in-house deployment of open-source Streamlit on AWS allows us fine control over different technical aspects of the Streamlit application, many of which are required by enterprise organizations like ours. These include user management, adherence to network security guidelines, tracking of usage analytics and tighter integration with existing AWS-hosted data and infrastructure.</p><p>We’ll dive into some of these custom implementations in the following sections.</p><h3>Overarching Principles</h3><p>In this section, we will outline some of the criteria we used for evaluating Streamlit and the guiding principles that informed the final architecture in our deployment of Streamlit open-source.</p><ol><li><strong>Technology Fit: </strong>Our team consists mainly of data scientists and engineers with expertise in data analytics and Python development. Since Python is our preferred development language, a tool allowing us to create beautiful user interfaces and visualizations without the need to use Javascript and other front-end centric languages is ideal.</li><li><strong>Multi-dashboard Applications: </strong>Our stakeholders will sometimes have the need for multiple dashboards. Our Streamlit design must then support this requirement without needing to spin up new cloud resources for each new dashboard separately.</li><li><strong>Scalable and Highly Available: </strong>The service hosting our dashboards should be scalable under various computational workloads and highly available. For this reason we are looking for a framework that can be easily deployed into an easily-scalable cloud orchestration service rather than a static compute resource. Since we expect our team to contribute new code to this application on a near-daily basis, seamless deployment of new features without application downtime is crucial.</li><li><strong>Handling Large Datasets: </strong>Data processing requirements on the dashboard server can at times be up to 5GB per dashboard. The selected solution must be able to process this amount of data without running out of memory. Additionally, use cases such as development of interactive applications that apply machine learning models in real-time must allow for computations across multi-GB datasets.</li><li><strong>Development Best Practices: </strong>Development of every new feature should begin with local testing, progress to a pre-production environment, and eventually be deployed into a production environment. We are committed to representing the artifacts we produce as code subject to versioning, peer-review and CI/CD processes. Different environments need to be representable as different git branches, and some branches should be deployable to live environments for testing by multiple users.</li><li><strong>Security/User Management: </strong>Our application should have a strong security framework, and user management relatively simple. Exposing dashboards that access modern warehouses such as Snowflake requires granular levels of access control. We should also integrate this application with an identity and access management service.</li></ol><h3>Architecture</h3><p>The diagram below represents our architectural design. Here, we will dive into the technical aspects associated with the deployment of a Streamlit web server into our existing cloud infrastructure on Amazon Web Services (AWS).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*-l-KeJXcmfATnV3Y_PXlFg.png" /></figure><h3>High Availability &amp; Scalable Compute</h3><p>AWS ECS (Elastic Container Services) orchestrates containers running the Streamlit web server based on images stored in AWS ECR (Elastic Container Registry). Streamlit web server containers are represented as tasks within a single service.</p><p>ECS supports auto-scaling policies based on metrics such as memory usage, CPU usage, or other custom metrics. Using auto-scaling, new tasks are provisioned if a selected metric reaches a threshold value. If a metric drops below a specific value, resources are de-provisioned.</p><p>Using this setup, we are able to support server-side computations on data in the multi-gigabyte range that is often demanded when having multiple concurrent users interacting with our Streamlit applications. We are also able to easily accommodate unforeseen spikes in usage.</p><h3>Updating Streamlit Applications</h3><p>Streamlit code is housed in a Github repository which contains a Dockerfile. Every new commit into a “live” branch of the repository where we want code deployed into our cloud environment (i.e. main or staging ) triggers a build of a new image from that base Dockerfile. This image is subsequently uploaded into AWS ECR and deployed onto our ECS cluster.</p><h3>Load Balancing &amp; User Management</h3><p>Exposing one or multiple ECS containers running Streamlit under a domain or subdomain requires an application load balancer. As Streamlit does not natively support HTTPS encryption, additional resources are required to encrypt traffic.</p><p>Our solution to this challenge involves <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html">SSL termination (SSL offloading)</a> and <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html">authentication</a> on the application load balancer level. This allows for easy integration with identity provider services such as Azure Active Directory and Auth0.</p><p>The load balancer is configured to authenticate incoming traffic and adds an additional JWT header “X-Amzn-Oidc-Data” to client requests. Access to this header is required to extract information such as current user email and group membership.</p><p>Streamlit does not natively support access to request headers. Therefore, a workaround such as the code snippet below is required to access the header from within the Streamlit session.</p><pre>import gc<br><br>for obj in gc.get_objects():<br>    if type(obj) is BrowserWebSocketHandler:<br>        socket_handler = obj<br>        break<br><br>headers = socket_handler.request.headers<br>encoded_jwt = headers[&quot;X-Amzn-Oidc-Data&quot;]</pre><h3>Development Patterns</h3><p>To ensure high availability of our Streamlit application, we require test, staging and production environments following the <a href="https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow">git flow branching model</a>. Each of these environments exists as a separate git branch.<br>Standardized PEP8 style and code formatting checks are implemented to ensure that no unformatted code is introduced into staging or production environments.</p><p>Once these format checks are complete, Streamlit code is built into a Docker image and uploaded into the corresponding AWS environment. ECS then automatically loads the latest version of our Streamlit application’s image. After a few minutes, ECS starts serving the latest version of our code.</p><p>Our GitHub repository consists of the following branches:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TKd94s8vY5_mGD7A2wCDvw.png" /></figure><p>Test environments can be deployed locally running in a virtual environment or container. These environments are intended for rapid feature development and debugging.</p><p>The staging environment is a near-clone of production and exposed only to developers. This environment is intended for testing of new features on our cloud infrastructure. Staging allows us to run integration tests to make sure dashboards have the appropriate access levels, pass performance tests, and pass user-acceptance testing.</p><p>The production environment is exposed to end users and has access to all production data sources.</p><p>Here is a graphical representation of the development process:</p><pre>main ============================ o ================== merge back ======== ... <br>                                  ^                      |<br>                                  |                      |<br>                                  |                      v<br>staging ================ o === Release ================= o =============== ...<br>         |               ^                            <br>         |               |                            <br>         v               |                            <br>         feature/* ======PR</pre><h3>Organizing Streamlit Code</h3><p>There are multiple ways to support multiple Streamlit dashboards running on a single server. Our solution involves defining a sidebar with a dropdown menu, allowing users to pick the dashboard they want to view (as described in this article). For this, we use the following file structure:</p><pre>.<br>├──streamlit-app<br>│   ├──common_utils             # Utility code common to multiple dashboards<br>│   |   ...<br>│   ├──dashboard_1              # First dashboard folder<br>│   │   ├──dashboard_1.py  <br>│   │   ├──data_utils.py        # Contains methods that deal with data<br>│   │   ├──ui_utils.py          # Contains methods associated with graphics<br>│   │   ├──utils.py             # Contains other utility methods and classes<br>│   │   ├──Readme.md            # Optional readme for the given dashboard<br>│   │   ...<br>│   ├──dashboard_2              # Second dashboard folder<br>│   │   ├──dashboard_2.pys<br>│   │   ├──data_utils.py        # Contains methods that deal with data<br>│   │   ├──ui_utils.py          # Contains methods associated with graphics<br>│   │   ├──utils.py             # Contains other utility methods and classes<br>│   ├──app.py                   # Main app file<br>│   ├──Dockerfile               # Dockerfile to containerize the application <br>│   ├──build.sh                 # script to build the Docker image and push application into ECR</pre><p>Here, the main application window with the sidebar is defined in the file app.py and individual dashboards that a user can switch between are defined in subdirectories.</p><p>The file app.py imports dashboards from all subdirectories and implement the logic for switching between dashboards. This solution uses the <a href="https://docs.streamlit.io/library/api-reference/widgets/st.selectbox">st.selectbox</a> widget located in the <a href="https://docs.streamlit.io/library/api-reference/layout/st.sidebar">st.sidebar</a> container.</p><pre>import dashboard_1.dashboard_1 as dashboard_1<br>import dashboard_2.dashboard_2 as dashboard_2<br>import streamlit as st<br><br>def main_page():<br>    &quot;&quot;&quot;<br>    Displays the main page that is shown right after the user lands on the<br>    dashboard website<br>    &quot;&quot;&quot;<br>    st.markdown(&quot;# Endeavor Streamlit dashboard&quot;)<br><br>    st.markdown(<br>        &quot;Welcome to the streamlit dashboard developed by CDP Data Science and Data&quot;<br>    )<br><br><br>def populate_sidebar(page_names_to_funcs):<br>    &quot;&quot;&quot;<br>    Populate the sidebar on the left. Display a logout button.<br>    &quot;&quot;&quot;<br><br>    selected_page = st.sidebar.selectbox(<br>        &quot;Select a dashboard&quot;, page_names_to_funcs.keys()<br>    )<br><br>    st.sidebar.markdown(f&quot;# {selected_page}&quot;)<br><br>    return selected_page<br><br><br>def main():<br>    &quot;&quot;&quot;<br>    Displays the welcome page of the dashboard website<br>    &quot;&quot;&quot;<br><br>    page_names_to_funcs = {<br>        &quot;Index Page&quot;: main_page,<br>        &quot;Dashboard 1&quot;: dashboard1.main,<br>        &quot;Dashboard 1&quot;: dashboard2.main<br>    }<br><br>    selected_page = populate_sidebar(page_names_to_funcs)<br><br>    page_names_to_funcs[selected_page]()<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Individual dashboard files then implement the main code for a dashboard that are typically subdivided into three sections.</p><pre>import streamlit as st<br>from common_utils.snowflake_utils import make_snowflake_conn<br>import .data_utils as data_utils<br>import .ui_utils as ui_utils<br>import package1 <br>import package2<br><br># Wrap all your code into main method<br>def main():<br>    &quot;&quot;&quot;<br>    Docstring with a brief description<br>    &quot;&quot;&quot;<br>    # Add titles, description and name of developer in this format<br>    st.markdown(f&quot;# Test dashboard &quot;)<br>    st.caption(&#39;Developed by John Doe&#39;)<br>    st.text(<br>        &quot;This dashboard is a test dashboard. Just an example.&quot;<br>    )<br>    <br>    # section containing queries and data work is wrapped in a spinner<br>    with st.spinner(&#39;Downloading some data...&#39;):<br>        # section dedicated to data work<br>        ...<br>        data_utils.method1(arg1, arg2)<br>        ...<br>        df = pd.Dataframe(...)<br>        ...<br><br>    # section showing results <br>    st.text(&quot;here is some description&quot;)<br>    ...<br>    ui_utils.method1(arg1, arg2)<br>    ...<br>    st.line_chart(<br>        data=pdf,<br>        x=&quot;AGE_GROUP&quot;,<br>        y=&quot;MY_MEAN&quot;,<br>        width=0,<br>        height=0,<br>        use_container_width=True,<br>    )</pre><p>The initial section of each dashboard includes the title of the dashboard, its author and its description.</p><p>Next, we have a data processing section wrapped in a <a href="https://docs.streamlit.io/library/api-reference/status/st.spinner">st.spinner</a> with statement. This provides a nice user experience by displaying an indicator that results are being computed, with a clear description of what is happening.</p><p>The third section includes code that displays a graphical representation of the data.</p><h3>Conclusion</h3><p>Our open-source deployment of Streamlit has allowed data scientists and engineers on our team to quickly and easily build interactive web applications for visualizing and exploring data. By utilizing an architecture capable of high availability and scalability, developers can deploy powerful applications into Streamlit capable of handling large volumes of data with no downtime to end-users.</p><p>In the future, we hope to do more testing with Streamlit in Snowflake to see what parts of our infrastructure we can simplify by possibly transitioning to a Snowflake-native solution. Moreover, we look forward to testing newer features available in open-source such as <a href="https://docs.streamlit.io/library/get-started/multipage-apps/create-a-multipage-app">multipage applications</a>, and building out support for dedicated UAT (user-acceptance testing) environments to streamline our deployment process.</p><p>We hope that this post can serve as a useful quick-start for teams looking to setup Streamlit on their cloud infrastructure with enterprise level security features as well as standard software development practices. To learn more — reach out to the writer <a href="https://www.linkedin.com/in/adamivansky/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=868530005833" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Second Generation Machine Learning Platform Using Snowpark]]></title>
            <link>https://medium.com/@endeavordata/building-a-second-generation-machine-learning-platform-using-snowpark-555463da04f7?source=rss-618e193438f6------2</link>
            <guid isPermaLink="false">https://medium.com/p/555463da04f7</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[snowpark]]></category>
            <dc:creator><![CDATA[Endeavor: Data Blog]]></dc:creator>
            <pubDate>Mon, 24 Jul 2023 14:12:58 GMT</pubDate>
            <atom:updated>2023-07-24T19:57:08.456Z</atom:updated>
            <content:encoded><![CDATA[<p>By Ilya Galperin</p><p><strong>Machine Learning at Endeavor</strong></p><p>Behind the best events, you’ll often find a group of folks laser-focused on making a remarkable, lasting impression on attendees. When a fan streams a UFC card, catches the world’s top bull riders at a Professional Bull Riders tour, or bids on a classic car at Barrett-Jackson, their experience is powered by a likewise passionate team at Endeavor.</p><p>In the data science and engineering group, we champion this ethos by helping build highly personalized experiences and applications for fans across a wide range of Endeavor businesses through the power of data. Machine learning (ML) models play a critical role in our approach. They allow us to make predictions based on patterns within datasets, leading to valuable insights about our customers and highly informed decisions on the best ways to serve them.</p><p>Data scientists at Endeavor frequently build new ML models and tune existing ones in service of this goal. Running these models requires significant computational power and technical expertise. To meet these challenges and power our machine learning platform, we have adopted Snowpark, the set of libraries and runtimes that enable users to deploy models directly in Snowflake with ease and highly scalable processing capacity.</p><p><strong>Guiding Principles</strong></p><p>Before getting into the specifics of Snowpark, it’s helpful to take a look at the <a href="https://medium.com/@endeavordata/machine-learning-platform-at-endeavor-93ba88b66986">homegrown ML platform</a> we have been using thus far, including the product objectives it aimed to fulfill, and any gaps we might want to address.</p><p>Prior to adopting Snowpark, machine learning models were deployed as purpose-built containers orchestrated with Kubernetes. Many models had their own sets of dependencies and a good amount of custom code for loading, handling, and eventually saving data back into Snowflake. Getting a model running on our infrastructure required continuous collaboration between a data engineer and data scientist, with the former responsible for deployment and the latter for maintaining core model logic.</p><p>While the flexibility inherent to this architecture was a big plus, we found that taking a model to production or iterating on an existing model took days, and sometimes weeks, longer than we would have liked.</p><p>Our business objectives demand that we work quickly and on a relatively short turn-around time to deploy models to applications. Therefore, we focused on the following guiding principles when designing our second-generation machine-learning platform:</p><ol><li><strong>Self-Serve: </strong>The design must enable a data scientist to move a machine learning model from inception all the way through deployment entirely by themselves. A successful end state here would remove the need for multiple engineers to be involved in the ML lifecycle. There should be zero dependency on a machine learning or DevOps engineer to get a model into production.</li><li><strong>Ease of Use:</strong> Given that data scientists would be responsible for model deployment, ease of use is critical. However, ease of use shouldn’t come at the expense of flexibility; we must still be able to run a large variety of models using a broad range of ML libraries and write custom Python code where needed.</li><li><strong>Repeatability:</strong> There must be a way to periodically retrain models in a simple manner. The lineage of models must be well understood and recreating an older version of a model possible at all times. Likewise, model training methods and routines should be stored in a git repo and be reviewable by others.</li><li><strong>Computationally Scalable and Cost Effective</strong>: As different models will have varying compute requirements, our host infrastructure must be capable of scaling up and down to accommodate a range of workloads.</li></ol><p><strong>Snowpark: What and Why?</strong></p><p>Snowpark is the set of libraries and runtimes in Snowflake that securely deploy and process non-SQL code, including Python, Java, and Scala. This code can be run server-side inside the Snowflake infrastructure using a virtual warehouse. Snowpark also offers a set of client-side libraries including the Snowpark DataFrame API, which is useful for querying and manipulating data directly in Snowflake using familiar DataFrame-style programming.</p><p>With this framework, Snowflake enables users to work with relatively large datasets without needing to worry too much about the underlying implementation for data distribution or parallel processing. The benefit of this approach becomes readily apparent when working with the sizeable datasets that are used to train ML models. Since Snowflake operates the infrastructure for us, much of the complexity in maintaining and scaling a machine-learning platform is abstracted away from our data scientists.</p><p>To scale compute, for example, a user can toggle between different virtual warehouse sizes. And because the compute resource and data live in the same infrastructure, the latency involved in moving data is decreased significantly compared to having to fetch it from Snowflake and process it in an external cluster. At the sizes of data necessary for machine learning, this ends up saving substantial runtime.</p><p>A less tangible but significant advantage of Snowpark from our perspective was the existing familiarity of our users with the ecosystem. Our data scientists are well-versed in Snowflake and its abstractions (i.e. stored procedures, user-defined functions), making this framework close to off-the-shelf ready. Data scientists are able to train and deploy models directly within Snowpark, without having to switch between different tools or programming languages.</p><p>It is important to note that Snowpark currently supports only certain packages available in the <a href="https://repo.anaconda.com/pkgs/snowflake/">Snowpark conda channel</a> (you can find supported back-versions via select * from information_schema.packages where package_name = &#39;name&#39;). Thankfully, the list of packages is extensive and includes a wide range of machine-learning libraries, with new ones being made available over time. Along with the <a href="https://www.snowflake.com/blog/snowpark-container-services-deploy-genai-full-stack-apps/">recent announcement of Snowpark Container Services</a>, this will no longer be a hard limit.</p><p><strong>General Architecture</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8FnguSAz2NGH8bfGQ3rMDg.png" /></figure><p>The diagram above shows the different stages of our machine learning model deployment lifecycle.</p><p>The first step is exploratory data analysis (EDA), feature engineering, and initial model development. Here, the <a href="https://docs.snowflake.com/en/developer-guide/snowpark/index">Snowpark API</a> allows our data scientists to easily interact with data directly in Snowflake, using virtual warehouses for data transformation and feature engineering (see related post about our feature store <a href="https://medium.com/@endeavordata/streamlining-machine-learning-development-with-a-feature-store-680ee6d45c64">here</a>).</p><p>The output of this step is a Python-stored procedure capable of preparing data, training, and re-training a machine learning model on demand. At the same time, we create a feature catalog. This artifact is useful to tell us what input data types our model expects and as a reference for when we are ready to generate a SQL query that uses our model for inference.</p><p>Finally, we register a user-defined function (UDF) representing the model. This UDF loads our model from an internal Snowflake stage and allows it to be run against datasets living in our warehouse.</p><p>With self-service being the main goal in our design, it was crucial to ensure that each step is easy to perform both locally and via an automated deployment process. We accomplished this by creating a script around each discrete step and orchestrating them in a CI/CD pipeline operating in production and pre-production environments, i.e.:</p><p>register_training_sproc.py -&gt; create_feature_catalog.py -&gt; register_inference_udf.py</p><p>Now, a data scientist can deploy their model and navigate it through the above lifecycle by making commits to a code repository.</p><p>Running the training stored procedure and inference UDF is as simple as executing a SQL query. Models can be trained, tested, and used for inference directly in the Snowflake console for rapid iteration, or orchestrated by an external tool for periodic re-training and batch inference.</p><p>At each phase in the pipeline, we log metadata about artifacts such as model versions and other information about the training dataset. This makes the process repeatable and model lineage well documented and readily available.</p><p>In the following sections, we’ll take a deeper look into some of these components to help illustrate how they fit into our overall design.</p><p><strong>Specifics: Model Training</strong></p><p>First, we create a script that is registered as a stored procedure that can later be called to train our machine-learning model. This stored procedure:<br>a) Handles and prepares the training dataset.<br>b) Fits the model based on the transformed training data.<br>c) Serializes and saves a trained model into an internal Snowflake stage.</p><pre>from joblib import dump<br>from snowflake.snowpark.session import Session<br>from sklearn.tree import DecisionTreeClassifier<br><br>def save_model(session: object, model: object, model_name: str) -&gt; None:<br>    &quot;&quot;&quot;<br>    Saves a serialized ML model to an internal Snowflake stage.<br><br>    Args:<br>        - session: snowpark session object<br>        - model: fitted machine learning model<br>        - model_name: name of the model<br>    &quot;&quot;&quot;<br>    file_path = &quot;/tmp/&quot; + model_name + &quot;.pkl&quot;<br>    dump(model, file_path)<br>    session.file.put(file_path, &quot;@MLP/models&quot;, auto_compress=False, overwrite=True)<br><br>def main(session: Session, training_table: str,) -&gt; list:<br>    &quot;&quot;&quot;<br>    The main model training procedure.<br>    <br>    Args:<br>        - session: snowflake session object<br>        - training_table: snowflake table to be used for the training task<br><br>    Returns:<br>        - list: here you can return feature importance or some other result of<br>            this training procedure to the caller.<br>    &quot;&quot;&quot;<br>    training_data = session.table(training_table).to_pandas()<br>    <br>    # create_transformer is a custom function used to prepare the training data<br>    transformer = create_transformer(training_data) <br>    <br>    X = transformer.transform(training_data)<br>    y = training_data[&quot;target&quot;]<br><br>    model = DecisionTreeClassifier()<br>    model.fit(X, y)<br><br>    result = save_model(session, model, &quot;MY_MODEL&quot;)<br><br>    return [result, MODEL_NAME] </pre><p>After the training script is prepared, we register it as a stored procedure using the Snowpark API.</p><pre># create_session() is a custom function used to connect to Snowpark<br>session = create_session(**snowflake_credentials)<br><br># model_requirements.txt contains all third party packages used by the training stored procedure<br>session.add_requirements(&quot;model_requirements.txt&quot;) <br>session.add_packages(&quot;snowflake-snowpark-python&quot;)<br><br>session.sproc.register_from_file(<br>    file_path=&quot;training_sproc.py&quot;,<br>    func_name=&quot;main&quot;,<br>    name=&quot;SPROC_TRAIN_MY_MODEL&quot;,<br>    is_permanent=True,<br>    replace=True,<br>    stage_location=&quot;@MLP&quot;,<br>)</pre><p>We can now easily call our procedure on demand or on a schedule to train and re-train our model using a SQL query like CALL SPROC_TRAIN_MY_MODEL(TRAINING_TABLE);.</p><p><strong>Specifics: Feature Catalog</strong></p><p>Our feature catalog serves as a reference point for the model’s features, their <a href="https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/latest/types">Snowpark data types</a> and ordinal position.</p><p>Here, we pass in the model’s name, the name of the table used to train the data, and any columns we need to exclude from the training set such as the label.</p><pre>def create_feature_catalog(<br>    session: object, model_name: str, table_name: object, exclude_columns: list = None<br>) -&gt; None:<br>    &quot;&quot;&quot;<br>    Creates the {model}_feature_catalog table which will be used downstream by the<br>    model&#39;s scoring UDF.<br><br>    The result is a table containing a list of feature names, their Snowpark data types,<br>    and their ordinal position used when training the model.<br><br>    Args:<br>        - session: snowpark session object<br>        - model_name: name of the model<br>        - table_name: Snowflake table containing a training dataset<br>        - exclude_columns: list of columns to exclude from the feature catalog, useful<br>            for y-axis, user identifiers, etc.<br><br>    &quot;&quot;&quot;<br>    df = session.table(table_name).drop(exclude_columns)<br><br>    data = []<br>    for e, row in enumerate(df.schema.fields):<br>        data.append((row.name, str(row.datatype), e))<br><br>    df_output = session.create_dataframe(data, schema=[&quot;NAME&quot;, &quot;DATA_TYPE&quot;, &quot;POSITION&quot;])<br><br>    df_output.write.mode(&quot;overwrite&quot;).save_as_table(&quot;MY_MODEL_FEATURE_CATALOG&quot;)</pre><p>The result is a feature catalog table used in the next steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*UMhElS7UR8T1qVBlMcapTA.png" /></figure><p><strong>Specifics: Inference User-defined Function</strong></p><p>To make use of our trained model, we write a UDF that loads our model from an internal stage, reads input as a Pandas dataframe, makes an inference and returns the result.</p><pre>def inference_udf(df):<br>    import sys<br>    from joblib import load<br><br>    import_dir = sys._xoptions[&quot;snowflake_import_directory&quot;]<br>    model_file = &quot;MY_MODEL.pkl&quot;<br>    model = load(import_dir + model_file)<br><br>    df.columns = feature_names<br>    <br>    return model.predict(df)</pre><p>Note the variable feature_names. This variable is used to assign column names to the input dataframe. Since the UDF object only processes raw data and does not assign column labels, this step will be important if your model expects named features.</p><p>Referencing the feature catalog in the previous step, feature_names can be set like so:</p><pre>catalog_names = session.sql(&quot;SELECT NAME FROM MY_MODEL_FEATURE_CATALOG ORDER BY POSITION&quot;)<br>feature_names = [row[&quot;NAME&quot;] for row in catalog_names.collect()]</pre><p>We can now register our user-defined function like so:</p><pre>from snowflake.snowpark.types import (  <br>  # import any Snowpark data types you will use here<br>    BooleanType,<br>    FloatType,<br>    PandasDataFrameType,<br>    PandasSeriesType,<br>    ...<br>)<br><br>session.add_requirements(&quot;model_requirements.txt&quot;)<br>session.udf.register(<br>    func=udf_main,<br>    return_type=PandasSeriesType(return_type),<br>    input_types=[PandasDataFrameType(input_types)],<br>    name=f&quot;UDF_MY_MODEL&quot;,<br>    stage_location=&quot;@MLP&quot;,<br>    max_batch_size=MAX_BATCH_SIZE,<br>    replace=True,<br>    is_permanent=True,<br>    imports=[&quot;@MLP/models/MY_MODEL.pkl&quot;],<br>    session=session,<br>)</pre><p>There are two important variables here to flag in the registration script: input_types and return_type.</p><p>return_type should be set to the Snowpark data type that the model is expected to return. For example, if we expect a boolean result, we would set return_type to BooleanType. If we expect a numeric value, we could set this to FloatType.</p><p>Our input_types variable will be constructed based on the data types stored in the feature catalog and tells the UDF what structure to expect from the input dataframe.</p><pre>def get_input_types(feature_catalog: object) -&gt; list:<br>    data_types = [row[&quot;DATA_TYPE&quot;] for row in feature_catalog.collect()]<br>    input_types = []<br>    for data_type in data_types:<br>        type_object = type_converter(data_type)<br>        input_types.append(type_object)<br>    return input_types<br><br>def type_converter(data_type: str) -&gt; object:<br>    if data_type == &quot;BooleanType()&quot;:<br>        return ArrayType()<br>    elif data_type == &quot;FloatType()&quot;:<br>        return FloatType()</pre><p>After registration, the UDF is ready for use and can be executed via a SQL statement:</p><pre>SELECT<br>  key,<br>  UDF_MY_MODEL(<br>    feature1,<br>    feature2,<br>    ...<br>  ) as inference<br>FROM my_inference_table;</pre><p>You can easily generate this select statement by referencing the feature catalog:</p><pre>catalog_names = session.sql(&quot;SELECT NAME FROM MY_MODEL_FEATURE_CATALOG ORDER BY POSITION&quot;)<br>feature_names = &quot;, &quot;.join([row[&quot;NAME&quot;] for row in catalog_names.collect()])<br>query = &quot;SELECT UDF_MY_MODEL({columns}) AS RESULT FROM &lt;TOSCORE_TABLE&gt;;&quot;</pre><p><strong>Conclusion</strong></p><p>In its current state, our Snowpark-powered machine learning platform has enabled data scientists to deploy ML models in a streamlined and automated fashion. The cost and time-saving benefits here are twofold: models can go to production days and weeks faster than before, and engineers are freed up from having to deploy models into our infrastructure and troubleshoot code that they did not write.</p><p>The time to run batch inference jobs has also been sped up significantly due to the fact that you can bring modeling and inference code to the data. Most importantly, data scientists have full ownership over the model deployment process and work with tooling and abstractions native to Snowflake that they are well accustomed to working with.</p><p>Recently, Snowflake <a href="https://www.snowflake.com/blog/snowpark-ml-api-python-develops-more/">announced</a> expanded ML offerings including a dedicated <a href="https://docs.snowflake.com/en/developer-guide/snowpark-ml/index">Machine Learning API</a>, a containerization service, and dev-ops enhancements including the ability to test locally and native git integration. As these features enter public preview and release, we feel confident that continuing investment into our Snowpark ML platform will allow our team to continue providing best-in-class machine-learning models powering unforgettable experiences.</p><p>To learn more- reach out to the writer <a href="https://www.linkedin.com/in/ilya-galperin/">here.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=555463da04f7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streamlining Machine Learning Development with a Feature Store]]></title>
            <link>https://medium.com/@endeavordata/streamlining-machine-learning-development-with-a-feature-store-680ee6d45c64?source=rss-618e193438f6------2</link>
            <guid isPermaLink="false">https://medium.com/p/680ee6d45c64</guid>
            <category><![CDATA[ufc]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-store]]></category>
            <dc:creator><![CDATA[Endeavor: Data Blog]]></dc:creator>
            <pubDate>Mon, 12 Dec 2022 20:31:03 GMT</pubDate>
            <atom:updated>2022-12-12T20:31:03.191Z</atom:updated>
            <content:encoded><![CDATA[<p>By John Thomas</p><p>In <a href="https://www.ufc.com/">UFC</a>, fighters prepare for their time in the Octagon with intense training regimens. Fighters and their teams will spend months analyzing their opponent’s style for any exploitable weaknesses, fine-tuning their body and their technique to offer the best possible match.</p><p>Data scientists at UFC’s parent organization, <a href="https://www.endeavorco.com/">Endeavor</a>, approach machine learning predictions with no less determination. While fighters throw down on camera, the data related to fans watching the fight also gets analyzed critically — the competition to attract and retain viewers is fierce. Customers need an engaging experience that keeps them hooked, and Endeavor has become a rising leader in leveraging machine learning and digital expertise to meet the challenge. Below we dive deeper into Endeavor’s work in maximizing viewership insights with their implementation of a feature store in streamlining machine learning development at UFC.</p><p>Many data scientists reading this know that the bulk of the work going into building machine learning models comes from feature creation. Features (i.e., predictors or attributes) like aggregated characteristics (e.g., days since last purchase or customer country of residence) are a necessary part of predictive machine learning models. For a model to make accurate predictions, it needs high-quality features that can explain sufficient variation in the output (in other words, determine what category your prediction belongs to). But with scaling machine learning operations, one may find it challenging or repetitive to upkeep an ML ecosystem. Furthermore, your team can risk definition inconsistency across your codebase by repeating the same efforts over and over. Endeavor has solved this problem through a <strong>Feature Store</strong> which standardizes and centralizes features to ensure consistency, discoverability, and reusability.</p><p>Features can take a significant time to develop, depending on the complexity of the calculations. You can spend weeks aggregating the raw data on hand into a digestible format that makes sense for your model. Some features can be common across several use cases — lifetime ticket sales or most-used streaming devices, for example — which can be predictive for both transactional and browsing behavior models. If multiple data scientists work on models using the same source, problems like definition inconsistency can emerge when the same features are created multiple times for different objective functions (i.e., the predictive question you want to be answered). Data engineering efforts can be streamlined, using a common hub of features that can be used across multiple clients and multiple models.</p><p>Enter the feature store.</p><p><strong>A feature store is a common code repository where multiple contributors can define features for model development.</strong></p><p>In a feature store, you can have a schema consisting of several feature tables, each providing a glance at the common identifier you are trying to predict upon. We can walk through an example using the work done with <a href="https://www.endeavorstreaming.com/">Endeavor Streaming.</a> Endeavor Streaming brings leading technologies and digital media expertise to the streaming world and provides an array of analytical capabilities for sports streaming giants. With each client (defined as the business Endeavor partners with to stream their content), Endeavor Streaming effectively collects data on their streaming and transactional behavior, which can then enhance the customer experience. Data scientists on our team then use these raw data streams to provide machine learning predictions for clients. If you were doing machine learning for a data source on video streaming data, you could have the following breakdown, for example:</p><p>Potential Objective Functions:</p><ul><li><strong>Churn</strong> — What is the likelihood a customer will cancel their subscription?</li><li><strong>Pay-Per-View Re-Purchase</strong> -What is the chance a customer who bought a Pay-Per-View license buys one again?</li><li><strong>Customer Re-Engagement</strong> -What is the probability that a customer would be likely to stream a specific piece of content?</li></ul><p>All these questions are very different to answer as far as marketing actionability and product insights are concerned, but these questions all pertain to a common identifier: the customer. If we know enough about the customer, we can determine answers for all these objective functions using the data on hand. Below, we outline how this feature store structure could be used for the UFC and how data on viewership during the weekly fights could be leveraged accordingly. We can utilize knowledge of engagement activity in minutes for a churn problem and continue using this aggregation for a Re-Engagement problem. Continuing with the example of streaming, one of these tables could look like:</p><p><strong>Table 1:</strong> Example Viewership Raw Data Stream for UFC</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*1-ouDKsgmhOZUn0ZbMSkWw.png" /></figure><p>Endeavor leverages Snowflake databases to store our data and we maximize efficiency by utilizing DBT models to store queries of pertinent data aggregations. Using DBT, we could build test and inference datasets at the customer level (identified by the customer_EXID to answer the objective functions on hand.</p><p>Some features that could be derived from these datasets:</p><ul><li>Devices Used by Customer</li><li>Minutes Watched by Customer</li><li>Country of Viewership by Customer</li><li>Content Viewed by Customer</li><li>First Main Card Content Viewed</li></ul><p>On the surface, you could use these traits individually and make the three models above for however many clients you have. In this example, we will assume Endeavor Streaming has two clients:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vqAZQ2yA5zz_kZ-LgkGsTg.png" /><figcaption>Chart 1: Feature Store Centralizes All Data Manipulation</figcaption></figure><p>This “works.” The data manipulation step of the viewership data for two clients can produce two data frames (test and inference) for each objective function (3). This brings us to twelve distinct models, all drawing from similar sources and producing similar output. Notice that not all features are used in every model, so this is specifically accounted for in each individual model’s creation via DBT. This however, is inefficient for a few reasons:</p><ul><li>Feature Definition Drift — As the same features get recreated for each of these models, there are no mechanisms in place for the definition of that feature to remain consistent across models</li><li>Repetition — As new clients get introduced, the process of recreating these exact models for those clients must be repeated.</li><li>Model Upkeep — As the data changes or new depth on the target variable is exposed over time, the smallest change must be manually implemented across all models one by one.</li></ul><p>The feature store solves these issues by storing all features in one place for all data sources to feed into and all models to draw from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVAUcmqXkgin1ZpBvS2zWQ.png" /><figcaption>Chart 2: Feature Store Centralizes All Data Manipulation</figcaption></figure><p>We are now able to mediate the above concerns by centralizing feature development in one location. All features, no matter the use case, live in the feature store. One model can simply “check out” any feature they like from the store, simply by joining on the row level identifier (customer). The beauty of this design doesn’t stop here; any client that is added henceforth will simply follow the logic of its predecessors went through to create the features. Since the raw viewership data structure coming into the database is identical, if we have a new “Client 3” onboard, we can simply apply the same logic as existing clients and generate these same features for Client 3 with relatively low effort. This also enables us to quickly build models (test and inference) for these new clients.</p><p>This is useful for a number of reasons:</p><ul><li>Feature development work can be easily accessed by other data scientists</li><li>Computation of features is now automated for all use cases</li><li>Training and inference datasets are consistent in features</li><li>Replication of models across different use cases is quickly scalable</li></ul><p>The entries in the feature store are still SQL Features extracted via DBT Models, each living as a feature table. Each of these feature tables has several attributes about the customer in them, all at the customer level. Devices Used by Customer (feature_devices) could have features inside them that describe the customer like first_device_used, most_commonly_used_device, multi-device_user, etc. These are all features that can now be chosen to help predict whichever objective function you are developing. Once the features exist, we would just need to join them all together based on our common identifier (customer).</p><p>While the data being processed from the unique clients serviced by Endeavor Streaming exist in the same feature store, the data becomes federated in model development simply by filtering the feature store based on the client. In the UFC example, the machine learning models can only be developed <em>after </em>UFC-specific<em> </em>training sets are generated, thereby guaranteeing no cross-client data leakage. Data security is built into the design and proves to be another benefit of the feature store here at Endeavor Digital.</p><p>We want to get all tables into a common row-level aggregation, and in the use case for the work with Endeavor Streaming, we care to boil things down to the customer level, denoted by CUSTOMER_EXID. Aggregations of the viewership start time/end time can be very useful and worthy of a feature table in our store.</p><pre>WITH minutes AS (<br>  SELECT <br>    customer_exid, <br>    DATEDIFF(<br>      &#39;minute&#39;, session_start, session_end<br>    ) AS session_length, <br>    DATE(session_start) AS session_date <br>  FROM <br>    viewership<br>) <br>SELECT <br>  customer_exid, <br>  AVG(session_length) AS avg_session_length, <br>  COUNT(DISTINCT session_date) AS distinct_days_active <br>FROM <br>  minutes <br>GROUP BY <br>  customer_exid</pre><p><strong>Table 2:</strong> Feature Table of Minutes Related Data (feature_minutes)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*0tcIb71fyMHR8FFR1FzgBg.png" /></figure><p>And conversely, we can do the same for our knowledge of what devices they use to stream.</p><pre>SELECT <br>  customer_exid, <br>  MODE(device) AS most_used_device <br>FROM <br>  viewership <br>GROUP BY <br>  customer_exid</pre><p><strong>Table 3:</strong> Feature Table of Device Related Data (feature_devices)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*ErjfnQX8n7qOw4p6nKJAMw.png" /></figure><p>Because Endeavor Streaming’s data streams are consistent in their structure, we can rely on these feature tables to automatically compute these features once it’s created.</p><p>With Churn, we could have a list of a subset of historic customers (represented as CUSTOMER_EXID) as well as an identifier as to if they have churned, or cancelled their subscription, or not (they are currently active).</p><p><strong>Table 4:</strong> Customer Table of Churn Status (churn_status)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*hlUW8GwpdP1_kWav0mXZPw.png" /></figure><p>Our goal with this objective function is to find quality predictors that can best classify a customer’s status as active or expired. While this part of the magic happens in the actual machine learning, we must create a dataset. But the beauty of the feature store is we can actually very quickly create a dataset to train based on the tables we already have. We do not need to make a complicated query to make this table, rather we just need to join all the features we want together.</p><pre>SELECT <br>  A.customer_exid, <br>  A.status, <br>  COALESCE(B.avg_session_length, 0) AS avg_session_length COALESCE(B.distinct_days_active, 0) AS distinct_days_active, <br>  COALESCE(C.most_used_device, &#39;Invalid&#39;) AS most_used_device <br>FROM <br>  churn_status A <br>  LEFT JOIN feature_minutes B ON A.customer_exid = B.customer_exid <br>  LEFT JOIN feature_devices C ON A.customer_exid = C.customer_exid</pre><p><strong>Table 5:</strong> Training Dataset to Predict Churn Status (ToTrain_churn_status)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*kN2LBP5FLwW38WWuilN7ow.png" /></figure><p>As you can see, we now have a dataset that we can make predictions of status on, using all my features as predictors at the CUSTOMER_EXID level. You may also notice something critical here: not all customers will appear in each of your feature tables. A customer who has an account but never viewed any content would never appear in the viewership table, or any feature tables which stem from it. As a result, you want to ensure you add COALESCE() to most if not all of your joins to account for these otherwise null values.</p><p>If another data scientist wants to predict the likelihood that a customer who bought a Pay-Per-View license buys one again, they can check out features that were already created in the churn use case for their objective function. This significantly reduces model development time and ensures definitions of features are common across models. Furthermore, when it comes time to productionize your models, you can pull from the feature store again when building your inference sets. In using a feature store, you optimize development of ML training and create a robust ecosystem for your data science team to move both effectively and efficiently.</p><p><em>Wanna chat more about data with the author? Contact John Thomas directly on </em><a href="https://www.linkedin.com/in/johnnytommy/"><em>LinkedIn</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=680ee6d45c64" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Platform at Endeavor]]></title>
            <link>https://medium.com/@endeavordata/machine-learning-platform-at-endeavor-93ba88b66986?source=rss-618e193438f6------2</link>
            <guid isPermaLink="false">https://medium.com/p/93ba88b66986</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[media-and-entertainment]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Endeavor: Data Blog]]></dc:creator>
            <pubDate>Thu, 18 Aug 2022 14:37:31 GMT</pubDate>
            <atom:updated>2022-08-18T14:37:31.456Z</atom:updated>
            <content:encoded><![CDATA[<p>By Saad Zaheer</p><p>Endeavor is a global sports and entertainment company at the intersection of all forms of culture. Whether you stream a UFC fight, attend New York Fashion Week, experience premium hospitality at the Super Bowl, or enjoy the latest Hollywood flick, there is an Endeavor business behind your experience, powering lifelong memories for you and your loved ones. Our goal is to reach, engage, and delight our customers wherever they are and however they choose to engage with us.</p><p>We have developed a machine learning platform (MLP) at Endeavor that combines the best features of the most modern data technologies to enable data scientists to develop and deploy machine learning applications at scale. This article explains the underlying architecture and components of Endeavor’s MLP. So far, we have deployed MLP-based solutions for UFC, On Location, Endeavor Streaming, and IMG Events. Examples of deployed models include ranking and scoring of leads for On Location, the likelihood of UFC users to stream additional content, the likelihood to re-engage lapsed UFC customers, and personalized event recommendations for IMG events such as Hyde Park Winter Wonderland and Hall Des Lumieres.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1kWmlT1ukUgGO1yNXX-v5g.png" /><figcaption>Diagram 1: Machine Learning Platform Architecture</figcaption></figure><p>The diagram above describes the technologies and data flow processes that comprise MLP. These should be familiar to machine learning practitioners. Endeavor’s MLP is an attempt to create a closed-loop around the disparate tasks of ML engineering, such as data ingestion, feature engineering, model research and development, deployment, inference, and export. Let us deep dive into each area.</p><h3>Overarching Principles</h3><p>Historically, typical machine learning workflows start out with data scientists building models in Python, and then machine learning engineers rewriting them in more performant libraries for production, along with data engineers rewriting entire feature engineering pipelines. A typical Machine Learning model can take months from research to deployment while changing multiple hands. Our goal here is to simplify and speed up this process by choosing technologies that can help close the loop from data ingestion to export of predictions within simply connected processes. Endeavor’s MLP consists of six distinct processes that have been designed with the following principles in mind:</p><ol><li>Data scientists should be able to research, develop, and deploy models in production with minimum support from engineering.</li><li>Processing of data into features, training sets, and inference sets is best handled by a large-scale data processing engine such as Snowflake or BigQuery and expressed as DBT models which data scientists can develop in standard SQL syntax.</li><li>Orchestration of data pipelines handled by standard orchestration frameworks such as Prefect or Airflow deployed on a cloud computing service, such as Kubernetes on AWS or GCP.</li><li>Model inference and training are best handled by standard machine learning libraries in Python.</li><li>Snowflake or BigQuery erves as a landing zone for raw data as well as materialized model inferences, while cloud storage such as AWS S3 or GCS serves as a storage zone for model objects.</li><li>Artifacts follow strict naming conventions, e.g. {artifact}_{client_name}_{model_name}.extension across all systems. This includes DBT models describing features, training, validation, inference sets, and model objects. This allows complete traceability of model artifacts from ingestion to inference and exposes operational analytics on system health.</li></ol><p>Our data engineers have written four generalizable and configurable data pipelines in Prefect for data ingestion, model deployment, model inference and data export. The data science team can add new models and data sources by modifying config files in Prefect pipelines:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*6r5bmOgXluI7d00wFsxN_Q.png" /></figure><p>The above snippet corresponds to the likelihood_to_stream model config in the Model Inference process. Pre-configured Prefect pipelines can connect to Snowflake and S3 using pre-defined roles. Secrets are declared as Kubernetes environment variables which read from AWS secrets manager. Model deployment takes as long as a simple code review by an engineer instead of weeks of rewriting Python code.</p><h3>1. Data Ingestion</h3><p>Data ingestion pipelines are perhaps the most customized components of the MLP. Largely driven by bespoke data sources such as application event streams coming through Segment, point of sale data from ticketing companies, customer relationship data from CRM systems such as Salesforce, demographic attributes from data enrichment providers, or custom data sources such as the Endeavor Streaming platform or On Location ticketing and reservations systems, our ingestion pipelines can break down into two types:</p><ol><li>Near real-time streams coming through a data platform such as Segment</li><li>Batch streams coming via SFTP or S3 that are read into Snowflake via a Prefect pipeline.</li></ol><p>Segment data follows well-defined schema patterns and our data scientists utilize <a href="https://hub.getdbt.com/dbt-labs/segment/0.1.7/">pre-defined DBT models</a> to organize this data for feature engineering.</p><p>Prefect ingestion pipelines usually require the data scientist to declare unique_keys and schema_configuration as well as one of the three ingestion modes merge , append , or full_refresh .</p><h3>2. Feature Engineering</h3><p>Here, we introduce the concept of a feature store. Each data source, e.g. point-of-sale data or segment data, follows consistent schema patterns and semantic definitions. Therefore, a recency feature for a specific data source applies to different business units such as On Location or Hyde Park Winter Wonderland. We organize features as DBT models inside a DBT folder called feature_store under the data source segment with naming convention as feature_feature1_defn.sql and so on. These features are documented in feature_feature1_defn.yml files and can be discovered in an internally hosted dbt-docs website (<a href="https://github.com/dbt-labs/dbt-docs">see here</a>). This lineage diagram from the Endeavor Streaming feature store illustrates how different features are used for the likelihood_to_stream model. Some of these features are clearly applicable to likelihood_to_churn or likelihood_to_repurchase_ppv models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T-2QaWw14Fq9QVpcrYoH1g.png" /><figcaption>Diagram 2: Feature Store Architecture</figcaption></figure><h3>3. Model Research and Development</h3><p>Rapid prototyping and iteration are the most critical process of the MLP. Our data scientists experiment in a Kubernetes hosted Jupyter Hub environment where they use standard Python ML libraries such as sklearn, xgboost etc. Hyperparameter tuning requires significant computing resources so the data scientists have access to a <a href="https://kubernetes.dask.org/en/latest/">Dask cluster</a> deployed on Kubernetes. Ultimately, the outcome of this process results in a model training object that is configured to read a DBT model that lives in Training store and a joblib object that is stored in AWS S3.</p><h3>4. Model Deployment</h3><p>The joblib object output from the model R&amp;D process is stored in AWS S3 under the following directory structure:{client_name}/{model_name}/active/joblib_{client_name}_{model_name}_{model_version}.pkl . The model version deployed in production lives in the /active directory. The deployment pipeline ensures that, a) only one model object lives in the /active directory and, b) the correct model has been deployed by comparing the filename.pkl string with the S3 directory path. A second object that performs feature calculations and filtering before inference step also gets exported to the S3 bucket under the same directory: {client_name}/{model_name}/active with a similar naming convention to the model object.</p><p>Finally, model deployment includes writing ToScore and ToExport DBT models in a Marts DB in Snowflake which respectively perform the following functions:</p><ol><li>ToScore models: this object organizes new data and features for ongoing scoring.</li><li>ToExport models: this object assumes that Model Inference materializes scored data into a predefined schema and organizes that data into a format that is ready for export to a destination. It includes, for example, logic for selecting the most recent predictions for a given user.</li></ol><h3>5. Model Inference</h3><p>This process is executed by a Prefect data pipeline that takes the location of the model objects in AWS S3 and ToScore data models as input parameters and, after performing pre-defined data transformations, applies a model.predict transformation to the inference set, and publishes new scores to a Snowflake landing database. The Prefect pipeline is completely generalized and can be adapted to new models and inference sets by declaring configuration variables as described in the Overarching Principles section above.</p><h4>Package Dependency Management</h4><p>Different ML models have different Python package dependencies, and since inference takes place on dockerized Kubernetes VMs, these dependencies need to be installed at runtime. However, requiring bespoke pip install package commands to run on demand can impact cluster network uncontrollably. Therefore, we pre-install the specific packages for each model into Python wheel objects when the docker image is built.</p><p>Each model’s dependencies are declared in a file titled requirements.{model_name}.txt while the general pipeline dependencies are declared in requirements.txt. Both files live in the \root directory of the Prefect inference pipeline and are read in by the above shell commands upon docker build.</p><h3>6. Model Export</h3><p>This process is executed by a Prefect pipeline that gets triggered by the Inference process whenever export_to_destination = True . Destinations can vary based on the business need, e.g. an AWS S3 bucket that connects with Snowflake via storage integration, a CRM system such as SalesForce that gets written through AWS AppFlow, an email serving platform such as DotDigital that gets written into via API orchestrated by Prefect, or a pre-configured destination in Segment such as MailChimp that can accept data via SQL computed traits. We use dbt post_hooks to grant appropriate export roles Select access to the ToExport views as well as apply any decrypt masks on PII data so personalized marketing actions can be taken in the appropriate platform as required by business needs (and as legally permissible).</p><h3>Conclusions</h3><p>Endeavor has developed a closed-loop Machine Learning Platform that processes customer interactions and organizes them into features used to make predictions against well-defined business objectives such as reducing customer churn or recommending event tickets to purchase. These predictions are actioned via personalized marketing messages or personalized user experiences, which generate more interactions that feed back into the MLP. Through the MLP, our data scientists have generalized data pipelines and compute and orchestration engines at their disposal which can be easily configured for multiple models, clients, and use case combinations. The novelty of our design is in utilizing two different systems: Snowflake for data storage and large-scale data processing, and Prefect on Kubernetes for pipeline orchestration, deployment, and configuration management. The versatility of this system has allowed us to unify the role of data scientist, data engineer, and machine learning engineer into one person who can concentrate on model development and feature engineering while simplifying deployment and inference. This has twofold benefits: machine learning teams can move faster without the cognitive load of translating technical concepts across different skill sets, and don’t need to hire specialized skill sets other than data scientists who are skilled in SQL and Python.</p><p><em>To learn about what the Endeavor Digital team is up to, reach out to us via </em><a href="/service/mailto:digital_comms@endeavorco.com"><em>digital_comms@endeavorco.com</em></a></p><p><em>Interested in continuing the conversation further with the author? Contact Saad Zaheer directly on </em><a href="https://www.linkedin.com/in/saad-zaheer-52429669/"><em>LinkedIn</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93ba88b66986" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Infrastructure-as-Code to Manage Segment Access]]></title>
            <link>https://medium.com/@endeavordata/using-infrastructure-as-code-to-manage-segment-access-877465bbf268?source=rss-618e193438f6------2</link>
            <guid isPermaLink="false">https://medium.com/p/877465bbf268</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[segment]]></category>
            <category><![CDATA[access-management]]></category>
            <category><![CDATA[data-team]]></category>
            <dc:creator><![CDATA[Endeavor: Data Blog]]></dc:creator>
            <pubDate>Fri, 29 Jul 2022 17:42:58 GMT</pubDate>
            <atom:updated>2022-07-29T17:43:59.859Z</atom:updated>
            <content:encoded><![CDATA[<p>By Ilya Galperin</p><h3>Introduction</h3><p>This article is an introduction to the Endeavor Digital engineering team’s approach to managing Segment, a widely used customer data platform. Our data engineering team is a part of Endeavor Digital, which sits at the intersection of technology, content, and celebrity talent. The Endeavor Digital team operates mature, large-scale businesses and launches new businesses that are driving the future of media and entertainment.</p><p>In this post, we will focus on our role-based access management design pattern and its implementation using infrastructure-as-code principles and the Segment Public API. Over the coming months, we will be sharing more about our engineering practices and their implementation in our data stack. We hope this article can be informative and useful for other engineering teams facing similar problems.</p><h3>What is Segment?</h3><p><a href="https://segment.com/">Segment</a> is a customer data platform (<a href="https://segment.com/resources/cdp/"><em>CDP</em></a>), which enables organizations to easily capture, process, and route customer data for analytics and marketing usage. Segment boasts over <a href="https://segment.com/catalog/?ref=nav">300 built-in integrations</a> including payment processors, web applications built in various languages, CRM and marketing platforms, and data warehouses.</p><p>At Endeavor, much of our data infrastructure and services are managed in-house through data orchestration and storage tools like Prefect, dbt, and Snowflake. However, we’ve found that including Segment as part of our data stack has enabled us to rapidly onboard additional data sources — especially websites and applications — and expose those sources to our data science team for further exploratory analysis. For example, when working with a new internal customer, i.e. the UFC, our team can send them a simple one-line JavaScript snippet to include in whichever websites we’d like to enable user activity tracking against.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XMBZj7gvxtB0daDvIcimkw.png" /><figcaption>Our Data Flows</figcaption></figure><h3>Our Business Needs</h3><p>Before we dive further into our implementation of role-based access control, let’s discuss some of the unique needs that we have at Endeavor and what this framework does to help achieve them.</p><p>Our data team manages pipelines and provides analytics for dozens of Endeavor brands or “business units” such as the UFC, World’s Strongest Man, Frieze, Professional Bull Riders, etc. Given that we serve a conglomeration of businesses, each with its own distinct customer data, responsible stewardship is always top of mind when architecting pipelines.</p><p>We are diligent in applying a federated model to separate different business units’ data within our infrastructure. In the realm of Segment, this means that every business unit has its own set of sources and <a href="https://segment.com/docs/personas/">personas</a> to prevent any sort of crossover of data where it shouldn’t exist.</p><p>Practically, this translates to us having dozens of sources and personas spaces, all of which are methodically created, named, and labeled according to which property they belong to. For example, sources are named in the following format:</p><p><em>BUSINESSNAME_SOURCETYPE_SOURCENAME_ENVIRONMENT</em> i.e. <em>UFC_JAVASCRIPT_MAINSITE_PROD</em></p><p>Moreover, all sources have a minimum of two labels — <em>property</em> and <em>environment,</em> i.e. <em>property:ufc</em> and <em>environment:prod</em></p><p>for the example above.</p><p>Being able to create these resources and labels programmatically through code is a crucial part of our overall strategy for governing Segment and we hope to publish a blog in the near future outlining our approach.</p><p>As you may be starting to suspect, <em>property</em> labels are instrumental in helping us identify which users should have access to which resources within Segment and are used directly in our implementation of role-based access control. In doing so, we enable users from different Endeavor business units to self-serve within Segment and fully realize some of the awesome capabilities of the platform.</p><h3>What is Role-Based Access Control (RBAC) and Why Use It?</h3><p>Broadly speaking, RBAC is a security framework used to manage user access to a resource based on their specific role in a group or organization. You’ve almost certainly encountered this paradigm in your professional journey. Think of a simple example of a blog post. A user whose role we can define as “writer” probably has the privileges to publish, edit, delete, or read a blog post. A user to which we assign the “reader” role will only have the read privilege.</p><p>One of the greatest benefits of RBAC is its scalability. In less mature organizations, you might find access governed at the level of an individual being assigned permissions on an ad hoc basis i.e. Aliyah has “read”, “publish” &amp; “edit” permissions, Miguel has “read” &amp; “delete” permissions, Faye has “read” &amp; “publish” permissions. As this team of three grows into a team of thirty and beyond, onboarding and managing access becomes increasingly difficult to scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CKvB5lE0hTgWNl_c_9_XPw.png" /><figcaption>Ad-hoc vs. Role-Based Access Control</figcaption></figure><p>Endeavor has dozens of users that interact with Segment, many of whom operate within different business units. This includes marketers, product managers, data engineers and scientists, analysts, and members of our IT team. As these users have discrete roles in our broader organization and often times their own distinct business units, using role-based access control principles to govern their access makes a lot of sense.</p><h3>RBAC-as-Code</h3><p>While Role-based Access Control is nice and possible to implement on Segment using just its web interface, combining it within an “infrastructure-as-code” (<em>IaC</em>) framework makes it even more useful. Infrastructure-as-code is exactly as it sounds: the managing and provisioning of applications through code and configuration files. Our engineering team is a big fan of IaC and uses it wherever advantageous and feasible, especially in the realm of access control. Thankfully, Segment exposes the <a href="https://api.segmentapis.com/docs/">Segment Public API</a> which enables us to write code that directly interacts with Segment’s identity-access management settings.</p><p>So why did we decide to use infrastructure-as-code to manage RBAC?</p><ol><li>“Nuclear Sub Principle”: By committing all of our access management documents as configuration files to a git repository, we can use peer review and require a minimum number of approvers to provision access. This removes the requirement of having a human administrator interacting with a web interface and instead allows 2+ developers with sufficient access to review a pull request, and do so with peer oversight.</li><li>Auditability: Having our configuration files version controlled in a shared repository allows our team to understand exactly which users had what level of access at any point in time, and who granted that access.</li><li>Scalability: Besides the advantages of RBAC described in the previous section, IAC makes expansion even more streamlined by removing admin-user bottlenecks and introducing a common, singular interface for multiple platforms’ access management settings.</li><li>Repeatability: We’re able to easily reproduce our permission structures across Segment workspaces, i.e. production and development, and make a full recovery of our access management configuration in case of catastrophic failure.</li></ol><h3>Implementation</h3><p>Before describing our method to implement RBAC-as-Code, let’s further define a few of the object types that we’ll be working with:</p><ul><li>Users:<br>- Individuals that can be assigned to a user group<br>- Never assigned permissions directly, rather they inherit permissions from their group<br>- Can belong to multiple user groups<br>- Identified by their email address</li><li>Roles: A basic grouping of permissions i.e.:<br>- Minimal Workspace Access | Access to view the workspace. Cannot view any sub-resources or make changes to the workspace.<br>- Source Read-only | Read-only access to assigned Source(s), Source settings, enabled Destinations, Schema, live data in the Debugger, and connected Tracking Plans.</li></ul><p>You can find a comprehensive list of roles available in Segment <a href="https://segment.com/docs/segment-app/iam/roles/">here</a>.</p><ul><li>User Groups:<br>- These hold users and one or more roles that users belonging to the user group are eligible to assume<br>- User groups are a useful abstraction above the level of a role that allows us to overlay our business-specific role definitions over Segment’s built-in roles</li></ul><p>By utilizing user groups, we’re able to define access based on our organization’s specific roles. At Endeavor, we may use groups like <em>Data Scientist</em>, <em>Taste of London Marketer</em>, and <em>IT Administrator</em> to categorize various levels of access.</p><p>Using infrastructure-as-code as principles, all configurations for manipulating user, role, and user group objects are defined in <a href="https://yaml.org/">YAML</a> files. Of course, JSON or any other data serialization language can be used in its place but we prefer YAML due to its readability.</p><p>We have two kinds of YAML files: users and groups. These files are processed using a wrapper that we’ve built around the <a href="https://api.segmentapis.com/docs/">Segment Public API</a>.</p><p><strong>On and Off-boarding Users:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/197/1*UhfHaplu31_WIADFsNnNIA.png" /></figure><p>User files accept two inputs: an email address and status.</p><p>The email address acts as a unique identifier for a user and the status as a toggle to determine whether we want to on or off-board a user. To send an invite out, we set the status to <em>active</em>, and when off-boarding access, we simply switch this field to <em>inactive</em>.</p><p>To accomplish this, we utilize the following Public API endpoints:</p><p>1 GET /invites: Returns a list of invitations to join a Workspace. <br>2 DELETE /invites: Removes a list of invitations to join a Workspace. <br>3 POST /invites: Invites a list of Users to join a Workspace.</p><p><strong>Assigning Users and Roles to Groups:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*9osS4ttAQhFn24QGcmu2AA.png" /><figcaption>sample group .yaml file</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/208/1*5W8YtGUZmu6J-hh75z85Eg.png" /><figcaption>sample group .yaml file</figcaption></figure><p>User group files accept the following configuration options:</p><ul><li><em>Status</em>: used for group on and off-boarding.</li><li><em>Permissions</em>:<br>- <em>Role</em>: a pre-defined Segment <a href="https://segment.com/docs/segment-app/iam/roles/">role</a>.<br>- <em>Type</em>: the subset of objects this role should apply to. We allow for three input parameters here: <em>all</em>, <em>label</em>, and <em>entity</em>. <br> a) <em>All</em> grants access to all objects belonging to the Segment role i.e.<em> </em>Type <em>“Source Read-only” </em>and type<em> “all” </em>would grant read-only privileges on all sources in the workspace.<br> b) <em>Label</em> grants access to all objects with a particular label or label(s) defined in the <em>Target</em> field.<em> “Source Read-only”</em> and targets<em> “property:taste_of_london”</em> and<em> “property:tol”</em> would grant read-only access to all sources that have property labels of “<em>taste_of_london</em>” or “<em>tol</em>”.<br> c) <em>Entity</em> grants access to specific entities based on their name(s) as defined in the <em>Target</em> field.<em> </em>For example<em>, “Source Read-only” </em>and target<em> “Taste_of_London_Main_Website”</em> would grant read-only access to a source named <em>“Taste_of_London_Main_Website”</em>.<br>- <em>Target</em>: Used to scope down access to specific labels or individual objects.</li><li><em>Users</em>: a list of users as defined by their email address that are part of this group. Removing a users email address from this list will also remove them from the group.</li></ul><p>We use the following Public API endpoints to manage user groups:</p><p>1 POST /groups: Creates a User Group. <br>2 DELETE /groups/{userGroupId}: Removes a user group from a workspace. <br>3 POST /groups/{userGroupId}/users: Adds a list of users or invites to a user group. <br>4 DELETE /group/{userGroupId}/users: Removes users or invites from a user group. <br>5 POST /groups/{userGroupId}/permissions: Adds access permissions to a User Group. <br>6 PUT /groups/{userGroupId}/permissions: Updates access permissions for a User Group. 7GET /labels: Returns a list of all available Labels</p><h3>Conclusion</h3><p>We hope that this article proves useful as a basic introduction to the use of role-based access control, infrastructure-as-code, and our implementation of these paradigms via the Segment Public API. Please feel free to look up the author on <a href="https://www.linkedin.com/in/ilya-galperin/">LinkedIn</a> if you’d like to further chat about this topic.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=877465bbf268" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>