<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ryan Lamb on Medium]]></title>
        <description><![CDATA[Stories by Ryan Lamb on Medium]]></description>
        <link>https://medium.com/@ryanelamb?source=rss-adce0c00c3e2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*E-ME61YnLvRGi4Ktc6rcVQ.jpeg</url>
            <title>Stories by Ryan Lamb on Medium</title>
            <link>https://medium.com/@ryanelamb?source=rss-adce0c00c3e2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:24:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ryanelamb/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A data engineering project with Prefect, Docker, Terraform, Google CloudRun, BigQuery and Streamlit]]></title>
            <link>https://medium.com/@ryanelamb/a-data-engineering-project-with-prefect-docker-terraform-google-cloudrun-bigquery-and-streamlit-3fc6e08b9398?source=rss-adce0c00c3e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fc6e08b9398</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[prefect]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[Ryan Lamb]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 17:29:38 GMT</pubDate>
            <atom:updated>2023-03-17T08:36:16.559Z</atom:updated>
            <content:encoded><![CDATA[<h4>Orchestrate data pipelines to find late buses in real-time — from raw data to live data visualisation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qIu_LvYz0BOjCqzB" /><figcaption>Photo by <a href="https://unsplash.com/@chan_lee94?utm_source=medium&amp;utm_medium=referral">chan lee</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article demonstrates how to set up a data pipeline with Google Cloud Services, orchestrate it with Prefect and present the transformed data to users in a live web app with Streamlit. The transformed data is also consolidated in BigQuery, ready for further analysis.</p><p>This workflow is useful because it provides an example of how data professionals, from data engineers to data scientists, can easily create and manage the full end-to-end process in delivering live, actionable insights to end-users. The combination of modern tools used, such as Prefect and Streamlit, makes it accessible to a wide range of skillsets. Even if you’re just starting out and want to learn the ropes, this entire project can be built and run for free (for a fixed period of time) using Google’s generous free trial.</p><p>The data used is from the <a href="https://www.bus-data.dft.gov.uk/">UK Bus Open Data Service</a> and the final app displays late-running buses in real-time on an interactive map. The workflow can, however, be replicated with any data of your choice — I’ve purposely chosen to focus more on the workflow and pipeline development process and less on the data itself.</p><p>By the end of this project you’ll have:</p><ul><li>Created and managed cloud resources with Terraform.</li><li>Changed ETL functions into tasks and flows that will be orchestrated with Prefect.</li><li>Created Prefect blocks to manage authentication and interaction with Google Cloud resources.</li><li>Created deployments to manage flow scheduling.</li><li>Created a Docker image for the script execution environment.</li><li>Pushed the Docker image to Google Artifact Registry.</li><li>Updated the Prefect deployment for Cloud Run to run the scripts.</li><li>Set up the Prefect agent on a Compute Instance.</li><li>Created a data visualisation app with Streamlit.</li></ul><p>The live app can be viewed <a href="https://latebuses.streamlit.app/">here</a> and the full project code <a href="https://github.com/RyanEricLamb/data-engineering-bus-tracker">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*kyHBqvuaBxvz-_3kOh7sDA.png" /><figcaption>Image by author</figcaption></figure><h3>1a. Project setup: Development environment</h3><p>I use <a href="https://code.visualstudio.com/">VS Code</a> as my editor and there are a number of useful tools and libraries to set up before you begin development. Some examples:</p><ol><li><strong>Black</strong></li></ol><p><a href="https://github.com/psf/black">Black</a> is a Python code formatter. It helps enforce a consistent code style (especially helpful if you have multiple developers working together), and speeds up development as you can spend less time formatting your code and more time coding.</p><p>Once Black is installed, you can simply run it on your Python file or a whole directory:</p><pre>black {source_file_or_directory}</pre><p>If you are using VS Code, you can set Black to automatically format your code whenever you save. Once Black is installed, go to VS Code settings, then check <em>Editor: Format on Save</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L8hbz7rSGsSQTrPRQXhZ3g.png" /><figcaption>Image by author</figcaption></figure><p><strong>2. Flake8</strong></p><p><a href="https://flake8.pycqa.org/en/latest/">Flake8</a> is a code linter that will check your code’s compliance to PEP8 coding conventions.</p><p>Note that there may be some conflicts between Black and flake8, so we can create a <em>setup.cfg</em> file to adjust some of flake8’s settings so that it is compatible with Black:</p><pre>[flake8]<br>ignore = E203, E266, E501, W503<br>max-line-length = 88<br>max-complexity = 18<br>select = B,C,E,F,W,T4<br>exclude =<br>    venv<br>    .venv<br>    .git,<br>    __pycache__,<br>    docs/source/conf.py,<br>    old,<br>    build,<br>    dist</pre><p><strong>3. Pre-commit hooks</strong></p><p><a href="https://pre-commit.com/">Pre-commit</a> is a framework that allows you to run scripts before committing your code. If the script fails then the commit isn’t made. We can run Black and flake8 as pre-commit hooks, so that whenever code is committed, Black will automatically format it and flake8 will check its compliance. If flake8 fails, then the commit is denied.</p><p>Once pre-commit is installed, you will need to define your pre-commit config file. Create a <em>.pre-commit-config.yaml </em>that will define the sources where the hooks are taken from. You can also reference local repos which you can read more about <a href="https://pre-commit.com/#repository-local-hooks">here</a>.</p><pre>repos:<br>-   repo: https://github.com/psf/black<br>    rev: stable<br>    hooks:<br>    - id: black<br>      language_version: python3.6<br>-   repo: https://gitlab.com/pycqa/flake8<br>    rev: 3.7.9<br>    hooks:<br>    - id: flake8</pre><p>Once you’ve configured your yaml file, run <em>pre-commit install</em> to install your hooks in your git directory. Now whenever you commit your code, both Black and flake8 will first run and you can make amendments as necessary.</p><p><strong>4. Gitignore templates</strong></p><p>If you aren’t certain about which files to include in your <em>.gitignore</em> file, you can use one of these comprehensive <a href="https://github.com/github/gitignore">gitignore templates.</a></p><p><strong>5. Pip-tools</strong></p><p>Using <em>pip-compile</em> from <a href="https://github.com/jazzband/pip-tools">pip-tools</a> is helpful when generating your <em>requirements.txt</em> file. This will make sure that builds are predictable and deterministic. You create a <em>requirements.in </em>file containing only your direct project dependencies, then running</p><pre>pip-compile .\requirements.in</pre><p>will generate a <em>requirements.txt</em> file that will contain the modules listed in your <em>requirements.in</em> file as well as the transitive dependencies of those modules. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*6fUAmgWXPP0RNU-aMqFPgA.png" /><figcaption>Image by author</figcaption></figure><h3>1b. Project setup: Google Cloud account</h3><p>For this project we’ll be using the following services from Google Cloud:</p><ul><li>Artifact Registry to store our Docker image</li><li>Cloud Run to run our Python scripts</li><li>Compute Engine to run our Prefect agent</li><li>Cloud Storage to store our data</li><li>BigQuery for our serverless data warehouse</li></ul><p>Google offers a generous <a href="https://cloud.google.com/free">free trial</a> that you can use to replicate this project or create your own using a wide variety of services.</p><h4>Account setup and authentication</h4><ol><li>Create a new project with a unique Project ID.</li></ol><p>2. In the same project, create a new Service Account under IAM. A service account can be used to manage access to Google Cloud services.</p><p>3. In the new service account, click on <em>Actions </em>then <em>Manage keys</em>. Create a new key which will be downloaded as a JSON file — keep it safe!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eTsgVjVROTnsNPdCZbO9Kg.png" /><figcaption>Image by author</figcaption></figure><p>4. Install the<a href="https://cloud.google.com/sdk/docs/install-sdk"> Google Cloud SDK</a></p><p>5. Set your credentials environment variable to the location of your JSON key file:</p><pre>export GOOGLE_APPLICATION_CREDENTIALS=&quot;your_key.json&quot;</pre><p>6. Authenticate with:</p><pre>gcloud auth application-default login</pre><p>which will redirect you to a browser login to sign in to your Google account. Once confirmed, you are authenticated and are able to locally interact with your Google Cloud environment.</p><h4>Permissions and APIs</h4><p>We need to add permissions to the Service Account so that we can interact with the various services we require. We are going to add broad admin permissions, but typically you would want to make permissions as restrictive as possible e.g. restrict access to a bucket/resource only.</p><ol><li>Go to IAM then next to your Service Account click on Edit Principal and assign the following roles:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*4fVULj5xu7k7vcc9v9t04A.png" /><figcaption>Image by author</figcaption></figure><p>2. We still need to enable the APIs for the services we are going to use so that we can interact with them. Go to each of these APIs and enable them:</p><ul><li><a href="https://console.cloud.google.com/apis/library/iam.googleapis.com">IAM</a></li><li><a href="https://console.cloud.google.com/apis/library/iamcredentials.googleapis.com">IAM credentials</a></li><li><a href="https://console.cloud.google.com/apis/library/compute.googleapis.com">Compute Engine</a></li><li><a href="https://console.cloud.google.com/apis/library/artifactregistry.googleapis.com">Artifact Registry</a></li><li><a href="https://console.cloud.google.com/apis/library/bigquery.googleapis.com">BigQuery</a></li><li><a href="https://console.cloud.google.com/apis/library/run.googleapis.com">Cloud Run</a></li><li><a href="https://console.cloud.google.com/apis/library/storage.googleapis.com">Cloud Storage</a></li></ul><h3>2. Infrastructure-as-Code with Terraform</h3><p><a href="https://www.terraform.io/">Terraform</a> is an open-source tool from Hashicorp that allows us to set up infrastructure resources from configuration files in a standardised way.</p><p>For example, instead of creating a Storage Bucket by clicking through multiple steps in the GCS console, we can simply define our bucket settings in a file and then create or destroy the bucket by running the file with Terraform. This gives us reproducibility and allows us to easily share infrastructure configurations. If we need to change anything in our infrastructure, we just update our file and apply it.</p><p>Hashicorp recommend a <a href="https://developer.hashicorp.com/terraform/language/modules/develop/structure">standard module structure </a>— for our example we are going to have four files in our root Terraform directory:</p><ul><li>.terraform.version</li><li>main.tf</li><li>variables.tf</li><li>terraform.tfvars</li></ul><h4>.terraform.version</h4><p>This just contains the version of Terraform that we have installed e.g. v1.3.7</p><h4>main.tf</h4><p>This is the primary entrypoint and is where we define the resources to be created.</p><p>We first define the minimum Terraform version required and the backend to be used (<a href="https://developer.hashicorp.com/terraform/language/settings/backends/local">where the state snapshot is stored</a>).</p><p>Next we define our service provider (Google in this case). This is like importing a library with modules — Terraform will use the ‘Google’ library to import predefined resource types such as the Google Cloud Storage Bucket. We also add the GCS project ID that we created earlier, the region and, optionally, we can provide credentials (we don’t have to in our case as we have already authenticated with GCS).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*4QblRLIn4XBXt3u_Q9dLPw.png" /><figcaption>Terraform registry providers. Image taken by author from <a href="https://registry.terraform.io/browse/providers">Terraform</a> website</figcaption></figure><p>We can then define the various resources that we want created. Terraform’s registry provides templates for creating resources with the various cloud providers. For example, <a href="https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket">here</a> is the template for creating a storage bucket on GCS and <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_container">here</a> is a template for managing a container in an Azure storage account.</p><p>For this project I will only be creating the storage bucket and artifact registry with Terraform. The Compute Engine and BigQuery configuration will be discussed in later steps.</p><p>For our storage bucket below, we define its name and region, and for our artifact registry we define its ID, region and format (Docker in our case). These variables are prefixed with var and the actual values for these are declared in our separate tfvars file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e201325bd047b7e20e28fa426049dc8f/href">https://medium.com/media/e201325bd047b7e20e28fa426049dc8f/href</a></iframe><h4>variables.tf</h4><p>This file contains the declarations for our variables. Each input variable is declared with a variable block:</p><p>It contains the name of the variable, the type, a brief description and a default value if required. We can set our variable values in this file but I have put them in a separate <em>.tfvars</em> file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a566f062a1c0f4750f3d3e1523cf269c/href">https://medium.com/media/a566f062a1c0f4750f3d3e1523cf269c/href</a></iframe><h4>terraform.tfvars</h4><p>This is our variable definition file and is just one way of <a href="https://developer.hashicorp.com/terraform/language/values/variables">setting variables</a> in the root module.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75407cbeacf4153afb6d2c96dde009b9/href">https://medium.com/media/75407cbeacf4153afb6d2c96dde009b9/href</a></iframe><h4>Terraform execution steps</h4><p>Once we have our Terraform files set up, we can execute with the following steps. Note that there is a useful <a href="https://github.com/github/gitignore/blob/main/Terraform.gitignore">Terraform .gitignore template</a> that can be added to your .gitignore file.</p><ol><li>terraform init</li></ol><p>This initializes the working directory containing configuration files — you need to run this first after creating your Terraform files.</p><p>2. terraform plan</p><p>This checks your local changes and proposes an execution plan. For example, if you’ve added a new resource to your <em>main.tf</em> file, you should be able to see it by running this command. It essentially tells you what changes it will make, without executing them.</p><p>3. terraform apply</p><p>Once you’ve reviewed the plan and are happy to proceed with the changes, run this command to approve the proposed plan and apply the changes to your provider account e.g. create the bucket and artifact registry in GCP.</p><p>4. terraform destroy</p><p>This removes all of our resources that are listed in our configuration files — running this will delete the bucket and artifact registry from GCP (and their contents!).</p><h3>3. Orchestrating ETL with Prefect</h3><p>For this project I use data from the <a href="https://data.bus-data.dft.gov.uk/">UK’s Bus Open Data Service</a>. My goal is to compare live bus location data against the bus timetables in order to see which buses are running late in real-time.</p><h4>Extract, transform and load data</h4><p>Typically you would extract and store your raw data first and then transform it (e.g. see the <a href="https://www.databricks.com/glossary/medallion-architecture">Medallion Architecture</a>), but in this smaller scale project I extracted and transformed the data together in single scripts:</p><ol><li>bus_live_locations.py : Pulls the live bus locations from the Open Bus Data <a href="https://gtfs.org/">GTFS</a> feed for the area specified by the bounding box coordinates (Leeds, UK in this case). The data is saved as a parquet file and uploaded to the Google storage bucket.</li><li>bus_timetables.py: Gets the latest bus timetable GTFS data for the relevant region, transforms it and loads the parquet file to the bucket.</li><li>compare_bus_times.py: Gets the bus live location and timetable files from the bucket, transforms them and calculates which buses are currently late to arrive at their next bus stop. The output is a small csv file containing a list of late buses and their associated trip information. The csv is uploaded to the bucket.</li><li>write_to_bq.py: Gets the late bus csv file and appends the data to our BigQuery table.</li></ol><p>The above are relatively straightforward data extraction and transformation steps, and the scripts can be run individually in a local environment, but our aim is to have them run on a set schedule in our cloud environment. This is where our orchestration tool, Prefect, comes in.</p><h4>Prefect overview and setup</h4><p><a href="https://www.prefect.io/">Prefect</a> is an orchestration tool that offers both a locally hosted UI and a cloud based UI (<a href="https://docs.prefect.io/ui/cloud/">Prefect Cloud</a>) to view and manage your workflows.</p><p><strong>Authentication with Prefect Cloud</strong></p><ol><li>Install Prefect on your local environment and also install the Prefect GCP module: pip install -U prefect and pip install &#39;prefect_gcp[cloud_storage]&#39;.</li><li>Sign in to <a href="https://www.prefect.io/cloud/">Prefect Cloud</a>.</li><li>Create a new workspace and <a href="https://docs.prefect.io/ui/cloud-api-keys/">generate an API key</a>.</li><li>Log in to Prefect Cloud from your local environment: prefect cloud login. You will have the option of authenticating with the above API key or with a browser sign-in.</li><li>Now when you run a flow locally (discussed below) you will see the flow execution in your Prefect Cloud UI.</li></ol><p><strong>Flows and tasks</strong></p><p>In Prefect, flows and tasks are the building blocks of your data workflow. A flow is like a function that can take inputs and produce outputs. Tasks are typically single, discrete pieces of work within a flow. For example, our function to load the bus timetable data to the GCS bucket is a single task. Flows can call tasks or other flows (known as subflows).</p><p>The great thing about Prefect is that we can make our existing functions a task or a flow simply by using the task or flow decorator. Below you will see that the load_live_locations_to_gcs function has the @task() decorator and the get_live_bus_locations function is defined as a flow with @flow() — this flow calls the load_live_locations_to_gcs and get_live_gtfs tasks.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/167c4b87dc2278255ce4e2fd5c877b13/href">https://medium.com/media/167c4b87dc2278255ce4e2fd5c877b13/href</a></iframe><p>That’s it! Now when we run this flow we can observe the function’s execution in Prefect’s UI. Prefect automatically adds logging so that you can view your flow run time and final state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*cLv3KsZAGAYWDtFFv1OM4w.png" /><figcaption>Image by author</figcaption></figure><p><strong>Creating a main flow with subflows</strong></p><p>As mentioned, a flow can call another flow (a subflow). I’ve imported the flows from three separate scripts that need to run on the same schedule and are dependent on each other. The parent flow will then call each subflow separately, execute and log it — so we don’t need to explicitly run or schedule the other scripts. The wait_for command can be used to force Prefect to wait for the previous flow run to complete before progressing.</p><p>Here we get the live bus location data, compare it against the scheduled bus times and write the results to BigQuery.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b12768b10336965f88b5d0818b7555b6/href">https://medium.com/media/b12768b10336965f88b5d0818b7555b6/href</a></iframe><p>Here we can see the three separate functions successfully being executed as subflows as part of the master flow run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*9Ndzz5LXNzuq3YWaTb2lYw.png" /><figcaption>Image by author</figcaption></figure><h4>Prefect blocks</h4><p><a href="https://docs.prefect.io/concepts/blocks/">Prefect blocks</a> are a handy feature that allow us to store configuration data to interact with external systems. For example, authenticating with GCP or interacting with a bucket. Blocks can be managed in the UI or through code.</p><p>If a block doesn’t appear in your Prefect UI then you need to register it with the block register command e.g. prefect block register -m prefect_gcp.</p><p>Once you’ve created your block you can use it by simply calling it in your script. Here I load the GCS Bucket block and upload my data to the bucket:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45192c0c1daa8fea27047d5e687b2f45/href">https://medium.com/media/45192c0c1daa8fea27047d5e687b2f45/href</a></iframe><p>The bucket block uses the credentials block for authentication. The same blocks can be used in multiple flows, so instead of having to manage credentials and changes to resource names in our code, we can simply change these details in our blocks.</p><p>Below are the six blocks used. The use of Cloud Run and GitHub blocks are discussed in the next section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*54oj-Bp6bIVua7VHKOPlTg.png" /><figcaption>Image by author</figcaption></figure><h4>Prefect deployment</h4><p>A deployment allows us to trigger and schedule our flows instead of running them manually.</p><p>Build the deployment file with:</p><p>prefect deployment build path-to-script:flow-function-name</p><p>We can use the -n option to specify the name we want to call the deployment. This will create a deployment yaml file. One can provide default parameter values in the deployment file which will then be used by flows associated with this deployment.</p><p>Create the deployment:</p><p>prefect deployment apply name-of-deployment-yaml</p><p>and you will see your new deployment listed in the Prefect UI Deployments section.</p><h4>Prefect scheduling</h4><p>Schedules can be set for any flow deployment. The schedule can be defined with the deployment file or in the UI itself:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*SMg5sCjX3qYb96QVzO2Wng.png" /><figcaption>Image by author</figcaption></figure><h3>Recap so far</h3><p>Up to now we have:</p><ul><li>Created and managed cloud resources with Terraform.</li><li>Changed our data ETL functions into tasks and flows that will be orchestrated with Prefect.</li><li>Created Prefect blocks to manage our authentication and interaction with our Google Cloud resources.</li><li>Created our deployments to manage flow scheduling.</li></ul><p>Next steps:</p><ul><li>Create a Docker image for our script execution environment</li><li>Push this Docker image to Google Artifact Registry</li><li>Update our Prefect deployment for Cloud Run to run our scripts</li><li>Set up the Prefect agent on a VM so that it can run in our cloud environment and not locally</li><li>Create our data visualisation app with Streamlit</li></ul><h3>4. Creating the execution environment with Docker and Cloud Run</h3><p>There are multiple ways to have our scripts running in a cloud environment — we will be using <a href="https://www.docker.com/">Docker</a> and <a href="https://cloud.google.com/run/">Google Cloud Run</a>.</p><p>Our Docker image will contain our base environment for execution and we’ll use Cloud Run to run the container. It’s important to remember the distinction between the <em>flow source</em> (which is the actual flow code e.g. our <em>bus_live_locations.py</em> script) and the <em>flow dependencies</em>. The flow source code is stored in and referenced from a repository (Github in our case). The flow dependencies are captured in our <em>requirements.txt </em>file and are installed in our base environment in the Docker image.</p><p>To put it simply: We store our scripts in a Github repo, we install Python and our required packages in a Docker image and we run them with Cloud Run. We can then update our data pipeline scripts on Github without having to build a new image each time.</p><h4>Build Docker image and push to Artifact Registry</h4><ol><li>Our Dockerfile to install Python and project packages. Note that we don’t copy over any of our scripts — just our Python version and our package requirements.</li></ol><pre>FROM  prefecthq/prefect:2.7.7-python3.9<br>COPY requirements.txt .<br>RUN pip install -r requirements.txt --trusted-host pypi.python.org</pre><p>2. We will store our image in <a href="https://cloud.google.com/artifact-registry/">Artifact Registry</a>. Before I can push images to the registry I need to <a href="https://cloud.google.com/artifact-registry/docs/docker/store-docker-container-images">authenticate</a>:</p><pre>gcloud auth configure-docker europe-west6-docker.pkg.dev</pre><p>3. Build the Docker image with our registry address:</p><pre>docker build -t bus-tracker:v1 .</pre><p>4. <a href="https://cloud.google.com/artifact-registry/docs/docker/store-docker-container-images#add-image">Tag the image</a> with the registry name in the format</p><pre>docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]</pre><p>for example:</p><pre>docker tag bus-tracker:v1 europe-west6-docker.pkg.dev/bus-tracking-376121/bus-tracking/bus-tracker:v1</pre><p>5. Push the image to the registry in the format</p><pre>docker push HOST-NAME/PROJECT-ID/REPOSITORY/IMAGE</pre><p>for example:</p><pre>docker push europe-west6-docker.pkg.dev/bus-tracking-376121/bus-tracking/bus-tracker:v1</pre><p>You’ll then see your Docker image available in Artifact Registry.</p><h4>Setup Cloud Run and GitHub blocks</h4><ol><li>Create a Cloud Run block and provide it with our Docker image location. The fields that you should complete are:</li></ol><ul><li>Type: cloud-run-job</li><li>Image name: This is the full location of your image in artifact registry (same as provided when pushing your image)</li><li>Region: Use the same region as the resource</li><li>Credentials: Use your GCP Credentials block</li></ul><p>2. Create a GitHub block and in the Repository field put the full URL to your project repo. An access token isn’t required if you are using a public repo.</p><h4>Update Prefect deployment</h4><p>We now have to update our deployment to let Prefect know that we will be running our scripts from GitHub and using Cloud Run.</p><pre>prefect deployment build master_flow.py:master_flow -n master__flow -sb github/bus-tracker  -ib cloud-run-job/bus-tracker-cloudrun -o prefect-master-flow-deployment --apply</pre><p>where</p><ul><li>master_flow.py is our flow script</li><li>master_flow is the name of the flow within the script</li><li>-sb is the name of the storage block of our flow code i.e. GitHub block</li><li>-ib is the name of our infrastructure block i.e. Cloud Run block</li><li>-o is the name of the deployment yaml file</li></ul><h3>5. Running Prefect agent on Google Compute Engine</h3><p>The last piece of the puzzle — up until now we have been using our local system to run the Prefect agent. The agent is responsible for checking for flow runs that are ready to run and starting their execution.</p><p>In order for us to have the project fully run in the cloud we need to set up our agent on a Google Compute Engine instance. <a href="https://medium.com/@danilo.drobac/7-a-complete-google-cloud-deployment-of-prefect-2-0-32b8e3c2febe">Danilo Drobac</a> has an excellent summary of this process which I have mostly referenced here.</p><ol><li>Create the compute instance:</li></ol><pre>gcloud compute instances create prefect-agent \<br>--image=ubuntu-2004-focal-v20230104 \<br>--image-project=ubuntu-os-cloud \<br>--machine-type=e2-micro \<br>--zone=europe-west6-a \<br>--service-account=[SERVICE-ACCOUNT-NAME]</pre><p>2. SSH into the VM. You can use the direct SSH connection button in the console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*x-9UuN8XQfSYN2kCCGGgEg.png" /><figcaption>Image by author</figcaption></figure><p>3. Create a new shell script to install the required packages — we’ll need to install Python and Prefect, then log in to Prefect with our Prefect API key:</p><pre>#!/bin/bash<br>sudo apt-get update -y<br>sudo apt-get upgrade -y<br>sudo apt-get install -y \<br>ca-certificates \<br>curl \<br>gnupg \<br>lsb-release \<br>software-properties-common \<br>python3-dateutil<br>sudo ln-s /usr/bin/python3 /usr/bin/python<br>wget https://bootstrap.pypa.io/get-pip.py<br>sudo python3 get-pip.py<br>PATH=&quot;$HOME/.local/bin:$PATH&quot;<br>exportPATH<br>pip3 install prefect prefect-gcp<br>prefect cloud login -k &lt;INSERT_PREFECT_API_KEY&gt;</pre><p>4. Save the file and make it executable:</p><pre>sudo chmod+x install_script.sh</pre><p>5. Execute the script:</p><pre>./install_script.sh</pre><p>6. Start the agent. We’ll use tmux so that the agent continues to run after we kill the SSH connection. Open a tmux session with tmux then start the agent</p><pre>prefect agent start -q default</pre><p>Press ctrl/cmd+b then d to exit the tmux session</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*Z7VfrPnxhatvmScBSSGFxg.jpeg" /><figcaption>Image by author</figcaption></figure><p>7. Confirm that the session is still running with tmux list-sessions</p><p>We now have our Prefect agent running on a Compute Engine instance, checking for any new flow runs to update our data.</p><h3>6. Data visualisation app with Streamlit</h3><p><a href="https://streamlit.io/">Streamlit</a> is an open-source library that lets you rapidly and easily build data web apps in pure Python. You can self-host or use Streamlit’s Cloud service to deploy your app for free. To create the bus map I used the <a href="https://folium.streamlit.app/">Streamlit-Folium</a> library that integrates Streamlit with the mapping library Folium.</p><p>To demonstrate how easy it is — this interactive map app below can be made with 4 lines of code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*L2zrT7Htz_8wNgtrr1Raww.png" /><figcaption>Image by author</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3e8251f4b62eea2a444648670883f80/href">https://medium.com/media/a3e8251f4b62eea2a444648670883f80/href</a></iframe><p>The final project app has more components however, as it connects directly to the latest bus data csv file in the Google bucket. It allows the user to refresh the map data and view delayed buses in (almost) real-time.</p><p>Deploying the app on <a href="https://streamlit.io/cloud">Streamlit Cloud</a> is as simple as pointing the service to your app code on GitHub and clicking deploy. The live app can be <a href="https://latebuses.streamlit.app/">viewed here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*kyHBqvuaBxvz-_3kOh7sDA.png" /><figcaption>Image by author</figcaption></figure><h3>7. Go-live steps</h3><p>To recap, here are the high-level project steps to go from setup to live web app:</p><ol><li>Create your Google Cloud account with necessary permissions and APIs enabled</li><li>Use Terraform to create a storage bucket and artifact registry on Google Cloud service</li><li>Run the <em>Create BigQuery</em> script to create the BigQuery table</li><li>Push the latest Docker image to Artifact Registry</li><li>Ensure that the Prefect Deployment files are up-to-date and that the latest ETL scripts have been pushed to your GitHub repo</li><li>Start the Prefect agent in Compute Instance</li><li>Set the flow run schedule(s) in Prefect Cloud</li><li>Deploy the Streamlit app</li></ol><h3>8. Conclusion</h3><p>This workflow has shown how you can extract value from modern data and infrastructure tools to create end-to-end pipelines that deliver actionable, live insights to users. The tools used, from GCS to Prefect and Streamlit, can be implemented by data professionals with various backgrounds and skill levels. With minimal tweaks, the process can be replicated and applied to any appropriate dataset, making it a beneficial toolset for a wide range of users.</p><p>Hope you enjoyed this walkthrough and it helps you in developing your data pipelines and projects. All relevant project files can be found <a href="https://github.com/RyanEricLamb/data-engineering-bus-tracker">here</a>.</p><h3>9. Resources</h3><ol><li><a href="https://www.bus-data.dft.gov.uk/">Bus Open Data Service</a></li><li><a href="https://github.com/psf/black">Black</a></li><li><a href="https://flake8.pycqa.org/en/latest/">flake8</a></li><li><a href="https://pre-commit.com/">pre-commit</a></li><li><a href="https://www.terraform.io/">Terraform</a></li><li><a href="https://www.prefect.io/">Prefect</a></li><li><a href="https://cloud.google.com/">Google Cloud Services</a></li><li><a href="https://code.visualstudio.com/">Visual Studio Code</a></li><li><a href="https://www.docker.com/">Docker</a></li><li><a href="https://streamlit.io/">Streamlit</a></li><li><a href="https://github.com/RyanEricLamb/data-engineering-bus-tracker">Project source code</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fc6e08b9398" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An end-to-end machine learning project with Python Pandas, Keras, Flask, Docker and Heroku]]></title>
            <link>https://medium.com/data-science/an-end-to-end-machine-learning-project-with-python-pandas-keras-flask-docker-and-heroku-c987018c42c7?source=rss-adce0c00c3e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c987018c42c7</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <dc:creator><![CDATA[Ryan Lamb]]></dc:creator>
            <pubDate>Wed, 09 Dec 2020 13:01:07 GMT</pubDate>
            <atom:updated>2020-12-15T22:13:40.876Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>Predicting sport scores, from data wrangling to model deployment</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*72_8Y9gPXMZspgUz8aYYfA.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/chrisreadingfoto-2723427/">Chris Reading </a>from <a href="https://pixabay.com/photos/team-huddle-togetherness-connection-1928848/">Pixabay</a></figcaption></figure><p>At the time of the Rugby World Cup in 2019 I did a small data science project to try and predict rugby match results, <a href="https://towardsdatascience.com/using-machine-learning-to-predict-sport-scores-a-rugby-world-cup-example-f699fd552673?sk=052e7a0d71f68421a97e1996f3b05727">which I wrote about here</a>. I’ve expanded this into an example end-to-end machine learning project to demonstrate how to deploy a machine learning model as an interactive web app.</p><h4>Goal</h4><p><strong>To provide a high-level overview of the key steps needed in going from raw data to a live deployed machine learning app.</strong></p><p>Once you’ve gone through this — pick a topic that you’re interested in, find some data, get your hands dirty and aim to build your own machine learning app, from data preparation to deployment.</p><h4>The key steps</h4><ol><li>Data wrangling with Pandas &amp; data storage with SQLite</li><li>Machine learning (Neural Network) with Keras</li><li>Web app with Flask (and a bit of CSS &amp; HTML)</li><li>App deployment with Docker and Heroku</li></ol><p>The code for this is available on <a href="https://github.com/RyanEricLamb/rugby-score-prediction">GitHub here</a> and <a href="https://rugby-score-prediction.herokuapp.com/">the live app can be viewed here</a>. (Note that this code isn’t necessarily production level, but meant to show what can be done as a starting point. The live app uses a snapshot of data at a specific point in time).</p><h3><strong>1. Data wrangling &amp; storage with Pandas &amp; SQLite</strong></h3><blockquote><a href="https://www.anaconda.com/state-of-data-science-2020">Data scientists spend about 45% of their time simply getting data ready for model development</a></blockquote><p>Data wrangling includes cleaning, structuring and enriching your data to get it ready for your machine learning model. For example, dealing with blank data or inconsistent formats.</p><p>Unless you’re using an already clean dataset (which seldom happens in the real world), you’d need to perform Exploratory Data Analysis (EDA) to better understand the data profile.</p><p><a href="https://github.com/pandas-profiling/pandas-profiling">Pandas-profiling</a> is an excellent module for EDA. It generates profile reports of DataFrames including quantile and descriptive statistics, correlations, missing values and histograms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wHeJ69sHsWS__QJvZquk5g.jpeg" /><figcaption>Example output from Pandas-profiling for Exploratory Data Analysis (EDA)</figcaption></figure><p>For this project, I needed data from <a href="https://www.espn.co.uk/rugby/">ESPN Scrum</a>. You can find detailed cleaning steps in the data_prep.py file, for example defining the Match Date format with the to_datetime() method. I also enriched the data with the <a href="https://www.world.rugby/tournaments/rankings/mru">World Rugby Rankings</a> and a measure of skill using Microsoft’s <a href="https://www.microsoft.com/en-us/research/project/trueskill-ranking-system/">TrueSkill </a>rating system.</p><p>If you’re looking for datasets to get started, <a href="https://archive.ics.uci.edu/ml/datasets.php#:~:text=%20%20%20Name%20%20%20Default%20Task,%20%20%20%2017%20more%20rows%20">UC Irvine’s Machine Learning repository</a> and <a href="https://www.kaggle.com/datasets">Kaggle</a> are good sources to explore. Alternatively you can get data from scraping using <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">BeautifulSoup</a>.</p><p>I stored my DataFrames as tables in a SQLite database. This is a lightweight database <a href="https://sqlite.org/about.html">and the mostly widely deployed in the world</a>. Plain csv files can of course be used, but this provides an opportunity to learn about SQLite. The code below connects to the match_results database and writes the latest_stats DataFrame to it as a table.</p><pre>conn = sqlite3.connect(&#39;match_results.db&#39;)<br>latest_stats.to_sql(&#39;latest_stats&#39;, conn, if_exists=&#39;replace&#39;)</pre><p>Now we’ve got the data - time to test and train the machine learning model.</p><h3>2. Machine learning — Neural Networks with Keras</h3><h4><strong>Testing</strong></h4><p>Mean Squared Error (MSE) was used as a loss function to minimize. Hyperparameter tuning was performed (Jason Brownlee has a tutorial <a href="https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/">here</a>), and I experimented with both wide (one layer with a lot of neurons) and deep (more layers but fewer neurons per layer) networks until the MSE didn’t improve significantly.</p><p>You want the model to be able to perform well with unseen data. If you overfit your model, you may have a low error on your training data but a high error on your test (unseen) data. When this happens, the model has learned the training data almost too well and cannot generalize to new data. There are <a href="https://www.kdnuggets.com/2019/12/5-techniques-prevent-overfitting-neural-networks.html">various techniques to avoid overfitting</a>.</p><p>Below you can see the MSE decreasing rapidly up until ~25 epochs and then remaining relatively stable for both training and testing data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*au0DrUfU-17QiR4OKwd6ew.jpeg" /><figcaption>MSE against the number of epochs</figcaption></figure><h4>Training</h4><p>The final model had a first layer of 15 neurons and a second layer of 8 neurons, both with the Rectified Linear Activation Unit (ReLU).</p><p>It was compiled with the A<em>dam </em>optimization algorithm, which is an <a href="https://arxiv.org/pdf/1412.6980.pdf">efficient gradient descent algorithm</a>.</p><pre>model = Sequential()<br>model.add(Dense(15, input_dim=5, kernel_initializer=&#39;normal&#39;, activation=&#39;relu&#39;))<br>model.add(Dense(8, kernel_initializer=&#39;normal&#39;, activation=&#39;relu&#39;))<br>model.add(Dense(2, kernel_initializer=&#39;normal&#39;))<br>model.compile(loss=&#39;mse&#39;, optimizer=&#39;adam&#39;)</pre><h4>Saving your model</h4><p>The <a href="https://keras.io/guides/serialization_and_saving/">save()</a> function is used to save the final Keras model. This saves the model architecture and weights, and will allow us to load the model later on in the app to make predictions on new data.</p><h3>3. Web app with Flask <strong>(and a bit of CSS &amp; HTML)</strong></h3><p><a href="https://palletsprojects.com/p/flask/">Flask</a> is a web framework that can be used for developing web applications relatively quickly. You can find a walk through for quick development <a href="https://towardsdatascience.com/deploying-a-keras-deep-learning-model-as-a-web-application-in-p-fc0f2354a7ff">here</a>.</p><p>The code below in the app.py file essentially sets up the home page and serves the <em>index.html</em> to the user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*928yy0rdddR7OUN8-P2h3Q.jpeg" /><figcaption>Home page of the app with the user input form</figcaption></figure><p><a href="https://wtforms.readthedocs.io/en/2.3.x/">WTForms</a> is used for the form so that the user can select their two rugby teams. When the button is clicked, the inputs are sent to the saved ML model via the model_predict function. The resulting model predictions are then sent as input variables to the <em>prediction.html </em>template.</p><pre>@app.route(&quot;/&quot;, methods=[&#39;GET&#39;, &#39;POST&#39;])<br>def home():<br><br>    form = TeamForm(request.form)<br><br>    # Get team names and send to model<br>    if request.method == &#39;POST&#39; and form.validate():<br>        team1 = request.form[&#39;team1&#39;]<br>        team2 = request.form[&#39;team2&#39;]<br>        return render_template(&#39;prediction.html&#39;, win_prob=model_predict(team1,team2)[0],<br>                               score=model_predict(team1, team2)[1])<br><br>    # Send template information to index.html<br>    return render_template(&#39;index.html&#39;, form=form)</pre><p>The<em> prediction.html</em> template does three key things:</p><ol><li>Gets the win probability and score sent from the output of the model to display to the end user (via the <em>win_prob</em> and <em>score</em> variables)</li><li>Provides a button for the user to return to the home page</li><li>References the main.css file to provide some styling</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UtJxDgY-got-uCUJBjTUQg.jpeg" /><figcaption>Prediction results page</figcaption></figure><pre>&lt;!DOCTYPE html&gt;<br>&lt;head&gt;<br>  &lt;title&gt;Prediction&lt;/title&gt;<br>  &lt;link rel=&quot;stylesheet&quot; href=&quot;/static/css/main.css&quot;&gt;<br>&lt;/head&gt;<br><br>&lt;body&gt;<br>  &lt;div class=&quot;container&quot;&gt;<br>    &lt;h1&gt;<br>      &lt;center&gt;Predictions&lt;/center&gt;<br>    &lt;/h1&gt;<br><br>    {% block content %}<br>    &lt;div class=&quot;flash&quot;&gt;{{win_prob}}&lt;/div&gt;<br><br>    &lt;div class=&quot;flash&quot;&gt;{{score}}&lt;/div&gt;<br><br>    &lt;form action=&quot;/button&quot; method = &quot;POST&quot;&gt;<br>    &lt;p&gt;&lt;input type = &quot;submit&quot; value = &quot;Return&quot; /&gt;&lt;/p&gt;<br>    &lt;/form&gt;<br><br>    {% endblock %}<br><br>  &lt;/div&gt;<br>&lt;/body&gt;<br><br>&lt;/html&gt;</pre><p>And that’s it! In summary, you have:</p><ul><li>Two templates, <em>index.html </em>and <em>prediction.html</em> for the home page and prediction results page</li><li>CSS files to provide styling</li><li>Your saved model file to make predictions</li><li>The <em>app.py </em>Flask code to define your pages and their functionality and to run your app</li></ul><p>Now to deploy your app and share it with the world.</p><h3>4. <strong>App deployment with Docker and Heroku</strong></h3><h4><strong>Docker container</strong></h4><p>I won’t go into much detail about Docker but the key thing to know is that it allows you to package your application in a self-sufficient container that can be used across systems i.e. it creates a portable container that makes it easier to deploy your app. You can dig into the details in the official Docker documentation <a href="https://docs.docker.com/get-started/overview/">here</a>.</p><ul><li>Download Docker Desktop <a href="https://www.docker.com/products/docker-desktop">here</a>.</li><li>Create a D<a href="https://docs.docker.com/engine/reference/builder/">ockerfile</a> in your app directory. This contains the instructions for building your Docker image. Best practices for Dockerfiles can be viewed <a href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/">here</a> and <a href="https://pythonspeed.com/articles/dockerizing-python-is-hard/">here</a>. Below you’ll see I’ve included instructions to run the requirements.txt file that contains the app dependencies.</li></ul><pre>FROM python:3.7.3-stretch</pre><pre>RUN mkdir /app<br>WORKDIR /app</pre><pre>#Copy all files<br>COPY . .</pre><pre># Install dependencies<br>RUN pip install --upgrade pip<br>RUN pip install -r requirements.txt</pre><pre># Run<br>CMD [&quot;python&quot;,&quot;./main.py&quot;]</pre><ul><li>Open a terminal and go to the directory containing your Dockerfile and app. Use the build command to build your image: docker build -t rugby_prediction . This builds the Docker image from the Dockerfile.</li><li>You can run the app locally using the runcommand which creates a container: docker run --name rugby -p 5000:5000 -d rugby_prediction. Here, the <em>-p</em> publishes the container’s port to the host and <em>-d</em> makes the container run in the background. Once this is running, you should be able to view your app running in your browser (in Docker Desktop, click on the <em>open in browser button</em>).</li><li>If you experience any errors when running your app, you can use the docker logs command to view the logs and debug.</li></ul><h4><strong>Deployment with Heroku</strong></h4><p>I used <a href="https://www.heroku.com/home">Heroku</a> for hosting because they have a good free tier for non-commercial apps.</p><ul><li>Once you’ve created a Heroku account, create a new app and select Container Registry as the deployment method</li><li>Download the <a href="https://devcenter.heroku.com/articles/heroku-cli">Heroku CLI</a> and log in to your Heroku account:heroku login</li><li>Sign in to the Container Registry: heroku container:login</li><li>Go to your app working directory and push the Docker image: heroku container:push web --app [YOUR APP NAME]</li><li>Deploy your app: heroku container:release web --app [YOUR APP NAME]</li></ul><p>Done! You are ready to share your live machine learning web app.</p><p>Hope this has been helpful to get you started. Feel free to post links to your own ML apps in the comments - would be great to see them.</p><p>If you’d like to collaborate on rugby (or sport) analytics, <a href="https://www.linkedin.com/in/ryanericlamb/">drop me a message on LinkedIn</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hdKYY0IBs-xo6LcxYRlpA.gif" /><figcaption>Final deployed web app</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c987018c42c7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-end-to-end-machine-learning-project-with-python-pandas-keras-flask-docker-and-heroku-c987018c42c7">An end-to-end machine learning project with Python Pandas, Keras, Flask, Docker and Heroku</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using machine learning to predict sport scores — a Rugby World Cup example]]></title>
            <link>https://medium.com/data-science/using-machine-learning-to-predict-sport-scores-a-rugby-world-cup-example-f699fd552673?source=rss-adce0c00c3e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f699fd552673</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[sports]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[rugby]]></category>
            <dc:creator><![CDATA[Ryan Lamb]]></dc:creator>
            <pubDate>Sat, 23 Nov 2019 20:26:21 GMT</pubDate>
            <atom:updated>2020-12-06T16:13:31.541Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using machine learning to predict sport scores — a Rugby World Cup example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J3ZtVjLR2QPWikLIhHWAgw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@jesus?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Thomas Serer</a> on <a href="https://unsplash.com/s/photos/rugby?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>One of the best ways to expand your coding skills is to get stuck in with a hands-on project on a subject you’re interested in. Just before the 2019 Rugby World Cup started in September I decided that I wanted to try and predict the outcomes of the tournament matches.</p><p>Below I’ll walk through the high-level process I went through to try and predict both the winners and the scores of the Rugby World Cup group stages. Hopefully it encourages you to try it yourself on your favourite sport!</p><p>For this project I used <a href="https://www.python.org/">Python</a>, <a href="https://jupyter.org/">Jupyter notebooks</a>, <a href="https://www.jetbrains.com/pycharm/">PyCharm</a> and <a href="https://powerbi.microsoft.com/">Power BI </a>(for quick data viz).</p><h3>Step 1: Get the data</h3><p><a href="https://www.espn.co.uk/rugby/">ESPN Scrum</a> has an extensive database of international rugby matches. I wanted all international rugby matches from 2003 onwards. I chose this date because the World Rugby Rankings were introduced just before the 2003 World Cup (more on this in the Features section).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vRbKm6Kn89ysf3PUSKAfpw.png" /><figcaption>ESPN Scrum match result database</figcaption></figure><p><em>But</em> this data was split over 175 pages - copy/paste was not a pragmatic solution. <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">BeautifulSoup </a>to the rescue. BeautifulSoup is a library for pulling data from HTML and XML files. Below you can see the raw scraped data containing information about a Wales match on 7 September 2019:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y3UQlnzOtXXPK7cFfzkE2w.png" /><figcaption>Result of BeautifulSoup scraping for match results</figcaption></figure><p>I noticed that the ESPN URL had a reference to <em>page,</em> so I iterated through each page and stored the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gybq_YZBseG427cGV7GqQg.png" /><figcaption>DataFrame of match results</figcaption></figure><p>I also added the <a href="https://www.world.rugby/rankings/mru?lang=en">World Rugby Rankings</a> as a feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*db9vP82voHCxI3R78ghqkg.png" /><figcaption>World Rugby Rankings</figcaption></figure><h3>Step 2: Data preparation</h3><p>The scraped data obviously didn’t come in a perfect state for machine learning, so I needed to clean up a few minor things such as removing the <em>‘v’</em> in the Opposition name field and ensuring that a country’s name was the same in both the Team and Opposition fields (I noticed the USA was <em>USA</em> in one and <em>United States of America</em> in the other). Pandas makes this relatively straightforward — see <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html"><em>apply</em></a> and <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.map.html"><em>map</em></a>, for example. I performed a few other preparation steps such as accounting for errors/blanks in the raw data and formatting dates correctly.</p><h3>Step 3: Features</h3><p>The World Rugby Rankings were merged on to the main dataset so that for each team, their ranking at the date closest to the date of each match was applied.</p><p>While I had the World Rugby Rankings, I still wanted to calculate a separate, simpler skill score for each team. I calculated relative skill using a generalisation of the <a href="https://en.wikipedia.org/wiki/Elo_rating_system">Elo rating system</a>. In this system, a player’s rating changes depending on the match result and the relative difference in rating between the two teams. In a simple example, if a team ranked 10th beats a team ranked 2nd, the 10th ranked team will receive more rating points than if they beat a 15th ranked (weaker) team. Similarly, the losing 2nd ranked team will lose more rating points for losing against a weaker opponent than if they lose to say, the top ranked team. The algorithm is therefore self-correcting and over time ‘learns’ the skill of each team.</p><p>The World Rugby Rankings take various match features into account such as match status (more points for a World Cup final), home advantage, and points scored in the match. The calculated skill score ignores all of that and simply looks at match result and relative skill between the two teams at the time of the match.</p><p>Let’s see how the rankings compare at a snapshot in time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/1*_wlqNMPG41sLkwOqeNoojQ.jpeg" /><figcaption>Official World Rugby rankings (23rd September) versus the calculated rankings</figcaption></figure><p>Some interesting results here — Australia ranked higher than expected, and Ireland and Wales lower than the true rankings. I suspect Australia’s high ranking is heavily influenced by their initial high starting point (they were ranked 3rd best team in the world in 2003). Below you can see how the skill scores of some teams have changed over time as the algorithm learns. Notice the steady rise of England, Ireland and Wales from about 2014 onwards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Njt81Lx4Xkui4oJOagaROA.png" /><figcaption>Relative team skill score over time</figcaption></figure><p>In summary, the data for each historical match included: match date, match result, points for, points against, World Rugby rankings for both teams and a relative skill score for both teams.</p><h3>Step 4: Train and test</h3><p>For score prediction I used <a href="https://keras.io/">Keras,</a> a high-level neural networks API. The simplest model in Keras is the Sequential model.</p><p>I performed hyperparameter tuning (check out Jason Brownlee’s great tutorial <a href="https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/">here</a>), and experimented with both wide (one layer with a lot of neurons) and deep (more layers but fewer neurons per layer) networks. I ended up using a first layer of 15 neurons and a second layer of 8 neurons, both with the Rectified Linear Activation Unit (ReLU). We don’t use an activation function for the output layer as we don’t want to transform the output values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cLjQr1nJ1aA8tW-9kyjiQw.jpeg" /><figcaption>Neural network topology</figcaption></figure><p>Below you can see a plot of the loss (Mean Squared Error) over epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*hfXlwUR3B-QudFwu7HCILw.jpeg" /><figcaption>Neural network model loss against number of epochs</figcaption></figure><p>I also tried the fan favourite <a href="https://xgboost.readthedocs.io/en/latest/#">XGBoost,</a> but obtained slightly better results with the neural network, at the expense of a longer training time.</p><h3>Step 5: Predict!</h3><p>For model input I created a dataset of the upcoming World Cup group stage matches and assigned each team their latest skill score and ranking.</p><p>So how did the predictions perform? Much better than I expected (at least for the first 15 games)! Of the 30 games it predicted, it correctly predicted the winner in 27 of them (predicting the Japan versus Ireland upset would have been astounding).</p><p>The model performed particularly well in the first 15 games, being within less than 5 points difference of the actual score difference in 10 of those.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PYbfz-Bg4AKOOO3RWfiSUQ.jpeg" /><figcaption>Model prediction results versus actual results</figcaption></figure><p>Note that I didn’t retrain the model during the group stages, the results above were obtained right at the beginning of the World Cup. It was interesting to note that once most teams had played at least one game, the predictions seemed to worsen. This makes sense as the relative rankings and skill scores would have changed, so one could assume that the predictions would have improved if I updated these features and retrained the model after the first round of games (which I didn’t do, too caught up in watching the actual games!).</p><p>Either way, not bad results for a couple of evenings of work! Machine learning is a fascinating field, do let me know if you decide to try this yourself, there is plenty to improve on and interesting features that could be added. Hope you enjoyed the walkthrough!</p><p><strong>Here are some useful resources to get you started:</strong></p><p><strong>Web scraping with BeautifulSoup</strong></p><ul><li><a href="https://news.codecademy.com/web-scraping-python-beautiful-soup-mlb-stats/">Web Scraping MLB Stats with Python and Beautiful Soup</a></li><li><a href="https://www.freecodecamp.org/news/how-to-scrape-websites-with-python-and-beautifulsoup-5946935d93fe/">How to scrape websites with Python and BeautifulSoup</a></li></ul><p><strong>Manipulating data with Pandas</strong></p><p><a href="https://www.datacamp.com/community/tutorials/pandas-tutorial-dataframe-python">Pandas Tutorial: DataFrames in Python</a></p><p><strong>Data science</strong></p><p>I highly recommend Jake VanderPlas’s excellent Python Data Science handbook (includes example Notebooks)</p><p><a href="https://jakevdp.github.io/PythonDataScienceHandbook/">Python Data Science Handbook</a></p><p><strong>Getting started with Keras and neural networks</strong></p><p><a href="https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/">Your First Deep Learning Project in Python with Keras Step-By-Step - Machine Learning Mastery</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f699fd552673" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/using-machine-learning-to-predict-sport-scores-a-rugby-world-cup-example-f699fd552673">Using machine learning to predict sport scores — a Rugby World Cup example</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>