<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ibinabo Bestmann on Medium]]></title>
        <description><![CDATA[Stories by Ibinabo Bestmann on Medium]]></description>
        <link>https://medium.com/@ibinaldo?source=rss-c9b33aa3086e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Rj8icpwKtGlRhaUMQdaJeg.jpeg</url>
            <title>Stories by Ibinabo Bestmann on Medium</title>
            <link>https://medium.com/@ibinaldo?source=rss-c9b33aa3086e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 09:52:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ibinaldo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Setting Up Apache Airflow on Kubernetes for Local Development]]></title>
            <link>https://blog.dataengineerthings.org/setting-up-apache-airflow-on-kubernetes-for-local-development-f873115fbe93?source=rss-c9b33aa3086e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f873115fbe93</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Ibinabo Bestmann]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 23:40:13 GMT</pubDate>
            <atom:updated>2024-01-26T03:22:06.268Z</atom:updated>
            <content:encoded><![CDATA[<p>In the ever-evolving realm of software development, transformative technologies come in wave after wave, shaping the way we conceive and create applications. From running multiple applications in virtual machines to the more lightweight approach of containerization, these advancements have not only improved efficiency but also simplified the process of application development.</p><p>One technology that sometimes sends shivers down the spines of both new and seasoned software developers alike is <a href="https://kubernetes.io/docs/concepts/overview/"><strong>Kubernetes</strong></a>. One such discipline of software engineering that benefits from having distributed, scalable applications is Data Engineering.</p><h3>What is Data Engineering?</h3><p>What exactly is a data engineer? If your life revolves around requests to “extract” data from Google sheet sources (😔), pulling files from FTP servers (yes, you read that correctly), or even setting up alerting on data freshness in Slack, then you might have an idea of what a data engineer does.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/0*tUAT6NhFUIrqvnvA.jpg" /><figcaption>Taken from <a href="https://www.reddit.com/r/ProgrammerHumor/comments/igerq0/data_engineering/">Reddit</a>. A data engineer’s life purpose.</figcaption></figure><p>At a fundamental level, a data engineer’s job is to build data pipelines to transport data from one location to another. Therefore, a pivotal aspect of Data Engineering is data orchestration — the process of consolidating data from multiple storage locations into a centralized repository for analysis. With the exponential growth in the data landscape, scalable systems are required to make copious amounts of data available to enable data-driven decision making. Various tools exist for accomplishing this task: Apache Airflow, Dagster, Prefect, etc., with <a href="https://airflow.apache.org/docs/apache-airflow/stable/index.html"><strong>Apache Airflow</strong></a><strong> </strong>being the most popular at the time of this writing.</p><p>While most, if not all cloud providers offer Kubernetes as a service, the same cannot be said for Apache Airflow. In this article, I will explore the series of steps necessary to set up a local Apache Airflow installation on Kubernetes. Whether you find yourself tasked with setting up a self-managed Airflow instance for your data engineering team, or you don’t have access to the cloud but would like to experiment with Airflow on Kubernetes and need a local setup for prototyping purposes, this article is tailored for you.</p><h3>Requirements</h3><ul><li><a href="https://go.dev/doc/install"><strong>go</strong></a> 1.16+</li><li><a href="https://docs.docker.com/get-docker/"><strong>Docker</strong></a></li><li><a href="https://kind.sigs.k8s.io/">kind</a></li><li><a href="https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/"><strong>kubectl</strong></a></li><li><a href="https://helm.sh/">helm</a></li></ul><h3>Deploying a Kubernetes Cluster</h3><p>For local development, a typical setup might use <a href="https://minikube.sigs.k8s.io/docs/start/">minikube</a>, a simple single-node cluster. However, since it only runs in a VM on a single node, it doesn’t provide the reliability of a distributed Kubernetes cluster. Most production setups have at least 3 nodes, with one node being the control plane, and the other 2 nodes acting as worker nodes. We don’t have that problem in local development, but we do want to mirror production as closely as possible so we will adopt this setup.</p><p>We can achieve this using <a href="https://kind.sigs.k8s.io/">kind</a> (Kubernetes in Docker), a tool that employs Docker containers to emulate multiple Kubernetes nodes, sidestepping the need for a virtual machine. kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.</p><p>Refer to the <a href="https://kind.sigs.k8s.io/docs/user/quick-start#installation">official documentation</a> for kind installation. I used go install since I am working on Ubuntu 20.04, but pick the installation method that works best for your setup. You’ll also need to have <a href="https://go.dev/doc/install"><strong>go</strong></a> and <a href="https://docs.docker.com/get-docker/"><strong>Docker</strong></a> installed, as well as <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/"><strong>kubectl</strong></a>, the command-line tool for communicating with the Kubernetes API server.</p><p>Let’s check kind is installed by running the command:</p><pre>kind --version</pre><p>We can then create our cluster in a declarative manner using a YAML file.</p><pre># three node (two workers) cluster config (cluster_config.yaml)<br>kind: Cluster<br>apiVersion: kind.x-k8s.io/v1alpha4<br>nodes:<br>- role: control-plane<br>- role: worker<br>  kubeadmConfigPatches:<br>  - |<br>    kind: JoinConfiguration<br>    nodeRegistration:<br>      kubeletExtraArgs:<br>        node-labels: &quot;node=airflow&quot;<br>  extraMounts:<br>  - hostPath: /dags<br>    containerPath: /tmp/data<br>- role: worker<br>  kubeadmConfigPatches:<br>  - |<br>    kind: JoinConfiguration<br>    nodeRegistration:<br>      kubeletExtraArgs:<br>        node-labels: &quot;node=airflow_2&quot;<br>  extraMounts:<br>  - hostPath: /dags<br>    containerPath: /tmp/data</pre><p>This configuration sets up a three-node Kubernetes cluster using kind, with one control-plane node and two worker nodes. Let’s explore this in detail (More information can be found in the official documentation).</p><p>The kind and apiVersion specify we are creating a KIND cluster, as well as the api version. There’s a nodes field that allows us to specify the number of nodes, as well as the ability to specify per-node configurations. let’s look at the node configs. The first node is the control-plane which makes global decisions about the cluster. The other two nodes are worker nodes with <strong>kubeadm</strong> <a href="https://kind.sigs.k8s.io/docs/user/configuration/#kubeadm-config-patches">config patches</a>, spefically the <strong>JoinConfiguration</strong>. You can think of it as <strong>kubeadm </strong>running kubeadm join for you, which is adding your nodes to the cluster, and downloading and configuring necessary components on the joining nodes. I’ve given them labels to allow us to differentiate between the workers if necessary.</p><p>An additional volume mount that maps the host path /dags to the container path /tmp/data has also been configured for both workers. This allows the worker node to access data stored at /dags on the host machine within the /tmp/data path in the container. Feel free to configure as many worker nodes as you’d like. I configured it this way since the DAGs will be stored in a directory called /dags. These are the configurations I went with, but you are welcome to try out different configurations as well.</p><p>Now we can create the cluster</p><pre>kind create cluster --name airflow --config cluster_config.yaml</pre><p>This creates the Kubernetes cluster appropriately named “<strong>airflow</strong>” (you can use a name of your choosing), using the configuration settings from our YAML file.</p><blockquote>You should use a more descriptive name than <strong>airflow</strong>, as this will help minimize confusion with other resource names. In production, you most certainly will not want to go with that name.</blockquote><p>You can confirm it works using both kubectl and kind commands:</p><pre>kind get clusters<br>kubectl get nodes</pre><p>Running the first command should show the cluster you just created, while the second command should show you the control-plane and worker nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*yoKIAM98MuXOwiENCMZVSA.png" /><figcaption>Output of the <strong>kubectl</strong> <strong>get</strong> <strong>nodes</strong> command</figcaption></figure><p>Finally, let’s create a namespace to keep any objects we create organized.</p><pre>kubectl create namespace airflow-workload</pre><h3>Setting up Apache Airflow on your cluster</h3><h4>Generating Kubernetes objects using Helm</h4><p>If we were to create our Airflow application from scratch, we would have to create our own YAML manifest files which would describe the resources we’d need to create our Airflow application. We would need to manually define our webserver, scheduler, database etc. We would also need to make sure they communicate with each other, and a whole lot of other steps.</p><p>This is very time consuming, and may be challenging to get done (See an <a href="https://dev.to/bhavaniravi/how-to-set-up-airflow-on-kubernetes-24i8">example</a> here). What if someone’s already done the difficult job of putting together the resources required for an Airflow application, packaged it up, and made it available for third-party use? This could save us a lot of time and headache, and would be easier to use and maintain.</p><p>It turns out there’s a way to do all this, and it involves using <a href="https://helm.sh/">Helm</a>! Helm is a package manager for Kubernetes that makes it easy to take repeatable applications and services and deploy them to a Kubernetes cluster. It is a tool for managing Kubernetes packages called <em>charts, </em>which<em> </em>is a collection of files that describe a related set of Kubernetes resources. What does this mean? Essentially the challenge of defining all the resources that would go into our Airflow application has already been done for us, and all we’d need is a Helm chart!</p><p>Now where would we find publicly available charts? You guessed it, in a chart repository! The distributed community Helm chart repository we will utilize is <a href="https://artifacthub.io/"><strong>Artifacthub</strong></a>. Different publishers sometimes have their own versions of a particular application. In this tutorial, I will use the <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow">official helm chart</a> provided by Apache Airflow. If you don’t have helm installed, please follow the guide <a href="https://helm.sh/docs/intro/install/">here</a>.</p><p>Let’s begin by adding the Apache Airflow helm chart repository to your local Helm client’s list of known repositories.</p><pre>helm repo add apache-airlow https://airflow.apache.org</pre><p>apache-airflow is the repo label along with its corresponding URL. Why is this important? Once you’ve registered a repository, it becomes relatively straightforward to view available charts in the repository, install a chart or upgrade to a particular release of your choice.</p><p>Now we can install our Airflow helm chart. On the repository page for this chart, they provide an installation command like so</p><pre>helm install my-airflow apache-airflow/airflow --version 1.11.0</pre><p>This installs the latest version of the remote Airflow helm chart directly into your cluster with the release name my-airflow . This might seem very convenient but there a number of caveats to this approach, the biggest of which is the ability to easily replicate any configurations we make. The chart’s configuration could change or it could disappear altogether.</p><p>A better approach is to pull a local copy of the remote chart. Any changes we make will now be repeatable and we’re not at risk if the remote chart suddenly disappears. I shall use the version just before the latest version moving forward. Let’s create a directory called</p><pre>mkdir chart_airflow &amp;&amp; cd chart_airflow<br>helm pull apache-airflow/airflow --version 1.10.0 --untar</pre><p>The above commands create a directory called <strong>chart_airflow</strong>, into which a local copy of the helm chart is pulled. The --untar flag automatically extracts the chart from the tar file after downloading.</p><h4>Configuring our helm chart</h4><p>Below is the typical structure of a chart (adapted from the <a href="https://helm.sh/docs/topics/charts/">official helm documentation</a>)</p><pre>chart_airflow/<br>  Chart.yaml          # A YAML file containing information about the chart<br>  LICENSE             # OPTIONAL: A plain text file containing the license for the chart<br>  README.md           # OPTIONAL: A human-readable README file<br>  values.yaml         # The default configuration values for this chart<br>  values.schema.json  # OPTIONAL: A JSON Schema for imposing a structure on the values.yaml file<br>  charts/             # A directory containing any charts upon which this chart depends.<br>  crds/               # Custom Resource Definitions<br>  templates/          # A directory of templates that, when combined with values,<br>                      # will generate valid Kubernetes manifest files.<br>  templates/NOTES.txt # OPTIONAL: A plain text file containing short usage notes</pre><p>Feel free to explore this structure. Our focus will be on the <strong>values.yaml </strong>file, which contains the default configurations for our application. It&#39;s usually advised to create your own values files, but in this example, I’ll make edits directly here. Let’s explore this <strong>values.yaml</strong> file</p><pre>242 # Airflow executor<br>243 # One of: LocalExecutor, LocalKubernetesExecutor, CeleryExecutor, KubernetesExecutor, CeleryKubernetesExecutor<br>244 executor: &quot;CeleryExecutor&quot;</pre><p><em>Line numbers have been included for convenience to highlight position within the </em><strong><em>values.yaml </em></strong><em>file.</em></p><p>The default Airflow executor in the chart is the CeleryExecutor. It uses Celery, a distributed task queue, to execute tasks. it is designed to be highly scalable, and it can handle a large number of tasks concurrently. However, it does not provide native support for containerization, something the Kubernetes executor handles fairly well. There are many use cases for the other <a href="https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html">executors</a> available but I am going to use the KubernetesExecutor. Feel free to try out the other executor options and see how they affect your performance. Let’s make that change in the executor field to KubernetesExecutor.</p><pre>675 # Airflow scheduler settings<br>676 scheduler:<br>677   #  hostAliases for the scheduler pod<br>678   hostAliases: []<br>.....<br>.....<br>.....<br>744   resources: {}<br>      #  limits:<br>      #   cpu: 100m<br>      #   memory: 128Mi<br>      #  requests:<br>      #   cpu: 100m<br>      #   memory: 128Mi</pre><p>It is usually best practice to set resource requests and limits for your application. requests usually refer to the minimum required resources for containers to run, while limits defining the resource usage limits. We can configure resources for the <strong>webserver </strong>and the <strong>triggerer (</strong>the Airflow triggerer config). Feel free to further define resources where necessary.</p><pre>2121 # Git sync<br>2122 dags:<br>2123   persistance:<br>2124     # Annotations for dags PVC<br>2125     annotations: {}<br>2126     # Enable persistent volume for storing dags<br>2127     enabled: false<br>2128     # Volume size for dags<br>2129     size: 1Gi<br>      .......<br>2138   gitSync:<br>2139      enabled: false<br>2140     <br>2141      # git repo clone url<br>2142      # ssh example: git@github.com:apache/airflow.git<br>2143      # https example: https://github.com/apache/airflow.git<br>2144      repo: https://github.com/apache/airflow.git<br>2145      branch: v2-2-stable<br>2146      depth: 1<br>2147      # the number of consecutive failures allowed before aborting<br>2148      maxFailures: 0<br>2149      # subpath within the repo where dags are located<br>2150      # should be &quot;&quot; if dags are at repo root<br>2151      subPath: &quot;tests/dags&quot;<br>.....<br>.....<br>.....</pre><p><a href="https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html">Airflow</a> suggests a number of ways to manage your DAG files. The ideal option is to host your DAG files locally. In this approach, Airflow will read the DAGs from a PVC. The PVC has to be populated/updated with the required DAGs. Most teams do development around a shared repository, and in production settings, they can either bake DAGs to a docker image, which they deploy in production or mount in DAGs using a git-sync approach which syncs in your DAG files from a repository. I am opting for the git-sync approach, but feel free to try out the other methods described in the official documentation which fits your use case better.</p><p>This section of the values file defines how Airflow will read in your DAG files assuming you enable git-sync. In order to sync from a private repository we’d need to set this to <strong>true</strong>. repo would correspond to the repository where your DAG files live if you were to employ the <strong>gitSync</strong> option. For now, set repo to github url where your DAGs will live, branch to <strong>main </strong>and subPath to <strong>“dags”. </strong>Notice by default gitSync is set to false. Set that to true.</p><p>So how do we set up the git-sync side car? You’d first need to create a Kubernetes secret YAML definition.</p><pre># private ssh key secret<br>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: airflow-ssh-secret<br>  namespace: airflow-workload<br>type: Opaque<br>data:<br>  gitSshKey: |<br>    &lt;base64_encoded_private_key_content&gt;</pre><p>This step assumes you’ve created a private repository, and have created ssh keys. You can then encode your ssh private key:</p><pre>base64 &lt;my-private-ssh-key&gt; -w 0 &gt; temp.txt</pre><p>Then use the string in temp.txt as your gitSshKeyvalue in your Kubernetes secret object YAML definition. Your private key can usually be found at /home/{user}/.ssh/id_rsa</p><p>Next, you’d create the secret object in your cluster:</p><pre>kubectl apply -f airflow-ssh-secret.yaml</pre><p>Alternatively, you could have just run the following command to create the secret object:</p><pre>kubectl create secret generic airflow-ssh-secret --from-file=gitSshKey=/home/{user}/.ssh/id_rsa -n airflow-workload</pre><p>You can confirm your secret was created by running the command:</p><pre>kubectl get secrets -n airflow-workload</pre><p>Finally, let’s address logs:</p><pre>2222  logs:<br>2223    persistence:<br>2224      # Enable persistent volume for storing logs<br>2225      enabled: false<br>2226      # Volume size for logs<br>2227      size: 100Gi<br>2228      # Annotations for the logs PVC<br>2229      annotations: {}<br>2230      # If using a custom storageClass, pass name here<br>2231      storageClassName:<br>2232      ## the name of an existing PVC to use<br>2233      existingClaim:</pre><p>If you’d like to avoid pulling your hair out in frustration when trying to fix errors, you probably want to implement proper logging. Besides configuring the enabled field to <strong>true</strong>, A <em>PersistentVolumeClaim</em> is required for the logs so let’s get that implemented. A <em>PersistentVolumeClaim</em> (PVC) is a request for storage by a user. Creating one is relatively straighforward</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: airflow-logs-pvc<br>  namespace: airflow-workload<br>spec:<br>  storageClassName: local-path<br>  accessModes:<br>    - ReadWriteMany<br>  resources:<br>    requests:<br>      storage: 100Gi<br>  volumeName: airflow-logs-pv  # Add this line to bind to the PV</pre><p>In the above PVC definition, we are specifically binding it to a <em>PersistentVolume</em> (PV). This isn’t necessary in production deployments since Kubernetes automatically bind the PVC to an available PV that satisfies the claim’s requirements. However we are working locally so I have defined a Persistent volume. The <em>PersistentVolume</em> manifest is like so:</p><pre>apiVersion: v1<br>kind: PersistentVolume<br>metadata:<br>  name: airflow-logs-pv<br>  namespace: airflow-workload<br>spec:<br>  storageClassName: local-path<br>  capacity:<br>    storage: 100Gi<br>  accessModes:<br>    - ReadWriteMany<br>  hostPath:<br>    path: &quot;/opt/airflow/logs&quot;</pre><h4>Deploying Airflow</h4><p>Now we’re ready to see if things work. Let’s deploy our Airflow application. First, in our private repository, let’s create a root level <strong>dags/ </strong>directory. Within this directory, let’s create the most basic of dags.</p><pre># hello.py<br>from airflow.decorators import dag<br>from airflow.operators.dummy import DummyOperator<br>from airflow.operators.bash import BashOperator<br>from datetime import datetime, timedelta<br><br>@dag(<br>    &quot;hello_world&quot;,<br>    default_args={<br>        &quot;owner&quot;: &quot;Ibinaldo&quot;,<br>        &quot;depends_on_past&quot;: False,<br>        &quot;email&quot;: &quot;ibi.best@gmail.com&quot;,<br>        &quot;email_on_failure&quot;: True,<br>        &quot;email_on_retry&quot;: False,<br>        &quot;retries&quot;: 2,<br>        &quot;retry_delay&quot;: timedelta(minutes=2),<br>        &quot;start_date&quot;: datetime(2023,7,23)<br>    },<br>    description=&quot;DAG to validate git sync&quot;,<br>    schedule_interval=&quot;@daily&quot;<br><br>)<br><br>def hello_world_dag():<br>    task_start= DummyOperator(task_id=&quot;task_start&quot;)<br>    task_end = DummyOperator(task_id=&quot;task_end&quot;)<br><br>    t1 = BashOperator(<br>        task_id=&quot;print_date&quot;,<br>        bash_command=&quot;date&quot;,<br><br>    )<br><br>    task_start &gt;&gt; t1 &gt;&gt; task_end<br><br>dag = hello_world_dag()</pre><p>Perfect. Its time to deploy our Airflow application. We run the following command from our <strong>chart-airflow </strong>directory:</p><pre>helm install helm-airflow ./airflow --namespace airflow-workload --debug --timeout 10m0s</pre><p>The debug option shows logs while the installation is occurring to prevent a situation where helm gets stuck. Once everything’s installed, you should see a series of instructions on how to access and login to the web server.</p><pre>NOTES:<br>Thank you for installing Apache Airflow 2.6.2!<br><br>Your release is named helm-airflow.<br>You can now access your dashboard(s) by executing the following command(s) and visiting the corresponding port at localhost in your browser:<br><br>Airflow Webserver:     kubectl port-forward svc/helm-airflow-webserver 8080:8080 --namespace airflow-workload<br>Default Webserver (Airflow UI) Login credentials:<br>    username: admin<br>    password: *****<br>Default Postgres connection credentials:<br>    username: postgres<br>    password: ******<br>    port: ****<br><br>You can get Fernet Key value by running the following:<br><br>    echo Fernet Key: $(kubectl get secret --namespace airflow-workload helm-airflow-fernet-key -o jsonpath=&quot;{.data.fernet-key}&quot; | base64 --decode)</pre><p>You can double check your helm installation worked by running this command</p><pre>helm list -n airflow-workload</pre><p>If everything worked as expected, you should see the image below when you go to <em>localhost:8080/</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1kwHt7TWUamZ7RBNzLJYxg.png" /><figcaption>Airflow UI login screen found at <em>localhost:8080/</em></figcaption></figure><p>Log in using the username and password defined in your <strong>values.yaml </strong>file. You should see something familiar:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6e4b2nLIpcBca4TOvyWecA.png" /><figcaption>Airflow UI DAGs list</figcaption></figure><p>Let’s make sure things run as expected. Trigger the DAG:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*25PgxzeqFFYx8QhlaaQVgg.png" /><figcaption>Airflow backfill job running for “<strong>hello_world</strong>” DAG, processing data from 2023–07–23 to present (2024–01–08)</figcaption></figure><p>Not only is our DAG running as expected, we are also able to see the logs which is crucial to having a positive developer experience.</p><p>You can always un-deploy your Airflow chart i.e. remove the resources associated with your helm application from the cluster, by running the command</p><pre>helm uninstall helm-airflow -n airflow-workload</pre><p>Feeling adventurous? You can even delete your kind cluster:</p><pre>kind delete cluster -n airflow</pre><p>Normally deleting everything would be a heart-in-mouth moment. But the declarative nature of deploying our applications means we could recreate everything we just did fairly quickly.</p><h3>Takeaways</h3><p>Now we have successfully deployed our Airflow application locally on a Kubernetes cluster.</p><p>Besides the configurations used in this tutorial, there are many other alternatives. For instance, although I chose to utilize the KubernetesExecutor, there could certainly have been a use case for all the other executors. In addition, while I was able to adopt the git-sync approach to load DAG files, I could have read in the DAGS locally via a persistent volume claim. Having total control over configuration gives us unlimited options in terms of application setup, and there isn’t a right answer.</p><p>The caveat to total control is things can get complicated very quickly, especially if we’re unsure about things. Helm charts don’t always have the greatest documentation or helpful examples, and the Airflow helm chart whilst having some documentation is still far from perfect in terms of what and how each value operates.</p><p>One thing I did notice was missing resource constraints i.e. requests and limits for some of our objects (scheduler, webserver, etc.) made it impossible to inspect logs. I never figured out why that was but perhaps someone out there might have an explanation.</p><p>So what can you do with this setup going forward? There are lots of cool projects you can do with this Airflow setup. You could send yourself a daily email of the 100 most traded stocks for the trading day, along with sentiment information about them (see <a href="https://medium.com/swlh/stock-market-screening-and-analysis-using-web-scraping-neural-networks-and-regression-analysis-f40742dd86e0">this</a> article for an example). Ultimately, it depends on your use case and where your interests lie.</p><p>The <a href="https://github.com/Ibinaldo/airflow">DAG code</a> and <a href="https://github.com/Ibinaldo/airflow_helm_chart">chart</a> for the example in this article can be found on GitHub. I hope you enjoyed the read.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f873115fbe93" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/setting-up-apache-airflow-on-kubernetes-for-local-development-f873115fbe93">Setting Up Apache Airflow on Kubernetes for Local Development</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[COVID-19: The data scientist strikes back]]></title>
            <link>https://medium.com/data-science/covid-19-the-data-scientist-strikes-back-7570c151ea72?source=rss-c9b33aa3086e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7570c151ea72</guid>
            <category><![CDATA[simulation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[modeling]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ibinabo Bestmann]]></dc:creator>
            <pubDate>Mon, 14 Dec 2020 05:42:12 GMT</pubDate>
            <atom:updated>2020-12-14T19:38:35.765Z</atom:updated>
            <content:encoded><![CDATA[<h4>Numerical simulations</h4><h4>How numerical simulations help determine the effects of public health safety measures on slowing the spread of COVID-19</h4><p>The year is 2020. It&#39;s a chilly December morning in San Diego, and you’re out of avocados for the fifth time that month. Ah well, guess it&#39;s time for a grocery store run. You get into the car, and just before turning on the ignition, realize you’ve forgotten your mask on the dresser. With a slight groan, you make a dash for the warm comfort of your home. At least there’s some exercise involved right?</p><p>If you haven’t been living under a rock these past few months, you would be aware of the COVID-19 pandemic that has drastically changed the lives of millions around the globe. In addition to the economic, physical, and mental strains it has put on our lives, it has also become quite a political issue with many protesting against the numerous public health safety measures put in place. Wearing masks, social distancing, and proper, vigilante use of hand sanitizers and hand washing are some of the measures that have sparked debate in terms of their effectiveness in slowing the spread of this contagion.</p><blockquote>With all these in mind it occurred to me;<strong> Could we simulate virus spread in a community in a way that allows us directly see the effects of these public health safety measures?</strong></blockquote><p>To answer that question, let’s have a brief understanding of simulations</p><h3>Numerical simulations</h3><p>A numerical simulation is essentially a computer program that mathematically models a physical system, allowing us to study the behavior of systems that are too complex to provide analytical solutions e.g non-linear systems. Some examples of these kinds of simulations include the finite difference method (FEM), finite element method, and agent-based modeling. My focus in this article will be on agent-based modeling. Agent-based modeling is a type of simulation modeling technique with a large number of real-world applications such as traffic, customer flow management, stock market, operational risk, diffusion of innovation and adoption dynamics, etc.</p><p>In agent-based modeling, a system is modeled as a collection of autonomous decision-making entities (agents). These agents make decisions based on a set of rules and their individual situation and their repetitive interactions are the main features of this type of modeling. Monte Carlo methods are used to introduce randomness. Their ability to exhibit complex behavior patterns can provide valuable information about the dynamics of the real-world system it emulates.</p><h3>Application to epidemiology</h3><p>A particular area where the benefits of this type of modeling are evident is in simulating the spread of infectious disease. In a population of <strong>N</strong> individuals, an individual at any given time (<strong>t</strong> which is measured in days), can be in one of three categories:</p><ol><li>Infected and prone to infect others (<em>infected</em>) (<strong>I(t)</strong>)</li><li>Healthy but susceptible to infection (<em>susceptible</em>) (<strong>S(t)</strong>)</li><li>Recovered or removed (<em>recovered</em>) (<strong>R(t)</strong>)</li></ol><p>In other to simplify calculations, working with the fraction of the total population in each category (i.e divided by <strong>N</strong>) is used instead. Both sets of variables will give us the same information about the progress of the epidemic. A couple of assumptions are made in the modeling process;</p><ul><li>The only way an individual leaves the susceptible group is by becoming infected i.e The susceptible group isn&#39;t influenced by external changes e.g births, immigration</li><li>The rate of change of the number of susceptible individuals (<strong>dS/dt</strong>) is dependent on the original number of susceptible individuals (<strong>S(t)</strong>)</li><li>Under the assumption of a homogenously mixed population, each infected individual has a fixed number of contacts (<strong>b</strong>) per day with individuals in our system, and not all these contacts are susceptible. Therefore on average, each infected individual generates (<strong>b S(t)/N</strong>) new infected individuals per day</li><li>A fixed fraction <strong>k</strong> of the infected group will recover during any given day.</li></ul><p>We can illustrate how three categories of agents, in this case, will change over time in the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*J_dJsAF6xWJqhX_atJjxMg.png" /><figcaption>A pictorial representation of the SIR model showing the change in the number of susceptible, infected, and recovered individuals in a given population over time, <a href="https://commons.wikimedia.org/wiki/File:SIRmodel.png">Public Domain</a></figcaption></figure><p>The above assumptions are the basis of the classical <strong>SIR</strong> models of disease dynamics. This model is very straightforward and relies on the fact that when a susceptible and infected individual are in close proximity, the susceptible individuals contract the disease and become infected. Although reasonably predictive for infectious diseases, it is sometimes criticized for simplifying and unrealistic assumptions. Nevertheless, they are still useful in informing public health safety measures to mitigate and suppress outbreaks. For an in-depth look into the theory please visit [<a href="https://www.maa.org/press/periodicals/loci/joma/the-sir-model-for-spread-of-disease-the-differential-equation-model">this link</a>]</p><h3>A simple python simulation</h3><p>So can we make our own simulation? The very first step involves defining the variables and rules governing our system. Specifically, we will be investigating the effects of following public safety guidelines in reducing the spread of COVID19. The basic guidelines are to</p><ol><li>Social distance (at least 6 feet apart)</li><li>Wear a mask</li><li>Wash your hands/hand sanitizer</li></ol><p>Recently a vaccine was developed by Pfizer-BioNtech to combat the virus, so we will also investigate its effect on reducing the spread.</p><h4>Simulation parameters</h4><p>A square grid is used to represent the community containing the individuals (Agents). In our exam, a 100 x 100 square grid. We also assume:</p><ul><li><strong>infection_radius</strong> (to reflect the 6 feet distance). which a</li><li><strong>recovery_threshold</strong> (to reflect recovering rate)</li><li><strong>death_threshold</strong> (to reflect death rate)</li></ul><p>The idea is simple: At each iteration, the locations of the individuals in our system are shifted by a random amount <strong>shift</strong>. Then we check to see if an individual is within the radius of an infected individual (<strong>infection_radius)</strong>. These individuals are then moved into the infected population. At every iteration, we can mirror (<strong>b S(t)/N</strong>) newly infected individuals per day, as well as <strong>k</strong> individual recoveries using Monte-Carlo methods. Samples the size of the infected population are drawn from a uniform distribution, and values that are higher and lower than our specified thresholds for recovery and deaths (<strong>recovery_threshold &amp; death_threshold) </strong>are moved into the recovered and removed populations respectively. A uniform distribution is used here because all infected individuals are assumed to be equally likely to recover or die (a simplistic assumption I know). We also assume the recovered and dead population can not be infected again.</p><h4>Public safety measures</h4><p>We will treat public safety measures as entities we can introduce into our system. Specifically, we’ll consider the following variables:</p><ul><li><strong>Mask</strong></li><li><strong>SD</strong></li><li><strong>Hygiene</strong></li><li><strong>Vacc</strong></li></ul><p><strong>Mask: </strong>If this entity is introduced, <strong>Mask</strong> is the factor by which the <strong>infection_radius</strong> drops i.e <strong>infection_radius</strong> X <strong>Mask </strong>if masks are used.</p><p><strong>SD:</strong> This corresponds to the social distancing protocol and is the factor by which the shifts in individual locations at each iteration drops i.e <strong>shift/SD</strong></p><p><strong>Hygiene: </strong>The use of cleaning agents will reduce the transmissibility of the contagion. We can simulate this effect by randomly assigning an infectivity score from 1 to 0 to each infected individual, and only infected individuals with scores greater than this value can infect. It is an alternate way of saying a fraction of the susceptible population will not be infected.</p><p><strong>Vacc: </strong>This corresponds to the use of a vaccine. If this variable is set in play, the assumption is the death rate will drop by a factor of <strong>Vacc i.e. death_threshold/Vacc.</strong></p><p>Note: These values are relative values and not absolute. It’s usually best to run models relative to each other to quantify how changes in one parameter affect a system. You are free to use any values you like!</p><h4><strong>Implementation</strong></h4><p>Let&#39;s first load in our libraries</p><pre>%matplotlib notebook<br>import matplotlib.pyplot as plt<br>import numpy as np<br>from matplotlib import style<br>style.use(&#39;dark_background&#39;)<br>from matplotlib import animation, rc<br>from IPython.display import HTML</pre><pre>rc(&#39;animation&#39;, html=&#39;html5&#39;)</pre><p>We utilize the %matplotlib notebook command to allow our jupyter notebook to Livestream our plots dynamically, as opposed to statically plotting just a single figure. This is necessary in order to actually see our simulation. Now we define our parameters:</p><pre>## Colors for the different states of the population<br>inf_col = &#39;red&#39;  # Infected<br>rem_col = &#39;lime&#39; # Removed/reoovered<br>ded_col = &#39;blue&#39; # Dead<br>sus_col = &#39;grey&#39; # Susceptable</pre><pre># Simulation parameters<br>infection_radius = 0.6 # Radius of infection (6 feet for covid-19)<br>recovery_threshold =  (1-0.009975)# Assuming 99.75% recovery rate<br>death_threshold = 0.00021  # 2.1% death rate<br>stage_dimension = 100 (dimension of square grid)</pre><pre># Initial population conditions<br>Pop = 800 # pop. size (per 100,000 of the population)     <br>init = [Pop, # Total population<br>        np.round(Pop*0.01), # infected (per 100 of the population)<br>        0, # recovered population<br>        0] # dead population</pre><pre># Public safety variables<br>Mask_on=False<br>Mask=0.7</pre><pre>Hygiene_on = False<br>Hygiene = 0.9</pre><pre>SD_on=False<br>SD= 10</pre><pre>Vacc_on = False<br>Vacc = 2</pre><p>We first define our colors to differentiate individuals/agents in the different population categories during the simulation. We then define the simulation parameters. The values I used were modeled based on values obtained at <a href="https://www.coronatracker.com/country/united-states/">coronatracker</a> at the time this article was written. We begin with a population of 800,0000 (scaled down to 800 for model purposes) and our assumption is 1% of the population has covid-19. The public safety variables are modeled to mimic strict policies i.e mask on, constant sanitization and disinfection, social distancing, and the use of vaccines.</p><p>Next, we define a class to model the system’s behavior: We begin by calculating how many individuals/agents belong in our susceptible population and then assigning colors to each point (individual) on our grid based on their population category. We then define the upper and lower bounds of the grid and generate random points within that grid from a uniform distribution. We then stream the data in a continuous loop and create the animation using the <em>FuncAnimation </em>method of the animation class. This method accepts a <em>figure</em> object (self.fig) upon which the plots are made, an initial function <em>init_func (self.setup) </em>which displays the locations of the individuals/agents before perturbation, and <em>self.update </em>which shows the population after perturbation and adjustment of population.</p><p>Now the most important bit, our data stream! So how does it look? well at each iteration we need to apply our random shifts to the population. If we have our social distancing flag on, this shift is reduced by a factor of <strong>SD. </strong>Then we isolate the infected population and calculate their distances to the susceptible population. We assign susceptible individuals whose distance from an infected individual is less than the <strong>infection_radius. </strong>Note that if the mask flag is on, this reduces the infection_radius by a factor <strong>Mask </strong>since the assumption is mask limits the spread of the disease. Next, we determine which infected individuals meet the criteria for being moved to either the recovered or removed population. If we apply the hygiene and vaccine flags, then only a certain fraction of infected individuals can infect to mimic fewer infections from using hand sanitizers and handwashing, and the death rate is decreased by a factor of <strong>Vacc </strong>to mimic the supposed drop in deaths due to vaccinations. To sum it all up, here is our simulation class defined below:</p><pre>class Sim():<br>    &#39;&#39;&#39;<br>    The class Sim creates a simulation of the spread of COVID19 using user defined population and public health parameters.<br>    &#39;&#39;&#39;</pre><pre>    def __init__(self,init,Mask_on,Hygiene_on,SD_on,Vacc_on,gif_name):<br>        self.N = init[0]<br>        suc = self.N-init[1]-(init[2]+init[3])<br>        self.cols = np.repeat([sus_col,inf_col,rem_col,ded_col],<br>                              [suc,init[1],init[2],init[3]])<br>        self.low,self.high = -stage_dimension/2,stage_dimension/2<br>        self.pos = np.random.uniform(self.low,self.high,size=(2,self.N))<br>        <br>        self.stream = self.data_stream()<br>        #print(self.removed) <br>        self.fig,self.ax = plt.subplots();<br>        <br>        self.ani = animation.FuncAnimation(self.fig, self.update, interval=200, <br>                                           init_func=self.setup, blit=True, save_count=1) <br>        <br>        ## uncomment to save mp4 or gif. Increase save_counts for longer simulation file<br>        #self.ani.save(&#39;simulation.mp4&#39;)<br>        #self.ani.save(gif_name+&#39;.gif&#39;, writer=&#39;imagemagick&#39;, fps=60)<br>        HTML(self.ani.to_html5_video())</pre><pre>def setup(self):<br>        &#39;&#39;&#39;<br>        Function constructs frame based on the initial population state<br>        &#39;&#39;&#39;<br>        p,c= next(self.stream)<br>        self.scat = self.ax.scatter(x=p[0,:],<br>                                    y=p[1,:],<br>                                    c=c.T)<br>        return self.scat</pre><pre>def arg_within_radius(self,inf,susceptible,r):<br>        &#39;&#39;&#39;<br>        This function calculates distances between an infected and susceptible individuals <br>        &#39;&#39;&#39;<br>        if(Mask_on):<br>            r=r*Mask       <br>        dist = np.sqrt(((inf-susceptible)**2).sum(axis=1))      <br>        return np.argwhere(dist&lt;r).ravel()</pre><pre>def data_stream(self):<br>        while True:<br>            removed = 0<br>            shift = 0.5*np.random.normal(0,1,size=(2,self.N))<br>            if(SD_on):<br>                shift=shift/SD<br>            self.pos[0:2,:]+=shift</pre><pre>inf = np.argwhere(self.cols == inf_col).ravel()<br>            inf_people = self.pos[:,inf].T<br>            <br>            suc = np.argwhere(self.cols == sus_col).ravel()<br>            suc_people = self.pos[:,suc].T</pre><pre>for i in inf_people:<br>                <br>                if(Hygiene_on):<br>                    risk = np.random.rand(1)<br>                    if(risk &gt; Hygiene):<br>                        infect = self.arg_within_radius(i,suc_people,infection_radius)<br>                    else:<br>                        pass<br>                else:<br>                    infect = self.arg_within_radius(i,suc_people,infection_radius)<br>                <br>                infect_index = suc[infect]<br>                self.cols[infect_index] = inf_col<br>            <br>            # adjust out of bound individuals<br>            x1=np.where(self.pos[0:2,:]&lt;self.low)<br>            x2=np.where(self.pos[0:2,:]&gt;self.high)<br>            self.pos[0:2,:][x1],self.pos[0:2,:][x2] = self.low,self.high<br>            <br>            r = np.random.uniform(0,1, size = inf.size)<br>            rec = np.argwhere(r&gt;recovery_threshold).T<br>            recovered_idx = inf[rec]<br>            self.cols[recovered_idx] = rem_col</pre><pre>d = np.random.uniform(0,1, size = (inf.size))<br>            if (Vacc_on):<br>                dead = np.argwhere(d&lt;(death_threshold/Vacc)).T<br>            else:<br>                dead = np.argwhere(d&lt;death_threshold).T<br>            dead_idx = inf[dead]<br>            self.cols[dead_idx] = ded_col<br>            ded=np.argwhere(self.cols == ded_col).ravel()<br>            ded_people = self.pos[:,dead].T<br>            for dd in ded_people:<br>                removed+=1<br>               #self.removed+=removed<br>                #print(&quot;An individual died&quot;)</pre><pre>yield self.pos,self.cols</pre><pre>def update(self,i):<br>        &#39;&#39;&#39;<br>        Function used in the animation function to plot population after pertubation and population assignment<br>        &#39;&#39;&#39;<br>        data,c= next(self.stream)<br>        self.scat.set_offsets(data[0:2,:].T)<br>        self.scat.set_sizes(np.zeros(self.N)+10)<br>        self.scat.set_color(c.T)<br>        return self.scat</pre><p>Now with everything set, let’s run two simulations; one where we follow public health safety guidelines, and one where we don’t. Let’s begin with not following the public health guidelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*BxwyMzRz0ifz7JtylV2B-w.gif" /><figcaption>Image by the author. A GIF snippet displaying our simulation into the spread of COVID19 in a community. <strong>No public health safety measures were implemented in this case</strong>. Grey dots = susceptible, red dots = infected, green dots = recovered, blue dots = removed (dead)</figcaption></figure><p>Now, how about all measures in place?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*AIDvW-P74b8x_Wyh9Fii9Q.gif" /><figcaption>Image by the author. A GIF snippet displaying our simulation into the spread of COVID19 in a community. <strong>All public health safety measures were implemented in this case</strong>. Grey dots = susceptible, red dots = infected, green dots = recovered, blue dots = removed (dead)</figcaption></figure><p>All though these are snippets from the simulation before reaching steady-state, we notice that there’s only a single death in our scenario where all public health safety measures were followed, as opposed to two deaths in the initial scenario lacking any safety measures. In fact, if you run both simulations till steady-state, you’ll find the second scenario with safety measures had fewer deaths and a slower rate of spread of the contagion than the first scenario. If you don’t believe me, I invite you to run the code yourself.</p><h3>Insights</h3><p>As you would have expected, following the safety guidelines appears to have slowed the spread of the contagion and resulted in fewer deaths. But there are some caveats. This model is inherently simplistic; we not only used very simple assumptions and mathematical formulas, we assumed conditions were fixed for all individuals but in reality, there are other factors that must be taken into account; underlying conditions, length of exposure, etc. If we wanted a more complex model we would need to account for those. On the other hand, by keeping the conditions as simple as possible we can get some insights on a broad scale and by running different simulations where some parameters are varied, gain some understanding as to which variables have more of an impact. Overall simulation appears to be a very useful tool in making decisions. So with all that being said, the question of the day is:</p><blockquote><strong>Mask on or Mask off?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fVZS1Fu8TwDEM2y1uqTI5Q.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@tiffanytertipes?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Tiffany Tertipes</a> on <a href="https://unsplash.com/s/photos/mask-darth-vader?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>This project was mainly inspired by these resources (<a href="https://www.coursera.org/projects/simulating-viral-pandemic-in-python">here</a>, and <a href="https://www.coursera.org/learn/modeling-simulation-natural-processes">here</a>). The code will be provided <a href="https://github.com/Ibinaldo?tab=repositories">here</a> in the not so distant future, and my Linkedin profile can be found <a href="https://www.linkedin.com/in/ibinabo-bestmann/">here</a>. Thank you for reading! (and hopefully clapping!)</p><p>Ibinabo Bestmann</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7570c151ea72" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/covid-19-the-data-scientist-strikes-back-7570c151ea72">COVID-19: The data scientist strikes back</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stock Market Screening and Analysis: Using Web Scraping, Neural Networks, and Regression Analysis…]]></title>
            <link>https://medium.com/swlh/stock-market-screening-and-analysis-using-web-scraping-neural-networks-and-regression-analysis-f40742dd86e0?source=rss-c9b33aa3086e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f40742dd86e0</guid>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ibinabo Bestmann]]></dc:creator>
            <pubDate>Sun, 23 Aug 2020 22:24:24 GMT</pubDate>
            <atom:updated>2020-08-24T17:53:29.539Z</atom:updated>
            <content:encoded><![CDATA[<h4>Trading with AI</h4><h3>Stock Market Screening and Analysis: Using Web Scraping, Neural Networks, and Regression Analysis in Investing</h3><h4>Increasing portfolio returns using python</h4><p>Stock markets tend to react very quickly to a variety of factors such as news, earnings reports, etc. While it may be prudent to develop trading strategies based on fundamental data, the rapid changes in the stock market are incredibly hard to predict and may not conform to the goals of more short term traders. This study aims to use data science as a means to both identify high potential stocks, as well as attempt to forecast future prices/price movement in an attempt to maximize an investor’s chances of success.</p><p>In the first half of this analysis, I will introduce a strategy to search for stocks that involves identifying the highest-ranked stocks based on trading volume during the trading day. I will also include information based on twitter and sentiment analysis in order to provide an idea of which stocks have the maximum probability of going up in the near future. The next half of the project will attempt to apply forecasting techniques to our chosen stock(s). I will apply deep learning via a <strong>Long short term memory (LSTM) neural network</strong>, which is a form of a recurrent neural network (RNN) to predict close prices. Finally, I will also demonstrate how <strong>simple linear regression</strong> could aid in forecasting.</p><h3>Part 1: Stock screening</h3><p>Let’s begin by web-scraping data on the most active stocks in a given time period, in this case, one day. Higher trading volume is more likely to result in bigger price volatility which could potentially result in larger gains. The main python packages to help us with this task are the <em>yahoo_fin</em>, <em>alpha_vantage,</em> and <em>pandas</em> libraries.</p><pre># Import relevant packages<br>import yahoo_fin.stock_info as ya<br>from alpha_vantage.timeseries import TimeSeries<br>from alpha_vantage.techindicators import TechIndicators<br>from alpha_vantage.sectorperformance import SectorPerformances<br>import pandas as pd<br>import pandas_datareader as web<br>import matplotlib.pyplot as plt<br>from bs4 import BeautifulSoup<br>import requests <br>import numpy as np</pre><pre># Get the 100 most traded stocks for the trading day<br>movers = ya.get_day_most_active()<br>movers.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*O3Kt_1RB-NgWt_XPTCOKZQ.png" /><figcaption>Stocks with the largest trading volume for the trading day</figcaption></figure><p>The <em>yahoo_fin </em>package<em> </em>is able to provide the top 100 stocks with the largest trading volume. We are interested in stocks with a positive change in price so let’s filter based on that.</p><pre>movers = movers[movers[&#39;% Change&#39;] &gt;= 0]<br>movers.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*TyScax4EQO1ZsuIrmdFIIA.png" /><figcaption>Stocks with the largest trading volume for the trading day, filtered by the price change</figcaption></figure><p>Excellent! We have successfully scraped the data using the yahoo_fin python module. it is often a good idea to see if those stocks are also generating attention, and what kind of attention it is to avoid getting into false rallies. We will scrap some sentiment data courtesy of <a href="http://www.sentdex.com/financial-analysis/">sentdex</a>. Sometimes sentiments may lag due to source e.g News article published an hour after the event, so we will also utilize <a href="https://www.tradefollowers.com/strength/twitter_strongest.jsp?tf=1d">tradefollowers</a> for their twitter sentiment data. We will process both lists independently and combine them. For both the sentdex and tradefollowers data we use a 30 day time period. Using a single day might be great for day trading but increases the probability of jumping on false rallies. NOTE: Sentdex only has stocks that belong to the S&amp;P 500. Using the <em>BeautifulSoup </em>library, this process is made fairly simple.</p><pre>res = requests.get(&#39;<a href="http://www.sentdex.com/financial-analysis/?tf=30d&#39;">http://www.sentdex.com/financial-analysis/?tf=30d&#39;</a>)<br>soup = BeautifulSoup(res.text)<br>table = soup.find_all(&#39;tr&#39;)</pre><pre># Initialize empty lists to store stock symbol, sentiment and mentions</pre><pre>stock = []<br>sentiment = []<br>mentions = []<br>sentiment_trend = []</pre><pre># Use try and except blocks to mitigate missing data<br>for ticker in table:<br>    ticker_info = ticker.find_all(&#39;td&#39;)<br>    <br>    try:<br>        stock.append(ticker_info[0].get_text())<br>    except:<br>        stock.append(None)<br>    try:<br>        sentiment.append(ticker_info[3].get_text())<br>    except:<br>        sentiment.append(None)<br>    try:<br>        mentions.append(ticker_info[2].get_text())<br>    except:<br>        mentions.append(None)<br>    try:<br>        if (ticker_info[4].find(&#39;span&#39;,{&quot;class&quot;:&quot;glyphicon  glyphicon-chevron-up&quot;})):<br>            sentiment_trend.append(&#39;up&#39;)<br>        else:<br>            sentiment_trend.append(&#39;down&#39;)<br>    except:<br>        sentiment_trend.append(None)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/1*CD6tT8rJTC2CB6tIE6L_HQ.png" /><figcaption>Sentiment data obtained from <a href="http://www.sentdex.com/financial-analysis/">sentdex</a></figcaption></figure><p>We then combine these results with our previous results about the most traded stocks with positive price changes on a given day. This done using a left join of this data frame with the original <em>movers</em> data frame</p><pre>top_stocks = movers.merge(company_info, on=&#39;Symbol&#39;, how=&#39;left&#39;)<br>top_stocks.drop([&#39;Market Cap&#39;,&#39;PE Ratio (TTM)&#39;], axis=1, inplace=True)<br>top_stocks</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*eBJSH6Q14H7EPtaU-gVrJw.png" /><figcaption>A merged data frame containing the biggest movers and their sentiment information (if available)</figcaption></figure><p>The <em>movers</em> data frame had a total of 45 stocks but for brevity only 10 are shown here. A couple of stocks pop up with both very good sentiments and an upwards trend in favourability. ZNGA, TWTR, and AES (not shown) for instance stood out as potentially good picks. Note, the mentions here refer to the number of times the stock was referenced according to the internal metrics used by <a href="http://localhost:8888/notebooks/sentdex.com">sentdex</a>. Let’s attempt supplementing this information with some data based on twitter. We get stocks that showed the strongest twitter sentiments within a time period of 1 month and were considered bullish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/1*WJKSi59Pt6dDHH7j2PH24Q.png" /><figcaption>Twitter sentiment data for stocks classified as bullish. Obtained from <a href="https://www.tradefollowers.com/strength/twitter_strongest.jsp?tf=1d">tradefollowers</a></figcaption></figure><p>Twit_Bull_score refers to the internal scoring used at <a href="http://localhost:8888/notebooks/tradefollowers.com">tradefollowers</a> to rank stocks that are considered bullish, based on twitter sentiments, and can range from 1 to as high as 10,000 or greater. With the twitter sentiments obtained, I’ll combine it with our sentiment data to get an overall idea of the stocks and their sentiments</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*gxgNStqUYND51Gs3eFslIw.png" /><figcaption>A merged data frame containing the biggest movers and their sentiment information (if available). Only the first 10 stocks shown</figcaption></figure><p>For completeness, stocks classified as having large momentum and their sentiment data will also be included in this analysis. These were scraped and merged with the <em>Final_list</em> data frame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*bTGOxQ0ZPTWRePhDtKIhfw.png" /><figcaption>Final merged data frame of the largest moving stocks and their associated sentiments (if present)</figcaption></figure><p>Our list now contains even more information to help us with our trades. Stocks that it suggests might generate positive returns include TSLA, ZNGA, and TWTR as their sentiments are positive, and they have relatively decent twitter sentiment scores. As an added measure, we can also obtain information on the sectors to see how they’ve performed. Again, we will use a one month time period for comparison. The aforementioned stocks belong to the Technology and consumer staples sectors.</p><pre># Checking sector performances</pre><pre>sp = SectorPerformances(key=&#39;0E66O7ZP6W7A1LC9&#39;, output_format=&#39;pandas&#39;)<br>plt.figure(figsize=(8,8))<br>data, meta_data = sp.get_sector()<br>print(meta_data)<br>data[&#39;Rank D: Month Performance&#39;].plot(kind=&#39;bar&#39;)<br>plt.title(&#39;One Month Performance (%) per Sector&#39;)<br>plt.tight_layout()<br>plt.grid()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*JN7lEYE0ZWWbgpN86CoIKQ.png" /><figcaption>One-month sector performance (July till August) obtained using the alpha_vantage python package</figcaption></figure><p>The industrials sector appears to be the best performing in this time period. Consumer staples sector appears to be doing better than the information technology sector, but overall they are up which bodes well for potential investors. Of course with stocks, a combination of news, technical and fundamental analysis, as well as knowledge of the given product/sector, is needed to effectively invest but this system is designed to find stocks that are more likely to perform.</p><p>To validate this notion, let’s construct a mock portfolio using TSLA, ZNGA, and TWTR as our chosen stocks and observe their cumulative returns. This particular analysis began on the 13th of August, 2020 so that will be the starting date of the investments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*cO5rB3IjePJmiBJwmqLPHg.png" /><figcaption>7-day cumulative returns for Tesla, Twitter, and Zynga using close prices from yahoo finance.</figcaption></figure><p>Overall all three investments would have netted a positive net return over the 7-day period. The tesla stock especially experienced a massive jump around the 14th and 19th respectively, while twitter shows a continued upward trend.</p><blockquote>Please note that this analysis is only a guide to find potentially positive return generating stocks and does not guarantee positive returns. It is still up to the investor to do the research.</blockquote><h3>Part 2: Forecasting close price using an LSTM</h3><p>In this section, I will attempt to apply deep learning to a stock of our choosing to forecast future prices. At the time this project was conceived (early mid-late July 2020), the AMD stock was selected as it experienced really high gains around this time. First I obtain stock data for our chosen stock. Data from 2014 data up till August of 2020 was obtained for our analysis. Our data will be obtained from yahoo using the <em>pandas_webreader</em> package</p><pre># Downloading stock data using the pandas_datareader library</pre><pre>from datetime import datetime</pre><pre>stock_dt = web.DataReader(&#39;AMD&#39;,&#39;yahoo&#39;,start,end)<br>stock_dt.reset_index(inplace=True)<br>stock_dt.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*zR322NqcbUvQnrdIkbrFoQ.png" /><figcaption>AMD stock price information</figcaption></figure><p>Next, some technical indicators were obtained for the data using the <em>alpha_vantage </em>python package<em>.</em></p><pre># How to obtain technical indicators for a stock of choice<br># RSI<br>t_rsi = TechIndicators(key=&#39;0E66O7ZP6W7A1LC9&#39;,output_format=&#39;pandas&#39;)<br>data_rsi, meta_data_rsi = t_rsi.get_rsi(symbol=&#39;AMD&#39;, interval=&#39;daily&#39;,time_period = 9,<br>                          series_type=&#39;open&#39;)</pre><pre># Bollinger bands<br>t_bbands = TechIndicators(key=&#39;0E66O7ZP6W7A1LC9&#39;,output_format=&#39;pandas&#39;)<br>data_bbands, meta_data_bb = t_bbands.get_bbands(symbol=&#39;AMD&#39;, interval=&#39;daily&#39;, series_type=&#39;open&#39;, time_period=9)</pre><p>This can easily be functionalized depending on the task. To learn more about technical indicators and how they are useful in stock analysis, I welcome you to explore <a href="https://www.investopedia.com/">investopedia</a>’s articles on different technical indicators. Differential data was also included in the list of features for predicting stock prices. In this study, differential data refers to the difference between price information on 2 consecutive days (price at time <strong>t</strong> and price at time <strong>t-1</strong>).</p><h4>Feature engineering</h4><p>Let’s visualize how the generated features relate to each other using a heatmap of the correlation matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*Lsoy15nQmQbpS3lH2UVtoQ.png" /><figcaption>Correlation matrix of stock features</figcaption></figure><p>The closing price has very strong correlations with some of the other price information such as opening price, highs, and lows. On the other hand, the differential prices aren&#39;t as correlated. We want to limit the amount of colinearity in our system before running any machine learning routine. So feature selection is a must.</p><p>I utilize two means of feature selection in this section. Random forests and mutual information gain. Random forests are very popular due to their relatively good accuracy, robustness as well as simplicity in terms of utilization. They can directly measure the impact of each feature on the accuracy of the model and in essence, give them a rank. Information gain, on the other hand, calculates the reduction in entropy from transforming a dataset in some way. Mutual information gain essentially evaluates the gain of each variable in the context of the target variable.</p><h4>Random forest regressor</h4><p>Let’s make a temporary training and testing data sets and run the regressor on it.</p><pre># Feature selection using a Random forest regressor<br>X_train, X_test,y_train, y_test = train_test_split(X,y,test_size=0.2,random_state=0)</pre><pre>feat = SelectFromModel(RandomForestRegressor(n_estimators=100,random_state=0,n_jobs=-1))<br>feat.fit(X_train,y_train)</pre><pre>X_train.columns[feat.get_support()]</pre><p>The regressor essentially selected the features that displayed a good correlation with the close price which is the target variable. However, although it selected the most important we would like information on the information gain from each variable. An issue with using random forests is it tends to diminish the importance of other correlated variables and may lead to incorrect interpretation. However, it does help reduce overfitting. Using the regressor, only three features were seen as being useful for prediction; High, Low, and Open prices.</p><h4>Mutual information gain</h4><p>Now quantitatively see how each feature contributes to the prediction</p><pre>from sklearn.feature_selection import mutual_info_regression, SelectKBest</pre><pre>mi = mutual_info_regression(X_train,y_train)<br>mi = pd.Series(mi)<br>mi.index = X_train.columns<br>mi.sort_values(ascending=False,inplace=True)</pre><pre>mi</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*GJsKxwRzQutoihZkUlM-WA.png" /><figcaption>Mutual information gain from each feature towards predicting close price</figcaption></figure><p>The results validate the results using the random forest regressor, but it appears some of the other variables also contribute a decent amount of information. In terms of selection, only features with contributions greater than 2 are selected, resulting in 8 features</p><h4>Data preparation for the LSTM</h4><p>In order to construct a Long short term memory neural network (LSTM), we need to understand its structure. Below is the design of a typical LSTM unit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*GMmntL9w7qBG6zm1C2IMHQ.png" /><figcaption>The architecture of a typical LSTM unit. Data source: <a href="https://commons.wikimedia.org/wiki/File:Long_Short-Term_Memory.svg">Wikimedia</a></figcaption></figure><p>As mentioned earlier, <strong>LSTM’s</strong> are a special type of recurrent neural network (<strong>RNN</strong>). Recurrent neural networks (<strong>RNN</strong>) are a special type of neural network in which the output of a layer is fed back to the input layer multiple times in order to learn from the past data. Basically, the neural network is trying to learn data that follows a sequence. However, since the RNNs utilize past data, they can become computationally expensive due to storing large amounts of data in memory. The <strong>LSTM </strong>mitigates this issue, using gates. It has a cell state, and 3 gates; <strong>forget</strong>, <strong>input</strong> and <strong>output</strong> gates.</p><p>The <strong>cell state</strong> is essentially the memory of the network. It carries information throughout the data sequence processing. Information is added or removed from this cell state using gates. Information from the previous hidden state and current input are combined and passed through a sigmoid function at the <strong>forget</strong> gate. The sigmoid function determines which data to keep or forget. The transformed values are then multiplied by the current <strong>cell state</strong>.</p><p>Next, the information from the previous hidden state (<strong>H_(t-1)</strong>)combined with the input (<strong>X_t</strong>) is passed through a sigmoid function at the <strong>input</strong> gate to again determine important information, and also a <em>tanh</em> function to transform data between -1 and 1. This transformation helps with the stability of the network and helps deal with the vanishing/exploding gradient problem. These 2 outputs are multiplied together, and the output is added to the current cell state with the sigmoid function applied to it to give us our new cell state for the next time step.</p><p>Finally, the information from the hidden state combined with the current input is combined and a sigmoid function applied to it. In addition, the new <strong>cell state</strong> is passed through a <em>tanh</em> function to transform the values and both outputs are multiplied to determine the new hidden state for the next time step at the <strong>output</strong> gate.</p><p>Now we have an idea of how the <strong>LSTM</strong> works, let’s construct one. First, we split our data into training and test set. An 80 – 20 split was used for the training and test sets respectively in this case. The data were then scaled using a <em>MinMaxScaler</em>.</p><h4>Structure of data input for LSTM</h4><p>The figure below shows how the sequential data for an LSTM is constructed to be fed into the network. Data source: <a href="https://ieeexplore.ieee.org/document/8355458">Althelaya et al, 2018</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*FzU9shCvIW9Hdog1q88Fbw.png" /><figcaption>Modified from <a href="https://ieeexplore.ieee.org/document/8355458">Althelaya et al, 2018</a>. Data input structure for LSTM</figcaption></figure><p>The idea is since stock data is sequential, information on a given day is related to information from previous days. Basically for data at time <strong>t</strong>, with a window size of <strong>N</strong>, the target feature will be the data point at time <strong>t</strong>, and the feature will be the data points [<strong>t-1</strong>, <strong>t-N</strong>]. We then sequentially move forward in time using this approach. This applies for the short-term prediction. For long term prediction, the target feature will be <strong>Fk</strong> time steps away from the end of the window where <strong>Fk </strong>is the number of days ahead we want to predict. We, therefore, need to format our data that way.</p><pre># Formatting input data for LSTM<br>def format_dataset(X, y, time_steps=1):<br>    Xs, ys = [], []<br>    for i in range(len(X) - time_steps):<br>        v = X.iloc[i:(i + time_steps)].values<br>        Xs.append(v)        <br>        ys.append(y.iloc[i + time_steps])<br>    return np.array(Xs), np.array(ys)</pre><p>Following the work of <a href="https://ieeexplore.ieee.org/document/8355458">Althelaya et al, 2018</a>, a time window of 10 days was used for <strong>N</strong>.</p><h4>Building the LSTM model</h4><p>The new installment of TensorFlow (Tensorflow 2.0) via Keras has made the implementation of deep learning models much easier than in previous installments. We will apply a bidirectional LSTM as they have been shown to more effective in certain applications (see <a href="https://ieeexplore.ieee.org/document/8355458">Althelaya et al, 2018</a>). This due to the fact that the network learns using both past and future data in 2 layers. Each layer performs the operations using reversed time steps to each other. The loss function, in this case, will be the mean squared error, and the adam optimizer with the default learning rate is applied. 20% of the training set will also be used as a validation set while running the model.</p><pre># Design and input arguments into the LSTM model for stock price prediction</pre><pre>from tensorflow import keras</pre><pre># Build model. Include 20% drop out to minimize overfitting<br>model = keras.Sequential()<br>model.add(keras.layers.Bidirectional(<br>        keras.layers.LSTM(<br>          units=32, <br>          input_shape=(X_train_lstm.shape[1], X_train_lstm.shape[2]))<br>  )<br>)</pre><pre>model.add(keras.layers.Dropout(rate=0.2))<br>model.add(keras.layers.Dense(units=1))</pre><pre># Define optimizer and metric for loss function<br>model.compile(optimizer=&#39;adam&#39;,loss=&#39;mean_squared_error&#39;)</pre><pre># Run model<br>history = model.fit(<br>    X_train_lstm, y_train_lstm, <br>    epochs=90, <br>    batch_size=40, <br>    validation_split=0.2,<br>    shuffle=False,<br>    verbose=1   <br>)</pre><p>Note we do not shuffle the data in this case. That’s because stock data is sequential i.e the price on a given day is dependent on the price from previous days. Now let’s have a look at the loss function to see how the validation and training data fair</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*fcX2HBf7cq5kn4Ql6mFd4g.png" /><figcaption>The loss function for training and validation sets as a function of epochs</figcaption></figure><p>The validation set appears to have a lot more fluctuations relative to the training set. This is mainly due to the fact that most neural networks use stochastic gradient descent in minimizing the loss function which tends to be stochastic in nature. In addition, the data is batched which can influence the behavior of the solution. Of course, ideally, these hyper-parameters need to be optimized, but this is merely a simple demonstration. Let’s have a look at the solution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*_mmEgIgI8uiaM9mozJ7etg.png" /><figcaption>Predicted vs actual closing prices for AMD stock</figcaption></figure><p>Overall, it seems the predictions follow the trend of the actual prices. Of course with more epochs and hyperparameter tuning, we could achieve better results. Let’s take a closer look</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*fRhT-5Za1RGw0WckgzBzEQ.png" /><figcaption>Predicted vs actual closing prices for AMD stock. A sub-section of the results</figcaption></figure><p>At first glance, we notice the LSTM has some implicit autocorrelation in its results since its predictions for a given day are very similar to those of the previous day. The predictions essentially lag the true results. Its basically showing that the best guess of the model is very similar to previous results. This should not be a surprising result; The stock market is influenced by a number of factors such as news, earnings reports, mergers, etc. Therefore, it is a bit too chaotic and stochastic to be accurately modeled because it depends on so many factors, some of which can be sporadic i.e positive or negative news. Therefore, in my opinion, this may not be the best way to predict stock prices. Of course with major advances in AI, there might actually be a way of doing this using LSTMs, but I don’t think the hedge funds will be sharing their methods with outsiders anytime soon.</p><h3>Part 3: Estimating price movement with regression analysis</h3><p>Of course, we could still make an attempt to have an idea of what the possible price movements might be. In this case, I will utilize the differential prices as there is less variance compared to using absolute prices. This is a somewhat naive way but let’s try it nevertheless. Let’s explore some relationships</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*R_LnDcTnI2a_5Aywnxr64A.png" /><figcaption>Differential close price as a function of some stock-price information. The X-axis is denoted by the legends in each subplot</figcaption></figure><p>Again to reiterate, the differential prices relate to the difference between the price at time <strong>t</strong> and the previous day value at time <strong>t-1</strong>. The differential high, differential low, differential high-low, and differential open-close price appear to have a linear relationship with the differential close. But only the differential open-close would be useful in analysis. This because on a given day (time <strong>t</strong>), we can not know what the highs or lows are beforehand till the end of the trading day. Theoretically, those values could end up coinciding with the closing price. However, we do know the open value at the start of the trading period. I will apply ridge regression to make our results more robust. Our target variable will be the differential close price, and our feature will be the differential open-close price.</p><pre>from sklearn.linear_model import LinearRegression, Ridge, Lasso<br>from sklearn.model_selection import GridSearchCV, cross_val_score<br>import sklearn<br>from sklearn.metrics import median_absolute_error, mean_squared_error</pre><pre># Split data into training and test<br>X_train_reg, X_test_reg, y_train_reg, y_test_reg = train_test_split(X_reg, y_reg, test_size=0.2, random_state=0)</pre><pre># Use a 10-fold cross validation to determine optimal parameters for a ridge regression<br>ridge = Ridge()<br>alphas = [1e-15,1e-8,1e-6,1e-5,1e-4,1e-3,1e-2,1e-1,0,1,5,10,20,30,40,45,50,55,100]<br>params = {&#39;alpha&#39;: alphas}</pre><pre>ridge_regressor = GridSearchCV(ridge,params, scoring=&#39;neg_mean_squared_error&#39;,cv=10)<br>ridge_regressor.fit(X_reg,y_reg)</pre><pre>print(ridge_regressor.best_score_)<br>print(ridge_regressor.best_params_)</pre><p>The neg_mean_squared_error returns the negative version of the mean squared error. The closer it is to 0, the better the model. The cross-validation returned an <em>alpha</em> value of 1e-15 and a score of approximately <strong>-0.4979.</strong> Now let’s run the actual model and see our results</p><pre>regr = Ridge(alpha=1e-15)<br>regr.fit(X_train_reg, y_train_reg)</pre><pre>y_pred = regr.predict(X_test_reg)<br>y_pred_train = regr.predict(X_train_reg)</pre><pre>print(f&#39;R^2 value for test set is {regr.score(X_test_reg,y_test_reg)}&#39;)<br>print(f&#39;Mean squared error is {mean_squared_error(y_test_reg,y_pred)}&#39;)</pre><pre>plt.scatter(df_updated[&#39;Open-Close&#39;][1:],df_updated[&#39;Diff_Close&#39;][1:],c=&#39;k&#39;)<br>plt.plot(df_updated[&#39;Open-Close&#39;][1:], (regr.coef_[0] * df_updated[&#39;Open-Close&#39;][1:] + regr.intercept_), c=&#39;r&#39; );<br>plt.xlabel(&#39;Open-Close&#39;)<br>plt.ylabel(&#39;Diff-Close&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*TI6Plw9plvIaxStAwJXxtQ.png" /><figcaption>Differential open-close vs differential close price for AMD stock</figcaption></figure><p>The model corresponds to a slope of <strong>$0.9599</strong> and an intercept of <strong>$0.01854. </strong>This basically says that for every $1 increase in price between the opening price on a given day, and its previous closing price we can expect the closing price to increase by roughly a dollar from its closing price the day before.<strong> </strong>I obtained a mean square error of <strong>0.58 </strong>which is fairly moderate all things considered. This R² value of 0.54 basically says 54% of the variance in the differential close price is explained by the differential open-close price. Not so bad so far. Note that the adjusted R² value was roughly equal to the original value.</p><p>To be truly effective, we need to make further use of statistics. Specifically, let’s define a prediction interval around the model. Prediction intervals give you a range for the prediction that accounts for any threshold of modeling error. Prediction intervals are most commonly used when making predictions or forecasts with a regression model, where a quantity is being predicted. We select the 99% confidence interval in this example such that our actual predictions fall into this range 99% of the time. For an in-depth overview and explanation please explore <a href="https://machinelearningmastery.com/prediction-intervals-for-machine-learning/">machinelearningmastery</a></p><pre>def predict_range(X,y,model,conf=2.58):<br>    <br>    from numpy import sum as arraysum<br>    <br>    # Obtain predictions<br>    yhat = model.predict(X)<br>    <br>    # Compute standard deviation<br>    sum_errs = arraysum((y - yhat)**2)<br>    stdev = np.sqrt(1/(len(y)-2) * sum_errs)<br>   <br>    # Prediction interval (default confidence: 99%)<br>    interval = conf * stdev<br>    <br>    lower = []<br>    upper = []<br>    <br>    for i in yhat:<br>        lower.append(i-interval)<br>        upper.append(i+interval)</pre><pre>return lower, upper, interval</pre><p>Let’s see our prediction intervals</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*Qs3MoVSVdhFlSM9Z3xgRjw.png" /><figcaption>Differential open-close vs differential close price for AMD stock with its prediction interval</figcaption></figure><p>Our prediction error corresponds to a value of <strong>$1.82</strong> in this example. Of course, the parameters used to obtain our regression model (slope and intercept) also have a confidence interval which we could calculate, but its the same process as highlighted above. From this plot, we can see that even with a 99% confidence interval, some values still fall outside the range. This highlights just how difficult it is to forecast stock price movements. So ideally we would supplement our analysis with news, technical indicators, and other parameters. In addition, our model is still quite limited. We can only make predictions about the closing price on the same day i.e at time <strong>t</strong>. Even with the large uncertainty, this could potentially prove useful in, for instance, options tradings.</p><h3>What does this all mean?</h3><p>The stock screening strategy introduced could be a valuable tool for finding good stocks and minimizing loss. Certainly, in future projects, I could perform sentiment analysis myself in order to get information on every stock from the biggest movers list. There is also some room for improvement in the LSTM algorithm. If there was a way to minimize the bias and eliminate the implicit autocorrelation in the results, I’d love to hear about it. So far, the majority of research on the topic haven’t shown how to bypass this issue. Of course, more model tuning and data may help so any experts out there, please give me some feedback. The regression analysis seems pretty basic but could be a good way of adding extra information in our trading decisions. Of course, we could have gone with a more sophisticated approach; the differential prices almost appear to form an ellipse around the origin. Using a radial SVM classifier could also be a potential way of estimating stock movements.</p><p>The codes, as well as the dataset, will be provided <a href="https://github.com/Ibinaldo?tab=repositories">here</a> in the not so distant future, and my Linkedin profile can be found <a href="https://www.linkedin.com/in/ibinabo-bestmann/">here</a>. Thank you for reading!</p><p>Ibinabo Bestmann</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f40742dd86e0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/stock-market-screening-and-analysis-using-web-scraping-neural-networks-and-regression-analysis-f40742dd86e0">Stock Market Screening and Analysis: Using Web Scraping, Neural Networks, and Regression Analysis…</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Facies classification using unsupervised machine learning in geoscience]]></title>
            <link>https://medium.com/data-science/facies-classification-using-unsupervised-machine-learning-in-geoscience-8b33f882a4bf?source=rss-c9b33aa3086e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b33f882a4bf</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[geology]]></category>
            <category><![CDATA[oil-and-gas]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ibinabo Bestmann]]></dc:creator>
            <pubDate>Sat, 22 Aug 2020 23:45:44 GMT</pubDate>
            <atom:updated>2020-08-24T03:03:02.466Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data science in Oil and Gas</h4><h4>Understanding Earth structure using K-means clustering</h4><p>Facies are uniform sedimentary bodies of rock which are distinguishable enough from each other in terms of physical characteristics (e.g. sedimentary structure, grain sizes), deposited under the action of a relatively uniform hydrodynamic regime in a given depositional setting. The different types of facies based on these properties include sedimentary facies, lithofacies, seismic facies, etc.</p><p>The physical and organic characteristics found in these rock units usually provide some insight into the different process and systems (e.g. depositional environments) which may have occurred in the region. Combinations of several facies with physical models and other geological data can help provide informative low-dimensional models of the geologic region, leading to better insights regarding the geology of the region.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*tOKIkxO-MEvzIPOib2p7qw.png" /><figcaption>Taken from <a href="https://commons.wikimedia.org/wiki/File:Geology_Santa_Cruz_Formation.png">Wikimedia</a>: An example of facies classification in a geological formation.</figcaption></figure><p>In oil and gas exploration, knowledge of the depositional environment is vital as it can help in providing a decent picture of the petroleum systems involved. A petroleum system consists of source rocks (organic matter rich rocks which generate hydrocarbons if heated sufficiently), reservoir rocks (set of rock units that holds hydrocarbon reserves which migrate from the source rock), and seal rock (relatively impermeable rock units that form a barrier around reservoir rocks, preventing hydrocarbon migration beyond the reservoir). A succession of facies with sandstone units, for instance, might be an indicator of a good reservoir, as they tend to have high permeability and porosity, ideal conditions to store hydrocarbons. A common rule known as Walther’s Law of facies stipulates that vertical succession of facies reflects lateral changes in the depositional environment i.e. Two adjacent facies within the rock record must have been deposited laterally in formation. Therefore, the amount of hydrocarbon in place may be estimated by observing the lateral extent and geometries of the facies containing the reservoir units.</p><p>The main sources of data with regards to sub-surface rocks come from a variety of sources, most of which involve drilling. The ideal source of data for facies classification is core (rock) samples obtained from drilled wells as they allow for direct assessment of the sedimentary structure. However, these are expensive to obtain and are not always a feasible option in some cases due to costs. In these situations, indirect measurements are necessary. In this study, I will focus on wireline logs. Wireline logging involves the process of lowering instruments into a borehole and recording measurements that detail the physical characteristics of the surrounding rock and fluids with depth. It is a means of assessing reservoirs since it aids in the differentiation between oil, gas, and water containing formations, determination of porosity, as well as the approximate amount of hydrocarbons present in each formation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*dkMgJQEqb8Yjfd7ThNb9zg.jpeg" /><figcaption>Taken from <a href="https://en.wikipedia.org/wiki/File:Wikipedia_petrophysics_figure_1.jpg">Wikipedia</a>: Schematic well-log responses for some common lithologies. Shale (SH) and Sandstone (SS) shown in this figure.</figcaption></figure><p>In order to assign facies using log data, an analyst is required to analyze the records and determine the approximate lithology with depth. With larger datasets, this can become quite tedious and very inefficient in terms of time. Automation of this procedure is therefore necessary and may be helpful in giving a quick and reasonably accurate picture of the regional geology. I will demonstrate how unsupervised learning can help with this process using K-means clustering.</p><h3>Study area</h3><p>The analysis is focused on the northwestern region of Kansas, USA, specifically the Forest city basin. This basin has predominantly been a shallow oil and gas province with some coal production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*UjJchvNqkkHQ65w7kabr2g.png" /><figcaption>Taken from <a href="https://en.wikipedia.org/wiki/File:Kansas_Structure_Map.png">Wikipedia</a>: Subsurface geologic structures of Kansas with the major topographic basins</figcaption></figure><p>My focus was on the Nemaha region within the Forest city basin, which is known to have occurrences of oil and gas and had a couple of wells drilled there. The data was obtained courtesy of the Kansas geological survey which has public records for different wells in the region (Please see <a href="https://www.shalexp.com/kinney-oil-company">https://www.shalexp.com/kinney-oil-company</a> for more details on wells).</p><p>The well logs were obtained in <strong>LAS</strong> file format from <a href="http://www.kgs.ku.edu/PRS/Scans/Log_Summary/index.html">here</a> and were examined for data quality and completeness. A criterion for selection was the presence of eight curves that were deemed necessary for the analysis. The table below shows the associated curves and the properties they provide about the rock formation. These values are all a function of depth measured in feet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-8Q0uWZ72yE-6Kuduqm-pg.png" /><figcaption>A list of logging tools and their properties</figcaption></figure><p>Using a wide array of logs is useful as they not only provide information on properties such as the lithology, porosity, and electrical conductivity of a formation, but used in conjunction with each other, can boost the accuracy of lithology estimation by performing better in situations where individual logs might fail. I analyzed several of the wells and was able to obtain 6 wells with the required curves available. These curves from the different wells were then combined into a single CSV file for processing. The well names and the formation at each depth were also included. The formation information was obtained at <a href="http://www.kgs.ku.edu/Magellan/Qualified/index.html">http://www.kgs.ku.edu/Magellan/Qualified/index.html</a>. Oil wells belonging to Kinney Oil</p><h3>Data pre-processing and cleaning</h3><p>Descriptive statistics were obtained on the raw data to determine the presence of missing values and potential outliers. Outlier removal is essential as their presence can increase variability in data, making the results of an experiment less likely to be statistically significant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zt__7ht90DGi8Po6FB4u1Q.png" /><figcaption>Descriptive statistics on raw well log data</figcaption></figure><p>The counts differed across the different well logs which indicates missing values for some logs. In addition, some well logs which contained no data had a value of 999 imputed instead. Normally, the issue of missing values can be addressed by imputing the mean or median of the available values. However, geology can be quite complex, and simply replacing missing values with the mean may not reflect the actual geology. The total missing values accounted for a small fraction of the total data so they were dropped instead. The Unnamed column was also dropped as it corresponded to the index for the data measurements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*yd4WbJtFzusbV0J7jQRehw.png" /><figcaption>Descriptive statistics on the well log data after removal of missing values</figcaption></figure><h3>Removing outliers</h3><p>So missing data have been removed, but if we pay attention to the maximum values for each well log, we will notice some situations where the maximum values are significantly greater than the mean which are red flags for outliers. Let’s visually inspect some of these data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BuwipwALGPqxoPRhsXrb4Q.png" /><figcaption>Well logs as a function of depth</figcaption></figure><p>We can see some data that significantly greater than the general population for some of the logs, especially in the resistivity log (<strong>ILD</strong>). Note that values as high as 10,000 Ω/m for <strong>ILD</strong> do occur in nature for certain lithologies. But resistivity values for most rock types fall within a range, and values as high as 10000 Ω/m also encapsulates 2000 Ω/m in that range. Judging by the relatively low prevalence of values that high, removing those values may improve clustering results. Using the <strong>ILD</strong> curve as a reference point, I found keeping values that fell within the 99.95th quantile resulted in a cut off value of around 2000 Ω/m for the <strong>ILD</strong>. It was gentle enough to minimize data loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EACb3JhpsQkEEKeGlZBbKg.png" /><figcaption>Well logs as a function of depth after outlier removal</figcaption></figure><p>As some lithologies may result in big spikes in certain physical measurements (e.g. <strong>ILD, CILD)</strong>, the natural log was opted for to minimize the variability in measurement values. This was applied to the <strong>ILD </strong>and <strong>CILD </strong>logs due to the large range in their values. Finally, the processed logs were saved to a CSV file.</p><h3>K-means Clustering</h3><p>After pre-processing the data, it is important to do some feature selection to ensure faster training, as well as less complexity in our model and improved accuracy. In our data set, we have two logs that give very similar information; the resistivity (<strong>ILD</strong>) and conductivity logs (<strong>CILD</strong>). We can verify this by checking the correlation between the logs to see how strongly related they are.</p><pre>features.corr() # compute correlation between the well log curves</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*fNc7iPc7zm8PzEklWRrFLw.png" /><figcaption>Correlation matrix for the well log data</figcaption></figure><p>While there are some high correlation values, for this study correlations of 0.8 and above were deemed statistically significant. I picked a high value because some logs do measure very similar properties in terms of lithology, but they also have important differences that can help differentiate lithologies apart. The logarithmic resistivity and conductivity values have a near-perfect correlation i.e they give us basically the same information. The conductivity was therefore dropped from the features list.</p><p>Next, the data were scaled using the <em>scale</em> module from the <em>sklearn </em>package. The issue with k-means clustering is we have no idea which cluster separation is the most accurate representation of our data. Therefore I employed two techniques that can give us an idea of what the optimal cluster size could be.</p><p>Let’s begin with the Elbow technique. This method runs K-means clustering for a specified number of clusters and calculates the within sum of squares i.e the sum of differences between each point in a cluster and its assigned cluster centroid. The optimal cluster is usually selected as the point in which the sum of squares is minimized and subsequent changes for increasing clusters are minimal. Let’s make an analysis for clusters ranging in size from 1 to 12</p><pre>wcss = []  # Store within sum of square values for each cluster size</pre><pre>cl_num = 12 # Total number of clusters<br>for i in range (1,cl_num):<br>    kmeans = KMeans(i, random_state=10) <br>    kmeans.fit(x_scaled)<br>    wcss_iter = kmeans.inertia_ # calculates the wcss<br>    wcss.append(wcss_iter)<br>wcss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/1*gZlp0yrINDvNjcAPlSQmvA.png" /><figcaption>Calculated within sum of squares for each number of clusters used</figcaption></figure><p>Next, we plot these values as a function of the number of clusters used</p><pre>number_clusters = range(1,cl_num)<br>plt.figure(figsize=(10,8))<br>plt.plot(number_clusters, wcss, &#39;*-&#39;)<br>plt.xlabel(&#39;Number of clusters&#39;,fontsize=20)<br>plt.ylabel(&#39;Within-cluster Sum of Squares&#39;,fontsize=20)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*UF0Xg0ZqSneKvakypFHwWg.png" /><figcaption>The within-sum of squares (WCSS) as a function of the number of assigned clusters</figcaption></figure><p>Although not very apparent, we can see the curve begins to somewhat flatten around <em>no of clusters = 5. </em>We can further evaluate the optimal cluster size by looking at the silhouette score. The silhouette score is a measure of how similar an object is to its own cluster (cohesion) compared to other clusters (separation). Silhouette coefficients near 1 indicate that the sample is far away from the neighboring clusters. A value of 0 indicates that the sample is on or very close to the decision boundary between two neighboring clusters and negative values indicate that those samples might have been assigned to the wrong cluster. Let’s calculate the silhouette scores for the different number of clusters starting at <em>no of clusters = 5.</em></p><pre>from sklearn.metrics import silhouette_score</pre><pre>range_n_clusters = [5,6,7,8,9] # Number of clusters</pre><pre>for n_clusters in range_n_clusters:<br>    <br>    clusterer = KMeans(n_clusters=n_clusters, random_state=10)<br>    cluster_labels = clusterer.fit_predict(x_scaled)<br>    <br>    silhouette_avg = silhouette_score(x_scaled, cluster_labels)<br>    <br>    print(&quot;For n_clusters =&quot;, n_clusters,<br>          &quot;The average silhouette_score is :&quot;, silhouette_avg)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*1qq7M6JGdGuP2bYGnwWzew.png" /><figcaption>Silhouette scores as a function of the number of assigned clusters.</figcaption></figure><p>The silhouette score had its highest values at <em>no of clusters = 8</em>. The value of 6 had a very similar silhouette score but having more separation is better in facies identification. This is due to the fact that some lithologies tend to get interbedded with each other so more clusters might reveal these patterns.</p><p>Using the <em>KMeans</em> algorithm from the <em>sklearn’s cluster </em>module, the well log data were clustered into 8 groups. I utilized a 100 random initial centroid seeds to increase the probability of finding a cluster separation that best describes the data.</p><h3>Data visualization and interpretation</h3><p>The well logs and facies as a function of depth for some of the well were plotted using codes modified from <a href="https://github.com/seg/2016-ml-contest/blob/master/Facies_classification.ipynb">Brendan Hall</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*MK60-yR7rJQZ9sPD0S56Pg.png" /><figcaption>Classification results for well: Woody Acres</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*9-Sqp1x-mLrmDS3iVOqhXQ.png" /><figcaption>Classification results for well: Baumgartner</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*8LfWn9OQy-BPr9m9muAhLA.png" /><figcaption>Classification results for well: Hartter</figcaption></figure><h3>Validating results</h3><p>The plots show there is a lateral variation of lithologies in this region due to variations in the facies around the same depth. We can validate these results by obtaining some knowledge of the Kansas stratigraphy. According to <a href="http://www.kgs.ku.edu/Publications/Bulletins/Sub9/page2.html">KGS geology</a>, the dominant lithologies in this region are; sandstones, shales, dolomite, limestones with some silt, chert, and chalk. Analysis of core samples also suggests the presence of gypsium and pyrite amongst others. My analysis highlighted 8 facies which are reasonable results based on the known information. Different rocks have different well log responses, and these responses used in combination can give us a pretty good idea of what the underlying lithology is. Different log responses for different lithologies can be found <a href="https://www.geoloil.com/VshFromNeutrDens.php">here</a> and <a href="http://www.kgs.ku.edu/Publications/Bulletins/LA/05_overlay.html">here</a>.</p><p>If we consider the Viola formation for instance, in the Woody Acres well its represented predominantly by <strong>Facies 2</strong>. It is characterized by extremely low gamma-ray values, density values between 2.5–2.8 g/cm3, DT values on the low end (47–67 µs/ft). The PE values show a jump halfway through the formation suggesting the presence of another lithology, but its overall properties suggest a <strong>dolomitic layer</strong> with some limestones inclusions in it. In the Baumgartner well, the PE values in the bottom half of the formation jump towards a value of 5 b/E, which is indicative of limestones. The top half shows similar properties to dolomite in terms of the PE value, and its decreasing Neutron porosity coupled with a relatively stable density suggests this facies to be a combination of dolomites and limestones i.e a dolomitic limestone layer. We can, therefore, attribute <strong>Facies 7</strong> and <strong>5</strong> as <strong>limestone</strong> and <strong>dolomitic limestone</strong> respectively.</p><p>The Kinderhook formation in the Hartter well shows a jump in the gamma readings, an increase in neutron porosity, with relatively stable density readings and a much lower resistivity reading. The sudden increase in the sonic porosity in addition to the previously mentioned properties points to a <strong>shale</strong> layer for <strong>Facies 4</strong>. The Viola formation in this well also has very similar properties, but the sonic porosity values are lower, and the slightly higher resistivity values compared to the Kinderhook formation suggest the inclusion of sands, limestone, or dolomites. However, the PE values are lower than that of dolomites or limestones suggesting <strong>Facies 1</strong> is a <strong>sandy shale</strong> layer. Finally, the low gamma values in the top parts of the Hunton formation, coupled with the drop in the neutron porosity and increase in density and PE values in the dolomitic range suggest a <strong>dolomitic sand</strong> regime for <strong>Facies 3</strong>.</p><p>Reference to <a href="http://www.kgs.ku.edu/Publications/Bulletins/Sub9/page2.html">KGS geology</a> reveals the Viola formation to be composed of fine to coarse-grained limestones and dolomites containing variable quantities of chert. It also highlights the Hunton formation to be generally composed of gray to brown, fine-grained, crystalline dolomite or limestone with minor chert in some parts. It is also slightly coarser-grained, and slightly sandy dolomite with vuggy porosity with some chert in other parts of the formation. These agree with my analysis which shows limestones and dolomites constitute the majority lithology in these formations.</p><h3>Summary</h3><p>Unsupervised machine learning can be a great way to derive some quick insights from your data in a cost-effective manner and could be a guide for additional supervised work as demonstrated in this study. However, as attractive as the prospects of this technique are, it is important to be aware of the caveats associated with this method.</p><p>The optimal number of clusters is not easy to predict and can be subjective in most cases. Most lithologies are not completely homogenous and tend to include different rock types, so finding an optimal <em>k</em> value is difficult and sometimes not possible, and may require some <em>a priori</em> knowledge. Additionally, the clusters can change depending on the initial centroid starting location. It is also not a practical solution for situations where there is an enormous amount of data. In those cases, it is preferable to use supervised learning techniques. In future work, I will demonstrate how a <em>convolutional neural network</em> can be applied to facies classification, as well as evaluating its effectiveness in terms of accuracy and other metrics.</p><p>Comments and feedback are welcome. The codes, as well as the dataset, will be provided <a href="https://github.com/Ibinaldo?tab=repositories">here</a> in the not so distant future, and my Linkedin profile can be found <a href="https://www.linkedin.com/in/ibinabo-bestmann/">here</a>. I hope you found the article interesting, this was a fun project to investigate!</p><p>Ibinabo Bestmann</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b33f882a4bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/facies-classification-using-unsupervised-machine-learning-in-geoscience-8b33f882a4bf">Facies classification using unsupervised machine learning in geoscience</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>