<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bryan McMaster on Medium]]></title>
        <description><![CDATA[Stories by Bryan McMaster on Medium]]></description>
        <link>https://medium.com/@bryanmcmaster?source=rss-9482c23f774d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Y5H-OPZ85hIyvxD9gNRgpA.jpeg</url>
            <title>Stories by Bryan McMaster on Medium</title>
            <link>https://medium.com/@bryanmcmaster?source=rss-9482c23f774d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:02:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bryanmcmaster/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Ingesting Data with Cribl Webhook using Databricks Zerobus]]></title>
            <link>https://bryanmcmaster.medium.com/ingesting-data-with-cribl-webhook-using-databricks-zerobus-cbf3e6f94e3e?source=rss-9482c23f774d------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbf3e6f94e3e</guid>
            <dc:creator><![CDATA[Bryan McMaster]]></dc:creator>
            <pubDate>Sat, 08 Aug 2026 11:39:10 GMT</pubDate>
            <atom:updated>2026-08-08T12:29:05.882Z</atom:updated>
            <content:encoded><![CDATA[<h3>Ingesting Data with the Cribl Webhook using Databricks Zerobus</h3><p>Quick Update: Databricks just released the <a href="https://docs.databricks.com/aws/en/ingestion/zerobus-kafka">Zerobus Kafka Producer API</a> into Beta. In my next post I will show how to set that up!</p><p>Recently, I was asked what the best way to ingest data from <a href="https://cribl.io/">Cribl</a> into Databricks was.</p><p>For those that do not know, Cribl is a platform that allows you to route security and telemetry data from different sources to different destinations. In this case, the destination is Databricks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r1hUgzKYsWe354ChhGT-BQ.jpeg" /></figure><h3>The different options</h3><p>Initially, we looked at the default cloud-storage destination in Cribl. This is relatively easy to set up. Cribl writes data to a Databricks volume in near real time, and then Databricks uses <a href="https://learn.microsoft.com/en-gb/azure/databricks/ingestion/cloud-object-storage/auto-loader/">Auto Loader</a> to load the data into Delta tables.</p><p>There are a few caveats with this approach. The main requirement was sub-minute end-to-end latency. Auto Loader can support near-real-time ingestion, but achieving a consistently low latency can require more aggressive trigger settings and additional compute. This can increase the cost, and file discovery and micro-batch scheduling can still add some variability.</p><p>The next option was to use a message broker as the Cribl destination, such as <a href="https://docs.cribl.io/stream/destinations-azure-event-hubs/">Azure Event Hubs</a> or <a href="https://docs.cribl.io/stream/4.3/destinations-kafka/">Kafka</a>. We could then attach a <a href="https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/concepts">Structured Streaming</a> job directly to the broker. For workloads with very low latency requirements, <a href="https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/real-time/concepts">Structured Streaming real-time mode</a> is another option.</p><p>From an architecture perspective, this is a sound approach. However, it introduces another system to operate, secure, monitor, and scale between Cribl and Databricks. For this use case, we decided that the additional operational overhead was not worth it.</p><p>I then investigated the <a href="https://docs.cribl.io/stream/destinations-webhook/">Cribl Webhook destination</a>. Databricks does not have a generic webhook endpoint out of the box, but it does offer <a href="https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-overview">Zerobus Ingest</a>. Zerobus is a push-based ingestion API that writes directly to Unity Catalog Delta tables.</p><p>This looked like it could be the solution. The problem was that the Zerobus REST API documentation did not provide guidance on how to configure it as a Cribl Webhook destination. So I started working on it. After several hours of testing and debugging, I had an approach that is now being used on a production workload.</p><h3>The difficult part: the configuration</h3><p>The difficult part was not sending an HTTP POST. Cribl already supports HTTP and OAuth. The difficult part was mapping the Zerobus authentication and ingestion requirements into the fields available in the Cribl Webhook configuration.</p><p>There are two different Databricks URLs involved, and keeping them separate is important:</p><ul><li>The <strong>workspace URL</strong> is used to obtain the OAuth token. This is the workspace hostname without the ?o=&lt;workspace-id&gt; part.</li><li>The <strong>Zerobus server endpoint</strong> is used to send the data. For Azure, this follows the format https://&lt;workspace-id&gt;.zerobus.&lt;region&gt;.azuredatabricks.net.</li></ul><p>The Databricks documentation for finding these values is <a href="https://learn.microsoft.com/en-gb/azure/databricks/ingestion/zerobus-ingest#get-your-workspace-url-and-zerobus-ingest-endpoint">Get your workspace URL and Zerobus Ingest endpoint</a>.</p><p>The OAuth request also needs more than a client ID and secret. It needs the Zerobus resource and an authorization_details parameter that describes the Unity Catalog privileges being requested. The service principal needs USE CATALOG, USE SCHEMA, and SELECT and MODIFY on the target table.</p><p>Finally, the request sent to Zerobus needs to be a JSON list of records. Each record must match the schema of the target table. These were the parts that took the most testing to get working together.</p><h3>Create the target table</h3><p>The first step is to create the target table in Unity Catalog. Zerobus does not create the table for you. The table and its schema are the contract for the incoming data. Records that do not match the schema are rejected by default.</p><p>The example below is for OneLogin data:</p><pre>CREATE TABLE tesco_demos_catalog.security.zb_onelogin_raw (<br>  timestamp STRING COLLATE UTF8_BINARY COMMENT &#39;Event time as provided by the source; cast downstream if required&#39;,<br>  event_id STRING COLLATE UTF8_BINARY COMMENT &#39;UUID per event&#39;,<br>  user_email STRING COLLATE UTF8_BINARY COMMENT &#39;Authenticating user email address&#39;,<br>  event_type STRING COLLATE UTF8_BINARY COMMENT &#39;login_success | sso_auth | mfa_challenge | login_failure | account_lockout | password_reset&#39;,<br>  status STRING COLLATE UTF8_BINARY COMMENT &#39;success | failure&#39;,<br>  mfa_method STRING COLLATE UTF8_BINARY COMMENT &#39;totp | push | sms | none&#39;,<br>  source_ip STRING COLLATE UTF8_BINARY COMMENT &#39;Client public IP; string supports IPv4 and IPv6&#39;,<br>  geo_country STRING COLLATE UTF8_BINARY COMMENT &#39;Geo-IP country, for example United Kingdom&#39;,<br>  geo_city STRING COLLATE UTF8_BINARY COMMENT &#39;Geo-IP city, for example London&#39;,<br>  app_name STRING COLLATE UTF8_BINARY COMMENT &#39;SSO target application, for example Salesforce&#39;,<br>  user_agent STRING COLLATE UTF8_BINARY COMMENT &#39;Raw User-Agent header&#39;,<br>  session_duration_sec INT COMMENT &#39;Session duration in seconds; 0 for failures&#39;,<br>  risk_score DOUBLE COMMENT &#39;Risk score from 0 to 100&#39;,<br>  source STRING COLLATE UTF8_BINARY COMMENT &#39;Logical source name, always onelogin for this feed&#39;<br>)<br>USING DELTA<br>COMMENT &#39;OneLogin SSO and identity events landed through Zerobus REST from the Cribl Webhook destination.&#39;<br>TBLPROPERTIES (<br>  &#39;delta.autoOptimize.autoCompact&#39; = &#39;true&#39;,<br>  &#39;delta.autoOptimize.optimizeWrite&#39; = &#39;true&#39;,<br>  &#39;delta.checkpoint.writeStatsAsJson&#39; = &#39;false&#39;,<br>  &#39;delta.checkpoint.writeStatsAsStruct&#39; = &#39;true&#39;,<br>  &#39;delta.enableDeletionVectors&#39; = &#39;true&#39;,<br>  &#39;delta.enableRowTracking&#39; = &#39;true&#39;,<br>  &#39;delta.feature.appendOnly&#39; = &#39;supported&#39;,<br>  &#39;delta.feature.deletionVectors&#39; = &#39;supported&#39;,<br>  &#39;delta.feature.domainMetadata&#39; = &#39;supported&#39;,<br>  &#39;delta.feature.invariants&#39; = &#39;supported&#39;,<br>  &#39;delta.feature.rowTracking&#39; = &#39;supported&#39;,<br>  &#39;delta.minReaderVersion&#39; = &#39;3&#39;,<br>  &#39;delta.minWriterVersion&#39; = &#39;7&#39;,<br>  &#39;delta.parquet.compression.codec&#39; = &#39;zstd&#39;<br>)</pre><p>The table does not need to use exactly the same physical types as the source, but the fields sent by Cribl must map to compatible columns.</p><p>If the source data is semi-structured, a VARIANT column may help reduce the amount of transformation needed. However, VARIANT does not make the Zerobus ingestion contract schema-free. Zerobus still validates the incoming record against the table schema. The <a href="https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/variant-type">VARIANT documentation</a> also notes that the type is available in Databricks Runtime 15.3 and above.</p><p>The table properties above are from this example. They are not all required for the Cribl-to-Zerobus pattern, so review them against your own Delta and governance requirements.</p><h3>Create the service principal</h3><p>Once the table has been created, set up a Databricks service principal. The instructions are available in <a href="https://learn.microsoft.com/en-gb/azure/databricks/ingestion/zerobus-ingest#create-a-service-principal-and-grant-permissions">Create a service principal and grant permissions</a>.</p><p>Create the service principal, generate a client ID and client secret, and then grant it access to the catalog, schema, and table:</p><pre>GRANT USE CATALOG ON CATALOG tesco_demos_catalog TO `&lt;APPLICATION_ID&gt;`;<br>GRANT USE SCHEMA ON SCHEMA tesco_demos_catalog.security TO `&lt;APPLICATION_ID&gt;`;<br>GRANT MODIFY, SELECT ON TABLE tesco_demos_catalog.security.zb_onelogin_raw TO `&lt;APPLICATION_ID&gt;`;</pre><p>Replace &lt;APPLICATION_ID&gt; with the application ID for the service principal. The client secret should be stored securely and should not be placed in source control or published in a blog post.</p><h3>Configure the Cribl Webhook</h3><p>Now that the Databricks setup is complete, move to the Cribl UI.</p><p>Under <strong>Data &gt; Destinations</strong>, select the <strong>Webhook</strong> tile and click <strong>Add Destination</strong>. Once the new destination screen is open, click <strong>Manage as JSON</strong>.</p><p>The JSON below is based on the working configuration for this pattern, so check the exported configuration against the version of Cribl you are running before applying it.</p><h3>Values to replace</h3><ul><li>SERVER_ENDPOINT: The endpoint for the Zerobus server</li><li>LOGIN_URL: This is how the credentials are passed, basically your workspace URL without the o= parameter</li><li>MY_REGION: The Azure region for the workspace</li><li>MY_CATALOG: Your Unity Catalog catalog</li><li>MY_SCHEMA: Your Unity Catalog schema</li><li>MY_TABLE: Your target table</li><li>MY_SECRET: The service principal client secret</li><li>MY_CLIENT_ID: The service principal application ID</li></ul><p>For example, if the full workspace URL is:</p><pre>https://abcd-teste2-test-spcse2.azuredatabricks.net/?o=12123234242323</pre><p>then the values are:</p><pre>SERVER_ENDPOINT: 12123234242323.zerobus.&lt;region&gt;.azuredatabricks.net<br>LOGIN_URL: abcd-teste2-test-spcse2.azuredatabricks.net</pre><p>The same idea applies to AWS and GCP, although the hostname suffix is different. Use the values from your own workspace rather than copying the example.</p><pre>{<br>  &quot;id&quot;: &quot;zerobus&quot;,<br>  &quot;systemFields&quot;: [],<br>  &quot;streamtags&quot;: [],<br>  &quot;method&quot;: &quot;POST&quot;,<br>  &quot;format&quot;: &quot;json_array&quot;,<br>  &quot;keepAlive&quot;: true,<br>  &quot;concurrency&quot;: 3,<br>  &quot;maxPayloadSizeKB&quot;: 9216,<br>  &quot;maxPayloadEvents&quot;: 0,<br>  &quot;compress&quot;: false,<br>  &quot;rejectUnauthorized&quot;: true,<br>  &quot;timeoutSec&quot;: 30,<br>  &quot;flushPeriodSec&quot;: 1,<br>  &quot;useRoundRobinDns&quot;: false,<br>  &quot;failedRequestLoggingMode&quot;: &quot;none&quot;,<br>  &quot;safeHeaders&quot;: [],<br>  &quot;responseRetrySettings&quot;: [<br>    {<br>      &quot;httpStatus&quot;: 408,<br>      &quot;initialBackoff&quot;: 250,<br>      &quot;backoffRate&quot;: 2,<br>      &quot;maxBackoff&quot;: 10000<br>    },<br>    {<br>      &quot;httpStatus&quot;: 429,<br>      &quot;initialBackoff&quot;: 1000,<br>      &quot;backoffRate&quot;: 2,<br>      &quot;maxBackoff&quot;: 10000<br>    },<br>    {<br>      &quot;httpStatus&quot;: 500,<br>      &quot;initialBackoff&quot;: 250,<br>      &quot;backoffRate&quot;: 2,<br>      &quot;maxBackoff&quot;: 10000<br>    },<br>    {<br>      &quot;httpStatus&quot;: 502,<br>      &quot;initialBackoff&quot;: 250,<br>      &quot;backoffRate&quot;: 2,<br>      &quot;maxBackoff&quot;: 10000<br>    },<br>    {<br>      &quot;httpStatus&quot;: 503,<br>      &quot;initialBackoff&quot;: 250,<br>      &quot;backoffRate&quot;: 2,<br>      &quot;maxBackoff&quot;: 10000<br>    },<br>    {<br>      &quot;httpStatus&quot;: 504,<br>      &quot;initialBackoff&quot;: 250,<br>      &quot;backoffRate&quot;: 2,<br>      &quot;maxBackoff&quot;: 10000<br>    },<br>    {<br>      &quot;httpStatus&quot;: 509,<br>      &quot;initialBackoff&quot;: 250,<br>      &quot;backoffRate&quot;: 2,<br>      &quot;maxBackoff&quot;: 10000<br>    }<br>  ],<br>  &quot;timeoutRetrySettings&quot;: {<br>    &quot;timeoutRetry&quot;: true,<br>    &quot;initialBackoff&quot;: 1000,<br>    &quot;backoffRate&quot;: 2,<br>    &quot;maxBackoff&quot;: 10000<br>  },<br>  &quot;responseHonorRetryAfterHeader&quot;: true,<br>  &quot;onBackpressure&quot;: &quot;block&quot;,<br>  &quot;authType&quot;: &quot;oauth&quot;,<br>  &quot;tls&quot;: {<br>    &quot;disabled&quot;: true,<br>    &quot;rejectUnauthorized&quot;: true<br>  },<br>  &quot;loadBalanced&quot;: false,<br>  &quot;authHeaderExpr&quot;: &quot;`Bearer ${token}`&quot;,<br>  &quot;tokenTimeoutSecs&quot;: 3500,<br>  &quot;customSourceExpression&quot;: &quot;__httpOut&quot;,<br>  &quot;customDropWhenNull&quot;: false,<br>  &quot;customEventDelimiter&quot;: &quot;\\n&quot;,<br>  &quot;customContentType&quot;: &quot;application/json&quot;,<br>  &quot;customPayloadExpression&quot;: &quot;`${events}`&quot;,<br>  &quot;excludeSelf&quot;: false,<br>  &quot;urls&quot;: [<br>    {<br>      &quot;weight&quot;: 1,<br>      &quot;url&quot;: &quot;https://SERVER_ENDPOINT/zerobus/v1/tables/MY_CATALOG.MY_SCHEMA.MY_TABLE/insert&quot;<br>    }<br>  ],<br>  &quot;dnsResolvePeriodSec&quot;: 600,<br>  &quot;loadBalanceStatsPeriodSec&quot;: 300,<br>  &quot;type&quot;: &quot;webhook&quot;,<br>  &quot;description&quot;: &quot;This is a zero bus&quot;,<br>  &quot;loginUrl&quot;: &quot;https://LOGIN_URL/oidc/v1/token&quot;,<br>  &quot;secret&quot;: &quot;MY_SECRET&quot;,<br>  &quot;secretParamName&quot;: &quot;client_secret&quot;,<br>  &quot;tokenAttributeName&quot;: &quot;access_token&quot;,<br>  &quot;oauthParams&quot;: [<br>    {<br>      &quot;value&quot;: &quot;&#39;client_credentials&#39;\n&quot;,<br>      &quot;name&quot;: &quot;grant_type&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;client_id&quot;,<br>      &quot;value&quot;: &quot;&#39;MY_CLIENT_ID&#39;\n&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;scope&quot;,<br>      &quot;value&quot;: &quot;&#39;all-apis&#39;\n&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;resource&quot;,<br>      &quot;value&quot;: &quot;&#39;api://databricks/workspaces/WORKSPACE_ID/zerobusDirectWriteApi&#39;&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;authorization_details&quot;,<br>      &quot;value&quot;: &quot;&#39;[{\&quot;type\&quot;:\&quot;unity_catalog_privileges\&quot;,\&quot;privileges\&quot;:[\&quot;USE CATALOG\&quot;],\&quot;object_type\&quot;:\&quot;CATALOG\&quot;,\&quot;object_full_path\&quot;:\&quot;MY_CATALOG\&quot;},{\&quot;type\&quot;:\&quot;unity_catalog_privileges\&quot;,\&quot;privileges\&quot;:[\&quot;USE SCHEMA\&quot;],\&quot;object_type\&quot;:\&quot;SCHEMA\&quot;,\&quot;object_full_path\&quot;:\&quot;MY_CATALOG.MY_SCHEMA\&quot;},{\&quot;type\&quot;:\&quot;unity_catalog_privileges\&quot;,\&quot;privileges\&quot;:[\&quot;SELECT\&quot;,\&quot;MODIFY\&quot;],\&quot;object_type\&quot;:\&quot;TABLE\&quot;,\&quot;object_full_path\&quot;:\&quot;MY_CATALOG.MY_SCHEMA.MY_TABLE\&quot;}]&#39;&quot;<br>    }<br>  ],<br>  &quot;url&quot;: &quot;https://SERVER_ENDPOINT/zerobus/v1/tables/MY_CATALOG.MY_SCHEMA.MY_TABLE/insert&quot;,<br>  &quot;pipeline&quot;: &quot;main&quot;<br>}</pre><p>The most important settings are:</p><ul><li>format: &quot;json_array&quot;, because Zerobus expects a JSON list of records</li><li>authType: &quot;oauth&quot;, because Cribl needs to obtain a Databricks bearer token first</li><li>loginUrl, which points to the Databricks workspace OAuth endpoint</li><li>resource and authorization_details, which identify the Zerobus API and the Unity Catalog privileges</li><li>url, which points to the Zerobus endpoint and fully qualified table</li><li>onBackpressure: &quot;block&quot;, which prevents Cribl from silently dropping data when the downstream service applies backpressure</li><li>The retry settings for transient HTTP responses such as 408, 429, and 5xx responses</li></ul><p>The custom* fields are used for Cribl’s custom payload format. In this configuration, format is json_array, so the important thing is that each event is serialized as an object matching the target table schema. If you change the format to custom, make sure the batch expression creates valid JSON and uses ${events} for the batched records.</p><h3>Why this approach worked</h3><p>This approach removes the need to operate a separate message broker while still using a push-based ingestion path. Cribl batches the events and sends them over HTTPS to Zerobus. Zerobus authenticates the request, checks the records against the Unity Catalog table, acknowledges durable ingestion, and writes the data to Delta.</p><p>There are still some things to monitor. These include rejected requests, authentication failures, HTTP 429 responses, retry volume, Cribl backpressure, and Zerobus usage and billing. The batch size and flush interval should also be tested against the actual workload. A one-second flush interval does not guarantee one-second end-to-end latency.</p><p>The main lesson for me was that the difficult part was at the boundary between the two products. Cribl could already send HTTP and OAuth requests, and Databricks already had a REST ingestion API. The work was mapping the authentication fields, endpoint construction, permissions, payload format, retries, and backpressure behavior between the two systems.</p><p>This solution also shows that it is possible that with any Webhook that allows Oauth configuration, this should work there as well.</p><p>A special thanks to Duncan Turnbull at Cribl for his assistance in getting the last touches in place!</p><p><em>Disclaimer: This article is strictly my own thoughts and opinions and may not reflect those of my employer.</em></p><h3>References</h3><ul><li><a href="https://cribl.io/">Cribl</a></li><li><a href="https://docs.cribl.io/stream/4.3/destinations-webhook/">Cribl Webhook destination</a></li><li><a href="https://docs.cribl.io/stream/4.3/destinations-azure-event-hubs/">Cribl Azure Event Hubs destination</a></li><li><a href="https://docs.cribl.io/stream/4.3/destinations-kafka/">Cribl Kafka destination</a></li><li><a href="https://learn.microsoft.com/en-gb/azure/databricks/ingestion/cloud-object-storage/auto-loader/">Databricks Auto Loader</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/concepts">Databricks Structured Streaming concepts</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/real-time/concepts">Databricks Structured Streaming real-time mode</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-overview">Zerobus Ingest overview</a></li><li><a href="https://learn.microsoft.com/en-gb/azure/databricks/ingestion/zerobus-ingest">Use the Zerobus Ingest connector</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/variant-type">Databricks VARIANT type</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbf3e6f94e3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streamlining Data Ingestion: Harnessing the Power of Databricks Auto Loader and Serverless…]]></title>
            <link>https://bryanmcmaster.medium.com/streamlining-data-ingestion-harnessing-the-power-of-databricks-auto-loader-and-serverless-72be9775c511?source=rss-9482c23f774d------2</link>
            <guid isPermaLink="false">https://medium.com/p/72be9775c511</guid>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[medallion-architecture]]></category>
            <category><![CDATA[databricks-workflows]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Bryan McMaster]]></dc:creator>
            <pubDate>Sun, 20 Oct 2024 16:52:43 GMT</pubDate>
            <atom:updated>2024-10-20T22:51:49.924Z</atom:updated>
            <content:encoded><![CDATA[<h3>Streamlining Data Ingestion: Harnessing the Power of Databricks Auto Loader and Serverless Workflows</h3><p>Remember back in the day (like less than 6 months ago) when you had to go through so much work to ingest your data into the bronze tables of your Delta lake? You may have even set up a complicated metadata driven approach with YAML files configuring all the rules and logic. This would include many parameter driven notebooks, setting up infrastructure and figuring out the right settings to optimize performance and run it all asynchronously.</p><p>In this post I am going to show how to quickly low-code a solution to ingest many different files into the Bronze layer with a combination of several Databricks features that include Autoloader, Parameterized notebooks and the new For Each task in Workflows, running completely in a powerful serverless infrastructure managed by Databricks</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N8UfeCaDfm40eASUAOlUrw.jpeg" /><figcaption>A bucket filled with data</figcaption></figure><h3>The Tools</h3><h4>Serverless Workflows</h4><p>Remember the days of manually configuring clusters and praying to the compute gods for optimal performance? Well, those days are gone, and it’s all thanks to serverless compute for Workflows. This feature is like having a tiny, hyper-efficient IT department living inside your Databricks workspace. This is because serverless workflows include automatic scaling and simplified resource management, all with the same support for many different types of tasks, including the new For Each task.</p><p>The recently introduced For Each task added to Workflows is quite amazing, it allows you to loop over dynamic parameters asynchronously, even calling the same parameterized notebook over and over. This dramatically improves and simplifies tasks, while increasing scalability.</p><h4>Auto Loader</h4><p>For those not in the know, Auto Loader is a powerful feature that simplifies the process of ingesting data from cloud storage into Delta Lake tables. Its the perfect feature for our low-code solution, as it automatically processes new files as they arrive, and has a robust capability to detect schema evolution. So when data changes over time it can add these new fields to the target table, and even rescue unexpected data for manual review. This flexibility ensures that data pipelines remain resilient and adaptable to change and can scale to process billions of records.</p><h3>Example</h3><h4>Use Case</h4><p>The marketing team is spending way too much money on pushing promotions to email and other channels without understanding their customers needs. The Data Science team is super excited to take on solving the problem but they need your help and fast, they don’t have access to the data, its all sitting out in your cloud buckets in JSON, as a matter of fact there are about 100 different types of json files that must be put into a more structured format, and joined together. Enter the aforementioned autoloader and workflows.</p><h4>The Solution</h4><p>We will start by creating a single notebook with a few commands that will drive the creation and modification of the bronze tables, as well as inserting all of the data from the JSON tables.</p><pre>file_name = dbutils.widgets.get(&quot;file_name&quot;)<br> df = (spark.readStream<br>  .format(&quot;cloudFiles&quot;)<br>  .option(&quot;cloudFiles.format&quot;, &quot;json&quot;)<br>  .option(&quot;cloudFiles.schemaLocation&quot;, &quot;/Volumes/ingestion_demo/schemas/{file_name}&quot;.format(file_name=file_name))<br>  .option(&quot;encoding&quot;, &quot;utf-8&quot;) <br>  .option(&quot;multiline&quot;, &quot;true&quot;) <br>  .option(&quot;cloudFiles.inferSchema&quot;, &quot;true&quot;)<br>  .option(&quot;cloudFiles.inferColumnTypes&quot;, &quot;true&quot;)<br>  .load(&quot;/your_cloud_bucket/ingestion_demo/raw_files_{file_name}&quot;.format(file_name=file_name))<br>)<br><br>(df.writeStream <br>.option(&quot;mergeSchema&quot;, &quot;true&quot;)<br>.option(&quot;checkpointLocation&quot;, &quot;/Volumes/bryan_mcmaster/ingestion_demo/checkpoint/{file_name}&quot;.format(file_name=file_name))<br>.trigger(availableNow=True)<br>.toTable(&quot;your_catalog.ingestion_demo.{file_name}&quot;.format(file_name=file_name))<br><br>)</pre><p>The above code does the following:</p><ul><li>Gets a parameter from a widget called “file_name”</li><li>Takes in a parameter for the name of a file, this can take in any number of parameters depending on the complexity of your file storage</li><li>With the keyword “cloudfiles” we are telling databricks to use Auto Loader</li><li>We do some housekeeping work in reading the files by specifying the format, the schema location, telling it to infer the schema and column types and finally the location of your files to process</li><li>Now after reading the files we are going to write them to our bronze table</li><li>The mergeschema option tells databricks to automatically add new columns to our table</li><li>We need to point to a location for our checkpoint</li></ul><p>With our code now complete, let’s turn to our attention to setting up the workflow and our configuration</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lKy4__Tjjc9kClcMBCOdwg.png" /><figcaption>Completed example workflow</figcaption></figure><p>Setting up the workflow is easy, but first we need a configuration, since our ingestion notebook requires a name for each file, we can create a table of those values and read them into a list of taskValues using dbutils from that in a notebook, here I have called the notebook get_configs:</p><pre>df = spark.read.table(&quot;bryan_mcmaster.ingestion_demo.configuration&quot;)<br>dbutils.jobs.taskValues.set(key = &quot;config&quot;, value = df.toPandas()[&#39;file&#39;].to_list())</pre><p>Then we will go into workflows and create a new Job, the first is to create a task to call the get_config notebook, and retrieve the parameters. Note that we want to select Serverless for compute if this isn’t automatically selected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P2RhXaiV6cgp-SGliB46ZQ.png" /><figcaption>Setting up the first task to get our parameters from the get_configs notebook</figcaption></figure><p>Now, for our For Each task, this is where the real magic happens, create a new Task by clicking Add Task and select the task type as a “For each”. Here you will reference the parameters from the notebook setting the Inputs field, note that the concurrency field is necessary in order to run the task asynchronously , so set this to at least the number of files you need to process at a time, this can be set to up to 100.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I5sxcopwDF72e4T2601v2Q.png" /><figcaption>Setting up the For Each task</figcaption></figure><p>Next we will add a task within the For Each task by clicking “Add a task to loop over”, we will create a notebook task and call our notebook we created earlier, we will add a parameter to this with the name of our widget called “file_name” and the value is set to our input value of the For Each task</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6fKEDA-kyfKiGIPXuj5nNw.png" /><figcaption>Adding a task within the For Each task</figcaption></figure><p>Now we can do the first run of the Job, creating all of the bronze tables automatically. Keep in mind, if any of these files change the schema in the future, we will automatically capture these in the bronze tables for future use in our downstream tables and logic.</p><p>When we click on the job run, and click on the get_parameters_iteraton task to see the tasks that ran in parallel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TN3ylrP_t-gYk074VQe3Ig.png" /><figcaption>The Job run</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ogTx7_tKru6jy-IYDoV9iw.png" /><figcaption>Showing each task that ran concurrently</figcaption></figure><p>We can see from the tasks that it used the input parameters and created 3 tables, users, clickstream and orders. The jobs took less than a minute from the time I clicked Run Job till the end, all without the need for setting up any infrastructure, running completely on serverless and creating my bronze layer that will always be updated with any new columns using Auto Loader.</p><p>Now we can add tasks for each of the more complicated silver logic, this is where our data is cleaned up, joined together and and ready for our downstream ML notebooks. In the end your Job may look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lKy4__Tjjc9kClcMBCOdwg.png" /></figure><h4>Conclusion</h4><p>By combining Databricks Auto Loader, serverless compute, and the new For Each task in Workflows, I’ve demonstrated a streamlined, low-code solution for data ingestion into the Bronze layer. This approach eliminates complex setups, automates schema evolution, and leverages serverless infrastructure, allowing data teams to focus on insights rather than pipeline management. The result is a flexible, scalable, and efficient data ingestion process that can easily adapt to changing business needs, setting the stage for more agile data operations.</p><p><strong><em>If you find this article interesting, you are interested in data, leadership and retail technology please be sure to follow me here and on LinkedIn: </em></strong><a href="https://www.linkedin.com/in/bryanmcmaster/"><strong><em>Bryan McMaster</em></strong></a></p><p><strong><em>Disclaimer: This article is strictly my own thoughts and opinions and may not reflect those of my employer</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72be9775c511" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adopting a Data Strategy]]></title>
            <link>https://bryanmcmaster.medium.com/adopting-a-data-strategy-414354bff3e4?source=rss-9482c23f774d------2</link>
            <guid isPermaLink="false">https://medium.com/p/414354bff3e4</guid>
            <category><![CDATA[data-strategy]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Bryan McMaster]]></dc:creator>
            <pubDate>Mon, 05 Feb 2024 03:42:48 GMT</pubDate>
            <atom:updated>2024-02-05T03:46:05.900Z</atom:updated>
            <content:encoded><![CDATA[<p>Building a world class data team involves more than technology and having a unified data platform. After a long career as a leader and data practitioner, now as a Solutions Architect for <a href="https://www.databricks.com/">Databricks</a> it is difficult for me to see customers that don’t have a clearly defined data strategy. This article is meant to convey the importance of data strategy, and an approach to get something off the ground quickly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CQZln0lydbxRBmQI" /><figcaption>Photo by <a href="https://unsplash.com/@mvdheuvel?utm_source=medium&amp;utm_medium=referral">Maarten van den Heuvel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>How will you know you don’t have a Data Strategy?</h3><p>When you don’t have a Data Strategy, there are some key indicators that will tell you this:</p><ul><li><strong>Silos:</strong> Teams work in silos across different business areas with disparate data sources with no unified approach to both the data and collaboration</li><li><strong>Shadow IT: </strong>Department heads and practitioners across the organization adopting new technology fit for purpose because “IT doesn’t move fast enough”</li><li><strong>BI Tool Sprawl: </strong>More than one BI tool used in silos or across departments</li><li><strong>Excel is the BI Tool</strong>: Don’t get me wrong, Excel is never going away, but it will never offer the value of a good Data Warehouse and BI tool</li><li><strong>Governed Metrics: </strong>There is no way to govern your metrics and KPI’s, these are calculated separately in different reports, dashboards, spreadsheets, there is no way to show lineage</li><li><strong>No Perceived Value: </strong>The C-Suite still has not seen any value in data investments</li><li><strong>Project Management:</strong> Analytics requests are handled through hallway conversations or one-off conversations</li></ul><h3><strong>Establishing a Data Strategy</strong></h3><p>Clearly understanding the business strategy is key to the success of yourself, the team, department and the business. Whether you are just starting your career or are the Chief Data Officer. You must understand the strategic goals of the organization, and then create and adjust your strategy to be successful. Strategies are built around how they will accomplish the mission in order to get to the future state vision.</p><p>The strategy will likely and should change with the external factors, companies that do not adapt their strategy and fail to innovate will become a thing of the past. There are two good examples along this spectrum, Blockbuster and Best Buy.</p><p>Blockbuster Video had over 9000 stores and a valuation of over 3 billion dollars, however with competitors like Netflix and advancements in streaming they didn’t shift their strategy to keep up with the pace of innovation and are now just a memory except for their one remaining holdout store in Bend Oregon.</p><p>In 2012 Best Buy took over a billion dollar hit to their revenue in a single quarter because of fierce competition from Amazon and other online retailers. Best Buy, quickly shifted their strategy making each store a mini warehouse where they could satisfy online and store pickup quickly and effciently for their customers. Best Buy today is still strong today and will not go anywhere anytime soon.</p><p>If you are leading a data team, you must have a strategy that aligns with the business strategy. The diagram below illustrates this point, the business and the vision, mission etc. is not one dimensional. The alignment with business strategy only can happen if the business accepts it and they both align.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3gQau5bjwTnN53EudKNRLQ.png" /><figcaption>Business Alignment with Data Strategy</figcaption></figure><h4>How do you create a Data Strategy?</h4><p>My suggestion is to first start your interview process. You will do this buy interviewing leaders in different business areas of your organization. Setup a way that will capture the leader, the department and how they run their business area. Try to understand their strategy, what is preventing them from helping to fulfill the vision and mission and what is already working well and what is barely or not working at all. It’s easy to get into technical aspects, they will even lean in this direction since they know you are the data leader, but what we are trying to do here is understand how they are achieving or not achieving their goals and tactics and what can be done to improve. It sometimes is easier to communicate this in terms of people and process since these are the resources that will generally be at top of mind. Later you can use technology to redefine people and process.</p><p>Once you have documented each business area you should start to see a picture of the similarities in deficiencies and new use cases. Now it’s time to look at your current data landscape. In it’s current state, how does it help fulfill the goals and objectives of the business? Then, plot out what the future state looks like. What does the future hold for setting up the business for success using data?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*y-5ZSVi9MhlNsMvkMkB0Yw.png" /><figcaption>Data Strategy is about People, Process and Technology</figcaption></figure><p>The gaps between current and future state will map to your data strategy. This does not mean you should just look at investments in technology. You will also need to understand what skills do you need on your team and in the business. This may involve hiring new people in both areas and getting appropriate training. Creating or refining existing business processes will also be key to the success of the strategy. The chosen technologies will serve to augment the people and process. The People, Process and Technology should be as detailed as possible for each area. Finally, you are ready to create a strategy from the gaps in current and future state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*RQXxKzek3pp-eXSeY0YahQ.png" /><figcaption>People, Process &amp; Technology</figcaption></figure><p>Once a draft has been created, it’s important to communicate this to the stakeholders and business leaders you interviewed earlier, as well as your team.</p><blockquote>You know that you have communicated your strategy effectively when everyone can finish each other’s sentences</blockquote><p>This approach will be sure to create alignment with the business as well as help you create a roadmap which will include your goals and tactics to achieve the strategy. Be sure that you communicate and are as transparent as possible to your team and everyone around you. When you have completed the setup of the data strategy in your organization, your team, your peers and the business should be able to finish each other’s sentences in understanding the strategy.</p><p>This article is short and the above process can get overwhelming. There are entire programs, trainings, and literature devoted to this topic. Considering reading more on this topic or hiring a consultant to help you design a data strategy. However, be cautious about hiring consultants to tell you the strategy, this should come from you and your work with the business. This is as much about building relationships with your peers as it is creating a data strategy.</p><p><strong>Please feel free to offer any comments, your advice to others and to me about creating an effective data strategy.</strong></p><p><strong><em>If you find this article interesting, you are interested in data, leadership and technology please be sure to follow me here and on LinkedIn: </em></strong><a href="https://www.linkedin.com/in/bryanmcmaster/"><strong><em>Bryan McMaster</em></strong></a></p><p><strong><em>Disclaimer: This article is strictly my own thoughts and opinions and may not reflect those of my employer</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=414354bff3e4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Data Science Story: Squirrel Friend]]></title>
            <link>https://bryanmcmaster.medium.com/a-data-science-story-squirrel-friend-41a5a4524edb?source=rss-9482c23f774d------2</link>
            <guid isPermaLink="false">https://medium.com/p/41a5a4524edb</guid>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Bryan McMaster]]></dc:creator>
            <pubDate>Tue, 14 Feb 2023 00:39:50 GMT</pubDate>
            <atom:updated>2023-02-14T01:06:07.578Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Data Science to find a new friend a Central Park</h4><p>This is a data science story about how to make friends with squirrels. This comes with perfect timing since Valentine’s Day is near and those without a significant other might still be able to make a new friend in New York’s Central Park. You just need to know where to look.</p><p><strong>Disclaimer: Before I get too far I need to make sure that everyone knows that this is strictly data science satire, using a real world dataset. I do not condone going into Central Park to try to make a friend with a squirrel. Squirrels approach humans to get handouts of food, leaving them to rely on us for their food. Please do not feed the squirrels and do not approach them.</strong></p><p>If while walking in Central Park, how do we discern the friendly squirrels from the not so friendly ones and where they might be located? Sometimes it can be obvious, they might approach you, expecting a handout of sorts (note: This is a squirrel friend), plus Central Park is huge, you probably need to know the best location so you aren&#39;t wandering around everywhere.</p><p>How do we find this information? We will need to start with a dataset. If you didn’t already know, New York City has some pretty awesome real- world datasets. The one we are using to find our friend is located here: <a href="https://data.cityofnewyork.us/Environment/2018-Central-Park-Squirrel-Census-Squirrel-Data/vfnx-vebw">Central Park Squirrel Census</a>. This data was collected by volunteers that collected over 30 different attributes regarding the squirrel population in Central Park.</p><p>Along with this dataset, we are going to do the following in this story:</p><ul><li>Investigate the dataset and see what kind of information it has about squirrels</li><li>Build and take a look at some data plotted against maps of Central Park</li><li>Apply some modification to transform our data</li><li>Create a model to predict where in Central Park you would likely find a squirrel friend</li><li>Talk a little about the accuracy of the model</li><li>Plot the final output on a map</li></ul><figure><img alt="A squirrel eating a nut" src="https://cdn-images-1.medium.com/max/1024/0*8aaCEp_dakK6BfP_" /><figcaption>Photo by <a href="https://unsplash.com/@sparrow24?utm_source=medium&amp;utm_medium=referral">Włodzimierz Jaworski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Mind you, this is a story about using data science to solve a fun problem. I will not go too deep into technical jargon here so that people like my mom can understand data science. I will show snippets of code however for you techy readers and provide a <a href="https://github.com/Bryan-McMaster/squirrelfriend">link here to the github repository</a></p><h4>Investigate the Squirrel dataset</h4><p>We start out by downloading the squirrel dataset and loading it into a <a href="https://towardsdatascience.com/everything-you-need-to-know-about-jupyter-notebooks-10770719952b">Python notebook</a>. Python is a coding language used in data science and not an actual snake. I think that our squirrels would not be so friendly if we brought a python around. After loading, we notice that the columns represent the data that has been collected about the squirrels and we have a total of 3023 surveys.</p><pre># load the squirrel data into a pandas dataframe and take a looks at the columns<br>df_squirrel = pd.read_csv(&quot;squirrel_data.csv&quot;)<br>print(&#39;Row count:&#39;,len(df_squirrel))<br>print(&#39;Columns:\n&#39;,df_squirrel.dtypes)<br><br>Row count: 3023<br>Columns:<br>X                                             float64<br>Y                                             float64<br>Unique Squirrel ID                             object<br>Hectare                                        object<br>Shift                                          object<br>Date                                            int64<br>Hectare Squirrel Number                         int64<br>Age                                            object<br>Primary Fur Color                              object<br>Highlight Fur Color                            object<br>Combination of Primary and Highlight Color     object<br>Color notes                                    object<br>Location                                       object<br>Above Ground Sighter Measurement               object<br>Specific Location                              object<br>Running                                          bool<br>Chasing                                          bool<br>Climbing                                         bool<br>Eating                                           bool<br>Foraging                                         bool<br>Other Activities                               object<br>Kuks                                             bool<br>Quaas                                            bool<br>Moans                                            bool<br>Tail flags                                       bool<br>Tail twitches                                    bool<br>Approaches                                       bool<br>Indifferent                                      bool<br>Runs from                                        bool<br>Other Interactions                             object<br>Lat/Long                                       object<br>dtype: object</pre><p>This looks interesting. Normally we would use a method to get the statistical summary of the dataset, but there are only four numerical values. Most of this is location information, true/false and handwritten notes. Note that the definitions of all the fields are available <a href="https://data.cityofnewyork.us/Environment/2018-Central-Park-Squirrel-Census-Squirrel-Data/vfnx-vebw">here</a>. There are some columns that look like they would be great to do some further analysis on: X (Longitude), Y (Latitude) and Approaches. Let’s look at these first.</p><p>Just a little bit of code and we can see the percentage of squirrels that approach as well as visualize these friendly squirrels. This tells us that about 6% of the time a squirrel approaches a human. This looks like we would want to predict this if we wanted to find a new friend.</p><pre># Calculate percentage of squirrels that approach<br>counts = df_squirrel[&#39;Approaches&#39;].value_counts()<br>perc_squirrel = &#39;{:.2%}&#39;.format(counts[1]/counts.sum())<br>print(f&#39;Percent of squirrels approaching:{perc_squirrel}&#39;)<br># Show a plot of squirrels that approach<br>sns.countplot(x=&#39;Approaches&#39;,data=df_squirrel,palette=&#39;hls&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*Q-huysJVKAn9aOomPzoWCA.png" /></figure><p>Now let’s see what “Approaches” looks like using our coordinates of X and Y. To see this in the perspective of the park I am doing an overlay of a screenshot I took from Google Maps of Central Park. I am sure there is another easier and better formatted way to do this using maps but for now this is what you get. Afterall, we are in a hurry here, Valentine’s Day is fast approaching and time is slipping away!</p><pre>sq_groups = df_squirrel.groupby(&#39;Approaches&#39;)<br># tried to do a map overlay.<br>extent = df_squirrel[&#39;X&#39;].min()   , df_squirrel[&#39;X&#39;].max()   , df_squirrel[&#39;Y&#39;].min() , df_squirrel[&#39;Y&#39;].max() <br>im = plt.imread(&#39;cp14.png&#39;)<br>implot = plt.imshow(im,interpolation=&#39;nearest&#39;,<br>                 extent=extent)<br>for name, group in sq_groups:<br>    plt.plot(group.X,group.Y,marker=&#39;o&#39;,linestyle=&#39;&#39;,markersize=3,label=name, alpha=.5)<br>plt.title(&#39;Squirrel Locations In Central Park - NYC&#39;)<br><br>ax = plt.gca()<br>ax.axes.xaxis.set_ticklabels([])<br>ax.axes.yaxis.set_ticklabels([])<br>ax.legend(labels=[&#39;Not A Friend&#39;,&#39;Friend&#39;])<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*1TrPXb4d2OpzXHns0aeBNg.png" /><figcaption>A plot overlaying a map of Central Park</figcaption></figure><p>As can be seen above, the orange dots represent where a squirrel approached a human during the survey. One might ask, why don’t I just go where those dots are concentrated and meet a squirrel there? Why am I going to research this even further? Well, the short answer is — This is a data science story, and we are interested in how these things work, also, there may other characteristics that make the squirrels more likely to approach, why just leave it to pure chance?</p><h4>Transforming our data</h4><p>So, when we refer to characteristics, we are talking about things that are in the data that influence the likelihood of an event occurring, in this case an approaching squirrel. From this point on we will call these data “features”, and for the computer to use these we need to do some feature engineering. The first feature we will engineer is our longitude and latitudes coordinates. We are going to group these into pre-defined clusters (geographic areas) in the park using a popular algorithm called K-Means.</p><pre>coordinates= df_squirrel[[&#39;X&#39;,&#39;Y&#39;]]<br>xloc, location = kmeans2(whiten(coordinates), 8, seed=50 )  # iter = 20,<br><br>extent = df_squirrel[&#39;X&#39;].min()   , df_squirrel[&#39;X&#39;].max()   , df_squirrel[&#39;Y&#39;].min() , df_squirrel[&#39;Y&#39;].max() <br>im = plt.imread(&#39;cp14.png&#39;)<br>implot = plt.imshow(im,interpolation=&#39;nearest&#39;,<br>                 extent=extent)<br>plt.scatter(coordinates[&#39;X&#39;], coordinates[&#39;Y&#39;], c=location, alpha=.2)<br><br>df_squirrel[&#39;cluster&#39;] = location<br>for i in np.unique(location):<br>    coords = df_squirrel[(df_squirrel.cluster == i)]<br>    long = coords[&#39;X&#39;].median()<br>    lat = coords[&#39;Y&#39;].median()<br>    plt.text(x=long,y=lat,s=i, <br>          fontdict=dict(color=&#39;red&#39;,size=10),<br>          bbox=dict(facecolor=&#39;yellow&#39;,alpha=0.5))<br>plt.title(&#39;Assigned Clusters to Geo Coordinates Central Park&#39;)<br><br>ax = plt.gca()<br>ax.axes.xaxis.set_ticklabels([])<br>ax.axes.yaxis.set_ticklabels([])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*-9wjfS08HqwiepyNJZ1xjA.png" /></figure><p>We have identified four clusters, and generated the map above, and we have also updated the data to include a new column for the cluster.</p><p>The next step is to convert the features columns, so that the computer can properly use them. When we do this, the clusters numbers in our rows become columns, so for every value of a cluster, we are going to make it a column and assign whether it’s true or false:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*SlDP9pB4uhy2kI9GjcgSFA.png" /></figure><pre>cat_vars = []<br>for col_name in df_squirrel.columns:<br>    if col_name not in [&#39;Lat/Long&#39;,&#39;Other Interactions&#39;,&#39;Other Activities&#39;,&#39;Specific Location&#39;,&#39;Location&#39;,&#39;Color notes&#39;,&#39;X&#39;,&#39;Y&#39;,&#39;Unique Squirrel ID&#39;, &#39;Hectare&#39;, &#39;Shift&#39;, &#39;Date&#39;,&#39;Hectare Squirrel Number&#39;,&#39;Combination of Primary and Highlight Color&#39;,&#39;Above Ground Sighter Measurement&#39;]:<br>        cat_vars.append(col_name)<br>df_final_squirrels = pd.get_dummies(df_squirrel.astype(str)[cat_vars],drop_first=True)</pre><p>We have excluded some columns above, because it’s really hard for a computer to take text in our “Other Interactions” column like “looked down on me like her imperial subject” and turn that into it can use unless we did more feature engineering. By the way, this squirrel acted more like a cat, only it decides when it wants to be your friend.</p><h4>Creating the Model</h4><p>Now for using our modified features to build a model. In data science and machine learning, a model is something that is trained to make a prediction about something, in this case we are trying to predict the probability that a squirrel will approach you in Central Park and be your friend. From this point on we will call this approach from a squirrel our “Dependent” variable, we are going to try to predict this variable using our features we have from the rest of the data. Another name for features is called Independent variables.</p><p>The type of model we are building is called a logistic regression, where we try to predict a value between 0 and 1, or in other words 0% to 100% that something will happen. More can be read here on <a href="https://en.wikipedia.org/wiki/Logistic_regression">logistic regression</a>.</p><p>For this type of model to perform well and predict with more accuracy, we need to balance the data. If you remember earlier, we found that about 6% of squirrels approached a human, if we used this to predict we likely would not have enough data points to make an accurate prediction. We need to add some synthetic data to this to make the dataset appear to have 50% of squirrels approaching.</p><p>In training models it’s important that we set aside some of the data to test the prediction, this is our “unseen” data, or data that the training didn’t see. An example of this would be learning about all the squirrels and knowing which ones would be friends instead of using a subset to learn and make predictions later. We want to be sure that if new squirrels come along that we can predict with some accuracy if they would be friends.</p><pre>#Balance our data<br>X = df_final_squirrels<br><br>y = df_squirrel[&#39;Approaches&#39;]<br>from imblearn.over_sampling import RandomOverSampler<br>os = RandomOverSampler(random_state=36)<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)<br>columns = X_train.columns<br>os_data_X,os_data_y=os.fit_resample(X_train, y_train)<br>os_data_X = pd.DataFrame(data=os_data_X,columns=columns )<br>os_data_y= pd.DataFrame(data=os_data_y,columns=[&#39;Approaches&#39;])<br># we can Check the numbers of our data<br><br>print(&quot;length of oversampled data is &quot;,len(os_data_X))<br>print(&quot;Number of no Approaches in oversampled data&quot;,len(os_data_y[os_data_y[&#39;Approaches&#39;]==0]))<br>print(&quot;Number of Approaches&quot;,len(os_data_y[os_data_y[&#39;Approaches&#39;]==1]))<br><br>length of oversampled data is  3990<br>Number of no Approaches in oversampled data 1995<br>Number of Approaches 1995</pre><p>Now we will do a test run of the model with the balanced data to see how it scores.</p><pre>import statsmodels.api as sm<br><br>y=os_data_y[&#39;Approaches&#39;]<br>X=os_data_X.drop(columns=[&#39;Approaches_True&#39;])<br><br>logit_model=sm.Logit(y.astype(float),X.astype(float))<br>result=logit_model.fit()<br>display(result.summary2())</pre><p>We did this test run so we can see the summary output below. This has a lot of information and can be overwhelming. For this story I will not try to explain everything. However, I believe that I should explain the columns Coef. and P&gt;|z|since they should be understood at some level, and we will be using P&gt;|z|to filter out some of our features we will consider not useful for our calculations.</p><pre>=================================================================<br>Model:              Logit             Pseudo R-squared: 0.334    <br>Dependent Variable: Approaches        AIC:              3748.4344<br>Date:               2023-02-12 16:02  BIC:              3943.4724<br>No. Observations:   3990              Log-Likelihood:   -1843.2  <br>Df Model:           30                LL-Null:          -2765.7  <br>Df Residuals:       3959              LLR p-value:      0.0000   <br>Converged:          0.0000            Scale:            1.0000   <br>No. Iterations:     35.0000                                      <br>-----------------------------------------------------------------------------------------------------<br>                                             Coef.    Std.Err.     z     P&gt;|z|      [0.025     0.975]                     <br>-----------------------------------------------------------------------------------------------------<br>Age_Adult                                    -1.7813    1.1027  -1.6154 0.1062     -3.9425     0.3799<br>Age_Juvenile                                 -1.4752    1.1059  -1.3340 0.1822     -3.6427     0.6922<br>Age_nan                                      -3.0685    1.1328  -2.7087 0.0068     -5.2888    -0.8482<br>Primary Fur Color_Cinnamon                    0.0491    0.2751   0.1784 0.8584     -0.4901     0.5883<br>Primary Fur Color_Gray                       -0.4502    0.2250  -2.0013 0.0454     -0.8912    -0.0093<br>Primary Fur Color_nan                        -0.4406    0.3875  -1.1368 0.2556     -1.2001     0.3190<br>Highlight Fur Color_Black, Cinnamon         -16.9279     &gt; 100  -0.0017 0.9986      &lt; -100      &gt; 100<br>Highlight Fur Color_Black, Cinnamon, White    3.1517    1.1292   2.7911 0.0053      0.9385     5.3648<br>Highlight Fur Color_Black, White            -14.2729     &gt; 100  -0.0032 0.9974      &lt; -100      &gt; 100<br>Highlight Fur Color_Cinnamon                  3.1561    1.0877   2.9017 0.0037      1.0243     5.2879<br>Highlight Fur Color_Cinnamon, White           3.8229    1.0929   3.4979 0.0005      1.6809     5.9650<br>Highlight Fur Color_Gray                      2.8696    1.1073   2.5915 0.0096      0.6993     5.0398<br>Highlight Fur Color_Gray, Black             -27.3505     &gt; 100  -0.0000 1.0000      &lt; -100      &gt; 100<br>Highlight Fur Color_Gray, White               2.4513    1.1316   2.1663 0.0303      0.2334     4.6691<br>Highlight Fur Color_White                     2.2252    1.0897   2.0421 0.0411      0.0895     4.3610<br>Highlight Fur Color_nan                       2.0421    1.0874   1.8780 0.0604     -0.0891     4.1734<br>Running_True                                  0.4561    0.1038   4.3951 0.0000      0.2527     0.6595<br>Chasing_True                                 -0.5958    0.1789  -3.3303 0.0009     -0.9464    -0.2451<br>Climbing_True                                 0.1066    0.1107   0.9631 0.3355     -0.1103     0.3236<br>Eating_True                                   0.7614    0.0924   8.2384 0.0000      0.5803     0.9426<br>Foraging_True                                 0.7644    0.0942   8.1110 0.0000      0.5797     0.9491<br>Kuks_True                                    -0.6309    0.2768  -2.2793 0.0227     -1.1735    -0.0884<br>Quaas_True                                  -26.9041     &gt; 100  -0.0002 0.9998      &lt; -100      &gt; 100<br>Moans_True                                 -191.8740     &gt; 100  -0.0000 1.0000      &lt; -100      &gt; 100<br>Tail flags_True                               0.4784    0.1806   2.6484 0.0081      0.1244     0.8324<br>Tail twitches_True                            1.0066    0.1181   8.5219 0.0000      0.7751     1.2381<br>Indifferent_True                             -3.1070    0.1126 -27.5956 0.0000     -3.3277    -2.8864<br>Runs from_True                               -1.8164    0.1121 -16.2036 0.0000     -2.0361    -1.5967<br>cluster_1                                    -0.5122    0.1191  -4.2992 0.0000     -0.7458    -0.2787<br>cluster_2                                     0.0740    0.1110   0.6668 0.5049     -0.1436     0.2916<br>cluster_3                                    -1.0714    0.1403  -7.6357 0.0000     -1.3464    -0.7964<br>=====================================================================================================</pre><p>Models use math. Math can at times get quite complex for machine learning, but for this interpretation of the logistic regression we will make it as easy as possible and try to leave the nuances out. For those interested in a good interpretation of the Coef (Coefficients) please take a look at this <a href="https://towardsdatascience.com/a-simple-interpretation-of-logistic-regression-coefficients-e3a40a62e8cf">post</a>. The coefficients are what is used to calculate the probability in our model. For each of our features we multiply that feature by the coefficient. A simplified example of this is below, the resulting answer is if you have a squirrel friend based on the math.</p><p>Squirrel Friend = Running_True * 0.4561 + Kuks_True * -0.6309 …etc…..</p><p>We are only going to use P-Value &lt; .05 to filter out our features. But why? Again, for a more comprehensive explanation, please see this article <a href="https://towardsdatascience.com/a-simple-interpretation-of-p-values-34db3777d907">here</a>. The P-Value provides the probability of getting a result much different than the one we observed. So, for each of the features, a lower value is better. Why did we choose .05? .05 is our level of significance, .05 is a common threshold in statistics. Anything less than .05 means that our model fits the data that we have given well. Anything that is ≥ .05 we will remove. After applying the filter below, we now have nineteen features.</p><pre>columns_to_use = result.pvalues[(result.pvalues &lt;.05)]<br>print(&#39;Our columns to use for predictions:&#39;)<br>columns_to_use.index<br><br>Our columns to use for predictions:<br>Index([&#39;Age_nan&#39;, &#39;Primary Fur Color_Gray&#39;,<br>       &#39;Highlight Fur Color_Black, Cinnamon, White&#39;,<br>       &#39;Highlight Fur Color_Cinnamon&#39;, &#39;Highlight Fur Color_Cinnamon, White&#39;,<br>       &#39;Highlight Fur Color_Gray&#39;, &#39;Highlight Fur Color_Gray, White&#39;,<br>       &#39;Highlight Fur Color_White&#39;, &#39;Running_True&#39;, &#39;Chasing_True&#39;,<br>       &#39;Eating_True&#39;, &#39;Foraging_True&#39;, &#39;Kuks_True&#39;, &#39;Tail flags_True&#39;,<br>       &#39;Tail twitches_True&#39;, &#39;Indifferent_True&#39;, &#39;Runs from_True&#39;, &#39;cluster_1&#39;,<br>       &#39;cluster_3&#39;],<br>      dtype=&#39;object&#39;)</pre><p>Now we are going to run the logistic regression again with our smaller set of features.</p><pre>from sklearn.linear_model import LogisticRegression<br>from sklearn import metrics<br>X_train, X_test, y_train, y_test = train_test_split(X[columns_to_use.index].astype(float), y.astype(float), test_size=0.3, random_state=0)<br>logreg = LogisticRegression()<br>logreg.fit(X_train, y_train)<br><br>y_pred = logreg.predict(X_test)<br>print(&#39;Accuracy of logistic regression classifier on test set: {:.2f}&#39;.format(logreg.score(X_test, y_test)))<br>print(&#39;Accuracy of logistic regression classifier on train set: {:.2f}&#39;.format(logreg.score(X_train, y_train)))<br><br>Accuracy of logistic regression classifier on test set: 0.77<br>Accuracy of logistic regression classifier on train set: 0.76</pre><h4>Scoring the model</h4><p>We see the results here of our accuracy on the test data is .77 out of 1.0, which means that 77% of the time our model is making a correct prediction. This isn’t bad. How was this 77% calculated? The model basically kept track of four different numbers, called a confusion matrix. These numbers are below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*wISOXxQvHP3po4z3rbe6Dg.png" /><figcaption>confusion matrix</figcaption></figure><p>The white boxes are where the model accurately predicted the correct value, the shaded is where the model got it wrong. So now, the model does some simple math to show our accuracy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*2VSzudXhhmGREywokEd74A.png" /><figcaption>Calculating the accuracy</figcaption></figure><p>Now we will create a plot to understand another metric: ROC Curve. The ROC Curve shows us how skilled the model is at making predictions.</p><pre>from sklearn.metrics import RocCurveDisplay<br><br>y_pred_proba = logreg.predict_proba(X_test)[::,1]<br><br><br><br>RocCurveDisplay.from_predictions(<br>    y_test,<br>    y_pred_proba,<br>    name=&quot;Skilled Prediction&quot;,<br>    color=&quot;darkorange&quot;,<br>)<br>plt.plot([0, 1], [0, 1], &quot;k--&quot;, label=&quot;Not Skilled (AUC = 0.5)&quot;)<br>plt.axis(&quot;square&quot;)<br>plt.xlabel(&quot;False Positive Rate&quot;)<br>plt.ylabel(&quot;True Positive Rate&quot;)<br>plt.title(&quot;ROC Curve:\nHow skilled is the model at making predictions?&quot;)<br>plt.legend()<br>plt.show()</pre><p>The orange curve in the plot below shows us the AUC (or area under the curve takes up 86% of the space of the plot. The higher the number, the better our model is at making predictions. I have drawn in a black dotted line that compares what a model would look like if it’s not skilled. An AUC of .5 would not be useful in telling us if we could find a squirrel friend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*AWPFdivYEcSCZ8bvarWaDw.png" /></figure><h4>Mapping our probabilities</h4><p>Finally, let’s apply our probabilities back to our original dataset and map these out.</p><pre>heatmap, xedges, yedges = np.histogram2d(xden, yden, bins=50)<br>extent = [xedges[0], xedges[-1], yedges[0], yedges[-1]]<br><br><br>plt.clf()<br><br>im = plt.imread(&#39;cp14.png&#39;)<br>implot = plt.imshow(im,interpolation=&#39;nearest&#39;,<br>                 extent=extent)<br>#cmap = clr.LinearSegmentedColormap.from_list(&quot;&quot;, [&quot;white&quot;,&quot;white&quot;,&quot;red&quot;,&quot;firebrick&quot;,&quot;darkred&quot;,&quot;black&quot;])<br><br>plt.imshow(heatmap.T, extent=extent, origin=&#39;lower&#39;, alpha=.6,interpolation = &#39;gaussian&#39; , cmap=&#39;Reds&#39;) <br>#plt.colorbar()<br>cbar = plt.colorbar()<br>cbar.set_ticks([*range(2000,10000,2000)])<br>cbar.set_ticklabels([&quot;20%&quot;, &quot;40%&quot;, &quot;60%&quot;, &quot;80%&quot;])<br><br>ax = plt.gca()<br>ax.axes.xaxis.set_ticklabels([])<br>ax.axes.yaxis.set_ticklabels([])<br><br>plt.title(&#39;% Chance of running into a squirrel friend in Central Park&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*qZfN7i2KD9rV-qTTcWeujg.png" /><figcaption>Central Park friendly squirrel density</figcaption></figure><p>We can see from this plot where we have a high degree of probability that we will find a squirrel friend in Central Park. Mostly concentrated in the lower half of the park with some smaller areas in the upper portion of the park.</p><h4>Conclusion</h4><p>Today I told a story about how data science can help us find a squirrel friend in New York’s Central Park. I have demonstrated the basics of:</p><ol><li>Analyzing the data through charts and maps</li><li>Why do we need to transform and do feature engineering on data</li><li>What is a model and how does it work</li><li>Interpreting the model</li><li>Checking the accuracy of the model</li><li>Showing our results so we can find a squirrel friend</li></ol><p><strong><em>If you find this article interesting, or if you are interested in data, leadership and technology please be sure to follow me here and on LinkedIn: </em></strong><a href="https://www.linkedin.com/in/bryanmcmaster/"><strong><em>Bryan McMaster</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41a5a4524edb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Quality and the Romans]]></title>
            <link>https://bryanmcmaster.medium.com/data-quality-and-the-romans-e41e8fa5d646?source=rss-9482c23f774d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e41e8fa5d646</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[history]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Bryan McMaster]]></dc:creator>
            <pubDate>Sun, 15 Jan 2023 21:49:04 GMT</pubDate>
            <atom:updated>2023-01-15T21:49:04.671Z</atom:updated>
            <content:encoded><![CDATA[<p>As a person curious about history, in particular history of the Romans I am fascinated at how well they built their empire. There are roads, aqueducts and buildings still functional today, more than 2000 years after they were built. How did these monuments withstand the test of time? There is one word for that: Quality. There are six core dimensions to data quality. How do we ensure that your data from cradle to grave follows these and other principles of quality and governance?</p><figure><img alt="At the Roman Colosseum" src="https://cdn-images-1.medium.com/max/1024/0*e5Yb7H0UN-68J7ue" /><figcaption>At the Roman Colosseum July 2021</figcaption></figure><p>Data quality is key to any company that creates any amount of data, and not just using it for analytics and data science. Quality keeps the business running properly, and with as little hiccups as possible. Quality keeps the lights on, and you having a full nights sleep at night while your business runs, just as it kept the Roman Empire running for hundreds of years.</p><p>There are six dimensions to data quality (<a href="https://a.co/d/8oDs0ls">DAMA-DMBOK 2nd Edition</a>):</p><ul><li>Accuracy: We measure the accuracy of data as the degree to which the data correctly describes the real world</li><li>Completeness: The proportion of data stored against the potential for 100%</li><li>Consistency: In data, we refer this to the absence of difference when comparing two or more representations of a thing against a definition</li><li>Validity: Data is valid if it conforms to the syntax (format, type, range) of its definition</li><li>Timeliness: In the world of data we refer to this as the degree to which data represent reality from the required point in time</li><li>Uniqueness: For Data we describe this as no entity instance (thing) will be recorded more than once based upon how that thing is identified</li></ul><p>If we look at Via Appia in Rome, or many of the roman highways we can see that the Romans followed a system of quality that would closely mirror these six dimensions. We do not expect our systems to last two thousand years, but what can we do so they last 3–5 years?</p><figure><img alt="Via Appia" src="https://cdn-images-1.medium.com/max/1024/0*Si8wRzZU8W3jY-Y7" /><figcaption>Photo by <a href="https://unsplash.com/@gabiontheroad?utm_source=medium&amp;utm_medium=referral">Gabriella Clare Marino</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>What if the business, data governance and engineering were all working together? The core benefits of having your data governance team tightly coupled with your engineering and business teams is that these data quality dimensions can be built-in to every application and they are not an after thought. So its always clean data in, and clean data out and not our often said “Garbage in, garbage out”. The business and product owners work with both the engineering and data team to communicate software requirements and data requirements .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*IrVVhCmt3vSK6q7PQGK0uw.png" /></figure><p>These data requirements and definitions should have at least the six dimensions of quality, but also contain everything across the data lifecycle, from cradle to grave that will include security, lineage, storage etc. This will ensure that you have the you have quality built-in to your systems for the business, analytics and data science.</p><p>I admire seeing these well architected systems that take are heavily designed around the quality of data that will be created, just as much as success if of the business. I believe these go hand in hand, just as much as the Romans would have placed the success of their empire on the quality put into their construction.</p><p><strong><em>If you find this article interesting, or if you are interested in data, leadership and technology please be sure to follow me here and on LinkedIn: </em></strong><a href="https://www.linkedin.com/in/bryanmcmaster/"><strong><em>Bryan McMaster</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e41e8fa5d646" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to become a successful data analyst: It is not only your hard skills!]]></title>
            <link>https://bryanmcmaster.medium.com/how-to-become-a-successful-data-analyst-it-is-not-only-your-hard-skills-97de79e04323?source=rss-9482c23f774d------2</link>
            <guid isPermaLink="false">https://medium.com/p/97de79e04323</guid>
            <category><![CDATA[soft-skills]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[business-strategy]]></category>
            <category><![CDATA[communication-skills]]></category>
            <category><![CDATA[problem-solving]]></category>
            <dc:creator><![CDATA[Bryan McMaster]]></dc:creator>
            <pubDate>Mon, 02 Jan 2023 23:47:02 GMT</pubDate>
            <atom:updated>2023-01-03T03:35:09.041Z</atom:updated>
            <content:encoded><![CDATA[<h3>A successful career in Data Analytics? Develop your Business Acumen and Problem Solving skills!</h3><p>Many of us have concentrated our efforts on hard skills necessary for our first jobs. Concentrating heavily on tools, libraries, algorithms will only get you so far. The one thing that I see most entry and junior level people needing to develop more is well formed problem solving skills. These skills, fused with a strong sense and understanding of the business, its strategy and acumen go further than just the hard skills.</p><h3>Develop Business Acumen</h3><p>Data analysts play a crucial role in helping businesses make informed decisions. They do this by sifting through a mountain of data and finding the nuggets that will further the success of the business. But without a firm understanding of the business how will you find these nuggets? This understanding will include knowing the company’s products or services, target market, competition, overall business model and value proposition.</p><p>How do you go about understanding the business? Research! Even if you haven’t yet started the job, or are still interviewing you should completely read through the entire website, using its sitemap or menu. A good place to start, if its a public company is to read the investor relations page, read the latest public filings and quarterly statements. This may seem boring, but keep in mind this is how you get paid! Find their mission statement, their vision, do you relate to it? Does it get you excited to learn more? If not, maybe this isn’t the right place for you! Also, do some Google searches, find out the positives and negatives about the business, what new products or services have they recently disclosed? If it is a Brick &amp; Mortar business, visits a few locations, buy something, interact with the employees.</p><p>Another option is to reach out to your new coworkers on <a href="https://www.linkedin.com/">LinkedIn</a>. Most of them will be more than happy to get to know you and help you along to understand the company. Look for senior managers and directors in your business area or even people who are in strategy. If you are already working there, its even better to ask your manager or peers to make introductions to the decision-makers and those that truly understand the business processes, and the overall strategy.</p><p>Having this knowledge will allow you to start asking the right questions and focus your analysis on the right data points. It will also help you contextualize your findings and communicate them in a meaningful way to decision-makers, once you have developed your problem solving skills.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1vpIETYrqTcPOlGa" /><figcaption>Photo by <a href="https://unsplash.com/@pavement_special?utm_source=medium&amp;utm_medium=referral">Riccardo Annandale</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Develop Your Problem Solving Skills</h3><p>With knowledge of the business now in hand, it is time to hone your problems solving skills. You may be saying to yourself — “Wait, I already have problem solving skills!”, after all you got the job, you passed the hard skills part of the interview, you spent months preparing solving <a href="https://leetcode.com/">LeetCode</a>. You aced research papers in grad school! What else do I need to learn?</p><blockquote>I already have problem solving skills, what else do I need to learn?</blockquote><p>Business acumen is a portfolio of knowledge, meaning technical skills and soft skills. This includes how to think critically in your domain, and solve problems with the best solution, with all the given data. This skills will take some time to develop but can be developed faster with a combination of your knowledge of the business and the project or dataset in hand.</p><p>When given a problem or a dataset, ask questions, even in a loop if necessary! Here is a start for questions you might ask:</p><ul><li>What are we trying to solve?</li><li>What is the goal of the analysis?</li><li>What are the constraints?</li><li>Where did the data come from? How was it collected and it is complete and accurate, what&#39;s missing?</li><li>Most importantly: How does this project relate to the overall strategy?</li></ul><blockquote>Make your hard work meaningful, if its not contributing to the overall strategy is it worth it?</blockquote><p>If the project sponsor cannot clearly articulate the last point, then maybe this analysis isn’t worth the effort. Make your hard work meaningful, if you toil away at trying to solve a problem or finish a project and there is no clear connection to the strategy, is it worth it?</p><p>Once you have this information, and the dataset in hand, its time to do what you do best! Take an approach to fully understand the dataset. I like to first gather summary statistics and build out a glossary and definition of the data. Its a good time now to keep the communication going, and review your findings so far with the team.</p><p>Depending on the business problem or question use the appropriate tools at your disposal . Are you asked to build KPI’s and descriptive reports or are you asked to make a prediction? This is where asking the right questions, and understanding the business comes into play. Having both of these will lead you to the correct tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K688BqVOzsSv1iTH" /><figcaption>Photo by <a href="https://unsplash.com/ja/@benjaminzanatta?utm_source=medium&amp;utm_medium=referral">Benjamin Zanatta</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Finally, after you have gathered your insights, made your prediction or developed the KPI’s, be ready to present your findings. When presenting your findings, you will articulate your understanding of the problem, how it relates to the business and your results. I look for these five answers in any analysis:</p><ol><li>Well articulated problem statement</li><li>How it relates to the business strategy</li><li>Summary statistics, visualizations and definitions</li><li>Results and findings of the analysis (both summary and detail if possible)</li><li>Conclusion and next steps</li></ol><p>These should all be wrapped into a nice, well articulated story presented in a document, deck and communicated verbally to the decision-makers and stakeholders.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97de79e04323" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>