<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matt Collins on Medium]]></title>
        <description><![CDATA[Stories by Matt Collins on Medium]]></description>
        <link>https://medium.com/@matt_collins?source=rss-8fd2c4b84128------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Nhk7GuWMxJEb90nzxLjDdA.jpeg</url>
            <title>Stories by Matt Collins on Medium</title>
            <link>https://medium.com/@matt_collins?source=rss-8fd2c4b84128------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:16:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@matt_collins/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Evolution of the Databricks Lakehouse Paradigm]]></title>
            <link>https://medium.com/@matt_collins/the-evolution-of-the-databricks-lakehouse-paradigm-71f613c6533a?source=rss-8fd2c4b84128------2</link>
            <guid isPermaLink="false">https://medium.com/p/71f613c6533a</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[slalom]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Matt Collins]]></dc:creator>
            <pubDate>Thu, 12 Nov 2020 14:22:47 GMT</pubDate>
            <atom:updated>2020-11-12T20:51:02.270Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4fTii4jyousJ7W4FScNZLg.png" /><figcaption>Databricks Lakehouse Paradigm</figcaption></figure><p>In January 2020, Databricks made a promising <a href="https://databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html">announcement</a>; the introduction of a new paradigm in data management called the Lakehouse — a blend of the best features of data warehouses and data lakes.</p><p>Databricks was founded by the creators of Apache Spark and has developed some of the most innovative and revolutionary technologies in data and analytics: the Unified Data Analytics Platform, Delta Lake, and MLflow.</p><p>As a thought and innovation leader in the field, Databricks was uniquely positioned to introduce the concept of the Lakehouse, as well as to lead the charge towards it. With the release of SQL Analytics, Databricks is quickly bringing the Lakehouse paradigm from concept to reality.</p><h3>Introducing SQL Analytics</h3><p>Today, Databricks has taken the Lakehouse paradigm another leap forward with the <a href="https://databricks.com/blog/2020/11/12/announcing-the-launch-of-sql-analytics.html">introduction of SQL Analytics</a>. SQL Analytics is a new SQL-first experience in Databricks, featuring a new, dedicated workspace. The interface provides the Databricks platform with a first-class environment that SQL users will feel right at home using.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZotyBxbm9mcsuUH8P6zOqg.png" /><figcaption>SQL Analytics Workspace</figcaption></figure><p>Traditionally, Databricks has been a notebook-only environment, with SQL capabilities contained within notebook cells. That all changes today. Databricks now has the ability to provide SQL-heavy data engineers and data analysts with a distinct workspace tailored specifically to meet their needs and ways of working.</p><h4><strong>Dedicated SQL Analytics Workspace</strong></h4><p>Databricks has developed a separate workspace specifically for SQL users. The UI features a rich development environment, allowing users to explore their data and even build dashboards and visualizations right below the query editor. The familiarity of this environment will improve usability for SQL-first users, especially those who may not prefer a notebook environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cxFf4iWxzXEgFs5KRjVHfQ.png" /><figcaption>SQL Analytics Query Pane</figcaption></figure><p>The SQL Analytics Workspace also features:</p><ul><li><strong>Simplified cluster management pane: </strong>Easily provision a new cluster right from the SQL Analytics workspace or restart a previously stopped cluster. Users have the ability to specify the cluster size and enable auto-stopping after a user-provided timeout period, with support for multi-cluster load balancing and Photon coming in the future</li><li><strong>Job scheduling: </strong>Just like in the notebook environment, Databricks has provided a simple-to-use scheduling feature, enabling users to set up a refresh schedule right from the Query pane</li><li><strong>Query history: </strong>View the history of executed queries and search queries</li><li><strong>Event alerts: </strong>Set alerts and receive messages for specified events</li></ul><h4><strong>Performance</strong></h4><p>SQL Analytics leverages the performance of <a href="https://databricks.com/product/delta-engine">Delta Engine</a> with Photon — a new execution engine built in C++ from the ground up — to provide all of the speed and snappiness you would expect from a SQL service, on the data lake.</p><p>When Databricks announced Delta Engine in June 2020, the potential impact of the performance gains on top of Spark 3.0 was immediately apparent, as noted in the announcement blog:</p><blockquote>The improved query optimizer extends the functionality already in Spark 3.0 (cost-based optimizer, adaptive query execution, and dynamic runtime filters) with more advanced statistics to deliver up to 18x increased performance in star schema workloads. <br>— <a href="https://databricks.com/blog/2020/06/24/introducing-delta-engine.html">Databricks</a></blockquote><p>Now, with the introduction of SQL Analytics, Databricks brings these massive performance improvements to the spotlight for SQL users.</p><h4><strong>Great BI Experience on the Data Lake</strong></h4><p>SQL Analytics easily connects to leading BI tools and has been optimized for business intelligence on data lakes. Databricks has developed new ODBC and JDBC drivers for BI tools with lower latency and higher data transfer speed, as well as improved metadata performance for increased read speeds for cold queries on Delta tables. Support for OAuth and Single Sign On provide a seamless authentication experience.</p><p>At launch, SQL Analytics will connect to Tableau and PowerBI, with planned integration for Qlik, ThoughtSpot, and Looker.</p><p>The introduction of SQL Analytics marks an important milestone for Databricks, in both the evolution of the Lakehouse paradigm and the maturation of the Unified Data Analytics Platform. Databricks is a platform for all personas, and that has never been truer than it is now.</p><p>Leveraging an integrated platform for data and AI, organizations are free to focus on creating an environment for innovation.</p><h3>Modern Culture of Data</h3><p>At Slalom, one way we help organizations achieve their boldest ambitions is by helping them build a <a href="https://www.slalom.com/modern-culture-data">Modern Culture of Data</a> — an environment of experimentation, empowerment, curiosity, critical thinking, and collaboration.</p><p>A Modern Culture of Data is enabled by five key elements:</p><ul><li><strong>Bold Vision: </strong>To build a successful culture of data, you need to know where you’re going, how you’ll get there, and why it’s important</li><li><strong>Access &amp; Transparency: </strong>A true data-driven culture is built on a modern technology foundation that provides easy access to data and tools</li><li><strong>Guardianship: </strong>Data guardianship ensures that the use, ownership, and maintenance of data is safe, secure, compliant and ethical</li><li><strong>Data Literacy: </strong>Data literacy ensures that people have a fundamental understanding of data, how to analyze it, and how to use it to make decisions and take action — at all levels and in all processes and parts of their business</li><li><strong>Ways of Working: </strong>An operating model and organizational structure with processes, roles and responsibilities that support the organization’s bold vision will ensure that data and analytics are embedded into day-to-day operations, planning, and decision-making</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m32kJvtd-drEy_SKT65tjA.png" /><figcaption>Slalom’s Modern Culture of Data Framework</figcaption></figure><p>According to a Forrester Research report from 2018, insights-driven businesses are growing at an average of more than 30% each year, and by 2021, they are predicted to earn $1.8 trillion annually.</p><p>Despite this substantial opportunity, Gartner’s 3rd Annual CDO Survey reported that Chief Data Officers say culture change is the #1 inhibitor to progress.</p><h3>A Holistic Approach to Data</h3><p>Realizing the full potential of data and analytics requires shifting the way your organization works and adopting a culture where people have the power to accelerate business outcomes with rapid insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*XyCpFSVFdA0Uq8BMMyaz8A.png" /></figure><p>With the introduction of SQL Analytics and the realization of the Lakehouse paradigm, Databricks is perfectly suited to help organizations transform by allowing them to focus on innovation by leveraging a unified platform that works for all of its users.</p><p>Data engineers, data scientists, ML engineers, and data analysts can actively collaborate in a single platform, shifting time from processing data and managing infrastructure to extracting value.</p><p>A Modern Culture of Data , powered by Databricks, gives organizations a platform to act as a catalyst for the massive cultural transformation necessary to achieve the vision of analytics at scale.</p><p><em>Join Databricks, Microsoft, and Slalom on December 15th for a webinar on </em><a href="https://go.slalom.com/building_your_ai_skyscraper"><em>Building Your AI Skyscraper</em></a><em> to learn how to break ground on AI in your organization.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71f613c6533a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BigQuery ML + Tableau]]></title>
            <link>https://medium.com/qash/bigquery-ml-tableau-34f8ecb0caa9?source=rss-8fd2c4b84128------2</link>
            <guid isPermaLink="false">https://medium.com/p/34f8ecb0caa9</guid>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Matt Collins]]></dc:creator>
            <pubDate>Thu, 04 Oct 2018 19:06:44 GMT</pubDate>
            <atom:updated>2019-02-26T19:06:37.995Z</atom:updated>
            <content:encoded><![CDATA[<h3>Integrating BigQuery ML &amp; Tableau</h3><p>Using BigQuery ML and Tableau for dynamic model predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjKJwRaigXq2IR7d5_jlXQ.jpeg" /><figcaption>Source:<a href="https://unsplash.com/@rawpixel?utm_source=medium&amp;utm_medium=referral"> rawpixel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Data Science Democratization</h3><p>A trend that has emerged over the past couple of years in the analytics space is that of democratizing data science. In other words, enabling business users to garner insights from data that, many years ago, might have taken a room full of experts to extract.</p><p>At Google Next ’18, Google announced a new entrant in this democratization race — BigQuery ML. The ML component builds onto BigQuery, Google’s serverless data warehouse. As Google tells it:</p><blockquote>BigQuery ML enables users to create and execute machine learning models in BigQuery using standard SQL queries. BigQuery ML democratizes machine learning by enabling SQL practitioners to build models using existing SQL tools and skills. BigQuery ML increases development speed by eliminating the need to move data. <br> — <a href="https://cloud.google.com/bigquery/docs/bigqueryml-intro">Google Cloud Platform Website</a></blockquote><p>This is an exciting prospect, as it will enable those who are familiar with SQL, but who may have never tried an object-oriented programming language such as Python or R, to utilize machine learning.</p><p>According to StackOverflow’s 2018 Developer Survey, in which over 100,000 developers responded to questions about their experiences and backgrounds, SQL ranked as the 4th most commonly used programming language. This is compared to Python at #7 and R at #19.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uLKH89eSneMqMUfOTAvBfg.png" /><figcaption>Source: StackOverflow</figcaption></figure><p>Given these results, it would appear that a technology like BigQuery ML could go a long way in enabling a large number of users.</p><p>BigQuery ML pairs well with another popular analytics tool — Tableau. Connecting a BigQuery ML model to Tableau creates some exciting opportunities for understanding and acting on machine learning predictions.</p><p>In this post, we’ll look at:</p><ul><li>Creating a model in BigQuery</li><li>Understanding predictions in Tableau</li><li>Using Tableau parameters for dynamic predictions</li></ul><p>To demonstrate a business use case, let’s explore using BigQuery ML to create a model to predict customer churn. We’ll use logistic regression to predict a binary value of whether we expect a given customer to discontinue service. Then we’ll use Tableau to understand the predictions and see how different actions might affect those predictions.</p><p><em>(Data used is from </em><a href="https://www.ibm.com/communities/analytics/watson-analytics-blog/guide-to-sample-datasets/"><em>IBM Analytics Communities Sample Data Sets</em></a><em>)</em></p><h3>Creating a Model in BigQuery</h3><p>Creating a machine learning model in BigQuery only requires a few commands in addition to a typical SELECT statement.</p><p>Here’s what the syntax looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*g9JY7RVzk7tKg6yReRWDkA.png" /></figure><p>The bare minimum needed here is: naming the model; selecting linear or logistic regression; and setting the label (unless its actual name is already “label”). Not too bad, eh?</p><p>Following the example above, the code for our model would be as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*9jq7xV5vnNPzA9pa1FigKQ.png" /></figure><p>Here, we are training the model on every feature in our dataset except for our ID field — <em>customerID.</em></p><p><em>Note: This post will not cover some very important stages of the machine learning process such as feature engineering, hyper-parameter tuning, cross validation or model performance monitoring. We will only cover what is needed to stand up an initial model.</em></p><p>After running the query, BigQuery allows you to view information on your model such as size, training statistics, and the model schema:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oPbHoIP9egZR0hs8Mpm5Cg.png" /></figure><h3>Understanding Predictions in Tableau</h3><p>Now that we have a working model to predict customer churn, we can bring it into Tableau and predict outcomes for a new sample of customers.</p><p>The dataset we want to predict is also loaded into BigQuery and contains data on separate customers from those in our training data. We’ll bring this data into Tableau and use Tableau’s Custom SQL feature to call our model and predict churn for this new subset of customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xY98LVOULgS2-Uw9DLK1Pg.png" /></figure><p>The above Custom SQL query is all it takes to call our model and use it to make predictions on our new <strong>test</strong> data. The first column in the data preview window — Churn? — is our newly predicted value.</p><p>Now that we have predicted values, we can use Tableau to visualize these predictions and see how they relate to our data.</p><p>Using the predicted values, I created a simple dashboard to view some metrics related to predicting customer churn:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FwyVyZN3yYEapmGBVi9jaw.png" /></figure><p>Looking at this dashboard, we can see that about 17.2% of our customers are expected to churn the next cycle. We’ll look at how different actions might effect these predictions a bit later.</p><p>Another thing you’ll notice in the dashboard above is that all of the customers predicted to churn are on month-to-month contracts. Additionally, looking at churn rates by tenure at the bottom right of the dashboard, we can see that we expect to lose almost half — 44% — of our customers that have a tenure between zero and six months. This rate drops significantly, however, once you pass the 6-month and 1-year tenure mark.</p><p>Based on this knowledge, it seems that converting subscribers off month-to-month plans and into plans that result in them staying longer than one year could have an impact on our predicted churn rates.</p><h3>Using Tableau Parameters for Dynamic Predictions</h3><p>Given that we want to get customers into longer-term contracts, an option may be to offer a special 1- or 2-year promotional pricing plan targeted to current month-to-month customers. Using parameters in Tableau, we can dynamically see how a response like this would impact our predicted churn.</p><p>Below, I have created another version of the dashboard we looked at earlier:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t1q-AbCWOz_A1AxPtyvoZw.png" /></figure><p>On the right-hand side, there are now two parameter options to select. The options I’ve included here are:</p><ul><li><em>% to Convert</em> — the % of current month-to-month customers we expect to convert based on previous promotion results</li><li><em>Convert to Contract</em> — the contract length we will offer to current month-to-month customers</li></ul><p>Let’s see how changing the parameters can impact our predictions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqkkftJlVvVt3XI0BLWp_Q.gif" /></figure><p>As you can see above, converting users off of month-to-month contracts has a positive impact on reducing our predicted churn rates. Let’s say a conservative estimate would be to convert 20% of our month-to-month customers to a 1-year contract. Using those parameters, we could expect to see our overall predicted churn rate drop from 17.2% to 14.9%.</p><h3>Bigger Picture</h3><p>In closing, the capabilities illustrated in this post provide a glimpse of how ML can be combined with other elements of BI and how these capabilities will continue to shape the analytics landscape. The integration of tools like BigQuery ML and Tableau provide an exciting outlook on the current state of machine learning and its continued march into the forefront of business intelligence and decision making.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34f8ecb0caa9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/qash/bigquery-ml-tableau-34f8ecb0caa9">BigQuery ML + Tableau</a> was originally published in <a href="https://medium.com/qash">Slalom Insights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>