<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Raghav Matta on Medium]]></title>
        <description><![CDATA[Stories by Raghav Matta on Medium]]></description>
        <link>https://medium.com/@raghavmatta?source=rss-fc418b483cb4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*VQw4vZYOev3bEmV2y-X11w.jpeg</url>
            <title>Stories by Raghav Matta on Medium</title>
            <link>https://medium.com/@raghavmatta?source=rss-fc418b483cb4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:03:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@raghavmatta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Give Databricks Genie Your Business Context by Hydrating Unity Catalog Pages]]></title>
            <link>https://medium.com/@raghavmatta/give-databricks-genie-your-business-context-by-hydrating-unity-catalog-pages-627fe47d5f2a?source=rss-fc418b483cb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/627fe47d5f2a</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[databricks-genie]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <dc:creator><![CDATA[Raghav Matta]]></dc:creator>
            <pubDate>Fri, 23 Oct 2026 06:43:02 GMT</pubDate>
            <atom:updated>2026-10-23T06:43:02.038Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 1 of X in the </em><strong><em>Building a Company Brain</em></strong><em> series.</em></p><h3>Summary</h3><ul><li><strong>Databricks Genie</strong> answers business questions in plain English, but only as well as it understands <em>your</em> vocabulary — your terms, metrics, and policies.</li><li>That vocabulary usually lives in <strong>SharePoint</strong>, Confluence, and wikis — not in your tables.</li><li>You can capture it as <strong>Unity Catalog Pages</strong> (governed definitions that enrich Genie) by connecting SharePoint through the <strong>Microsoft 365 MCP connector</strong> and using <strong>Genie Code bulk-import</strong> — entirely in the UI, with no custom pipeline.</li><li>The payoff is immediate: ask Genie the same question before and after, and watch it answer in your business’s own language.</li></ul><h3>Why does Databricks Genie need business context to answer accurately?</h3><p><strong>Databricks Genie is Databricks’ AI coworker for business users: it turns a plain-English question about your data into a trusted answer.</strong> But Genie can only be as accurate as its understanding of your business. To answer <em>“what was our sell-through on the fall drop?”</em>, Genie needs to know what your company means by “sell-through,” how it is calculated, and which tables carry it. That knowledge — your <strong>business glossary</strong> — rarely lives in the warehouse. It lives in SharePoint sites, Confluence spaces, and onboarding docs.</p><p>Building a “company brain” is the work of capturing that glossary where the AI can use it, without re-typing years of institutional knowledge by hand, and without losing control of the official definition.</p><h3>What are Unity Catalog Pages?</h3><p><strong>Unity Catalog Pages are governed objects in Databricks Unity Catalog that capture a business concept — a term, metric, or policy — with its definition, synonyms, owner, and links to the data assets it describes.</strong> Because Pages enrich the Genie Ontology (the context layer Genie reasons over), the more of your real vocabulary you capture as Pages, the better Genie understands your business.</p><p>That gives a simple, source-agnostic pattern:</p><ul><li><strong>Source</strong> — your knowledge lives in SharePoint, Confluence, and wikis.</li><li><strong>Capture</strong> — it becomes governed Unity Catalog Pages (definition, synonyms, owner, and asset links).</li><li><strong>Payoff</strong> — Databricks Genie understands your business and answers in your language.</li></ul><p>This post uses <strong>SharePoint</strong>, reached through the <strong>Microsoft 365 MCP connector</strong>. (MCP, the Model Context Protocol, is the open standard that lets AI assistants reach external systems through a connector.) <strong>SharePoint is just the example here — the same pattern hydrates Pages from Confluence, Google Drive, or any knowledge source that has a connector; swap the connector and the steps are identical.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_BCRl67dLqcbOvK7kHf-Bw.png" /><figcaption>Flow diagram: SharePoint content becomes Unity Catalog Pages, which enrich Databricks Genie.</figcaption></figure><h3>How do you connect SharePoint to Databricks with the Microsoft 365 MCP connector?</h3><p>Genie Code — the assistant that builds and edits Pages — reaches external systems through MCP connectors. The Microsoft 365 MCP connector puts your SharePoint content within reach of the assistant using your tenant’s own authentication, so nothing is copied or re-keyed by hand.</p><ol><li><strong>Add the Microsoft 365 MCP connector</strong> and authenticate to your tenant (OAuth). This grants read access to the SharePoint sites you approve.</li><li><strong>Scope it to the right content</strong> — the sites and libraries that hold glossary-grade material (the data dictionary, metric definitions, policy playbooks), not every document in the org. A tight, authoritative scope beats a firehose every time.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zFBDKNanWTFUBQ0Rjub8zA.png" /><figcaption>Connecting Microsoft 365 so Genie Code can read SharePoint.</figcaption></figure><h3>How do you bulk-import SharePoint content into Unity Catalog Pages?</h3><p>With SharePoint reachable, hydrating Pages is a guided, UI-driven flow:</p><ol><li><strong>Open the bulk-import flow:</strong> Discover → Pages → Create page, then open the <strong>Genie Code</strong> pane and choose <strong>Bulk import pages</strong>.</li><li><strong>Attach the source and run a scoped prompt.</strong> Attach the Microsoft 365 connector, choose the target <strong>domain</strong>, and give Genie Code a precise instruction:</li></ol><p><em>Create a Unity Catalog Page for each SharePoint document in the selected library. Use the document title as the term, the body as the definition, and extract any synonyms. Set the source link to the SharePoint URL. Do not invent content that isn’t in the source. Leave domain, owner, and certification for me to set. Produce drafts only — do not publish.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wbuHf53o9r0Bu7d_XvXZmA.png" /><figcaption>Discover → Pages → Create page → Genie Code → Bulk import pages.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*9nLYUWlj0d_LYR3HRzCxLA.png" /><figcaption>Pick the domain the new Pages belong to (for example, Finance).</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*YXAPqRGUymCZ8NMFvEbj0w.png" /><figcaption>Genie Code searches the SharePoint library and reads the source documents.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gv7YOxwrBur2xh6k4hDuPQ.png" /><figcaption>The proposed Pages, each with a definition and synonyms.</figcaption></figure><h3>How do you review and publish the Pages?</h3><p>This is where governance lives, and it is deliberate.</p><ol><li><strong>Review each draft:</strong> set the <strong>domain</strong> and <strong>owner</strong>, confirm the <strong>synonyms</strong>, screen the content, and resolve duplicates. The assistant proposes; a person decides.</li><li><strong>Publish:</strong> accepted drafts move through the Page lifecycle (draft → review → published) and immediately begin enriching Databricks Genie.</li></ol><p><strong>Why is publishing a UI action and not an API call?</strong> Writing a Page is intentionally a human-in-the-loop step: publishing a definition is a governed act, so automation can <em>propose</em> a draft but a person <em>publishes</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aZG6Uxq9nyyTPwpZblEcng.png" /><figcaption>Draft Pages land in the domain for review before anyone publishes.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7maiaDn4unsRbvc7YNFpA.png" /><figcaption>A published Page — definition, synonyms, owner, and a link back to the SharePoint source.</figcaption></figure><h3>What does Genie answer before vs. after hydration?</h3><p>Pick a term you just published — say <strong>Sell-Through Rate</strong> — and ask Databricks Genie a question that depends on it. (Recreate this on synthetic data, e.g. the retail demo at go/geniedemo-retail.)</p><ul><li><strong>Before hydration</strong> — you ask <em>“What was sell-through on the fall drop?”</em> and Genie has no definition for “sell-through,” so it guesses at columns or asks you to clarify.</li><li><strong>After hydration</strong> — you ask the same question and Genie uses your published Page (the definition, the formula, the mapped assets) to answer in your business’s language.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2c8MVNmJHtNA41SgG2JF9w.png" /></figure><p>That contrast is the whole point of a company brain: the same question, a dramatically better answer, because the platform finally knows what <em>you</em> mean.</p><h3>Key takeaways</h3><ul><li>Databricks Genie’s accuracy depends on business context, which lives in SharePoint and wikis, not tables.</li><li>Unity Catalog Pages capture that context as governed definitions that enrich Genie.</li><li>The Microsoft 365 MCP connector + Genie Code bulk-import hydrates Pages directly from SharePoint — no custom pipeline to start.</li><li>Publishing stays human-approved by design; automation proposes, a person publishes.</li><li>Prove value by asking Genie the same question before and after hydration.</li></ul><h3>FAQ</h3><p><strong>How do I give an AI data agent the business context it needs to answer accurately?</strong> Capture your business glossary as Unity Catalog Pages — definitions, synonyms, and asset links — so Databricks Genie reasons over your actual vocabulary.</p><p><strong>Do I need a data pipeline to create Unity Catalog Pages?</strong> No. For the initial hydration, Genie Code reads the source directly through the connector; a pipeline is only needed later to keep Pages current (Part 2).</p><p><strong>Can Pages be created through an API?</strong> No — Pages are created and published through the UI / Genie Code by design, so a human approves every published definition.</p><p><strong>Which knowledge sources work?</strong> The pattern is source-agnostic — SharePoint via the Microsoft 365 MCP connector here, but Confluence or other stores follow the same shape.</p><h3>Sources</h3><ul><li><a href="https://www.databricks.com/blog/operationalizing-genie-ontology-your-data-stack">Operationalizing Genie Ontology in Your Data Stack</a></li><li><a href="https://www.databricks.com/blog/data-ontology-defined-context-layer-your-ai-agents-are-missing">Data Ontology defined: the context layer your AI agents are missing</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=627fe47d5f2a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How do you make an AI assistant respect document permissions on Databricks?]]></title>
            <link>https://medium.com/@raghavmatta/how-do-you-make-an-ai-assistant-respect-document-permissions-f4a18336cc3a?source=rss-fc418b483cb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4a18336cc3a</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[ai-search]]></category>
            <category><![CDATA[enterprise-ai]]></category>
            <category><![CDATA[databricks-genie]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Raghav Matta]]></dc:creator>
            <pubDate>Mon, 12 Oct 2026 09:47:35 GMT</pubDate>
            <atom:updated>2026-11-06T08:00:42.684Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The problem:</strong> You build an AI assistant over your company’s documents. Finance can open the payroll file; Sales can’t. Will the assistant honor that — or will it quote the payroll file to anyone who asks?</p><p><strong>The answer:</strong> It honors it, if you enforce the permission at query time. On Databricks AI Search you add an accessible_principals list to each document chunk and filter every search by the asking user’s identity. The index returns only chunks that user is allowed to open, so the model can’t ground an answer in anything else.</p><p>That’s the whole idea. We’ll use <strong>SharePoint</strong> as the running example throughout — but the same pattern applies to any document source that has per-item permissions (Google Drive, Confluence, a document management system). Here’s how it works and how to prove it.</p><h3>Where does permission enforcement belong?</h3><p>At the query layer — and that’s by design. A document’s permissions live in the source system, not inside the document’s text, so the clean place to apply them is at the moment of retrieval: check the asking user against each document’s access list, then return only what they’re allowed to see.</p><p>Databricks AI Search gives you exactly that hook. The docs point you straight to it: <em>“Row and column level permissions are not supported. However, you can implement your own application level ACLs using the filter API.”</em> The <strong>filter API</strong> is purpose-built for application-level access control — you pass the user’s identity on every query, and the index does the filtering for you. Reference: (<a href="https://docs.databricks.com/aws/en/ai-search/ai-search#limitations">AI Search docs</a>)</p><h3>The pattern, in four steps</h3><ol><li><strong>Ingest the docs</strong> — chunk and embed the text as usual.</li><li><strong>Ingest the permissions separately</strong> — read each document’s access list (users, groups, or an org-wide marker) into its own table.</li><li><strong>Attach them</strong> — join the access list onto each chunk as an accessible_principals array, in a continuous Delta Sync index.</li><li><strong>Filter per user</strong> — on every query, pass filters={“accessible_principals”: [user, “__org__”]}. Only permitted chunks come back.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cGmnTo72Mc-B3L5-04KhkA.png" /></figure><h3>How do the permissions actually get from SharePoint?</h3><p>Steps 2 and 3 are a small job that runs <strong>separately from document ingestion</strong> and keeps the index’s permissions current. It does three things:</p><ul><li><strong>Read permissions from the source via the Microsoft Graph API.</strong> Using an app-only (client-credentials) OAuth token, the job lists the files in the SharePoint library and calls each item’s permissions endpoint to get its access list — which users and groups are granted, and whether the item is shared organization-wide.</li><li><strong>Write them to a permissions table</strong>, keyed by document ID, and <strong>expand groups into their member identities</strong> so a user who inherits access through a group is matched the same way as a direct grant.</li><li><strong>Merge onto the chunks.</strong> The job MERGEs the access list onto each document’s chunks as the accessible_principals array. Because the index is a continuous Delta Sync index, that merge streams into the index automatically — no rebuild.</li></ul><p>You run this on a schedule or as a continuous loop; the refresh interval is your freshness-versus-cost knob. In the proof of concept it runs continuously and picks up a SharePoint change in about 30 seconds.</p><p>Document <strong>content</strong> can be pulled the same way through Microsoft Graph, or — in production — through a managed Lakeflow Connect SharePoint ingestion pipeline. The permission sync above is the part that makes the result access-aware.</p><h3>Does a live permission change actually flow through?</h3><p>Yes — in seconds, with no re-indexing. Here’s the exact sequence from a proof of concept with synthetic documents and personas.</p><ol><li><strong>No access, no answer.</strong> The user asks a payroll question but isn’t on any document, so the assistant has nothing to ground on and says so.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u-WIwmEPDRF_7PH9eRw6rQ.png" /></figure><p><strong>2. Grant access — in SharePoint, not in the app.</strong> An admin adds the user to payroll_summary.csv through SharePoint’s native <strong>Grant access</strong> dialog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IyWEvSYHOYjCeoSIrhkMuA.png" /></figure><p><strong>3. Same question, now answered.</strong> Within the ACL job’s refresh window (~30s), the grant reaches the index. The same question now returns the answer, cited to payroll_summary.csv — no re-embedding, no rebuild.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A7HMkPTVpjgjTSFRBdyz3g.png" /></figure><p>The permission lived in SharePoint the whole time. We didn’t re-create it in the AI layer — we propagated it: <strong>SharePoint → ACL sync (~30s) → index → answer.</strong> The gate is per user on every query: a different persona who can only open the travel policy gets no answer to the same payroll question.</p><h3>What about structured data?</h3><p>Same principle, handled for you. <strong>Genie One</strong> answers natural-language questions over <strong>Unity Catalog</strong> data and enforces the asking user’s own permissions automatically. Documents need the filter-API pattern above; tables get it natively.</p><h3>Key takeaways</h3><ul><li>A document’s permissions aren’t in its text — sync them alongside the content.</li><li>Enforce document access <strong>at query time</strong> via the filter API (the documented Databricks pattern).</li><li>Use a <strong>continuous</strong> index so a grant/revoke at the source shows up in seconds, no rebuild.</li><li>For structured data, <strong>Genie One + Unity Catalog</strong> enforce permissions natively.</li></ul><h3>FAQ</h3><p><strong>Does AI Search enforce per-user document access on its own?</strong> No — it has no row-level ACL. You add it at the app layer via the filter API.</p><p><strong>How fast does a permission change take effect?</strong> Seconds. In the PoC a SharePoint grant/revoke reflected in answers in ~30s, no re-embedding.</p><p><strong>What about group permissions?</strong> Expand group membership into accessible_principals during the sync step.</p><p><strong>How does this relate to Genie?</strong> It’s the document-side counterpart. Genie One governs structured data in Unity Catalog; this governs documents.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4a18336cc3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Secrets API to Mount Azure Blob securely on Databricks File Storage]]></title>
            <link>https://medium.com/swlh/using-secrets-api-to-mount-azure-blob-storage-securely-to-databricks-30e7f2501703?source=rss-fc418b483cb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/30e7f2501703</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[swabluanalytics]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <dc:creator><![CDATA[Raghav Matta]]></dc:creator>
            <pubDate>Sun, 17 May 2020 17:37:17 GMT</pubDate>
            <atom:updated>2020-06-19T19:26:33.298Z</atom:updated>
            <content:encoded><![CDATA[<p>In this tutorial, you will learn Databricks CLI -Secrets API to achieve the below objectives:</p><ul><li>Create an Azure Storage Account using Azure Portal</li><li>Install and configure Databricks CLI - Secrets API</li><li>Mount Blob storage on your Azure Databricks File Storage</li></ul><h3>Requirements</h3><ul><li>Microsoft Azure subscription must be pay-as-you-go / MSDN / Trial subscription</li><li>An existing Databricks Workspace with a cluster, steps to create one can be found <a href="https://medium.com/@raghavmatta/getting-started-with-databricks-on-azure-f3bceda8b918">here</a></li><li>Python version 2.7. 9 and above if you’re using Python 2 or Python 3.6 and above if you’re using Python 3</li></ul><h3>What are the advantages of using Secrets API?</h3><p>Databricks provide your team with a collaborative environment using Notebooks, which has many advantages but one challenge which comes with this is that you end up passing all your access secrets ( resource keys, service principal passwords, server address, etc.) between your team members.</p><p>Secrets API is a REST API 2.0 provided by Databricks CLI and helps you with below:</p><ul><li>Store all your secrets under a single repository</li><li>Access these secrets through REST API 2.0 encrypted channel</li><li>No need to enter or display secrets on your Databricks Notebook</li><li>The secrets stored cannot be viewed once stored through CLI</li><li>Manage your secrets category wise under a given scope</li><li>Scope of the secrets is limited to given Databricks Workspace</li><li>Version control your Notebooks on Git Repo without pushing your Secrets</li></ul><p><strong>Exercise 1: Create an Azure Storage Account using Azure Portal</strong></p><ul><li>Navigate to <a href="https://portal.azure.com/">https://portal.azure.com/</a> and sign in using your Microsoft account credentials with Azure Subscription</li><li>From your Azure dashboard Click on <strong>+Create A Resource</strong>, then scroll down and click on <strong>Storage </strong>and select <strong>Storage account -blob, file, table, and queue</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*8g3Zkt_9GmXgA6ybcyndIg.png" /></figure><ul><li>Under the <strong>Basic </strong>tab in the <strong>Create Storage Account </strong>page, enter the below details:</li></ul><pre><strong>Subscription: </strong>Select Your Azure Subscription<br><strong>Resource Group(Create New):</strong> adb-secrets-rg<br><strong>Storage Account Name:</strong> &lt;Your Name&gt;adbmount<br><strong>Location:</strong> East US<br><strong>Performance: </strong>Standard<strong><br>Account Kind: </strong>StorageV2<strong><br>Replication: </strong>Read-access geo-redundant storage (RA-GRS)<strong><br>Access tier(default): </strong>Hot</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*vxQS1C0U2VUOqbjGBUHBKg.png" /></figure><ul><li>Click on <strong>Review + Create,</strong> and then click on <strong>Create</strong>. It will take less than 5 minutes for your deployment to complete</li><li>After your deployment is successful, click on <strong>Go To Resource,</strong> this will open your <strong>Azure Storage Account</strong></li><li>On your <strong>Storage Account </strong>blade, from the left-hand menu, click on <strong>Access Keys</strong> under <strong>Settings</strong></li><li>Copy and temporarily save the <strong>Storage Account Name </strong>and <strong>Storage Account Key1 </strong>on a Notepad in your PC</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*__4N8NJlsPl3w21r9_W4-g.png" /></figure><ul><li>Now we will create a container and upload a sample file. Click on the <strong>Overview </strong>tab, and then click on <strong>Container</strong></li><li>In the <strong>Containers </strong>tab, click on <strong>+Container</strong>, enter the Name of the container, make sure the <strong>Public access level</strong> is set to <strong>Private</strong>, and click on <strong>Create</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GzoTJ2gLuoHmO_qUAe7QCw.png" /></figure><ul><li>Open the <strong>sample-container</strong>, and click on <strong>+Upload</strong> to upload a Sample file which you will be accessing through Databricks Notebook</li><li>Sample data used in this tutorial can be exported from the below link:</li></ul><pre><a href="https://data.cityofchicago.org/Public-Safety/Crimes-2018/3i3m-jwuy">https://data</a><a href="http://data.cityofchicago.org/api/views/3i3m-jwuy/rows.csv?accessType=DOWNLOAD">http://data.cityofchicago.org/api/views/3i3m-jwuy/rows.csv?accessType=DOWNLOAD</a></pre><p><strong>Exercise 2: Install and configure Databricks CLI - Secrets API</strong></p><ul><li>From your command-line interpreter with Python installed, run the following pip command to install the <strong>Databricks CLI</strong>:</li></ul><pre>pip install databricks-cli</pre><ul><li>You can confirm that everything is working by running the following command:</li></ul><pre>databricks --version</pre><ul><li>Now, we will have to set up authentication to the Databricks workspace from the CLI. Navigate to your <strong>Databricks Workspace</strong>, and from the top right-hand side click on <strong>User Settings</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*wyx2zzso6YoiXOasRKN5-g.png" /></figure><ul><li>Under <strong>User Settings</strong>, click on <strong>Generate New Token</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P3CFlvy7ynVKK6HdS6CPwQ.png" /></figure><ul><li>Provide a <strong>Comment </strong>(optional) and set a <strong>Lifetime </strong>of the token in days and click on <strong>Generate</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*SJVn_8Ef5mRl4u5_4TRRxA.png" /></figure><ul><li>Make sure you copy the token and click on <strong>Done</strong>. Verify now that your token is visible under the <strong>Access Tokens</strong> tab in the <strong>User Settings</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*zWZTfFayyBW8pJAId_oxRg.png" /></figure><ul><li>Navigate back to your command-line interpreter and type in the following command:</li></ul><pre>databricks configure --token</pre><ul><li>Enter your <strong>Workspace URL</strong> and <strong>Token </strong>to authenticate your CLI, as shown below:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ng1pEU9FhFhP4Iol3IqrQ.png" /></figure><ul><li>You can confirm that you have successfully authenticated to your workspace and view all the notebooks created within your workspace by running the following command:</li></ul><pre>databricks workspace ls</pre><ul><li>Create a new <strong>Scope (‘secrets-demo-storage-key’) </strong>by running the following command:</li></ul><pre>secrets create-scope --scope <strong>secrets-demo-storage-key </strong>--initial-manage-principal users</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*yTSOFBdyTgcu0yBhtOJpdw.png" /></figure><ul><li>Add a new <strong>Secret (‘mattadb-accesskey’) </strong>to the existing scope, by running the following command:</li></ul><pre>databricks secrets put --scope <strong>secrets-demo-storage-keys</strong> --key <strong>mattadb-accesskey</strong></pre><ul><li>In the pop-up text editor, paste the <strong>Storage Account Key1</strong> you had copied in the previous exercise and Save</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W7JnRTqsp-zxe-cSN2E_mQ.png" /></figure><ul><li>You can confirm that you have successfully created <strong>Scope </strong>and added a <strong>Secret </strong>by running the following command:</li></ul><pre>databricks secrets list --scope secrets-demo-storage-keys</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*ZiyoIsoeAt49HCsW6TnAQg.png" /></figure><ul><li>If you want to get some help as to what command parameters you can use in the CLI, you can run the following command:</li></ul><pre>databricks &lt;resource&gt; -h</pre><ul><li>For example, if you want to know what commands I could use when working with the<strong> Secrets API</strong>, I’d run the following command:</li></ul><pre>databricks secrets -h</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*N5Ct7R9gcfnI7tHbmDYEiA.png" /></figure><p><strong>Exercise 3: Mount Blob storage on your Azure Databricks File Storage</strong></p><ul><li>Navigate to your <strong>Databricks Workspace</strong> homepage, and click on <strong>New Notebook</strong>. Enter a <strong>Name </strong>for your Notebook, Select <strong>Language </strong>as Python and click on <strong>Create</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*KOsc9wzo6uce7FVJ0qeTuw.png" /></figure><ul><li>In your Notebook, copy and paste the below command:</li></ul><pre>dbutils.fs.mount(<br>  source = &quot;wasbs://<strong>&lt;ContainerName &gt;</strong>@<strong>&lt;StorageAccountName&gt;</strong>.blob.core.windows.net&quot;,<br>  mount_point = &quot;/mnt/<strong>&lt;MountName&gt;</strong>/&quot;,<br>  extra_configs = {&quot;fs.azure.account.key.<strong>&lt;StorageAccountName&gt;</strong>.blob.core.windows.net&quot;:dbutils.secrets.get(scope = &quot;<strong>&lt;Scope Name&gt;</strong>&quot;, key = &quot;<strong>&lt;Secrets Name&gt;</strong>&quot;)})</pre><ul><li>Make sure you have replaced values of <strong>Storage Account Name</strong>, <strong>Container Name</strong>, <strong>Scope Name</strong>, and <strong>Secrets Name </strong>created in Exercise 1 and Exercise 2 respectively. Specify a name for the <strong>Mount Point</strong> and Press <strong>Ctrl + Enter</strong> to run the cell</li></ul><pre>dbutils.fs.mount(<br>  source = &quot;wasbs://<a href="/service/mailto:sample-container@mattaadbmount.blob.core.windows.net"><strong>sample-container</strong>@<strong>mattaadbmount</strong>.blob.core.windows.net</a>&quot;,<br>  mount_point = &quot;/mnt/<strong>sample-container</strong>/&quot;,<br>  extra_configs = {&quot;fs.azure.account.key.<strong>mattaadbmount</strong>.blob.core.windows.net&quot;:dbutils.secrets.get(scope = &quot;<strong>secrets-demo-storage-keys</strong>&quot;, key = &quot;<strong>mattadb-accesskey</strong>&quot;)})</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yzoo_bgGhdFoJSjAVIMkhw.png" /></figure><ul><li>You can confirm that you have successfully mounted your container and list its content by running the following command:</li></ul><pre>%fs<br>ls &#39;/mnt/sample-container/&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aKUBvAXBq8SlhzEv0XYnzQ.png" /></figure><ul><li>You can try reading the data uploaded earlier on your Blob Storage and Create a temporary view by running the following command:</li></ul><pre>crime_records = spark.read.option(&#39;header&#39;,&#39;True&#39;).option(&#39;delimiter&#39;,&#39;,&#39;).csv(&#39;/mnt/sample-container/crime_data/&#39;)<br>crime_records.createOrReplaceTempView(&#39;crime_records&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qxf4VVS2Me-v3-D1TSxCbw.png" /></figure><ul><li>Try running some sample queries, and start analyzing your data</li></ul><pre>%sql<br>SELECT COUNT(*) as Count,`Primary Type`,Arrest FROM crime_records<br>GROUP BY 2,3<br>ORDER BY 1 DESC</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VjMxSfQ5oG9m5NRsKRCYCQ.png" /></figure><ul><li>Next, you can start using this Mount Point with RWX permission. To set up more granular access a file/folder level, you must have Databricks Premium Workspace and setup ACL which will be covered in the next tutorial</li></ul><p><strong>References:</strong></p><ol><li><a href="https://docs.databricks.com/dev-tools/api/latest/secrets.html#secretsecretserviceputacl">https://docs.databricks.com/dev-tools/api/latest/secrets.html#secretsecretserviceputacl</a></li><li><a href="https://data.cityofchicago.org/Public-Safety/Crimes-2018/3i3m-jwuy">https://data.cityofchicago.org/Public-Safety/Crimes-2018/3i3m-jwuy</a></li><li><a href="https://medium.com/@willvelida/installing-configuring-and-using-the-azure-databricks-cli-5d0381e662a1">https://medium.com/@willvelida/installing-configuring-and-using-the-azure-databricks-cli-5d0381e662a1</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30e7f2501703" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/using-secrets-api-to-mount-azure-blob-storage-securely-to-databricks-30e7f2501703">Using Secrets API to Mount Azure Blob securely on Databricks File Storage</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated ML using Azure Machine Learning]]></title>
            <link>https://medium.com/@raghavmatta/automated-ml-using-azure-machine-learning-f5a505fa748?source=rss-fc418b483cb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5a505fa748</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[automl]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[swabluanalytics]]></category>
            <category><![CDATA[azure-machine-learning]]></category>
            <dc:creator><![CDATA[Raghav Matta]]></dc:creator>
            <pubDate>Tue, 05 May 2020 21:21:38 GMT</pubDate>
            <atom:updated>2020-05-05T21:21:38.894Z</atom:updated>
            <content:encoded><![CDATA[<p>In this tutorial, you will learn the Automated ML feature of Azure ML to achieve the below objectives:</p><ul><li>Create an Azure Machine Learning Workspace</li><li>Create an Automated ML Experiment</li><li>Review the Experiment run results and deploy the best model</li></ul><h3>Requirements</h3><p>Microsoft Azure subscription must be pay-as-you-go / MSDN</p><h3>Introduction</h3><p>We have observed that most automobile companies are providing IoT enabled systems that help in remotely monitoring and providing alerts based on the different metrics generated by a car. We will be implementing a subset of this use case in which we will be using the telemetry stream of time series data from a car to monitor the battery performance and generate an alert if we detect a battery failure is imminent within the next 30 days.</p><h3><strong>What is Automated ML?</strong></h3><p>The most time-intensive task in any data science project is the model generation process, in which you will be iterating over a number of algorithms to find the best or most accurate one for your use case. With automated machine learning, you can automate away this time-intensive tasks. Automated machine learning rapidly iterates over many combinations of algorithms and hyperparameters to help you find the best model based on a success metric of your choosing.</p><p><strong>Exercise 1: Create an Azure Machine Learning Workspace</strong></p><ul><li>Navigate to <a href="https://portal.azure.com/">https://portal.azure.com/</a> and sign in using your Microsoft account credentials with Azure Subscription</li><li>From your Azure dashboard Click on <strong>+Create A Resource</strong>, then <strong>AI + Machine Learning </strong>and select <strong>Machine Learning</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*VCOcMcKLjrEoUSe8V6CkYw.png" /></figure><ul><li>Under the <strong>Main </strong>tab in Create Machine Learning Service Page, enter the below details:</li></ul><pre><strong>Workspace Name:</strong> &lt;Your Name&gt;-aml<strong><br>Subscription: </strong>Select Your Azure Subscription<br><strong>Resource Group(Create New):</strong> aml-rg<br><strong>Location:</strong> East US<br><strong>Workspace Edition:</strong> Enterprise</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xdtN7Xn4om64zW88N1TjuA.png" /></figure><ul><li>Click on <strong>Review + Create,</strong> and then click on <strong>Create</strong>. It may take 5–10 minutes for your deployment to complete</li><li>After your deployment is successful, click on <strong>Go To Resource,</strong> this will open your <strong>Azure Machine Learning Workspace</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*croMZN7tZIOh3OOZCt08Xw.png" /></figure><p><strong>Exercise 2: Create an Automated ML Experiment</strong></p><ul><li>Navigate to your <strong>Azure Machine Learning Workspace </strong>created in the previous exercise, under <strong>Try the new Azure Machine Learning studio </strong>click on <strong>Launch</strong></li><li>In the <strong>Azure Machine Learning studio, </strong>on the left-hand navigation bar click on <strong>Automated ML</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*O2HeN-OQek6_JBlm4C2Gng.png" /></figure><ul><li>Select <strong>+ New automated ML run, </strong>then Select <strong>+ Create dataset, </strong>from the drop-down menu click on <strong>From web files</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*PMCMzWfexaUM-r_ZCMJRWw.png" /></figure><ul><li>Under the <strong>Basic Info </strong>tab, provide the below details and click on <strong>Next</strong>:</li></ul><pre><strong>Web URL: </strong><a href="https://databricksdemostore.blob.core.windows.net/data/connected-car/daily-battery-time-series-v2.csv">https://databricksdemostore.blob.core.windows.net/data/connected-car/daily-battery-time-series-v2.csv</a></pre><pre><strong>Name:</strong> daily-battery-time-series</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BOURg-kX0paeDuzljjqjig.png" /></figure><ul><li>Under <strong>Settings and preview</strong> tab, set the <strong>Column headers </strong>drop-down to <strong>All files have same headers,</strong> and then select <strong>Next</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oUEjGRVjc8RI5yD12WrdpQ.png" /></figure><ul><li>Under the <strong>Schema </strong>tab, include the columns relevant for training and click on <strong>Next</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*21QWExVmxUSAIaI4_tCSTA.png" /></figure><ul><li>Review the dataset details in the <strong>Confirm details</strong> tab and select <strong>Create</strong></li><li>In the <strong>Select dataset </strong>tab, check the daily-battery-time-series dataset and click on <strong>Next</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*ybIVw-O8s6juLn-A01TBSw.png" /></figure><ul><li>Under the <strong>Configure run </strong>tab, set <strong>Experiment Name</strong>: Battery-monitor and <strong>Target Column: </strong>Daily_Cycles_Used and Select <strong>Create a new compute</strong></li><li>In the <strong>New Compute Cluster</strong> pop-up, enter the below details and click on <strong>Create. </strong>New compute creation will take 5-10 minutes to complete. Once completed, click on <strong>Next</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*maRLTfasveJgV9m_9yr6Qw.png" /></figure><ul><li>Under <strong>Task Type and Settings, </strong>select <strong>Time series forecasting, </strong>set <strong>Time column</strong>: Date, <strong>Group by column(s)</strong>: Battery_ID and uncheck Autodetect for <strong>Forecast horizon</strong> and set it for 30 days as we plan to monitor battery performance for next 30 days only. Next, click on <strong>View additional configuration settings</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PFXBMZx_E2AKwyRCq8AJhQ.png" /></figure><ul><li>In the Additional configurations, view the <strong>Primary metrics</strong> available for Time Series Forecasting. Choose <strong>Normalized root mean squared error</strong>, this will be used during model training for optimization</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*V8hcCodexYoISVSXDnKZoA.png" /></figure><ul><li>Given the use case at hand, you may choose to block algorithm(s) which may not be considered during training. Leave this option default and consider all algorithms within the scope of this experiment</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*UaENH7e5LwGLB6ZasXWlwQ.png" /></figure><ul><li><strong>Forecast target lags</strong> can be used to limit the number of lagging rows the target value is based on and <strong>Target rolling window size</strong> is used to limit the history of the training set. Set the <strong>Training job time (hours)</strong>: 1, this will terminate the job in case it is unable to find the best model within the given time. Leave rest of the values as default and click on <strong>Save</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*2D-7Mv7FNmw9zOA5QKwQew.png" /></figure><ul><li>Select <strong>Finish </strong>to trigger your Automated ML Run</li></ul><p><strong>Exercise 3: Review the Experiment Run Results and Deploy the best Model</strong></p><ul><li>After the experiment is complete, you will be able to view the recommended model and the run summary under the <strong>Run Details </strong>tab.</li><li>If the <strong>Run Details </strong>tab does not open automatically, Navigate to <strong>Azure Machine Learning Workspace</strong> and from the left-hand navigation bar click on <strong>Automated ML. </strong>Select <strong>Run 1 </strong>under <strong>Recent Automated ML Runs</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*J6l1F23S9azLTi7p5guKcQ.png" /></figure><ul><li>In the Details tab, you will be able to see the <strong>Best Model Summary</strong> and <strong>Run Summary</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6QEq2R6V7FS_zURWSuOxiQ.png" /></figure><ul><li>Switch to the <strong>Models </strong>tab, to see a tabular view of different iterations and the primary metric i.e. Normalized root mean squared error score for each iteration, which provides us the difference between predicted and actual value to determine the model accuracy.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qcHq3vNejFTeZQOZ48t-PQ.png" /></figure><ul><li>Navigate back to <strong>Details </strong>tab, and under the <strong>Best Model Summary </strong>click on the recommended <strong>Algorithm Name</strong>, in my case it is <strong>Prophet Model</strong></li><li>Click on <strong>View all other metrics</strong>, to evaluate the run performance for the recommended model and then click on <strong>Close</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p9sHhOXUq47PrKX8TS07UA.png" /></figure><ul><li>Next, click on <strong>Visualization </strong>to view the model performance curve i.e. Predicted vs True</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZrnMm-FKFRbwGSMcSi3Rdw.png" /></figure><ul><li>Navigate back to <strong>Model details, </strong>and click on <strong>Deploy. </strong>In <strong>Deploy a Model </strong>pop-up set <strong>Name: </strong>battery-monitor and <strong>Compute Type:</strong> Azure Container Instance (ACI) and click on <strong>Deploy. </strong>The model deployment process will register the model, create the deployment image, and deploy it as a scoring web service in an Azure Container Instance (ACI).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*2vU93JE5yAMXOCZxShlGsw.png" /></figure><ul><li>Once the Deployment is completed, it will be updated in <strong>Deployment Status</strong> under the <strong>Model Details</strong> tab</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*zD63gQEIZ7UF5dMJAuSbfQ.png" /></figure><ul><li>Click on <strong>battery-monitor </strong>in the Deploy Status, this will open the deployment <strong>Endpoints </strong>tab with the Deployment State, URI and other details</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*JH6yOd0tSIpMpDMfbo1-bA.png" /></figure><p>In the Next Part, we will apply this forecast model to a Spark streaming job in order to make predictions against streaming data using Azure Databricks</p><p><strong>References:</strong></p><ol><li><a href="https://github.com/microsoft/MCW-Machine-Learning">https://github.com/microsoft/MCW-Machine-Learning</a></li><li><a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#primary-metric">https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#primary-metric</a></li><li><a href="https://docs.microsoft.com/en-in/azure/machine-learning/tutorial-first-experiment-automated-ml">https://docs.microsoft.com/en-in/azure/machine-learning/tutorial-first-experiment-automated-ml</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5a505fa748" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with Databricks on Azure]]></title>
            <link>https://medium.com/swlh/getting-started-with-databricks-on-azure-f3bceda8b918?source=rss-fc418b483cb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3bceda8b918</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[swabluanalytics]]></category>
            <dc:creator><![CDATA[Raghav Matta]]></dc:creator>
            <pubDate>Mon, 04 May 2020 21:03:08 GMT</pubDate>
            <atom:updated>2020-05-04T21:03:08.347Z</atom:updated>
            <content:encoded><![CDATA[<p>In this tutorial, you will learn Azure Databricks to achieve the below objectives:</p><ul><li>Create an Azure Databricks workspace using Azure portal</li><li>Create and configure a spark cluster in Azure Databricks</li></ul><h4>Requirements</h4><p>Microsoft Azure subscription must be pay-as-you-go / MSDN / Trial subscription</p><h4>What is Databricks?</h4><p>Databricks with Azure provides self-managed analytical platform, it breaks the silos between Data Engineers, Business Users, Data Scientists to collaborate and deliver a single solution from ingestion to production</p><h4>How can Databricks help your organization?</h4><p>Databricks can help organizations streamline the below tasks:</p><ul><li>Enables data teams to work with big data efficiently, through ingestion to production</li><li>A single source to connect disparate data sources and build data pipelines</li><li>Data preparation and feature engineering for machine learning models</li><li>Data exploration and profiling using ML and BI applications</li><li>Extracting active insights from data for decision making</li></ul><h4>Exercise 1: Create an Azure Databricks workspace using Azure portal</h4><ul><li>Navigate to <a href="https://portal.azure.com/">https://portal.azure.com/</a> and sign in using your Microsoft account credentials with Azure Subscription</li><li>From your Azure dashboard Click on <strong>+Create A Resource</strong>, then <strong>Analytics</strong> and select <strong>Azure Databricks</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*-j3eMlFRSoE929qBuUBZ5A.png" /></figure><ul><li>Under the <strong>Basics </strong>tab in Create Azure Databrick Service Page, enter the below details:</li></ul><pre><strong>Subscription: </strong>Select Your Azure Subscription<br><strong>Resource Group(Create New):</strong> databricks-rg<br><strong>Workspace Name:</strong> &lt;Your Name&gt;dbc<br><strong>Location:</strong> East US<br><strong>Pricing Tier:</strong> Standard</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5B8Gh6FXIKxWoZq28422mw.png" /></figure><ul><li>Click on <strong>Review + Create,</strong> and then click on <strong>Create</strong>. It may take 5–10 minutes for your deployment to complete</li><li>After your deployment is successful, click on <strong>Go To Resource,</strong> this will open your Azure Databricks Service</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*VwXSd362gElix1JCZZGu1Q.png" /></figure><ul><li>Click on <strong>Launch Workspace</strong> to open Azure Databricks and sign in using the same Microsoft credentials associated with your Azure subscription</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VhDBxf3OiLRqSxI0ZuqhDQ.png" /></figure><h4>Exercise 2: Create and configure a spark cluster in Azure Databricks</h4><ul><li>Navigate to your Azure Databricks Workspace homepage, from the left-hand menu click on <strong>Clusters, </strong>and then click on <strong>+ Create Cluster</strong></li><li>In the New Cluster Tab, enter the <strong>Cluster Name </strong>as spark-cluster</li><li><strong>Cluster Mode</strong>, can be configured as Standard/ High Concurrency. Standard is the default cluster mode and is recommended for single users supporting Python, R, SQL, and Scala whereas High Concurrency mode is optimized to run concurrent Python, R, and SQL workloads. Use <strong>Standard </strong>Mode for this demo</li><li><strong>Pool</strong>, can be used in a collaborative environment where users are creating and releasing computes, it will keep a defined number of ready instances to reduce cluster startup time. Choose the <strong>default </strong>option</li><li>Azure Databricks offers several types of runtime and several versions of those runtime types in the <strong>Databricks Runtime Version</strong> drop-down when you create or edit a cluster. Choose the <strong>default </strong>Runtime</li><li>Make sure you <strong>enable</strong> autoscaling, and to save on cost select to terminate cluster after <strong>30 minutes</strong> of inactivity</li><li>A cluster consists of one driver node and worker nodes. You can pick separate cloud provider instance types for the driver and worker nodes, although by default the driver node uses the same instance type as the worker node. Choose <strong>default </strong>options for driver and worker nodes</li><li>Verify the Cluster configuration below, and click on <strong>Create Cluster</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*ktBoMuTly289l0bvdQviRg.png" /></figure><ul><li>After the cluster is created, you can view the same in the <strong>Clusters </strong>tab</li><li>Now you can create a Notebook and start designing your analytical solutions</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3bceda8b918" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/getting-started-with-databricks-on-azure-f3bceda8b918">Getting Started with Databricks on Azure</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>