<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sharma Aman on Medium]]></title>
        <description><![CDATA[Stories by Sharma Aman on Medium]]></description>
        <link>https://medium.com/@sharma.aman7823?source=rss-10cf33dcd78f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*shIpP5lScx-VJMwcjNtfYA@2x.jpeg</url>
            <title>Stories by Sharma Aman on Medium</title>
            <link>https://medium.com/@sharma.aman7823?source=rss-10cf33dcd78f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:50:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sharma.aman7823/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Salesforce Ingestion Connector - Databricks]]></title>
            <link>https://medium.com/@sharma.aman7823/salesforce-ingestion-connector-databricks-085a3b9e4d85?source=rss-10cf33dcd78f------2</link>
            <guid isPermaLink="false">https://medium.com/p/085a3b9e4d85</guid>
            <category><![CDATA[microsoft-azure]]></category>
            <category><![CDATA[databricks-connector]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[salesforce-ingestion]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Sharma Aman]]></dc:creator>
            <pubDate>Sat, 08 Aug 2026 12:27:17 GMT</pubDate>
            <atom:updated>2026-08-23T09:07:20.394Z</atom:updated>
            <content:encoded><![CDATA[<p>The Salesforce Ingestion Connector simplifies the process by enabling secure, configurable, and scalable ingestion of Salesforce objects into Databricks. This article walks through the implementation, configuration, usage, advantages and limitations for building a reliable ingestion pipeline using salesforce ingestion connector.</p><h4>Features</h4><ol><li>Supports both SCD type 1 and type 2</li><li>SCD type 1 by default for non formula fields.</li><li>Supports incremental ingestion for formula fields(BETA feature).</li><li>Supports Automated Schema Evolution</li><li>Automatically creates tables reducing the setup overhead</li><li>Supports Row-Filtering</li></ol><h4>Understanding Salesforce Objects</h4><p>Salesforce objects contain two broad categories of fields: <strong>Non-Formula Fields</strong> and <strong>Formula Fields</strong>. Understanding the difference between them is important when designing an incremental ingestion strategy.</p><ul><li><strong>Non-Formula Fields:</strong> These are fields whose values are not derived using any formula. Whenever one of these fields is created or updated, Salesforce updates system cursor fields like <strong>LastModifiedDate</strong> and <strong>SystemModstamp</strong>, making them suitable for incremental data extraction.</li><li><strong>Formula Fields:</strong> Formula fields values are based on any formula such as TODAY() and values for these are computed at runtime when any ingestion starts. Since values are computed at runtime , <strong>LastModifiedDate</strong> and <strong>SystemModstamp</strong> remain unchanged even though the formula computes a different result, meaning changes in formula values alone are not detected by cursor-based columns which is a huge drawback.</li></ul><h4>Configuration and Setup</h4><p>I will be using salesforce free trial account and Databricks community edition to showcase the steps.</p><p><strong>Step 1</strong>: Create a free 30-day Salesforce trial account. By default, Salesforce provides sample objects (tables) and data that you can use for practice and exploration. Once your account has been created, you will be redirected to the Salesforce home page. From there, click the <strong>Setup</strong> icon in the top-right corner and select <strong>Object Manager</strong>. In Object Manager, you will find a list of all the available objects (tables) in your Salesforce account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hWQy6PagbaBtwMDMApWL4A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PkODXn2hOpw2V-uIP4l4Eg.png" /></figure><p><strong>Step 2</strong>: Open the <strong>Databricks workspace</strong> and, from the left navigation pane, go to the <strong>Data Engineering</strong> section and select <strong>Salesforce Connector</strong>.</p><p>You will be prompted to enter a <strong>Connection Name</strong>. Provide a user-defined name based on your organization’s naming conventions. Then, click <strong>Authenticate and Create Connection</strong>.</p><p>You will be asked to provide your <strong>Salesforce username and password</strong>. Once the credentials are entered and successfully authenticated, the connection will be created and will be visible in the Salesforce Connector section.</p><p><strong><em>Note: For any production grade solution, you might need a service account and installing the Databricks ingestion app in the salesforce. The user should have necessary permission set mapped for any table access.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OoMvqesijsBaqHy1pctFow.png" /></figure><p><strong>Step 3</strong>: Select the <strong>Catalog</strong> and <strong>Schema</strong> where you want to load the data. Next, select the <strong>table</strong> you want to ingest and configure a <strong>schedule</strong> if you want the ingestion to run at specific intervals.</p><p>Since I already have a Salesforce connection named <strong>“demo”</strong>, I will use the same connection for this setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hEB0w4SFjdt-2f6QC9rytw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4VMg_25TiKajBDY06-ZDbA.png" /></figure><p>Select the <strong>tables</strong> you want to ingest. You can either retain the original table names or provide user-defined names based on your requirements.</p><p>For this example, I have selected the <strong>Contact</strong> and <strong>Contact Feed</strong> tables. You can choose the tables and table names according to your use case. As shown in screenshot below i am renaming contact table as contact_demo.</p><p>You can also enable or disable <strong>History Tracking</strong> for each table:</p><ul><li>If <strong>History Tracking is disabled</strong>, the default behavior is <strong>SCD Type 1</strong> for non-formula fields, while formula fields are ingested using a <strong>full load</strong>.</li><li>If <strong>History Tracking is enabled</strong>, the table uses <strong>SCD Type 2</strong> for non-formula fields. In this case, additional <strong>Start Date</strong> and <strong>End Date</strong> columns are added to table to track changes over time.</li></ul><p>For this example, I have enabled <strong>History Tracking</strong> for the <strong>Contact Feed</strong> table.</p><p>Additionally, you can select or deselect individual <strong>columns</strong> based on the data you want to ingest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*116qK8M0JbTbP4cF5AXLsA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sot-pECbx798wP97oISguw.png" /></figure><p>If <strong>Include Future Columns</strong> is enabled, any new fields added to the Salesforce table in the future will be automatically ingested into the Databricks table.</p><p>If a column is <strong>deleted from the source</strong>, the pipeline will not fail. Instead, the deleted column will continue to remain in the Databricks table.</p><p>Similarly, if a column is <strong>renamed in Salesforce</strong>, the Databricks table will retain the existing column with the old name and also ingest the renamed column as a new column.</p><p>If you want to synchronize the Databricks table schema with Salesforce, you can trigger a <strong>Full Load</strong> for the table.</p><p><strong>Step4 :</strong> Select the <strong>Target Catalog</strong> and <strong>Target Schema</strong> where you want to store the ingested data. If you want to automate the ingestion process, configure a <strong>schedule</strong> based on your requirements.</p><p>Once all the required settings are configured, click <strong>Save and Close</strong> to complete the setup.</p><h4><strong>Lets run the pipeline!!</strong></h4><p>Just head to jobs and pipeline section where you will be able to see the pipeline you just created. Click on the pipeline name and you will see an interface similar to this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CddiddQKT1E1YKHe-4qrnA.png" /></figure><p>If you look closely at the <strong>Canvas</strong>, you will notice that the <strong>contact_demo (contact)</strong> table has two incoming streams, while the <strong>contactfeed</strong> table has only one.</p><p>This is because the <strong>Contact</strong> table in Salesforce contains formula fields. Databricks separates the formula and non-formula fields into two different views:</p><ul><li>The <strong>non-formula fields</strong> view is used for <strong>incremental ingestion</strong> based on cursor columns.</li><li>The <strong>formula fields</strong> view is always processed as a <strong>full snapshot</strong>.</li></ul><p>When the pipeline is triggered, both streams for the Contact table run <strong>in parallel</strong>.</p><p>To manually trigger the pipeline, click the <strong>Start</strong> button in the top-right corner.</p><p><strong>Lets have look at the data:</strong></p><p><strong>Contact Feed</strong>: START_AT, END_AT added at the end as it was configured as SCD type2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kXD59yN_ptdabX2N4W2H9g.png" /></figure><p><strong>Contact_Demo</strong>: Configured as SCD type1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EquhsOuMVzosSTEqMYOasQ.png" /></figure><p>Now i am going to change the LastName for this contact in salesforce from <strong>Benioff (Sample) </strong>to just <strong>Benioff </strong>and rerun the pipeline. Below is the result where LastName is updated for the record as it is SCD type1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWLZJDKn6zVDgT5WqsEUiw.png" /></figure><h4><strong>Incremental Ingestion for formula fields</strong></h4><p>Since Salesforce formula fields are calculated dynamically at runtime, the pipeline performs a <strong>full snapshot</strong> of all formula field by default.</p><p>Since Salesforce <strong>formula fields</strong> are calculated dynamically at runtime, the pipeline performs a <strong>full snapshot</strong> of all formula fields by default.</p><p>When <strong>incremental ingestion for formula fields</strong> is enabled, the formula logic is recreated and evaluated within Databricks using <strong>materialized views</strong>. As new or updated records are ingested, the materialized views are automatically refreshed to recalculate the formula fields.</p><p>The <strong>materialized view</strong> serves as the final output and provides the latest calculated values while retaining the performance and scalability benefits of <strong>incremental data ingestion</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CqEcDOy-2HBgrwhk6bTHqA.png" /></figure><p><strong>Here are few limitations of incrementally ingesting formula fiel</strong>ds:</p><ol><li>It can only be applied to a new pipeline.</li><li>Some of the formula fields may produce different result and some functions are not supported.</li><li>It disables the pipeline UI feature.</li><li>If any formula fields in a table is referencing a field from another table, both the tables should be part of same pipeline.</li><li>It is still a beta feature and no specific timeline when it will become GA.</li></ol><p>To enable incremental ingestion for a pipeline, you must add the below line of code to pipeline’s YAML code.</p><pre>pipelines.enableSalesforceFormulaFieldsMVComputation: &quot;true&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*E7ABpRqLOWE9oxpSir8qsA.png" /></figure><h4><strong>Pipeline API to trigger full load</strong></h4><p>While the UI capabilities of the Salesforce Ingestion Pipeline provide a simple and user-friendly experience, they are not as flexible as invoking the pipeline through the REST API, particularly for automation and operational workflows.</p><p>With the UI-based approach, this would require manual intervention to identify the failed tables and initiate the full load. However, the <strong>API-based approach</strong> provides a more automated solution.</p><p>Think of it this way: the tables that fail during a UI-based pipeline run can be identified from the <strong>Event Log</strong> using the query below, with some modifications as required.</p><p>The list of failed tables can then be passed to the <strong>Pipeline API</strong> as <strong>full refresh parameters</strong>, allowing a full refresh to be triggered only for the affected tables.</p><p>This approach makes the solution more <strong>robust and automated</strong>, while significantly reducing the need for manual intervention.</p><pre>result_df = spark.sql(f&quot;&quot;&quot;<br>SELECT update_id, pipeline_id<br>FROM system.lakeflow.pipeline_update_timeline<br>WHERE trigger_details.job_task.job_task_run_id = &#39;add your job run id&#39; <br>  AND trigger_details.job_task.job_id = &#39;add your job id&#39;<br>  AND result_state IN (&#39;FAILED&#39;)<br>LIMIT 1<br>&quot;&quot;&quot;)<br><br># Extract the values from the DataFrame<br>if(result_df.count()&gt;0):<br>  first_row = result_df.first()<br>  update_id = first_row[&#39;update_id&#39;]<br>  pipeline_id = first_row[&#39;pipeline_id&#39;]<br>else:<br>  dbutils.notebook.exit(&quot;No failed tables found&quot;)<br><br><br>print(update_id,&#39;  &#39;,pipeline_id)<br><br>table_names_df = spark.sql(f&quot;&quot;&quot;<br>SELECT <br>  origin.flow_name as table_name<br>FROM event_log(&#39;{pipeline_id}&#39;) <br>WHERE level = &#39;ERROR&#39; <br>  AND event_type = &#39;flow_progress&#39; <br>  AND origin.update_id = &#39;{update_id}&#39;<br>&quot;&quot;&quot;)<br>table_names_list = [row[&#39;table_name&#39;] for row in table_names_df.collect()]<br>print(table_names_list)</pre><p>Trigger full load for a pipeline using REST API.</p><pre>import requests<br>import json<br><br># 1. Dynamically retrieve the workspace URL and authentication token<br>workspace_url = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().get()<br>workspace_instance = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().get()<br># 2. Configure endpoint variables<br>pipeline_id = &quot;add your pipeline id&quot;<br>url = f&quot;{workspace_instance}/api/2.0/pipelines/{pipeline_id}/updates&quot;<br>headers = {<br>    &quot;Authorization&quot;: f&quot;Bearer {workspace_url}&quot;,<br>    &quot;Content-Type&quot;: &quot;application/json&quot;<br>}<br># Optional parameters (e.g., {&quot;full_refresh&quot;: &quot;true&quot;} to reprocess everything)<br>body = {&quot;full_refresh_selection&quot;: table_names_list}#failed tables <br># 3. Post request to trigger the pipeline<br>response = requests.post(url, headers=headers, data=json.dumps(body))<br>if response.status_code == 200:<br>    print(&quot;Pipeline update triggered: &quot;, response.json())<br>else:<br>    print(f&quot;Failed to trigger pipeline. Error {response.status_code}: {response.text}&quot;)</pre><p><strong>Happy Learning!!</strong></p><p>Regards!<br>Learning is a journey best taken together — sharing ideas today creates better solutions tomorrow.</p><p><strong>Remember!!</strong></p><ol><li>Every setback or failure in life may be part of a divine plan, protecting you from something you cannot see yet.</li><li>(1.01) ³⁶⁵ is 37 times (1) ³⁶⁵<br> Small growth × consistency → massive difference</li></ol><h4><strong>About me!!</strong></h4><p>Follow Me on Linked In : <a href="https://www.linkedin.com/in/aman-sharma-a2a7141b1/">Aman Sharma</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=085a3b9e4d85" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>