<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arkimetrix Analytics on Medium]]></title>
        <description><![CDATA[Stories by Arkimetrix Analytics on Medium]]></description>
        <link>https://medium.com/@arkimetrix.analytics?source=rss-da38359c288e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YGXFpNMqcUe73p5pcmTQXQ.jpeg</url>
            <title>Stories by Arkimetrix Analytics on Medium</title>
            <link>https://medium.com/@arkimetrix.analytics?source=rss-da38359c288e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:40:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arkimetrix.analytics/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The “No Plan, No Code” Rule: How we scale AI coding without the chaos]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@arkimetrix.analytics/the-no-plan-no-code-rule-how-we-scale-ai-coding-without-the-chaos-dd7874cacdca?source=rss-da38359c288e------2"><img src="https://cdn-images-1.medium.com/max/1376/1*kcL8he3it7rkYTrEp8jngg.png" width="1376"></a></p><p class="medium-feed-snippet">AI coding agents are incredibly productive &#x2014; until they&#x2019;re not. Here&#x2019;s how our team went from &#x201C;AI entropy&#x201D; to a structured, auditable&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@arkimetrix.analytics/the-no-plan-no-code-rule-how-we-scale-ai-coding-without-the-chaos-dd7874cacdca?source=rss-da38359c288e------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@arkimetrix.analytics/the-no-plan-no-code-rule-how-we-scale-ai-coding-without-the-chaos-dd7874cacdca?source=rss-da38359c288e------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd7874cacdca</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[developer-tools]]></category>
            <dc:creator><![CDATA[Arkimetrix Analytics]]></dc:creator>
            <pubDate>Tue, 24 Mar 2026 18:23:00 GMT</pubDate>
            <atom:updated>2026-03-24T19:22:01.640Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Part 7. Python Polars — Working with Fixed Width Files (edge case)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@arkimetrix.analytics/part-7-python-polars-working-with-fixed-width-files-edge-case-9a7b1e710b98?source=rss-da38359c288e------2"><img src="https://cdn-images-1.medium.com/max/720/1*V8ML4RyNoFuVb0CZyPQLIQ.png" width="720"></a></p><p class="medium-feed-snippet">Navigating and Manipulating Fixed Width Files using Python Polars</p><p class="medium-feed-link"><a href="https://medium.com/@arkimetrix.analytics/part-7-python-polars-working-with-fixed-width-files-edge-case-9a7b1e710b98?source=rss-da38359c288e------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@arkimetrix.analytics/part-7-python-polars-working-with-fixed-width-files-edge-case-9a7b1e710b98?source=rss-da38359c288e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a7b1e710b98</guid>
            <category><![CDATA[data-munging]]></category>
            <category><![CDATA[data-wrangling]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Arkimetrix Analytics]]></dc:creator>
            <pubDate>Tue, 02 Jan 2024 00:05:54 GMT</pubDate>
            <atom:updated>2024-01-02T00:05:54.402Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Optimize Business Performance with Acterys: Top Use Cases for Extended Planning & Analysis]]></title>
            <link>https://medium.com/@arkimetrix.analytics/optimize-business-performance-with-acterys-top-use-cases-for-extended-planning-analysis-9d4773cf5a10?source=rss-da38359c288e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d4773cf5a10</guid>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[financial-forecasting]]></category>
            <category><![CDATA[anaplan]]></category>
            <category><![CDATA[budgeting-and-planning]]></category>
            <category><![CDATA[hyperion-planning]]></category>
            <dc:creator><![CDATA[Arkimetrix Analytics]]></dc:creator>
            <pubDate>Fri, 03 Nov 2023 17:05:12 GMT</pubDate>
            <atom:updated>2023-11-03T17:05:12.029Z</atom:updated>
            <content:encoded><![CDATA[<h3>Even Microsoft Agrees: Acterys is a Breakthrough in EPM Technology</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*jBiDOfDhteKBWzWR7f833g.png" /><figcaption>Seamless Synchronization: Acterys’s Robust Integration Connects with Any Data Source with Unmatched Ease</figcaption></figure><p><strong>Enterprise Performance Management (EPM)</strong> is a process and software framework that helps organizations link their strategies to their plans and execution. EPM includes methods for managing an organization’s performance through the planning, budgeting, forecasting, and reporting of business activities to achieve a comprehensive view of its financial health and operational efficiency.</p><p>Navigating the complexities of enterprise performance management is more daunting than ever. Teams are swamped with data from multiple systems, leading to inconsistent information, while dealing with a need for real-time insights. The result? A chaotic jumble of spreadsheets, and a struggle to produce accurate forecasts, budgets, and reports.</p><p><strong><em>Time lost to manual data wrangling is time stolen from strategic thinking.</em></strong></p><p>We all know the frustration when the data doesn’t line up, leading to costly mistakes. This inefficiency is more than a nuisance — it’s a direct threat to your business’s nimbleness and expansion. Missed opportunities and sluggish decision-making are the unfortunate byproducts of a disjointed data landscape.</p><h3>Enter the dynamic duo of Acterys and Power BI.</h3><p><a href="https://arkimetrix.com/acterys/">Acterys</a> brings unparalleled agility to your financial planning and analysis, automating data consolidation from various sources into a single version of the truth.</p><p>Underneath the surface, <a href="https://arkimetrix.com/acterys/">Acterys</a> operates on proven Microsoft technologies like SSAS and Power BI, and that brings the following advantages:</p><ol><li>Ensures that you have access to a wide pool of skilled yet cost-effective developers.</li><li>The familiar interfaces of Power BI and Excel make adoption a breeze, simplifying change management.</li></ol><blockquote>Built on Power BI, Acterys delivers an EPM solution that is robust yet flexible, turning what was once a tangled web of spreadsheets into a streamlined, easy to work with interconnected platform, or replace any existing expensive proprietary system and lower your costs while improving performance.</blockquote><h3>Need Proof?</h3><p>See this announcement from May 2023: “<a href="https://acterys.com/acterys-licenses-xpa-technology-to-microsoft/">Acterys Licenses xP&amp;A Technology to Microsoft: Revolutionizing Planning &amp; Analytics Integration with Dynamics 365</a>”</p><blockquote>Acterys’s xP&amp;A technology now underpins Microsoft’s Dynamics 365 xP&amp;A offering, underscoring its revolutionary impact. This integration works flawlessly with Azure cloud, Power BI, Teams, and Excel, creating dynamic dashboards and enabling AI-driven planning processes with rapid writeback features.</blockquote><p>The confidence in this partnership is echoed by <a href="https://www.linkedin.com/in/mkratky?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAAWkvQBrO_fwDzbafzLfgck-DDW8SoH_-8"><strong>Martin Kratky</strong></a><strong>, CEO and Founder of </strong><a href="https://acterys.com/"><strong>Acterys</strong></a>, who sees it as a “significant step forward” in empowering businesses with integrated solutions that drive meaningful outcomes. <strong>Georg Glantschnig, VP of Dynamics 365 Finance at Microsoft,</strong> also highlights the shared vision to transform business planning and analytics, indicating that <a href="https://acterys.com/">Acterys’s</a> technology is pivotal in this advancement. <a href="https://landing.acterys.com/financial-reporting-planning-in-power-bi?hs_preview=NcRNXhqg-134634385997"><strong>Learn more by viewing this webinar</strong></a><strong> by </strong><a href="https://www.linkedin.com/in/mike-zack-98023bb?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAH-CrAByQa0z-MIoumrduYjv1pM3kuy2Fs"><strong>Mike Zack</strong></a><strong>, Acterys COO.</strong></p><p>With 82% of finance leaders acknowledging the importance of AI and automation for strategic business development and innovation, Acterys and Microsoft are addressing the complexity that has hindered adoption. This collaboration is not just enhancing the EPM landscape; it’s setting new industry benchmarks and unlocking unprecedented possibilities for all stakeholders.</p><h3>Drive Results, Not Data Doubts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*ba0jorUSMVNFJ7DWXGdYDg.png" /><figcaption>Arkimetrix: Mastering Data Integration from Cloud to Ground, Powering Planning &amp; Analysis with Acterys</figcaption></figure><h4><a href="https://arkimetrix.com/acterys/">Arkimetrix</a> is an Acterys Augmented Business Application Delivery Partner.</h4><blockquote>We bring not just extensive technology experience but also deep domain knowledge to turn this unparalleled EPM solution into your competitive edge.</blockquote><p><a href="https://arkimetrix.com/acterys/">Arkimetrix and Acterys</a> can help elevate your enterprise’s performance with the and take your strategic planning from reactive to proactive. We can help with solutions for any use case, and here is a partial list:</p><blockquote>#Demand Planning, #Headcount and Labor Planning, #Financial Close and Consolidation, #Sales and Operations Planning (S&amp;OP), #Cash Flow Forecasting, #Capital Expenditure (CapEx) Planning, #Revenue Forecasting, #Supply Chain Optimization, #Risk Management, #Project Portfolio Management, #Marketing Budget Allocation, #Product Profitability Analysis, #Strategic Future Scenario Modeling and more.</blockquote><p><strong>Contact us for a no obligation consult:</strong></p><p>North America Toll-free +1 (800) 701–4342 or email <a href="/service/mailto:info@arkimetrix.com">info@arkimetrix.com</a></p><p>#Acterys #PowerBI #EPMsolutions #FinancialPlanning #DataIntegration #BusinessIntelligence #Analytics #Forecasting #MicrosoftPartnership #PlanningAndAnalysis #AIinFinance #Arkimetrix #StrategicPlanning #PerformanceManagement #DataDrivenDecisions #Oracle Hyperion #SAP BPC #IBM Planning Analytics #Anaplan #Workday Adaptive Planning #OneStream #Host Analytics #Infor d/EPM #CCH Tagetik #Board #Vena Solutions #Jedox #Longview #Prophix #Planful #Host Analytics</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d4773cf5a10" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 6 — From Pandas to Polars: Breaking Free from Index Dependency]]></title>
            <link>https://medium.com/@arkimetrix.analytics/part-6-breaking-free-from-index-dependency-from-pandas-to-polars-1532943eed18?source=rss-da38359c288e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1532943eed18</guid>
            <category><![CDATA[polar]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Arkimetrix Analytics]]></dc:creator>
            <pubDate>Wed, 21 Jun 2023 21:25:56 GMT</pubDate>
            <atom:updated>2023-06-25T18:46:46.437Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*-B4Kzb008t3_h86U7bGauw.png" /><figcaption>Python Polars Cookbook = POOK BOOK</figcaption></figure><p>I recently had a fascinating conversation with young data scientist from a prestigious financial institution, and our discussion naturally gravitated towards the topic of data manipulation. As a passionate advocate of Polars, I eagerly shared my enthusiasm, explaining how Polars surpasses Pandas in most transformations. However, the young gent expressed a valid concern: Polars does not have indexes. True confession, I did not do a good job explaining the true power of Polars and why it does not use or need indexes.</p><blockquote>So, in this blog post, I will try to address the concerns of Pandas users who may be hesitant to explore Polars, and hopefully provide useful insights on how Polars can liberate you from index dependency.</blockquote><h3>A Simple Transformation Example</h3><p>Let’s begin with a simple transformation example that highlights the difference between Pandas and Polars. Consider a DataFrame with columns ‘A’ and ‘B’, where we want to select a subset of data using row numbers ([0, 2, 4]) and the column name (‘A’).</p><p><strong><em>In Pandas:</em></strong></p><pre>import pandas as pd<br><br># Create a DataFrame<br>data = {<br>    &#39;A&#39;: [1, 2, 3, 4, 5],<br>    &#39;B&#39;: [6, 7, 8, 9, 10]<br>}<br>df = pd.DataFrame(data)<br><br># Select a subset of data using Pandas default numerical index<br>subset_pandas = df.loc[[0, 2, 4], &#39;A&#39;]</pre><p><strong><em>In Polars:</em></strong></p><pre>import polars as pl<br><br># Create a DataFrame<br>data = {<br>    &#39;A&#39;: [1, 2, 3, 4, 5],<br>    &#39;B&#39;: [6, 7, 8, 9, 10]<br>}<br>df = pl.DataFrame(data)<br><br># Select a subset of data using Polars default numerical index<br>subset_polars = df.select(pl.col(&#39;A&#39;)).with_row_count(<br>    name=&quot;index&quot;).filter(pl.col(&quot;index&quot;).is_in([0, 2, 4]))</pre><blockquote>While the Polars expression may appear more verbose, it’s important to note that operations requiring selection by specific row numbers are relatively infrequent in most transformations. Polars provides alternative approaches that do not heavily rely on indexes, resulting in concise and efficient code for common data manipulation tasks.</blockquote><h3>A More Complex Transformation Example:</h3><p>To showcase a more complex transformation, let’s consider an example adapted from the <a href="https://pandas.pydata.org/docs/user_guide/cookbook.html#cookbook-multi-index">Pandas Cookbook</a>. We start with a DataFrame that contains columns with hierarchical names.</p><p><strong><em>In Pandas:</em></strong></p><pre>import pandas as pd<br><br>df = pd.DataFrame(<br>    {<br>        &quot;row&quot;: [0, 1, 2],<br>        &quot;One_X&quot;: [1.1, 1.1, 1.1],<br>        &quot;One_Y&quot;: [1.2, 1.2, 1.2],<br>        &quot;Two_X&quot;: [1.11, 1.11, 1.11],<br>        &quot;Two_Y&quot;: [1.22, 1.22, 1.22],<br>    }<br>)<br><br>In [1]: df<br>Out[1]: <br>   row  One_X  One_Y  Two_X  Two_Y<br>0    0    1.1    1.2   1.11   1.22<br>1    1    1.1    1.2   1.11   1.22<br>2    2    1.1    1.2   1.11   1.22<br><br># With Hierarchical Columns<br>df.columns = pd.MultiIndex.from_tuples(<br>    [tuple(c.split(&quot;_&quot;)) for c in df.columns])<br><br>In [2]:df<br>Out[2]: <br>  row  One        Two      <br>  NaN    X    Y     X     Y<br>0   0  1.1  1.2  1.11  1.22<br>1   1  1.1  1.2  1.11  1.22<br>2   2  1.1  1.2  1.11  1.22<br><br># Now stack &amp; Reset<br>df = df.stack(0).reset_index(1)<br><br>In [3]:df<br>Out[3]: <br>  level_1     X     Y<br>0     One  1.10  1.20<br>0     Two  1.11  1.22<br>1     One  1.10  1.20<br>1     Two  1.11  1.22<br>2     One  1.10  1.20<br>2     Two  1.11  1.22<br><br># And fix the labels (Notice the label &#39;level_1&#39; got added automatically)<br>df.columns = [&quot;Sample&quot;, &quot;All_X&quot;, &quot;All_Y&quot;]<br><br>In [4]: df<br>Out[4]: <br>  Sample  All_X  All_Y<br>0    One   1.10   1.20<br>0    Two   1.11   1.22<br>1    One   1.10   1.20<br>1    Two   1.11   1.22<br>2    One   1.10   1.20<br>2    Two   1.11   1.22</pre><p>The example above is somewhat reminiscent of spreadsheet operations and the starting dataframe is not the best data structure for fast calculations and transformations.</p><p>Let’s replicate the operations above using Polars. The objective is to create the ending df with “All_X” and “All_Y” columns. The steps are first broken out for clarity, but feel free to jump ahead if you want to see that in one codeblock.</p><pre>import polars as pl<br><br># Create a DataFrame with two columns<br>df = pl.DataFrame(<br>    {<br>        &quot;row&quot;: [0, 1, 2],<br>        &quot;One_X&quot;: [1.1, 1.1, 1.1],<br>        &quot;One_Y&quot;: [1.2, 1.2, 1.2],<br>        &quot;Two_X&quot;: [1.11, 1.11, 1.11],<br>        &quot;Two_Y&quot;: [1.22, 1.22, 1.22],<br>    }<br>)<br><br>In [5]:df<br>Out[5]: <br>shape: (3, 5)<br>┌─────┬───────┬───────┬───────┬───────┐<br>│ row ┆ One_X ┆ One_Y ┆ Two_X ┆ Two_Y │<br>│ --- ┆ ---   ┆ ---   ┆ ---   ┆ ---   │<br>│ i64 ┆ f64   ┆ f64   ┆ f64   ┆ f64   │<br>╞═════╪═══════╪═══════╪═══════╪═══════╡<br>│ 0   ┆ 1.1   ┆ 1.2   ┆ 1.11  ┆ 1.22  │<br>│ 1   ┆ 1.1   ┆ 1.2   ┆ 1.11  ┆ 1.22  │<br>│ 2   ┆ 1.1   ┆ 1.2   ┆ 1.11  ┆ 1.22  │<br>└─────┴───────┴───────┴───────┴───────┘<br><br># melt the df<br>df = (df<br>      .melt(id_vars=&#39;row&#39;,<br>            value_vars=[&#39;One_X&#39;,&#39;One_Y&#39;,&#39;Two_X&#39;,&#39;Two_Y&#39;])<br>      )<br><br>In [6]df<br>Out[6]: <br>shape: (12, 3)<br>┌─────┬──────────┬───────┐<br>│ row ┆ variable ┆ value │<br>│ --- ┆ ---      ┆ ---   │<br>│ i64 ┆ str      ┆ f64   │<br>╞═════╪══════════╪═══════╡<br>│ 0   ┆ One_X    ┆ 1.1   │<br>│ 1   ┆ One_X    ┆ 1.1   │<br>│ 2   ┆ One_X    ┆ 1.1   │<br>│ 0   ┆ One_Y    ┆ 1.2   │<br>│ …   ┆ …        ┆ …     │<br>│ 2   ┆ Two_X    ┆ 1.11  │<br>│ 0   ┆ Two_Y    ┆ 1.22  │<br>│ 1   ┆ Two_Y    ┆ 1.22  │<br>│ 2   ┆ Two_Y    ┆ 1.22  │<br>└─────┴──────────┴───────┘<br><br># split and drop &#39;variable&#39; column <br>df = (df<br>      .with_columns(<br>          pl.col(&#39;variable&#39;).str.slice(0,length=3).alias(&quot;level&quot;),<br>          pl.col(&#39;variable&#39;).str.slice(-1).alias(&quot;x_y&quot;),<br>          )<br>      .drop(&#39;variable&#39;)<br>      )<br>In [7]: df<br>Out[7]: <br>shape: (12, 4)<br>┌─────┬───────┬───────┬─────┐<br>│ row ┆ value ┆ level ┆ x_y │<br>│ --- ┆ ---   ┆ ---   ┆ --- │<br>│ i64 ┆ f64   ┆ str   ┆ str │<br>╞═════╪═══════╪═══════╪═════╡<br>│ 0   ┆ 1.1   ┆ One   ┆ X   │<br>│ 1   ┆ 1.1   ┆ One   ┆ X   │<br>│ 2   ┆ 1.1   ┆ One   ┆ X   │<br>│ 0   ┆ 1.2   ┆ One   ┆ Y   │<br>│ …   ┆ …     ┆ …     ┆ …   │<br>│ 2   ┆ 1.11  ┆ Two   ┆ X   │<br>│ 0   ┆ 1.22  ┆ Two   ┆ Y   │<br>│ 1   ┆ 1.22  ┆ Two   ┆ Y   │<br>│ 2   ┆ 1.22  ┆ Two   ┆ Y   │<br>└─────┴───────┴───────┴─────┘<br><br># Pivot and rename columns<br>df = (df<br>      .pivot(<br>          values=&quot;value&quot;, index=[&quot;row&quot;, &quot;level&quot;],<br>          columns=&quot;x_y&quot;, aggregate_function=&quot;sum&quot;<br>      )<br>      .sort(&#39;row&#39;)<br>      .rename({&#39;level&#39;: &#39;Sample&#39;, &#39;X&#39;: &#39;All_X&#39;, &#39;Y&#39;: &quot;All_Y&quot;})<br>      )<br><br>In [8]: df<br>Out[8]: <br>shape: (6, 4)<br>┌─────┬────────┬───────┬───────┐<br>│ row ┆ Sample ┆ All_X ┆ All_Y │<br>│ --- ┆ ---    ┆ ---   ┆ ---   │<br>│ i64 ┆ str    ┆ f64   ┆ f64   │<br>╞═════╪════════╪═══════╪═══════╡<br>│ 0   ┆ One    ┆ 1.1   ┆ 1.2   │<br>│ 0   ┆ Two    ┆ 1.11  ┆ 1.22  │<br>│ 1   ┆ One    ┆ 1.1   ┆ 1.2   │<br>│ 1   ┆ Two    ┆ 1.11  ┆ 1.22  │<br>│ 2   ┆ One    ┆ 1.1   ┆ 1.2   │<br>│ 2   ┆ Two    ┆ 1.11  ┆ 1.22  │<br>└─────┴────────┴───────┴───────┘</pre><blockquote><strong>Voila, same outcome, without any multi-index.</strong></blockquote><p>The example above split out the steps for clarity, but Polars allows method chaining and it could all have been done in one block like below.</p><pre>df = (df<br>      .melt(id_vars=&#39;row&#39;,<br>            value_vars=[&#39;One_X&#39;,&#39;One_Y&#39;,&#39;Two_X&#39;,&#39;Two_Y&#39;])<br>      .with_columns(<br>          pl.col(&#39;variable&#39;).str.slice(0,length=3).alias(&quot;level&quot;),<br>          pl.col(&#39;variable&#39;).str.slice(-1).alias(&quot;x_y&quot;),<br>          )<br>      .drop(&#39;variable&#39;)<br>      .pivot(<br>          values=&quot;value&quot;, index=[&quot;row&quot;, &quot;level&quot;],<br>          columns=&quot;x_y&quot;, aggregate_function=&quot;sum&quot;<br>      )<br>      .sort(&#39;row&#39;)<br>      .rename({&#39;level&#39;: &#39;Sample&#39;, &#39;X&#39;: &#39;All_X&#39;, &#39;Y&#39;: &quot;All_Y&quot;})<br>      )</pre><p>Pandas was developed with a primary focus on providing a high-performance, easy-to-use data analysis toolkit. In pandas, indexes play a crucial role in facilitating efficient data access, alignment, and computation. The index serves as a label for each row in a DataFrame or Series and enables quick data retrieval and alignment based on these labels. It allows for convenient indexing, slicing, and grouping operations, as well as alignment of data across different data structures.</p><blockquote>On the other hand, Polars is a newer library inspired by pandas but built with a different approach. Polars aims to provide high-performance data manipulation capabilities specifically optimized for large-scale datasets, leveraging multi-threading and SIMD (Single Instruction, Multiple Data) instructions for faster computation. In Polars, the primary data structure is the DataFrame, but it does not have an explicit index object like in pandas.</blockquote><h3>Advantages of Polars without explicit indexes:</h3><ol><li><strong>Reduced memory overhead</strong>: Polars, by not having explicit indexes, can reduce memory consumption for large datasets compared to pandas. This can be beneficial when working with memory-intensive operations or when memory optimization is a priority.</li><li><strong>Simplified parallelism:</strong> Polars leverages parallelism and SIMD instructions for efficient data processing. By avoiding indexes, Polars can potentially execute operations more effectively in parallel, resulting in faster computation on large datasets.</li><li><strong>Columnar memory layout:</strong> Polars utilizes a columnar memory layout, which enables efficient processing of columns and facilitates vectorized operations. This layout can enhance the performance of certain operations, such as filtering, aggregation, and column-wise computations.</li><li><strong>Lazy evaluation: </strong>Polars employs lazy evaluation, which allows for optimized execution plans and avoids unnecessary computations. This can lead to improved performance when working with complex operations on large datasets.</li></ol><p><strong>By exploring Polars, Pandas users can unlock the benefits of high-performance data manipulation on large datasets.</strong></p><blockquote>Polars offers a range of powerful operations, including filtering, grouping, joining, and aggregation, using predicates or column expressions. Although the syntax may differ in some cases, Polars provides concise and efficient solutions for common data transformation tasks.</blockquote><p>So, don’t let the absence of traditional indexes deter you from experiencing the speed and power of Polars. Embrace the paradigm shift, and you’ll discover a powerful tool that liberates you from index dependency and opens up new possibilities for data analysis and manipulation.</p><p>Stay tuned for more insights into Polars in upcoming articles! <em>And, please do clap if you found this article useful.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1532943eed18" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 5 — Python Polars: Leveraging Fluent Interfaces for Efficient and Reusable Code]]></title>
            <link>https://medium.com/@arkimetrix.analytics/part-5-pythons-polars-streamlining-data-processing-fluent-interfaces-in-action-ab8cd31e83d2?source=rss-da38359c288e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab8cd31e83d2</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Arkimetrix Analytics]]></dc:creator>
            <pubDate>Sun, 11 Jun 2023 23:57:35 GMT</pubDate>
            <atom:updated>2023-06-12T04:12:15.580Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 5 — Python Polars: Leveraging Fluent Interfaces for Efficient and Reusable Code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*-B4Kzb008t3_h86U7bGauw.png" /><figcaption>Python Polars Cookbook = POOK BOOK</figcaption></figure><p>Welcome back! This is the fifth installment in our exploration of Polars, a high-performance library in Python. In our earlier discussions, we introduced the basics of Polars.</p><h4>Now, we will examine how Polars employs the fluent interface design pattern to create more readable and efficient code.</h4><blockquote>Before diving into the specifics of Polars, let’s brush up on the concept of <a href="https://en.wikipedia.org/wiki/Fluent_interface"><strong>fluent interface design pattern</strong></a>. <strong><em>If you’re already acquainted with it, please skip ahead to next section.</em></strong></blockquote><p>Fluent interfaces streamline your code by setting a logical order of methods or functions, much like a symphony conductor leading an orchestra.</p><p>This creates a flow in your code that emulates natural language, all thanks to method chaining. In this process, the return value of each method determines the context for the subsequent one. To put it simply, imagine the following sequence:</p><p>Start -&gt; Create Object -&gt; Call Method A -&gt; Call Method B -&gt; Call Method C -&gt; Get Result</p><p>Each transition between stages depends on the return value of the previous method. All these methods stem from the same object, enabling each method to return the object itself (either as ‘return this’ or ‘return self’). This lets you chain method calls on a single line of code, for example:</p><p>result = Object().methodA().methodB().methodC().getResult()</p><p>This design pattern enhances readability and reduces clutter by aligning the syntax more closely with human thought patterns. However, it’s essential to be mindful of potential drawbacks. Over-reliance on this pattern can make your code less clear, especially when exceptions occur or object states change in less apparent ways.</p><h4>Polars integrates the principles of fluent interface design into its foundation through the use of <strong>method and function chaining</strong>.</h4><p>The DataFrame object, a core component of Polars, offers a multitude of methods for data manipulation and transformation. These methods all return DataFrame objects, facilitating method chaining and thereby improving code readability and compactness.</p><p>Take a look at the following Polars code where we filter rows, sort the resultant data, and then perform some aggregation:</p><pre>import polars as pl<br><br># Create a DataFrame<br>df = pl.DataFrame(<br>    {<br>        &quot;Fruit&quot;: [&quot;Apple&quot;, &quot;Banana&quot;, &quot;Cherry&quot;, &quot;Date&quot;,<br>                   &quot;Elderberry&quot;, &quot;Fig&quot;, &quot;Guava&quot;],<br>        &quot;Quantity&quot;: [15, 25, 5, 12, 14, 8, 10],<br>        &quot;Price&quot;: [1.5, 0.5, 2.0, 1.0, 2.5, 1.75, 1.0]<br>    }<br>)<br><br># Example method chaining<br>df = (df<br> .filter(pl.col(&quot;Quantity&quot;) &gt; 10)<br> .sort(&quot;Price&quot;)<br> .groupby(&quot;Fruit&quot;)<br> .agg(pl.mean(&quot;Price&quot;))<br>)</pre><p>This code reads almost like an English sentence, showcasing the power of the fluent interface design pattern.</p><h4>Polars also incorporates a “pipe” function that allows for explicit function chaining.</h4><p>This proves particularly useful when you want to apply custom functions or operations not included as DataFrame methods.</p><p><em>The “pipe” method in Polars is similar to the pipe operator in some other languages and libraries (like the </em><em>%&gt;% operator in R&#39;s dplyr, or the </em><em>pipe method in pandas).</em></p><p>Consider the following example where we refactor the previous code using the “pipe” function for chaining:</p><pre>def filter_by_quantity(df):<br>    return df.filter(pl.col(&quot;Quantity&quot;) &gt; 10)<br><br>def sort_by_price(df):<br>    return df.sort(&quot;Price&quot;)<br><br>def group_and_agg(df):<br>    return df.groupby(&quot;Fruit&quot;).agg(pl.mean(&quot;Price&quot;))<br><br>df = (df<br>      .pipe(filter_by_quantity)<br>      .pipe(sort_by_price)<br>      .pipe(group_and_agg)<br>      )</pre><p>Each function here takes a DataFrame as its argument, operates on it, and returns the transformed DataFrame. The “pipe” function then passes the DataFrame to the next function in the chain.</p><h4>In routine operations and complex scenarios, the combination of lambda functions and the pipe method offers both flexibility and concise code.</h4><p>Let’s refactor the previous example again using lambda functions:</p><pre>df = (df<br>      .pipe(lambda df: df.filter(pl.col(&quot;Quantity&quot;) &gt; 10))<br>      .pipe(lambda df: df.sort(by=&quot;Price&quot;))<br>      .pipe(lambda df: df.groupby(&quot;Fruit&quot;).agg(pl.mean(&quot;Price&quot;)))<br>     )</pre><p>Here, each lambda function takes a DataFrame as an argument and applies a transformation before returning it. The pipe method strings these operations together into a chain.</p><h4>Lambda functions or User-Defined Functions (UDFs) can be a game changer in complex scenarios. They allow you to apply custom operations not readily available in Polars, enabling you to craft reusable operations.</h4><p>Imagine you’re handling dataframes representing sales from different branches of a retail company. You need to perform the same series of transformations on each dataframe for analysis.</p><blockquote>Let’s first create a toy dataset with information about a variety of fruits sold in a store, including the quantity sold, the price, and the date of sale. Let’s create dataframes for N stores:</blockquote><pre>import random<br>import polars as pl<br>import datetime<br><br># Create a list of products.<br>products = [&#39;Apples&#39;, &#39;Bananas&#39;, &#39;Cherries&#39;, &#39;Dates&#39;, &#39;Elderberries&#39;]<br><br># Generate random data for the quantity and price columns.<br>quantities = [random.randint(1, 50) for _ in range(100)]<br>prices = [round(random.uniform(0.5, 3.0), 2) for _ in range(100)]<br><br># Generate dates for the last 14 days for the date column.<br>dates = [(datetime.datetime.now() - datetime.timedelta(days=i)).date() for i in range(100)]<br><br># Create a DataFrame for store 1<br>store1_df = pl.DataFrame(<br>    {<br>        &quot;Product&quot;: products * 20,<br>        &quot;Quantity&quot;: quantities,<br>        &quot;Price&quot;: prices,<br>        &quot;Date&quot;: dates<br>    }<br>)<br><br># Create a DataFrame for store 2<br>store2_df = pl.DataFrame(<br>    {<br>        &quot;Product&quot;: products * 20,<br>        &quot;Quantity&quot;: quantities,<br>        &quot;Price&quot;: prices,<br>        &quot;Date&quot;: dates<br>    }<br>)<br><br># Create a DataFrame for store N<br>...</pre><p>In this case, you could define a series of functions to perform these transformations and then use them with the <strong>pipe method.</strong></p><pre># Define a function to filter rows from the last week.<br>def filter_last_week(df):<br>    df = (df<br>          .filter(pl.col(&quot;Date&quot;) &gt;= (datetime.datetime.now() - datetime.timedelta(weeks=1)))<br>          )<br>    return df<br><br># Define a function to sort the dataframe by date.<br>def sort_by_date(df):<br>    return df.sort(&quot;Date&quot;)<br><br># Define a function to calculate total revenue for each type of product.<br>def calculate_revenue(df):<br>    df = (df<br>          .with_columns(<br>              (pl.col(&quot;Quantity&quot;) * pl.col(&quot;Price&quot;)).alias(&#39;revenue&#39;)<br>          )<br>          )<br>    return df<br><br># Define a function to group by product and calculate total quantity and average price.<br>def group_and_agg(df):<br>    df = (df<br>          .groupby(&quot;Product&quot;)<br>          .agg([<br>              pl.sum(&quot;Quantity&quot;).alias(&quot;Total_Qty&quot;),<br>              (pl.sum(&quot;revenue&quot;)/pl.sum(&#39;Quantity&#39;)).alias(&quot;Avg_Price&quot;)<br>              ])<br>          )<br>    return df<br></pre><p>Let’s now create a function that combines all the steps above:</p><pre>def prepare_data(df):<br>    return (df<br>            .pipe(filter_last_week)<br>            .pipe(sort_by_date)<br>            .pipe(calculate_revenue)<br>            .pipe(group_and_agg)<br>            )</pre><p>With the prepare_data function, you can apply the same set of transformations to any DataFrame.</p><pre># Now you can use the prepare_data function to transform any DataFrame in the same way.<br>store1_df = prepare_data(store1_df)<br>store2_df = prepare_data(store2_df)<br># ... and so on for each branch&#39;s dataframe.</pre><blockquote>Polars effectively utilizes the fluent interface design, offering the benefits of method and function chaining.</blockquote><p>By adopting this approach, you’ll be writing code that is not only more readable and compact, but also more efficient and reusable.</p><p>Stay tuned for more insights into Polars in our upcoming articles! <em>And, please do clap if you found this article useful.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab8cd31e83d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 4. Handling Large Datasets with Polars and Spark: Creating a 100 million Row Fake Dataset in…]]></title>
            <link>https://medium.com/@arkimetrix.analytics/polars-and-spark-100-million-row-dataset-18fd29f46d2c?source=rss-da38359c288e------2</link>
            <guid isPermaLink="false">https://medium.com/p/18fd29f46d2c</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[polar]]></category>
            <dc:creator><![CDATA[Arkimetrix Analytics]]></dc:creator>
            <pubDate>Sat, 01 Apr 2023 20:38:11 GMT</pubDate>
            <atom:updated>2023-05-23T21:41:26.510Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 4. Handling Large Datasets with Polars and Spark: Creating a 100 million Row Fake Dataset in Just 2 Minutes</h3><h4>Polars and Pyspak: A Powerful Combination for Efficient Data Processing and Manipulation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PCiAgm8hrHLIuO9bitVe7g.png" /><figcaption>Python Polars Cookbook = POOK BOOK</figcaption></figure><p>In this article, we explore the use of <strong>Python Polars</strong> and <strong>Apache Spark</strong> to handle and combine large datasets. While the exercise below is almost just for ‘fun’, it has practical applications for data analysts and engineers who frequently work with large datasets.</p><blockquote>If you haven’t already, check out my previous articles for a gentle introduction to the Polars library (<a href="https://medium.com/@danny.bharat/phenomenal-python-polars-the-new-go-to-for-etl-and-data-analysis-6766a92f3bbf">Part 1</a>, <a href="https://medium.com/@danny.bharat/python-polars-part2-fe7a0ca4435c">Part 2</a> &amp; <a href="https://medium.com/@danny.bharat/python-polars-transform-recipes-date-extract-explode-4aafa58bd046">Part 3</a>). Stay tuned for more upcoming articles in this series.</blockquote><p>With the increasing volume and variety of data available today, scalable cloud solutions are often used for tasks like these. However, not everyone has access to those solutions or would like to avoid the compute costs associated with them. This is where tools like Polars and Spark come in handy.</p><ul><li><strong>Polars</strong> is a Python library for data manipulation that utilizes <strong>Apache Arrow</strong> as its underlying data structure. With Polars, users can efficiently work with large datasets in memory using tools like filtering, aggregation, grouping, and manipulation. Apache Arrow is designed for efficient in-memory processing and transferring data between different systems and languages. Its columnar data format enables efficient data processing, while support for zero-copy memory allocation ensures that Polars can handle very large datasets with minimal memory usage.</li><li><strong>Apache Parquet</strong>, a columnar storage format, is ideal for storing and retrieving large datasets efficiently. Parquet’s compressed, columnar format supports various data types and complex structures, making it suitable for many data processing tasks.</li><li><strong>Apache Spark</strong> is a distributed data processing framework that can work with both Apache Arrow and Apache Parquet. With Spark, users can work with large datasets in parallel across multiple nodes in a cluster.</li></ul><blockquote><strong>Together, Polars, Spark, and Parquet provide a powerful combination for working with large datasets in memory and for storage, enabling efficient data processing and manipulation for a wide range of data-intensive applications.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QE2vHgskku8Gas4TAQNRyA.png" /></figure><p>On to the fun. In this exercise, we <strong>create a</strong> <strong>100 million row fake dataset</strong> using Polars and Spark. The example code below has two functions:</p><ul><li><em>The “create_fake_data” function generates a 20 million row dataframe using Polars and the faker library, and materializes it as a Parquet file.</em></li><li><em>The “spark_union” function then takes the 20 million rows and self-appends 5 times to create a 100 million row dataframe using Spark.</em></li></ul><blockquote><strong><em>The two functions combined average just around 2 minutes on my laptop (12th gen Intel i7–1260P and 16 GB RAM).</em></strong></blockquote><pre>import os<br>from datetime import datetime, timedelta<br>import time<br>import random<br><br>import polars as pl<br>from faker import Faker<br><br># optional, in case your files are not in the working directory<br>os.chdir(&quot;/path/&quot;)<br>file_path = &quot;/path/&quot;<br><br>Faker.seed(0) #optional, seed value if you want to return consistent data<br>fake = Faker(&quot;en_CA&quot;) #change this to your locale<br>random.seed(10) #optional, seed value if you want to return consistent data<br><br># added to measure elapsed time<br>start = time.perf_counter()<br><br># Set number of rows and chunks for the Polars to create<br>n_rows = 20_000_000<br>n_chunks = 15<br>chunk_size = int(n_rows / n_chunks)<br><br># Set date range for the dataset<br>start_date = datetime(2022, 1, 1)<br>end_date = datetime(2022, 12, 31)<br><br>def create_fake_data():<br>    &#39;&#39;&#39;<br>    Generates a fake dataset with 5 columns. <br>    Number of rows can be defined. Data generated in chunks using <br>    Python Polars library (faster for initial smaller number of rows).<br>    Returns a dataframe materialized as a parquet file.<br>    &#39;&#39;&#39;<br>    #Generate random dates between the date range<br>    days = (end_date - start_date).days<br>    dates = [<br>        start_date + timedelta(days=random.randint(0, days))<br>        for i in range(chunk_size)<br>    ]<br>    random.shuffle(dates)<br>    <br>    # Create a list of 10 random alpha-numeric values for the &#39;Item&#39; column<br>    items = [<br>        &quot;&quot;.join(<br>            random.choices(<br>                &quot;abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890&quot;,<br>                k=10,<br>            )<br>        )<br>        for i in range(100)<br>    ]<br>    <br>    # Create a list of 20 fake company names<br>    company = [fake.company() for i in range(20)]<br>    <br>    # Create a DataFrame with five columns<br>    # loop through each chunk and append it to the Parquet file<br>    <br>    df = None<br>    df_chunk = None<br>    <br>    for i in range(1, n_chunks + 1):<br>    <br>        # Generate random dates between the date range<br>        days = (end_date - start_date).days<br>        chunk_dates = [<br>            start_date + timedelta(days=random.randint(0, days))<br>            for i in range(chunk_size)<br>        ]<br>        random.shuffle(chunk_dates)<br>    <br>        # Create a DataFrame with five columns<br>        df_chunk = pl.DataFrame(<br>            {<br>                &quot;Date&quot;: [d.date() for d in chunk_dates],<br>                &quot;Company&quot;: [random.choice(company) for i in range(chunk_size)],<br>                &quot;Item&quot;: [random.choice(items) for i in range(chunk_size)],<br>                &quot;On_hand&quot;: [random.randint(0, 100) for i in range(chunk_size)],<br>                &quot;sales&quot;: [random.randint(0, 1000) for i in range(chunk_size)],<br>            }<br>        ).with_columns(pl.col(&quot;Date&quot;).cast(pl.Date)) #<br>    <br>        if i == 1:<br>            df = df_chunk<br>        else:<br>            df = df.collect().vstack(df_chunk)<br>    <br>        df = df.lazy()<br>        <br>    <br>        if df is not None:  <br>            df.sink_parquet(<br>                file_path + &quot;chunk.parquet&quot;,<br>                row_group_size=chunk_size,<br>                slice_pushdown=True,<br>                compression=&quot;zstd&quot;,<br>            )<br><br>create_fake_data()<br><br>stop = time.perf_counter()<br>elapsed = stop - start<br>print(f&quot;creating dataframe with {n_rows} rows took {elapsed}&quot;)<br><br># See next section which takes the 20M row df created by Polars<br># and append that 5 times using<br># Spark to create a 100M dataframe materialized as a parquet file<br># Function “create_fake_data” takes around 111 seconds to return 20 million rows (approximately 165 MB). Next section is the pyspark module.</pre><pre>import sys<br>from datetime import datetime, timedelta<br>import time<br>import os<br><br>from pyspark.sql import SparkSession<br><br>os.chdir(&quot;/path/&quot;)<br>file_path = &quot;/path/&quot;<br><br>start = time.perf_counter()<br><br>def spark_union():<br>    # create a SparkSession<br>    spark = SparkSession.builder.appName(&quot;Append Parquet Files&quot;).getOrCreate()<br><br>    df1 = spark.read.parquet(file_path + &quot;chunk.parquet&quot;)<br>    df2 = spark.read.parquet(file_path + &quot;chunk.parquet&quot;)<br>    df3 = spark.read.parquet(file_path + &quot;chunk.parquet&quot;)<br>    df4 = spark.read.parquet(file_path + &quot;chunk.parquet&quot;)<br>    df5 = spark.read.parquet(file_path + &quot;chunk.parquet&quot;)<br><br>    # union the two dataframes<br>    df_combined = df1.union(df2).union(df3).union(df4).union(df5)<br><br>    # write out the combined dataframe to a new Parquet file<br>    # replace &#39;overwrite&#39; with &#39;append&#39;, even faster but creates multiple files<br>    df_combined.write.mode(&quot;overwrite&quot;).parquet(<br>        file_path + &quot;large.parquet&quot;<br>    )  <br><br>    # read in the Parquet file lazily<br>    df = spark.read.parquet(file_path + &quot;large.parquet&quot;)<br><br>    # view the contents of the DataFrame<br>    df.show()<br>    df.printSchema()<br><br>    # get the row count of the DataFrame<br>    row_count = df.count()<br><br>    # print the row count<br>    return row_count<br><br>    # stop the SparkSession<br>    spark.stop()</pre><blockquote>Function “spark_union” takes around 16 seconds to return 100 million rows (approximately 460 MB).</blockquote><p>In conclusion, although this exercise was done for fun, I hope it also demonstrates the practical applications of Polars and Spark. Both tools are powerful and belong in the toolkit of businesses and individuals working with large datasets.</p><h4>I hope you are finding this series useful, dear reader. If you have experience working with Polars and Spark, but you prefer different tools for handling large datasets, please share your thoughts in the comments. Your insights can help fellow data analysts and engineers choose the best tools for their specific needs.</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18fd29f46d2c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 3. Python Polars — Data Type Operations, Replacements, Extractions, and Explosions]]></title>
            <link>https://medium.com/@arkimetrix.analytics/python-polars-transform-recipes-date-extract-explode-4aafa58bd046?source=rss-da38359c288e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4aafa58bd046</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dask]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[vaex]]></category>
            <dc:creator><![CDATA[Arkimetrix Analytics]]></dc:creator>
            <pubDate>Fri, 20 Jan 2023 00:17:52 GMT</pubDate>
            <atom:updated>2023-04-30T16:06:17.238Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A polar bear with the Python Polars logo stripes overlaid on it. Meant to convery the power of the Python Polars library." src="https://cdn-images-1.medium.com/max/1024/1*PCiAgm8hrHLIuO9bitVe7g.png" /><figcaption>Python Polars Cookbook = POOK BOOK</figcaption></figure><h3>Part 3. Python Polars - Data Type Operations, Replacements, Extractions, and Explosions</h3><h4>Easier Syntax and Faster than Pandas, Dask, Vaex etc.</h4><p><em>I want to add some qualifications at the outset . While the title suggests that Polars is both easier to use and faster than competitors, the article itself does not provide proof of that. The claim about speed is supported by a multitude of articles on the topic and our own tests, some covered in </em><a href="https://medium.com/@danny.bharat/python-polars-part2-fe7a0ca4435c"><em>Part 2 of the series</em></a><em>. The claim about easier syntax is my opinion and my team’s experience using Polars, but the article does not compare and contrast Polars versus the rest.</em></p><p>This series of articles are targeted at analysts who are getting started with Python, and might be useful for Python experts not familiar with the Python Polars library. It is focussed on Data Type Operations defined in the next section.</p><p>The <a href="https://medium.com/@danny.bharat/phenomenal-python-polars-the-new-go-to-for-etl-and-data-analysis-6766a92f3bbf">first article covered why Polars is phenomenal</a> (as a replacement for Pandas, Dask etc.). <a href="https://medium.com/@danny.bharat/python-polars-part2-fe7a0ca4435c">Part 2 covers some basic operations</a> on how to read and write data etc.</p><p>This article examines frequently used transformations for data analysis or modeling after data extraction, as part of an ongoing series. Part 3 covers:</p><ol><li>Data type transforms</li><li>Replace strings and drop columns</li><li>Split or Explode strings</li></ol><p>We will continue to use the the dataframe created in <a href="https://medium.com/@danny.bharat/python-polars-part2-fe7a0ca4435c">Part 2</a>. The fake dataframe ‘df’ has the following structure:</p><pre>df.fetch(5)<br>Out[]: <br>shape: (5, 5)<br>┌────────────────────────────┬─────────────────────────────┬─────────┬─────────────┬──────┐<br>│ Transaction_date           ┆ Ref                         ┆ Cust_ID ┆ Ship_method ┆ Item │<br>│ ---                        ┆ ---                         ┆ ---     ┆ ---         ┆ ---  │<br>│ str                        ┆ str                         ┆ i64     ┆ str         ┆ i64  │<br>╞════════════════════════════╪═════════════════════════════╪═════════╪═════════════╪══════╡<br>│ 10/15/2021 21:10:17.889427 ┆ id-1094579svb;id-9754901svb ┆ 87      ┆ Truck       ┆ 969  │<br>│ 01/03/2021 23:00:20.705979 ┆ id-6778571yva;id-1758776yva ┆ 57      ┆ Rail        ┆ 463  │<br>│ 10/25/2021 08:21:38.892811 ┆ id-9868666djm;id-6668689djm ┆ 40      ┆ Rail        ┆ 442  │<br>│ 12/24/2021 10:24:11.919099 ┆ id-8161638iyl;id-8361618iyl ┆ 37      ┆ Rail        ┆ 42   │<br>│ 03/12/2021 16:18:25.103390 ┆ id-8669336nsx;id-6339668nsx ┆ 13      ┆ Air         ┆ 27   │<br>└────────────────────────────┴─────────────────────────────┴─────────┴─────────────┴──────┘</pre><p>We are also using ‘Lazy evaluation’ throughout, since we are using the ‘scan_’ function to load the parquet file to the df dataframe. Execution can be requested at any time by the .collect method. So, for example: df = df.collect() would query all available data.</p><h3>Data type conversions</h3><h4>Column renaming</h4><p>Before we get started with the transformations, your boss/customer complains that the ‘Transaction_date’ is too long and needs to be shortened to ‘date, and ‘Ref’ should instead be ‘reference’. Easily done with a single line:</p><pre># Rename columns<br>df = df.rename({&quot;Transaction_date&quot;: &quot;date&quot;,&#39;Ref&#39;:&#39;reference&#39;})</pre><h4>Data types: Int to Str to Int</h4><p>Polars data types can be found in the <a href="https://pola-rs.github.io/polars/py-polars/html/reference/datatypes.html#">documentation</a>.</p><p>To change datatypes, Polars uses the expression ‘<em>polars.Expr.cast</em>’. So for example, let’s say you want the ‘Cust_ID’ columns to be string format instead of i64, you can use the expression below.</p><pre># Illustrative: Change column datatypes <br>df = df.with_columns([<br>    pl.col(&#39;Cust_ID&#39;).cast(pl.Utf8), #from &#39;i64&#39; to string (&#39;Utf8&#39;)<br>    pl.col(&#39;Cust_ID&#39;).cast(pl.Int64) #change back to &#39;i64&#39;<br>])</pre><h4>Data type String to Date</h4><p>It is almost certain that date operations will be required on df. As you can see above, the ‘Transaction_date’ column (since renamed to ‘date’) is in string format. Let’s change that to date format, but you can’t just use the expression ‘<em>polars.Expr.cast</em>’ to do that.</p><p>Here is a script to illustrate how that would return an error:</p><pre># illustrative, useful for multipe columns<br>convert_to_date = [&#39;Transaction_date&#39;]<br><br>&#39;&#39;&#39; This block of code is wrong, here just to illustrate possible errors.&#39;&#39;&#39;<br>    <br>df = (<br>    df.with_columns([<br>      pl.col(convert_to_date).cast(pl.Date()).alias(&quot;Date_Transact&quot;)<br>      ])<br>      )<br><br>&#39;&#39;&#39;Converting a string (utf8 data type in Polars) to date will fail<br>    and return an error of type: ComputeError: Strict conversion from Utf8<br>    to Date failed for values [&quot;12/24/2021 10:24:11.919099&quot;, <br>    0/25/2021 08:21:38.892811&quot;, ... &quot;01/03/2021 23:00:20.705979&quot;]...<br>    Ideally should have been fixed before saving to parquet file&#39;&#39;&#39;</pre><p>To convert a string to date, we need to use the ‘<em>strptime</em>’ function as below:</p><pre>&#39;&#39;&#39; Right way to convert string to date&#39;&#39;&#39;<br>df = (<br>    df.with_columns([<br>        pl.col(convert_to_date).str.strptime(<br>            pl.Date,<br>            format=&#39;%m/%d/%Y %H:%M:%s.%f&#39;, strict=False<br>        )<br>    ])<br>)</pre><p>If you are not familiar, <strong><em>the strptime() method is a Python function</em></strong> from the datetime module. It parses a string representation of a date and time and convert it to a datetime object. <strong><em>This method takes two arguments</em></strong>: a string representing the date and time (pl.Date), and a string representing the format of the date and time (fmt).</p><p>For example: from df, the first date (string) = “10/15/2021 21:10:17.889427”. The date format here is “%m/%d/%Y %H:%M:%s.%f” (the string has time down to microsceonds, represented by %f. Here is a <a href="https://www.programiz.com/python-programming/datetime/strptime">good article that covers the various Python strptime() date time objects</a>.</p><h3>Parsing column values</h3><p>For the next operation, let’s look at the ‘reference’ column (originally ‘Ref’). The ‘reference’ column holds string values of the type ‘<em>id-1094579svb;id-9754901svb</em>’.</p><p>Let’s say a business SME has just informed you that the column actually holds two different items that are separated by “;”. They would also like the prefix “<em>id-</em>” to be removed.</p><p>Let’s do the transformations one step at a time.</p><h4>Replace</h4><p>First, let’s get rid of the prefix ‘id-’</p><pre>df = df.with_columns([<br>    pl.col(&#39;reference&#39;).str.replace(r&quot;id-&quot;, &quot;&quot;)<br>])</pre><p>The script above returns:</p><pre>df.fetch(5) # returns first 5 rows from the Lazy frame<br>Out[]: <br>shape: (5, 5)<br>┌────────────┬──────────────────────────┬─────────┬─────────────┬──────┐<br>│ date       ┆ reference                ┆ Cust_ID ┆ Ship_method ┆ Item │<br>│ ---        ┆ ---                      ┆ ---     ┆ ---         ┆ ---  │<br>│ date       ┆ str                      ┆ i64     ┆ str         ┆ i64  │<br>╞════════════╪══════════════════════════╪═════════╪═════════════╪══════╡<br>│ 2021-10-15 ┆ 1094579svb;id-9754901svb ┆ 87      ┆ Truck       ┆ 969  │<br>│ 2021-01-03 ┆ 6778571yva;id-1758776yva ┆ 57      ┆ Rail        ┆ 463  │<br>│ 2021-10-25 ┆ 9868666djm;id-6668689djm ┆ 40      ┆ Rail        ┆ 442  │<br>│ 2021-12-24 ┆ 8161638iyl;id-8361618iyl ┆ 37      ┆ Rail        ┆ 42   │<br>│ 2021-03-12 ┆ 8669336nsx;id-6339668nsx ┆ 13      ┆ Air         ┆ 27   │<br>└────────────┴──────────────────────────┴─────────┴─────────────┴──────┘</pre><p>Here you see only the first instance of ‘id-’ was removed, <em>but we want both instanced gone!</em></p><p>Easy, replace ‘<strong>str.replace</strong>’ with ‘<strong>str.replace_all</strong>’</p><pre>df = df.with_columns([<br>    pl.col(&#39;reference&#39;).str.replace_all(r&quot;id-&quot;, &quot;&quot;)<br>])<br><br>df.fetch(5)<br>Out[]: <br>shape: (5, 5)<br>┌────────────┬───────────────────────┬─────────┬─────────────┬──────┐<br>│ date       ┆ reference             ┆ Cust_ID ┆ Ship_method ┆ Item │<br>│ ---        ┆ ---                   ┆ ---     ┆ ---         ┆ ---  │<br>│ date       ┆ str                   ┆ i64     ┆ str         ┆ i64  │<br>╞════════════╪═══════════════════════╪═════════╪═════════════╪══════╡<br>│ 2021-10-15 ┆ 1094579svb;9754901svb ┆ 87      ┆ Truck       ┆ 969  │<br>│ 2021-01-03 ┆ 6778571yva;1758776yva ┆ 57      ┆ Rail        ┆ 463  │<br>│ 2021-10-25 ┆ 9868666djm;6668689djm ┆ 40      ┆ Rail        ┆ 442  │<br>│ 2021-12-24 ┆ 8161638iyl;8361618iyl ┆ 37      ┆ Rail        ┆ 42   │<br>│ 2021-03-12 ┆ 8669336nsx;6339668nsx ┆ 13      ┆ Air         ┆ 27   │<br>└────────────┴───────────────────────┴─────────┴─────────────┴──────┘</pre><h4>Split, Split_exact, Drop and Explode</h4><p>For the second step, we want to split the ‘reference’ strings that are separated by ‘;’ into two parts .</p><pre>df = df.with_columns(<br>    [<br>        pl.col(&#39;reference&#39;).str.split(by=&quot;;&quot;)<br>    ]<br>)</pre><p>This returns the ‘reference’ column with a list of strings. Not exactly what we want.</p><pre>df.fetch(5)<br>Out[]: <br>shape: (5, 5)<br>┌────────────┬──────────────────────────────┬─────────┬─────────────┬──────┐<br>│ date       ┆ reference                    ┆ Cust_ID ┆ Ship_method ┆ Item │<br>│ ---        ┆ ---                          ┆ ---     ┆ ---         ┆ ---  │<br>│ date       ┆ list[str]                    ┆ i64     ┆ str         ┆ i64  │<br>╞════════════╪══════════════════════════════╪═════════╪═════════════╪══════╡<br>│ 2021-10-15 ┆ [&quot;1094579svb&quot;, &quot;9754901svb&quot;] ┆ 87      ┆ Truck       ┆ 969  │<br>│ 2021-01-03 ┆ [&quot;6778571yva&quot;, &quot;1758776yva&quot;] ┆ 57      ┆ Rail        ┆ 463  │<br>│ 2021-10-25 ┆ [&quot;9868666djm&quot;, &quot;6668689djm&quot;] ┆ 40      ┆ Rail        ┆ 442  │<br>│ 2021-12-24 ┆ [&quot;8161638iyl&quot;, &quot;8361618iyl&quot;] ┆ 37      ┆ Rail        ┆ 42   │<br>│ 2021-03-12 ┆ [&quot;8669336nsx&quot;, &quot;6339668nsx&quot;] ┆ 13      ┆ Air         ┆ 27   │<br>└────────────┴──────────────────────────────┴─────────┴─────────────┴──────┘</pre><p>There are two ways to solve the problem depending on what output is required.</p><p><strong>Case 1. Split into columns: </strong>We want to split the string values in ‘reference’ column into exactly 2 parts and assign each part to a new column. To do that, instead of <strong><em>‘str.split’</em></strong> we can use <strong><em>‘str.split_exact’.</em></strong></p><pre>df = df.with_columns(<br>    [<br>        pl.col(&#39;reference&#39;).str.split_exact(&quot;;&quot;, 1)<br>        .struct.rename_fields([&quot;first_part&quot;, &quot;second_part&quot;])<br>        .alias(&quot;fields&quot;),<br>    ]<br>).unnest(&quot;fields&quot;).drop(&#39;reference&#39;)</pre><p>This returns what we wanted, two new columns and we used the <strong><em>df.drop(‘col name’)</em></strong> method to remove the original ‘reference’ column.</p><pre>df.fetch(5)<br>Out[]: <br>shape: (5, 6)<br>┌────────────┬─────────┬─────────────┬──────┬────────────┬─────────────┐<br>│ date       ┆ Cust_ID ┆ Ship_method ┆ Item ┆ first_part ┆ second_part │<br>│ ---        ┆ ---     ┆ ---         ┆ ---  ┆ ---        ┆ ---         │<br>│ date       ┆ i64     ┆ str         ┆ i64  ┆ str        ┆ str         │<br>╞════════════╪═════════╪═════════════╪══════╪════════════╪═════════════╡<br>│ 2021-10-15 ┆ 87      ┆ Truck       ┆ 969  ┆ 1094579svb ┆ 9754901svb  │<br>│ 2021-01-03 ┆ 57      ┆ Rail        ┆ 463  ┆ 6778571yva ┆ 1758776yva  │<br>│ 2021-10-25 ┆ 40      ┆ Rail        ┆ 442  ┆ 9868666djm ┆ 6668689djm  │<br>│ 2021-12-24 ┆ 37      ┆ Rail        ┆ 42   ┆ 8161638iyl ┆ 8361618iyl  │<br>│ 2021-03-12 ┆ 13      ┆ Air         ┆ 27   ┆ 8669336nsx ┆ 6339668nsx  │<br>└────────────┴─────────┴─────────────┴──────┴────────────┴─────────────┘</pre><p><strong>Case 2. Split into rows: </strong>We want to split the string values in ‘reference’ column into exactly 2 parts, but want them to remain in the same column.</p><p>This can be accomplished using <strong><em>polars.Expr.explode</em></strong></p><pre>df = (df<br>      .with_columns(<br>          pl.col(&#39;reference&#39;).str.split(by=&quot;;&quot;)<br>      )<br>      .explode(pl.col(&#39;reference&#39;))<br>      )</pre><p>This returns a long dataframe where every split item is expanded to a new row.</p><pre>df.fetch(4)<br>Out[]: <br>shape: (8, 5)<br>┌────────────┬────────────┬─────────┬─────────────┬──────┐<br>│ date       ┆ reference  ┆ Cust_ID ┆ Ship_method ┆ Item │<br>│ ---        ┆ ---        ┆ ---     ┆ ---         ┆ ---  │<br>│ date       ┆ str        ┆ i64     ┆ str         ┆ i64  │<br>╞════════════╪════════════╪═════════╪═════════════╪══════╡<br>│ 2021-10-15 ┆ 1094579svb ┆ 87      ┆ Truck       ┆ 969  │<br>│ 2021-10-15 ┆ 9754901svb ┆ 87      ┆ Truck       ┆ 969  │<br>│ 2021-01-03 ┆ 6778571yva ┆ 57      ┆ Rail        ┆ 463  │<br>│ 2021-01-03 ┆ 1758776yva ┆ 57      ┆ Rail        ┆ 463  │<br>│ 2021-10-25 ┆ 9868666djm ┆ 40      ┆ Rail        ┆ 442  │<br>│ 2021-10-25 ┆ 6668689djm ┆ 40      ┆ Rail        ┆ 442  │<br>│ 2021-12-24 ┆ 8161638iyl ┆ 37      ┆ Rail        ┆ 42   │<br>│ 2021-12-24 ┆ 8361618iyl ┆ 37      ┆ Rail        ┆ 42   │<br>└────────────┴────────────┴─────────┴─────────────┴──────┘</pre><p><em>Note, we see 8 rows even though we specified df.fetch(4). This is because Polars is still working with the original Lazy Frame, and is not just returning 4 rows, it is returning 4 partitions of the dataframe and each partition has 2 rows.</em></p><p><em>An aside: Readers somewhat familiar with the Polars library might have noticed that the </em><strong><em>.lazy() method</em></strong><em> was not used in any of the examples.</em></p><p><em>The </em><strong><em>.lazy()</em></strong><em> method is used in the Polars library to indicate that the computation should be performed lazily, rather than eagerly. This means that the computation is not executed immediately, but instead is delayed until the result is needed. Not the expert on this, but it appears that functions that do not need the .lazy() method are for data transformation rather than computation.</em></p><h3><strong>Conclusion and Recap</strong></h3><p>Part 3 has covered some transformations that can be done easily and quickly using Python Polars. The different scripts above can be combined into one, of course. It would look something like this:</p><pre>import polars as pl<br>import time<br><br>df = pl.scan_parquet(<br>    &#39;/home/dhb/pythonscripts/final_scripts/Database/fake_data.parquet&#39;)<br><br># illustrative, useful for multipe columns<br>convert_to_date = [&#39;Transaction_date&#39;]<br><br># case 1: split reference into two columns<br>df = (<br>    df<br>    .with_columns([<br>        pl.col(convert_to_date).str.strptime(<br>            pl.Date,<br>            fmt=&#39;%m/%d/%Y %H:%M:%s.%f&#39;, strict=False<br>        )<br>    ])<br>    .rename({&#39;Transaction_date&#39;: &#39;date&#39;, &#39;Ref&#39;: &#39;reference&#39;})<br>    .with_columns([<br>        pl.col(&#39;reference&#39;).str.replace_all(r&quot;id-&quot;, &quot;&quot;),<br>        pl.col(&#39;reference&#39;).str.split_exact(&quot;;&quot;, 1)<br>        .struct.rename_fields([&quot;first_part&quot;, &quot;second_part&quot;])<br>        .alias(&quot;fields&quot;)<br>    ])<br>    .unnest(&quot;fields&quot;).drop(&#39;reference&#39;)<br>)</pre><p>Do clap if you found this article useful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4aafa58bd046" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 2. Python Polars: recipes to create Lazy Frames, combine tables, deduplicate a dataframe]]></title>
            <link>https://medium.com/@arkimetrix.analytics/python-polars-part2-fe7a0ca4435c?source=rss-da38359c288e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe7a0ca4435c</guid>
            <category><![CDATA[dask]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[polar]]></category>
            <category><![CDATA[vaex]]></category>
            <dc:creator><![CDATA[Arkimetrix Analytics]]></dc:creator>
            <pubDate>Mon, 16 Jan 2023 17:35:03 GMT</pubDate>
            <atom:updated>2023-01-20T05:29:04.595Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PCiAgm8hrHLIuO9bitVe7g.png" /><figcaption>Python Polars Cookbook = POOK BOOK</figcaption></figure><h3>Part 2: Efficient Data Manipulation with Python Polars: Lazy Frames, Table Combining and Deduplication</h3><h4>Deliciously Simple Recipes to Kickstart Your Data Transformations</h4><p>In this article, the second in a series of practical Polars recipes, we will explore some of the transformations we have used in our recent projects.</p><p>In the <a href="https://medium.com/@danny.bharat/phenomenal-python-polars-the-new-go-to-for-etl-and-data-analysis-6766a92f3bbf">previous article</a>, we discussed our fondness for Polars and provided code on how to connect to supported databases such as Postgres, as well as a workaround for connecting to SQL Server.</p><h4>In this article, we will cover the following:</h4><ol><li>Create a fake data set and it save as a parquet file</li><li>Lazy (or eager) load the data into a Polars data frame</li><li>Learn how to combine multiple data frames</li><li>Remove duplicate rows from a data frame</li></ol><p>To begin, we will create a fake dataset to demonstrate practical transformations. <strong><em>This dataset is intentionally slightly more complex than the usual toy data frame to reflect real-life complexity.</em></strong></p><pre>import random<br>from datetime import datetime<br>import polars as pl<br><br><br>customers = []<br>LETTERS = &#39;abcdefghijklmnopqrstuvwxyz&#39;<br><br><br>def create_fake_data(nr_of_rows, path, file):<br>    &#39;&#39;&#39;Create a fake transactions dataset and materialize to defined format.<br>    Dataframe can be made as large as required using required argument &#39;nr_of_row&#39;<br>    Other required arguments are &#39;path&#39; and &#39;filename&#39; &#39;&#39;&#39;<br><br>    for customers_id in range(nr_of_rows):<br><br>        # Create transaction date in range<br>        d1 = datetime.strptime(&#39;1/1/2021&#39;, &#39;%m/%d/%Y&#39;)<br>        d2 = datetime.strptime(&#39;12/31/2021&#39;, &#39;%m/%d/%Y&#39;)<br><br>        # get the timestamp of the beginning and end of the range<br>        start_timestamp = int(d1.timestamp())<br>        end_timestamp = int(d2.timestamp())<br><br>        # generate a random timestamp within the range.<br>        random_timestamp = random.uniform(start_timestamp, end_timestamp)<br>        transaction_date = datetime.fromtimestamp(random_timestamp)<br>        transaction_date = transaction_date.strftime(&#39;%m/%d/%Y %H:%M:%S.%f&#39;)<br><br>        # create customer IDs<br>        cust_id = random.randint(1, 135)<br><br>        # Create shipping method<br>        ship_method = random.choice([&#39;Truck&#39;, &#39;Air&#39;, &#39;Rail&#39;])<br><br>        # Create Item #s<br>        item = random.randint(1, 1000)<br><br>        # create reference numbers<br>        random_letters = &#39;&#39;.join(random.sample(LETTERS, k=3))<br>        random_numbers = &#39;&#39;.join(str(random.randint(0, 9)) for _ in range(7))<br>        ref = f&#39;id-{random_numbers}{random_letters}&#39;+&#39;;&#39; + \<br>            f&#39;id-{random_numbers[::-1]}{random_letters}&#39;<br><br>        # create list with the generated elements<br>        customers.append([transaction_date, ref, cust_id, ship_method, item])<br><br>    # covert list to dataframe using Polars<br>    df = pl.DataFrame(customers,<br>                      columns=[&#39;Transaction_date&#39;, &#39;Ref&#39;,<br>                               &#39;Cust_ID&#39;, &#39;Ship_method&#39;, &#39;Item&#39;])<br><br>    #print(df)  # debug<br><br>    # write file to desired format, Parquet used in this example.<br>    # The arguments &#39;use-arrow&#39; and &#39;compression&#39; are optional,<br>    # but we found that specifying values resulted in faster write speeds<br>    df.write_parquet(f&#39;{path}/{file}.parquet&#39;,<br>                     use_pyarrow=True, compression=&#39;zstd&#39;)<br><br>start_time = time.time() #optional: record start time<br><br>create_fake_data(<br>    10000000, &#39;/path&#39;, &#39;output_file_name&#39;)<br><br>#optional: record execution time<br>end_time = time.time()<br>execution_time = end_time - start_time<br>print(f&quot;Execution time: {execution_time} seconds&quot;)</pre><p>The size of the dataframe can be adjusted using the ‘nr_of_rows’ argument. In the examples provided, the number of rows is set to 1000000 (10⁶). Benchmarking is performed locally on a laptop with a 12th Gen Intel i7–1260P CPU (2100 MHz, 12 Cores), 16 GB RAM, and Ubuntu 20.04 (WSL2) operating system. To generate 10⁶ rows takes ~26 seconds. The parquet file is ~24.5GB. The function will return a data frame like this:</p><pre>shape: (1000000, 5)<br>┌────────────────────────────┬─────────────────────────────┬─────────┬─────────────┬──────┐<br>│ Transaction_date           ┆ Ref                         ┆ Cust_ID ┆ Ship_method ┆ Item │<br>│ ---                        ┆ ---                         ┆ ---     ┆ ---         ┆ ---  │<br>│ str                        ┆ str                         ┆ i64     ┆ str         ┆ i64  │<br>╞════════════════════════════╪═════════════════════════════╪═════════╪═════════════╪══════╡<br>│ 10/15/2021 21:10:17.889427 ┆ id-1094579svb;id-9754901svb ┆ 87      ┆ Truck       ┆ 969  │<br>│ 01/03/2021 23:00:20.705979 ┆ id-6778571yva;id-1758776yva ┆ 57      ┆ Rail        ┆ 463  │<br>│ 10/25/2021 08:21:38.892811 ┆ id-9868666djm;id-6668689djm ┆ 40      ┆ Rail        ┆ 442  │<br>│ 12/24/2021 10:24:11.919099 ┆ id-8161638iyl;id-8361618iyl ┆ 37      ┆ Rail        ┆ 42   │<br>│ ...                        ┆ ...                         ┆ ...     ┆ ...         ┆ ...  │<br>│ 04/13/2021 19:50:16.876760 ┆ id-7819853vwc;id-3589187vwc ┆ 14      ┆ Rail        ┆ 626  │<br>│ 11/07/2021 06:54:43.312792 ┆ id-5154523ukt;id-3254515ukt ┆ 41      ┆ Truck       ┆ 798  │<br>│ 07/16/2021 10:36:44.143773 ┆ id-1350711rpi;id-1170531rpi ┆ 15      ┆ Air         ┆ 349  │<br>│ 04/11/2021 20:08:08.716959 ┆ id-7493362bni;id-2633947bni ┆ 38      ┆ Rail        ┆ 942  │<br>└────────────────────────────┴─────────────────────────────┴─────────┴─────────────┴──────┘</pre><h3><strong>Lazy loading data</strong></h3><p>Let’s create a dataframe by ‘Lazy Loading’ the parquet file from the previous step into a Polars dataframe.</p><p>“<em>Polars allows you to scan a </em><em>Parquet input. Scanning delays the actual parsing of the file and instead returns a lazy computation holder called a </em><em>LazyFrame.”</em></p><p>Execution time to load 1 million row into a dataframe: 0.00879 seconds.</p><pre>import polars as pl<br>import time<br><br>df = pl.scan_parquet(&quot;path.parquet&quot;)</pre><p>The data can also be read ‘eagerly’ from the parquet file into memory:</p><pre># replace pl.scan_parquet with read_parquet<br>df = pl.read_parquet(&quot;path.parquet)</pre><p>Execution time in this case is : 0.0840 seconds, 10 times longer than the lazy load, but still l<strong>ightning fast</strong>.</p><h3>A little about LazyFrame Objects</h3><p>Several libraries such as Dask, Vaex and Modin provide Lazy DataFrame functionality.</p><p>Lazy DataFrames are a concept in Python that refers to a way of working with data in a DataFrame format, but without loading all the data into memory at once. A LazyFrame holds information found during the brief scan and reads the rest of the file only when it is needed.</p><p>This can be useful when working with large datasets that would otherwise consume too much memory to fit into RAM. Instead of loading the entire dataset into memory, Lazy DataFrames only load and process the data as it is needed. This allows for more efficient use of resources and can make it possible to work with much larger datasets than would otherwise be possible.</p><blockquote>Some downsides of using Lazy DataFrames include:</blockquote><blockquote>Increased complexity: Adds an additional layer of complexity to the process of working with data, as the data is not loaded into memory all at once. This can make it more difficult to understand and debug issues that may arise.</blockquote><blockquote>Reduced performance for small datasets: Lazy DataFrames may not provide any performance benefits for small datasets, and in some cases, may even be slower than working with regular DataFrames due to the overhead of managing the lazy operations.</blockquote><blockquote>Incompatibility with certain libraries: Some libraries and tools are not designed to work with Lazy DataFrames, which can limit the functionality that is available when working with the data.</blockquote><blockquote>Limitations on operations: Some operations that can be performed on regular DataFrames cannot be performed on Lazy DataFrames.</blockquote><p>The Lazy frames can not be simply viewed or printed, and if you try, you will get output like this:</p><pre>df<br>Out[25]: &lt;polars.LazyFrame object at 0x7FF963E705B0&gt;<br><br>print(df)<br>naive plan: (run LazyFrame.describe_optimized_plan() to see the optimized plan)<br><br>  PARQUET SCAN /path/fake_data.parquet<br>  PROJECT */5 COLUMNS</pre><p>Some useful commands to get details about a Lazy Frame:</p><p><strong>DataFrame.Schema:</strong> returns a dict with column names and DataType</p><pre>df.schema<br>Out[28]: <br>{&#39;Transaction_date&#39;: Utf8,<br> &#39;Ref&#39;: Utf8,<br> &#39;Cust_ID&#39;: Int64,<br> &#39;Ship_method&#39;: Utf8,<br> &#39;Item&#39;: Int64}</pre><p><strong>LazyFrame.collect </strong>Polars will eagerly evaluate the query by calling the .collect method. To run a query in eager mode you can also just replace scan_xxx with read_xxx in the Polars code.</p><pre>df.collect()<br>Out: <br>shape: (1000000, 5)<br>┌────────────────────────────┬─────────────────────────────┬─────────┬─────────────┬──────┐<br>│ Transaction_date           ┆ Ref                         ┆ Cust_ID ┆ Ship_method ┆ Item │<br>│ ---                        ┆ ---                         ┆ ---     ┆ ---         ┆ ---  │<br>│ str                        ┆ str                         ┆ i64     ┆ str         ┆ i64  │<br>╞════════════════════════════╪═════════════════════════════╪═════════╪═════════════╪══════╡<br>│ 10/15/2021 21:10:17.889427 ┆ id-1094579svb;id-9754901svb ┆ 87      ┆ Truck       ┆ 969  │<br>│ 01/03/2021 23:00:20.705979 ┆ id-6778571yva;id-1758776yva ┆ 57      ┆ Rail        ┆ 463  │<br>│ 10/25/2021 08:21:38.892811 ┆ id-9868666djm;id-6668689djm ┆ 40      ┆ Rail        ┆ 442  │<br>│ 12/24/2021 10:24:11.919099 ┆ id-8161638iyl;id-8361618iyl ┆ 37      ┆ Rail        ┆ 42   │<br>│ ...                        ┆ ...                         ┆ ...     ┆ ...         ┆ ...  │</pre><p><strong>LazyFrame.fetch </strong>Collect a small number of rows for debugging purposes. Fetch is like the <a href="https://pola-rs.github.io/polars/py-polars/html/reference/lazyframe/api/polars.LazyFrame.collect.html#polars.LazyFrame.collect"><strong>collect()</strong></a> operation, but it overwrites the number of rows read by every scan operation.</p><pre>df.fetch(2)<br>Out: <br>shape: (2, 5)<br>┌────────────────────────────┬─────────────────────────────┬─────────┬─────────────┬──────┐<br>│ Transaction_date           ┆ Ref                         ┆ Cust_ID ┆ Ship_method ┆ Item │<br>│ ---                        ┆ ---                         ┆ ---     ┆ ---         ┆ ---  │<br>│ str                        ┆ str                         ┆ i64     ┆ str         ┆ i64  │<br>╞════════════════════════════╪═════════════════════════════╪═════════╪═════════════╪══════╡<br>│ 10/15/2021 21:10:17.889427 ┆ id-1094579svb;id-9754901svb ┆ 87      ┆ Truck       ┆ 969  │<br>│ 01/03/2021 23:00:20.705979 ┆ id-6778571yva;id-1758776yva ┆ 57      ┆ Rail        ┆ 463  │<br>└────────────────────────────┴─────────────────────────────┴─────────┴─────────────</pre><h3><strong>Vertical concatenation — let’s make the dataset larger</strong></h3><p>Polars allows both vertical and horizontal concatenation of two or more dataframes. Horizontal concatenation will be covered another time, but for now let’s double the size of dataframe from the previous section.</p><p>Why? Why not :). Data analyts have to often join tables and this is an extremely fast way to do that. Of course, Vertical concatenation works only when the dataframes have the same column names.</p><pre># continuing from the previous section where df has been loaded<br># the syntax is the same whether the df is a scan or read<br># Lazy dfs returns lazy df<br><br>df_vertical_concat = pl.concat([<br>    df,<br>    df<br>])<br><br># df_vertical_concat Out: shape: (2000000, 5)</pre><p>Lazy data frame took only ~2.5 seconds. Of course, concatenation of in-memory data frames (using read_parquet instead of scan_parquet) took less time 0.0636 seconds. Just for kicks, concatenating it ten times to create a 10 million row data frame took 1.001 seconds.</p><h3>Remove Duplicates</h3><p>In the previous step we duplicated each row of the orginal dataframe. Let’s get it back into the original form. Easy!</p><pre>df_back = df_vertical_concat.unique()<br>#Execution time: 0.0010294914245605469 seconds</pre><h3><strong>In Conclusion</strong></h3><p>This article has provided an overview of some basic actions that can be achieved using Python Polars. The article demonstrated how to create a fake dataset that reflects real-life complexity, and provided examples of how some transformations can be applied.</p><p>As a reminder, this is part of a series and these recipes are currently somewhat random. Subsequent articles will cover more detailed transformations.</p><p>Hope that this series will informative and useful for the Python Polars community, and that it inspires further exploration and experimentation with this powerful tool.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe7a0ca4435c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1. Phenomenal Python Polars: The new go-to for ETL and data analysis]]></title>
            <link>https://medium.com/@arkimetrix.analytics/phenomenal-python-polars-the-new-go-to-for-etl-and-data-analysis-6766a92f3bbf?source=rss-da38359c288e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6766a92f3bbf</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[dask]]></category>
            <category><![CDATA[vaex]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Arkimetrix Analytics]]></dc:creator>
            <pubDate>Sun, 15 Jan 2023 01:26:18 GMT</pubDate>
            <atom:updated>2023-01-16T17:44:50.866Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*60EZ_ZFd_BDaDZiIhkgRrA.png" /><figcaption><a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fgithub.com%2Fpola-rs%2Fpolars&amp;psig=AOvVaw3bEjYZ8F0cOaY645vjSm_Q&amp;ust=1673831049872000&amp;source=images&amp;cd=vfe&amp;ved=0CBAQjRxqFwoTCIid9NuwyPwCFQAAAAAdAAAAABAJ">Image source</a></figcaption></figure><h4>Polars is a fast and easy-to-learn library for ETL and data analysis. Makes connecting to databases really easy and fast.</h4><p>Polars offers several advantages, including fast data ingestion and export, no memory constraints, and optimized caching and performance. If you haven’t checked it out yet, there are a number of articles on Medium and elsewhere about <a href="https://medium.com/search?q=python+polars"><strong><em>how quick it is against benchmarks (Pandas) etc.</em></strong></a>, so I will not repeat the details. Here is a link to an <a href="https://www.codemag.com/Article/2212051/Using-the-Polars-DataFrame-Library"><strong><em>excellent introduction to Polars</em></strong></a></p><p>The <a href="https://pola-rs.github.io/polars-book/user-guide/">documentation</a> and <a href="https://medium.com/search?q=python+polars">resources </a>may not be as extensive as Pandas, but there is an extremely helpful community, including the <a href="https://www.ritchievink.com/">library’s author</a>, available for support.</p><p>This article covers one first step: how to load the data. It is important to first decide whether you want to ‘Lazy’ or ‘Eager’ load the data. Lazy versus Eager is a large topic in itself, and I will not attempt to discuss that here.</p><p>There is so much that can be done with Polars and as indicated above, a lot of introductory articles and other reviews are available.</p><p>This article covers the very first step, how to read data. It is important to first decide whether you want to ‘Lazy’ or ‘Eager’ load the data.</p><p><em>We’d prefer to lazy load the data if possible. As Polars is so quick, we have never had to worry about load times versus fully loading the data in memory. However, the Lazy API documentation is sparse and at times doesn’t translate 1:1 from the eager expressions.</em></p><p>Polars supports reading data from various formats (CSV, Parquet, and JSON) and connecting to databases like Postgres, MySQL, and Redshift. Reading or ‘scanning’ data from CSV, Parquet, JSON etc. is quite straightforward and well covered in the usage guide.</p><p>Connecting to most databases is usually simple as well, but currently there are some gaps. First, the easy one and it works beautifully straight out of the box: <a href="https://pola-rs.github.io/polars-book/user-guide/howcani/io/read_db.html"><strong>connecting to Postgres (</strong>or MySQL, Postgres, Sqlite, Redshift, Clickhouse)</a>.</p><h3>How to connect to Postgres using Polars</h3><p>Polars uses and requires the connector-x library to be installed. If this saves you time, here’s the script we use:import polars as pl</p><pre>dialect = &#39;postgresql&#39;<br>username = &#39;username&#39; <br>password = &#39;password&#39;<br>host = &#39;host&#39;<br>port = &#39;port&#39; #usually 5432<br>database = &#39;database&#39;<br>table = &#39;table&#39;<br>partition_on = &#39;quantity&#39; # Optional: a numeric field in the source table<br>num_part = 12 # Optional: number of parallel operations to run<br>def connect_download(tablename,partition_on,num_part):<br>    conn = f&quot;{dialect}://{username}:{password}@{host}:{port}/{database}&quot;<br>    query = &#39;SELECT * FROM public&#39;+tablename<br>    df = pl.read_sql(query, conn,partition_on = partition_on,\<br>           partition_num=num_part )<br>    #if not defining the partition, comment line above and use one below<br>    #df = pl.read_sql(query, conn)<br>    <br>    # alternatively, you can read in parallel using 2 threads<br>    # by explicitly providing two SQL queries<br>    #queries = [<br>    #     &quot;SELECT * FROM lineitem WHERE partition_col &lt;= 10&quot;,<br>    #     &quot;SELECT * FROM lineitem WHERE partition_col &gt; 10&quot;,<br>    #]<br>    #df = pl.read_sql(queries, conn)<br>    #modify as required if you want to materialize the data frame<br>    df.write_parquet(f&#39;{vv.data_path}/{database}_{table}.parquet&#39;)</pre><p>Polars allows data to be <a href="https://pola-rs.github.io/polars-book/user-guide/howcani/io/csv.html">materialized in many different formats, including CSV, JSON etc.</a></p><p>Polars (or more accurately, connector-x) currently doesn’t support SQL Server. We do need to connect to SQL Server very often, and still use Pandas or alternatives, depending on the data size.</p><p>Regardless, we still prefer to use Polars and clone the pandas data frame using the Polars function <em>polars.from_pandas() .</em></p><p>This isn’t related to Polars, but if you need to connect to SQL Server, here’s a script we use. <em>Make sure you have the MS SQL Server Driver installed correctly, in our experience it tends to be an adventure (at least on Linux). You will also need pyodbc.</em></p><h3>How to connect to SQL Server (Polars workaround)</h3><pre>import pyodbc<br>from sqlalchemy import create_engine<br>from sqlalchemy.sql import text<br>import pandas as pd<br>from tenacity import retry, wait_exponential, stop_after_attempt<br>import os<br>from dotenv import load_dotenv<br>load_dotenv()<br><br>server = r&#39;00.000.0.0\Abc123&#39;<br>database = &#39;database&#39; <br>username = os.environ.get(&#39;DBUSER&#39;) #replace with username if not using .env<br>password = os.environ.get(&#39;PASSWORD&#39;) #replace with password if not using .env<br>driver=r&#39;ODBC Driver 18 for SQL Server&#39;<br><br># Function to materialize sql output to the specified location<br>def sql_download(tablename,output_loc):<br>    &#39;&#39;&#39; Function connect to SQL Server and fetches the required table<br>        Pequires following arguments: <br>       (SQL Server) Username, Password, tablename, output path<br>       requires following libraries: os, dotenv, pyodbc, pandas, tenacity, pyarrow, sqlalchemy&#39;&#39;&#39;<br>    DATABASE_CONNECTION = f&#39;mssql+pyodbc://{username}:{password}@{server}/{database}?driver={driver}&amp;TrustServerCertificate=yes&#39;<br>    engine = create_engine(DATABASE_CONNECTION)<br>    connection = engine.connect()<br><br>    # This section is optional. Use to monitor or log<br>    # Use `exponential backoff` to poll the database for a status<br>    # and make sure that it is available for a connection<br>    # by sending a basic query at an incremental delay.<br>    # The number of retry attempts as well as the multiplier<br>    # and time between attempts can be modified as required<br>    <br>    @retry(wait=wait_exponential(multiplier=2, min=1, max=10),\<br>       stop=stop_after_attempt(5))<br>    def try_connection():<br>        try:    <br>            with connection:<br>                stmt = text(&quot;SELECT 1&quot;)<br>                connection.execute(stmt)<br>            print(&quot;Connection to database successful.&quot;)<br>            <br>        except Exception as e:<br>            print(&quot;Connection to database failed, retrying.&quot;)<br>            raise Exception<br>    <br>    try_connection()<br><br>    # This section is required to create the connection object<br>    # generated by the `Engine.connect()` method and close automatically<br>    # by `Connection.close()` at the end of the codeblock<br><br>    connection = engine.connect()<br>    with connection:  <br>        sql = (&#39;select * from &#39;+tablename)<br>        df = pd.read_sql_query(sql, connection)<br>        df = pl.from_pandas(df) #convert pandas df to Polars<br>        df.write_parquet(output_loc) #replace with required output format<br><br># download the query to a df and materialized as parquet file<br>sql_download(&#39;table_name&#39;,&#39;file_path\file_name.parquet&#39;)</pre><h3><strong>Final thoughts</strong></h3><p>A recommendation for anyone new to data engineering or data analysis would be to skip Pandas and start with Polars. My team has been using the <a href="https://pola-rs.github.io/polars-book/user-guide/">Polars library</a> for the past few months and it has been a huge productivity boost for all our projects. It is fast becoming our ETL tool of choice, both for the speed but especially for the simplicity and the power of Polars Expressions. More in the next article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6766a92f3bbf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>