<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eshwaran Venkat on Medium]]></title>
        <description><![CDATA[Stories by Eshwaran Venkat on Medium]]></description>
        <link>https://medium.com/@eash98?source=rss-7c131006709c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mc3E7qwvDbN7ks0C9puTAQ.jpeg</url>
            <title>Stories by Eshwaran Venkat on Medium</title>
            <link>https://medium.com/@eash98?source=rss-7c131006709c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:16:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eash98/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building Virtual F&B Brands by asking Data-Driven Questions]]></title>
            <link>https://eash98.medium.com/building-virtual-f-b-brands-by-asking-data-driven-questions-7f6460be1a14?source=rss-7c131006709c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f6460be1a14</guid>
            <category><![CDATA[virtual-brands]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[food-and-beverage]]></category>
            <category><![CDATA[restaurant-owners]]></category>
            <category><![CDATA[cloud-kitchens]]></category>
            <dc:creator><![CDATA[Eshwaran Venkat]]></dc:creator>
            <pubDate>Sun, 24 Dec 2023 14:35:37 GMT</pubDate>
            <atom:updated>2023-12-24T14:35:37.580Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fZE2AobV0FnMM-Ea" /><figcaption>Photo by <a href="https://unsplash.com/@jsnbrsc?utm_source=medium&amp;utm_medium=referral">Jason Briscoe</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article is a playbook guide to formulating better data-driven questions for creating virtual brands and positioning them. Listed are a set of targeted questions that can help materialize abstract concepts into actionable insights that can help grow your business, and align your operations teams with strategy. At <a href="http://www.dotlas.com/">Dotlas</a>, we specialize in assisting F&amp;B companies, including cloud kitchens with the insights they need to gain a competitive advantage. Find out more at:</p><p><a href="https://www.dotlas.com/">Dotlas</a></p><h3>What is a Cloud Kitchen?</h3><p>A <strong>cloud kitchen</strong> is like a restaurant, but does not have the dine-in service component. Customers interact with the offerings of a cloud kitchen using food-delivery apps, or custom brand-specific apps. This allows a cloud kitchen business to focus on takeout-style food, food-quality, cuisine selection, etc. This business model is beneficial for increasing profit margins since the overhead of managing a dine-in setup with staff is reduced. However, the optimization can be improved threefold with some innovative strategies.</p><p>A cloud kitchen can essentially become a “house of brands” where a single kitchen could span the size of a food-truck to the size of a warehouse. Consequently, a single kitchen can host tens or hundreds of brands. As far as the customer is concerned, they’re ordering from a different brand, but all of them fall under the umbrella of a single business to the owner(s). These type of brands are called “<strong>Virtual Brands”, </strong>as they do not possess a physical presence like a standard restaurant, and only exist as logical separations in the cloud kitchen and as separate entities in the food delivery app. This opens up certain degrees of freedom for cloud kitchens that are otherwise not available to a brick-and-mortar (traditional) restaurant.</p><ol><li><strong>Static Backend &amp; Cycling Frontend<br></strong>Same staff, same equipment and same ingredients in the kitchen deliver different kinds of cuisines, brands and food to the customer. This enables cloud kitchens to “kill” a brand that does not perform well in favour of a new brand that fixes the shortcomings of the predecessor in an effort to “keep their market” without an overhaul of their infrastructure and capital.</li><li><strong>SKU optimization</strong>. SKUs are the ingredients used by different food items, such as Tomatoes, etc. A burger brand and pizza brand can share common SKUs, in this case the tomato. This creates common “buckets” or “channels” of SKUs that can be shared resources across multiple brands.</li><li><strong>Increases Flexibility. </strong>Since everything can be an experiment or a process of trial-and-error, owners are free to be flexible in trying new marketing strategies, new cuisines (such as fusions), targeting different demographics, use different promotions and so on. They can keep rotating ideas until they find the one that just sticks with the market, at a very minimal cost of production.</li><li><strong>Ease of Expansion. </strong>A cloud kitchen unlike a restaurant doesn’t have to be on Main street. One can move a cloud kitchen to a less expensive area and still serve the same target market based on proximity, traffic and other parameters. Due to less strict location constraints, cloud kitchens can expand quickly and take the brands that matter to new locations or new markets faster.</li></ol><h3>The Competitive Advantage</h3><p>A cloud kitchen performs better than a competing kitchen:</p><ol><li>If its virtual brands generate more revenue than a competitor’s brand.</li><li>If its infra, capital cost &amp; spillage are less than a competitor across all brands.</li><li>By reducing the time taken to put a new virtual brand on market. This includes everything from marketing the brand to preparing menus, cuisine selection and more.</li></ol><h3>The Big Questions</h3><ul><li>How can a cloud kitchen create a specification for a brand that is <em>statistically</em> likely to perform well?</li><li>What is the definition of a good brand? How can 2 brands be compared to each other.</li><li>What kind of indicators need to be measured that can result in brand discovery?</li><li>Where should a brand be positioned with respect to other brands in the city, and in the world at large with respect to cuisines, categories and price offerings?</li><li>What are the hottest trends where demand needs to be served?</li><li>How can a brand stand out with respect to the competition, in the eyes of the customer?</li><li>How can a brand induce customers to order. How can demand be created, or trends be set?</li></ul><h3>The Key Ingredient 🍜</h3><p>Revenue is the objective optimized against the variables of Average Order Value and Order Volume. Every brand, based on the drivers of Revenue will sit in a different point of AOV and OV distributions.</p><p>When optimizing for Order Volume, the main focus should be on creating brands around the concept of <strong>Revenue Frequency</strong>, which we can categorize through the following high-level brand archetypes:</p><p><strong>Daily consumption:</strong> Brands serving low-cost, high-frequency items such as coffees that ensure a reliable and consistent client-base.</p><p><strong>Multi-weekly consumption: </strong>Lunch-focused cheap brands with quick and fast meals. These could be anything from brands serving salads, wraps, simple burgers.</p><p><strong>Weekly consumption: </strong>Any type of brand whose consumption pattern is not frequent enough to happen several times a week, but seen more as an indulgence by consumers. These brands would be serving cakes or premium dinners such as high-end burgers.</p><p><strong>Monthly consumption:</strong> These are high-end pricy meals, focused on special occasions. These could be expensive items such as sushi or seafood, or event-oriented meals such as items servicing 10+ people.</p><p>Conversely, in order to optimize for <strong>Average Order Value</strong>, a brand could be sub-divided as follows:</p><p><strong>Single Meal:</strong> Typically simple, fast meals for working professionals during lunch, or one-person dinner combos.</p><p><strong>Group Meal: </strong>Meals oriented at couples for special occasions such as weekly dinners or menus serving whole families including kid’s meals.</p><p>Lastly, there is yet another important sub-division based on the brand’s potential for revenue generation throughout its <strong>lifetime</strong>:</p><p><strong>M Brands (Marathon): </strong>Long-Term brands that continuously serve a regular order flow over a long period of time. A go-to lunch / dinner brand that behaves as an everyday eatery.</p><p><strong>S Brands (Sprint):</strong> Short-Term sprint (or seasonal) brands that serve during a particular season or to serve a culinary trend at its peak, but dies out later.</p><p>This means that an S brand serves a shock revenue over a short period of time, whereas an M brand serves a sustainable revenue over a long period of time.</p><h3>Factors Influencing Brand Revenue 💵</h3><p>Key Factors influencing revenue of a brand, which in turn influences a brand’s rank in a cloud kitchen portfolio include:</p><h3>Positioning 🛍</h3><p><strong><em>Marketing the brand to customers</em></strong></p><ul><li>Influencing perception of the brand by USP, attraction, offers, etc.</li><li>Building long-term consumer trust in the brand.</li><li>Identify all points in the flow of order to consumption that can hamper relations.</li></ul><p><strong>Deciding on the theme of the brand (different from cuisine and category):</strong></p><ul><li>Is the theme of the menu and delivery retro-like, classic, festive, etc.</li><li>Under what occasions does the brand offer discounts or craft special items</li></ul><p><strong>Customer Targeting</strong></p><ul><li>Segmenting consumers of other brands in the cloud-kitchen based on behaviour, eating habits, etc.</li></ul><h3>Location 📌</h3><p><strong><em>What are the needs of the consumers around the Kitchen</em></strong></p><ul><li>What is the purchasing power of demographics across varying contours of delivery time?</li><li>What is the distribution of other restaurants around this area?</li><li>Finding a brand that works to satisfy underserved consumers.</li><li>Adaptation of the brand to neighbourhood trends (on short and long scales)</li><li>Can the brand upsize or downsize its menu offering based on consumption patterns?</li><li>When to offer discounts and when to squeeze the customer, while maintaining retention.</li></ul><p><a href="https://towardsdatascience.com/geospatial-site-selection-analysis-using-cosine-similarity-dcde4be458e7">Geospatial Site-Selection Analysis Using Cosine Similarity</a></p><h3>Experience 📲</h3><p><strong>Food Quality</strong></p><ul><li>Does the food disintegrate easily. The balance of flavour compared to competition.</li></ul><p><strong>Food Selection (Cuisines and Categories)</strong></p><ul><li>What is the cuisine served by the brand?</li><li>What are the categories served in this cuisine and average price of category?</li><li>What is the distribution of menu items per category?</li><li>How does each menu item differ from what the competition offers?</li></ul><p><strong>Delivery Experience</strong></p><ul><li>Is Unpacking the food a bad / rewarding experience</li><li>Is the food prone to lose structure / form over delivery time</li><li>Is there a food satisfactory level that changes over varying contours of delivery time</li></ul><h3>The Brand Portfolio: An Ecosystem of Brands 💼</h3><p>The true power of cloud-based brands comes from the selection and management of a universe of brands, that can together serve a certain amount of demand.</p><h3>Failover 🔄</h3><ul><li>When a customer switches from a current brand to order on a new brand, how likely is it that they will choose a different brand served by one of the cloud kitchen’s own sites. I.e, how can we keep customers retained within the same brand universe where there is something for everyone?</li><li>How can the company tactically recommend brand hops that always lead back to our brands and never keep the customers’ orders a few hops away from our selection</li></ul><h3>Coverage &amp; Redundancy 🍰</h3><ul><li>How can you reach maximum amount of coverage with n different kinds of categorical brands.</li></ul><p>Is there a need to have more than one brand of a specific category? How can this demand be justified.</p><h3>Cannibalization 🤺</h3><ul><li>How can we keep our brands in a most stable and non-compete form to avoid cannibalization</li><li>How do we build a brand portfolio where every brand complements the next in the overall selection, but is capable of sustaining its own orders independent of other brands (at a customer level and at an ingredients (SKU) level)</li></ul><p>Here’s an example where Subway setup stores too close to each other that each store started eating off of the other’s revenue</p><p><a href="https://www.forbes.com/sites/garyocchiogrosso/2019/07/01/is-subway-taking-a-bite-of-the-sandwich-from-both-ends/">Is Subway Taking A Bite Of The Sandwich From Both Ends?</a></p><h3>Agility 🏎</h3><ul><li>How quickly can you switch brands in and out of our own kitchens</li><li>What is the turnaround time to put up a brand on a food aggregator and take them off. What are the related costs?</li><li>How can you incentivize food aggregators (food delivery sites) to ease your experimentations?</li></ul><h3>Discovering Underserved Demand 📉</h3><h3>User Studies 🧪</h3><p>Consider the following scenario: There exists a number of restaurants serving cuisine X in city A of country M. Assume that that the highest standard of cuisine X in city A is only half as good as the average standard of cuisine X in city B, perhaps in a different country N. Consumers in A <em>have only ever known</em> cuisine X as good as A is providing it, whereas a better version of X exists elsewhere. How can these gaps be identified? This repressed acceptance of X in A will not normally show up online in reviews / ratings.</p><blockquote><em>“A lot of times, people don’t know what they want until you show it to them” ~ Steve Jobs</em></blockquote><p>“How can we identify any underlying or inferred psychological indicators to serve, capture and even monopolize on cuisine service to underserved areas”?</p><h3>Tailored Experience 🪡</h3><p>When we reach sufficient order volumes and have the requisite customer information on their frequency of ordering, likes and dislikes of items, etc. Can we tailor a delivery experience that’s different for each customer, or even for a group of customers? Does this make sense to do cost-wise, and what is the gain.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f6460be1a14" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debunking 5 Myths on Adopting Databricks]]></title>
            <link>https://eash98.medium.com/debunking-5-myths-on-adopting-databricks-18140cc3d183?source=rss-7c131006709c------2</link>
            <guid isPermaLink="false">https://medium.com/p/18140cc3d183</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Eshwaran Venkat]]></dc:creator>
            <pubDate>Sun, 24 Dec 2023 09:15:03 GMT</pubDate>
            <atom:updated>2023-12-24T10:10:35.200Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BV04yQvgts1gpaB-" /><figcaption>Photo by <a href="https://unsplash.com/@waldemarbrandt67w?utm_source=medium&amp;utm_medium=referral">Waldemar</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article is primarily for any teams within companies that are performing a survey or due diligence for their data engineering, analytics, machine learning or business intelligence needs. I’m not affiliated with Databricks, and all thoughts and opinions in the article are my own, based on tech stack evaluations at our company.</p><h3>About Databricks</h3><p>Databricks is a company founded by the creators of Apache Spark, one of the major frameworks for big data processing in the modern era, born out of UC Berkeley. They still maintain a major portion of the Spark OSS, and have created and maintain additional open source tooling like <a href="https://delta.io/">Delta Lake</a>, a data storage format and <a href="https://mlflow.org/">Mlflow</a>, for tracking machine learning experiments.</p><p>When using Databricks, all data is stored within the company cloud provider, and all resources to compute, transform and work on that data is used from company cloud resources. As such, your data never leaves your tech ecosystem (without your explicit directive).</p><h3>The Myths / Misconceptions About Databricks</h3><ol><li><strong>It’s specifically built only for MS Azure 🧢</strong></li><li><strong>It’s a Jupyter notebook with add-ons 🧶</strong></li><li><strong>It’s only for big enterprises with big data 👔</strong></li><li><strong>It’s no good if you don’t use Apache Spark ⚡️</strong></li><li><strong>It’s cost compared to alternatives is unjustifiable 💰</strong></li></ol><h3>Databricks is only on Azure <strong>🧢</strong></h3><p>While <a href="https://azure.microsoft.com/en-in/products/databricks">Azure Databricks</a> is a Microsoft Azure service that does indeed exist, Databricks can be deployed on AWS and Google Cloud as well, and has no Azure dependencies when run elsewhere. The deployment is straightforward because Databricks maintains IaaC that can run the platform out of your cloud without tedious configuration (For example, using existing CloudFormation templates to get fully setup on AWS).</p><blockquote>Microsoft offers a cloud service <a href="https://news.microsoft.com/2017/11/15/microsoft-announces-azure-databricks-powered-by-apache-spark-new-ai-iot-and-machine-learning-tools-for-developers/">with “Databricks” in the name</a>, but Databricks is the one that’s running it ~ <a href="https://www.cnbc.com/2019/02/04/microsoft-invests-in-databricks-funding-at-2point7-billion-valuation.html">CNBC</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*9w1boSlprfidFoCzywdrVg.png" /><figcaption>Databricks documentation even provides specific documentation for different clouds (like S3 for AWS, and GCS for Google Cloud). Image by Author. Screenshot from docs.databricks.com</figcaption></figure><p><a href="https://www.geekwire.com/2019/microsoft-joins-250m-funding-round-boring-ai-company-databricks/">Microsoft has invested a large capital into Databricks</a>, which is a part of the reason why Azure Databricks exists. The Databricks features, services and support however, are very much apples-to-apples on AWS and Azure, with minor caveats on GCP. AWS and other companies have also recently been a part of <a href="https://www.databricks.com/company/newsroom/press-releases/databricks-closes-series-h-with-additional-participation-from-strategic-partners">Databricks’ funding round</a>, so the stakeholder portfolio is certainly getting diverse.</p><h3>Databricks is a Jupyter Notebook with Frills 🧶</h3><p>Databricks offers a <a href="https://www.databricks.com/product/collaborative-notebooks">Jupyter-Notebook-like interface</a> to develop code within their platform, and they have improved on the interface in multiple ways such as google-docs-like live editing of notebooks, highlighting code and leaving comments, a notebook section browser and more. However, the core philosophy of Databricks is to have all aspects of your company’s data lifecycle in one place, and on one managed platform. Three (of many) additional features include:</p><p><strong>Workflows</strong>: A fully managed orchestration tool, to run, manage and track jobs. Each task of a job can be a notebook, SQL query, event, etc and they can be put together to run batch or streaming pipelines</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rdit_EpaMfykrPvF.png" /></figure><p><strong>Unity Catalog: </strong>A managed view of the data lake as a set of coherent tables, schemas and catalogs with lineage information and controlled access for data governance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/0*wlucGpIoDvcrhwVx.png" /></figure><p><strong>Databricks SQL: </strong>It uses Delta Lake under the hood to store data, which provides a SQL interface to query data in a fast and clean way that lives on Object stores like S3, GCS or Blob storage. Databricks SQL adds on to ANSI-SQL with useful functions for slicing and dicing data a lot easier, and even incorporates frameworks like <a href="https://www.uber.com/en-IN/blog/h3/">H3</a> directly into their SQL language reference to use on large swaths of data with SELECT statements.</p><p><a href="https://towardsdatascience.com/uber-h3-for-data-analysis-with-python-1e54acdcc908">Uber H3 for Data Analysis with Python</a></p><h3>Databricks is only for Big Data &amp; Big Enterprises 👔</h3><p>While Databricks is already in use by large enterprises including many Fortune-500s, the goal here is to assess its viability for startups, or small data teams. At <a href="http://www.dotlas.com/">Dotlas</a>, we’ve been using Databricks for a while now and I’m personally of the opinion that choosing and evaluating options like Databricks is just as crucial as getting setup with platforms like AWS or Azure when starting out.</p><p>If you’re a company that has an application or platform that produces data in one or many databases at the beginning, that’s still good impetus for executives to start pushing for basic reporting. It’s preferable to use a platform like this instead of rebuilding the wheel, or even worse: <a href="https://betterprogramming.pub/data-engineering-is-not-software-engineering-af81eb8d3949">thinking that software and data engineering are identical</a>, and underestimating the problem in terms of getting a basic reporting pipeline and dashboard setup.</p><p>The Databricks pay-as-you-go model can help you get setup with a grassroots data lake ecosystem, that can scale up as your business grows, while keeping initial cost at no more than a couple hundred USD per month.</p><h3>Databricks can’t be adopted without up-skilling Apache Spark ⚡️</h3><p>The short answer is yes, Databricks is heavily built on top of Spark and when you’re not using Spark, you’re not harnessing the full capability of Databricks. But here’s the catch; When starting out — or migrating from an existing solution — you don’t always need to go max throttle and there are a number of reasons for this.</p><ol><li><strong>Databricks uses a Spark Runtime, but a lot of it is also abstracted</strong></li></ol><p>In SQL queries within your data lake, Spark knowledge isn’t necessary, as the underlying engine utilizes Spark SQL. Efficiency depends on delta partitions, but technologies like <a href="https://medium.com/@tsiciliani/liquid-clustering-with-databricks-delta-lake-57dc251d7870">Liquid clustering</a> can enhance performance. You can query in SQL as if you’re using a regular database, with full table and schema support.</p><p>For complex pipelines involving Python, Scala, or R, Spark isn’t mandatory. Tools like Pandas or Polars are viable for tasks like processing a 12M row dataset, especially with a large machine (32GB RAM) and optimized techniques like <a href="https://medium.com/analytics-vidhya/understanding-vectorization-in-numpy-and-pandas-188b6ebc5398">vectorized functions</a>. Databricks offers a smooth transition between Pandas and Spark DataFrames, with compatibility in data types.</p><p>However, using Spark is crucial for handling large batches of data or streaming tasks and for substantial data volumes.</p><p><strong>2. For basic pipelines, don’t use multi-node clusters</strong></p><p>Let’s say that you have a use-case where you have to transform sales data and store it in the warehouse. You may want to perform some aggregations, timestamp casting, fixing erroneous sales values and more. Perhaps your daily sales ingestion volume is a (1 Million x 25 Col) dataset. You may be tempted to bust your Jupyter notebook and create a Pandas transformation, or perhaps setup a job that loads this into a Postgres or mySQL database and then use SQL to do your thing.</p><p>Databricks will by default recommend multi-node clusters for the job. You can define a cluster configuration per specific job or pipeline requirement, or spin up a cluster for development too. Here’s an example in the image — when you click “Create Compute” and view the default (customisable) configs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*wYrwDw4dJVK30oRVBF-HKQ.png" /><figcaption>Default Configuration when Creating a New Cluster. This can be customized to upsize or downsize the hardware requirement. Image by Author.</figcaption></figure><p>It starts with 2–8x 30.5 GB machines. However, you can customize the hardware needs for that job, and in this case — perhaps a single node (1x 16 GB) or (1x 32 GB) machine will suffice. Keep in mind that Pandas or Polars do not natively use additional machines or cores, even if they’re available. Databricks allows you to track cluster metrics such as memory usage and CPU usage as well, so that you can make more informed decisions on downsizing or upsizing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*1SSi6VMq7i0OJrTEW7Hkhg.png" /><figcaption>Default Job Cluster Recommendation by Databricks. Image by Author.</figcaption></figure><p>I’ve heard a story or two where prototyping Databricks for small jobs broke the bank due to using the default configurations. Take this <a href="https://docs.databricks.com/en/compute/cluster-config-best-practices.html#cluster-sizing-considerations">cluster sizing article by Databricks</a> for example, where the cluster recommendations for data analysis is attached in the image (Omg!). While this could certainly be relevant for certain volumes, it’s largely a factor on the size of data you’re working with, and our initial example of sales data would not require this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/0*tpsOlmR-Q_zBlxKS.png" /><figcaption>Databricks recommended Cluster Sizing Configurations for Data Analysis. <a href="https://docs.databricks.com/en/compute/cluster-config-best-practices.html#data-analysis">Source</a></figcaption></figure><p>Databricks infra recommendations will be for mid-size jobs. The target market for Databricks is companies that have a threshold data volume and data related activity. If you’re willing to forego some support, talking to an account manager, etc. then you could use the pay-as-you-go model until you’ve scaled to the point where a dedicated plan or support will be beneficial.</p><h3>Databricks vs. Competition &amp; Cost 💰</h3><h4>Understanding Databricks’ Cost Structure</h4><p>With Databricks, you only pay for compute costs. Specifically Databricks Units (DBUs), which are a unit of processing capability per hour, billed per second of usage. There’s no Databricks invoiced cost of storing your data, or the ability to orchestrate workflows, lineage information, etc. Let’s take some simple scenarios and unfurl the cost factors.</p><ol><li><strong>Scenario A:</strong> Running a data ingestion for 32 minutes on AWS hosted Databricks to ingest data from a Postgres DB in Australia, and saving it to a data lake in California. The cost factors would be:<br>1. Databricks compute cost to run the AWS VM for 32 minutes<br>2. AWS EC2 Compute Cost to run a VM for 32 minutes<br>3. AWS Data Transfer Costs b/w Australia to California<br>4. AWS S3 Data Storage Costs for ingested GBs in California</li></ol><p>Keep in mind that there’s only one VM on AWS, and you pay Databricks through DBUs and AWS is paid based on standard VM charges. The implication is not that you pay for 2 separate VMs.</p><ol><li><strong>Scenario B:</strong> Training a Machine Learning Model on a GPU in AWS Hosted Databricks from Data in Unity Catalog (Delta Lake)<br>1. Databricks GPU compute cost in minutes<br>2. AWS GPU compute cost in minutes<br>3. AWS Data Transfer cost from S3 (Delta) to AWS/Databricks VM in GBs<br>4. AWS S3 Storage costs where initial data is stored</li></ol><p>Scenario B is interesting because you don’t pay for the lineage insights and governance from Unity Catalog provided by Databricks.</p><h4>Comparison Overview</h4><p>This section will further (non-meticulously, cause that’s for you to do) compare:</p><ol><li>Databricks with Snowflake</li><li>Databricks with Native Cloud Analytics Tools (AWS / GCP)</li><li>Databricks compared with Open Source &amp; Other Managed Tooling</li></ol><h4>Snowflake</h4><p>Snowflake is probably one of the closest competitors to Databricks, and have seen a meteoric rise (partially due to effective marketing) in the last few years. Snowflake also bills customers by compute costs only. Databricks excels in advanced data processing, particularly for machine learning and data science tasks, leveraging Apache Spark. Snowflake, while also powerful, is more focused on data warehousing and SQL-based analytics, although have recently introduced <a href="https://www.snowflake.com/en/data-cloud/snowpark/"><strong>Snowpark</strong></a>, to allow python based data science and big data engineering capabilities. Their acquisitions of <a href="https://techcrunch.com/2022/03/02/snowflake-acquires-streamlit-for-800m-to-help-customers-build-data-based-apps/"><strong>Streamlit</strong></a>, a popular data science web application development kit, and <a href="https://www.snowflake.com/blog/snowflake-to-acquire-ponder/"><strong>Ponder</strong></a>, a <a href="https://www.ischool.berkeley.edu/">UC Berkeley School of Information</a> based startup born from the creators of <a href="https://modin.readthedocs.io/">Modin</a>, suggest that they want to play the data science platform game head-on with Databricks for market share. The larger industry almost treats Spark as a standard for big-data processing, which is Databricks’ home turf. This is a bridge Snowflake (and its users) have to cross, and acquisitions of data processing frameworks like Ponder can be the first step to building their own big-data processing engine to rival Spark.</p><p>Both Snowflake and Databricks are considered leaders in this space, on a tier of companies right after big tech giants by Gartner:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4KiKGuEbdN_IZ8QX.png" /><figcaption>Gartner Magic Quadrant for Data Analytics Providers December 2023. <a href="https://www.databricks.com/blog/databricks-named-leader-2023-gartner-magic-quadrant-cloud-database-management-systems">Source</a>.</figcaption></figure><h4>Native Cloud Tools</h4><p>Each cloud provider has native tools for data ingestion, engineering, analysis and machine learning. Here are some examples:</p><ol><li><strong>AWS</strong>: AWS <strong>Glue</strong> for Ingestion, <strong>Athena</strong> for Analysis, <strong>SageMaker</strong> for training models, <strong>Step Functions</strong> &amp; <strong>Lambda</strong> for launching jobs.</li><li><strong>Azure</strong>: <strong>Data Factory (ADF) </strong>for data ingestion, integrating with various data stores. <strong>Synapse Analytics </strong>combines big data and data warehousing for analysis. <strong>Azure ML</strong> for building and training machine learning models. <strong>Logic Apps</strong> and <strong>Azure Functions</strong> for orchestrating and launching jobs.</li><li><strong>GCP</strong>: <strong>Dataflow</strong> for ingestion and data processing, particularly for stream and batch data. <strong>BigQuery</strong> for large-scale data analysis using SQL queries. AI Platform for training and deploying machine learning models. <strong>Cloud Functions</strong> and <strong>Cloud Composer</strong> for job orchestration and automation.</li></ol><p>In contrast to Databricks’ unified approach, AWS, Azure, and GCP offer a more compartmentalized suite of services, each tailored to specific data management tasks. While Databricks tends to offer a more streamlined workflow, the choice between it and cloud services depends on your budget and team expertise. If your team is adept in managing cloud services (through certificatons or experience), then they may be viable options. Furthermore, it’s important to consider the user-friendliness of cloud services, especially for team members more directly involved in business operations. For example, accessibility is crucial when considering tools for team members who may not be as technically versed</p><p>Hence consider not just infrastructure costs, but also the savings in developer time, ease of maintenance and accessibility (coupled with security). For instance, <strong>setting up workflows in Databricks can be more straightforward compared to configuring AWS CloudWatch Triggers and Lambda functions</strong>. Ultimately, the choice should align with your business goals, emphasizing efficiency and outcome.</p><h4>Open Source &amp; Other Managed Services</h4><p>Finally, there’s been an explosion of data infrastructure and analytics products in the past few years that either start off, or build on an existing open source technology. You just need to search “Modern Data Stack” for web or image results and they come pouring in. The truth is, you do need to use a service as it’s often worse to build a platform for your pipelines from scratch, as that’s almost akin to setting up your own datacenter in your garage in 2023. This could be through in-built cloud services as described in the last section, or integrating an open source or managed service.</p><p>Some common open source examples include: Mage, Airflow, DBT, Prefect, Dagster, Ploomber, Apache Superset, etc. Managed services include Airbyte, Fivetran, DBT, Census, Mode, Dataiku, Metabase and many more.</p><p>A lot of these tools focus on doing a specific part of the data lifecycle and they do it well. The choice of whether to use these or opt for a more comprehensive solution like Databricks depends on your unique requirements and willingness to invest in learning. Opting for a comprehensive platform like Databricks could streamline learning and application, as it integrates multiple features of these diverse tools into one.</p><p>Keep in mind that Databricks is also a managed service that’s built on top of open source. Databricks has also built some proprietary additions on top of these and are transparent about these in their documentation. Databricks, as a company are also very agile in adding new features and re-working their platform based on user inputs. Just take a look at the volume of changes and additions they’ve <a href="https://docs.databricks.com/en/release-notes/product/index.html">added in 2023</a>.</p><blockquote>Ultimately, the decision hinges on prioritizing efficient business outcomes over the intricacy of development processes. Databricks in my opinion, is great at getting to results than tinkering with a myriad of tools in the wake.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18140cc3d183" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create a Shared Python Dev Environment for your Team — Part 2: Configuring a Devcontainer]]></title>
            <link>https://eash98.medium.com/create-a-shared-python-dev-environment-for-your-team-part-2-configuring-a-devcontainer-974a83833c0f?source=rss-7c131006709c------2</link>
            <guid isPermaLink="false">https://medium.com/p/974a83833c0f</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[devcontainer]]></category>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[collaboration]]></category>
            <dc:creator><![CDATA[Eshwaran Venkat]]></dc:creator>
            <pubDate>Sat, 25 Feb 2023 16:53:29 GMT</pubDate>
            <atom:updated>2023-02-25T17:30:13.792Z</atom:updated>
            <content:encoded><![CDATA[<h3>Create a Shared Python Dev Environment for your Team — Part 2: Configuring a Devcontainer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*65pmg8CJt2BhIO-d" /><figcaption>Photo by <a href="https://unsplash.com/@sigmund?utm_source=medium&amp;utm_medium=referral">Sigmund</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This is Part-2 of a two-part series on creating a shared Python development environment for a team. In both these articles, we shall introduce and use <a href="http://containers.dev/">D<strong>evcontainers</strong></a> to create a shared development environment. This will cover configuration and basic setup syntax. We shall also cover <a href="https://github.com/codespaces"><strong>GitHub Codespaces</strong></a><strong> </strong>as a viable option for a shared workspace on the cloud using devcontainers. Pros and cons are also highlighted using Python packages as examples. In Part-1, we introduced GitHub Codespaces along with some utilities and best practices to manage shared python package projects. <strong>If you’re familiar with GitHub Codespaces as a concept, then I’d recommend skipping Part 1</strong>.</p><p>Link to Part-1: <a href="https://eash98.medium.com/create-a-shared-python-dev-environment-for-your-team-part-1-github-codespaces-project-7a32d2bbea33"><em>Introduction to GitHub Codespaces &amp; Project Utilities</em></a></p><h3>Table of Contents for Part 2</h3><blockquote>1. <a href="#d754">Setting up Codespaces for Teams</a><br>2. <a href="#c269">Introduction to Devcontainers</a><br>3. <a href="#c3f7">Creating a Setup Script</a><br>4. <a href="#19a5">Creating a Codespace Prebuild</a><br>5. <a href="#5175">Jupyter Notebook on a Codespace</a></blockquote><h3>Setting up Codespaces for Teams</h3><p>The general template for Codespaces involves getting a machine setup to develop on a single repository. This may still not be of much use when using in teams when variables like conda environments, package management and git hooks come to play. You still need to go into a new Codespace and spend some time performing an initial setup before you can use the repository.</p><p>Also, GitHub by-default blocks private repositories from being cloned in a Codespace. If you have some of your team’s private repositories that you would like to switch between in a single Codespace, you need to setup additional configurations. Else, you will be working on one Codespace per repository. While there are use-cases to work on different projects independently, we’ll consider a different scenario as given below.</p><p><strong>Let’s create a shared python development environment in GitHub Codespaces based on the following scenario / requirements.</strong></p><ul><li><strong>Access and work with 2 (inter-dependent) repositories in a single Codespace</strong>. Both repositories are internal (or <strong>private</strong>) organization repositories and are python packages. This means that a direct pip install &lt;package_name&gt; for instance, won’t work as the public internet (and PyPi) does not know that these packages exist outside your organization. Let the first package be a dependency for the second package repository. This means that the second package won’t work in an environment where the first doesn’t exist and is installed.</li></ul><blockquote>For example, let the first package be <em>apple</em> and the second package be <em>apple-juice</em> . <em>apple</em> can have public dependencies like <em>pandas</em> , <em>Django</em>, etc. And <em>apple-juice</em> can also have public dependencies like <em>fastapi</em>, but also has a dependency <em>apple</em>. You can <em>import apple</em> and <em>import apple-juice</em> inside some python application. <em>apple-juice</em> will import from <em>apple</em> implicitly.</blockquote><ul><li><strong>A single </strong><strong>conda (or equivalently a </strong><a href="https://anaconda.org/conda-forge/mamba"><strong>mamba</strong></a><strong> ) environment is available on starting the Codespace where both the package repositories are installed in an </strong><a href="https://stackoverflow.com/questions/35064426/when-would-the-e-editable-option-be-useful-with-pip-install"><strong>editable form</strong></a><strong>.</strong> This means that live changes to the package code will affect its installed functionality, thereby having no separation between development and usage.</li></ul><blockquote>For example, say <em>apple</em> has a function <em>apple.squeeze()</em> which is used by <em>apple-juice</em>. Then editing this function inside the <em>apple</em> codebase will affect what <em>apple-juice</em> sees of that function live.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIpNZOzvrt6-gYAXKLW0Kg.png" /><figcaption>Illustration of Conda Environments for managing dependencies. Image by <a href="https://nbisweden.github.io/workshop-scRNAseq/conda_instructions.html">NBIS</a> (National Bioinformatics Infrastructure Sweden)</figcaption></figure><ul><li><strong>Utilities such as </strong><strong>awscli, </strong><strong>npm, </strong><strong>pnpm, </strong><strong>black, </strong><strong>flake8, </strong><strong>isortetc. are also installed and available for usage.</strong> awscli is used to interface with Amazon Web Services from the terminal. npm and pnpm are package managers for JavaScript bundles (which can sometimes be useful even inside Python projects, as we’ll see). The remaining utilities assist with linting and formatting in Python.</li><li><strong>VSCode IDE Settings and Recommended Extensions come pre-installed. </strong>This ensures that productivity hacks are shared between colleagues.</li></ul><blockquote>For example, if your organization supports <a href="https://www.freecodecamp.org/news/how-to-use-github-copilot-with-visual-studio-code/">GitHub Copilot</a> subscriptions, then having the Codespace initialize with the <a href="https://marketplace.visualstudio.com/items?itemName=GitHub.copilot">Copilot extension</a> is a cherry on top.</blockquote><blockquote>I’ve used Copilot for over a year now, and can’t recommend it enough. It allows me to focus on the meat of the problem when programming and avoid writing boilerplates. Fun fact: Copilot can also write documentation when prompted.</blockquote><h3>Introduction to Devcontainers</h3><p>A devcontainer (development container) is a development environment that is defined and packaged as a Docker container, which can be used by developers to ensure that everyone working on a project is using the same development environment. Devcontainers can be used with various programming languages, including Python, and can include tools like code editors, compilers, linters, and other development tools. By defining a development environment as a devcontainer, developers can ensure that everyone working on a project has access to the same dependencies and tools, making it easier to collaborate and debug code issues.</p><p>Devcontainers can be defined using a configuration file, which specifies the tools and dependencies needed for a particular development environment. Visual Studio Code, a popular code editor, includes built-in support for devcontainers, allowing developers to easily create and use a devcontainer for their projects. In the context of Python development, a devcontainer can include tools like a specific version of Python, the pip package manager, and tools like Black and Flake8 for code formatting and style checks. Overall, devcontainers are a powerful tool for ensuring consistency in development environments and improving collaboration among developers.</p><blockquote>Note that devcontainers can be used without GitHub Codespaces. It is an independent project. <strong>You can create shared python development environments by distributing a devcontainer file with your team, and they can work out of their local systems. </strong>GitHub codespaces only provides a managed infrastructure service for running devcontainers.</blockquote><p>To launch a development environment using a devcontainer locally, you will need to have Docker installed on your local machine. Here are the steps to launch a Python development environment using a devcontainer in Visual Studio Code:</p><ol><li>Open Visual Studio Code and navigate to the root directory of your Python project.</li><li>Create a .devcontainer directory at the root of your project, if it doesn&#39;t already exist.</li><li>Create a devcontainer.json file inside the .devcontainer directory, and specify the Docker image and any additional tools and dependencies that you want to include in the devcontainer.</li><li>Open the Command Palette in Visual Studio Code (using Ctrl+Shift+P on Windows/Linux or Cmd+Shift+P on macOS) and select &quot;Remote-Containers: Reopen in Container&quot;.</li><li>Visual Studio Code will rebuild the Docker container based on the devcontainer.json file and open a new window inside the container. You can now use this window to develop your Python code using the specified tools and dependencies.</li></ol><p>Note that the specific steps for creating and launching a devcontainer may vary depending on your project’s specific requirements and the tools and dependencies you want to include in the devcontainer. However, using devcontainers can be an efficient way to set up a consistent development environment for your Python project and ensure that all developers are using the same tools and dependencies.</p><p>Devcontainers for GitHub Codespaces lends itself to be free to add more than one json file that has multiple devcontainer configurations. One of these can be chosen when instantiating a Codespace.</p><p>In the example of apple and apple-juice, we can either add the same devcontainer at the repository root of both codebases, or we can create a third repository called orchard in which the Codespace will clone into and install both apple and apple-juice.</p><p><strong>Here’s a format of a devcontainer for a codespace to enable </strong><strong>apple and </strong><strong>apple-juice repositories. </strong>Let’s assume these are part of the dotlas organization. For ex: github.com/dotlas/apple</p><pre>{<br>  &quot;image&quot;: &quot;mcr.microsoft.com/devcontainers/universal:2&quot;,<br>  &quot;customizations&quot;: {<br>    &quot;codespaces&quot;: {<br>      &quot;repositories&quot;: {<br>        &quot;dotlas/apple&quot;: {<br>          &quot;permissions&quot;: &quot;write-all&quot;<br>        },<br>        &quot;dotlas/apple-juice&quot;: {<br>          &quot;permissions&quot;: &quot;write-all&quot;<br>        }<br>      }<br>    }<br>  }<br>}</pre><blockquote>Note that the Codespace will gain <em>write</em> permissions to the repositories in question. These permissions can be changed based on the level of authorization given to various members of a team. To view more information about permissions, view the GitHub docs on <a href="https://docs.github.com/en/codespaces/managing-your-codespaces/managing-repository-access-for-your-codespaces">managing repo access to codespaces.</a></blockquote><p>The image key is used to pull and build a basic container. In this case we are using Microsoft’s universal image for Codespaces (which comes with tools like conda , git , nvm, etc. pre-installed). There is a marketplace of multiple images from various open source organization and others which can also be used.</p><p>If one of the team members who will use the Codespace do not have permissions to the specific repository mentioned in devcontainer.json, then they will not be able to clone that specific repository within the Codespace, despite a write permission at the container level.</p><p>Let’s say you want to add some VSCode Extensions pre-configured when the Codespace pops up. You can specify this within /customizations -&gt; vscode. Where customizations is a top-level JSON key as seen above.</p><pre>{<br>  &quot;image&quot;: &quot;mcr.microsoft.com/devcontainers/universal:2&quot;,<br>  &quot;customizations&quot;: {<br>    &quot;vscode&quot;: {<br>      &quot;extensions&quot;: [<br>        // Basic Extensions<br>        &quot;github.codespaces&quot;,<br>        &quot;github.copilot&quot;,<br>        <br>        // Python Extensions<br>        &quot;ms-python.python&quot;, // Python Language Support<br>        &quot;ms-python.vscode-pylance&quot;, // Python Language Support<br>        &quot;njpwerner.autodocstring&quot;, // Function Docstring Templater<br>        &quot;bungcip.better-toml&quot;, // TOML Syntax Highlighting<br>        <br>        // Shell Extensions<br>        &quot;bmalehorn.shell-syntax&quot;, // Shell Syntax Highlighting<br>        <br>        // Helpers<br>        &quot;grapecity.gc-excelviewer&quot;, // Previewing CSVs<br>        &quot;randomfractalsinc.geo-data-viewer&quot;, // View Maps in VSCode<br>        &quot;yzhang.markdown-all-in-one&quot;, // Markdown helpers<br>        &quot;gruntfuggly.todo-tree&quot;, // TODO Tree - for viewing code tasks<br>        &quot;streetsidesoftware.code-spell-checker&quot;, // Natural Language Spell Checker<br>        &quot;amazonwebservices.aws-toolkit-vscode&quot;, // Viewing AWS Resources<br>        &quot;mark-tucker.aws-cli-configure&quot; // Configuring AWS CLI made easy<br>      ]<br>    }<br>  }<br>}</pre><p>Adding this devcontainer will allows us to create a Codespace where we can clone the repositories we require and also have the extensions working on the VSCode session we’re using to remotely access this Codespace. While this doesn’t meet the goals of the scenario we described earlier, we are certainly getting close.</p><h3>Creating a Setup Script</h3><p>While a new Codespace does come with some tools installed, and has the ability to clone the repositories we need, we can still automate the setup process. For example, the ideal scenario is to enter the codespace and just get to work! No conda initialization, no installing packages or initializing hooks, etc. With the devcontainer setup so far, we would still need to enter the Codespace and run some commands to get make our workspace feel like home.</p><p>A devcontainer can be configured to run a script (bash script / python script, etc.) at various stages of creating a Codespace. View the <a href="https://containers.dev/implementors/json_reference/#lifecycle-scripts">Lifecycle Docs</a> for more information on the various creation stages. Note that the script can only run during <strong>creation</strong> and does not run every-time a Codespace is resumed. In our scenario, we’ll use the onCreateCommand key in our devcontainer. This signals to the codespace that once it receives the image and builds the barebones of the container, we want the script to run immediately after that. A postCreateCommand will run the script after the Codespace is created and the user session has begun within the remote VSCode IDE. We shall use a bash script to setup the remaining parts of our shared development environment. We will store it in the project repository’s root under the scripts directory. For ex. github.com/dotlas/orchard/main/scripts/setup.sh</p><pre>{<br>  &quot;image&quot;: &quot;mcr.microsoft.com/devcontainers/universal:2&quot;,<br>  // Setup - Runs on creation and installs all dependencies<br>  &quot;onCreateCommand&quot; : &quot;/workspaces/orchard/scripts/setup.sh&quot;<br>}</pre><p>In the setup.sh script, we will create a bash script which will initialize various aspects of our development environment. In our example scenario, we shall initialize a condaenvironment called cider which will have our packages.</p><pre>#!/bin/bash<br><br># declare globals<br><br># directory where all the repos will be downloaded<br>MAIN_DIR=&quot;/workspaces&quot;<br><br># directory where hive scripts will be stored<br>SCRIPT_DIR=&quot;$MAIN_DIR/orchard/scripts&quot;<br><br># list of repos to download and optionally install<br>REPOS=(&quot;apple&quot; &quot;apple-juice&quot;)<br><br># name of the conda environment<br>ENV_NAME=&quot;cider&quot;<br><br># python version<br>PYTHON_VERSION=&quot;3.10&quot;</pre><p>Note how apple precedes apple-juice since apple is its dependency and needs to be downloaded and installed first. We shall check if the environment exists, and create it if it does not.</p><pre># Check if the environment already exists<br>if (conda env list | grep $ENV_NAME); then<br>    echo &quot;$ENV_NAME environment already exists&quot;<br>else<br>    echo &quot;Creating $ENV_NAME environment&quot;<br>    conda create --name $ENV_NAME python=$PYTHON_VERSION -y<br>fi<br><br># Activate the environment through the conda shell<br>source /opt/conda/etc/profile.d/conda.sh<br>conda activate $ENV_NAME<br><br># Update the pip version for installing editable packages<br>python -m pip install --upgrade pip</pre><p>A few points of note:</p><ul><li>The Codespace can run the command conda env list because conda comes pre-installed with the Microsoft Universal image for Codespaces.</li><li>We need to source into the conda.sh because conda activate commands in bash / zsh to enable environments requires initialization, and then a restart of the terminal. We want to perform the setup in such a way that this “new launch” of a terminal happens when the user enters the Codespace and launches for the first time with the condaenvironment ready-to-go.</li></ul><pre># Download (clone) the repos<br><br>### REPOS=(&quot;apple&quot; &quot;apple-juice&quot;) declared previously<br>### MAIN_DIR=&quot;/workspaces&quot; declared previously<br><br>cd $MAIN_DIR<br><br>for repo in &quot;${REPOS[@]}&quot;; do<br>    if (ls | grep $repo); then<br>        echo &quot;$repo already exists - download skipped&quot;<br>    else<br>        echo &quot;Downloading $repo&quot;<br>        git clone https://github.com/dotlas/$repo<br>    fi<br>done</pre><p>Finally, we can install the repositories in an editable fashion. We first check if the repos are already installed successfully, and only install them if they are not found in the current condaenvironment.</p><pre># Check if the repos are installed, and install them if not<br><br>### REPOS=(&quot;apple&quot; &quot;apple-juice&quot;) declared previously<br>### MAIN_DIR=&quot;/workspaces&quot; declared previously<br><br>cd $MAIN_DIR<br>for repo in &quot;${REPOS[@]}&quot;; do<br>    if (python -m pip show $repo | grep -q &quot;Name: $repo&quot;)<br>    then<br>        echo &quot;$repo already installed&quot;<br>    else<br>        echo &quot;$repo not installed&quot;<br>        <br>        # Reset pip cache to avoid errors<br>        mv ~/.cache/pip ~/.cache/pip.bk<br>        <br>        # Install the repo<br>        echo &quot;Installing $repo&quot;<br>        cd $repo<br>        python -m pip install -e .<br>        cd $MAIN_DIR<br>    fi<br>done</pre><p>Now we can install any additional dependencies as well</p><pre># initialize python linters and formatters<br>python -m pip install black isort ruff pre-commit bandit<br><br># install pnpm package manager for JS bundles<br>npm i -g pnpm<br><br># Install AWS CLI<br>python -m pip install awscli boto3</pre><p>Finally, <strong>remember to initialize the </strong><strong>conda environment</strong> in the shell so that the user can view and use the conda environments.</p><pre># initialize conda environment to bash, so that new terminal has `base` environment loaded<br>conda init bash &gt;/dev/null</pre><p>All of the above snippets constitute setup.sh broken down into different functionalities. Feel free to spruce up setup.sh with additional tools, technologies or configurations. <strong>You can also </strong><a href="https://docs.github.com/en/actions/security-guides/encrypted-secrets"><strong>store secrets</strong></a><strong> as environment variables when your Codespace is created through GitHub Actions</strong>. This is done by storing a secret at the repository level or organization level in GitHub. This can be done through the Github UI or API.</p><p>For example, if you save ORCHARD_AWS_KEY in GitHub as a secret, then it can be accessed within your Codespace through the environment variable $ORCHARD_AWS_KEY. Secrets like this can be used to make additional configurations.</p><p>Once your setup script and devcontainer are ready, you can create a Codespace and watch for a few mins while GitHub gets it ready for you. Anyone from your team can now enter this repository (orchard) and use it to work on apple and apple-juice. <strong>When they create the Codespace for the first time, the setup script runs and then further syncs to the repositories will need to be done through </strong><strong>git pull from within the Codespace.</strong></p><p>Debugging devcontainers can be a bit slow when done through Codespaces. It might make sense to debug locally, and when you have a working container — then add to the repository to make a Codespace from. Remember to enter the VSCode session once changes are made and try out a few commands from the terminal to ensure that everything is working as intended.</p><h3>Creating a Codespace Prebuild</h3><p>A prebuild lets you create a Codespace from a devcontainer (which in turn may setup shop using a setup script as mentioned in the previous section), and store it so that new Codespace “creations” do not take as much time. <strong>It’s a way to <em>cache</em> your built development environment so that it can spawn quickly</strong>. A prebuild is created through a GitHub action (CI) and can be configured to run manually, on a schedule, on every push or to a devcontainer configuration change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2BcqWbVE1jZgWLVdjjGAPg.png" /><figcaption>Option to setup prebuilds from Github UI. Image by Author.</figcaption></figure><h3>Jupyter Notebook on a Codespace</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Mm-V8Rqnvm0WU1kqzCKkA.png" /><figcaption>Despite creating a Jupyter Notebook on Localhost, Codespaces points you to the machine’s web UI. Image by Author.</figcaption></figure><p>One way to run Jupyter notebooks on a Codespace is by installing Jupyter in the anaconda environment created prior and running jupyter notebook or jupyter lab from there. A new browser window / tab will take you to your Jupyter session. You can also optionally start a Codespace purely on Jupyter, similar to the way Jupyterhub works. This means, you don’t SSH into the machine, but only access the machine through the Jupyter UI and terminal commands are first received through HTTP and passed to the underlying OS through the Jupyter system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U0DiSfmQSiGwBDjZ0r4kDg.png" /><figcaption>A Jupyter Notebok template option provided by GitHub to use as a Codespace. Image by Author.</figcaption></figure><p>It is however recommended if you’re running large ML or Data engineering jobs to use <a href="https://jupyter.org/hub">Jupyterhub</a>. Jupyterhub allows you to share Jupyter Notebook instances between your team by orchestrating variable size docker containers, or just simply deploy on VMs. The setup time required to get Jupyterhub up and running is significantly greater than using GitHub Codespaces. That is of course, more of an <em>apples</em> to <em>oranges</em> comparison. Ba-dum-tss! (cause Jupyter is orange, and we used the apple repository for Codespaces, get it? :P)</p><p>Nevertheless, you could also run Jupyter notebooks locally when devcontainers are used to build containers on local systems instead of on the cloud with Codespaces.</p><p>Thanks for reading!</p><p>P.S, don’t go looking for dotlas/apple and dotlas/apple-juice since they regrettably do not exist.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=974a83833c0f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create a Shared Python Dev Environment for your Team — Part 1: Github Codespaces & Project…]]></title>
            <link>https://eash98.medium.com/create-a-shared-python-dev-environment-for-your-team-part-1-github-codespaces-project-7a32d2bbea33?source=rss-7c131006709c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a32d2bbea33</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[collaboration]]></category>
            <category><![CDATA[devcontainer]]></category>
            <category><![CDATA[jupyterhub]]></category>
            <category><![CDATA[cookie-cutters]]></category>
            <dc:creator><![CDATA[Eshwaran Venkat]]></dc:creator>
            <pubDate>Sat, 25 Feb 2023 16:53:00 GMT</pubDate>
            <atom:updated>2023-02-25T17:32:06.735Z</atom:updated>
            <content:encoded><![CDATA[<h3>Create a Shared Python Dev Environment for your Team — Part 1: GitHub Codespaces &amp; Project Utilities</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WR3CiZMv_m475pQ_" /><figcaption>Photo by <a href="https://unsplash.com/@disruptxn?utm_source=medium&amp;utm_medium=referral">Desola Lanre-Ologun</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This is Part-1 of a two-part series on creating a shared Python development environment for a team. In both these articles, we shall introduce and use <a href="http://containers.dev/">D<strong>evcontainers</strong></a> to create a shared development environment. This will cover configuration and basic setup syntax. We shall also cover <a href="https://github.com/codespaces"><strong>GitHub Codespaces</strong></a><strong> </strong>as a viable option for a shared workspace on the cloud using devcontainers. Pros and cons are also highlighted using Python packages as examples. In Part-2, we get into the nitty-gritty details and use a hypothetical team scenario to set up a shared Devcontainer configuration from scratch. <strong>If you’re familiar with GitHub Codespaces as a concept, then I’d recommend skipping to Part 2</strong>.</p><p><a href="https://eash98.medium.com/create-a-shared-python-dev-environment-for-your-team-part-2-configuring-a-devcontainer-974a83833c0f">Link to Part-2: <em>Configuring a Devcontainer</em></a></p><h3>Table of Contents for Part 1</h3><blockquote>1. <a href="#15f1">The Problem</a><br>2. <a href="#6644">Goals of a Potential Shared Development Environment</a><br>3. <a href="#215f">What is a GitHub Codespace</a><br>4. <a href="#2c56">Codespace States</a><br>5. <a href="#1884">Codespace Management from VSCode</a><br>6. <a href="#698c">Shortcomings of Codespaces</a><br>7. <a href="#0eaa">Project Setup Best-Practices</a></blockquote><h3>The Problem</h3><p>Docker and related containerization products are widely used to ship and test production code uniformly. Besides scaling easier, they also ensure that the code running on prod is in a stable, deterministic and consistent environment. There is however no prevailing consistency in environment during development, and this is a good thing. Developers, data scientists, researchers, engineers and other related professionals prefer to have control over the customization of the environment they’re building and testing on, as it enables freedom and ownership of the nuts &amp; bolts used to bring their workspace together.</p><p>There are however situations where it makes sense to abstract away the build and config process of a development environment and distribute a common environment among a team should they need it. Essentially, “<strong>to</strong> <strong>pick up an environment from off the shelf and get started”</strong>. To list a couple of these scenarios:</p><ul><li><strong>Onboarding new team members. </strong>Onboarding pipelines can be sometimes extensive when bringing in fresh blood into a team. The bottleneck is usually the time spent understanding internal systems and projects. These are often not introduced all at once either due to complexity reasons, compliance, etc. This process also becomes difficult to standardize in a startup environment where tools and configs can change frequently at the beginning, with new gizmos coming in all the time. In other scenarios, new colleagues consequently start out sometimes with a “<a href="https://en.wikipedia.org/wiki/Fog_of_war">fog-of-war</a>”-type view of their work or project. As a result, this could affect what tools (and versioning) they setup for themselves for development, and end up having to reconfigure or retrofit in the short and medium term when a dependency is not met, or breaks.</li><li><strong>Teams with skill variance. </strong>Not all team members are developers, and not all developers are made equal. This is often a strength in diversity, because no-one chooses the entire squad to be mages when playing DnD do they? The problem is that helping with configuration or setup is sometimes frustrating for someone ‘with experience’ to share with someone ‘without’. It’s natural to want to spend your time solving the real problems, and not wanting to spend 2 hours to realize that two projects used different versions of a library. This introduces friction in terms of asking and offering support to setup shop. Unbeknownst to most, this fabricates an invisible barrier into an org’s tech stack &amp; projects.</li></ul><h3>Goals of a Shared Development Environment</h3><p>So, let’s outline some <strong>goals we want to achieve with a shared development environment:</strong></p><ul><li>Comes in-built with <a href="https://en.wikipedia.org/wiki/Command-line_interface">CLI</a> tools, extensions and other technologies that are used for development and general productivity</li><li>Code repositories required for a specific project or team are made available and installed for building, debugging and testing with minimal action required from a developer to setup or initialize.</li><li>Allows one to switch the type of machine based on the workload of their development, and further configure the machine or machine template for sharing.</li><li>Standardize automated-best practices such as linting, formatting, etc.</li><li>Access the development environment with ease (allowing for some security-based friction).</li></ul><p>Now that we’ve established the problem and goals, we need only explore the tools available to bring a shared development environment to life. This occurs in two stages.</p><ul><li>The first stage is the <strong>workspace</strong> itself, such as making a machine available to work on, with team-wide configs and productivity tools in-place.</li><li>The second stage is the <strong>project-level configurations</strong> that make it easy to get started building and modifying one or more projects within the workspace from stage 1.</li></ul><h3>What is a GitHub Codespace</h3><p>Let’s get the trivialities out of the way. I’m not a huge fan of the GitHub docs so I’ve attempted to simplify a lot of the bare-bones setup.</p><p>GitHub Codespaces are a product offered by GitHub to develop your code on the cloud. This means that you receive a full-fledged virtual machine that comes pre-configured with a repository of your choosing. You need to have a GitHub account to access and use Codespaces. Think of a “Codespace” as meaning a “VM” (Virtual Machine).</p><blockquote>For all intents and purposes, when you think “Codespace”, just think “VM asterisk”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*enqhS4BJVoWnqJj98ffjNQ.png" /><figcaption>Creating a Codespace for <a href="https://github.com/dotlas/mission_dotlas">Mission Dotlas</a> repository from GitHub UI. Image by Author.</figcaption></figure><p>This will open a new tab and display a VSCode IDE on the web with the repository loaded-in and ready for development.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MWLquu44JhGjBfBBOYaRQw.png" /><figcaption>An in-browser VSCode IDE from the <a href="https://github.com/dotlas/mission_dotlas">Mission Dotlas</a> Codespace. Image by Author.</figcaption></figure><p>Notice how codespace is indicated in the bottom-left of the IDE</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*pCsxCkNXwzK3HI_ypr-5wg.png" /><figcaption>Remote connection to codespace in browser VSCode IDE. Image by Author.</figcaption></figure><blockquote>Codespaces are priced based on <strong><em>storage used and hours consumed</em></strong>. GitHub has a free-tier of the maximum amount of these parameters that can be used before being charged. Refer to <a href="https://docs.github.com/en/billing/managing-billing-for-github-codespaces/about-billing-for-github-codespaces">GitHub docs</a> for more info.</blockquote><h3>Codespace States</h3><ul><li><strong>Creating</strong> a Codespace means you request GitHub to get you a VM with the repository loaded-in. GitHub will allocate a machine from one of their server farms (usually the one closest to you by latency) and spin up a machine for you to use.</li><li>A Codespace is <strong>deleted </strong>if the user requests the Codespace to be deleted. This means that all saved data or work done in the Codespace that has not been pushed to remote will be lost. This includes data on-disk.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*skUQhs_IO9VHqIZRf6JezA.png" /><figcaption>Delete option for the created Codespace. Image by Author.</figcaption></figure><ul><li>A Codespace is <strong>Stopped </strong>when the user requests it to stop, or if the Codespace is idle for a specific amount of time. Data saved on disk (such as new files in the repository) will persist when the Codespace is stopped, but data on RAM (i.e, running jobs) will be terminated.</li><li>A Codespace is <strong>Started</strong> or <strong>Resumed</strong> when it goes from a stopped state to running .</li><li>Multiple codespaces can be setup on a single branch.</li><li>When the machine-type of a Codespace is changed (i.e, modifying CPUs or RAM), then the Codespace is <em>stopped</em> and then <em>resumed </em>with the new system configuration.</li><li><strong>TLDR; A Codespace is <em>created</em> or <em>deleted</em> once. It can be <em>started</em> or <em>stopped</em> multiple times while persisting data on disk</strong></li></ul><h3>Codespace Management from VSCode</h3><p>While we used the browser in the previous section to create and manage codespaces, this can also be achieved from inside the VSCode Desktop Application IDE. Download and install the <a href="https://marketplace.visualstudio.com/items?itemName=GitHub.codespaces">GitHub Codespaces Extension</a> or search for it by the extension id github.codespaces</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hKni48P1G-UbnC4F_BuLyA.png" /><figcaption>Official GitHub Codespaces extension available on VSCode. Image by Author.</figcaption></figure><p>This makes Codespaces available through the command palette to manage, create, stop and resume our machines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CHzGUjByrlkj-MmqFnYdcw.png" /><figcaption>Ctrl / Cmd + Shift + P and type `Codespaces` to see options. Image by Author.</figcaption></figure><h3>Shortcomings of Codespaces</h3><p>Codespaces are not great if you want to run large jobs on the cloud and are just looking for a VM to do so. One reason is because the maximum idle timeout is 4 hours. Another reason is that lower tiers of machines on Codespaces offer very low disk space (and remember that the OS also has to be loaded in, along with swap space for memory). Codespaces are primarily meant for providing as much compute and memory necessary to develop and push code which can then go run large jobs on the cloud elsewhere.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*ZHQkKbLrtf6ri99e6UwanA.png" /><figcaption>GitHub Codespaces Machine Types as of Feb 2023. Image by Author.</figcaption></figure><p>The idle timeout can also be a boon to use Codespaces, as it can lead to significant cost-savings as opposed to running VMs when no-one on the team is using them. Codespaces are also useful for situations when you want to spend less time on setup and infrastructure. GitHub enterprise lets you manage Codespace creation and maintenance policies across an organization.</p><h3>Project Setup Best-Practices &amp; Utilities</h3><p>In a shared development environment like GitHub Codespaces (or devcontainers), it is important to have consistent tools and dependencies to ensure that all developers are working in the same environment. There are five supporting pillars to make a great, healthy and reliable Python package project. They are <strong>Tests, Linting, Documentation, Hooks and Releases</strong>. Let’s explore each of these briefly.</p><p><strong>Tests: </strong>Any Python project or general software project that’s worth its weight has tests that have good code coverage. This means that different functions have a wide variety of test-cases and these tests span the entire codebase such that different lines of code that are conditionals execute based on each test-case. The ideal set of tests will cover every possible route your program takes. <a href="https://pytest.org/">Pytest</a> is a battle-tested testing library, and you can use <a href="https://coverage.readthedocs.io/">Coverage</a> along with pytest to build a reliable testing suite.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*Xug0By7uNyp5U9t1VHVgIQ.png" /><figcaption>Sample Pytest run with a code coverage report. Image by Author.</figcaption></figure><p><strong>Linting &amp; Formatting: </strong>Linting and formatting are two essential processes for maintaining high-quality Python code. Linting involves analyzing the source code for potential errors, inconsistencies, and style violations, while formatting ensures that the code is written in a consistent, readable, and maintainable style. By performing these processes, developers can detect and fix potential issues early, reduce code review time, and improve code readability and maintainability, ultimately leading to better software quality and developer productivity.</p><p>There are several tools available for linting and formatting Python code, such as <a href="https://pylint.readthedocs.io/">PyLint</a>, <a href="https://flake8.pycqa.org/">Flake8</a>, <a href="https://black.readthedocs.io/">Black</a>, and <a href="https://pypi.org/project/autopep8/">autopep8</a>. PyLint is a widely used static analysis tool that checks for syntax errors, coding standards, and potential bugs. Flake8 combines PyLint with additional tools for checking style and formatting, and it can be easily integrated with most text editors and IDEs. Black is a code formatter that automatically reformats Python code to follow the <a href="https://peps.python.org/pep-0008/">PEP 8</a> style guide, while autopep8 is a tool that automatically applies PEP 8 formatting rules to Python code.</p><p><strong>Documentation: </strong>Documentation is essential for creating maintainable and understandable Python code. One common method for documenting Python code is by using <a href="https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring">function docstrings</a>, which are triple-quoted strings that provide a detailed description of the code’s purpose and how to use it. There are several styles of docstrings, including Google-style, reStructuredText, and NumPy-style, among others. Tools like <a href="https://www.mkdocs.org/">MkDocs</a>, a static site generator, can be used to create visually appealing and easy-to-navigate documentation sites, with the<a href="https://squidfunk.github.io/mkdocs-material/"> MkDocs Material theme</a> being a popular choice. By following a consistent style and using docstrings to document code, developers can ensure that their code is easy to understand and maintain, both for themselves and for others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1K5cIs6rX_sO5GDYqZ7WqA.png" /><figcaption>Sample Documentation page on Mkdocs with Material theme. Image by author.</figcaption></figure><p><strong>Hooks: </strong><a href="https://pre-commit.com/">Pre-commit</a> is a Python package that automates the process of running code checks and tests before committing changes to a version control system like Git. It includes a wide range of hooks for checking code quality, including code formatting, style checks, and static analysis tools like Black, Flake8, and <a href="https://mypy.readthedocs.io/">Mypy</a>. By using pre-commit hooks, developers can ensure that their code meets quality standards and is ready for review before it is merged into the codebase. Pre-commit can be used in conjunction with <a href="https://typicode.github.io/husky/">Husky</a>, a similar tool for Node.js projects, to ensure that the check runs automatically during a commit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*LlD01r6beqNoUZH7i4NSQA.png" /><figcaption>Pre-commit hooks running before a commit is finalized. Image by author.</figcaption></figure><p><strong>Releases &amp; Conventional Commits: </strong><a href="https://www.conventionalcommits.org/en/v1.0.0/">Conventional Commits</a> is a specification that provides a structured way of committing code changes, making it easier to generate changelogs and automate the release process. A changelog is a document that records the changes made to a software project between different versions or releases. It typically includes information about new features, bug fixes, and other changes that may affect how the software behaves or how it should be used. By using Conventional Commits, developers can categorize their code changes into types such as feat for new features, fixfor bug fixes, and chorefor maintenance tasks, among others. This structure makes it easier to understand the scope and purpose of each commit, simplifying the process of generating release notes and creating release cycles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*0E4uoheiy7xwInXz2KJoxA.png" /><figcaption>Adding conventional commits through <a href="https://commitizen-tools.github.io/commitizen/">commitizen</a> (cz). Image by author.</figcaption></figure><p>In the Python ecosystem, Google’s <a href="https://github.com/googleapis/release-please">release-please</a> tool is a popular choice for automating the release process using Conventional Commits. Release-please uses commit messages to generate release notes, update version numbers, and create release branches and tags automatically. By using Conventional Commits and release-please, Python developers can automate the tedious parts of the release process and focus on writing code instead. Additionally, Conventional Commits and release-please can help maintainers of Python packages ensure that their releases are consistent and well-documented, improving the overall quality of the package and its adoption by other developers.</p><h3>Cookiecutter Template with Utilities</h3><p>The open source <a href="https://github.com/dotlas/cookiecutter-pypackage">Dotlas cookiecutter for Python packages</a> comes pre-configured with all of the tools mentioned above to manage your package end-to-end right from the get-go. Give it a spin by running this from your terminal (with cookiecutter installed — pip install cookiecutter):</p><pre>cookiecutter gh:dotlas/cookiecutter-pypackage</pre><p><strong>Continue Reading —</strong><a href="https://eash98.medium.com/create-a-shared-python-dev-environment-for-your-team-part-2-configuring-a-devcontainer-974a83833c0f"><strong> Link to Part 2: <em>Configuring a Devcontainer.</em></strong></a></p><blockquote>Shoutout to <a href="http://kelvindecosta.com/">Kelvin</a> for being an advocate of the best practices mentioned above!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a32d2bbea33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why SQLAlchemy should no longer be your ORM of choice for Python projects]]></title>
            <link>https://eash98.medium.com/why-sqlalchemy-should-no-longer-be-your-orm-of-choice-for-python-projects-b823179fd2fb?source=rss-7c131006709c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b823179fd2fb</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[prisma]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[sqlalchemy]]></category>
            <category><![CDATA[orm]]></category>
            <dc:creator><![CDATA[Eshwaran Venkat]]></dc:creator>
            <pubDate>Fri, 20 Jan 2023 23:24:48 GMT</pubDate>
            <atom:updated>2023-01-20T23:24:48.818Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IWr3yeYZiV51S_hu" /><figcaption>Photo by <a href="https://unsplash.com/@jankolar?utm_source=medium&amp;utm_medium=referral">Jan Antonin Kolar</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>SQLAlchemy is a powerful tool to interface with a backend database, irrespective of whether the goal is to build a frontend web application, administrate a database “as-code”, or something similar. SQLAlchemy has both an <strong>ORM (Object Relational Mapper)</strong> and a core library and is used heavily for python backends to I/O a SQL database. As an ORM, it is also database agnostic meaning that a database can be migrated (for example) from a mySQL backend to a PostgreSQL one without changing the SQLAlchemy models on the app. On the flip side, it provides database-specific dialects to use the full power of a specific SQL database (such as JSONBin Postgres).</p><p>ORMs have been around for a while and they’re great for building basic CRUD applications without writing raw SQL. They’re also the number one choice to avoid SQL injection vulnerabilities. Most applications will not require complex SQL at all — as you’ll mostly be fetching some user data, session data, or the like which can be accomplished with some simple select and filter statements which most ORMs almost always come pre-packaged with, and work pretty seamlessly.</p><p><strong>Popular Python ORMs include:</strong></p><ul><li><a href="https://www.sqlalchemy.org/"><strong>SQLAlchemy</strong></a>: The oldest and most proven ORM for Python. Note that it is a database toolkit and the ORM is a feature.</li><li><a href="https://sqlmodel.tiangolo.com/"><strong>SQLModel</strong></a><strong>:</strong> Built on top of SQLAlchemy &amp; <a href="https://docs.pydantic.dev/">Pydantic</a>. Functionality is largely overlapping with SQLAlchemy and its main purpose is to support the <a href="http://fastapi.tiangolo.com/">FastAPI</a> project as an ORM.</li><li><a href="https://docs.djangoproject.com/en/4.1/topics/db/queries/"><strong>Django ORM</strong></a><strong>:</strong> ORM for Django projects, but can be extended for generic Python projects as well.</li></ul><p><strong>ORMs are considered both necessary by some developers and as an </strong><a href="https://blog.codinghorror.com/object-relational-mapping-is-the-vietnam-of-computer-science/"><strong>anti-pattern by others</strong></a><strong>.</strong> A quote from this <a href="https://stackoverflow.com/questions/404083/is-orm-still-the-vietnam-of-computer-science">StackOverflow post</a> goes:</p><blockquote>In many people’s eyes, SQL is ugly, complex, and confusing. But trying to make an object-oriented interface to perform the same functionality is always uglier, more complex, and has a steeper learning curve.</blockquote><p>Of course, there are productivity gains but you don’t see a third dimension of the “learning curve” mentioned in the quote above to the figure below.</p><figure><img alt="A 2D Coordinate System with Control &amp; Productivity from a SQL database as X and Y axes. ORMs are right in the middle of the graph, and SQL is to the bottom right." src="https://cdn-images-1.medium.com/max/1024/1*cfgPgWix7K6LAM4Wqui-gg.png" /><figcaption>Source: <a href="https://www.prisma.io/docs/concepts/overview/why-prisma#problems-with-sql-orms-and-other-database-tools">Prisma</a></figcaption></figure><p>That is not to say that there aren’t upsides to using ORMs either, which is also something discernible from this figure.</p><ul><li>Engineers deal with a lot of data models on a day-to-day basis, and the shape of the data is prone to change; sometimes often — especially for projects in development or projects that scale up quickly. Managing these in SQL is not ideal, to say the least, and using a GUI SQL Client (like pgAdmin, mySQLWorkbench, DBeaver, etc) makes the process highly manual.</li><li><strong>DDL</strong> (Data Definition Language) of SQL is great but unproductive. Having the data schema “as-code” is a lot easier to track and administer changes instead of “as-SQL”.</li></ul><h3><strong>The Case Against SQLAlchemy</strong></h3><blockquote>No native support for database change management. The syntax is complex compared to modern ORMs. Nitpicks: A very ancient-looking official documentation, and not the best support for introspection.</blockquote><h3>Database Change Management</h3><p>A <strong>database migration</strong> (which may sound like you’re migrating the whole database or moving or backing up your data, but instead) means changing the schema of your data models. For example:</p><ul><li>Adding or removing a column from a table</li><li>Renaming a column or changing its type definition</li><li>Adding or removing a table, and so on.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TYG7dOjITcTrZqLFxr0Enw.png" /><figcaption>Source: <a href="https://www.bytebase.com/blog/what-is-database-change-management">Bytebase</a></figcaption></figure><p>This is topically called <strong>Database Change Management (DCM)</strong> and <a href="https://www.bytebase.com/blog/what-is-database-change-management">Bytebase has a great article detailing the idea</a>. It is honestly a rookie mistake to think that a database schema for any kind of application will remain unchanged over time, and happens more frequently than you may initially assume. This is because a data model is subject to morphing based on where the perceived value of the contained data truly lies. This means that a data model using an ORM is not as straightforward as declaring the models on file in the app and calling it a day.</p><p>SQLAlchemy does not support DCM out of the box. If you declare models and push them to your database to reflect the schema, the database is no longer prone to changes to existing models and only detects new tables. If you want to add a column to the SQLAlchemy models file and expect those changes to take place in the database again, not going to happen. Admins will pretty much have to use raw SQL ALTER statements, manually plugging changes for all edits to those model tables in the future. Take the following snippet for example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Mosl9dfkOv3212yuPS7pA.png" /><figcaption>sample models.py of an app using sqlalchemy for a user table. Image by Author.</figcaption></figure><p>If the model has been reflected in the database already, and now you want to add an attribute called phone_number to the User model, then adding phone_number to this class in the models.py file will not reflect that column in the database.</p><p>This is where you, the reader may say “<strong>Hey, there’s always the Alembic tool to handle migrations</strong>”, and you would be correct. <a href="https://alembic.sqlalchemy.org/en/latest/tutorial.html">Alembic</a> is a tool created by the Author of SQLAlchemy that overcomes this limitation and allows SQLAlchemy migrations. Problem solved? Not quite. In my opinion, Alembic is probably not the best way that DCM can be solved at least in terms of developer productivity.</p><p>Here’s an example of an alembic script to create a migration to add phone_number to the user table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P9l1hyH8mmGSqjRgbtdBsw.png" /><figcaption>Alembic script. Image by Author</figcaption></figure><p>To use this script, you will need to have Alembic and SQLAlchemy installed. You will also need to have a .ini configuration file and a env.py script, which define the connection to the database and provide necessary context.</p><p>To run the script, you can use the alembic upgrade head command, which will apply the changes defined in the upgrade function. To revert the changes, you can use the alembic downgrade base command, which will run the downgrade function.</p><p>Seems a bit much, but gets the job done. One cool thing about Alembic is the time-travel aspect like git where you can rollback as well. Adding these scripts each time a change is made just seems tedious, especially for multiple alterations. However, <strong>I do acknowledge that alembic can be exactly what you may be looking for</strong>.</p><p><a href="https://docs.djangoproject.com/en/4.1/topics/migrations/">Migrations in the Django ORM are similar</a>, with some precise code definitions to get you on your way. The alembic migrations can be made slightly better by utilizing <a href="https://flask-migrate.readthedocs.io/en/latest/">Flask-Migrate</a> , which can deploy migrations easier but requires a flask declaration of the app (even if the app doesn’t need to use flask).</p><h3>Syntax &amp; Docs</h3><p>Okay, let’s take a look at this file again. You may be indifferent to the declarations here citing that “it doesn’t get simpler than this”, or you feel like there are parts of this file syntactically that don’t need to exist explicitly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Mosl9dfkOv3212yuPS7pA.png" /><figcaption>Image by Author</figcaption></figure><p>The SQLModel project does a good job of reducing the explicit syntax by using type annotations with pydantic’s BaseModel as follows: (excluding the __repr__)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*rE6-0RtRLQqvv1ks8j9Crw.png" /><figcaption>Sqlmodel equivalent of user table. Image by Author</figcaption></figure><p>And this might honestly be good enough for most folks. <strong>A combination of SQLModel and Alembic can get your DataOps and Backend development off the ground</strong>. <a href="http://sqlmodel.tiangolo.com/">SQLModel has a great user guide</a> (though no API documentation since it overlaps with pydantic and SQLAlchemy), and looks better than SQLAlchemy’s docs.</p><h3>Introspection</h3><p>The final feature we’re looking at is introspection. Introspection is like “crawling” a database. The program examines the schemas and generates a file, or set of files to reflect the data modeling as objects. Most GUI SQL clients use introspection features. This is especially useful for ORM use cases where a schema already exists. <strong>SQLAlchemy has either some good in-built introspection features or some add-on libraries that can be configured to deliver decent end-to-end introspection</strong>.</p><ul><li><a href="https://docs.sqlalchemy.org/en/14/orm/extensions/automap.html">SQLAlchemy has Automap functionality</a> which tries to map relational objects to object-oriented models. Scripts still have to be developed around the building blocks SQLAlchemy provides, which is not too bad.</li><li><a href="https://github.com/agronholm/sqlacodegen">sqlacodegen</a> is an open-source project that can introspect a database and create SQLAlchemy Table models. It also has support for sqlmodel but this feature is not an official stable release and has to be used from the master branch (as of Jan 2023).</li></ul><h3>Conclusion?</h3><p>Perhaps this whole article is nit-picky but the ORM selection for python consistently seems to be in a state of “almost but not quite”, and most of the places where these apparent holes exist are really in the developer productivity area. <strong>In 2023, an age where tools like ChatGPT exist — it just seems unacceptable to me that dealing with SQL databases needs to involve even close to the iota of effort that any of these tools seem to warrant</strong>. Of course, this may seem like very much a “first world” or “entitlement” problem and that might very well be true.</p><p>Speaking purely from a database administration point of view, tools like <a href="https://www.bytebase.com/">ByteBase</a> and <a href="https://docs.getdbt.com/docs/introduction">DBT</a> seem to do a good job. Google cloud also has a <a href="https://cloud.google.com/architecture/devops/devops-tech-database-change-management">document detailing DCM for production databases</a> by using CI/CD which reduces downtime. Documentation of these schemas also seems to be made quite easy with tools like <a href="https://dbdocs.io/">dbdocs</a>, which certainly seems crucial for managing tables and relations as they scale for SQL Databases (I’m not sponsored or associated with these companies). But speaking from managing a database for small projects or for big projects handled by small teams when using the open-source options: things get quite tedious indeed.</p><h3>Prisma &amp; Prisma Python Client</h3><p>In my hunt for ORMs and Dataops tools, the tools highlighted in the article thus far always appeared to be the recommendations. Even online articles and publications favour the de-facto standard that is apparently being introduced by these libraries in the Python ecosystem.</p><p>So what surprised me was when a couple of recommendations came through to try <a href="https://www.prisma.io/">Prisma</a>, from the folks over in the Typescript / Javascript ecosystem. I’m not sponsored or associated with Prisma, but t<strong>hey do seem to get a solution closest to the ideal ORM in my opinion</strong>. I do want to highlight that the following features address the “almost but not quite” problem in the previous section. This means that some of these features can feel like marginal improvements, but that’s the idea.</p><ul><li>The data models are <strong>syntactically simple</strong> but convey necessary information concisely in a near-JSON-like format.</li><li>Prisma <strong>documentation is fantastic</strong> and has both user guides as well as API references. It also has a neat set of DIYs in terms of ORMs.</li><li>Database migrations and<strong> DCM are easy</strong> as changing the data models in-file and running a migrate CLI command.</li><li><strong>Introspection is straightforward</strong> and has standard syntax instead of separating Table from Class like SQLAlchemy.</li><li>An easy-to-use query builder for the ORM.</li><li>Support for NoSQL document databases like MongoDB, which is more of a nice-to-have from a SQL PoV.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*F7Ts-rbQv6rxKAMNhQ5JkA.png" /><figcaption>Prisma’s data model for the User table. Image by Author.</figcaption></figure><p>One thing that stands out here is that <strong>Prisma does not use object models (like classes of the ORM)</strong> and instead uses a standard schema. This lends to Prisma being less affected by the anti-patterns introduced in using object mappings on relations that the Python ORMs suffer from.</p><p>Adding this model to the database as simple as:</p><pre>prisma migrate dev --name init</pre><p>Adding a new phone number field would involve adding it to the prisma file and then running the following command:</p><pre>prisma migrate dev --name added_phone_number</pre><p>Introspecting an existing schema on the database would be:</p><pre>prisma db pull</pre><p>Which would load the entire database schema into a file as prisma data models.</p><p><strong>But what about the fact that this is a TypeScript ORM I hear you say?</strong> Well, that’s where the <a href="https://prisma-client-py.readthedocs.io/">Prisma Python Client</a> comes in. Simply pip install prisma and include the following in the Prisma schema file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*dyqWsBaubv-o84MMChJ_Rw.png" /></figure><p>You can now query the tables using the Prisma ORM which is similar to the query generators on SQLAlchemy or SQLModel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7rojVQTkukFmOlX5W8-vw.png" /><figcaption>Prisma ORM Python file. Derived from example in <a href="https://prisma-client-py.readthedocs.io/en/stable/#accessing-your-database-with-prisma-client-python">Prisma Python Client</a>. Image by Author.</figcaption></figure><p>Naturally, you can learn about a lot more you can do with Prisma in their <a href="https://www.prisma.io/docs">docs</a>.</p><h3>Complex SQL Queries</h3><p>A point raised about ORMs is how they’re not the tools for complex SQL queries. This is 100% correct, as a complex SQL query is never far away, even as a developer. The best way to manage this issue while still making use of an ORM would probably be to create a <a href="https://www.w3schools.com/sql/sql_view.asp">VIEW</a> or <a href="https://en.wikipedia.org/wiki/Materialized_view">MATERIALIZED VIEW</a> which would serve as a table that does the hard SQL work from where an ORM can query the data easily. Think of this view as reshaping the data or normalizing it using specific SQL operations, and then using an ORM to utilize simple select or filter queries to extract data for the application.</p><h3>Conclusion.</h3><p>I’ve been using Prisma for a while and it really overcomes the shortcomings of ORMs in the standard python ecosystem. The idea is naturally to use it in projects of different shapes and sizes to see how it pans out there too. The simplicity in developer UX appears to be worth it, for now at least.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b823179fd2fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Uber H3 for Data Analysis with Python]]></title>
            <link>https://medium.com/data-science/uber-h3-for-data-analysis-with-python-1e54acdcc908?source=rss-7c131006709c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e54acdcc908</guid>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[uber]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Eshwaran Venkat]]></dc:creator>
            <pubDate>Mon, 11 Jan 2021 19:58:55 GMT</pubDate>
            <atom:updated>2023-03-02T14:24:37.135Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Edit 2023-March</strong>: Since the time of writing this article (H3 v3), H3 v4 has released which has different function names from the ones mentioned in this article. You can install v3 to follow the article, or install v4 and view the <a href="https://github.com/uber/h3/blob/master/CHANGELOG.md#400---2022-08-23">changelog</a> and <a href="https://medium.com/foursquare-direct/introducing-h3-version-4-0-0-c60eb2fffaaa">related article</a>, and make the changes as they appear in this article.</blockquote><p>If you’ve ever played <a href="https://en.wikipedia.org/wiki/Civilization_VI">Civilization VI </a>or <a href="https://en.wikipedia.org/wiki/Catan">Settlers of Catan</a>, you might have noticed that the board map is shaped into regular hexagons. This is a famed game design technique called <a href="https://en.wikipedia.org/wiki/Hex_map">Hex Map</a> and is used in board games such as Hexagonal chess and video games such as Forge of Empires. This is a natural feature of the game because these maps bring in a lot of advantages for gameplay, especially in movement and tiling.</p><p>What if this design could prove useful in the real world if we re-modelled our own maps with a layer of hexagons? Perhaps we can find some actual business problems that could be solved with such a model. Enter Uber H3.</p><h3>Why H3 and Why Hexagons? 🌏</h3><p>Currently there exist a number of ways to analyze data related to maps. One of the most rudimentary ways is to plot a list of co-ordinates and see where the densities and clusters arise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*apKGP_8Ql68R4gYFCLovsQ.png" /><figcaption><a href="http://data.bayanat.ae/en_GB/dataset/dubai-civil-defence-fire-stations/resource/c877382c-a545-482a-9cf4-72f8bb033542">Fire Station Locations in Dubai, United Arab Emirates</a></figcaption></figure><p>Analyzing individual points has certain drawbacks. There is no clear form of classification for points within a certain area and the analysis we often wish to carry out may be area-wise or regional.</p><p>It would help if we had a grid system where points could be contained within a certain fence and <strong>we analyze</strong> <strong>the behavior of the grid itself as representing the behavior of all points underneath</strong>. This means each grid represents the data of the points beneath it. Another issue is that when using points directly, while useful for humans to see the map and notice where the data is bunching up may be computationally a bit more expensive for a computer system. A computer needs to calculate distances of neighboring points to group them together to gather the same visual insights as a human being.</p><p>However, coordinate points are as granular as you can go with respect to geolocation data so don’t dismiss them just yet. <strong>We simply need to find a grid framework around which we, or a computer can view geospatial data and derive insights from it.</strong></p><p>A common form of using a grid system more locally is to define zones in cities, by drawing out a shape over a neighborhood, city, district or country. Administrative and political boundaries while useful to section the geospatial analysis into realistic areas may still be problematic in a number of ways. Some zones may be larger than other zones within a city, and there could be overlaps between two or more zones. Furthermore, <strong>from a mathematical/absolute perspective, defining zones in this form are highly arbitrary</strong>. <strong>Zones or districts may be defined based on natural features, population, economy, etc. and are subject to change over time.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cLVcDa5sl9Rn2d2I9fteiw.png" /><figcaption>Dubai, UAE zones as defined in <a href="https://gadm.org/download_country_v3.html">Level 3 Layer of the UAE GeoPackage of GADM Academic Database</a></figcaption></figure><p>An absolute method of using uniform grid systems is to cover a the Earth with repeated tiling. Assuming that we want to tile our plane regularly and completely, <strong>we need to choose the shape that acts as a building block facilitating the complete tiling</strong>. <strong>This shape needs to be closed and uniformly repeating</strong>. Our candidates are the triangle (3 sides), square (4 sides), and hexagon (6 sides).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/0*nwGf5CqFhS-N3fTq.gif" /><figcaption><a href="https://plus.maths.org/content/trouble-five">https://plus.maths.org/content/trouble-five</a></figcaption></figure><p>There is no right answer here, so choosing one of these depends on your use-case. At Uber, one of the most in-demand use-cases we have is to determine distances for both rides and deliveries.</p><blockquote>Using a hexagon as the cell shape is critical for H3.<a href="http://eng.uber.com/h3"> <strong>Hexagons have only one distance between a hexagon’s center-point and its neighbors</strong></a><strong>’</strong>, compared to two distances for squares or three distances for triangles. This property greatly simplifies performing analysis and smoothing over gradients.</blockquote><p>Besides distances, it is generally a good idea to choose a hexagon as the base shape because <a href="https://www.youtube.com/watch?v=thOifuHs6eY">Hexagons are the Bestagons!</a> ;)</p><h3>About H3 ⚫️</h3><p>H3 is an open source framework developed by Uber in the C Programming Language. At its core, H3 is a geospatial analysis tool that provides a hexagonal, hierarchical spatial index to gain insights from large geospatial datasets. <strong>The building blocks of H3 are different sized regular hexagonal polygons.</strong> These polygons are spread out over the entire projection of the earth map from pole to pole. This means that any location on the planet can be attributed to a H3 Hexagon down to a precision of 0.0000009 km² area.</p><p>Imagine it as a layer over the planet where each unit of the layer is a hexagon and each hexagon has a unique ID and can very quickly perform geospatial calculations. Each H3 hexagon can be thought of as its own object and each object can be accessed in a very short amount of time given its ID.</p><h3>H3 Resolutions 📷</h3><p><strong>A core strength of H3 is that it covers the entire world with different sized hexagons</strong>. This means that the resolution of the layer can be adjusted based on the problem being solved, like scaling the entire grid up and down. H3 contains a total of 16 resolutions as described in the table below, and each resolution has a certain number of hexagons that span the entire earth as a layer ranging from 122 hexagons in the highest layer and about 500 trillion hexagons at the lowest layer. Each layer consists of a more granular level of hexagons and each hexagon of every layer has its own unique ID.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*2pK-bAH_y45Y7pZ7vgEJtQ.png" /><figcaption><a href="https://h3geo.org/docs/core-library/restable">H3 Resolutions</a></figcaption></figure><p>H3 defines its indices via hexadecimal format (16-bits, and therefore 16 resolutions), so a cell resolution can be immediately determined by looking at the ID of the cell. <strong>The layered approach of different sized hexagons is what lends H3 its power of “hierarchy”</strong>. Every low resolution hexagon contains a set of child hexagons in higher resolutions. Every hexagon of a resolution can have sibling hexagons that share the same set of parent hexagons. The layers in essence define a tree of hexagons with the last layer (resolution) containing 500 trillion siblings.</p><p>It’s worth checking out this <a href="https://observablehq.com/@four43/h3-index-visualizer">notebook</a>. The resolution can be adjusted accordingly and the hexagons can be viewed over the entire earth.</p><h3>H3 Functions 🔩</h3><p>You can find some examples of using H3 functions directly in the C language <a href="https://github.com/uber/h3/tree/master/examples">here</a>. We’ll be using the H3 library binding with the Python programming language since it relatively easier to analyze data with Python.</p><pre>pip install h3</pre><p>Let’s say you had a coordinate point, or even a list of coordinates. You can fetch the H3 index of each point with the following function:</p><pre>h3.<strong>geo_to_h3(<br>    </strong><em>lat</em>=25.32,<br>    <em>lng</em>=55.46,<br>    <em>resolution</em>=7<strong><br>)</strong></pre><p>Get the resolution of a H3 index if the index represented in a string is valid</p><pre>h3_result = lambda id_str: h3.<strong>h3_get_resolution(</strong>id_str<strong>)</strong> if h3.<strong>h3_is_valid(</strong>id_str<strong>)</strong> else None</pre><pre>h3_result(&#39;8843a13687fffff&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wxCzTbx0ZQY0Cs-4EcudaQ.png" /><figcaption>The H3 index from the snippet above visualized with <a href="https://kepler.gl/">Kepler.GL</a></figcaption></figure><p>Let’s try a full fledged function that returns a set of H3 attributes, given a H3 index</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/803152de60004a0c10d5d06f2beb3e1d/href">https://medium.com/media/803152de60004a0c10d5d06f2beb3e1d/href</a></iframe><p>Another useful function in H3 is its ability to quickly return an index’s k-nearest neighbors. This means that for k =1, a hexagon’s 1st degree neighbors are returned, and for k = 2, it is second degree neighbors (or neighbors of neighbors) and so on.</p><pre>h3_id = &quot;<em>8843a13687fffff</em>&quot;<br>h3.<strong>k_ring(</strong>h3_id,1<strong>)</strong><br>h3.<strong>k_ring(</strong>h3_id,2<strong>)</strong><br>h3.<strong>k_ring(</strong>h3_id,10<strong>)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FlKCnqnlYYCA-RExjg6SDA.jpeg" /><figcaption>K-ring for 1 and 2 from Hex ID: 8843a13687fffff</figcaption></figure><h3>Optional Sprinkles: Populating Hexagons with Maps Data 📍</h3><p>Now that we have some absolute methods to define grids over the Earth, it would suit us to begin assigning real-world details to hexagons. This is because despite having been given a H3 index and its centroid coordinate, we don’t have much to go on in terms of the physical location which that hexagon represents.</p><p>One way to extend the power of H3 is to combine it together with Maps APIs that contain on-the-ground information. Common choices include <a href="https://developers.google.com/maps/documentation">Google Maps API</a>, <a href="https://docs.mapbox.com/api/overview/">Mapbox API</a> and <a href="https://developer.mapquest.com/documentation/open/nominatim-search/">Nominatim</a>. These services allow us to assign more useful geographical information per hexagon.</p><p>One common function is <em>reverse geocoding,</em> which is a technique of converting a given coordinate into a physical text address by means of a lookup. Let’s find the reverse geocode result of the centroid of a H3 index using the <a href="https://github.com/googlemaps/google-maps-services-python/blob/master/googlemaps/geocoding.py">Google Maps Client for Python</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13522604e677d03f1e5a59c8c396163b/href">https://medium.com/media/13522604e677d03f1e5a59c8c396163b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*7NJMH1iXUAt3uhRnNq1ubw.png" /><figcaption>Reverse Geocoded Hexagon — Tooltip contains Geo-Information</figcaption></figure><h3>Polyfill 🌐</h3><p><strong>If H3 contains different resolution hexagons that span the entire earth, what if you wanted to select a fraction of those hexagons that represented a country, city or neighborhood?</strong></p><p>The <a href="https://h3geo.org/docs/api/regions">poly-fill function</a> fills a polygon with H3 Hexagons. A geofence is a polygon spread out over a map. Zones in cities, or country shapes can be modelled as geofences. Geofences are often represented by means of <a href="https://en.wikipedia.org/wiki/GeoJSON">Geo-JSON</a> files or <a href="https://stackoverflow.com/questions/30457089/how-to-create-a-shapely-polygon-from-a-list-of-shapely-points">Shapely polygons</a>. An example of a shapely polygon’s WKT(well known text representation):</p><pre><strong>POLYGON </strong>((<em>55.13977696520102 25.09805053895709</em>, 55.14002932545401 25.09743871100549, 55.1407574981263 25.0972787358399, 55.14123332293791 25.09773058763684, 55.14098097073993 25.09834242241949, 55.14025278567518 25.09850239857415, <em>55.13977696520102 25.09805053895709</em>))</pre><p>A WKT is, as the name implies, a textual representation of the polygon’s vertex points. Notice how the first set of coordinates and last set of coordinates in the polygon are the same (as highlighted in the code block). This is a closed polygonal figure.</p><p>Every “Point” object of a polygon can be of the form of (lat, lng) or (lng, lat). The latter is usually a Geo-JSON representation of the polygon, since the Geo-JSON specification requires longitudes to appear first in the Point object.</p><p>The shapely polygon object needs to be extracted from the WKT, and the WKT is an attribute of the shapely polygon as demonstrated below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f17455455263a88525513e2d701fca7e/href">https://medium.com/media/f17455455263a88525513e2d701fca7e/href</a></iframe><p>The <em>loads </em>function parses out and builds a shapely polygon from its WKT representation.</p><p>I’ve taken a H3 Hexagon’s own WKT here, but you can get a WKT of pretty much any polygon. Another point to note is that the shape has no reference to a map. This is because we need a base map to plot our polygons on, as a layer.</p><p>Let’s take a geofence of an entire city, say Dubai, UAE and see what the Multi-polygon object looks like. I’ll be using the <a href="https://gadm.org/download_country_v3.html">GADM </a>database to extract the UAE country’s Geo-Package. In this way, we’ll be able to view an entire city/district as a geometrical shape. We’ll process the Geo-Package and output the Multi-Polygon for sector 3, Dubai as shown below:</p><blockquote>It is a “multi-polygon” because it contains a list of polygons, as shown in the world islands off the shore of Dubai where each island is its own polygon.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/86737eb889e6cc399b204d0194be33da/href">https://medium.com/media/86737eb889e6cc399b204d0194be33da/href</a></iframe><p>Below is an example of a Geo-JSON file, which can be another useful form to represent geofences. Note again how the first and last coordinates of the list are the same.</p><pre>{<br>  &quot;type&quot;: <strong>&quot;FeatureCollection&quot;</strong>,<br>  &quot;features&quot;: [<br>    {<br>      &quot;type&quot;: <strong>&quot;Feature&quot;</strong>,<br>      <strong>&quot;properties&quot;</strong>: {},<br>      &quot;geometry&quot;: {<br>        &quot;type&quot;: <strong>&quot;Polygon&quot;</strong>,<br>        &quot;coordinates&quot;: [<br>          [<br>            [<br>            <em>  55.07652282714844,<br>              25.11731056144692</em><br>            ],<br>            [<br>              55.15205383300781,<br>              25.055745117015316<br>            ],<br>            [<br>              55.223464965820305,<br>              25.112958466940725<br>            ],<br>            [<br>              55.15068054199219,<br>              25.18070920440447<br>            ],<br>            [<br>              <em>55.07652282714844,<br>              25.11731056144692</em><br>            ]<br>          ]<br>        ]<br>      }<br>    }<br>  ]<br>}</pre><p>A Geo-JSON file is a standard JSON file with special key names that help certain frameworks understand the nature of the data being passed. The Geo-JSON specification defines a list of <em>Features </em>called a<em> Feature Collection, </em>and in this case, the feature is of type <em>Polygon</em>. The <em>properties</em> dictionary can be populated with custom data and may be useful to provide <em>attributes</em> for different <em>feature IDs.</em> Say for example, if you wanted to store the population per polygon of a city, then <em>properties </em>is where you can save that information. It is an extension of your polygon that allows each polygon/feature to carry relevant information specific to itself for later analysis.</p><p>You can use <a href="https://geojson.io/">geojson.io</a>, or any GIS tool/software to create custom geo-fences by drawing out your own shapes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BzKka8cXsluMAo9hRu4HkQ.png" /></figure><blockquote>Defining zones allow us to construct <a href="https://en.wikipedia.org/wiki/Choropleth_map">choropleth maps</a> (maps with zonal coloring). The Plotly graphing libary has <a href="https://plotly.com/python/choropleth-maps/">native choropleth support</a> using Geo-JSON files. Kepler natively supports H3 visualizations by using just the indices of H3 hexagons from the data.</blockquote><p>Now we know what we need to deal with the elephant in the room. How can we reconcile geofences with H3? We poly-fill the entire boundary with specific resolution hexagons (resolution 10 in our case). H3 has capabilities to poly-fill both Shapely Polygons and Geo-JSON objects.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/721ade2d5890111e87e9e1d304ca635e/href">https://medium.com/media/721ade2d5890111e87e9e1d304ca635e/href</a></iframe><h3>Polyfill Gallery 💠</h3><p>The output of this data-frame visualized from Uber’s Kepler.GL below with different resolutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U0YUeyj9S1jSvCfWerF3iA.png" /><figcaption>Close-Up of Resolution 10 Hexagons</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8JzhbBGCRSk70KbknAEHTg.png" /><figcaption>Dubai Poly-filled with Resolution 10 Hexagons (100,000 hexagons spanning the city)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kGJzKXoxyJMvL07NZNXpNQ.png" /><figcaption>Dubai Poly-filled with Resolution 8 Hexagons ( 2,500 hexagons spanning the city)</figcaption></figure><p>Looks like a retro <em>16-bit</em> city to me (ba dum tss).</p><h3>Analysis 📈</h3><p>Now that we have our hexagons populated across a city, and we have each hexagon’s ID stored in a table/data-frame,</p><ol><li>We can perform any H3 related function on each hexagon (Find its parent for a more aggregate analysis, store its geo-fence, find its neighboring hexes, etc)</li><li>We can track coordinate level data grouped by their respective hexagons. In this case, hexagons become buckets which we can use to perform grouped analysis on a set of coordinate points per bucket/hex</li></ol><p>Some non-exhaustive ways to fully utilize H3 would be:</p><ol><li>Using it as an “on/off” indicator to highlight activity in a particular region. Remember that H3 hexagons can be of different resolutions so you can go as granular as you want or aggregate it upwards to lower resolutions.</li><li>View time series data by analyzing the evolution of hexagons over a time frame. Each hexagon is representative of the neighborhood of a number of coordinates that reside within its boundaries.</li><li>Clustering areas with specific levels of activity.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TaEQ9l8m_6TJWq1VLR1MKQ.png" /></figure><p>The rest is up to you ;)</p><p>Thanks for reading!</p><p><strong>A few more links:</strong></p><p><a href="https://www.youtube.com/watch?v=wDuKeUkNLkQ">Uber Open Source: Engineering Sub-City Geos for a Hyper-Local Marketplace with Uber</a></p><p>Veritasium’s <a href="https://www.youtube.com/watch?v=48sCx-wBs34">“The Infinite Pattern That Never Repeats”</a></p><p><strong><em>References:</em></strong></p><p><a href="https://eng.uber.com/h3/">Uber Engineering Blog: H3: Uber’s Hexagonal Hierarchical Spatial Index</a></p><p><a href="https://h3geo.org/docs">H3 Docs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e54acdcc908" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/uber-h3-for-data-analysis-with-python-1e54acdcc908">Uber H3 for Data Analysis with Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Analytics in Tableau]]></title>
            <link>https://medium.com/data-science/automating-analytics-in-tableau-5516e1e7134c?source=rss-7c131006709c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5516e1e7134c</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[tabcmd]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Eshwaran Venkat]]></dc:creator>
            <pubDate>Mon, 18 May 2020 16:47:28 GMT</pubDate>
            <atom:updated>2020-05-26T17:59:18.653Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automating Analytics in Tableau — Python &amp; Tabcmd</h3><h4>An overview with demos on using Python to automate insights on Tableau with a look at Tabcmd, the Tableau command line utility.</h4><h3><strong>Introduction</strong></h3><p>Tableau has been recognized as a leading tool in Analytics and Business Intelligence Platforms for seven consecutive years by <a href="https://www.tableau.com/about/blog/2019/2/tableau-7-years-leader-2019-gartner-magic-quadrant-101644">Gartner</a>. BI/Data teams in companies of all shapes and sizes in nearly every industrial sector develop dashboards to effectively communicate key business metrics across the organization.</p><p>One major advantage of using Tableau (which is often overlooked) is that it exposes a public API to programmatically handle nearly all tasks that can be carried out over the application GUI — Including administration.</p><p>This is the part where Python comes in. The scripting capability and support from a vast number of libraries makes it a popular choice to handle end-to-end process automation in BI workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8T5vuoCY1nEeou7aRMfUyA.png" /><figcaption>From <a href="http://tableau.com/products/viewer">tableau.com/products/viewer</a></figcaption></figure><blockquote><strong>Before You Proceed…</strong></blockquote><blockquote><em>To effectively use the integrations, the Tableau workbook(s) should be available on one of the following:</em></blockquote><blockquote>1. <a href="https://www.tableau.com/products/server"><strong>Tableau Server</strong></a><strong> <br></strong>Lets you host workbooks and data sources in-house on your organisational servers or using cloud service providers.</blockquote><blockquote><strong>2. </strong><a href="https://www.tableau.com/products/cloud-bi"><strong>Tableau Online</strong></a><strong><br></strong>Tableau manages the server infrastructure and you can host your workbooks through them at some additional cost.</blockquote><blockquote><em>It’s often good practice to have workbooks hosted on a Tableau server through one of the above mentioned methods — especially if your dashboards are viewed frequently by multiple parties and involve periodic data refreshes (Daily/Hourly).</em></blockquote><blockquote><strong>Tableau Online</strong> has a 14 day free-trial with some sample workbooks loaded in by default. You can <a href="https://www.tableau.com/products/online/request-trial">create a free trial account</a> to try out some of the test integrations and code snippets listed below.</blockquote><p>Upon creation of a Tableau Online account, save the credentials for <em>username, password, tableau server address</em> and <em>site name</em>. A single tableau server can have multiple sites and each of these sites hosts a collection of workbooks and datasources.</p><h3>All Roads Lead to Rome</h3><p>There are a number of ways to integrate Python with Tableau Server, each based off a different use-case.</p><ol><li><strong><em>Tableau Server Client (TSC)</em></strong> library <br>(Python 3.5 or later)</li></ol><p>Probably the best option out there for performing server related tasks from a Python script. It’s an open source project managed by Tableau that wraps around the <strong><em>Tableau Server REST API.</em></strong></p><pre>pip install tableauserverclient</pre><p>2.<strong><em> TabPy</em></strong> library<br>Allows you to embed and execute Python scripts within Tableau’s table calculations. It essentially allows Tableau to remotely execute Python code.</p><pre>pip install tabpy</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LOYgitIXTVZ1hbKFy-6HyQ.png" /><figcaption>From this <a href="https://www.tableau.com/about/blog/2017/1/building-advanced-analytics-applications-tabpy-64916">TabPy blog</a> — embed Python within Tableau calculations</figcaption></figure><p>3. Using <strong><em>Requests </em></strong>library to directly consume the <em>Tableau Server REST API</em><strong> .<br></strong>A more bare-bones approach but you can make Tableau work in any way you deem fit as allowed by the API.</p><p><strong>Others integration routes and variants include:</strong></p><p>1. <strong><em>JupyTab</em></strong></p><p>Brings Jupyter Notebook data into Tableau visualizations.</p><pre>pip install jupytab</pre><p>2. <strong><em>Tabcmd</em></strong> <br>Issue commands to Tableau Server from the Windows Shell — Unsupported for Mac/Linux natively, but there is a <a href="https://community.tableau.com/thread/118422">hack</a>.</p><p><a href="https://help.tableau.com/current/online/en-us/tabcmd.htm"><em>Download Tabcmd for Tableau Server</em></a></p><p>Tabcmd executions can be primarily automated by setting them up inside a <a href="https://www.windowscentral.com/how-create-and-run-batch-file-windows-10">batch file</a> under a <a href="https://www.windowscentral.com/how-create-automated-task-using-task-scheduler-windows-10">scheduled task</a>. Add Tabcmd to your windows PATH variable or switch over to the directory where it&#39;s installed to run its commands.</p><blockquote>You can execute tabcmd commands from within Python source code by using the <strong><em>os</em></strong> or <strong>subprocess</strong> modules. Construct tabcmd statements as strings and pass them into the shell using <strong><em>os.system()</em></strong>or<strong><em>subprocess.call()</em></strong>functions</blockquote><p>You can find more projects with support for other languages (such as JavaScript) in <a href="https://www.github.com/tableau">Tableau’s GitHub Page</a></p><p><strong>We’ll focus on the usage of TSC and Tabcmd here</strong>.</p><p>The dummy variables listed below will be assumed to be accessible for each TSC snippet. Substitute these with your own credentials.</p><pre>tableau_user = &#39;cricks@gmail.com&#39;<br>tableau_passwd = &#39;123&#39;<br>tableau_server = &#39;http://172.16.22.2/&#39;<br>tableau_server_site = &#39;my_site&#39;<br>server_version = &#39;3.7&#39;</pre><p>For Tableau Online, the tableau_server field would look something like this: <br>prod-apnortheast-a.online.tableau.com based on your server location.</p><p>Each of the following sections details how to perform a particular task on the Tableau Server using both TSC and Tabcmd.</p><h3><strong>Authentication</strong></h3><p><strong>TSC</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9db9faa5fbf1456c4b7a11e20a53d751/href">https://medium.com/media/9db9faa5fbf1456c4b7a11e20a53d751/href</a></iframe><p><strong>Tabcmd</strong></p><pre>&gt; tabcmd login -s http://172.16.22.2/ -t my_site -u cricks@gmail.com -p 123</pre><p>Each parameter of the credential is passed as a flag in the shell.</p><p>The output should look like this:</p><pre>===== redirecting to http://172.16.22.2/auth<br>===== Signed out<br>===== Creating new session<br>=====     Server:   http://172.16.22.2/<br>=====     Username: cricks@gmail.com<br>=====     Site:     my_site<br>===== Connecting to the server...<br>===== Signing in...<br>===== Succeeded</pre><blockquote>In the event of a failed authentication in either routes, check your user privileges/user authorization with a Tableau Server admin.</blockquote><h3>Fetch &amp; Download a Workbook</h3><blockquote><em>The ‘Superstore’ workbook is available as a sample in all Tableau distributions. It will be the default workbook in the demonstrations for our code snippets.</em></blockquote><p><strong>Finding the Workbook &amp; Downloading it as a </strong><strong>.twbx file</strong></p><p><strong>TSC</strong><br>A workbook can be found by using<br> — TSC.Pager()<br> — TSC.RequestOptions() with a TSC.Filter() on workbooks.get().</p><p>TSC returns a workbook object with attributes such as workbook ID, workbook name, etc. Each of these options is detailed as a function in the following gist.</p><p>A workbook can be downloaded as twbx (Tableau packaged workbook) with<br>- TSC.Server().workbooks.download(workbook_id)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea2571b243e7ad775b68670d3492ccc5/href">https://medium.com/media/ea2571b243e7ad775b68670d3492ccc5/href</a></iframe><p>You can retrieve the workbook’s ID on the server by referencing the id attribute on the workbook object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jVPG8Sv2b3BHEMepXnEszA.png" /><figcaption>Product View of the Superstore Workbook</figcaption></figure><p><strong>Tabcmd</strong></p><p>Tabcmd keeping it short and simple:</p><p>tabcmd get “/workbooks/Superstore.twbx”</p><blockquote><em>There could be a couple of caveats especially with permissions for the </em><em>get command in Tabcmd. Make sure to go through the full checklist </em><a href="https://help.tableau.com/current/server/en-us/tabcmd_cmd.htm#id7e0a0627-ad89-4135-a1c2-85b1d8472568"><em>here</em></a><em>. You can use </em><em>get to download the workbook as a PDF or PNG as well.</em></blockquote><h3>Apply a Filter to a View and Download as a File</h3><p>We can query a particular view within the workbook and apply a filter to it.</p><p><em>How it works:</em><br>We issue a request for a view with filters as parameters. The API applies the request to the view and fetches a snapshot back to the requesting client.</p><p><strong>Before Filter:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x0LcfOfcXnE4P_zBKXgspg.png" /><figcaption>Superstore — Overview with all regions</figcaption></figure><p><strong>After Filter</strong> for Region = West:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VL0MPjxgH5DB0Oq8S8MY3A.png" /><figcaption>Superstore — Overview with Region field filtered to one value</figcaption></figure><p><strong>TSC</strong></p><p>Assuming workbook id can be fetched, we’ll use it as a dummy variable.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ddd492b701eb70b17b15e1ef5c06f40/href">https://medium.com/media/2ddd492b701eb70b17b15e1ef5c06f40/href</a></iframe><p>view_items contains a list of view objects. A view object has attributes such as id, workbook_id, name, csv. A personal favorite is the total_views attribute which returns the total number of times that the currently referenced view has been seen by users on the Server.</p><p>We are required to “populate” views as PDFs because it allows TSC to fetch only necessary attributes from over the web. The workbook or view object contains a reference to all other attributes of that object on the server. When one of these objects needs to be populated, TSC makes the request fetch the required attributes to satisfy the populate params.</p><p>Images can be saved as PNG with ImageRequestOptions() and CSVs can be saved with CSVRequestOptions().</p><p>See the docs for their usage <a href="https://tableau.github.io/server-client-python/docs/api-ref#requests">here</a>.</p><p><strong>Tabcmd</strong></p><p>The export command takes in the URL of the specified view and optional parameters for download options.</p><pre>tabcmd export -t my_site &quot;Superstore/Overview?refresh=yes&amp;Region=West&quot; --pdf --pagesize a4 --pagelayout landscape -f &quot;Overview.pdf&quot;</pre><p>The view can be exported as a PNG or CSV too.</p><blockquote><em>The URI specified within quotes is exactly the same as the sub-string of the URL used to access the view on Tableau Server over a browser.</em></blockquote><p>Here’s what the URL on the browser would be for the same command:</p><pre>http://172.16.22.2/#/site/my_site/views/Superstore/Overview?Region=West</pre><p>This helps experiment with what a view would look like on the browser before using Tabcmd to export it locally.</p><blockquote>&#39;refresh=yes&#39; <em>forces a server side refresh on the view and fetches the most up-to-date results.</em></blockquote><blockquote><em>Apply </em>multiple filters<em> by separating them out in the params with an</em> &#39;&amp;&#39;</blockquote><blockquote><em>Dates can be filtered as: ‘</em>..?report_date=2020–01–01&#39;</blockquote><h3>Data Sources &amp; Connections</h3><p>One of Tableau’s core features as a software has been its ability to allow users to handle the data sources, extracts and connections of data that feeds into the visualizations.</p><p><strong>TSC</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9104e37cc087fbcd16140ef82f9fc310/href">https://medium.com/media/9104e37cc087fbcd16140ef82f9fc310/href</a></iframe><p>the datasource object can be filtered or paged through as well, similar to workbook or view objects.</p><p><strong>Tabcmd</strong></p><pre>tabcmd refreshextracts --datasource sales_ds</pre><h3>TSC vs Tabcmd</h3><blockquote>Both TSC and Tabcmd allow you to perform crucial CRUD operations on Data Sources, Connections, Workbooks, Views, Users, etc.</blockquote><p>While TSC is natively OS agnostic, Tabcmd is still sufficient to carry out most day-to-day tasks on the server, provided you have a windows system or are able to configure it to run manually on Mac/Linux.</p><p>One particular use-case where TSC outbids Tabcmd is its ability to handle a large number of requests to the server <em>in parallel</em>. With Tabcmd, requests are always serialized per shell session. TSC can be multithreaded to tens or hundreds of workers providing a massive boost in the performance for its server requests. On Python, you can implement the ThreadPoolExecutor() with a map() on a list to a function that uses TSC and have it work to your advantage.</p><p>This could potentially be useful in kick-starting fast and efficient BI reporting. Bringing this together with <a href="https://towardsdatascience.com/replacing-sheets-with-python-f1608e58d2ca">Python for Google Sheets/Excel</a> or <a href="https://towardsdatascience.com/pivoting-your-data-using-python-sql-or-spreadsheets-62736ba23921">Python for SQL</a> can supercharge your workflow.</p><p>Tableau has rich documentation on using both TSC and Tabcmd. The snippets in this article scratch the surface of the multitude of actions that can be performed with them on a Tableau Server. Check them out in detail below.</p><blockquote><a href="https://tableau.github.io/server-client-python/#">TableauServerClient Docs</a></blockquote><blockquote><a href="https://help.tableau.com/current/server/en-us/tabcmd_cmd.htm">Tabcmd Docs</a></blockquote><h3>Other References</h3><ol><li><a href="https://visualbi.com/blogs/tableau/automate-admin-tasks-tabcmd/">VisualBI blog on basic Tabcmd</a></li><li><a href="https://www.youtube.com/watch?v=sqrTUUay8uk">Take command with Tabcmd</a></li><li><a href="https://www.youtube.com/watch?v=NHomg6M9RcA">Tableau Server Client and REST API demo</a></li></ol><p>Hopefully, this was a gentle enough introduction to get you on your feet and to get started with these integrations.</p><p>Cheers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5516e1e7134c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/automating-analytics-in-tableau-5516e1e7134c">Automating Analytics in Tableau</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>