<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alex Volok on Medium]]></title>
        <description><![CDATA[Stories by Alex Volok on Medium]]></description>
        <link>https://medium.com/@alexandr.volok?source=rss-cb832394c0a0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0FLp_0vFq4nxtG9QbV36_w.png</url>
            <title>Stories by Alex Volok on Medium</title>
            <link>https://medium.com/@alexandr.volok?source=rss-cb832394c0a0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:37:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alexandr.volok/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Python Data Engineering: 10 Steps From a Slow Team to a Delivering One]]></title>
            <link>https://blog.dataengineerthings.org/python-data-engineering-10-steps-from-a-slow-team-to-a-delivering-one-315694e77dc5?source=rss-cb832394c0a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/315694e77dc5</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Alex Volok]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 13:31:04 GMT</pubDate>
            <atom:updated>2026-07-13T14:13:01.156Z</atom:updated>
            <content:encoded><![CDATA[<h4>Pre-commit hooks, trunk-based development, ephemeral environments, and other practices the team fully controls</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wq3zThvpP4vBdOsmQUVi4A.png" /><figcaption>From a slow team to a delivering one, in ten steps. Image by the author.</figcaption></figure><p>As a freelance data engineer, I must switch teams occasionally — more often than an average full-time employee. Frequent switching can sometimes be painful and require an extra cognitive load. Still, it also allows me to see recurring trends, common and shared team goals, and, most importantly, blockers on the way to achieving them.</p><h3>The common goal</h3><p>Nearly every data team I joined had a common goal — it must deliver. Delivery is something that is expected by the business and brings added value. Ironically, the delivery pressure is one of the reasons why I joined the industry as a freelancer — to help clients by adding my capacity and expertise quickly.</p><p>Even though every team must deliver, their delivery velocities vary significantly.</p><h3>The team’s delivery velocity</h3><p>Data teams in different companies have different processes, routines, gatekeepers, constraints, and rituals.</p><p>In my experience, some clients have teams where the average developer is expected to implement, test, document, and deploy a minor two-point story feature to production within a few hours in the morning.</p><p>In contrast, on the other side of the spectrum, I have seen clients where the team might spend a whole day just preparing the development environment and then wait till next week’s Thursday evening to deliver that minor feature to the stakeholders.</p><p>Many factors might affect team velocity, and some of them might even be outside of the control of the data team, such as:</p><ul><li>Cross-team constraints.</li><li>Blocking security measures (every change must be formally approved).</li><li>Poor development processes.</li><li>Blame culture.</li></ul><p>If such negative factors are present, they must be recognized by the business leaders and fixed individually. A single developer, or even a team, cannot always resolve them, but this might trigger a discussion.</p><p>However, there is one more factor that, in my opinion, might have the most significant impact. And it is <strong>empowering developers to contribute independently</strong>. The good news is that the developer teams fully control it.</p><p>This factor reduces in-team constraints and ambiguity. Eventually, it enables developers to deliver faster. The team will focus more on adding business value, while automation handles routine steps. This happens because the tooling often brings the industry’s best practices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*17C2ILe1Qy4Nk6PcH7sH0Q.png" /><figcaption>Constant coordination. Image by the author.</figcaption></figure><h3>What does an empowered team look like?</h3><p>Before I go any further, let’s define the baseline.</p><p>Imagine a typical modern data team of 4 to 6 developers. It uses Python, Apache Spark, Kafka, Terraform, and GitHub to build a data platform and shares the following characteristics and wishes:</p><ul><li>Each developer might work on a few user stories in the current sprint, so the team can potentially deliver 5–15 changes to the product in the coming weeks.</li><li>Developers want to ensure that their changes will not break existing logic, so they rely heavily on testing. The new feature means new tests.</li><li>Developers want to run end-to-end testing in the development environment without affecting other developers. They want a fresh and isolated copy of the production environment with sample data pre-deployed.</li><li>Developers do not want to wait for other developers when they are ready to deliver changes to production.</li><li>Developers might have coding style preferences and habits, but the team would rather have tools like Black or Ruff automatically enforce commonly accepted standards on each commit.</li><li>The team wants to maintain documentation as code and refresh feature-specific pages when the related PR is merged into the main branch.</li><li>The team wants to understand the pipelines&#39; behavior baselines, so observability in the form of dashboards, alerts, and operational logs is part of their data platform.</li></ul><p>The items above explain development as an independent process. Each contributor can deliver contributions at their own pace and without constant coordination with others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*RL7XiNPOE8mIfeRpR8EWLg.png" /><figcaption>Independent contributions. Image by the author.</figcaption></figure><h4>Purposeful developer</h4><p>Besides financial benefits (3h vs. 20h to deliver a user story), the independent and unconstrained approach described above brings a desired sense of purpose to the developer’s job, as he/she feels enabled to deliver, which might mean a lot for the team’s stability, performance and satisfaction surveys.</p><h3>So, 10 Practical Steps That Empower Developers</h3><p>I noticed ten level-changers that impacted my productivity the most and helped my teams increase theirs:</p><ol><li>Get a Unix-like development environment.</li><li>Enable pre-commit git hooks.</li><li>Create tests.</li><li>Validate user input files using Pydantic.</li><li>Protect the collaboration branch with required checks.</li><li>Switch to a trunk-based branching strategy.</li><li>Infrastructure as code.</li><li>Give each feature branch its own environment.</li><li>Observability.</li><li>Documentation as code.</li></ol><p>Preferably, these steps must be enabled in the given order.</p><p>Let’s discuss each in a bit more detail.</p><h3><strong>Step 1: Get a Unix-like development environment</strong></h3><p>In my opinion, this is the most essential step. I often see developers forced to use corporate Windows laptops, and their tooling is not yet configured. So they run VS Code but have difficulty configuring Spark or even Python environments locally.</p><p>Sometimes, this turns into an extreme case, like when the developer edits code locally, but validation happens by pushing it to the remote feature branch and running unit tests on the CI agent, only because their Windows laptop cannot run them locally.</p><p>If you are in similar circumstances and must write Python/PySpark code, consider getting a “*nix” environment by checking what is possible. A few examples:</p><ul><li>Enable the Windows Subsystem for Linux (WSL) on the company laptop. Luckily, modern versions of WSL2 natively support VPN and proxy connections.</li><li>Check if you can get macOS-based hardware instead of Windows. Maybe your Windows laptop can be replaced, or your employer has a “Bring Your Own Device” (BYOD) policy so you can bring your own Mac to work.</li></ul><p>The next action is to install and get comfortable with a shell of choice. An example is a combination of ZSH + Oh-My-ZSH, optionally topped with the Rust-based starship.rs prompt.</p><p>The command-line shell is where you will spend a significant amount of time daily: switching branches, activating virtual environments, running tests, and checking pipelines. A well-configured shell shows the current git branch and the active Python environment directly in the prompt and completes half of the typing for you. Each of these features saves seconds, and when seconds are multiplied by hundreds of daily commands, the result becomes a noticeable difference by the end of the week.</p><p>It might look like a cosmetic step. In my experience, it is the foundation for everything that follows: pre-commit hooks, local test runs, and infrastructure tooling all assume a working local environment.</p><h3>Step 2: Enable pre-commit git hooks</h3><p>Pre-commit is a framework for managing and maintaining git hooks. Simply put, it is a developer tool that forces each commit to be pre-validated by invoking popular linting hooks before the commit completes.</p><p>In the Python/Data Engineering world, consider the following hooks: ruff, black, mypy, flake8, check-yaml, and check-json.</p><p>They can be configured to be triggered automatically at each commit to validate the code changes early. If the validation fails, the commit is aborted with descriptive messages, so corrective actions can be taken.</p><p>Pre-commit stores its configuration as part of the project codebase, so each developer uses the exact same version of the git hooks. This means no tooling conflicts or accidental overwrites of each other’s changes just because someone in the team uses an ancient version of black.</p><p>Preferably, pre-commit should run not only on developers’ laptops but also as a required check for a Pull Request (more on this in step 5).</p><p>For more details, check:</p><ul><li>The official page: <a href="https://pre-commit.com/">pre-commit</a></li><li>Easy-to-digest write-up: <a href="https://medium.com/marvelous-mlops/welcome-to-pre-commit-heaven-5b622bb8ebce">Welcome to pre-commit heaven</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*cCOI7JKQLFHDhXK4KwpEmQ.png" /><figcaption>Credits: Maria Vechtomova for the flow visualization</figcaption></figure><h3>Step 3: Create tests</h3><p>Automated testing is an indispensable part of software engineering and one of the most significant investments a development team can make to improve their processes and development experience.</p><p>The basic idea is that the automated test runs the data pipeline with a given input and then checks that the generated output matches the expected one.</p><p>The creation of tests is an ongoing process. The team can start with a “happy flow” integration test and eventually enrich the test collection by adding edge cases. This is especially important for bug-fixing tasks, as it is crucial not only to fix a bug but also to create a reproducible test to ensure the issue is not reintroduced.</p><p>An established collection brings the following benefits:</p><ul><li><strong>Early bug detection.</strong> Tests can reveal bugs immediately after a code change, significantly improving the feedback loop.</li><li><strong>Consistency and the human factor.</strong> Every member of the team might have their own style of manual testing. Automation brings consistency and reproducibility, even if the test’s author is no longer part of the team.</li><li><strong>CI/CD.</strong> Automated testing is a prerequisite for the next steps of this list. A PR build policy pipeline can only protect the main branch if there is a test collection to run. Once it exists, every pull request is verified the same way, and the team knows the change is safe to merge before a reviewer even opens it.</li><li><strong>Documentation.</strong> A well-named test describes what the pipeline is supposed to do better than a wiki page written a year ago. Unlike the wiki page, the test fails when the described behavior changes, so this form of documentation cannot silently become outdated.</li><li><strong>Cost savings.</strong> Of course, creating tests is an investment, but it pays off quickly by catching errors in the early phase of development, well before they reach production.</li></ul><p>In my recent article</p><h3><strong>Step 4: Validate user input files using Pydantic</strong></h3><p>Data pipelines are rarely driven by code alone. Quite often, a significant part of the pipeline behavior is defined in YAML or JSON configs that describe sources, destinations, schedules, and schema mappings. These files are edited by humans, so they are prone to typos.</p><p>Without validation, a typo surfaces at the worst possible moment — as a KeyError somewhere in the middle of a pipeline run, often long after the run has started. The error message points to the line of Python code that failed, not to the configuration mistake that caused it. Debugging such failures is a routine time sink.</p><p>Pydantic solves this by declaring the expected structure once, as a typed model. The following snippet shows a model that validates an ingestion configuration file:</p><pre>from pydantic import BaseModel<br><br>class SourceConfig(BaseModel):<br>    name: str<br>    path: str<br>    file_format: str = &quot;delta&quot;<br>    partition_columns: list[str] = []<br><br>config = SourceConfig.model_validate(yaml_payload)</pre><p>If the file is missing a mandatory field or contains a string where a list is expected, the pipeline fails immediately, before any processing starts, and the error message identifies the exact field and the reason. The model also serves as documentation: a new team member can open it to see all supported options and their default values in one place.</p><p>Additionally, such a model-based validation of the configs can be included in a PR validation pipeline; however, this possibility will be covered in the next step.</p><h3>Step 5: Protect the collaboration branch with required checks</h3><p>The previous steps introduced linting, tests, and validations. This step turns them into a quality gateway to protect the collaboration branch.</p><p>GitHub branch protection rules (and Azure DevOps build policies) allow the team to declare the policy that no change merges into the main branch unless the required status checks pass:</p><ul><li>pre-commit hooks</li><li>the set of test scenarios</li><li>Pydantic validation for sensitive user input files</li><li>and a peer review by a teammate</li></ul><p>The merge button remains grey until the pipeline that runs the verification check turns green.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*EU0hjXQA--PA-WfDlPeyWQ.png" /><figcaption>Pull request can be merged when all required checks turn green</figcaption></figure><p>The point is not distrust. Developers forget to run tests locally, especially on a Friday afternoon, and that is normal. The required PR validation pipeline removes the human factor from the equation: the same checks run for every contributor, including the most senior one.</p><p>There is also a cultural benefit. When the pipeline is the gatekeeper, no person has to take that role anymore. Pull request discussion shifts from “Did you test this?” to the actual change itself. In environments with traces of blame culture, this shift alone might be worth the setup effort.</p><p>Eventually, the main branch stays in a safe-to-release state. That property is a prerequisite for the next step.</p><h3>Step 6: Trunk-based branching strategy</h3><p>Many data teams still use some variation of GitFlow: long-lived develop and acceptance branches, features that wait for days before being released to production, and a merge ceremony that often occurs at a specific time of the week. There is nothing wrong with GitFlow as such — it has served software teams for years and provides a sense of control over releases.</p><p>The price of that feeling can be high, though. Long-lived branches drift apart; nobody remembers which commits are pending in the acceptance branch for release, and the chance of a large release in which many smaller individual features go to production at the same moment for the first time increases the risk of regressions and potential rollbacks or hot-fixing.</p><p>Trunk-based development takes the opposite approach: there is one long-lived branch — the trunk, or main. Feature branches are short-lived, ideally a day or two, and are merged back as soon as the required pipeline is green. Each change is small, easy to review, and easy to roll back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FVAUM7fY5w0yqhhicIMzWw.png" /></figure><p>Note the dependency on the previous steps: merging into main several times a day is only safe when tests exist (step 3), user input is validated (step 4), and the branch is protected by required checks (step 5). Without them, trunk-based development would just deliver bugs faster.</p><p>In my experience, this switch is more of a mindset change than a technical one. However, not every team has the choice — versioned products or strict release windows can genuinely require GitFlow. Where the constraint is only a habit, though, trunk-based development has always felt faster to me.</p><h3>Step 7: Infrastructure as code</h3><p>Data platforms are not just pipelines. They are also workspaces, clusters, storage accounts, secret scopes, and access permissions. When these resources are created by clicking in a portal, the environment becomes a snowflake: nobody can tell precisely how it was configured, and nobody can reproduce it.</p><p>Infrastructure as code — (of course, Terraform) — turns every resource into a text definition stored in the same repository as the pipeline code. This brings the infrastructure changes into the same contribution flow as everything else: modifications are proposed in a pull request, reviewed by a teammate, potentially validated by a pipeline, and applied automatically after the merge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NlyINZVFIsH54Z1mXRyo-g.png" /><figcaption>The same three environments, created from a Terraform definition</figcaption></figure><p>The primary benefits are reproducibility and an audit trail. The same definitions create the development, acceptance, and production environments, so differences between them are rather the exception.</p><p>Clicking resources together in the portal is still fine for a one-off experiment in a development or feature environment. For anything the team depends on, though, the portal should be read-only with Privileged Identity Management (PIM) gatekeeping enabled.</p><p>Infrastructure as code also softens a cross-team dependency that many data teams know well: resources that only a central infrastructure team may create, whether a storage account, a Databricks workspace, or an Event Hub. Each request becomes a ticket and a wait, and the team that owns the pipelines does not own the ground they run on — a genuinely disempowering setup. Terraform enables a middle way: the infrastructure team publishes approved modules and policies, and the data team creates what it needs within those guardrails, through the same pull-request flow as any other change.</p><p>There is one more reason this step matters: without it, the next step is simply not achievable.</p><h3>Step 8: Give each feature branch its own environment</h3><p>A shared development environment is a classic source of friction and disappointment. While one developer loads test data, and another one truncates the same table an hour later, the third one adds a new column. End-to-end test results in such a development environment depend on who else is testing that day. Famous “do not touch dev today, I am testing” messages appear in the team chat.</p><p>The alternative is an isolated, short-lived environment for each feature branch that a developer can spin up on demand, so the pipeline deploys a fresh copy of the platform — with sample data preloaded — using the Terraform definitions from step 7. End-to-end tests run there, in complete isolation. When the branch is merged, the environment is destroyed.</p><p>In the Databricks reality, this means isolated Python wheels, notebooks, configs, storage, jobs, and schema catalogs per feature branch, created by the same Terraform definitions and named after the branch or pull request ID.</p><p>It might sound expensive at first glance. In practice, the costs are moderate: the footprint is small, lives for hours rather than weeks, and most cloud resources involved are billed per use. The hours no longer spent on debugging each other’s side effects in the shared environment are usually more expensive.</p><p>The benefit maps directly to the article&#39;s baseline idea: every developer can test their end-to-end change at any time, without waiting for others or affecting them.</p><h3>Step 9: Observability</h3><p>The goal of this step is to understand the platform&#39;s baseline behavior and detect deviations early, especially when new features are frequently delivered to production.</p><p>In practice, it consists of three layers:</p><ul><li><strong>Operational logs.</strong> Structured and centralized, so a failed run can be diagnosed from a single place instead of opening the driver logs for each cluster one by one.</li><li><strong>Dashboards.</strong> Run durations, processed row counts, data freshness, and the time gap between message enqueuing and processing. The trends matter more than the absolute numbers: a microbatch duration that slowed by 30% after the last release is visible on a chart but invisible behind a green checkmark.</li><li><strong>Alerts.</strong> Notifications on deviations from the baseline: a table that has not received data since yesterday, a spike in a certain error code, or a sudden jump in costs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*ABSpEoo-pyLohVTtFo3tcg.png" /><figcaption>Incident ticket, weeks later. Image by the author.</figcaption></figure><p>Observability plays the same role in production as tests play in development — it shortens the feedback loop. The team that sees the degradation on a dashboard fixes it during the sprint. The team that does not receive an incident ticket may potentially suffer a loss of reputation.</p><p>The exact tooling is secondary. It might be Databricks system tables, Application Insights, or Grafana dashboards fed by custom metrics. What matters is that the signals exist and that checking them is routine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*K1f34jNrcbc0eFG0ud1m8w.png" /><figcaption>The same slowdown, hidden behind green checkmarks and visible on a dashboard. Image by the author.</figcaption></figure><h3>Step 10: Documentation as code</h3><p>Documentation has a well-known failure mode: it is written once, with the best intentions, and then quietly lives unchanged in a third-party SaaS service such as Confluence. The root cause is distance — the documentation lives too far away from the code it describes.</p><p>Documentation as code closes that distance. The pages are Markdown files in the same repository, so updating a feature and updating its documentation happen in the same pull request, visible to the same reviewer.</p><p>Tools like MkDocs with the Material theme turn these files into a browsable site, and plugins like mkdocstrings generate the API reference directly from the docstrings that already exist in the code.</p><p>The final piece is automation: a pipeline rebuilds and publishes the site on every merge into the main branch. No need to remember to “update the docs later”, so a decrease in the human factor here as well.</p><p>Autogeneration will not write the narrative parts for you, though. The architecture overview and the onboarding guide still require a human author. What automation removes is the publishing friction and the silent drift between the code and its description.</p><h3>Final words</h3><p>In this article, I covered ten practical steps to improve the productivity of a Python data engineering team: from a properly configured local environment to automated validation and testing, to isolated feature environments, observability, and self-maintaining documentation.</p><p>These steps are not a silver bullet. They will not fix cross-team constraints or a blame culture on their own. They are, though, fully within the development team&#39;s control — no budget approval or reorganization required, only some engineering time and agreement within the team.</p><p>The payoff is the difference described at the beginning: delivering a minor feature within a few morning hours instead of waiting for the next release window. I was part of multiple such team transitions, and the feeling of being enabled to deliver is worth every step. Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=315694e77dc5" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/python-data-engineering-10-steps-from-a-slow-team-to-a-delivering-one-315694e77dc5">Python Data Engineering: 10 Steps From a Slow Team to a Delivering One</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unit Testing PySpark Without the Friction]]></title>
            <link>https://medium.com/@alexandr.volok/unit-testing-pyspark-without-the-friction-755e87225d59?source=rss-cb832394c0a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/755e87225d59</guid>
            <category><![CDATA[pytest]]></category>
            <category><![CDATA[databricks-delta-lake]]></category>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Alex Volok]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 09:51:12 GMT</pubDate>
            <atom:updated>2026-07-10T09:51:12.505Z</atom:updated>
            <content:encoded><![CDATA[<h4>How a small pytest plugin removes the boilerplate of Spark sessions, Delta Lake JARs, and test datasets</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4DdO5KSHA_V96N--C5ZeKw.png" /><figcaption>Test data goes in as plain files, comes back as cached Delta tables — and pytest stays fast.</figcaption></figure><p>I have worked with PySpark for some years, mostly via Azure Databricks, and I keep observing the same pattern in project after project: transformation logic grows, pipelines go to production, and unit tests are still missing. Then, a few sprints later, somebody changes a schema, and the pipeline breaks silently.</p><p>The honest reason is not that data engineers are lazy or do not care about quality. The reason is friction. Before the very first PySpark test can run, you have to learn pytest with its fixtures and conftest files, write a SparkSession builder, find the matching delta jar, and prepare test datasets by hand. That is easily a day of work, and the perceived payoff from a single test is small. Therefore, testing is postponed to the never-coming later.</p><p>So, a while ago, I decided to remove this friction once and for all. That all resulted in <a href="https://github.com/avolok/pytest-pyspark-utils">pytest-pyspark-utils</a>, a small pytest plugin, and this article explains what exactly it does and why.</p><h3>The friction</h3><p>Let’s name the individual pieces first, because each of them will be addressed later:</p><ul><li><strong>pytest itself.</strong> Fixtures, scopes, conftest.py — a whole framework to learn just to get a Spark session into a test function.</li><li><strong>SparkSession boilerplate.</strong> Master, shuffle partitions, log levels, timezone, warehouse directory settings. Copied and pasted between repositories and subtly different in each.</li><li><strong>The Delta Lake version puzzle.</strong> The Maven coordinates must match the exact PySpark version. Was it delta-core or delta-spark? Scala 2.12 or 2.13?</li><li><strong>Test data as code.</strong> Walls of createDataFrame() calls with dozens of tuples that nobody can read and nobody wants to review.</li><li><strong>Slow feedback loop.</strong> Every run spins up Spark and reloads the data, so the suite gets slow, and people stop running it.</li><li><strong>Test pollution.</strong> Tables and Hive state leak between tests; a DELETE in one test silently breaks another one.</li></ul><p>Each of these issues is small on its own; however, when combined, they exceed the effort budget for a regular sprint task.</p><h3>Getting Started. Prepare a clean environment</h3><p>First things first, let’s install the plugin together with the PySpark version of choice:</p><pre>pip install &quot;pytest-pyspark-utils[pyspark4]&quot;       # PySpark 4.x<br># or<br>pip install &quot;pytest-pyspark-utils[pyspark3]&quot;       # PySpark 3.x<br># or<br>pip install pytest-pyspark-utils pyspark==3.5.1    # Exact PySpark release</pre><p>That is the entire setup. The plugin registers itself via the pytest entry point, so there is nothing to import and nothing to add to conftest.py.</p><h3>The spark fixture you did not have to write</h3><p>Right after the installation, every test can request the spark fixture as a regular argument:</p><pre>def test_something(spark):<br>    df = spark.createDataFrame([(1, &quot;a&quot;), (2, &quot;b&quot;)], [&quot;id&quot;, &quot;value&quot;])<br>    assert df.count() == 2</pre><p>Behind the scenes, the fixture builds a session-scoped SparkSession tuned for local testing: local[*] master, a single shuffle partition, disabled UI, quiet logs, UTC timezone everywhere, and a randomly named database per run, so parallel executions do not collide in the Hive metastore.</p><p>Some of these settings are not only about speed. A single shuffle partition and the forced UTC make the runs deterministic: stable ordering, stable timestamps. That removes a whole class of flaky assertions.</p><h3>Delta Lake without the version puzzle</h3><p>Delta support traditionally is the annoying part. The delta-spark JAR coordinates must match the installed PySpark version, and this mapping is not something people keep in their heads.</p><p>The plugin resolves it automatically. On the session start, it detects the installed PySpark version and picks the correct Maven coordinates:</p><pre>PySpark    Delta JAR<br>4.0.x      io.delta:delta-spark_2.13:4.0.1<br>3.5.x      io.delta:delta-spark_2.12:3.3.2<br>3.4.x      io.delta:delta-core_2.12:2.4.0<br>3.3.x      io.delta:delta-core_2.12:2.3.0<br>3.2.x      io.delta:delta-core_2.12:2.0.2</pre><p>If auto-detection is not desired, the JAR can be explicitly pinned in pyproject.toml or passed as — delta-jar on the command line. Luckily, in most cases, nothing has to be configured at all.</p><h3>Test data as files, not walls of code</h3><p>This is, in my opinion, the most helpful of the plugin, so let me spend some extra time explaining here.</p><p>Test datasets in PySpark projects usually live inside the test code as long as <em>createDataFrame()</em> blocks. They are hard to read, hard to review in a pull request, and impossible to hand over to a colleague who does not write Python.</p><p>The plugin takes a different approach: the data lives next to the test as plain CSV or JSONL files:</p><pre>tests/test_my_feature/<br>├── conftest.py<br>├── input/<br>│   ├── users.csv<br>│   └── orders.csv<br>├── expected/<br>│   └── results.csv<br>└── test_my_feature.py</pre><p>The files are described declaratively in conftest.py using TableConfig:</p><pre>import pytest<br>from pytest_pyspark_utils import TableConfig<br><br>@pytest.fixture(scope=&quot;module&quot;)<br>def delta_tables_config():<br>    return {<br>        &quot;users&quot;: TableConfig(source=&quot;input&quot;, schema=users_schema),<br>        &quot;orders&quot;: TableConfig(source=&quot;input&quot;, schema=orders_schema),<br>        &quot;results&quot;: TableConfig(source=&quot;expected&quot;, schema=results_schema),<br>    }</pre><p>And that is all. The delta_tables fixture converts those files into Delta tables and registers them as Spark SQL tables, so the test can simply call spark.table(“users”). TableConfig also supports explicit schemas, partitioning, and even Delta liquid clustering; however, the defaults are enough for most cases.</p><p>The expected folder for the correct output of a transformation, stored as a CSV or JSONL file, is compared against the actual result. A data analyst can open such a file, review it in a PR diff, and edit it without touching a single line of Python. Data becomes data again, not code.</p><p>Some teams use delta/parquet files directly as the input for tests. They push them into git so those data objects reside next to the test code. I intentionally avoid such an approach. As binary files, they are not editable in VS Code and are not git-friendly.</p><h3>Fast because it caches, isolated because it copies</h3><p>The delta_tables fixture works in two layers, and both address a classic issue in data test suites.</p><p>The first layer is module-level caching. The CSV-to-Delta (or JSON-to-Delta) conversion occurs once per test file and is stored in a _delta_cache directory, keyed by an MD5 hash of the file content and the schema. If nothing has changed, the next run immediately reuses the cache; if the data or the schema has changed, the cache invalidates itself. No manual cleanup is involved. On my laptop, the first run incurs a conversion cost of a few seconds per table, while subsequent runs start almost immediately.</p><p>The second layer is test-run-level isolation. Every single test run receives a fresh copy of the tables. Before the test starts, the cached Delta directories are copied to a temporary location, the old tables are dropped, and fresh ones are registered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*bROt2P3PAbpUdXjrRiv25w.png" /><figcaption>_delta_cache in two layers: convert once and reuse the cache across runs, then hand every test its own disposable copy.</figcaption></figure><p>That combination matters. Caching kills the “tests are too slow to run” problem. Isolation kills the “a DELETE in test A poisoned test B” problem:</p><pre>def test_another_scenario(spark, delta_tables):<br>    spark.sql(&quot;DELETE FROM users WHERE id = 1&quot;)<br>    # sibling tests still see all rows — every test gets its own copy</pre><h3>Writing the actual test</h3><p>With everything above in place, the test itself collapses into the classic arrange-act-assert shape, where the arrange part is fully handled by the fixture:</p><pre>from chispa import assert_df_equality<br>from my_project.transformations import enrich_orders<br><br>def test_enrich_orders(spark, delta_tables):<br>    result = enrich_orders(spark.table(&quot;orders&quot;), spark.table(&quot;users&quot;))<br>    assert_df_equality(result, spark.table(&quot;results&quot;), ignore_row_order=True)</pre><p>Three lines. The input data comes from CSV files; the expected output is a golden file; and chispa produces a readable diff when something does not match.</p><h3>Some limitations</h3><p>To be fair, the plugin is opinionated, and it is worth naming the boundaries.</p><p>It is Delta-centric and runs Spark in local mode. It is a unit testing tool; it does not replace end-to-end tests against a real cluster or Unity Catalog to catch Databricks-specific issues. Furthermore, it writes a _delta_cache directory next to the tests, which should be added to .gitignore.</p><p>That said, for the daily loop of writing and refactoring transformation logic, these boundaries were never a problem for me.</p><h3>Final Words</h3><p>Unit tests in PySpark projects are usually not skipped because of laziness. They are skipped because the setup tax exceeds the expected payoff of the first test. pytest-pyspark-utils pays that tax upfront: the Spark session, the correct delta jar, test data handling, caching, and isolation are already in place.</p><p>And once the first test takes minutes instead of an afternoon, the second and third tests come almost for free. As of now, this is the closest thing to a testing culture I have managed to establish in my current data projects, and I hope it can do the same for yours.</p><p>The source code and the full documentation are available on <a href="https://github.com/avolok/pytest-pyspark-utils">GitHub</a>. <br>Thank you for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=755e87225d59" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Approaches to ad hoc Dataframes in PySpark]]></title>
            <link>https://medium.com/plumbersofdatascience/five-different-approaches-to-ad-hoc-dataframe-creation-in-pyspark-665545d656c7?source=rss-cb832394c0a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/665545d656c7</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Alex Volok]]></dc:creator>
            <pubDate>Sun, 30 Jul 2023 18:05:35 GMT</pubDate>
            <atom:updated>2023-10-02T19:36:05.394Z</atom:updated>
            <content:encoded><![CDATA[<h4>Various ways to create small Spark dataframes in place without external dependencies</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WyXBVldSU9vvPbhf" /><figcaption>Photo by <a href="https://unsplash.com/@mbaumi?utm_source=medium&amp;utm_medium=referral">Mika Baumeister</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The common task of PySpark data engineers is to build data processing logic that is mostly a bunch of classes and functions they must test. Modern spark-based data platforms can contain hundreds, if not thousands, of various unit tests that help data teams to keep the codebase from falling apart.</p><p>Crafting small dataframes for those unit tests as input and out values is a part of the engineering routine.</p><p>In this article, I am ambitious to enumerate all known (by me) ways of creating ad-hoc dataframes containing synthetic data. Such approaches are based on ad-hoc and in-place input and do not use source data from the data lake, like parquet, delta, or CSV, nor use any other sources like databases or Kafka streams. So, purely in-place creation without any external dependencies.</p><p>This story is also logically related to my previous one: <a href="https://medium.com/plumbersofdatascience/scripting-pyspark-dataframes-6527ea291e96">Scripting PySpark Dataframes</a>. However, the current write-up focuses entirely on synthetic data instead of making scripts of the existing datasets or dataframes.</p><h4>How making small custom dataframes can be helpful?</h4><p>There are multiple scenarios where an in-place new dataframe might be needed:</p><ul><li>As stated earlier, synthetic data is the input for unit tests.</li><li>Synthetic data and mini dataframes can also be helpful during development time as the validators of the processing logic.</li><li>Reproduction of the bugs in certain parts of PySpark by narrowing down to using a certain functionality.</li></ul><h4>A real-life example</h4><p>Recently, during the upgrade of our platform to Spark 3.4.0, we spotted that a few percent of unit tests failed. All such tests used JSON files as the input, and the session ran in a PERMISSIVE mode.</p><p>Something was going wrong with that new release of OSS Spark. Databricks, which uses the same version of Spark, runs those unit tests correctly. Also, previous versions of OSS and Databricks Spark handled them without errors.</p><p>We quickly crafted a small snippet and created a <a href="https://issues.apache.org/jira/browse/SPARK-44079">Spark JIRA</a> ticket to show that the JSON handler has a newly introduced issue:</p><pre>from pyspark.sql.types import StructType, StructField, IntegerType, StringType<br>import json<br><br><br>data = &quot;&quot;&quot;[{&quot;a&quot;: &quot;incorrect&quot;, &quot;b&quot;: &quot;correct&quot;}]&quot;&quot;&quot;<br>schema = StructType(<br>    [<br>        StructField(&quot;a&quot;, IntegerType(), True),<br>        StructField(&quot;b&quot;, StringType(), True),<br>        StructField(&quot;_corrupt_record&quot;, StringType(), True),<br>    ]<br>)<br><br>(<br>    spark.read.option(&quot;mode&quot;, &quot;PERMISSIVE&quot;)<br>    .option(&quot;multiline&quot;, &quot;true&quot;)<br>    .schema(schema)<br>    .json(spark.sparkContext.parallelize([data]))<br>    .show(truncate=False)<br>)<br><br># returns:<br># java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1</pre><p>As a result, the core team quickly built and merged the fix, and Spark 3.4.2 will contain a bug fix for this issue.</p><h3>The different approaches to generating dataframes</h3><p>Let&#39;s first define a helper method to retrieve the dataframe content and schema so that each example will include the same related information:</p><pre>from pyspark.sql import DataFrame<br><br>def check_dataframe(df: DataFrame):<br>    &quot;&quot;&quot;Helper for checking dataframe content and schema&quot;&quot;&quot;<br>    <br>    print(&#39;&quot;&quot;&quot; output:&#39;)<br>    df.show()<br>    df.printSchema()<br>    print(&#39;&quot;&quot;&quot;&#39;)</pre><h3>Approach # 1 - Create a dataframe using Spark SQL</h3><p>This is my favorite choice because of my DBA background. The SQL syntax provides precise control over the dataframe schema and types, and it is relatively simple to generate from easy to complex data structures:</p><h4>1.1. Create a simple dataframe using Spark SQL</h4><pre>df = spark.sql(<br>    &quot;&quot;&quot;<br>    select * from <br>    values <br>        (1, &#39;abc&#39;), <br>        (2, &#39;def&#39;) as (id, value)<br>    &quot;&quot;&quot;<br>)<br>check_dataframe(df)<br><br>&quot;&quot;&quot; output:<br>+---+-----+<br>| id|value|<br>+---+-----+<br>|  1|  abc|<br>|  2|  def|<br>+---+-----+<br><br>root<br> |-- id: integer (nullable = false)<br> |-- value: string (nullable = false)<br>&quot;&quot;&quot;</pre><h4>1.2. Create a dataframe with nested attributes using Spark SQL</h4><pre>df = spark.sql(<br>    &quot;&quot;&quot;<br>    select <br>      &#39;a&#39; as col1,        <br>      struct(1 as child1, current_date as child2) as col2,<br>      array(1,2,3) as col3<br>    &quot;&quot;&quot;<br>)<br>check_dataframe(df)<br><br>&quot;&quot;&quot; output:<br>+----+---------------+---------+<br>|col1|           col2|     col3|<br>+----+---------------+---------+<br>|   a|{1, 2023-07-29}|[1, 2, 3]|<br>+----+---------------+---------+<br><br>root<br> |-- col1: string (nullable = false)<br> |-- col2: struct (nullable = false)<br> |    |-- child1: integer (nullable = false)<br> |    |-- child2: date (nullable = false)<br> |-- col3: array (nullable = false)<br> |    |-- element: integer (containsNull = false)<br>&quot;&quot;&quot;</pre><h3>Approach # 2 - Create a dataframe from CSV or JSON strings using RDD</h3><p>Sometimes the data team has data defined in a common format like CSV or JSON; however, the content is delivered as available as a variable. In this case, one of the approaches is to generate the RDD and then read it using a proper format handler.</p><h4>2.1. Ingesting JSON from a string variable using RDD:</h4><pre>json_string = &quot;&quot;&quot;<br>[<br>    {&quot;id&quot;: 1, &quot;value&quot;: &quot;abc&quot;},<br>    {&quot;id&quot;: 2, &quot;value&quot;: &quot;def&quot;}<br>]&quot;&quot;&quot;<br><br>rdd_data = spark.sparkContext.parallelize([json_string])<br>df = spark.read.json(rdd_data)<br><br>check_dataframe(df)<br><br><br>&quot;&quot;&quot; output:<br>+---+-----+<br>| id|value|<br>+---+-----+<br>|  1|  abc|<br>|  2|  def|<br>+---+-----+<br><br>root<br> |-- id: long (nullable = true)<br> |-- value: string (nullable = true)<br>&quot;&quot;&quot;</pre><h4>02.2. Ingesting CSV from a string variable using RDD:</h4><pre>csv_string = &quot;&quot;&quot;<br>id,value<br>1,abc<br>2,def<br>&quot;&quot;&quot;<br><br>rdd_data = spark.sparkContext.parallelize(csv_string.splitlines())<br>df = spark.read.options(header=True, inferSchema=True).csv(rdd_data)<br><br>check_dataframe(df)<br><br>&quot;&quot;&quot; output:<br>+---+-----+<br>| id|value|<br>+---+-----+<br>|  1|  abc|<br>|  2|  def|<br>+---+-----+<br><br>root<br> |-- id: integer (nullable = true)<br> |-- value: string (nullable = true<br>&quot;&quot;&quot;<br></pre><h3>Approach # 3 - Create a dataframe using Pandas</h3><p>Another alternative is Pandas, which overlaps with the RDD approach but might be handy for specialists who use this package extensively.</p><h4>3.1. Ingesting JSON from a string variable using Pandas:</h4><pre>import pandas as pd<br>import io<br><br>json_string = &quot;&quot;&quot;<br>[<br>    {&quot;id&quot;: 1, &quot;value&quot;: &quot;abc&quot;},<br>    {&quot;id&quot;: 2, &quot;value&quot;: &quot;def&quot;}<br>]<br>&quot;&quot;&quot;<br><br>pd_data = pd.read_json(io.StringIO(json_string))<br>df = spark.createDataFrame(pd_data)<br><br>check_dataframe(df)<br><br>&quot;&quot;&quot; output:<br>+---+-----+<br>| id|value|<br>+---+-----+<br>|  1|  abc|<br>|  2|  def|<br>+---+-----+<br><br>root<br> |-- id: long (nullable = true)<br> |-- value: string (nullable = true)<br>&quot;&quot;&quot;</pre><h4>3.2. Ingesting CSV from a string variable using Pandas:</h4><pre>import pandas as pd<br>import io<br><br>csv_string = &quot;&quot;&quot;<br>id,value<br>1,abc<br>2,def<br>3,xyz<br>&quot;&quot;&quot;<br><br>data_io = io.StringIO(csv_string)<br>pd_df = pd.read_csv(data_io, sep=&quot;,&quot;)<br>df = spark.createDataFrame(pd_df)<br><br>check_dataframe(df)<br><br>&quot;&quot;&quot; output:<br>+---+-----+<br>| id|value|<br>+---+-----+<br>|  1|  abc|<br>|  2|  def|<br>|  3|  xyz|<br>+---+-----+<br><br>root<br> |-- id: long (nullable = true)<br> |-- value: string (nullable = true)<br>&quot;&quot;&quot;</pre><h4>3.3. Create a pyspark dataframe from the pandas dataframe</h4><pre>import pandas as pd<br><br>pd_df = pd.DataFrame(<br>    {<br>        &quot;id&quot;: [1, 2], <br>        &quot;value&quot;: [&quot;abc&quot;, &quot;def&quot;]<br>    }<br>)<br>df = spark.createDataFrame(pd_df)   <br><br>check_dataframe(df)<br><br>&quot;&quot;&quot; output:<br>+---+-----+<br>| id|value|<br>+---+-----+<br>|  1|  abc|<br>|  2|  def|<br>+---+-----+<br><br>root<br> |-- id: long (nullable = true)<br> |-- value: string (nullable = true)<br>&quot;&quot;&quot;</pre><h3>Approach # 4 - Create a dataframe from a list of Row objects</h3><p>This approach is based on the class pyspark.sql.row and is used by PySpark by default to represent the content of the dataframe. My other story <a href="https://medium.com/plumbersofdatascience/scripting-pyspark-dataframes-6527ea291e96">Scripting PySpark Dataframes</a>, describes it in more detail.</p><h4>4.1. Create a dataframe from the collection of Row objects</h4><p>By default, PySpark cast types automatically infer them from the object attribute values. However, this does not always work as expected. In the example below, the column idhas a type long, but the values specified are integers:</p><pre>from pyspark.sql import Row<br><br>rows_data = [Row(id=1, value=&quot;abc&quot;), Row(id=2, value=&quot;def&quot;)]<br>df = spark.createDataFrame(rows_data)<br><br>check_dataframe(df)<br><br>&quot;&quot;&quot; output:<br>+---+-----+<br>| id|value|<br>+---+-----+<br>|  1|  abc|<br>|  2|  def|<br>+---+-----+<br><br>root<br> |-- id: long (nullable = true)<br> |-- value: string (nullable = true)<br>&quot;&quot;&quot;</pre><h4>4.2. Create a dataframe from the collection of Row objects and explicit schema</h4><p>This example casts column id explicitly to type int:</p><pre>from pyspark.sql import Row<br><br>rows = [Row(id=1, value=&quot;abc&quot;), Row(id=2, value=&quot;def&quot;)]<br>schema = &quot;id int, value string&quot;<br>df = spark.createDataFrame(rows, schema)<br><br>check_dataframe(df)<br><br>&quot;&quot;&quot; output:<br>+---+-----+<br>| id|value|<br>+---+-----+<br>|  1|  abc|<br>|  2|  def|<br>+---+-----+<br><br>root<br> |-- id: integer (nullable = true)<br> |-- value: string (nullable = true)<br>&quot;&quot;&quot;</pre><h3>Approach # 5 - Create a dataframe using Python native data types</h3><p>It is also worth mentioning the Pythonic way, which uses native Python data structures to describe the dataset, like dictionaries, tuples, and lists.</p><h4>5.1. Create a dataframe from the list of dictionaries</h4><pre>data_dicts = [{&quot;id&quot;: 1, &quot;value&quot;: &quot;abc&quot;}, {&quot;id&quot;: 2, &quot;value&quot;: &quot;def&quot;}]<br>df = spark.createDataFrame(data_dicts)<br>check_dataframe(df)<br><br>&quot;&quot;&quot; output:<br>+---+-----+<br>| id|value|<br>+---+-----+<br>|  1|  abc|<br>|  2|  def|<br>+---+-----+<br><br>root<br> |-- id: long (nullable = true)<br> |-- value: string (nullable = true)<br>&quot;&quot;&quot;</pre><h4>5.2. Create a dataframe from the list of tuples</h4><p>This example and the next are cases when the explicit schema definition is needed to determine the right type and the column&#39;s name.</p><pre>data_tuples = [(1, &quot;abc&quot;), (2, &quot;def&quot;)]<br>schema = &quot;id int, value string&quot;<br>df = spark.createDataFrame(data_tuples, schema)<br>check_dataframe(df)<br><br>&quot;&quot;&quot; output:<br>+---+-----+<br>| id|value|<br>+---+-----+<br>|  1|  abc|<br>|  2|  def|<br>+---+-----+<br><br>root<br> |-- id: integer (nullable = true)<br> |-- value: string (nullable = true)<br>&quot;&quot;&quot;</pre><h4>5.3. Create a dataframe from the list of lists</h4><pre>data_lists = [[1, &quot;abc&quot;], [2, &quot;def&quot;]]<br>schema = &quot;id int, value string&quot;<br>df = spark.createDataFrame(data_lists, schema)<br>check_dataframe(df)<br><br>&quot;&quot;&quot; output:<br>+---+-----+<br>| id|value|<br>+---+-----+<br>|  1|  abc|<br>|  2|  def|<br>+---+-----+<br><br>root<br> |-- id: integer (nullable = true)<br> |-- value: string (nullable = true)<br>&quot;&quot;&quot;</pre><h3>Final words</h3><p>I started this article as Jupyter Notebook in the form of a helper reference to the “future me” so I can always come back and re-use the snippet. Over time it turned into this write-up which still acts as a reference that I update occasionally. I hope someone else might find it also helpful. And if you know any other ad-hoc method of creating a Spark dataframe, please leave a comment so I can include it in this text.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=665545d656c7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/plumbersofdatascience/five-different-approaches-to-ad-hoc-dataframe-creation-in-pyspark-665545d656c7">5 Approaches to ad hoc Dataframes in PySpark</a> was originally published in <a href="https://medium.com/plumbersofdatascience">Plumbers Of Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scripting PySpark Dataframes]]></title>
            <link>https://medium.com/plumbersofdatascience/scripting-pyspark-dataframes-6527ea291e96?source=rss-cb832394c0a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/6527ea291e96</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Alex Volok]]></dc:creator>
            <pubDate>Mon, 12 Jun 2023 08:15:42 GMT</pubDate>
            <atom:updated>2023-06-12T08:15:42.984Z</atom:updated>
            <content:encoded><![CDATA[<h4>Reproducing and transporting dataframes by generating plain Python scripts</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*LpC0rry7TdCeOSDyPPog_w.jpeg" /><figcaption>Figure 01: Generating a script, Source: by DALL-E</figcaption></figure><p>Developing Spark applications means dealing with Spark DataFrames. These objects are in-memory data structures accessible via various APIs but locally scoped. Sometimes we need to use them outside the runtime environment. Scripting a dataframe as the collection of Python commands that fully reproduce this object is a possible and occasionally preferred solution.</p><h3>The use cases</h3><p>The most common example is a debugging of a production issue. So, think of a Data Processing framework built around Spark. It runs in a production environment, and you spot that one of the workflows fails. Further checks show that there is a flaw in the processing logic. It is handy to identify those few rows on which the processing crashes, create a dataframe and then transfer it to your development environment.</p><p>The transported dataframe might then be used for the following:</p><ol><li>Debugging the issue in the development/feature environment</li><li>Creating a unit test for preventing such kind of errors</li><li>Creating a default sample data when the new development/feature environment must be deployed</li></ol><h4>Why not just save the dataframe as parquet?</h4><p>Some will ask: “Why bother with scripting if the dataframe can be dumped as a parquet file?”.</p><p>The quick answer is that parquet is a binary format; it is very efficient but hard to be viewed and edit using text tools.</p><p>The Python script of the dataframe contains both data and schema and can be edited in place when needed, bringing flexibility. The dataframe script can eventually be turned into a parameterized Pytest fixture to enable multiple testing scenarios without crafting a set of binary files for each one.</p><p>Another aspect is Git. Git handles plain text/code better than binary content. The data team might use small parquet/delta files as the input for unit tests. If its data content must be updated, the peer reviewer cannot see what exactly changed.</p><p>And as the analogy from the database world: there are scenarios when binary backup might be the most robust and preferred solution. Still, sometimes a script with a table definition and few insert statements is better.</p><h3>Scripting a dataframe in practice</h3><p>Before we proceed with an example, let’s define a sample dataframe with a complex enough schema:</p><pre>df = spark.sql(<br>    &quot;&quot;&quot; <br>    SELECT <br>        1 AS ID, <br>        &quot;John Doe&quot; AS Name, <br>        STRUCT(<br>            &quot;line1&quot; AS Line1, <br>            &quot;line2&quot; AS Line2<br>        ) AS Address, <br>        TO_DATE(&quot;1980-01-01&quot;) AS BirthDate,<br>        ARRAY(<br>            STRUCT(<br>                012345678 AS AccountNumber, <br>                &quot;NL&quot; AS CountryCode,<br>                1005.50 AS Balance<br>            ),<br>            STRUCT(<br>                012345678 AS AccountNumber, <br>                &quot;UK&quot; AS CountryCode,<br>                NULL AS Balance<br>            )<br>        ) AS BankAccounts<br>    &quot;&quot;&quot;<br>)<br>df.printSchema()<br><br><br>&quot;&quot;&quot;<br>root<br> |-- ID: integer (nullable = false)<br> |-- Name: string (nullable = false)<br> |-- Address: struct (nullable = false)<br> |    |-- Line1: string (nullable = false)<br> |    |-- Line2: string (nullable = false)<br> |-- BirthDate: date (nullable = true)<br> |-- BankAccounts: array (nullable = false)<br> |    |-- element: struct (containsNull = false)<br> |    |    |-- AccountNumber: integer (nullable = false)<br> |    |    |-- CountryCode: string (nullable = false)<br> |    |    |-- Balance: decimal(6,2) (nullable = true)<br>&quot;&quot;&quot;<br></pre><p>The dataframe can be one-line scripted using a custom helper function script_dataframe() which returns generated code as the string output:</p><pre><br>dataframe_script = script_dataframe(df)<br>print(dataframe_script)</pre><p>The content of the dataframe_script is a Python snippet that is small enough to be copy-pasted, edited if needed (to remove PII data, for instance), and executed in any other environment. It contains __data, a collection of rows, and __schema that holds an exact schema of the scripted dataframe:</p><pre>from pyspark.sql import Row<br>import datetime<br>from decimal import Decimal<br>from pyspark.sql.types import *<br><br># Scripted data and schema:<br>__data = [Row(ID=1, Name=&#39;John Doe&#39;, Address=Row(Line1=&#39;line1&#39;, Line2=&#39;line2&#39;), BirthDate=datetime.date(1980, 1, 1), BankAccounts=[Row(AccountNumber=12345678, CountryCode=&#39;NL&#39;, Balance=Decimal(&#39;1005.50&#39;)), Row(AccountNumber=12345678, CountryCode=&#39;UK&#39;, Balance=None)])]<br>__schema = StructType.fromJson({&#39;type&#39;: &#39;struct&#39;, &#39;fields&#39;: [{&#39;name&#39;: &#39;ID&#39;, &#39;type&#39;: &#39;integer&#39;, &#39;nullable&#39;: False, &#39;metadata&#39;: {}}, {&#39;name&#39;: &#39;Name&#39;, &#39;type&#39;: &#39;string&#39;, &#39;nullable&#39;: False, &#39;metadata&#39;: {}}, {&#39;name&#39;: &#39;Address&#39;, &#39;type&#39;: {&#39;type&#39;: &#39;struct&#39;, &#39;fields&#39;: [{&#39;name&#39;: &#39;Line1&#39;, &#39;type&#39;: &#39;string&#39;, &#39;nullable&#39;: False, &#39;metadata&#39;: {}}, {&#39;name&#39;: &#39;Line2&#39;, &#39;type&#39;: &#39;string&#39;, &#39;nullable&#39;: False, &#39;metadata&#39;: {}}]}, &#39;nullable&#39;: False, &#39;metadata&#39;: {}}, {&#39;name&#39;: &#39;BirthDate&#39;, &#39;type&#39;: &#39;date&#39;, &#39;nullable&#39;: True, &#39;metadata&#39;: {}}, {&#39;name&#39;: &#39;BankAccounts&#39;, &#39;type&#39;: {&#39;type&#39;: &#39;array&#39;, &#39;elementType&#39;: {&#39;type&#39;: &#39;struct&#39;, &#39;fields&#39;: [{&#39;name&#39;: &#39;AccountNumber&#39;, &#39;type&#39;: &#39;integer&#39;, &#39;nullable&#39;: False, &#39;metadata&#39;: {}}, {&#39;name&#39;: &#39;CountryCode&#39;, &#39;type&#39;: &#39;string&#39;, &#39;nullable&#39;: False, &#39;metadata&#39;: {}}, {&#39;name&#39;: &#39;Balance&#39;, &#39;type&#39;: &#39;decimal(6,2)&#39;, &#39;nullable&#39;: True, &#39;metadata&#39;: {}}]}, &#39;containsNull&#39;: False}, &#39;nullable&#39;: False, &#39;metadata&#39;: {}}]})<br><br>outcome_dataframe = spark.createDataFrame(__data, __schema)</pre><p>Let’s verify if the script works correctly by running it and exploring the dataframe content:</p><pre>outcome_dataframe.show(truncate=False)<br><br>&quot;&quot;&quot;<br>+---+--------+--------------+----------+-----------------------------------------------+<br>|ID |Name    |Address       |BirthDate |BankAccounts                                   |<br>+---+--------+--------------+----------+-----------------------------------------------+<br>|1  |John Doe|{line1, line2}|1980-01-01|[{12345678, NL, 1005.50}, {12345678, UK, null}]|<br>+---+--------+--------------+----------+-----------------------------------------------+<br>&quot;&quot;&quot;<br><br>outcome_dataframe.printSchema()<br><br>&quot;&quot;&quot;<br>root<br> |-- ID: integer (nullable = false)<br> |-- Name: string (nullable = false)<br> |-- Address: struct (nullable = false)<br> |    |-- Line1: string (nullable = false)<br> |    |-- Line2: string (nullable = false)<br> |-- BirthDate: date (nullable = true)<br> |-- BankAccounts: array (nullable = false)<br> |    |-- element: struct (containsNull = false)<br> |    |    |-- AccountNumber: integer (nullable = false)<br> |    |    |-- CountryCode: string (nullable = false)<br> |    |    |-- Balance: decimal(6,2) (nullable = true)<br>&quot;&quot;&quot;</pre><h4>Limitations</h4><p>The logic of script_dataframe() internally relies on DataFrame.collect(), which means it gathers all scripted rows in Spark’s driver. Therefore, the function is intended to dump relatively small dataframes. By default, it limits the number of rows to 20, but that limit can be increased using an extra parameter limit_rows.</p><h3>Source code</h3><p>The following code was tested and runs in PySpark and Databricks:</p><pre>from pprint import pformat<br>from pyspark.sql import DataFrame<br><br>def script_dataframe(<br>    input_dataframe: DataFrame, limit_rows: int = 20<br>) -&gt; str:<br>    &quot;&quot;&quot;Generate a script to recreate the dataframe<br>    The script includes the schema and the data<br><br>    Args:<br>        input_dataframe (DataFrame): Input spark dataframe<br>        limit_rows (int, optional): Prevents too large dataframe to be processed. \<br>            Defaults to 20.<br><br>    Raises:<br>        ValueError: when the dataframe is too large (by default &gt; 20 rows)<br><br>    Returns:<br>        The script to recreate the dataframe<br><br>    Examples:<br>        &gt;&gt;&gt; script = script_dataframe(input_dataframe=df)<br>        &gt;&gt;&gt; print(script)<br>    &quot;&quot;&quot;<br><br>    if input_dataframe.count() &gt; limit_rows:<br>        raise ValueError(<br>            &quot;This method is limited to script up &quot;<br>            f&quot;to {limit_rows} row(s) per call&quot;<br>        )<br>    <br>    __data = pformat(input_dataframe.collect())<br><br>    __schema = input_dataframe.schema.jsonValue()<br><br>    __script_lines = [<br>        &quot;from pyspark.sql import Row&quot;,<br>        &quot;import datetime&quot;,<br>        &quot;from decimal import Decimal&quot;,<br>        &quot;from pyspark.sql.types import *&quot;,<br>        &quot;&quot;,<br>        &quot;# Scripted data and schema:&quot;,<br>        f&quot;__data = {__data}&quot;,<br>        f&quot;__schema = StructType.fromJson({__schema})&quot;,<br>        &quot;&quot;,<br>        &quot;outcome_dataframe = spark.createDataFrame(__data, __schema)&quot;,<br>    ]<br><br>    __final_script = &quot;\n&quot;.join(__script_lines)<br><br>    return __final_script</pre><h3>Final words</h3><p>While the PySpark dataframe scripting is not (yet) part of the native functionality, it is a relatively simple operation. It can be done via the custom helper function script_dataframe()I shared above. It helps me to craft unit tests faster and to make the processing frameworks more reliable. I hope it will also help you in your PySpark journey.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6527ea291e96" width="1" height="1" alt=""><hr><p><a href="https://medium.com/plumbersofdatascience/scripting-pyspark-dataframes-6527ea291e96">Scripting PySpark Dataframes</a> was originally published in <a href="https://medium.com/plumbersofdatascience">Plumbers Of Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delta Properties and Check Constraints at Scale]]></title>
            <link>https://medium.com/plumbersofdatascience/delta-properties-and-check-constraints-at-scale-2b581ee04e01?source=rss-cb832394c0a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b581ee04e01</guid>
            <category><![CDATA[delta-tables]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[Alex Volok]]></dc:creator>
            <pubDate>Fri, 26 May 2023 14:14:37 GMT</pubDate>
            <atom:updated>2023-05-26T14:14:37.697Z</atom:updated>
            <content:encoded><![CDATA[<h4>Managing Delta Table Properties and Check Constraints using Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ku0vQUL_PR2Pdm-l7OrkJg.jpeg" /><figcaption>Figure 01: Random image generated by DALL-E</figcaption></figure><p>Delta arguably is the most popular data format in the current Data Engineering landscape. It is mainly used with Databricks/Spark ecosystem; however, this is optional. It is a universal open-source format and works great out of the box. However, it might be that the default behavior is not sufficient for some use cases, and then custom table-level properties must be defined.</p><p>It is not that hard to hit that point. To define the Delta table property or the check constraint, you have the only option — ALTER TABLE statement. This command is easy to embed in an ad-hoc notebook. What to do when there are no manual processes and delta objects are delivered in a fully automated way using config-driven Python pipelines?</p><p>This story answers the question above and reveals my path from the exploration to the solution that scales.</p><h3>Prepare a clean environment</h3><p>First thing first, let’s prepare a clean Python environment to run showcase examples.</p><p>Traditionally, I am going to run the code on Apple Silicon; however, all snippets must be compatible with Linux and WSL of Windows 10/11:</p><pre>mkdir delta-properties<br>cd delta-properties<br>python3.10 -m venv .venv<br>source .venv/bin/activate<br>pip install pyspark==3.3.1 delta-spark==2.3.0<br>touch delta-properties.ipynb<br>code .</pre><p>Then initialize minimalistic configuration of spark with the latest delta jar, and disable all Spark UI:</p><pre>from pyspark.sql import SparkSession<br><br>spark = (<br>    SparkSession.builder.master(&quot;local[*]&quot;)<br>    .appName(&quot;MyApp&quot;)<br>    .config(&quot;spark.jars.packages&quot;, &quot;io.delta:delta-core_2.12:2.3.0&quot;)<br>    .config(&quot;spark.sql.extensions&quot;, &quot;io.delta.sql.DeltaSparkSessionExtension&quot;)<br>    .config(<br>        &quot;spark.sql.catalog.spark_catalog&quot;,<br>        &quot;org.apache.spark.sql.delta.catalog.DeltaCatalog&quot;,<br>    )    <br>    .config(&quot;spark.ui.showConsoleProgress&quot;, &quot;false&quot;)<br>    .config(&quot;spark.ui.enabled&quot;, &quot;false&quot;)<br>    .config(&quot;spark.ui.dagGraph.retainedRootRDDs&quot;, &quot;1&quot;)<br>    .config(&quot;spark.ui.retainedJobs&quot;, &quot;1&quot;)<br>    .config(&quot;spark.ui.retainedStages&quot;, &quot;1&quot;)<br>    .config(&quot;spark.ui.retainedTasks&quot;, &quot;1&quot;)<br>    .config(&quot;spark.sql.ui.retainedExecutions&quot;, &quot;1&quot;)<br>    .config(&quot;spark.worker.ui.retainedExecutors&quot;, &quot;1&quot;)<br>    .config(&quot;spark.worker.ui.retainedDrivers&quot;, &quot;1&quot;)<br>    .config(&quot;spark.driver.memory&quot;, &quot;4g&quot;) <br>).getOrCreate()</pre><p>When the spark is up, create a sample delta table that will be a subject of coming experiments:</p><pre>df = spark.sql(<br>    &quot;&quot;&quot;<br>    SELECT <br>        1 as id, <br>        &quot;John&quot; AS name, <br>        struct(<br>            &quot;Longstreet 1&quot; AS line1, <br>            &quot;1234AB&quot; AS zipcode<br>        ) AS address<br>&quot;&quot;&quot;)<br>df.printSchema()<br><br>&quot;&quot;&quot;<br>root<br> |-- id: integer (nullable = false)<br> |-- name: string (nullable = false)<br> |-- address: struct (nullable = false)<br> |    |-- line1: string (nullable = false)<br> |    |-- zipcode: string (nullable = false)<br>&quot;&quot;&quot;<br><br>df.write.format(&quot;delta&quot;).mode(&quot;overwrite&quot;).save(&quot;/tmp/sample_delta_dataset&quot;)</pre><h3>A real-world example</h3><p>So you build a Spark-based data processing framework. You want to improve Structured Streaming latencies by modifying default table properties. Because the streaming logic uses.ForEachBatch with a MERGE statement, you also plan to add NOT NULL constraints on Primary Key columns in the silver layer.</p><h4>Table Properties</h4><p>This part is easy to construct with a bit of dynamic SQL and then embed into the framework, and it will work:</p><pre>spark.sql(<br>    &quot;&quot;&quot;<br>    ALTER TABLE delta.`/tmp/sample_delta_dataset`<br>    SET TBLPROPERTIES (<br>        delta.checkpoint.writeStatsAsStruct = true,<br>        delta.checkpoint.writeStatsAsJson = false<br>    );<br>    &quot;&quot;&quot;<br>)</pre><p>However, the framework triggered the data load five times, and you noticed that Delta History is polluted by the SET TBLPROPERTIES messages:</p><pre>(<br>    spark.sql(<br>        &quot;DESCRIBE HISTORY delta.`/tmp/sample_delta_dataset`;&quot;<br>    ).select(&quot;version&quot;, &quot;operation&quot;, &quot;operationParameters&quot;)<br>    .show(truncate=False)<br>)<br><br>&quot;&quot;&quot;<br>+-------+-----------------+----------------------------------------------------------------------------------------------------------+<br>|version|operation        |operationParameters                                                                                       |<br>+-------+-----------------+----------------------------------------------------------------------------------------------------------+<br>|5      |SET TBLPROPERTIES|{properties -&gt; {&quot;delta.checkpoint.writeStatsAsStruct&quot;:&quot;true&quot;,&quot;delta.checkpoint.writeStatsAsJson&quot;:&quot;false&quot;}}|<br>|4      |SET TBLPROPERTIES|{properties -&gt; {&quot;delta.checkpoint.writeStatsAsStruct&quot;:&quot;true&quot;,&quot;delta.checkpoint.writeStatsAsJson&quot;:&quot;false&quot;}}|<br>|3      |SET TBLPROPERTIES|{properties -&gt; {&quot;delta.checkpoint.writeStatsAsStruct&quot;:&quot;true&quot;,&quot;delta.checkpoint.writeStatsAsJson&quot;:&quot;false&quot;}}|<br>|2      |SET TBLPROPERTIES|{properties -&gt; {&quot;delta.checkpoint.writeStatsAsStruct&quot;:&quot;true&quot;,&quot;delta.checkpoint.writeStatsAsJson&quot;:&quot;false&quot;}}|<br>|1      |SET TBLPROPERTIES|{properties -&gt; {&quot;delta.checkpoint.writeStatsAsStruct&quot;:&quot;true&quot;,&quot;delta.checkpoint.writeStatsAsJson&quot;:&quot;false&quot;}}|<br>|0      |WRITE            |{mode -&gt; Overwrite, partitionBy -&gt; []}                                                                    |<br>+-------+-----------------+----------------------------------------------------------------------------------------------------------+<br>&quot;&quot;&quot;</pre><p>This is not a perfect situation, but still not a roadblock. At the end of the day, you can write the script that checks current table properties and changes them only when needed.</p><h4>Table Check Constraints</h4><p>Similarly to table properties, you extend the framework to create NOT NULL constraints for each Primary Key column of the input dataset config, so the logic gets one more dynamically generated SQL command:</p><pre>spark.sql(<br>    &quot;&quot;&quot;<br>    ALTER TABLE delta.`/tmp/sample_delta_dataset`<br>    ADD CONSTRAINT id_is_not_null CHECK (id is not null)<br>    &quot;&quot;&quot;<br>)</pre><p>The first time it runs correctly, but the consecutive runs break the execution flow:</p><pre>AnalysisException: Constraint &#39;id_is_not_null&#39; already exists. Please delete the old constraint first.<br>Old constraint:<br>id is not null</pre><p>This time, things get nastier. The error is a blocker and must be fixed in a structured way.</p><h3>An elephant in the room</h3><p>This elephant is a generation of SQL DDL statements. It is not just a non-Pythonic way to deal with this problem. The embedded DDL statements are hard for abstraction, not always idempotent, and inconvenient for unit testing.</p><p>Ideally, the official Delta-Spark package must contain the logic to deal with such tasks. However, at this point, the existing official class DeltaTable does not have any Python abstraction for properties, constraints, table metadata, etc. So it still must be defined using SQL DDL format.</p><h3>What about native NOT NULL column constraint</h3><p>Some may ask why I define a table-level constraint to enforce NOT NUL for a particular column while the documentation says it is possible to do on the column level:</p><pre>spark.sql(<br>    &quot;&quot;&quot;<br>    ALTER TABLE delta.`/tmp/sample_delta_dataset`<br>    CHANGE COLUMN id SET NOT NULL;<br>    &quot;&quot;&quot;<br>)</pre><p>Even though such column-level <a href="https://docs.databricks.com/tables/constraints.html">Constraints</a> are documented and must be supported, I failed to make them work in Databricks 11.3 and Spark 3.3.1. The command fails with the same error in both offerings:</p><pre>AnalysisException: Cannot change nullable column to non-nullable: id; line 2 pos 4;<br>AlterColumn resolvedfieldname(StructField(id,IntegerType,true)), false</pre><p>I hope that this issue will be solved soon, but for the time being, check constraints are used instead.</p><p>The internal benchmarks show that the overhead of table constraint checks is relatively tiny in data manipulation operations; however, due to the mentioned issue, I cannot directly compare table and column-level constraints.</p><h3>An alternative using Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4juIRASj0It2VB-ZhOUHhg.jpeg" /><figcaption>Figure 02: (Hard) Thinking Data Engineer. Source: DALL-E</figcaption></figure><p>The limitations mentioned earlier, and the vision of having a <em>“desired state configuration”</em> kind of experience brought me to the need to write a Python abstraction around the discussed subject. So, to abstract the configuration process of the Delta tables. That all resulted in the Python class DeltaTableConfig.</p><h4>Table Properties</h4><p>So, the first example. The framework knows that the table sample_delta_dataset is a streaming dataset, and it prescribes the following two standard options for such a kind table:</p><pre>STREAMING_PROPERTIES = {<br>    &quot;delta.checkpoint.writeStatsAsStruct&quot;: &quot;true&quot;,<br>    &quot;delta.checkpoint.writeStatsAsJson&quot;: &quot;false&quot;,<br>}<br>dtc = DeltaTableConfig(table_path=&quot;/tmp/sample_delta_dataset&quot;)<br>dtc.properties = STREAMING_PROPERTIES</pre><p>the first run enables the properties:</p><pre>2023-05-21 16:03:35 INFO   Applying table properties on &#39;delta.`/tmp/sample_delta_dataset`&#39;:<br>2023-05-21 16:03:35 INFO   Checking if &#39;delta.checkpoint.writeStatsAsStruct = true&#39; is set on delta.`/tmp/sample_delta_dataset`<br>2023-05-21 16:03:36 INFO   The property has been set<br>2023-05-21 16:03:36 INFO   Checking if &#39;delta.checkpoint.writeStatsAsJson = false&#39; is set on delta.`/tmp/sample_delta_dataset`<br>2023-05-21 16:03:36 INFO   The property has been set</pre><p>And the following runs detect that properties already exist so it has nothing to do:</p><pre>2023-05-21 16:09:06 INFO   Applying table properties on &#39;delta.`/tmp/sample_delta_dataset`&#39;:<br>2023-05-21 16:09:06 INFO   Checking if &#39;delta.checkpoint.writeStatsAsStruct = true&#39; is set on delta.`/tmp/sample_delta_dataset`<br>2023-05-21 16:09:06 INFO   The property already exists on the table<br>2023-05-21 16:09:06 INFO   Checking if &#39;delta.checkpoint.writeStatsAsJson = false&#39; is set on delta.`/tmp/sample_delta_dataset`<br>2023-05-21 16:09:06 INFO   The property already exists on the table</pre><p>The check of delta history confirms that the property-related messages appear in a single instance:</p><pre>(<br>    spark.sql(<br>        &quot;DESCRIBE HISTORY delta.`/tmp/sample_delta_dataset`;&quot;<br>    ).select(&quot;version&quot;, &quot;operation&quot;, &quot;operationParameters&quot;)<br>    .show(truncate=False)<br>)<br><br>+-------+-----------------+--------------------------------------------------------------+<br>|version|operation        |operationParameters                                           |<br>+-------+-----------------+--------------------------------------------------------------+<br>|2      |SET TBLPROPERTIES|{properties -&gt; {&quot;delta.checkpoint.writeStatsAsJson&quot;:&quot;false&quot;}} |<br>|1      |SET TBLPROPERTIES|{properties -&gt; {&quot;delta.checkpoint.writeStatsAsStruct&quot;:&quot;true&quot;}}|<br>|0      |WRITE            |{mode -&gt; Overwrite, partitionBy -&gt; []}                        |<br>+-------+-----------------+--------------------------------------------------------------+</pre><p>Imagine another scenario: the table type changed from <em>“streaming”</em> to <em>“append-only.”</em> The framework might have different options for that kind of table:</p><pre>APPEND_ONLY_PROPERTIES = {<br>    &quot;delta.appendOnly&quot;: &quot;true&quot;,<br>    &quot;delta.enableChangeDataFeed&quot;: &quot;true&quot;,<br>}<br>dtc.properties = APPEND_ONLY_PROPERTIES</pre><p>The DeltaTableConfig unsets previous properties and sets two new items:</p><pre>2023-05-21 16:17:06 INFO   Applying table properties on &#39;delta.`/tmp/sample_delta_dataset`&#39;:<br>2023-05-21 16:17:06 INFO   Checking if &#39;delta.appendOnly = true&#39; is set on delta.`/tmp/sample_delta_dataset`<br>2023-05-21 16:17:07 INFO   The property has been set<br>2023-05-21 16:17:07 INFO   Checking if &#39;delta.enableChangeDataFeed = true&#39; is set on delta.`/tmp/sample_delta_dataset`<br>2023-05-21 16:17:07 INFO   The property has been set<br>2023-05-21 16:17:08 INFO   The property &#39;delta.checkpoint.writeStatsAsStruct = true&#39; has been unset because it is not defined in the original dict<br>2023-05-21 16:17:08 INFO   The property &#39;delta.checkpoint.writeStatsAsJson = false&#39; has been unset because it is not defined in the original dict</pre><p>If the new properties must be added with keeping all existing ones, use the flag .keep_existing_properties = True. So, let’s re-add previously removed delta properties:</p><pre>dtc.keep_existing_properties = True<br>dtc.properties = STREAMING_PROPERTIES</pre><p>The attribute .properties acts as a getter and setter, so the existing values can be retrieved by getting the value of the attribute:</p><pre>from pprint import pprint<br>pprint(dtc.properties)<br><br>{&#39;delta.appendOnly&#39;: &#39;true&#39;,<br> &#39;delta.checkpoint.writeStatsAsJson&#39;: &#39;false&#39;,<br> &#39;delta.checkpoint.writeStatsAsStruct&#39;: &#39;true&#39;,<br> &#39;delta.enableChangeDataFeed&#39;: &#39;true&#39;}</pre><p>Under the hood the helper class runs DeltaTable.detail()each time to get the current state.</p><h4>Check Constraints</h4><p>Similarly to the Table Properties, the data processing framework can pre-generate the input dictionary PK_CHECK_CONSTRAINTS with the constraints definitions to add. DeltaTableConfig then adds prescribed check constraints if they still need to be created.</p><pre>dtc = DeltaTableConfig(table_path=&quot;/tmp/sample_delta_dataset&quot;)<br><br>PK_CHECK_CONSTRAINTS = {<br>    &quot;id_is_not_null&quot;: &quot;id is not null&quot;,<br>}<br>dtc.check_constraints = PK_CHECK_CONSTRAINTS</pre><p>results to the output:</p><pre>2023-05-21 16:30:00 INFO   Applying check constraints on &#39;delta.`/tmp/sample_delta_dataset`&#39;:<br>2023-05-21 16:30:00 INFO   Checking if constraint &#39;id_is_not_null&#39; was already set on delta.`/tmp/sample_delta_dataset`<br>2023-05-21 16:30:01 INFO   The constraint id_is_not_null has been successfully added to &#39;delta.`/tmp/sample_delta_dataset`&#39;</pre><p>In case of new constraints are to be added instead of existing ones, provide the new dictionary to apply:</p><pre>NEW_CHECK_CONSTRAINTS = {<br>    &quot;address_zipcode_is_not_null&quot;: &quot;address.zipcode is not null&quot;,<br>}<br>dtc.check_constraints = NEW_CHECK_CONSTRAINTS</pre><p>As you might expect, the old constraint is removed, and a new one is added:</p><pre>2023-05-21 16:32:00,180 INFO    Applying check constraints on &#39;delta.`/tmp/sample_delta_dataset`&#39;:<br>2023-05-21 16:32:01,085 INFO    The constraint &#39;id_is_not_null&#39; has been dropped from the table &#39;delta.`/tmp/sample_delta_dataset`&#39;<br>2023-05-21 16:32:01,085 INFO    Checking if constraint &#39;address_zipcode_is_not_null&#39; was already set on delta.`/tmp/sample_delta_dataset`<br>2023-05-21 16:32:01,616 INFO    The constraint address_zipcode_is_not_null has been successfully added to &#39;delta.`/tmp/sample_delta_dataset`&#39;</pre><p>If existing check constraints must be preserved, use the optional argument .keep_existing_check_constraints=True</p><pre>dtc.keep_existing_check_constraints = True<br>dtc.check_constraints = PK_CHECK_CONSTRAINTS</pre><p>All existing constraints can be programmatically retrieved via getter .check_constraints:</p><pre>print(dtc.check_constraints)<br><br>{<br>    &#39;address_zipcode_is_not_null&#39;: &#39;address.zipcode is not null&#39;,<br>    &#39;id_is_not_null&#39;: &#39;id is not null&#39;<br>}</pre><h3>Delta Table Check Constraints in Action</h3><p>Let’s then run a final smoke test to ensure that all those constraint efforts payback:</p><pre>spark.sql(<br>    &quot;&quot;&quot;<br>    INSERT INTO delta.`/tmp/sample_delta_dataset`<br>    VALUES (1, &#39;John Doe&#39;, STRUCT(&#39;123 Main St&#39;, &#39;10001&#39;))<br>    &quot;&quot;&quot;<br>)</pre><p>So, the insert with non-null values works as expected.</p><p>The following command intentionally inserts a NULL in the nested attribute address.zipcode, which is protected by the recently added check constraint:</p><pre>spark.sql(<br>    &quot;&quot;&quot;<br>    INSERT INTO delta.`/tmp/sample_delta_dataset`<br>    VALUES (1, &#39;John Doe&#39;, STRUCT(&#39;123 Main St&#39;, NULL))<br>    &quot;&quot;&quot;<br>)<br><br>&quot;&quot;&quot;<br>org.apache.spark.sql.delta.schema.DeltaInvariantViolationException:<br>CHECK constraint address_zipcode_is_not_null (address.zipcode IS NOT NULL) violated by row with values:<br> - address.zipcode : null<br>&quot;&quot;&quot;</pre><p>The code fails with a clear message that must eventually trigger the data team to fix the issue.</p><h3>The source code</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07dd69d0876fd12bbf6f586d363e311d/href">https://medium.com/media/07dd69d0876fd12bbf6f586d363e311d/href</a></iframe><p>Alternatively, the source code is documented, maintained, tested, and constantly updated as a GitHub-hosted open-source <a href="https://github.com/avolok/spalah">project</a>. So the usage is a few lines of the code away:</p><pre># %pip install spalah<br><br>from spalah.dataset import DeltaProperty<br><br>dt = DeltaProperty(table_path=&#39;/tmp/sample_delta_dataset&#39;)<br>print(dt.properties)</pre><h3>Final Words</h3><p>Although Delta is a self-sufficient and mature dataset format, sometimes it must be additionally configured, and specific table options, constraints, and metadata to be added.</p><p>While all that is possible via SQL DDL commands, it is just not handy to use at scale. Therefore, Python abstractions come into the picture with the hope that, eventually, something similar will be a part of the official Delta-Spark package.</p><p>As of now, DeltaTableConfig fills the missing functionality and helps with managing delta properties and constraints in a Pythonic way and at scale.</p><p>Thank you for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b581ee04e01" width="1" height="1" alt=""><hr><p><a href="https://medium.com/plumbersofdatascience/delta-properties-and-check-constraints-at-scale-2b581ee04e01">Delta Properties and Check Constraints at Scale</a> was originally published in <a href="https://medium.com/plumbersofdatascience">Plumbers Of Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Towards Databricks Certified Data Engineer Professional]]></title>
            <link>https://medium.com/plumbersofdatascience/towards-databricks-certified-data-engineer-professional-5fdc81ff12e6?source=rss-cb832394c0a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fdc81ff12e6</guid>
            <category><![CDATA[data-engineer-skills]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[databricks-certification]]></category>
            <dc:creator><![CDATA[Alex Volok]]></dc:creator>
            <pubDate>Fri, 12 May 2023 15:32:48 GMT</pubDate>
            <atom:updated>2023-05-12T15:32:48.745Z</atom:updated>
            <content:encoded><![CDATA[<h4>A compilation of observations and tips to achieve this credential</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eE1R6ivteL9jY_is" /><figcaption>Photo by <a href="https://unsplash.com/@mayamaceka?utm_source=medium&amp;utm_medium=referral">Maya Maceka</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I decided to obtain the Databricks Certified Data Engineer Professional recognition within 2023. It was a New Year Resolution goal that resonated well with my career path. Surprisingly, few related study materials, blog posts, or forum discussions were available. I passed the exam on the first try, though, going blindly. This story is a compilation of memories and bits of advice on how to get ready and perform optimally during the exam.</p><h3>How was it different from Databricks Certified Data Engineer Associate examination experience?</h3><p>This is the common question I’ve heard recently, and it makes sense because the first logical step before passing the Professional examination is to complete an Associate’s one.</p><p>The Databricks Data Engineer Associate certification is more accessible due to various reasons:</p><ul><li>The difficulty and deepness of the questions.</li><li>It has fewer questions (however, also equally, the duration is shorter).</li><li>The available free <a href="https://files.training.databricks.com/assessments/practice-exams/PracticeExam-DataEngineerAssociate.pdf?_gl=1*r7qtic*_gcl_aw*R0NMLjE2ODI0MjExODEuQ2p3S0NBanc5SjJpQmhCUEVpd0FFcndwZVV5YXZCcXVlcnNBMnVvMVRPVUQzQ3NxU04yd3RyUlo0dlg3QjhmcGpUTTM0UGk2cGZxWUtCb0NUQ01RQXZEX0J3RQ..&amp;_ga=2.186278377.1307112911.1683537318-206e399d-5801-4da2-8aa3-f87273b716e2&amp;_gac=1.82508516.1682421181.CjwKCAjw9J2iBhBPEiwAErwpeUyavBquersA2uo1TOUD3CsqSN2wtrRZ4vX7B8fpjTM34Pi6pfqYKBoCTCMQAvD_BwE">practice test</a> precisely represents the difficulty of the exam so that an adequate self-estimation can be accomplished ahead.</li></ul><p>That said, the candidate with six months of daily Databricks usage may be ready for the associate-level testing. Getting the associate-level certification is a wise step before running professional accreditation.</p><h3>Getting ready for the exam</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oi0pAMhudA8ZlkcJ" /><figcaption>Photo by <a href="https://unsplash.com/@imkirk?utm_source=medium&amp;utm_medium=referral">Iewek Gnos</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><strong>Get</strong> sufficient <strong>Databricks experience</strong></h4><p>I would advise considering professional certification if you spent with Databricks (or Spark) at least 18–24 months and already had a chance for hands-on experience with things like Structured Streaming, Jobs APIs, Compute Pools, and the choice between Delta Merge vs. DataFrame.Write is not an issue.</p><h4><strong>Schedule the right exam time</strong></h4><p>Each person has (his/her own) optimal alerting time. This is the daily repeating period of complete focus and cognitive performance. Mostly this is a time when it is easy for you to execute linear tasks sequentially. In my case, this period is the morning hours supplemented by enough caffeine and a night of good sleep.</p><p>I highly recommend scheduling the exam at this time and avoiding hours of energy deep or evening hours when your body and brain are ready for sleep. This is going to be especially important for such an intense exam.</p><h4><strong>Prepare the workspace</strong></h4><p>If you schedule the online exam, ensure that nothing will interrupt you.</p><p>You will not be able to use noise-canceling headphones, so choose a quiet room and close the windows to avoid potential noise from outside.</p><p>It might be that just before the start of the exam, the environment is ambient, you hear only birds from the backyard, and nothing can go wrong. However, in half an hour, the situation might change, and road workers started fixing something nearby. You cannot leave the seat during the exam because your webcam tracks you. If that happens and you will stand up for a walk to the other side of the room, for instance, to close the door or window, be ready that exam might be canceled.</p><p>For a few hours, consider keeping your family members, pets, and other home inhabitants away from the examination room. All you need is a complete focus during that time.</p><h3>During the exam</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gRRnLD0dV1vUTtKd" /><figcaption>Photo by <a href="https://unsplash.com/@alexandrelion?utm_source=medium&amp;utm_medium=referral">Alex Lion</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><strong>Time management</strong></h4><p>This is probably the main advice — do not get stuck on complex questions. Some of them will take plenty of time even to read. If you get into such one and do not have a quick answer, mark it for a later review and skip it.</p><h4><strong>Evaluate all available answers</strong></h4><p>Because of the time pressure, I tended to choose the “good enough” answer to win some time. Luckily, I quickly realized that there were better strategies than this. The answers are often very similar and look correct at first glance, but this exam intends to determine if you understand those more minor details hidden behind each variant. Therefore, scanning and evaluating each answer before moving to the next question is better.</p><h4><strong>Keep the most challenging questions for later</strong></h4><p>This examen has 60 questions and only 2 minutes per each to read, understand, evaluate all answers, and then choose the proper one. After answering all “doable” questions, you will likely have some time in reserve. Use it for challenging items.</p><p>The hint to the correct answer might be a part of sub-sequential questions.</p><h3>Exam topics to put an extra attention</h3><p>I do not think that I am allowed to disclose exact exam questions and answers. However, I believe I can share a general notion of the exam. The questions were mainly constructed to test the practical knowledge and understanding of the data platform.</p><p>In my case, the following few categories were dominant. Also, in contrast to the Associate exam, this time, I didn&#39;t get any Delta Live Table questions.</p><h4><strong>Structured Streaming:</strong></h4><ul><li>Types of triggers: <em>availableNow</em>, <em>Once</em>, <em>processingTime</em>.</li><li>Watermarking and processing windows</li><li>Role of streaming checkpoints</li></ul><h4>Processing Strategies:</h4><ul><li>Scenarios to use Delta Merge</li><li>Cases when DataFrame.Write is optimal</li><li>The order of operations and the impact on deduplication</li><li>The logical outcome of the chain of transformations on the DataFrame</li></ul><h4><strong>Databricks Management and Automation:</strong></h4><ul><li>The use of various rest APIs. For instance, what will happen if <em>Create Job API</em> runs three times with the same payload?</li><li>The use of Compute Pools</li><li>User Access management and privileges, especially in the light of the Unity Catalog</li><li>Integration with the source control using Git Repos</li><li>Databricks SQL Alerts and Dashboards</li></ul><h3>Final Words</h3><p>The Databricks Certified Data Engineer Professional is a challenging exam. It is also a fair one. It touches perhaps all aspects of the Databricks platform and checks your knowledge and understanding of the optimal use of the components in various situations. So I wrote this article to put all the memories together. I hope someone will find it helpful and actionable and eventually experience the same satisfaction and relief by observing the final score page. Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fdc81ff12e6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/plumbersofdatascience/towards-databricks-certified-data-engineer-professional-5fdc81ff12e6">Towards Databricks Certified Data Engineer Professional</a> was originally published in <a href="https://medium.com/plumbersofdatascience">Plumbers Of Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delta-RS and DuckDB — Read and Write Delta Without Spark]]></title>
            <link>https://medium.com/better-programming/delta-rs-duckdb-read-and-write-delta-without-spark-c4d3db580b25?source=rss-cb832394c0a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4d3db580b25</guid>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[delta]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Alex Volok]]></dc:creator>
            <pubDate>Wed, 26 Apr 2023 06:37:15 GMT</pubDate>
            <atom:updated>2023-04-26T12:35:29.182Z</atom:updated>
            <content:encoded><![CDATA[<h4>An alternative toolset to Spark for low-latency operations with small to medium-size datasets</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/0*fXLscVXEAzffSVGf.jpg" /><figcaption>Figure 1: Image generated by DALL-E</figcaption></figure><p>I have used Apache Spark (often as Azure Databricks) for some years and see it as a sledgehammer in data processing. It is a reliable tool built on JVM, which does in-memory processing and can spin up multiple workers to distribute workload to handle various use cases. It does not matter: whether small or considerable datasets to process; Spark does a job and has a reputation as a de-facto standard processing engine for running Data Lakehouses.</p><p>There is an alternative to Java, Scala, and JVM, though. Open-source libraries like delta-rs, duckdb, pyarrow, and polars written in more performant languages. These newcomers can act as the performant option in specific scenarios like low-latency ETLs on small to medium-size datasets, data exploration, etc.</p><p>This article is a form of POC exploration with a bit of benchmark to see what else is currently achievable outside of spark.</p><h3>So why Delta-RS and DuckDB?</h3><ul><li><strong>Delta-RS</strong> library provides low-level access to Delta tables in Rust, which data teams can use in data processing frameworks. It also provides bindings to another higher-level language — <a href="https://delta-io.github.io/delta-rs/python/">Python</a>.</li><li><strong>DuckDB</strong> is an open-source in-process SQL OLAP database management system.</li></ul><p>The solution is a combination of those two libraries with Apache Arrow as the columnar in-memory format:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/0*9rn-q1s6ZNtU7zfR.png" /><figcaption>Figure 2: Data processing high-level</figcaption></figure><h3>Prepare a clean environment.</h3><p>First thing first, let’s prepare the clean Python environment.</p><p>I am going to run the code on Apple M2; however, all snippets must be compatible with Linux and WSL of Windows 10/11:</p><pre>mkdir delta-rs<br>cd delta-rs<br>python3.10 -m venv .venv<br>source .venv/bin/activate<br>touch delta-rs-exploration.ipynb<br>code .</pre><p>When VS Code is launched, don’t forget to choose a recently created Python environment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vjwf3ft-mF7AXZym.png" /><figcaption>Figure 3: Select the Python environment</figcaption></figure><p>That’s it, the sandbox is ready, and we are good to go.</p><h3>Generate Sample Dataset using PySpark.</h3><p>We must generate sample data first to proceed with delta-rs testing. So let’s first install pyspark and dbldatagen. The last one is a Databricks Labs project and data generation tool.</p><pre>%pip install pyspark==3.3.1<br>%pip install dbldatagen==0.3.4<br>%pip install pyparsing==3.0.9</pre><p>The next step is to initialize spark. The following code will run spark in the minimalistic version to avoid UI overhead:</p><pre>from pyspark.sql import SparkSession<br>spark_jars = (<br>    &quot;org.apache.spark:spark-avro_2.12:3.2.1&quot;<br>    &quot;,io.delta:delta-core_2.12:2.1.0&quot;<br>    &quot;,com.databricks:spark-xml_2.12:0.14.0&quot;<br>)<br>spark = (<br>    SparkSession.builder.master(&quot;local[*]&quot;)<br>    .appName(&quot;MyApp&quot;)<br>    .config(&quot;spark.jars.packages&quot;, spark_jars)<br>    .config(&quot;spark.sql.extensions&quot;, &quot;io.delta.sql.DeltaSparkSessionExtension&quot;)<br>    .config(<br>        &quot;spark.sql.catalog.spark_catalog&quot;,<br>        &quot;org.apache.spark.sql.delta.catalog.DeltaCatalog&quot;,<br>    )    <br>    .config(&quot;spark.ui.showConsoleProgress&quot;, &quot;false&quot;)<br>    .config(&quot;spark.ui.enabled&quot;, &quot;false&quot;)<br>    .config(&quot;spark.ui.dagGraph.retainedRootRDDs&quot;, &quot;1&quot;)<br>    .config(&quot;spark.ui.retainedJobs&quot;, &quot;1&quot;)<br>    .config(&quot;spark.ui.retainedStages&quot;, &quot;1&quot;)<br>    .config(&quot;spark.ui.retainedTasks&quot;, &quot;1&quot;)<br>    .config(&quot;spark.sql.ui.retainedExecutions&quot;, &quot;1&quot;)<br>    .config(&quot;spark.worker.ui.retainedExecutors&quot;, &quot;1&quot;)<br>    .config(&quot;spark.worker.ui.retainedDrivers&quot;, &quot;1&quot;)<br>    .config(&quot;spark.driver.memory&quot;, &quot;16g&quot;) <br>).getOrCreate()</pre><p>Since spark must be already installed and initialized by now, it is time to generate a simple dataframe with ten million rows of sample data using dbldatagen</p><pre>import dbldatagen as dg<br>from pyspark.sql.types import IntegerType<br>row_count = 10 * 1000 * 1000<br>test_data_spec = (<br>    dg.DataGenerator( spark, name=&quot;test_data_set1&quot;, rows=row_count, partitions=4, <br>                      randomSeedMethod=&quot;hash_fieldname&quot;, verbose=True, )<br>    .withColumn(&quot;purchase_id&quot;, IntegerType(), minValue=1000000, maxValue=2000000)<br>    .withColumn(&quot;product_code&quot;, IntegerType(), uniqueValues=10000, random=True)<br>    .withColumn(<br>        &quot;purchase_date&quot;,<br>        &quot;date&quot;,<br>        data_range=dg.DateRange(&quot;2017-10-01 00:00:00&quot;, &quot;2018-10-06 11:55:00&quot;, &quot;days=3&quot;),<br>        random=True,<br>    )<br>    .withColumn(<br>        &quot;return_date&quot;,<br>        &quot;date&quot;,<br>        expr=&quot;date_add(purchase_date, cast(floor(rand() * 100 + 1) as int))&quot;,<br>        baseColumn=&quot;purchase_date&quot;,<br>    )<br>    .withColumn(&quot;name&quot;, &quot;string&quot;, percentNulls=0.01, template=r&#39;\\\\w \\\\w|\\\\w A. \\\\w|test&#39;)<br>    .withColumn(&quot;emails&quot;, &quot;string&quot;, template=r&#39;\\\\w.\\\\w@\\\\w.com&#39;, random=True,<br>                 numFeatures=(1, 6), structType=&quot;array&quot;)<br>)<br>df_test_data = test_data_spec.build()</pre><p>And then check the dataframe content before materializing it:</p><pre># returns: 10.0<br>df_test_data.count() / 1000_000<br>df_test_data.limit(5).show(truncate=True)<br><br>&quot;&quot;&quot;<br>+-----------+------------+-------------+-----------+--------------------+--------------------+<br>|purchase_id|product_code|purchase_date|return_date|                name|              emails|<br>+-----------+------------+-------------+-----------+--------------------+--------------------+<br>|    1000000|        7503|   2017-10-07| 2017-11-28|      pariatur F. ut|[elit.sunt@veniam...|<br>|    1000001|        7740|   2018-04-02| 2018-04-12|          ut G. quis|[do.eu@laborum.co...|<br>|    1000002|        6891|   2018-03-14| 2018-03-22|anim H. reprehend...|[irure.aute@velit...|<br>|    1000003|        3751|   2017-12-26| 2018-02-25|                test|[nostrud.adipisci...|<br>|    1000004|        8015|   2017-11-05| 2017-12-07|         qui laboris|[aute.voluptate@n...|<br>+-----------+------------+-------------+-----------+--------------------+--------------------+<br>&quot;&quot;&quot;<br></pre><p>Save it as a delta dataset (generated by pyspark):</p><pre>df_test_data.write.format(&quot;delta&quot;).mode(&quot;overwrite&quot;).save(&quot;delta/sample_dataset&quot;)</pre><p>For the sake of the performance comparison between PySpark and DuckDB, I will read it back from Delta and then run a sample analytics query:</p><pre>df_read_delta = spark.read.format(&quot;delta&quot;).load(&quot;delta/sample_dataset&quot;)<br>df_read_delta.createOrReplaceTempView(&quot;sample_dataset&quot;)<br>query =  &quot;&quot;&quot; <br>select <br>  purchase_date,<br>  count(distinct purchase_id) as purchases,<br>  count(emails) as emails<br>from sample_dataset<br>group by purchase_date<br>order by 2 desc<br>limit 5<br>&quot;&quot;&quot; <br>spark.sql(query).show()<br>&quot;&quot;&quot;<br>output:<br>+-------------+---------+------+<br>|purchase_date|purchases|emails|<br>+-------------+---------+------+<br>|   2018-08-21|    79148| 82085|<br>|   2018-08-12|    79132| 82024|<br>|   2018-02-03|    78976| 81911|<br>|   2018-04-29|    78913| 81813|<br>|   2018-05-23|    78897| 81849|<br>+-------------+---------+------+<br>&quot;&quot;&quot;</pre><p>The query execution duration was <strong>4.5 seconds</strong> on my laptop, and I will execute this query eventually, also using DuckDB, to match the performance of data engines.</p><h3>Run a query on a sample dataset using Delta-RS and DuckDB.</h3><p>Firstly, let’s add missing libraries. Delta-RS currently has two implementations: in Python and Rust.</p><p>Because I continue exploring a Python version, I have to install and import a package deltalake: It also installs dependencies like pyarrow and numpy if they are missing. In the same step, I include pandas and duckdb required for querying and data manipulation.</p><pre>%pip install deltalake==0.8.1<br>%pip install pandas=1.5.0<br>%pip install duckdb==0.7.1</pre><p>Let’s check if we can use delta-rs to reveal the dataset’s history generated by PySpark</p><pre>from deltalake import DeltaTable<br>dt = DeltaTable(&quot;delta/sample_dataset&quot;)<br>dt.history()<br>&quot;&quot;&quot;<br>[{&#39;timestamp&#39;: 1682098489708,<br>  &#39;operation&#39;: &#39;WRITE&#39;,<br>  &#39;operationParameters&#39;: {&#39;mode&#39;: &#39;Overwrite&#39;, &#39;partitionBy&#39;: &#39;[]&#39;},<br>  &#39;isolationLevel&#39;: &#39;Serializable&#39;,<br>  &#39;isBlindAppend&#39;: False,<br>  &#39;engineInfo&#39;: &#39;Apache-Spark/3.3.1 Delta-Lake/2.1.0&#39;,<br>  &#39;operationMetrics&#39;: {&#39;numFiles&#39;: &#39;4&#39;,<br>   &#39;numOutputBytes&#39;: &#39;480488931&#39;,<br>   &#39;numOutputRows&#39;: &#39;10000000&#39;},<br>  &#39;txnId&#39;: &#39;ee54da7d-4dad-4652-b24f-813d6cf5633f&#39;}]{&#39;type&#39;: &#39;struct&#39;,<br> &#39;fields&#39;: [{&#39;name&#39;: &#39;purchase_id&#39;,<br>   &#39;type&#39;: &#39;integer&#39;,<br>   &#39;nullable&#39;: True,<br>   &#39;metadata&#39;: {}},<br>  {&#39;name&#39;: &#39;product_code&#39;,<br>   &#39;type&#39;: &#39;integer&#39;,<br>   &#39;nullable&#39;: True,<br>   &#39;metadata&#39;: {}},<br>  {&#39;name&#39;: &#39;purchase_date&#39;, &#39;type&#39;: &#39;date&#39;, &#39;nullable&#39;: True, &#39;metadata&#39;: {}},<br>  {&#39;name&#39;: &#39;return_date&#39;, &#39;type&#39;: &#39;date&#39;, &#39;nullable&#39;: True, &#39;metadata&#39;: {}},<br>  {&#39;name&#39;: &#39;name&#39;, &#39;type&#39;: &#39;string&#39;, &#39;nullable&#39;: True, &#39;metadata&#39;: {}},<br>  {&#39;name&#39;: &#39;emails&#39;,<br>   &#39;type&#39;: {&#39;type&#39;: &#39;array&#39;, &#39;elementType&#39;: &#39;string&#39;, &#39;containsNull&#39;: True},<br>   &#39;nullable&#39;: True,<br>   &#39;metadata&#39;: {}}]}<br>&quot;&quot;&quot;</pre><p>The library also returns schema, while handy Spark’s df.printSchema() is indeed missing:</p><pre>dt.schema().json()<br>&quot;&quot;&quot;<br>{&#39;type&#39;: &#39;struct&#39;,<br> &#39;fields&#39;: [{&#39;name&#39;: &#39;purchase_id&#39;,<br>   &#39;type&#39;: &#39;integer&#39;,<br>   &#39;nullable&#39;: True,<br>   &#39;metadata&#39;: {}},<br>  {&#39;name&#39;: &#39;product_code&#39;,<br>   &#39;type&#39;: &#39;integer&#39;,<br>   &#39;nullable&#39;: True,<br>   &#39;metadata&#39;: {}},<br>  {&#39;name&#39;: &#39;purchase_date&#39;, &#39;type&#39;: &#39;date&#39;, &#39;nullable&#39;: True, &#39;metadata&#39;: {}},<br>  {&#39;name&#39;: &#39;return_date&#39;, &#39;type&#39;: &#39;date&#39;, &#39;nullable&#39;: True, &#39;metadata&#39;: {}},<br>  {&#39;name&#39;: &#39;name&#39;, &#39;type&#39;: &#39;string&#39;, &#39;nullable&#39;: True, &#39;metadata&#39;: {}},<br>  {&#39;name&#39;: &#39;emails&#39;,<br>   &#39;type&#39;: {&#39;type&#39;: &#39;array&#39;, &#39;elementType&#39;: &#39;string&#39;, &#39;containsNull&#39;: True},<br>   &#39;nullable&#39;: True,<br>   &#39;metadata&#39;: {}}]}<br>&quot;&quot;&quot;</pre><p>Let’s prepare the dataset for querying:</p><pre>import duckdb<br>pyarrow_dataset = dt.to_pyarrow_dataset()<br>sample_dataset = duckdb.arrow(pyarrow_dataset)</pre><p>And then eventually query it using the Spark-like syntax of DuckDB:</p><pre>sample_dataset.filter(&quot;purchase_id = 1000000&quot;).project(&quot;purchase_id, product_code, name&quot;)<br>&quot;&quot;&quot;<br>┌─────────────┬──────────────┬────────────────────┐<br>│ purchase_id │ product_code │        name        │<br>│    int32    │    int32     │      varchar       │<br>├─────────────┼──────────────┼────────────────────┤<br>│     1000000 │         7503 │ test               │<br>│     1000000 │         9981 │ aliquip M. eiusmod │<br>│     1000000 │         5467 │ nisi Z. quis       │<br>│     1000000 │         9854 │ test               │<br>│     1000000 │         7499 │ test               │<br>│     1000000 │         4288 │ test               │<br>│     1000000 │         9674 │ NULL               │<br>│     1000000 │         3593 │ ipsum laboris      │<br>│     1000000 │         4885 │ fugiat C. commodo  │<br>│     1000000 │         2343 │ commodo H. in      │<br>├─────────────┴──────────────┴────────────────────┤<br>│ 10 rows                               3 columns │<br>└─────────────────────────────────────────────────┘<br>&quot;&quot;&quot;</pre><p>The next step is to run the analytics query that I used earlier in this article:</p><pre>query =  &quot;&quot;&quot; <br>select <br>  purchase_date,<br>  count(distinct purchase_id) as purchases,<br>  count(emails) as emails<br>from sample_dataset<br>group by purchase_date<br>order by 2 desc<br>limit 5<br>&quot;&quot;&quot; <br>duckdb.query(query)<br>&quot;&quot;&quot;<br>┌───────────────┬───────────┬────────┐<br>│ purchase_date │ purchases │ emails │<br>│     date      │   int64   │ int64  │<br>├───────────────┼───────────┼────────┤<br>│ 2018-08-21    │     79148 │  82085 │<br>│ 2018-08-12    │     79132 │  82024 │<br>│ 2018-02-03    │     78976 │  81911 │<br>│ 2018-04-29    │     78913 │  81813 │<br>│ 2018-05-23    │     78897 │  81849 │<br>└───────────────┴───────────┴────────┘<br>&quot;&quot;&quot;</pre><p>It was finished in 0.7 seconds in DuckDB versus 4.5 seconds in PySpark.</p><p>The query output can also be materialized to the pandas dataframe:</p><pre>duckdb.query(query).to_df()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/0*w79h7usWYPIx7YiU.png" /><figcaption>Figure 4: Query result as pandas dataframe</figcaption></figure><h3>Create new datasets using Delta-RS and DuckDB.</h3><p>Let’s now write the outcome of the query back to the delta lake and check if the dataset exists:</p><pre>from deltalake.writer import write_deltalake<br>arrow_table = duckdb.query(query).to_arrow_table()<br>write_deltalake(<br>    data=arrow_table, <br>    table_or_uri=&quot;delta/demo_dataset_groupby_rust&quot;, <br>    mode=&quot;overwrite&quot;,<br>    overwrite_schema=True<br>)<br>DeltaTable(&quot;delta/demo_dataset_groupby_rust&quot;).history()<br>&quot;&quot;&quot;<br>[{&#39;timestamp&#39;: 1682098595580, &#39;delta-rs&#39;: &#39;0.8.0&#39;},<br> {&#39;timestamp&#39;: 1682098622494,<br>  &#39;operation&#39;: &#39;WRITE&#39;,<br>  &#39;operationParameters&#39;: {&#39;partitionBy&#39;: &#39;[]&#39;, &#39;mode&#39;: &#39;Overwrite&#39;},<br>  &#39;clientVersion&#39;: &#39;delta-rs.0.8.0&#39;}]<br>&quot;&quot;&quot;</pre><h3>Check the dataset generated by Delta-RS in PySpark.</h3><p>As the final exploration, I ensure that PySpark can read datasets generated outside and that there are no computability issues:</p><pre>spark.read.format(&quot;delta&quot;).load(&quot;delta/demo_dataset_groupby_rust&quot;).show()<br>&quot;&quot;&quot;<br>results:<br>+-------------+---------+------+<br>|purchase_date|purchases|emails|<br>+-------------+---------+------+<br>|   2018-08-21|    79148| 82085|<br>|   2018-08-12|    79132| 82024|<br>|   2018-02-03|    78976| 81911|<br>|   2018-04-29|    78913| 81813|<br>|   2018-05-23|    78897| 81849|<br>+-------------+---------+------+<br>&quot;&quot;&quot;</pre><p>Luckily, this is not the case. Spark can still use delta objects generated outside.</p><p>Let’s also check how Pyspark can read metadata of history created by the package outside. This time I install spark’s version of delta:</p><pre>%pip install delta-spark</pre><p>I noticed that not all fields of the delta history formed by the side delta writer are available in the PySpark:</p><pre>from delta import DeltaTable<br>(<br>    DeltaTable<br>    .forPath(path=&quot;delta/demo_dataset_groupby_rust&quot;, sparkSession=spark)<br>    .history()<br>    .show()<br>)<br>&quot;&quot;&quot;<br>+-------+--------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+----------------+------------+----------+<br>|version|           timestamp|userId|userName|operation| operationParameters| job|notebook|clusterId|readVersion|isolationLevel|isBlindAppend|operationMetrics|userMetadata|engineInfo|<br>+-------+--------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+----------------+------------+----------+<br>|      2|2023-04-21 19:37:...|  null|    null|    WRITE|{partitionBy -&gt; [...|null|    null|     null|       null|          null|         null|            null|        null|      null|<br>|      1|2023-04-21 19:37:...|  null|    null|    WRITE|{mode -&gt; Overwrit...|null|    null|     null|       null|          null|         null|            null|        null|      null|<br>|      0|2023-04-21 19:36:...|  null|    null|     null|                null|null|    null|     null|       null|          null|         null|            null|        null|      null|<br>+-------+--------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+----------------+------------+----------+<br>&quot;&quot;&quot;</pre><p>So only essential columns are filled: <em>version</em>, <em>timestamp</em>, <em>operation</em>, and operationParameters.</p><h3>Some potential use cases</h3><p>Delta-RS is a relatively young and actively developed library; however, it can already be a helpful addition to the tool stack of the data teams:</p><ul><li><strong>Data Science:</strong> The discussed combination of libraries might benefit data exploration because it offers an SQL interface and provides a more robust response with less involved resources.</li><li><strong>Data Engineering:</strong> Low-latency ETLs on small to medium-size datasets.</li><li><strong>Cost Saving:</strong> Reading and writing Delta Lake datasets in Python without the need for spark, which could be helpful for smaller teams or projects with limited resources.</li></ul><h3>Conclusion</h3><p>Delta-RS and DuckDB offer an alternative to spark for querying and writing Delta Lake datasets.</p><p>With the ability to run analytics queries and create new datasets, these libraries can be a useful addition to a data team’s toolbox.</p><p>Furthermore, they can provide cost savings for smaller teams or projects with limited resources.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4d3db580b25" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/delta-rs-duckdb-read-and-write-delta-without-spark-c4d3db580b25">Delta-RS and DuckDB — Read and Write Delta Without Spark</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Observability: Processing Collected Ganglia Metrics]]></title>
            <link>https://medium.com/@alexandr.volok/databricks-processing-collected-ganglia-metrics-25852d49b758?source=rss-cb832394c0a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/25852d49b758</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[ganglia]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Alex Volok]]></dc:creator>
            <pubDate>Sun, 16 Apr 2023 07:12:29 GMT</pubDate>
            <atom:updated>2023-04-16T07:12:29.780Z</atom:updated>
            <content:encoded><![CDATA[<h3>Databricks Observability: Processing Collected Ganglia Metric</h3><h3>Introduction</h3><p>In this article, we will explore the use of collected Ganglia metrics and their preparation for observability tools like Grafana. We will begin by examining how the metrics files are organized and what kind of content they hold. Then, we will discuss how the metrics going to be processed by creating a notebook to ingest XML files. Finally, we will push the metrics to Azure Data Explorer.</p><p>This article depends on the previous story: <a href="https://medium.com/@alexandr.volok/databricks-observability-collecting-ganglia-metrics-d2848d5944ee">Databricks Observability: Collecting Ganglia Metrics</a> and uses the outcome of the init-script discussed there.</p><h3>The processing flow</h3><p>Before the implementation part starts let’s briefly discuss the logic behind the processing.</p><p>In a nutshell, it has a few steps:</p><ul><li>Move logs from the landing to the processing directory</li><li>Generate Pyspark dataframe by processing xml logs</li><li>Push the content of the dataframe to Azure Data Explorer</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/0*Lz4CoXfBTggHM5dj.png" /><figcaption>Figure 1: The processing flow</figcaption></figure><p>Each of these steps will be covered later in the implementation part.</p><h3>Folder structure and the metrics content</h3><p>The init script generates metrics using the following structure:</p><pre>/dbfs/logs/ganglia_metrics/in/&lt;cluster-name&gt;/&lt;driver_ip&gt;_&lt;timestamp&gt;.xml</pre><p>Each cluster has its own directory and the filename of the log contains a driver IP as the first token. Driver’s IP is needed to separate such kind of host from workers since ganglia do not distinguish them:</p><pre>&lt;GANGLIA_XML VERSION=&quot;3.6.0&quot; SOURCE=&quot;gmetad&quot;&gt;<br>    &lt;GRID NAME=&quot;unspecified&quot; AUTHORITY=&quot;&lt;http://0411-080835-6k8t703k-172-16-0-5/ganglia/&gt;&quot;<br>        LOCALTIME=&quot;1681494185&quot;&gt;<br>        &lt;CLUSTER NAME=&quot;cluster&quot; LOCALTIME=&quot;1681494184&quot; OWNER=&quot;unspecified&quot; LATLONG=&quot;unspecified&quot;<br>            URL=&quot;unspecified&quot;&gt;<br>            &lt;HOST NAME=&quot;172.16.0.4&quot; IP=&quot;172.16.0.4&quot; REPORTED=&quot;1681494182&quot; TN=&quot;3&quot; TMAX=&quot;20&quot; DMAX=&quot;0&quot;<br>                LOCATION=&quot;unspecified&quot; GMOND_STARTED=&quot;1681493542&quot; TAGS=&quot;&quot;&gt;<br>                &lt;METRIC NAME=&quot;cpu_num&quot; VAL=&quot;4&quot; TYPE=&quot;uint16&quot; UNITS=&quot;CPUs&quot; TN=&quot;43&quot; TMAX=&quot;1200&quot;<br>                    DMAX=&quot;0&quot; SLOPE=&quot;zero&quot; SOURCE=&quot;gmond&quot;&gt;<br>                    &lt;EXTRA_DATA&gt;<br>                        &lt;EXTRA_ELEMENT NAME=&quot;GROUP&quot; VAL=&quot;cpu&quot; /&gt;<br>                        &lt;EXTRA_ELEMENT NAME=&quot;DESC&quot; VAL=&quot;Total number of CPUs&quot; /&gt;<br>                        &lt;EXTRA_ELEMENT NAME=&quot;TITLE&quot; VAL=&quot;CPU Count&quot; /&gt;<br>                    &lt;/EXTRA_DATA&gt;<br>                &lt;/METRIC&gt;<br>            &lt;/HOST&gt;<br>            &lt;HOST NAME=&quot;172.16.0.5&quot; IP=&quot;172.16.0.5&quot; REPORTED=&quot;1681494177&quot; TN=&quot;8&quot; TMAX=&quot;20&quot; DMAX=&quot;0&quot;<br>                LOCATION=&quot;unspecified&quot; GMOND_STARTED=&quot;1681493528&quot; TAGS=&quot;&quot;&gt;<br>                &lt;METRIC NAME=&quot;cpu_num&quot; VAL=&quot;4&quot; TYPE=&quot;uint16&quot; UNITS=&quot;CPUs&quot; TN=&quot;56&quot; TMAX=&quot;1200&quot;<br>                    DMAX=&quot;0&quot; SLOPE=&quot;zero&quot; SOURCE=&quot;gmond&quot;&gt;<br>                    &lt;EXTRA_DATA&gt;<br>                        &lt;EXTRA_ELEMENT NAME=&quot;GROUP&quot; VAL=&quot;cpu&quot; /&gt;<br>                        &lt;EXTRA_ELEMENT NAME=&quot;DESC&quot; VAL=&quot;Total number of CPUs&quot; /&gt;<br>                        &lt;EXTRA_ELEMENT NAME=&quot;TITLE&quot; VAL=&quot;CPU Count&quot; /&gt;<br>                    &lt;/EXTRA_DATA&gt;<br>                &lt;/METRIC&gt;<br>            &lt;/HOST&gt;<br>        &lt;/CLUSTER&gt;<br>    &lt;/GRID&gt;<br>&lt;/GANGLIA_XML&gt;</pre><p>Each file holds multiple metrics of the cluster, however one xml payload per moment in time.</p><h3>Implementation of the processing logic</h3><h4>Step 1: Install extra libraries</h4><p>Because gathered metrics are xml payloads and the destination target is Azure Data Explorer, the following library must be installed:</p><ul><li><em>Maven: com.databricks:spark-xml_2.12:0.16.0</em></li><li><em>PyPI: azure-kusto-data</em></li><li><em>PyPI: azure-kusto-ingest</em></li></ul><p>This can be done manually by adjusting cluster settings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/0*eCjOOoJK81aOplKF.png" /><figcaption>Figure 2: Installed custom libraries</figcaption></figure><p>If the cluster creation is managed by Terraform the missing module can be added using an attribute library in the scope of a cluster:</p><pre>...<br>library {<br>      maven {<br>        coordinates = &quot;com.databricks:spark-xml_2.12:0.16.0&quot;<br>      }<br>    }<br>library {<br>      pypi {<br>        package = &quot;azure-kusto-data&quot;<br>      }<br>}<br>library {<br>      pypi {<br>        package = &quot;azure-kusto-ingest&quot;<br>      }<br>}<br>...</pre><h4>Step 2: Move logs to the processing folder</h4><p>The following snippet moves files from /logs/ganglia_metrics/in to /logs/ganglia_metrics/processing. It terminates the notebook if the root ganglia_metrics folder does not exist or there are no input logs to process.</p><pre>path_ganglia_root = &quot;/logs/ganglia_metrics&quot;<br>path_ganglia_input = f&quot;{path_ganglia_root}/in&quot;<br>path_ganglia_processing = f&quot;{path_ganglia_root}/processing&quot;<br>path_ganglia_processing_fetch_pattern = f&quot;{path_ganglia_processing}/*/*&quot;<br><br># COMMAND ----------<br><br># Check if ganglia root folder even created<br># and if not then terminates the notebook flow<br>try:<br>    dbutils.fs.ls(path_ganglia_root)<br>except Exception as e:<br>    print(e)<br>    dbutils.notebook.exit(&quot;Ganglia metrics gathering is not enabled on any cluster in this opco, terminating...&quot;)<br><br># Check if the metrics files available in the input folder<br>if not &quot;in/&quot; in [x.name for x in dbutils.fs.ls(path_ganglia_root)]:<br>    dbutils.notebook.exit(&quot;No input to process, terminating...&quot;)<br><br># COMMAND ----------<br><br># Moves incoming logs to the processing folder to isolate processing files from other processes<br># 3 tries to move the files<br>try_nr = 1 <br>moved = False<br>while not moved and try_nr &lt;= 3:<br>    print(f&quot;Moving files to processing folder. Try {try_nr}&quot;)<br>    try:<br>        dbutils.fs.mv(path_ganglia_input, path_ganglia_processing, True)<br>        moved = True<br>    except Exception as e: # Replace Exception with something more specific.<br>        print(e)<br>        try_nr += 1</pre><h4>Step 3: Generate dataframe by processing moved files</h4><p>The following Pyspark snippet reads metric payloads and extracts clusters and metrics details:</p><pre>from pyspark.sql.functions import input_file_name<br>df = (<br>    spark<br>    .read<br>    .format(&quot;xml&quot;)<br>    .option(&quot;rowTag&quot;, &quot;HOST&quot;)<br>    .load(path_ganglia_processing_fetch_pattern)<br>    .withColumn(&quot;path&quot;, input_file_name())<br>    .selectExpr(<br>        &quot;SPLIT(path, &#39;/&#39;)[4] as cluster&quot;,<br>        &quot;_NAME AS host&quot;,<br>        &quot;METRIC._NAME AS metric&quot;,<br>        &quot;METRIC._VAL AS value&quot;,<br>        &quot;METRIC._UNITS AS units&quot;,<br>        &quot;from_unixtime(_REPORTED) AS timestamp&quot;,<br>        &quot;CASE WHEN SPLIT(SPLIT(path, &#39;/&#39;)[5], &#39;_&#39;)[0] = _NAME THEN &#39;Driver&#39; ELSE &#39;Worker&#39; END AS host_type&quot;<br>    )<br>)</pre><p>The outcome is a tabular flat dataset:</p><pre>df.limit(5).show(truncate=False)</pre><pre>+----------------------------+----------+---------+---------+-----+-------------------+---------+<br>|cluster                     |host      |metric   |value    |units|timestamp          |host_type|<br>+----------------------------+----------+---------+---------+-----+-------------------+---------+<br>|ganglia-metrics-demo-cluster|172.16.0.4|mem_total|9324544.0|KB   |2023-04-14 17:36:42|Worker   |<br>|ganglia-metrics-demo-cluster|172.16.0.5|mem_total|9324544.0|KB   |2023-04-14 17:36:48|Driver   |<br>|ganglia-metrics-demo-cluster|172.16.0.4|mem_free |7675972.0|KB   |2023-04-14 17:36:42|Worker   |<br>|ganglia-metrics-demo-cluster|172.16.0.5|mem_free |3332040.0|KB   |2023-04-14 17:36:48|Driver   |<br>|ganglia-metrics-demo-cluster|172.16.0.4|cpu_idle |95.0     |%    |2023-04-14 17:36:42|Worker   |<br>+----------------------------+----------+---------+---------+-----+-------------------+---------+<br><br></pre><p>The dataframe df might be saved as the delta table. In some scenarios, this should be enough to analyze the health and utilization of the clusters. However, for a better user experience in observability tools like Grafana, the data will be exported to a more robust append-only query engine – Azure Data Explorer.</p><h4>Step 4: Push the content of the dataframe to Azure Data Explorer</h4><p>Warning: this is going the be the longest step since a few prerequisites must be implemented before running the code.</p><p><strong>Prerequisites:</strong></p><ol><li><a href="https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal">Create</a> or re-use existing SPN. <a href="https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret">Create</a> an application secret. The Kusto client requires this for authentication.</li><li><a href="https://learn.microsoft.com/en-us/azure/data-explorer/create-cluster-database-portal">Create</a> an instance of Azure Data Explorer and the database <em>ganglia-metrics</em>.</li><li><a href="https://learn.microsoft.com/en-us/azure/data-explorer/kusto/api/netfx/kusto-ingest-client-permissions">Grant</a> permission <em>Ingestor</em> to the SPN created in step 1 for the database created in step 2:<br>.add database ganglia-metrics ingestors (&#39;aadapp=&lt;Step 1 Application ID&gt;&#39;)</li><li>Run the following DDL script in <em>ganglia-metrics</em> to create a table <em>cluster_usage</em>:<br>.create table cluster_usage ( <br> cluster: string, <br> host: string, <br> metric: string, <br> value: decimal, <br> units: string, <br> timestamp: datetime, <br> host_type:string<br>)</li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes#create-a-databricks-backed-secret-scope">Create</a> databricks secret scope <em>ganglia-observability</em></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secrets#create-a-secret-in-a-databricks-backed-scope">Create</a> the following secrets in the added earlier secret scope:</li></ol><ul><li><em>ingest-connectionstring</em>: Use the Data Ingestion URI of ADX, an example is in a screenshot below</li><li><em>client-id</em>: Application (client) ID</li><li><em>client-secret</em>: The secret string created in Step 1</li><li><em>tenant-id</em>: Directory (tenant) ID</li></ul><p>Data Ingestion URI can be extracted from the Azure Data Explorer overview page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/0*Xt9zfKTmqJXgI71E.png" /><figcaption>Figure 3: ADX endpoints</figcaption></figure><p>As soon as the ADX prerequisites are fulfilled the next step is to execute the following script. It converts spark dataframe into the CSV in a form of a string variable and pushes it asynchronously to Azure Data Explorer using a native Python client QueuedIngestClient:</p><pre>from azure.kusto.data import KustoConnectionStringBuilder<br>from azure.kusto.ingest import QueuedIngestClient<br>from azure.kusto.ingest.ingestion_properties import IngestionProperties, DataFormat<br>from datetime import datetime<br>import io<br><br>kusto_database = &quot;ganglia-metrics&quot;<br>kusto_table = &quot;cluster_usage&quot;<br>secret_scope = &quot;ganglia-observability&quot;<br><br>kusto_ingest_uri = dbutils.secrets.get(secret_scope, &quot;ingest-connectionstring&quot;)<br>client_id = dbutils.secrets.get(secret_scope, &quot;client-id&quot;)<br>client_secret = dbutils.secrets.get(secret_scope, &quot;client-secret&quot;)<br>tenant_id = dbutils.secrets.get(secret_scope, &quot;tenant-id&quot;)<br><br># initialises client to push messages<br>kusto_ingest_client = QueuedIngestClient(<br>    KustoConnectionStringBuilder.with_aad_application_key_authentication(<br>        connection_string=kusto_ingest_uri,<br>        aad_app_id=client_id,<br>        app_key=client_secret,<br>        authority_id=tenant_id<br>    )<br>)<br><br># controls ingestion behavior<br>ingestion_properties = IngestionProperties(<br>    database=kusto_database,<br>    table=kusto_table,<br>    data_format=DataFormat.CSV,<br>    flush_immediately=True,<br>)<br><br># converts the dataframe to csv string<br>adx_csv_output = df.toPandas().to_csv(index=False, header=False,)<br>payload_text = io.StringIO(adx_csv_output)<br><br># pushes the payload message to data explorer<br>kusto_ingest_client.ingest_from_stream(<br>    payload_text, <br>    ingestion_properties<br>)<br><br>processed_rows = adx_csv_output.count(&quot;\\n&quot;)<br>print(f&quot;{processed_rows} rows pushed appended into {kusto_database}.{kusto_table}&quot;)<br><br># COMMAND ----------<br># Final step: cleanup of the processing directory<br>dbutils.fs.rm(path_ganglia_processing, True)</pre><p>If everything went well the target table cluster_usage must contain rows with metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/0*pMBD9Qha8qE86wAp.png" /><figcaption>Figure 4: The content of ADX table ‘cluster_usage’</figcaption></figure><p>In the next articles, I will talk about the usage of the metrics data in Grafana and more detailed overview of the ADX as the observability engine.</p><h3>The processing notebook code put together</h3><p>The following notebook might be scheduled to process data periodically and refresh the target dataset</p><pre># Databricks notebook source<br>path_ganglia_root = &quot;/logs/ganglia_metrics&quot;<br>path_ganglia_input = f&quot;{path_ganglia_root}/in&quot;<br>path_ganglia_processing = f&quot;{path_ganglia_root}/processing&quot;<br>path_ganglia_processing_fetch_pattern = f&quot;{path_ganglia_processing}/*/*&quot;<br><br># COMMAND ----------<br><br># Check if ganglia root folder even created<br># and if not then terminates the notebook flow<br>try:<br>    dbutils.fs.ls(path_ganglia_root)<br>except Exception as e:<br>    print(e)<br>    dbutils.notebook.exit(&quot;Ganglia metrics gathering is not enabled on any cluster in this opco, terminating...&quot;)<br><br># Check if the metrics files available in the input folder<br>if not &quot;in/&quot; in [x.name for x in dbutils.fs.ls(path_ganglia_root)]:<br>    dbutils.notebook.exit(&quot;No input to process, terminating...&quot;)<br><br># COMMAND ----------<br><br># Moves incoming logs to the processing folder to isolate processing files from other processes<br># 3 tries to move the files<br>try_nr = 1 <br>moved = False<br>while not moved and try_nr &lt;= 3:<br>    print(f&quot;Moving files to processing folder. Try {try_nr}&quot;)<br>    try:<br>        dbutils.fs.mv(path_ganglia_input, path_ganglia_processing, True)<br>        moved = True<br>    except Exception as e: # Replace Exception with something more specific.<br>        print(e)<br>        try_nr += 1<br><br># COMMAND ----------<br><br># Ingests xmls using pyspark then converts to pandas to have control over generation of CSV output<br>from pyspark.sql.functions import input_file_name<br>df = (<br>    spark<br>    .read<br>    .format(&quot;xml&quot;)<br>    .option(&quot;rowTag&quot;, &quot;HOST&quot;)<br>    .load(path_ganglia_processing_fetch_pattern)<br>    .withColumn(&quot;path&quot;, input_file_name())<br>    .selectExpr(<br>        &quot;SPLIT(path, &#39;/&#39;)[4] as cluster&quot;,<br>        &quot;_NAME AS host&quot;,<br>        &quot;METRIC._NAME AS metric&quot;,<br>        &quot;METRIC._VAL AS value&quot;,<br>        &quot;METRIC._UNITS AS units&quot;,<br>        &quot;from_unixtime(_REPORTED) AS timestamp&quot;,<br>        &quot;CASE WHEN SPLIT(SPLIT(path, &#39;/&#39;)[5], &#39;_&#39;)[0] = _NAME THEN &#39;Driver&#39; ELSE &#39;Worker&#39; END AS host_type&quot;<br>    )<br>)<br><br># COMMAND ----------<br><br>from azure.kusto.data import KustoConnectionStringBuilder<br>from azure.kusto.ingest import QueuedIngestClient<br>from azure.kusto.ingest.ingestion_properties import IngestionProperties, DataFormat<br>from datetime import datetime<br>import io<br><br>kusto_database = &quot;ganglia-metrics&quot;<br>kusto_table = &quot;cluster_usage&quot;<br><br># to establish a connection with a data explorer<br>secret_scope = &quot;ganglia-observability&quot;<br>kusto_ingest_uri = dbutils.secrets.get(secret_scope, &quot;ingest-connectionstring&quot;)<br>client_id = dbutils.secrets.get(secret_scope, &quot;client-id&quot;)<br>client_secret = dbutils.secrets.get(secret_scope, &quot;client-secret&quot;)<br>tenant_id = dbutils.secrets.get(secret_scope, &quot;tenant-id&quot;)<br><br># initialises client to push messages<br>kusto_ingest_client = QueuedIngestClient(<br>    KustoConnectionStringBuilder.with_aad_application_key_authentication(<br>        connection_string=kusto_ingest_uri,<br>        aad_app_id=client_id,<br>        app_key=client_secret,<br>        authority_id=tenant_id<br>    )<br>)<br><br># controls ingestion behavior<br>ingestion_properties = IngestionProperties(<br>    database=kusto_database,<br>    table=kusto_table,<br>    data_format=DataFormat.CSV,<br>    flush_immediately=True,<br>)<br><br># converts the dataframe to csv string<br>adx_csv_output = df.toPandas().to_csv(index=False, header=False,)<br>payload_text = io.StringIO(adx_csv_output)<br><br># pushes the payload message to data explorer<br>kusto_ingest_client.ingest_from_stream(<br>    payload_text, <br>    ingestion_properties<br>)<br><br>processed_rows = adx_csv_output.count(&quot;\\n&quot;)<br>print(f&quot;{processed_rows} rows pushed appended into {kusto_database}.{kusto_table}&quot;)<br><br># COMMAND ----------<br><br># Final step: cleanup of the processing directory<br>dbutils.fs.rm(path_ganglia_processing, True)</pre><h3>Final words</h3><p>In this article, we explored the processing flow of the ganglia metrics. The XMLs parsing is done by databricks and then the data content pushed to chosen query engine which is Azure Data Explorer. In the following article, I will discuss the combination of Azure Data Explorer and Grafana and build charts for rendering CPU and memory stats.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25852d49b758" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Observability: Collecting Ganglia Metrics]]></title>
            <link>https://medium.com/@alexandr.volok/databricks-observability-collecting-ganglia-metrics-d2848d5944ee?source=rss-cb832394c0a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2848d5944ee</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[ganglia]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Alex Volok]]></dc:creator>
            <pubDate>Tue, 04 Apr 2023 19:21:14 GMT</pubDate>
            <atom:updated>2023-04-16T07:14:04.512Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>This blog post is about a few host metrics like CPU and RAM usage and their importance in understanding cluster utilisation. The simple task of tracking them might turn into a non-trivial task if the number of clusters growing over time. The real challenge is to track such metrics for short-living job clusters. This post is the first one in related series and I will discuss further the problem of getting metrics gathered and the possible solution.</p><h3>The problem: Data team has many clusters to monitor</h3><p>Databricks cluster utilisation monitoring might be a simple task if the data team has a few “permanent” clusters to perform all data processing. These clusters have ganglia endpoints, so bookmarking the ganglia pages and checking each cluster’s health by looking into CPU, Memory, and Network charts from time to time might be enough to understand how clusters are doing.</p><p>However, things might be more complicated if the data team has dozens or even hundreds of clusters. Checking each cluster’s ganglia page to see bottlenecks or underutilisation will be tedious daily OPS work.</p><p>It might be even worse in case of workflows scheduled to run on <strong>job clusters</strong> that are created/destroyed during each run. In this case, the number of clusters might be counted in thousands per week. How then data team can see the average utilisation of the cluster that has about 15 minutes of the lifetime?</p><h3>The solution: Collect cluster metrics centrally</h3><p>The solution is to collect cluster metrics centrally and store them for further observability using the push approach. Ganglia is a monitoring system that can be used to collect cluster statistics. But it will not be available when the cluster has already been destroyed.</p><p>A bash init script can be used to collect cluster metrics, and then store them in a central location. The script will start in the background during cluster start-up and will continuously collect the metrics at a specified interval and store them in a dbfs location shared among all clusters in the workspace.</p><p>These metrics can then be used with observability tools like <a href="https://azure.microsoft.com/en-us/products/managed-grafana/">Grafana</a> to monitor cluster health and performance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*2EeDze5umDlN0nXM54ur_g.png" /><figcaption>Figure 1: The topology of the solution</figcaption></figure><p>This blog post will cover the first part of the challenge: gathering data centrally to DBFS location.</p><h3>What is Ganglia?</h3><p>Before we proceed to the script writing, let’s make sure that the term Ganglia is clear.</p><p>According to Wikipedia:</p><blockquote>Ganglia is <strong>a scalable, distributed monitoring tool for high-performance computing systems, clusters, and networks</strong>. The software is used to view either live or recorded statistics covering metrics such as CPU load averages or network utilization for many nodes.</blockquote><p>Ganglia is part of the Databricks offering and is available in interactive and job clusters by default in the Metrics tab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/0*ZCaXoJJaFOqesdCh.png" /><figcaption>Figure 2: Open Ganglia Metrics</figcaption></figure><p>The link will open a dashboard with a few basic metrics on top of the page and hundreds more hidden per category at the bottom:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v8jFedbsGLTXkQ21.png" /><figcaption>Figure 3: Ganglia page</figcaption></figure><p>The databricks driver also runs an endpoint which is available on port 8652 that returns an XML payload with instant values of the tracked metrics. It can be accessed directly from the notebook using the “magic” command <em>“%sh”</em> which enables a “shell” kernel and give access to the driver’s shell.</p><p>The following example can be executed directly in the notebook and it returns all metrics of the cluster:</p><pre>%sh<br>curl localhost:8652/cluster</pre><p>In the case of databricks, the number of metrics is measured in hundreds so the size of XML payload is measured in hundreds of kilobytes to a few megabytes, depending on the number of workers.</p><p>The URL can be tuned to get metrics of the single host:</p><pre>%sh<br>curl localhost:8652/cluster/172.16.0.4/</pre><p>The endpoint can return also specified metrics only.</p><p>In the example below worker name is replaced with an asterisk to get the metric “cpu_idle” from all hosts:</p><pre>%sh<br>curl localhost:8652/cluster/*/cpu_idle</pre><p>and the outcome is:</p><pre>&lt;GANGLIA_XML VERSION=”3.6.0&quot; SOURCE=”gmetad”&gt;<br>&lt;GRID NAME=&quot;unspecified&quot; AUTHORITY=&quot;&lt;http://0401-154024-wfoq75oh-172-16-0-4/ganglia/&gt;&quot; LOCALTIME=&quot;1680364998&quot;&gt;<br>&lt;CLUSTER NAME=&quot;cluster&quot; LOCALTIME=&quot;1680364993&quot; OWNER=&quot;unspecified&quot; LATLONG=&quot;unspecified&quot; URL=&quot;unspecified&quot;&gt;<br>&lt;HOST NAME=&quot;172.16.0.4&quot; IP=&quot;172.16.0.4&quot; REPORTED=&quot;1680364991&quot; TN=&quot;6&quot; TMAX=&quot;20&quot; DMAX=&quot;0&quot; LOCATION=&quot;unspecified&quot; GMOND_STARTED=&quot;1680363793&quot; TAGS=&quot;&quot;&gt;<br>&lt;METRIC NAME=&quot;cpu_idle&quot; VAL=&quot;96.8&quot; TYPE=&quot;float&quot; UNITS=&quot;%&quot; TN=&quot;24&quot; TMAX=&quot;90&quot; DMAX=&quot;0&quot; SLOPE=&quot;both&quot; SOURCE=&quot;gmond&quot;&gt;<br>&lt;EXTRA_DATA&gt;<br>&lt;EXTRA_ELEMENT NAME=&quot;GROUP&quot; VAL=&quot;cpu&quot;/&gt;<br>&lt;EXTRA_ELEMENT NAME=&quot;DESC&quot; VAL=&quot;Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request&quot;/&gt;<br>&lt;EXTRA_ELEMENT NAME=&quot;TITLE&quot; VAL=&quot;CPU Idle&quot;/&gt;<br>&lt;/EXTRA_DATA&gt;<br>&lt;/METRIC&gt;<br>&lt;/HOST&gt;<br>&lt;HOST NAME=&quot;172.16.0.5&quot; IP=&quot;172.16.0.5&quot; REPORTED=&quot;1680364986&quot; TN=&quot;11&quot; TMAX=&quot;20&quot; DMAX=&quot;0&quot; LOCATION=&quot;unspecified&quot; GMOND_STARTED=&quot;1680363806&quot; TAGS=&quot;&quot;&gt;<br>&lt;METRIC NAME=&quot;cpu_idle&quot; VAL=&quot;98.4&quot; TYPE=&quot;float&quot; UNITS=&quot;%&quot; TN=&quot;11&quot; TMAX=&quot;90&quot; DMAX=&quot;0&quot; SLOPE=&quot;both&quot; SOURCE=&quot;gmond&quot;&gt;<br>&lt;EXTRA_DATA&gt;<br>&lt;EXTRA_ELEMENT NAME=&quot;GROUP&quot; VAL=&quot;cpu&quot;/&gt;<br>&lt;EXTRA_ELEMENT NAME=&quot;DESC&quot; VAL=&quot;Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request&quot;/&gt;<br>&lt;EXTRA_ELEMENT NAME=&quot;TITLE&quot; VAL=&quot;CPU Idle&quot;/&gt;<br>&lt;/EXTRA_DATA&gt;<br>&lt;/METRIC&gt;<br>&lt;/HOST&gt;<br>&lt;/CLUSTER&gt;<br>&lt;/GRID&gt;<br>&lt;/GANGLIA_XML&gt;</pre><h3>Why not a native Spark Dropwizard metrics library?</h3><p>Spark out of the box supports another metrics library which is also well documented: <a href="https://spark.apache.org/docs/latest/monitoring.html#executor-metrics">Dropwizard</a>.</p><p>There is nothing wrong with it. It works, is flexible, and has many bundled sinks, like a console, CSV, Prometheus, JMX, etc.</p><p>It was not suitable in my research though because the metrics it gathers are Spark-centric.</p><p>For example, it shows the amount of memory and CPU time used by a certain task. But it does not show the memory and CPU utilisation on the hosts of the cluster. Therefore, this library is a great tool to analyse the resource usage by a certain process, but it does not help understand the main requirement — general cluster utilisation.</p><h3>How-to configure databricks cluster to collect metrics</h3><p>Since we already discussed the idea of a solution and the reasoning behind choices, it is time to get into the practical steps of configuration.</p><h4>Step 1: Create an init script</h4><p>The following snipped must be executed in the databricks notebook. It creates an init script in the DBFS location. In a such way, all clusters might be configured to have it attached.</p><pre># Databricks notebook source<br>init_script_path = &quot;/init-scripts/configure_ganglia_metrics.sh&quot;<br>init_script = r&quot;&quot;&quot;#!/bin/bash<br><br># This script must run only on the driver node<br>if [[ $DB_IS_DRIVER != &quot;TRUE&quot; ]]; then<br>    exit 0<br>fi<br><br>if [[ $GANGLIA_METRICS_ENABLED != &quot;true&quot; ]]; then<br>    echo &quot;Ganglia metrics on $GANGLIA_METRICS_CLUSTER_NAME are not enabled&quot;<br>    exit 0<br>fi<br><br>echo &quot;Enabling collection of metrics on $GANGLIA_METRICS_CLUSTER_NAME&quot;<br><br>cat &lt;&lt;&#39;EOF&#39; &gt;&gt; /tmp/gather_ganglia_metrics.sh<br>ROOT_PATH=&quot;/dbfs/logs/ganglia_metrics/in&quot;<br>LOGS_DIR=&quot;$ROOT_PATH/$GANGLIA_METRICS_CLUSTER_NAME&quot;<br><br># Assign poll interval<br>re=&#39;^[0-9]+$&#39;<br>if [[ $GANGLIA_METRICS_POLL_INTERVAL =~ $re ]] ; then<br>    POLL_INTEVAL=$GANGLIA_METRICS_POLL_INTERVAL<br>else<br>    POLL_INTEVAL=60<br>fi<br>echo &quot;Poll interval is $POLL_INTEVAL seconds&quot;<br><br>if [[ ! -d $LOGS_DIR ]] ; then<br>sudo mkdir -p $LOGS_DIR<br>fi<br><br>while true; do<br><br>    LOG_TIMESTAMP=$(date &#39;+%Y%m%d%H%M%S&#39;)<br>    LOG_PATH=&quot;$LOGS_DIR/${DB_DRIVER_IP}_${LOG_TIMESTAMP}.xml&quot;<br><br>    echo $LOG_PATH<br><br>    curl http://localhost:8652/cluster/*/{mem_total,mem_free,cpu_idle,cpu_num} &gt;&gt; $LOG_PATH<br><br>    sleep $POLL_INTEVAL<br>done<br>EOF<br>chmod a+x /tmp/gather_ganglia_metrics.sh<br>/tmp/gather_ganglia_metrics.sh &amp; disown<br>&quot;&quot;&quot;<br>dbutils.fs.put(init_script_path, init_script, True)</pre><h4>Step 2: Attach the init script to the cluster</h4><p>This can be done by editing the cluster and opening Advanced options:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/0*TgCuw8BDF63BJbCk.png" /><figcaption>Figure 4: Attach init script</figcaption></figure><h4>Step 3: Enable the ganglia script by setting-up environment variables</h4><p>Open tab Spark and place required environment variables:</p><pre>GANGLIA_METRICS_ENABLED=true<br>GANGLIA_METRICS_CLUSTER_NAME=myClusterName</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/0*qKklIbP6DMFD_60H.png" /><figcaption>Figure 5: Configure environment variables</figcaption></figure><p>An optional one GANGLIA_METRICS_POLL_INTERVAL might also be used. It controls the frequency of polling in seconds. By default, it is set to 60.</p><h4>Terraform</h4><p>The cluster configuration might be simplified by adding: a few extra clauses to the cluster definition:</p><pre>new_cluster {<br>  …<br>  init_scripts {<br>    dbfs {<br>      destination = &quot;dbfs:/init-scripts/configure_ganglia_metrics.sh&quot;<br>    }<br>  }<br>  spark_env_vars = {<br>    PYSPARK_PYTHON = &quot;/databricks/python3/bin/python3&quot;<br>    GANGLIA_METRICS_ENABLED = true<br>    GANGLIA_METRICS_POLL_INTERVAL = 30<br>    GANGLIA_METRICS_CLUSTER_NAME = &quot;myClusterName&quot;<br>  }<br>  …<br>}</pre><h4>Checking the result</h4><p>Restart the cluster, attach the notebook, and check if the destination location has any file generated:</p><pre>%fs<br>ls /logs/ganglia_metrics/in/myClusterName</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/0*Xc_-AxkurgVIMxE9.png" /><figcaption>Figure 6: Check the content of the log directory</figcaption></figure><p>Congratulations, if the directory content looks similar to the image above. This means that the cluster generates XML payloads with specified metrics which are ready to be ingested into the observability tool of choice. However, the ingestion part if the topic of the next post.</p><h3>Final words</h3><p>In this blog post, I covered the problem of gathering databricks cluster metrics and explained how the gathering of them might be automated. The next article of the series <a href="https://medium.com/@alexandr.volok/databricks-processing-collected-ganglia-metrics-25852d49b758">Databricks Observability: Processing Collected Ganglia Metric</a> covers the parsing logic and ingestion of the cleaned dataframe into the observability data engine.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2848d5944ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>