<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michal Molka on Medium]]></title>
        <description><![CDATA[Stories by Michal Molka on Medium]]></description>
        <link>https://medium.com/@michalmolka?source=rss-7c158f145f51------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*uYm11Bq_xZMHXdtiNifr6g.png</url>
            <title>Stories by Michal Molka on Medium</title>
            <link>https://medium.com/@michalmolka?source=rss-7c158f145f51------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 08:45:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@michalmolka/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Codex Skills, Automations, and AGENTS]]></title>
            <link>https://michalmolka.medium.com/codex-skills-automations-and-agents-f123be9ffd5e?source=rss-7c158f145f51------2</link>
            <guid isPermaLink="false">https://medium.com/p/f123be9ffd5e</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Michal Molka]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 08:01:03 GMT</pubDate>
            <atom:updated>2026-11-03T08:06:50.642Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gACOZiNs4_IaJuP5bNEFLQ.png" /></figure><p>Everyone who has been using Codex and its ecosystem quite heavily for a while knows: skills, automations, and AGENTS.md files.</p><p>At first glance, all of these things can look pretty similar. A skill does something. An automation also does something. AGENTS.md also influences how the agent behaves. When you see this for the first time, it’s easy to assume these are just different versions of the same mechanism. The boundaries between them can feel pretty vague.</p><p>But once I started using them in practice, I realized that each of them solves a completely different problem. Interestingly, at a very simplified level, AGENTS.md and Skills are largely built around human-readable instructions, while Automations add scheduling and execution on top of prompts and agent capabilities.</p><p>When you look at them this way, the whole thing becomes much easier to understand.</p><p><strong>AGENTS</strong>: How do we work in this project?</p><p><strong>Skill</strong>: How do we perform a specific task?</p><p><strong>Automation</strong>: When should the task be executed?</p><h3>AGENTS: How do we work in this project?</h3><p>Once you look at it from this perspective, most of the confusion disappears.</p><p>AGENTS.md is probably the closest thing to a README for an AI agent. Codex reads this file before starting work and treats it as local context for a given repository or directory. This is where you define the working rules and context of the project. Rules that should apply regardless of the specific task.</p><p>You can include things like:</p><ul><li>coding standards,</li><li>testing requirements,</li><li>branch management rules,</li><li>required steps before merge,</li><li>security constraints,</li><li>project architecture overview.</li></ul><p>Put simply, AGENTS.md does not tell the agent what to do. It tells the agent how work is done in this project. Just like a new team member needs to understand the rules of an organization, an AI agent needs to understand the environment it’s operating in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zI_njCWYIxDNWWBfE8I7Ug.png" /></figure><h3>Skill: How do we perform a specific task?</h3><p>A skill is responsible for executing a specific process.</p><p>Despite the name, it does not have to be some complicated algorithm. Very often, it is simply a well-defined workflow described through instructions, checklists, and prompts.</p><p>For example, you can create a skill for analyzing Pull Requests or CI/CD.</p><p>Such a skill can perform steps like:</p><ol><li>Check the CI/CD status.</li><li>Analyze the scope of changes.</li><li>Assess test quality.</li><li>Look for potential security issues.</li><li>Prepare a final report.</li></ol><p>In the attached repository, there are two skills:</p><ul><li>check-pr-cicd</li><li>assess-pr-readiness</li></ul><p>The first one analyzes pipeline status. The second one evaluates whether a Pull Request is ready for review or merge. And this is where skills become useful. They let you capture a repeatable process in one place and execute it consistently, using the same rules every time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*326Ob4yg6_AxtJ-pGU_BTQ.png" /></figure><h3>Automation: When should the task be executed?</h3><p>A skill defines a reusable process.</p><p>An automation decides when an agent should run a task. That task can use a skill, but it does not have to. The idea is somewhat similar to scheduled jobs in GitHub Actions, Azure DevOps Pipelines, or Jenkins. The difference is that one of the workflow steps can involve an AI agent.</p><p>Examples of automations are simple:</p><ul><li>check Pull Request status every morning,</li><li>monitor GitHub Actions workflows every hour,</li><li>prepare a daily report,</li><li>periodically check for new Pull Requests and analyze them.</li></ul><p>A simple flow could look like this:</p><ol><li>A developer creates a Pull Request.</li><li>An automation triggers the task according to a defined schedule.</li><li>A skill responsible for code analysis is executed.</li><li>A skill responsible for CI/CD analysis is executed.</li><li>The agent publishes a summary or performs additional actions.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z3C-5MnKtR7CpZjE4r10FQ.png" /></figure><h3>How does it all work together?</h3><p>Let’s say we want to automate Pull Request analysis in a FastAPI project.</p><p>First, we create an AGENTS.md file that describes the architecture, coding standards, and testing requirements. Then we build a skill responsible for analyzing code changes. Finally, we configure an automation that checks, according to a defined schedule, whether a new Pull Request has appeared.</p><p>As a result, we get a system that:</p><ul><li>understands the project rules,</li><li>knows the analysis process,</li><li>has access to the right tools,</li><li>knows when to start working.</li></ul><p>And this is when it starts to look like an intelligent agent.</p><p>When we look at AGENTS, Skills, and Automations as separate layers of one solution, most of the confusion disappears. They complement each other. They have different functions. They work like one coherent system.</p><p>And that is exactly why it is worth understanding their roles separately.</p><p>Repository: <a href="https://github.com/michalmolka/codex_ci_cd">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f123be9ffd5e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-Tuning an LLM: Teaching a Data Warehouse Schema Without RAG]]></title>
            <link>https://michalmolka.medium.com/fine-tuning-an-llm-teaching-a-data-warehouse-schema-without-rag-1122ff567c41?source=rss-7c158f145f51------2</link>
            <guid isPermaLink="false">https://medium.com/p/1122ff567c41</guid>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mlx]]></category>
            <dc:creator><![CDATA[Michal Molka]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 07:01:03 GMT</pubDate>
            <atom:updated>2026-10-01T07:01:03.649Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Owot1Ud8ISf4rerCFiU5_g.png" /></figure><p>In the AI world, it is often assumed that any additional knowledge should be stored in a vector database and made available to a model through Retrieval-Augmented Generation (RAG).</p><p>In many cases, that is an excellent approach. However, it is not the only option. When a model is expected to answer a relatively narrow set of questions within a specific domain, fine-tuning can be a viable alternative.</p><p>So let’s train a model and see how it works. In this case, fine-tuning is used to teach the model to answer questions about a sales data warehouse schema. All operations were run locally on macOS.</p><h3>Goal</h3><p>The objective was to enable the model to answer questions related to:</p><ul><li>Tables</li><li>Relationships</li><li>Keys</li><li>Data grain</li><li>Business meaning of columns</li></ul><p>The schema consisted of:</p><ul><li>The iowa_sales fact table</li><li>Five dimension tables</li><li>Relationships between tables</li><li>Business descriptions of individual attributes</li></ul><p>The model received no additional context at inference time. All knowledge came exclusively from the fine-tuning process.</p><h3><strong>Data Preparation</strong></h3><p>A dataset of approximately 2,500 examples was created in JSONL format. Each record contained a question and the expected answer.</p><p>Example:</p><pre>{<br>    &quot;messages&quot;: [<br>        {<br>            &quot;role&quot;: &quot;system&quot;,<br>            &quot;content&quot;: &quot;You are an assistant specialized in the Iowa sales data model. Give concise, business-friendly answers grounded only in the schema.&quot;<br>        },<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: &quot;How would you explain iowa_store in a quick way?&quot;<br>        },<br>        {<br>            &quot;role&quot;: &quot;assistant&quot;,<br>            &quot;content&quot;: &quot;iowa_store is a dimension table. It is the store and location dimension and its grain is one store.&quot;<br>        }<br>    ]<br>}</pre><p>The dataset was split into three parts:</p><ul><li>Train</li><li>Validation</li><li>Test</li></ul><pre>head -n 2200 data.jsonl &gt; train.jsonl<br>sed -n &#39;2201,2350p&#39; data.jsonl &gt; valid.jsonl<br>sed -n &#39;2351,2500p&#39; data.jsonl &gt; test.jsonl</pre><h3>Model</h3><p>For this PoC, I used <strong>Qwen3–8B-MLX-bf16</strong> running locally on Apple Silicon with MLX. The model was a good fit for a narrow, domain-specific use case: compact enough to fine-tune locally, while still capable enough to handle schema relationships, keys, data grain, and business meaning.</p><p>MLX was used to keep the entire fine-tuning workflow local on the Mac, without requiring external GPU infrastructure or cloud services. The objective was to build a lightweight setup focused on one specific domain and keep the architecture as simple as possible.</p><h3>Training</h3><p>Fine-tuning was performed using LoRA.</p><pre>mlx_lm lora \<br>  --model ./Qwen3-8B-MLX-bf16 \<br>  --train \<br>  --data ./data \<br>  --fine-tune-type lora \<br>  --adapter-path ./Qwen3-8B-MLX-bf16-iowa-lora-v1 \<br>  --batch-size 1 \<br>  --iters 2500 \<br>  --learning-rate 1e-4 \<br>  --num-layers 16 \<br>  --mask-prompt \<br>  --save-every 500 \<br>  --max-seq-length 1024</pre><p>These parameters are not intended to be universal. They were selected specifically for a small, highly specialized dataset focused on a single data model.</p><h3>Results</h3><p>Before training, the responses were vague and inaccurate. The model had no usable knowledge of the schema.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PMufeAm1yzsvYOI-fZUXfg.png" /></figure><p>After training, the model was able to correctly identify:</p><ul><li>The fact table</li><li>Dimension tables</li><li>Relationships</li><li>Keys</li><li>Data grain</li><li>Business meaning of columns</li></ul><p>The answers were concise and precise. Most importantly, the model achieved this without receiving any additional context in the prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bg63nD6P0lM6kRt6HhACYQ.png" /></figure><h3>Does Fine-Tuning Replace RAG?</h3><p>No. Fine-tuning and RAG solve different problems. RAG works well for large, dynamic knowledge bases that change frequently. Fine-tuning is useful when a model needs to learn domain-specific knowledge or adopt a particular response style. In practice, both approaches are often used together.</p><h3>Conclusion</h3><p>Fine-tuning did not disappear with the rise of RAG. A well-prepared dataset and a relatively small LoRA adapter can effectively teach a model domain-specific knowledge without requiring additional context at inference time. For stable domain knowledge, in some narrow use cases, this can be a simpler and more cost-effective solution than building and maintaining a complete RAG pipeline.</p><p>Code available here: <a href="https://github.com/michalmolka/python_llm_code/tree/main/fine-tuning">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1122ff567c41" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG, Vectors, Elasticsearch, and Graphs. What, When, How, and Where.]]></title>
            <link>https://michalmolka.medium.com/rag-vectors-elasticsearch-and-graphs-what-when-how-and-where-4e958d6aaaa5?source=rss-7c158f145f51------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e958d6aaaa5</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Michal Molka]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 07:01:03 GMT</pubDate>
            <atom:updated>2026-09-01T07:01:03.899Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xfN5B7Z-2MTe7E3UGRe4cg.png" /></figure><p>In RAG, vector search is usually the first choice. And rightly so. There is usually a real reason behind this decision: the user asks in natural language, and the system needs to find pieces of text with a similar meaning. Embeddings do exactly that job. No incense, no grand philosophy. In most cases, this works, but there are cases where we can choose better solutions.</p><p>A quick note before we go further: all of these technologies are versatile and can handle far more than the use cases discussed here. This article focuses on their strongest and most characteristic roles in retrieval, not on drawing hard boundaries between them.</p><p><strong>When Vectors Make Sense</strong></p><p>Vector search is great when meaning matters more than exact words. When the user asks using different wording than what appears in the document. When documents are long, descriptive, and loosely structured. When we need to find similar cases, tickets, contracts, procedures, or documentation fragments. When the question is imprecise, but the intent is clear. This is the natural environment for embeddings: knowledge bases, FAQs, instructions, procedures, policies, support notes, and case descriptions. That is where vectors do a good job, and there is no need to pretend someone just discovered electricity. Things get more complicated when we try to use vectors to solve every type of search problem. Some questions are semantic. Some are filters. Some are conditions, sorting rules, relationships, or business rules with a “needed yesterday” label attached.</p><p><strong>When Elasticsearch Does Its Job</strong></p><p>Elasticsearch is a mature tool for working with text, metadata, filters, ranking, and aggregations. It has survived several hype cycles and still does its job, because search in business systems rarely ends with one nice semantic similarity score. If the question includes conditions, Elasticsearch quickly becomes very attractive. Categories, statuses, departments, customer types, partial names, codes, text fields, sorting by freshness, facets, top values, fuzzy search, autocomplete, typos, and ranking explanations — these are not exotic features. This is a normal day in a normal system.</p><p>In RAG, Elasticsearch works well as a candidate-narrowing layer. Hard conditions, dates, document types, and text matching can narrow the search space before vector retrieval, reranking, and the LLM.</p><p>Instead of asking a vector database: “find everything that might match,” we can first say: “search only within documents from this country, this time period, and this type.” Elasticsearch gives us a rich set of search options for narrowing the candidate set before further retrieval or reranking.</p><p><strong>And Graphs?</strong></p><p>Graphs make sense when the problem is relationships.</p><p>Which products are connected to this supplier? Which contracts touch the same business process? What is the chain of dependencies between an incident, a system, a service, and a customer? Who has access to data related to this project? Which entities are indirectly connected, even if they never appear together in the same document? Vector search can find semantically similar text. Elasticsearch is strong at lexical search, filtering, and ranking. A graph can show how things are connected. These are three different superpowers. The whole trick is not asking one superpower to do everything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nk4MVSxrieOZlzQnJtRO5A.png" /></figure><p><strong>We are talking about them here in the context of LLMs and RAG, but these tools have a life outside RAG:</strong></p><p>Elasticsearch has been powering application search, logs, observability, monitoring, event analysis, security analytics, product catalogs, autocomplete, suggestions, and text data exploration for years.</p><p>Neo4j and graph databases also have very concrete use cases: fraud detection, knowledge graphs, access analysis, recommendations, master data management, dependency mapping, impact analysis, and network analysis.</p><p>And they are damn good at it. So what solution should we choose? Most of the time, the hybrid approach wins. In practice, the best RAG systems often combine several types of retrieval. Elasticsearch narrows the data down by filters, permissions, dates, types, and keywords. Vector search finds semantic similarity within a reasonable candidate set. Graphs look for related objects. A reranker orders the results for the specific question. The LLM receives only the context that has a real chance of being useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kN1n0EANiCWkXZzemdNxfA.png" /></figure><h3><strong>Simple Rule</strong></h3><p>If the question is about meaning, similarity, and paraphrasing, vectors are a strong candidate. If the question is about filters, text, sorting, aggregations, statuses, and business conditions, Elasticsearch will be very strong. If the question is about relationships, paths, dependencies, and connections between entities, a graph starts to make sense. If the question is about all of that at once, welcome to a real system.</p><p>A vector search is a very good first choice for many RAG scenarios. But the first choice only ends the discussion in simple cases.</p><p>Sometimes you need embeddings. Sometimes you need a text index. Sometimes you need a graph. Sometimes SQL. Often, a regular SQL table with well-designed fields and metadata is the best or fully sufficient solution. But as usual, when a stakeholder asks, the standard answer is… it depends…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e958d6aaaa5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Polars vs DuckDB vs Pandas: Benchmark on 33 Million Rows]]></title>
            <link>https://michalmolka.medium.com/polars-vs-duckdb-vs-pandas-benchmark-on-33-million-rows-02386be8c37f?source=rss-7c158f145f51------2</link>
            <guid isPermaLink="false">https://medium.com/p/02386be8c37f</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[polar]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Michal Molka]]></dc:creator>
            <pubDate>Wed, 29 Jul 2026 08:01:03 GMT</pubDate>
            <atom:updated>2026-07-29T08:01:03.453Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*32KNuCBA4zVpst__Jx0x_w.png" /></figure><p>There has recently been a lot of discussion about Polars, DuckDB, Apache Arrow, and the limitations of Pandas. Pandas has been around for a long time. It has a history — and baggage. Nobody expected it to win this benchmark. The more interesting question was how large the gap would be on a realistic analytical workload rather than on the kind of synthetic micro-benchmark that is all over the internet.</p><p>For this test, I used a simple star schema consisting of a 33.2 million-row fact table and five dimension tables stored as Parquet files, approximately 535 MB in total. The benchmark was executed on a MacBook Pro M5 with 32 GB RAM.</p><p>To avoid reusing Python objects and library state, the notebook kernel was restarted between benchmark runs. So the results represent clean-process runs with a potentially warm filesystem cache.</p><p>The complete code is available on: <a href="https://github.com/michalmolka/polars_pandas_duckdb_benchamrk_33m">GitHub</a></p><p>The goal was to compare how the three tools perform in a typical analytical workflow:</p><ul><li>Pandas (DataFrame API)</li><li>Polars (DataFrame API)</li><li>DuckDB (SQL engine)</li></ul><p>The workload included:</p><ul><li>Multiple joins</li><li>Filtering</li><li>Grouping</li><li>Aggregations</li><li>Sorting</li></ul><p>In other words, something much closer to a real analytical workload than a simple “sum a column” benchmark.</p><p>Here is a SQL version of the query:</p><pre>SELECT<br>        d.CalendarYear,<br>        s.County,<br>        s.StoreName,<br>        v.VendorName,<br>        c.CategoryName,<br>        i.ItemName,<br>        SUM(f.SaleDollars)       AS total_sales,<br>        SUM(f.BottlesSold)       AS bottles_sold,<br>        SUM(f.VolumeSoldLiters)  AS volume_liters,<br>        AVG(f.StateBottleRetail) AS avg_retail,<br>        AVG(f.StateBottleCost)   AS avg_cost,<br>        COUNT(f.ID)              AS transactions<br>    FROM read_parquet(&#39;iowa_sales.snappy.parquet&#39;) f<br>    JOIN read_parquet(&#39;iowa_category.snappy.parquet&#39;) c<br>        ON f.CategoryID = c.CategoryID<br>    JOIN read_parquet(&#39;iowa_date.snappy.parquet&#39;) d<br>        ON f.DateID = d.DateID<br>    JOIN read_parquet(&#39;iowa_store.snappy.parquet&#39;) s<br>        ON f.StoreID = s.StoreID<br>    JOIN read_parquet(&#39;iowa_vendor.snappy.parquet&#39;) v<br>        ON f.VendorID = v.VendorID<br>    JOIN read_parquet(&#39;iowa_item.snappy.parquet&#39;) i<br>        ON f.ItemID = i.ItemID<br>    WHERE d.CalendarYear &gt;= 2022<br>    GROUP BY<br>        d.CalendarYear,<br>        s.County,<br>        s.StoreName,<br>        v.VendorName,<br>        c.CategoryName,<br>        i.ItemName<br>    ORDER BY<br>        d.CalendarYear,<br>        s.County,<br>        s.StoreName,<br>        v.VendorName,<br>        c.CategoryName,<br>        i.ItemName,<br>        total_sales DESC,<br>        bottles_sold DESC</pre><p><strong>A Note About the Benchmark</strong></p><p>This benchmark is not an attempt to manually tune each library to its absolute limits. The goal was to compare a typical way many developers work with data.</p><p>In Pandas, this pipeline is executed eagerly, meaning each operation runs immediately and often materializes intermediate DataFrames in memory. Pandas does not build a global execution plan, so optimizations such as predicate pushdown, projection pushdown, or join reordering remain largely the responsibility of the developer.</p><p>Polars and DuckDB take a different approach. Polars in LazyFrame mode and DuckDB SQL can treat the entire workflow as a query plan and optimize it before execution. In practice, this means less data read from disk, fewer intermediate structures in memory, and more efficient execution of joins and aggregations.</p><p>Another important piece of the puzzle is Apache Arrow. Polars follows the Apache Arrow columnar memory format and can exchange Arrow data with little or no copying. DuckDB, however, uses its own vectorized, DataChunk-based execution engine while providing efficient interoperability with Arrow. In this benchmark, the main performance advantages come from optimized Parquet scans, query planning, vectorized execution, parallelism, and efficient memory management.</p><p>So, a large part of the performance difference comes from modern execution engines built around columnar processing and optimized memory access patterns.</p><p>Because of that, this benchmark should not be viewed simply as “library A is faster than library B.” Among other things, it is also a comparison between two execution models:</p><ul><li>Eager execution (Pandas)</li><li>Optimized query planning (Polars LazyFrame and DuckDB)</li></ul><p>Could the Pandas code be optimized further? Absolutely. Columns could be pruned earlier, filters pushed closer to the source, and parts of the workflow rewritten to reduce memory usage. But at that point, the developer effectively becomes the optimizer. One of the advantages of Polars and DuckDB is that many of those decisions can be made automatically by the engine itself.</p><p><strong>Average Results (5 Runs)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*w-jumLSI80fZBPmHPJmXoA.png" /></figure><p>Important: The RAM and CPU numbers should be treated as directional indicators rather than precise measurements. They were collected from a local notebook environment and are intended to illustrate general resource usage patterns.</p><p><strong>Observations</strong></p><p>The first observation is the size of the gap. A roughly 10× speedup changes the experience from waiting through every iteration to getting results almost instantly.</p><p>The second observation is how close Polars and DuckDB were. Despite very different APIs, both engines delivered virtually identical execution times on this workload.</p><p>The third observation is memory consumption. DuckDB achieved an execution time comparable to Polars while using noticeably less memory, reducing peak usage from approximately 10.46 GB to 8.3 GB. For developers working on laptops with 16 GB RAM, that difference can be significant.</p><p>The fourth observation is parallelism. While their execution engines and implementation details differ, the peak CPU measurements suggest that both Polars and DuckDB made substantially better use of the available CPU cores than Pandas in this workload.</p><p>However, the most interesting takeaway is not that Pandas was slower. That was expected. The interesting part is that a workload involving 33 million rows, five joins, aggregations, and sorting can now be executed in around <strong>1.3</strong> seconds on a laptop.</p><p>Not long ago, many teams would have reached for a distributed engine by default. Today, local analytical engines are capable of handling workloads that were traditionally associated with much larger platforms.</p><p><strong>Why Pandas Is Still Everywhere</strong></p><p>Looking at the benchmark results, it may be tempting to conclude that Pandas is obsolete. That would be the wrong conclusion.</p><p>Despite being slower on this workload, Pandas remains one of the most successful data processing libraries ever created.</p><p>Some of the reasons include:</p><ul><li>A massive ecosystem built around it</li><li>Excellent integration with the Python data stack</li><li>Countless tutorials, books, and learning resources</li><li>Millions of existing notebooks, scripts, and production workloads</li><li>Strong backward compatibility</li><li>A familiar API that many developers already know</li></ul><p>In many organizations, developer productivity is more important than squeezing out the last few seconds of execution time.</p><p><strong>Final Thoughts</strong></p><p>Pandas remains an incredibly useful library and is still the default choice for many data professionals. Processing this workload in around 13 seconds on a laptop is far from a poor result.</p><p>However, the benchmark clearly shows how much the landscape has changed. Modern analytical engines such as Polars and DuckDB combine query optimization, vectorized execution, efficient Parquet scanning, optimized memory access patterns, and parallel processing to deliver performance that would have seemed unrealistic for local analytics only a few years ago.</p><p>The most interesting question is no longer whether Polars and DuckDB are faster than Pandas. The more interesting question is how many workloads we still need a cluster for.</p><p><strong>One More Thing: Data Locality</strong></p><p>Data locality is another aspect worth remembering.</p><p>In this benchmark, all data was processed locally from Parquet files stored on a laptop. That makes the comparison straightforward, but it does not reflect every production environment.</p><p>In many organizations, datasets remain inside governed platforms such as Databricks, MS Fabric, Snowflake, or cloud data lakes. While it may be technically possible to download data and process it locally with tools such as Polars or DuckDB, that is not always a good idea.</p><p>Moving data outside a controlled environment introduces additional considerations around security, governance, compliance, auditing, and data ownership. In some organizations, it may not be allowed at all. In others, it may be allowed but strongly discouraged.</p><p>For that reason, architecture decisions should not be driven solely by execution time, cost, or developer convenience. A local analytical engine may be faster, cheaper, and easier to use, but keeping data inside a governed platform can still be the better engineering decision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=02386be8c37f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vendor-Neutral Observability with OpenTelemetry and the Grafana Stack]]></title>
            <link>https://michalmolka.medium.com/vendor-neutral-observability-with-opentelemetry-and-the-grafana-stack-fdfe8d8378a9?source=rss-7c158f145f51------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdfe8d8378a9</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[alloy]]></category>
            <category><![CDATA[tempo]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[loki]]></category>
            <dc:creator><![CDATA[Michal Molka]]></dc:creator>
            <pubDate>Wed, 01 Jul 2026 08:01:03 GMT</pubDate>
            <atom:updated>2026-07-01T08:01:03.597Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DChDJom0SnqO6ulNQcjagg.png" /></figure><p>Modern applications generate enormous amounts of telemetry data. Metrics, logs, and traces help us understand what is happening inside our systems, troubleshoot issues faster, and improve reliability.</p><p>While many organizations use managed observability platforms, it is often beneficial to understand the underlying building blocks first. This is especially true when learning observability, building proof-of-concepts, or avoiding vendor lock-in.</p><p>In this article, we will build a complete open-source observability stack using:</p><ul><li>Grafana</li><li>Alloy</li><li>Prometheus</li><li>Loki</li><li>Tempo</li><li>OpenTelemetry</li><li>FastAPI</li></ul><p>This walkthrough was developed and tested on a local macOS environment using Docker Desktop. Treat this as a minimal example, not a production-ready deployment. In a production-grade Grafana stack, Grafana Mimir could be used for metrics. For this local implementation, Prometheus is used instead to keep the setup simpler.</p><p>The complete source code is available on <a href="https://github.com/michalmolka/observability_lgtm">GitHub</a>.</p><h4><strong>Why an Open-Source Observability Stack?</strong></h4><p>One of the biggest advantages of this stack is that it is based on open-source components and open standards, which reduces vendor lock-in. You can run the entire observability platform locally, on-premises, or in any cloud provider without being locked into a single vendor. OpenTelemetry provides a standardized way of collecting telemetry data, making migrations and multi-cloud strategies significantly easier. The same concepts, telemetry data, and instrumentation can later be used with managed platforms such as Azure Monitor, Datadog, New Relic, Dynatrace, or Grafana Cloud. More importantly, each component can be replaced independently. You can use Grafana without Loki, Prometheus without Tempo, or OpenTelemetry with a completely different backend.</p><h4><strong>Architecture Overview</strong></h4><p>The following components make up our observability platform:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BzI2w_HRCrjioACwFD2KiA.png" /></figure><ul><li>The application generates telemetry data.</li><li>OpenTelemetry instrumentation generates logs and traces and exports them to Alloy.</li><li>Metrics are exposed through the /metrics endpoint and scraped by Alloy before being forwarded to Prometheus.</li><li>Alloy acts as the central collection and routing layer.</li><li>Metrics are sent to Prometheus, logs to Loki, and traces to Tempo.</li><li>Grafana connects to all three backends and provides a unified observability experience.</li></ul><h4><strong>Why Alloy?</strong></h4><p>Grafana Alloy is the telemetry collector used in this solution. Its primary responsibility is to receive telemetry data and route it to the appropriate destination. Think of Alloy as the traffic controller of the observability stack. Instead of configuring every application to communicate directly with multiple backends, applications send telemetry to Alloy and Alloy handles the rest.</p><p>Benefits include:</p><ul><li>Centralized telemetry collection</li><li>OpenTelemetry native support</li><li>Flexible routing and processing</li><li>Simplified application configuration</li><li>Easier scaling and maintenance</li></ul><h4><strong>Why Prometheus?</strong></h4><p>Prometheus is responsible for metrics.</p><p>Metrics answer questions such as:</p><ul><li>How many requests are we processing?</li><li>What is the average response time?</li><li>Are error rates increasing?</li><li>Is CPU or memory usage growing?</li></ul><p>Prometheus stores time-series data and allows querying through PromQL.</p><h4><strong>Why Loki?</strong></h4><p>Loki is responsible for logs.</p><p>Logs provide detailed information about individual events occurring within an application.</p><p>Examples include:</p><ul><li>Request processing details</li><li>Errors and exceptions</li><li>Business events</li><li>Debug information</li></ul><p>Unlike traditional log management systems, Loki stores labels separately and keeps log storage relatively lightweight.</p><p>This makes it a popular choice within the Grafana ecosystem.</p><h4><strong>Why Tempo?</strong></h4><p>Tempo is responsible for distributed tracing.</p><p>Traces answer questions such as:</p><ul><li>Which services participated in a request?</li><li>Where is the latency coming from?</li><li>Which operation is slowing down the application?</li><li>What happened during a specific transaction?</li></ul><p>Tracing becomes increasingly important as applications grow and evolve into distributed systems. In short, traces help visualize the complete lifecycle of a request.</p><h4><strong>Why Grafana?</strong></h4><p>Grafana provides the user interface for the entire observability stack.</p><p>Instead of switching between multiple tools, Grafana allows us to visualize:</p><ul><li>Metrics from Prometheus</li><li>Logs from Loki</li><li>Traces from Tempo</li></ul><p>The real power comes from connecting metrics, logs, and traces in a single place. Instead of switching between tools during an incident, engineers can move between telemetry signals with a few clicks.</p><p><strong>Trace to Logs and Logs to Traces</strong></p><p>One of the most useful features in Grafana is navigating directly from a trace to the related logs and the other way around. Starting with a trace, we can immediately view all log entries associated with that request. When viewing a log entry, Grafana can extract the Trace ID and jump directly to the corresponding trace. This requires Trace IDs to be included in logs and properly configured in Grafana.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HQsMFdzEPEYx_KrWt6Wqmg.png" /></figure><h4><strong>The Sample Application</strong></h4><p>To generate telemetry data, we use a minimal FastAPI application. The application exposes two endpoints:</p><ul><li>/signal — generates telemetry data used throughout the walkthrough</li><li>/metrics — exposes Prometheus metrics collected by the application</li></ul><p>The purpose of the <strong>/signal</strong> endpoint is not business functionality, but to generate metrics, logs, and traces that we can analyze throughout the observability pipeline. The /metrics endpoint returns application metrics in Prometheus format. Alloy scrapes these metrics and forwards them to Prometheus for storage and querying.</p><p>After starting the application, we generate traffic using a simple loop:</p><pre>for i in {1..100}; do<br>  curl -s http://localhost:8000/signal &gt; /dev/null<br>done</pre><p>This creates enough telemetry data to demonstrate the entire stack.</p><h4><strong>End-to-End Telemetry Flow</strong></h4><p>The complete flow looks like this:</p><p>1. A request hits the <strong>/signal</strong> endpoint.</p><p>2. OpenTelemetry instrumentation generates logs and traces.</p><p>3. Logs and traces are sent to Alloy.</p><p>4. The application exposes metrics through the <strong>/metrics</strong> endpoint.</p><p>5. Alloy scrapes metrics from <strong>/metrics</strong>.</p><p>6. Alloy forwards:</p><ul><li>metrics to Prometheus</li><li>logs to Loki</li><li>traces to Tempo</li></ul><p>7. Grafana visualizes and correlates telemetry across all three backends.</p><p>This creates a unified observability experience across the entire stack.</p><h4><strong>Run the </strong>solution<strong> — step by step</strong></h4><p><strong>#1 Create Docker containers</strong></p><pre>docker compose \<br>  -f 01_alloy.yaml \<br>  -f 02_prometheus.yaml \<br>  -f 03_loki.yaml \<br>  -f 04_tempo.yaml \<br>  -f 05_grafana.yaml \<br>  up -d</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lwxBlAwfGU8LbZ9CkTaxtA.png" /></figure><p><strong>#2 Check endpoints</strong></p><pre>alloy: http://localhost:12345<br>prometheus: http://localhost:9090/<br>loki: http://localhost:3100/ready<br>tempo: http://localhost:3200/ready<br>grafana: http://localhost:3000</pre><p><strong>#3 Run the FastAPI app</strong></p><pre>uvicorn main:app --reload</pre><p><strong>#4 Call the endpoint</strong></p><pre>for i in {1..100}; do<br>  curl -s http://localhost:8000/signal &gt; /dev/null<br>done</pre><p><strong>#5 Open Grafana</strong></p><pre>http://localhost:3000<br>Default credentials:<br>username: admin<br>password: admin</pre><p><strong>#6 Verify telemetry</strong></p><p>At this point, metrics, logs, and traces should be visible in Grafana.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RxOIj9sGIh3st4_NTFdIOQ.png" /></figure><h4><strong>Final Thoughts</strong></h4><p>By combining OpenTelemetry, Alloy, Prometheus, Loki, Tempo, and Grafana, we can build a complete, vendor-neutral observability solution using open-source technologies and open standards.</p><p>More importantly, understanding these building blocks gives us greater flexibility, control, and freedom of choice. Each component can be replaced independently.</p><p>It also becomes much easier to work with managed observability platforms later, because the underlying concepts remain largely the same.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdfe8d8378a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Power BI assets — seamless PRs]]></title>
            <link>https://michalmolka.medium.com/power-bi-assets-seamless-prs-96582595af58?source=rss-7c158f145f51------2</link>
            <guid isPermaLink="false">https://medium.com/p/96582595af58</guid>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[pull-request]]></category>
            <dc:creator><![CDATA[Michal Molka]]></dc:creator>
            <pubDate>Mon, 01 Jun 2026 16:01:00 GMT</pubDate>
            <atom:updated>2026-06-01T16:01:00.865Z</atom:updated>
            <content:encoded><![CDATA[<h3>Power BI — seamless PRs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sjy6zXPs5w98t5YUBGO4Tg.png" /></figure><p>In these two posts, I’ve been writing about Power BI datasets differential update through ALM Toolkit: <a href="https://medium.com/p/f3e749f818f6">Power BI — update dataset — ALM Toolkit</a> and a a visual layer control versioning: <a href="https://medium.com/p/b71a45f16ca6">Power BI report — control versioning</a></p><p>As I wrote, saving Power BI models and reports as text files makes them easier maintainable from a control version standpoint. There is another benefit. Pull request reviews are much easier.</p><p>Let’s look at the exemplary PR. The most important area is the Files section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QahGOkerxgQWyVYV5Ql36g.png" /></figure><p>All the changes are clear as day. We see every part of the report and the model. In the case above, a column had been added to a user-aggregation table.</p><p>Here, a measure has been added to the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sEG2jUdazxxj5YUyAe68Sw.png" /></figure><p>Let’s look at the visualization layer. One of the measures has changed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SbevKSgIhCKsG5zRTdwdyw.png" /></figure><p>We can track changes through commits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TcPCBOscVkjwOxksH5C9nQ.png" /></figure><p>And after the PR is merged into the main branch, all the history is more readable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zDWF_eZTOBbNjF7CJwHUnw.png" /></figure><p>Besides the Azure DevOps portal, we can use our favorite text editor/IDE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oTZ3S5N0PMTnLZM9ih_uAA.png" /></figure><p>And track changes in the same way directly through the Git console or functionality baked into an app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FkcbxR2XvVH23MrqK71_NA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96582595af58" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Power BI — update dataset — ALM Toolkit]]></title>
            <link>https://michalmolka.medium.com/power-bi-update-dataset-alm-toolkit-f3e749f818f6?source=rss-7c158f145f51------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3e749f818f6</guid>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[msfabric]]></category>
            <category><![CDATA[dataset]]></category>
            <dc:creator><![CDATA[Michal Molka]]></dc:creator>
            <pubDate>Fri, 01 May 2026 16:01:02 GMT</pubDate>
            <atom:updated>2026-06-01T17:58:34.317Z</atom:updated>
            <content:encoded><![CDATA[<h3>Power BI — update dataset — ALM Toolkit</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_4AiXmvHmUcOHVcTvV3Xnw.png" /></figure><p>Today we will talk about updating models through ALM toolkit. Why use an app, you can ask. We can publish our report from Power BI Desktop or Tabular Editor and forget about it. ALM Toolkit is useful when we want to apply a differential change to a dataset without overwriting everything. Not too big of an argument, you might say. So, when can we take advantage from this approach. For example:</p><ul><li>We have a big model containing a lot of data, partitions, etc., and refreshing it after every deployment is a bit problematic…costs money,</li><li>we use large semantic models, so downloading a .pbix file isn’t possible,</li><li>we want to separate the data model layer from the report layer,</li><li>we want to have detailed control over changes (what and when is changed).</li></ul><p>In the last post, <a href="https://medium.com/p/d610dc44faf5">Power BI — user defined aggregations</a>, I created an exemplary report, and it will be our base.</p><p>Here is the report deployed to the MS Fabric portal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kDbRM5jWcP4VZYeqNbZp0w.png" /></figure><p>We don’t need and don’t want to change anything related to the report/virtual layer. I&#39;ve made a few changes in a semantic model:</p><ul><li>Added a measure,</li><li>Added a column in a table,</li><li>Added a relationship between two tables,</li><li>Changed settings of a user-aggregation table.</li></ul><p>In order to use ALM toolkit, we need to save the model into a <strong>.bim</strong> file using Tabular Editor or save it to a <strong>.pbib</strong> format from Power BI Desktop.</p><ul><li><strong>.bim</strong> file is one big <strong>.json</strong> file,</li><li><strong>.pbib </strong>— saves the model and the report into several text files like <strong>.json</strong>, <strong>.tmdl</strong>, etc.</li></ul><p>A similar topic focused on the reporting layer is available in this post: <a href="https://medium.com/p/b71a45f16ca6">Power BI report — control versioning</a></p><p>In this example, I will use the second option (<strong>.pbib</strong>).</p><p>Run ALM Toolkit and hit the <strong>Compare </strong>button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*fNAwjyvGvKu9uuQA4gXw-w.png" /></figure><p>A source is a model we developed locally and want to deploy. A target is a dataset existing in the Power BI service.</p><p>In the options menu, you can set how the comparison should behave, what to take into consideration and what not. Here as well, you can set partitions consideration and how to process them if there is a need for it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*tBvllYKZI-esgQOr4Zp2qg.png" /></figure><p>Now, we see what changes when we update the target dataset. We can filter out objects that don’t change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z468D-j1DdBCDWDEWCv32w.png" /></figure><p>If everything is according to our expectations, then we can <strong>Validate</strong> the changes and <strong>Update</strong> the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/1*a5So7PMTJKjr-rurl5eqCA.png" /></figure><p>One more thing. Before you deploy the model, you need to have an XMLA Endpoint set to [<strong>Read Write</strong>].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0U16ta9xdvuR2ohaWz-s3w.png" /></figure><p>You can look at a post realated to this topics: <a href="https://medium.com/p/96582595af58">Power BI assets — seamless PRs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3e749f818f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Power BI — user defined aggregations]]></title>
            <link>https://michalmolka.medium.com/power-bi-user-defined-aggregations-d610dc44faf5?source=rss-7c158f145f51------2</link>
            <guid isPermaLink="false">https://medium.com/p/d610dc44faf5</guid>
            <category><![CDATA[msfabric]]></category>
            <category><![CDATA[aggregates]]></category>
            <category><![CDATA[power-bi]]></category>
            <dc:creator><![CDATA[Michal Molka]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 16:01:02 GMT</pubDate>
            <atom:updated>2026-04-01T16:01:02.214Z</atom:updated>
            <content:encoded><![CDATA[<h3>Power BI — user defined aggregations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NP3rjylbs5BXlA_iBdr2OA.png" /></figure><p>From time to time, we can have a problem with model size. We own a model with huge fact table/s. The model reaches capacity size or is just expensive in cost and slow in refreshes. I’m thinking about imported tables. We can consider changing the storage mode to Direct Query. It has its advantages — no cost at the model size, no long refreshes, etc. Depending on the source, reading data may be slower or much slower.</p><p>If for some reason, we need to keep data at a very high granular level, for example, at a transaction or sold product level, then we often need to deal with a large amount of data.</p><p>End reports are mainly based on aggregates. Decision-makers use highly detailed reports less often. If we deal with such a case, then we can consider user-defined aggregations.</p><p>Initially, our example contains four tables:</p><ul><li>A fact table: iowa_sales [DirectQuery] — let’s pretend that it’s a very big monster which we can’t fit into the capacity limit,</li><li>Three dimension tables [Import] — pretty smal, nothing special.</li></ul><p>In order to define user-defined aggregation, we need to add an aggregated version of our fact table. Let’s assume that we’ve made an evaluation of reports and queries. Then we typed that the most common aggregations are performed on a Category and/or Store level for expressions like: SUM(iowa_sale.SalesDollars) or COUNTROWS(iowa_sale).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GvNEYIWSckLHEFm3bMBItg.png" /></figure><p>So, we have to create an aggregated version of the iowa_sale table. Here are the settings.</p><ul><li>storage mode: <strong>Import</strong>,</li><li>relationships to dims — same as the original table. If we have matching dims, in this case, we don’t reference the iowa_date dim table, thus no relationship is created,</li><li>grouping: <strong>Category_ID</strong>, <strong>Store_ID</strong>,</li><li>aggregation columns: <strong>COUNTABLEROWS</strong>(iowa_sale), <strong>SUM</strong>(iowa_sale.Sales_Dollars)</li></ul><p>The best practice is to have such a table or view in a backend system and import it directly.</p><p>After we had imported the table and created relationships, we can do the last step. In the Manage aggregations menu, we need to set grouping and aggregations. The pattern is the same as in the settings I presented above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EE2bPm2NlVoiO276JgLecg.png" /></figure><p>We can verify settings in Tabular Editor in the <strong>Alternate of </strong>section for respective columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V30mVax3A4WK9hp6P9K29Q.png" /></figure><p>Alright, we created a solution; now it would be sufficient to check whether it works. DAX studio is the best tool to do the job.</p><pre>EVALUATE<br>SUMMARIZECOLUMNS(<br> iowa_category[CategoryName],<br> &quot;Qty&quot;,<br> COUNTROWS(iowa_sale)<br> )</pre><p>The aggregation table has grouping on a Category level and the <strong>COUNTROWS</strong> aggregation calculation set. Thus, it’s coherent with the query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KsGYKmpmmTD3xlsjs89AaQ.png" /></figure><p>DAX Studio confirms:</p><p><strong>&lt;matchFound&gt;</strong> means that the aggregated table has been used instead the original one.</p><p>Here’s the next proof.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*OxImu0B8z9-NUWvUuV2COQ.png" /></figure><p>The next query:</p><pre>EVALUATE<br>SUMMARIZECOLUMNS(<br> iowa_category[CategoryName], iowa_store[County],<br> &quot;sum&quot;,<br> sum(iowa_sale[SaleDollars])<br> )</pre><p>The grouping and the aggregate suit to the aggregation table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oVVn-3U7d_JTzTr9Jo3WHA.png" /></figure><p>The last query:</p><pre>EVALUATE<br>SUMMARIZECOLUMNS(<br> iowa_category[CategoryName], iowa_store[County], iowa_date[CalendarYear],<br> &quot;sum&quot;,<br> sum(iowa_sale[SaleDollars])<br> )</pre><p>We have a <strong>SUM </strong>aggregation — it exists in the aggregated table. But we don’t have grouping on a <strong>Date </strong>level, so the original iowa_sales table is queried. <strong>&lt;attemptFailed&gt;</strong> says more than a thousand words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9u344nm4zBAnUuG6iyrQYQ.png" /></figure><p>You can notice that query timings are much longer for this Direct Query table in comparison to Imported ones.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d610dc44faf5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Power BI — calculation groups]]></title>
            <link>https://michalmolka.medium.com/power-bi-calculation-groups-d2e92539f776?source=rss-7c158f145f51------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2e92539f776</guid>
            <category><![CDATA[calculation-group]]></category>
            <category><![CDATA[power-bi]]></category>
            <dc:creator><![CDATA[Michal Molka]]></dc:creator>
            <pubDate>Sun, 01 Mar 2026 17:01:00 GMT</pubDate>
            <atom:updated>2026-03-01T17:01:00.745Z</atom:updated>
            <content:encoded><![CDATA[<h3>Power BI — calculation groups</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SntLpROllbFfaV6OqQTaVA.png" /></figure><p><strong>Calculation groups</strong> are one of the topics that are all over the internet. I can’t count how many posts and tutorials are available. It looks like everyone knows them; they are obvious for developers, and there is nothing more to say about them. But still, during my work with clients, I see some monstrous models containing way too many measures. I won’t write anything new about this topic. This also isn’t a deep dive. Without further ado, let’s start with an example.</p><p>The <strong>calculation group</strong> can be added in the Model view, inside the <strong>Calculation groups</strong> section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*COmueaN0omfq7tvAzMAQtQ.png" /></figure><p>Rename the Calculation group column. This object will be used in slicers or as columns in visualizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*Ms-GsGzjsBRU6PWtLOSSmA.png" /></figure><p>Then create calculation items with the following measures.</p><pre>Current = SELECTEDMEASURE()<br>PD = CALCULATE(SELECTEDMEASURE(), PREVIOUSDAY(iowa_date[CalendarDate]))<br>PM = CALCULATE(SELECTEDMEASURE(), PREVIOUSMONTH(iowa_date[CalendarDate]))<br>PQ = CALCULATE(SELECTEDMEASURE(), PREVIOUSQUARTER(iowa_date[CalendarDate]))<br>PY = CALCULATE(SELECTEDMEASURE(), PREVIOUSYEAR(iowa_date[CalendarDate]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*73YkQ5vm8j_e2TgwkCTszQ.png" /></figure><p>We need some arguments for the created measures. SELECTEDMEASURE() will call them during calculations, depending on a context.</p><pre>AvgOfSales = CALCULATE(SUM(iowa_sale[SaleDollars]))<br>CountOfSales = CALCULATE(COUNTROWS(iowa_sale))<br>SumOfSales = CALCULATE(AVERAGE(iowa_sale[SaleDollars]))</pre><p>OK, everything is ready; it’s time for check them out in practice.</p><p>Place the [<strong>CalculationType</strong>] column into a slicer and into the column pane on the matrix. Then, some other measures you want to calculate. For example, place [<strong>SumOfSales</strong>] into the values pane and a chosen column from the Date dimension.</p><p>Here, the calculation group column on a slicer works like a filter for columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*oK3Uc-0Qj4q8BNSupXsYew.png" /></figure><p>We can add more measures to the table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0RNjHkXnFTwD3awxDsMh5Q.png" /></figure><p>With one or two period types, it is readable. But, we can go wild and select everything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ntog23l77y5NhxV5kSVKBw.png" /></figure><p>We aren’t restricted only to the matrix visual. It look pretty good as well on a chart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ApVxnHq_Z-lgldylnEHBDA.png" /></figure><p>Creating and editing calculation groups weren’t supported by Power BI Desktop for a long time. To avoid this inconvenience, developers were using Tabular Editor. So, we can compare how it looks in TE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89090k01iwmi4XXOJYv9pw.png" /></figure><p>Why calculation groups, you may ask. In this case, we have three measures and five calculation items. With these objects, we can create fifteen combinations of calculations. When we need more combinations, let’s say 10 calculation items and 10 measures. Then, instead of 20 objects, we would need to create 100 measures.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2e92539f776" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Power BI — TMDL — Tabular Model Definition Language]]></title>
            <link>https://medium.com/microsoft-power-bi/power-bi-tmdl-tabular-model-definition-language-2cca06e87921?source=rss-7c158f145f51------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cca06e87921</guid>
            <category><![CDATA[power-bi-tutorials]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Michal Molka]]></dc:creator>
            <pubDate>Sun, 01 Feb 2026 07:01:01 GMT</pubDate>
            <atom:updated>2026-06-01T17:57:26.327Z</atom:updated>
            <content:encoded><![CDATA[<h3>Power BI — TMDL — Tabular Model Definition Language</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GU85qXVDzlsKQnmQyCtR3g.png" /></figure><p>In the last post, I touched on a control versioning topic in a Power BI context. Today’s subject is partially related. You can check it out before: <a href="https://medium.com/p/b71a45f16ca6">Power BI report — control versioning</a> — we were working with a visualization layer back then. Today we are going to look at a model layer and how we can operate on it with a TMDL language.</p><p>So, we have a model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EISVnvnaghy0xu6uJSuIag.png" /></figure><p>Like in the last post. I saved the report as a <strong>.pbip</strong> project. Thanks to this, we are able to look at TMDL files. You need to remember to have this option turned on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*kixAyqcNwrmerRA5a72oiQ.png" /></figure><p>Let’s take a use case. I created a model where every table is sourced from a separate .csv file lying on a local drive. I want to change all the files to be sourced from OneDrive. A classic approach would take us to the Tabular Editor application or have us do the job in the Power Query window.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LZFUp8gYq-AFKuVRqwbQ2w.png" /></figure><p>With TMDL we have other ways to make it work.</p><p><strong>🎁 </strong><a href="https://powerbi-masterclass.short.gy/learning-database?utm_source=medium&amp;utm_medium=medium-post&amp;utm_campaign=database-medium-post-start"><strong>Get friend links for all of our 1500&gt; Power BI learning articles here</strong></a><strong> 🎁</strong></p><p><strong>The first option: Power BI Desktop — a TMDL editor.</strong></p><p>In the TMDL editor -&gt; Script TMDL to -&gt; Script tab</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*kghGoq3UMXT2Vz5rfTpr-Q.png" /></figure><p>Then, we can edit most aspects related to the table through code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J5LRW98B24jopgYX1qAU5Q.png" /></figure><p>A change we want to apply is at the end.</p><pre>  partition &#39;Fact Sale&#39; = m<br>   mode: import<br>   source =<br>     let<br>         Source = Csv.Document(File.Contents(&quot;C:\Users\d******l\Downloads\iowa_csv\Iowa_FactSale.csv\part-00000-e1a14094-35f0-448c-b973-4bcf22858e25-c000.csv&quot;),[Delimiter=&quot;,&quot;, Columns=17, Encoding=1252, QuoteStyle=QuoteStyle.None]),<br>         #&quot;Promoted Headers&quot; = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),<br>         #&quot;Changed Type&quot; = Table.TransformColumnTypes(#&quot;Promoted Headers&quot;,{{&quot;ID&quot;, Int64.Type}, {&quot;InvoiceItemNumber&quot;, type text}, {&quot;Date&quot;, Int64.Type}, {&quot;StoreID&quot;, Int64.Type}, {&quot;CategoryID&quot;, Int64.Type}, {&quot;VendorID&quot;, Int64.Type}, {&quot;ItemID&quot;, Int64.Type}, {&quot;Pack&quot;, Int64.Type}, {&quot;BottleVolumeMl&quot;, Int64.Type}, {&quot;StateBottleCost&quot;, type number}, {&quot;StateBottleRetail&quot;, type number}, {&quot;BottlesSold&quot;, Int64.Type}, {&quot;SaleDollars&quot;, type number}, {&quot;VolumeSoldLiters&quot;, type number}, {&quot;VolumeSoldGallons&quot;, type number}, {&quot;PartitionKeyYear&quot;, Int64.Type}, {&quot;Inserted&quot;, type datetime}})<br>     in<br>         #&quot;Changed Type&quot;<br><br>  annotation PBI_ResultType = Table</pre><p>The source is a local .csv file. Let’s change it to OneDrive.</p><pre>  partition &#39;Fact Sale&#39; = m<br>   mode: import<br>   source =<br>     let<br>         Source = Csv.Document(Web.Contents(&quot;https://e***y.sharepoint.com/personal/m***e_onmicrosoft_com/Documents/iowa_csv/part-00000-e1a14094-35f0-448c-b973-4bcf22858e25-c000.csv&quot;), [Delimiter = &quot;,&quot;, Columns = 17, QuoteStyle = QuoteStyle.None]),<br>         #&quot;Promoted Headers&quot; = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),<br>         #&quot;Changed Type&quot; = Table.TransformColumnTypes(#&quot;Promoted Headers&quot;,{{&quot;ID&quot;, Int64.Type}, {&quot;InvoiceItemNumber&quot;, type text}, {&quot;Date&quot;, Int64.Type}, {&quot;StoreID&quot;, Int64.Type}, {&quot;CategoryID&quot;, Int64.Type}, {&quot;VendorID&quot;, Int64.Type}, {&quot;ItemID&quot;, Int64.Type}, {&quot;Pack&quot;, Int64.Type}, {&quot;BottleVolumeMl&quot;, Int64.Type}, {&quot;StateBottleCost&quot;, type number}, {&quot;StateBottleRetail&quot;, type number}, {&quot;BottlesSold&quot;, Int64.Type}, {&quot;SaleDollars&quot;, type number}, {&quot;VolumeSoldLiters&quot;, type number}, {&quot;VolumeSoldGallons&quot;, type number}, {&quot;PartitionKeyYear&quot;, Int64.Type}, {&quot;Inserted&quot;, type datetime}})<br>     in<br>         #&quot;Changed Type&quot;<br><br>  annotation PBI_ResultType = Table</pre><p>After the change is applied, you get a confirmation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*LIBrJ5fhtOrRKZNMhK2Tmw.png" /></figure><p>If there is an error, you will get proper information on what is wrong.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3u34A2DyyhEJ5KNNoEZ5bw.png" /></figure><p>If you don’t want to change every separate object in a different TMDL file, you can do it in a model script and handle everything in one place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3VbQbPneW3SHy6LuZUFQA.png" /></figure><p>As a second example, we can add a measure with this code.</p><pre>createOrReplace<br><br> ref table &#39;Fact Sale&#39;<br><br>  measure Sale = CALCULATE(SUM(&#39;Fact Sale&#39;[SaleDollars]))<br>   lineageTag: f01731e0-13e6-4536-abc7-081a3ce57f2d<br><br>   annotation PBI_FormatHint = {&quot;isGeneralNumber&quot;:true}</pre><p>Now, we can look at the <strong>second option: Visual Studio Code</strong> or any editor of your choice. In the case of VS Code, we can install a TMDL language extension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8gi9YWhrTDvJCnJ52TzGJw.png" /></figure><p>Now, we are going to check how the measure is reflected in the project code.</p><p>When we open a <strong>Fact Sale.tmdl</strong> file, we can track the measure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hC_RT3sj79Z3tCHvHbvApg.png" /></figure><p>…and make a change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kb1o_8HgQwx9LqG2ig0eJA.png" /></figure><p>Switch over to Power BI Desktop (restart it if the report is open). Then you can confirm that the change has been applied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2nDP4y8bpN4iER-DL8TEYQ.png" /></figure><p>One key takeaway at the end: The functionality provides a better experience in the case of:</p><ul><li>version control,</li><li>task automation,</li><li>CI/CD,</li><li>etc.</li></ul><p>There is one perk I love the most: practically seamless movement of model objects between reports. It’s only a matter of copying a part of the code you are interested in, running it in another report, and it’s done.</p><p>If you want to check related topics out, you can look at: <a href="https://medium.com/p/f3e749f818f6">Power BI — update dataset — ALM Toolkit</a></p><p>and <a href="https://medium.com/p/96582595af58">Power BI assets — seamless PRs</a></p><p><strong>💡 </strong><a href="https://powerbi-masterclass.short.gy/pbi-gpt?utm_source=medium&amp;utm_campaign=pbi-gpt-medium-post-end"><strong>MUST TRY — Power BI GPT — Personal Power BI Learning Coach</strong></a><strong>💡</strong></p><p><strong>Power BI Masterclass Article Classification</strong></p><p><strong>Level:</strong> Beginner</p><p><strong>Category:</strong> Data Model</p><p><strong>Tags:</strong> Tutorial, Data Model</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cca06e87921" width="1" height="1" alt=""><hr><p><a href="https://medium.com/microsoft-power-bi/power-bi-tmdl-tabular-model-definition-language-2cca06e87921">Power BI — TMDL — Tabular Model Definition Language</a> was originally published in <a href="https://medium.com/microsoft-power-bi">Power BI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>