<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Adil Said on Medium]]></title>
        <description><![CDATA[Stories by Adil Said on Medium]]></description>
        <link>https://medium.com/@adilsaid64?source=rss-bc57e1ee366e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nBLWFAWCzJx3q1dgSPG92w.jpeg</url>
            <title>Stories by Adil Said on Medium</title>
            <link>https://medium.com/@adilsaid64?source=rss-bc57e1ee366e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:21:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@adilsaid64/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Refactoring a FastAPI Inference Endpoint That Wouldn’t Stop Growing]]></title>
            <link>https://pub.towardsai.net/refactoring-a-fastapi-inference-endpoint-that-wouldnt-stop-growing-ef3e13a067f6?source=rss-bc57e1ee366e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef3e13a067f6</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Adil Said]]></dc:creator>
            <pubDate>Thu, 01 Jan 2026 23:16:43 GMT</pubDate>
            <atom:updated>2026-01-05T09:23:22.372Z</atom:updated>
            <content:encoded><![CDATA[<h3>FastAPI Inference Design Patterns</h3><p>It’s common for API endpoints to start simple and slowly accumulate complexity as systems evolve. Most complex systems begin this way.</p><p>In this article, I’ll walk through a refactor I worked on in a production system at work. We’ll being with what the problem looked like, why it became painful, and how I approached fixing it.</p><p>In my case at work, we had a FastAPI endpoint responsible for orchestrating inference on user data.</p><p>On paper, the flow was simple: accept user data, extract features, send those features to a model service, and return the result.</p><p>In reality, the endpoint kept growing into a branching headache.</p><p>The request payload contained not just the user data, but also metadata describing how inference should run.</p><pre>@dataclass<br>class InferencePayload:<br>  raw_data: dict<br>  user_id: str<br>  model_type: str<br>  model_version: str<br>  feature_version: str</pre><p>The payload included the model type, the model version, and the feature extraction version.</p><p>Each model lived behind its own self-hosted HTTP microservice, and each of those services evolved independently. Some expected different payload shapes, some required different preprocessing, and new versions were added regularly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ObBVNRn85Rw_LH5VJ5f6ZA.png" /><figcaption>Source image by author— Image shows the request lifecycle. A single request would be routed to one of the models based on the payload contents</figcaption></figure><p>The business flow rarely changed but implementation details changed constantly. All of that change accumulated in the API layer.</p><h3>How the endpoint became a problem</h3><p>The original implementation handled all variation directly inside the endpoint.</p><p>It relied on branching logic to decide which model to call and which feature extraction logic to use.</p><p>At first, this was manageable. With only a couple of models and feature versions, a few if statements felt reasonable.</p><p>Over time, as more models were deployed alongside new feature extraction versions, the endpoint slowly turned into a dense block of conditional logic. Bellow is a code example of the structure:</p><pre>@app.post(&quot;/infer&quot;)<br>def infer(payload: InferencePayload):<br>    # Feature extraction<br>    if payload.feature_version == &quot;v1&quot;:<br>        features = extract_features_v1(payload.user_data)<br>    elif payload.feature_version == &quot;v2&quot;:<br>        features = extract_features_v2(payload.user_data)<br>    else:<br>        raise HTTPException(status_code=400, detail=&quot;Unsupported feature version&quot;)<br>    <br>    # Model selection<br>    if payload.model_type == &quot;model_a&quot;:<br>        # Some model types could have additional if statements<br>        model_payload = {<br>            &quot;user_features&quot;: features,<br>            &quot;metadata&quot;: payload.metadata,<br>        }<br>        response = requests.post(<br>            f&quot;http://model-a-version-{payload.model_version}/predict&quot;,<br>            json=model_payload,<br>            timeout=2,<br>        )<br>    elif payload.model_type == &quot;model_b&quot;:<br>        model_payload = {<br>            &quot;features&quot;: features,<br>            &quot;context&quot;: payload.context,<br>        }<br>        response = requests.post(<br>            f&quot;http://model-b-version-{payload.model_version}/predict&quot;,<br>            json=model_payload,<br>            timeout=2,<br>        )<br>    else:<br>        raise HTTPException(status_code=400, detail=&quot;Unsupported model type&quot;)<br>    return response.json()</pre><p>Every new model required editing the endpoint. Even when the business logic stayed the same, the API code changed.</p><p>Code reviews became harder, regressions became more likely, the endpoint became harder to test, and understanding what each new branch did required reading more and more code.</p><p>The problem was that we were expressing runtime variation using conditionals.</p><h3>Reframing the problem</h3><p>When we stepped back, two things stood out.</p><ol><li>Feature extraction varied.</li><li>Model inference varied.</li></ol><p>Everything else stayed the same.</p><p>What we needed was not a cleaner endpoint with fewer if statements, but a way to make those variations explicit, isolate them, and select between them at runtime without constantly editing the API layer.</p><p>This led to a single refactor that introduced three ideas:</p><ol><li>Interfaces to define contracts</li><li>Strategies and Adapters to hide implementation details</li><li>Factory Registry to handle selection</li></ol><h3>Feature extraction as a contract</h3><p>We started by defining a simple interface for feature extraction.</p><pre>from abc import ABC, abstractmethod<br><br>class FeatureExtractor(ABC):<br>    @abstractmethod<br>    def extract(self, user_data: dict) -&gt; dict:<br>        pass</pre><p>Each feature extraction version became its own implementation of that interface.</p><pre>class FeatureExtractorV1(FeatureExtractor):<br>    def extract(self, user_data: dict) -&gt; dict:<br>        # feature extraction logic here<br>        return {...}</pre><pre>class FeatureExtractorV2(FeatureExtractor):<br>    def extract(self, user_data: dict) -&gt; dict:<br>        # feature extraction logic here<br>        return {...}</pre><p>The endpoint no longer needed to know how features were produced. It only needed something that could produce them.</p><h3>Making feature selection explicit</h3><p>Feature extractors were registered in a small factory.</p><pre>class FeatureExtractorFactory:<br>    def __init__(self):<br>        self._extractors: dict[str, FeatureExtractor] = {}<br><br>    def register(self, version: str, extractor: FeatureExtractor):<br>        self._extractors[version] = extractor<br><br>    def get(self, version: str) -&gt; FeatureExtractor:<br>        if version not in self._extractors:<br>            raise ValueError(f&quot;Unsupported feature version: {version}&quot;)<br>        return self._extractors[version]</pre><p>Feature extractors are registered once at startup. Adding a new version means writing a new implementation and registering it.</p><pre>feature_factory = FeatureExtractorFactory()<br>feature_factory.register(&#39;V1&#39;, FeatureExtractorV1())<br>feature_factory.register(&#39;V2&#39;, FeatureExtractorV2())</pre><h3>Adapting model services</h3><p>Each model lived behind its own HTTP service, with its own payload format and response shape.</p><p>We handled this the same way: by defining a stable interface and hiding the details behind adapters.</p><pre>class ModelService(ABC):<br>    @abstractmethod<br>    def categorise(self, features: dict) -&gt; dict:<br>        pass</pre><p>Each model implementation handled payload translation and HTTP communication internally.</p><pre>class ModelAService(ModelService):<br>    def prep_payload(self, features: dict) -&gt; dict:<br>        payload = ...<br>        return payload<br>    def categorise(self, features: dict) -&gt; dict:<br>        model_url = f&quot;http://model-a-version-{features.version}/predict&quot;<br>        payload = self.prep_payload(features)<br>        response = requests.post(<br>            model_url,<br>            json=payload,<br>            timeout=2,<br>        )<br>        return response.json()</pre><pre>class ModelBService(ModelService):<br>    def prep_payload(self, features: dict) -&gt; dict:<br>        payload = ...<br>        return payload<br>    def categorise(self, features: dict) -&gt; dict:<br>        model_url = f&quot;http://model-b-version-{features.version}/predict&quot;<br>        payload = self.prep_payload(features)<br>        response = requests.post(<br>            model_url,<br>            json=payload,<br>            timeout=2,<br>        )<br>        return response.json()</pre><p>All HTTP and payload-specific logic moved out of the API layer.</p><h3>Selecting models without branching</h3><p>Model selection followed the same approach as feature extraction.</p><pre>class ModelServiceFactory:<br>    def __init__(self):<br>        self._services: dict[str, ModelService] = {}<br><br>    def register(self, model_type: str, service: ModelService):<br>        self._services[model_type] = service<br><br>    def get(self, model_type: str) -&gt; ModelService:<br>        if model_type not in self._services:<br>            raise ValueError(f&quot;Unsupported model type: {model_type}&quot;)<br>        return self._services[model_type]</pre><p>Adding a new model means creating a new adapter and registering it. Existing code paths remain untouched.</p><pre>model_factory = ModelServiceFactory()<br>model_factory.register(&#39;ModelA&#39;, ModelAService())<br>model_factory.register(&#39;ModelB&#39;, ModelBService())</pre><h3>Wiring everything together with FastAPI</h3><p>FastAPI’s dependency injection system handles runtime selection. The request payload determines which implementations are used, and the endpoint receives fully constructed dependencies.</p><pre>def get_feature_extractor(payload: InferencePayload) -&gt; FeatureExtractor:<br>    return feature_factory.get(payload.feature_version)</pre><pre>def get_model_service(payload: InferencePayload) -&gt; ModelService:<br>    return model_factory.get(payload.model_type)</pre><h3>The final endpoint</h3><p>After the refactor, the endpoint became small and stable.</p><pre>@app.post(&quot;/infer&quot;)<br>def infer(<br>    payload: InferencePayload,<br>    feature_extractor: FeatureExtractor = Depends(get_feature_extractor),<br>    model_service: ModelService = Depends(get_model_service),<br>):<br>    <br>    features = feature_extractor.extract(payload.user_data)<br>    prediction = model_service.categorise(features)<br><br>    return prediction</pre><p>The endpoint now expresses only the business flow. It doesn’t care:</p><ul><li>How features are extracted</li><li>How models are called</li><li>What payload shape each model expects</li></ul><p>If the endpoint changes now, it’s because the business logic changed, not because a new model was introduced.</p><h3>What changed in practice</h3><p>After this refactor, adding a new model or feature extraction version became a predictable process.</p><p>A developer writes a new adapter or strategy, registers it in the factory, and deploys.</p><p>The endpoint stopped being a headache that changed every week. The complexity didn’t disappear, but it moved to places where it was easier to reason about.</p><p>Isolating feature extraction and model inference behind interfaces also made testing and mocking individual components much simpler.</p><h3>Tradeoffs and closing thoughts</h3><p>To be honest, this approach introduces more abstractions and more files.</p><p>For a small system with only one or two models, it would probably be unnecessary. But once models and feature versions became open-ended, keeping the API layer stable was worth the additional structure.</p><p>This refactor didn’t eliminate complexity, but it gave us a structure that could evolve without constant changes to the API layer.</p><p>Let me know what you think, I’m curious how others would have apprached this problem. What would you have done differently?</p><p>Note, the examples here are simplified and the real endpoint handled additional responsibilities, but the core problem was the same.</p><p>Although this post focuses on architecture, the refactor was guided by a golden master–style test to ensure the system’s behavior didn’t change. The goal was to improve the look and structure of the code while preserving existing functionality.</p><p>Thanks for reading. Find me on <a href="https://www.linkedin.com/in/adil-s64/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef3e13a067f6" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/refactoring-a-fastapi-inference-endpoint-that-wouldnt-stop-growing-ef3e13a067f6">Refactoring a FastAPI Inference Endpoint That Wouldn’t Stop Growing</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG in Practice: Exploring Versioning, Observability, and Evaluation in Production Systems]]></title>
            <link>https://pub.towardsai.net/rag-in-practice-exploring-versioning-observability-and-evaluation-in-production-systems-85dc28e1d9a8?source=rss-bc57e1ee366e------2</link>
            <guid isPermaLink="false">https://medium.com/p/85dc28e1d9a8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Adil Said]]></dc:creator>
            <pubDate>Wed, 11 Jun 2025 18:08:09 GMT</pubDate>
            <atom:updated>2025-06-12T12:12:52.633Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve seen a few posts on LinkedIn recently declaring RAG systems are dead.</p><p>The core argument? “Context windows are getting bigger, so who needs retrieval anymore?”</p><p>It got me thinking. RAG only really entered the mainstream about a year or two ago. So why the premature funeral?</p><p>Sure, context windows have expanded, but <strong>does that eliminate the need for retrieval?</strong></p><p>I don’t think so. Here’s why:</p><ul><li><strong>Cost &amp; latency</strong> scale linearly with context size. Dumping your entire knowledge base into a prompt isn’t efficient.</li><li><strong>Relevance filtering</strong> still matters. RAG gives you control over what context is passed.</li><li><strong>Traceability</strong> and <strong>source attribution</strong> are key in regulated or high-stakes domains.</li><li><strong>Most enterprise knowledge bases exceed even long context windows.</strong></li></ul><p>So no, I don’t think RAG is dead. But it is maturing alongside LLM-powered applications.</p><p>Which is why I wanted to do this project. I wanted to explore <strong>what LLMOps really means in practice, </strong>essentially <strong>MLOps, adopted for large language models</strong>. So I decided to explore RAG systems.</p><p>Here are the questions I wanted to answer while doing this project:</p><ul><li><strong>How do we track and evaluate quality without labelled data?</strong></li></ul><p>Most traditional machine learning (ML) systems rely on ground-truth labels. RAG systems might not have that luxury.</p><ul><li><strong>What does observability mean in a retrieval-based or LLM-powered system?</strong></li></ul><p>It’s not just about latency and uptime, but how do we get visibility into retrieval quality, context relevance, and user satisfaction?</p><ul><li><strong>How do you design a stack that’s modular, monitorable, and scalable?</strong></li></ul><p>Microservices? Monolith? Where do these services live, and how do they talk to each other?</p><ul><li><strong>What does model versioning mean if you’re not training the model yourself?</strong></li></ul><p>You might be using hosted or open-weight models, so how do you track upgrades, API changes, or fine-tuning?</p><ul><li><strong>What does data versioning look like when your “data” is a knowledge base, not training sets?</strong></li></ul><p>In RAG systems, your documents are your data. So, how do you track changes across documents and embeddings? How do you track prompt templates?</p><p>Before diving in, it’s worth saying that this field is a rapidly evolving space.</p><p>What’s considered best practice today might shift tomorrow.</p><p>One thing to know is that the core principles of observability, evaluation/testing, reproducibility, modularity and versioning always hold up.</p><h3>System Architecture Overview</h3><p>To explore these questions hands-on, I built a RAG system.</p><p>It’s fully containerised and orchestrated with Docker Compose, so you can spin it up yourself locally. GitHub link at the bottom of the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/0*9D9g2g4-UcCrVxG4.png" /><figcaption>Source: Image by the author — System Architecture Overview — Arrows showcase the flow of data and which services talk to which</figcaption></figure><p>I went for a <strong>microservice architecture</strong>, with each core part of the RAG pipeline handled by its own containerised service. At the centre is a <a href="https://fastapi.tiangolo.com/"><strong>FastAPI </strong></a><strong>orchestrator</strong>, which exposes three endpoints — /metrics, /embed, and /query — and acts as the glue that coordinates ingestion, embedding, vector storage, retrieval, generation, and logging.</p><p>This approach gives a few benefits:</p><ul><li>Easier to swap components (e.g. different embedding or generation models)</li><li>Clear separation of concerns</li><li>Scalable services that can evolve independently</li></ul><p>In the next sections, I explore how I approached the core principles I mentioned earlier — versioning, observability, evaluation, and model deployment and what they look like in a real RAG system.</p><h4>Data Versioning</h4><p>In traditional ML workflows, data versioning usually refers to tracking training, validation, and test datasets. It’s mostly about ensuring reproducibility and understanding how model performance changes as the data evolves. It’s also just good practice — it gives you a safety net, so you can roll back if something breaks or goes sideways.</p><p>But in the context of <strong>RAG systems</strong>, you’re most likely not training your own model, so you don’t have training data in the traditional sense. So, what does data versioning mean in this context?</p><p>Instead, your “data” is the <strong>knowledge base</strong>, the raw documents, articles, PDFs, and internal content that you embed and retrieve from. That means versioning here is about ensuring that your embeddings and responses can be traced back to the specific version of a document they came from.</p><p>This becomes important because:</p><ul><li>Documents change over time, and so do the embeddings.</li><li>Once embedded, content is detached from its source version.</li><li>Without tracking, it’s impossible to know which version of a document influenced a model’s response.</li></ul><p>To handle this, I used<strong> </strong><a href="https://lakefs.io/"><strong>lakeFS</strong> </a>to version the document store. When a document is ingested, it’s committed to a lakeFS repo. The resulting <strong>commit hash</strong> is stored alongside the document’s embedding in <a href="https://qdrant.tech/"><strong>Qdrant</strong></a>. That way, every embedding is tied to a specific snapshot of the document state at the time of ingestion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*uJy9GrVwqcd3yjLo5GfrAg.png" /><figcaption>Source: Image by the author — How this system is preserving data lineage between embeddings and documents</figcaption></figure><p>One way to improve traceability even further is by tracking <strong>how documents were split and tagged</strong>, and <strong>which embedding model</strong> was used.</p><p>Small changes in chunking logic or model versions can lead to very different results. You could do this by recording the <strong>Git commit/tag</strong> or <strong>Docker image hash</strong> that performed the ingestion, and storing the <strong>embedding model version</strong> as metadata. This would make it easier to trace back exactly how a particular answer was generated.</p><h4>Observability</h4><p>One of the things I wanted to understand while building this system was: what does observability even look like in a RAG pipeline?</p><p>In traditional ML, you’re usually monitoring things like model accuracy or drift over time. But with RAG, there’s more going on. You’re not just calling a model — you’re ingesting documents, embedding them, storing vectors, retrieving chunks, generating text, and logging results. I wanted some visibility into that whole process.</p><p>To get started, I added monitoring using <a href="https://prometheus.io/"><strong>Prometheus</strong> </a>and <a href="https://grafana.com/"><strong>Grafana</strong></a>. I exposed an /metrics endpoint in the FastAPI orchestrator, which Prometheus scrapes on a regular interval. Grafana is then used to visualise that data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1u6s5vnQx8dmlisy.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WrpK-tGbnMTxDR22.png" /><figcaption>Source: Image by the author — Grafana Dashboard</figcaption></figure><p>Here’s what I decided to monitor:</p><ul><li><strong>Ingestion latency</strong>: how long it takes to process and embed a document.</li><li><strong>Query latency</strong>: total time to return an answer to the user. I wanted to understand the end-to-end performance.</li><li><strong>Context retrieval stats</strong>: how many chunks were retrieved and what their relevance scores were. This gave me some idea of how well the retrieval step was working.</li><li><strong>Generation health</strong>: counts of failures, timeouts, or any exceptions during the model response. Basically, is the model behaving or not?</li><li><strong>Basic usage metrics</strong>: counters for how many ingestion and query requests were made — useful just to get a pulse on what’s happening.</li></ul><p>There’s definitely more I could be tracking — things like user feedback loops, context drift over time, or retrieval quality across different document sets. But even with this basic setup, it gave me a clearer picture of what’s happening under the hood. It also sets up the structure to add more metrics in the future.</p><h4>Evaluating and Testing RAG</h4><p>One of the biggest challenges I ran into was figuring out how do you actually evaluate a RAG system.</p><p>In traditional ML, you usually have a clear ground truth — a label, a category, a number — and you can compare your model’s output against that. With LLMs, especially in open-ended question-answering, it’s rarely that clear-cut. There isn’t always one “correct” answer.</p><p>So, how do you measure quality?</p><p>First, we need a way to track model outputs. I decided to log user queries, the retrieved documents, and the generated response in <a href="https://www.mongodb.com/">MongoDB</a>. This gave me a structured dataset I could evaluate offline. One way to do this is with <a href="https://github.com/explodinggradients/ragas">RAGAS</a>, a tool that uses LLMs themselves to score responses across a few key dimensions:</p><ul><li><strong>Retrieval Precision</strong> — Were the retrieved documents actually relevant to the query?</li><li><strong>Groundedness</strong> — Is the generated answer supported by the retrieved content?</li><li><strong>Factual Consistency</strong> — Does the answer reflect what the documents actually say?</li><li><strong>Completeness</strong> — Does the answer fully address the user’s question?</li></ul><p>Each evaluation run gets logged in <a href="https://mlflow.org/">MLflow</a>, so I can compare results across different model versions, prompt templates, or retrieval strategies.</p><p>Of course, using an LLM to evaluate another LLM has its own challenges — you’re introducing another layer of subjectivity. But in the absence of labelled data, it’s a helpful starting point.</p><p>I’ve also thought about incorporating lightweight human-user feedback. Something like a thumbs up/down on each answer, to collect signals over time. Even simple binary ratings can be valuable for identifying patterns or spotting failure cases.</p><p>There’s more to explore here. Things like long-term answer consistency, hallucination detection, or task-specific scoring would be great next steps. But this setup gave me enough structure to start reasoning about quality.</p><h4>Model Deployment</h4><p>For this project, I deployed two separate services using BentoML:</p><ul><li>An <strong>embedding service</strong> using all-MiniLM-L6-v2</li><li>A <strong>generation service</strong> using EleutherAI/gpt-neo-1.3B</li></ul><p>These models were chosen arbitrarily. The goal wasn’t performance tuning, just to get a working setup to explore system design and deployment patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*LOESmuDaGmx0ZXzlwbgk1A.png" /><figcaption>Source: Image by the author — Query flow</figcaption></figure><p>Keeping them separate gave me more control during experimentation, and BentoML made it easy to spin up standalone model servers with clear API contracts. But realistically, <strong>you probably wouldn’t self-host a generation model in production</strong> — not unless you have serious infrastructure and ops resources. Hosting your own LLM is expensive, operationally complex, and often less efficient than using managed APIs.</p><p>So what if you’re using a hosted LLM, like from OpenAI, Anthropic, or OpenRouter?</p><p>In that case, you’re not training or fine-tuning anything, and <strong>the model can silently change behind the scenes</strong>. That makes model versioning tricky, but still important. Here’s what you can do:</p><ul><li><strong>Log the model name and version string</strong> (if available) with every request</li><li><strong>Store the prompt and output pairs</strong> to compare behaviour over time</li><li><strong>Periodically re-run fixed evaluation sets</strong> (same prompts, same documents) to catch shifts in answer quality or consistency</li></ul><p>Even when using a managed API, I think versioning still plays an important role. It’s less about checkpoints or model weights and more about keeping track of how the model behaves at a given time, so you can spot any unexpected changes or regressions that might affect users later on.</p><h3>What I Would Change</h3><p>Right now, the system follows a <strong>synchronous orchestration</strong> pattern using FastAPI and HTTP between services. It works fine for now, but I’d like to explore an <strong>event-driven architecture</strong> in the future, introducing a message broker (like <a href="https://kafka.apache.org/">Kafka</a>) so services can publish and consume events asynchronously. This also helps to decouple the current system.</p><p>As mentioned earlier, I’d also like to invest more in <strong>evaluation infrastructure</strong>. While RAGAS gave me a great starting point, combining it with <strong>live user feedback</strong> or lightweight human review would give a better picture of model performance in the real world. Having an interface where users can upvote/downvote answers or flag bad results could feed into evaluation pipelines and model iteration.</p><p>On the database side, <strong>MongoDB and Postgres support vector search</strong>, so the system could be simplified by reducing the number of services and sticking to more general-purpose infrastructure. I went with Qdrant here to explore a purpose-built vector DB — and it’s been great — but in a production setting, consolidating services could reduce operational complexity.</p><h3>Wrapping Up</h3><p>This project was my way of getting hands-on with the real-world challenges of deploying RAG systems — not just building something that works, but trying to understand what “production-ready” actually means in this space.</p><p>There’s still plenty I’d like to explore: event-driven orchestration, tighter evaluation loops, smarter feedback collection, and maybe even experimenting with different model serving setups. As mentioned earlier, evaluation especially feels like a space that deserves more thought — if you’ve tackled this before, I’d love to learn from your experience.</p><p>And if you’re reading this thinking I’ve missed something important, I probably have. This project was a learning exercise, and there’s always more to uncover. I’m always open to feedback or ideas, so don’t hesitate to connect or drop me a message.</p><p><a href="https://github.com/adilsaid64/open-rag-stack">GitHub - adilsaid64/open-rag-stack: A playground for building and serving Retrieval-Augmented Generation (RAG) systems using best practices in MLOps and LLMOps, with open-source tools.</a></p><p>Let&#39;s connect: <a href="https://www.linkedin.com/in/adil-s64/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85dc28e1d9a8" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/rag-in-practice-exploring-versioning-observability-and-evaluation-in-production-systems-85dc28e1d9a8">RAG in Practice: Exploring Versioning, Observability, and Evaluation in Production Systems</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting Up Apache Kafka with Docker]]></title>
            <link>https://blog.dataengineerthings.org/setting-up-apache-kafka-with-docker-ad33ac88f6af?source=rss-bc57e1ee366e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad33ac88f6af</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Adil Said]]></dc:creator>
            <pubDate>Thu, 09 Jan 2025 22:02:30 GMT</pubDate>
            <atom:updated>2025-01-09T22:02:30.762Z</atom:updated>
            <content:encoded><![CDATA[<p>This tutorial will guide you through setting up Apache Kafka using Docker and creating basic Python producers and consumers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*leoMIUgnVA83JT-_FgIeiA.png" /></figure><p>This guide pairs well with my other article <a href="https://medium.com/@adilsaid64/apache-kafka-explained-be4b6f6d1591"><strong>Apache Kafka Explained</strong></a> which covers the internals and architecture of Kafka. That article covers concepts like topics, partitions, and more which are used in this article.</p><p>Having an understanding of the internals of Kafka first can help make sense of what some of the code below is doing.</p><p><a href="https://medium.com/@adilsaid64/apache-kafka-explained-be4b6f6d1591">Apache Kafka Explained</a></p><h4>Why do it in Docker?</h4><p>Docker packages everything we need into containers, making it easy to start, stop and deploy our Kafka setup with just a few commands.</p><h3>Prerequisites</h3><ul><li><a href="https://www.docker.com/products/docker-desktop/">Docker and Docker</a> Compose are installed</li><li>Python 3.7+</li><li>Basic understanding of messaging systems</li></ul><h3>Part 1: Project Structure</h3><p>Create the following project structure:</p><pre>kafka-docker-python/<br>├── docker-compose.yml<br>├── producer/<br>│   ├── Dockerfile<br>│   ├── requirements.txt<br>│   └── producer.py<br>├── consumer/<br>│   ├── Dockerfile<br>│   ├── requirements.txt<br>│   └── consumer.py<br>└── kafka-init/<br>    ├── Dockerfile<br>    └── init-kafka.sh</pre><p>We will be creating Python-based producers and consumers. The producer will generate data into Kafka and the consumer will subscribe to the Kafka topic, consume the data, and print it to the console.</p><h3>Part 2: Setting up Kafka with Docker</h3><p>We will be using Docker Compose to configure our setup. A docker-compose is like a blueprint that tells Docker how to set up multiple containers and how they should work together.</p><p>In your docker-compoase.yml paste the following:</p><pre>version: &#39;3&#39;<br><br># custom networks for container communication<br>networks:<br>  kafka-network:<br>    name: kafka-network<br>    driver: bridge<br><br>services:<br>  # zookeeper service - needed for Kafka coordination and broker management<br>  zookeeper:<br>    image: confluentinc/cp-zookeeper:latest<br>    networks:<br>      - kafka-network<br>    environment:<br>      - ZOOKEEPER_CLIENT_PORT=2181<br>      - ZOOKEEPER_TICK_TIME=2000    <br>    ports:<br>      - &quot;2181:2181&quot;                     # expose zookeeper port to host<br>    healthcheck:<br>      test: [ &quot;CMD&quot;, &quot;nc&quot;, &quot;-z&quot;, &quot;localhost&quot;, &quot;2181&quot; ]  # check if zookeeper is listening<br>      interval: 10s                      <br>      timeout: 5s        <br>      retries: 5  <br><br>  # kafka broker service<br>  kafka:<br>    image: confluentinc/cp-kafka:latest<br>    networks:<br>      - kafka-network<br>    depends_on:<br>      zookeeper:<br>        condition: service_healthy       # wait for zookeeper to be healthy<br>    ports:<br>      - &quot;9092:9092&quot;                    <br>    environment:<br>      - KAFKA_BROKER_ID=1               <br>      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181  # zookeeper connection string<br>      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092  # address clients will use to connect<br>      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT<br>      - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT<br>      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1    <br>    healthcheck:<br>      test: [ &quot;CMD&quot;, &quot;kafka-topics&quot;, &quot;--bootstrap-server&quot;, &quot;localhost:9092&quot;, &quot;--list&quot; ]  # verify kafka is operational<br>      interval: 30s<br>      timeout: 10s<br>      retries: 3<br><br>  # kafka initialisation service - sets up initial topics and other configs can be added here<br>  init-kafka:<br>    build:<br>      context: ./kafka-init<br>      dockerfile: Dockerfile<br>    networks:<br>      - kafka-network<br>    depends_on:<br>      kafka:<br>        condition: service_healthy <br><br>  # producer service - generates and sends messages to Kafka<br>  producer:<br>    build: ./producer<br>    networks:<br>      - kafka-network<br>    depends_on:<br>      kafka:<br>        condition: service_healthy<br>      init-kafka:<br>        condition: service_completed_successfully  # waits for initialisation to complete<br>    environment:<br>      KAFKA_BOOTSTRAP_SERVERS: kafka:9092 <br><br>  # consumer service - receives and processes messages from kafka<br>  consumer:<br>    build: ./consumer<br>    networks:<br>      - kafka-network<br>    depends_on:<br>      kafka:<br>        condition: service_healthy<br>      init-kafka:<br>        condition: service_completed_successfully<br>    environment:<br>      KAFKA_BOOTSTRAP_SERVERS: kafka:9092</pre><p>This script will set up 5 services. The first is Zookeeper. Zookeeper helps to manage our Kafka brokers. Kafka won&#39;t start without this so this needs to run first.</p><p>Now run in your terminal from the root directory:</p><pre>docker-compose up --build</pre><p>Once you build you can close by running:</p><pre>docker-compose down -v</pre><p>Now we need to set up our Kafka initialisation script. This will create our Kafka topic.</p><p>In your kafka-init/init-kafka.sh file paste the following lines.</p><pre>#!/bin/bash<br><br># wait for kafka using a simple connection check<br>echo &quot;Waiting for Kafka to be ready...&quot;<br>while ! nc -z kafka 9092; do <br>  sleep 1<br>done<br><br>echo &quot;Creating Kafka topics...&quot;<br>kafka-topics \<br>    --bootstrap-server kafka:9092 \<br>    --create \<br>    --if-not-exists \<br>    --topic test_topic \<br>    --replication-factor 1 \<br>    --partitions 2 \<br>    --config cleanup.policy=delete \<br>    --config retention.ms=604800000<br><br>echo &quot;Successfully created topics:&quot;<br>kafka-topics --bootstrap-server kafka:9092 --list<br><br>echo &quot;Topic details:&quot;<br>kafka-topics \<br>    --bootstrap-server kafka:9092 \<br>    --describe \<br>    --topic test_topic</pre><p>Create a Dockerfile for the Kafka initialisation. kafka-init/Dockerfile .</p><pre># kafka-init/Dockerfile<br>FROM confluentinc/cp-kafka:latest<br><br># Create a directory for logs with proper permissions<br>RUN mkdir -p /tmp/kafka-logs &amp;&amp; \<br>    chmod -R 777 /tmp/kafka-logs<br><br># Set log directory environment variable<br>ENV LOG_DIR=/tmp/kafka-logs<br><br>COPY --chmod=0755 init-kafka.sh /init-kafka.sh<br>CMD [&quot;/init-kafka.sh&quot;]</pre><p>This initialisation script will create a Kafka topic called test_topic with 2 partitions.</p><p>Create requirements.txt in both producer and consumer directories:</p><pre>kafka-python==2.0.2</pre><p>In the Dockerfile for the producer (producer/Dockerfile):</p><pre>FROM python:3.9-slim<br><br>WORKDIR /app<br>COPY requirements.txt .<br>RUN pip install --no-cache-dir -r requirements.txt<br><br>COPY producer.py .<br>CMD [&quot;python&quot;, &quot;producer.py&quot;]</pre><p>In the Dockerfile for the consumer (consumer/Dockerfile):</p><h3>Part 3: Creating a Basic Producer</h3><p>In the file named producer.py:</p><pre>from kafka import KafkaProducer<br>from typing import Dict, Any<br>import json<br>import time<br>import os<br>import random<br><br>def create_producer() -&gt; KafkaProducer:<br>    &quot;&quot;&quot;Create a Kafka producer with default settings.&quot;&quot;&quot;<br>    return KafkaProducer(<br>        bootstrap_servers=[os.environ.get(&#39;KAFKA_BOOTSTRAP_SERVERS&#39;, &#39;kafka:9092&#39;)],<br>        value_serializer=lambda x: json.dumps(x).encode(&#39;utf-8&#39;),<br>        key_serializer=lambda k: k.encode(&#39;utf-8&#39;)<br>    )<br><br><br>def send_message(producer: KafkaProducer, topic: str, key: str, message: Dict[str, Any]) -&gt; None:<br>    &quot;&quot;&quot;Send a message to a Kafka topic.&quot;&quot;&quot;<br>    try:<br>        future = producer.send(topic, key=key, value=message)<br>        metadata = future.get(timeout=5)<br>        print(f&quot;Message sent - Key: {key}, Partition: {metadata.partition}, Offset: {metadata.offset}&quot;)<br>    except Exception as e:<br>        print(f&quot;Error sending message: {repr(e)}&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    producer = create_producer()<br>    topic = &quot;test_topic&quot;<br>    i = 0<br>  <br>    unique_ids = [&#39;id1&#39;, &#39;id2&#39;, &#39;id3&#39;, &#39;id4&#39;]<br>    message_counts = {uid: 0 for uid in unique_ids}<br>    try:<br>        while True:<br>            key = random.choice(unique_ids)<br>            message_counts[key] += 1<br>            message = {<br>                &#39;unique_id&#39; : key,<br>                &quot;message_count&quot;: message_counts[key],<br>                &quot;timestamp&quot;: time.time(),<br>                &quot;message&quot;: f&quot;Test message {i}&quot;<br>            }<br>            send_message(producer, topic, key, message)<br>            i += 1<br>            time.sleep(1)<br>    finally:<br>        producer.close()</pre><p>By using unique_idas the message key in our Kafka producer, we ensure all messages with the same unique_idare processed in order. This works because Kafka guarantees message order within each partition, and messages with the same key always go to the same partition.</p><p>You can check for this when reading the log messages from the consumer.</p><h3>Part 4: Creating a Basic Consumer</h3><p>In the file named consumer.py:</p><pre>from kafka import KafkaConsumer<br>from typing import Any<br>import json<br>import os<br><br>def create_consumer(topic: str) -&gt; KafkaConsumer:<br>    &quot;&quot;&quot;Create a Kafka consumer with default settings.&quot;&quot;&quot;<br>    return KafkaConsumer(<br>        topic,<br>        bootstrap_servers=[os.environ.get(&#39;KAFKA_BOOTSTRAP_SERVERS&#39;, &#39;kafka:9092&#39;)],<br>        auto_offset_reset=&#39;earliest&#39;,<br>        enable_auto_commit=True,<br>        group_id=&#39;my_consumer_group&#39;,<br>        value_deserializer=lambda x: json.loads(x.decode(&#39;utf-8&#39;))<br>    )<br><br>if __name__ == &quot;__main__&quot;:<br>    topic = &quot;test_topic&quot;<br>    consumer = create_consumer(topic)<br><br>    try:<br>        print(&quot;Starting consumer... waiting for messages&quot;)<br>        for message in consumer:<br>            print(f&quot;&quot;&quot;<br>                Topic: {message.topic}<br>                Partition: {message.partition}<br>                Offset: {message.offset}<br>                Key: {message.key}<br>                Value: {message.value}<br>            &quot;&quot;&quot;)<br>    except KeyboardInterrupt:<br>        print(&quot;Stopping consumer...&quot;)<br>    finally:<br>        consumer.close()</pre><h3>Part 5: Running the Example</h3><p>Build and start all services:</p><pre>docker-compose up --build</pre><p>This will:</p><ol><li>Start Zookeeper</li><li>Start Kafka (waiting for Zookeeper to be healthy)</li><li>Start Kafka initialisation script to create our Kafka topic</li><li>Build and start the Python producer</li><li>Build and start the Python consumer</li></ol><p>Thanks to the dependency and health check configurations in the docker-compose.ymlfile, the services will start in the correct order.</p><p>Once you are done and want to stop the containers and all associated mock data we just produced. Run:</p><pre>docker-compose down -v</pre><h3>Some Screenshots</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HM9pONlIwXxAOo7PCv7WJQ.png" /><figcaption>Topic details</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*MxS1i30bf5ZwkLhGauoGcQ.png" /><figcaption>Producer Logs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5aKuZLsYHGcFvrz02MbXsw.png" /><figcaption>Consumer Logs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-G-Jn5rDRN6_qZE08Th1BA.png" /><figcaption>Docker Desktop</figcaption></figure><h3>End</h3><p>Hope this guide was useful. You can now use this setup as a foundation for building more complex Kafka projects.</p><p>Feel free to reach out in the comments below or message me directly. I’m always happy to help troubleshoot or explain concepts further.</p><p>Find me on <a href="https://www.linkedin.com/in/adil-s64/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad33ac88f6af" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/setting-up-apache-kafka-with-docker-ad33ac88f6af">Setting Up Apache Kafka with Docker</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Kafka Explained]]></title>
            <link>https://blog.dataengineerthings.org/apache-kafka-explained-be4b6f6d1591?source=rss-bc57e1ee366e------2</link>
            <guid isPermaLink="false">https://medium.com/p/be4b6f6d1591</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Adil Said]]></dc:creator>
            <pubDate>Fri, 03 Jan 2025 17:08:26 GMT</pubDate>
            <atom:updated>2025-01-09T16:56:44.675Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zpRB-ijDLslwWUh7" /><figcaption>Photo by <a href="https://unsplash.com/@hharritt?utm_source=medium&amp;utm_medium=referral">Hunter Harritt</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In modern systems, managing data flow between multiple applications has become increasingly complex. If you’re working with data-intensive applications or building event-driven systems, you’ve likely encountered Apache Kafka. But what problem does Kafka solve, and why has it become such a fundamental tool in modern architecture?</p><h4>The Data Integration Challenge</h4><p>At its core, software systems need to move data from source systems to target systems, often with transformations or aggregations along the way. In its simplest form, this might involve a straight forward point-to-point integration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*9CL4YGxaZK-_xE91C29rgA.png" /></figure><p>However, as organizations grow, this simple pattern rarely suffices. Consider a modern e-commerce company with:</p><p><strong>Source Systems:</strong></p><ul><li>Customer-facing web application generating user activity data</li><li>Mobile app tracking user behaviour</li><li>Payment processing system recording transactions</li><li>Inventory management system tracking stock levels</li><li>Order fullfilment system tracking shipments</li></ul><p><strong>Target Systems:</strong></p><ul><li>Data warehouse for business intelligence</li><li>Real-time analytics platform for monitoring</li><li>Customer relationship management (CRM) system</li><li>Fraud detection system</li><li>Recommendation engine</li><li>Email marketing platform</li></ul><p>Each source system might need to send data to multiple targets. For example, when a customer places an order:</p><ul><li>The data warehouse needs it for historical analysis</li><li>The fraud detection system needs it for real-time verification</li><li>The CRM needs it to update customer records</li><li>The inventory system needs it to adjust stock levels</li><li>The marketing platform needs it to trigger follow-up emails</li></ul><p>This type of system could look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*FWhpFVYvENHYWoxs46RRnQ.png" /><figcaption>An example of a complex point to point integration</figcaption></figure><h4>The Problems with Direct Integration</h4><p>This direct integration approach introduces several critical challenges:</p><ol><li><strong>Tight Coupling</strong>: Systems become tightly coupled, creating brittle architectures where changes in one system can cascade throughout the entire infrastructure.</li><li><strong>Complexity at Scale</strong>: As systems multiply, the number of point-to-point connections grows rapidly. With 5 source systems and 6 target systems, you could potentially need up to 30 different integrations, each with its own maintenance requirements.</li><li><strong>Maintenance Overhead</strong>: Managing and monitoring these direct integrations becomes increasingly difficult. Each connection might require:<br> — Custom error handling<br> — Data transformation logic<br> — Monitoring and alerting<br> — Rate limiting and back pressure mechanisms</li><li><strong>Limited Flexibility</strong>: Making changes to data flows or adding new consumers requires coordinating with multiple teams and systems. For instance, adding a new analytics service would require new connections to all relevant source systems.</li></ol><h3>Enter Apache Kafka: The Decoupling Layer</h3><p>This is where an event bus like Apache Kafka comes in. Instead of direct system-to-system communication, Kafka introduces a powerful publish-subscribe model that changes how systems interact.</p><p>Apache Kafka was developed by LinkedIn engineers in 2011 to handle their large-scale data feeds. It was later open-sourced and became a top-level Apache project, now widely used across industries for real-time data streaming.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/0*MPwv5cQ3NXr1CcU-.png" /></figure><h4>How Kafka Transforms the Architecture</h4><p>Kafka acts as a central system for your data, providing:</p><ol><li><strong>Decoupled Communication</strong>: Producers (source systems) publish data to Kafka without the knowledge of downstream consumers. For example, the e-commerce platform can publish order events without knowing whether they’ll be consumed by the analytics system, fraud detection, or both.</li><li><strong>Unified Interface</strong>: Consumers (target systems) retrieve data from Kafka using a standardised interface, eliminating the need for custom integration code. The fraud detection system can consume order events using the same interface as the analytics platform.</li><li><strong>Scalable Design</strong>: New producers and consumers can be added without affecting existing systems. For instance, adding a new recommendation engine only requires connecting it to Kafka, not to each individual source system.</li><li><strong>Data Persistence</strong>: Unlike traditional message queues, Kafka stores data for configured retention periods, allowing for replay and multiple consumption patterns. This enables scenarios like:<br> — Reprocessing historical data after fixing bugs<br> — Adding new analytics systems that need historical data<br> — Recovering from failures by replaying missed events</li></ol><p>The resulting architecture becomes more resilient, scalable, and maintainable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*m7QMT65rg0xtiylWnqicxA.png" /><figcaption>Kafka Based Integration</figcaption></figure><p>With this foundation established, let’s dive deeper into Kafka’s architecture and core concepts to understand how it achieves these benefits.</p><h3>Kafka Topics</h3><p>A Kafka topic is a named channel that stores a sequence of messages in a specific order. Think of it as a log where messages are continuously appended and stored.</p><p>Each topic has a unique name, like “payment_events” or “user_activity”.</p><p>Producers write messages to topics, and consumers read these messages independently.</p><p>For example, a payment service might produce messages to a “payments” topic, while fraud detection and analytics services consume from it.</p><p>Topics are format-agnostic, supporting any data structure from simple text to complex objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*OhI2xXzgoa_nzFFwzNpcCw.png" /><figcaption>An example of a Kafka setup with two topics</figcaption></figure><h3>Kafka Topic Partitions</h3><p>Let’s dive into how a topic is structured.</p><p>A Kafka topic is divided into multiple partitions, each acting as an ordered sequence of messages. When data is written to a partition, it’s assigned a sequential ID called an offset. This offset uniquely identifies each message within its partition.</p><p>Key properties:</p><ul><li>Message ordering is guaranteed only within a partition, not across partitions</li><li>Once written, messages are immutable (cannot be changed)</li><li>Offsets are never reused, even after message deletion</li><li>The producer decides which partition to write to. More on this in the next section</li><li>A consumer subscribes to a partition or multiple partitions within a topic. More on this in the consumer section</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h80rAoJhRXyYaZOB" /></figure><p>The diagram above shows a topic with three partitions, with messages ordered by their offsets within each partition. The dotted lines indicate where the next message would be written in each partition.</p><h3>Kafka Producers</h3><p>A producer is a system that writes data to Kafka.</p><p>Producers write messages to specific topic partition. Each message has an optional key that determines the partition assignment.</p><p>Messages with the same key are sent to the same partition, ensuring ordered processing of related messages. When a key is provided, the destination partition is computed from the hash of the key. This ensures that all messages with the same key are sent to the same partition.</p><p>If no key is provided, messages are distributed across partitions in a round-robin fashion. This way all partitions get an even share of the data. However, the order of messages in this case is not guaranteed. This makes keys important when message order matters (e.g., events from the same user/device).</p><p>All data is serialised to bytes before sending (using Murmur2 hash by default). The producer handles this serialisation process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*npwUi5X0RHAN_V0QpKsD1A.png" /><figcaption>Diagram to illustrate how partitioning would work if you wanted to preserve the order of events for each user. In this case the user_id is used as the key.</figcaption></figure><p>This diagram illustrates message routing in Kafka based on keys. Messages from user_123 consistently go to Partition 0, while user_456’s messages go to Partition 1. Within each partition, messages receive sequential offsets, maintaining order for each user’s events.</p><p>For messages without keys in Kafka, the distribution follows a round-robin pattern across partitions. In a topic with 2 partitions:</p><ul><li>Message 1 → Partition 0</li><li>Message 2 → Partition 1</li><li>Message 3 → Partition 0</li><li>Message 4 → Partition 1</li></ul><h3>Kafka Consumers</h3><p>A Kafka consumer pulls data from topic partitions rather than having Kafka push data to it. This pull model lets consumers control their data consumption rate and processing. Consumers can subscribe to one or multiple partitions depending on their needs.</p><p>Since Kafka stores data as binary, consumers must deserialise it using the correct schema and deserialisation method. Maintaining consistent data schemas throughout a topic’s lifecycle is crucial — if schema changes are needed, create a new topic rather than modifying an existing one to avoid breaking consumers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NuT4-o9xlUFp2KR-Ml7rtQ.png" /><figcaption>An example of a topic with three partitions. Consumer 1 consumes from partition 0 and partition 1 and Consumer 2 consumes from partition 2.</figcaption></figure><h3>Consumer Groups</h3><p>Consumer groups in Kafka allow multiple consumers to work together to process data from a topic in a distributed manner. When a topic has multiple partitions, consumers in the same group divide these partitions among themselves. Each partition is read by one consumer within a group. This is to avoid preprocessing the same data within the group. However, one consumer can read multiple partitions.</p><p>You can have more consumers in a group than you have partitions. In this case, you will have inactive consumers within your group. If a consumer in your group fails, then the inactive consumer can take over.</p><p>Multiple consumer groups can read from the same topic independently.</p><p>Why would you want to have multiple consumer groups?</p><p>Suppose you have two services, like analytics and fraud detection. You can create a group for each service. Within each group, having multiple consumers allows you to distribute the processing of data for each service. You can process multiple partitions in parallel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ee5jhWS2F2TmE_QMa7AowQ.png" /><figcaption>An example with a topic that has 5 partitions with two separate consumer groups subscribed to it.</figcaption></figure><p>In the example above, we have a payment topic with 4 partitions. We have an analytics group and a fraud detection group. The analytics group has 2 consumers. Each takes 2 partitions each. The fraud group has 5 consumers. The first 4 consumers are taking a partition each. The 5th consumer in this group is idle and will become active if a consumer within its group fails.</p><p>To use consumer groups, you can use the consumer property group.id.</p><h3>Consumer Offsets</h3><p>How do we know where a consumer has successfully read into a partitioned topic?</p><p>When a consumer from a consumer group reads data from a topic, it will commit the offset from which it last read into an internal Kafka topic called __consumer-offsets .</p><p>This is to keep track of where the consumer should pick up from the next time it reads data from Kafka.</p><p>Java consumers automatically commit offsets to Kafka, but this can also be manually triggered.</p><ol><li>At least Once (preferred) — This is the automatic method too. With this mode, an offset is committed after a message has been successfully processed. If an error occurs during the processing, and you need to restart, you can reply to it exactly where you last were.</li><li>At Most Once — Commits offsets as soon as the message is read. If processing goes wrong, there is a chance you won’t reply to this last event.</li></ol><h3>Kafka Brokers and Clusters</h3><p>A Kafka cluster consists of multiple brokers (servers) working together to store and manage data. Each broker is responsible for storing specific topic partitions, ensuring data is distributed across the cluster. When a client connects to any broker in the cluster, they gain access to all brokers through that connection. This is known as the bootstrap connection.</p><h3>Broker and Topics</h3><p>So how are Topics and Partitions stored in a Kafka Cluster?</p><p>Say you have a Topic called user_events_topic with 3 partitions, and another topic orders_topics with 2 partitions. Say you have 3 brokers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E-6kCp77S1vmOElh" /><figcaption>Partition distribution across multiple brokers in a Kafka cluster, showing how two topics (user_events and orders) are split across three broker nodes.</figcaption></figure><p>A Broker is essentially a server that can store data from multiple partitions from different topics. Looking at our example now:</p><ul><li>user_events_topic has 3 partitions distributed across the brokers</li><li>orders_topic has 2 partitions distributed across the brokers</li></ul><p>Each broker acts as a storage unit, capable of holding partitions from any topic.</p><p>This broker-based architecture enables three key benefits:</p><ol><li>Load balancing: Data storage is distributed across multiple servers</li><li>Parallel processing: Multiple brokers can handle requests simultaneously</li><li>Horizontal scaling: You can add more brokers as your data grows</li></ol><h3>Topic Replication</h3><p>One great aspect of Kafka is that its fault tolerant. So if a Broker goes down, the idea is that you don’t lose access to your data. But how?</p><p>Kafka achieves fault tolerance through partition replication. Each partition is replicated across multiple brokers in your cluster to prevent data loss during broker failures.</p><p>Key points about replication:</p><ul><li>One broker serves as the leader for each partition</li><li>Other brokers maintain synchronised copies (replicas) of that partition</li><li>Producers write to partition leaders, with additional ability to write to In-Sync Replicas (ISR)</li><li>Consumers traditionally read only from leaders, but since Kafka 2.4+ can also read from ISRs</li></ul><p>The replication-factor setting controls how many copies of each partition exist:</p><ul><li>Production environments: Should be &gt; 1 (typically 3)</li><li>Local development: Can be 1 for simplicity</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2Fi6Suuz9pQkxZWO" /><figcaption>Kafka cluster with replication factor=3. Each partition has one leader (orange) and two replicas (grey) distributed across brokers.</figcaption></figure><p>We can see from this example, if Broker 1 goes down, we don’t lose access to any data because:</p><ul><li>partition-0’s data is still available from its replicas on Broker 2 and 3</li><li>partition-1’s leader is on Broker 2, so it’s unaffected</li></ul><p>In this case a new leader for partition-0 will be elected automatically. We will get into this more in the Zookeeper section.</p><h3>Producer Acknowledgements (acks) and Topic Durability</h3><p>Producer acknowledgments (acks) solve data durability and consistency challenges in distributed systems. Here’s why they exist:</p><p>Without acks, you wouldn’t know if:</p><ul><li>Messages were actually received by Kafka</li><li>Data was properly replicated before a broker failure</li><li>Messages were lost in transit due to network issues</li></ul><p>Kafka producers can configure acknowledgment levels (acks):</p><ul><li>acks=0: Producers do not wait for write confirmations</li><li>acks=1: Producers wait for write confirmation from only the leader of a partition.</li><li>acks=all: Producers wait for write confirmation from leaders + replicas.</li></ul><h3>Zookeeper and KRaft</h3><p>Zookeeper is a service that manages Kafka’s cluster state, monitoring broker health and handling leader elections when brokers fail. It’s been integral to Kafka’s distributed coordination since the beginning, serving as the central coordinator for the cluster.</p><p>However, Kafka is evolving beyond this dependency. Kafka 3.x introduced Kafka Raft (KRaft) as an alternative to Zookeeper, and Kafka 4.0 will eliminate Zookeeper dependency entirely. This change streamlines Kafka’s architecture by handling cluster management internally rather than relying on an external service.</p><h3>End!</h3><p>And that’s the end of this article! I hope it helped you better understand Kafka and its capabilities.</p><p>If you have any questions, suggestions or want to share how you’ve used Kafka in your projects, feel free to drop a comment.</p><p>Find me on <a href="https://www.linkedin.com/in/adil-s64/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be4b6f6d1591" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/apache-kafka-explained-be4b6f6d1591">Apache Kafka Explained</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQL Practice Made Easy: Using SQLite and Python in Jupyter Notebooks]]></title>
            <link>https://medium.com/@adilsaid64/sql-practice-made-easy-using-sqlite-and-python-in-jupyter-notebooks-13512b725dd3?source=rss-bc57e1ee366e------2</link>
            <guid isPermaLink="false">https://medium.com/p/13512b725dd3</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Adil Said]]></dc:creator>
            <pubDate>Mon, 09 Sep 2024 12:25:19 GMT</pubDate>
            <atom:updated>2024-09-09T12:25:19.409Z</atom:updated>
            <content:encoded><![CDATA[<p>So, you want to practice SQL, but you don’t want to deal with setting up a Postgres or MySQL server on your computer?</p><p>I totally get it, configuring a database server can feel like overkill when you just want to start running some queries.</p><p>That’s where SQLite comes in.</p><p>With Python and a Jupyter Notebook, you can jump right into practicing SQL without any complicated setup.</p><p>In this guide, I’ll show you how to use SQLite with Python in a Jupyter Notebook to practice SQL the easy way.</p><p>Whether you’re a beginner or just want a quick and simple way to work on your skills, this guide has you covered. Let’s dive in!</p><h3>Step 1: Set Up SQLite and Load the Chinook Database</h3><p>Let&#39;s start by creating a new notebook. Then we import our requirements and download the database. We will be using the Chinook Database.</p><pre>import sqlite3<br>import pandas as pd<br>import urllib.request<br><br># Download the Chinook SQLite database<br>url = &#39;https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite&#39;<br>urllib.request.urlretrieve(url, &#39;Chinook_Sqlite.sqlite&#39;)<br><br># Connect to the Chinook SQLite database<br>conn = sqlite3.connect(&#39;Chinook_Sqlite.sqlite&#39;)</pre><p>You should see a .sqlite file in your directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*lujNwXyNN0kXJ-sYdnVIGg.png" /><figcaption>Output</figcaption></figure><h3>Step 2: View the Tables in the Chinook Database</h3><p>This is how we view the tables in our database.</p><pre># Query to view all tables<br>tables_query = &quot;SELECT name FROM sqlite_master WHERE type=&#39;table&#39;;&quot;<br>tables = pd.read_sql(tables_query, conn)<br>tables</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*Op5fUzeghxHAzNjmDLv1LA.png" /></figure><h3>Step 3: Query Data from a Table</h3><p>Now that we see the avaliable tables, lets dive into querying the data. We can start by taking a look at the first 10 rows of the Customer table.</p><pre># Query the customers table<br>query = &quot;SELECT * FROM Customer LIMIT 10;&quot;  # Limiting to 10 rows for demonstration<br>customers = pd.read_sql(query, conn)<br>customers</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OYsscSOXcMVYKHCb6r3qyg.png" /></figure><p>And thats it! Feel free to do some experimentation and playing around.</p><h3>Step 4: Try some Joins</h3><p>Let’s try to join Track and Album tables to get track names and album titles.</p><pre># Join the Track and Album tables to get track names and album titles<br>query = &quot;&quot;&quot;<br>SELECT Track.Name AS TrackName, Album.Title AS AlbumTitle<br>FROM Track<br>JOIN Album ON Track.AlbumId = Album.AlbumId<br>LIMIT 10;<br>&quot;&quot;&quot;<br>track_album = pd.read_sql(query, conn)<br>track_album</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*6URnBiuAz2URlHORATqdYQ.png" /></figure><h3>That’s a wrap!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MIFM8Nr2bu5vBW0L" /><figcaption>Photo by <a href="https://unsplash.com/@japhethmast?utm_source=medium&amp;utm_medium=referral">Japheth Mast</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Now you’ve got the basics of using SQLite and Python to practice SQL without any of the setup headaches.</p><p>If you’re ready to dive deeper into SQL joins, be sure to check out my other article on <a href="https://medium.com/@adilsaid64/understanding-sql-joins-a-quick-guide-with-examples-b0985419ea85">Understanding SQL Joins</a> for a more detailed look.</p><p>Hope this was useful!</p><p>Follow for more Data Science/Engineering content!</p><p>Find me on:</p><p><a href="https://www.linkedin.com/in/adil-s64">LinkedIn</a>, <a href="https://github.com/adilsaid64">GitHub</a>, <a href="https://adilsaid64.github.io/">Portfolio</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13512b725dd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding SQL Joins. A Quick Guide with Examples]]></title>
            <link>https://medium.com/@adilsaid64/understanding-sql-joins-a-quick-guide-with-examples-b0985419ea85?source=rss-bc57e1ee366e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0985419ea85</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[sql-joins]]></category>
            <category><![CDATA[relational-databases]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Adil Said]]></dc:creator>
            <pubDate>Thu, 18 Jul 2024 21:02:36 GMT</pubDate>
            <atom:updated>2024-07-18T21:02:36.232Z</atom:updated>
            <content:encoded><![CDATA[<p>SQL joins allow us to combine information from multiple tables based on a related column. In this article, we go through:</p><ul><li>Inner Joins</li><li>Outer Joins (Left, Right and Full)</li><li>Cross Joins</li><li>Self Joins</li></ul><p>If you are new to SQL joins, these examples are really easy to go through and with some patience and persistence you’ll understand them too.</p><h3>Inner Join</h3><p>This is the simplest join.</p><p>Inner joins return the records that have matching values in both tables.</p><p>This is a symmetrical joint. This means you get the same result if you join table A with table B or table B with table A.</p><p>Examples:</p><pre><strong>Employees Table<br></strong>| EmployeeID | EmployeeName |<br>|------------|--------------|<br>| 1          | John Doe     |<br>| 2          | Jane Smith   |<br>| 3          | Emily Jones  | <br><br><strong>Departments Table<br></strong>| EmployeeID | Department   |<br>|------------|--------------|<br>| 1          | Marketing    |<br>| 2          | Sales        |<br>| 4          | HR           |</pre><p>An inner join query would look like:</p><pre>SELECT Employees.EmployeeName, Departments.Department<br>FROM Employees<br>INNER JOIN Departments<br>ON Employees.EmployeeID = Departments.EmployeeID;</pre><p>And the result:</p><pre>| EmployeeName | Department |<br>|--------------|------------|<br>| John Doe     | Marketing  |<br>| Jane Smith   | Sales      |</pre><h3>Outer Joins</h3><p>These can be broken down into 3.</p><ol><li>Left Outer Join</li><li>Right Outer Join</li><li>Full Outer Join</li></ol><pre><strong>Employees</strong> <strong>Table</strong><br>| EmployeeID | EmployeeName |<br>|------------|--------------|<br>| 1          | John Doe     |<br>| 2          | Jane Smith   |<br>| 3          | Emily Jones  |<br><br><br><strong>Departments Table<br></strong>| EmployeeID | Department   |<br>|------------|--------------|<br>| 1          | Marketing    |<br>| 2          | Sales        |<br>| 4          | HR           |</pre><h3>Left Outer Join</h3><p>A left outer join returns <strong>all rows from the left table</strong> and the <strong>matched rows from the right table.</strong></p><p>If there is no match it returns a <strong>NULL in the right table.</strong></p><pre>SELECT Employees.EmployeeName, Departments.Department<br>FROM Employees<br>LEFT JOIN Departments<br>ON Employees.EmployeeID = Departments.EmployeeID;</pre><p>The result.</p><pre>| EmployeeName | Department |<br>|--------------|------------|<br>| John Doe     | Marketing  |<br>| Jane Smith   | Sales      |<br>| Emily Jones  | NULL       |</pre><h3>Right Outer Join</h3><p>Similar to a left join. But the other way.</p><p>A right outer join <strong>returns all rows from the right table</strong> and the <strong>matched rows from the left table.</strong></p><p>If there is no match it returns a <strong>NULL in the left table.</strong></p><pre>SELECT Employees.EmployeeName, Departments.Department<br>FROM Employees<br>RIGHT JOIN Departments<br>ON Employees.EmployeeID = Departments.EmployeeID;</pre><p>The result.</p><pre>| EmployeeName | Department |<br>|--------------|------------|<br>| John Doe     | Marketing  |<br>| Jane Smith   | Sales      |<br>| NULL         | HR         |</pre><h3>A Full Outer Join</h3><p>Returns <strong>all records when there is a match in either the left or right table.</strong></p><p>If there is no match,<strong> there result is NULL on the side that does not have a match.</strong></p><pre>SELECT Employees.EmployeeName, Departments.Department<br>FROM Employees<br>FULL OUTER JOIN Departments<br>ON Employees.EmployeeID = Departments.EmployeeID;</pre><p>The result.</p><pre>| EmployeeName | Department |<br>|--------------|------------|<br>| John Doe     | Marketing  |<br>| Jane Smith   | Sales      |<br>| Emily Jones  | NULL       |<br>| NULL         | HR         |</pre><h3>Cross Join</h3><p>A cross-join returns the <strong>cartesian product</strong> of two tables.</p><p>Meaning it combines each row from the first table with each row from the second table.</p><p>So this generates <strong>all possible combinations of records</strong> in the two tables.</p><p>Example:</p><pre><strong>Colours Table<br></strong>| Colour  |<br>|---------|<br>| Red     |<br>| Blue    |<br>| Green   |<br><br><br><strong>Size Table<br></strong>| Size    |<br>|---------|<br>| Small   |<br>| Medium  |<br>| Large   |</pre><pre>SELECT Colours.Colour, Sizes.Size<br>FROM Colors<br>CROSS JOIN Sizes;</pre><p>The result</p><pre>| Colour | Size   |<br>|--------|--------|<br>| Red    | Small  |<br>| Red    | Medium |<br>| Red    | Large  |<br>| Blue   | Small  |<br>| Blue   | Medium |<br>| Blue   | Large  |<br>| Green  | Small  |<br>| Green  | Medium |<br>| Green  | Large  |</pre><h3>Self Join</h3><p>This is used to join a table to itself.</p><p><strong>As if the table was two separate tables.</strong></p><p>A typical use case is when the table has a <strong>hierarchical structure.</strong></p><p>For example, an Employee Table that contains all the employees within a company. Each row represents an individual employee with attributes like EmployeeId , Name ManagerId . The ManagerId the column points back to the EmployeeId .</p><pre><strong>Employees Table<br></strong>| EmployeeID | EmployeeName | ManagerID |<br>|------------|--------------|-----------|<br>| 1          | John Doe     | NULL      |<br>| 2          | Jane Smith   | 1         |<br>| 3          | Emily Jones  | 1         |<br>| 4          | Eric Johnson | 2         |</pre><p>So in this table, the manager of Jane Smith is John Doe.</p><pre>SELECT E1.EmployeeName AS Employee, E2.EmployeeName AS Manager<br>FROM Employees E1<br>LEFT JOIN Employees E2 ON E1.ManagerID = E2.EmployeeID;</pre><p>The result.</p><pre>| Employee     | Manager    |<br>|--------------|------------|<br>| John Doe     | NULL       |<br>| Jane Smith   | John Doe   |<br>| Emily Jones  | John Doe   |<br>| Eric Johnson | Jane Smith |</pre><h3>Summary</h3><p>Before you choose a join, think about what your desired outcome is. Then look at how your tables are related. Here is a quick summary of the main SQL joins we discussed above.</p><ul><li><strong>Inner Join:</strong> When you need rows that have matching values in both tables. This is the most performance-efficient join when you only care about matching rows.</li><li><strong>Left Outer Join:</strong> When you want all records from the left table to show up even if there is no corresponding entry in the right table.</li><li><strong>Right Outer Join:</strong> Similar to the left. But when you care about returning all the entries in the right table even if they have no matching entries in the left.</li><li><strong>Full Outer Join:</strong> When you need all records from both tables.</li></ul><p>Full Outer Joins and Cross Joins can be resource intensive, especially on large amounts of data.</p><p>This is a useful illustration and cheatsheet to give an intuition on the different join types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/0*Nrjozs1i0XIiACKd.png" /><figcaption>Source: <a href="https://dsin.wordpress.com/2013/03/16/sql-join-cheat-sheet/">https://dsin.wordpress.com/2013/03/16/sql-join-cheat-sheet/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qUMVTHgbP627CHOo" /><figcaption>Photo by <a href="https://unsplash.com/@peet818?utm_source=medium&amp;utm_medium=referral">Pete Pedroza</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Thank you for reading. Hopefully, this article was beneficial in breaking down the different types of SQL joins.</p><p>Follow for more Data Science/Engineering content!</p><p>Find me on:</p><p><a href="https://www.linkedin.com/in/adil-s64">LinkedIn</a>, <a href="https://github.com/adilsaid64">GitHub</a>, <a href="https://adilsaid64.github.io/">Portfolio</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0985419ea85" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Slowly Changing Dimensions: Types 0 to 3 and How to Choose the Right Strategy]]></title>
            <link>https://medium.com/@adilsaid64/understanding-slowly-changing-dimensions-types-0-to-3-and-how-to-choose-the-right-strategy-acd7c4a61ed3?source=rss-bc57e1ee366e------2</link>
            <guid isPermaLink="false">https://medium.com/p/acd7c4a61ed3</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[slowly-changing-dimension]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Adil Said]]></dc:creator>
            <pubDate>Mon, 08 Jul 2024 21:13:56 GMT</pubDate>
            <atom:updated>2024-07-08T21:17:35.027Z</atom:updated>
            <content:encoded><![CDATA[<h3>What are slowly changing dimensions (SCD)?</h3><p>SCD is quite a popular term in Data Engineering, and today we gonna discuss what they are how we deal with them.</p><p>Most of the time in a data warehouse we expect facts to change but not so much the dimensions. But in the real world, dimensions may also change.</p><p>For example, imagine a company has a dimension called <em>customers</em>, and there is an attribute called <em>favourite colour</em>.</p><p>Maybe initially someone&#39;s favourite colour on signup is Red but later on that changes to Blue.</p><p>We need to decide how to handle this change. This is where slowly changing dimensions comes into play.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dGl84PdEK-2-Lxvo" /><figcaption>Photo by <a href="https://unsplash.com/@chrislawton?utm_source=medium&amp;utm_medium=referral">Chris Lawton</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>How do you go about SCD?</h3><p>Often times, you wont be told about SCD when designing your data warehouse.</p><p>Business owners and analysts might not be expecting changes to dimensions.</p><p><strong>So</strong> <strong>be</strong> <strong>proactive</strong> <strong>and</strong> <strong>initiate conversations</strong> with the analyst and business owners/managers to develop strategies on how deal with dimensions that might change.</p><p>Its important to remember that this process is a business decision and not a technical one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R3rO7VjXrLpO9qmd" /><figcaption>Photo by <a href="https://unsplash.com/@alterego_swiss?utm_source=medium&amp;utm_medium=referral">AbsolutVision</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Here are strategies introduced by Kimball in 1995.</p><h4>Type 0 — Retain Original</h4><p>This strategy is applicable when you are sure there wont be any changes to a dimension table. This is most common with date dimension tables which dont really change.</p><h4>Type 1 — Overwrite</h4><p>In this strategy we overwrite the old attribute with the new.</p><p>It’s a very simple approach and no fact table needs to be modified.</p><p><strong>But we dont get a historical record of the changes.</strong> And this may be problematic for significant changes.</p><p>This is okay for small changes. For example, a product name changing from ‘<em>Choco Shake</em>’ to ‘<em>Best Choco Shake</em>’. If the analysts and end users agree that this change is okay.</p><p>This might look like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*auGZIajzXBSXgXMcyqV8kw.png" /><figcaption>Replacing the older value with the newer value. Row 2.</figcaption></figure><p>Type 1 might affecting existing queries as the values are different. If you query contained <em>Choco Shake</em> it would need to be changed.</p><h4>Type 2 — Additional Row (This should be your default if you want to maintain history)</h4><p>Type 1’s main problem is lack of history.</p><p>Type 2 strategy aims creates a new row in the data. This creates a new primary key (PK) in the dimension.</p><p>Then start using the new FK in the fact table.</p><p><strong>This should be your default strategy when expecting changes in a dimensions.</strong></p><p>An example might be a customers favourite foods. The end users of our data warehouse might be interested in tracking this change. Here is an example of how that might look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aSuEMmQFi7I2vXVPJT-wvw.png" /><figcaption>Adding a new row. There is a new PK for the new row. In the fact table, this is what you refer to from now. But we keep the employee id the same. Also notice the adddition of 3 new columns. Which are explained bellow.</figcaption></figure><p>Some problems and how you can overcome them.</p><p>You may have a problem of counting unique values in your dimension table. How do you resolve? You can create a <em>id</em> column. Then the new row shares the same <em>id</em> as the older one. And you can refer to this instead. This may also be already given as the <em>natural id</em>. We can see the example above has an <em>Employee Id</em>. So the analyst can count for unique employee ids in this case.</p><p>Another problem is how do you query for the current product? Because the same product now has two rows (An older and newer). You can include <em>Effective Date</em> and <em>Expiration Date</em> columns. And also create a third flag (boolean) column indicating if the current value is the active one to make filtering easier.</p><p>When does it not making sense to use Type 2?</p><p>Well, just depends on context. Does the business care about tracking this historical change? Does the analyst or end users want this change tracked?These are questions you should ask your end users.</p><h4>Type 3 — Additional Attribute</h4><p>Instead of a new row like type 2 we add an additional column.</p><p>The benefits of this method is that we can aggregate data using the previous and new state.</p><p>An example, after some restructuring, existing items are in a new categories. <em>Choco</em> <em>Shakes</em> have been moved to a new category called <em>Shakes</em>.</p><p>This might look like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OiRXum4YCzwUrbVdCV2zxQ.png" /><figcaption>We create a new column called Prev Category.</figcaption></figure><p>This method allows the analysts to group by the older structure and newer structure.</p><p>This is common for significant changes in your business like restructuring your data warehouse. This allows analysts to switch back between the new and older state. This allows them to compare figures between the older and new structure.</p><p>This strategy is not suitable for frequent changes. The number of columns will be too much. In this case opt for type 2. For minor changes, opt for type 1.</p><h3>Conclusion</h3><ol><li>Be proactive and initiate conversation with analysts and business owners when creating a data warehouse. They might not expect changes, so discuss whats important for them. This is not a technical decision but a business one. So this step is crucial.</li><li>Type 0 — Do nothing. Most common in date tables.</li><li>Type 1— Overwrite the original data. For minor changes that analysts dont care about.</li><li>Type 2 — Create a new row in the dimension. With a new primary key. Refer to this new key in your fact table for newer data. Useful for things you expect to change a lot.</li><li>Type 3 — Create new column. Indicating the pervious and current value. Great for non frequent changes and larger restructuring tasks.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JI4A5KLa_3BFoFtz" /><figcaption>Photo by <a href="https://unsplash.com/@wilhelmgunkel?utm_source=medium&amp;utm_medium=referral">Wilhelm Gunkel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Thanks for reading. Hope this article was useful in breaking down slowly changing dimensions. Would love to know your thoughts on this.</p><p>For more content on Data Engineering and Data Science check out my other articles.</p><p>Find me on <a href="https://www.linkedin.com/in/adil-s64">LinkedIn</a>, <a href="https://github.com/adilsaid64">GitHub</a> and checkout my <a href="https://adilsaid64.github.io/">portfolio</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acd7c4a61ed3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create Spatial Temporal Choropleth Maps in Python]]></title>
            <link>https://medium.com/@adilsaid64/create-spatial-temporal-choropleth-maps-in-python-a347a94ac71d?source=rss-bc57e1ee366e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a347a94ac71d</guid>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Adil Said]]></dc:creator>
            <pubDate>Sun, 21 Apr 2024 19:10:05 GMT</pubDate>
            <atom:updated>2024-04-21T19:11:33.326Z</atom:updated>
            <content:encoded><![CDATA[<p>This article aims to demonstrate how to create spatial-temporal choropleth maps in Python.</p><p>Such maps are useful in conducting spatial-temporal exploratory analysis of some phenomena. For example, I’ve used them in my final year dissertation to analyse violent crime in London. I’ve also used them in this article <a href="https://medium.com/@adilsaid64/a-decade-of-crime-in-london-insights-from-spatial-temporal-data-9781369dccb6">here</a> (which is not my dissertation but an exploratory analysis of violent crime in London).</p><p>In this article, we will go through how to use Python to map suicide counts in London.</p><p>By the end of this article, you will know how to create plots like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3XB1X605nM213hY1QbnfoA.png" /><figcaption>Suicide Counts in London from 2011 to 2022</figcaption></figure><p>Let&#39;s get started.</p><h3>Requirements</h3><ol><li>Data. I will be using Suicide Data from Ons.gov.uk. (2019).</li><li>A shapefile of your location of interest. I will use one from the London Datastore.</li></ol><h3>Steps</h3><ol><li>Load the raw data into Pandas. Load the shapefile with GeoPandas.</li><li>Combine the shapefile and the raw data into one dataset.</li><li>Create cutpoints.</li><li>Create plots.</li></ol><h4>1. Loading The Data</h4><p>The data should be in this structure.</p><ol><li>The rows represent each geographic location.</li><li>The columns represent the time.</li><li>The values should represent the occurrences or ratio or some statistic about the phenomena you are exploring at that location and time.</li></ol><p>Here is an example:</p><pre>import pandas as pd<br>data = pd.read_csv(&#39;data/london-suicide.csv&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UUN2ZCL2A0penPQ3H247JA.png" /><figcaption>The raw suicide incident (count) data. Each column represents a year. Each row represents a location.</figcaption></figure><p>It&#39;s unlikely to find data that comes ready in this nicely structured format. You will likely have to do some preprocessing beforehand.</p><p>Next, we load the shapefile.</p><pre>import geopandas as gpd<br><br>shapefile_borough = gpd.read_file(&quot;statistical-gis-boundaries-london/ESRI/London_Borough_Excluding_MHW.shp&quot;)<br>shapefile_borough.plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*ptGW8iuL487Hp-mTv1tdwQ.png" /><figcaption>London Borough shapefile.</figcaption></figure><h4>2. Combine The Data and Shapefile.</h4><p>We need to align the shapes of each geographic location with the corresponding data.</p><p>We will set the index of each row to the borough name. Then align the shapefile data and the raw data by borough name.</p><pre>shapefile_borough.index = shapefile_borough[&#39;NAME&#39;]<br>data.index = data[&#39;name&#39;]<br><br>df = shapefile_borough.join(data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cAc3lKB84KmZ79aou1drw.png" /><figcaption>The combined data and aligned data.</figcaption></figure><p>We can plot one of the columns as follows:</p><pre>df.plot(column = &#39;2022&#39;, figsize=(20, 7), legend = True, cmap=&#39;Blues&#39;)<br>plt.title(&#39;London Suicide Counts in 2022&#39;)<br>plt.axis(&#39;off&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*4lMGybMvcyZVQUAwm3-1sQ.png" /><figcaption>Suicide Counts in 2022.</figcaption></figure><h4>3. Creating Consistent Cutpoints.</h4><p>We need consistent cutpoints because we will want to compare our data across time. In our case, we want to compare suicide counts across different years.</p><p>It will be very difficult to fairly compare the data if they are plotted at different cutpoints.</p><p>Here is how we do that:</p><pre>stacked_data = data.drop(columns=[&#39;code&#39;, &#39;name&#39;]).stack()<br>quantiles = stacked_data.quantile([0.25, 0.5, .75, 1])</pre><p>We dropped the non-numerical columns and stacked the data. This created one row containing all the values. Then we take the quantiles of those values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*NRgOElIqvOvU47y2zgbrNQ.png" /><figcaption>The stacked data. Indexed first by location (the original index) and the years (the column names). The values are last.</figcaption></figure><p>And the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*TaE7hCjQgdLBgKk9hjwilw.png" /><figcaption>The quantiles.</figcaption></figure><h4>4. Creating Spatial-Temporal Choropleth Maps.</h4><p>We will create a plot for each year on a grid.</p><pre>fig, axes = plt.subplots(nrows=2, ncols=6, figsize=(30, 10)) <br>axes = axes.flatten() <br><br>for i, year in enumerate(range(2011, 2023)):<br>    ax = axes[i]<br>    df.plot(column=str(year), <br>            cmap=&#39;Blues&#39;, <br>            edgecolor = &#39;0.8&#39;,<br>            legend=True, <br>            scheme=&#39;user_defined&#39;, <br>            classification_kwds={&#39;bins&#39;: quantiles.round(2).to_list()},<br>            ax=ax)<br>    ax.set_title(f&#39;London Suicide Counts in {year}&#39;)<br>    ax.axis(&#39;off&#39;)<br><br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3XB1X605nM213hY1QbnfoA.png" /><figcaption>The final plot! Each map corresponds to a certain year. The cutpoints are the quantiles (0–0.25, 0.25–0.5, 0.5–0.75, 0.75–1)</figcaption></figure><p>Thats it! You can view the complete code on my GitHub <a href="https://github.com/adilsaid64/uk-suicide-analysis">here</a>.</p><p>Thank you for reading. Find me on <a href="https://www.linkedin.com/in/adil-s64/">LinkedIn</a> and <a href="https://github.com/adilsaid64">GitHub</a>.</p><h3>References</h3><ol><li>Said, A. (2024). <em>adilsaid64/uk-suicide-analysis</em>. [online] GitHub. Available at: <a href="https://github.com/adilsaid64/uk-suicide-analysis">https://github.com/adilsaid64/uk-suicide-analysis</a></li><li>Ons.gov.uk. (2019). <em>Suicides in England and Wales by local authority — Office for National Statistics</em>. [online] Available at: <a href="https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/datasets/suicidesbylocalauthority.">https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/datasets/suicidesbylocalauthority.</a>‌</li><li>London Datastore. (n.d.). <em>Statistical GIS Boundary Files for London — London Datastore</em>. [online] Available at: <a href="https://data.london.gov.uk/dataset/statistical-gis-boundary-files-london.">https://data.london.gov.uk/dataset/statistical-gis-boundary-files-london.</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a347a94ac71d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Premier League match wins using Bayesian Modelling]]></title>
            <link>https://pub.towardsai.net/predicting-premier-league-match-wins-using-bayesian-modelling-32eec733472e?source=rss-bc57e1ee366e------2</link>
            <guid isPermaLink="false">https://medium.com/p/32eec733472e</guid>
            <category><![CDATA[math]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[football]]></category>
            <dc:creator><![CDATA[Adil Said]]></dc:creator>
            <pubDate>Mon, 01 Apr 2024 22:55:59 GMT</pubDate>
            <atom:updated>2025-08-29T11:42:49.200Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_pT3b5fYMgZz0fVH" /><figcaption>Source: Photo by <a href="https://unsplash.com/@reganography?utm_source=medium&amp;utm_medium=referral">Samuel Regan-Asante</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Football is the greatest and most-watched sport in the world, and according to <a href="https://publications.fifa.com/en/vision-report-2021/the-football-landscape/">Fifa</a>, it has around 5 billion fans worldwide.</p><p>The Premier League is arguably the most difficult and exciting. That is because it thrives on its unpredictability. It&#39;s the league where underdogs can triumph and giants can fall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*mNseT7sEpYIWotIVkGMccA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*YbP5VhNDLgwjM-zOe1D_bg.png" /><figcaption>Source: Image from google — Who would have predicted these results?</figcaption></figure><p>And that&#39;s the reason why we love to watch it. Anything can happen from the starting minute till the last whistle. And as a Man Utd fan, I know that well.</p><p>Having said that, I was looking for a weekend project to do that was difficult and where I could combine two things I like.</p><p>Football and Bayesian Modelling.</p><p>In this article, I want to take you through my attempt and thought process for this problem. Predicting Premier League match wins using Bayesian Modelling.</p><p>There are similar articles that predict football match wins. Many of them like this article on <a href="https://www.youtube.com/watch?v=0irmDBWLrco&amp;ab_channel=Dataquest">Dataquest</a> use traditional machine learning models like Random Forrest algorithms. The closest article I could find was by <a href="https://towardsdatascience.com/using-bayesian-modeling-to-predict-the-champions-league-8ebb069006ba">Pol Marin</a>, who used Bayesian Modelling to predict Champions League wins.</p><p>From what I could find a good accuracy metric for this type of project is around 50–60%.</p><h3>The Data</h3><p>The data came from <a href="http://football-data.co.uk/">football-data.co.uk</a>. Data from the current 2023–2024 season was used.</p><p>Each row represents a Football match played. With columns representing the details of the match like <em>HomeTeam</em>, <em>AwayTeam</em>, <em>FTHG</em> (Full-time home goals), <em>FTAG</em> (Full-time away goals), <em>FTR</em> (Full-time results) and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4l-kX3CNnwCFKMQgv9pUZw.png" /><figcaption>Source: Image by the author — The first few rows and columns that were used for this project.</figcaption></figure><p>The data also contains details about match bet odds. These were discarded from this project because the aim is to let the model learn from team performance rather than information from betting odds.</p><h3>Pre-processing and Feature Engineering</h3><p>The first task was to get the data into a format needed for a Machine Learning model. My preferred tool for data processing is Python.</p><p>Essentially, the go-to questions that I ask myself on any Machine Learning project are (when it comes to the data):</p><ol><li>What does each row represent?</li><li>What does the target column represent?</li><li>How do the features describe each row and how are they related to the target?</li></ol><p>For this project, we needed to know what team was playing, who was their opponent and a binary target column indicating if the team playing won.</p><p>This means, for each match, we needed two rows. One row from the perspective of the home team and another from the away team.</p><p>So for example, a match like Manchester United vs Arsenal will have two rows. In one row, the team playing will be Manchester United and their opponent Arsneal. In the next row, the team playing will be Arsenal, their opponent Manchester United.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7uu75l9FV_TRoiVWrRuaeQ.png" /><figcaption>Source: Image by the author</figcaption></figure><p>The target column will be a binary column that represents if the team playing won (1) or didn&#39;t win (0).</p><p>In the current form, the dataset describes the statistics of each match. The dataset tells us, how many goals the <em>team_playing</em> has scored, shots they took, number of shots and goals conceded and more.</p><p>When making future predictions we won&#39;t have these statistics. Because these games won&#39;t have been played yet.</p><p>So the next task was to feature engineer statistics we could get before the game was played.</p><p>So I grouped the dataset by the <em>team_playing</em> column and then calculated the rolling statistics of the past 5 games (excluding the current row). These included <em>mean</em>, <em>std</em>, <em>min</em>, and <em>max</em>. We had a total of 90 features.</p><p>This resulted in the first five rows for each team being missing, so I filled them using the next available value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yb7UK7_RNg-rEhFLEYf6Ew.png" /><figcaption>Source: Image by the author — The ‘<strong>opponent_ft_goals_rolling_mean</strong>’ column shows the average goals conceded by the team, based on a rolling average from the last 5 games, while the ‘<strong>team_playing_ft_goals_rolling_mean</strong>’ column displays the team’s average goals scored, also calculated as a rolling average over the last 5 games.</figcaption></figure><p>So now, each row tells us, the team playing, their opponent, the mean number of goals scored by the team playing, goals conceded by the team playing, shots taken and more from the past 5 games.</p><p>Choosing to use the last 5 games for the rolling averages was chosen semi-arbitrarily and more experimentation could be done to find an optimum value.</p><h3>Feature Selection</h3><p>Although 90 features were extracted. Only 6 features were used for this Bayesian logistic regression. This was because there was a lot of multicollinearity. See the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H8PsiBHjiAOJsrR0_4-dJA.png" /><figcaption>Source: Image by the author — Feature Correlation Matrix. Only looking at the rolling mean features.</figcaption></figure><p>As you would expect, a team that had been winning recently would have its <em>target_rolling_mean</em> highly correlated with <em>team_playing_ft_goals_rolling mean</em> etc.</p><p>The final features set I went with was:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-bRtuymg_dM_ICEfglCDeg.png" /><figcaption>Source: Image by the author — Final features to go into the model.</figcaption></figure><p>The names of the team playing would not be used as features for this project. This is because they would require encoding.</p><p>Either done through label encoding, where each team gets given a unique number. Since we are using a linear model, it will assume a linear relationship between the target and features. It would incorrectly interpret numerical values as having an ordinal relationship. In the context of team names, it wouldn&#39;t make sense.</p><p>It would have been okay with a non-linear model like a tree-based model since they can capture non-linear relationships.</p><p>Another way would be through one hot encoding but this would have increased the dimensionality of our dataset more. For these reasons, I decided to drop them.</p><h3>The Model: Bayesian Logistic Regression</h3><p>We are trying to predict match wins. Either a team won or didn&#39;t win. Thus our target is binary. We define <em>y </em>to be our target and we assume it follows a Bernoulli distribution with parameter <em>p.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*avpdWWOifGS9LRUd8G-tzw.jpeg" /><figcaption>Source: Image by the author — Model structure</figcaption></figure><p>To model the probability <em>p</em> of a team winning a match, we employ a logistic regression model. In logistic regression, the log odds of the event of interest (in this case, a team winning) are modelled as a linear combination of predictor variables. Specifically, we use the <em>logit</em> function to transform the probability <em>p</em> into the log-odds scale.</p><p>Here, each β​ is the coefficient associated with each predictor variable <em>X</em>​. These coefficients represent the change in the log odds of winning for a one-unit change in the corresponding predictor variable, holding other variables constant.</p><p>To account for uncertainty in the estimation of these coefficients, we introduce weakly informative prior distributions for the coefficients <em>β</em>. In our model, we assume a normal prior distribution for each coefficient, centred at 0 with a variance of 1000.</p><p>I employed MCMC using the <em>rjags</em> package in R to estimate the parameter coefficients and ran 2 chains with 100000 samples. A burn-in period of 10000 and thinning set to 20 to reduce autocorrelation. The <em>rjags </em>model string is below.</p><pre># JAGS model<br>model_string &lt;- &#39;model{<br>  for(i in 1:N){<br>    y[i] ~ dbern(p[i])<br>    logit(p[i]) &lt;- eta[i]<br>    # linear predictors using inner product notation.<br>    eta[i] &lt;- inprod(X[i,], beta[]) <br>  }<br>  <br>  # Weakly informative priors for coefficients<br>  for(j in 1:P){<br>    beta[j] ~ dnorm(0, 0.001)<br>  }<br>}&#39;</pre><p>For training, all data up to and including matchday 28 of the current 23–24 season was used. Matchday 29 will be our test set. This is what we will use to assess our models.</p><p>We are training the model on 516 matches and then testing on 50.</p><h3>Results</h3><p>First, we need to validate our model. To assess convergence the Gelman-Rubin statistic was used. All point estimates are less than less than 1.1 suggesting good convergence. Trace plots and moving average plots were also assessed visually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*DNBF4PylSuB-YAUQcB1mGA.png" /><figcaption>Source: Image by the author — Gelman Rubin Statistics. All &lt;1.1 suggest good convergence.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tUOzfu68eSa7l7d5QJyZCw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MPcQynistL4VHlZLO7Qlxw.png" /><figcaption>Source: Image by the author — Trace plots of beta[1] to beta[4] on the left. Moving average plots of beta[1] and beta[2] on the right. See full results on my <a href="https://github.com/adilsaid64/bayesian-football-match-prediction">GitHub</a>.</figcaption></figure><p>From a visual analysis of the trace plots, we see that they look stationary and mixing seems to be good (They have that ‘hairy caterpillar’ look).</p><p>The moving average plots also show signs of convergence. We see the two chains start from different locations but come together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*owDwX3dYqmJXxXRIcDV70A.png" /><figcaption>Source: Image by the author — Auto-correlation</figcaption></figure><p>A high autocorrelation means that successive samples are similar to each other. This can result in the chain taking longer to converge as it might get stuck in a certain place.</p><p>At lag 20, we see autocorrelation is quite high for most parameters except beta[2]. At lag 100, autocorrelation is at a minimum for all parameters except beta[1]. At lag 200 we see that the autocorrelation is minimal for all betas. There&#39;s quite a slow decay here.</p><p>The posterior means were used to calculate the probabilities of the test set. The matches from matchday 29.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*4vGEKk4odbr4KUpZlSb96g.png" /><figcaption>Source: Image by the author — Posterior Summary</figcaption></figure><p>I used a threshold of 0.5 to classify the model&#39;s probabilities. So turning them from probabilities to binary predictions. The full probabilities from the test set are available <a href="https://github.com/adilsaid64/bayesian-football-match-prediction">here</a>.</p><p>The model achieved an overall accuracy of 78%. The recall was 61% which means it correctly identified 61% of the wins in the test set. The precision was 73% which means when the model predicted a win, it was correct 73% of the time.</p><p>We managed to get an overall F1 score of 67%. Not too bad.</p><p>Let&#39;s take a look at some of the distributions from the test set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8gURN6lv2wmgZGcCIhNNsw.png" /><figcaption>Source: Image by the author</figcaption></figure><p>This first result looks reasonable. Man City are currently the favourites over Man United. Manchester City ended up winning this game 3–1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FdxYX3E0llMWD6uemXxAKg.png" /><figcaption>Source: Image by the author</figcaption></figure><p>Taking a look at Fulham vs Tottenham. I think it&#39;s safe to assume that Tottenham would have been favourites to win this game.</p><p>Tottenham who are competing for a Champions League spot, sit in the top 6 while Fulham sit mid-table. The model predicted Fulham to win with an expected probability of 0.64 while Tottenham had 0.34.</p><p>Fulham won this game 3–0. Our model predicted that correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*tOS7FR438dMMjLI7j3Wvpw.png" /><figcaption>Source: Google</figcaption></figure><h3>Future Predictions</h3><p>Now comes the fun of predicting next week’s (match week 30) fixtures. These games haven’t been played so let’s see.</p><p><strong>Man City vs Arsenal</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o4hmgOgu_DHhS6NCKPHOjg.png" /><figcaption>Source: Image by the author</figcaption></figure><p>The expected win probability for Man City is 0.77 and Arsenal 0.79.</p><p>This game is tricky as both teams are on a great run of form and challenging for the league. The winner of this game takes second place in the league. As we can see, there is quite a lot of uncertainty in the model’s predictions because it predicts both teams to win. Although, Arsenal takes the edge in the expected win.</p><p><strong>Brentford vs Man Utd</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EoYuHanTu9x5HxYEEuTzfA.png" /><figcaption>Source: Image by the author</figcaption></figure><p>The expected win probability for Man Utd is 0.49 and Brentford 0.29.</p><p>Both teams have struggled for form recently with Manchester United winning just 1 of their last 5. Brentford winning None. The expected probability of either team winning is less than 0.5 for both teams.</p><p><strong>Liverpool vs Brighton</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_5E9e5FzBuj-u_9ksrgOA.png" /><figcaption>Source: Image by the author</figcaption></figure><p>The expected win probability for Liverpool is 0.84 and Brighton 0.39.</p><p>Liverpool currently sit first in the table. They are by far the favourites and the model shows that.</p><p><strong>Chelsea vs Burnley</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vU8zGp9AoVivJ7CdTHRrng.png" /><figcaption>Source: Image by the author</figcaption></figure><p>The expected win probability for Chelsea is 0.52 and Burnley 0.20.</p><p>Chelsea takes the edge here.</p><p><strong>Aston Villa vs Wolves</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dnvNGHmY6JThJTMIbCK2mQ.png" /><figcaption>Source: Image by the author</figcaption></figure><p>The expected win probability for Aston Villa is 0.46 and Wolves 0.36.</p><p>Aston Villa takes the edge.</p><h3>Improvements and Next Steps</h3><p>For this model, we used 6 features. Next, we could apply feature selection like Boruta or MRMR. Also, some of the features I used were correlated with each other, so using a better feature selection process and more advanced feature engineering would be good.</p><p>Also, we used a rolling window of the last 5 games. We could experiment with different window sizes to see how that affects model performance.</p><p>I would have liked to add features like ball possession, player ratings and league position at the time of the match.</p><p>Another idea could be to train a logistic model per team rather than one general model.</p><p>Thank you for reading. All the code for preprocessing, feature engineering and modelling are available on my <a href="https://github.com/adilsaid64/bayesian-football-match-prediction">GitHub</a>.</p><p>Find me on <a href="https://www.linkedin.com/in/adil-s64/">LinkedIn</a>, <a href="https://github.com/adilsaid64">GitHub</a> and my <a href="https://adilsaid64.github.io/">Portfolio</a>. Always open and appreciate feedback.</p><h3>References</h3><ol><li>Said, A. (2024). <em>adilsaid64/bayesian-football-match-prediction</em>. [online] GitHub. Available at: <a href="https://github.com/adilsaid64/bayesian-football-match-prediction">https://github.com/adilsaid64/bayesian-football-match-prediction</a></li><li>Marin, P. (2024). <em>Using Bayesian Modeling to Predict The Champions League</em>. [online] Medium. Available at: <a href="https://towardsdatascience.com/using-bayesian-modeling-to-predict-the-champions-league-8ebb069006ba">https://towardsdatascience.com/using-bayesian-modeling-to-predict-the-champions-league-8ebb069006ba</a>.</li><li>Hu, J.A. and J. (n.d.). <em>Chapter 12 Bayesian Multiple Regression and Logistic Models | Probability and Bayesian Modeling</em>. [online] <em>bayesball.github.io</em>. Available at: <a href="https://bayesball.github.io/BOOK/bayesian-multiple-regression-and-logistic-models.html">https://bayesball.github.io/BOOK/bayesian-multiple-regression-and-logistic-models.html</a></li><li>‌Plummer, M., Stukalov, A. and Denwood, M. (2023). <em>rjags: Bayesian Graphical Models using MCMC</em>. [online] R-Packages. Available at: <a href="https://cran.r-project.org/web/packages/rjags/index.html">https://cran.r-project.org/web/packages/rjags/index.html</a></li><li><a href="http://www.youtube.com./">www.youtube.com.</a> (n.d.). <em>Predict Football Match Winners With Machine Learning And Python</em>. [online] Available at: <a href="https://www.youtube.com/watch?v=0irmDBWLrco&amp;ab_channel=Dataquest">https://www.youtube.com/watch?v=0irmDBWLrco&amp;ab_channel=Dataquest</a></li><li>football-data.co.uk. (n.d.). <em>Football Betting | Football Results | Free Bets | Betting Odds</em>. [online] Available at: <a href="https://football-data.co.uk/">https://football-data.co.uk/.</a></li><li>FIFA (2021). <em>The football landscape — The Vision 2020–2023</em>. [online] FIFA Publications. Available at: <a href="https://publications.fifa.com/en/vision-report-2021/the-football-landscape/">https://publications.fifa.com/en/vision-report-2021/the-football-landscape/.</a>‌‌</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32eec733472e" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/predicting-premier-league-match-wins-using-bayesian-modelling-32eec733472e">Predicting Premier League match wins using Bayesian Modelling</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Web Scraping: A Step-by-Step Guide in Python.]]></title>
            <link>https://medium.com/@adilsaid64/introduction-to-web-scraping-a-step-by-step-guide-in-python-5ed87d67fd4a?source=rss-bc57e1ee366e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ed87d67fd4a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Adil Said]]></dc:creator>
            <pubDate>Sun, 17 Mar 2024 18:03:58 GMT</pubDate>
            <atom:updated>2024-03-17T19:33:01.095Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cUrl_7JiZ56cetyQ" /><figcaption>Photo by <a href="https://unsplash.com/@altumcode?utm_source=medium&amp;utm_medium=referral">AltumCode</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As data scientists one of the first things we must think about regarding any project is knowing how to get the data we need to solve the problem at hand.</p><p>This is often the most difficult and crucial part because we don’t have a project without access to good data.</p><p>But for most problems, data is out there and it’s everywhere. We just need to know how to get it.</p><p>Incomes web scraping. If you know how to scrape the web you can get access to almost all the data you need.</p><p>This article aims to break down web-scraping and show you how to do it.</p><p>First, we’ll go through some basics of the web and some theory. Then we’ll go through a simple project I’ve done in Python to scrape data from Google Finance.</p><p>Some basic knowledge of Python and HTML is useful.</p><h3>Disclaimer</h3><p>This article provides information on web scraping for educational purposes only. Web scraping, the process of extracting data from websites, can raise legal, ethical, and technical issues.</p><p>It is important to be aware of and respect the Terms of Service (ToS) of any website you wish to scrape. Some websites explicitly prohibit web scraping in their ToS, and doing so could potentially result in legal action against the scraper. Additionally, excessive requests to a website can overload servers, affecting the website’s operation and accessibility for others.</p><p>Before undertaking any web scraping project, it is advisable to:</p><ul><li>Review the website’s Terms of Service or robots.txt file to understand any restrictions on web scraping.</li><li>Consider the ethical implications and potential impact on the website’s resources and the availability of the data to other users.</li><li>Ensure compliance with all applicable laws and regulations, including data protection and privacy laws.</li></ul><h3>The basics of web scraping</h3><p>Here is a simple illustration of how we use websites and how we scrape it. This is just the basics to give you some intuition but there is more to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*DZ-6Kk6mZi3bJ7AGgydHKQ.png" /><figcaption>HTTP Request-Response Cycle</figcaption></figure><p>The client is us. Every time we click on a link, we make a <em>request</em> to that server. The server processes that <em>request </em>and then sends a <em>response</em>.</p><p>This is known as the HTTP (Hypertext Transfer Protocol) request-response cycle.</p><p>From the server, we receive the HTML. Once we receive the HTML we can do what we like with it. Whenever we browse the web, our websites make a <em>request</em> to the server and the server returns the HTML. Our powerful web browsers then render the HTML we receive and that&#39;s how we see it.</p><p>Anything that makes a request to a server is called a <strong>user agent</strong>. These can be web browsers, scripts, or web crawlers. User agents are typically responsible for rendering. The degree to which they render differ.</p><p>A server can detect what user agent is making a request by something called a <strong>header</strong>. A header is some piece of information sent with the request. When web scraping we can manipulate these headers and pretend to be something we aren’t.</p><p>Often pretending to be a Google Web Crawler gives you access to sites because these sites want their information displayed on Google search results.</p><p>There is more to web scraping than this and we could get into other parts in more detail in another article, this is just the basics.</p><p>The essence is we make a request to a website through code, get that response and then search for the data we want that&#39;s in that response.</p><p>We could write our own algorithms to navigate the HTML responses but luckily there exist powerful libraries in Python that we can use.</p><p>The best way is to learn through practical projects. Let’s start with a simple one.</p><h3>Scraping Google Finance</h3><p>We will build a simple portfolio evaluation tracker that scrapes the latest stock prices.</p><p>This article focuses only on the web scraping part of the project. The complete project then uses data classes to structure the data and then stores the data in an SQLite database.</p><p>For the complete project code please see my GitHub <a href="https://github.com/adilsaid64/google-finance-scraper/">here</a>.</p><p>First, let’s go to Google Finance and see the information we want to scrape.</p><p>Let’s go to <a href="https://www.google.com/finance/quote/TSLA:NASDAQ"><em>https://www.google.com/finance/quote/TSLA:NASDAQ</em></a><em>.</em></p><p>Notice the structure of the HTML? We have the stock we are interested in TSLA (Tesla) on the Nasdaq exchange. We can manipulate these in our code to get different stock prices on different exchanges.</p><p>We’ll get to that later, first, we need to find which HTML tag contains the information we want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*HFDmyqH2AGRwWoXQeMFx_Q.png" /></figure><p>We are interested in the large text that says $163.57. By inspecting the element on the web page we can find it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*xcN0aez-oZdMDmKqT0MK0g.png" /></figure><p>We need to extract this <em>div</em> that contains the ‘data-last-price’. Now that we know what we want let’s switch to Python.</p><p>We start with importing <em>requests and BeautifulSoup</em>.</p><pre>import requests as r<br>from bs4 import BeautifulSoup</pre><p>With the <em>requests </em>library<em>, </em>we can make a request to the server and receive the HTML parse of the page.</p><p>We can then use <em>BeautifulSoup</em> to extract the information we need from that HTML.</p><p>Let’s create a function that gets the HTML contents of the page. The input will be the URL and the function will return the HTML contents.</p><pre>def fetch_html(url):<br>    &quot;&quot;&quot;Fetch HTML content.&quot;&quot;&quot;<br>    resp = r.get(url)<br>    return resp.content</pre><p>We now need to parse this HTML content for the information we need. Remember that was ‘data-last-price’. We’ll use BeautifulSoup to get the div that contains the element we need.</p><pre>def parse_html_for_price_info(html_content):<br>    &quot;&quot;&quot;Gets Price Information from the HTML content.&quot;&quot;&quot;<br>    soup = BeautifulSoup(html_content, &#39;html.parser&#39;)<br>    # we want to target the div that has the data we want. which is &#39;data-last-price&#39;<br>    # get the first div that has the data-last-price attribute. True means, this attribute is set to some value.<br>    price_div = soup.find(&#39;div&#39;, attrs={&#39;data-last-price&#39;:True}) <br>    return price_div</pre><p>This function will return the div tag we need. Now that we have that div, we can create a function to get the ‘data-last-price’ information.</p><p>This function will also call our two functions from above. The parameters will be the ticker and exchange we are inserted in. Notice how they are made parameters of the function and how they are used in the URL.</p><pre>def get_price_information(ticker, exchange):<br>    &quot;&quot;&quot;Gets price information for a given ticker and exchange.&quot;&quot;&quot;<br>    url = f&#39;https://www.google.com/finance/quote/{ticker}:{exchange}&#39;<br>    html_content = fetch_html(url)<br>    price_div = parse_html_for_price_info(html_content)<br>    return {<br>        &#39;ticker&#39;: ticker,<br>        &#39;exchange&#39;: exchange,<br>        &#39;price&#39;: float(price_div[&#39;data-last-price&#39;]),<br>        &#39;timestamp&#39;: int(price_div[&#39;data-last-normal-market-timestamp&#39;]),<br>        &#39;currency&#39;: price_div[&#39;data-currency-code&#39;],<br>    }</pre><p>The div also contains other useful information like the timestamp and currency which is useful for our portfolio so we can extract them too.</p><p>And that’s it. You can now run the function <em>get_price_information</em> to get the latest stock prices and print the dictionary to view it.</p><pre>if __name__ == &#39;__main__&#39;:    <br>    print(get_price_information(ticker = &#39;TSLA&#39;, exchange = &#39;NASDAQ&#39;)</pre><p>We get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*8hgZJ8fwhcItvD6xssX4gA.png" /><figcaption>Console Output</figcaption></figure><p>Thats it! Hopefully, this article will have given you a taste of web scraping.</p><p>A web scraping project will most likely involve some use of database and data management to keep track of the data you just scraped.</p><p>In the full project, I used Data Classes to structure the data and stored the data in an SQLite database. This way we can create a historical tracker of the data we just scraped.</p><p>Thank you for reading. Feel free to check out my other articles and some feedback is always appreciated.</p><p>Find me on <a href="https://www.linkedin.com/in/adil-s64/">LinkedIn</a>, <a href="https://github.com/adilsaid64">GitHub</a> and <a href="https://adilsaid64.github.io/">My Portfolio</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ed87d67fd4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>