<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Takealot Engineering - Medium]]></title>
        <description><![CDATA[Takealot.com’s vision is to provide the most customer-centric online shopping experience in Africa. Here we’ll share more about our engineering journey of how we use technology to bring positive change and impact to the lives of our customers and communities we operate in. - Medium]]></description>
        <link>https://medium.com/takealot-engineering?source=rss----55bbf27f0c2c---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Takealot Engineering - Medium</title>
            <link>https://medium.com/takealot-engineering?source=rss----55bbf27f0c2c---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:08:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/takealot-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Getting TFX to Run on Apple Silicon]]></title>
            <link>https://medium.com/takealot-engineering/getting-tfx-to-run-on-apple-silicon-ce903861ee0b?source=rss----55bbf27f0c2c---4</link>
            <guid isPermaLink="false">https://medium.com/p/ce903861ee0b</guid>
            <category><![CDATA[tfx]]></category>
            <category><![CDATA[tensorflow-extended]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Henrico Bekker]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 08:51:52 GMT</pubDate>
            <atom:updated>2026-08-13T08:51:50.887Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How we built a pure-Python shim, a family of Docker images, a companion QA container and beat two stubborn native dependencies to learn what TensorFlow Extended is actually made of.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zacWc9ZB7VYRN727NNk6uA.png" /></figure><p>I work for Takealot Engineering, South Africa’s biggest e-commerce platform.</p><p>As an Engineering department, we bought into Apple Silicon for our day-to-day ML laptops and workstations, and the first serious thing we tried on it failed on line one:</p><pre>from tfx import v1 as tfx</pre><h3>Why “import tfx” dies on ARM</h3><p>TFX is the framework Google uses to wrap a model in everything that makes it production ready: data validation, schema inference, feature transforms, training, evaluation, blessing, and pushing. Underneath all of that sits a piece you rarely think about called ML Metadata, or MLMD - the system of record that tracks every artifact, execution, and lineage edge a pipeline produces.</p><ul><li>Every component logs to MLMD before and after it runs.</li><li>MLMD has a C++ extension at its core.</li><li>The Python you import is a thin set of wrappers; the actual local store - the thing that reads and writes the SQLite database holding all that metadata, is compiled native code.</li><li>On x86_64 there’s a prebuilt wheel and you never notice.</li><li>On arm64, that C++ extension simply does not build. No wheel, no clean compile.</li></ul><p>So the import chain reaches for a native module that isn’t there, and the whole framework collapses early on.</p><p>This has been a shared issue in the Tensorflow community for a while, <a href="https://github.com/tensorflow/tfx/issues/5804">https://github.com/tensorflow/tfx/issues/5804</a>, with no real solution, until now.</p><p>The usual escape routes are all bad:</p><ul><li><strong>Source-build the entire stack for arm64.</strong> Possible in theory, miserable in practice. We know, because we ended up doing part of it and the part we did fought back hard enough to earn its own section below.</li><li><strong>Run everything under x86 emulation.</strong> Slow, fragile, and it defeats the entire point of having fast Apple Silicon.</li><li><strong>Just use the cloud.</strong> To be clear, we don’t run production pipelines on local hardware anyway, but for development and experimentation, the cloud means you can’t iterate locally, can’t run a reproducible pipeline on a plane, and you’re paying for compute to debug a typo. Local dev saves us money and removes complexity from daily operations.</li></ul><p>None of those was a fix for us.</p><h3>The breakthrough: MLMD is two layers, and we only had to replace one</h3><p>Here’s the insight the whole thing turns on. MLMD isn’t one monolith. It’s two layers stacked on top of each other:</p><ol><li><strong>The Python surface</strong>: MetadataStore, the protobuf message bindings, the gRPC stubs. This is pure Python. It runs fine on arm64. It comes straight from upstream’s x86_64 wheel and doesn’t care what CPU you’re on.</li><li><strong>The native backend: </strong>the compiled C++ that actually turns MLMD’s API calls into SQLite reads and writes. This is the part that won’t compile on ARM.</li></ol><p>So we kept the top half and rewrote the bottom half. The native backend got replaced with a pure-Python implementation, a single file, <em>metadata_store_extension.py</em>, that pretends to be the missing C++ module and translates MLMD’s RPC calls into ordinary SQLite queries against the same database TFX expects.</p><p>The Python surface keeps making the same calls it always made. It has no idea the floor underneath it changed. As far as TFX is concerned, MLMD is present and working. As far as our Macs are concerned, nothing native ever had to compile.</p><p>To be precise about where we fought and where we dodged: TFX has <em>three</em> native dependencies with no aarch64 wheels, MLMD, TFDV (TensorFlow Data Validation), and tfx-bsl.</p><p>We do compile the latter two from source inside the image build, because they’re <em>easy</em> fights (see the gotchas below). MLMD is the one we replaced instead of compiled, because its native layer turned out to be a middleman between a Python API and a SQLite file - exactly the kind of middleman you can rewrite in the language that runs anywhere.</p><p>We didn’t reimplement all of it. MLMD’s store exposes 52 RPC methods; the shim implements 38 - the ones a real pipeline actually exercises: putting and getting artifacts, executions, contexts, events, the type system, and the launcher-path filter queries TFX issues between components.</p><p>That’s enough to run the full DAG end to end. (What the other 14 cost us, we’ll come back to - being honest about the gaps is part of the point.)</p><p>The shim gets mounted into the container’s site-packages at startup, dropped exactly where the C++ module would have lived.</p><p>The import succeeds. The pipeline runs.</p><h3>Wrapping it so it’s actually usable</h3><p>A clever shim you have to hand-mount every time is not a tool, it’s a party trick. The day-to-day breakthrough was making it disappear.</p><p>Everything ships as CPU-only Docker images built for linux/arm64, in four variants selected by image tag:</p><ul><li><strong>TFX 1.17</strong> (TensorFlow 2.17.1, Apache Beam 2.55.0)</li><li><strong>TFX 1.21</strong> (TensorFlow 2.21.0, Beam 2.70.0), each available for Python 3.10 and 3.11, for example tfx:1.21-arm64-py311.</li></ul><p>All variants run Beam’s DirectRunner and carry the full offering of tensorflow-* siblings (TFDV, TFT, TFMA, tfx-bsl).</p><p>On top sits a small bash wrapper, tfx-run, that handles the three fiddly things you’d otherwise get wrong (but optional):</p><pre># run the included smoke test against the Palmer Penguins dataset<br>tfx-run pipeline penguin_smoke.py</pre><p>That one command quietly does what would otherwise be a paragraph of docker run flags: it mounts the MLMD shim into site-packages, it bypasses the image’s default Beam SDK entrypoint (/opt/apache/beam/boot, which otherwise hijacks the container and greets you with a cryptic “No id provided”), and it mounts your working directory plus a shared volume so artifacts land on the host.</p><p>Three to five minutes later, on CPU, you get:</p><pre>SUCCESS: pushed model at /workspace/penguin/serving_model_dir/penguin-model/&lt;timestamp&gt;</pre><p>The full TFX graph - <em>example gen, statistics, schema, transform, train, evaluate, bless, push</em>, running locally on a laptop that the framework officially refuses to run on.</p><h3>The gotchas that weren’t code problems at all</h3><p>A note for anyone who builds the images themselves rather than loading the prebuilt ones: the wheel-build stage compiles TFDV and <em>tfx-bsl</em> from source, and two things will try to stop you. Neither of them is a bug in your setup, and neither error message will tell you what’s actually wrong.</p><p><strong>1. Google renamed the ZetaSQL repo, and Bazel noticed.</strong></p><p>TFDV and <em>tfx-bsl</em> both pin a sha256 checksum against the google/zetasql archive tarball. Google has since renamed the repository to google/googlesql. GitHub redirects the old URL - but the tarball the redirect serves hashes differently (the internal path prefix changed along with the name), so Bazel rejects the download as corrupted.</p><p>The content is fine; the checksum and the strip-prefix are simply describing a repo that no longer exists under that name. The fix is to patch the WORKSPACE in both projects: swap in the sha256 of the tarball GitHub actually returns today, and change the <em>zetasql-&lt;commit&gt;</em> prefix to <em>googlesql-&lt;commit&gt;</em>. We pass the sha as a build arg, because when it changes again, and it will, we’d rather change one flag than hunt through two WORKSPACE files.</p><p><strong>2. The compile that dies without an error.</strong></p><p>Assuming the downloads succeed, the build runs for about thirty minutes and then, if you’re unlucky, dies at the ZetaSQL compile step. We lost real time to this before realizing it isn’t a code error, it’s a memory error.</p><p>ZetaSQL’s <em>resolved_ast_rewrite_visitor.cc</em> needs roughly 10 GB of RAM to compile a single translation unit. Docker Desktop’s default memory ceiling is well below that, so the compiler gets OOM-killed mid-file and you go hunting for a bug that doesn’t exist.</p><p>The fix is boring and total: give Docker at least 16 GB (24 is comfortable) and cap Bazel at one job - our Dockerfiles default to: jobs=1 and drop the hot files to -01 for exactly this reason. We’re flagging it because every minute we spent diagnosing it was a minute the error message never spent explaining itself.</p><p>(There’s a third, smaller patch worth knowing about: Apache Arrow ships AVX2/AVX-512 source files that <em>#include &lt;immintrin.h&gt; </em>unconditionally, and that header doesn’t exist on arm64. Our build scripts neutralize those files before compiling. It’s mechanical once you know, and baffling before you do, which is the whole “miserable in practice” of source-building, condensed into one include line.)</p><h3>Getting the GPU back: the hybrid workflow</h3><p>There’s one thing the container genuinely cannot do, and it’s worth being clear-eyed about: no GPU inside Docker on a Mac. This isn’t a Docker bug or a missing flag. The macOS hypervisor doesn’t expose Metal to guest VMs. There is no workaround, because there’s nothing to work around - the GPU simply isn’t on the other side of the wall.</p><p>Most of a TFX pipeline is happy on CPU anyway - ingestion, statistics, evaluation, blessing, pushing are all Beam work that never wanted a GPU in the first place.</p><p>But two stages sit differently. Training, obviously. And, less obviously, the steps just before it: schema validation and the feature transforms.</p><p>The Transform component executes real TensorFlow graphs over your data, and if those transformations are the kind that benefit from a GPU - heavy vocabulary computations, embeddings, anything tensor-shaped rather than row-shaped, then Transform benefits too.</p><p>More importantly, deciding <em>what</em> the transformations should be is one of the fastest-looping parts of pipeline development: you inspect the statistics, adjust the schema, rewrite a preprocessing_fn , look at the output, and go again. That loop is exactly where you want your fastest hardware, not a CPU-bound container.</p><p>So our setup splits across two environments that hand work to each other through a shared folder:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IP0kobSIir_3T_pPrrULeA.jpeg" /></figure><p>We prototype in a Metal-accelerated virtual environment - fast iteration on architecture and hyperparameters with the GPU doing real work.</p><p>We save the proven model to the shared directory, then the container’s Trainer loads that pre-trained model instead of training from scratch, and the rest of the pipeline does what pipelines are for:</p><p>validate &gt; evaluate &gt; track lineage &gt;bless &gt; push.</p><p>GPU where it helps, reproducible container everywhere else.</p><h3>What it still can’t do</h3><p>The most useful section of any infrastructure write-up is the honest list of what doesn’t work, so here’s ours.</p><p>Those 14 unimplemented RPC methods include GetLineageGraph and GetLineageSubgraph. They’re stubbed. Pipeline execution doesn’t touch them, so everything runs, but lineage-graph tooling like tfx model_card won’t work locally. The shim’s filter parser also only understands the launcher-path queries TFX itself issues; the property-based filters some Resolver strategies use aren’t supported.</p><p><strong>Beyond the shim</strong>: there’s no official tensorflow/serving image for arm64, so production HTTP serving means building from source, using a community image, or deploying to an x86 box. And this whole setup is for the local DirectRunner, point it at Dataflow or Flink or Spark and you’re in different infrastructure entirely.</p><p>None of these breaks the thing we set out to do.</p><p>They’re the edges of a tool that does one job well: run real, reproducible TFX pipelines on an ARM laptop.</p><h3>Closing the loop: a companion QA container</h3><p>The honest-limits list above flags one gap that stings more than the others, no TF Serving on arm64. A pipeline that trains, evaluates, blesses, and pushes a model is only half the job; at some point you have to point real inputs at the served model and check that the numbers coming out are sane. That’s exactly the step the pipeline image can’t do on a Mac, and it’s the step we’d been doing by hand with awkward base64-encoded <em>tf.Example</em> payloads.</p><p>So we built a second, smaller tool that picks up precisely where the pipeline puts the model down: a QA container, <em>tensorflow-qa</em>, that takes a pushed model and a CSV of test cases and gives you back predictions. Crucially, it ships for both architectures<strong>:arm64 </strong>and<strong> :amd64,</strong> selected purely by the image tag, and internally it stands up TensorFlow Serving (plus a TensorFlow sibling) through the Docker socket. In other words, it does the one thing the pipeline image admits it can’t: it runs TF Serving on Apple Silicon.</p><p>The arm64-serving gap doesn’t get worked around so much as quietly handed to a tool built for it.</p><p>The handoff is as direct as it sounds.</p><p>The pipeline pushes a model to serving_model_dir/&lt;model&gt;/&lt;timestamp&gt;/; you hand that path, plus a CSV, to the QA container:</p><pre>docker run -v app-shared:/app -v /var/run/docker.sock:/var/run/docker.sock \<br>eu.gcr.io/…/tensorflow-qa:arm64 ~/model ~/data/qa.csv</pre><p>The CSV is deliberately boring, first row the feature names, second row the data types, the rest your test cases:</p><pre>text,rating<br>string,int<br>&quot;I love this product, best company ever.&quot;,5<br>&quot;Really bad product, the colours are ugly.&quot;,1</pre><p>…and at the end of a fairly verbose log you get the thing you actually wanted:</p><pre>predictions<br>[0.235232338]<br>[0.774663508]</pre><p>No hand-rolled serving payloads, no standing up TF Serving yourself, no remembering the serving_default signature dance the pipeline’s troubleshooting section warns about. (You still need to know the model’s serving signature, but that’s a question for the team that trained it, not a thing to reverse-engineer at 11pm.)</p><p>Two ways this genuinely complements the first solution, beyond just filling the serving gap:</p><p><strong>Cross-architecture parity.</strong> The QA container’s interface is identical on arm64 and amd64. You can validate a model on your Mac during development and re-run the exact same check on a Linux CI box, and a divergence in predictions becomes a real signal rather than noise from two different setups. That’s the same reproducibility goal that made us keep the pipeline images CPU-only - extended one step further, into validation.</p><p><strong>A clean division of labour.</strong> The pipeline image, with its MLMD shim, answers <em>how the model was made </em>every artifact, execution, and lineage edge. The QA container answers does the model behave, black-box, prediction-level, no metadata required. One is provenance; the other is behaviour. Together they cover a model from <em>CsvExampleGen</em> all the way to “does it actually score this review correctly,” without ever leaving an ARM laptop.</p><p>The two even share the same operational discipline: both run on a plain Docker runtime, and on a Mac that means a socket-exposing runtime like colima rather than Docker Desktop. Learn the setup once, use it for both halves.</p><h3>The takeaway</h3><p>The headline is “TFX on Apple Silicon,” but the lesson underneath is more general than that. When a dependency won’t build on your platform, the reflex is to fight the compiler. The better question is which layer of that dependency you actually consume.</p><p>We fought the compiler where the fight was winnable TFDV and tfx-bsl compile for arm64 once you patch around a renamed repo, a missing x86 header, and a 10 GB translation unit. MLMD looked like the same kind of fight.</p><p><strong>It wasn’t.</strong></p><p>It turned out to be a thin Python API sitting on a SQLite database, with a C++ middleman we could simply replace RPC by RPC, in a language that runs anywhere.</p><p>The breakthrough wasn’t building TFX for ARM, it was realizing that, for the piece that mattered most, we didn’t have to.</p><p>Below link to the essential code you’ll need to replicate what we’ve built.</p><p><a href="https://gist.github.com/hbk-dml/00109f24ae1665645de21876ab1dfd66">https://gist.github.com/hbk-dml/00109f24ae1665645de21876ab1dfd66</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce903861ee0b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/takealot-engineering/getting-tfx-to-run-on-apple-silicon-ce903861ee0b">Getting TFX to Run on Apple Silicon</a> was originally published in <a href="https://medium.com/takealot-engineering">Takealot Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Refactoring with AI: From “Good Luck” Legacy Code to Clean, Testable Design]]></title>
            <link>https://medium.com/takealot-engineering/refactoring-with-ai-from-good-luck-legacy-code-to-clean-testable-design-d83362ed355d?source=rss----55bbf27f0c2c---4</link>
            <guid isPermaLink="false">https://medium.com/p/d83362ed355d</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[code-refactoring]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Sascha Barrett Lenz]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 08:16:22 GMT</pubDate>
            <atom:updated>2026-08-07T10:14:59.522Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8p8YiavEaN_uZmX7Aeshnw.png" /></figure><p>Every engineering team has one. That function. The one where your senior colleague leans over with a knowing look and says: “<em>Good luck with that one.</em>”</p><p>I got that ticket. What followed was challenging, frustrating, and ultimately rewarding — made possible in large part by AI. But not the way the hype suggests.</p><p>This is the honest story of how I used AI to tackle a legacy refactor, what worked, what spectacularly didn’t, and what I’d do differently.</p><h4>The Legacy Code Situation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YfZ4BAZaYzblGj17aqnYUw.png" /></figure><p>The ticket seemed simple: optimize a product submission workflow where even minor changes triggered a 30-to-120-second validation pipeline by skipping unnecessary revalidation. Then I opened the function — the <strong>Pandora box</strong>. Organic, overgrown code that no one fully understood, not due to poor engineering, but because software naturally accumulates complexity over time.</p><p>The red flags were everywhere:</p><ul><li>A single <strong>~700-line function</strong></li><li>Brittle <strong>if-elif-else chains</strong></li><li><strong>Tangled logic</strong>: validation, DB queries, and side effects all in one block</li><li><strong>Deeply nested conditions</strong> that are impossible to trace</li><li>A <strong>300-line secondary function</strong> called only by the first</li></ul><p>The danger of the Pandora box is that the impact of any change is unknowable — making it impossible to reason about safely.</p><h3>Starting the Conversation with AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xenCJz_xkuGGzFwmfUofiQ.png" /></figure><p>My initial prompt was poor — I dumped the file in and asked what design patterns could clean it up. I expected vague platitudes. Instead, the AI came back with five concrete recommendations: <strong>Command, Strategy, Repository, Chain of Responsibility, and Builder</strong>. Not because it wrote the solution, but because it acted like a well-read reference book, pointing me toward the right vocabulary. That alone was worth the conversation.</p><h3>What the AI Suggested — And Where It Started Falling Apart</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MgrCiw9HOyVJkzAy9jP9Tg.png" /></figure><p>Encouraged, I pushed further and asked it to apply those patterns directly to my function. This is where things went sideways. The code it generated looked plausible — clean, well-structured, confident. It was also <strong>mostly wrong</strong> for my context. AI had no idea how our data was shaped or what our conventions were, so it filled the gaps with assumptions that didn’t hold.</p><h3><strong>The Before and After</strong></h3><p>After several iterations, a clear design emerged. The 700-line monolith became a set of focused, single-purpose components:</p><ul><li>One <strong>entry point</strong> that simply assembles and runs the pipeline — no logic, just orchestration</li><li>A <strong>pipeline</strong> where each task lives in its own isolated, testable class — replacing the long if-elif chain</li><li>A <strong>strategy layer</strong> that handles variations in behaviour cleanly, without nested conditions</li><li>A <strong>post-processor</strong> that handles downstream effects separately from the core logic</li><li>A <strong>data cache</strong> that fetches once, works in memory, and commits at the end — eliminating repeated round-trips to the database</li></ul><p>The result: code where you can open any single file and understand exactly what it does, change it safely, and test it in isolation. That’s the real win.</p><h3>How NOT to Use AI for Refactoring</h3><p>Here’s what I learned the hard way:</p><ul><li><strong>Don’t throw the whole problem at it at once.</strong> Large, complex functions overwhelm the context window — the bigger the task, the more AI hallucinates to fill in what it doesn’t know.</li><li><strong>Don’t assume it understands your codebase.</strong> AI builds context from what it can see, not from years of institutional knowledge your team has built up.</li><li><strong>Don’t mix manual edits with AI edits without communicating.</strong> I manually corrected something, went back to the AI, and it confidently re-applied its original wrong version — overwriting my fix.</li><li><strong>Don’t skip the analysis phase.</strong> The most valuable thing AI can do in a refactor is help you <em>understand</em> the code before you change it.</li></ul><h3>The Right Mental Model</h3><p>The shift that made everything click was changing how I thought about AI as a collaborator.</p><blockquote><em>Think of AI as a brilliant new senior developer who just joined your team.</em></blockquote><p>They have deep knowledge and strong technical skills. But they’ve never seen your codebase. They don’t know your domain, your conventions, or the weird edge cases your system has accumulated over years. You wouldn’t hand them a 700-line function and say “fix this” on their first day. You’d brief them properly.</p><p>The same applies to AI. You are the domain expert. You bring the context, the constraints, and the judgement. AI brings the craft. Together, you’re more effective than either alone.</p><h3>The Incident — Being Honest About What Went Wrong</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZSw97nmakvS_MaCETbM9yA.png" /></figure><p>After implementation and two weeks of QA regression testing, we deployed.</p><p>It didn’t go well.</p><p>I had assumed that each product in a submission would have a unique identifier. It seemed reasonable. It was wrong — and it brought down the workflow for affected seller partners who submitted the product requests. Two important lessons:</p><ol><li><strong>Validate assumptions with your teammates.</strong> A 60-second conversation with a senior colleague would have caught this immediately.</li><li><strong>Use real production data to build regression tests before you refactor.</strong> AI is excellent at helping you build that kind of black-box test harness — it just needs real data to work from.</li></ol><h3>The 3-Step Playbook</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqMjQSMhZKCcUO3rt0XMtA.png" /><figcaption>Each step is iterated until you are confident the AI has the full picture.</figcaption></figure><p><strong>Step 1 — Analyse</strong></p><p>Don’t ask AI to change anything yet. Ask it to help you understand the code. What does this function actually do? What are the edge cases?</p><p><strong>Step 2 — Design</strong></p><p>Ask AI to propose a structure, not implement it. Iterate until it fits your context. Document this — you’ll use it as a reference in step 3</p><p><strong>Step 3 — Implement</strong></p><p>One small task at a time. Review each output before moving on. Always tell it about any manual changes before continuing.</p><h3>Closing — AI Is Not the Enemy (But It’s Not Magic Either)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*siG-QmbQq_B6FmKr_s7jvg.png" /></figure><blockquote>AI isn’t magic, but it’s a genuine force multiplier. Somewhere between 10% and 100% more productive on the right tasks — and refactoring legacy code is absolutely one of them.</blockquote><p>It makes the Pandora box less intimidating when you have a collaborator who can help you read it, understand it, and design a path through it.</p><p>If you haven’t adopted AI in your day-to-day development yet, give it a try — not to do the work for you, but to make the tough jobs more approachable.</p><p>That “good luck” function you’ve been avoiding? Maybe now is the time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d83362ed355d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/takealot-engineering/refactoring-with-ai-from-good-luck-legacy-code-to-clean-testable-design-d83362ed355d">Refactoring with AI: From “Good Luck” Legacy Code to Clean, Testable Design</a> was originally published in <a href="https://medium.com/takealot-engineering">Takealot Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How AI Helped To Cut Our Test Suite from 8 Minutes to 19 Seconds]]></title>
            <link>https://medium.com/takealot-engineering/how-ai-helped-to-cut-our-test-suite-from-8-minutes-to-19-seconds-108e636fa0ba?source=rss----55bbf27f0c2c---4</link>
            <guid isPermaLink="false">https://medium.com/p/108e636fa0ba</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pytest]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[spec-driven-development]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Johann Eicher]]></dc:creator>
            <pubDate>Thu, 06 Aug 2026 11:04:59 GMT</pubDate>
            <atom:updated>2026-08-06T11:04:57.862Z</atom:updated>
            <content:encoded><![CDATA[<h4>With a spec and a plan, we avoided dead-end solutions, and got a blazingly fast test suite</h4><p>If you’ve used an AI coding assistant, you’re probably familiar with this experience: sometimes it solves a problem in one shot; other times it ends up in a <a href="https://bitsofrandomness.com/breaking-the-doom-prompting-loop-with-spec-driven-development">“doom-prompting loop”</a>, circling the issue without getting anywhere. We decided to see if we could get around that trap using <a href="https://martinfowler.com/articles/exploring-gen-ai/sdd-3-tools.html">spec-driven development</a>: write a technical specification and implementation plan first, then use both to guide the AI. We applied this to a painfully slow Python test suite in our team&#39;s largest repository: roughly 8 minutes a run, and every pushed commit an excuse for a coffee break. The documents give the agent a concrete guideline to navigate by, so it can&#39;t keep going in circles with seemingly plausible edits that never converge.</p><h3>Overview</h3><p>AI suggested and implemented six optimisations, which took the suite from roughly 8 minutes to about 19 seconds, a 25× speedup. None of them touched test logic or assertions: only fixtures, database setup, or execution plumbing, so coverage couldn’t be affected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*72Xt5YFJN9Ek4Qk4BBePBw.png" /></figure><p>Each stage had to run faster and stay green before we continued. Afterwards, out of curiosity, we checked out each phase in git, ran the suite in triplicate, and recorded mean and standard deviation.</p><h3>The approach: spec first, then implement</h3><p>We worked in Cursor. Planning ran on Claude Opus 4.8; implementation alternated between Composer 2.5 and Claude Sonnet 4. We used the heavier model for the thinking, and lighter ones for the implementation work. That split is <a href="https://code.claude.com/docs/en/model-config.md">a well-established practice</a>.</p><p>The technical spec captured “what” we wanted: a phased approach with a hard stop under 30 seconds, and a preference for boring mechanical changes over clever ones. We didn’t want to build a Death Star to shave off half a second of test time. From there the AI wrote an implementation plan — the “how” — detailing each stage and which parts of the codebase needed to change.</p><p>It wasn’t perfectly linear. Parallel execution started out of scope and only became Phase 6 once the other optimisations hadn’t hit the 30 second goal. When the plan missed something in practice, we’d send the AI back to revise the docs before continuing. That back-and-forth is the approach working: the AI still did the vast majority of the work.</p><h3>The six optimisations</h3><h4>1. Stop paying for a fresh database on every run.</h4><p>pytest-mock-resources spun up a fresh Docker Postgres every session, with production fsync = on for data we’d throw away minutes later. The AI pointed us at our existing local-dev container with --no-fsync instead — fine for throwaway test data, never for production.</p><p><strong>Impact:</strong> 475s → 261s.</p><h4>2. Stop paying DB overhead on tests that don’t need a DB.</h4><p>All 781 tests connected to Postgres; 204 were pure logic tests. Dropping autouse=True made seeding opt-in.</p><p><strong>Impact:</strong> 261s → 129s.</p><h4>3. Batch the cleanup query.</h4><p>Cleanup ran as 20+ individual TRUNCATE statements per test. The AI collapsed those into one.</p><p><strong>Impact: </strong>129s → 84s.</p><h4>4. Load immutable reference data once per session.</h4><p>Static lookup rows were re-inserted on every test — almost 4000 identical inserts per run. The AI moved them to a session-scoped fixture.</p><p><strong>Impact:</strong> 84s → 79s.</p><h4>5. Bulk-insert fixtures instead of one-by-one.</h4><p>About 50 fixtures were going in via separate session.add() / flush() calls. The AI switched to one session.add_all() and one flush().</p><p><strong>Impact:</strong> 79s → 75s.</p><h4>6. Use all the CPU cores.</h4><p>With progress having plateaued, the AI added pytest-xdist and gave each worker its own database. Fourteen workers, fourteen databases (test_gw0 through test_gw13), no shared state.</p><p><strong>Impact:</strong> 75s → 19s.</p><h3>Results</h3><p><strong>Total: 475s → 19s, a 25× speedup, or 96% reduction.</strong></p><p>What used to be a coffee break is now short enough to sit through. No <a href="https://xkcd.com/303/">swords</a> required, and CI is much faster too.</p><h3>Takeaways</h3><p>None of these optimisations would surprise an experienced engineer. The difference was implementing them carefully, one at a time, against a plan that said what to do next and a spec that said when to stop. Spec-driven development didn’t give the AI superpowers; it gave it a roadmap and a finish line.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=108e636fa0ba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/takealot-engineering/how-ai-helped-to-cut-our-test-suite-from-8-minutes-to-19-seconds-108e636fa0ba">How AI Helped To Cut Our Test Suite from 8 Minutes to 19 Seconds</a> was originally published in <a href="https://medium.com/takealot-engineering">Takealot Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Realities of Generative AI in Software Engineering]]></title>
            <link>https://medium.com/takealot-engineering/the-realities-of-generative-ai-in-software-engineering-e1dcb2a92836?source=rss----55bbf27f0c2c---4</link>
            <guid isPermaLink="false">https://medium.com/p/e1dcb2a92836</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Pieter Rautenbach]]></dc:creator>
            <pubDate>Fri, 09 Jan 2026 12:51:30 GMT</pubDate>
            <atom:updated>2026-01-09T12:51:26.322Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*8ZgYEPleQigp6M3FMibZnA.png" /></figure><h4><em>Verification Above All</em></h4><p>There’s still a gap between the excitement around generative AI (GenAI) and the reality of how it actually works. Many see it as a shortcut to higher productivity; few grasp its limits. The truth is simple: <strong>GenAI is powerful — but fundamentally unreliable without human verification.</strong></p><p><em>Note: GenAI is a subset of AI, even though the two terms are used interchangeably. AI ranges from rule-based systems to the most sophisticated large learning models (LLMs), including the ultimate goal in autonomy, often called “artificial general intelligence” (AGI).</em></p><h3>Human in the Loop</h3><p>Generative AI excels at the <em>mechanical</em> parts of software creation, such as generating boilerplate and accelerating the initial draft. However, the core of Software Engineering is inherently <a href="https://terriblesoftware.org/2025/12/11/ai-can-write-your-code-it-cant-do-your-job/">a human and social endeavor</a>, not a purely technical one.</p><p>The difference between an LLM that generates code and an engineer lies in the critical human factors it cannot replicate. This includes the social context and synthesis found in the conversations needed to unpack requirements, negotiate trade-offs, and explore solutions. Engineering relies on architectural foresight, the ability of experienced developers to “see the future” of a system under scale and changing demands, a long-term predictive modeling that goes beyond an LLM’s statistical pattern matching.</p><p>Furthermore, a successful system is built on institutional knowledge — the unwritten rules, historical compromises, and domain-specific details that inform every major design decision. Beyond skill, the engineer holds ethical and legal accountability for the output, ensuring the code meets security, legal, and maintainability standards. The LLM can generate a technical artifact, but it cannot bear the liability.</p><p>The human remains essential for validation, verification, ethical oversight, and connecting the technical artifact to its ultimate purpose of solving a human problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_CG4C8rU6NIL8-Idej5g2Q.jpeg" /><figcaption>Hallucinations are real.</figcaption></figure><h3>Hallucination: It’s Not a Bug; It’s a Feature</h3><p>LLMs don’t <em>understand</em> the world; they generate text by predicting what’s statistically most likely to come next based on patterns in data. Their “reasoning” isn’t logical — it’s <strong>statistical reasoning</strong>. What looks like deduction or inference is really pattern completion at scale.</p><p>That’s why <em>hallucination</em> isn’t a rare glitch — it’s a natural consequence of how these systems work. They can simulate reasoning impressively well, but they don’t actually know when they’re right.</p><p>They can’t self-verify because they don’t reason about <em>truth</em> — their only mechanism is <em>statistical</em> <em>likelihood</em>. And because their outputs are fluent, coherent, and confident, humans tend to trust them more than they should. Even under strict deductive logic, a perfectly valid statement can still be false if its premise is wrong. For example: “All cats can fly. Luna is a cat. Therefore, Luna can fly.” Thus, if an LLM learns a false premise, it will produce deductions that are logically sound but fundamentally untrue — and sometimes absurd.</p><p>The real risk isn’t hallucination itself — it’s <strong>misplaced trust</strong>. Humans are wired to believe articulate systems, even when they’re confidently wrong. It is the same psychological lever that pseudo-science exploits.</p><p>Verification isn’t optional; it’s essential.</p><h3>Fast, But Efficient?</h3><p>AI is undeniably fast at generating output — but that speed comes at a cost. Running large models requires enormous computational power, translating into significant energy and infrastructure expense. Users rarely perceive that cost directly, even if they pay for a service, but compared to the human brain, it’s <em>vastly</em> less efficient. What looks instant is actually the result of immense parallel computation happening elsewhere.</p><p>Beyond resource cost, speed often sacrifices <em>clarity</em>. AI compresses <strong>generation time</strong> but inflates <strong>verification time</strong>. You get something quickly — but you spend longer checking, debugging, refining and reworking it. For many experienced engineers, that <a href="https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/">makes the overall process <em>slower</em></a> than doing it properly from scratch.</p><p>The real advantage isn’t pure velocity — it’s <em>exploration</em>. GenAI is great for brainstorming and breaking creative inertia, not for bypassing the thinking process. Crafting effective prompts can itself require significant thought, effort and time, and the insights you get depend on how carefully you guide the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J9nE6KIsEMt-8guLsihuXQ.jpeg" /></figure><h3>Use AI Where It Fits</h3><p>GenAI works well when the task is narrow and well-defined: generating boilerplate, exploring design alternatives, or drafting repetitive code, but it’s no substitute for deep engineering. There’s no prompt that will architect and deploy a large, scalable and resilient system on its own or from a couple of interactions. It’s also not capable of autonomously maintaining and improving a system.</p><p>The rule is simple: <strong>use AI to accelerate, not to abdicate</strong>. Keep humans responsible for testing and verification — that’s where accountability lives.</p><h3>Learning vs. Mastery</h3><p>AI works best in expert hands. Senior engineers can spot inconsistencies and know when something “feels off”, but it can also accelerate learning — <em>if</em> used deliberately.</p><p>The danger is when junior developers use AI as a crutch instead of a coach. Without the incentive to question and understand what’s being produced, they risk becoming surface-level operators — able to prompt, but unable to debug or solve problems effectively. By asking well-defined, targeted questions on topics that a developer has some familiarity with, AI can become a powerful tool for discovery and learning. For instance, it can clarify abstract concepts, suggest alternative approaches to coding challenges, or offer fresh perspectives on solving problems.</p><p><strong>Access without understanding is a liability.</strong> Democratisation only works if it’s paired with education and curiosity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RktFEHBspz03YHtmFzhDwg.png" /></figure><h3>Context Is Everything</h3><p>Even the most capable models struggle with complex systems that depend on implicit context, integration details, and domain-specific rules. Larger context windows help, but they don’t fix the core problem: <strong>AI doesn’t know what it doesn’t know.</strong></p><p>Humans navigate complex systems with implicit understanding; a terse sentence or shared context can convey volumes of unspoken institutional knowledge. The LLM, by contrast, demands extreme verbiage and relentless prompting. The engineer must externalise every assumption, constraint, and piece of implicit context into a series of explicit, unambiguous instructions. This inflation of the required conversational bandwidth, coupled with the high cost of the back-and-forth refinement loop, often negates the initial speed advantage. The process trades rapid generation for slow, painstaking communication and verification, leading to the feeling that a “simple task” requires too many iterations to get done.</p><p>Human verification remains essential, not only to catch mistakes, but to ensure the output even <em>fits</em> the problem it’s meant to solve.</p><h3>Democratising AI, Responsibly</h3><p>GenAI’s growing accessibility is a double-edged sword. Developers can now embed GenAI capabilities through APIs or prompt engineering without deep knowledge of how LLMs work — a genuine step forward for productivity, and for addressing coding challenges that were traditionally difficult to solve using purely syntactic checks or weak semantic validation.</p><p>At a fundamental level, someone still needs to architect, build, and maintain the AI models and systems themselves, and that is still software engineering. The real risk of democratisation is the erosion of foundational skill. When powerful tools are used blindly, they create a dependency that prevents learning and obscures crucial flaws. This makes professional judgment and expertise more vital, not less. The balance lies in using AI to extend developer capability, not to replace foundational skill or judgement.</p><p>As an analogy, handing out scalpels to everyone doesn’t make people surgeons. For simple tasks — like bandaging a wound — everyday users manage fine. But intricate surgeries, like removing an appendix, require professionals. Similarly, while everyday users can likely generate functional code for a quick fix, specialist engineers are indispensable for creating scalable, ethical, and resilient systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yW6gBysWh44-Jp3QmxLXkQ.png" /></figure><h3>A Voice of Reason in the Hype</h3><p>Generative AI has shown real utility in software engineering, particularly in automating repetitive coding tasks, generating boilerplate code, and assisting with documentation. It’s a powerful tool for <a href="https://addyo.substack.com/p/my-llm-coding-workflow-going-into">accelerating development workflows</a> and reducing the cognitive load of routine tasks, freeing engineers to focus on more complex and creative problem-solving. When used thoughtfully, it can amplify productivity and unlock new possibilities in software development.</p><p>But every wave of new technology brings hype, and AI is no exception. The bigger danger today isn’t that companies and software engineers ignore AI, but that they adopt it <em>without purpose</em>. Many projects chase “AI for AI’s sake,” without defining a problem or evaluating value. In such cases, the result could be wasted effort, inefficiency, and technical debt that skilled professionals will eventually need to untangle.</p><p>Caution isn’t resistance. It’s maturity. Verification, governance, and critical thinking turn AI from a shiny demo into a durable capability.</p><p><strong>The bottom line:</strong> GenAI is here to stay — but it predicts; it doesn’t know. Treat its output as inspiration, not truth. Use it with expertise, curiosity, and skepticism, and it will make you faster <em>and</em> smarter. Use it blindly, and it will just make your mistakes arrive sooner.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1dcb2a92836" width="1" height="1" alt=""><hr><p><a href="https://medium.com/takealot-engineering/the-realities-of-generative-ai-in-software-engineering-e1dcb2a92836">The Realities of Generative AI in Software Engineering</a> was originally published in <a href="https://medium.com/takealot-engineering">Takealot Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine learning at Takealot and MrD]]></title>
            <link>https://medium.com/takealot-engineering/machine-learning-at-takealot-and-mrd-87246795f9f1?source=rss----55bbf27f0c2c---4</link>
            <guid isPermaLink="false">https://medium.com/p/87246795f9f1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <dc:creator><![CDATA[Axel Tidemann]]></dc:creator>
            <pubDate>Wed, 08 Oct 2025 08:10:05 GMT</pubDate>
            <atom:updated>2025-10-08T08:10:01.520Z</atom:updated>
            <content:encoded><![CDATA[<p>The field of AI has developed significantly since the <a href="https://medium.com/takealot-engineering/machine-learning-at-takealot-4c1a7faa01c4">previous post</a> about the machine learning space at Takealot, published in 2022. We’d like to bring you up to speed with the current state of machine learning at Takealot and MrD.</p><h3>The technologies</h3><p>The Google Cloud Platform (GCP) is central to Takealot engineering. Kubernetes is the foundation for our micro services architecture, upon which machine learning services are built.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i_TEIS7hcLIvm5NSTFwEjA.png" /><figcaption>The Takealot AI architecture, running on GCP.</figcaption></figure><p>For almost all use cases, data already exists in BigQuery. The data engineering and analytical engineering teams are fundamental in maintaining data pipelines for that purpose. Dataform pipelines are used to create train, validation and test sets from the various BigQuery tables. In cases where data lives in Sheets or in Cloud Storage buckets, ad-hoc programmatic solutions are developed to transform the data into BigQuery tables.</p><p>Jupyter Notebooks play an important part for exploratory data analysis, but for production use cases, machine learning models are exclusively orchestrated via Kubernetes. For example, a training job is started through the use of a CronJob, and the model is subsequently served via TensorFlow Serving.</p><p>The TensorFlow ecosystem plays a central role in developing predictive AI models. TensorFlow Extended (TFX) is used for machine learning development, TensorBoard is used to monitor training. TensorFlow Serving is how models are deployed in production.</p><p>Not all models are served online, some instead make batch predictions on regular cadences, which are stored as CloudSQL or BigQuery tables. Some form the basis for Looker reports. Batch predictions and online predictions are served via REST APIs or Kafka producer/consumers.</p><p>Generative AI development is done using Google’s flagship model, Gemini, accessed via Vertex AI APIs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pRlC2rwEMSWo6KBsigjpIg.png" /><figcaption>A typical machine learning pipeline, that comprises a Dataform pipeline and a TFX pipeline, orchestrated via Kubernetes (here abbreviated as K8s).</figcaption></figure><p>The machine learning models are monitored using Google Sheets, Grafana and Looker. “Monitoring” in this sense means both technical resources, as well as metrics (e.g. precision and recall for a particular model).</p><p>Streamlit is used to build functional prototypes, which is a great way to show off machine learning capabilities both within the teams, as well as to stakeholders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RxwwKB9KMWm0OnIxTS6uXw.png" /><figcaption>The MrD AI architecture, a hybrid of both AWS and GCP.</figcaption></figure><p>The majority of MrD’s infrastructure runs on Amazon Web Services (AWS), where Amazon Athena and S3 buckets make up the data storage backbone. Similar to Takealot, RESTful APIs and Kafka is used. SageMaker plays a crucial part for model development and deployment, in particular SageMaker AI and Notebooks for model development. Weights and Biases is used to monitor model training.</p><p>MrD uses several GCP functions to complement its machine learning operations, effectively having a hybrid cloud architecture.</p><h3>The teams</h3><p>Takealot has four machine learning teams: <em>Discovery</em> (search and recommendations), <em>Merchant</em> (catalogue, pricing, promotions, retail and marketplace), <em>Fulfilment</em> (supply chain and logistics) and <em>MLOps</em> (similar to DevOps).</p><p>MrD is in its early growth phase as a business, and for that reason has a singular machine learning team. However, the team fulfils all the machine learning functions as the more specialised Takealot teams.</p><h3>The machine learning projects</h3><p>There is vast array of machine learning projects at any given time at Takealot and MrD. Head over to the <a href="https://medium.com/takealot-engineering">Takealot Engineering publication</a> to read about some of them.</p><h3>We are hiring!</h3><p>Does working with Silicon Valley-grade technologies and so much data you must use the cloud to process and train models, solving real-world machine learning problems that serve customers, sound exciting? <a href="https://grnh.se/5uyo1ptl1us">Join us</a>! Reach out to ai@takealot.com if you have questions, we are looking for both junior and senior engineers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87246795f9f1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/takealot-engineering/machine-learning-at-takealot-and-mrd-87246795f9f1">Machine learning at Takealot and MrD</a> was originally published in <a href="https://medium.com/takealot-engineering">Takealot Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Applying the TiDE deep learning model to demand forecasting]]></title>
            <link>https://medium.com/takealot-engineering/applying-the-tide-deep-learning-model-to-demand-forecasting-2360552ffc65?source=rss----55bbf27f0c2c---4</link>
            <guid isPermaLink="false">https://medium.com/p/2360552ffc65</guid>
            <dc:creator><![CDATA[Izak Marais]]></dc:creator>
            <pubDate>Tue, 15 Apr 2025 09:20:47 GMT</pubDate>
            <atom:updated>2025-04-15T09:20:46.959Z</atom:updated>
            <content:encoded><![CDATA[<h3>Context</h3><p>At Takealot warehouses, goods are handled by both permanent and temporary staff as part of our workforce management strategy. Temporary staff help to match the cyclical variation in demand: some days we need more people to ship more goods. To help determine how many temporary staff would be needed on a given day, our machine learning team needed to predict the daily aggregated unit sales for the next 7 days. Sales are grouped by warehouse, stock size and lead time. All in all we needed to predict 18 time series each day, each with 7 output values.</p><p>We had already done the bulk of the work: we had combined and transformed features in <a href="https://cloud.google.com/dataform/docs/overview">Dataform</a> and experimented with various models using <a href="https://www.tensorflow.org/tfx">TFX</a>. The best model was running in production: a <a href="https://kubernetes.io/">Kubernetes</a> cronjob would trigger a daily re-training and bulk inference.</p><p>Our best performing solution used a collection of <a href="https://en.wikipedia.org/wiki/Long_short-term_memory">LSTM</a>s. We grouped together time series with similar distributions into 6 groups and trained a model for each group. For example, one group is <em>lead-time</em> goods, small enough to <em>fit on a conveyor belt,</em> for <em>all warehouses</em>. We also trained a different model for each day in the forecast horizon. This approach achieved a satisfactory performance with a mean absolute percent error (MAPE) value of 9.9%.</p><h4>The TiDE model: Time-series Dense Encoder</h4><p>While our current solution was adequate, when we learned about a promising <a href="https://research.google/pubs/long-horizon-forecasting-with-tide-time-series-dense-encoder/">new time series prediction model out of Google Research</a>, we were keen to try it out. The claims from the abstract sounded promising:</p><blockquote>We propose a simple MLP based encoder decoder architecture for long-term time-series forecasting that can handle non-linear dependencies and dynamic covariates. Our method can achieve better results in several long term forecasting benchmarks while being 5–10x faster in terms of training and inference compared to the best transformer-based baselines</blockquote><p>We appreciated how the model architecture seemed relatively simple to recreate: we could reproduce it in a couple of days<em> — </em>and about 200 lines of code<em> — </em>by straightforwardly connecting readily available <a href="https://keras.io/">Keras</a> layers.</p><figure><img alt="A high level diagram of the TiDE model architecture" src="https://cdn-images-1.medium.com/max/937/1*8t_LLnjU3NPYRqkYtpHDfQ.png" /><figcaption>The model architecture can be built with standard Keras layers: <a href="https://keras.io/api/layers/core_layers/dense/">Dense</a>, <a href="https://keras.io/2.17/api/layers/regularization_layers/dropout/">Dropout</a>, <a href="https://keras.io/api/layers/normalization_layers/layer_normalization/">LayerNormalization</a>, <a href="https://keras.io/api/layers/merging_layers/add/">Add</a>, <a href="https://keras.io/api/layers/recurrent_layers/time_distributed/">TimeDistributed</a>.</figcaption></figure><p>There exists a <a href="https://github.com/google-research/google-research/blob/master/tide/README.md">reference implementation</a>. However, given the differences in data preparation and batching between the paper (which relies on fixed batch sizes equal to the number of time series in its CSV datasets) and our existing data TFX preparation (built on <a href="https://www.tensorflow.org/datasets">Tensorflow datasets</a>, where batch size is independent from the number time series) it was easier to integrate a recreated model than try to reuse the reference code as is.</p><h4>Results</h4><p>Initially, we struggled to get our TiDE model to converge during training. While we preprocess and normalise our <em>input</em> features, our LSTM models had been predicting the <em>target</em> variable at its true scale (un-normalised). This is sound in theory: the model’s final linear layer should be able to handle the scaling. It has an advantage over normalising the target variable: there is no need to remember to convert the normalised model output back the target scale.</p><p>We discovered that converting the target variable into the single digit range (from the 100k range) was necessary to get training to converge. With training working, we could experiment with the various hyper-parameters mentioned in the paper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*GK_QWQso03OE8ZM5-58Mjg.png" /><figcaption>Validation error across training runs with different hyper-parameters (e.g. hidden layer counts and embedding sizes), batch sizes and epochs trained.</figcaption></figure><p>From error analysis plots we could see the predictions where looking reasonable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qdXHzmAosiYq8g3Ot1Xjlw.png" /><figcaption>Example of a single day forecast from the evaluation set with a 365 day lookback window.</figcaption></figure><p>Unfortunately, they where not beating our LSTM baseline. Results below compare one of the LSTM models with a TiDE model trained on a similar slice of the data using different lookback windows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yHB_QPg_kuFgCkFD-XB78A.png" /><figcaption>The top row is the baseline implementation. None of the the TiDE models beat it.</figcaption></figure><p>One interesting observation is that the TiDE model performs better with larger context window sizes (the 30 day and 365 day MAPE values are better than the 7 day value for TiDE). On the other hand we had observed the opposite when selecting the best LSTM model (longer context window sizes decreased performance). This behaviour for TiDE echos the results from the paper, while it is a well known shortcoming of LSTM models that cannot optimally utilise long context windows.</p><p>So, what might be the cause of TiDE not out-performing our baseline?</p><p>We speculate that the main reason is the difference in datasets. The TiDE model focusses on <em>long-term</em> forecasts. The eval datasets from the TiDE paper have between 20k and 50k observations, each with forecast horizons between 96 and 720. We have 12k observations with a forecast horizon of 7.</p><p>Usually, long forecast horizons are problematic for LSTM models due to their autoregressive nature: errors can propagate and accumulate over time. We side-stepped this short-coming by training a different model for each day, thereby creating a strong baseline.</p><h4>Conclusion</h4><p>While it was disappointing that we could not improve our performance, we had several learnings:</p><ol><li>LSTMs are still powerful for short forecast horizons, with a limited lookback window.</li><li>The paper introduced some clever data pre-processing tricks that we are applying in other projects.</li><li>The importance of target value scaling.</li></ol><p>Lastly, it was fun and interesting to recreate a recent model from literature and apply it to our use case.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2360552ffc65" width="1" height="1" alt=""><hr><p><a href="https://medium.com/takealot-engineering/applying-the-tide-deep-learning-model-to-demand-forecasting-2360552ffc65">Applying the TiDE deep learning model to demand forecasting</a> was originally published in <a href="https://medium.com/takealot-engineering">Takealot Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using generative AI and vector search for product recommendations]]></title>
            <link>https://medium.com/takealot-engineering/using-generative-ai-and-vector-search-for-product-recommendations-baa1a04ca44f?source=rss----55bbf27f0c2c---4</link>
            <guid isPermaLink="false">https://medium.com/p/baa1a04ca44f</guid>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <dc:creator><![CDATA[Stefan Dominicus]]></dc:creator>
            <pubDate>Mon, 24 Feb 2025 10:08:25 GMT</pubDate>
            <atom:updated>2025-02-24T10:08:25.715Z</atom:updated>
            <content:encoded><![CDATA[<h4>Takealot’s first steps towards richer complementary product recommendations</h4><p>On-platform recommendations are one of the core product discovery mechanisms helping <a href="https://www.takealot.com/">takealot.com</a> customers “shop anything they can imagine”. In 2024 we embarked on a project that would use Generative AI as the basis for a brand new <em>“Complementary Products”</em> recommender.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BOmHRLXy7qIeW2hy" /></figure><h3>Background</h3><p>Our recommenders can be loosely grouped according to the kind of data each is trained on and what behaviour we expect their predictions to model.</p><ul><li><strong>Popularity-based</strong> models aggregate customer interactions such as clicks or purchases across various marginals such as product category. Think <em>“Best Selling in Sportswear”</em>.</li><li><strong>Browsing-based</strong> recommenders aim to promote items typically viewed in the same session by other customers. These capture comparative behaviour — that is, customers comparing similar items before making a purchase decision. For example, <em>“You Might Also Like”</em>.</li><li>In a similar vein, <strong>purchase-based</strong> recommenders model frequent co-occurrences in customer orders. In some cases, this can uncover complementary product pairs, but more often these models are hampered by data sparsity. Remember this for later…</li><li>Finally, <strong>personalised</strong> recommenders aim to model customer or cohort preferences in order to provide a more tailored experience for each individual. This is a complex topic and warrants a dedicated post.</li></ul><h3>The brief</h3><blockquote>Given a context product, recommend complementary products with the aim of increasing order sizes, and growing customer awareness of our vast catalogue.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OnaFIPLifZKfCAYB" /><figcaption>Context product: A Kookaburra cricket bat.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1LjY9jIxMu3NK5xV" /><figcaption>Complementary recommendations: A selection of cricket accessories (stumps, gloves, helmets, etc).</figcaption></figure><p>The catch? Our product data model does not capture any kind of compatibility relationship between products, and our purchase activity is too sparse (relative to the breadth of our catalogue) to inform a purely analytical model.</p><p>In fact, we already have a purchase-based model in place — <em>“Customers Also Bought” </em>— which picks out frequent co-occurrences in customer purchase history. Unfortunately the complementary product signal is still very weak, and as a result, this model struggles with recall. Practically, this means that only 60% of requests yield non-empty recommendations, which average only 4 items per response, despite many more good candidates existing in our catalogue.</p><h3>Thinking outside the box</h3><p>The stupendous pace of Generative AI innovation makes it easy to forget the state of things back in May of 2024. Let’s recap:</p><ul><li>Generative AI and Large Language Models (LLMs) are the hot topics.</li><li>Tooling and support for AI workflows are rapidly improving.</li><li><a href="https://en.wikipedia.org/wiki/Retrieval-augmented_generation">Retrieval Augmented Generation (RAG)</a> workflows promise to fix relevance and hallucinations.</li><li>Google Cloud Platform (GCP) is punting their own RAG tooling.</li></ul><p>The barrier to entry had never been lower. While searching for a way around the data sparsity discussed above, our brainstorming lead us to an idea that unlocked a new kind of recommender architecture:</p><blockquote>Use an LLM to generate complementary product types for each item.</blockquote><p>To our knowledge, this approach was novel. It sounded plausible, and we’d be able to test it quite quickly.</p><h3>Filling in the details</h3><p>In a typical RAG workflow, relevant information is first retrieved and then used as a grounding mechanism in the generative process. What we propose here is <em>not</em> a typical RAG workflow, although it makes use of many of the same building blocks.</p><ul><li>First we use an LLM to generate a number of “complementary product types” for each product in our catalogue.</li><li>Then we embed those “complementary product types” alongside “product embeddings” for all products.</li><li>Finally, we can perform a <a href="https://en.wikipedia.org/wiki/Nearest_neighbor_search">Nearest Neighbour Search</a> to find those products which are closest to each of the “complementary product types” generated in the first step.</li></ul><p>By keeping track of each of these steps, it is then possible to map each product to a set of complementary products, <strong>using the generated “complementary product types” as the link that was missing in our customer interaction data</strong>. The diagram below shows how these pieces fit together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6HyQ4moQ309ycTsqrpJnbg.png" /><figcaption>Simplified system architecture.</figcaption></figure><h4>Generation</h4><p>This can be simplified to just two decisions: which model to use, and how to prompt it.</p><ul><li>Given our extensive use of GCP, the choice of model was a no-brainer: use Google’s latest gemini-1.5-flash model, offered through Vertex AI.</li><li>As for prompting, if you’ve ever tried it yourself you’ll know <a href="https://en.wikipedia.org/wiki/Prompt_engineering">prompt engineering</a> to be a bit of a dark art. One can spend a lot of time and effort <em>and money</em> iteratively tuning prompts, often yielding diminishing returns. We settled on a fairly simple prompt which looks something like this:</li></ul><blockquote>You are an online shopping assistant, trying to help customers find products which complement the item they’re already looking at.</blockquote><blockquote>For example, when a customer is looking at a camping tent, do not suggest they look at other tents. Instead, suggest they look for camp chairs, sleeping bags, backpacks, torches, and other complementary products.</blockquote><blockquote>A customer is looking at the product described by the JSON document below, and they’d like to browse the store for other complementary products. <strong>Your task is to generate a list of complementary product types that the customer may want to look for later.</strong></blockquote><blockquote>&lt;&lt;Product JSON Document&gt;&gt;</blockquote><p>Once each product is passed through the LLM using this prompt, we are left with a mapping from products to “complementary product types”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*8iSRYXEQOzAEarSbR_0YrA.png" /><figcaption>Example: Generated complementary product types for the same cricket bat shown above.</figcaption></figure><h4>Embedding</h4><p>The architecture diagram above shows the same <a href="https://en.wikipedia.org/wiki/Word_embedding">text embedding</a> model being used twice:</p><ul><li>Once to generate embeddings of each “complementary product type”.</li><li>And again to generate embeddings of each product in our catalogue.</li></ul><p>As before, GCP’s Vertex AI had a state-of-the-art model ready for us to use: the latest text-embedding-004 model (version 005 was released shortly after this project went live). Once processed, we’re left with a high-dimensional embedding space containing both product vectors and “complementary product type” vectors.</p><h4>Retrieval</h4><p>A number of vector retrieval algorithms exist, but in real-world applications the choice is often informed by the <a href="https://en.wikipedia.org/wiki/Vector_database">vector store</a> being used. We chose to use <a href="https://cloud.google.com/bigquery/docs/vector-search-intro">BigQuery Vector Search</a> (more about this in the next section), which offers either brute force retrieval or Approximate Nearest Neighbour (ANN) retrieval using an <a href="https://cloud.google.com/bigquery/docs/vector-index#ivf-index">IVF index</a>. Performance at scale is always an important consideration at takealot.com, so we opted to use TopK ANN rather than brute force retrieval.</p><h4>Orchestration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*16tT6t_Mw5bIXFzZ-hZ_Ig.gif" /></figure><p>GCP had recently unveiled some useful tooling and <a href="https://cloud.google.com/blog/products/data-analytics/how-to-use-rag-in-bigquery-to-bolster-llms">tutorials</a> aimed at supporting RAG workflows — not <em>exactly</em> what we’re building, but close enough. Here’s what we ended up with:</p><ul><li>We store all our data in <strong>BigQuery</strong> (product details, generated “complementary product types”, and embedding vectors).</li><li>We use off-the-shelf text generation and text embedding models from <strong>Vertex AI</strong>. These models plug into <strong>BigQueryML</strong> functions via BigQuery’s Vertex AI connector.</li><li>We create a <strong>Vector Index</strong> on the embedding output table, and then use <strong>BigQuery’s Vector Search</strong> to retrieve nearby rows.</li><li>We rely on <strong>DataForm</strong> to hold it all together. This gives us scheduling and configuration management on top of BigQuery, without needing to mess around with custom Python scripts and other scheduled tasks.</li><li>Finally, to serve recommendations after each build, we stream the output table into a <strong>PostgreSQL</strong> database (running on <strong>CloudSQL</strong>).</li></ul><h3>Results</h3><p>We launched the new <em>“Complementary Products”</em> recommender in a head-to-head A/B test against the existing <em>“Customers Also Bought”</em> model. And the results were…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*kpBJrddLr2BJ7wW_xAyZiw.gif" /></figure><p><em>Terrible! </em>😢</p><p>Despite achieving similar <em>coverage</em> (the number of context products for which we return non-empty recommendations), and much better <em>fill</em> (the number of recommended items in each response), both of the conversion rate metrics we monitor tanked horribly:</p><ul><li>Click-through rate dropped by ~70%</li><li>Add-to-cart rate dropped by ~90%</li></ul><p>But why? At the risk of speculating, here are some thoughts:</p><ul><li>Our order data shows that customers don’t typically purchase complementary pairings. We expected this behaviour to change when exposed to complementary recommendations, but perhaps our customers don’t <em>want</em> complementary items — one might already have cricket pads, gloves, shoes, etc, and only need a new bat.</li><li>This model doesn’t account for product popularity, quality, price, etc. “cricket gloves” may be a perfectly good “complementary product type” for a cricket bat, but our catalogue likely contains many different cricket gloves— which one deserves to be recommended?</li></ul><h3>Conclusions</h3><p>In hindsight, this was always a <em>quantity over quality</em> approach, and it shows in these results. Nevertheless, there are still a few important takeaways:</p><ul><li>Anecdotes and spot-checks don’t always generalise — widespread testing and validation cannot be overlooked.</li><li><a href="https://en.wikipedia.org/wiki/Open-loop_controller">Open-loop</a> generation (to borrow from control theory) is unlikely to align with customer behaviour — feedback loops are critical.</li><li>DataForm and BigQueryML can enable powerful experimentation. In just a few weeks we were able to build our first fully cloud-native generative recommender system.</li></ul><p>We didn’t see the results we hoped for, but this was just an experiment after all. In our next iteration, we hope to incorporate customer engagement as a feedback loop and refine the LLM prompt to produce more robust complementary relationships.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*MYxxyUXQH7pnHzIVNor6hg.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=baa1a04ca44f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/takealot-engineering/using-generative-ai-and-vector-search-for-product-recommendations-baa1a04ca44f">Using generative AI and vector search for product recommendations</a> was originally published in <a href="https://medium.com/takealot-engineering">Takealot Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons learned from a failed machine learning project]]></title>
            <link>https://medium.com/takealot-engineering/lessons-learned-from-a-failed-machine-learning-project-d36eef239fb6?source=rss----55bbf27f0c2c---4</link>
            <guid isPermaLink="false">https://medium.com/p/d36eef239fb6</guid>
            <category><![CDATA[failure]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Axel Tidemann]]></dc:creator>
            <pubDate>Wed, 05 Feb 2025 09:32:31 GMT</pubDate>
            <atom:updated>2025-02-05T09:30:46.333Z</atom:updated>
            <content:encoded><![CDATA[<p>Failure is a funny thing, in particular when it happens to others. It is often lamented how many machine learning models never leave the laptop, however the topic of this article is a machine learning project that was running in production, but still ended up as a failed project. This is what we learned.</p><h3>Short summary of the project lifetime</h3><p>When I joined <a href="https://www.takealot.com/">Takealot</a> in July 2020, I took over the management of a project named <em>capacity delay</em>. The aim was to predict how much time a driver for <a href="https://www.mrd.com/">Mr D</a> would need to pick up an order.</p><p>The team trained deep learning and tree-based models, all with more or less the same performance, all beating the baseline: the mean absolute percent error (MAPE) went down from 36% to 30%. We settled on the deep learning model, due to the support of the <a href="https://www.tensorflow.org/">TensorFlow</a> ecosystem.</p><p>We did a gradual roll out, first to three hubs. On the first day of deployment, we had our <a href="https://en.wikipedia.org/wiki/Ariane_flight_V88">Ariane 5 moment</a>: we had trained the model to predict in seconds, however the consumers expected milliseconds. Oops. Luckily, a quick fix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/0*7iEueNH4UaryI48i" /></figure><p>In production, the performance was OK, but not stellar. It was continuously monitored, and re-trained on a monthly basis. Eventually, there was a decision to revert to the old mathematical equation that was in place earlier, while we would try to improve the model. During that time, it was discovered that our target variable was, in fact, the wrong one. Luckily, the other target variable worked fairly well — but not by intention. It was also realised that the old solution was more than good enough, some key people in the project had left, and there were bigger fish to fry. Unceremoniously, it was decided to archive the whole thing.</p><h3>Lesson 1: understand the machine learning potential</h3><p>When a baseline is already in place, the machine learning model must be a lot better for it to be worth developing and maintaining it. In this case, that was an optimistic assumption. Furthermore, the problem must be thoroughly examined to understand what is the potential for success when training a model. This means looking at the data: visually inspect the target variable and the baseline (if you have one). The figure below reveals a target value that is highly irregular. It would be hard for any machine learning model to learn to predict this pattern, and the next time I see something like this, I will say, “no, thank you”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*GxUpWueve4-W2SUofXt9tQ.png" /><figcaption>The baseline (red) is a mathematical formula based on <a href="https://en.wikipedia.org/wiki/Little%27s_law">Little’s law</a>; the prediction (green) is the deep learning model implemented in TensorFlow.</figcaption></figure><h3>Lesson 2: what to predict</h3><p>Even though we spent a lot of time making sure we got everything right, there still was a misunderstanding of what the target variable was. Or rather, <em>where to find it. </em>This sounds mind-boggling in retrospect, however it should be said that machine learning teams are not necessarily domain experts — this was certainly the case for this project. So the team was entirely dependent on the consuming team to understand what to predict, and where to find the target variable. A key issue was the lack of documentation. A lot of discussions and meetings were done to extract the required knowledge, but somehow we messed up, and had a target variable that was “adjacent” — not completely wrong, but still not the correct one. (Side note: this is how I learned to love <a href="https://cloud.google.com/dataform?hl=en">Dataform</a>, to break down SQL queries into small chunks.)</p><h3>Lesson 3: infrastructure readiness</h3><p>Takealot conforms to the shared microservices paradigm, but there was a lot to build in terms of machine learning infrastructure. The project moved from ad-hoc, bespoke solutions to solidifying on the TensorFlow ecosystem, which eventually morphed into our modern <a href="https://cloud.google.com/vertex-ai?hl=en">Vertex AI</a> infrastructure on the <a href="https://cloud.google.com/gcp?hl=en">Google Cloud Platform</a>. What is the lesson here? <em>This takes a lot of time.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*J2vNoEi1r2Wp9Ur8" /></figure><h3>Conclusion</h3><p>Failure <em>can</em> be fun. This was not a total waste of time, it was challenging and exciting to train the models, build the infrastructure, and grow the team. And we learned a lot about machine learning product management, building data and machine learning pipelines, how to write custom components for error analysis in <a href="https://www.tensorflow.org/tfx">TensorFlow Extended</a>, and asymmetric loss functions, just to name a few examples.</p><p>Hopefully, we won’t make the same mistakes again.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d36eef239fb6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/takealot-engineering/lessons-learned-from-a-failed-machine-learning-project-d36eef239fb6">Lessons learned from a failed machine learning project</a> was originally published in <a href="https://medium.com/takealot-engineering">Takealot Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hybrid product search at Takealot]]></title>
            <link>https://medium.com/takealot-engineering/hybrid-product-search-at-takealot-24c6e97a84eb?source=rss----55bbf27f0c2c---4</link>
            <guid isPermaLink="false">https://medium.com/p/24c6e97a84eb</guid>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[search]]></category>
            <dc:creator><![CDATA[Grant Zietsman]]></dc:creator>
            <pubDate>Tue, 19 Nov 2024 09:51:01 GMT</pubDate>
            <atom:updated>2024-11-19T09:51:01.817Z</atom:updated>
            <content:encoded><![CDATA[<p>In June 2024 Takealot launched hybrid search to customers by adding a semantic retrieval component to the existing lexical search engine. This marks a paradigm shift in how search is done at Takealot.</p><p>For example, consider the query <em>“something to keep my feet warm”</em>. The following results show relevant products: socks, slippers, etc. Yet, the query doesn’t specifically mention terms such as <em>“socks”</em> or <em>“slippers”. </em>This is because semantic search matches based on latent contextual similarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rRJTIM6zyI8BNzTkwDfk4Q.png" /><figcaption>hybrid product search results for the query “something to keep my feet warm”</figcaption></figure><p>Of course, a lexical search engine could still leverage synonyms and query expansion to handle such queries but it requires complex tuning and has high maintenance overheads.</p><h4>What is hybrid search?</h4><p>Hybrid search is the composition of lexical and semantic search which fulfil somewhat complementary retrieval objectives.</p><ul><li><strong>Lexical Search</strong> excels at exact matching (e.g. terms, ISBNs, etc.). It typically uses the <a href="https://en.wikipedia.org/wiki/Okapi_BM25">BM25</a> algorithm which performs <a href="https://en.wikipedia.org/wiki/Bag-of-words_model">bag-of-words</a> term matching with <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a> inspired relevance scores.</li><li><strong>Semantic Search</strong> excels at concept matching which is less sensitive to exact terms. It encodes both the query and results as dense embeddings which are semantic representations of the information. Vector similarity is used for relevance scores.</li></ul><p>Reciprocal Rank Fusion (RRF) or score based fusion are typically used to combine results.</p><h3>Prototyping</h3><p>The initial prototype was developed using <a href="https://openai.com/index/clip/">OpenAI CLIP</a> as the semantic embedding model. This model had compelling zero shot performance eliminating the need for fine tuning. Furthermore, the multimodal capability enabled the embedding of both product text and image content.</p><p>The goal was to represent products as a single embedding for scalability and performance. Note, techniques such as <a href="https://www.pinecone.io/learn/chunking-strategies">chunking</a> remain useful especially for searching larger bodies of text. In this case, product embeddings were represented as the weighted addition of text and image embeddings.</p><p>To preview this, the <a href="https://www.marqo.ai/">marqo</a> engine with <a href="https://streamlit.io/">streamlit</a> search interface was used.</p><h4>Multimodal Pitfalls</h4><p>During tuning it became apparent that multimodal <em>text-to-image</em> similarity was state-of-the-art while multimodal <em>text-to-text </em>similarity fell short on benchmarks. Given the enormous variety of products that Takealot sell, accurately representing textual content is crucial.</p><p>To compensate, prototypes were developed by concatenating <em>text </em>embeddings from state-of-the-art text embedding models with the multimodal <em>text-to-image </em>embeddings from <em>OpenAI CLIP</em>. Concatenation was chosen because embedding spaces of independent models are typically not semantically compatible. Although this resulted in improved quality, it was not sufficient to compensate for the performance and cost overheads of the increased embedding dimensions.</p><p>Jina AI responded to this issue by developing <a href="https://jina.ai/news/jina-clip-v1-a-truly-multimodal-embeddings-model-for-text-and-image"><em>jina-clip-v1</em></a><em>. </em>They managed improve multimodal <em>text-to-text </em>performance on <a href="https://huggingface.co/spaces/mteb/leaderboard">MTEB</a> from 0.162 to 0.429 (see table below). However, at the time, state-of-the-art text embedding models were achieving closer to 0.7 on MTEB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sZaEApWlhYYqawBo" /><figcaption><a href="https://jina.ai/news/jina-clip-v1-a-truly-multimodal-embeddings-model-for-text-and-image">table benchmarking jina-clip-v1 to openai-clip-vit-b16</a></figcaption></figure><p>Although marking a significant improvement, the performance gap remains. Therefore, Vertex text embeddings were selected.</p><h4>Search Console</h4><p>In order to iterate efficiently, a search console was developed that provides dynamic configuration of the entire search engine and decouples backends to enable high malleability. Additionally, this serves as tool to investigate and understand how the search engine responds to different queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c7Tm8XlMvBxs_AQqlZDCig.png" /></figure><h3>Serving and Architecture</h3><h4>Vertex AI</h4><p>The <a href="https://cloud.google.com/vertex-ai">Google Vertex AI platform</a> provides extensive tooling for machine learning. This includes model hosting and managed model endpoints. The managed model endpoints are billed on a <a href="https://cloud.google.com/vertex-ai/generative-ai/pricing#embedding-models">pay-per-token</a> basis while model hosting is billed based on the underlying resources. Pay-per-token is by far the most cost effective option available for embeddings, unless <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/models/tune-embeddings">fine tuning</a> is required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SNbvkBkCW7sMAikzmwtY4A.png" /><figcaption><a href="https://cloud.google.com/vertex-ai">Google Vertex AI</a></figcaption></figure><p>The <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings-api#model_versions"><em>multimodalembedding@001</em></a> model suffered from a similar <em>text-to-text</em> deficiency as observed with <em>OpenAI CLIP</em>. Additionally, the latency and cost was substantially higher than the <em>text-embedding-004 </em>counterpart<em>. </em>Therefore, <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings-api#model_versions"><em>text-embedding-004</em></a><em> </em>was selected (with 512 dimensions and 8-bit quantisation for performance reasons).</p><p>It is anticipated that <em>multimodalembedding@002 </em>(when/if released)<em> </em>will resolve the above mentioned deficiency.</p><h4>Dynamic Model Upgrades</h4><p>A core component of the implementation was the capability to seamlessly upgrade and support multiple embedding models. This is especially important for A/B testing.</p><h4>Caching</h4><p>Since customer search queries are often repeated and the same products are indexed frequently, both query and product embedding caches were incorporated.</p><h3>Results</h3><p>A successful A/B test led to promoting the upgrade. A ~6% increase in search listing click through rate was measured (top of the funnel).</p><h3>Conclusion</h3><p>This feature marks a paradigm shift and a milestone in our strategy to provide the ultimate (proudly South African 🇿🇦) online shopping experience for customers.</p><h3>What’s Next</h3><p>This has already paved the way for future features such as visual search, which is currently an internal function.</p><p>There is always room for improvement and some key focal areas for search are query understanding and generative AI capabilities. For example, LLMs can enrich both query intent extraction as well as product annotation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24c6e97a84eb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/takealot-engineering/hybrid-product-search-at-takealot-24c6e97a84eb">Hybrid product search at Takealot</a> was originally published in <a href="https://medium.com/takealot-engineering">Takealot Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Takealot customer service chatbot with generative AI]]></title>
            <link>https://medium.com/takealot-engineering/building-a-takealot-customer-service-chatbot-with-generative-ai-cc2af809bdd7?source=rss----55bbf27f0c2c---4</link>
            <guid isPermaLink="false">https://medium.com/p/cc2af809bdd7</guid>
            <category><![CDATA[chatbot-development]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <dc:creator><![CDATA[Axel Tidemann]]></dc:creator>
            <pubDate>Thu, 19 Sep 2024 12:16:59 GMT</pubDate>
            <atom:updated>2024-09-19T12:14:02.903Z</atom:updated>
            <content:encoded><![CDATA[<p>Co-authors: <a href="https://medium.com/u/9b01e9889701">Schroodinger</a> and <a href="https://medium.com/u/35425a82d12">Ceasor Mufamadi</a>.</p><p>Takealot has built its first chatbot using generative AI, enabling a software engineering team (i.e. <em>not</em> one of the machine learning teams, but still under machine learning guidance) to harness the power of large language models (LLMs).</p><p>The rapid growth of open-source LLMs (as well as those hosted by cloud providers), makes it easier than ever to develop generative AI applications. The advent of LLMs has changed the landscape dramatically when it comes to use cases which include unstructured data (e.g. text, images, audio and video). Previously, gathering data and training a model was a complex operation. Now, time can be spent on <em>testing</em> instead, and iterations on the final product can be done much faster.</p><h3>Where is my order? I want a refund!</h3><p>The Takealot chatbot answers customer questions based on existing help centre articles. This makes for the perfect use case for retrieval augmented generation (RAG). In essence, the knowledge base is split into chunks, and embeddings are gathered (using an LLM) for each of these chunks. These are stored in a vector database. A customer queries the chatbot, and the embedding of the query is matched with the chunk in the vector database. The full text of the chunk along with the query is then provided as input to <em>another</em> (typically bigger and more powerful) LLM, which provides the answer back to the customer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*qDvgeZS8cbIPS_85dPivLw.png" /><figcaption>A typical RAG architecture.</figcaption></figure><h3>Guardrails and hallucinations</h3><p>It is understood that hallucinations are, in fact, <a href="https://x.com/karpathy/status/1733299213503787018?lang=en">all that LLMs do</a>. These hallucinations vary from useful to nonsensical. Using RAG is a great first step to steer the hallucinations into useful territory. It was also necessary to create guardrails so that the chatbot would not respond to questions that were outside its help centre knowledge base. A lot of time was spent on creating these guardrails and getting the prompt design right. In particular, providing clear, concise and short prompts proved to be a winning strategy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*vBONHXGNNwfjpkfPwGNetA.png" /><figcaption>A sample interaction with the chatbot.</figcaption></figure><h3>The challenge of navigating a rapidly evolving landscape</h3><p>After a bit back and forth on software packages and open source models, the project solidified with <a href="https://www.langchain.com/">LangChain</a>, using <a href="https://pypi.org/project/chromadb/">ChromaDB</a> as the vector database, and all-MiniLM-L6-v2 to create embeddings for each chunk. <a href="https://gpt4all.io/">GPT4All</a> was used to harness the power of Mistral-7B-Instruct as the final output LLM, which was the prevailing open source model at the time. This model could be run on a developer’s laptop, although the response times were painfully slow (a few minutes).</p><p>We opted to self-host the Mistral model on a GPU on Vertex AI to serve customer traffic. The cost was roughly $400/month for a single instance. Just a few weeks before the go-live date, <a href="https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-flash">Gemini Flash</a> was announced, which dramatically altered the playing field. The cost dropped to a point where it became absolutely pointless to self-host. We instead opted for Gemini Flash, which was faster, cheaper and better in terms of responses given the same prompts as the Mistral model.</p><h3>Conclusion</h3><blockquote>The underlying goal, which was to enable software engineering teams to develop generative AI applications, was a success.</blockquote><p>This opens up for a lot of potential in the immediate future for the software engineering teams at Takealot. It also seems like the generative AI landscape (in particular the APIs) seem to stabilise on Vertex AI, making it easier to make long-term decisions on which technology to use.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc2af809bdd7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/takealot-engineering/building-a-takealot-customer-service-chatbot-with-generative-ai-cc2af809bdd7">Building a Takealot customer service chatbot with generative AI</a> was originally published in <a href="https://medium.com/takealot-engineering">Takealot Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>