<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kees C. Bakker on Medium]]></title>
        <description><![CDATA[Stories by Kees C. Bakker on Medium]]></description>
        <link>https://medium.com/@keestalkstech?source=rss-5e9a43654081------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nT1bknsOH_-tDs1lCd1Srw.png</url>
            <title>Stories by Kees C. Bakker on Medium</title>
            <link>https://medium.com/@keestalkstech?source=rss-5e9a43654081------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:48:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@keestalkstech/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How we switched 150 stores from Redis to Valkey, and nobody cared!]]></title>
            <link>https://medium.com/wehkamp-techblog/how-we-switched-150-stores-from-redis-to-valkey-and-nobody-cared-308b0a8a9fc7?source=rss-5e9a43654081------2</link>
            <guid isPermaLink="false">https://medium.com/p/308b0a8a9fc7</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[valkey]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[blog]]></category>
            <dc:creator><![CDATA[Kees C. Bakker]]></dc:creator>
            <pubDate>Mon, 22 Jun 2026 13:52:08 GMT</pubDate>
            <atom:updated>2026-07-11T09:09:57.808Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BSeRgA8td2tweeRm.jpg" /></figure><p>Ever since we started our move from the on-premises data center to the cloud, delivery teams haven’t had to worry about their infrastructure. We’ve built a platform that lets delivery teams focus on solving business problems. At Wehkamp, teams don’t have to worry about Kubernetes, network infrastructure, or writing Terraform code to provision data stores. They don’t even need to know about AWS! Team infrastructure is easily provisioned through our Slack bot. <em>Basically, the teams have been running serverless since 2013!</em></p><p>Having such a platform comes with a peculiar downside: when teams don’t <em>have to care</em> about infrastructure, they usually… don’t!</p><p><em>So what if we want to migrate our AWS ElastiCache Redis stores to Valkey?</em> In this article we’ll explain how we switched 150+ AWS ElastiCache stores from Redis to Valkey.</p><p><strong>A big shout-out to </strong><a href="https://www.linkedin.com/in/klaastalsma/"><strong>Klaas Talsma</strong></a><strong> and </strong><a href="https://www.linkedin.com/in/chris-v-4b9a62b/"><strong>Chris Vahl</strong></a><strong> for working together on this story. We started this migration at the beginning of 2026, so this article was long overdue.</strong></p><h3>Why?</h3><p>Back in 2024, <a href="https://redis.io/blog/redis-adopts-dual-source-available-licensing/">Redis switched to another license</a>, and that got developers around the world thinking about a fork. Valkey was born in no time! <a href="https://aws.amazon.com/blogs/opensource/why-aws-supports-valkey/">AWS was heavily involved from the start</a>. Honestly, the results were very impressive. And they didn’t stop there, as Valkey 8.1 was released with <a href="https://aws.amazon.com/blogs/database/year-one-of-valkey-open-source-innovations-and-elasticache-version-8-1-for-valkey/#:~:text=Innovations%20that%20matter">even more power</a>.</p><p>When we read that it was <a href="https://aws.amazon.com/blogs/database/get-started-with-amazon-elasticache-for-valkey/#:~:text=On%20ElastiCache%20for%20Valkey%20self%2Ddesigned%20(node%2Dbased)%20clusters%2C%20you%20can%20benefit%20from%20up%20to%2020%25%20lower%20cost%20compared%20to%20other%20engines.">20% cheaper to run Valkey</a> than Redis, the decision became a no-brainer. So we threw together some slides, gave a Tech Talk, and started adding Valkey to our platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VhFMZaiZO98NBAbT.jpg" /><figcaption>This was only the beginning, as these slides are from early 2025.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2StIYHKXKzNughu1.png" /><figcaption>To sum up: it is faster &amp;&amp; cheaper =&gt; let’s migrate.</figcaption></figure><h3>Step 1: Stop the bleeding</h3><p>Whenever we introduce new technology, we work according to a two-step principle:</p><ol><li>Stop the bleeding: make sure <em>newly</em> provisioned infrastructure uses the new technology by <em>default</em>.</li><li>Migration: present a migration path and let teams move at their own pace.</li></ol><p>Our delivery teams don’t have to write Terraform code (they can); we provide a Slack bot to guide developers to provisioning a data source. Here we swapped out the provisioning of Redis resources for that of Valkey. In the beginning we even kept the Redis icon, but over time that got swapped as well, as the name caused confusion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*hhKcyXLq8LnlnYWkAeZqlw.png" /><figcaption>The Provisioner provides an easy dialog in Slack for Valkey provisioning. In an earlier version you could select Redis or Valkey engines.</figcaption></figure><p>Our Provisioner provisions Valkey according to Tech Hub standards, including TLS, authentication, and multi-AZ. It generates Terraform code, commits to the IAC Git repository, creates a PR, and watches it roll out to production, all while keeping the user informed on Slack.</p><h3>Step 2: Migration</h3><p>Again, we made some slides:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wn87-Zq5KAR7Paz3.png" /><figcaption>We made TLS mandatory for our new Valkey setups, which requires application changes.</figcaption></figure><p>The usual suspects moved right away, but the bulk of the teams did not migrate 😱. So we talked to the teams to find out if there were any blockers.</p><ul><li>Some older services did not use AUTH or a TLS connection, and the teams did not want to refactor those applications to include both features.</li><li>Some teams still had their service on a <em>single fascia</em> environment, so they needed to move to a multi-fascia environment before they could benefit from the new setup.</li><li>Some teams had very busy backlogs, so they couldn’t prioritize it.</li></ul><h3>Step 3? Now what?</h3><p>What could we, as a platform team, do? If we wanted to reap the performance and financial benefits, we needed to make an in-place upgrade possible. Fortunately for us, Valkey is meant as a drop-in replacement.</p><p>However, we hit a technical hurdle: our latest infrastructure standards mandate encryption and TLS, but applying those specific enhancements to existing environments would have made an in-place engine upgrade impossible. Fortunately there was a pragmatic solution: we rewrote an older version of the Terraform module, so old environments could also benefit from the new engine.</p><p>By testing this extensively on clusters with at least two nodes, we confirmed we could swap the engine under the hood with no downtime, because with a multi-node cluster, there’s always a node left to serve traffic during the switch.</p><p>The result was a process so streamlined that teams could migrate to Valkey by updating just a few lines of code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5xBcZj6sksb9fmA3.png" /><figcaption>Only 4 lines need to be changed.</figcaption></figure><h3>A New Year’s Resolution</h3><p>The next step was to talk to <a href="https://www.linkedin.com/in/kmroumen/">Koen Roumen, Head of Technology</a>. We explained that there was no reason not to move, and we came up with a plan: <em>either</em> the teams move <em>before</em> a certain date, or <em>we would make the move for them.</em> Maybe it was the optimism that comes with starting a new year, or maybe the stars aligned, but we all agreed on a pretty tight timeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jgp1srV_hTa1I11-.png" /><figcaption>We wanted to move within the month.</figcaption></figure><p>Again, some teams woke up and moved, but most teams were like: <em>if you’re sure there is no impact, go for it!</em> And so we did. We ran the migration in batches. On the morning of the 27th, we migrated a batch of 47 stores for 6 delivery teams across 3 AWS development environments to Valkey. It went incredibly smoothly. Yes, it took AWS a while to provision the stores (no idea why), but by 9 o’clock we were done. Our delivery teams did not report any problems.</p><p>Production was scheduled for a week later. As dev went so smoothly, we dropped our requirement to send a backend engineer to the call, and we decided to start at 09:00 instead of 08:00. Again: one smooth transition.</p><h3>Step 4: Right sizing</h3><p>Now that the migration to Valkey was behind us, we shifted our focus from “making it work” to “making it right”. Because our in-place migration kept the exact same instance types that were previously running Redis, much of our fleet was likely over-provisioned. On a number of stores we <em>observed</em> lower resource utilization after the switch.</p><ul><li>Valkey runs leaner. Where we saw genuinely lower load after the swap, the engine itself is the likely driver.</li><li>Some stores were simply oversized under Redis. Safety margins had been chosen years ago and never revisited. That capacity was “ghost capacity” regardless of engine, the migration was just the moment we finally went looking for it.</li></ul><p>To do this well, we started a deep-dive analysis, leveraging LLMs and the AWS CLI to parse two weeks of CloudWatch metrics since the migration. Guided by our FinOps tooling, this data-driven approach identified consistently over-provisioned stores, which we then scaled down, keeping headroom for peaks. It’s a pragmatic approach to cloud spending: optimize the baseline first, so we only pay for the performance we actually use.</p><h3>So what did we learn?</h3><p>155 stores migrated, no production downtime, and zero complaints. We switched the engine, we’re seeing the savings, and, just as we intended, nobody even noticed. Great.</p><p>Our main lessons:</p><ol><li>Valkey really is a great replacement for Redis.</li><li>Our delivery teams are very busy, and that’s great! But that shouldn’t stop us, as a platform team, from moving forward. This is where the real benefit of Infrastructure as Code and AWS shows: we can move the whole fleet without asking every team to stop what they’re doing.</li><li>There is <a href="https://aws.amazon.com/blogs/database/announcing-valkey-9-1-for-amazon-elasticache/#:~:text=Improve%20price%2Dperformance%20at%20scale">even more power</a> ahead in Valkey 9.1!</li></ol><p><em>I work as a Pathfinder at the </em><a href="https://wehkampretailgroup.nl/"><em>Wehkamp Retail Group</em></a><em>, one of the biggest e-commerce companies of the Netherlands. This article is </em><a href="https://medium.com/wehkamp-techblog"><em>part of our Tech Blog</em></a><em>, check it out &amp; subscribe. Looking for a great job? </em><a href="https://wehkampretailgroup.nl/vacatures/"><em>Check our job offers</em></a><em> or </em><a href="https://www.linkedin.com/in/keescbakker/"><em>drop me a line on LinkedIn</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://keestalkstech.com/how-we-switched-150-stores-from-redis-to-valkey-and-nobody-cared/"><em>keestalkstech.com</em></a><em> on June 22, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=308b0a8a9fc7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wehkamp-techblog/how-we-switched-150-stores-from-redis-to-valkey-and-nobody-cared-308b0a8a9fc7">How we switched 150 stores from Redis to Valkey, and nobody cared!</a> was originally published in <a href="https://medium.com/wehkamp-techblog">wehkamp-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When the Planning Wall Doesn’t Scale]]></title>
            <link>https://medium.com/wehkamp-techblog/when-the-planning-wall-doesnt-scale-ba87356e63da?source=rss-5e9a43654081------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba87356e63da</guid>
            <category><![CDATA[jira]]></category>
            <category><![CDATA[blog]]></category>
            <category><![CDATA[roadmaps]]></category>
            <dc:creator><![CDATA[Kees C. Bakker]]></dc:creator>
            <pubDate>Mon, 16 Mar 2026 07:04:39 GMT</pubDate>
            <atom:updated>2026-07-09T10:47:15.408Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*IyTIngEftFLlwBkq.jpg" /></figure><p>At the Wehkamp Tech Hub, we support Wehkamp with technology in the broadest sense of the word: almost everything is technology. Larger initiatives, like the replacement of our order management system, require careful planning across teams; we have many initiatives like that.</p><p>When I walk into the office of my boss, Jon Higgins (CTO), I’m greeted by a huge planning wall. It feels a bit like one of those large charts the British Navy used in World War II to plan operations and track the current situation. While I love the clarity it provides, it also saddens my IT heart a little: <em>shouldn’t we be able to create something digital?</em> If it were up to me, I’d replace the entire wall with a full-touch LED panel.</p><p>Our former Head of Product, Sander Hoekman, also wanted to share more of the roadmap with the organization, so he asked me to look into generating a roadmap <em>report</em> that we could share more broadly.</p><h3>What did it used to look like?</h3><p>Céleste, one of our Product Owners, did some digging and found an early version of the planning wall from 2023:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FaNqZnrEjq9lta0z.jpeg" /><figcaption>The early version of the planning wall was only concerned with getting insights on the current quarter.</figcaption></figure><p>Later versions of the wall tried to plan further ahead. Unfortunately, I wasn’t able to find a more recent wall with stickies, so this is what we could come up with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_zvmmO_4m21_AYrV.jpg" /><figcaption>Our plans used to be on this wall. Big thanks to Renske for snapping this photo.</figcaption></figure><h3>Goals of the project</h3><p>I like solving multiple problems at once, so I wasn’t going to build just a report, but a <em>tool</em> that would give our organization insight into the roadmap. First, we needed to lay the groundwork by defining the requirements:</p><ol><li><strong>The data will always live within Jira<br></strong>The tool will be a read-only view on top of the current state in <a href="https://www.atlassian.com/software/jira/guides/advanced-roadmaps/overview#what-is-advanced-planning">Jira Advanced Planning</a> (with some caching for performance).</li><li><strong>Reports will be generated from the tool<br></strong>There was a need for a PDF report, but it usually ended up in a Google Slides presentation, often as a low-resolution screenshot. To address this, we&#39;ll generate full-HD slides that can be added as images.</li><li><strong>Give more attention to <em>non-functional</em> programs<br></strong>Technical programs should be included in the roadmap, as they impact teams and availability.</li><li><strong>Give more insight into the <em>complexity</em> of a milestone<br></strong>When multiple teams are working on the same topic, it is likely to be complex. Let’s show a list of teams that are working on a milestone. This will also give us insight into which team or teams might become a bottleneck because of their workload.</li><li><strong>Help tech leadership<br></strong>Which topics are late? Which topics have been replanned? Which topics are being investigated? The roadmap should give leadership useful input for their weekly meetings.</li><li><strong>Access<br></strong>Not everyone has access to Jira, so let’s give our stakeholders access to this tool so they know what is being worked on and what is coming up.</li></ol><h3>What’s the (Jira) Plan?</h3><p>Due to the nature of the topics we work on, we use many approaches at the <em>team</em> level. The replacement of our VMware cluster or a network component is a <em>project</em> that follows a waterfall approach with clearly defined phases. A <em>redesign of our branding</em> is typically a <em>multi-team effort</em> that uses agile (Scrum or Kanban). No matter the approach, all of these topics need to be weighed, planned, and discussed. That’s where our roadmap comes in.</p><p>When moving away from our on-premises Jira server to SaaS, we knew we wanted to invest in Jira Advanced Planning. We wanted to be able to give an overview of our topics, plan them, and visualize our roadmap. We also wanted to separate long-term planning from team planning (agile and on a per-sprint basis).</p><p>So we came up with the following structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Eq7kD_CXYE7SseU9.png" /><figcaption>So, we have two levels: the operational level and the team level. Note that teams are agile and have their own sprints, boards, and backlogs, as not all work ties into the central roadmap.</figcaption></figure><p>It took some time to set it up, but we had some useful overviews almost overnight. While this was a good improvement, we still lacked a few features:</p><ul><li>Which teams are actually working on which roadmap topics?</li><li>How can we give non-technical people access to our roadmap without them having access to Jira?</li><li>How can we be transparent about what we <em>replan</em> ? Plans are never static in the real world!</li><li>How can we export our charts to be part of our regular update presentations?</li><li>Personal: how can I build something that can compete with “the wall”?</li></ul><h3>What did we come up with?</h3><p>Without going into too many technical details, we used the following technology stack:</p><ul><li>We use <a href="https://istio.io/">Istio</a> + <a href="https://dexidp.io/">Dex</a> + <a href="https://oauth2-proxy.github.io/oauth2-proxy/">OAuth2 Proxy</a> + Google login to give our Wehkamp users access to our application.</li><li>A small Node.js container that fetches the data from Jira.</li><li>A fast <a href="https://en.wikipedia.org/wiki/Single-page_application">single page application</a> that allows users to interact with the data.</li><li>Some clever CSS to plot our milestones and progress bars in the right position on the calendar.</li><li>The excellent <a href="https://github.com/bubkoo/html-to-image">HTML to Image library</a> to generate the full-HD slides.</li></ul><p>For obvious reasons, I cannot share the full Wehkamp roadmap, but I can blur the topics and show a single team. Here you can see what the <em>customer service</em> team is working on:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FOezBMFUr2Mc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DOezBMFUr2Mc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FOezBMFUr2Mc%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ffb2bac2adca82dacebda5740be63f55/href">https://medium.com/media/ffb2bac2adca82dacebda5740be63f55/href</a></iframe><h3>More features</h3><p>A project like this is never really finished, so we added a few more features:</p><ul><li><strong>Placeholder items</strong><br>A roadmap is for planning, so it may contain work that has not been assigned to a milestone yet. We visualize this with a dotted line.</li><li><strong>Analysis visualization<br></strong>Analysis is work too, and work takes time. We show analysis items on the roadmap as rounded milestones with diagonal bars. We also provide a filter to hide these items, as they might not be relevant for everyone.</li><li><strong>Delayed and moved items<br></strong>If work is late, we may need to discuss what is going on. We visualize delayed work with a red bar. Milestones that were moved get a yellow indicator, so it is clear that the date changed.</li><li><strong>Navigation through time<br></strong>At the bottom, we added navigation controls so users can move back and forth to look at past or future items.</li></ul><h3>Acknowledgments</h3><p>Many thanks to <a href="https://www.linkedin.com/in/sanderhoekman/">Sander Hoekman</a>, <a href="https://www.linkedin.com/in/celeste-polstra/">Céleste Polstra</a>, <a href="https://www.linkedin.com/in/vincentwolff/">Vincent Wolff</a>, <a href="https://www.linkedin.com/in/bart-jansen-90055733/">Bart Jansen</a>, and <a href="https://www.linkedin.com/in/lisetteberghuis/">Lisette Berghuis</a> for collaborating on this tool and on implementing the roadmap across the organization.</p><p><em>I work as a Pathfinder at the </em><a href="https://wehkampretailgroup.nl/"><em>Wehkamp Retail Group</em></a><em>, one of the biggest e-commerce companies of the Netherlands. This article is </em><a href="https://medium.com/wehkamp-techblog"><em>part of our Tech Blog</em></a><em>, check it out &amp; subscribe. Looking for a great job? </em><a href="https://wehkampretailgroup.nl/vacatures/"><em>Check our job offers</em></a><em> or </em><a href="https://www.linkedin.com/in/keescbakker/"><em>drop me a line on LinkedIn</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://keestalkstech.com/when-the-planning-wall-doesnt-scale/"><em>keestalkstech.com</em></a><em> on March 16, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba87356e63da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wehkamp-techblog/when-the-planning-wall-doesnt-scale-ba87356e63da">When the Planning Wall Doesn’t Scale</a> was originally published in <a href="https://medium.com/wehkamp-techblog">wehkamp-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Istio and Argo CD helped us to debug platform migration issues for our App]]></title>
            <link>https://medium.com/wehkamp-techblog/how-istio-and-argo-cd-helped-us-to-debug-platform-migration-issues-for-our-app-e52c93cd5e15?source=rss-5e9a43654081------2</link>
            <guid isPermaLink="false">https://medium.com/p/e52c93cd5e15</guid>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[istio-service-mesh]]></category>
            <category><![CDATA[argo-cd]]></category>
            <dc:creator><![CDATA[Kees C. Bakker]]></dc:creator>
            <pubDate>Thu, 31 Oct 2024 14:43:50 GMT</pubDate>
            <atom:updated>2025-01-06T15:51:10.386Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QcuWn2wf5OXjcIM878tR-g.jpeg" /></figure><p>At Wehkamp, we’ve been very busy with the migration of our container platform. Our (old) Blaze platform runs on Mesos, Marathon, and NGINX, but we’re shifting to a new Atlas platform, powered by Kubernetes and Istio. This setup gives us more resilience, security and <a href="https://keestalkstech.com/2022/10/devops-days-eindhoven-talk-how-to-handle-the-immense-traffic-of-a-successful-influencer-campaign/">auto scaling</a>. A year ago, we successfully migrated our www-ingress to this new platform. This week, we set our sights on migrating the app-ingress for our Android and IOS backend services. Unfortunately, things didn’t go as planned…</p><p><em>A big shout-out to </em><a href="https://www.linkedin.com/in/chris-v-4b9a62b/"><em>Chris Vahl</em></a><em> for working together on this story.</em></p><h3>Key differences: routing and tokens</h3><p>Atlas works with an Istio service mesh. <a href="https://istio.io/latest/docs/overview/what-is-istio/">Istio adds traffic management, security, and observability</a> by deploying sidecar proxies alongside our services to intercept and control traffic. To make adoption easier, we’ve added new traffic classes to our internal helm chart. ArgoCD will apply this and generate the proper ingress rules by convention (routes + constraints). Services need to be more explicit about <em>what</em> they expose and <em>where</em>, which helps to reduce the attack surface.</p><p>We’ve also enhanced JWT token security with stricter validation and encryption. One of our biggest challenges during the migration was dealing with old JWT tokens from our Blaze platform. Since Blaze and Atlas tokens aren’t backwards compatible, we had to exchange them during the migration. Because old versions of our app might still be in use, we needed to handle this dynamically. So we’ve built a Cloudflare worker to assist with the token exchange, ensuring a smooth transition for all Wehkamp users.</p><h3>Problems ahead!</h3><p>During the switch, we ran into problems almost immediately after switching our endpoints to the new platform, which forced us to revert to maintain stability. Unfortunately, finding the root cause wasn’t straightforward. There could have been subtle configuration mismatches between our development and production environments, like differences in Cloudflare settings (e.g., WAF or Workers) or Kubernetes configurations (Istio routes or constraints) that didn’t fully align.</p><p>We have two main environments: dev and production. The dev environment is used for Q/A, but no matter how much effort we put in… dev and prod are never exactly the same. To debug effectively, we needed to connect a new subdomain for testing without disrupting production…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*MdFatAuCaFsc4ebU.png" /><figcaption>Via <a href="https://x.com/thejewberwocky/status/793830760495013892">Joshua Zimmerman</a>, a slide by <a href="https://x.com/stahnma">@stahnma</a>.</figcaption></figure><h3>Leveraging Istio and Argo CD to generate routes</h3><p>So what are our options?</p><ul><li>Reconfigure our app ingress to point to a different domain. However, due to the nature of our charts, this would require a redeploy of over 50 services in production.</li><li>Add the extra domain and create additional virtual services for every app, but modifying the charts would still trigger a redeploy, which we want to avoid.</li><li>🤔 So… <em>what if we add a separate Argo CD ApplicationSet that only contains virtual services</em>? This would let us avoid a redeploy altogether while handling the routing changes we needed.</li></ul><h3>Script it</h3><p>Let’s create a script that:</p><ol><li>Uses the Argo CD cli to query all production applications (e.g. prod- convention).</li><li>Lists all manifests that belong on each app and find the VirtualService that ends in -app (convention).</li><li>Modifies the manifest, by dropping the labels, changing the name and replacing the hosts.</li><li>Stores the manifests in a single file with the --- YAML file separator</li><li>And we can write an Argo CD ApplicationSet that uses the the YAML file.</li></ol><h4>Bash to the rescue!</h4><p>Here’s what the script would look like (assuming you’re already logged into Argo CD):</p><pre>#!/bin/bash<br>set -e<br><br># Output file for all manifests<br>output_file=&quot;combined_virtualservices.yaml&quot;<br><br># Clear the output file if it already exists<br>&gt; &quot;$output_file&quot;<br><br># Get a list of all Argo CD apps<br>apps=$(argocd app list --output name)<br>for app in $apps; do<br>  # Skip any apps that do not contain the word &#39;prod-&#39;<br>  if [[ $app != *prod-* ]]; then<br>    continue<br>  fi<br><br>  echo &quot;Processing app: $app&quot;<br><br>  # Get the VirtualService manifests for this app<br>  manifest=$(argocd app manifests &quot;$app&quot; | yq e &#39;select(.kind == &quot;VirtualService&quot; and .metadata.name == &quot;*-app&quot;) | .&#39;)<br><br>  # No manifest, go to the next app<br>  if [[ -z &quot;$manifest&quot; ]]; then<br>    continue<br>  fi<br><br>  # Modify the manifest: <br>  # delete metadata.labels, <br>  # replace spec.hosts, <br>  # and update name<br>  modified_manifest=$(echo &quot;$manifest&quot; | \<br>    yq e &#39;del(.metadata.labels) | .spec.hosts = [&quot;new-app-ingress.domain.tld&quot;] | .metadata.name += &quot;-new-ingress&quot;&#39; -)<br><br>  # Append the modified manifest to the output file<br>  echo &quot;---&quot; &gt;&gt; &quot;$output_file&quot;<br>  echo &quot;$modified_manifest&quot; &gt;&gt; &quot;$output_file&quot;<br><br>  echo &quot;  Modified and appended manifest for app: $app&quot;<br>done</pre><p>The resulting YAML file can be quite large and should be stored in folder manifests/app2-vs/prod of the repository. These services will handle the routing on the new ingress URL, which allows us to debug and analyze discrepancies without interfering with production traffic.</p><h4>Referencing the Virtual Services in Argo CD</h4><p>Once we had the modified virtual service definitions, we needded an Argo CD ApplicationSet to deploy those to production. Here’s a snippet of how we defined it:</p><pre>---<br>apiVersion: argoproj.io/v1alpha1<br>kind: ApplicationSet<br>metadata:<br>  name: app2-vs<br>  namespace: argo-cd<br>spec:<br>   generators:<br>     - matrix:<br>         generators:<br>           - list:<br>               elements:<br>                 - envType: prod<br>           - clusters:<br>               selector:<br>                 matchLabels:<br>                   type: workload<br>                 matchExpressions:<br>                   - {key: environment, operator: In, values: [&quot;{{envType}}&quot;]}<br>   template:<br>     metadata:<br>       name: &quot;{{name}}-app2-vs&quot;<br>       namespace: argo-cd<br>     spec:<br>       destination:<br>         server: &quot;{{server}}&quot;<br>         namespace: app2-vs<br>       project: default<br>       syncPolicy:<br>         automated:<br>           prune: true<br>           selfHeal: true<br>         syncOptions:<br>           - CreateNamespace=true<br>           - PruneLast=true<br>           - ApplyOutOfSyncOnly=true<br>       sources:<br>       - repoURL: https://github.com/our-org/our-workload-services.git<br>         path: manifests/app2-vs/{{metadata.labels.environment}}<br>         targetRevision: HEAD</pre><p>Once this config was picked up by Istio, all of our productions services were available on the new domain. And our debugging began. In a matter of hours all of the issues were identified and solved by our platform and application teams.</p><h3>Lessons learned</h3><p>What did we learn from this?</p><ul><li>Environments are never the same, so there is value if we can test our production changes on a different URL before swithing over our backend.</li><li>Describing Kubernetes / Istio resources using Argo CD is very powerful and makes it easy to extract information from a running cluster. Istio makes it simple to replicate a secondary routing strategy.</li><li>YQ makes it easy to extract the right manifest and edit it.</li><li>GitOps for the win! 💪</li></ul><p>The combination of GitOps, Argo CD, Istio, and Kubernetes gives us a flexible and secure way to debug production issues with a secondary domain without any disruptions to the production workloads.</p><p><em>I work as a Pathfinder at </em><a href="https://wehkamp.nl/"><em>Wehkamp.nl</em></a><em>, one of the biggest e-commerce companies of the Netherlands. This article is </em><a href="https://medium.com/wehkamp-techblog"><em>part of our Tech Blog</em></a><em>, check it out &amp; subscribe. Looking for a great job? </em><a href="https://wehkampretailgroup.nl/vacatures/?favourites=%5B%5D&amp;filter_favourites=false&amp;search=&amp;smartrecruiters_department_label%5B%5D=Tech"><em>Check our job offers</em></a><em> or </em><a href="https://www.linkedin.com/in/keescbakker/"><em>drop me a line on LinkedIn</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://keestalkstech.com/how-istio-and-argo-cd-helped-us-to-debug-ap-migration-issues/"><em>https://keestalkstech.com</em></a><em> on Oktober 31, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e52c93cd5e15" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wehkamp-techblog/how-istio-and-argo-cd-helped-us-to-debug-platform-migration-issues-for-our-app-e52c93cd5e15">How Istio and Argo CD helped us to debug platform migration issues for our App</a> was originally published in <a href="https://medium.com/wehkamp-techblog">wehkamp-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wehkamp TechHub Journey towards cost optimization on AWS]]></title>
            <link>https://medium.com/wehkamp-techblog/wehkamp-techhub-journey-towards-cost-optimization-on-aws-17702bb0ea92?source=rss-5e9a43654081------2</link>
            <guid isPermaLink="false">https://medium.com/p/17702bb0ea92</guid>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[talks]]></category>
            <dc:creator><![CDATA[Kees C. Bakker]]></dc:creator>
            <pubDate>Wed, 12 Jun 2024 12:37:53 GMT</pubDate>
            <atom:updated>2026-07-09T13:51:57.166Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*elcmD9zSEkwrCYre.png" /></figure><p>Embedding FinOps into our <a href="https://www.linkedin.com/showcase/wehkamp-tech-hub/">Wehkamp Retail Group</a> development culture has led to substantial cost savings and more cost awareness. While we keep running a scalable and stable platform. All to give our customers a pleasant and smooth journey when shopping in our webshops 🛒 .</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXoG41FR6IhE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXoG41FR6IhE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXoG41FR6IhE%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/60d5fd315c341ae5f07a3c974e2418fa/href">https://medium.com/media/60d5fd315c341ae5f07a3c974e2418fa/href</a></iframe><p><em>I work as a Pathfinder at the </em><a href="https://wehkampretailgroup.nl/"><em>Wehkamp Retail Group</em></a><em>, one of the biggest e-commerce companies of the Netherlands. This article is </em><a href="https://medium.com/wehkamp-techblog"><em>part of our Tech Blog</em></a><em>, check it out &amp; subscribe. Looking for a great job? </em><a href="https://wehkampretailgroup.nl/vacatures/"><em>Check our job offers</em></a><em> or </em><a href="https://www.linkedin.com/in/keescbakker/"><em>drop me a line on LinkedIn</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://keestalkstech.com/embedding-finops-into-our-wehkamp-retail-group/"><em>keestalkstech.com</em></a><em> on June 12, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17702bb0ea92" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wehkamp-techblog/wehkamp-techhub-journey-towards-cost-optimization-on-aws-17702bb0ea92">Wehkamp TechHub Journey towards cost optimization on AWS</a> was originally published in <a href="https://medium.com/wehkamp-techblog">wehkamp-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grafana, I want slugs back for my Hubot! ]]></title>
            <link>https://medium.com/wehkamp-techblog/grafana-i-want-slugs-back-for-my-hubot-51f579fab2f8?source=rss-5e9a43654081------2</link>
            <guid isPermaLink="false">https://medium.com/p/51f579fab2f8</guid>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[chatops]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[hubot]]></category>
            <category><![CDATA[grafana]]></category>
            <dc:creator><![CDATA[Kees C. Bakker]]></dc:creator>
            <pubDate>Sat, 05 Nov 2022 02:49:22 GMT</pubDate>
            <atom:updated>2024-05-24T08:36:33.152Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ewupyjz5aKVLtv3JGlIMJw.jpeg" /></figure><p>At Wehkamp, we use the <a href="https://github.com/stephenyeargin/hubot-grafana">Hubot Grafana</a> project to make our Grafana dashboard available in Slack. It mainly helps our standby team to make quick assessments on what’s going on with our website. Last year, we saw we could not upgrade the package from 2.5.2 to 3.0.1, <a href="https://github.com/stephenyeargin/hubot-grafana/issues/140">because <em>slugs</em> could not be used anymore</a>. <em>But we use slugs and… quite heavily!</em> So we did not upgrade our Grafana v7… <a href="https://github.com/grafana/grafana/issues/54535">until we got hit by the input field bug</a>. We decided to see what we could do to bring our slug feature back to our ChatOps bot!</p><p>A big shout-out to <a href="https://www.linkedin.com/in/duvekot/?originalSubdomain=nl">Kees Duvekot</a>, <a href="https://www.linkedin.com/in/niekrosink/">Niek Rosink</a> and <a href="https://www.linkedin.com/in/chris-v-4b9a62b/">Chris Vahl</a> for the collaboration on the Grafana v9 upgrade.</p><p><em>At Wehkamp </em><a href="https://keestalkstech.com/2018/08/jump-starting-slack-bot-projects-bot-zero/"><em>we use bot-zero</em></a><em>, a Hubot project that has its CoffeeScript removed in favor of TypeScript. Hence, the code in this blog is written in TypeScript. A “normal” Hubot understands JavaScript, so it should not be hard to convert the code to JavaScript and add it to your Hubot instance.</em></p><p><em>The </em><a href="https://web.archive.org/web/20230323195208/https://keestalkstech.com/2022/11/grafana-i-want-slugs-back-for-my-hubot-%F0%9F%98%AD/"><em>initial article was written in 2022</em></a><em> and things have evolved a bit. I’ve </em><a href="https://github.com/stephenyeargin/hubot-grafana/pull/180"><em>added some of the code of the earlier article to the hubot-grafana package</em></a><em>, so the article now uses the code provided by the package.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*650Dy4DSdilMON9c.png" /><figcaption>When I ask “500” to our ChatOps bot, it returns a nice overview with HTTP 500 errors for our <a href="https://www.wehkamp.nl/">wehkamp.nl</a> website. Our standby teams rely heavily on the feature to query dashboards from Slack.</figcaption></figure><h3>No slug, but we have a URL</h3><p><a href="https://grafana.com/docs/grafana/latest/developers/http_api/folder_dashboard_search/">Grafana provides a nice search API</a> that can search for dashboards. Let’s see what it returns:</p><pre>{<br>  id: 904,<br>  uid: &#39;ICRkSGv4k&#39;,<br>  title: &#39;[Team tooling] consumer-gateway - Nginx metrics (nl.wehkamp.prod)&#39;,<br>  uri: &#39;db/team-tooling-consumer-gateway-nginx-metrics-nl-wehkamp-prod&#39;,<br>  url: &#39;/d/ICRkSGv4k/team-tooling-consumer-gateway-nginx-metrics-nl-wehkamp-prod&#39;,<br>  slug: &#39;&#39;,<br>  type: &#39;dash-db&#39;,<br>  tags: [ &#39;nl.wehkamp&#39;, &#39;prod&#39;, &#39;tooling&#39; ],<br>  isStarred: false,<br>  sortMeta: 0<br>}</pre><p>I see two fields that are <em>very</em> similar to my old <em>slug</em> field: uri and url. When you read the docs, you&#39;ll see that the uri field is deprecated since Grafana v5.0. So we might get somewhere by parsing the url field.</p><p><em>You might say: why not switch to </em>UID<em>, as Grafana advices you to do? Well... our dashboard generator setup contains a bug which generates a new UID once in a while, but the name and URI are fairly constant. Also: it is nice to have a human readable string.</em></p><h3>Make Hubot understand</h3><p>Let’s <em>alter the message </em>before it reaches the Hubot Grafana package. We can do this by using the <a href="https://github.com/hubotio/hubot/blob/main/docs/scripting.md#listener-middleware-examples">Hubot Receive Middleware API</a>; this API makes it easy to alter the text of a message (we’ve used the same technique to <a href="https://github.com/KeesCBakker/hubot-command-mapper/blob/master/src/middleware/removeMarkdownFromIncommingMessages.ts">remove markdown from incoming Slack messages</a>). <em>Note: the Hubot middleware was rewritten in modern versions of Hubot. My code uses the newer async middleware.</em></p><p>Let’s match messages that are for Grafana dashboards, extract the old slug and see if we can come up with a matching UID. If we can find such an ID, we use it to alter the message, otherwise we let Hubot Grafana have the original message.</p><p>Here is the code:</p><pre>import { Bot } from &quot;hubot-grafana/src/Bot&quot;<br><br>function parseGrafanaSlugs(robot: Hubot.Robot) {<br>  if (!robot) throw &quot;Argument &#39;robot&#39; is empty.&quot;<br><br>  const bot = new Bot(robot)<br>  const dashboardRequestRegex = /(grafana|graph|graf) (dash|dashboard|db) ([^ :]+)/i<br><br>  robot.receiveMiddleware(async context =&gt; {<br>    const text = context.response.message.text<br><br>    const match = dashboardRequestRegex.exec(text)<br>    if (match) {<br>      let dashboardId = match[3]<br>      let uid = null<br><br>      try {<br>        uid = await bot.createService(context.response).getUidBySlug(dashboardId)<br>      } catch (ex) {<br>        robot.logger.error(&quot;Error finding dashboard&quot;, ex)<br>      }<br><br>      if (uid) {<br>        const newText = text.replace(dashboardRequestRegex, &quot;graf db &quot; + uid)<br>        robot.logger.info(`Changing &quot;${text}&quot; to &quot;${newText}&quot;.`)<br>        context.response.message.text = newText<br>      }<br>    }<br><br>    return true<br>  })<br>}<br><br>module.exports = (robot: Hubot.Robot) =&gt; parseGrafanaSlugs(robot)</pre><p>When we add this middleware, our Hubot logs:</p><pre>INFO Changing &quot;jarvis graf db team-tooling-consumer-gateway-nginx-metrics-nl-wehkamp-prod:1 now-6h now&quot; to &quot;jarvis graf db ICRkSGv4k:1 now-6h now&quot;.</pre><p>🎉 Hurray! 🎉</p><h4>You might want to load your middleware a bit late(r)</h4><p>Our code uses the <a href="https://github.com/KeesCBakker/hubot-command-mapper#alias">Hubot Command Mapper Alias</a> feature to turn complex Grafana commands into simple ones, like @jarvis 500 -- which will return a dashboard on all the HTTP 500 responses of <a href="https://www.wehkamp.nl/">wehkamp.nl</a>. Be careful with the order, we needed to add our aliases to _alias.ts and our Grafana script to _grafana-slugs.ts. This makes sure both the alias and the Grafana middleware is executed before the reset.</p><h3>Final thoughts</h3><p>Love Hubot. Love Grafana v9 and all that comes with it. I think we’ve shown that with a bit of tinkering, you can make the software work for you.</p><p><em>I work as a Pathfinder at </em><a href="https://wehkamp.nl/?source=post_page---------------------------"><em>Wehkamp.nl</em></a><em>, one of the biggest e-commerce companies of the Netherlands. This article is </em><a href="https://medium.com/wehkamp-techblog?source=post_page---------------------------"><em>part of our Tech Blog</em></a><em>, check it out &amp; subscribe. Looking for a great job? </em><a href="https://werkenbij.wehkamp.nl/vacatures/?source=post_page---------------------------"><em>Check our job offers</em></a><em> or </em><a href="https://www.linkedin.com/in/keescbakker/?source=post_page---------------------------"><em>drop me a line on LinkedIn</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://keestalkstech.com/2022/11/grafana-i-want-slugs-back-for-my-hubot-%f0%9f%98%ad/"><em>https://keestalkstech.com</em></a><em> on November 5, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51f579fab2f8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wehkamp-techblog/grafana-i-want-slugs-back-for-my-hubot-51f579fab2f8">Grafana, I want slugs back for my Hubot! 😭</a> was originally published in <a href="https://medium.com/wehkamp-techblog">wehkamp-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DevOps Days Eindhoven Talk: How To Handle The Immense Traffic of a Successful Influencer Campaign]]></title>
            <link>https://medium.com/wehkamp-techblog/devops-days-eindhoven-talk-how-to-handle-the-immense-traffic-of-a-successful-influencer-campaign-8ee726dbf01b?source=rss-5e9a43654081------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ee726dbf01b</guid>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[karpenter]]></category>
            <category><![CDATA[talks]]></category>
            <category><![CDATA[keda]]></category>
            <dc:creator><![CDATA[Kees C. Bakker]]></dc:creator>
            <pubDate>Sun, 18 Sep 2022 16:56:55 GMT</pubDate>
            <atom:updated>2026-07-09T14:20:55.400Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*aZ9F2SNbj8a2hQ9G.jpg" /></figure><p><a href="https://www.linkedin.com/in/annekekeller/">Anneke Keller</a> (CTO), <a href="https://www.linkedin.com/in/kmroumen/">Koen Roumen</a> (Head of Platfom) and me did a talk on <a href="https://devopsdays.org/events/2022-eindhoven/welcome/">DevOpsDays Eindhoven 2022</a>:</p><p><em>A true story on how we learned how to handle the immense web traffic caused by a successful influencer campaign. How we built a platform with </em><a href="https://karpenter.sh/"><em>Karpenter</em></a><em> and </em><a href="https://keda.sh/"><em>Keda</em></a><em> for rapid and massive auto-scaling and the effects this had on our platform.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FRErltyToxz0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DRErltyToxz0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FRErltyToxz0%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/17eaf29a728bf79a228bf467fb9e4636/href">https://medium.com/media/17eaf29a728bf79a228bf467fb9e4636/href</a></iframe><p><em>I work as a Pathfinder at the </em><a href="https://wehkampretailgroup.nl/"><em>Wehkamp Retail Group</em></a><em>, one of the biggest e-commerce companies of the Netherlands. This article is </em><a href="https://medium.com/wehkamp-techblog"><em>part of our Tech Blog</em></a><em>, check it out &amp; subscribe. Looking for a great job? </em><a href="https://wehkampretailgroup.nl/vacatures/"><em>Check our job offers</em></a><em> or </em><a href="https://www.linkedin.com/in/keescbakker/"><em>drop me a line on LinkedIn</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://keestalkstech.com/devops-days-eindhoven-talk-how-to-handle-the-immense-traffic-of-a-successful-influencer-campaign/"><em>keestalkstech.com</em></a><em> on October 20, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ee726dbf01b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wehkamp-techblog/devops-days-eindhoven-talk-how-to-handle-the-immense-traffic-of-a-successful-influencer-campaign-8ee726dbf01b">DevOps Days Eindhoven Talk: How To Handle The Immense Traffic of a Successful Influencer Campaign</a> was originally published in <a href="https://medium.com/wehkamp-techblog">wehkamp-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t use Slack Incoming WebHooks; App Creation is Dead Simple!]]></title>
            <link>https://medium.com/wehkamp-techblog/dont-use-slack-incoming-webhooks-app-creation-is-dead-simple-af9ea8ff41da?source=rss-5e9a43654081------2</link>
            <guid isPermaLink="false">https://medium.com/p/af9ea8ff41da</guid>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[blog]]></category>
            <category><![CDATA[chatops]]></category>
            <dc:creator><![CDATA[Kees C. Bakker]]></dc:creator>
            <pubDate>Mon, 05 Sep 2022 17:32:09 GMT</pubDate>
            <atom:updated>2025-01-06T15:55:25.173Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wME53XV481EEtGEpFVKLsQ.jpeg" /></figure><p>At Wehkamp we ❤️ Slack! Seriously, in order to improve our efficiency, we’ve connected many of our applications, alerts and dashboards to Slack channels. But, as with all things, there is a <em>right</em> way and a <em>wrong</em> way of integrating a webhook at enterprise level.</p><p>As a Slack admin I see the old integration being requested every now and then, so let’s see how easy it is to move away from Slack Incoming WebHooks. This guide was created with the help of <a href="https://www.linkedin.com/in/harry-kroes-339ba015/">Harry Kroes</a>, one of our cloud engineers, as we needed to create a webhook to connect alerts from Nifi to Slack.</p><h3>Why is it a problem?</h3><p>Let me be clear: the <a href="https://wehkampgroup.slack.com/apps/A0F7XDUAZ-incoming-webhooks?tab=more%5Finfo">Slack Incoming WebHooks</a> integration will work, but <em>is</em> <strong>legacy</strong> and you should migrate as soon as possible.</p><blockquote>Please note, this is a legacy custom integration — an outdated way for teams to integrate with Slack. These integrations lack newer features and they will be deprecated and possibly removed in the future. <strong>We do not recommend their use.</strong></blockquote><blockquote><em>Source: </em><a href="https://wehkampgroup.slack.com/apps/A0F7XDUAZ-incoming-webhooks?tab=more%5Finfo"><em>Slack Incoming WebHooks</em></a></blockquote><p>At Wehkamp, that’s not even the main problem! Whenever the <em>creator</em> of a legacy webhook leaves the organization, the webhook is disabled. The <strong>code</strong> of the reactivated webhook changes; so vital integrations break.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/0*ImfLHpj9n0GkL9Ly.png" /><figcaption>Nooooo: some configurations of Incoming WebHooks have been deactivated.</figcaption></figure><h3>Fast Guide To Create An App</h3><p>It is super simple to create an application, especially now that we have <a href="https://api.slack.com/reference/manifests">Slack App Manifests</a>. Follow this wizard and you’ll have an <strong>app in 2 minutes</strong>:</p><p>1. Goto <a href="https://api.slack.com/apps?new%5Fapp=1">api.slack.com/apps?new_app=1</a></p><p>2. Choose <em>From An App Manifest</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/0*TqEh2uvh1IsCUmPZ.png" /><figcaption>Slack wizard for app creation.</figcaption></figure><p>3. Select your workspace and click <em>Next</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*6TM0lAKVW9XjqAG6.png" /><figcaption>Slack wizard for app creation.</figcaption></figure><p>4. Add this manifest, change the contents to your liking and click the <em>Next</em> button.</p><pre>display_information:<br>  name: My Alert<br>  description: Alerts from the My Monitoring System. Used for My App monitoring.<br>  background_color: &quot;#8B0000&quot;<br>features:<br>  bot_user:<br>    display_name: My Alert<br>    always_online: true<br>oauth_config:<br>  scopes:<br>    bot:<br>      - incoming-webhook<br>settings:<br>  org_deploy_enabled: false<br>  socket_mode_enabled: false<br>  token_rotation_enabled: false</pre><p>5. Review your app and hit the <em>Create</em> button. 🎉 <em>Congratz, your app is now created</em> 🎉</p><p>6. Next, we need to install the app. Navigate to <em>Settings &gt; Basic Information</em> and click the <em>Install your app</em> section and hit the <em>Install to Workspace</em> button:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*hc-NX4sA33ZiXJsx.png" /><figcaption>Install the app.</figcaption></figure><p>7. Select a default channel and hit the <em>Allow</em> button:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/0*AlQjRBTSdAGlSz5v.png" /><figcaption>Select a default Slack channel.</figcaption></figure><p>8. 🥳 <em>Your app is now installed and you can copy the webhook!</em> 🥳 Navigate to <em>Features &gt; Incoming Webhooks &gt; Webhook URLs for Your Workspace</em> and do your thing!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*kvw_zPyK1bP-kQuf.png" /><figcaption>Configure one or more webhooks.</figcaption></figure><h3>Add collaborators</h3><p>It is super easy to add a new application. Now you can add <em>collaborators</em> to your app, so when you leave the organization, the application is still managed by your peers. Navigate to <em>Settings &gt; Collaborators</em> to add more members.</p><p>Enjoy!</p><p><em>I work as a Lead Developer at </em><a href="https://www.wehkamp.nl/"><em>Wehkamp.nl</em></a><em>, one of the biggest e-commerce companies of the Netherlands. This article is </em><a href="https://medium.com/wehkamp-techblog"><em>part of our Tech Blog</em></a><em>, check it out &amp; subscribe. Looking for a great job? </em><a href="https://wehkampretailgroup.nl/vacatures/?favourites=%5B%5D&amp;filter_favourites=false&amp;search=&amp;smartrecruiters_department_label%5B%5D=Tech"><em>Check our job offers</em></a><em> or </em><a href="https://www.linkedin.com/in/keescbakker/"><em>drop me a line on LinkedIn</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://keestalkstech.com/dont-use-slack-incoming-webhooks-app-creation-is-dead-simple/"><em>https://keestalkstech.com</em></a><em> on September 5, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af9ea8ff41da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wehkamp-techblog/dont-use-slack-incoming-webhooks-app-creation-is-dead-simple-af9ea8ff41da">Don’t use Slack Incoming WebHooks; App Creation is Dead Simple!</a> was originally published in <a href="https://medium.com/wehkamp-techblog">wehkamp-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dependency Injection based on request headers]]></title>
            <link>https://medium.com/wehkamp-techblog/dependency-injection-based-on-request-headers-f05d79eea8c5?source=rss-5e9a43654081------2</link>
            <guid isPermaLink="false">https://medium.com/p/f05d79eea8c5</guid>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[dependency-injection]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Kees C. Bakker]]></dc:creator>
            <pubDate>Wed, 03 Jun 2020 06:12:21 GMT</pubDate>
            <atom:updated>2020-06-04T13:57:07.932Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XAnoJL5qwwBNj1hB.jpg" /></figure><p>Dependency Injection (DI) helps us to change the behavior of parts of our application on the fly. This is especially neat when you want to test your domain services against a mocked data-store. <em>But what if you need to change the behavior in your API based on a request header?</em></p><p>Yesterday I had a discussion with my colleague <a href="https://www.linkedin.com/in/robert-kranenburg-a7b616a7">Robert Kranenburg</a> about this subject. He showed an example of a console application changing its behavior based on an argument. I took the idea and converted it into .NET Core 3.1 code for a Web API.</p><h3>Let’s change the message</h3><p>Our objective: <em>let’s change the response message based on the presence of a cookie named “hidden”, using dependency injection </em>.</p><p>First, we need to define the controller and the service interface:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94207c9d9801c4b804579900d3765e95/href">https://medium.com/media/94207c9d9801c4b804579900d3765e95/href</a></iframe><p>.NET Core uses <em>constructor injection</em> and will provide an instance for each constructor parameter. It will inject an IMessageService instance into the controller.</p><p>Now, let’s define 2 implementations of the IMessageService:</p><ul><li>DefaultMessageService just returns a &quot;Hello world!&quot; message</li><li>HiddenMessageService returns a different message. To make things more interesting, it uses a dependency to construct that message.</li></ul><p>Let’s view the code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e5d0db9a2785550d3f2667dd26f7285/href">https://medium.com/media/2e5d0db9a2785550d3f2667dd26f7285/href</a></iframe><p>Before we set up the dependency injection, let’s dive deeper into lifetimes.</p><h3>3 lifetime types</h3><p>A DI service is added with a <em>lifetime</em>:</p><ul><li>AddTransient: Adding a <strong>transient </strong>service means that each time the service is requested, a new instance is created.</li><li>AddSingleton: A <strong>singleton </strong>is an instance that will last the entire lifetime of the <strong>application</strong>. In web terms, it means that after the initial request of the service, every subsequent request will use that same instance, across all requests.</li><li>AddScoped: A <strong>scoped</strong> service is instantiated per scope. For web this means the same instance per request, but you can actually create your own scopes.</li></ul><p>Because we want to change the behavior based on request information, it makes sense to use a <strong>scoped</strong> service.</p><h3>Accessing request information</h3><p>In the olden days we could do anything we wanted with the static HttpContext.Current and be done with it. In .NET Core we use the IHttpContextAccessor and dependency injection to interact with the HttpContext. We can use the AddHttpContextAccessor to set this up.</p><h3>DI config</h3><p>Let’s go to the Startup.cs and set up the dependency injection:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d5e41dc0e5f1239d7ce5c3d50569058/href">https://medium.com/media/1d5e41dc0e5f1239d7ce5c3d50569058/href</a></iframe><p><em>Why do we add </em><em>DefaultMessageService and </em><em>HiddenMessageService without interfaces to the service collection? </em>Well, this helps us to use DI in the classes themselves. HiddenMessageService <em>needs</em> this, because this is how we get an ISecretKey instance. Adding these classes will make our lives way easier when we need to instantiate them. If we would &quot;new them up&quot; ourselves, we would need to resolve all the dependencies as well, leading to a lot of unnecessary code.</p><p>The IMessageService is resolved via a <em>factory method</em>. This method gets an IServiceProvider as input to locate any services that are required. We use this provider to locate the HTTP accessor and to check if a cookie named <em>hidden</em> is present. If so, we resolve the IMessageService as HiddenMessageService; otherwise as DefaultMessageService.</p><h3>The result</h3><p>When we add the code together we get this behavior:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JRfDm63TpQDdf7s0.gif" /><figcaption>The contents of the message is changed based on the presence of a cookie named <em>hidden</em>.</figcaption></figure><h3>Final thoughts</h3><p>It is easy to change de behavior of your program based on a request header or cookie. But things can get messy quite fast when you are doing dependency injection. Startup classes get large quickly when you need to inject many classes.</p><p>To make things easier, you can use an extension method like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74a1b2568d6d97ad48e68b430e90b51e/href">https://medium.com/media/74a1b2568d6d97ad48e68b430e90b51e/href</a></iframe><p>Which can be used like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9d268f64c692316ceadc4cac30e2955/href">https://medium.com/media/b9d268f64c692316ceadc4cac30e2955/href</a></iframe><p>In this example we used a cookie, but any header can be used to change the behavior of your application.</p><h3>Further reading</h3><p>While working on this topic I found some excellent sources for reading:</p><ul><li><a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/dependency-injection?view=aspnetcore-3.1">Dependency injection in ASP.NET Core</a></li><li><a href="https://dotnetcoretutorials.com/2017/03/25/net-core-dependency-injection-lifetimes-explained/">.NET Core Dependency Injection Lifetimes Explained</a></li><li><a href="https://keestalkstech.com/2018/04/dependency-injection-with-ioptions-in-console-apps-in-net-core-2/">Dependency injection (with IOptions) in Console Apps in .NET Core</a></li></ul><p><em>I work as a Lead Developer at </em><a href="https://wehkamp.nl/?source=post_page---------------------------"><em>Wehkamp.nl</em></a><em>, one of the biggest e-commerce companies of the Netherlands. This article is </em><a href="https://medium.com/wehkamp-techblog?source=post_page---------------------------"><em>part of our Tech Blog</em></a><em>, check it out &amp; subscribe. Looking for a great job? </em><a href="https://werkenbij.wehkamp.nl/vacatures/?source=post_page---------------------------"><em>Check our job offers</em></a><em> or </em><a href="https://www.linkedin.com/in/keescbakker/?source=post_page---------------------------"><em>drop me a line on LinkedIn</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://keestalkstech.com/2020/06/dependency-injection-based-on-request-headers/"><em>https://keestalkstech.com</em></a><em> on June 3, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f05d79eea8c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wehkamp-techblog/dependency-injection-based-on-request-headers-f05d79eea8c5">Dependency Injection based on request headers</a> was originally published in <a href="https://medium.com/wehkamp-techblog">wehkamp-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streaming Kafka topic to Delta table (S3) with Spark Structured Streaming]]></title>
            <link>https://medium.com/wehkamp-techblog/streaming-kafka-topic-to-delta-table-s3-with-spark-structured-streaming-2bb3027c7565?source=rss-5e9a43654081------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bb3027c7565</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[structured-streaming]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Kees C. Bakker]]></dc:creator>
            <pubDate>Sat, 09 Nov 2019 09:25:36 GMT</pubDate>
            <atom:updated>2023-07-10T07:48:36.340Z</atom:updated>
            <content:encoded><![CDATA[<h3>Streaming a Kafka topic in a Delta table on S3 using Spark Structured Streaming</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a1EI_hqpPre1Mf5p.jpg" /></figure><p>At Wehkamp we use <a href="https://www.wikiwand.com/en/Apache_Kafka">Apache Kafka</a> in our event driven service architecture. It handles high loads of messages really well. We use <a href="https://spark.apache.org/">Apache Spark</a> to run analysis, data processing and machine learning.</p><p>When I work with Kafka, the words of <a href="https://www.linkedin.com/in/mark-van-gool/">Mark van Gool</a>, a data architect, always echo in my head: “<em>Kafka should not be used as a data store</em>!” It is really tempting for me to do so, but most event topics have a small retention period. Our data strategy specifies that we should store data on S3 for further processing. Raw S3 data is not the best way of dealing with data on Spark, though. In this blog I’ll show how you can use <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#overview">Spark Structured Streaming</a> to write JSON records on a Kafka topic into a <a href="https://delta.io/">Delta table</a>. This will make the data structured and faster to process.</p><p>Note: <em>This article assumes that you’re dealing with a JSON topic without a schema.</em> <em>It also assumes that the buckets are mounted to the file system, so we can read and write to them directly (without the need for boto3). Also: I’m using Databricks Cloud, so some parts may be specific to that environment.</em></p><h3>Design</h3><p>To make things easier to understand, I’ve made a diagram of the setup we’re trying to create. <em>Let’s assume we have 2 topics that we need to turn into Delta tables. We have another notebook that consumes those delta tables.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K-3fwTL1QS_JVaMQzvxHbQ.png" /><figcaption>Visualization of the setup. The notebook is now decoupled from Kafka and uses the delta table. Delta tables will make your setup more scalable.</figcaption></figure><p>Each <em>topic </em>gets <em>its own Delta table</em> in <em>its own bucket</em>. The topics are read by parametrized jobs that will use Spark Structured Streaming to stream updates into the table. The jobs can run every hour or continuously, depending on your needs. Each job will save the Kafka <em>group id</em>, so it continues reading from Kafka where it left of.</p><p>The notebook that needs data from the topics now connects to the delta table and consumes that data. This way the notebook becomes decoupled from Kafka. The Delta table now works like a database, it contains a single row per key.</p><h4>Steps</h4><p>We need to do the following steps:</p><ol><li>Add widgets to parametrize the notebook. Use the widgets content to determine the configuration.</li><li>Infer the Kafka schema of the topic and persist it for later use.</li><li>Create a schema from the schema file and read the Kafka topic.</li><li>Ensure the Delta table has the right schema and write the table using an <em>upsert</em>.</li><li>Clean up the Delta table.</li></ol><p>Let’s start!</p><h3>1. Configuration</h3><p>We will build a generic notebook, so we must add some <em>widgets </em>to influence the way the notebook runs:</p><ul><li>Let’s add a <em>debug</em> setting to support debug runs that don’t touch production files.</li><li>We’ll use the <em>streaming</em> setting to indicate if we want to use continues streaming or only <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#triggers">trigger</a> it once (False). Most of our jobs are only scheduled once every hour.</li><li>The <em>topic</em> holds the Kafka topic we want to turn into a delta table.</li><li>The <em>update Kafka schema</em> indicates that we want to re-infer the Kafka schema. We need this if the structure of the topic changes.</li></ul><p>The widgets look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*E2yU3fMIgkR1bEelSeifAw.png" /><figcaption>Widgets are shown on top of the notebook. Unfortunately, widgets are shown in alphabetical order.</figcaption></figure><h4>Global variables</h4><p>Let’s start out with a cell with global variables that will not be parametrised.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd0fed481ad8cacdc6af38212efc2233/href">https://medium.com/media/cd0fed481ad8cacdc6af38212efc2233/href</a></iframe><p>At Wehkamp we use prefixes for buckets.</p><h4>Widgets</h4><p>Let’s set up the widgets:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3566c6121a1219738ffa0138c0aa4023/href">https://medium.com/media/3566c6121a1219738ffa0138c0aa4023/href</a></iframe><h4>Parse widgets</h4><p>Now that we have our widgets, we should parse them to variables. I’ve lifted some code from <a href="https://keestalkstech.com/2019/08/adding-true-false-and-list-widgets-to-your-databricks-notebook/">this blog</a> to help me get the values from the widgets:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e62a7c7a71aea91d0813376dcc27d433/href">https://medium.com/media/e62a7c7a71aea91d0813376dcc27d433/href</a></iframe><p>Let’s convert the widgets values to variables:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f68dccfb997be0f4075ce0237caa67b/href">https://medium.com/media/0f68dccfb997be0f4075ce0237caa67b/href</a></iframe><h4>Locations by convention</h4><p>To write a Delta table, we need 3 settings: the <em>location </em>of the delta table, the <em>location of the checkpoints</em> and the <em>location of the schema file</em>. Let’s use the name of the topic as a convention to get these locations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94428d8555515cc9a322bedc148d5730/href">https://medium.com/media/94428d8555515cc9a322bedc148d5730/href</a></iframe><h3>2. Schema</h3><p>The Kafka topic contains JSON. To properly read this data into Spark, we must provide a schema. To make things faster, we’ll infer the schema once and save it to an S3 location. Upon future runs we’ll use the saved schema.</p><h4>Schema inference</h4><p>Before we can read the Kafka topic in a streaming way, we must infer the schema. Let’s use the code from this <a href="https://keestalkstech.com/2019/11/kafka-spark-and-schema-inference/">blog to infer the schema</a>. We’ll start with some imports:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9957d8025812ad72100f647bbe89a101/href">https://medium.com/media/9957d8025812ad72100f647bbe89a101/href</a></iframe><p>Now, let’s define a <em>method </em>to infer the schema of a Kafka topic and return it in the JSON format:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f096889beef087d80ae7756827d50ebc/href">https://medium.com/media/f096889beef087d80ae7756827d50ebc/href</a></iframe><p>Inferring a schema might take a while, as Spark has to read the entire topic to determine the schema. That’s why we cache it in the S3 bucket, so we only have to infer the schema once.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66f9c5cc311406eadd22a3be2d167ba4/href">https://medium.com/media/66f9c5cc311406eadd22a3be2d167ba4/href</a></iframe><p><em>Note: I’m using </em><strong><em>dbutils.fs</em></strong><em> because writing a file with </em><strong><em>file.write</em></strong><em> will write the file to the driver, but not to S3.</em></p><h4>Load schema</h4><p>Loading the JSON from S3 into a schema is super simple:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4670d06e039e06df88506734d30f4e9/href">https://medium.com/media/e4670d06e039e06df88506734d30f4e9/href</a></iframe><h3>3. Read Kafka Stream</h3><p>Now we can finally start to use <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#overview">Spark Structured Streaming</a> to read the Kafka topic. The function we’ll use looks a lot like the infer_topic_schema_json function. The main difference is the usage of readStream that will use <em>structured streaming</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/beb73082d56a86d08a8510921e9fe70f/href">https://medium.com/media/beb73082d56a86d08a8510921e9fe70f/href</a></iframe><p>Let’s read our topic into a dataframe:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a95eceb5d9785c201a9f18db3999fd88/href">https://medium.com/media/a95eceb5d9785c201a9f18db3999fd88/href</a></iframe><h3>4. Delta table</h3><p>Now that we have a (streaming) dataframe of our Kafka topic, we need to write it to a Delta table.</p><h4>Ensure the Delta table</h4><p>First, we need to make sure the Delta table is present. Here is where we can use the schema of the dataframe to make an empty dataframe. This dataframe will create an empty Delta table if it does not exist.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2560df4337c881cec9773ed1bcdc8848/href">https://medium.com/media/2560df4337c881cec9773ed1bcdc8848/href</a></iframe><h4>Upsert by Kafka key</h4><p>The kafka_key is a unique identifier for each record. We can use the key to update the data in the Delta table. We&#39;ll use this script to make that <em>upsert</em> happen:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c41c16fbe6d05957986fdb8ff7c56d3b/href">https://medium.com/media/c41c16fbe6d05957986fdb8ff7c56d3b/href</a></iframe><p>We want to update or insert <em>all </em>the columns of our dataframe into the Delta table, so we are using whenNotMatchedInsertAll and whenMatchedUpdateAll. More information can be found in the documentation of the <a href="https://docs.delta.io/0.4.0/api/python/index.html#delta.tables.DeltaMergeBuilder">DeltaMergeBuilder</a>.</p><h4>Write stream data</h4><p>Now that we have everything in place, we can write to our delta table:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d803ba14806a53b406694f6243696ef/href">https://medium.com/media/2d803ba14806a53b406694f6243696ef/href</a></iframe><h3>5. Clean up</h3><p>A few notes on clean up. When you do streaming, you might want to schedule a <em>separate</em> job that does the clean up periodically. When you only “trigger once”, you can wait for the cell to finish with the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9de15b27ad570a0506856e45adf437a0/href">https://medium.com/media/9de15b27ad570a0506856e45adf437a0/href</a></iframe><h4>Optimize</h4><p>Let’s <a href="https://docs.databricks.com/spark/latest/spark-sql/language-manual/optimize.html">optimize</a> the delta table first. We use the default <a href="https://docs.databricks.com/delta/optimizations/file-mgmt.html#compaction-bin-packing"><em>bin-packing</em></a> to<em> produce evenly-balanced data files with respect to their size on disk</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91fa9a9fc31927073939115bb5462acb/href">https://medium.com/media/91fa9a9fc31927073939115bb5462acb/href</a></iframe><p><em>Note: when you use partitioning and z-ordering to optimize the layout of your Delta table, you might benefit from running the optimize on a separate job and cluster. We’ve seen some use-cases where it was cheaper to run the optimization once every 4 hours on a massive CPU cluster.</em></p><h4>Vacuum</h4><p>Next, we’ll <a href="https://docs.databricks.com/spark/latest/spark-sql/language-manual/vacuum.html">vacuum</a> to throw away unused files that are older than 7 days. You might not want this in your specific use case, but we see delta tables growing way too big because we did not vacuum.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73e9c0e2611d64390ebaac61e49de3b5/href">https://medium.com/media/73e9c0e2611d64390ebaac61e49de3b5/href</a></iframe><p>The impact of VACUUM on some of our older tables is massive:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*t_qCWJy2bN6dc0AHxaS7_Q.png" /></figure><p>This tables shows the savings after applying the VACUUM command. We <strong>saved</strong> <strong>99.54%</strong> on our storage. Some of these tables have been accumulating for quite some time.</p><p>You might want to read up on the <a href="https://databricks.com/blog/2019/02/04/introducing-delta-time-travel-for-large-scale-data-lakes.html">Delta Time Travel</a> feature and make a conscious decision if you want to keep using it.</p><h3>6. Databricks Database Table</h3><p>You can add the table to the <a href="https://docs.databricks.com/data/tables.html">Databricks database</a>, so it is easier is to query:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8e660337fd62e6b4d88370b7f4a5c98/href">https://medium.com/media/e8e660337fd62e6b4d88370b7f4a5c98/href</a></iframe><p>First we need to ensure the database:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/65947980a101cb0748139884db12e911/href">https://medium.com/media/65947980a101cb0748139884db12e911/href</a></iframe><p>Then we can ensure the table:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/19cfa27001ba701a300a9b185ee1d96e/href">https://medium.com/media/19cfa27001ba701a300a9b185ee1d96e/href</a></iframe><h3>7. Reading a Delta table</h3><p>Reading a Delta table based on the location is a piece of pie:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/37470f86f821fa1a93061445aa71bc53/href">https://medium.com/media/37470f86f821fa1a93061445aa71bc53/href</a></iframe><p>It is even easier when <a href="https://keestalkstech.com/2019/11/streaming-a-kafka-topic-to-a-delta-table-on-s3-with-spark-structured-streaming/#databricks-database-table">using a table</a>:</p><p><a href="https://gist.github.com/KeesCBakker/08fa5de568a8fb947c1863ff8b1b4393">https://gist.github.com/KeesCBakker/08fa5de568a8fb947c1863ff8b1b4393</a></p><h3>Final thoughts</h3><p>I’ve shown one way of using Spark Structured Streaming to update a Delta table on S3. The combination of Databricks, S3 and Kafka makes for a high performance setup. But the real advantage is not in just serializing topics into the Delta Lake, but combining sources to create new Delta tables that are updated on the fly and provide relevant data to your domain.</p><p>We’ve seen an uplift in the performance of scripts that used to query Kafka themselves (some had an uplift of 25%). A delta table is faster and makes code easier to read. We see that the total cost goes down.</p><p>A big shout-out to <a href="https://www.linkedin.com/in/jesse-bouwman-610b49a5/">Jesse Bouwman</a> (Data Scientist) and <a href="https://www.linkedin.com/in/martinus-slomp-44800317/">Martinus Slomp</a> (Senior .NET Developer) for the collaboration on this code.</p><h3>Further reading</h3><p>While working on this topic I found some excellent sources for reading:</p><ul><li><a href="https://docs.databricks.com/delta/quick-start.html">Introduction to Delta Lake: Delta Lake Quickstart</a></li><li><a href="https://docs.databricks.com/delta/delta-update.html">Delta Lake: Table Deletes, Updates, and Merges</a></li><li><a href="https://www.linuxfoundation.org/press-release/2019/10/the-delta-lake-project-turns-to-linux-foundation-to-become-the-open-standard-for-data-lakes/https://www.linuxfoundation.org/press-release/2019/10/the-delta-lake-project-turns-to-linux-foundation-to-become-the-open-standard-for-data-lakes/">The Delta Lake Project Turns to Linux Foundation to Become the Open Standard for Data Lakes</a></li><li><a href="https://databricks.com/blog/2019/10/03/simple-reliable-upserts-and-deletes-on-delta-lake-tables-using-python-apis.html">Simple, Reliable Upserts and Deletes on Delta Lake Tables using Python APIs</a></li><li><a href="https://keestalkstech.com/2019/11/easy-spark-optimization-for-max-record-aggregate-instead-of-join/">Easy Spark optimization for max record: aggregate instead of join?</a></li><li><a href="https://keestalkstech.com/2019/11/add-more-color-to-the-python-code-of-your-databricks-notebook/">Add more color to the Python code of your Databricks notebook</a></li></ul><p><em>I work as a Lead Developer at </em><a href="https://wehkamp.nl/?source=post_page---------------------------"><em>Wehkamp.nl</em></a><em>, one of the biggest e-commerce companies of the Netherlands. This article is </em><a href="https://medium.com/wehkamp-techblog?source=post_page---------------------------"><em>part of our Tech Blog</em></a><em>, check it out &amp; subscribe. Looking for a great job? </em><a href="https://werkenbij.wehkamp.nl/vacatures/?source=post_page---------------------------"><em>Check our job offers</em></a><em> or </em><a href="https://www.linkedin.com/in/keescbakker/?source=post_page---------------------------"><em>drop me a line on LinkedIn</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://keestalkstech.com/2019/11/streaming-a-kafka-topic-to-a-delta-table-on-s3-with-spark-structured-streaming"><em>https://keestalkstech.com</em></a><em> on November 9, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bb3027c7565" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wehkamp-techblog/streaming-kafka-topic-to-delta-table-s3-with-spark-structured-streaming-2bb3027c7565">Streaming Kafka topic to Delta table (S3) with Spark Structured Streaming</a> was originally published in <a href="https://medium.com/wehkamp-techblog">wehkamp-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Validate strongly typed options when using config sections]]></title>
            <link>https://medium.com/wehkamp-techblog/validate-strongly-typed-options-when-using-config-sections-fffbadd30821?source=rss-5e9a43654081------2</link>
            <guid isPermaLink="false">https://medium.com/p/fffbadd30821</guid>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <dc:creator><![CDATA[Kees C. Bakker]]></dc:creator>
            <pubDate>Fri, 20 Sep 2019 05:19:24 GMT</pubDate>
            <atom:updated>2019-09-25T11:59:23.202Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4n3u61jnf6An4-0S.jpg" /><figcaption>Sometimes you need to validate what’s in the box.</figcaption></figure><p>I like to validate my application configuration upon startup. Especially when doing local development, I want to know which application settings are missing. I also like to know where I should add them. This blog shows how to implement validation of your configuration classes using data annotations.</p><h3>Use data annotations</h3><p>The configuration classes were completely rewritten when .NET Core was introduced. <em>We came a long way since the old .config files.</em></p><p>I love how we can now use <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/options?view=aspnetcore-2.2">the options pattern</a> to bind our settings classes. Strongly typed instances are injected as <strong>IOptions&lt;T&gt;</strong> or <strong>IOptionsSnapShot&lt;T&gt;</strong> into our services.</p><p>Let’s define a settings class and use data annotations to make the fields required:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/61ed350bf477ec4881e85c69c00340ab/href">https://medium.com/media/61ed350bf477ec4881e85c69c00340ab/href</a></iframe><p>In ASP.NET we already use data annotation to validate our models. Some better known attributes are: <a href="https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.dataannotations.requiredattribute?view=netcore-2.2">Required</a>, <a href="https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.dataannotations.minlengthattribute?view=netcore-2.2">MinLength</a>, <a href="https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.dataannotations.maxlengthattribute?view=netcore-2.2">MaxLength</a>, <a href="https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.dataannotations.rangeattribute?view=netcore-2.2">Range</a>, <a href="https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.dataannotations.stringlengthattribute?view=netcore-2.2">StringLength</a>, <a href="https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.dataannotations.emailaddressattribute?view=netcore-2.2">EmailAddress</a>, <a href="https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.dataannotations.urlattribute?view=netcore-2.2">Url</a> and <a href="https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.dataannotations.regularexpressionattribute?view=netcore-2.2">RegularExpression</a>.</p><h3>What does .NET Core provide?</h3><p>Microsoft introduced a validation method that can validate the options (since .NET Core 2.2). You can use it with data annotations and it can look something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/223a707354510f9859fb4be49a1bf57d/href">https://medium.com/media/223a707354510f9859fb4be49a1bf57d/href</a></iframe><p>This is what happens when you try to read the <strong>Value</strong> of the <strong>IOptions&lt;T&gt;</strong>object:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hi8qmVW2KEIrhSdUsnSeMg.png" /><figcaption>An error is thrown inside the code that tries to use the provided option. It is validated when the valid is accessed.</figcaption></figure><p>.NET throws an error upon usage and the error message does not provide many clues on what to do or where to look.</p><p>If you, like me, are looking for eager validation on startup, you must wait until the next version:</p><blockquote>Eager validation (fail fast at startup) is under consideration for a future release.<em><br></em>Source: <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/options?view=aspnetcore-2.2#options-validation-1">Options pattern in ASP.NET Core — Options Validation</a></blockquote><p>So, we need to code something our selves.</p><h3>PostConfigure to the rescue!</h3><p>Let’s use PostConfigure to validate the created option and throw an error if something is invalid:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b48548d15a53ad6d951048c9e6053424/href">https://medium.com/media/b48548d15a53ad6d951048c9e6053424/href</a></iframe><p>Now, when we run the code, it breaks in (and during) Startup.cs. This is eager validation, and it provides us with an exception that helps to solve the configuration-error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kCnTNKkE3qEix2gxrAEXFA.png" /><figcaption>The application now breaks in the right time + place and with a useful error message.</figcaption></figure><h3>Extension methods</h3><p>Your startup can get quite verbose if you have to configure many options in your application. That’s why I’ve created the following extension methods to help set up the configuration and validation of options:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45e908468482629d9f544e42b6e125a1/href">https://medium.com/media/45e908468482629d9f544e42b6e125a1/href</a></iframe><p>Coming up with a good name for an extension method is usually the hardest part. Normally, I set up options using <strong>services.Configure&lt;KafkaOptions&gt;(Configuration.GetSection(&quot;Kafka&quot;))</strong>, so <strong>ConfigureAndValidate&lt;KafkaOptions&gt;(&quot;Kafka&quot;, Configuration)</strong> looks good to me.</p><h3>Example application</h3><p>Let’s look at an example application. I like to use the <strong>appsettings.json</strong> to define the configuration of my services. This is the <strong>Program.cs</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1b9bc5a1105c9af2cf07450556556f8/href">https://medium.com/media/a1b9bc5a1105c9af2cf07450556556f8/href</a></iframe><p>This setup allows us to <a href="https://keestalkstech.com/2019/01/setup-multiple-setting-files-with-a-net-console-application/">use environment-specific configuration files</a>. Next, we’ll set up the loading of the configuration information into an object in the <strong>startup.cs</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/709ec24463ca24e02502726fd989e73b/href">https://medium.com/media/709ec24463ca24e02502726fd989e73b/href</a></iframe><p><em>Note: the configuration is injected in the startup class.</em></p><h3>Final thoughts</h3><p>Because we’ve used <strong>config.AddEnvironmentVariables()</strong> in our program, the setting values could also come from the environment. Technically, we can add a <strong>Kafka__GroupId</strong> to satisfy the requirement.</p><p>Configuration can come from a <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/index?view=aspnetcore-2.2#providers">myriad of places</a>, not just from the config files. Pointing your fellow developers into the right direction can save a lot of time.</p><p><em>I work as a Lead Developer at </em><a href="https://wehkamp.nl/?source=post_page---------------------------"><em>Wehkamp.nl</em></a><em>, one of the biggest e-commerce companies of the Netherlands. This article is </em><a href="https://medium.com/wehkamp-techblog?source=post_page---------------------------"><em>part of our Tech Blog</em></a><em>, check it out &amp; subscribe. Looking for a great job? </em><a href="https://werkenbij.wehkamp.nl/vacatures/?source=post_page---------------------------"><em>Check our job offers</em></a><em> or </em><a href="https://www.linkedin.com/in/keescbakker/?source=post_page---------------------------"><em>drop me a line on LinkedIn</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://keestalkstech.com/2019/09/validate-strongly-typed-options-when-using-config-sections/"><em>keestalkstech.com</em></a><em> on September 20, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fffbadd30821" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wehkamp-techblog/validate-strongly-typed-options-when-using-config-sections-fffbadd30821">Validate strongly typed options when using config sections</a> was originally published in <a href="https://medium.com/wehkamp-techblog">wehkamp-techblog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>