<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hans Christian Ekne on Medium]]></title>
        <description><![CDATA[Stories by Hans Christian Ekne on Medium]]></description>
        <link>https://medium.com/@hc.ekne?source=rss-ab0bff3c2cc9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qHQx407X8oRT9H0vNELp7g.png</url>
            <title>Stories by Hans Christian Ekne on Medium</title>
            <link>https://medium.com/@hc.ekne?source=rss-ab0bff3c2cc9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:50:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hc.ekne/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Pricing-as-a-Service]]></title>
            <link>https://medium.com/@hc.ekne/pricing-as-a-service-996155cf9261?source=rss-ab0bff3c2cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/996155cf9261</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[entrepreneurship]]></category>
            <category><![CDATA[pricing-strategy]]></category>
            <category><![CDATA[saas-pricing]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Hans Christian Ekne]]></dc:creator>
            <pubDate>Sat, 08 Nov 2025 11:31:25 GMT</pubDate>
            <atom:updated>2025-11-21T10:32:05.575Z</atom:updated>
            <content:encoded><![CDATA[<h4>Make Pricing Always-On, Not a One-Off Project</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZLNJEyJ4DGMZCG8eG1rzPg.png" /><figcaption>Continuous pricing / image create by author with GPT-5</figcaption></figure><p>What if your prices were always on-strategy, always current, and you didn’t have to run a pricing project every quarter to get there? Pricing-as-a-Service (PaaS) turns pricing from an episodic effort into a reliable ongoing capability.</p><p>Think of pricing like a football team. You assemble the squad, assign positions, and set a game plan, that’s the initial project. But as the season unfolds, you adjust the lineup and tactics to the competition. Pricing is the same: to stay effective and current, you need a continuous pricing function.</p><p>For years, only the biggest firms could afford advanced pricing, data-driven models, experimentation platforms, and change-management consulting. My mission is to democratise pricing: to make rigorous, data-driven price setting available to far more businesses.</p><p>For a lot of companies, the best way to achieve that is through a managed service: a centrally run pricing hub that consumes relevant customer and purchasing data and continuously delivers optimal prices, discounts, and promotions, not as a one-off, but as an always-on capability.</p><h3>Why Now: The Case for Pricing-as-a-Service</h3><p>Over the years, I’ve built quite a few data-driven pricing engines. While these projects usually achieve their initial goals, clients often find it difficult to capture lasting value from the models. Running them in production tends to be more complex than expected, and day-to-day operations can be too technical for the team. In my experience, there are four main reasons why maintaining your own pricing infrastructure is so challenging:</p><ol><li><strong>People turnover and capacity.</strong> When key resources leave, core competencies tied to the model are lost. Even when a new person is appointed, she or he must split time between value‑creating work and keeping the engine and data flows running — maintenance that steals focus.</li><li><strong>Data drift and breakage.</strong> Upstream data changes. New values appear that weren’t in the training data. Labels on categorical fields get renamed. I’ve even seen data warehouses overwrite historical product names, effectively erasing customer history. The model keeps receiving data, but it now needs re‑interpretation and relabeling to stay on track.</li><li><strong>DevOps complexity.</strong> Even “simple” models need reliable pipelines, monitoring, and cost‑efficient infrastructure. It gets trickier when multiple in‑house and external data sources feed ensemble models that should auto‑retrain and hit a clear SLA.</li><li><strong>Model Complexity.</strong> Even though we try to follow the principle of Occam’s Razor and strive not to develop models more complex than they need to be, the pricing optimization still requires a minimum level of complexity and technical pricing skills which can sometimes be difficult for a company to attract or retain.</li></ol><p>PaaS addresses these pain points by providing a steady operating backbone while you focus on commercial outcomes. We handle the complexity of DevOps and price modelling, along with dealing with and alerting your team of inconsistencies in the data. If the case for PaaS resonates, the next question is control: what do you keep vs. outsource?</p><h3>You Set the Why and We Run the How</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*20Igtuuu1wkCgPVB" /><figcaption>You set the Why we run the How / image created by author using GPT-5</figcaption></figure><p>Some parts of the pricing process shouldn’t be outsourced, especially overall pricing strategy, which is a core part of company strategy. Strategy speaks to the heart and soul of the company: your positioning, value narrative, and where you want to compete.</p><p>Once strategy is set, execution becomes an operating function. Outsourcing that execution is as natural as outsourcing parts of IT, sales development, or fulfilment and logistics. Day‑to‑day pricing is something you must do well, but you don’t have to do it all yourself.</p><ul><li><strong>Keep in‑house:</strong> Pricing strategy — market positioning, value narrative, brand promise, and where you want to compete.</li><li><strong>Outsource:</strong> Price execution and operations — elasticity estimation, list/pack architecture, discount ladders, tender guardrails, promo governance, experimentation, monitoring, and change controls.</li></ul><p>With roles clear, here’s how the service actually runs day to day.</p><h3>The ins-and-outs of PaaS (how it works)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*znIEW5N7u83t767D" /><figcaption>Continuous pricing pipeline / image created by author using GPT-5</figcaption></figure><p>In practice, a PaaS solution ingests a defined set of inputs, applies business guardrails and configurable settings, runs them through a modular pricing engine, and pushes the resulting prices and recommendations back into the sales and marketing channels where decisions are made. The workflow also measures performance and learns, so the next round is smarter.</p><h4>Inputs</h4><p>A typical PaaS setup pulls from transaction history; product hierarchy and pack structure; cost files or indices; CRM activity and churn/win signals; promotion history; inventory or capacity; competitor benchmarks; and policy constraints. In less‑mature environments (e.g., start‑ups or early scale‑ups) those inputs can be sparse; the system starts with stronger priors, expert rules, external benchmarks, and simple demand assumptions, and then quickly adjusts the journey (tests, tags, feedback) to build the data needed for modelling.</p><h4>The engine</h4><p>Specialised modules work together: elasticity models estimate demand response; willingness‑to‑pay segmentation separates value seekers from premium buyers; price/pack architecture aligns offers and price points; discount ladders and tender corridors codify deal‑making boundaries; promotion simulators forecast uplift and cannibalisation; and anomaly detection flags outliers or leakage. Each module is governed by parameters you can tune without rewriting code.</p><h4>Outputs</h4><p>The engine produces operational artefacts the business can use immediately: price lists and rate cards by region, tier, or segment; personalised save/win‑back offers for at‑risk or high‑potential customers; tender “blue sheets” with target, stretch, and walk‑away logic; promotion calendars with pre‑ and post‑analysis; plus exception logs and approval prompts where human judgement is required.</p><h4>Governance and control</h4><p>Guardrails such as floors, ceilings, and exposure limits, ensure prices stay within strategic bounds. An approval matrix routes exceptions to the right level; audit trails preserve who changed what and when. A compact KPI set tracks impact and adoption: price realisation, margin %, churn/win‑rate, promo ROI, and contribution per customer. These metrics feed back into the engine to recalibrate elasticities, refine segments, and tighten guardrails over time.</p><h4>The operating rhythm</h4><p>Day to day, PaaS runs a simple cycle: ingest → decide → approve → deliver → learn. New data lands, the engine generates recommendations, exceptions are reviewed, prices are published to channels (ERP, CPQ, e‑commerce, CRM), and performance is measured to inform the next iteration. This cadence turns pricing from one‑off projects into a repeatable capability.</p><h3>Is PaaS Right for You?</h3><p>A useful rule of thumb: if revenue, risk, and customer value are determined by your live clearing/quoting engine, you can buy tools, but you shouldn’t outsource the execution. If other elements make up your core business, outsourcing pricing can be a great fit.</p><p><strong>PaaS is generally a great fit if you:</strong></p><ul><li>Sell subscriptions or repeat purchases where price realization, churn/win‑rate, and promo ROI matter.</li><li>Manage many SKUs/customers/regions where price discipline and guardrails are hard to maintain manually.</li><li>Have frequent cost changes and need indexation or pass‑through discipline.</li><li>Want faster learning via controlled tests, not anecdote‑driven debates.</li><li>Want to scale quickly and leanly without hiring a full CRO or Pricing Manager.</li></ul><p>Examples of sectors where this applies:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jvsg6J63L3EtCBpf" /><figcaption>Industries where it makes sense to outsource price execution</figcaption></figure><p><strong>PaaS is not a fit if you:</strong></p><ul><li>Compete in markets where pricing is the product.</li><li>Run a minimal assortment where pricing is simple and differentiation is elsewhere.</li><li>Cannot share even minimally necessary data (we can work with anonymization and minimization, but zero data means zero lift).</li></ul><p>Examples of sectors where outsourcing pricing is a bad idea:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2VRZJEgB8sdgRP3t" /><figcaption>Industries where it doesn’t make sense to outsource price execution</figcaption></figure><h3>Your 90-Day Onramp to Always-On Pricing</h3><p>So let’s say your industry is a good match for pricing as a service, what would an implementation look like? Below I outline a standard 30/60/90-day high-level plan to showcase the steps needed to reach a fully operational PaaS solution.</p><h4>Days 0–30: Baseline &amp; Controls</h4><p>We start by agreeing the big picture: what you want your prices to achieve, the guardrails you’re comfortable with (floors, ceilings, and approval rules), and how we’ll measure success. In parallel, we do a light data handshake to confirm what’s available and how often it can refresh. Then we build your baseline: price realization, a margin waterfall, win-rate/churn, and a short list of “high-leverage” moves to test first (for example, disciplined indexation, a cleaner discount ladder, or targeted save-offers).</p><p>To make this initial process be repeatable and more reliable we use an established method for analyzing how customer and product centric the industry is, as well the degree of personalisation you can can offer on the pricing side. Read more about that <a href="https://www.linkedin.com/pulse/data-driven-pricing-you-products-people-hans-christian-ekne-n5gle/">here.</a></p><h4>Days 31–60: Pilot &amp; Prove</h4><p>Next we run small, controlled pilots on selected segments/SKUs/regions. Each pilot includes a comparison group (“holdout”) so we can show impact with real numbers, not opinions. We stand up simple live dashboards so your team can see uplift as it happens (e.g., margin points, win-rate changes, promo ROI), and we document how exceptions will be handled and who approves what (your SLAs).</p><h4>Days 61–90: Scale &amp; Embed</h4><p>With evidence in hand, we roll out price lists/rate cards, automate the data refresh (API or secure file drops), and set up a monthly price council with clear inputs, decisions, and outputs. We finalize the governance pack (who can change what, and when) and align on the KPIs we’ll report each month.</p><h4>Right-sized, not one-size</h4><p>Does goes getting started with PaaS always take 90 days? No. We tailor scope and investment to your maturity. I’ve done pricing projects that took less than a month and others that stretched closer to half a year. Implementation speed largely depends on the complexity of the business, the number of stakeholders and the organizational willingness to implement.</p><p>Early-stage teams like startups or scale-ups might start with a lightweight data hookup and weekly price lists; mature enterprises may want multi-market rate cards, APIs into ERP/CRM, and formal approval workflows. Same path, different load: we scale the depth of analysis, the level of automation, and the governance to fit where you are today.</p><h3>Ready to See It in Your Numbers?</h3><p>Pricing is too important to be ad‑hoc and too operational to consume your leadership time every month. Keep the strategy where it belongs — with you. Let a service run the machinery so your prices are always on‑strategy, always current, and always earning their keep.</p><p>Curious about PaaS for your business (or a portfolio company)? Message me for a 30‑minute fit check.</p><p><strong>Want to be notified whenever I publish a new article? ➡️ </strong><a href="https://www.linkedin.com/newsletters/ekne-insights-7266419471009951744/?lipi=urn%3Ali%3Apage%3Ad_flagship3_pulse_read%3BEEfV9UAzQ%2F2osSrzqWon0Q%3D%3D"><strong>Subscribe to my LinkedIn newsletter here</strong></a><strong> ⬅️. It’s free &amp; you can unsubscribe at any time!</strong></p><p><em>If you enjoyed reading this article and would like to access more content from me please feel free to connect with me on LinkedIn at </em><a href="https://www.linkedin.com/in/hanschristianekne/"><em>https://www.linkedin.com/in/hanschristianekne/</em></a> <em>or visit my webpage at </em><a href="https://www.ekneconsulting.com/"><em>https://www.ekneconsulting.com/</em></a><em> to explore some of the services I offer. Don’t hesitate to reach out via email at hce@ekneconsulting.com</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=996155cf9261" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are You Pricing for Products or People?]]></title>
            <link>https://medium.com/data-science-collective/are-you-pricing-for-products-or-people-ba9495af40d4?source=rss-ab0bff3c2cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba9495af40d4</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[pricing-strategy]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hans Christian Ekne]]></dc:creator>
            <pubDate>Tue, 26 Aug 2025 09:32:59 GMT</pubDate>
            <atom:updated>2025-08-26T09:32:59.928Z</atom:updated>
            <content:encoded><![CDATA[<h4>DATA DRIVEN PRICING</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fBlS14L7rG22YfYQpdHi2g.png" /><figcaption>Customer-centric vs product-centric pricing / Image generated by author using GPT-5</figcaption></figure><p>Two companies can sell the same product and have the same costs, but one makes 30% more profit. The difference? How they set their prices. Pricing is the final touchpoint of your commercial strategy, the moment where all your marketing, segmentation, and advertising efforts are put to the test. Get it wrong and the effort is wasted; get it right and you create a win-win for your business and your customer.</p><p>So how do you get pricing “right”? Increasingly, companies are turning to data-driven approaches. And in practice, these approaches often fall along a spectrum with two distinct poles: <strong>customer-centric pricing</strong> vs <strong>product-centric pricing.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ox_jzpXkh9XaViia91SoVQ.png" /><figcaption>Customer-Centric vs Product-Centric Pricing</figcaption></figure><h3>Customer-Centric Pricing</h3><p>In customer-centric pricing, the customer is at the heart of the equation. The goal is to understand each customer’s price sensitivity and willingness to pay, then use that insight to tailor offers or prices.</p><p>This approach is common in industries with relatively few products but very large customer bases, for example utilities, telcos, or online services like LinkedIn, which uses behavioural data to decide which tier, offer, or trial period to show to each user. The process often starts with purchase history, enriched with profile information such as demographics, usage, or engagement, to build models that predict individual elasticities.</p><blockquote>Think of a telco offering 10 products to millions of customers — customer-centric pricing means figuring out which customer is willing to pay a premium for unlimited data, and which one will only upgrade if offered a short-term discount.</blockquote><p>We do not need long purchase histories for everyone, and often we only have one or two observations for each customer, but by pooling transactions across a large customer base, we can model how different factors affect purchase likelihood. Using the prices and purchasing events as the response variable, you estimate demand curves using regression or classification models (these models model the probability of a buy/churn event and are then translated back into elasticities), then simulate different price or discount options to find the one that maximizes profit, revenue, or other goals. This can be done in batch (e.g., setting quarterly prices) or dynamically in real time.</p><p>While this method shines when there are limited products, the opposite challenge arises in industries with thousands of products, where the focus shifts from tailoring prices for people to optimising prices for the products themselves.</p><h3>The Product-Centric Approach to Pricing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ei0rud_gvJWQL2bOC86OMA.png" /><figcaption>Product-centric pricing / Image generated by author using GPT-5</figcaption></figure><p>If you sell thousands of products, pricing each one individually is impossible without data. Product-centric pricing starts with the products, and is common in retail, grocery, and other industries with large SKU counts. The aim is to understand how price changes affect sales and use that insight to set prices and plan promotions. Below, I give a high-level view of how to identify price elasticities, map product relationships, and use this to inform an effective promotion calendar.</p><h4>How Your Sales History Reveals Price Sensitivity</h4><p>The first step is to work out how sensitive each product’s sales are to changes in its own price — its <em>own-price elasticity</em>. This can be estimated from transaction history by looking at how sales vary at different price points over time. When some products don’t have enough price variation to give a stable estimate, we can “borrow strength” from similar items in the same category, pooling the data so that all products get a sensible baseline elasticity. Hierarchical Bayesian models are a subset of models well suited for this task. Sometimes we often also need to group similar items together to get enough observations for statistically significant results.</p><p>Once we have each product’s own-price elasticity, the next step is to see how changes in one product’s price affect demand for others — the <em>cross-price elasticities</em>. This is where we find the substitutes and complements: does putting Romaine lettuce on sale steal volume from Iceberg or does it boost tomato sales because people make more salads? The result is a cross-elasticity matrix, with own-price effects on the diagonal and cross-effects off-diagonal.</p><h4>Building a Data-Driven Discount Calendar</h4><p>Using the cross-elasticity matrix, we can forecast the uplift and profit impact of any potential promotion, not just for the promoted item but also simultaneously for the rest of the portfolio. We can define realistic promotion “options” for each SKU — e.g. a modest discount, a deep discount, a multi-buy — and attach estimated sales lifts and margin impacts to each, based on the elasticities. These options, along with business constraints (limited promo slots, endcap space, budget caps, category rules, seasonality), feed into an optimisation model.</p><p>Using an optimisation model such as a mixed-integer optimiser allows us to search across all possible combinations of SKU–week–promotion choices to find the promotion calendar that maximizes profit (or revenue, or traffic), while respecting the constraints. Because the optimiser “knows” about both own- and cross-effects, it avoids wasteful combinations (like discounting two near-identical items in the same week) and finds combinations that lift total basket spend. The result is a data-driven calendar that is optimal given both the elasticity model and the real-world limits the business operates under.</p><h3>Hybrid Approaches: Blending Customer and Product Views</h3><p>Of course, the world is not black and white, and most businesses do not sit neatly at one end of the customer-product spectrum. In practice, many combine elements of both customer and product-centric pricing — using product-level analysis to set the structure, and customer-level insights to fine-tune execution.</p><p>For example, in SaaS or telco with a small portfolio, product-centric pricing means optimising the product options themselves: plan prices, bundles, and their trade-offs. Then, customer-centric tactics personalise who gets which offer (trial length, intro discount, renewal save-offer). In other words: set a strong rate card at the product level, then tailor the offer at the customer level.</p><p>Another example would be in B2B energy and utilities where the product-centric element might be setting standard tariff structures, peak/off-peak rates, and contract menus based on cost and market factors. The customer-centric element comes in when tailoring offers for specific clients, adjusting terms for a factory with predictable base load versus a retailer with volatile demand, offering green energy bundles to sustainability-focused accounts or tweaking the price based on our knowledge of competitors in the bidding process. Like in the SaaS example, design the market-facing options at the product level, then use customer data to customise the deal. This is also where personalisation becomes a powerful lever, taking a well-structured product offer and tailoring it to the right customer at the right time.</p><h3>The Twist: Personalisation Adds another Dimension</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OVXIwjDj1JmlRXy4WXoaCQ.png" /><figcaption>2D-pricing matrix, Customer/Product-Centric vs Personalization</figcaption></figure><p>So far, we have looked at the spectrum between customer-centric and product-centric pricing. But there is another layer worth considering: the degree of personalisation. This measures how much the price or offer is tailored to the individual, versus being the same for everyone.</p><p>Some industries are obvious fits: airlines, hotels, and digital advertising are masters of personalising offers. But counterexamples show the picture is not so clean. Some highly customer-centric industries, like regulated utilities, have very low degrees of personalisation. (Although that doesn’t mean they cannot benefit from optimal pricing across their product ranges.) And some traditionally product-centric industries, like luxury fashion or marketplaces, are introducing targeted deals and loyalty perks that edge them toward the “personalised” side.</p><p>In a traditional mass market industry such as grocery retail, personalisation has already been around for years in some geographies. For example in UK supermarkets, the top players have long sent vouchers for items shoppers regularly buy. What’s changing is the sophistication and reach of these tactics. Instead of just discounting the same products you’ve bought before, retailers can now predict what might trigger your next visit — say, a personal coupon for a new craft beer similar to your usual brand — and deliver it through an app or loyalty card system. Done well, this doesn’t just increase sales of that item; it can lift the entire basket because once the shopper is in the store, they buy more.</p><p>Why is personalisation especially interesting now?</p><ul><li>Tech costs have dropped</li><li>We can use AI to make personalisation feel <em>truly personal</em></li><li>Loyalty data collection is easier than ever before</li><li>Integration into everyday channels like email, apps and social media</li><li>Customer experience shifts (Customers are expecting more as everything is becoming more personal)</li></ul><p>Personalisation is no longer tied to being “customer-centric” in the traditional sense. Even the most SKU-driven businesses can incorporate personalised levers to drive incremental value.</p><h3>Factoring in the Competition</h3><p>Like all strategy, pricing does not happen in a vacuum and apart from just considering how your own customers have historically reacted to your prices, it is also critical to consider the competitors.</p><p>Competitor prices influence customer expectations, perceived value, and ultimately the elasticity of your own products. In some industries, competitor monitoring is almost real-time, for example e-commerce, where price-matching and undercutting algorithms run constantly. In others, competitor prices change less frequently but still shape the “reference price” in customers’ minds.</p><p>Integrating competitor pricing into your models can be as simple as including competitor price indices in your regression models, or as sophisticated as simulating “game theory” scenarios where you anticipate and model competitor reactions to your own price moves. The importance varies by industry — telcos may change prices rarely, but airlines can update fares multiple times a day — so the level of integration should match your competitive environment.</p><h3>The way forward</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hf0__vI-lqKybbIYcQYHFA.png" /><figcaption>Personlised pricing / Image generated by author using GPT-5</figcaption></figure><p>Data-driven pricing is not just for the big players with endless resources — it’s a practical advantage any business can use. Whether you lean toward customer-centric models, product-centric analysis, or a blend of the two, the real results come from combining price elasticity insights, cross-product effects, and the right level of personalisation for your market. Done well, it means pricing that works better for your customers and your bottom line.</p><p>Drop a comment below or send me a message to continue the conversation on how data-driven pricing can unlock new value in your business.</p><p><strong>Want to be notified whenever I publish a new article? ➡️ </strong><a href="https://www.linkedin.com/newsletters/ekne-insights-7266419471009951744/?lipi=urn%3Ali%3Apage%3Ad_flagship3_pulse_read%3BEEfV9UAzQ%2F2osSrzqWon0Q%3D%3D"><strong>Subscribe to my LinkedIn newsletter here</strong></a><strong> ⬅️. It’s free &amp; you can unsubscribe at any time!</strong></p><p><em>If you enjoyed reading this article and would like to access more content from me please feel free to connect with me on LinkedIn at </em><a href="https://www.linkedin.com/in/hanschristianekne/"><em>https://www.linkedin.com/in/hanschristianekne/</em></a> <em>or visit my webpage at </em><a href="https://www.ekneconsulting.com/"><em>https://www.ekneconsulting.com/</em></a><em> to explore some of the services I offer. Don’t hesitate to reach out via email at hce@ekneconsulting.com</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba9495af40d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/are-you-pricing-for-products-or-people-ba9495af40d4">Are You Pricing for Products or People?</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Integrate AI and Data Science into Your Business Strategy]]></title>
            <link>https://medium.com/data-science/how-to-integrate-ai-and-data-science-into-your-business-strategy-29767ca82b8f?source=rss-ab0bff3c2cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/29767ca82b8f</guid>
            <category><![CDATA[consulting]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[strategy]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[workshop]]></category>
            <dc:creator><![CDATA[Hans Christian Ekne]]></dc:creator>
            <pubDate>Fri, 06 Dec 2024 15:02:11 GMT</pubDate>
            <atom:updated>2024-12-06T15:02:11.421Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@hc.ekne/list/data-science-consulting-bcf6e4b59652">DATA SCIENCE CONSULTING</a></h4><h4>Insider consulting guide to conducting a successful 2-day executive workshop</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bMWMuXGiP8aTbvCxLGULvQ.png" /><figcaption>Image by author using Canva</figcaption></figure><blockquote><a href="https://news.microsoft.com/2014/02/04/satya-nadella-email-to-employees-on-first-day-as-ceo/">“Our industry does not respect tradition — it only respects innovation.” </a>— <em>Satya Nadella, CEO Microsoft, Letter to employees in 2014</em></blockquote><p>While not all industries are as competitive and cutthroat as the software and cloud industries, innovating and applying the latest technological developments is a fundamental theme for executives. Seasoned business leaders understand that staying up-to-date with the relevant technologies is necessary for success.</p><p>As a data science consultant, some of the questions clients often ask me are: “How do we effectively integrate the right AI and machine learning tools into our business?”, and “How do we prioritize our AI initiatives, and integrate them with our broader company strategy?”. Especially now, after the latest AI-boom, these questions are higher on the agenda and seem even more urgent than before.</p><p>What makes these questions difficult is that good answers requires both knowledge of the technological innovations, but also domain and business expertise. In addition, it requires a fundamental understanding of the current company strategy in order to prioritize and select initiatives. As such, a comprehensive strategy workshop with the executive leadership of a company, or a division, is one of the best ways to uncover the answers.</p><p>In this article, I share a blueprint for how to conduct a 2-day strategy workshop with the aim of figuring out how to best apply AI and data science tools to a business. I cover everything, from what needs to be done to prepare, who should attend, how to identify the right topics for deep dives, what needs to be done after the workshop, and much more. The idea is that this can be used as a template to conduct a workshop in any industry for a company of almost any size.</p><p>I have worked a lot with energy and financial services companies in my years as a consultant, so you will find example cases from those industries throughout the article, however the blueprint is by design industry agnostic, and the methods and principles are general in nature.</p><h3>Preliminary Work</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oE6bSW_W5m70JaqPtQk9AA.png" /><figcaption>Image generated by the author using DALL-E</figcaption></figure><p>Most of the work associated with a workshop like this is actually done before the workshop even starts. To quote one of my favorite inventors and statesman:</p><blockquote>“By failing to prepare, you are preparing to fail.” — Benjamin Franklin</blockquote><h4>Functional Areas Research and Alignment</h4><p>Depending on your level of industry knowledge, be prepared to put in a lot of time on pre-workshop research. There are several topics that need to be addressed before you can draft the outline for the workshop.</p><ul><li><strong>A high-level understanding of the industry: </strong>Who are the major players, what are the key drivers, what are the trends, and what are the current challenges</li><li><strong>Functional business areas:</strong> Thoroughly investigate what are the key functional business areas for the business you are working with and then do a deep dive in each of these</li></ul><p>Try to segment the functional areas one level down to get a more granular view. Using an energy utility as an example, a typical list of functional areas could be like the list below:</p><ol><li><strong>Power Generation and Energy Resources Management:</strong> Traditional Power Plants, Renewable Energy (Solar, Wind, Hydro), Distributed Generation, Energy Storage Systems, Generation Optimization</li><li><strong>Grid Management and Asset Maintenance:</strong> Transmission Networks, Distribution Networks, Smart Grid Technologies, Predictive Maintenance, Outage Management, Asset Lifecycle Management</li><li><strong>Customer Base Management, Marketing, and Sales:</strong> Customer Service, Billing and Payments, Customer Relationship Management (CRM), Marketing Campaigns, Sales Operations, Customer Analytics</li><li><strong>Energy Trading, Market Operations, and Risk Management:</strong> Energy Procurement, Wholesale Trading, Price Forecasting, Market Analysis, Hedging Strategies, Risk Assessment</li><li><strong>Supply Chain Management and Operational Efficiency:</strong> Procurement, Supplier Management, Inventory Management, Logistics, Process Optimization, Cost Reduction</li><li><strong>Finance, Compliance, and Regulatory:</strong> Financial Planning, Budgeting, Accounting, Regulatory Compliance, Auditing, Government Relations, Policy Advocacy</li><li><strong>Human Resources and Workforce Management:</strong> Talent Acquisition, Training and Development, Employee Engagement, Performance Management, Workforce Planning, Health and Safety</li><li><strong>Information Technology, Cybersecurity, and Innovation:</strong> IT Infrastructure, Cybersecurity Measures, Data Analytics, Business Intelligence, Innovation Programs, Research and Development (R&amp;D), Emerging Technologies (IoT, AI, Blockchain)</li><li><strong>Environmental Sustainability and Corporate Social Responsibility:</strong> Emission Reduction Initiatives, Sustainability Reporting, Environmental Compliance, Renewable Energy Certificates, Community Engagement, CSR Programs</li></ol><p>You have now completed the first part of the research and should, ideally, align with the client as to whether this list is what they want to focus on or if they want to expand on some areas while excluding others. The above structure will help you specify the agenda for the workshop in more detail and also help steer the rest of the research for the workshop.</p><h4>Functional Area Deep Dives</h4><p>After aligning on the structure, we can start doing deep dives into each of the subcategories to understand where and how AI and data science is being applied to generate value. This is usually where I need to spend the most time on research.</p><p>I typically start out with specific queries, like: “How is AI being used in power generation, specifically in wind generation?” Results for this query might yield the following topics:</p><ul><li>Use of AI and quantum computing to better understand how to plan and optimize turbine location in onshore wind farms</li><li>Time series modelling for fault detection and diagnostics for turbines</li><li>Time series modelling for predictive maintenance for turbines</li></ul><p>If available, try to also quantify the possible value that comes from using the technology. For example, if Equinor, an energy company, was able to reduce unplanned downtime of wind turbines by 40% after implementing a predictive maintenance project, how does this translate into monetary value? How would this example translate into your specific business if you for example had a wind farm with 1000 wind turbines? The quantification aspect is important because it will help in the later work of prioritizing initiatives.</p><p>At this research stage, it is also OK to think outside the box and perhaps explore how a specific technology could be borrowed from one industry to another. Many technologies start out being used in one industry and then transition into others with similar functional areas. For example, data driven churn management started out being used by the telco and banking companies but was quickly adopted in almost all industries.</p><h4>Drafting the Agenda</h4><p>With an understanding of the industry, functional business areas, and technological possibilities, it’s time to draft an agenda for the workshop.</p><p>For a two-day workshop, I would recommend at least 30 minutes for an introduction to present the workshop and its goals. I would also schedule time to review pre-workshop findings, as this gives the participants insights into their collective a priori views, expectations and prioritizations. The rest of the workshop we would then be devoted to sessions on the selected functional areas. Finally, end the workshop with a summary of the topics covered and next steps.</p><p>A 2-day workshop with 9 functional area deep dives, could be planned using the structure below:</p><p><strong>Day 1</strong></p><p>9:00 AM — 9:30 AM: Welcome and Introduction<br>9:30 AM — 10:00 AM: Review of Pre-Workshop Findings<br>10:15 AM — 11:30 PM: Session 1<br>1:00 PM — 2:15 PM: Session 2<br>2:30 PM — 3:45 PM: Session 3<br>4:00 PM — 5:15 PM: Session 4<br>5:15 PM — 5:30 PM: Day 1 Wrap-Up</p><p><strong>Day 2</strong></p><p>9:00 AM — 9:15 AM: Recap of Day 1<br>9:15 AM — 10:30 AM: Session 5<br>10:45 AM — 12:00 PM: Session 6<br>1:00 PM — 2:15 PM: Session 7<br>2:30 PM — 3:45 PM: Session 8<br>4:00 PM — 5:15 PM: Session 9<br>5:15 PM — 5:45 PM: Final Wrap-Up and Next Steps</p><p>The above structure leaves room for breaks between the sessions and uses the time effectively to run through each of the different functional areas. In each of the sessions I will typically spend time on the following:</p><ul><li>Interactive discussion on current processes</li><li>Presentations of case studies and feasibility analysis</li><li>Brainstorming on further AI and data science development</li><li>Prioritizing key initiatives</li></ul><h4>Involving the Right People</h4><p>Given the technical nature of AI and data science, the CTO or similar executive role is the natural contact point for the workshop. You ideally want someone who really understands the business from a technological point of view and is senior enough to command the attention of the rest of the executive team.</p><p>In addition, for the results of the workshop to be meaningful, you typically want most of the senior leadership of the company to attend. It’s a red flag if the CEO or managing director can’t attend. If possible, reschedule to keep her attending at least part of the workshop.</p><h4>Pre-Workshop Interviews or Questionnaire</h4><p>To make sure the content at the workshop fits the maturity level, ambition, and general strategy of the company, it’s preferable to conduct interviews with the main players in the leadership team. (Well written questionnaires also work fine for this purpose.) This lets you understand how far along they are with AI and data science initiatives across parts of the business, and lets you tailor the content to that level.</p><p>For example, if they are highly mature and already have a well-tuned in-house data science team, you can have a much more aggressive strategy than if they are starting from scratch.</p><h4>The Slide Deck</h4><p>One of the reasons I switched from management consulting to data science was to avoid making too many PowerPoint slides (😅), but even as data scientists, it’s hard to escape the inexplicable pull of PowerPoint. Maybe you’ve switched to Canva at this point; nonetheless, the fact is that if you want the workshop to be effective, it’s critical to have a solid slide deck.</p><p>The presentation deck serves as the guide and reference point as you progress along the workshop, allowing you to visually represent the ideas and concepts you are exploring. A good slide deck that keeps you on track is essential for a successful workshop.</p><h4>Final Alignment on Content</h4><p>You should always get a final go-ahead on the content of the workshop before kick-off. Alignment with key stakeholders is important for a couple of reasons. Firstly, you ensure that the content is correct and relevant, and you can identify any knowledge gaps that you need to cover. Secondly, and perhaps most importantly, by involving key players in the planning process, you increase stakeholder buy-in and increase the chances of the workshop’s success.</p><h3>Facilitating the Workshop</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KcMfHXeXiGjRwlsWoE18qw.png" /><figcaption>Image generated by the author using DALL-E</figcaption></figure><p>Running the workshop should be relatively straightforward once all the preliminary steps have been performed, but there are a few key things to be aware of.</p><h4>The Facilitator Role</h4><p>In your role as facilitator, keep in mind that what are really looking for is engagement from the participants. You want to avoid the workshop turning into a facilitator presentation and monologue. The input of the participants is key to the success. They are typically the ones who have the deep industry knowledge, and as executives they also have the power to act on various initiatives.</p><p>Ultimately, their participation will help garner a feeling of ownership to the process and make future steps easier to implement.</p><h4>Time Management</h4><p>The agenda serves as a guide for how to manage time between the various topics, however, time management can still be challenging. It is natural that some topics spark more interest than others and this needs to be considered. Allow for adjusting your agenda if some discussions go on longer than expected and avoid rushing the participants through topics to meet the timeline.</p><h4>Remote Vs Onsite</h4><p>While it would be possible to run the workshop remotely, I would strongly advice to have the key participants together in the same room. There are plenty of times remote work is a good option, but this is not one of them.</p><p>Ideally, the meeting is also hosted on Teams or a similar platform so you can record the process and get a transcript of the workshop later. Before we had AI transcripts from meetings, I would always have a dedicated person taking notes to make sure we documented everything. If you don’t have satisfactory recording options, this should be considered.</p><h4>Tools and Artifacts</h4><p>One of my previous employers loved to use brown paper (large rolls of wide paper we could hang on the walls) and Post-it notes to engage participants and document results. I think this can be a good approach but is by no means necessary. Tools like digital white boards are also great to use. The main point is that you get engagement from the participants and that you document the findings.</p><h3>Post-Workshop Activities</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5cP3XTxJdFC0FOkXsip-Yg.png" /><figcaption>Image generated by author using DALL-E</figcaption></figure><p>Having concluded the workshop you now need to analyze all the findings and insights and draft a strategy document that can act as guide for further implementation work.</p><p>The key points that need to be included in this document are:</p><ul><li>A list of prioritized AI and data science opportunities.</li><li>A data and infrastructure assessment.</li><li>An AI and data science roadmap.</li></ul><p>Let’s break down each of the points above.</p><h4>A list of prioritized AI and data science opportunities</h4><p>After the workshop you should be able to compile a list of prioritized AI and data science opportunities that the company can focus on. The opportunities should be ranked according to their potential impact, difficulty of implementation, cost of implementation and alignment with business goals. This makes it easier to choose which activities and opportunities to pursue.</p><h4>Data and Infrastructure Assessment</h4><p>Once all the various opportunities have been identified you can begin to understand how this will impact the current data and IT infrastructure. Unless the organization already is at a high maturity level with respect to using AI and data science, there might be significant steps that need to be taken to upgrade the infrastructure. If for example one of the prioritized activities is to start doing predictive maintenance on wind turbines, you need start adding sensors to the turbines — if they don’t already have them installed — and create the data pipelines and data infrastructure to be able to digest the sensor data and format it into actionable time series data.</p><h4>AI and Data Science Roadmap</h4><p>Putting everything together in a plan, you can craft a roadmap that details out the steps, timelines and resources needed to implement the opportunities. For my timeline and resource allocation I prefer to use Gant charts. However, for a visual understanding of how the various activities fit together — under the different functional areas — I like to use a sun ray map. The map below visualizes how the different opportunities come together to make the complete transformation into the future state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bqbDWc94Xx4VgyQh0LqroQ.png" /><figcaption>Example sun ray transformation map by the author</figcaption></figure><h4>A Follow-Up Workshop to Align on Results</h4><p>My last step would be to schedule another workshop to align on the strategy document. The roadmap and prioritization of AI and data science initiatives that you have found, now needs to be agreed on by the leadership, and integrated into their overall strategy.</p><p>It is counterproductive to a have a separate AI and data science strategy, instead the aim should be to integrate their IT and AI initiatives into their company wide strategy.</p><h3>Concluding remarks</h3><p>By now, you should have a comprehensive guide for planning and executing a strategy workshop that identifies the most valuable AI and data science opportunities for your business.</p><p>We have gone into detail as to how to prepare a workshop, including:</p><ul><li>Splitting company activities into functional areas</li><li>Investigating how AI and data science can be applied to each area</li><li>Drafting an agenda to allocate time effectively</li><li>Identifying the key people to involve in the process</li></ul><p>We also covered how to run the workshop effectively, emphasizing good facilitation, time management, the use of appropriate tools, and the benefits of conducting the workshop onsite versus remotely.</p><p>A workshop like we discussed in the article can be an important first step in integrating AI and data science into your business strategy. It helps secure executive alignment and is a starting point for a transformation journey.</p><p>Thanks for reading!</p><p><strong>Want to be notified whenever I publish a new article? ➡️ </strong><a href="https://ekneconsulting.com/register_subscriber/"><strong>Subscribe to my newsletter here</strong></a><strong> ⬅️. It’s free &amp; you can unsubscribe at any time!</strong></p><p><em>If you enjoyed reading this article and would like to access more content from me please feel free to connect with me on LinkedIn at </em><a href="https://www.linkedin.com/in/hans-christian-ekne-1760a259/"><em>https://www.linkedin.com/in/hans-christian-ekne-1760a259/</em></a><em> or visit my webpage at </em><a href="https://www.ekneconsulting.com/"><em>https://www.ekneconsulting.com/</em></a><em> to explore some of the services I offer. Don’t hesitate to reach out via email at hce@ekneconsulting.com</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29767ca82b8f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-integrate-ai-and-data-science-into-your-business-strategy-29767ca82b8f">How to Integrate AI and Data Science into Your Business Strategy</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[16 Ways To Generate Value From Churn Models]]></title>
            <link>https://medium.datadriveninvestor.com/16-ways-to-generate-value-from-churn-models-e3f0d539198c?source=rss-ab0bff3c2cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3f0d539198c</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[churn-prediction]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[customer-management]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Hans Christian Ekne]]></dc:creator>
            <pubDate>Thu, 28 Nov 2024 15:15:22 GMT</pubDate>
            <atom:updated>2024-11-29T06:26:15.239Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@hc.ekne/list/customer-base-management-32bbc1c1cf76">CUSTOMER BASE MANAGEMENT</a></h4><h4>Are You Maximizing the Value From Your Churn Models?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d7Wqrvica83Ge8ktWHTjLQ.png" /><figcaption>Image generated by author using Canva</figcaption></figure><p>Unlike Gen AI, churn models have been around for quite a while. Probably since the mid 1990s. In the fauna of the data science world they are comparable to dinosaurs at this point.</p><p>Given their relatively long history in the data and analytics space, you might think we had exhausted all the possible value from these models and that there weren’t any opportunities left? Wrong. Traditional machine learning methods — like churn modelling — still have plenty of gas left, and most companies are not near to utilizing them to their full potential. These models have the potential to improve businesses for years to come.</p><p>Below I have collected a selection of 16 different ways companies can utilize churn models to create value. The topics are diverse and range from retention campaigns to financial planning, but the ability to understand when a customer is likely to leave remains the central theme.</p><h3>1. Contract Renewal</h3><ul><li><strong>Use Case:</strong> Identify customers close to the end of their contract who are likely to switch to competitors</li><li><strong>Example: </strong>A B2B IT service provider creates special renewal offers for at-risk clients.</li><li><strong>Example 2: </strong>A magazine with a subscription services contacts customers who are predicted to churn and offers them early renewal benefits.</li></ul><h3>2. Product Pricing &amp; Price Elasticity</h3><ul><li><strong>Use Case: </strong>Optimize pricing strategies and products to match customer price elasticity (model churn as a function of price)</li><li><strong>Example:</strong> An energy company uses a churn model with price as an explanatory variable to optimize their product pricing in line with their strategic objectives.</li></ul><h3>3. Loyalty Card Program</h3><ul><li><strong>Use Case: </strong>Use promotions, discounts and special bonus points with loyalty programs to motivate customers at risk of churn.</li><li><strong>Example:</strong> An airline rewards bonus loyalty points to customers who have not flown in a while.</li></ul><h3>4. Financial Planning</h3><ul><li><strong>Use Case: </strong>Forecast future revenue based on expected churn rates.</li><li><strong>Example:</strong> A utility uses the churn rates to accurately forecast the next years revenue from a cohort of customers.</li></ul><h3>5. Scenario Building</h3><ul><li><strong>Use Case: </strong>Companies can predict how their future customer base will evolve under various future scenarios and develop robust strategies to deal with the expected outcomes.</li><li><strong>Example:</strong> An energy company wants to understand how its customer base will develop depending on the future market price of electricity and uses a churn function that has the price of power as an explanatory variable to estimate multiple different scenarios.</li></ul><h3>6. Promotional Campaigns</h3><ul><li><strong>Use Case: </strong>Use a churn model to identify customers at high risk for churn and target them with specific offers and promotions</li><li><strong>Example:</strong> An online retailer creates a targeted promotional campaign towards high-risk churn customers</li></ul><h3>7. Optimal Sourcing</h3><ul><li><strong>Use Case: </strong>By using a combination of short-term and long-term churn forecasting, companies can adapt more flexible sourcing strategies for products that need to be delivered to their customers.</li><li><strong>Example:</strong> An energy company switches to short-term sourcing of back-end energy contracts and while using 1-month churn models to predict demand for energy, this lets them exploit the flexibility of short-term sourcing to increase expected profits on fixed-term contracts.</li></ul><h3>8. Workforce planning</h3><ul><li><strong>Use Case: </strong>Churn predictions help inform staffing requirements for sales and customer support roles.</li><li><strong>Example:</strong> A B2B SaaS company can adjust its call center workforce based on predicted number of customers that haven’t churned.</li></ul><h3>9. Segmented Product Creation</h3><ul><li><strong>Use Case: </strong>Product creation based on customer segments at differing risk of churn, optimizing revenues and retention within each segment.</li><li><strong>Example:</strong> A teclo company segments customers based on churn risk and creates different products to cater to the different categories of customers.</li></ul><h3>10. Customer Life Time Value (CLV) Estimation</h3><ul><li><strong>Use Case:</strong> Combine churn predictions with customer value data to focus retention efforts on high-value customers.</li><li><strong>Example:</strong> A P&amp;C insurance company uses the churn risk together with the risk of accident to predict the value of individual customers and of the total customer portfolio.</li></ul><h3>11. Targeted Discounting</h3><ul><li><strong>Use Case:</strong> Offer targeted discounts to customers at high risk of churning</li><li><strong>Example:</strong> At risk customers at a telco are offered specific discounts to mitigate the churn risk.</li><li><strong>Example 2:</strong> A subscription-based video streaming service offers a temporary price reduction to retain customers.</li></ul><h3>12. Lead Generation</h3><ul><li><strong>Use Case: </strong>Build churn models on existing customers and score prospects, identity the best potential new customers with low churn risk.</li><li><strong>Example:</strong> A financial services group uses the churn risk of its banking customers to identify possible leads for their insurance products.</li></ul><h3>13. Company Value Estimation</h3><ul><li><strong>Use Case: </strong>Use churn rates to predict future cash flow.</li><li><strong>Example:</strong> An PE company is considering the takeover of gym chain. It uses the churn risk of the existing customer base together with the acquisition rate and expected monthly membership fee to determine the future discounted cash flow of the gym.</li></ul><h3>14. Upselling and Cross-Selling</h3><ul><li><strong>Use Case:</strong> Use churn predictions to identify opportunities for cross-selling or upselling complementary products or services.</li><li><strong>Example:</strong> An online retailer offers related products to customers who haven’t purchased recently.</li></ul><h3>15. Product or Service Improvements</h3><ul><li><strong>Use Case:</strong> Use churn risk along with customer surveys to identify pain points or dissatisfaction areas</li><li><strong>Example:</strong> A B2B SaaS company creates a better onboarding process base on feedback from high churn risk customers.</li></ul><h3>16. Proactive Support Outreach</h3><ul><li><strong>Use Case:</strong> Flag customers for personalized support based on their churn risk.</li><li><strong>Example:</strong> A B2C software company reaches out to users who encountered frequent issues and have an increased churn rate.</li></ul><p>Having reviewed the list above, do you honestly think your company is using churn models to their full advantage? Before planning that next Gen AI project, consider getting value from some proven machine learning models first.</p><p>Did I miss anything? Is your company using churn models in a different or unique way? I would love to hear about it! Let me know in the comments!</p><h3>Special Note to European Readers</h3><p>While most of the use cases below are fine to use, some of them might intersect with current GDPR regulation. Especially for B2C, some of the more targeted approaches can be hard to implement without jumping through a lot of hoops related to consent etc. B2B customers are generally expected to professionals so the regulation is more relaxed for them.</p><p>Check with your legal team if you are unsure, or send me a private message, and I’ll be happy to have a discussion. I’ve advised multiple companies on how to stay within the bounds of GDPR while still retaining value from their churn and customer base management models.</p><p><strong>Want to be notified whenever I publish a new article? ➡️ </strong><a href="https://ekneconsulting.com/register_subscriber/"><strong>Subscribe to my newsletter here</strong></a><strong> ⬅️. It’s free &amp; you can unsubscribe at any time!</strong></p><p><em>If you enjoyed reading this article and would like to access more content from me please feel free to connect with me on LinkedIn at </em><a href="https://www.linkedin.com/in/hans-christian-ekne-1760a259/"><em>https://www.linkedin.com/in/hans-christian-ekne-1760a259/</em></a><em> or visit my webpage at </em><a href="https://www.ekneconsulting.com/"><em>https://www.ekneconsulting.com/</em></a><em> to explore some of the services I offer. Don’t hesitate to reach out via email at hce@ekneconsulting.com</em></p><p>Visit us at <a href="https://www.datadriveninvestor.com/"><em>DataDrivenInvestor.com</em></a></p><p>Subscribe to DDIntel <a href="https://www.ddintel.com/"><em>here</em></a>.</p><p>Join our creator ecosystem <a href="https://join.datadriveninvestor.com/"><em>here</em></a>.</p><p>DDI Official Telegram Channel: <a href="https://t.me/+tafUp6ecEys4YjQ1">https://t.me/+tafUp6ecEys4YjQ1</a></p><p>Follow us on <a href="https://www.linkedin.com/company/data-driven-investor"><em>LinkedIn</em></a>, <a href="https://twitter.com/@DDInvestorHQ"><em>Twitter</em></a>, <a href="https://www.youtube.com/c/datadriveninvestor"><em>YouTube</em></a>, and <a href="https://www.facebook.com/datadriveninvestor"><em>Facebook</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3f0d539198c" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/16-ways-to-generate-value-from-churn-models-e3f0d539198c">16 Ways To Generate Value From Churn Models</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build a Data-Driven Customer Management System]]></title>
            <link>https://medium.com/data-science/how-to-build-a-data-driven-customer-management-system-7d1597998dd5?source=rss-ab0bff3c2cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d1597998dd5</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[pricing]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Hans Christian Ekne]]></dc:creator>
            <pubDate>Wed, 20 Nov 2024 18:14:16 GMT</pubDate>
            <atom:updated>2024-11-20T19:00:03.578Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@hc.ekne/list/customer-base-management-32bbc1c1cf76">CUSTOMER BASE MANAGEMENT</a></h4><h4>A High-Level Framework for Building a CBM System with Strategic Impact</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7-akiPHQl6b3zF-dVWGGtQ.png" /><figcaption>Image created by the author using Canva</figcaption></figure><p>Imagine if your customer data could tell a story — one that drove key decisions, optimized pricing, and forecast future revenue with accuracy.</p><p>As a data scientist, I have spent a large part of my career designing and building customer base management (CBM) systems. These are systems that monitor and predict anything from customer churn and customer price elasticity to recommending the products customers are most likely to buy.</p><p>I have witnessed how CBM systems have been successfully adopted by organizations, and enabled them to execute their strategy through effective pricing, improved retention and more targeted sales activities. Combining forecasting with the visual effect of dashboards, CBM systems can provide an effective way for managers to communicate with executive leadership. This enhances decision-making and helps leaders better understand the consequences of their actions, particularly regarding the customer base and projected future revenues from that base.</p><p>In this article we will explore both the foundational components of a CBM system, as well as the more advanced features that really ramp up the value generation. By the end of this guide, you’ll have a clear understanding of the key components of a CBM system and how advanced modules can enhance these foundations to give your company a competitive edge.</p><h3><strong>The Foundation of a Strong CBM System</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*__8VKaanNP7Fyo0Eyz3Ceg.png" /><figcaption>Image created by the author using Canva</figcaption></figure><p>The three main foundational components of a CBM systems are:</p><ul><li><strong>ELT (extract-load-transform)</strong></li><li><strong>Basic Churn Modelling</strong></li><li><strong>Dashboards</strong></li></ul><p>With these three elements in place, managers are afforded a basic understanding of churn, a visual understanding of the data, and furthermore it allows them to communicate any findings to the leadership and other stakeholders. Below we detail each of these components in depth.</p><h4>ELT — Getting the Data</h4><p>Extract-Load-Transform, ELT for short, is the first and most critical part of a customer base management system. This is the component that feeds the system with data, and is often the first component to be built when creating a CBM system. To get started, you will typically interact with some kind of data platform where most of the rudimentary data manipulation has already been performed (in this case you are technically only required to do the “load” and “transform”), but sometimes you need get directly from source systems as well and “extract” the data.</p><p>Irrespective of where the data comes from, it needs to be loaded into your system and transformed into a format that is easy to input into the machine learning models you are using. You will also likely need to transform the data into formats that makes it easier to make dashboards from the data. For some dashboards it might also be necessary to pre-aggregate a lot of data in smaller tables to improve query and plotting performance.</p><p>It is easy to see that if there are mistakes in the ELT process, or if there are errors in the incoming data, this has the potential to severely affect the CBM system. Since everything in the system is based on this incoming data, extra care needs to be taken to ensure accuracy and alignment with business rules.</p><p>I have seen multiple times where the ELT process has failed and led to mistakes in the churn predictions and dashboard. One way to monitor the consistency of the data coming into the system is to keep records of the distribution of each variable and track that distribution over time. In case of radical changes to the distribution you can then quickly check whether something is going wrong with the logic in the ELT process or if the problem is arising from data issues further upstream.</p><h4>Basic Churn Modelling</h4><p>The second critical component in understanding your customer base is identifying who, when and why customers churn (for non-practitioners, “churn” refers to the point at which a customer stops using a product or service). A good churn prediction algorithm allows businesses to focus retention efforts where they matter most, and can help identify customers that are at an elevated risk of leaving.</p><p>Back in the mid-1990s, telcos, banks, insurance companies and utilities where some of the first to start using churn modelling on their customer base, and developing basic churn models is relatively straightforward.</p><p>The first task at hand is to decide the definition of churn. In many cases this is very straightforward, for example when a customer cancels a telco contract. However, in other industries, such as e-commerce, one needs to use some judgement when deciding on the definition of churn. For example, one could define a customer as having churned if that customer had not had a repeat shop 200 days after their last shop.</p><p>After churn has been defined, we need to select and outcome period for the model, that is a time frame within which we want to observe churn. For example, if we want to create a churn model with outcome period of 10 weeks, that would gives a a model that would predict the likelihood that a customer would churn at any point between the time of scoring and the next 10 weeks. Alternatively, we could have an outcome period of a year, which would give us a model that predicted churn at any time within the next year.</p><p>After the outcome period and churn definition has been decided, analysts need to transform the data into a format which makes it easy for the machine learning models to train on and also later for inference.</p><p>After the models are trained and predictions are being run on the active customer base, there are multiple different use cases. We can for example use the churn scores to identify customers at high risk of leaving and target them with specific retention campaigns or pricing promotions. We can also create differentiated marketing material to different groups of customers based on their churn score. Or we can use the churn score together with the products the customer has to develop customer lifetime value models, which in turn can be used to prioritize various customer activities. It is clear that proper churn models can give a company a strategic advantage in how it manages it customer base compared to competitors who neglect this basic component of CBM.</p><h4>Dashboards</h4><p>Dashboards, BI, and analytics used to be all the rage back in the 2000s and early 2010s, before the maturity of machine learning algorithms shifted our focus toward prediction over descriptive and often backwards looking data. However, for a CBM tool, dashboards remain a critical component. They allow managers to communicate effectively with leadership, especially when used alongside advanced features like price optimization. Visualizing the potential impact of a specific pricing strategy, can be very powerful for decision-making.</p><p>As with any data science project, you may invest thousands of hours in building a system, but often, the dashboard is the main point of interaction for managers and executives. If the dashboard isn’t intuitive or doesn’t perform well, it can overshadow the value of everything else you’ve built.</p><p>Additionally, dashboards offer a way to perform visual sanity checks on data and can sometimes reveal untapped opportunities. Especially in the early phases after a system has been implemented, and perhaps before all control routines have been set into production, maintaining a visual check on all variables and model performance can act as a good safety net.</p><p>With the main foundational pieces in place we can explore the more advanced features that have the potential to deliver greater value and insights.</p><h3><strong>Enhancing Strategic Advantage with Advanced Components</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G3RjVqnZ5gWAvxCjBfQKqQ.png" /><figcaption>Image created by the author using Canva</figcaption></figure><p>Although a basic CBM system will offer some solid benefits and insights, to get the maximum value out of a CBM system, more advanced components are needed. Below we discuss a few of the most important components, such as having churn models with multiple time horizons, adding price optimization, using simulation-based forecasting and adding competitor pricing data.</p><h4>Multiple Horizon Churn Models</h4><p>Sometimes it makes sense to look at churn from different perspectives, and one of those angles is the time horizon — or outcome period — you allow the model to have. For some business scenarios, it makes sense to have a model with a short outcome period, while for others it can make sense to have a model with a 1-year outcome period.</p><p>To better explain this concept, assume you build a churn model with 10-week outcome period. This model can then be used to give a prediction whether a given customer will churn within a 10-week period. However, assume now that you have isolated a specific event that you know causes churn and that you have a short window of perhaps 3 weeks to implement any preventative measure. In this case it makes sense to train a churn model with a 3-week horizon, conditional on the specific event you know causes churn. This way you can focus any retention activities on the customers most at risk of churning.</p><p>This kind of differentiated approach allows for a more strategic allocation of resources, focusing on high-impact interventions where they’re needed most. By adapting the model’s time horizon to specific situations, companies can optimize their retention efforts, ultimately improving customer lifetime value and reducing unnecessary churn.</p><h4>Pricing Optimization &amp; Customer Price Elasticity</h4><p>Price is in many cases the final part of strategy execution, and the winners are the ones who can effectively translate a strategy into an effective price regime. This is exactly what a CBM system with prize optimization allow companies to do. While the topic of price optimization easily warrants its own article, we try to briefly summarize the key ideas below.</p><p>The first thing needed to get started is to get data on historic prices. Preferably different levels of price across time and other explanatory variables. This allows you to develop an estimate for price elasticity. Once that is in place, you can develop expected values for churn at various price points and use that to forecast expected values for revenue. Aggregating up from a customer level gives the expected value and expected churn on a product basis and you can find optimal prices per product. In more complex cases you can also have multiple cohorts per product that each have their optimal price points.</p><p>For example, assume a company has two different products, product A and product B. For product A, the company wishes to grow its user base and are only willing to accept a set amount of churn, while also being competitive in the market. However, for product B they are willing to accept a certain amount of churn in return for having an optimal price with respect to expected revenues. A CBM system allows for the roll out of such a strategy and gives the leadership a forecast for the future expected revenues of the strategy.</p><h4>Simulation-Based Forecasting</h4><p>Simulation based forecasting provides a more robust way generating forecast estimates rather than just doing point estimation based on expected values. By using methods like Monte Carlo simulation, we are able generate probability densities for outcomes, and thus provide decision makers with ranges for our predictions. This is more powerful than just point estimates because we are able to quantify the uncertainty.</p><p>To understand how simulation based forecasting can be used, we can illustrate with an example. Suppose we have 10 customers with given churn probabilities, and that each of these customers have a yearly expected revenue. (In reality we typically have a multivariate churn function that predicts churn for each of the customers.) For simplicity, assume that if the customer churns we end up with 0 revenue and if they don’t churn we keep all the revenue. We can use python to make this example concrete:</p><pre>import random<br># Set the seed for reproducibility<br>random.seed(42)<br><br># Generate the lists again with the required changes<br>churn_rates = [round(random.uniform(0.4, 0.8), 2) for _ in range(10)]<br>yearly_revenue = [random.randint(1000, 4000) for _ in range(10)]<br><br>churn_rates, yearly_revenue</pre><p>This gives us the following values for churn_rates and yearly_revenue:</p><pre>churn_rates: [0.66, 0.41, 0.51, 0.49, 0.69, 0.67, 0.76, 0.43, 0.57, 0.41]<br>yearly_revenue: [1895, 1952, 3069, 3465, 1108, 3298, 1814, 3932, 3661, 3872]</pre><p>Using the numbers above, and assuming the churn events are independent, we can easily calculate the average churn rate and also the total expected revenue.</p><pre># Calculate the total expected revenue using (1 - churn_rate) * yearly_revenue for each customer<br>adjusted_revenue = [(1 - churn_rate) * revenue for churn_rate, revenue in zip(churn_rates, yearly_revenue)]<br>total_adjusted_revenue = sum(adjusted_revenue)<br><br># Recalculate the expected average churn rate based on the original data<br>average_churn_rate = sum(churn_rates) / len(churn_rates)<br><br>average_churn_rate, total_adjusted_revenue</pre><p>With the following numbers for average_churn_rate and total_adjusted_revenue:</p><pre>average_churn_rate:0.56, <br>total_adjusted_revenue: 13034.07</pre><p>So, we can expect to have about 56% churn and a total revenue of 13034, but this doesn’t tell us anything about the variation we can expect to see. To get a deeper understanding of the range of possible outcomes we can expect, we turn to Monte Carlo simulation. Instead of taking the expected value of the churn rate and total revenue, we instead let the situation play out 10000 times (10000 is here chosen arbitrarily; the number should be chosen so as to achieve the desired granularity of the resulting distribution), and for each instance of the simulation customers either churn with probability churn_rate or they stay with probability 1- churn_rate.</p><pre>import pandas as pd<br><br>simulations = pd.DataFrame({<br>    &#39;churn_rate&#39;: churn_rates * 10000,<br>    &#39;yearly_revenue&#39;: yearly_revenue * 10000<br>})<br><br># Add a column with random numbers between 0 and 1<br>simulations[&#39;random_number&#39;] = (<br>    [random.uniform(0, 1) for _ in range(len(simulations))])<br><br># Add a column &#39;not_churned&#39; and set it to 1, then update it to 0 based on the random number<br>simulations[&#39;not_churned&#39;] = (<br>    simulations[&#39;random_number&#39;] &gt;= simulations[&#39;churn_rate&#39;]).astype(int)<br><br># Add an &#39;iteration&#39; column starting from 1 to 10000<br>simulations[&#39;iteration&#39;] = (simulations.index // 10) + 1</pre><p>This gives a table like the one below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*z1Mel0so8zNPEcLmhYuaQw.png" /><figcaption>head of simulations data frame / image by the author</figcaption></figure><p>We can summarize our results using the following code:</p><pre># Group by &#39;iteration&#39; and calculate the required values<br>summary = simulations.groupby(&#39;iteration&#39;).agg(<br>    total_revenue=(&#39;yearly_revenue&#39;, <br>       lambda x: sum(x * simulations.loc[x.index, &#39;not_churned&#39;])),<br>    total_churners=(&#39;not_churned&#39;, lambda x: 10 - sum(x))<br>).reset_index()</pre><p>And finally, plotting this with plotly yields:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*iPFv_A42ADAQNhG3_xo_uQ.png" /><figcaption>Histogram of total revenues / image by the author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*fE3K6dv15-PJ9S6mSXOwlA.png" /><figcaption>Histogram of total churners / image by the author</figcaption></figure><p>The graphs above tell a much richer story than the two point estimates of 0.56 and 13034 we started with. We now understand much more about the possible outcomes we can expect to see, and we can have an informed discussion about what levels of churn and revenue we we find acceptable.</p><p>Continuing with the example above we could for example say that we would only be prepared to accept a 0.1 % chance of 8 or more churn events. Using individual customer price elasticities and simulation based forecasting, we could tweak the expected churn_rates for customers so that we could exactly achieve this outcome. This kind of customer base control is only achievable with an advanced CBM system.</p><h4>The Importance of Competitor Pricing</h4><p>One of the most important factors in pricing is the competitor price. How aggressive competitors are will to a large degree determine how flexible a company can be in its own pricing. This is especially true for commoditized businesses such as utilities or telcos where it’s hard for providers to differentiate. However, despite the importance of competitor pricing, many business choose not to integrate this data into their own price optimization algorithms.</p><p>The reasons for not including competitor pricing in price algorithms are varied. Some companies claim that it’s too difficult and time consuming to collect the data, and even if they started now, they still wouldn’t have all the history they need to train all the price elasticity models. Others say the prices of competitor products are not directly comparable to their own and that collecting them would be difficult. Finally, most companies also claim that they have price managers who manually monitor the market and when competitors make moves, they can adjust their own prices in response, so they don’t need to have this data in their algorithms.</p><p>The first argument can increasingly be mitigated by good web scraping and other intelligence gathering methods. If that is not enough, there are also sometimes agencies that can provide historic market data on prices for various industries and sectors. Regarding the second argument about not having comparable products, one can also use machine learning techniques to tease out the actual cost of individual product components. Another method is also to use different user personas that can be used to estimate the total monthly costs of a specific set of products or product.</p><p>Ultimately, not including competitor prices leaves the pricing algorithms and optimization engines at a disadvantage. In industries where price calculators and comparison websites make it increasingly easy for customers to get a grasp of the market, companies run a risk of being out-competed on price by more advanced competitors.</p><h3><strong>Conclusion</strong></h3><p>In this article we have discussed the main components of a customer base management system as well as some of the advanced features that contribute to making these systems invaluable. Personally, having built a few of these systems, I think the combination of price optimization algorithms — running on a broad dataset of internal and external data — coupled with a powerful visual interface in the form a dashboard is one of the most effective tools for managing customers. This combination of tools allows managers and executive leadership really take control of the customer management process and understand the strategic consequence of their actions.</p><p>As Jeff Bezos — one of the most customer-obsessed leaders — put it:</p><blockquote><a href="https://s2.q4cdn.com/299287126/files/doc_financials/annual/AMZN_Shareholder-Letter-2009-(final).pdf">“We can assure you that we’ll continue to <strong>obsess over customers</strong>. We have strong conviction that that approach — in the long term — is every bit as good for owners as it is for customers.” </a>— <em>Jeff Bezos, Amazon 2009 Letter to Shareholders</em></blockquote><p>A commitment to customer management, underpinned by data and AI-driven insights, not only enhances customer satisfaction but also secures long-term value for stakeholders.</p><p>Thanks for reading!</p><p><strong>Want to be notified whenever I publish a new article? ➡️ </strong><a href="https://ekneconsulting.com/register_subscriber/"><strong>Subscribe to my newsletter here</strong></a><strong> ⬅️. It’s free &amp; you can unsubscribe at any time!</strong></p><p><em>If you enjoyed reading this article and would like to access more content from me please feel free to connect with me on LinkedIn at </em><a href="https://www.linkedin.com/in/hans-christian-ekne-1760a259/"><em>https://www.linkedin.com/in/hans-christian-ekne-1760a259/</em></a><em> or visit my webpage at </em><a href="https://www.ekneconsulting.com/"><em>https://www.ekneconsulting.com/</em></a><em> to explore some of the services I offer. Don’t hesitate to reach out via email at hce@ekneconsulting.com</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d1597998dd5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-build-a-data-driven-customer-management-system-7d1597998dd5">How to Build a Data-Driven Customer Management System</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Machines Think Ahead: The Rise of Strategic AI]]></title>
            <link>https://medium.com/data-science/when-machines-think-ahead-the-rise-of-strategic-ai-91052e4c5da9?source=rss-ab0bff3c2cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/91052e4c5da9</guid>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[strategy]]></category>
            <category><![CDATA[games]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Hans Christian Ekne]]></dc:creator>
            <pubDate>Mon, 04 Nov 2024 10:01:56 GMT</pubDate>
            <atom:updated>2024-11-07T13:08:07.874Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@hc.ekne/list/strategic-ai-72a460668137">STRATEGIC AI</a></h4><h4>Exploring the advancements in strategic AI and how large language models fit into the bigger picture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Pb7PIr1XmSKfEGX91TVGQ.jpeg" /><figcaption>Image generated by the author using Canva Magic Studio</figcaption></figure><h3>Prologue</h3><h4>11. May 1997, New York City.</h4><p>It was a beautiful spring day in New York City. The skies were clear, and temperatures were climbing toward 20 degrees Celsius. The Yankees prepared to play the Kansas City Royals at Yankee Stadium, and the Rangers were facing off against the Devils at Madison Square Garden.</p><p>Nothing seemed out of the ordinary, yet the people gathering at the Equitable Center in Midtown Manhattan were about to experience something truly unique. They were about to witness the historic event when a computer, for the first time, would beat a reigning world champion in chess under standard tournament conditions.</p><p>Representing humans was Gary Kasparov, widely recognized as the world’s top chess player at the time. And representing the machines, Deep Blue — a chess computer developed by IBM. Going into the final and 6th game of the match, both players had 2.5 points. It was today that the winner was to be decided.</p><p>Gary started out as black, but made an early error and faced a strong, aggressive attack from Deep Blue. After just 19 moves it was all over. Kasparov, feeling demoralized and under pressure, resigned, believing his position was untenable. A symbolic, and by many hailed as one of the most important moments between man and machine was a fact. This landmark event marked a turning point in AI development, highlighting the potential — and challenges — of strategic AI.</p><h3>Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Rn-BNLx0M9z8e4SiTVKVg.jpeg" /><figcaption>Image generated by the author using Canva Magic Studio</figcaption></figure><p>Inspired by the recent advancements in generative AI — and my own experiments with large language models and their strategic capabilities — I have increasingly been thinking about strategic AI. How have we tried to approach this topic in the past? What are the challenges and what remains to be solved before we have a more generalist strategic AI agent?</p><p>As data scientists, we are increasingly implementing AI solutions for our clients and employers. For society at large, the ever-increasing interaction with AI makes it critical to understand the development of AI and especially strategic AI. Once we have autonomous agents with the ability to maneuver well in strategic contexts, this will have profound implications for everyone.</p><p>But what exactly do we mean when we say <em>strategic AI</em>? At its core, strategic AI involves machines making decisions that not only consider potential actions, but also anticipate and influence the responses of others. It’s about maximizing expected outcomes in complex, uncertain environments.</p><p>In this article, we’ll define strategic AI, explore what it is and how it has developed through the years since IBM’s Deep Blue beat Kasparov in 1997. We will try to understand the general architecture of some of the models, and in addition also examine how large language models (LLMs) fit into the picture. By understanding these trends and developments, we can better prepare for a world where autonomous AI agents are integrated into society.</p><h3>Defining Strategic AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pJhXlIDB3qYcqAUnq0gOFw.jpeg" /><figcaption>Image generated by the author using Canva Magic Studio</figcaption></figure><p>A deeper discussion around strategic AI starts with a well-formulated definition of the topic.</p><p>When we consider strategy in a commercial setting, we often tend to associate it with topics like long-term thinking, resource allocation and optimization, a holistic understanding of inter-dependences in an organization, alignment of decisions with the purpose and mission of the company and so on. While these topics are useful to consider, I often prefer a more game theoretical definition of strategy when dealing with AI and autonomous agents. In this case we define being strategic as:</p><blockquote>Choosing a course of action that maximizes your expected payoff by considering not just your own potential actions but also how others will respond to those actions and how your decisions impact the overall dynamics of the environment.</blockquote><p>The critical part of this definition is that strategic choices are choices that do not occur in a vacuum, but rather in the context of other participants, be they humans, organizations or other AIs. These other entities can have similar or conflicting goals of their own and may also try to act strategically to further their own interests.</p><p>Also, strategic choices always seek to <strong>maximize expected payoffs</strong>, whether those payoffs are in terms of money, utility, or other measures of value. If we wanted to incorporate the more traditional “commercial” topics related to strategy we could imagine that we want to maximize the value of a company 10 years from now. In this case, to formulate a good strategy we would need to take a “long term” view, and might also consider the “purpose and mission” of the company as well, to ensure alignment with the strategy. However, pursuing these endeavors are merely a consequence of what it actually means to act strategically.</p><p>The game-theoretic view of strategy captures the essence of strategic decision-making and consequently lets us clearly define what we mean by strategic AI. From the definition we see that if an AI system or agent is to act strategically, it needs to have a few core capabilities. Specifically, it will need to be able to:</p><ul><li><strong>Model other agents</strong> (using predictive techniques or probabilistic reasoning; those agents being anything from humans, AIs or organizations).</li><li><strong>Optimize actions</strong> based on expected utility.</li><li><strong>Adapt dynamically</strong> as they gather new information about other agents’ strategies.</li></ul><p>There is currently no well-known, or well published system, that is capable of all of these actions in an autonomous way in the real world. However, given the recent advances in AI systems and the rapid rise of LLMs that might be about to change!</p><h3>Other Important Concepts from Game Theory</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nU6gWJpCSpF8MHC3LVHqIg.jpeg" /><figcaption>Image generated by the author using Canva Magic Studio</figcaption></figure><p>Before we proceed with further discussion into strategic AI, it might be useful to review some concepts and ideas from game theory. A lot of the work that has been done around strategic AI has a foundation in game theoretic concepts and using theorems from game theory can show the existence of certain properties that make some games and situations easier to deal with than others. It also helps to highlight some of the shortcomings of game theory when it comes to real world situations and highlights where we might be better off looking in other directions for inspiration.</p><h4>What is a Game?</h4><p>We define a game as a mathematical model comprising three key components:</p><ol><li><strong>Players</strong>: The individuals or entities making decisions.</li><li><strong>Strategies</strong>: The possible actions or plans each player can adopt.</li><li><strong>Payoffs</strong>: The rewards or outcomes each player receives based on the chosen strategies.</li></ol><p>This formal structure allows for the systematic study of strategic interactions and decision-making processes.</p><h4>Finite vs Infinite Games</h4><p>When speaking on games it also makes sense to look at the distinction between finite and infinite games.</p><p>Finite games have a fixed set of players, defined rules, and a clear endpoint. The objective is to win, and examples include chess, go, checkers, and most traditional board games.</p><p>Infinite games on the other hand have no predetermined endpoint, and the rules can evolve over time. The objective is not to win but to continue playing. Real-world scenarios like business competition or societal evolution can be viewed as infinite games. The Cold War can be viewed as an example of an infinite game. It was a prolonged geopolitical struggle between the United States and its allies (the West) and the Soviet Union and its allies (the East). The conflict had no fixed endpoint, and the strategies and “rules” evolved over time.</p><h4>Subgames</h4><p>Sometimes we might be able to find smaller games within a larger game context. Mathematically, subgames are self-contained games in their own right, and the need to satisfy a few different criteria:</p><ol><li>A subgame starts at a point where the player knows exactly where they are in the game.</li><li>It includes every possible action and outcome that could follow from that point.</li><li>It encompasses all the players’ knowledge and uncertainties relevant to those actions.</li></ol><p>We can visualize a subgame if we imagine a large tree representing an entire game. A subgame is like selecting a branch of this tree starting from a certain point (node) and including everything that extends from it, while also ensuring that any uncertainties are fully represented within this branch.</p><p>The core idea behind a subgame makes it useful for our discussion around strategic AI. The reason is primarily that some infinite games between players might be very complex and difficult to model while if we choose to look at smaller games within that game, we can have more success applying game theoretical analysis.</p><p>Coming back to our example with the Cold War as an infinite game, we can recognize several subgames within that context. Some examples include:</p><p><strong>The Cuban Missile Crisis (1962):</strong></p><ul><li><strong>Players</strong>: The United States and the Soviet Union.</li><li><strong>Strategies</strong>: The U.S. considered options ranging from diplomatic negotiations to military invasion, while the Soviet Union had to decide whether to remove the missiles or escalate the confrontation.</li><li><strong>Payoffs</strong>: Avoiding nuclear war, maintaining global image, and strategic military positioning.</li></ul><p><strong>The Berlin Blockade and Airlift (1948–1949):</strong></p><ul><li><strong>Players</strong>: The Western Allies and the Soviet Union.</li><li><strong>Strategies</strong>: The Soviets blocked Berlin to push the Allies out, while the Allies had to decide between abandoning the city or supplying it via air.</li><li><strong>Payoffs</strong>: Control over Berlin, demonstrating political resolve, and influencing European alignment.</li></ul><p>Although of course very difficult and complex to deal with, both “subgames” are easier to analyze and develop responses to than to the whole of the Cold War. They had a defined set of players, with a limited set of strategies and payoffs, and also a clearer time frame. This made them both more applicable for game theoretical analysis.</p><p>In the context of strategic AI, analyzing these sub-games is crucial for developing intelligent systems capable of making optimal decisions in complex, dynamic environments.</p><h4>Two Player Games</h4><p>Two player games are simply a game between two players. This could for example be a game between two chess players, or coming back to our Cold War example, the West vs the East. Having only two players in the game simplifies the analysis but still captures essential competitive or cooperative dynamics. Many of the results in game theory are based around two player games.</p><h4>Zero-Sum Games</h4><p>Zero-sum games are a subset of games where one player’s gain is another player’s loss. The total payoff remains constant, and the players are in direct competition.</p><h4>Nash Equilibrium and Optimal Actions</h4><p>A Nash Equilibrium (NE) is a set of strategies where no player can gain additional benefit by unilaterally changing their own strategy, assuming the other players keep theirs unchanged. In this state, each player’s strategy is the best response to the strategies of the others, leading to a stable outcome where no player has an incentive to deviate.</p><p>For example, in the game Rock-Paper-Scissor (RPS), the NE is the state where all players play rock, paper and scissors, randomly, each with equal probability. If you as a player choose to play the NE strategy, you ensure that no other player can exploit your play and in a two player zero-sum games it can be shown that you will not lose in expectation, and that the worst you can do is break even.</p><p>However, playing a NE strategy might not always be the optimal strategy, especially if your opponent is playing in a predictably sub-optimal way. Consider a scenario with two players, A and B. If player B starts playing paper more, player A could recognize this and increase its frequency of playing scissors. However, this deviation from A could again be exploited by B again which could change and play more rock.</p><h4>Key Takeaways Regarding Strategic AI</h4><p>Reviewing the game theoretic concepts, it would seem the idea of a subgame is especially useful for strategic AI. The ability to find possible smaller and easier to analyze games within a larger context makes it easier to apply already know solutions and solvers.</p><p>For example, let’s say you are working on developing your career, something which could be classified as an infinite game and difficult to “solve”, but suddenly you get the opportunity to negotiate a new contract. This negotiation process presents an opportunity for a subgame within your career and would be much more approachable for a strategic AI using game theoretic concepts.</p><p>Indeed, humans have been creating subgames within our lives for thousands of years. About 1500 years ago in India, we created the origins of what is now known as chess. Chess turned out to be quite a challenge for AI to beat, but also allowed us to start developing more mature tools and techniques that could be used for even more complicated and difficult strategic situations.</p><h3>A Short History of Strategic AI in Games</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WayZVMp8EkBvLCkORX-6bA.jpeg" /><figcaption>Image generated by the author using Canva Magic Studio</figcaption></figure><p>Games have provided an amazing proving ground for developing strategic AI. The closed nature of games makes it easier to train models and develop solution techniques than in open ended systems. Games are clearly defined; the players are known and so are the payoffs. One of the biggest and earliest milestones was Deep Blue, the machine that beat the world champion in chess.</p><h4><strong>Early Milestones</strong>: Deep Blue</h4><p>Deep Blue was a chess-playing supercomputer developed by IBM in the 1990s. As stated in the prologue, it made history in May 1997 by defeating the reigning world chess champion, Garry Kasparov, in a six-game match. Deep Blue utilized specialized hardware and algorithms capable of evaluating 200 million chess positions per second. It combined brute-force search techniques with heuristic evaluation functions, enabling it to search deeper into potential move sequences than any previous system. What made Deep Blue special was its ability to process vast numbers of positions quickly, effectively handling the combinatorial complexity of chess and marking a significant milestone in artificial intelligence.</p><p>However, as Gary Kasparov notes in his interview with Lex Fridman¹, Deep Blue was more of a brute force machine than anything else, so it’s perhaps hard to qualify it as any type of intelligence. The core of the search is basically just trial and error. And speaking of errors, it makes significantly less errors than humans, and according to Kasparov this is one of the features which made it hard to beat.</p><h4><strong>Advancements in Complex Games</strong>: AlphaGo</h4><p>19 years after the Deep Blue victory in chess, a team from Google’s DeepMind produced another model that would contribute to a special moment in the history of AI. In 2016, AlphaGo became the first AI model to defeat a world champion go player, Lee Sedol.</p><p>Go is a very old board game with origins in Asia, known for its deep complexity and vast number of possible positions, far exceeding those in chess. AlphaGo combined deep neural networks with Monte Carlo tree search, allowing it to evaluate positions and plan moves effectively. The more time AlphaGo was given at inference, the better it performs.</p><p>The AI trained on a dataset of human expert games and improved further through self-play. What made AlphaGo special was its ability to handle the complexity of Go, utilizing advanced machine learning techniques to achieve superhuman performance in a domain previously thought to be resistant to AI mastery.</p><p>One could argue AlphaGo exhibits more intelligence than Deep Blue, given its exceptional ability to deeply evaluate board states and select moves. Move 37 from its 2016 game against Lee Sedol is a classic example. For those acquainted with Go, it was a shoulder hit at the fifth line and initially baffled commentators, including Lee Sedol himself. But as would later become clear, the move was a brilliant play and showcased how AlphaGo would explore strategies that human players might overlook and disregard.</p><h4>Combining Chess and Go: AlphaZero</h4><p>One year later, Google DeepMind made headlines again. This time, they took many of the learnings from AlphaGo and created AlphaZero, which was more of a general-purpose AI system that mastered chess, as well as Go and shogi. The researchers were able to build the AI solely through self-play and reinforcement learning without prior human knowledge or data. Unlike traditional chess engines that rely on handcrafted evaluation functions and extensive opening libraries, AlphaZero used deep neural networks and a novel algorithm combining Monte Carlo tree search with self-learning.</p><p>The system started with only the basic rules and learned optimal strategies by playing millions of games against itself. What made AlphaZero special was its ability to discover creative and efficient strategies, showcasing a new paradigm in AI that leverages self-learning over human-engineered knowledge.</p><h4>Integrating Speed and Strategy: Star Craft II</h4><p>Continuing its domination in the AI space, the Google DeepMind team changed its focus to a highly popular computer game, StarCraft II. In 2019 they developed an AI called AlphaStar² which was able to achieve Grandmaster level play and rank higher than 99.8% of human players on the competitive leaderboard.</p><p>StarCraft II is a real time strategy game that provided several novel challenges for the team at DeepMind. The goal of the game is to conquer the opposing player or players, by gathering resources, constructing buildings and amassing armies that can defeat the opponent. The main challenges in this game arise from the enormous action space that needs to be considered, the real-time decision making, partial observability due to fog of war and the need for long-term strategic planning, as some games can last for hours.</p><p>By building on some of the techniques developed for previous AIs, like reinforcement learning through self-play and deep neural networks, the team was able to make a unique game engine. Firstly, they trained a neural net using supervised learning and human play. Then, they used that to seed another algorithm that could play against itself in a multi-agent game framework. The DeepMind team created a virtual league where the agents could explore strategies against each other and where the dominant strategies would be rewarded. Ultimately, they combined the strategies from the league into a super strategy that could be effective against many different opponents and strategies. In their own words³:</p><blockquote>The final AlphaStar agent consists of the components of the <a href="https://papers.nips.cc/paper/7588-re-evaluating-evaluation.pdf">Nash distribution of the league</a> — in other words, the most effective mixture of strategies that have been discovered — that run on a single desktop GPU.</blockquote><h4><strong>Deep Dive into Pluribus and Poker</strong></h4><p>I love playing poker, and when I was living and studying in Trondheim, we used to have a weekly cash game which could get quite intense! One of the last milestones to be eclipsed by strategic AI was in the game of poker. Specifically, in one of the most popular forms of poker, 6-player no-limit Texas hold’em. In this game we use a regular deck of cards with 52 cards, and the play follows the following structure:</p><ol><li><strong>The Preflop:</strong> All players are given 2 cards (hole cards) which only they themselves know the value of.</li><li><strong>The Flop:</strong> 3 cards are drawn and laid face up so that all players can see them.</li><li><strong>The Turn: </strong>Another card is drawn and laid face up.</li><li><strong>The River:</strong> A final 5th card is drawn and laid face up.</li></ol><p>The players can use the cards on the table and the two cards on their hand to assemble a 5-card poker hand. For each round of the game, the players take turns placing bets, and the game can end at any of the rounds if one player places a bet that no one else is willing to call.</p><p>Though reasonably simple to learn, one only needs to know the hierarchy of the various poker hands, this game proved to be very difficult to solve with AI, despite ongoing efforts for several decades.</p><p>There are multiple factors contributing to the difficulty of solving poker. Firstly, we have the issue of hidden information, because you don’t know which cards the other players have. Secondly, we have a multiplayer setup with many players, with each extra player increasing the number of possible interactions and strategies exponentially. Thirdly, we have the no-limit betting rules, which allow for a complex betting structure where one player can suddenly decide to bet his entire stack. Fourth, we have an enormous game tree complexity due to the combinations of hole cards, community cards, and betting sequences. In addition, we also have complexity due to the stochastic nature of the cards, the potential for bluffing and the opponent modelling!</p><p>It was only in 2019 that a couple of researchers, Noam Brown and Tuomas Sandholm, finally cracked the code. In a paper published in Science, they describe a novel poker AI — Pluribus — that managed to beat the best players in the world in 6-player no-limit Texas hold’em.⁴ They conducted two different experiments, each consisting of a 10000 poker hands, and both experiments clearly showed the dominance of Pluribus.</p><p>In the first experiment, Pluribus played against 5 human opponents, achieving an average win rate of 48 mbb/game, with a standard deviation of 25 mbb/game. (mbb/game stands for milli big blind per game, how many big blinds is won per 1000 games played.) 48 mbb/game is considered a very high win rate, especially among elite poker players, and implies that Pluribus is stronger than the human opponents.</p><p>In the second experiment, the researchers had 5 versions of Pluribus play against 1 human. They set up the experiment so that 2 different humans would each play 5000 hands each against the 5 machines. Pluribus ended up beating the humans by an average of 32 mbb/game with a standard error of 15 mbb/game, again showing its strategic superiority.</p><p>The dominance of Pluribus is quite amazing, especially given all the complexities the researchers had to overcome. Brown and Sandholm came up with several smart strategies that helped Pluribus to become superhuman and computationally much more efficient than previous top poker AIs. Some of their techniques include:</p><ol><li>The use of two different algorithms for evaluating moves. They would first use a so called “blueprint strategy” which was created by having the program play against itself using a method called Monte Carlo counterfactual regret minimization. This blueprint strategy would be used in the first round of betting, but in subsequent betting rounds, Pluribus conducts a real-time search to find a better more granular strategy.</li><li>To make its real-time search algorithm be more computationally efficient, they would use a dept-limited search and evaluate 4 different possible strategies that the opponents might choose to play. Firstly, they would evaluate each strategy for 2 moves ahead. In addition, they would only evaluate four different strategies for the opponents, including the original blueprint strategy, a blueprint strategy biased towards folding, a blueprint strategy biased towards calling and a final blueprint strategy biased towards raising.</li><li>They also used various abstraction techniques to reduce the number of possible game states. For example, because a 9 high straight is fundamentally similar to a 8 high straight these can be viewed in a similar way.</li><li>Pluribus would discretize the continuous betting space into a limited set of buckets, making it easier to consider and evaluate various betting sizes.</li><li>In addition, Pluribus also balances its strategy in way that for any given hand it is playing, it would also consider other possible hands it could have in that situation and evaluate how it would play those hands, so that the final play would be balanced and thus harder to counter.</li></ol><p>There are quite a few interesting observations to draw from Pluribus, but perhaps the most interesting is that it doesn’t vary its play against different opponents, but instead has developed a robust strategy that is effective against a wide variety of players. Since a lot of poker players think they have to adjust their play to various situations and people, Pluribus shows us that this is not needed and probably not even optimal, given how it beat all the humans it played against.</p><p>In our short foray into game theory, we noted that if you play the NE strategy in two-player zero-sum games you are guaranteed not to lose in expectation. However, for a multiplayer game like 6-player poker there is no such guarantee. Noam Brown speculates⁵ that it is perhaps the adversarial nature of a game like poker which still makes it suitable to try to approach it with a NE strategy. Conversely, in a game like Risk where players can cooperate more, pursuing a NE strategy is not guaranteed to work, because, if you are playing a risk game with 6 people, there is nothing you can do if your 5 opponents decide to gang up on you and kill you.</p><h4>Evaluating the Trend in Strategic AI</h4><p>Summarizing the history of strategic AI in games, we see a clear trend emerging. The games are slowly but surely becoming closer to the real-world strategic situations that humans find themselves in on an everyday basis.</p><p>Firstly, we are moving from a two-player to a multiplayer setting. This can be seen from the initial success in two-player games to multiplayer games like 6-player poker. Secondly, we are seeing an increase in the mastery of games with hidden information. Thirdly we are also seeing an increase in mastery of games with more stochastic elements.</p><p>Hidden information, multiplayer settings and stochastic events are the norm rather than the exception in strategic interactions among humans, so mastering these complexities is key in achieving a more general superhuman strategic AI that can navigate in the real world.</p><h3>Large Language Models and Strategic AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z2xK58d58d2Xr0xbk_3mhQ.png" /><figcaption>Image generated by the author using Canva Magic Studio</figcaption></figure><p>I recently ran an experiment where I let LLMs play the boardgame Risk against each other. My objective with the experiment was to gauge how well the LLMs could perform in a strategic setting, more of less out of the box. Quite a lot of detailed prompting were given to the agents to provide the right context, however, and perhaps not surprisingly, the LLM performance was rather mediocre.</p><p>You can find an article about the experiment here:</p><p><a href="https://towardsdatascience.com/exploring-the-strategic-capabilities-of-llms-in-a-risk-game-setting-43c868d83c3b">Exploring the Strategic Capabilities of LLMs in a Risk Game Setting</a></p><p>Summarizing some of the key findings from the experiment, the current generation of LLMs struggles with basic strategic concepts like fortification and recognizing winning moves. They also fail to eliminate other players when it would have been strategically beneficial for them to do so.</p><p>The above experiment indicates that even though we have seen a rapid improvement in the LLMs, they still lack the sophistication for strategic reasoning. Given their very general training data and how they have been constructed this shouldn’t come as a surprise.</p><p>So how do they fit into the discussion around strategic AI? To understand that, we need to understand what the LLMs really excel at. Perhaps the most promising feature of the LLMs is their ability to digest and generate vast amounts of text. And now with multimodal models, video and audio too. In other words, LLMs are great for interacting with the real world, both in human and other contexts. Recently, an AI team at Meta was able to combine the general language capabilities of a language model with the strategic insights of a strategy engine.</p><h4>Case Study: Cicero and Diplomacy</h4><p>The game of Diplomacy is a 2 to 7-player strategy game, which Meta describes as a mix between Risk, Poker and the TV show Survivor. The players start out with a map of Europe ca. 1900, and the objective is to gain control over a majority of supply centers. Specifically, a player aims to control 18 out of 34 supply centers to achieve victory. By doing so, a player effectively dominates the map, representing their nation’s ascendancy over Europe in the period leading up to World War I.</p><p>What sets Diplomacy apart from many of the other games we have discussed so far is its reliance on negotiations between players. It’s a much more cooperative form of play than for example poker. Each player uses natural language to communicate with the other players before each turn, and they make plans to ally with each other. When the preparations are finished all players reveal their plans at the same time and the turn is executed. This type of game obviously resembles actual diplomacy and real-life negotiations closer than most other boardgames, however because of the natural language component, it has been very difficult for AI to master.</p><p>This changed in 2022, when the AI team at Meta developed Cicero. Using the latest advancements in language modelling, combined with a strategic module, Cicero was a game engine that was able to achieve more than “double the average score of the human players and ranked in the top 10% of participants who played more than one game.”⁶ As Meta describes it, their model is able to produce a strategy-grounded dialogue and generate a dialogue aware-strategy.</p><h4>Differences Between Cicero and Other Strategic AI Models</h4><p>There are a few key differences between Diplomacy and some of the other games where we have had recent strategic AI advancements. Most notably is the cooperative nature of the game — compared to the adversarial nature of the other games — and the open-ended natural language format it uses. I would argue that these differences makes the game more like real human interaction, however it also places restrictions on how the researches could train the algorithms that power Cicero.</p><p>Unlike Pluribus and AlphaZero, Cicero is not primarily trained through self-play and reinforcement learning. Instead, the Meta team used a data set with over 125,000 games and 40,000,000 messages to help train the algorithm. They thought that given the negotiating, persuading and trust-building aspects of the game, they might see strange behavior if they let the AI negotiate with itself through self-play, and that it might not capture the essence of human interaction. Quoting their research article:</p><blockquote>“…we found that a self-play algorithm that achieved superhuman performance in 2p0s versions of the game performed poorly in games with multiple human players owing to learning a policy inconsistent with the norms and expectations of potential human allies.”</blockquote><p>However, reinforcement learning was used to train part of the strategy engine, specifically it was used to train Cicero’s value function — which it needs to predict the utility of its actions. The researchers used a modified version of behavioral cloning, piKL, which seeks to maximize the expected utility from an action and at the same time minimize the divergence from human behavior.⁶ Simply put, they wanted the model to be able to find strategically sound actions while at the same time staying close to human actions.</p><p>The above features of Diplomacy highlight some important issues related to creating a strategic AI that can operate in a real-world human setting, and need to be taken into consideration when we evaluate how strategic AI will evolve moving forward.</p><h3>The Future of Strategic AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DubFcO1fOmSO0bf5WK-6lA.png" /><figcaption>Image generated by the author using Canva Magic Studio</figcaption></figure><p>Predicting the future is always tricky, however, one approach can be to use the current trends and extrapolate into future scenarios. Below, we investigate a few topics that closely relate to our previous discussion and evaluate how they can influence the future of strategic AI.</p><h4><strong>General Symbolic Strategy Engines vs. Specialized Modules</strong></h4><p>If we examine the trajectory of strategic AI engines so far, one thing that strikes us is how specialized each game engine is. Even though the architectures can be similar — like with AlphaZero learning how to play multiple different games — the AI still plays millions of games with itself for each specific game. For chess, AlphaZero played 44 million games and for Go 130 million games!⁷ A natural question to ask is whether we should try to build more general strategy engines or continue to focus on specialized modules for specific tasks?</p><p>A general strategy engine would aim to understand and apply broad strategic principles across different situations. Perhaps by creating games that capture many aspects of human strategic interaction, AI could learn through play against itself and develop strategies that apply to real-world scenarios. This approach could help AI generalize its learning, making it useful in various contexts.</p><p>On the other hand, specialized modules are AI systems designed for particular scenarios or tasks. We could envision that we could create a general strategic AI by combining multiple specialized agents. AI agents could be trained to excel in each specific area, providing deep expertise where it’s most needed. While this method might limit the AI’s ability to generalize, it ensures high performance in specific domains, which can lead to practical applications more quickly.</p><p>Given the issues with using AI for self-play in cooperative settings — as we observed with Diplomacy — and the current trend which seems to favor specialized modules for different strategic situations, it seems likely that for the near future we will have specialized strategic modules for different contexts. However, one could also envision a mixed system where we used general strategy engines to provide insights into broader topics, while specialized modules handle complex, specific challenges. This balance could allow AI systems to apply general strategic insights while adapting to the details of particular situations.</p><h4><strong>LLMs Bridging the Gap Between Strategic Modules and Real-World Applications</strong></h4><p>Large language models have changed how AI interacts with human language, offering a powerful way to connect strategic AI modules with real-world use cases. LLMs are great at understanding and generating human-like text, making them ideal as an intermediary that can translate real-world situations into structured data that strategy engines can process. As seen with Meta’s Cicero, combining LLMs with strategic reasoning allowed the AI to understand human communication, negotiate, and plan actions in collaborative environments.</p><p>Given the current trend towards more multimodal models, the LLMs are also increasingly able to translate not just text, but any real-world context into a machine readable syntax. This makes the models even more useful as intermediaries.</p><p>If we build on the ideas developed for Cicero, we could also envision fine-tuning different language models for specific tasks — like diplomatic communication — perhaps by fine tuning the models on historic diplomatic correspondence and then training separate strategy engines to come up with optimal actions.</p><h4><strong>Human-AI Collaboration: The Centaur Model</strong></h4><p>The future of strategic AI isn’t just about machines taking over decision-making; for a transition period it’s also about humans and AI working together effectively. This partnership is often called the “Centaur Model,” combining human intuition with AI’s computing power. In this model, humans bring creativity, ethical judgment, and flexibility, while AI systems offer powerful data processing and consistent application of strategic principles.</p><p>Real-world examples of this model include areas where human-AI teams outperform either humans or machines working alone. In chess, for example, Garry Kasparov promoted the idea of teaming up with AI, combining human strategic insight with AI’s precise calculations. The centaur model seemed to work well in chess until the programs started to become really good. At that point the human contribution wasn’t worth anything and was in the worst case detrimental.</p><p>However, in other areas that are more open-ended and real-world-like than chess, the centaur model is probably a good bet going forward. Simply consider how human collaboration with modern LLMs has the potential to drastically improve productivity.</p><p>This collaborative approach improves decision-making by combining human judgment with AI analysis, possibly leading to more informed and balanced outcomes. It allows for quick adaptation to new and unexpected situations, as humans can adjust strategies in real-time with AI support.</p><h4><strong>Real-World Applications Beyond Games</strong></h4><p>Games have been a great testing ground for developing strategic AI, but the real impact comes from applying these advancements to real-world challenges. Below we highlight a few examples.</p><p>One field that has seen tremendous development in the last few years is self-driving cars, and how they use strategic AI to navigate roads safely. They must predict and respond to the actions of other drivers, pedestrians, and cyclists. For example, an autonomous vehicle needs to anticipate if a pedestrian is about to cross the street or if another driver is about to change lanes unexpectedly.</p><p>Just this year, Waymo — a company that develops autonomous vehicles and ride-hailing services — started using fully autonomous taxis in three US cities: Phoenix, Arizona, and California’s Los Angeles and San Francisco. In the coming years we can probably expect to see a massive rise in fully autonomous vehicles due to the improvements in strategic AI.</p><p>In the financial markets, AI-driven trading systems analyze enormous amounts of data to make investment decisions. These systems consider the likely actions of other market participants, such as traders and institutions, to anticipate market movements. They use strategic reasoning to execute trades that maximize returns while minimizing risks, often in highly volatile environments.</p><p>AI systems also optimize supply chains by considering the actions of suppliers, competitors, and customers. They can strategically adjust production schedules, inventory levels, and logistics based on anticipated demand and competitor behavior. For example, if a competitor is expected to launch a new product, the AI can recommend increasing stock levels to meet potential increases in demand.</p><p>Strategic AI is also used to manage energy distribution efficiently. Smart grids employ AI to predict consumption patterns and adjust supply accordingly. They consider how consumers might change their usage in response to pricing signals or environmental factors. The AI strategically allocates resources to balance load, prevent outages, and integrate renewable energy sources.</p><p>The examples above clearly show how strategic AI is being integrated into various industries and fields. By considering the actions of others, these AI systems make informed decisions that optimize outcomes, enhance efficiency, and often provide a competitive advantage. As strategic AI continues to improve so will these systems, and we will likely see their emergence in many other domains as well.</p><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-UtbFRJOTo9qYMAbefW2kw.png" /><figcaption>Image generated by the author using Canva Magic Studio</figcaption></figure><p>Strategic AI has come a long way since Deep Blue’s victory over Garry Kasparov. From mastering complex board games to engaging in human-like negotiations, AI systems are increasingly exhibiting strategic reasoning abilities.</p><p>In this article we investigated the foundational concepts of strategic AI, emphasizing the importance of game theory and how some of the concepts from the field can be applied to strategic AI. We also looked at how specialized AI systems have achieved superhuman performance in specific games by focusing on narrow domains and extensive self-play. This raises the question of whether the future of strategic AI lies in developing general symbolic strategy engines capable of broader application or continuing with specialized modules tailored to specific tasks.</p><p>As we saw with Cicero, language models will also likely have a future in the space of strategic AI. The new models from providers like OpenAI, Anthropic and Meta make it easier than ever before to integrate these tools into autonomous agents that can use them to translate the real-world into structured data that AI systems can process.</p><p>However, the journey toward a general-purpose strategic AI that can navigate the complexities of the real world is just beginning. Challenges remain in developing systems that can generalize across domains, adapt to unforeseen situations, and integrate ethical considerations into their decision-making processes.</p><p>Thanks for reading!</p><p><strong>Want to be notified whenever I publish a new article? ➡️ </strong><a href="https://ekneconsulting.com/register_subscriber/"><strong>Subscribe to my newsletter here</strong></a><strong> ⬅️. It’s free &amp; you can unsubscribe at any time!</strong></p><p><em>If you enjoyed reading this article and would like to access more content from me please feel free to connect with me on LinkedIn at </em><a href="https://www.linkedin.com/in/hans-christian-ekne-1760a259/"><em>https://www.linkedin.com/in/hans-christian-ekne-1760a259/</em></a><em> or visit my webpage at </em><a href="https://www.ekneconsulting.com/"><em>https://www.ekneconsulting.com/</em></a><em> to explore some of the services I offer. Don’t hesitate to reach out via email at hce@ekneconsulting.com</em></p><h3>References</h3><ol><li>Lex Fridman. (2019, October 27). Garry Kasparov: Chess, Deep Blue, AI, and Putin | Lex Fridman Podcast #46 [Video File]. Youtube. <a href="https://youtu.be/8RVa0THWUWw?si=1ErCnwlAn4myoK9W">https://youtu.be/8RVa0THWUWw?si=1ErCnwlAn4myoK9W</a></li><li>Vinyals, O., Babuschkin, I., Czarnecki, W.M. <em>et al.</em> Grandmaster level in StarCraft II using multi-agent reinforcement learning. <em>Nature</em> 575, 350–354 (2019). <a href="https://doi.org/10.1038/s41586-019-1724-z">https://doi.org/10.1038/s41586-019-1724-z</a></li><li><a href="https://deepmind.google/discover/blog/alphastar-mastering-the-real-time-strategy-game-starcraft-ii/">https://deepmind.google/discover/blog/alphastar-mastering-the-real-time-strategy-game-starcraft-ii/</a></li><li>Brown et al. (2019, August 30). Superhuman AI for multiplayer poker. <em>Science 365, 885–890, (2019).</em> <a href="https://www.science.org/doi/epdf/10.1126/science.aay2400">https://www.science.org/doi/epdf/10.1126/science.aay2400</a></li><li>Lex Fridman. (2022, December 6). Noam Brown: AI vs Humans in Poker and Games of Strategic Negotiation | Lex Fridman Podcast #344 [Video File]. Youtube. <a href="https://youtu.be/2oHH4aClJQs?si=AvE_Esb42GNGIPRG">https://youtu.be/2oHH4aClJQs?si=AvE_Esb42GNGIPRG</a></li><li><a href="https://www.science.org/action/doSearch?ContribAuthorRaw=Meta+Fundamental+AI+Research+Diplomacy+Team+FAIR">Meta Fundamental AI Research Diplomacy Team (FAIR)†</a> <em>et al., </em>Human-level play in the game of <em>Diplomacy</em> by combining language models with strategic reasoning.<em>Science</em><strong>378</strong>,1067 1074(2022).DOI:<a href="https://doi.org/10.1126/science.ade9097">10.1126/science.ade9097</a>, <a href="https://noambrown.github.io/papers/22-Science-Diplomacy-TR.pdf">https://noambrown.github.io/papers/22-Science-Diplomacy-TR.pdf</a></li><li>David Silver <em>et al. , </em>A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play.<em>Science</em><strong>362</strong>,1140–1144(2018).DOI:<a href="https://doi.org/10.1126/science.aar6404">10.1126/science.aar6404</a> <a href="https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphazero-shedding-new-light-on-chess-shogi-and-go/alphazero_preprint.pdf">https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphazero-shedding-new-light-on-chess-shogi-and-go/alphazero_preprint.pdf</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91052e4c5da9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/when-machines-think-ahead-the-rise-of-strategic-ai-91052e4c5da9">When Machines Think Ahead: The Rise of Strategic AI</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring the Strategic Capabilities of LLMs in a Risk Game Setting]]></title>
            <link>https://medium.com/data-science/exploring-the-strategic-capabilities-of-llms-in-a-risk-game-setting-43c868d83c3b?source=rss-ab0bff3c2cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/43c868d83c3b</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[strategy]]></category>
            <dc:creator><![CDATA[Hans Christian Ekne]]></dc:creator>
            <pubDate>Tue, 27 Aug 2024 00:13:15 GMT</pubDate>
            <atom:updated>2024-11-07T13:07:38.647Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@hc.ekne/list/strategic-ai-72a460668137">STRATEGIC AI</a></h4><h4>In a simulated Risk environment, large language models from Anthropic, OpenAI, and Meta showcase distinct strategic behaviors, with Claude Sonnet 3.5 edging out a narrow lead</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7guIFVKv_hArqNw83A4sIA.png" /><figcaption>Image generated by the author using DALL-E</figcaption></figure><h3>Introduction</h3><p>In recent years, large language models (LLMs) have rapidly become a part of our everyday lives. Since OpenAI blew our minds with GPT-3 we have witnessed a profound increase in the capabilities of the models. They are excelling in a myriad of different tests, in anything from language comprehension to reasoning and problem-solving tasks.</p><p>One topic that I find particularly compelling — and perhaps under-explored — is the ability of LLMs to reason strategically. That is, how the models will act if you insert them into a situation where the outcome of their decisions depends not only on their own actions but also on the actions of others, who are also making decisions based on their own goals. The LLMs’ ability to think and act strategically is increasingly important as we weave them into our products and services, and especially considering the emerging risks associated with powerful AIs.</p><p>A decade ago, philosopher and author Nick Bostrom brought AI risk into the spotlight with his influential book <em>Superintelligence</em>. He started a global conversation about AI, and it brought AI as an existential risk into the popular debate. Although the LLMs are still far from Bostrom’s superintelligence, it’s important to keep an eye on their strategic capabilities as we integrate them tighter into our daily lives.</p><p>When I was a child, I used to love playing board games, and Risk was one of my favorites. The game requires a great deal of strategy and if you don’t think through your moves, you will likely be decimated by your opponents. Risk serves as a good proxy for evaluating strategic behavior, because making strategic decisions often involves weighing potential gains against uncertain outcomes, and while for small troop sizes, luck clearly plays an big part, given enough time and larger army sizes, the luck component becomes less pronounced and the most skillful players emerge. So, what better arena to test the LLMs’ strategic behavior than Risk!</p><p>In this article I explore two main topics related to LLMs and strategy. Firstly, which of the top LLM models is the most strategic Risk player and how strategic is the best model in its actions? Secondly, how have the strategic capabilities of the models developed through the model iterations?</p><p>To answer these questions, I built a virtual Risk game engine and let the LLMs battle it out. The first part of this article will explore some of the details of the game implementation before we move on to analyzing the results. We then discuss how the LLMs approached the game and their strategic abilities and shortcomings, before we end with a section on what these results mean and what we can expect from future model generations.</p><h3>Setting the Stage</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DnSGVXBHmwrJRyD062jvxA.png" /><figcaption>Image generated by author using DALL-E</figcaption></figure><h4><strong>Why Risk?</strong></h4><p>My own experience playing Risk obviously played a part in choosing this game as a testbed for the LLMs. The game requires players to understand how their territories are linked, balance offense with defense and all while planning long-term strategies. Elements of uncertainty are also introduced through dice rolls and unpredictable opponent behavior, challenging AI models to manage risk and adapt to changing conditions.</p><p>Risk simulates real-world strategic challenges, such as resource allocation, adaptability, and pursuing long-term goals amid immediate obstacles, making it a valuable proxy for evaluating AI’s strategic capabilities. By placing LLMs in this environment, we can observe how well they handle these complexities compared to human players.</p><h4><strong>The Modelling Environment</strong></h4><p>To conduct the experiments, I created a small python package creatively named risk_game. (See the appendix for how to get started running this on your own machine.) The package is a Risk game engine, and it allows for the simulation of games played by LLMs. (The non-technical reader can safely skip this part and continue to the section “The Flow of the Game”.)</p><p>To make it easier to conceptually keep track of the moving parts, I followed an object-oriented approach to the package development, where I developed a few different key classes to run the simulation. This includes a game master class to control the flow of the game, a player class to control prompts sent to the LLMs and a game state class to control the state of the game, including which player controls which territories and how many troops they hold at any given time.</p><p>I tried to make it a flexible and extensible solution for AI-driven strategy simulations, and the package could potentially be modified to study strategic behavior of LLMs in other settings as well. See below for a full overview of the package structure:</p><pre>risk_game/<br>│<br>├── llm_clients/<br>│   ├── __init__.py<br>│   ├── anthropic_client.py<br>│   ├── bedrock_client.py<br>│   ├── groq_client.py<br>│   ├── llm_base.py<br>│   ├── llm_client.py<br>│   └── openai_client.py<br>│<br>├── utils/<br>│   ├── __init__.py<br>│   ├── decorators.py<br>│   └── game_admin.py<br>│<br>├── card_deck.py<br>├── experiments.py<br>├── game_config.py<br>├── game_constants.py<br>├── game_master.py<br>├── game_state.py<br>├── main.py<br>├── player_agent.py<br>├── rules.py<br>│<br>├── scripts/<br>│   ├── example_run.py<br>│<br>└── tests/</pre><p>To run an experiment, I would first instantiate a GameConfig object. This config objects holds all the related game configuration settings, like whether we played with progressive cards, whether or not capitals mode was active, and how high percent of the territories needed to be controlled to win, in addition to multiple other game settings. I would then used that to create an instance of the Experiment class and call the run_experiment method.</p><p>Diving deeper behind the scenes we can see how the Experimentclass is set up.</p><pre>from risk_game.llm_clients import llm_client<br>import risk_game.game_master as gm<br>from risk_game.rules import Rules<br>from typing import List<br>from risk_game.game_config import GameConfig <br><br>class Experiment:<br>    def __init__(self, config: GameConfig, agent_mix: int= 1, num_games=10<br>      ) -&gt; None:<br>        &quot;&quot;&quot;<br>        Initialize the experiment with default options.<br>        <br>        Args:<br>        - num_games (int): The number of games to run in the experiment.<br>        - agent_mix (int): The type of agent mix to use in the experiment.<br>        - config (GameConfig): The configuration for the game.<br><br>        &quot;&quot;&quot;<br>        self.config = config    <br>        self.num_games = num_games<br>        self.agent_mix = agent_mix<br><br>    def __repr__(self) -&gt; str:<br><br>        if self.config.key_areas:<br>            key_areas = &#39;, &#39;.join(self.config.key_areas)<br>        else:<br>            key_areas = &#39;None&#39;<br><br>        return (f&quot;Experiment Configuration:\n&quot;<br>                f&quot;Agent Mix: {self.agent_mix}\n&quot;<br>                f&quot;Number of Games: {self.num_games}\n&quot;<br>                f&quot;Progressive: {self.config.progressive}\n&quot;<br>                f&quot;Capitals: {self.config.capitals}\n&quot;<br>                f&quot;Territory Control Percentage: +&quot;<br>                f&quot;{self.config.territory_control_percentage:.2f}\n&quot;<br>                f&quot;Required Continents: {self.config.required_continents}\n&quot;<br>                f&quot;Key Areas: {key_areas}\n&quot;<br>                f&quot;Max Rounds: {self.config.max_rounds}\n&quot;)<br>    <br>    def initialize_game(self)-&gt; gm.GameMaster:<br>        &quot;&quot;&quot;<br>        Initializes a single game with default rules and players.<br>        <br>        Returns:<br>        - game: An instance of the initialized GameMaster class.<br>        &quot;&quot;&quot;<br>        # Initialize the rules<br>        rules = Rules(self.config)<br>        game = gm.GameMaster(rules)<br>        <br>        if self.agent_mix == 1:<br>            # Add strong AI players<br>            game.add_player(name=&quot;llama3.1_70&quot;, <br>                        llm_client=llm_client.create_llm_client(&quot;Groq&quot;, 1))<br>            game.add_player(name=&quot;Claude_Sonnet_3_5&quot;, <br>                        llm_client=llm_client.create_llm_client(&quot;Anthropic&quot;, 1))<br>            game.add_player(name=&quot;gpt-4o&quot;, <br>                        llm_client=llm_client.create_llm_client(&quot;OpenAI&quot;, 1))<br>        <br>        elif self.agent_mix == 3:<br>            # Add mix of strong and weaker AI players from Open AI<br>            game.add_player(name=&quot;Strong(gpt-4o)&quot;, <br>                        llm_client=llm_client.create_llm_client(&quot;OpenAI&quot;, 1))<br>            game.add_player(name=&quot;Medium(gpt-4o-mini)&quot;, <br>                        llm_client=llm_client.create_llm_client(&quot;OpenAI&quot;, 2))<br>            game.add_player(name=&quot;Weak(gpt-3.5-turbo)&quot;, <br>                        llm_client=llm_client.create_llm_client(&quot;OpenAI&quot;, 3))<br><br>        elif self.agent_mix == 5:<br>            # Add mix extra strong AI players<br>            game.add_player(name=&quot;Big_llama3.1_400&quot;, <br>                        llm_client=llm_client.create_llm_client(&quot;Bedrock&quot;, 1))<br>            game.add_player(name=&quot;Claude_Sonnet_3_5&quot;, <br>                        llm_client=llm_client.create_llm_client(&quot;Anthropic&quot;, 1))<br>            game.add_player(name=&quot;gpt-4o&quot;, <br>                        llm_client=llm_client.create_llm_client(&quot;OpenAI&quot;, 1))<br><br><br>        return game<br><br>    def run_experiment(self)-&gt; None:<br>        &quot;&quot;&quot;<br>        Runs the experiment by playing multiple games and saving results.<br>        &quot;&quot;&quot;<br>        for i in range(1, self.num_games + 1):<br>            print(f&quot;Starting game {i}...&quot;)<br>            game = self.initialize_game()<br>            game.play_game(include_initial_troop_placement=True)</pre><p>From the code above, we see that the run_experiment() method will run the number of games that are specified in the initialization of the Experiment object. The first thing that happens is to initialize a game, and the first thing we need to do is to create the rules and instantiate at game with the GameMaster class. Subsequently, the chosen mix of LLM player agents are added to the game. This concludes the necessary pre-game set-up and we use the games’ play_game()method to start playing a game.</p><p>To avoid becoming too technical I will skip over most of the code details for now, and rather refer the interested reader to the Github repo below. Check out the README to get started:</p><p><a href="https://github.com/hcekne/risk-game.git">GitHub - hcekne/risk-game</a></p><h4>The Flow of the Game</h4><p>Once the game begins, the LLM player agents are prompted to do initial troop placement. The agents take turns placing their troops on their territories until all their initial troops have been exhausted.</p><p>After initial troop placement, the first player starts its turn. In Risk a turn is comprised of the 3 following phases:</p><ul><li><strong>Phase 1: Card trading and troop placement.</strong> If a player agent wins an attack during its turn, it gains a card. Once it has three cards, it can trade those in for troops if has the correct combination of infantry, cavalry, artillery or wildcard. The player also receives troops as a function of how many territories it controls and also if controls any continents.</li><li><strong>Phase 2: Attack. </strong>In this phase the player agent can attack other players and take over their territories. It is a good idea to attack because that allows the player to gain a card for that turn and also gain more territories. The player agent can attack as many times as it wishes during a turn.</li><li><strong>Phase 3: Fortify.</strong> The last phase is the fortify phase, and now the player is allowed to move troops from one of its territories to another. However, the territories must be connected by territories the player controls. The player is only allowed one such fortify move. After this the is finished, the next player starts his turn.</li></ul><p>At the beginning of each turn, the LLM agents receive dynamically generated prompts to formulate their strategy. This strategy-setting prompt provides the agent with the current game rules, the state of the board, and possible attack vectors. The agent’s response to this prompt guides its decisions throughout the turn, ensuring that its actions align with an overall strategic plan.</p><p>The request for strategy prompt is given below:</p><pre>prompt = &quot;&quot;&quot;<br>  We are playing Risk and you are about to start your turn, but first <br>  you need to define your strategy for this turn.<br>  You, are {self.name}, and these are the current rules we are <br>  playing with:<br><br>  {rules}<br><br>  {current_game_state}<br><br>  {formatted_attack_vectors}<br><br>  Your task is to formulate an overall strategy for your turn, <br>  considering the territories you control, the other players, and the <br>  potential for continent bonuses. <br><br>  Since the victory conditions only requires you to control <br>  {game_state.territories_required_to_win} territories, and you already <br>  control {number_of_territories} territories, <br>  you only need to win an extra {extra_territories_required_to_win}<br>  to win the game outright. Can you do that this turn?? If so lay <br>  your strategy out accordingly.<br><br>  **Objective:**<br><br>  Your goal is to win the game by one of the victory conditions given<br>  in the rules. Focus on decisive attacks that reduce <br>  your opponents&#39; ability to fight back. When possible, eliminate <br>  opponents to gain their cards, which will allow you to trade them <br>  in for more troops and accelerate your conquest.<br><br><br>  **Strategic Considerations:**<br><br>  1. **Attack Strategy:**<br>  - Identify the most advantageous territories to attack.<br>  - Prioritize attacks that will help you secure continent bonuses or <br>  weaken your strongest opponents.<br>  - Look for opportunities to eliminate other players. If an opponent <br>  has few territories left, eliminating them could allow you to gain <br>  their cards, which can be especially powerful if you’re playing with <br>  progressive card bonuses.<br>  - Weigh the risks of attacking versus the potential rewards.<br><br>  2. **Defense Strategy:**<br>  - Identify your most vulnerable territories and consider fortifying <br>  them.<br>  - Consider the potential moves of your opponents and plan your defense <br>  accordingly.<br><br>  Multi-Turn Planning: Think about how you can win the game within <br>  the next 2-3 turns. What moves will set you up for a decisive victory?<br>  Don&#39;t just focus on this turn; consider how your actions this turn <br>  will help you dominate in the next few turns.<br><br><br>  **Instructions:**<br><br>  - **Limit your response to a maximum of 300 words.**<br>  - **Be concise and direct. Avoid unnecessary elaboration.**<br>  - **Provide your strategy in two bullet points, each with a <br>  maximum of four sentences.**<br><br>  **Output Format:**<br><br>  Provide a high-level strategy for your turn, including:<br>  1. **Attack Strategy:** Which territories will you target, and why? <br>  How many troops will you commit to each attack? If you plan to <br>  eliminate an opponent, explain how you will accomplish this.<br>  2. **Defense Strategy:** Which territories will you fortify, and <br>  how will you allocate your remaining troops?<br><br>  Example Strategy:<br>  - **Attack Strategy:** Attack {Territory B} from {Territory C} with <br>  10 troops to weaken Player 1 and prevent them from securing the <br>  continent bonus for {Continent Y}. Eliminate Player 2 by attacking <br>  their last remaining territory, {Territory D}, to gain their cards.<br>  - **Defense Strategy:** Fortify {Territory E} with 3 troops to <br>  protect against a potential counter-attack from Player 3.<br><br>  Remember, your goal is to make the best strategic decisions that<br>      will maximize your chances of winning the game. Consider the <br>      potential moves of your opponents and how you can position <br>      yourself to counter them effectively.<br><br>  What is your strategy for this turn?<br>        &quot;&quot;&quot;</pre><p>As you can see from the prompt above, there are multiple dynamically generated elements that help the player agent better understand the game context and make more informed strategic decisions.</p><p>These dynamically produced elements include:</p><ul><li><strong>Rules:</strong> The rules of the game such as whether capitals mode is activated, how many percent of the territories are needed to secure a win, etc.</li><li><strong>Current game state</strong>: This is presented to the agent as the different continents and the</li><li><strong>Formatted Attack Vectors:</strong> These are a collection of the possible territories the agent can launch an attack from, to which territories it can attack and the maximum number of troops the agent can attack with.</li><li><strong>The extra territories needed to win the game:</strong> This represents the remaining territories the agent needs to capture to win the game. For example, if the total territories required to win the game are 28 and the agent holds 25 territories, this number would be 3 and would maybe encourage the agent to develop a more aggressive strategy for that turn.</li></ul><p>For each specific action during the turn — whether it’s placing troops, attacking, or fortifying — the agent is given tailored prompts that reflect the current game situation. Thankfully, Risk’s gameplay can be simplified because it adheres to the Markov property, meaning that optimal moves depend only on the current game state, not on the history of moves. This allows for streamlined prompts that focus on the present conditions</p><h3>The Experimental Setup</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3DEsM4xx4JSEr5coedZ6sA.png" /></figure><p>To explore the strategic capabilities of LLMs, I designed two main experiments. These experiments were crafted to address two key questions:</p><ol><li><em>What is the top performing LLM, and how strategic is it in its actions?</em></li><li><em>Is there a progression in the strategic capabilities of the LLMs through model iterations?</em></li></ol><p>Both of these questions can be answered by running two different experiments, with a slightly different mix of AI agents.</p><h4><strong>Experiment-1: Evaluating the Top Models</strong></h4><p>For the first question, I created an experiment using the following top LLM models as players:</p><ul><li>OpenAI’s GPT-4o running off the OpenAI API endpoint</li><li>Anthropic’s claude-3–5-sonnet-20240620 running off the Anthropic API endpoint</li><li>Meta’s llama-3.1–70b-versatile running of the Groq API endpoint</li></ul><p>I obviously wanted to try Meta’s meta.llama3–1–405b-instruct-v1:0 and configured it to run off AWS Bedrock, however the response time was painfully slow and made simulating games take forever. This is why we run Meta’s 70b model on Groq. It’s much faster than AWS bedrock. (If anyone knows how to speed up llama3.1 405b on AWS please let me know!)</p><p>And we formulate our null and alternative hypotheses as follows:</p><blockquote><strong>Experiment-1, H0 : </strong>There is no difference in performance among the models; each model has an equal probability of winning.</blockquote><blockquote><strong>Experiment-1, H1​</strong>: At least one model performs better (or worse) than the others, indicating that the models do not have equal performance.</blockquote><h4><strong>Experiment-2: Analyzing the Model Generations</strong></h4><p>The second experiment aimed to evaluate how strategic capabilities have progressed through different iterations of OpenAI’s models. For this, I selected three models:</p><ul><li>GPT-4o</li><li>GPT-4o-mini</li><li>GPT-3.5-turbo-0125</li></ul><p><strong>Experiment-2</strong> allows us to see how the strategic capabilities of the models have developed across model generations, and also allows us to analyze the difference between different size models in the same model generation (GPT-4o vs GPT-4o-mini). I chose OpenAI’s solutions because they didn’t have the same restrictive rate limits as the other providers.</p><p>Similarly, as for <strong>Experiment-1</strong>, for this experiment we can formulate our null and alternative hypotheses:</p><blockquote><strong>Experiment-2, H0: </strong>There is no difference in performance among the models; each model has an equal probability of winning</blockquote><blockquote><strong>Experiment-2, H1A​</strong>: GPT-4o is better than GPT-4o-mini</blockquote><blockquote><strong>Experiment-2, H1B: </strong>GPT-4o and GPT-4o-mini are better than GPT-3.5-turbo</blockquote><h4>Game Setup, Victory Conditions &amp; Card Bonuses</h4><p>Both experiments involved 10 games, each with the same victory conditions. There are multiple different victory conditions in Risk, and typical victory conditions that players can agree upon are:</p><ol><li>Number of controlled territories required for the winner. “World domination” is subset of this where one player needs to control all the territories. Other typical territory conditions are 70% territory control.</li><li>Number of controlled continent(s) required for the winner</li><li>Control / possession of key areas required for the winner</li><li>Preset time / turn count: whoever controls the most territories after x hours or x turns wins.</li></ol><p>In the end I settled for a more pragmatic approach which was a a combination of victory conditions that would be easier to fulfill and progressive cards. The victory conditions for the games in the experiments were finally chosen to be:</p><ol><li>First agent to reach 65% territory dominance or</li><li>The agent with the most territories after 17 game rounds of play (Making the full game be concluded after at most 51 turns distributed across the three players.)</li></ol><p>For those of you unfamiliar with Risk, progressive cards means that the value of the traded cards increase progressively as the game goes on, which is contrasted by fixed cards, where the troop value of traded cards are the same throughout the game. (4,6,8,10 for the different combinations.) Progressive is generally accepted to be a faster game mode.</p><h3>The Results — Who Conquered the World?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*38Jdks5JRfFOyVFnbaVprQ.png" /><figcaption>Image generated by the author using DALL-E</figcaption></figure><h4>Experiment-1: The Top Models</h4><p>The results were actually quite astounding — for both experiments. Starting with the first, below we show the distribution of wins amongst the three agents. Anthropic’s Claude is the winner with 5 wins in total, second place goes to OpenAI’s GPT-4o with 3 wins and last place to Meta’s llama3.1 with 2 wins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*qG1yguQJsVBD5o-9XsPkHg.png" /><figcaption>Figure 3. Experiment-1 wins by player, grouped by victory condition / image by author</figcaption></figure><p>Because of their long history and early success with GPT-3 I was expecting OpenAI&#39;s model to be the winner, but it ended up being Anthropic’s Claude which took a lead in overall games. I guess if we take a look at how Claude is performing on <a href="https://www.anthropic.com/news/claude-3-5-sonnet">benchmark tests</a>, it shouldn’t be too unexpected that they come out ahead.</p><h4>Territory Control and Game Flow</h4><p>If we dive a little deeper in the overall flow of the game and evaluate the distribution of territories throughout the game, we find the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*aPAFW9HFArMoSMT3j92pSA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mLqzB9LDgqjLFlaAhSC6QA.png" /><figcaption>Figure 5. Experiment-1 territory control per turn / image by author</figcaption></figure><p>When we examine the distribution of territories throughout the games, a clearer picture emerges. On average, Claude managed to gain a lead in territory control midway through most games and maintained that lead until the end. Interestingly, there was only one instance where a player was eliminated from the game entirely — this happened in Game 8, where Llama 3.1 was knocked out around turn 27.</p><blockquote>In our analysis, a “turn” refers to the full set of moves made by one player during their turn. Since we had three agents participating, each game round typically involved three turns, one for each player. As players were eliminated, the number of turns per round naturally decreased.</blockquote><p>Looking at the evolution of troop strength and territory control we find the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*6bSbLyjTSGSVou_NOD4a4w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*---rP82FdenORE5Hxc0OJQ.png" /><figcaption>Figure 6. Experiment-1 change in troop strength throughout the game / image by author</figcaption></figure><p>The troop strength seems to be relatively even, on average, for all the models, so that is clearly not the reason why Claude is able to pull off the most wins.</p><h4>Statistical Analysis: Is Claude Really the Best?</h4><p>In this experiment, I aimed to determine whether any of the three models demonstrated significantly better performance than the others based on the number of wins. Given that the outcome of interest was the frequency of wins across multiple categories (the three models), the chi-square goodness-of-fit test is a good statistical tool to use.</p><p>The test is often used to compare observed frequencies against expected frequencies under the null hypothesis, which in this case was that all models would have an equal probability of winning. By applying the chi-square test, I could assess whether the distribution of wins across the models deviated significantly from the expected distribution, thereby helping to identify if any model performed substantially better.</p><pre>from scipy.stats import chisquare<br><br># Observed wins for the three models<br>observed = [5, 3, 2]<br><br># Expected wins under the null hypothesis (equal probability)<br>expected = [10 / 3] * 3<br><br># Perform the chi-square goodness-of-fit test<br>chi2_statistic, p_value = chisquare(f_obs=observed, f_exp=expected)<br><br>chi2_statistic, p_value<br><br>(1.4, 0.4965853037914095)</pre><p>The chi-square goodness-of-fit test was conducted based on the observed wins for the three models: 5 wins for Claude, 3 wins for GPT-4o, and 2 wins for llama3.1. Under the null hypothesis:</p><blockquote><strong>Experiment-1, H0 : </strong>There is no difference in performance among the models; each model has an equal probability of winning.</blockquote><p>each model was expected to win approximately 3.33 games out of the 10 trials. The chi-square test yielded a statistic of 1.4 with a corresponding p-value of 0.497. Since this p-value is much larger than the conventional significance level of 0.05, we can’t really say with any statistical rigor that Claude is better than the others.</p><blockquote>We can interpret the p-value such that there is a 49.7% chance that we would observe an outcome as extreme as (5,3,2) under the null hypothesis, which assumes each model has the same probability of winning. So this is actually quite a likely scenario to observe.</blockquote><p>To make a definitive conclusion, we would need to run more experiments with a larger sample size. Unfortunately, rate limits — particularly with Llama 3.1 hosted on Groq — made this impractical. I invite the eager reader to follow up and test themselves. See the appendix for how to run the experiments on your own machine.</p><h4>Experiment-2: Model Generations</h4><p>The results of Experiment-2 were equally surprising. Contrary to expectations, GPT-4o-mini outperformed both GPT-4o and GPT-3.5-turbo. GPT-4o-mini secured 7 wins, while GPT-4o managed 3 wins, and GPT-3.5-turbo failed to win any games.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*IubQVESZ0TFAHH1nLPBlAw.png" /><figcaption>Figure 8. Number of wins by player and victory condition / image by author</figcaption></figure><p>GPT-4o-mini actually went off with the overall victory. This was also rather substantial, with 7 wins of GPT-4o’s 3 and GPT-3.5 turbo’s 0 wins. While GPT-4o on average had more troops GPT-4o-mini won most of the games.</p><h4>Territory Control and Troop Strength</h4><p>Again, diving deeper and looking at performance in individual games, we find the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*1rM5c4WF9I_iqYB3hjGrmA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*taRqT4YLOvmnF722F04wBA.png" /><figcaption>Figure 9. Experiment-2 Average territory control per turn, for all games / image by author</figcaption></figure><p>The charts above show territory control per turn, on average, as well as for all the games. These plots show a confirmation of what we saw in the overall win statistics, namely that GPT-4o-mini is on average coming out with the lead in territory control by the end of the games. GPT-4o-mini is beating its big brother when it actually counts, close to the end of the game!</p><p>Turning around and examining troop strength, a slightly different picture emerges:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*VnYVOumhCNx_b8AzXq34Zw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pqB9ajJwaZuJ4Zk6sSFRGg.png" /><figcaption>Figure 10. Experiment-2 Average total troop strength per turn, for all games / image by author</figcaption></figure><p>The above chart shows that on average, the assumed strongest player, GPT-4o manages to keep the highest troop strength throughout most of the games. Surprisingly it fails to use this troop strength to its advantage! Also, there is a clear trend between troop strength and model size and model generation.</p><p>To get some more insights we can also evaluate a few games more in detail and look at the heatmap of controlled territories across the turns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*62iO1OjX7i7OfKJF8DvFfw.png" /><figcaption>Figure 11. Experiment 2, heatmap of territory control, game 2 / image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d4ICxQfguUahTdmi2aIrRQ.png" /><figcaption>Figure 12. Experiment 2, heatmap of territory control, game 7 / image by author</figcaption></figure><p>From the heatmaps we see how the models trade blows and grab territories from another. Here we have selected two games which seemed reasonably representative for the 10 games in the experiment.</p><p>Regarding specific territory ownership, a trend we saw play out frequently was GPT-4o trying to hold North America while GPT-4o-mini often tried to get Asia.</p><h4>Statistical Analysis: Generational Differences</h4><p>With the above results, let’s again revisit our initial hypotheses:</p><blockquote><strong>Experiment-2, H0 : </strong>There is no difference in performance among the models; each model has an equal probability of winning.</blockquote><blockquote><strong>Experiment-2, H1A​</strong>: GPT-4o is better than GPT-4o-mini</blockquote><blockquote><strong>Experiment-2, H1B: </strong>GPT-4o and GPT-4o-mini are better than GPT-3.5-turbo</blockquote><p>Let’s start with the easy one, <strong>H1B, </strong>namely that GPT-4o and GPT-4o-mini are better than GPT-3.5-turbo. This is quite easy to see, and we can do a chi-squared test again, based on equal probabilities of winning for each model.</p><pre>from scipy.stats import chisquare<br><br># Observed wins for the three models<br>observed = [7, 3, 0]<br><br># total observations<br>total_observations = sum(observed)<br><br># Expected wins under the null hypothesis (equal probability)<br>expected_probabilites = [1/3] * 3<br><br>expeceted_wins = [total_observations * p for p in expected_probabilities]<br><br># Perform the chi-square goodness-of-fit test<br>chi2_statistic, p_value = chisquare(f_obs=observed, f_exp=expected_wins)<br><br>chi2_statistic, p_value<br><br>(7.4, 0.0247235265)</pre><p>This suggests that the observed distribution of wins is unlikely to have occurred if every model had the same probability of winning, 33.3%. In fact, a case as extreme as this could only be expected to have occurred in 2.5% of cases.</p><p>To then evaluate our <strong>H1A</strong> hypothesis we should first update our null hypothesis adjusting for unequal probabilities of winning. For example, we can now assume that:</p><ul><li>GPT-4o-mini: Higher probability</li><li>GPT-4o: Higher probability</li><li>GPT-3.5-turbo: Lower probability</li></ul><p>Putting some numbers on these, and given the results we just observed, let’s assume GPT-4o-mini:</p><ul><li>GPT-40-mini: 45% chance of winning each game</li><li>GPT-4o: 45% chance of winning each game</li><li>GPT-3.5-turbo: 10% chance of winning each game</li></ul><p>Then, for 10 games, the expected wins would be:</p><ul><li>GPT-4o-mini: 0.45×10=4.5</li><li>GPT-4o: 0.45 ×10=4.5</li><li>GPT-3.5-turbo: 0.1×10=10 → .1×10=1</li></ul><p>In addition, given the fact that GPT-4o-mini won 7 out of the 10 games, we also revise our alternative hypothesis:</p><blockquote><strong>Experiment-2 Revised Hypothesis, H1AR</strong>: GPT-4o-mini is better than GPT-4o.</blockquote><p>Using python to calculate the chi-squared test, we get:</p><pre>from scipy.stats import chisquare<br><br># Observed wins for the three models<br>observed = [7, 3, 0]<br><br># Expected wins under the null hypothesis (equal probability)<br>expected_wins = [0.45 * 10, 0.45 * 10, 0.1 * 10] <br><br># Perform the chi-square goodness-of-fit test<br>chi2_statistic, p_value = chisquare(f_obs=observed, <br>  f_exp=expected_wins)<br><br>chi2_statistic, p_value<br><br>(2.8888888888888890, .23587708298570023)</pre><p>With our updated probabilities, we see from the code above that seeing a result as extreme as we did, (7,3,0) is in fact not very unlikely under our new updated expected probabilities. Interpreting the p-value tells us that a results at least as extreme as what we observed would be expected 23% of the time. So, we cannot conclude with any statistical significance that there is a difference between GPT-4o-mini and GPT-4o and we reject the revised alternative hypothesis, <strong>H1AR.</strong></p><h4><strong>Key Takeaways</strong></h4><p>Although there is only limited evidence to suggest Claude is the more strategic model, we can with reasonably high confidence state that there is a difference in performance across model generations. GPT-3.5-turbo is significantly less strategic than its newer iterations. Obviously this implication works in reverse, which means we are seeing an increase in the strategic abilities of the models as they improve through the generations, and this is likely to profoundly impact how these models will be used in the future.</p><h3>Analyzing the Strategic Behavior of LLMs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7LTIQ75iC7nwy7ldONBTw.png" /><figcaption>Image generated by the author using DALL-E</figcaption></figure><p>One of the first things I noticed after running some initial tests were how different the LLMs play than humans. The LLM games seem to have more turns than human games, even after I prompted them to be more aggressive and try to go after weak opponents.</p><p>While many of the observations about player strategy can be made just from looking at plots of territory control and troop strength; some of the more detailed observations below first became clear as I watched the LLMs play turn-by-turn. This is slightly hard to replicate in an article format, however all the data from both experiments are stored in .csv files in the Github repo and loaded into pandas dataframes in the Jupyter notebooks used for analysis. The interested reader can find them in the repo here: /game_analysis/experiment1_analysis_notebook.ipynb. The dataframe experiment1_game_data_dfholds all relevant game data for Experiment-1. By looking at territory ownership and troop control turn-by-turn more details about the playstyles emerge.</p><h4>Distinctive Winning Play Styles</h4><p>What seemed to distinguish Anthropic’s model was its ability to claim a lot of territory in one move. This can be seen in some of the plots of territory control, when you look at individual games. But even though Claude had the most wins, how strategic was it really? Based on what I observed in the experiments, it seems that the LLMs are still rather immature when it comes to strategy. Below we discuss some of the typical behavior observed through the games.</p><h4>Poor Fortifying Strategies</h4><p>A common issue across all models was a failure to adequately fortify their borders. Quite frequently, the times the agents were also stuck with a lot of troops inside their internal territory instead of guarding their borders. This made it easier for neighbors to attack their territories and steal continent bonuses. In addition, it made it more difficult for the player agents to actually do a larger land-grab since often their territories with large troop strengths were surrounded by other territories it controlled.</p><h4>Failure to See Winning Moves</h4><p>Another noticeable shortcoming was the models’ failure to recognize winning moves. They don’t seem to realize that they can win in a turn if they play correctly. Less so with the stronger models but still present.</p><p>For example, for all the games in the simulations we played with 65% territory control to win. This means you just need to acquire 28 territories. In one instance during Experiment-2 Game 2, OpenAI’s GPT-4o had 24 territories and 19 troops in Greenland. It could easily just have taken Europe which has several territories with just 1 troop, however, it fails to see the move. This is a move that even a relatively inexperienced human player would likely recognize.</p><h4>Failure to Eliminate Other Players</h4><p>The models also frequently failed to eliminate opponents with only a few troops remaining, even when it would have been strategically advantageous. More specifically, they fail to remove players with only a few troops left and more than 2 cards. This would be considered an easy move for most human players, especially when playing with progressive cards. The card bonuses quickly escalate, and if an opponent only has 10 troops left, but 3 or more cards, taking him down for the cards is almost always the right move.</p><h4>GPT-4o Likes North America</h4><p>One of the very typical strategies that I saw GPT-4o pursue was to get early control over North America. Because of the strong continent bonus and the fact that it only requires to be guarded in 3 places means that is a strategically good starting point. I suspect the reason that GPT-4o does is because it has read as part of its training data that it is a strategically good location.</p><h4>Top Models Finish More Games</h4><p>Overall, there is a trend amongst the top models to finish more of the games, and achieve the victory conditions than weaker models. Of the games played with the top models, only 2 games went to the max game limit, while this happened 6 times for the weaker models.</p><h4><strong>Limitations of Pre-Trained Knowledge</strong></h4><p>A limitation of classic Risk is of course that the LLMs have read about strategies for playing Risk online, and then the top models are simply the best ones at executing on this. I think the tendency to quickly try to dominate North America highlights this. This limitation could be mitigated if we played with randomly generated maps instead. This would increase the difficulty level and would provide a higher bar for the models. However, given their performance on the current maps I don’t think we need to increase the difficulty for the current model generations.</p><h4>General observations</h4><p>Even the strongest LLMs are still far from mastering strategic gameplay. None of the models exhibited behavior that could challenge even an average human player. I think we have to wait at least one or two model generations before we can start to see a substantial increase in strategic behavior.</p><p>That said, dynamically adjusting prompts to handle specific scenarios — such as eliminating weak opponents for card bonuses — could improve performance. With different and more enhanced prompting the models might be able to put up more of a fight. To get that to work though, you would need to manually program in a range of possible scenarios that typically occur and offer specialized prompts for each scenario.</p><p>Consider a concrete example where we could see this come into play: Player B is weak and only has 4 territories with 10 troops, however player B has 3 Risk cards, and you are playing progressive cards and the reward for trading in cards is currently 20 troops.</p><p>For the sake of this experiment, I didn’t want to make the prompts too specialized, because the goal wasn’t to optimize agent behavior in Risk, but rather to test their ability to do that themselves, given the game state.</p><h3>What These Results Mean for the Future of AI and Strategy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XJM-sqMLXAeViMaj3IhVaA.png" /><figcaption>Image generated by the author using DALL-E</figcaption></figure><p>The results from these experiments highlight a few key considerations for the future of AI and its strategic applications. While LLMs have shown remarkable improvements in language comprehension and problem-solving, their ability to reason and act strategically is still in its early stages.</p><h4><strong>Strategic Awareness and AI Evolution</strong></h4><p>As seen in the simulations, the current generation of LLMs struggles with basic strategic concepts like fortification and recognizing winning moves. This indicates that even though AI models have improved in many areas, the sophistication required for high-level strategic thinking remains underdeveloped.</p><p>However, as we clearly saw in Experiment-2, there is a trend towards improved strategic thinking, and if this trend keeps going for future generations we probably don’t have to wait too long until the models are much more capable. There are people claiming the LLMs have already plateaued, however I would be very careful assuming that.</p><h4><strong>Implications for Real-World Applications</strong></h4><p>The real-world applications of a strategically aware and capable AI agent are obviously enormous and cannot be understated. They could be used in anything from business strategy to military planning and complex human interaction. A strategic AI that can anticipate and react to the actions of others could be incredibly valuable — and of course also very dangerous. Below we present three possible applications.</p><p>If we consider a more positive application first, we could imagine everyone having a helpful strategic agent guiding them through their daily lives, helping make important decisions. This agent could help with anything from financial planning, planning daily tasks, to optimizing social interactions and behavior that involves the actions of other humans. It could act on your behalf and be goal oriented to optimize your interests and well-being.</p><p>There are obviously many insidious application areas as well. Think: Autonomous fighter drones with on-board strategic capabilities. This might not be too far-fetched especially when we consider the relative strength of the smaller models compared to their big-brother counterparts, for example GPT-4o vs GPT-4o-mini. Smaller models are much easier to deploy on edge devices like drones, and when we see how <a href="https://www.atlanticcouncil.org/blogs/ukrainealert/fpv-drones-in-ukraine-are-changing-modern-warfare/">popular</a> drones have become the Russian-Ukraine war, taking the step from first person view (FPV) drone to unmanned AI-driven drone might be considered feasible. Perhaps even as a back-up option if the drone operator lost contact with the drone.</p><p>Detailed simulation of social interaction is a third way to use strategically aware agents. We could for example create simulations to model specific economic or other social phenomena, blending classic agent-based methods with LLMs. Agent based modelling (ABM) as a field of research and toolkit for understanding complex adaptive systems has existed for decades — I used in my Masters&#39; thesis back in 2012 — but coupled with much smarter and strategic agents this could potentially be game changing.</p><h4><strong>The Importance of Dynamic Prompting</strong></h4><p>Detailed dynamic prompting is probably one of the best ways to use and interact with LLMs in the near future — and perhaps also for the next few model generations (GPT-5, Claude 4, etc.). By providing more dynamic scenario-specific prompts and letting LLM agents execute specific plans, we might see more sophisticated strategic behavior in the next generation of models.</p><p>This type of “handholding” requires a lot more work from human programmers — than just prompting agents directly — however it could be a crucial stepping stone until the models become more capable of independent strategic thinking.</p><p>One could of course argue that if we provide too detailed and specific prompts we are working against the generalized nature of these models, and at that point we might as well introduce a different type of optimization algorithm, however I think there are many problems where the more open-ended problem-solving abilities of the LLMs could be paired with some form of dynamic prompting.</p><h4><strong>The Need for New Benchmarks</strong></h4><p>As the LLMs continue to improve, it will also be necessary to develop new benchmarks to study them. The traditional benchmarks and tests are well suited to study problem-solving in isolated environments, but moving forward we might want to introduce more strategic tests, that allow us to understand how the agents behave in situations where they need to consider how their actions influence others over time. Games like Risk provide a reasonable starting point because of their strategic nature and elements of uncertainty.</p><h4><strong>Future Considerations</strong></h4><p>Looking ahead, as AI models continue to evolve, it will be important to monitor their strategic capabilities closely. We need to ensure that as these models become more powerful, they are aligned with human values and ethical considerations. The risks associated with strategic AI — such as unintended consequences in high-stakes environments — must be carefully managed.</p><p>As smaller models like GPT-4o-mini have shown competitive performance in strategic tasks, there is potential for deploying highly capable AI on edge devices, such as drones or autonomous systems. This opens up new possibilities for decentralized AI applications that require real-time decision-making in dynamic environments.</p><h3>Conclusion</h3><p>I think it’s safe to say that that while the strategic capabilities of LLMs are improving with each new generation, they still have a long way to go before they can rival even a moderately skilled human player. Models like Claude and GPT-4o are beginning to show some level of strategic thinking, but their shortcomings in areas such as fortification and recognizing winning moves highlight the current limitations of AI in complex, multi-agent environments. Nevertheless, the trend toward better performance across newer models shows promise for future advancements in AI strategy.</p><p>As we continue to integrate AI into more aspects of life, from business to military strategy, understanding and refining the strategic capabilities of these systems will become increasingly important. While we’re not there yet, the potential for AI to handle complex decision-making processes in dynamic environments is incredible. It will be super interesting to see how the capabilities of the LLMs evolve over time, and if our results that show the improvements of LLMs across model generations continue through to GPT-5, GPT-6, Claude 4, Claude 5 etc. I think we are in for a wild ride!</p><p>If you are interested in developing your own AI driven tools, feel free to reach out! I am always happy to explore collaborative opportunities!</p><h3>Appendix</h3><p>Here I aim to provide some extra details that while interesting for the more technically inclined reader, might not be necessary for the full flow of the article. The first topic we touch on is rate limit issues. Then we describe more detailed analysis of errors, accumulated turn time used by the agents and parsing of responses from the LLMs. In addition, I provide the reader with a short description of how to test the code base out by cloning the Github repo and getting started with the docker setup.</p><h4>Rate Limit Issues</h4><p>There are many actions that needs to be considered every turn, and this leads to quite a lot of back-and-forth interaction between the program and the LLM providers. One issues that turned out to be slightly problematic for running longer experiments was rate limiting.</p><p>Rate limits are something the LLM providers set in place to protect against spamming an other potentially disrupting behavior, so even though you have funds in the accounts, the providers still limit the amount of tokens you can query. For example, Anthropic does a rate limit of 1M token / per day for their best model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZQlClm0kgNBCeCr3cQui7w.png" /><figcaption>Rate limits for Anthropic models, taken from Anthropic console</figcaption></figure><p>And when you hit your rate limit, your LLM queries are answered with</p><pre>Rate Limit Error: Rate limit reached for model `llama-3.1-70b-versatile` <br>in organization `org_01j440c04tfr3aas7qctr0ejtk` <br>on : Limit 1000000, Used 999496, Requested 1573. <br>Please try again in 1m32.2828s. <br>Visit https://console.groq.com/docs/rate-limits for more information.</pre><p>For many application areas this might not be a problem, however the simulation queries each of the providers multiple times per turn (for strategy evaluation, card choice, troop placement, multiple attacks and fortification) so this quickly adds up, especially in long games that go over many rounds. I was initially planning on doing 10 experiments on with victory condition set to World Domination (this means the winner would need to control all 42 territories in the game to win), but because of how the LLMs play the game this wasn’t feasible in my time frame. The victory conditions had to be adjusted so a winner could be determined at an earlier stage.</p><h4>Tracking Errors</h4><p>Some of the LLMs in the experiments were also struggling with a large number of errors when prompted for moves, this could be anything from trying to place troops in territories they didn’t control to fortifying to territories that were not connected. I implemented a few variables to track these errors. This was way more common with the weaker models as the plots below suggests:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*dA5AcGmgtiC0IlvIR8PRUg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*o-iCqEal3A3jXC8plb0fJA.png" /><figcaption>Experiment-1 attack and fortify errors / image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*6nyKsxslFeW2YYbQvS34yg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*xbOxfVVpT4iWWEt4vI0A_g.png" /><figcaption>Experiment-2 attack and fortify errors / image by author</figcaption></figure><h4>Accumulated Turn Time</h4><p>The last thing I tracked during the experiments was how much time each of the LLMs used on their actions. As expected, the largest and more complex models used the most time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*jhkM5TzhTNWhNRy0jpF1Tw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*G4hLjs6_G1aVdtv1PkcMfA.png" /><figcaption>Accumulated turn time by player / image by author</figcaption></figure><p>What is clear is that Claude seems to be really taking it’s time. For experiment-1 GPT-4 is coming out better than llama3.1 70b running on Groq but this is likely due to the fact that there were more issues with internal server response errors etc., in addition to errors in the returned answers, which lead to the turn time going up. For pure inference, when it provides the correct response, Groq was marginally faster than OpenAI.</p><h4>Trending Towards Less Mistakes and More Robust Output</h4><p>As we could see from the model improved model generations, the new LLMs are generating far less erroneous output than the older models. This is important as we continue to build data products with the models and integrate them into pipelines. There will likely still be the need for some post-prompt error handling but less than before.</p><h4>Parsing Responses</h4><p>A key issue in interacting with the LLMs is to parse the output that they produce. OpenAI recently <a href="https://openai.com/index/introducing-structured-outputs-in-the-api/">revealed </a>that GPT-4o, can <em>“now reliably adhere to developer-supplied JSON Schemas.” </em>So, this is of course amazing news, but many of the other models, such as llama 3.1 70B still struggled to consistently return JSON output in the right format.</p><p>The solution to the parsing problem ended up packing the output into special text strings such as ||| output 1 |||, +++ output 2+++ and then using regex to parse those output strings. I simply prompt the LLM to format the output using the special text strings, and also provide examples of correctly formatted output. I guess because of how the LLMs are inherently sequence based this type of formatting is easier out of the box than for example asking it to return a complex JSON object. For a concrete example, see below:</p><pre>&#39;&#39;&#39;Your response should be in the following format:<br>    Move:|||Territory, Number of troops|||<br>    Reasoning:+++Reasoning for move+++p<br><br>For example:<br>    Move:|||Brazil, 1|||<br>    Reasoning:+++Brazil is a key territory in South America.+++&#39;&#39;&#39;<br></pre><h4>Trying Out the Code and Running Your Own Experiments</h4><p>I developed the package for the risk_game engine in addition to the modules and Jupyter notebook inside a docker container, and everything is self contained. So for anyone interested in trying out the simulator and run your own experiments all the code is available and should be very easy to run from the Github repo.</p><p><a href="https://github.com/hcekne/risk-game.git">GitHub - hcekne/risk-game</a></p><p>Clone the repo and follow the instructions in the README.md file. It should be pretty straightforward. The only thing you need to change to get everything running on your own machine is the .env_example file. You need to put in your own API keys for the relevant LLM providers and change the name of the file to .env.</p><p>Then run the start_container.sh script. This is just a bash script that initializes some environment variables and runs a docker compose .yml file. This file configures the appropriate settings for the docker container, and everything should start up by itself. (The reason we feed these environment variables into the docker container is because when doing in-container development you can run into an issue with file permissions on the files that are created in the container. This is fixed if we change the container user to your user, then the files created by the container will have the same owner as the user on the machine running the container.)</p><p><em>If you enjoyed reading this article and would like to access more content from me please feel free to connect with me on LinkedIn at </em><a href="https://www.linkedin.com/in/hans-christian-ekne-1760a259/"><em>https://www.linkedin.com/in/hans-christian-ekne-1760a259/</em></a><em> or visit my webpage at </em><a href="https://www.ekneconsulting.com/"><em>https://www.ekneconsulting.com/</em></a><em> to explore some of the services I offer. Don’t hesitate to reach out via email at hce@ekneconsulting.com</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43c868d83c3b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/exploring-the-strategic-capabilities-of-llms-in-a-risk-game-setting-43c868d83c3b">Exploring the Strategic Capabilities of LLMs in a Risk Game Setting</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leading Data Science Teams to Success]]></title>
            <link>https://medium.com/data-science/leading-data-science-teams-to-success-8292fe67de4f?source=rss-ab0bff3c2cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/8292fe67de4f</guid>
            <category><![CDATA[data-science-consulting]]></category>
            <category><![CDATA[office-hours]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[teamwork]]></category>
            <category><![CDATA[project-management]]></category>
            <dc:creator><![CDATA[Hans Christian Ekne]]></dc:creator>
            <pubDate>Wed, 31 Jul 2024 13:21:56 GMT</pubDate>
            <atom:updated>2024-07-31T13:21:56.941Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@hc.ekne/list/data-science-consulting-bcf6e4b59652">Data Science Consulting</a></h4><h4>Practical Strategies for Successful Project Delivery in Challenging Collaborative Environments</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pEyObNkVseu9g9MSy1XUCA.png" /><figcaption>Image generated by the author using DALL-E</figcaption></figure><h3>Introduction</h3><p>Have you ever asked a team member for a deliverable and received something completely different? Or sent an email to a client project manager and received no response, even after multiple follow-ups?</p><p>Delivering products with a team can sometimes be challenging, and if you are the team leader, it is your responsibility to work through those challenges and still have a successful project. This can be particularly demanding in consulting data science projects, where collaboration with client resources is essential.</p><p>In addition, when delivering end-to-end data science solutions, the need for close integration with client infrastructure, access to data, and frequent stakeholder feedback means you will typically work with a cross-functional team. This team may include business analysts, data scientists, DevOps engineers, data engineers, project coordinators, and domain experts. Each member brings unique skills and perspectives, but they also come with their own sets of challenges.</p><p>The key to navigating these issues lies in an understanding of individual team members capabilities, weaknesses and strengths, proper planning and a focus on goal achievement. As the team leader, you must be agile and adapt your plans based on the strengths and weaknesses of your team, but also sometimes evaluate your own behavior, to see how your own actions are perceived. Below, I explore some common team challenges in data science projects and offer some key takeaways to ensure successful project delivery.</p><h3>Challenges and Interventions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qxu8Dd6V3wGr5a1WjqHzoQ.png" /><figcaption>Image generated by the author using DALL-E</figcaption></figure><p>Leading data science projects involves overcoming a variety of challenges, from misunderstandings and lack of commercial insight to low productivity and disengaged stakeholders. Effective intervention often requires a combination of reflection, introspection, adaptive planning, one-on-one meetings and hands-on leadership. Here, I discuss specific challenges I have encountered and the strategies I employed to address them successfully.</p><h4>Lack of understanding and underlying commercial insight</h4><p><strong>Challenge:</strong> On one of my earlier projects, I was put in charge of a couple of data scientists that were struggling to deliver on a project. They had built an algorithm that was giving the opposite answer to what they were expecting. In this case their model predicted that churn would decrease as price increased. The consulting partners on the project were not impressed, and needed a viable solution that the client could accept.</p><p><strong>Intervention:</strong> My first course of action was to establish what we intuitively should expect from the algorithm. Then, I immersed myself in the data preparation and pipeline. (I totally support the common wisdom that data scientist spends 80% of their time cleaning and prepping data, and only 20% on generating insights.) This helped me discover that the issue was caused by how the data was being prepared and transformed from the source data, what data was included in the model and what other assumptions they made during the modelling process.</p><p>In addition to trying to <strong>lead by example</strong> and being hands on with the data, I also had frequent<strong> one-on-one meetings</strong> with the data scientist. These meetings let me clarify my expectations and better understand why they had gone down the wrong path with regards to modelling and data. Furthermore, I provided feedback on how we could course correct and get back on track. It was also important to constantly remind the data scientists about the <strong>end goal</strong> we were trying to achieve with the project.</p><h4>Too academic without focus on the end goal of the project</h4><p><strong>Challenge</strong>: As is often common with data science and data engineering projects there are a lot smart people involved. Usually this is great, but sometimes good ideas can get in the way of delivering on your goal. I have often seen data scientists — and I myself have been guilty of this — delve too deep into a given problem and perhaps slightly loosing focus on what we are trying to achieve. This issue becomes extra acute in a hectic project setting, where we face budget, time and resource constraints.</p><p>On one occasion, I tasked a data scientist with developing a model for predicting churn, however he got stuck on the details and kept going down an analysis path that was not fruitful — despite not making significant gains. The data scientist was very rigorous in his approach and had difficulty taking a more pragmatic view of the situation. In this case we were also looking for a connection between churn and price, however the way the data was structured, and the algorithms that were being applied were not conducive to the end goal.</p><p>This challenge doesn’t only apply to team members, and as team leader you yourself can also easily fall into this trap. On a recent project, I was working a business case and was determined I had a great solution for approximating the ROI of our project. It was rigorous, bottom up and didn’t require any wild assumptions to be able to show our case. However, after working on it for a few days it turned out it was way too time consuming and complex for what we really needed. They client didn’t fully understand the methodology and it was clear we needed a simpler and more intuitive approach.</p><p><strong>Intervention:</strong> In both cases, the solution was <strong>introspection and reflection </strong>on outcomes and use of time. Regarding the situation with the data scientist, I had <strong>one-on-one meetings</strong> with him to align on what we needed to achieve and deliver. What our <strong>end goal</strong> was. And for myself, by listening to feedback from my team and reflecting on my time consumption and progress, I was able to course correct and adjust the approach using a simpler technique that still met our objectives and allowed us to quantify the ROI of our project.</p><h4>Low productivity team members</h4><p><strong>Challenge</strong>: Not all team members will be able to produce the expected output when initially planning a project. However, as a consultant, you are often required to make development plans that integrate client personnel — despite not being able to assess their productivity beforehand. Various factors, such as being overloaded with projects or dealing with personal issues, can contribute to lower productivity. Beware of making overly detailed plans without understanding the available resources, as this can be a trap.</p><p>On one project I was informed by the client they would provide we me 2 data scientist FTEs (full time equivalents), 1 business analyst FTE and one data engineer FTE that would help me deliver the project. Based on this I tried to plan out how we might be able to deliver the project within the given timeframe. However, when time came to deliver it turned out one of my data scientists FTEs had a couple of weeks planned leave, and there were no one to replace him. It also turned out that the data engineer FTE didn’t really deliver and wasn’t able to produce the level of output we needed for the project.</p><p><strong>Intervention</strong>: The situation above was resolved by having <strong>short planning cycles</strong> and quickly adapting to changing conditions. I directed one of our more productive data science FTEs to pick up the slack and became very <strong>hands-on</strong> myself. This flexibility would have been difficult with longer, less adaptable plans.</p><h4>Disengaged or uncooperative client side project manager</h4><p><strong>Challenge</strong>: In many consulting projects you will have a counter party on the client side that manages the project internally. This will typically be a person who either will own the end product you are building or someone whose role it is to manage projects internally and knows how to help the project move forwards.</p><p>Usually, it’s great to have someone on the client side who can contribute and help drive the project — especially when you need to align with multiple stakeholders. But sometimes you end up with a disengaged or uncooperative manager that doesn’t respond to emails and becomes more of a bottleneck than a help. For data science projects, where you ideally want frequent feedback, this can be problematic.</p><p><strong>Intervention</strong>: Get <strong>regular meetings</strong> into their calendar. If you are used to weekly meetings with other managers, consider ramping that up to two meetings a week instead. Even if they don’t respond to emails, at least it is not long between <strong>in-person catch ups</strong>. If all else fails, another strategy can also be to use other channels to connect with the client. On one of my projects we had a consulting partner, who was not directly involved in the day-to-day operations of the project, reach out directly to the CMO on the client side and prompt the project manager to take more action.</p><p>Again, this is also a situation where <strong>reflection and introspection</strong> might be beneficial. Why is the project manager is behaving the way he is, is there anything about your own behavior that could have prompted it? Perhaps you did something or presented something in a way that rubbed off the wrong way? Try a <strong>one-on-one</strong> meeting to iron out any misunderstandings and align the focus on the <strong>end goal</strong> of the project.</p><h3>Key Takeaways and Guiding Principles</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*spbyUz16aS4re-NdSgGDkg.png" /><figcaption>Image generated by the author using DALL-E</figcaption></figure><p>This probably shouldn’t come as a huge surprise; it turns out many of the interventions we discussed above have a lot in common with the original principles underlying the agile software development movement. In the “12 Principles Behind the <a href="https://agilemanifesto.org/">Agile Manifesto</a>” we find a clear emphasis on a “<em>preference to the shorter timescale…</em> ”, and also ideas regarding reflection “<em>…reflects on how to become more effective, then tunes and adjusts its behavior accordingly.</em>” Additionally, there is also a focus on meeting and talking with people: “<em>The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.</em>” Since agile was originally created as a method for software teams to improve their development processes in response to the prevailing waterfall methodology, it makes sense that many of its principles also apply to data science projects.</p><p>I have tried to consolidate my interventions into a few guiding principles, and of course some of the strategies discussed here will apply to many different types of projects, not just data science.</p><h4>Planning with short iterations</h4><p>If you rely too heavily on a predetermined large plan, you risk setting yourself up for failure. Especially when you don’t know the team very well. This is why having shorter development cycles with being able to adapt to quickly will increase your chances of success.</p><h4>Keep the end goal in mind</h4><p>This is a very strong guiding principle, and I often see it being a good way to approach problems and issues with team members. Especially when their efforts and analysis have drifted too far away from what we are trying to achieve.</p><p>In many of the larger data science projects I have been involved in, the end goal has been to implement some kind of end-to-end machine learning system into the client architecture. This could be anything from custom-made pricing solutions to a fully fledged customer management platform. For example, if you are working on a price sensitivity algorithm that is going into the end customer pricing function, frequently evaluate whether your efforts and research is really advancing the end goal of the project.</p><h4>Lead by example</h4><p>In most of my projects, I have had positive experiences with taking a hands-on approach and working in the details with team. Leading by example is effective because it builds trust and respect. When leaders demonstrate the behavior and work ethic they expect from their team members, it sets a clear standard and motivates others to follow suit.</p><p>Leading by example can be especially important when you have low productive team members. Also, it lets you be more flexible in your planning and it’s easier to jump in whenever parts of the development need more attention. Ultimately, I believe leading by example helps create a culture of integrity, collaboration, and mutual support.</p><h4>One-to-one meetings</h4><p>One-to-one meetings are particularly beneficial when leading data science projects due to the personalized communication and individual focus they provide. These meetings create a space for deeper understanding between managers and team members, allowing for tailored support and guidance on complex tasks. This is extra beneficial when you are a consultant and don’t know all the team members.</p><p>When you need to align with team members, one-to-one meetings offer a private space for constructive feedback, and ensure coordination between individual contributions and project goals.</p><h4>Evaluate yourself and how you are perceived</h4><p>Lastly, resolving issues and moving a team forward sometimes requires introspection from the team leader. Working with new people can be challenging, especially when coming from different countries and cultures. For example, I am from Norway, where direct and informal communication is common. This contrasts with the UK, where communication tends to be more subtle and polite.</p><p>I have personally experienced times when I was too focused on the end goal, assuming everyone was on the same page, without stopping to coordinate with the team. Additionally, I sometimes find myself becoming too technical when explaining difficult subjects. If team members don’t understand, they can feel excluded and frustrated, and you might come across as a know-it-all or arrogant. This is particularly challenging in data science, where complex topics often need to be explained quickly.</p><h3>It’s Not All Doom and Gloom</h3><p>While this article focused mainly on the challenging aspects of working in a data science team, there are of course many aspects of it that are amazing, and why I continue to do it. I have met some really smart and engaging people who have contributed to my data science journey and taught me invaluable skills. In addition, together as teams we have been able to deliver way more than any of one of us could have done.</p><p>I would not be without those experiences, and I am grateful for the opportunity to be able to work on a diverse set of projects, interesting people and data challenges. (I even met my future wife and the mother of two of my boys on a project!!)</p><h3>Conclusion</h3><p>Leading data science teams comes with its unique set of challenges. From dealing with varying productivity levels and ensuring alignment with client resources, to maintaining focus on the end goals and fostering effective communication, the role demands a versatile and adaptive approach. By maintaining short planning cycles, leading by example, and utilizing one-on-one meetings for personalized guidance, team leaders can steer through these challenges more effectively.</p><p>While obstacles may be numerous, the rewards of working with a great team to solve complex problems are substantial. The experiences enhance professional growth and contribute to personal development and meaningful connections. Embrace the challenges, learn from each project, and continue to grow as a leader!</p><p><em>If you enjoyed reading this article and would like to access more content from me please feel free to connect with me on LinkedIn at </em><a href="https://www.linkedin.com/in/hans-christian-ekne-1760a259/"><em>https://www.linkedin.com/in/hans-christian-ekne-1760a259/</em></a><em> or visit my webpage at </em><a href="https://www.ekneconsulting.com/"><em>https://www.ekneconsulting.com/</em></a><em> to explore some of the services I offer. Don’t hesitate to reach out via email at hce@ekneconsulting.com</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8292fe67de4f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/leading-data-science-teams-to-success-8292fe67de4f">Leading Data Science Teams to Success</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Deliver Successful Data Science Consulting Projects]]></title>
            <link>https://medium.com/data-science/how-to-deliver-successful-data-science-consulting-projects-d4c25f826c36?source=rss-ab0bff3c2cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4c25f826c36</guid>
            <category><![CDATA[data-science-consulting]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[consulting]]></category>
            <dc:creator><![CDATA[Hans Christian Ekne]]></dc:creator>
            <pubDate>Sat, 13 Jul 2024 01:17:05 GMT</pubDate>
            <atom:updated>2024-07-31T16:13:30.789Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@hc.ekne/list/data-science-consulting-bcf6e4b59652">DATA SCIENCE CONSULTING</a></h4><h4>Key recommendations for how to succeed with data science consulting projects and build lasting client relationships</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5xDxHaopa5KiKUFDuMbNvQ.jpeg" /><figcaption>Image generated by the author using DALL-E</figcaption></figure><h3>Introduction</h3><p>I am not ashamed to say it: Data science consulting is not always easy! It can be brutal — especially at senior levels when you need to generate sales to stay in the game. Even if keeping your clients happy is at the top of your priority list, it isn’t always a trivial exercise to do that with data science projects.</p><p>Reflecting on over a decade of delivering data science and data engineering projects — the majority of which has been as a consultant — I have seen projects deliver incredible value to clients, but I have also seen projects stumble and falter, delivering mediocre results, often due to poor planning, misaligned expectations and technical difficulties.</p><p>It is clear that successful data science consulting isn’t just about being a Python and R wizard — acing Hackerrank data science programming competitions; it’s much deeper than that, being able to blend strategy, technology and analytics in project deliverables. In this article, I will draw insights from both successful and challenging projects and illustrate how data science consulting intersects with management, IT and analytics consulting. By understanding the differences and similarities between these more traditional and established consulting roles it becomes easier to see how we can deliver lasting and impactful solutions in our data science projects.</p><h3>Similarities and Differences with Traditional Consulting Roles</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2gI3zZ8usLIRBUXpNMTi3A.jpeg" /><figcaption>Data science consulting / image by the author</figcaption></figure><h4>Management Consulting vs. Data Science Consulting</h4><p>Management consulting projects and data science consulting projects usually share many of the same characteristics. They both try to enhance business performance and often deliver projects of strategic importance. They also usually involve senior executives — at least as project owners — and they both require detailed upfront planning. However, while I find that management consulting projects usually follow a traditional waterfall approach, data science consulting can benefit more from a hybrid approach that combines some of the structured planning from waterfall with agile iterations.</p><p>Regarding deliverables, management consulting projects will typically produce strategic plans, organizational assessments or pricing recommendations, and they are often contained in a power point deck or and an excel workbook. This is in contrast to data science projects, where the main deliverables will typically be predictive models, data pipelines and dashboards.</p><p>The feedback cycles are also different, while the traditional management consulting projects will typically have planned periodic feedback, the data science consulting projects are more successful with continuous stakeholder involvement and iterative feedback cycles.</p><h4>IT Consulting vs. Data Science Consulting</h4><p>IT consulting and data science consulting both require strong technical expertise and an understanding of IT infrastructure, and some people might argue that data science consulting is a subcategory of IT consulting. I don’t subscribe to that belief, but the two fields have a considerable overlap.</p><p>While IT consulting projects produces system architecture designs, implementation plans and more core system software development, data science consulting is more centered around generating predictive models, the pipelines that feed the models and data-driven insights.</p><p>Additionally, data science projects are often very closely connected to the business side of the organizations and will in many cases have leaders outside of IT as project owners and sponsors. For example, you can find yourself delivering directly to the CMO (Chief Marketing Officer) or the CDO (Chief Data Officer) instead of the CIO (Chief Information Officer) or CTO (Chief Technology Officer), as is often the case with more IT-driven projects. This means that your approach and method needs to be tailored differently than with pure IT projects, and you often need to make more considerations regarding continuity, operations and the maturity of the organization. (The CIOs and CTOs will already typically have more elaborate systems in place to deal with the above issues.)</p><p>However, as data science has matured over the last years, we have seen fields like MLOps evolve significantly, and also a trend in projects to develop and write code closer to how modern software development is undertaken. I see this as a natural part of the evolution of data science towards a more professional and structured field.</p><h4>Analytics Consulting vs. Data Science Consulting</h4><p>Lastly, comparing analytics to data science consulting, we find that they both share a focus on data and typically involve shorter project durations with targeted deliverables. They also emphasize rapid iterations and continuous refinement based on data and feedback. However, analytics consulting often addresses specific business questions with more narrow scopes, while data science consulting integrates a broader strategic alignment and more complex models.</p><p>The classical deliverables within analytics consulting would usually be data analysis reports, data marts and dashboards, whereas data science consulting deliverables — as we have mentioned previously — are more centered around pipelines, models and strategic and integrated data products.</p><p>Finally, how the projects are evaluated if usually also different. While the analytics projects are assessed more around the analysis accuracy and insights, the data science projects will typically in addition be measured on predictive accuracy and return on investment.</p><p><em>Notes to the critical reader: The above descriptions are by no means set in stone and are meant to show the general differences and similarities between the various classes of consulting. In many cases they will obviously diverge or merge more than above.</em></p><h3>Key Insights for Successful Data Science Consulting</h3><p>Given the above similarities and differences between data science consulting and other classes of consulting, it is natural to ask how we might adapt our approach to ensure the long term success and viability of our projects. Apart from the obvious elements such as quality deliverables, timely project delivery and strong stakeholder management, what are the other components that need to be in place to succeed?</p><h4>Ensuring Robust Data Products</h4><p>While management consulting typically focuses on immediate organizational changes and one-off deliverables, data science consulting requires a long-term perspective on robustness and sustainability. This has a couple of consequences. You can and will be judged on the continued performance of your work and should take steps to ensure you deliver good results not just at the moment of handover, but also potentially for years to come. (This is similar to IT consulting, where ongoing performance and maintenance are essential.)</p><p>For instance, I’ve built data products that have been in production for over 6 years! I have seen the direct effects of having data pipelines that are not robust enough, leading to system crashes and erroneous model results. I have also seen model variables and labels drift significantly over time, leading to degradation of system performance and in some cases completely wrong insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lBZKiLZ9zR16AtgxiLTNlg.jpeg" /><figcaption>Image by the author using DALL-E</figcaption></figure><p>I know that this is obviously not the most sexy topic, and in a project with tight budgets and short timelines it can be hard to make the argument to spend extra time and resources on robust data pipelines and monitoring of variable drift. However, I strongly compel you to spend time with your client on these topics, integrating them directly into your project timeline.</p><p>- Focus on long-term sustainability.<br>- Implement robust data pipelines.<br>- Monitor for model and variable drift continuously.</p><p>I have written about one aspect of data pipelines (one-hot encoding of variables) in a previous article that aims to illustrate the topic and provide solutions in Python and R.</p><p><a href="https://towardsdatascience.com/robust-one-hot-encoding-930b5f8943af">Robust One-Hot Encoding</a></p><h4>Documentation and Knowledge Transfer</h4><p>Proper documentation and knowledge transfer is critical in data science consulting. Unlike analytics consulting, which might involve less complex models, data science projects require thorough documentation to ensure continuity. Clients often face personnel changes, and well-documented processes help mitigate the loss of information. I have on multiple occasions been contacted by previous clients and asked to explain various aspects of the models and systems we built. This is not always easy — especially when you haven’t seen the codebase for years— and it can be very handy to have properly documented Jupyter Notebooks or Markdown documents, describing the decision process and analysis. This ensures that any decisions or initial results can easily be traced back and resolved.</p><p>- Ensure thorough documentation.<br>- Use Jupyter Notebooks, Markdown documents or similar.<br>- Facilitate knowledge transfer to mitigate personnel changes.</p><h4>Building End-to-End Solutions</h4><p>Building end-to-end solutions is another key consideration in data science consulting. Unlike analytics consulting, which might focus on delivering insights and reports, data science consulting needs to ensure the deployability and operationalization of models. This is similar to IT consulting, where integration into existing CI/CD pipelines is crucial.</p><p>I’ve seen companies waste years from the development of a model to its production deployment due to personnel changes and unfinished integration tasks. If we had insisted on seeing the project through to full production ready status, the client would have had the full benefits of the model much earlier. A significant improvement when project costs can be in the millions of euros.</p><p>- Build deployable models.<br>- Ensure operationalization.<br>- Integrate into existing CI/CD pipelines.</p><h4>Visual Artifacts</h4><p>Including visual artifacts, such as dashboards or widgets, helps demonstrate the value created by the project. While management consulting deliverables include strategic plans and assessments — usually in the form of a one-off power point deck — data science consulting benefits from visual tools that provide ongoing insights into the impact and benefits of the solution. These artifacts serve as reminders of the project’s value and help in measuring its success, similar to the role of visualizations in analytics consulting.</p><p>One of my most successful projects was when we built a pricing solution for a client and they started using the dashboard component directly in their monthly pricing committee meetings. Even though the dashboard was only a small fraction of the project, it was the only thing that management and company executives could interact with, and thus provided a powerful reminder of our work.</p><p>- Create visual artifacts like dashboards.<br>- Demonstrate project value visually.<br>- Use artifacts to measure success and stay relevant to the client.</p><h4>Evaluating Organizational Maturity</h4><p>Evaluating organizational maturity while planning the project is essential to avoid over-engineering the solution. Tailoring the complexity of the solutions to the client’s maturity level ensures better adoption and usability. Always remember that when you are finished with the project, ownership usually shifts to internal data scientists and data engineers. If the client has a team of 20 data scientists and a modern data infrastructure ready to integrate your models directly into their existing DevOps, that’s amazing, but frequently not the case. Consider instead the scenario where you are developing a tool for a company with 20 employees, with only one junior data scientist and and an over-worked data engineer. How would you adapt your strategy?</p><p>- Assess organizational and analytical maturity.<br>- Avoid over-engineering solutions.<br>- Tailor complexity to client readiness.</p><h4>Following Best Practices in IT Development</h4><p>Following best practices in IT development is becoming increasingly important and often required in data science consulting. Unlike analytics consulting, which might not involve extensive coding, data science consulting should stay true to software development practices to ensure scalability and maintainability. This is similar to modern IT consulting, where writing modular, well-documented code and including sample data for testing are essential practices.</p><p>This also ties back to the previous point around documentation and knowledge transfer. Properly documented and structured code, packaged into easy to install software packages and libraries is much easier to maintain and manage than 1000s of lines of spaghetti code. When personnel changes occur, you will be in a much better spot if the code has been properly developed.</p><p>- Follow IT development best practices.<br>- Write modular and well-documented code.<br>- Include sample data for testing.</p><p>I want to end this article with video of Steve Jobs talking about consulting. He clearly doesn’t have too much sympathy for traditional consultants, however I feel that as data scientist consultants we need to be more true to his ideas around really taking ownership of the advice and products that we build. We are measured not just on the successful completion of the project but the ongoing and lasting value we create.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F-c4CNB80SRc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D-c4CNB80SRc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F-c4CNB80SRc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/173879aa0034f1d9cea308b2217fd50d/href">https://medium.com/media/173879aa0034f1d9cea308b2217fd50d/href</a></iframe><h3>Conclusion</h3><p>Data science consulting is an exciting and complex profession that draws from management, IT, and analytics consulting. By understanding the similarities and differences and applying best practices from each, you can deliver successful data science consulting projects that drive long term value for your clients. I believe this is a necessity if you want to build a successful data science consulting business. My personal experiences highlight the importance of robust solutions, thorough documentation, end-to-end deliverables, visual artifacts, evaluating organizational maturity, and following IT development best practices in ensuring the success of data science consulting projects.</p><p>Thanks for reading!</p><p><em>If you enjoyed reading this article and would like to access more content from me please feel free to connect with me on LinkedIn at </em><a href="https://www.linkedin.com/in/hans-christian-ekne-1760a259/"><em>https://www.linkedin.com/in/hans-christian-ekne-1760a259/</em></a><em> or visit my webpage at </em><a href="https://www.ekneconsulting.com/"><em>https://www.ekneconsulting.com/</em></a><em> to explore some of the services I offer. Don’t hesitate to reach out via email at hce@ekneconsulting.com</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4c25f826c36" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-deliver-successful-data-science-consulting-projects-d4c25f826c36">How to Deliver Successful Data Science Consulting Projects</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Robust One-Hot Encoding]]></title>
            <link>https://medium.com/data-science/robust-one-hot-encoding-930b5f8943af?source=rss-ab0bff3c2cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/930b5f8943af</guid>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[one-hot-encoding]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Hans Christian Ekne]]></dc:creator>
            <pubDate>Fri, 26 Apr 2024 17:28:58 GMT</pubDate>
            <atom:updated>2024-04-26T17:28:58.408Z</atom:updated>
            <content:encoded><![CDATA[<h4>Production grade one-hot encoding techniques in Python and R</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z1bwjCLEH00GH5hcd91lZA.png" /><figcaption>Image generated by the author using DALL-E / or Dali?;)</figcaption></figure><p>Have you faced a crash in your machine learning production environments?</p><p>It’s not fun, and especially when it comes to issues that could be avoided. One issue that frequently causes problems is one-hot encoding of data. Drawing from my own experience, I’ve learned that many of these issues can largely be avoided by following a few best practices related to one-hot encoding. In this article I will briefly introduce the topic with a few simple examples and share some best practices to ensure stability of your machine learning models.</p><h3>One-hot encoding</h3><h4>What is one-hot encoding?</h4><p>One-hot encoding is the practice of turning a factor variable that is stored in a column into dummy variables stored over multiple columns and represented as 0s and 1s. A simple example illustrates the concept.</p><p>Consider for example this dataset with some numbers and some columns for colours:</p><pre>import pandas as pd<br><br># Creating the training_data DataFrame in Python<br>training_data = pd.DataFrame({<br>    &#39;numerical_1&#39;: [1, 2, 3, 4, 5, 6, 7, 8],<br>    &#39;color_1_&#39;: [&#39;black&#39;, &#39;black&#39;, &#39;red&#39;, &#39;green&#39;, <br>                &#39;green&#39;, &#39;black&#39;, &#39;red&#39;, &#39;blue&#39;],<br>    &#39;color_2_&#39;: [&#39;black&#39;, &#39;blue&#39;, &#39;pink&#39;, &#39;purple&#39;, <br>                &#39;black&#39;, &#39;blue&#39;, &#39;pink&#39;, &#39;purple&#39;]<br>})</pre><p>Or more visually:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*u-oi5VnAa8wnuoiI0-Y8iA.png" /><figcaption>Training data with 3 columns / image by author</figcaption></figure><p>The column color_1_could also be represented like in the table below:</p><figure><img alt="encoded variables of color_1" src="https://cdn-images-1.medium.com/max/667/1*C14Y61ad93EcG5xQ6ECcdw.png" /><figcaption>One-hot encoded representation of “color_1_” / image by author</figcaption></figure><p>Changing color_1_ from a one-column compact representation of a categorical variable into a multi-column binary representation is what we call one-hot encoding.</p><h4>Why do we use it?</h4><p>There are multiple reasons to use one-hot encoding. They could be related to avoiding implicit ordering, improving model performance, or just making the data compatible with various algorithms.</p><p>For example, when you encode a categorical variable like colour, into a numerical structure, (e.g. 1 for black, 2 for green, 3 for red) without converting it to dummy variables, a model could mistakenly misinterpret the data to imply an order ( black &lt; green &lt; red) when no such order exists.</p><p>Also, when training neural nets, it is best practice to normalize the data before sending it into the neural net, and with categorical variables, one-hot encoding can be a good method. Other linear models, like logistic and linear regression assume linear relationships and numerical inputs so for this class of models, one-hot encoding can be a good idea as well.</p><p>In addition, the process of doing one-hot encoding forces us to ensure we don’t feed unseen factor levels into our machine learning models.</p><p>Ultimately, one-hot encoding makes it easier for the machine learning models to interpret the data and thus make better predictions.</p><h3>The main reasons why one-hot encoding fails</h3><p>The way we build traditional machine learning models is to first train the models on a “training dataset” — typically a dataset of historic values — and then later we generate predictions on a new dataset, the “inference dataset.” If the columns of the training dataset and the inference dataset don’t match, your machine learning algorithm will usually fail. This is primarily due to either missing or new factor levels in the inference dataset.</p><h4>The first problem: Missing factors</h4><p>For the following examples, assume that you used the dataset above to train your machine learning model. You one-hot encoded the dataset into dummy variables, and your fully transformed training data looks like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FZVITiRYd5VIVJC4CNqM1w.png" /><figcaption>Transformed training dataset with pd.get_dummies / image by author</figcaption></figure><p>Now, let’s introduce the inference dataset, this is what you would use for making predictions. Let’s say it is given like below:</p><pre># Creating the inference_data DataFrame in Python<br>inference_data = pd.DataFrame({<br>    &#39;numerical_1&#39;: [11, 12, 13, 14, 15, 16, 17, 18],<br>    &#39;color_1_&#39;: [&#39;black&#39;, &#39;blue&#39;, &#39;black&#39;, &#39;green&#39;, <br>                &#39;green&#39;, &#39;black&#39;, &#39;black&#39;, &#39;blue&#39;],<br>    &#39;color_2_&#39;: [&#39;orange&#39;, &#39;orange&#39;, &#39;black&#39;, &#39;orange&#39;, <br>                &#39;black&#39;, &#39;orange&#39;, &#39;orange&#39;, &#39;orange&#39;]<br>})</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*7FtIFNFnm1NDy5O4QXuN9Q.png" /><figcaption>Inference data with 3 columns / image by author</figcaption></figure><p>Using a naive one-hot encoding strategy like we used above (pd.get_dummies)</p><pre># Converting categorical columns in inference_data to <br># Dummy variables with integers<br>inference_data_dummies = pd.get_dummies(inference_data, <br>  columns=[&#39;color_1_&#39;, &#39;color_2_&#39;]).astype(int)</pre><p>This would transform your inference dataset in the same way, and you obtain the dataset below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*LL8zZxQa0IH6FhN7ddqHjw.png" /><figcaption>Transformed inference dataset with pd.get_dummies / image by author</figcaption></figure><p>Do you notice the problems? The first problem is that the inference dataset is missing the columns:</p><pre>missing_colmns =[&#39;color_1__red&#39;, &#39;color_2__pink&#39;, <br>  &#39;color_2__blue&#39;, &#39;color_2__purple&#39;]</pre><p>If you ran this in a model trained with the “training dataset” it would usually crash.</p><h4>The second problem: New factors</h4><p>The other problem that can occur with one-hot encoding is if your inference dataset includes new and unseen factors. Consider again the same datasets as above. If you examine closely, you see that the inference dataset now has a new column: color_2__orange.</p><p>This is the opposite problem as previously, and our inference dataset contains new columns which our training dataset didn’t have. This is actually a common occurrence and can happen if one of your factor variables had changes. For example, if the colours above represent colours of a car, and a car producer suddenly started making orange cars, then this data might not be available in the training data, but could nonetheless show up in the inference data. In this case you need a robust way of dealing with the issue.</p><p>One could argue, well why don’t you list all the columns in the transformed training dataset as columns that would be needed for your inference dataset? The problem here is that you often don’t know what factor levels are in the training data upfront.</p><p>For example, new levels could be introduced regularly, which could make it difficult to maintain. On top of that comes the process of then matching your inference dataset with the training data, so you would need to check all actual transformed column names that went into the training algorithm, and then match them with the transformed inference dataset. If any columns were missing you would need to insert new columns with 0 values and if you had extra columns, like the color_2__orange columns above, those would need to be deleted. This is a rather cumbersome way of solving the issue, and thankfully there are better options available.</p><h3>The solution</h3><p>The solution to this problem is rather straightforward, however many of the packages and libraries that attempt to streamline the process of creating prediction models fail to implement it well. The key lies in having a function or class that is first fitted on the training data, and then use that same instance of the function or class to transform both the training dataset and the inference dataset. Below we explore how this is done using both Python and R.</p><h4>In Python</h4><p>Python is arguably one the best programming language to use for machine learning, largely due to its extensive network of developers and mature package libraries, and its ease of use, which promotes rapid development.</p><p>Regarding the issues related to one-hot encoding we described above, they can be mitigated by using the widely available and tested scikit-learn library, and more specifically the sklearn.preprocessing.OneHotEncoder class. So, let’s see how we can use that on our training and inference datasets to create a robust one-hot encoding.</p><pre>from sklearn.preprocessing import OneHotEncoder<br><br># Initialize the encoder<br>enc = OneHotEncoder(handle_unknown=&#39;ignore&#39;)<br><br># Define columns to transform<br>trans_columns = [&#39;color_1_&#39;, &#39;color_2_&#39;]<br><br># Fit and transform the data<br>enc_data = enc.fit_transform(training_data[trans_columns])<br><br># Get feature names<br>feature_names = enc.get_feature_names_out(trans_columns)<br><br># Convert to DataFrame<br>enc_df = pd.DataFrame(enc_data.toarray(), <br>                          columns=feature_names)<br><br># Concatenate with the numerical data<br>final_df = pd.concat([training_data[[&#39;numerical_1&#39;]], <br>                      enc_df], axis=1)</pre><p>This produces a final DataFrameof transformed values as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GFzZ9jT7Dmzlf7Q87uZbhw.png" /><figcaption>Transformed training dataset with sklearn / image by author</figcaption></figure><p>If we break down the code above, we see that the first step is to initialize the an instance of the encoder class. We use the option handle_unknown=&#39;ignore&#39; so that we avoid issues with unknow values for the columns when we use the encoder to transform on our inference dataset.</p><p>After that, we combine a fit and transform action into one step with the fit_transform method. And finally, we create a new data frame from the encoded data and concatenate it with the rest of the original dataset.</p><p>Now the task remains to use the encoder to transform our inference dataset.</p><pre># Transform inference data<br>inference_encoded = enc.transform(inference_data[trans_columns])<br><br>inference_feature_names = enc.get_feature_names_out(trans_columns)<br><br>inference_encoded_df = pd.DataFrame(inference_encoded.toarray(), <br>                                    columns=inference_feature_names)<br><br>final_inference_df = pd.concat([inference_data[[&#39;numerical_1&#39;]], <br>                                inference_encoded_df], axis=1)</pre><p>Unlike earlier, when we used the naive pandas.get_dummies ,we now see that our new final_inference_df dataset has the same columns as our training dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NjhVlhUw2KhF1aIe4mPoRQ.png" /><figcaption>Transformed Inference dataset with the correct columns / image by author</figcaption></figure><p>In addition to what we showed in the code above, the OneHotEncoder class from sklearn.preprocessing has a lot of other functionality that can be useful as well.</p><p>For example, it allows you set the min_frequency and max_categories options. As its name implies the min_frequency options allow you to specify the minimum frequency below which a category will be considered infrequent and then grouped together with other infrequent categories, or the max_categories option which limits the total number of categories. The latter can be especially useful if you don’t want to create too many columns in your training dataset.</p><p>For a full overview of the functionality, visit the documentation pages here:</p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder">sklearn.preprocessing.OneHotEncoder</a></p><h4>In R</h4><p>Several of my clients use R for running machine learning models in production — and it has a lot of great features. Before polars came out for Python, R’s data.table package was superior to what pandas could offer in terms of speed and efficiency. However, R doesn’t have access to the same type of production level packages as scikit-learn for python. (There are a few libraries, but they are not as mature as scikit-learn.) In addition, while some packages might have the required functionality, they require loads of other packages to run and can introduce dependency conflicts into your code. Consider running the line below in a docker container build with the r-base image:</p><pre>RUN R -e &quot;install.packages(&#39;recipes&#39;, dependencies=TRUE, repos=&#39;https://cran.rstudio.com/&#39;)&quot;</pre><p>It takes forever to install and takes up a lot of space on your container image. Our solution in this case — instead of using functions from a pre-built package like recipes — is to introduce our own simple function implemented using the data.table package:</p><pre>library(data.table)<br><br>OneHotEncoder &lt;- function() {<br>  # Local variables<br>  categories &lt;- list()<br>  <br>  # Method to fit data and extract categories<br>  fit &lt;- function(dt, columns) {<br>    for (column in columns) {<br>      categories[[column]] &lt;&lt;- unique(dt[[column]])<br>    }<br>  }<br>  <br>  # Method to turn columns into factors and <br>  factorize &lt;- function(dt) {<br>    for (column_name in names(categories)) {<br>        set(dt, j = column_name, <br>        value = factor(dt[[column_name]], <br>        levels = categories[[column_name]]))<br>    }<br>    return(dt)<br>  }<br>  <br>  # Method to transform columns in categories list to <br>  # dummy variables<br>  transform &lt;- function(dt) {<br>    dt = factorize(dt)<br>    # add row number for joins later<br>    dt[, rn := .I]<br>    for (col in names(categories)) {<br>      print(col)<br>      # Construct the formula dynamically<br>      formula_str &lt;- paste(&quot;~&quot;, col, &quot;- 1&quot;)<br>      formula_obj &lt;- as.formula(formula_str)<br>      # Create a model model.matrix object<br>      mm = model.matrix(formula_obj, dt)<br>      mm_dt &lt;- as.data.table(mm, keep.rownames = &quot;rn&quot;)<br>      mm_dt[, rn := as.integer(rn)]<br>      <br>      # Perform a merge based on these row numbers<br>      dt &lt;- merge(dt, mm_dt, by = &quot;rn&quot;, all = TRUE)<br>      <br>       # remove the original column<br>      dt[, (col) := NULL]<br><br>      # set any new NAs to 0<br>      for (ncol in names(mm_dt)) {<br>        set(dt, which(is.na(dt[[ncol]])), ncol, 0)<br>      }<br>    }<br>    dt[, rn := NULL]<br>    return(dt)<br>  }<br><br>  # Method to get categories<br>  get_categories &lt;- function() {<br>    return(categories)<br>  }<br>  <br>  # Return a list of methods<br>  list(<br>    get_categories = get_categories,<br>    fit = fit,<br>    transform = transform<br>  )<br>}</pre><p>Let’s go through this function and see how it works on our training and inference datasets. (R is slightly different from Python and instead of using a class, we use a parent function instead, which works in a similar way.)</p><p>First, we need to create an instance of the function:</p><pre> encoder = OneHotEncoder()</pre><p>Then, just like with the OneHotEncoder class from sklearn.preprocessing, we also have a fit function inside our OneHotEncoder. We use the fit function on the training data, supplying both the training dataset and the columns we want to one-hot encode.</p><pre># Columns to one-hot encode<br>fit_columns = c(&quot;color_1_&quot;, &quot;color_2&quot;)<br># Use the fit method<br>encoder$fit(dt=training_data, columns=fit_columns)</pre><p>The fit function simply loops through all the columns we want to use to for training and finds all the unique values each of the columns contain. This list of columns and their potential values is then used in the transform function. We now have a instance of a fitted one-hot encoder function and we can save it for later use using a R .RDS file.</p><pre>saveRDS(encoder, &quot;~/my_encoder.RDS&quot;)</pre><p>To generate the one-hot encoded dataset we need for training, we run the transform function on the training data:</p><pre>transformed_training_data = encoder$transform(training_data)</pre><p>The transform function is a little bit more complicated than the fit function, and the first thing it does is to convert the supplied columns into factors — using the original unique values of the columns as factor levels. Then, we loop through each of the predictor columns and create model.matrix objects of the data. These are then added back to the original dataset and the original factor column is removed. We also make sure to set any of the missing values to 0.</p><p>We now get the exact same dataset as before:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GFzZ9jT7Dmzlf7Q87uZbhw.png" /><figcaption>Transformed training dataset using R algorithm / image by author</figcaption></figure><p>And finally, when we need to one-hot encode our inference dataset, we then run the same instance of the encoder function on that dataset:</p><pre>transformed_inference_data = encoder$transform(inference_data)</pre><p>This process ensures we have the same columns in our transformed_inference_data as we do in our transformed_training_data.</p><h3>Further considerations</h3><p>Before we conclude there are a few extra considerations to mention. As with many other things in machine learning there isn’t always an easy answer as to when and how to use a specific technique. Even though it clearly mitigates some issues, new problems can also arise when doing one-hot encoding. Most commonly, these are related to how to deal with high cardinality categorical variables and how to deal with memory issues because of increasing the table size.</p><p>In addition, there are alternative coding techniques such as label encoding, embeddings, or target encodings which sometimes could be preferable to one-hot encoding.</p><p>Each of these topics is rich enough to warrant a dedicated article, so we will leave those for the interested reader to explore further.</p><h3>Conclusion</h3><p>We have shown how naive use of one-hot encoding techniques can lead to mistakes and problems with inference data, and we have also seen how to mitigate and resolve those issues using both Python and R. If left unresolved, poor management of one-hot encoding can potentially lead to crashes and problems with your inference, so it is strongly recommended to use more robust techniques—like either sklearn’s OneHotEncoder or the R function we developed.</p><p>Thanks for reading!</p><p><em>All the code presented and used in the article can be found in the following Github repo: </em><a href="https://github.com/hcekne/robust_one_hot_encoding"><em>https://github.com/hcekne/robust_one_hot_encoding</em></a></p><p><em>If you enjoyed reading this article and would like to access more content from me please feel free to connect with me on LinkedIn at </em><a href="https://www.linkedin.com/in/hans-christian-ekne-1760a259/"><em>https://www.linkedin.com/in/hans-christian-ekne-1760a259/</em></a><em> or visit my webpage at </em><a href="https://www.ekneconsulting.com/"><em>https://www.ekneconsulting.com/</em></a><em> to explore some of the services I offer. Don’t hesitate to reach out via email at hce@ekneconsulting.com</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=930b5f8943af" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/robust-one-hot-encoding-930b5f8943af">Robust One-Hot Encoding</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>