<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Simon Thornewill von Essen on Medium]]></title>
        <description><![CDATA[Stories by Simon Thornewill von Essen on Medium]]></description>
        <link>https://medium.com/@sthornewillve?source=rss-b0734c922eea------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_AuLdM2hDd9WfiLQbSkt5g.jpeg</url>
            <title>Stories by Simon Thornewill von Essen on Medium</title>
            <link>https://medium.com/@sthornewillve?source=rss-b0734c922eea------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:12:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sthornewillve/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Chaos moves fast, good data science moves slow]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sthornewillve.medium.com/chaos-moves-fast-good-data-science-moves-slow-3a2a653276d9?source=rss-b0734c922eea------2"><img src="https://cdn-images-1.medium.com/max/2600/0*ZOl047FpAsc38gU1" width="3278"></a></p><p class="medium-feed-snippet">Why slowing down actually accelerates impact</p><p class="medium-feed-link"><a href="https://sthornewillve.medium.com/chaos-moves-fast-good-data-science-moves-slow-3a2a653276d9?source=rss-b0734c922eea------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sthornewillve.medium.com/chaos-moves-fast-good-data-science-moves-slow-3a2a653276d9?source=rss-b0734c922eea------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a2a653276d9</guid>
            <category><![CDATA[risk]]></category>
            <category><![CDATA[stakeholder-management]]></category>
            <category><![CDATA[speed]]></category>
            <category><![CDATA[trust]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Simon Thornewill von Essen]]></dc:creator>
            <pubDate>Fri, 05 Dec 2025 10:32:05 GMT</pubDate>
            <atom:updated>2026-08-23T09:50:14.076Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[What you need to know to land your first job in data]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sthornewillve.medium.com/what-you-need-to-know-to-land-your-first-job-in-data-0a1af79cb903?source=rss-b0734c922eea------2"><img src="https://cdn-images-1.medium.com/max/2600/0*QcxEskOvigM7-74x" width="5760"></a></p><p class="medium-feed-snippet">A Map of the Data Career Landscape</p><p class="medium-feed-link"><a href="https://sthornewillve.medium.com/what-you-need-to-know-to-land-your-first-job-in-data-0a1af79cb903?source=rss-b0734c922eea------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sthornewillve.medium.com/what-you-need-to-know-to-land-your-first-job-in-data-0a1af79cb903?source=rss-b0734c922eea------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a1af79cb903</guid>
            <category><![CDATA[career-change]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[skills-development]]></category>
            <category><![CDATA[career-advice]]></category>
            <dc:creator><![CDATA[Simon Thornewill von Essen]]></dc:creator>
            <pubDate>Sat, 16 Aug 2025 13:43:45 GMT</pubDate>
            <atom:updated>2025-08-16T13:47:46.719Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How to stop your LLM from glazing you]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://ai.plainenglish.io/how-to-stop-your-llm-from-glazing-you-5ddc67209d3d?source=rss-b0734c922eea------2"><img src="https://cdn-images-1.medium.com/max/2600/0*kOi2OlyL9bwMeJmK" width="6016"></a></p><p class="medium-feed-snippet">Prompting for Discomfort</p><p class="medium-feed-link"><a href="https://ai.plainenglish.io/how-to-stop-your-llm-from-glazing-you-5ddc67209d3d?source=rss-b0734c922eea------2">Continue reading on Artificial Intelligence in Plain English »</a></p></div>]]></description>
            <link>https://ai.plainenglish.io/how-to-stop-your-llm-from-glazing-you-5ddc67209d3d?source=rss-b0734c922eea------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ddc67209d3d</guid>
            <category><![CDATA[problem-solving]]></category>
            <category><![CDATA[personal-growth]]></category>
            <category><![CDATA[reflections]]></category>
            <category><![CDATA[critical-thinking]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Simon Thornewill von Essen]]></dc:creator>
            <pubDate>Mon, 04 Aug 2025 06:33:09 GMT</pubDate>
            <atom:updated>2026-08-23T09:52:10.027Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How to Use AI Without Letting It Think for You]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://ai.plainenglish.io/how-to-use-ai-without-letting-it-think-for-you-4d6a9e90a663?source=rss-b0734c922eea------2"><img src="https://cdn-images-1.medium.com/max/2600/0*rw3w73vR1-3pq8y1" width="6000"></a></p><p class="medium-feed-snippet">Keeping Your Edge in the Age of LLMs</p><p class="medium-feed-link"><a href="https://ai.plainenglish.io/how-to-use-ai-without-letting-it-think-for-you-4d6a9e90a663?source=rss-b0734c922eea------2">Continue reading on Artificial Intelligence in Plain English »</a></p></div>]]></description>
            <link>https://ai.plainenglish.io/how-to-use-ai-without-letting-it-think-for-you-4d6a9e90a663?source=rss-b0734c922eea------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d6a9e90a663</guid>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[cursor]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[gen-ai-tools]]></category>
            <dc:creator><![CDATA[Simon Thornewill von Essen]]></dc:creator>
            <pubDate>Sun, 13 Jul 2025 18:25:21 GMT</pubDate>
            <atom:updated>2025-07-27T20:22:45.581Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How I Modelled My Cat’s Chaos using Linear Models in PyMC]]></title>
            <link>https://sthornewillve.medium.com/how-i-modelled-my-cats-chaos-using-linear-models-in-pymc-aa8de3c24c4a?source=rss-b0734c922eea------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa8de3c24c4a</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Simon Thornewill von Essen]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 10:34:56 GMT</pubDate>
            <atom:updated>2026-08-23T09:54:05.411Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Basic Linear Models in PyMC</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wo-GzsoHNCr68AKE" /><figcaption>Photo by <a href="https://unsplash.com/@pawel_czerwinski?utm_source=medium&amp;utm_medium=referral">Pawel Czerwinski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In my previous post, I wrote a tutorial about <a href="https://medium.com/@sthornewillve/babys-first-bayesian-model-using-pymc-in-python-13af85cf4239">fitting distrib</a>utions to data using Bayesian statistics with PyMC. It’s a good place to start understanding the basic statistical workflow while using Bayesian statistics, but we usually want to fit fully kitted-out models to our data.</p><p>This post takes that next step. We’ll make the outcomes we will be modelling conditional on specific situations so that we can move up to linear models.</p><p>While this step seems small, a lot can be done using them if you know what you are doing, so this blog post will be an enormous leap for your Bayesian skills.</p><p>This post assumes that you are familiar with the concepts of my last post, which includes the following:</p><ol><li>An understanding of Bayes’s Rule (the role of priors and the likelihood)</li><li>Understanding the basic workflow of building Bayesian models in PyMC, doing prior predictive checks and posterior predictive checks</li><li>Bayesian model evaluation using trace plots, effective sample size and Gelman-Rubin diagnostic</li><li>Importing the required packages (PyMC, Arviz and the usual data science stack along with knowledge of their use)</li></ol><p>If you need to reinforce your understanding, please see the previous post.</p><p><a href="https://sthornewillve.medium.com/babys-first-bayesian-model-using-pymc-in-python-13af85cf4239">Baby’s First Bayesian Model Using PyMC in Python</a></p><h3>My cat’s antics; a progression of our last problem</h3><p>Let’s continue with the cat knocking over a mug from the previous tutorial. This time, we will make the outcome a function of some independent variables.</p><p>For instance, my cat can’t knock over my mug if I don’t brew any coffee, and I’m not very regular with my brewing habits because I drink decaf, which allows me to be more random.</p><pre># Simulate data<br>brewed_coffee = stats.poisson(mu=4).rvs(14)<br>cat_knocks = stats.binom(n=brewed_coffee, p=0.75).rvs()<br><br># Assemble into data frame<br>df = pd.DataFrame(dict(brewed_coffee=brewed_coffee,<br>                       cat_knocks=cat_knocks))</pre><p>Instead of me brewing a new cup every time my cat knocks it over, let’s say that I brew 4 cups on average per day and that my cat will knock 75% of them over. We can visualise this below;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sQHpDnpbzKL1aYBgwCIJEw.png" /><figcaption>We can see a trend in the number of times my cat knocks my mug over by how many cups I brew.</figcaption></figure><p>We can see that the number of times my cat knocks my mug over depends linearly on the number of cups brewed. We can understand how often my cat does so using a linear model.</p><h3>Modelling Process</h3><h4>Building our initial model</h4><p>The difference to the previous example is that now we will have a likelihood function that is a function of the inputs to our model rather than being comprised entirely of parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*E-rqUue7lDj6E2JwZvaZPQ.png" /><figcaption>A description of our model</figcaption></figure><p>The number of times our cat knocks the mug over (K) is normally distributed around some mean (μ) and a standard deviation (σ), similar to our previous model.</p><p>What changes is that the mean is now a deterministic combination of the parameters and our data instead of <em>just</em> a parameter. If we know the parameters with certainty, then we know the mean.</p><p>The mean can be broken down into an intercept/baseline (b0) and the rate at which the outcome increases given the number of coffees (b1). Note that each of these unknown parameters gets its distribution. Meanwhile, our data are known as C.</p><p>Note that here I know that if we brew no coffee, our cat can’t knock it over. Therefore, I set the priors for b0 to be tight around 0. I expect the rate to be between 0 and 1 because our cat can’t knock over the mug more often than I brew coffee, but let’s keep it simple and say that it’s a number that we’re reasonably confident is close to 0.5.</p><p>Once again, our only expectation is that sigma is greater than 0. I used an exponential distribution instead of a half-normal, which puts more prior probability closer to 0 and allows a long tail of events.</p><p>Why is it still okay to use a <em>Normal</em> likelihood? It’s because we expect the distribution of the number of cups my cat knocks my mug over, conditional on the number of coffees I brew, to be explained using a <em>Normal</em> distribution. It’s also possible to use a general linear model (GLM) to explain it using a <em>Poisson</em> distribution instead, but we’re keeping it simple.</p><p>We can instantiate this model in PyMC using the following code;</p><pre>with pm.Model() as cat_model:<br>    σ = pm.Exponential(&quot;σ&quot;, lam=1)<br><br>    β0 = pm.Normal(&quot;β0&quot;, mu=0, sigma=0.5)<br>    β1 = pm.Normal(&quot;β1&quot;, mu=0.5, sigma=0.5)<br>    <br>    μ = β0 + β1 * df[&quot;brewed_coffee&quot;]<br><br>    likelihood = pm.Normal(&quot;likelihood&quot;, mu=μ, sigma=σ, observed=df[&quot;cat_knocks&quot;])</pre><p>This is already quite powerful; you can now do the usual traditional linear model tricks from here, such as using indicators, interactions, etc. All that experience is now available for use in a Bayesian framework.</p><h4>An alternative model with different priors</h4><p>Let’s also have a little fun. We know that b1 must be between 0 and 1, so let’s change the <em>Normal</em> prior to a <em>Beta</em> prior.</p><p>The <em>Beta</em> distribution spans between 0 and 1 and is a conjugate prior that you hear about if you learn Bayesian statistics the traditional way. That is, it’s a special distribution whose parameters can be made functions of the data so that we can correctly calculate our posterior distribution from our data.</p><p>For example, a beta(alpha=1, beta=1) distribution is uniform between 0 and 1, where each parameter can be considered as the number of successes and failed trials, respectively. For instance, if you flip a coin and find that it’s heads, then you increment alpha by one and if it’s tails, you increment beta by 1.</p><pre>x = np.linspace(0, 1, 100)<br><br>plt.plot(x, stats.beta(1, 1).pdf(x=x), color=sns.color_palette()[0], alpha=1/5, label=&quot;Flips = 1&quot;)<br>plt.plot(x, stats.beta(2, 1).pdf(x=x), color=sns.color_palette()[0], alpha=1/4, label=&quot;Flips = 2&quot;)<br>plt.plot(x, stats.beta(2, 2).pdf(x=x), color=sns.color_palette()[0], alpha=1/3, label=&quot;Flips = 3&quot;)<br>plt.plot(x, stats.beta(3, 2).pdf(x=x), color=sns.color_palette()[0], alpha=1/2, label=&quot;Flips = 4&quot;)<br>plt.plot(x, stats.beta(3, 3).pdf(x=x), color=sns.color_palette()[0], alpha=1/1, label=&quot;Flips = 5&quot;)<br>plt.legend()<br>plt.xlabel(&quot;Probability&quot;)<br>plt.ylabel(&quot;Probability Density&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-JXcnFq6mpfzfMPm7CGAgQ.png" /><figcaption>Updating of a beta distribution based on the coin flips HTHTH, the distribution becomes more confident that the probability of heads (H) is 50% as more data is gathered.</figcaption></figure><p>Above, we can see how I increment each parameter by 1, as if we had the data HTHTH, leading to increased confidence of being close to 0.5 with each flip.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*i2yul3iBPcz3SvDH4nqdEQ.png" /><figcaption>Description for an alternative model using a beta distribution instead, constraining b1 to be between 0 and 1 with uniform prior probability.</figcaption></figure><p>The only difference is with b1, which I’ve now constrained to have uniform probability between 0 and 1, which we can instantiate as follows:</p><pre>with pm.Model() as cat_model2:<br>    σ = pm.Exponential(&quot;σ&quot;, lam=1)<br><br>    β0 = pm.Normal(&quot;β0&quot;, mu=0, sigma=0.5)<br>    β1 = pm.Beta(&quot;β1&quot;, alpha=1, beta=1)  # Changed from `pm.Normal`<br>    <br>    μ = β0 + β1 * df[&quot;brewed_coffee&quot;]<br><br>    likelihood = pm.Normal(&quot;likelihood&quot;, mu=μ, sigma=σ, observed=df[&quot;cat_knocks&quot;])</pre><h4>Prior Predictive Check</h4><p>Doing prior predictive checks works the same as before.</p><pre>with cat_model:<br>        prior_pred = pm.sample_prior_predictive(samples=4000)<br><br>with cat_model2:<br>        prior_pred2 = pm.sample_prior_predictive(samples=4000)<br><br>az.plot_trace(prior_pred)<br>plt.tight_layout()<br>plt.show()<br><br>az.plot_trace(prior_pred2)<br>plt.tight_layout()<br>plt.show()</pre><p>We can then plot the distributions using az.plot_trace().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MLB_UmRFz5_gqD623lKdyQ.png" /><figcaption>Prior distributions for model 1. Naturally, all of the distributions are as you would expect.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bom2bQjy6s8kjd7lVK7UAw.png" /><figcaption>Prior distribution for b1 for the second model. The only difference is with b1, which is now uniformly distributed.</figcaption></figure><p>We can now write a function to map our priors to our simulated prior-predicted data. Writing a function is useful because it lets you quickly make many plots without repeating code. This is also useful to ensure the quality of your inferences. In a real-world setting, you can expect to make edits to your code after the fact as you discuss with others; having this first level of modularity is important.</p><pre>def plot_prior(trace, n_sims=50, alpha=1/5):<br><br>    C = np.linspace(0, 10, 10)<br>    K = trace.β0.values[0, :n_sims] + np.outer(C, trace.β1.values[0, :n_sims])<br><br>    for i in range(n_sims):<br>        plt.plot(C, K[:, i], alpha=alpha, color=sns.color_palette()[0])<br>    plt.xlabel(&quot;Number of Coffees Brewed&quot;)<br>    plt.ylabel(&quot;Number of times Cat Knocked Mug Over&quot;)<br>    plt.show()<br><br>    return<br><br>plot_prior(prior_pred.prior)<br>plot_prior(prior_pred2.prior)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A9jqO5D8PYftxSLo6lj1xQ.png" /><figcaption>Prior predicted data for the first model. The estimates are mostly reasonable, except that they allow for negative rates of my cat knocking over my mug, which can lead to impossible outcomes.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJA4tvU0m3e1Y43YPIfR-A.png" /><figcaption>Prior predicted data for the second model. We get more realistic prior predictions because we constrained b1 to never be negative.</figcaption></figure><p>Above, we can see the prior predicted distribution for our models, cat_model and cat_model2, respectively. We can see that having a <em>Beta</em> distribution for b1 removes the impossible negative slope, resulting in my cat knocking over my mug negative times.</p><p>We can also see that the intercept is tightly clustered around 0. However, this is a mistake because we know from the logic of our problem that it has to be 0.</p><p>It’s important to get this right because once we fit our model to the data, we should not change it to prevent overfitting to our data. Taking the time to make sure that the prior predicted data fits our expectations before moving forward, given our understanding of the problem, is essential.</p><h4>A note on building custom models for our problems</h4><p>Note how even with relatively basic knowledge, we’re already building models created to fit our specific use case. This can lead to more precise and accurate estimates. It isn’t easy to do this using traditional methods because fitting the data would require different optimisation processes from usual, requiring a statistician to work out.</p><p>Bayesian models are not like this, because sampling from our posterior distributions means we can calculate all of the interesting properties that come from a model without analytically evaluating the best estimators for each model.</p><p>This is <em>not </em>a minor point. The model-building techniques for inference that most people use are not flexible in this way, and it can lead to very biased estimates because the data-generating process violates the assumptions of the models used to fit them.</p><p>There’s nothing necessarily wrong with biased estimates so long as your data leads to good decisions or deeper understanding. This only becomes a problem when there is a lack of proper acknowledgement or understanding, which is common even in scientific literature.</p><h4>Fitting our Models</h4><p>Let’s fit our models now. As before, what is happening in this step <em>still</em> isn’t that important. For most linear models, you should find that the sampling algorithm should easily explore the posterior distribution. It’s when you take that next step to GLMs and hierarchical models that you need to understand what is going on.</p><pre>with cat_model:<br>    trace = pm.sample()<br><br>with cat_model2:<br>    trace2 = pm.sample()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VlbK30lRrqkiDWySP8EUqA.png" /></figure><p>Density plots and trace plots for the parameters of our first model. We can see the fuzzy caterpillar in our trace plots, which suggests that the model fits well. We can also see one divergent transition, which we’ll ignore for this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*solf25zLXdv_fm10xMU4-g.png" /></figure><p>We can see that the models are giving that “fuzzy caterpillar” shape, which explores the distribution such that each draw looks independent from the same stationary distribution.</p><p>When I showed the prior parameter distribution for cat_model2 I didn’t show the values for b1 and sigma, but I’m showing them now. That’s because the posterior parameters are jointly credible with the other parameters, whereas the prior distributions are all independent. This means we shouldn’t expect the posterior beliefs for b1 and sigma to be the same as for cat_model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*fooqwRM7dbvy-74UeAwt1Q.png" /></figure><p>Furthermore, we can see that the models give similar posterior estimates of the parameters. However, the second model has a worse effective sample size, suggesting we should use the first for posterior predictions.</p><p>We can also use a forest plot to compare the parameters side by side.</p><pre>az.plot_forest([trace, trace2], hdi_prob=0.89, combined=True, figsize=(7, 4))<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqAtjsehLZ2yoRHu1_Wj0A.png" /></figure><p>Using a forest plot is a great way to compare the parameters side by side, but we lose the information about the trace plot.</p><h4>Posterior Predictive Check</h4><p>It’s time to see what the models predict. I update plot_prior to plot_posterior by allowing it to plot both the mean and the simulated data.</p><pre>def plot_posterior(trace, n_sims=50, alpha=1/5):<br><br>    C = np.linspace(0, 10, 10)<br>    K = trace.β0.values.reshape(-1) + np.outer(C, trace.β1.values)<br>    Ksim = K + np.random.normal(loc=0, scale=trace.σ.values.reshape(-1), size=K.shape)<br><br>    Kq = np.quantile(K, [(1-0.89)/2, 1 - ((1-0.89)/2)], axis=1)<br>    Ksimq = np.quantile(Ksim, [(1-0.89)/2, 1 - ((1-0.89)/2)], axis=1)<br>    <br>    plt.fill_between(C, Kq[0, :], Kq[1, :], alpha=0.6, color=sns.color_palette()[0], label=&quot;mean (89% CI)&quot;)<br>    plt.fill_between(C, Ksimq[0, :], Ksimq[1, :], alpha=0.2, color=sns.color_palette()[0], label=&quot;simulated (89% CI)&quot;)<br><br>    plt.axhline(0, color=&quot;black&quot;)<br>    <br>    sns.scatterplot(x=&quot;brewed_coffee&quot;, y=&quot;cat_knocks&quot;, data=df, color=sns.color_palette()[1])<br>    <br>    return</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cnvREffWdhduEYx8CrK7CQ.png" /></figure><p>The plot above shows the observed data alongside the model’s posterior distributions for both the mean and the simulated data. The mean represents our best understanding of the centre of mass (area of least “wrongness”) of the data, whereas the simulated data are where we expect the actual data points to land. Our 89% credible interval for the simulated data captures our observed data very well, which indicates (but does not guarantee) that it’s a good description of the data-generating process.</p><p>As we move away from our data, our estimates become less certain. This is important because if we make predictions from parts of our model with little data, our estimates will reflect this. It is so vital that it’s always best practice to<em> </em>quantify the uncertainty in our data. Always looking at point estimates without quantifying uncertainty, which is common, can easily lead you to over-analysing noise by underestimating how unstable your estimate is getting.</p><p>Fortunately, this sample of 14 data points gives us reasonable estimates, but this may not always be the case, which can cause the estimates for the intercept and slope to be pulled slightly away from their actual values. (If I simulate with 4000 days, the posterior converges to the expected parameters.)</p><p>A well-specified prior and model can still provide functional inference with limited data. Still, accuracy improves significantly as more evidence accumulates and, crucially, when models better fit the data-generating process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fvKvID8qp4W_DdWUU9L3CQ.png" /></figure><p>The two models perform similarly in the posterior predictive check. Typically, we would score the models to choose between them, especially if plotting them isn’t as straightforward as in this case.</p><p>Although we know that the priors for cat_model2 are more realistic, we’ll use the first model (cat_model) for instance. Still, in either case, we can get reasonable posterior estimates even if the model isn&#39;t entirely correct. This is important because we often don&#39;t know the real model in the real world, but by using simulated data, we can be confident that our models should work, at least in theory, and often, that&#39;s good enough.</p><h4>Using Parameters for Inference</h4><p>We can show our posterior belief for how frequently our cat knocks over our mug by looking at the distribution for b1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0T4UKVXpNAnN_9ty-GrkBA.png" /></figure><p>In the plot above, the posterior distribution for the total probability that my cat knocks over a mug using cat_model (the sum of b0 and b1) occasionally exceeds 1, which isn’t possible for a probability.</p><p>This is a reflection of how modelling works. We did not place a constraint in the model that the output must be a valid probability. Meanwhile, the model was uncertain, especially with limited data, so it explored all regions consistent with the prior and likelihood, even if some combinations produced values above 1.</p><p>It’s not meaningful to ask if the model is “wrong”, all models are abstractions of reality, it just doesn’t know enough to rule out those values, and so this is the best estimate it can come up with.</p><p>We <strong>shouldn’t</strong> go back and “fix” the model just because we don’t like the results. Retroactively narrowing priors or hard-coding constraints based on posterior behaviour leads to overfitting and can make us overly confident in our estimates. A better approach is acknowledging the misspecification and suggesting a more realistic model for future use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T9mEcMxphf3s0xHI7NBZAw.png" /></figure><p>Meanwhile, cat_model2 does a better job estimating b1 even if we saw that the effective sample size is lower. Here, we’re trading off accuracy (distance from the real value), precision (spread of the data) and the model reliability (stability of our model estimates). We know that the effective sample size of cat_model2 is lower, but it will at least give us estimates that make sense.</p><p>I’ll choose it for our estimate and conclude that our cat knocks over our mug with a probability of 76% with a 89% credible interval between 59% and 93%. We can say with a 99% probability that our cat knocks over our mug more than 50% of the time, but still doesn’t <em>always</em> do it.</p><h3>Conclusion</h3><p>In this article, we’ve taken a big step in our modelling abilities by being able to fit linear models to data. Depending on how much you know about linear models, this opens up the Bayesian inference that you can now perform immensely without too much study.</p><p>We applied our understanding of prior/posterior predictive modelling and allowed our predictions to be changed based on independent variables. We also saw how we can make valid inferences even if our model doesn’t perfectly capture our data-generating process.</p><p>We also discussed modelling procedure to prevent overfitting and ethics surrounding communicating our results and being critical of our models, so we use them rather than necessarily believing any single one as we compare the models and choose between them.</p><p>Thanks a lot for making it to the end of this article, and I wish you the best of luck using the knowledge you now have!</p><p>To see the full code, please see the notebook <a href="https://github.com/SThornewillvE/medium_blogposts/blob/main/introductory_bayesian_posts/02_Linear-Models_Pymc_Bambi.ipynb">here</a>.</p><p>If you enjoyed this post, please remember to clap and follow me here or on <a href="https://bsky.app/profile/sthornewillve.bsky.social">BlueSky</a>. Thanks a lot for your support in advance!</p><p>If you want to learn more about why Bayesian statistics are useful in a business context, see my previous post here:</p><p><a href="https://sthornewillve.medium.com/why-bayesian-methods-very-useful-for-business-decision-making-c533573de514">The Hidden Power of Bayesian Methods in Business Decisions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa8de3c24c4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Baby’s First Bayesian Model Using PyMC in Python]]></title>
            <link>https://sthornewillve.medium.com/babys-first-bayesian-model-using-pymc-in-python-13af85cf4239?source=rss-b0734c922eea------2</link>
            <guid isPermaLink="false">https://medium.com/p/13af85cf4239</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[visualisation]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Simon Thornewill von Essen]]></dc:creator>
            <pubDate>Fri, 18 Apr 2025 18:44:19 GMT</pubDate>
            <atom:updated>2025-04-25T09:52:41.694Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*72eWKGiD9fzjybde" /><figcaption>Photo by <a href="https://unsplash.com/@shs521?utm_source=medium&amp;utm_medium=referral">Susan Holt Simpson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A couple of weeks ago (at the time of writing), I wrote a blog post about the benefits of Bayesian statistics in a business context. (<a href="https://medium.com/@sthornewillve/why-bayesian-methods-very-useful-for-business-decision-making-c533573de514">Link</a>.) It was quite a popular post, because there’s a lot of hype surrounding Bayesian statistics. However, the issue is that while most of us agree that Bayesian statistics are exciting, there is often a gulf between that excitement and applying it in practice.</p><p>In my opinion, the main reason for this is the bottom-up approach that we use to learn most things. My journey looked like this:</p><ol><li>Watched all of the YouTube lectures associated with Harvard’s Stats110.</li><li>Took an entire course on Markov chains.</li><li>Took a course on basic Bayesian statistics surrounding conjugate priors.</li><li>Read most of the way through Statistical Rethinking.</li></ol><p>I definitely learned a lot along the way. Unfortunately though, I also forgot a lot of things along the way.</p><p>Furthermore, even though I’ve given multiple talks on Bayesian statistics I have been no better, because I never gave people the tools to be dangerous right away.</p><p>This article aims to fix this. Instead of learning everything from the bottom up, I will give you enough to start fitting the simplest model you can <em>today.</em></p><h4>In this post: Fitting a normal distribution to data</h4><p>What’s the simplest operation we perform on data? People always take this first step when data is presented to them. That’s right, it’s calculating the mean/average of some data given to you.</p><p>When we calculate the mean, we normally calculate the centre of mass of the data points we are given. However, to do this in a Bayesian way, we need to create a prior distribution of what we expect our data to look like.</p><p>Generally, we will make these following steps:</p><ol><li>Construct a model with priors.</li><li>Perform a prior predictive check.</li><li>Fit the model and run diagnostics.</li><li>Perform a posterior predictive check.</li></ol><p>One can add other steps, but this will be our workflow for now, while we are keeping things as simple as possible.</p><p>To help us, we will be using a probabilistic programming language. A probabilistic programming language lets you write models with uncertain parameters, which it then automatically updates using the data. Examples of probabilistic programming languages include <a href="https://www.pymc.io/welcome.html">PyMC</a>, <a href="https://mc-stan.org/">Stan</a> and <a href="https://pyro.ai/">Pyro</a>.</p><p>We’ll be doing this using PyMC, since most people have python installed and it’s a great place to start before working up to more heavy duty probabilsitic programming languages like Stan, which expert statisticians use.</p><h3>Understanding Bayes’s Theorem</h3><p>In order to create a Bayesian model, we do need to talk about Bayes’s rule a little bit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*sHaagwc_qcK1CwTirzLqYg.png" /><figcaption>Bayes’s rule, The posterior is equal to the likelihood of the data multiplied by our prior divided by the evidence.</figcaption></figure><p>Bayes’s rule allows us to update our prior using data through a likelihood.</p><p>The issue is that the “evidence” (the marginal probability of B) is difficult to calculate for non-trivial models. This is why we sample from it instead of calculating our posterior distribution directly (the probability of A conditional on B). Don’t worry about the details about how that happens for now, for basic models, it’s not important.</p><p>What’s important for us right now is that to obtain our posterior, we need a model that defines our likelihoods and priors, which is what this article is going to help you to do.</p><h3>Practicing the Workflow</h3><h4>Set Up</h4><p>In order to run the python code for this example, I’ll use the following packages;</p><pre># Import Packages<br>import arviz as az<br>import matplotlib.pyplot as plt<br>import numpy as np<br>import pandas as pd<br>import pymc as pm<br>import seaborn as sns<br><br>from scipy import stats<br><br># Set Seed<br>np.random.seed(42)</pre><p>The Matplotlib, Numpy, Pandas, Scipy and Seaborn packages are standard for Data Scientists who work with Python, so I won’t explain them here.</p><p>The new packages to note are Arviz and PyMC. PyMC is the probabilistic programming language I mentioned above, while Arviz gives us some common tools to help plot and assess samples that we get from PyMC.</p><h4>Simulate the Data</h4><p>Let’s say we have a cat who knocks over my favourite mug of coffee multiple times a day. I always diligently clean it up and brew myself another cup immediately, and over 14 days, I count how often my cat comes and knocks over my mug.</p><p>I can simulate this data using a <em>Poisson</em> distribution. Let’s say that my cat knocks over the cup 4 times on average per day. This means that we expect the count of how often my cat knocks over my mug to be a positive integer. I can then find a count over 14 days like so:</p><pre>cat_knocks = stats.poisson(mu=4).rvs(14)</pre><p>We can then visualise this data using a histogram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GA1OMeuwAHocRrgdStijOg.png" /><figcaption>Histogram of the simulated times that my cat knocked over my mug of coffee over 14 days. We can see that sometimes our cat is unhappy with the mug and knocks it over a lot, and sometimes it does it less.</figcaption></figure><h4>Building the Model</h4><p>Of course, we can build a model using a <em>Poisson</em> likelihood. However, we want this to be “baby’s first model”, so we’ll assume that the data are normally distributed because that’s easier to think about.</p><p>Note that your likelihood is a part of your model; There is no objectively right answer as to which one you choose. The goal is to get something that describes the data somewhat well. While a <em>Nor</em>mal likelihood is easier to think about, it does come with the drawback that it allows for a negative number of times that my cat knocked over my mug.</p><p>There are various ways that you can fix this. For instance, you can choose specific priors or a truncated likelihood, making this outcome impossible. But we’ll keep it open for now since this problem with the model won’t be a problem for us.</p><pre>with pm.Model() as cat_model:<br>    σ = pm.HalfNormal(&quot;σ&quot;, sigma=1)<br>    μ = pm.Normal(&quot;μ&quot;, mu=7, sigma=1)<br><br>    likelihood = pm.Normal(&quot;likelihood&quot;, mu=μ, sigma=σ, observed=cat_knocks)</pre><p>To build a model, we need to create it using Python’s with statement. This creates a PyMC model called cat_model, and then we define the structure of the model within it.</p><p>As mentioned, our likelihood is normally distributed with priors for that model’s mean and standard deviation. We’ll also pass our data to the model using the observed parameter.</p><p>Let’s say that we are very frustrated with our cat, and so we expect that most of the data will be between 5 and 9, without seeing the data, so the mean of our normal distribution will have an average of 7 with a standard deviation of 1.</p><p>You might ask why we can’t just look at the data and use that to create our priors. The issue is that your priors represent your best knowledge <em>without</em> looking at the data you intend to collect. You can use any data that you have available to you before collecting the data, but if you use the data you plan to analyse to create your priors, then they aren’t priors and you’ll make your posterior overly confident. (i.e. You’ll risk <a href="https://en.wikipedia.org/wiki/Overfitting">overfitting</a> your model.)</p><p>Meanwhile, we don’t have any specific opinion about how big our standard deviation should be, other than it should be positive. I have used a half-normal with a standard deviation of 1.</p><h4>Prior Predictive Checks</h4><p>Now that we have defined our model, we can see what kind of data it generates and see if it’s reasonable given what we know about our cat and how often he knocks glasses over.</p><pre>with cat_model:<br>        prior_pred = pm.sample_prior_predictive(samples=1000)</pre><p>Once again, we call the model using with and we sample a prior predictive simulation.</p><pre># Prepare the data<br>samples = prior_pred.prior_predictive.likelihood.squeeze()<br><br>df = pd.DataFrame(samples, columns=[f&quot;Day {i+1}&quot; for i in range(samples.shape[1])])<br>df_long = df.melt(var_name=&quot;Day&quot;, value_name=&quot;Number of Knocks&quot;)<br><br>plt.figure(figsize=(10, 8))<br>sns.violinplot(data=df_long, x=&quot;Number of Knocks&quot;, y=&quot;Day&quot;, density_norm=&quot;width&quot;, inner=&quot;quartile&quot;)<br>plt.title(&quot;Prior Predictive Distributions per Day&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gfi0RVncanWKB4JYsFYY-A.png" /><figcaption>Prior predictive distribution of knocks for each observation (day).</figcaption></figure><p>We can see that, as I mentioned, it’s possible to have a negative number of knocks, but we’ll ignore that since we accept that this will be one of the ways our model is wrong, despite still being useful.</p><p>The important thing is that it covers most of the outcomes between 0 and 15, which is the number of knocks that we may expect to see. It’s not inappropriately high by having 100s of knocks.</p><p>Note how we use our domain knowledge to say what is remotely feasible. It’s still possible to have data that changes our understanding of the mean to get values higher than 100, and it would just take a lot of data to convince us of that. In this way, Bayesian statistics protects us from making inferences about things that would be difficult to believe.</p><h4>Fitting and Assessing the Model</h4><p>Now that our model is instantiated, we can fit it to the data.</p><pre>with cat_model:<br>    trace = pm.sample()</pre><p>A lot has happened under the hood with just these two lines of code. However, I wouldn’t worry about it until you start writing more complicated models.</p><p>You should rarely need to adjust what’s going on under the hood, and if you are having difficulty sampling from the posterior distribution, then it’s usually because your model needs work, not the sampling process.</p><p>In the words of <em>Andrew Gelman:</em></p><blockquote>If your model is having problems then that’s usually a problem with your model.</blockquote><p>The advice is usually to start with simple models before working up to more complex ones. Fitting complex models is a skill, which is one of the big reasons why Bayesian statistics struggles with popularity.</p><pre>az.plot_trace(trace)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ww8SM-GdIDaCSeSQYtMoUw.png" /><figcaption>KDE Plots (left) and Trace plots (right) of the posterior samples of mu and sigma. The model seems to have fit well.</figcaption></figure><p>Note that we have multiple lines for each distribution here. This is because the model is sampling the parameters using four independent chains. If you can come to the same distribution despite running the algorithm several times, then your inference is more likely to be robust.</p><p>Furthermore, we can see the samples in the trace plot to the right. We can see that the distribution is being sampled from well, it’s not wandering the space and each draw looks sufficiently random from the same distribution such that we get a “fuzzy caterpillar”.</p><p>We should also look at the effective sample size (ESS) and the <a href="https://en.wikipedia.org/wiki/Gelman-Rubin_statistic">Gelman-Rubin diagnostic</a> (R-hat) for our model.</p><pre>az.summary(trace, round_to=2)[[&quot;mean&quot;, &quot;ess_bulk&quot;, &quot;r_hat&quot;]]</pre><p>This command reveals that the effective sample size is perhaps a little low at ~3000 out of 4000 actual samples, which means that the sampling wasn’t as efficient as we may hope.</p><p>Since we’re not as interested in the tails of our distributions, this is okay. However, if your ESS is low, then that means that the estimates for part of your distribution where the posterior probability is thin can be unstable.</p><p>We can also see that R-hat isn’t above 1. If all chains are sampling from the same underlying posterior, their estimates should be similar, and R-hat will be close to 1.00.</p><p>If R-hat is greater than 1, the chains are still exploring different regions of the parameter space — a sign that the sampler hasn’t fully converged yet.</p><p>Given that this is a simple model, we don’t expect anything to go wrong. But these are some ways to see if your model is having trouble. Note that your model can still fit well and not necessarily be specified well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s6jxuPwdmm9pWNyi_GgUew.png" /><figcaption>Posterior (orange) and prior (blue) distributions for the mean of our distribution. We can see that we initially overestimated how much our cat knocks over our mug.</figcaption></figure><p>We can see from this that the number of times my cat knocks over my mug is much lower than I expected, and so given my prior, the model and the data, it should be much lower.</p><pre>(trace.posterior.μ.values.reshape(-1)[:1000] &lt; prior_pred.prior.μ.values.squeeze()).mean()</pre><p>With this single line of code, we can quantify the probability that our posterior is lower than our prior given the data, of which the probability is 99.5%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LfNKLdXut7TZrQFeyqDDGg.png" /></figure><p>Posterior (orange) and prior (blue) distributions for the standard deviation of our distribution. We can see that there is more noise in the data than we expected.</p><p>Meanwhile, there was more variability in the data than we expected. It could be that since the data are <em>Poisson</em> distributed, our model struggled to fit the data, which is why it struggled with the ESS. The only way to test this hypothesis would be to build a more complex model and see how that changes the summary statistics.</p><h4>Posterior Predictive Checks and Statistics</h4><p>Now that we have fitted our model, let’s take a look at what the posterior predictive distribution looks like.</p><pre>with cat_model:<br>    post_pred = pm.sample_posterior_predictive(trace, random_seed=42)<br><br># Add a small amount of random jitter to your observed data<br>jittered_knocks = cat_knocks + np.random.normal(0, 0.2, size=len(cat_knocks))<br><br>az.plot_posterior(post_pred.posterior_predictive.likelihood[0, :, 0],<br>                  hdi_prob=0.89)<br>sns.rugplot(jittered_knocks, label=&quot;Data (jittered)&quot;, alpha=0.6, color=sns.color_palette()[1])<br><br>plt.legend()<br>plt.xlabel(&quot;Knocks&quot;)<br>plt.title(&quot;Posterior Predictive vs Observed Data&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NfyFH3dJMeHyzNczsomp2g.png" /><figcaption>Posterior predictive (simulated) distribution against actual data (orange). The model is a little uncertain but we also only have 14 data points.</figcaption></figure><p>It seems that the model fits the data somewhat well, but we can also see that the model is still pretty uncertain. Then again, we only have two weeks of data.</p><p>It seems that 89% of simulated knocks occur between 1.3 and 6.7, which means that our initial impression that the mean of the distribution was 7 was just too high. In reality, the model estimates that it’s closer to 4, which is what we programmed the simulated data to have!</p><p>We can also ask questions, such as “what’s the probability that the average times my cat knocks over my mug is less than 7, and I find that it’s 99.98%.<br>Meanwhile, the probability of seeing the mug being knocked over less than 7 times is ~94%.</p><p>To be fair, these questions aren’t that interesting because the model is so simple. In a real life setting we would want to ask “so what”? We would be thinking about what decisions or insight/understanding this model would drive. Perhaps we can say that maybe we shouldn’t be too hard on our cat, but we still need to train him so that we don’t need a model to estimate how much of a pain he’s being.</p><h3>Conclusion</h3><p>That’s all there is to it. In this article, we simulated some data and fit a Bayesian model, assuming the data were normally distributed. We used prior predictive checks to ensure that our priors created “reasonable enough” data given our expectations. After fitting our data and assessing how well it sampled from the posterior distribution, we found that our initial impressions were too harsh and that our cat knocked over our mug close to 4 times a day instead of 7.</p><p>The way you build your models is important. We made many choices along the way, such as what likelihood function and priors we used to fit our model, and depending on how you do that, you can get different answers. The point is to justify your decisions at each step so that if someone disagrees, you know how to change your model and see what kinds of results it makes and what implications it might have.</p><p>That’s all for today. Next time, we’ll take a look at linear models. Keep your eyes peeled for the next post!</p><p>To see the full code, please see the notebook <a href="https://github.com/SThornewillvE/medium_blogposts/blob/main/introductory_bayesian_posts/01_Babys-First-Model.ipynb">here</a>.</p><p>If you enjoyed this post, please remember to clap and follow me here or on <a href="https://bsky.app/profile/sthornewillve.bsky.social">BlueSky</a>. Thanks a lot for your support in advance!</p><p>If you want to learn more about Bayesian statistics, see my previous post here:</p><p><a href="https://sthornewillve.medium.com/why-bayesian-methods-very-useful-for-business-decision-making-c533573de514">The Hidden Power of Bayesian Methods in Business Decisions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13af85cf4239" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I’m a 1-dan Go player with a decade of exerpeince; here is what the game has taught me about myself]]></title>
            <link>https://sthornewillve.medium.com/im-a-1-dan-go-player-with-a-decade-of-exerpeince-here-is-what-the-game-has-taught-me-about-myself-5ba42c6c6757?source=rss-b0734c922eea------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ba42c6c6757</guid>
            <category><![CDATA[strategy]]></category>
            <category><![CDATA[board-games]]></category>
            <category><![CDATA[improvement]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[balance]]></category>
            <dc:creator><![CDATA[Simon Thornewill von Essen]]></dc:creator>
            <pubDate>Sat, 12 Apr 2025 18:33:08 GMT</pubDate>
            <atom:updated>2025-04-12T18:38:53.397Z</atom:updated>
            <content:encoded><![CDATA[<h3>I’m a 1-dan Go player with a decade of experience; here is what the game has taught me about myself</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vQ9m8PBXSEqtn_LK" /><figcaption>Photo by <a href="https://unsplash.com/@elenapopova?utm_source=medium&amp;utm_medium=referral">Elena Popova</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When you’re freshly single, diving into something new is a great idea. I had just gotten out of my first relationship at University, and I needed to do something new. Luckily, I had a friend at the time who was going to the local Go Club in Edinburgh, so I tagged along.</p><p>Little did I know, this was the start of a lifelong obsession.</p><p><a href="https://en.wikipedia.org/wiki/Go_(game)">Go</a> is <em>the oldest</em> board game that is still played today. It’s an abstract strategy game where two players take turns with black and white stones to surround the most territory without losing stones.</p><p>The beauty of Go (<em>aka</em> Baduk or Wei-chi) is that it has a relatively simple ruleset and set-up, but from these rules comes near infinite complexity.<br>The patterns are recognisable and easy to learn, yet have infinite variations that keep them compelling while keeping you off balance.</p><p>This game is also highly mysterious even after the progress that has been made to understand this game over millennia. Because the game is so flexible and complex, your personality becomes reflected in the way you play, and what holds you back over the board is usually what you need to learn in life as well.</p><p>Here is what I learned;</p><h3>Letting go of rigid thinking and preconceived notions</h3><p>In <em>The Art of War,</em> Sun Tsu says the following:</p><blockquote>An army may be <strong>likened to water</strong>; water leaves dry the high places and seeks the hollows; an army <strong>turns from strength and attacks emptiness</strong>.</blockquote><p>As with many statements in this book, there is a deep profundity to this statement.</p><p>The fundamental learning from <em>The Art of War</em> is that you should not attack the enemy where they are strong but attack them where they are weak. It sounds like an obvious statement, but how many times in your life can you think of where you attempted to achieve something directly only for it to fail?</p><p>The real question is, why do we do what’s obviously stupid?</p><p>The answer: <em>We are often committed to something without even knowing, which means we can’t think or do anything else. This is what it means to be human.</em></p><p>When analysing a game of Go, you reach situations where you have no good answers and where the position or the entire game is lost. You scour the position for better answers but can’t find any. This is because your goal was wrong from the start. <em>The mistake happened earlier.</em></p><p>As you review your games, you’ll repeatedly find examples of situations where you cling to some belief or assumption, only to find that it was just another fixed idea. The thoughts you hold most dearly are often the ones you need to question.</p><p>The same thing also happens when interpreting your opponent’s moves. They’ll often play with the intention of “You must listen to me. My move is right, and you are weak!” But should you believe them? Once again, you look ahead to see if the threat is valid.</p><p>Therefore, strategy is about disbelieving not only your ideas but also your opponent’s to find where the weakness lies and either covering it up or exploiting it. You can only do this by challenging your assumptions so that you don’t leave yourself open while attacking where your opponent is strong and to “attack emptiness”.</p><h3>I learned to be assertive but not aggressive</h3><p>From the previous section, you may also get the feeling that you can separate the personalities that we have over a Go board into two camps. Those who prioritise aggression and live the savage thrill of the hunt in hopes of crushing their opponent underfoot and those who prioritise defence and enjoy peacefully reading their book by the campfire as the wind fails to breach sealed walls.</p><p>I falsely learned that if I just made stronger shapes. If I fixed my problems beforehand, that would be enough to win a game. But that is not so because if you spend all of your time protecting your weaknesses, your opponent will get away with the weakness in their shape.</p><p>In all honesty, I never wanted to be a defensive player. I have always admired the free-thinking nature of an aggressive player, but I could not let go of my ideas enough to acknowledge this as I looked down on them. I was taught and believed that this way of playing ultimately holds players back. It was wrong.</p><p>This meant that when I struggled to improve, I continued to learn theory when what I needed was to get more experience by playing more in combination with a teacher or another stronger player who could correct the tactical mistakes and missed opportunities in my game. It was ironic that I couldn’t learn what I increasingly needed to improve by looking down on my opponents as being incorrect.</p><p>I still struggle with this problem somewhat, but I am slowly learning to look at my opponent’s shape and, if it is weaker, say, “No, you”. “How dare you point the figure at me when you are so clearly in the wrong?” This gives a different point of view on shape that would allow me to take an opponent’s weak connections and snap them like a branch in a forest.</p><p>This is important in real life because many live under the illusion that less is possible than we think. After all, we need to follow the rules. The rules and the law are there for a reason, but you must also learn to think beyond them rather than always playing within them.</p><h3>What Go Taught me about learning</h3><p>Go is an incredibly difficult game to learn. Due to its complexity and unpredictability, it demands dedication and discipline.</p><p>Of course, you can take this to mean learning various opening patterns (Fuseki, Joseki) and doing problems (Tsumego, Tesuji). All this study away from the board, attempting to memorise everything you need to know for the game.</p><p>But here’s the rub: Go is so expansive that you can <strong>never</strong> understand everything you need before a game. You can waste a lot of energy learning everything when you learn what you need to learn from just playing and reviewing those games with a stronger player.</p><p>And beyond that, you need to play many games against various opponents in different contexts. This means different board sizes, different time controls, and different types of players.</p><p>There’s also value in a community of people to challenge your ideas. To collaborate and to have a rivalry with. To learn and to teach. Do you want to become a strong Go player? Find those who share your passion and spend time together with them.</p><p>I spent so much time memorising shapes on my own that my progress completely stalled, and it was once I started seeking feedback from a teacher, seeking community and finding patterns between my losses that I began to improve again.</p><h3>Seeing Past The Illusion of Endless Improvement</h3><p>One way of thinking about how to play a great game of Go is about efficiency. No stone is wasted, and the moves work together as you perfectly defend what you need to while squeezing every drop out of your opponent’s weaknesses.</p><p>Yet, I was spending so much time on the game that it distracted me from my wider life. I wasn’t focusing on my education and, later, my career with the same intensity. Perhaps that’s healthy, but my relationship with Go hasn’t always been like that.</p><p>The more you play, the more you get sucked into trying to improve and as you get stronger, that gets more difficult until your natural ability bottlenecks you. I say this even after breaking my plateau and believing I can still get stronger. I can still improve, but the cost is too great, and there are other things I need to focus on outside of Go. (Perish the thought.)</p><p>When we’re young, we are taught to childishly strive to <a href="https://www.youtube.com/watch?v=6xKWiCMKKJg">be the best like no one ever was.</a> But what makes you successful in an economy and life is being “pretty good” at one/some things and knowing “just enough” at many other things that you can combine to do what you need to do.</p><p>When I stepped back, I realised that on online-go.com (OGS), I’m a ~93rd percentile player, and if I worked hard, I could be a 96th percentile-ish player. But was it worth it? It depends a little on what you value, but since I’m in my early 30s, I have other things that I need to focus on, as well as life, my wife, my career, and my friends, to say nothing of things like children and so on.</p><p>Learning to give up my obsession with being a stronger player was one of the hardest things I’ve ever done because of the sunk cost I put into the game to get to where I am. Yet, the decision is clear: while I can continue to play and teach others at my local club, it truly must be casual, and for now, I cannot continue to attempt to improve and be competitive while also doing everything else that I want to do before I die.</p><p>In Go, sometimes (much more than you may think), the best move is to tenuki — “play away” — and I applied that lesson to my life.</p><h3>Go is the ultimate game, no others will <em>ever</em> come close</h3><p>Although I don’t play as much these days. Go has fundamentally changed the way how I view problem-solving and approach life. It’s taught me to see past what I believed was true, question my assumptions, balance my life, and improve at the things that need improvement without getting lost in academics.</p><p>Still, Michaela from the YouTube channel <a href="https://www.youtube.com/results?sp=mAEB&amp;search_query=in+sente">In Sente</a> said it best when she <a href="https://www.youtube.com/watch?v=ZEtrNR6_v20">announced</a> that she would also take a hiatus from her Go-playing career.</p><blockquote>Games are supposed to help us live better lives and if we sacrifice our life for the game, we’re not learning the lessons we need to learn.</blockquote><p>Still, I enjoy returning to the game and, with fresh eyes, looking upon the beauty of a finished game.</p><p>I love this game for what it teaches about us. This is why the game is so linked with Buddhist philosophy and mindfulness.</p><p>It will teach you to live like no other game can.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ba42c6c6757" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hidden Power of Bayesian Methods in Business Decisions]]></title>
            <link>https://sthornewillve.medium.com/why-bayesian-methods-very-useful-for-business-decision-making-c533573de514?source=rss-b0734c922eea------2</link>
            <guid isPermaLink="false">https://medium.com/p/c533573de514</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[uncertainty]]></category>
            <dc:creator><![CDATA[Simon Thornewill von Essen]]></dc:creator>
            <pubDate>Sat, 05 Apr 2025 10:52:17 GMT</pubDate>
            <atom:updated>2025-04-18T18:54:30.290Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IiQ2zEdfPvykC2Ts" /><figcaption>Photo by <a href="https://unsplash.com/@kmuza?utm_source=medium&amp;utm_medium=referral">Carlos Muza</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Making decisions in business can be difficult. There are often multiple ways to evaluate a decision, making navigating and evaluating choices complex. Furthermore, one of the most significant issues is having incomplete or little information to work off of.</p><p>Luckily, we often have access to information that would help us to structure and inform decisions, often more information than we realise. Bayesian methods allow us to combine the little data that we do have with our current understanding of a business situation to make the best decisions possible, even when the picture is unclear.</p><p>Bayesian methods provide a robust framework that allows you to quantify uncertainty, update your beliefs in real time, and make decisions even with all of the problems outlined above. There are substantial advantages to the framework that one should know about.</p><h3>What is Bayesian Statistics?</h3><p>Bayesian statistics is a branch of mathematics which uses <a href="https://en.wikipedia.org/wiki/Bayes%27_theorem">Bayes’s Rule</a> to model outcomes of interest. Its defining feature is that it conceptualises statistics as about degrees of belief.</p><p>The distinction is important because it questions whether statistics are “real” insofar as they are part of the world as opposed to something that we construct to understand that world.</p><p>Bayesian models were unpopular because of how old-school scientists idealised objectivity, leading them to want to believe that the statistics that we measure exist in reality and that we should minimise the gap between our understanding and what exists in that world by allowing data to speak for itself.</p><p>Both methods are valid and useful within their specific contexts. However, Bayesian statistics are under-represented because of how old-school scientists avoided its use. It wasn’t until the 20th century that Bayes rule was used extensively for many applications such as telecommunications and spam filtering. It even had success in AI applications such as <a href="https://www.nature.com/articles/s41598-022-12509-6">self-driving cars</a>.</p><p>It has also gained some popularity as a result of the replication crisis that took place in the 2010s. Bayesian statistics are often touted as a solution to this problem, and it slowly continues to gain popularity to this day.</p><p>For more information, I would recommend reading this quick medium article here.</p><p><a href="https://medium.com/data-science/what-is-bayes-rule-bb6598d8a2fd">What is Bayes Rule?</a></p><h3>Bayesian statistics are ideal for quantifying risk</h3><p>Bayesian statistics are mostly used to quantify risk in low-information, high-stakes environments. This makes it very popular in the following fields:</p><ul><li>Cybersecurity</li><li>Fraud</li><li>Financial Analysis</li><li>Forcasting election results</li><li>Gambling/Betting</li></ul><p>The reason why Bayesian statistics are so useful in these areas is because often there is very limited data with which to make inferences. For instance, with election results, you will only see the result once. Often, you have some analogous but not completely identical examples you may draw from that can be difficult to put together. Bayesian statistics gives you the structure to do this in a logically coherent way.</p><p>Not only do businesses face the problems that I outlined above, but within the area of business itself, where you make bets all the time and understanding the risks that come with them is essential to making good decisions.</p><h3>Explicit Articulation of Assumptions</h3><p>You can think of Bayesian modelling as having a similar benefit to your analysis of data that <a href="https://medium.com/@sthornewillve/writing-and-note-taking-will-10x-your-productivity-1b7967b3067f">writing has on your thinking</a>. It requires you to clearly state your priors and model, revealing the assumptions behind mapping your data to your estimates.</p><p>Doing this gives the same benefits as writing does because it forces you to think coherently about how you are interpreting your data and the process that generates it from beginning to end rather than making quick assumptions without having a deep understanding of how those assumptions change the interpretation of your data.</p><p>The result is that you will eventually construct a model tailored to your problem with knowledge of where it performs well and where it struggles. It’s not that traditional methods won’t lead to this understanding, but a Bayesian workflow creates this understanding as a requirement rather than as an optional afterthought.</p><p>For simple problems, it may not be necessary to go to these lengths. However, when there is a lot of money riding on a specific decision, you want to be able to look at the data from various angles and consider many scenarios to understand the risks of those decisions.</p><p>When there is a lot on the line, you would rather be more rigorous in your thinking and more aware of your assumptions and risks in each case.</p><h3>A more persuasive and inclusive form of science</h3><p>The key advantage of Bayesian statistics is that it defines probability as a degree of <em>belief</em>. At first glance, it may not be clear what this means or why it is advantageous.</p><p>Many people think this difference is merely philosophical and thus either irresolvable or inconsequential; both conclusions are misguided. You resolve this tension by examining what the framework allows you to do.</p><p>Because Bayesian statistics allows for subjectivity by allowing you to mix the data with your priors through a model, it’s possible to consider multiple points of view. This is important because if you come to different conclusions based on how you interpret the data, it’s important to investigate the models that create those conclusions.</p><p>Investigating how we believe things to be true epistemologically is more convincing to people than just presenting them with conclusions and forcing them to believe them because people always interpret data based on their beliefs. That is, we are all implicitly Bayesian.</p><p>However, people don’t like to admit that they think this way because it does not give neat answers where the data are not strong enough to warrant them. However, if your data are not strong enough to create a unified signal for (reasonably built) models, you are running away from the problem of likely having no single truth given current reasonable states of knowledge.</p><p>For example, if you are in a casino and suspect someone has a loaded dice, you may consider different starting points for your suspicion of the dice in question. If the data you gathered are strong enough to convince even a rational but sceptical person that the dice are loaded, then that creates a convincing case for accusation.</p><p>On the other hand, if you are attempting to evaluate evidence for an event that you know is very unlikely, then it will require a lot of data to convince you despite having an open mind.</p><p>The answer is not to enforce certain assumptions or claim that they are the only “most objective” ones in an attempt to force that answer through. By engaging with other ways of constructing a problem and understanding the reasoning that led someone to those conclusions, you can understand those you disagree with and discuss the assumptions that matter.</p><p>By allowing for a plurality of thought, you will not only be able to more easily arrive at a consensus with those on your team, using a method of discourse that is a lot more likely to convince people, but you will also have the discussions that enable strong decision making, by challenging assumptions and considering a problem from multiple perspectives.</p><h3>Estimating the risks your stakeholders are interested in</h3><p>Bayesian statistics encourages you to think probabilistically. You operate from a space of uncertainty, assigning a risk to every decision you take while detaching yourself from the outcomes themselves. This is important because making good decisions is not necessarily just about avoiding “wrong decisions”; it’s about making the best choices <em>given the information and understanding you have.</em></p><p>In <em>thinking in bets,</em> Annie Duke discusses a habit that poker players have called “resulting”, where they evaluate their based on what happens in a game. However, Poker is a game of information asymmetries where it is challenging to know what the opponent’s hands contain. Resulting leads to learning the wrong lessons because it’s possible to make good bets and still lose. The key is to understand which plays maximise returns over many games and be able to make those plays given the information that one has access to.</p><p>Many of the questions that stakeholders have are also probabilistic. For instance, they often ask…</p><ul><li>What is the probability that this campaign will return more than £X in Y time?</li><li>Given the distribution of possible costs/revenues, what is the optimal decision?</li><li>If I make different assumptions, what will the likely outcomes be?</li><li>Where do I have enough data to justify certain conclusions, and where should I try and get more to increase my certainty?</li></ul><p>The answers to these questions are also often directly related to what we want to understand about the data. Science often wastes time not answering these questions, leaving business stakeholders in a tricky situation.</p><p>This is important because it allows you to <em>always</em> say <em>something </em>about the outcomes you are interested in. This is important when there isn’t a lot of data because gathering more of it is often expensive, time-consuming and even sometimes impossible. In these situations, it’s especially important to think about what you can and can’t say from the data to make the most informed decision possible.</p><h3>Estimation in the absence of data</h3><p>Another advantage of constructing distributions as a matter of belief is that you can reason about probability even when you have <em>no</em> data.</p><p>In business, there are often various quantities that we do not know because we simply have no data. However, we often have an idea of what we may expect the distribution of this missing data to look like. You can then construct this distribution and perform (Monte-Carlo) simulations to understand what could happen given this assumption.</p><p>This is how consultants such as <a href="https://hubbardresearch.com/">Douglas Hubbard</a> quantify risk and other challenging quantities in “<a href="https://www.howtomeasureanything.com/">How to Measure Anything</a>”. It turns out that humans can be trained to reliably develop decent distributions for things that are challenging to measure using <a href="https://en.wikipedia.org/wiki/Fermi_problem">Fermi estimation</a>.</p><p>Fermi estimation is when you want to derive an estimate by breaking it down into its parts and attempting to estimate each component separately to calculate that estimate. Consider this popular interview problem; “How many piano tuners are there in Chicago?” You answer this question by breaking it down into parts. This may include the population of Chicago, the number of pianos per household, the tuning frequency of pianos, and so on. Using this method tends to <a href="https://en.wikipedia.org/wiki/Fermi_problem#Justification">make the estimand more reliable</a> on average.</p><p>Of course, if you have data, you can use Bayes’s rule to update these distributions to get more accurate ones or build models to create a posterior distribution for each one.</p><p>It’s true that assuming a distribution for missing data is a pretty strong assumption. However, at least you know that the assumption is being made, and if someone disagrees, you can freely change it based on what they think would be more reasonable.</p><h3>Leveraging Simulations to Address Jensen’s Inequality</h3><p>I mentioned using Monte-Carlo simulations above. The reason why that is so important is because of <a href="https://en.wikipedia.org/wiki/Jensen%27s_inequality">Jensen’s Inequality</a> leading to the “<a href="https://www.flawofaverages.com/">Flaw of Averages</a>” as coined by Sam Savage.</p><p>“The Flaw of Averages” is what happens when one attempts to estimate an outcome while assuming that each of the components of that calculation can be combined linearly.</p><p>This matters because if a distribution has a long tail, it has an outsized effect on the target distribution and the kinds of probabilities that you calculate it with. This means your estimate will not correctly capture the probability of extreme events and thus underestimate risks leading to bad decisions.</p><p>Most of the time, people don’t know they are making these assumptions. Monte-Carlo estimation allows for the estimation of probabilities without calculating everything by hand.</p><p>Since Bayesian methods rely so much on distributions, they can be used easily with Monte Carlo sampling. This means that you are quantifying the uncertainty in your estimates, giving you many benefits already explained and protecting yourself from bad mathematical errors that could cause you to underestimate uncertain outcomes and unknowingly make risky decisions.</p><h3>Bayesian statistics can be cumbersome, but the effort is worth it</h3><p>The main drawback for Bayesian methods, one of the major reasons they struggled to gain traction even though Bayes’s rule has been around for hundreds of years, is that simulations and calculating posterior distributions are often computationally intensive.</p><p>It still makes sense to use non-Bayesian approaches. For instance, traditional approaches make sense if you try to understand the probability of seeing your data against a known parameter. For example, if you consider an industrial process where you expect some concentration to be around a specific value, then Bayesian methods are not necessary because you have a good idea of what the parameter’s value is in real life, and thus, it has no distribution associated with it.</p><p>Secondly, if your priors can be sufficiently weak, your model is sufficiently simple, or you have so much data such that a Bayesian model would not be meaningfully different, then it would be permissible to use traditional methods.</p><p>A third (bad) reason is that non-Bayesian approaches are faster and easier to set up. Defining models, assuming that the priors are flat and calculating the maximum likely estimate is fine, but then you lose all of the benefits I just outlined and hamper your ability to communicate with stakeholders and address their needs.</p><p>However, if you need a relatively complex model, you have prior information that matters or relatively sparse data, then traditional models will be misleading, and Bayesian methods will be <em>necessary</em> for calculating rigorous results. They are not some neat trick.</p><p>The real roadblock is that when those difficult situations present themselves, you will need a lot of practice to get these methods to work. Therefore, you should be using them even for simple cases to gain experience with the tools and how posterior distributions are used in decision-making so that when the opportunity arises, you are prepared for it.</p><h3>Conclusion</h3><p>As this blog post shows, Bayesian methods provide massive value to those making business decisions. This is because the subjectivist approach of a Bayesian framework allows you to think more clearly about your modelling process, include a plurality of perspectives in your analysis, reason about decisions even when data are sparse or absent and avoid various mathematical mistakes through Monte-Carlo simulation.</p><p>The output is much easier for stakeholders to understand. It will encourage probabilistic thinking within them while allowing you to engage with them more deeply rather than forcing them to accept the “objective” results analysis you have performed, which they’ll interpret subjectively anyway.</p><p>Bayesian methods are often computationally intensive, and due to increased computational power, they have become more popular. They still require some practice and education to be able to use well. However, if you have a decision that you need to make and the cost of a wrong decision is high, would you not want to make the most informed decision possible?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c533573de514" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[These are the most important data science skills I learned since boot camp]]></title>
            <link>https://sthornewillve.medium.com/these-are-the-most-important-data-science-skills-i-learned-since-boot-camp-4cc1a413eed8?source=rss-b0734c922eea------2</link>
            <guid isPermaLink="false">https://medium.com/p/4cc1a413eed8</guid>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[initative]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[skills]]></category>
            <dc:creator><![CDATA[Simon Thornewill von Essen]]></dc:creator>
            <pubDate>Sat, 29 Mar 2025 09:04:31 GMT</pubDate>
            <atom:updated>2025-03-29T09:04:31.119Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DWMy-eJWqx8dr76H" /><figcaption>Photo by <a href="https://unsplash.com/@wocintechchat?utm_source=medium&amp;utm_medium=referral">Christina @ wocintechchat.com</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Between graduating with a master’s degree in Chemistry in 2017, first getting hired as a data analyst in 2018 and progressing to a data science position in 2021, I have learned a lot.</p><p>I feel that one of the most useful differentiating factors was taking multiple nanodegrees with Udacity, which gave me a way to build these skills in a focused way that was also credible by creating my own projects which I published online.</p><p>However, since I had just come out of academia, I was still taking a very “bottom-up” approach to learning. This means learning subjects in isolation, working your way up to applying them in the real world. This is as opposed to starting with a problem that you need to solve and learning only the skills you need to solve those problems when needed.</p><p>This left a considerable blind spot in my skills in that as a data scientist I was very technically focused, but often missed on other skills such as understanding…</p><ul><li>how to take initiative</li><li>what the focus of your analyses are</li><li>how to focus on business needs</li><li>how to track your impact so that you can communicate them in written and spoken form</li></ul><p>I wouldn’t go so far as to say that building these technical skills was a waste of time. For instance, what I learned through linear algebra and statistics does give me a level of knowledge that supports my work, but that is also exactly the problem. Bottom-up learning leads to an excess of information. That information is often <em>supporting</em> in nature so that you know what your doing, but it is highly wasteful.</p><p>This is important because in my career I have been told by stakeholders time and time again that what matters is business problem-solving rather than building some kind of process automation with machine learning.</p><p>My aim with this blog post is to write down the skills that I have learned since then so that you (someone early in their data science career) are aware that they are the limiting factors to making an impact. The technical aspect is just one part of the job.</p><h3>Learn to take initiative and actively find problems to solve</h3><p>The first problem that I had when I started my first position as a data analyst was that I was in a junior position. The problem with being in a junior position is that expectations are set very low such that they can hold you back because you are expected to do as you are told.</p><p>When I arrived at my first position on the job there was a machine learning project that was scoped by colleagues who had left the team. When I joined the team I thought that it would be a good idea to go to the stakeholders and ask them what they expected/needed from what we wanted to build for them. However, when I did that my manager at the time said “The scoping has already been done, you just need to follow through on it”.</p><p>My manager at the time was half-right that it would be redundant work. Where they were wrong was that we were new colleagues and there was no document saying what their requirements were and why. Furthermore, it would have been a good opportunity to build rapport with the stakeholders.</p><p>This happened multiple times, where I attempted to take initiative on an idea only to have it shut down. If that happens multiple times then you begin to lose your spark. This is what hierarchy does, it actively breaks down the initiative of those who are at the bottom because the expectations are so low that they are told what to do instead of thinking for themselves.</p><p>Thus, I forgot everything that I had learned by listening to podcasts about being proactive and a go-getter and focused on following what my manager told me to do.</p><p>This is a problem because in order to progress in your career, you need to show evidence of exceeding the scope of your current role rather than being defined by it. In order to learn, you <em>need </em>to take this kind of decision-making into your own hands so that you can fail and learn from those experiences.</p><p>Yes, this can potentially make you a loose cannon in the short term and you need to be careful to not always be “doing the most” by getting in other people’s way or acting like you can know or do more than you can. Sometimes it really is necessary to just deliver the expected work quickly, but you should be taking your own approaches to your own problems and acting on them rather than waiting for someone to tell you what to do.</p><p>Ideally, you should learn to engage in collaborative problem-solving with your manager so that you come to good ideas together. Yes, I could have just gone to my stakeholder, built that relationship and got their requirements and problems, but I should have said to my manager “I understand that this feels like redundant work, but how much of a waste will it be for us newbies to build our relationship with our stakeholders and understand their requirements better?” That would have started a much better conversation than just shutting up and doing what I was told.</p><p>I did not know to do this and so I believe this held me back for a long time.</p><p>The key is to spot the risk-asymmetry here. So long as you do not build up a bad reputation, the pay-offs for succeeding with your ideas are far greater than if you “fail” in a way that is limited and not that big of a deal.</p><p>Beyond just following through on ideas that you have, In order to be proactive as a data scientist, you need to talk to your stakeholders and understand their problems inside and out and understand the roadmap that they have and identify the key uncertainties in that roadmap and think about how you can provide value to plug them up.</p><h3>Do not vaguely explore data, use a plan instead</h3><p>Another huge misunderstanding that I had was how to analyse data. Bootcamps will often teach exploratory data analysis by saying that you explore each variable’s distribution, plotting them against each other and then taking those plots and segmenting them in interesting ways for multi-variate analysis.</p><p>Most data analysis does not look like this, this is far too open-ended and you will get lost in your data among all of the columns, and there are usually enough so that this kind of exploration is highly time inefficient.</p><p>Instead, before you start looking at your data, open a Google doc and write down the following:</p><ul><li><strong>Uncertainties that inhibit action</strong>: What are the major decisions that you want to make and what is blocking you from making those decisions?</li><li><strong>The required data</strong>: What data will you need for this analysis? Where can you find it?</li><li><strong>Analysis plan</strong>: How will you use the data you pull to answer the questions from the top?</li><li><strong>Assumptions and Limitations</strong>: You can stack an arbitrary number of assumptions to make your analysis feasible so long as you can justify that it generalises enough to the real world. However, what parts of your analysis will not generalise because of this?</li></ul><p>All of this will give you a focus and purpose, you can even write another section underneath about what you found during your analysis so that you have everything you need for your final write-up.</p><p>This is important because when doing data analysis, you want to focus on driving decisions rather than surfacing information that is “nice to know”. It is true that facilitating understanding is something that you want to support but your greatest value comes from driving action, so focus on that.</p><h3>Focus on what the business needs</h3><p>You may be able to tell from my publications on Medium that I do not shy away from being technical in my role as a data scientist. I do not want to lean into the kind of anti-intellectualism that can happen at a business where all that matters are techniques that can deliver the minimum viable analysis using only high-school math. It is not why I became a data scientist.</p><p>It still needs to be said, however, that your job as a data scientist is to deliver that business value even if the tools that you use to do it are simple and you can definitely become a highly ranked data scientist by doing so. Even if you lean into more complex methods, you <strong>must</strong> start with meeting business needs first.</p><p>The best way to do this is to 1) learn about business in general and 2) learn about <em>your</em> business.</p><h4>Learn about business in general</h4><p>There are a number of ways to learn business skills and nobody expects the data scientist to be an entrepreneur or consultant. That doesn’t mean you won’t meet a bunch of data scientists who were at one point at least consultants anyway!</p><p>There are all sorts of books that you can read about this. Some nice ones that I have on my bookshelf are “<em>Case in Point</em>”, “<em>Lean Analytics</em>”, “<em>Crossing the Chasm</em>”, “<em>The Innovator’s Dilemma</em>”, “<em>Only the Paranoid Survive</em>” and “<em>What They Teach You at Harvard Business School</em>”.</p><p><strong>Full disclosure</strong>: I have not read all of these books, although I have read some. If I had to prioritise one that I have read then pick “<em>Lean Analytics</em>”.</p><p>Having some idea about business theory will help you to be critical about where the business is going and to understand how to better play your role in delivering value. This helps you to guide your analyses and to create findings that even your stakeholders didn’t think of.</p><h4>Learn about your business</h4><p>Once you understand business in general, understanding more about your own specific business by taking notes during company-wide meetings will also be important. The high-level financial reporting and goal communications meetings will set the key performance indicators that you want to move and lower down you will get to the department and team level plans which contain the “how to get there”.</p><p>Once again, the hierarchy strikes back. It may well be that you disagree with company direction, but I would recommend that you only focus on the decisions that happen one level above you because they are proximate enough that if you inform a decision then it will get you noticed but not so far away that you are challenging assumptions that you are just too junior to meaningfully do anything about yet.</p><p>Once you have an idea about business in general and what your business is up to and your role in it, that’s when you can start being helpful in what you do.</p><h3>Always be looking to log your achievements, estimate your impact and communicate them</h3><p>A lot of people get to the end of the year and when the calibration starts they then need to remember a whole year’s worth of achievements. If you do not have them written down somewhere then you are going to undersell yourself. Do not be that person.</p><p>Keep an active log (a <a href="https://jvns.ca/blog/brag-documents/">brag doc</a>, if you will) of what you have done. For instance, I have weekly and monthly planning sessions where I take my to-do lists and keep track of what I achieved for that week. I take the major highlights and put them into my brag doc.</p><p>My current workflow is also to write my self-reflection during the year by comparing what I do against the expectations for the next level up, but it seems that this only really makes sense as you reach project completion.</p><p>This does not mean you need to be arrogant and over-emphasise your work. Managers will roughly know how much value you are bringing and if you are delivering all value and have no development areas then they will be sceptical of your account. It is okay and trust-building to have your humanity on display, but if you start actively diminishing yourself then your manager may not necessarily know to correct for this.</p><p>The best part about accurately reflecting on your performance is that not only will you know what to work on but it will also mean that your feedback conversation will be more productive because you will either agree or you will be in a position to understand what you misunderstood or under-estimated.</p><h3>Conclusion</h3><p>Real impact comes not from just analyzing data, but from understanding what the business needs, driving decisions with clarity and confidence and estimating/communicating the value of your insights. Bridging this gap requires developing problem-solving instincts, refining your communication skills, and learning to think critically about the data you work with.</p><p>While data science boot camps can help you to build those technical skills, you can see that there are a number of skills when it comes to navigating the workplace and delivering value that I needed to learn. This took me a little time and I hope that it can help to accelerate you on your journey.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cc1a413eed8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing and note-taking will 10x your productivity]]></title>
            <link>https://sthornewillve.medium.com/writing-and-note-taking-will-10x-your-productivity-1b7967b3067f?source=rss-b0734c922eea------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b7967b3067f</guid>
            <category><![CDATA[thinking]]></category>
            <category><![CDATA[writing]]></category>
            <category><![CDATA[notetaking]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[time-management]]></category>
            <dc:creator><![CDATA[Simon Thornewill von Essen]]></dc:creator>
            <pubDate>Wed, 19 Mar 2025 09:17:15 GMT</pubDate>
            <atom:updated>2025-03-19T09:26:21.183Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eP7Vjb4emx5dNDpl" /><figcaption>Photo by <a href="https://unsplash.com/@aaronburden?utm_source=medium&amp;utm_medium=referral">Aaron Burden</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I have published written content online in the past, but I was always doing it as a means to an end. Now I realise that writing is not just some way to signal that you are working harder than the competition, it is how you think better and improve yourself so that you don’t have to signal it.</p><p>In 2018, I was trying to break into the data space, and I had to do some writing on Medium as a part of my Udacity nanodegrees. Back then, my understanding of writing was that it was one form of media creation among many. Sure, you could choose to write if you were passionate about it but you could also choose to make Youtube videos or podcasts. Writing seemed to be for someone else — something that was academic rather than something that had its own benefits and value.</p><p>Let me tell you, I couldn’t have been more wrong.</p><p>This is the story of how I used writing to organise my thoughts and to capture information that I was otherwise forgetting. It is about how the act of being able to access my thoughts across time, in a form that <em>never</em> degrades, created an unexpected force-multiplier to my work.</p><h3><strong>Life before writing</strong></h3><p>As someone with diagnosed ADHD, I struggle to keep track of details. I did not have any kind of system for keeping track of what I was learning, thinking or even just planning what I was doing. If what you are doing is simple or you do not have a lot to keep track of then that can work, but unfortunately, I do not think anyone who can read this article has such a simple life. This meant that for a long time I lived in a constant state of overwhelm.</p><p>You could see this in the work I was doing. While analysing data, I did not have a lot of focus and I struggled to keep track of what I found and put it all together in a coherent way. People would tell me things in conversation, interesting ideas would strike me away from my desk but all of those ideas disappear as suddenly as they appeared.</p><p>Beyond the disorganisation and always struggling to remember what I was forgetting (the known-unknowns), I had no idea of the insights and potential that I was missing because of that forgetting (unknown-unknowns).</p><h3>The last straw that got me started</h3><p>Around a year ago, I was starting in a new team at my company and during that time I was also starting to launch my own initiatives in the business surrounding Bayesian modelling. I had so much in my head and I was really struggling to understand how everything fit together.</p><p>I thought to myself “I just need to get these ideas out of my head” and remembered that I had watched some short videos by<a href="https://www.youtube.com/watch?v=QXIa0NAycGo&amp;t=303s&amp;pp=ygUacHl0aG9uIHByb2dyYW1tZXIgb2JzaWRpYW4%3D"> Python Programmer</a> and<a href="https://www.youtube.com/watch?v=hSTy_BInQs8&amp;t=226s&amp;pp=ygURb2Rlc3N5YXMgb2JzaWRpYW4%3D"> Odysseas</a> talking about the popular note-taking app <a href="https://obsidian.md/">Obsidian</a> which allows you to connect notes together through links. I downloaded it and started writing.</p><p>Was this some kind of well researched 4-d chess move? No. The pain of trying to keep everything in my head became too great and so I just needed to get them the hell out of there.</p><h3>Writing immediately expanded my horizons and developed my thoughts</h3><p>As I started writing using Obsidian, I would link every keyword that could possibly become its own note, as though I was trying to build my own Wiki. It was great to get the ideas on to “paper” and as I typed I realised that they became more articulated and interconnected than when they were inside my head.</p><p>Imagine how difficult it would be doing high-school level mathematics without a sheet of paper in front of you, instead doing everything mentally. This is how many people attempt to think about their lives, there is a level of structure that they do not realise they are missing.</p><p>Attempting to think without writing is like trying to drive somewhere while having your car suspended in the air. Writing is when the metaphorical rubber meets the road — you can finally get somewhere.</p><p>The only problem was that I was linking <em>everything</em>, and when you do that you start seeing all sorts of connections that you never saw before. There would be too many ideas that made it difficult to communicate because my thoughts were only coherent to myself in the moment.</p><p>Once I realised this, I started using fewer links as tags to specific ideas, directing references and creating some amount of hierarchical structure when the links to one idea became too big. By adding the minimal amount of structure, the hallucinogenic feeling went away and everything became a lot more coherent.</p><h3>As I continued to write, things began to improve</h3><p>Writing has a lot of very immediate benefits. The obvious use-case is to just remember things, to have more structure whilst filling in the gaps. But that was just the beginning.</p><p>As I started to remember more from meetings, and understand the key questions and decision points, my analyses became a lot more detailed and focused. I began to keep more of a grip on every-day tasks as well as put those tasks next to each other to better prioritise and understand the commonalities between them.</p><p>I cannot believe I lived my entire life not understanding this power.</p><h4>Writing helped me in ways that I didn’t even expect</h4><p>Towards the end of 2024, I was struggling to be productive. When I started writing about why I was so disengaged from my work, I found that it was because I thought taking my own initiative would not be seen as valuable or even noticed.</p><p>This gave me something concrete to talk about with my manager and we resolved the issue together by including everything in writing during my performance review with a self-reflection. Guess who had all the notes ready to put everything together? I did. She later told me that the self-reflection I wrote was among the best she had ever read.</p><p>Even if that did not mean an immediate promotion, my growth was more evident and more potently argued for than ever before <strong>and</strong> I isolated my development areas and what to do about them very clearly.</p><p>The lesson… write! Your mind will either quiet itself or you will find practical things that you can do to help your situation and your actions will become more aligned with what you want.</p><h3>The benefits from my writing continue to compound</h3><p>When I started writing, things took a turn for the better very quickly, but it does not stop there. Why? Because those notes, the things I write, don’t vanish after I use them once. They persist and I can use them again for other projects.</p><p>This means that I’ve likely only scratched the surface, so who knows where this will lead?</p><p>You can see part of this in the existence of this blog post. I now write so much that I can easily start publishing some of it, because my Obsidian vault already has the thoughts and explanations within it. Perhaps my actions are now more oriented towards my goals, and maybe my goals now have more focus, but who knows what ideas I haven’t even thought of yet. The sky’s the limit.</p><h3>In the age of LLMs, human writing is going to become more and not less important</h3><p>When ChatGPT hit the scene in 2022, I thought “sweet, maybe I can get this help me write up this analysis I’m working on”. I wrote down a bunch of bullet points that I wanted ChatGPT to turn into paragraphs and let it churn something out.</p><p>My manager at the time (different from above) did not appreciate it, ChatGPT’s default style is to be wordy and I was leaning into it, which is a big no-no.</p><p>But verbose and convoluted writing is not what delivers value. Insights that have not been considered or well communicated before are!</p><p>Therefore, as automation becomes more widespread, your value as a human is increasingly driven by your ability to think critically and express your ideas in a way that connects with people. Writing is going to be more important than ever.</p><p>At my company, we are all-in on GenAI. We’re hooked up with Github Copilot, Google Gemini and we all have our own work API connected to ChatGPT. That’s great, but if we are going to be super-enabled then we’d better be super-enabled to go in the right direction and not into a brick wall.</p><p>To do that, you need to think — and to think you had better write.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b7967b3067f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>