<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Steffen Sjursen on Medium]]></title>
        <description><![CDATA[Stories by Steffen Sjursen on Medium]]></description>
        <link>https://medium.com/@sasjurse?source=rss-3286966ddae5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*bEHHt4TwZdIz0eLv</url>
            <title>Stories by Steffen Sjursen on Medium</title>
            <link>https://medium.com/@sasjurse?source=rss-3286966ddae5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:30:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sasjurse/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The hard part of improving retrieval augmented generation is measuring progress — let’s figure it…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@sasjurse/the-hard-part-of-improving-retrieval-augmented-generation-is-measuring-progress-lets-figure-it-a39d0439534d?source=rss-3286966ddae5------2"><img src="https://cdn-images-1.medium.com/max/1023/0*-N2IlugbjYe4Nrg_" width="1023"></a></p><p class="medium-feed-snippet">Product ideas based on RAGs (Retrieval Augmented Generation) are super fast to prototype. You can have an application that kinda works&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@sasjurse/the-hard-part-of-improving-retrieval-augmented-generation-is-measuring-progress-lets-figure-it-a39d0439534d?source=rss-3286966ddae5------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@sasjurse/the-hard-part-of-improving-retrieval-augmented-generation-is-measuring-progress-lets-figure-it-a39d0439534d?source=rss-3286966ddae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/a39d0439534d</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Steffen Sjursen]]></dc:creator>
            <pubDate>Wed, 20 Nov 2024 13:39:35 GMT</pubDate>
            <atom:updated>2024-11-21T09:48:16.758Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Test the Result First when Validating Data Science Pipelines]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@sasjurse/test-the-result-first-when-validating-data-science-pipelines-952c5985e070?source=rss-3286966ddae5------2"><img src="https://cdn-images-1.medium.com/max/2600/1*I12Ic10SD__tZGt7fKijkw.png" width="3834"></a></p><p class="medium-feed-snippet">Nobody wants to have the wrong estimates in a business critical analysis or introduce errors or unexpected behavior in production. So how&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@sasjurse/test-the-result-first-when-validating-data-science-pipelines-952c5985e070?source=rss-3286966ddae5------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@sasjurse/test-the-result-first-when-validating-data-science-pipelines-952c5985e070?source=rss-3286966ddae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/952c5985e070</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[test-driven-development]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Steffen Sjursen]]></dc:creator>
            <pubDate>Mon, 23 Mar 2020 19:05:36 GMT</pubDate>
            <atom:updated>2020-03-23T19:05:36.893Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The Pros and Cons of Using Jupyter Notebooks as Your Editor for Data Science Work]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/better-programming/pros-and-cons-for-jupyter-notebooks-as-your-editor-for-data-science-work-tip-pycharm-is-probably-40e88f7827cb?source=rss-3286966ddae5------2"><img src="https://cdn-images-1.medium.com/max/1280/1*DiRx0qapibqypyy91W8vBA.jpeg" width="1280"></a></p><p class="medium-feed-snippet">TL;DR: PyCharm&#x2019;s probably better</p><p class="medium-feed-link"><a href="https://medium.com/better-programming/pros-and-cons-for-jupyter-notebooks-as-your-editor-for-data-science-work-tip-pycharm-is-probably-40e88f7827cb?source=rss-3286966ddae5------2">Continue reading on Better Programming »</a></p></div>]]></description>
            <link>https://medium.com/better-programming/pros-and-cons-for-jupyter-notebooks-as-your-editor-for-data-science-work-tip-pycharm-is-probably-40e88f7827cb?source=rss-3286966ddae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/40e88f7827cb</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Steffen Sjursen]]></dc:creator>
            <pubDate>Sun, 01 Mar 2020 20:30:31 GMT</pubDate>
            <atom:updated>2020-03-04T02:06:29.741Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[“I have an idea involving Data Science” — a corporate checklist]]></title>
            <link>https://medium.com/@sasjurse/i-have-an-idea-involving-data-science-a-corporate-checklist-e2405c66e842?source=rss-3286966ddae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2405c66e842</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[consulting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <dc:creator><![CDATA[Steffen Sjursen]]></dc:creator>
            <pubDate>Sun, 02 Feb 2020 12:07:23 GMT</pubDate>
            <atom:updated>2020-03-26T20:02:45.739Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>“I have an idea involving Data Science” — a corporate checklist</strong></h3><p>As a data scientist and consultant, we do discuss a lot of ideas involving data science. Ideas of type:</p><blockquote>Data -&gt; Data Science -&gt; Valuable results</blockquote><p>Let’s look at some questions that can help you evaluate if this is a good idea. Not all of these questions needs to have a positive answer — there is no perfect project. It’s about managing risk vs. potential, how your work will enable better ideas in the future and making good bets. These are approximately what I try to ask when planning large projects or evaluating new initiatives. It is not a checklist we use for every day planning in an ongoing project.</p><p>For many cases, the organization will be a corporation looking for profits, but the same general checklist should work for other goals such as healthcare related data science where you are trying to improve outcomes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-It-rONE19tC-eFafGG1Cw.png" /></figure><h3>The hard and soft NO</h3><ul><li><strong>Is it legal to use the data for this purpose. Do you have the appropriate rights to use the data for this purpose?</strong> There are two main concerns. Just because you have access data to data doesn’t imply ownership — do we have consent from the owner of the data? Are your use case within relevant regulations such as GDPR?</li><li><strong>‘The headline test’. Would we be happy if this was a headline in a major national newspaper or all your customers knew about it? </strong>This one is pretty straightforward — if this was reported in an unbiased manner would your friends still like you? Is there a reputational risk to your organization?</li></ul><h3>What value can we create?</h3><ul><li><strong>How fast and cheap can we test our hypothesis? </strong>Return on Investment is a fraction, and one way of seeing large numbers in fractions is small divisors. This is also important for the ability to make changes. Your initial idea is not going to be perfect, but to improve it you need relevant feedback. How fast can you test and get a good ‘build measure learn’ loop going?</li><li><strong>If we found a perfect solution, how much money would we make? </strong>If your best case is not great, maybe we should look elsewhere. This is also useful for comparative purposes, are we starting with ideas that have the biggest impact?</li><li><strong>If we found a perfect solution, are the additional steps required to unlock value? </strong>Are there any additional sales or implementation required? Is that costly or risky? If so, that diminishes the value of our data science work.</li><li><strong>What is the baseline to beat? </strong>It is just not about creating a good model, but also something that is more useful than what you already have. There is a lot of value in predicting weather. But being better at predicting weather than existing solutions built on decades of experience and testing is going to be … hard.</li><li><strong>What will have learned if the idea is a failure? </strong>Sometimes, exploring a topic has independent value because you learn about your core business. It can help you see other use cases or ideas. Many of the truly great insights comes from accidental discovery. But then it helps to be in the right neighborhood of what you are trying to achieve.</li></ul><h3><strong>Is relevant data available?</strong></h3><ul><li><strong>Is the hypothesis testable?</strong> Can we disprove our idea or hypothesis? What would it take to kill the project? If we are not sure how to disprove the hypothesis, it is not a good hypothesis.</li><li><strong>Are there plausible links between what you are trying to estimate and data. </strong>This one is surprisingly easy — don’t overthink it. If you have data on sales in UK retail, that is probably relevant for customer preferences in the UK, but less so for temperatures in Antarctica. Maybe an extreme example, but what we are looking for are the really weird suggestions. However, weird suggestions show up quite frequently.</li><li><strong>Do we have labels on data (if applicable)? </strong>With a label, we mean if our data includes the ground truth about what we want to achieve. Example: If you are detecting cats in pictures, do you have a data set where the truth (there is a cat in this picture) is included? A lot of machine learning is about training the computer, but then you need to know what to train it for. Adding labels can in many cases be a very manual process that significantly drives costs. (Not applicable for unsupervised learning of course).</li></ul><h4><strong>Who are using this data today?</strong></h4><p>How much have we explored the data already?<strong> </strong>If someone have explored or used or data for other use cases, they probably learnt a lot about this data set. And understand it’s potential and limitations. Suggestions from people who understand that data set is usually more valuable.</p><ul><li><strong><em>Don’t know</em></strong><em>.</em> Bad and high risk. Signals extra exploratory work just to understand the underlying premise.</li><li><strong><em>Stored but not used</em></strong>. Bad. Without inspection, there will be problems that nobody saw or corrected before you.</li><li><strong><em>Stored and used in production</em>.</strong> High variance. Getting stuff out from production systems can be a pain. And it varies how well structured our data is. However, it can also mean very consistent data — fewer or no changes mean to data schemas. And data from production systems is a truth in it’s own — this is what the business actually used at that time.</li><li><strong><em>Stored and used for visualizations or analytics</em>.</strong> Good. When someone is visualizing the data, they do typically get a good understanding of that data set. However, sometimes analysts cannot make any improvements to how the raw data is produced. So that any underlying data quality issues are fixed very late in the pipeline. At this point, we should investigate how issues with data quality is handled? Is it fixed in preprocessing steps in a dashboard or fixed at the source? The latter is clearly preferable.</li><li><strong><em>Curated by analysts</em>.</strong> Better. Hopefully this means that at the end of curated pipelines, data have a meaningful interpretation in business terms that is also adaptable for more analysis.</li></ul><h3><strong>Organizational questions:</strong></h3><ul><li><strong>Can we create or have support from stakeholders? </strong>It’s not just about you, but having the resources and people available to explore it.</li><li><strong>Do we have access to to right competencies? </strong>If you want to do image recognition and the existing data science team only have experience with tabular problems — making that adjustment will require extra effort.</li><li><strong>Can we get relevant feedback and discussion about the problem you are trying to solve with data science? </strong>When doing data science in complex environments, there is a lot going on! Most data science is about complex environments, especially once there are humans involved. Have you considered all the relevant factors? Having good feedback from people with domain knowledge can help you guide towards what your data science tools need to look at. Or why your model suddenly creates unexpected results?</li><li><strong>Can we create a good balance between agility and perseverance. </strong>This one is tricky, because it’s very much a balancing act. If you need to show value too fast, there is no room for real experiments. But your initial hypothesis is also likely to be wrong — the truly great results are from repeated iterations that you learn from. Getting too locked in on the initial hypothesis can be very bad as well. Do you have time to explore while also having the option to make adjustments as you learn more?</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2405c66e842" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Arguments for using the ICE (Impact, Confidence, Ease) as your planning tool in data science teams]]></title>
            <link>https://medium.com/@sasjurse/arguments-for-using-the-ice-impact-confidence-ease-as-your-planning-tool-in-data-science-teams-8bcf423e0df6?source=rss-3286966ddae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/8bcf423e0df6</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[Steffen Sjursen]]></dc:creator>
            <pubDate>Fri, 24 Jan 2020 10:31:46 GMT</pubDate>
            <atom:updated>2020-01-24T10:31:46.262Z</atom:updated>
            <content:encoded><![CDATA[<p>ICE is a framework for selecting feature work based on 3 key dimensions, enabling teams to make better decisions on what features work to prioritise:</p><ul><li><strong>Impact</strong>: How much will this change our business (if it works)</li><li><strong>Confidence</strong>: How confident are we that this will work?</li><li><strong>Ease</strong>: How much work is it to implement?</li></ul><p>Rate each feature on scale from 1–10, multiply to get the score</p><h4>ICE = (Impact) * (Confidence) * (Ease)</h4><p>The feature with the highest score wins. For more information, see for example<a href="https://medium.com/@nimay/inside-product-introduction-to-feature-priority-using-ice-impact-confidence-ease-and-gist-5180434e5b15"> here</a> or <a href="https://productcraft.com/best-practices/ice-scoring-feature-prioritization/">here</a>.</p><blockquote>This scheme might seem simple, but that is also one of the upsides. Estimation is really hard, so trying to be more detailed might not add any value. Also, for simple schemes it is a lot easier to update once you get more information and that should be happening a lot in your data science team.</blockquote><p><strong>Pros:</strong></p><ul><li>Easy to update, so you use it more. Frequent updates give better estimation.</li><li>Capturing confidence is often overlooked, and is especially important for data science work where most outcomes have massive uncertainty.</li></ul><p><strong>Cons:</strong></p><ul><li>It can be tricky to handle dependencies between ideas/features. Some of the value of data science work can be enabling other features to be built. Example: “Can we use clustering techniques on click data to identify similar items in our ecommerce store”, where the output could be a recommendation algorithm. But you also want to use the result as a feature for your pricing optimization scheme! What is the combined value?</li><li>Some organisations might have a strong preference for business cases with a fixed $ amount attached or other prioritization schemes. In an organisation where resources are highly contested, ICE will not work as a tool if your project needs to fight for those resources. Using ICE requires a strong mandate with clear ownership and an agile mindset.</li></ul><h3>Why is ICE especially suited for data science?</h3><p>Data science work has a lot of continuous learning and uncertainty. Often you are trying to build a model using some data — but you don’t really know how much predictive power you can really get out of that data set. Being able to adapt quickly to new information is especially important.</p><p>Examples:</p><ul><li>“Your data set had a lot of strange observations that indicate data quality issues — maybe this is going to be harder than we thought. Time to adapt and maybe see if something else can create more value.”</li><li>“We worked a lot on that data set recently, so this should be feasible. Adjust immediately.”</li></ul><p>Since ICE is easy to work with, changing your estimations/bets on what is the most valuable next feature is easy. Also, the value of continuous dialogue and stimulating the team to act on short feedback loops keeps everyone looped in on what you want to achieve long term which may be easily forgotten amidst tasks / work in progress.</p><p>We don’t use it to prioritize on small tasks on our Kanban board, but for larger features on the size of epics. Below is a fictional example of a simple setup in Google Sheets for prioritizing ideas for improving credit scoring for a analytics department in a bank:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m8ZRWLmFytxkFLo8IN3s6A.png" /></figure><blockquote>For me, I like ICE because I actually use it. We can take a 30 minute overview of existing feature ideas and get a structured analysis. Comparing features is easy, and Impact, Confidence and Easy covers the most important dimensions to make an informed decision.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bcf423e0df6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Kanban works well for our data science teams]]></title>
            <link>https://medium.com/@sasjurse/why-kanban-works-well-for-our-data-science-teams-ffd74c87fc07?source=rss-3286966ddae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffd74c87fc07</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[kanban]]></category>
            <dc:creator><![CDATA[Steffen Sjursen]]></dc:creator>
            <pubDate>Thu, 16 Jan 2020 13:57:49 GMT</pubDate>
            <atom:updated>2020-01-16T13:57:49.815Z</atom:updated>
            <content:encoded><![CDATA[<p>Summary: Because most data science tasks have uncertain outcomes, it is easy to create a lot of work in progress that doesn’t go anywhere. Using Kanban enables us to keep track of that work in progress. We also find it to be a very lightweight tool for project management, with minimum ceremonies for our estimation work.</p><p>We are assuming you are already looking at an agile process methodology. As our main background is consulting, we are often working on early phase and immature problems where the level of confidence is naturally low, and with many, open solution spaces explore.</p><p><strong>Some examples of work we might want to explore:</strong></p><ul><li>Can we use clustering techniques on click data to identify similar items in our ecommerce store?</li><li>Why was conversion rates at our ecommerce site so low this week?</li><li>Is the temperature data from this sensor relevant for the quality issues in our factory?</li><li>Why did the distributions of the income feature in our credit scoring model shift dramatically?</li></ul><p><strong>These questions have some issues in common:</strong></p><ul><li>It is really hard to create a good definition of done. We don’t really know what is feasible or not before we try it. Your data might not be useful for the problem you are trying to solve. Or that every iteration of model building simply yields better and better results.</li><li>Since we don’t have a good definition of done, it is easy to create a lot of work in progress. “We might get a better result if we do X”. “Data is inconclusive but maybe some additional transformations or data cleaning could help”. “We have ROC AUC 80% now, but maybe we could achieve ROC AUC 82% by doing XYZ”.</li><li>Finding out how to evaluate the solution is part of the problem. It might not be obvious up front what the evaluation criteria for your model should be.</li><li>Since we don’t have a good definition of done, estimation of the duration and complexity of a task can be really hard. It might not even add meaningful value, as that time could be instead spent on experimenting and getting actionable facts. Also, be careful as estimation work can lead to writing less functional user stories.</li></ul><p>If we try to work in sprints, it can actually be hard to scope out two weeks. <em>Also note the emergence of issues that require urgent attention from the business side, that also are of high business value. </em>Kanban helps us identify work in progress, and leads to fruitful discussions about when to stop.</p><p>Having scoping discussion throughout each user story, and having less work in progress also lead us to another happy benefit —<strong> we deployed to production <em>way</em> more often</strong>. Frequent deployments have several benefits in terms of output, errors and employee stress. An especially relevant source here is the book <em>Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations by Forsgren and Humble, </em>which is one of the few times you see solid empirical analysis of different management techniques, and frequent deployments is one of the key predictors of performance.</p><blockquote><strong>With open ended questions estimating the duration of any task is extra hard, and the work spent on estimation will probably not add any value. It is also more difficult to accept ad hoc work, or rescope user stories.</strong></blockquote><h4>Criterias for our success with Kanban</h4><p>Enablers of a good kanban setup:</p><ul><li>A clear mandate on what the team should achieve. Since prioritization is an ongoing process it can be less transparent for outside stakeholders.</li><li>A tech lead or product manager with a strong data science background. Since we are more permissive about very open ended questions in our user stories, we also need to be more diligent about trying to find simple solutions that provide value early (and send the next iteration on that problem to the backlog).</li><li>Frequent reviews of work in progress so that non-solvable or less valuable problems are identified and scoped down as early as possible. Use explicit WIP-limits as needed.</li><li>A good continuous integration and continuous deployment setup. If deployments to production are costly, getting the benefits of more frequent deployments will be hard. However, the best way to get better at deployments is to do them more often, so Kanban can help your organization improve in that area.</li></ul><p>Possible downsides to be aware of:</p><ul><li>With continuous updates to the Kanban board, the team might find it harder to keep track of the big picture and why something is a priority. Make sure that the team has and feels ownership.</li><li>Increased uncertainty on exactly which features the team will be working on two weeks or more from now can be hard to communicate in certain organizations.</li></ul><h4>Conclusion</h4><p>For us, switching to Kanban had several benefits.</p><ul><li>Higher frequency of deployments.</li><li>Better feedback due to faster deployments.</li><li>Better discussions in the team on what to prioritize.</li><li>Less stress.</li></ul><blockquote><strong>However, the greatest benefit was how it enabled us to quickly adapt to whichever results the data was telling us. Data science became more data driven. We became a lot better at rescoping user stories to deliver value as we learnt more about the problems we were solving.</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffd74c87fc07" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>