<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bill Steimel on Medium]]></title>
        <description><![CDATA[Stories by Bill Steimel on Medium]]></description>
        <link>https://medium.com/@billsteimel?source=rss-e0d3d18cfca2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*1tlo7atpQskkYeHT.</url>
            <title>Stories by Bill Steimel on Medium</title>
            <link>https://medium.com/@billsteimel?source=rss-e0d3d18cfca2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:59:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@billsteimel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The AI Coding Workflow That Works for Me]]></title>
            <link>https://medium.com/@billsteimel/the-ai-coding-workflow-that-worked-for-me-9672d2edf9d1?source=rss-e0d3d18cfca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9672d2edf9d1</guid>
            <dc:creator><![CDATA[Bill Steimel]]></dc:creator>
            <pubDate>Thu, 29 Jan 2026 15:55:59 GMT</pubDate>
            <atom:updated>2026-01-31T04:32:21.102Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k3v-Awhl8oNfsiBe6E6CJg.png" /></figure><p>AI coding tools didn’t really click for me at first.</p><p>Not because the tools aren’t powerful — but because I was using them without a clear way of working.</p><p>The real gains only show up when you stop treating AI as a helper and start treating it like a <strong>team of engineers</strong> you can direct, constrain, and review.</p><p>That paradigm shift only works with deliberate process, upfront planning, and clear standards.</p><p>A small group of engineers who have invested in clear process and repeatable workflows has already crossed this line. Once I invested in a clearer process and more repeatable habits, the difference wasn’t subtle. Things moved with less friction, experimentation felt easier, and I could explore multiple ideas in parallel without feeling overwhelmed.</p><p>Andrej Karpathy captured this shift well in a recent post on X <a href="https://x.com/karpathy/status/2015883857489522876">https://x.com/karpathy/status/2015883857489522876</a>:</p><blockquote><strong><em>“Coding workflow.</em></strong><em> Given the latest lift in LLM coding capability, like many others I rapidly went from about 80% manual+autocomplete coding and 20% agents in November to 80% agent coding and 20% edits+touchups in December. </em><strong><em>I really am mostly programming in English now — telling the LLM what code to write in words</em></strong><em>. It hurts the ego a bit, but the power to operate over software in large “code actions” is too net useful to ignore. This is the biggest change to my coding workflow in ~2 decades, and it happened over a few weeks.”</em></blockquote><p>That resonated strongly with me, because my own workflows have shifted in the same direction.</p><p>Over the last few months, I’ve moved to roughly <strong>80% agent-written code and 20% human refinement and review</strong>.</p><p>Below is the workflow that made this sustainable for me — not because it’s perfect, but because it’s process-driven.</p><h3>Prerequisite: Clear Constraints</h3><p>Before this flow worked consistently for me, I had to be explicit about my own constraints.</p><p>At minimum, this means maintaining a shared instruction file (for example, Claude.md or .github/copilot-instructions.md) that clearly defines:</p><ul><li>coding standards and formatting rules</li><li>architectural patterns and boundaries</li><li>explicit anti-patterns and what not to do</li></ul><p>I primarily use GitHub Copilot, and GitHub has a solid write-up on best practices for custom instructions that’s worth reading.</p><p><a href="https://github.blog/ai-and-ml/github-copilot/5-tips-for-writing-better-custom-instructions-for-copilot/"><em>https://github.blog/ai-and-ml/github-copilot/5-tips-for-writing-better-custom-instructions-for-copilot/</em></a></p><p>These instructions are how I keep agents aligned with my expectations when working in a shared codebase. Think of them as <strong>living documentation</strong> — they must evolve alongside the codebase. Without them, you’ll end up repeatedly re-specifying the same context to your coding agent, which quickly becomes a pain.</p><p>Without shared standards, AI output becomes inconsistent fast. With them, agents become reliable collaborators. The best practices on writing them is a bit of an art as well, and the approach can differ from team to team.</p><h3>The Workflow (Step by Step)</h3><p>This article focuses primarily on how I apply this workflow in Python, but I’ve applied the same principles to TypeScript and Next.js for frontend development.</p><h3>1️⃣ Define the Problem</h3><p>Everything starts here.</p><ul><li>What problem are we solving?</li><li>Who is it for?</li><li>What does success look like?</li></ul><p>If this step isn’t clear, agents will build the wrong thing — very quickly.</p><p>Clarity at the problem-definition stage saves orders of magnitude of time downstream.</p><h3>2️⃣ Design / Plan</h3><p>This is the most important phase — and it always has been.</p><p>Design and planning turned out to matter more than I expected. What’s changed is that <strong>AI amplifies people who are already good at them</strong>. If you can reason clearly, decompose problems well, and write crisp specifications, AI unlocks disproportionate gains.</p><p>Think less about “prompting” and more about <strong>technical leadership</strong>.</p><p>At this stage:</p><ul><li>Write a clear specification (include the problem, goal, and scope)</li><li>Define inputs, outputs, and constraints</li><li>Call out edge cases and failure modes explicitly</li><li>Break the work into small, independently solvable tasks</li><li>Use agent plan or design modes per task and review them before any code is written (ensure it is aligned with your expectation)</li></ul><p>Agents can handle a surprising amount of complexity when constraints are clear. What they struggle with is ambiguity.</p><p>Clear requirements are the single biggest factor in whether an agent succeeds or fails — no different from working with human engineers.</p><h3>3️⃣ Agent Implementation</h3><p>Once the plan is locked:</p><ul><li>Let agents implement <strong>one task at a time</strong></li><li>Avoid fully autonomous runs except for very small, well-defined tasks (this boundary will likely shift as the technology improves).</li><li>Treat all AI-generated code as a <strong>first draft</strong>, not the final answer</li><li>Review changes against the plan, watching for misalignment and code smells</li><li>Actively steer the agent when direction drifts (make manual corrections as necessary)</li><li>Commit early and often so course correction is cheap</li></ul><p>The goal here is controlled execution, not blind automation.</p><h3>4️⃣ Refine &amp; Review</h3><p>Once the first draft is functionally correct, I move into a dedicated refinement step. Custom agents handle the mechanical work here, while humans remain responsible for review and judgment. With clear instructions, this phase becomes fast and consistent — I treat it much like the traditional pre-commit hook in the workflow.</p><p>In practice, I run a refiner agent to:</p><ul><li>Format the code (e.g. <a href="https://github.com/astral-sh/ruff">Ruff</a>, <a href="https://github.com/astral-sh/ty">Ty</a>)</li><li>Refactor and simplify without changing behavior</li><li>Improve readability and maintainability</li></ul><p>This mirrors how tools like Claude structure their own internal code-simplification agents. <a href="https://github.com/anthropics/claude-plugins-official/blob/main/plugins/code-simplifier/agents/code-simplifier.md">code-simplifier.md</a></p><p>The <strong>implementation</strong> and <strong>refinement</strong> steps repeat until the full solution is solid.</p><h3>5️⃣ Finalize (Documentation)</h3><p>This step used to be tedious. It doesn’t have to be.</p><p>Instead of “writing docs,” think about <strong>capturing context while it’s fresh</strong>.</p><p>Use a documentation agent to generate:</p><ul><li><em>Why</em> the change exists (problem context)</li><li><em>What</em> was built (high-level technical overview)</li><li><em>How</em> to use it (only what’s necessary)</li></ul><p>Because documentation is generated directly from the plan and final code, it stays accurate with minimal effort.</p><p>AI turns documentation from a chore into a byproduct of good planning.</p><h3>What Changed for Me</h3><p>The impact of this workflow was immediate:</p><ul><li>🚀 <strong>Less friction</strong> — ideas move from thought to code more smoothly</li><li>🧪 <strong>Lower barrier to trying ideas</strong> — rapid prototyping without heavy upfront effort</li><li>🔀 <strong>Holding more in parallel</strong> — handling frequent context switches with less mental strain than before</li><li>🧠 <strong>Higher-level thinking</strong> — more attention on design and decisions</li></ul><p>My day-to-day feels less like “writing code” and more like <strong>orchestrating a team of agents to solve problems</strong>.</p><h3>Key Learnings</h3><p>A few things became clear very quickly:</p><ul><li><strong>Process beats clever prompts</strong><br>The biggest gains came from structured workflows and planning — not from prompt tricks.</li><li><strong>Clear guidelines matter</strong><br>Being explicit about my own constraints is what turned AI from something interesting into something more reliable.</li><li><strong>Human-in-the-loop still matters</strong><br>Agents move fast, but I stay responsible for decisions, trade-offs, and final calls.</li></ul><p>AI hasn’t replaced anything for me — it’s been most useful when I’m deliberate about my own process.</p><p>For me, <strong>it’s about extending reach</strong>: operating at a higher level, with less friction, and creating space to explore ideas that previously felt too time-constrained.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9672d2edf9d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Clarity in AI Development: Lessons from Hans Rosling’s ‘Factfulness’]]></title>
            <link>https://medium.com/@billsteimel/unlocking-clarity-in-ai-development-lessons-from-hans-roslings-factfulness-6ea5e265e0b3?source=rss-e0d3d18cfca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ea5e265e0b3</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[factfulness]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[leadership]]></category>
            <dc:creator><![CDATA[Bill Steimel]]></dc:creator>
            <pubDate>Sun, 26 Jan 2025 05:15:35 GMT</pubDate>
            <atom:updated>2025-01-27T12:48:16.225Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*jo2u9Avn6N32zNDT26TpGA.jpeg" /></figure><p>In my role as an AI/ML software engineer, I recently encountered profound insights by reading <em>Factfulness</em> by Hans Rosling. Although the book focuses more on understanding the world, it also addresses our susceptibility to instinctual misjudgments and offers strategies for informed decision-making — principles that are directly applicable in the realms of AI development and project leadership. I will share the core principles gained from reading as well as how I feel “factfulness” can be applied within the context of AI software development and project leadership.</p><h3>Core Instincts</h3><h4><strong>1. Gap Instinct</strong></h4><p>The gap instinct leads us to perceive dichotomies such as success vs. failure, rather than seeing the nuanced spectrum between extremes. This often results in oversimplifying complex scenarios by focusing on outliers instead of the majority. In AI development, this could manifest when teams hastily adopt new technologies without considering existing solutions that may already be effective. It might lead individuals to perceive an exaggerated divide between new and old technologies, which may not be the case upon closer examination.</p><p><em>To Mitigate:</em></p><ul><li><strong>Encourage Nuanced Perspectives:</strong> Promote diverse viewpoints and discussions beyond binary thinking in systems design — like balancing traditional rule-based methods with newer technologies LLMs— to acknowledge successful hybrid strategies.</li><li><strong>Facilitate Inclusive Decision-Making Processes: </strong>Encourage team input across the board to capture a wide range of insights, preventing the neglect of effective hybrid solutions.</li></ul><h4><strong>2. Negativity Instinct</strong></h4><p>The negativity instinct is our tendency to focus on negative aspects which can overshadow successes and progress within our projects. By concentrating attention highly on bugs or defective code, we often overlook how each small pull request positively contributes by enhancing or improving the system.</p><p><em>To Mitigate:</em></p><ul><li><strong>Constructive Feedback:</strong> Foster an environment where code reviews aim for improvement with supportive feedback.</li><li><strong>Celebrate Little Wins:</strong> Recognize milestones and successful features or fixes, emphasizing growth over setbacks.</li></ul><h4><strong>3. Straight Line Instinct</strong></h4><p>The straight line instinct leads us to assume progress follows a straight line or is linear in nature, overlooking the iterative processes essential to software development. Genuine advancement often involves cycles of exploration, iteration, and adaptation.</p><p>To Mitigate:</p><ul><li><strong>Foster a Culture of Experimentation:</strong> Value every attempt as part of growth, acknowledging that failures contribute to deeper understanding.</li><li><strong>Embrace Refactoring as Progress:</strong> While it may not be regarded as glamorous, regular refactoring is essential — not only for enhancing performance but also for exploring new solutions throughout the development process. We should embrace its significant role in maintaining system health over the long term whenever possible.</li></ul><h4><strong>4. Fear Instinct</strong></h4><p>The fear instinct is an exaggerated fear about rare events which can stifle innovation and lead to risk aversion. For example, many organizations develop deep familiarity and dependency on existing technologies or processes, despite potential inefficiencies. This reliance often stems from operational comfort and an exaggerated fear of the unknown risks associated with transitioning to newer systems.</p><p><em>To Mitigate:</em></p><ul><li><strong>Promote a Data-Driven Culture:</strong> Encourage decisions based on data analysis rather than fear, providing teams with tools for effective data interpretation. Having substantial pain points documented through data makes it significantly more straightforward to convince stakeholders.</li></ul><h4><strong>5. Size Instinct</strong></h4><p>The size instinct is when we overemphasize metrics such as user counts or financial figures without context which can lead to misleading decision-making. It’s crucial not just to view these numbers in isolation but to consider them within a broader comparative framework for an accurate understanding of their significance.</p><p><em>To Mitigate:</em></p><ul><li><strong>Apply the 80/20 Rule for Prioritization:</strong> Focus on key factors that significantly impact outcomes first.</li><li><strong>Mitigate Misleading Single Numbers with Comparative Analysis:</strong> Encourage comprehensive data analysis over isolated numbers to uncover true implications.</li></ul><h4><strong>6. Generalization Instinct</strong></h4><p>The generalization instinct is our tendency to oversimplify complex situations by making broad generalizations, like assuming all users have identical needs. For instance, while analyzing user data, teams might conclude that most follow a certain pattern, only to discover smaller subgroups upon further investigation — where “majority” often means just over 50%.</p><p><em>To Mitigate:</em></p><ul><li><strong>Encourage Detailed Analysis in Reporting:</strong> Advocate for thorough exploration of data to understand nuances and specific user requirements.</li></ul><h4><strong>7. Destiny Instinct</strong></h4><p>The destinty instinct is a focus solely on inevitable outcomes which can overshadow incremental progress, which is vital for sustained success in software development. While major milestones often capture attention, it’s the consistent 1% improvements each day that accumulate significantly over time — a principle echoed in “Atomic Habits” as the power of compounding small changes.</p><p><em>To Mitigate:</em></p><ul><li><strong>Recognize the Value of Gradual Change:</strong> Emphasize continuous refinement rather than waiting for perfection; acknowledge every small improvement.</li></ul><h4><strong>8. Single Perspective Instinct</strong></h4><p>The single perspective instinct is our bias to rely solely on singular or familiar solutions which can limit our ability to reach optimal outcomes. As developers, we often favor familiar tools without exploring alternatives. To address this limitation, it’s important to build a diverse toolbox and remain open to solutions from various fields and others own unique experiences to innovate.</p><p><em>To Mitigate:</em></p><ul><li><strong>Promote Inclusive Dialogue:</strong> Foster an environment where all voices are heard and multiple viewpoints are considered.</li><li><strong>Challenge Existing Assumptions:</strong> Regularly question assumptions with “why” questions during problem-solving sessions to reveal biases or overlooked factors. Prevent resistance to change by encouraging flexibility and openness to new methods in development processes.</li></ul><h4><strong>9. Blame Instinct</strong></h4><p>The blame instinct is the tendency to focus on individual mistakes instead of systemic problems when an issue arises. This often can lead to mistrust and demoralization within teams. Even if a developer makes changes that result in an incident — which often happens — it doesn’t mean we should assign blame. Instead, by examining the broader system, we can enhance effectiveness and develop strategies to prevent similar issues from occurring.</p><p><em>To Mitigate:</em></p><ul><li><strong>Focus on Systemic Issues: </strong>Guide discussions toward identifying process gaps rather than pinpointing individuals.</li><li><strong>Celebrate Learning Opportunities:</strong> Treat mistakes as valuable learning opportunities that spur innovation to prevent future challenges.</li></ul><h4><strong>10. Urgency Instinct</strong></h4><p>Under pressure, there’s often a bias toward quick reactions, which can lead to rushed and suboptimal outcomes in software projects. For instance, when faced with bugs, developers might hurriedly attempt fixes without fully understanding the issue — though we recognize time constraints are sometimes unavoidable. It’s crucial, however, to pause and think through solutions by leveraging our experience and process.</p><p><em>To Mitigate:</em></p><ul><li><strong>Pause and Assess:</strong> Advocate for thorough assessments before urgent actions to ensure decisions are data-driven.</li><li><strong>Set Clear Operating Procedures: </strong>Establish guidelines to maintain consistency during crises, preventing rash decisions that could compromise quality. Be prepared for a rainy day.</li></ul><h3>Conclusion</h3><p>In reflecting on my insights from <em>Factfulness</em> by Hans Rosling, I’ve identified several of my own key strategies from a software perspective that can significantly enhance decision-making in AI development teams:</p><p><em>For those interested in enhancing their understanding of cognitive biases and improving decision-making processes,</em> I highly recommend reading <em>Factfulness</em>. It provides a compelling framework for viewing problems through a more informed lens — whether you’re leading an AI project or navigating any complex domain. Embracing these principles can empower us to build systems that are not only advanced but also fair, inclusive, and aligned with our broader human values.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ea5e265e0b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Signate] JPX Fundamentals Analysis Challenge]]></title>
            <link>https://medium.com/game-of-data/signate-jpx-fundamentals-analysis-challenge-c6023fe7977c?source=rss-e0d3d18cfca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6023fe7977c</guid>
            <category><![CDATA[stock-trading]]></category>
            <category><![CDATA[competition]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[signate]]></category>
            <dc:creator><![CDATA[Bill Steimel]]></dc:creator>
            <pubDate>Sat, 19 Jun 2021 05:28:43 GMT</pubDate>
            <atom:updated>2021-07-10T05:18:14.104Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mN5sEoffJsOa0R1Idxnxcw.png" /><figcaption>Screenshot of competition page.</figcaption></figure><h3>Competition Summary</h3><p>The Competition we participated was the JPX Fundamentals Analysis Challenge on the Signate platform. In case you are not familiar, Signate is a very popular platform for Data Science competitions in Japan which can be helpful for any one wishing to develop their data science skills further on the tried and tested battlefield on real live data. The goal of this competition was given a specific market day, to estimate the high and low price of a stock in 20 business days. The datasets included Financial Statements data and stock prices data for each listed company on the Tokyo Stock Exchange (TSE). In this article, we will discuss our approach for this competition as well as some key tips for anyone who wants to improve their speed and time to develop solutions in competitions. A link to our solution can be found <a href="https://github.com/dc-aichara/signate-jpx"><strong>here</strong></a>.</p><p>You can additionally find more information about the competition at the two links below. (In Japanese)</p><p><a href="https://signate.jp/competitions/423">日本取引所グループ ファンダメンタルズ分析チャレンジ | SIGNATE - Data Science Competition</a></p><p>An English overview of the competition can also be found below which interested us more in working with stock data.</p><p><a href="https://www.jpx.co.jp/english/corporate/news/news-releases/0010/20210319-01.html">https://www.jpx.co.jp/english/corporate/news/news-releases/0010/20210319-01.html</a></p><h3>The Importance of Pipelines for Rapid Experimentation and Iteration</h3><p>You can think of a pipeline as an approach for automating the typical machine learning workflow. What is the typical machine learning workflow? You may be familiar with some typical steps in the machine learning process like data cleaning/preparation, feature engineering, model training, model deployment, and model serving. <a href="https://medium.com/u/c6c6dcd9f4ff">Dayal Chand Aichara</a> and myself always codify our work into pipelines as having it in this structure always us to collaborate together and iterate faster. You can see an example of the pipeline we used in this competition below which is a typical shell script. Each python line is a step in the pipeline or script we execute which uses a configuration_ID to determine which parameters to use. You should think of a data pipeline like a real pipeline where data inputs/outputs are being passed from one part of the pipeline to the other.</p><pre># Competition Pipeline<br>config_id=$1<br>echo $config_id </pre><pre>python ./competition_code/make_dataset.py --config_id ${config_id}<br>python ./competition_code/create_features.py --config_id ${config_id}<br>python ./competition_code/train_model.py --config_id ${config_id}<br>python ./competition_code/model_serving.py --config_id ${config_id}</pre><p>You may be wondering what is this config_id in the above shell script. We also often create a config.yml file to store parameters of our pipelines so that we can toggle on and off functions of the pipeline we would like to use, for example- Cross Validation. The below configuration was used in our final result for the competition and as you can see there are many different parameters which are consumed by our pipeline. The below config_id is called baseline_model but a developer can develop any number of unique config_ids to use.</p><pre>## Insert Params as you see fit in this file<br>baseline_model:<br>  test_model: private<br>  random_seed: 255<br>  low_memory_mode: False<br>  use_fin_data: False<br>  data_date_limit: &quot;2021-01-01&quot;<br>  train_split_date: &quot;2020-01-01&quot;<br>  test_split_date: &quot;2020-01-01&quot;<br>  drop_data: False<br>  drop_data_train_date: &quot;2016-02-01&quot;<br>  drop_data_test_date: &quot;2020-01-01&quot;<br>  cross_validation: False<br>  lgb_model: True<br>  use_test_as_validation: False<br>  train_with_all_data: True<br>  seed: 227<br>  lgb_params: {<br>      &quot;application&quot;: &quot;fair&quot;,<br>      &quot;num_iterations&quot;: 130,<br>      &quot;learning_rate&quot;: 0.1,<br>      &quot;early_stopping_round&quot;: 10,<br>      &quot;feature_fraction&quot;: 1.0,<br>      &quot;bagging_fraction&quot;: 0.9,<br>      &quot;subsample_freq&quot;: 1,<br>      &quot;min_data_in_leaf&quot;: 1016,<br>      &quot;metric&quot;: &quot;lgb_spearmanr&quot;,<br>      &quot;num_leaves&quot;: 1016,<br>      &quot;reg_alpha&quot;: 0.3899,<br>      &quot;reg_lambda&quot;: 0.648,<br>      &quot;verbose&quot;: -1,<br>      &quot;device_type&quot;: &#39;cpu&#39;<br>    }</pre><p>A key benefit to these pipelines for iteration is that we can create multiple configurations when we want to try different experiments. If interested in taking your pipelines to the next level you can also look into open source software like DVC which will make pipeline building a breeze.</p><p><a href="https://dvc.org/">https://dvc.org/</a></p><p>If you are interested, we even have a <a href="https://github.com/dc-aichara/competition_template"><strong>template</strong></a> for this type of work which can be found below in case you are interested (Still a work in progress).</p><h3>Data</h3><p><strong>Overview</strong></p><p>The competition hosts provided the following datasets:</p><ul><li><strong>Stock price data</strong> : This data includes information about open price, closing price, trading volume, and other stock related factors.</li><li><strong>Stock list data: </strong>This data has information about stock industries and sectors.</li><li><strong>Stock fin data:</strong> Financial Statement data are reports or data about companies that tell about financial performance.</li><li><strong>Stock label data:</strong> Label data has labels for low_5, high_5, low_10, high_10, low_20, and high_20.</li></ul><p><strong>Feature Engineering</strong></p><p>We first experimented without any feature engineering for our baseline model. Our Cross Validation score was around 1.50 with baseline model. We then added technical features like volatility, simple moving average, exponential average, percentage change for interval of 10, 14, 20, 28, 30, 42 days. With these features, Cross Validation score reduced to 0.95 from 1.50. We only used stock price data as other data didn’t help us but you can read more about that in the things that didn’t work section of this article for more details. It is was clear at this point that technical features are very important. We added more technical features like Bollinger Bands (Lower and Upper BB), Relative Strength Index (RSI), Moving Average Convergence Divergence (MACD). These indicators tell us about price momentum, trend, and volatility. Our Cross Validation score was 0.525784 with new features. We used <a href="https://github.com/dc-aichara/PriceIndices"><strong>PrinceIndices</strong></a> python library to calculate price technical indicators which was created by <a href="https://medium.com/u/c6c6dcd9f4ff">Dayal Chand Aichara</a>.</p><h3>Model</h3><p>For Models we tried Ridge Regression, LightGBM, and TabNet models. We used LightGBM as our final model based on Cross Validation analysis results. We trained 2 models one for prediction of low and high stock values.</p><h3>Evaluation</h3><p>The evaluation metric used in this competition was the Spearman’s rank correlation coefficient which was calculated on both the high and low predictions from the previous models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*LNPbkA1swiuTGzFqxSnmQg.png" /></figure><p>d: The difference between two ranks of each observation</p><p>n: number of observations</p><p>Then an integrated or combined score of the rank correlation coefficients are calculated for both the high and low models where:</p><p>phigh: Spearman rank correlation coefficient of Stock High Predictions</p><p>plow: Spearman rank correlation coefficient of Stock Low Predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*Lv5bvkdvQ7dBbCj_2OLFkw.png" /></figure><p>We relied on a custom coded 5-fold Time Series Split Cross Validation on stock dates with sklearn to evaluate the combination of models properly based on the above integrated metric to select our submission.</p><p>An interesting component of this competition was definitely the real time serving aspect as we were all evaluated on stocks in a live market setting for 4 weeks. This competition required significant engineering skills as many needed to make their models not only train but serve in production type settings. I hope for more competitions like this in the future that are more realistic to the current industry problems.</p><h3>Things that didn’t work</h3><p>In this section, we will go over briefly some things we tried to experiment with that did not work so well for us.</p><p><strong>Ridge Regression </strong>— Before building LightGBM models we first tried with a simple Ridge Regression to get a baseline performance. After that we tried LightGBM and found that the LightGBM model performed significantly better on our cross-validation. At first we wanted to build an ensemble of LightGBM and RidgeRegression models but found that the LightGBM performance was so much better that this ensemble may be detrimental to us in the competition. Ridge Regression however was much faster at inference.</p><p><strong>TabNet </strong>— There have recently been a bunch of different Neural Network based models trying to compete with Gradient Boosted Trees models in the tabular domain. The research papers often argue that they can beat algorithms like XGBoost or LightGBM in predictive performance. We decided to give TabNet a try as it seemed to be interesting — you can see the paper and implementation below if you are interested.</p><p>Paper: <a href="https://arxiv.org/abs/1908.07442">https://arxiv.org/abs/1908.07442</a></p><p>Implementation: <a href="https://github.com/dreamquark-ai/tabnet">https://github.com/dreamquark-ai/tabnet</a></p><p>I would say that it was not that TabNet did not work as it seemed to give us quite competitive performance. It was just that the performance was about the same with a bigger cost on inference speed. There was also a degree of risk as we were not so familiar with the framework in comparison to well established libraries like LightGBM. We thought about ensembling this with LightGBM but this would have doubled our inference speed so we opted to not use it. Its possible that it could have boosted our performance to the top spot but who wants a real time stock prediction model that takes forever to predict?</p><p><strong>Financial Statement Data-</strong> Financial Statement data are reports or data about companies that tell you essentially about its financial health and performance. Many may be wondering how or why we did not use such a core dataset or indicators as part of our solution. The key response to that question is simply that based on our Cross-validation results it did not improve our performance significantly enough to warrant the extra cost of using it. It is possible other solutions used it but in our case we saw no benefit of using it and stuck with the basic stock prices data.</p><h3>Why Signate?</h3><p>For those not in Japan, we recommend that if you haven’t tried Signate before, it may be another good alternative to your Kaggle addiction. There are occasionally competitions where the datasets and competition details are in English but usually competitions related to computer vision are typically more accessible to all people regardless of language. In addition, if you are working in Japan as a foreign resident, Signate gives you the opportunity to show others that you are able to work with Japanese datasets which employers may find to be an asset. In fact, one of the key reasons I have been participating in Signate Competitions is to improve my skills in working with Japanese datasets which can have some slight differences to tabular datasets you may be familiar with in English. One negative point about Signate is that the prize money is typically lower than what you would see on Kaggle but I am hoping in the future as demand for AI specialists in Japan increases that the reward pools will also increase. Also a second negative is, you may find that rules are not often written in English which can be challenging if you are not confident with the language.</p><h3>Conclusion</h3><p>In this article, we discussed our solution in simple terms for predicting high and low values of stock with LightGBM models. From this competition, we can recommend that new data scientists should always translate their code into pipelines for faster experimentation and always trust cross-validation over any public leaderboard score. It is key to note that agility in competitions with the use of Machine Learning pipelines and time to prototype much like the real world will provide an advantage against others in competitive settings. We hope you find this advice helpful and look forward to others putting it to good use against us in the competition battlefield.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6023fe7977c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/game-of-data/signate-jpx-fundamentals-analysis-challenge-c6023fe7977c">[Signate] JPX Fundamentals Analysis Challenge</a> was originally published in <a href="https://medium.com/game-of-data">Game of Data</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>