<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Luka Brkljacic on Medium]]></title>
        <description><![CDATA[Stories by Luka Brkljacic on Medium]]></description>
        <link>https://medium.com/@luka_brkljacic?source=rss-a36222816d6e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Luka Brkljacic on Medium</title>
            <link>https://medium.com/@luka_brkljacic?source=rss-a36222816d6e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 06:57:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@luka_brkljacic/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[We Built a Tool That Turns Any Paper Into Working Code. It’s Now Open Source (Research2Code)]]></title>
            <link>https://medium.com/99p-labs/we-built-a-tool-that-turns-any-paper-into-working-code-its-now-open-source-research2code-51ca93ea5e44?source=rss-a36222816d6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/51ca93ea5e44</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[research]]></category>
            <dc:creator><![CDATA[Luka Brkljacic]]></dc:creator>
            <pubDate>Fri, 25 Sep 2026 14:59:46 GMT</pubDate>
            <atom:updated>2026-09-25T14:59:46.792Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GvvMSCQsBfWuvkjyrJjdvA.png" /></figure><h3>Table of Contents</h3><ol><li><a href="#83cc">Introduction</a></li><li><a href="#c6c8">Why We Built R2C</a></li><li><a href="#ad24">What R2C Delivers</a></li><li><a href="#edff">How a Run Works</a></li><li><a href="#3d50">Repo Overview</a></li><li><a href="#06b2">Getting Started</a></li><li><a href="#9174">What We’ve Learned So Far</a></li><li><a href="#9e9b">What’s Next</a></li><li><a href="#d9a4">Get Involved</a></li><li><a href="#f41e">Acknowledgements</a></li></ol><h3>Introduction</h3><p>Anyone who has tried to use a machine learning paper knows the gap between reading it and running it. The paper describes a method in equations and prose. Turning that into code that imports, trains, and behaves the way the paper says takes days, and at the end you are still not sure whether what you built is what the authors meant.</p><p>At 99P Labs we have been working on a tool that closes that gap. It is called R2C (Research2Code) and you can access it here: <a href="https://github.com/honda-research-institute/research2code">https://github.com/honda-research-institute/research2code</a>. You give it a research paper, as a PDF or markdown, and it delivers working, explained, importable code on your laptop, along with an honest report about how much of the result was actually verified.</p><p>Today we are open-sourcing the <a href="https://github.com/honda-research-institute/research2code">R2C repository</a>. As with <a href="https://medium.com/99p-labs/99p-labs-f1tenth-project-somethings-goes-open-source-514d5cd8b9e5">our F1TENTH project</a>, we believe these problems are best worked on in the open. By sharing R2C we hope to:</p><ul><li><strong>Make papers usable faster:</strong> Give researchers, students, and engineers a starting point that runs, instead of a PDF and a blank editor.</li><li><strong>Give back to the community:</strong> R2C is built on open tools (opencode, Marker, Jupyter, PyTorch) and on the open research it reads.</li><li><strong>Learn from other people’s papers:</strong> The pipeline was built and tuned on a handful of papers and one model. More papers, more models, and more eyes will show us where it breaks.</li></ul><p>The rest of this post explains what R2C does, how it works, how to get started, and where we want to take it.</p><h3>Why We Built R2C</h3><p>Reimplementing a paper is mechanical work with a few genuinely hard judgment calls buried inside it. Most of the time goes to the mechanical part: reading the method section closely enough to write a spec, scaffolding a package, writing a training loop, building a small demo so you can see it do something, and checking that the pieces fit together.</p><p>The judgment calls are where things go wrong. Which constant did the authors mean? Does this equation apply per sample or per batch? Is the optimizer the one named in the paper, or the one that made the experiments work? An LLM can do the mechanical work well, but it also makes its judgment calls silently, and will happily tell you everything is fine.</p><p>R2C is our attempt at the split we keep coming back to in our other AI work: let agents do the scaffolding, and make every judgment call visible so a human can check it. The system never edits code to make a check pass. It reports what it did, what it assumed, and what it could not verify, and then it stops.</p><h3>What R2C Delivers</h3><p>Every run produces a folder under r2c_runs/&lt;paper&gt;/ with the same shape:</p><ul><li><strong>REPORT.md</strong>: The entry point for a completed run. Contains the delivery label (how much was verified), every finding in plain language, and a claims table with a verdict for each claim the paper makes.</li><li><strong>METHOD.md</strong>: The paper&#39;s method explained equation by equation. This gets produced even when the later stages fail so that even if the user doesn’t get code they can at least get an explanation.</li><li><strong>notebook.ipynb</strong>: A tutorial that runs the method end to end on a small demo-scale dataset, and builds and explains the method from scratch.</li><li><strong>method/</strong>: The importable Python package (from method import ...) with its own README and tests.</li></ul><p>Supporting evidence lives in details/: the assumptions the system made (each with a line telling you how to override it), known issues, review findings left for your judgment, and a machine-readable manifest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*8JJwy6WNGxAgZ05GeCgSkg.png" /></figure><h3>How a Run Works</h3><p>The pipeline is paper-agnostic, so the same driver runs every paper without code changes. Knowledge about methods and tasks lives in a taxonomy (a single YAML file) plus specs derived from the paper itself. A Python driver sequences the stages and dispatches LLM agents through opencode’s HTTP API. Every judgment call lives in an agent, whereas all sequencing, validation, retry, and routing is deterministic driver code.</p><p>At a high level, a run has five phases:</p><ol><li><strong>Understand the paper.</strong> Parse it, map its sections and claims, and extract the method into a structured spec.</li><li><strong>Explain the method.</strong> Write METHOD.md.</li><li><strong>Write the code.</strong> Generate the model, training, and method code, then validate that the package imports and satisfies its declared architecture contract.</li><li><strong>Build and smoke-run the notebook.</strong> Generate the tutorial, then execute it end to end. This is usually the longest phase.</li><li><strong>Verify and report.</strong> Run behavioral probes against the package, route review findings, and write REPORT.md with the delivery label.</li></ol><p>Each phase is gated by three checks:</p><ol><li>Does the output exist?</li><li>Is it mechanically sound?</li><li>Does a reviewer agent agree it says what it should?</li></ol><p>When a check cannot be satisfied, the run either continues with the issue logged or stops on purpose. Halts are most common in phases 3 and 4, where the generated code first meets reality.</p><p>A run takes roughly 20 minutes to a few hours depending on the paper and the model. It detaches and runs on its own, with a local live view in your browser showing the current stage and a plain-language timeline.</p><h3>Repo Overview</h3><p>The R2C repository contains:</p><ul><li><strong>scripts/:</strong> The pipeline driver, the per-stage scripts, the deterministic validators, the PDF parser, and the live-view server.</li><li><strong>.opencode/:</strong> The LLM producer agents the driver dispatches, the /r2c-run and /r2c-chat slash commands, and the launch plugin.</li><li><strong>docs/ssot/taxonomies.yaml:</strong> The method and task taxonomy, the single source of truth for what the pipeline knows about ML methods.</li><li><strong>schemas/:</strong> Pydantic models, the data contracts every stage reads and writes.</li><li><strong>paradigms/:</strong> File templates the scaffolder uses to lay out a generated package.</li><li><strong>input_papers/:</strong> Two starter papers, each as the original PDF and its parsed markdown.</li><li><strong>example_output/:</strong> Complete deliveries of one paper on several models, with a README comparing time, cost, and what changed.</li><li><strong>tests/:</strong> The pytest suite and regression fixtures. No GPU, model endpoint, or server needed to run it.</li></ul><h3>Getting Started</h3><p>Detailed instructions are in the repo’s main README. The short version:</p><ol><li>Install R2C’s own (light) dependencies with pip install -r requirements.txt.</li><li>Copy .env.example to .env and set a model and API key. The default is Google&#39;s gemini-flash-latest, which has a free tier and needs no credit card. OpenAI and Anthropic work the same way.</li><li>Start the R2C environment with ./r2c-start.sh 4096. This brings up an opencode server, the TUI, and the live view in your browser.</li><li>In the opencode window, run one of the starter papers: /r2c-run bayesian-active-learning.</li><li>When it finishes, open r2c_runs/bayesian-active-learning/REPORT.md, then run the notebook.</li></ol><p>To run your own paper, drop it into input_papers/ and run /r2c-run &lt;its-filename&gt;. Markdown files runs directly, whereas PDFs are converted locally using Marker.</p><p>Once a run finishes, /r2c-chat &lt;slug&gt; opens a conversation grounded in that run&#39;s own artifacts. You can ask why a run stopped, whether a piece of code does what the paper says, or request a change. Every change is re-checked, recorded, and committed to the run folder&#39;s own git history, so &quot;undo that&quot; always works.</p><h3>What We’ve Learned So Far</h3><p><strong>You do not need a frontier model, you need a good harness.</strong> R2C was built and tuned on a self-hosted Qwen 27B model, a size class that fits on a single workstation GPU. We expected that to be a serious limitation, but on every paper we ran, that 27B model read the method, wrote an importable package with tests, produced a notebook that trained and improved, and explained the equations one by one. In our example comparison it wrote the most thorough explanation and found the most claims of any run.</p><p>What made that possible was the stack around the model, and that is the real learning from this project. The taxonomy tells the agents what kind of method they are looking at. Every stage has a spec to fill and a validator that rejects malformed output before it moves downstream. Retries, routing, and gating are deterministic driver code. The behavioral probes check what the code does rather than what the model says it does. A small model inside that structure makes small mistakes that get caught. The same model prompted freely would make the same mistakes and nobody would notice.</p><p>To see the tradeoff, example_output/ holds complete deliveries of the same paper (Ash et al., &quot;Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds&quot;) on different models, untouched by hand. Every run implemented BADGE faithfully. The hosted OpenAI models were terser than the reference run and just as correct on the algorithm. The cheapest working run cost under a dollar and took twenty minutes. The strongest hosted run reached verified for about five dollars in the same time. The self-hosted reference took four hours, but it wrote the most thorough explanation. Spending 60 percent more on a hosted model&#39;s reasoning bought a longer explanation and a worse label. The model tier mattered more than the effort knob, and the harness mattered more than either.</p><p>We think this is where the industry is heading. Small teams with modest models and a well designed pipeline are producing output that a year ago would have needed the largest model available. Four runs of one paper is a small sample, so we suggest treating it as a frame of reference and not an authoritative benchmark.</p><p><strong>Judgment calls belong in agents, everything else belongs in code.</strong> Sequencing, validation, retry, and routing are deterministic. This made the pipeline debuggable in a way a chain of prompts never was.</p><p><strong>Halt and report beats auto-fix.</strong> Early versions tried to patch generated output when a check failed. That produced layers of band-aids and outputs nobody could trust. Stopping on purpose and writing down why turned out to be the more useful behavior.</p><p><strong>Fixes have to be paper-agnostic.</strong> Every bug shows up on one paper first. The rule that saved us was to treat that paper as the reproducer, never the target, and fix the failure class where all papers route through.</p><p><strong>Presenting uncertainty honestly is the hard part.</strong> A label is easy to print and easy to over-read. Getting the labels, the claims table, and the assumptions file to say exactly as much as the evidence supports, and no more, took longer than the code generation did.</p><h3>What’s Next</h3><ul><li><strong>Broader taxonomy coverage.</strong> The behavioral probes that earn a verified label exist for a handful of method families. New families land as &quot;uncertified, new territory&quot; until someone writes the checks.</li><li><strong>More models, more papers.</strong> The example comparison is four runs of one paper. We want to see how the pipeline behaves across domains and providers we did not tune for.</li><li><strong>Closing the gaps the comparison exposed.</strong> The example runs surfaced a few pipeline follow-ups (a return-type check at the coding stage, earlier routing for a reviewer check, a smarter notebook probe). Those are the first issues in the tracker.</li></ul><h3>Get Involved</h3><p>We’d love to hear from you.</p><ul><li><strong>Technical questions:</strong> Reach us at <a href="/service/mailto:research@99plabs.com">research@99plabs.com</a> or visit <a href="https://www.99plabs.com/">our website</a>.</li><li><strong>Contribute:</strong> See CONTRIBUTING.md in the repo (<a href="https://github.com/honda-research-institute/research2code">https://github.com/honda-research-institute/research2code</a>) for setup, the test gates, and the ground rules. Open an issue with the halt artifact if a run stops on you twice the same way.</li><li><strong>Share your runs:</strong> If you run R2C on a paper we have not seen, we want to know what label it got and why.</li></ul><h3>Acknowledgements</h3><p>R2C stands on a lot of open work. In particular, we greatly appreciate the the people behind these projects:</p><ul><li><a href="https://opencode.ai/">opencode</a>, the agent runtime R2C dispatches through</li><li><a href="https://github.com/datalab-to/marker">Marker</a>, for local PDF parsing</li></ul><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51ca93ea5e44" width="1" height="1" alt=""><hr><p><a href="https://medium.com/99p-labs/we-built-a-tool-that-turns-any-paper-into-working-code-its-now-open-source-research2code-51ca93ea5e44">We Built a Tool That Turns Any Paper Into Working Code. It’s Now Open Source (Research2Code)</a> was originally published in <a href="https://medium.com/99p-labs">99P Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rethinking Innovation with AI Agents: How 99P Labs Is Exploring Ideation and Decision Intelligence]]></title>
            <link>https://medium.com/99p-labs/rethinking-innovation-with-ai-agents-how-99p-labs-is-exploring-ideation-and-decision-intelligence-1158aa5c938e?source=rss-a36222816d6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1158aa5c938e</guid>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Luka Brkljacic]]></dc:creator>
            <pubDate>Mon, 06 Apr 2026 14:42:24 GMT</pubDate>
            <atom:updated>2026-04-06T14:42:24.266Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0QPkL0IfzDCB5XJzFNyESw.png" /></figure><h3>Table of Contents</h3><ol><li>The Innovation Frameworks Problem</li><li>A Pipeline for Structured Innovation</li><li>Structuring the Problem</li><li>Generating Ideas</li><li>Filtering for Novelty</li><li>Organizing and Evaluating</li><li>Making It Actionable</li><li>What We’ve Learned So Far</li></ol><h3>The Innovation Frameworks Problem</h3><p>Everyone has been in a brainstorming session that starts with energy and ends with a whiteboard full of ideas that all sound vaguely the same. Someone suggests ride-sharing but for groceries. Someone else suggests an app that connects neighbors. A third person suggests a platform that does something similar to what already exists but with a twist. By the end, the team has thirty sticky notes and a familiar feeling: we’ve been here before.</p><p>This isn’t because the people in the room aren’t creative, it’s just that unstructured brainstorming has a well-documented tendency to converge on the obvious. The first ideas anchor the conversation and social dynamics favor ideas that sound reasonable over ideas that sound strange. Without a systematic way to track what’s already been said, the group circles the same territory without realizing it.</p><p>Innovation practitioners have thought about these problems for decades, and good frameworks exist to address them. There are frameworks for structuring vague questions into something specific enough to act on. Frameworks for grounding ideas in market reality. Frameworks for decomposing a big problem into the specific customer needs it contains. Frameworks for evaluating options through structured comparison rather than gut feel.</p><p>These frameworks work. The problem is that they’re labor-intensive. A proper Jobs to Be Done decomposition can take a team days. Mapping and tracking assumptions across a strategic initiative requires cross-functional workshops and sustained discipline. Running a structured ideation process that genuinely avoids repetition takes deliberate scaffolding that most teams don’t have time to build. So the frameworks sit in books and consulting decks, widely acknowledged as valuable but rarely practiced in day-to-day work.</p><p>At 99P Labs, we’ve been exploring a different angle: what if AI agents could handle the mechanical parts of these frameworks while humans stay in the loop for judgment? Not replacing the thinking, but providing the infrastructure that makes proven innovation frameworks practical to actually use.</p><p>This post walks through the frameworks we draw from, why each one matters, and how we’re thinking about putting them together into a cohesive pipeline. To make it concrete, we’ve built an interactive prototype that demonstrates what this could look like in practice for a realistic strategic question: <em>should a company enter the EV charging infrastructure market in Southeast Asia?</em></p><h3>A Pipeline for Structured Innovation</h3><p>When we looked at the innovation frameworks we wanted to draw from, a natural sequence emerged. Each framework addresses a specific limitation of unstructured ideation or decision-making, and they chain together in a way where each stage’s output becomes the next stage’s input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*VyL6F_fsWXEjeF2KaNFQLw.png" /></figure><p>The pipeline has eight stages, organized into four phases:</p><ul><li><strong>Structuring the problem</strong>: Prompt Extraction, Market Context, and Jobs to Be Done Extraction turn a vague question into something specific, grounded, and decomposed into actionable pieces.</li><li><strong>Generating ideas</strong>: Idea Generation produces options in adaptive batches, with each round informed by what came before.</li><li><strong>Filtering for novelty</strong>: Novelty Detection catches duplicates that look different on the surface but mean the same thing, and provides a signal for when the idea space is getting exhausted.</li><li><strong>Organizing and evaluating</strong>: Clustering groups ideas by meaning rather than labels, and Tournament/Debates rank them through structured comparison rather than arbitrary scoring.</li><li><strong>Making it actionable</strong>: Reporting and Storage ensure that the output is usable and that knowledge accumulates across sessions rather than disappearing.</li></ul><p>We’ll walk through each stage below: what the framework is, why it matters, and what it looks like in practice.</p><h3>Structuring the Problem</h3><p>The first phase is about taking a vague starting point and turning it into something structured enough for the rest of the pipeline to work with. This is the part that people most often skip when they sit down to brainstorm or when they prompt an LLM, and it’s arguably where the most value gets left on the table.</p><h4>Prompt Extraction</h4><p><strong>The framework:</strong> Take a free-text question and extract the structured dimensions that are hiding inside it.</p><p><strong>Why it matters:</strong> A question like “mobility solutions for older adults in an urban setting” sounds specific, but it’s actually underspecified in a dozen ways. Who exactly is the target audience? What’s the budget range? What does success look like? What domain constraints exist? When generation starts without answering these questions, the LLM (or the brainstorming team) fills in the blanks with its own assumptions, and those assumptions are usually generic.</p><p>Prompt extraction surfaces these implicit dimensions and makes them explicit. The result is a structured brief that includes fields like target audience, domain, budget range, timeframe, success metrics, and geographic scope. Each field gets a suggested value based on the original question, and the user can accept, modify, or override any of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*OPbwx4reh1z2m2SallmTyA.png" /></figure><p>In the context of a strategic decision, this same principle applies. Our prototype starts with a setup step where the user defines not just the strategic question (“Should we enter the EV charging market in Southeast Asia?”) but also the decision owner, time horizon, linked business objectives, and decision type. This structures the problem before any analysis begins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*Lq49HWUshbJsvDa7uMNiDw.png" /></figure><h4>Market Context</h4><p><strong>The framework:</strong> Generate a structured, multi-dimensional situational analysis before producing any ideas or strategies.</p><p><strong>Why it matters:</strong> Ideas generated in a vacuum tend to be generic. When the system (or the team) knows who the key players are, where the gaps in the market sit, what the regulatory landscape looks like, and what adjacent trends are shaping the space, the output becomes dramatically more specific and grounded.</p><p>In innovation work, this usually takes the form of a market brief that covers the domain overview, competitive landscape, known gaps, and recent trends. In strategic decision-making, the same principle extends further to include internal capabilities, global macro factors, consumer readiness, and legal considerations.</p><p>Our prototype illustrates this with five context dimensions, each summarized with key signals and supporting evidence. The market tab, for example, shows that the SE Asia EV charging market is $1.8B and growing at 28% CAGR, with no player above 15% share, which means the market is fragmented and a first-mover window is still open. That kind of grounding changes the character of every option that gets generated downstream.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*XN0Kap30GQw24OaHcCMq-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*Baehjnu0ToL8zJ0WqR0Oug.png" /></figure><h4>Jobs to Be Done</h4><p><strong>The framework:</strong> Decompose the problem into the specific customer needs it contains, including both functional needs (what someone is trying to accomplish) and emotional needs (how they want to feel).</p><p><strong>Why it matters:</strong> Most ideation organizes ideas by category: “here are our transportation ideas, here are our technology ideas, here are our policy ideas.” Categories tell you <em>what</em> an idea is, but not <em>why</em> it matters. Two ideas in completely different categories might serve the same customer need, while a critical need might have no ideas addressing it at all.</p><p>Jobs to Be Done (JTBD) provides a second axis for analysis. You might discover that 40 mobility ideas all serve the same functional job (“help me get places faster”) while a critical emotional job like “let me feel independent” has zero ideas addressing it. That’s the kind of gap you’d never find by looking at categories alone.</p><p>In the prototype, the JTBD decomposition feeds directly into how options are generated and evaluated. Each sub-job becomes a lens for asking “do we have good coverage here, or is this an underexplored area?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*fKfNR95BkdehdLCihHibpg.png" /></figure><h3>Generating Ideas</h3><h4>Idea Generation</h4><p><strong>The framework:</strong> Produce ideas in adaptive batches where each round’s output shapes the next round’s input, rather than generating everything from a single static prompt.</p><p><strong>Why it matters:</strong> If you ask an LLM the same question multiple times, you get roughly the same answers each time. The model defaults to its most statistically likely outputs, the ideas it’s encountered most frequently in its training data. Without intervention, more prompting just yields more of the same.</p><p>The key insight is that the generation prompt needs to evolve. After every batch, the prompt changes based on what’s already been generated. It incorporates three kinds of information:</p><ul><li><strong>Fixed context</strong> from the structured prompt and market analysis (this stays constant)</li><li><strong>Feedback</strong> from what’s been generated so far: which categories are crowded, which are sparse, which recent ideas to avoid repeating</li><li><strong>Diversity strategies</strong> that rotate on a cycle: one round might focus on filling gaps in underrepresented categories, the next might invert assumptions from recent ideas, the next might borrow mechanisms from unrelated industries, and the next might vary constraints like budget and timeframe</li></ul><p>This mirrors how effective brainstorming naturally progresses. Early rounds cast wide and explore broadly. Later rounds shift to targeted gap-filling based on what’s already been covered. The difference is that the system maintains awareness of everything that’s been generated across hundreds of ideas, something that’s impossible for a human team to do manually.</p><p>The same principle shows up in our decision intelligence prototype, where strategy generation doesn’t just produce one option. It generates multiple structured alternatives (in the prototype, a focused single-market entry, a broader multi-country approach, and a strategic deferral) each with explicit tradeoffs, so the decision-maker has genuine options to compare rather than a single recommendation to accept or reject.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*CKsMKIIz1N_vbEGo81TFDQ.png" /></figure><h3>Filtering for Novelty</h3><h4>Novelty Detection</h4><p><strong>The framework:</strong> Compare each new idea against everything that’s already been generated using semantic similarity, not just surface-level word matching, and flag duplicates in real time.</p><p><strong>Why it matters:</strong> “Autonomous delivery robot” and “self-driving delivery bot” are the same idea in different words. As you generate ideas at scale, this kind of duplication is inevitable, and simple string matching can’t catch it. You need to compare meaning rather than words.</p><p>Embedding-based similarity detection handles this by converting each idea into a vector representation that captures its semantic meaning. That vector is compared against all existing ideas. If a new idea is too similar to something already generated, it’s flagged as a duplicate. Otherwise, it’s kept and assigned a novelty score based on how different it is from everything that came before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*zuedKH5_5_kABCnZQk9gTQ.png" /></figure><p>This also provides a natural signal for when to stop generating. When the rate of genuinely novel ideas starts dropping across batches, the idea space is likely exhausted for the current angle. Rather than continuing to produce diminishing returns, the system can either stop or reallocate effort to areas that are still producing fresh output.</p><p>In the decision intelligence context, a similar principle applies to assumption tracking. Our prototype flags when new information overlaps with or contradicts existing assumptions. When Thailand extends its EV incentives through 2030, for example, the system recognizes that this directly affects an existing assumption (previously at 75% confidence) and updates it automatically rather than treating it as an isolated data point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*YIQbh8fKzz1ldBv8nZIaNw.png" /></figure><h3>Organizing and Evaluating</h3><h4>Clustering</h4><p><strong>The framework:</strong> Group ideas by semantic similarity rather than by their assigned labels, then label the resulting clusters and merge any that overlap.</p><p><strong>Why it matters:</strong> Categories can tell you what an idea is labeled as, but not what it actually means. Two ideas labeled “Transportation” and “Healthcare” might both be fundamentally about reducing isolation for elderly people. Clustering by meaning rather than labels reveals these deeper connections and produces a more honest picture of where ideas actually concentrate.</p><p>The clustering process works in three stages: first, a geometric grouping based on the idea embeddings; then, an LLM pass that assigns a theme label and selects a representative idea for each cluster; and finally, a merge step that catches cases where the initial grouping created clusters that are really the same theme from slightly different angles.</p><p>In the strategic context, our prototype demonstrates a similar organizational principle. The Context Intelligence section organizes information across five dimensions (Internal, Market, Global, Consumer, Legal) and surfaces a scorecard that shows at a glance where conditions are favorable and where they’re mixed or concerning. This gives the decision-maker a structured overview rather than an undifferentiated wall of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*_x3-ET32HrG3JSfmC7xxHg.png" /></figure><h4>Tournament and Debates</h4><p><strong>The framework:</strong> Rank options through structured pairwise comparison and adversarial argumentation, rather than through absolute scoring.</p><p><strong>Why it matters:</strong> The most intuitive approach to evaluation, having someone score each option from 1 to 10, turns out to be surprisingly unreliable. Absolute scores are noisy, don’t force the evaluator to weigh tradeoffs between options, and tend to cluster in a narrow range that makes it hard to differentiate.</p><p>A more effective approach borrows from competitive chess. Instead of asking “how good is this option?”, you ask “which of these two options is better, and why?” Arguments are made for and against each one, and a judgment is rendered with a margin of victory. Over multiple rounds, a ranking emerges that reflects genuine comparative assessment.</p><p>The reason this works better is that forcing structured argumentation from both sides surfaces information that single-pass scoring misses. A scoring rubric might give two ideas similar feasibility scores. A debate might reveal that one depends on an unrealistic supply chain assumption while the other is grounded in existing infrastructure. That distinction only comes out when someone has to argue against each option, not just for it.</p><p>Our prototype applies this principle to scenario modeling. Rather than just scoring each strategy, the system stress-tests the recommended strategy across base, bull, and bear cases, each built on explicit assumptions. It also runs sensitivity analysis to show which variables have the most impact on outcomes. In the prototype, EV adoption rate swings the odds by up to 22 points, making it the single most important factor to monitor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*HZxVdqwN6RswPetQq6QOVg.png" /></figure><h3>Making It Actionable</h3><h4>Reporting and Knowledge</h4><p><strong>The framework:</strong> Make results exportable and actionable, and ensure that knowledge accumulates across sessions rather than resetting to zero each time.</p><p><strong>Why it matters:</strong> Most AI interactions are ephemeral. You have a conversation, copy the useful parts, and the rest disappears. The next session starts from scratch. All the near-misses, the patterns in what the model gravitates toward, the prompts that produced the best results, the categories that saturated early: all of that signal is lost.</p><p>Treating output as an accumulating asset changes this. Every idea, every assumption, every scenario, every evaluation becomes a structured record that feeds future work. The next session knows what territory has been covered, which assumptions have been validated or invalidated, and where gaps remain.</p><p>In our prototype, this principle takes the form of two things: a structured decision record that captures the full context of a strategic choice (the question, the analysis, the assumptions, the alternatives considered, and the rationale for the chosen path), and a living monitoring system that tracks how confidence evolves over time as the world changes around the decision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*tyc7ZCihQ1wz_WMqgkMxjg.png" /></figure><p>The monitoring aspect is especially interesting to us. In the prototype, when Thailand extends its EV incentives, the system automatically updates the relevant assumption and the overall confidence ticks up from 72% to 76%. When a competitor announces a $200M expansion four months later, the confidence drops to 64% and a contingency trigger approaches. Each change is logged with its source and impact, creating a clear audit trail that the decision-maker can review at any time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*1RxaX7PnBSaIJNkvVmF8-Q.png" /></figure><p>The system also pre-defines contingency plans for the top risks, each with a specific trigger and a pre-approved response. This converts what would normally be reactive scrambling into a prepared playbook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*uQGrwUCycpSm8M3Xw0NVkQ.png" /></figure><h3>What We’ve Learned So Far</h3><p>We’re early in this work and still forming our views on what works and what doesn’t. Here’s where our thinking stands today.</p><p><strong>These frameworks translate well to agentic workflows.</strong> JTBD decomposition, assumption mapping, scenario comparison, adversarial debate: these are all structured processes with defined inputs and outputs, which makes them well-suited for AI agents to execute. The mechanical work involved is substantial, and letting agents handle it genuinely makes the frameworks more accessible than they’d otherwise be.</p><p><strong>The feedback loop is where the real value comes from.</strong> In ideation, the biggest gain isn’t from any single technique. It’s from chaining them together into a loop where each step’s output reshapes the next step’s input, so the process gets smarter as it goes rather than starting from scratch each time. We’re seeing early signs that the same principle applies on the decision intelligence side: the value isn’t in any individual analysis, but in the continuous connection between assumptions, scenarios, and real-world signals over time.</p><p><strong>Human judgment is essential at the key moments.</strong> The agent can structure the problem, generate options, and surface tradeoffs. But the human still needs to decide which assumptions are truly blocking, which strategy to pursue, and when a contingency should fire. We think the right model is augmentation, where AI handles the scaffolding and humans handle the calls that require experience and context that the model doesn’t have.</p><p><strong>Presenting uncertainty honestly is harder than it looks.</strong> Confidence scores, probability estimates, and success odds are useful as relative signals and conversation starters, but they can easily create a false sense of precision. A “72% success odds” figure is the output of a model built on assumptions, and communicating that effectively without either overstating or understating its value is something we’re still working through.</p><p><strong>Knowledge accumulation is the long game.</strong> We believe the real value compounds when each session’s output feeds the next: which parts of the idea space have been explored, which assumptions have been validated or invalidated, which strategies were considered and why they were set aside. Making that accumulation practical and reliable is one of the harder problems in this space, and it’s an area we’re actively working on.</p><p>These are experiments, and we’re sharing them in that spirit. We think the direction is promising: established innovation frameworks, made practical through AI agent workflows, with humans in the loop for the decisions that matter. If you’d like to explore the decision intelligence prototype yourself, you can try the interactive click-through here: <a href="https://idea-generation-system.netlify.app/">https://idea-generation-system.netlify.app/</a>. That’s the space we’re exploring at 99P Labs, and we’ll keep sharing what we learn as we go.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1158aa5c938e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/99p-labs/rethinking-innovation-with-ai-agents-how-99p-labs-is-exploring-ideation-and-decision-intelligence-1158aa5c938e">Rethinking Innovation with AI Agents: How 99P Labs Is Exploring Ideation and Decision Intelligence</a> was originally published in <a href="https://medium.com/99p-labs">99P Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[99P Archivist Project]]></title>
            <link>https://medium.com/99p-labs/99p-archivist-project-d557b32fe0da?source=rss-a36222816d6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d557b32fe0da</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Luka Brkljacic]]></dc:creator>
            <pubDate>Mon, 10 Nov 2025 17:58:15 GMT</pubDate>
            <atom:updated>2025-11-11T19:53:29.344Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*oVD07S4NIAHO87E34gLBFA.png" /></figure><h3>Table of Contents</h3><ol><li><a href="#9234"><strong>Project Motivation</strong></a></li><li><a href="#8eb3"><strong>Project Challenges</strong></a></li><li><a href="#9d75"><strong>Project Overview</strong></a><strong><br></strong>a. <a href="#8e9e">Markdown Conversion</a><br>b. <a href="#a43f">Processing &amp; Embedding</a><br>c. <a href="#79ad">Chatbot &amp; Knowledge Graph</a></li><li><a href="#561c"><strong>Technical Deep Dive</strong></a></li><li><a href="#faec"><strong>Conclusion</strong></a></li></ol><h3>Project Motivation</h3><p>The 99P Archivist began as a fork of Neo4J’s <a href="https://github.com/neo4j-labs/llm-graph-builder">LLM-Graph-Builder</a> and is purpose-built to capture the collective knowledge generated across 99P Labs, Honda Research Institute (HRI), and the broader Honda ecosystem. By adapting the original project, we gain a proven technical foundation, while tailoring the experience to 99P’s specific needs. The result is a knowledge platform that surfaces institutional expertise through both a conversational chatbot and a navigable graph, making it easier for anyone in the organization to connect people, projects, and ideas.</p><p>Our teams have produced years of technical papers, design documents, and planning artifacts. Each artifact captures a different slice of organizational memory: what work is underway, who is leading it, and how initiatives interrelate. Unfortunately, those answers often remain buried in disconnected repositories or implicit knowledge held by individual employees. The Archivist addresses this fragmentation by ingesting those documents, disambiguating authorship even when it is only implied, and unifying everything into a single, structured representation.</p><p>Once the “what” and the “who” are linked inside the graph, the Archivist becomes an always-on guide to the organization’s body of work. Employees can interrogate the system through a chatbot for quick answers, or dive into the graph for richer exploration.</p><p><strong>Potential Use-Cases:</strong></p><p><strong><em>Bridging parallel efforts</em></strong></p><p>Large organizations often discover that multiple teams are unknowingly pursuing similar goals. By querying the Archivist, program managers can surface overlaps early, reduce duplicated effort, and encourage collaboration that unlocks cross-functional learning.</p><p><strong><em>Accelerating subject-matter discovery</em></strong></p><p>When an employee is assigned to a new topic, the knowledge base acts as a directory of in-house expertise. A single question to the chatbot can reveal who has explored the domain before, which deliverables they produced, and provide the context needed for productive outreach.</p><p><strong><em>Informing strategic planning</em></strong></p><p>Executives and senior leaders gain an on-demand, data-informed view of the portfolio. Understanding current trajectories and the historical decisions that shaped them makes it easier to prioritize investments, identify capability gaps, and align future initiatives with broader corporate strategy.</p><h3>Project Challenges</h3><p>Although there were many small challenges to overcome and problems to solve throughout this project, two big challenges stand out: how to tame the large amount of files, all of different file types; and how best to expose the results so that users can interact with it and come away with the desired learnings.</p><p>The file-processing puzzle started with dozens of formats, so we narrowed the scope to the types teams touch every day (PowerPoint, Word, Excel, PDFs, images, plain text) then grouped them by how similar their structure and conversion needs were (PowerPoint, PDF, Image, All Others). JSON looked promising at first, but Markdown ultimately fit better because it preserves headings, tables, and narrative flow without overcomplicating downstream parsing. Microsoft’s MarkItDown library became the converter of choice: it handled most files well right out of the box and felt like a safe bet for Microsoft-native documents. The library also worked very well on the All Other category, but wasn’t very good at converting the other file type categories out of the box. After experimenting with the built-in LLM support, however, I landed on a two-step approach: rendering PowerPoint decks and PDFs into image sequences, and then handing those images to MarkItDown with LLM assistance. This finally yielded Markdown outputs that honored slide layouts and visual hierarchy.</p><p>With the content standardized, the next question was how people would explore it. I briefly considered a quick Streamlit interface and even revisiting <a href="https://medium.com/99p-labs/99gpt-building-a-chatbot-fdde8b689df4">a RAG chatbot we had previously built</a>, but neither option felt maintainable or extensible enough for the Archivist’s needs. Neo4j’s LLM-Graph-Builder, however, struck the right balance: its interface already blended table views, graph exploration, and retrieval-augmented chat, and the underlying repo was flexible enough to fork and adapt. By extending that foundation, the project gained a polished UI and a modular backend without starting from scratch.</p><h3>Project Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*236c-uu1SFbH9YohiDYVVA.png" /><figcaption>High-level project overview</figcaption></figure><p>At a high level, the project consists of three main parts. We’ll go through a general overview of each part here, with a more detailed and technical explanation provided in the <a href="#561c">Technical Deep Dive</a> section.</p><ol><li><strong>Convert files to Markdown format<br></strong>- Organized by Project</li><li><strong>Process and Embed all Markdown files<br></strong>- Outputs are stored as Vector Indices and Graph Documents<br>- This makes up the knowledge base</li><li><strong>Users query the data stored in the knowledge base through the Chatbot and/or Knowledge Graph</strong></li></ol><h4>1) Markdown Conversion</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*3ka4tliOCOk4nnKtXN7GYw.png" /><figcaption>High-level flow for the markdown conversion</figcaption></figure><ul><li>This is the preprocessing stage, which goes through all of the files within a project and converts every file to Markdown format.</li><li>The preprocessing pipeline uses <a href="https://github.com/microsoft/markitdown">Microsoft’s MarkItDown library</a> for conversion with the following logic:</li></ul><pre>IF <strong>PowerPoint </strong>THEN <em>convert_to </em><strong>PDF</strong>;<br>ELSE IF <strong>PDF </strong>THEN <em>convert_to </em><strong>PNG</strong>;<br>ELSE <em>convert_to </em><strong>Markdown</strong>;</pre><ul><li>Each project has a manually-created <em>metadata.json</em> file which provides project-level context. This metadata file is sent along with the conversion request, to provide context for a file that may not have any explicit tie to the project on its own (something like an image of a board or a diagram).</li></ul><pre>{<br>  &quot;project_title&quot;: &quot;Haunted Mobility Experience&quot;,<br>  &quot;project_director&quot;: {<br>    &quot;name&quot;: [&quot;Selena Nightshade&quot;],<br>    &quot;organization&quot;: [&quot;99P&quot;]<br>  },<br>  &quot;project_pic&quot;: {<br>    &quot;name&quot;: [&quot;Jack Lantern&quot;, &quot;Holly Grimm&quot;],<br>    &quot;organization&quot;: [&quot;99P&quot;, &quot;Ohio State University&quot;]<br>  },<br>  &quot;project_collaborators&quot;: {<br>    &quot;name&quot;: [&quot;Draven Crow&quot;, &quot;Wendy Frost&quot;],<br>    &quot;organization&quot;: [&quot;99P&quot;, &quot;Frost Robotics&quot;]<br>  },<br>  &quot;project_overview&quot;: &quot;This project prototypes an immersive mobility experience inspired by classic haunted attractions. Participants travel through sensor enhanced routes that blend audio cues, projection effects, and safe physical feedback. The platform studies how themed designs influence comfort, attention, and perception. Findings will inform seasonal mobility experiences and community focused activations.&quot;,<br>  &quot;project_end_user&quot;: &quot;Ohio State University&quot;,<br>  &quot;project_start_date&quot;: &quot;10/01/2025&quot;,<br>  &quot;project_end_date&quot;: &quot;10/31/2025&quot;<br>}</pre><ul><li>As noted above, most file types are processed without conversion to a different intermediate file type or the use of an LLM: Word (.docx), Excel (.xlsx, .xls, .csv), HTML (.html, .htm, .html_string), Text (.txt, .log)</li><li>PowerPoint (.pptx), PDF (.pdf), and Image (.png) file types are processed using the LLM.</li><li>PowerPoint (.pptx) and PDF (.pdf) files go through an intermediate conversion step that converts each slide to an Image (.png). These images are then fed into MarkItDown with the addition of the project metadata and LLM.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*E18OTq4uEvzxw0YKGRsXsQ.png" /></figure><p>Once every file in the project has been converted to markdown format, we are ready to move on to processing and embedding.</p><h4>2) Processing &amp; Embedding</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*lNRVjBRHJXIfnfyPcb-HsQ.png" /><figcaption>Ingestion Process</figcaption></figure><ol><li>Markdown files are processed into Documents &amp; Chunks<br>- Documents store various metadata<br>- Chunks are nodes linked back to the parent document</li><li>Chunks are processed into Embeddings &amp; Graph Structure<br>- Embeddings are created and stored on the chunk node using OpenAI’s text-embedding-3-small embedding model<br>- Entity &amp; Relationship nodes are created using LangChain. Each Entity is connected to the chunk(s) it came from</li><li>Embeddings &amp; Graph Structure are processed into Vector Indices<strong> </strong>&amp; Graph Documents<br>- Vector Indices are created using the embeddings. This allows the embeddings to be queried by the user through the chatbot<br>- Graph documents — which store graph-related data — are created and stored in the database</li><li>Vector Indices &amp; Graph Documents create the knowledge based which feeds the Knowledge Graph &amp; Chatbot</li></ol><h4>3) Chatbot &amp; Knowledge Graph</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HgcJRRvBtGGoqbL3XP4_ig.png" /><figcaption>App’s main UI</figcaption></figure><p>The app’s main UI consists of a few key elements. The largest component is the main table, which displays all of the processed files and their metadata. It also provides access to the Knowledge Graph and Text Chunks. The side panel provides access to the Chatbot.</p><h4>Chatbot</h4><ul><li>The chatbot allows the user to query the knowledge base using natural language</li><li>Chatbot’s system prompt includes information and FAQ about 99P labs</li><li>Chatbot can answer questions only about the contents of the knowledge base and information about 99P</li></ul><p><strong>Example use-cases:</strong></p><ol><li>User who has been given a task that they have limited experience in and they would like guidance and/or advice on how to approach it</li><li>User who is interested in researching a topic but wants to know if anyone else has explored similar topics within the company</li></ol><p><strong>Example 1:</strong></p><p>In this example, the user has been tasked with working with a university partner which is something that they have never done before. They would like to connect with someone within the company who has worked with university partners to better understand the process.</p><p>The chatbot returns several people who have the relevant experience. In this screenshot contact details are not provided for privacy, but in real-world use the contact information (company email) would also be provided. The answer also includes a Details button that opens up a dialog box with several tabs. The first, and most important, one is the Sources used tab, which lists the actual documents the chatbot used to generate the answer and gives the user the option to downloaded these files if they would like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iyBBO9NXWU6jCa_J-NGZsw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WA0FfneGH4VN5GLfsaBZLw.png" /></figure><p><strong>Example 2:</strong></p><p>In the second example, the user is a researcher who is interested in the intersection of drunk driving and mobility access. Before they get too deep into their own research, they want to understand what work has already been done on that topic or similar topics, within the organization.</p><p>The chatbot returns several relevant documents within the knowledge base and gives a brief description. If the user would like to learn more about a particular document, they can again click on the Details button and download the actual source documents from the Sources used tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YcrRosUjsbUtIKeyJuPCOQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uR7hyBHAK7eiensIk65wPQ.png" /></figure><p>The Details dialog also provides access to the Top Entities tab, which shows the entities used to generate the response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*ZX3WvLU6h_RkAjAc8awG2A.png" /></figure><p>This dialog allows the user to pull up a knowledge graph at two levels:</p><p><strong>Individual Entities</strong></p><p>The individual entities can be clicked on to bring up a knowledge graph showing the connections for that specific entity. For example, clicking on the Joan Smith entity (under the Person tag) returns the following knowledge graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g7LH_GbN3xpHOgjS-dabsQ.png" /></figure><p>From this graph we can see all of the chunks Joan Smith is associated with, but more importantly we can also see that she is affiliated with 99P, works on Mobility &amp; Wellness, and was the PIC of the Mobility x Wellness workshop.</p><p><strong>All Entities</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A8sEUoRCaIyAIuKB7yS-SA.png" /></figure><p>By clicking the Graph Entities used for Answer Generation button instead, we get the full knowledge graph of all entities used to generate the response. Here we can view the relationships within the context of the answer given, instead of investigating them individually.</p><h4>Knowledge Graph</h4><p>We have already seen what knowledge graphs look like and how they can be used, in the above examples. These examples are focused on chatbot responses and are therefore limited in their context and scope. The main UI provides access to document-level or entire-knowledge-base-level knowledge graphs, which can then be filtered down and searched through as needed.</p><p>In the image below, the user selects the documents they want to generate a graph for and then clicks on the Preview Graph button to generate the graph</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k5VwvLvMcKwj0Aqq8K9hPw.png" /></figure><p>The resulting graph is fully interactive and can be zoomed in to explore in more detail. The user can also use the search function to filter down the graph for a specific question or use-case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2mTyUvvKhC4zH5k7Pfq9w.png" /></figure><p>Alternatively, the user can also set up a Neo4j Aura connection and explore the knowledge base through the web interface (see below). This gives the user much more granular control, allowing them to fine-tune queries as needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zpRj3o5ljEZotN0leYn_yg.png" /></figure><h3>Technical Deep Dive</h3><p>So far, we’ve explored why the 99P Archivist exists, how it transforms project files into a shared knowledge base, and the real-world problems it helps teams solve. The Technical Deep Dive now zooms in on the machinery behind those outcomes, walking through each major subsystem in the order the data flows: preprocessing, ingestion, chatbot orchestration, and knowledge-graph exploration. Every section is split into a quick TLDR for orientation and a richer Detail walkthrough, so you can skim for the high points or dig into the implementation as needed.</p><h4>Preprocessing</h4><p>The preprocessing pipeline is run from the Archivist_Project_Pipeline notebook. At a high level, the pipeline handles the conversion of project files into markdown format while preserving project context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*3ka4tliOCOk4nnKtXN7GYw.png" /><figcaption>Preprocessing Pipeline Overview</figcaption></figure><p><strong>Preprocessing Pipeline TLDR:</strong></p><p>1. Processes each folder in the <em>/not_processed</em> directory as a separate project</p><p>2. Reads metadata.json from each project folder to provide context</p><p>3. Adds project context to the beginning of each Markdown file</p><p>4. Stores processed files in corresponding folders in the <em>/processed</em> directory</p><p>5. Handles different file types appropriately:</p><ul><li>PPTX files: Convert to PDF, then PNGs, then to Markdown</li><li>PDF files: Convert to PNGs, then to Markdown</li><li>PNG files: Direct Markdown conversion with LLM support</li><li>Other formats: Standard Markdown conversion</li></ul><p>6. Maintains original file relationships and structure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*UQyFhNhOIhQcATMj5Qq5sw.png" /><figcaption>The preprocessing folder structure</figcaption></figure><p><strong>Preprocessing Pipeline Detail:</strong></p><p>The first step of the pipeline is to set some basic definitions and variables. The two most important ones are the LLM we’d like to use (along with the associated API key) and the system prompt mode we’d like to use (detailed vs summarized). The detailed system prompt will attempt to keep as much of the original content as possible, at the cost of speed and cost. The summarized system prompt will summarize much more of the content but it will be fast and cost less to process the data.</p><p>Next, we define some intermediary conversion functions. These functions (pptx_to_pdf() and pdf_pages_to_pngs()) convert PowerPoint files into PDFs and PDFs into PNGs. These functions are the result of much testing in which we found that the highest fidelity results were obtained by passing PNGs to the MarkItDown library along with an LLM.</p><p>Finally, we define the main markdown conversion function, convert_to_markdown_file(). This function will:</p><ol><li>Read in a file</li><li>Append the appropriate project context from metadata.json</li><li>Build the MarkItDown converter</li><li>Convert the file to Markdown depending on file type</li><li>Write the output to the <em>/processed</em> directory and make sure that there is a symlink to the backend (to enable downloading through the UI).</li></ol><p>Now that all the pieces have been defined, we define the process_project() function, which will go through the project structure and systematically process all files for all projects. This function works like so:</p><p>Defines input and output paths and creates directories if they don’t already exist. The function is designed to read in projects under the <em>/not_processed</em> directory and write them to the <em>/processed/{project_name}/markdown</em> and <em>/processed/{project_name}/original_files</em> directories.</p><p>The actual processing logic depends on the file type:</p><p><strong>If </strong><strong>PPTX:</strong></p><ul><li>Convert to a temporary PDF</li><li>Convert that temporary PDF to PNGs, stored in a temp directory</li><li>Convert the images to a single combined Markdown file</li><li>Clean up the temporary directory and PDF</li></ul><p><strong>Else If </strong><strong>PDF:</strong></p><ul><li>Convert PDF to PNGs, stored in a temp directory</li><li>Convert the images to a single combined Markdown file</li><li>Clean up the temporary directory and PDF</li></ul><p><strong>Else If </strong><strong>PNG:</strong></p><ul><li>Convert to Markdown using MarkItDown with LLM support</li></ul><p><strong>Else:</strong></p><ul><li>Convert to Markdown using MarkItDown without LLM support</li></ul><p>Once the process_project() function finishes running, we have the following:</p><ul><li>All of the original files and their corresponding Markdown representation stored in the <em>/processed/{project_name}/markdown</em> and /<em>processed/{project_name}/original_files</em> directories.</li><li>Symlinks for the Markdown files created and stored under the <em>/backend/merged_files</em> directory. This enables downloading of said Markdown documents through the UI.</li></ul><p>We are now ready to move on to the Ingestion part of the app.</p><h4>Ingestion</h4><p>The programmatic ingestion pipeline is accessed through the programmatic_ingestion.ipynb notebook. This notebook specifies a project name and optional post-processing tasks, and then calls the ingest_project_via_api() function to process and embed the markdown documents stored in the <em>/processed/{project_name}/markdown </em>directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*lNRVjBRHJXIfnfyPcb-HsQ.png" /><figcaption>Ingestion Process</figcaption></figure><p><strong>Ingestion Pipeline TLDR:</strong></p><ol><li>Every markdown file is registered as a LangChain Document node with rich metadata (size, origin, project, per-stage counters) so the pipeline can track progress and retry state.</li><li>Pages are split with LangChain’s TokenTextSplitter, respecting configurable chunk size/overlap, before each chunk receives a deterministic hash ID, positional metadata, and FIRST/NEXT wiring back to its Document for RAG-friendly traversal.</li><li>When embeddings are enabled (on by default), every chunk is vectorized with the configured model (OpenAI’s text-embedding-3-small by default), stored on the node, and guaranteed to sit inside the shared Neo4j vector index for similarity search.</li><li>Schema-guided extraction loads allowed node labels and relationship triplets from <em>frontend/src/assets/newSchema.json</em>, and forwards the curated CSV strings to the extractor.</li><li>Chunks are fed through the LangChain LLMGraphTransformer with user-selected node/relationship whitelists, chunk-combination windows, and extra instructions to shape extraction cost and schema.</li><li>Extracted entities are persisted as labeled nodes and stitched back to their source chunks via HAS_ENTITY.</li><li>Optional post-processing tasks include: refreshing KNN-style SIMILAR edges, hybrid search/full-text indexes, optional entity embeddings, schema cleanup, and multi-level community detection plus LLM-authored summaries/titles.</li></ol><p><strong>Ingestion Pipeline Detail:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NFnCYDnGPYM4Lcwj-hb_3A.png" /><figcaption>Ingestion Detail</figcaption></figure><p><strong>Data Ingestion</strong></p><ul><li>Each document is represented as a source node in the Neo4j database. Metadata such as file name, type, source URL, and size are stored as node properties. This ensures traceability and allows the application to efficiently manage different data sources.</li></ul><p><strong>Chunk Creation</strong></p><ul><li>To handle large documents effectively, the system splits them into smaller, manageable chunks using a token-based splitter. Configurable constants like CHUNKS_TO_BE_CREATED and CHUNK_SIZE define the size and number of chunks. These parameters can be provided from the UI in the graph enhancement section or defined in the codebase.</li><li>The chunk-size restriction is crucial because the vector embedding of each chunk should capture the essence of its meaning, and chunks that are too large get too broad and vague. Chunking ensures that the retrieved data fits within the LLM’s input window, enabling efficient and accurate processing.</li><li>Each chunk is represented as a Chunk Node in Neo4j with properties like document position, offset, and length of text.</li><li>Relationships are established between chunks and their parent document:<br>- PART_OF: Links each chunk to the parent document<br>- NEXT_CHUNK: Establishes sequential relationships between chunks</li></ul><p><strong>Embedding Generation &amp; Indexing</strong></p><ul><li>The chunked data transforms into a vector representation using OpenAI’s text-embedding-3-small embedding model.</li><li>These embeddings are stores as a property of the chunk node and enable similarity searches through Neo4j’s vector index.</li><li>A vector index named “vector” is created for chunk nodes, making it possible to retrieve semantically similar chunks during queries. These vectors capture the semantic meaning of each chunk. This built-in Neo4j vector index (“vector”) allows for efficient similarity searches (using the Hierarchical Navigable Small Worlds (HNSW)) — finding chunks most relevant to a given user question or query.</li></ul><p><strong>Entity Creation &amp; Relations Mapping</strong></p><ul><li>After chunk creation, entities are extracted from chunk text using LLMs and Neo4j’s LLMGraphTransformer.</li><li>The LLMGraphTransformer process leverages the LLMGraphTransformer library from LangChain. Specifically, the convert_to_graph_documents method within this library transforms the chunk data into a structured representation called a GraphDocument. A GraphDocument is essentially a graph data structure consisting of a list of nodes (representing entities) and their relationships (edges connecting the nodes). This graph representation allows for a more nuanced and contextual understanding of the extracted entities and their connections within the original data.</li><li>The LLMGraphTransformer correspondingly enables the creation of knowledge graphs with a focus on nodes, relationships, and their properties — all while adhering to the following configurable constraints:<br>- Users can specify which node types (allowed_nodes) and relationship types (allowed_relationships) to include in the graph. This ensures that only relevant elements extract, reducing noise and focusing on the desired graph structure.<br>- Rich property extraction — The transformer supports extracting custom properties for nodes and relationships. Users can allow unrestricted property extraction or define specific properties to include, enhancing the granularity of the resulting graph.<br>- Additional instructions — Users can provide additional instructions in the graph enhancement section to pass on to the LLMGraphTransformer to focus on particular topics (e.g., focus on healthcare terms or ignore technical jargon, etc.). This helps to extract entities in line with requirements.</li><li>Each extracted entity is represented as an entity node in the graph and links to the originating chunk through a HAS_ENTITY relationship.</li><li>The system allows further refinement by merging related/duplicate entities in the graph enhancement section.</li><li>The application’s entity extraction process combines smaller text embedding chunks into larger text fragments. A configurable parameter (NUMBER_OF_CHUNKS_TO_COMBINE) controls the size of these fragments, currently defaulted to 1. Consequently, if this value increases, that many chunks will group together before processing.</li></ul><p><strong>Schema Configuration &amp; Graph Construction</strong></p><ul><li>The schema defaults to the output_markdown schema defined in <em>frontend/src/assets/newSchema.json</em>. Those Subject — Predicate — Object triplets enumerate the people, projects, organizations, events, and other relationships that make up 99P’s graph, so the extractor never improvises labels outside that curated vocabulary.</li><li>When an ingestion run starts, _load_schema_defaults reads the schema file, falls back if it is missing, and converts the selected triplets into comma-separated lists of node labels and relationship triples.</li><li>Those lists become hard constraints inside get_graph_from_llm. The backend merges chunk windows, strips whitespace, rejects malformed triplets, and ensures every relationship references a permitted source and target before it hands the request to LangChain’s LLMGraphTransformer. The transformer then emits graph documents whose nodes and edges mirror the approved schema, preventing stray labels from reaching Neo4j.</li><li>Once the model returns graph documents, processing_chunks continues the pipeline by updating chunk embeddings, writing the extracted entities and relationships into Neo4j, and wiring each entity back to its source chunk so provenance survives downstream analytics and chatbot answers.</li></ul><p><strong>Post Processing</strong></p><p>Processing includes the following tasks. You can enable them by adding them to the list defined in the POST_PROCESSING_TASKS variable within the programmatic_ingestion.ipynb notebook.</p><p><strong>1. Updating text chunk similarities</strong></p><ul><li>This step utilizes a k-nearest neighbor (KNN) algorithm to update graph similarities between chunks. By analyzing embedding-based distances, the graph captures stronger connections between similar content.</li></ul><p><strong>2. Hybrid and full-text search enablement</strong></p><ul><li>Hybrid search combines vector-based similarity with full-text search capabilities. This step creates vector and full-text indexes, enabling efficient search queries in tools like Neo4j Bloom. Users can perform semantic and keyword searches seamlessly.</li></ul><p><strong>3. Entity embeddings</strong></p><ul><li>Entities in the graph embed using a language model, allowing for similarity calculations between entities. This enables tasks like clustering and entity disambiguation. However, entity embeddings create only if the ENTITY_EMBEDDING flag is enabled.</li></ul><p><strong>4. Graph schema consolidation</strong></p><ul><li>The graph schema consolidation step standardizes node labels and relationships to maintain a clean and logical structure. Using a language model, redundant or inconsistent labels and relationships are unified, ensuring clarity and consistency in the graph’s schema. Users can enable this option through the graph enhancement tab.</li></ul><p><strong>5. Community detection</strong></p><ul><li>Users can enable the communities option in the graph enhancement section. Communities represent areas where nodes are more tightly interconnected with one another than with nodes in other groups. Communities are formed using algorithms like Leiden clustering. Community notes are created in the knowledge graph to represent clusters of related entities or chunks. Relationships such as IN_COMMUNITY and PARENT_COMMUNITY link nodes to their communities and parent-child relationships between communities. Communities have a hierarchy from 0 to 2 (0 being the base community, 1 being a parent community connecting to base communities, and 2 being a parent community to first-level parent communities).</li></ul><p><strong>6. Community summaries</strong></p><ul><li>Summaries generate for each community using LLMs to provide a concise overview of its content. Properties like summary and title are added to community nodes. Summaries enable broader questions like summarization or multi-context questions to be answered effectively. Community titles generate through an LLM to give concise titles to summaries, enabling proper identification of community nodes.</li></ul><h4>Chatbot</h4><p><strong>Chatbot TLDR:</strong></p><ol><li>Each prompt calls the /chatbot FastAPI endpoint, which selects a retrieval strategy (vector, full-text, graph, entity+vector, global search, etc.) based on the active chat mode and environment.</li><li>Session transcripts persist in Neo4j (or a local in-memory fallback) with automatic summarization threads that keep long conversations performant while still grounding answers in retrieved chunks.</li><li>Responses return rich metadata — ranked sources with download links, chunk/entity/community IDs, model version, token counts, and mode indicators — so the UI can render source pills, provenance modals, and inspection drawers.</li><li>Supporting endpoints power lifecycle operations: /clear_chat_bot wipes the saved history, /chunk_entities exposes chunk-level nodes and relationships, and /metric plus /additional_metrics run RAGAS-based quality evaluations when teams need automated scoring.</li></ol><p><strong>Chatbot Detail:</strong></p><p>The Chatbot package in the frontend manages all presentation and state for multi-view chat experiences. It tracks user messages, simulated typing, audio playback, copy-to-clipboard, and the expanded metrics panel. When the user submits a question, the component posts to /chatbot with the selected mode, any document filters, and the current session identifier. The backend’s QA_RAG entry point loads or creates a Neo4j-backed Neo4jChatMessageHistory, appends the new HumanMessage, and routes the request either through process_graph_response (pure graph mode) or process_chat_response (retrieval-augmented generation modes).</p><p>process_chat_response assembles the toolchain for the chosen mode: it instantiates the configured embedding index via Neo4jVector.from_existing_graph, optionally constrains it to selected documents, and wraps it in a LangChain ContextualCompressionRetriever that first rewrites follow-ups, splits oversized docs, and filters redundant chunks by embedding similarity. Retrieved chunks are formatted with contextual metadata and passed to the LLM designated by the user (OpenAI‘s gpt-5-mini by default). The resulting AIMessage is enriched with provenance (sources, chunk scores, entity IDs, or community summaries for global modes), total token usage, and execution mode before it is streamed back to the UI and appended to history. Parallel threads keep the persisted conversation trimmed by summarizing prior turns with the same LLM.</p><p>For teams validating answer quality, the frontend can trigger /metric to compute faithfulness and answer relevance or /additional_metrics for context-entity recall, semantic similarity, and ROUGE when ground-truth data is available. Power users can reset conversations through /clear_chat_bot, and deeper inspections come from /chunk_entities, which fetches all entity nodes, relationships, and original chunk text tied to a cited element ID so analysts can audit why a response was produced.</p><h4>Knowledge Graph</h4><p><strong>Knowledge Graph TLDR:</strong></p><ol><li>The ingestion outputs feed directly into the interactive Graph view, where /graph_query returns document, chunk, entity, relationship, and community nodes for visualization and drill-down.</li><li>Users can launch the graph explorer from the file table or graph actions menu to inspect node properties, relationship counts, and post-processing statistics (communities, similarities, embeddings, etc.).</li><li>The /chunk_entities endpoint underpins the chunk/entity side panels—surfacing the exact text span, connected entities, and relationship types so graph analysts can validate extraction quality without leaving the UI.</li><li>Graph enhancement controls let teams define or regenerate schemas, merge duplicates, drop unconnected nodes, and pass extra LLM instructions, keeping the knowledge graph clean and domain-aligned over time.</li><li>Graph-only chat mode leverages LangChain’s GraphCypherQAChain to translate natural language into Cypher, execute it against Neo4j, and relay both the generated query and the resulting subgraph for transparency.</li></ol><p><strong>Knowledge Graph Detail:</strong></p><p>Once documents finish ingestion and post-processing, each source exists as a Document node with linked Chunk, Entity, Community, and similarity relationships inside Neo4j. The frontend’s Graph tooling allows users to explore that structure directly. From the file table, analysts can open the Graph View action, which triggers /graph_query with the selected file list. The API responds with a full subgraph (including attributes such as processing time, node counts, embedding stats, and community totals) so the React graph canvas can render documents as hubs, chunks as intermediate nodes, entities with their extracted labels, and community clusters with the appropriate legend styling.</p><p>Contextual side panels call /chunk_entities whenever a user inspects a cited chunk from the chatbot or graph view. This returns the raw chunk text, positional metadata, and every entity/relationship that originated from that fragment, enabling auditors to verify extraction fidelity or debug noisy nodes. When broader schema adjustments are needed, the Graph Enhancement dialog exposes the backend’s schema APIs: /schema lists existing labels and relationship types, /populate_graph_schema regenerates schema suggestions from sample text, while /delete_unconnected_nodes and /merge_duplicate_nodes clean up isolated or redundant entities after review. Additional instructions and allowed-node filters are persisted alongside ingestion settings so future runs stay aligned with organizational knowledge standards.</p><p>For investigative querying, the dedicated graph chat mode wires LangChain’s GraphCypherQAChain to the active Neo4j instance. Natural-language prompts are converted into Cypher, executed, and the intermediate steps (query text plus matched nodes/relationships) are returned to the UI. That integration pairs with the richer visualization to let users alternate between visual exploration, table summaries, and declarative graph Q&amp;A, all backed by the same Neo4j dataset produced during preprocessing and ingestion.</p><h3>Conclusion</h3><p>This project represents our ongoing exploration at 99P Labs into making AI a more practical and useful tool. By leveraging AI to surface our organization’s hidden connections through the 99P Archivist, we believe we can create connections, increase workplace efficiency, and ultimately increase both the quantity and quality of our organization’s output.</p><p>That spirit of collaboration is central to our mission. If you found this glimpse into our work rewarding, we invite you to connect with our community. Consider subscribing to the <a href="https://medium.com/99p-labs">99P Labs blog</a>, connecting with us on <a href="https://www.linkedin.com/company/99p-labs">LinkedIn</a>, or checking out <a href="http://99plabs.com/">our website</a>. We’re always looking for new ways to push the boundaries of innovation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d557b32fe0da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/99p-labs/99p-archivist-project-d557b32fe0da">99P Archivist Project</a> was originally published in <a href="https://medium.com/99p-labs">99P Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[99P Labs F1TENTH: Project SOMEThings Goes Open Source]]></title>
            <link>https://medium.com/99p-labs/99p-labs-f1tenth-project-somethings-goes-open-source-514d5cd8b9e5?source=rss-a36222816d6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/514d5cd8b9e5</guid>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[autonomous-vehicles]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Luka Brkljacic]]></dc:creator>
            <pubDate>Mon, 02 Dec 2024 16:02:07 GMT</pubDate>
            <atom:updated>2024-12-02T16:02:07.192Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZLEbze7sypW6arI8UluwqA.jpeg" /><figcaption>99P’s F1TENTH vehicle driving around our custom-built track</figcaption></figure><h3>Introduction</h3><p>As detailed in our <a href="https://medium.com/99p-labs/project-somethings-creating-a-cyber-physical-environment-for-the-future-of-mobility-979a1b9c4e55">previous post</a>, the 99P Labs team has been working on creating a connected ecosystem proving ground through our extension of the <a href="https://github.com/f1tenth">F1TENTH project</a>, internally named SOMEThings. A significant part of this endeavor involved utilizing and enhancing various open-source projects, particularly the F1TENTH platform.</p><p>Today, we are excited to announce that we have officially open-sourced the entire SOMEThings project. By sharing our work, we aim to contribute to the community and foster collaboration in the field of autonomous vehicles and robotics. We believe that all challenges are best addressed through collaboration and shared knowledge. As such, we hope that by open sourcing this project we will be able to:</p><ul><li><strong>Accelerate Innovation:</strong> Empower researchers, students, and enthusiasts to build upon our platform, fostering rapid advancements in the field.</li><li><strong>Give Back to the Community:</strong> Contribute to the open-source ecosystem that has provided us with invaluable tools and frameworks.</li><li><strong>Encourage Collaboration:</strong> Invite others to contribute improvements, fixes, and new features.</li></ul><p>The rest of this blog post will provide an overview of the codebase, explain how to get started, and share ideas on how you might adapt the project for your own needs.</p><h3>Repo Overview</h3><p>The SOMEThings project consists of two repositories:</p><ul><li><a href="https://github.com/honda-research-institute/99plabs-f1tenth">99plabs-f1tenth</a></li><li><a href="https://github.com/honda-research-institute/99plabs-f1tenth-sim">99plabs-f1tenth-sim</a></li></ul><h3>99plabs-f1tenth</h3><h4>Overview</h4><p>The 99plabs-f1tenth repository contains three main components:</p><ul><li><strong>base:</strong> Handles communication between the driving hardware (VESC, motors, etc.) and software. This is where you can set up the joystick and deadman switch and adjust speed and odometry settings.</li><li><strong>nav:</strong> Responsible for localization and autonomous navigation using the <a href="https://docs.nav2.org/">nav2 package</a>. It includes the vehicle description defining the physical characteristics of the vehicle.</li><li><strong>task_commander:</strong> Stores, submits, and executes waypoints and poses for the vehicle to navigate.</li></ul><h4>Getting Started</h4><p>Detailed instructions are available in the <a href="https://github.com/honda-research-institute/99plabs-f1tenth/blob/main/README.md">99plabs-f1tenth README</a>.</p><ol><li>Turn on the car and build the images using Docker Compose.</li><li>Enter the nav container to execute commands.</li><li>Launch the nav2 stack.</li><li>Send the car a navigation goal.</li></ol><p>If everything is set up correctly, your vehicle should drive autonomously, resembling <a href="https://www.youtube.com/watch?v=hpUR0EYhCzw">this demonstration</a>.</p><h3>99plabs-f1tenth-sim</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*qFcDxTTb5_oWor5Tk162zg.png" /><figcaption>Simulated version of the 99P labs track and vehicle</figcaption></figure><h4>Overview</h4><p>The 99plabs-f1tenth-sim repository provides a simulated environment to help developers build and test software efficiently before deploying it to a physical vehicle. While designed to closely mirror real-world behavior, please note that due to hardware limitations and physics, there may be slight differences between the simulated and physical vehicles.</p><h4>Getting Started</h4><p>Detailed instructions are available in the <a href="https://github.com/honda-research-institute/99plabs-f1tenth-sim/blob/main/README.md">99plabs-f1tenth-sim README</a>.</p><ol><li>Ensure your Docker image has finished building, then enter the container.</li><li>Source and build the workspace.</li><li>Launch the simulator.</li><li>Initialize the car (optional).</li><li>Send the car navigation goal(s).</li></ol><p>If everything is functioning correctly, your simulated vehicle should navigate autonomously, similar to <a href="https://www.youtube.com/watch?v=U_jUog8-W-c">this video</a>.</p><h3>How You Can Use the SOMEThings Platform</h3><p>SOMEThings is a versatile platform that can be adapted for various purposes:</p><ul><li><strong>Hackathons and Competitions:</strong> Quickly prototype and test autonomous vehicle algorithms.</li><li><strong>Educational Projects:</strong> Ideal for student clubs and university courses focusing on robotics and autonomous systems.</li><li><strong>Research Labs:</strong> Test new navigation algorithms, sensor fusion techniques, or AI models in both real-world and simulated environments.</li><li><strong>Personal Projects:</strong> Hobbyists interested in robotics can experiment with autonomous driving technologies using accessible hardware.</li></ul><p>We encourage you to tailor SOMEThings to fit your specific needs and projects (and share them with us!).</p><h3>What’s Next for SOMEThings</h3><p>We’ve learned extensively from our experience and are working on evolving our project with a focus on:</p><ul><li><strong>Software-Defined Embedded Systems:</strong> Creating intelligent, hardware-independent software frameworks that foster seamless cooperation and intuitive interaction within diverse, multi-system environments.</li><li><strong>Cloud Continuum and Connectivity:</strong> Designing and evaluating sustainable, resilient, and ethical cloud continuum models using open-source frameworks to ensure scalable and adaptive connectivity.</li><li><strong>Informative Data:</strong> Developing data-driven methods for simulating and analyzing environments to extract valuable insights, leveraging AI for actionable intelligence.</li></ul><p>In the near future, we hope to expand the platform to a larger scale by adding larger vehicles, drones, and smart infrastructure. Stay tuned for updates as we continue to push the boundaries in autonomous systems and robotics.</p><h3>Conclusion</h3><p>By open-sourcing SOMEThings, we aim to empower others tackling similar challenges. If you’re working on autonomous vehicles, robotics, or related fields, we highly recommend exploring our repositories and those linked in the Acknowledgements section below.</p><p>Feel free to reach out if you have any questions or would like to discuss further. Happy coding!</p><h3>Get Involved</h3><p>We’d love to hear from you!</p><ul><li><strong>Technical Questions:</strong> If you have any technical inquiries or need assistance, feel free to reach out to us at <a href="http://research@99plabs.com/">research@99plabs.com</a> or visit <a href="https://www.99plabs.com/">our website</a>.</li><li><strong>Contribute Ideas: </strong>Have suggestions for new features or improvements? Submit them via our GitHub issues or email us directly.</li></ul><h3>Acknowledgements</h3><p>These projects were invaluable to the development of the SOMEThings project. We sincerely thank all of the people behind these projects and hope that you will check them out!</p><ul><li><a href="https://github.com/f1tenth">F1TENTH</a></li><li><a href="https://github.com/CL2-UWaterloo/f1tenth_ws">UWaterloo’s F1TENTH repo</a></li><li><a href="https://stevengong.co/notes/F1TENTH">Steven Gong’s F1TENTH notes</a></li><li><a href="https://foxglove.dev/ros">Foxglove</a></li><li><a href="https://docs.nav2.org/">nav2</a></li><li><a href="https://github.com/ros-visualization/rviz">RViz</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=514d5cd8b9e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/99p-labs/99p-labs-f1tenth-project-somethings-goes-open-source-514d5cd8b9e5">99P Labs F1TENTH: Project SOMEThings Goes Open Source</a> was originally published in <a href="https://medium.com/99p-labs">99P Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Overview of Machine Learning — Part 3]]></title>
            <link>https://medium.com/99p-labs/an-overview-of-machine-learning-part-3-acf71fe16d69?source=rss-a36222816d6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/acf71fe16d69</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Luka Brkljacic]]></dc:creator>
            <pubDate>Thu, 23 Mar 2023 14:04:20 GMT</pubDate>
            <atom:updated>2023-03-23T14:04:20.499Z</atom:updated>
            <content:encoded><![CDATA[<h3>An Overview of Machine Learning — Part 3</h3><p>Written by: Luka Brkljacic</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*v9kL0ntAeJaB7jR7" /><figcaption>Photo by <a href="https://unsplash.com/@elsbethcat?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Beth Macdonald</a> on <a href="https://unsplash.com/photos/P3rS8J1THi4?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Table of Contents</h3><ol><li><a href="#d4c9"><strong>Introduction</strong></a></li><li><a href="#28bd"><strong>Decision Trees</strong></a><strong><br> a. </strong><a href="#bd47"><strong>Classification Trees</strong></a><strong><br> b. </strong><a href="#ad94"><strong>Regression Trees</strong></a></li><li><a href="#ac23"><strong>Python Examples</strong></a><strong><br> a. </strong><a href="#a655"><strong>Classification Trees</strong></a><strong><br> b. </strong><a href="#d45b"><strong>Regression Trees</strong></a></li></ol><h3>Introduction</h3><p>Welcome to the 3rd edition of our Overview of Machine Learning! In this post, we will go deeper into the mechanisms of machine learning by doing a deep dive on <strong>Decision Trees</strong>. To get the complete picture, make sure you first check out <a href="https://medium.com/99p-labs/an-overview-of-machine-learning-part-1-7784ba3b3540">Part 1</a> and <a href="https://medium.com/99p-labs/an-overview-of-machine-learning-part-2-all-about-regression-2f991281932e">Part 2</a>.</p><p>If at any point you find yourself getting overwhelmed but want to keep learning, I highly suggest checking out this great YouTube playlist on Machine Learning from <a href="https://youtube.com/playlist?list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF">StatQuest</a>!</p><h3>Decision Trees</h3><p>The purpose of <strong>Decision Trees</strong>, just like with <strong>Regression</strong>, is to predict some outcome. This technique is usually associated with classification (<strong>Classification Trees</strong>) but can also be used to predict numeric variables (<strong>Regression Trees</strong>).</p><p><strong><em>Classification Trees</em></strong></p><p>In general, <strong>Classification Trees</strong> predict some outcome by doing the following:</p><ol><li>Make a statement</li><li>Make a decision based on whether or not that statement is True or False</li><li>Repeat until the bottom of the tree</li></ol><p>Example: You are looking to buy your first home and are considering the following independent variables:</p><ul><li><strong>Size</strong> (sqft)</li><li><strong>Neighborhood </strong>(“Bad”, “Okay”, “Great”)</li><li><strong>Price </strong>($)</li></ul><p>Your requirements are:</p><ul><li><strong>Size </strong>≥ 2,000sqft</li><li><strong>Neighborhood </strong>!= “Bad”</li><li><strong>Price </strong>≤ $300,000</li></ul><p>Below is a <strong>Classification Tree</strong> you might use when trying to classify available homes to identify those that meet your requirements. Important terminology to keep in mind:</p><ul><li>The starting node is called the <strong>Root Node</strong> (arrows pointing only from)</li><li>The rest of the gray nodes are called <strong>Branches</strong> (arrows pointing to and from)</li><li>The red and green nodes are called <strong>Leaf Nodes</strong> (arrows only point to)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-8brff5QP9dvm7Boi-z6aA.png" /></figure><p>Now, let’s adjust our example so that instead of trying to help you buy a house, we are an appraiser trying to determine whether the <strong>Value</strong> of a house is High or Low. We have the following data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*6P40hzE5M8IseGUZUbmerA.png" /></figure><p>Now we need to decide which of our three independent variables should serve as our <strong>Root Node</strong>. To do this, we calculate a <strong>Gini Impurity</strong> score (which is defined by the following equations) for each of our predictor variables:</p><p><strong>Leaf Gini Impurity</strong> = 1 — Probability(“High”)² — Probability(“Low”)²</p><p><strong>Total Gini Impurity</strong> = Weighted Average(Leaf Gini Impurities)</p><p>For our categorical variable, <strong>Type</strong>, calculating the <strong>Total Gini Impurity </strong>score is pretty straightforward. We start by making <strong>Type </strong>our root node. Then, we calculate the number of occurrences for each leaf node and get each leaf’s <strong>Gini Impurity</strong> score. Finally, we take the weighted average of the two <strong>Leaf Gini Impurity</strong> scores to get the <strong>Total Gini Impurity</strong> score for the <strong>Type</strong> variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5U1lvhGhxaYvnLWF7betCA.png" /></figure><p>For the continuous variables, calculating the score is a little bit more involved. We start by sorting our data frame by <strong>Size</strong>, from smallest to largest. Next, we calculate the average <strong>Size</strong> between each row:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F1ERxyOi5Fqae0xmCXDBcA.png" /></figure><p>Now we calculate the <strong>Gini Impurity</strong> using each of those average values as the root node and following the exact same procedure as with categorical variables. We do this until we have a <strong>Total Gini Impurity</strong> value for every average value above. Once we have all of those values, the lowest one will be kept as the <strong>Total Gini Impurity</strong> for the <strong>Size</strong> column as a whole.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*gHUZrEQ60FC1iuDqSubFWg.png" /></figure><p>We will repeat the above process with the <strong>Age</strong> column, at which point we will have three <strong>Total Gini Impurity</strong> scores — one for each one of our independent variables. The variable with the lowest score will be the <strong>Root Node</strong> of our Decision Tree because we have determined that it does the best job at predicting a home’s <strong>Value </strong>category. At the next level of the tree, we will perform the same process, recalculating the <strong>Gini </strong>to determine which of the remaining variables has the lowest impurity and should be used as the next decision node.</p><p>We repeat this process until we have no more ways to split up our data frame and have reached the leaves of the tree. We can then use that tree to classify new data and make predictions.</p><p><strong><em>Regression Trees</em></strong></p><p>Regression Trees are simply a specialized version of <strong>Decision Trees</strong> where we are trying to predict a numeric variable instead of a categorical one.</p><p><strong>Regression Trees</strong> — just like their Classification counterpart — predict some outcome by doing the following:</p><ol><li>Making a statement</li><li>Making a decision based on the value in the statement</li><li>Repeating until the bottom of the tree</li></ol><p>Example: Let’s return to our house buying example from above. In this example, you are looking to buy your first home and have the following data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*qGprEFpTzQIlBEc-Gb0Dqg.png" /></figure><p>Note that we added the <strong>Price </strong>column, which gives us the exact cost of each house in the dataset. This will be the variable we are trying to predict.</p><p>Next, we need to decide which of our three independent variables (<strong>Type, Size, Age</strong>) should serve as our <strong>Root Node</strong>. We do this by calculating the <strong>Sum of Squared Residuals (SSR)</strong> for each variable and plotting it against each independent variable. To illustrate, let’s start with <strong>Size</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*blkHY_rMnI1luuDoiFWM4w.png" /></figure><p>First, we divide our dataset into two by focusing on only the first two data points; this will serve as our first <strong>Root Node</strong> candidate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*hcvPYzys83M-tief3ptooQ.png" /></figure><p>The <strong>Leaf Nodes</strong> will be the average of <strong>Price</strong> on each side of our line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*4Ti-Rit7VI8MuwfFL_IRqA.png" /></figure><p>Plotting these values on our scatterplot, we can then visualize the quality of using Size &lt; 1,315 as our Root Node:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*yAUnWHSfLSlSGGwFu_1iLQ.png" /></figure><p>To have a better understanding of how good of a <strong>Root Node</strong> this is, however, we can calculate its <strong>SSR </strong>by taking the difference between each point and the green dotted line, squaring it, and then adding all of those values together. When we do this, we get an <strong>SSR </strong>of 235,988,888,889 for this choice of <strong>Root Node</strong>. Finally, we plot this value on a scatterplot against the value of <strong>Size</strong> used as our <strong>Root Node</strong> criterion:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*8AvkUM3L7eKw3aEAr4AfXw.png" /></figure><p>We will then repeat the above process, moving our cutoff between the next pair of <strong>Size</strong> values until we reach the end. For each cutoff, we will calculate its SSR and add it to the plot. In the end, we end up with this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*fNHTKAakgieDJ_wLTM5lHg.png" /></figure><p>According to the plot above, using a <strong>Size</strong> threshold of 1,755 results in the best prediction. As such, this is our best <strong>Root Node</strong> candidate for the <strong>Size </strong>variable. To finish creating the tree, we will simply repeat this process until we can no longer split the observations into groups, thereby creating our <strong>Leaf Nodes</strong>.</p><p>To extend this process to multiple independent variables like we have in our example, we simply perform the exact same process for each of the variables. The variable with the lowest <strong>SSR </strong>will serve as our <strong>Root Node</strong>. We will then repeat the process at each level of the three, calculating the <strong>SSR </strong>for each of the variables, and choosing the lowest one.</p><p>Note that in our example, we have a very small about of data points. This means that the model resulting from our tree will likely be overfit, since a lot of our <strong>Leaf Nodes</strong> will be based on very few observations. To get around this, we can simply set a minimum amount of observations needed within a group. In practice this number is usually set to 20, but can be tweaked based on your specific use-case.</p><h3>Python Examples</h3><p><strong><em>Decision Trees</em></strong></p><p>Dataset: <a href="https://www.kaggle.com/datasets/uciml/pima-indians-diabetes-database/download?datasetVersionNumber=1">Pima Indians Diabetes Database</a> (info page <a href="https://www.kaggle.com/datasets/uciml/pima-indians-diabetes-database">here</a>)</p><p>To start, import all required libraries and read in the file.</p><pre># Load all libraries<br>import pandas as pd<br>import numpy as np<br>from sklearn.tree import DecisionTreeClassifier<br>from sklearn.model_selection import train_test_split <br>from sklearn import metrics</pre><pre># Load the dataset<br>diabetes = pd.read_csv(&quot;/diabetes.csv&quot;)<br><br>diabetes = diabetes.dropna()</pre><p>Our goal with this dataset is to create a model that is able to accurately predict whether a patient has diabetes. We start off by defining our independent (x) and dependent (y) variables.</p><pre># Define the independent and dependent variables<br>independent_cols = [&#39;Pregnancies&#39;, &#39;Glucose&#39;, &#39;BloodPressure&#39;, &#39;SkinThickness&#39;, &#39;Insulin&#39;, &#39;BMI&#39;, &#39;DiabetesPedigreeFunction&#39;, &#39;Age&#39;]<br>x = diabetes[independent_cols] # Independent Variables<br>y = diabetes[&#39;Outcome&#39;] # Dependent Variables</pre><p>Next, we split the data into training and testing sets so we can evaluate the performance of our model.</p><pre># Split the dataset into training and testing sets<br>x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.3, random_state = 1) # 70% training and 30% test</pre><p>We then create the Decision Tree Regressor, train it, and use the model to make predictions.</p><pre># Create the Decision Tree Regressor object<br>classifier = DecisionTreeClassifier()<br><br># Train the classifer<br>classifier = classifier.fit(x_train, y_train)<br><br># Predict the response for test dataset<br>y_pred = classifier.predict(x_test)</pre><p>Now that we have the model, we can:</p><ul><li>Investigate the results visually by plotting the Decision Tree. Note that this specific tree is quite large, so it would be hard to interpret the results from a single image.</li></ul><pre>from sklearn import tree<br>from matplotlib.pyplot import figure<br><br>figure(figsize = (30, 10), dpi = 100)<br>tree.plot_tree(classifier)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ntSFy_J1aifQfo4VYL3mog.png" /></figure><ul><li>Evaluate the model’s accuracy</li></ul><pre># Evaluate the model<br>print(f&quot;Accuracy: {round(metrics.accuracy_score(y_test, y_pred), 4)*100}%&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*e22XTD-ZKLhSf-atcGzbaA.png" /></figure><p><strong><em>Regression Trees</em></strong></p><p>Dataset: <a href="https://www.kaggle.com/datasets/kumarajarshi/life-expectancy-who/download?datasetVersionNumber=1">Life Expectancy (WHO)</a> (info page <a href="https://www.kaggle.com/datasets/kumarajarshi/life-expectancy-who">here</a>)</p><p>To start, import all required libraries and read in the file.</p><pre># Load all libraries<br>import pandas as pd<br>import numpy as np<br>from sklearn.tree import DecisionTreeRegressor<br>from sklearn.model_selection import train_test_split <br>from sklearn import metrics</pre><pre># Read in a subset of the data<br>cols_of_interest = [&#39;Alcohol&#39;, &#39;infant deaths&#39;, &#39;BMI&#39;, &#39;Total expenditure&#39;, &#39;GDP&#39;, &#39;Population&#39;, &#39;Schooling&#39;, &#39;Life expectancy&#39;]<br><br>data = pd.read_csv(&quot;/Life Expectancy Data.csv&quot;, usecols = cols_of_interest)<br><br>data = data.dropna()</pre><p>Our goal with this dataset is to create a model that is able to accurately predict a country’s life expectancy. Just like before, we start off by defining our independent (x) and dependent (y) variables.</p><pre># Define the independent and dependent variables<br>independent_cols = [&#39;Alcohol&#39;, &#39;infant deaths&#39;, &#39;BMI&#39;, &#39;Total expenditure&#39;, &#39;GDP&#39;, &#39;Population&#39;, &#39;Schooling&#39;]<br>x = data[independent_cols] # Independent Variables<br>y = data[&#39;Life expectancy&#39;] # Dependent Variables</pre><p>Next, we split the data into training and testing sets so we can evaluate the performance of our model.</p><pre># Split the dataset into training and testing sets<br>x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.3, random_state = 1) # 70% training and 30% test</pre><p>We then create the Decision Tree Regressor, train it, and use the model to make predictions.</p><pre># Create the Decision Tree Regressor object<br>classifier = DecisionTreeRegressor()<br><br># Train the classifer<br>classifier = classifier.fit(x_train, y_train)<br><br># Predict the response for test dataset<br>y_pred = classifier.predict(x_test)</pre><p>Now that we have the model, we can:</p><ul><li>Investigate the results visually by plotting the Decision Tree. Note that this specific tree is quite large, so it would be hard to interpret the results from a single image.</li></ul><pre>from sklearn import tree<br>from matplotlib.pyplot import figure<br><br>figure(figsize = (30, 10), dpi = 100)<br>tree.plot_tree(classifier)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XYTgLvIF4yW5k0Qayv7uCw.png" /></figure><ul><li>Evaluate the model’s accuracy</li></ul><pre># Evaluate the model<br>print(f&quot;MAPE: {round(metrics.mean_absolute_percentage_error(y_test, y_pred), 4)}&quot;)<br><br>print(f&quot;R2: {round(metrics.r2_score(y_test, y_pred), 3)}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*CGV-haFnwjK1u6YarZxnSg.png" /></figure><h3>Final Thoughts</h3><p>So far in this series, we’ve covered the basics of machine learning in general, dove deep into regression topics, and finished with an overview of decision trees. These topics define the foundation upon which most of the more advanced ML techniques are based. In the future, we hope to create a few more posts detailing these techniques, so be on the lookout for that!</p><p>If you’d like to discuss this topic with us, you can comment on this post or connect with us on <a href="https://www.linkedin.com/company/99p-labs/">LinkedIn</a>.</p><h3>Resources &amp; Further Reading</h3><p>99P DevPortal: <a href="https://developer.99plabs.io/home/">https://developer.99plabs.io/home/</a></p><p>Beyond Algorithms: <a href="https://www.routledge.com/Beyond-Algorithms-Delivering-AI-for-Business/Luke-Porter-Santhanam/p/book/9780367613266">https://www.routledge.com/Beyond-Algorithms-Delivering-AI-for-Business/Luke-Porter-Santhanam/p/book/9780367613266</a></p><p>StatQuest: <a href="https://statquest.org/">https://statquest.org/</a></p><p>3Blue1Brown: <a href="https://www.3blue1brown.com/#lessons">https://www.3blue1brown.com/#lessons</a></p><p>An Introduction to Statistical Learning e-book: <a href="https://hastie.su.domains/ISLR2/ISLRv2_website.pdf">https://hastie.su.domains/ISLR2/ISLRv2_website.pdf</a></p><p>The Elements of Statistical Learning e-book: <a href="https://hastie.su.domains/Papers/ESLII.pdf">https://hastie.su.domains/Papers/ESLII.pdf</a></p><p>Computer Age Statistical Inference e-book: <a href="https://hastie.su.domains/CASI_files/PDF/casi.pdf">https://hastie.su.domains/CASI_files/PDF/casi.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acf71fe16d69" width="1" height="1" alt=""><hr><p><a href="https://medium.com/99p-labs/an-overview-of-machine-learning-part-3-acf71fe16d69">An Overview of Machine Learning — Part 3</a> was originally published in <a href="https://medium.com/99p-labs">99P Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Overview of Machine Learning — Part 2: All About Regression]]></title>
            <link>https://medium.com/99p-labs/an-overview-of-machine-learning-part-2-all-about-regression-2f991281932e?source=rss-a36222816d6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f991281932e</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Luka Brkljacic]]></dc:creator>
            <pubDate>Thu, 19 Jan 2023 21:32:08 GMT</pubDate>
            <atom:updated>2023-03-28T15:25:00.824Z</atom:updated>
            <content:encoded><![CDATA[<h3>An Overview of Machine Learning — Part 2: All About Regression</h3><p>Written by: Luka Brkljacic</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Dk3mBgHVtakb0-Whli8sg.png" /><figcaption>Photo by <a href="https://unsplash.com/@isaacmsmith?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Isaac Smith</a> on <a href="https://unsplash.com/s/photos/data-science?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Table of Contents</h3><ol><li><a href="#82c1"><strong>Introduction</strong></a></li><li><a href="#15fa"><strong>Regression</strong></a><strong><br></strong><a href="#200b"><strong> a. Linear Regression</strong></a></li><li><a href="#f69a"><strong>Extensions of Regression</strong></a><strong><br> a. </strong><a href="#2d72"><strong>Multiple Regression</strong></a><strong><br> b. </strong><a href="#c8a4"><strong>Logistic Regression</strong></a></li><li><a href="#dc62"><strong>Python Examples<br></strong></a><strong> a. </strong><a href="#30c0"><strong>Linear Regression</strong></a><strong><br>b. </strong><a href="#0401"><strong>Multiple Regression</strong></a><strong><br>c. </strong><a href="#aea4"><strong>Logistic Regression</strong></a></li></ol><h3>Introduction</h3><p>Welcome to the 2nd installment of our Overview of Machine Learning (Part 1 can be found <a href="https://medium.com/99p-labs/an-overview-of-machine-learning-part-1-7784ba3b3540">here</a>)! In this post, we will go deeper into the mechanisms of machine learning by doing a deep dive into regression. If at any point you find yourself getting overwhelmed but want to keep learning, I highly suggest checking out this great YouTube playlist on Machine Learning from <a href="https://youtube.com/playlist?list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF">StatQuest</a>!</p><h3>Regression</h3><p>Regression is one of the simplest and most easily extendable methods of machine learning. While there are many different types of regression — some of which we will cover in this post — at its core, regression is simply a way of using some variable (or combination of variables) to predict another variable.</p><p><strong><em>Linear Regression</em></strong></p><p>The most common and familiar application of regression is Linear Regression, which can be defined with the familiar formula y = mx + b where:</p><ul><li>y is the dependent variable to be predicted</li><li>m is the slope</li><li>x is the independent variable used to predict y</li><li>b is the y-intercept</li></ul><p><strong>Example:</strong> You are a biologist studying wombats in the wild. Your research goal is to determine which factors contribute to wombat size. Your hypothesis is that <strong>Wombat Size</strong> (y) can be accurately predicted by the <strong>Amount of Grass</strong> it eats (x). You collect data on ten wombats and plot it on a line graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBbmz913N5ntnCZmIPjY9g.png" /><figcaption>Sample data was collected on ten wombats, documenting their Size and Amount of Grass Eaten</figcaption></figure><p>Just by eyeballing the chart, you can probably tell that your hypothesis has some truth to it — as grass eaten increases, so does wombat size. Unfortunately, eyeballing won’t be accepted as a valid analysis method when publishing your research. To formalize our findings, we need to:</p><ol><li>Define a trendline through the data, which can be defined by using the y = mx + b formula</li><li>Define a way to quantify how well that line fits our actual data points</li><li>Determine the statistical significance of our goodness-of-fit measure</li></ol><p>For the first step, we can just draw a random line through the points, like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*23yKBJEmENPbEdQguX5tsQ.png" /><figcaption>Fitting a random regression line to our observed (x, y) data points.</figcaption></figure><p>Next, we figure out how well this line fits our data by calculating a goodness-of-fit measure. One of the most common metrics for calculating the goodness-of-fit is <strong>R²</strong>, defined by the formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JJ8kFvHtZDxaqQaiuefKHA.png" /></figure><p>So, to get our goodness-of-fit measure, we first need to calculate Sum of Squares around the Mean (<strong><em>SS_mean</em></strong>) and Sum of Squares around the Fit (<strong><em>SS_fit</em></strong>). Although these terms seem complicated, we are really just calculating the variation in <strong>Wombat Size</strong> when taking into account <strong>Grass Eaten</strong> (<strong><em>SS_fit</em></strong>) and the variation in <strong>Wombat Size</strong> on its own (<strong><em>SS_mean</em></strong>). Then we just divide the two to obtain a ratio (<strong>R²</strong>) that tells us how well our regression line fits the data.</p><p>At its core, <strong>R² </strong>is a way to see how well our model does at predicting the data. The closer<strong> R² </strong>is to 1.0, the better our model is at predicting y-variable. While the previous sentence is true, there are lots of nuances to the statement, so make sure to read through the<strong> </strong><a href="#8c96"><strong><em>Limitations of R²</em></strong></a><strong> </strong>section below for a full explanation. For even more detail, check out <a href="https://www.youtube.com/watch?v=2AQKmw14mHM">this video</a> from StatQuest.</p><p><strong><em>Calculating SS_mean</em></strong></p><p>The <strong><em>SS_mean </em></strong>is a measure of the raw variation in our data points, in our case <strong>Wombat Size</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*LYv6I1bB6DszYnveCXenVA.png" /></figure><p>Calculate the distance from each point to the mean of all of the points, like so:</p><ol><li>Calculate the distance from each point to the line like so:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*Kn3Sa8Feg8WkCWj-0hbwBQ.png" /><figcaption>Comparing observed y values to the mean of those y values (we are ignoring the x-axis)</figcaption></figure><p>2. Square each of those values to remove negatives</p><p>3. Sum up all of the values</p><p><strong><em>Calculating SS_fit</em></strong></p><p>The <strong><em>SS_fit </em></strong>measures how well our regression line fits the data and is simply the total of all squared distances from the observed data to our regression line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*jcPYCFReAOiYJMD-EOzR7A.png" /></figure><ol><li>Calculate the distance from each point to the line, like so:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*ZoyZFHnFg8xZSeKWa1kiCw.png" /><figcaption>Comparing the observed y values to those predicted by our random line. This difference is called a residual.</figcaption></figure><p>2. Square each of those values to remove negatives</p><p>3. Sum up all of the values</p><p><strong><em>Calculating R²</em></strong></p><p>Now that we have <strong><em>SS_fit </em></strong>and <strong><em>SS_mean</em></strong>, we can calculate our goodness-of-fit measure, <strong>R²</strong>. In this case, let’s say that our random line produced an <strong>R²</strong> of 0.51. This means that when using our model (the regression line defined by y = mx + b), <strong>Grass Eaten</strong> explains 51% of the variation in <strong>Wombat Size</strong>. Another helpful way of thinking about <strong>R²</strong> is as the following ratio:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7K68uPLKMrkeZdDuXmI7zA.png" /></figure><p><strong><em>Rinse and Repeat</em></strong></p><p>If we were doing this by hand, we would repeat the above process — generate a random line, calculate <strong><em>SS_fit </em></strong>and <strong><em>SS_mean</em></strong>, calculate <strong>R²</strong> — many times. Eventually, we would find a model which optimizes <strong><em>SS_fit</em></strong>, thereby giving us our line of best fit (y = mx + b) and its corresponding <strong>R²</strong> value. In our wombat example, these values work out to be:</p><p>y = 0.815x + 43.69</p><p><strong><em>SS_fit </em></strong>= 149.2</p><p><strong><em>SS_mean </em></strong>= 359.6</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*2gpk-9rrnqwjcJ0fOS7G4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ST0aWcyeFNuqnsH_lrh8ag.png" /><figcaption>Data and calculations used to obtain the above</figcaption></figure><p><strong><em>Limitations of R²</em></strong></p><p>While <strong>R²</strong> is a very useful metric, it is important to be aware of its limitations and how to determine the significance of your <strong>R²</strong> value.</p><p>Example: Imagine that we have two points on a graph, one at (5, 10), and the other at (10, 20). We can draw a line perfectly connecting these two points, resulting in a <strong><em>SS_fit </em></strong>of 0 (all observed minus predicted values are zero) and <strong><em>SS_mean </em></strong>of 50 ((20–15)² + (10–15)²) = 50). Plugging these into our <strong>R²</strong> equation, we get <strong>R²</strong> = (50–0) / (50) = 100%.</p><p>That <strong>R²</strong> value means that using that x-variable, we can explain 100% of the variation in the y-variable. But since any two random points will give you an <strong>R²</strong> of 100% and our goal is to use our model to predict future data, <strong>R²</strong> can’t be used in a vacuum. To solve this problem, we can find the statistical significance of the <strong>R²</strong> value by finding its <em>p-value</em>.</p><p><strong><em>Finding R²’s p-value</em></strong></p><p>Before we start, let’s make sure we understand what a <em>p-value</em> is. At its core, a <em>p-value</em> gives us a measure of confidence in the results from some statistical analysis. The closer the <em>p-value</em> is to 0, the more confidence we have in our result. This is because a <em>p-value</em> can be interpreted as the percentage of time we expect to see an incorrect result given some experiment or analysis. For a more detailed explanation of p-values check out <a href="https://www.youtube.com/watch?v=vemZtEM63GY">this video</a> from StatQuest.</p><p>To start, we start off by choosing our <em>alpha</em> level, which is the <em>p-value</em> that we’ll use as our cutoff. We set this value to 0.05. A resulting <em>p-value </em>of less than 0.05 will mean that our <strong>R²</strong> is significant.</p><p>Recall that above we defined <strong>R²</strong> with the following ratio:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tm19XQf4jCoqA4WjBD8fkw.png" /></figure><p>The <em>p-value</em> of <strong>R²</strong> comes from something called the <em>F-statistic</em>, which is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PJggZxvrwbj3bcho9Tjkqg.png" /></figure><p>The equation seems a lot more complicated than it is, so let’s break it down. To start, let’s ignore the second half of both the numerator and denominator. Once we do that, notice that the numerators in the <strong>R²</strong> and <strong><em>F</em></strong> equations are the same, and the denominator in the <strong><em>F</em></strong><em> </em>equation is something that we’ve already calculated — the Sum of Squares around the Fit, or <strong><em>SS_fit</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hcGUNXyqaTmW9ij0SQ7MlA.png" /></figure><p>The rest of the equation deals with an important concept, <strong><em>degrees of freedom</em></strong>, which is out of scope for this blog post. For now, we can simply substitute <strong><em>p_fit </em></strong>and <strong><em>p_mean </em></strong>with the number of parameters in the fit and mean calculations. <strong><em>p_fit </em></strong>refers to the number of parameters estimated in our regression line, defined by y = mx + b. Since we are estimating two parameters — slope and y-intercept — <strong><em>p_fit </em></strong>equals 2. For our mean line, we only had to use parameter, the y-intercept, because the average of the y-values had no slope. Therefore, <strong><em>p_mean </em></strong>equals 1. Finally, <strong><em>n</em></strong> is equal to the number of total data points, which in our case is 10. Plugging all of this into the <strong><em>F</em></strong> equation, we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*vPLMMHDZaY6B0JtRuw2cgw.png" /></figure><p>The <a href="https://en.wikipedia.org/wiki/F-distribution"><strong><em>F</em> distribution</strong></a> is defined by two parameters <strong><em>df_1</em></strong> and <strong><em>df_2</em></strong>. These are the two numbers we used in our calculation, which we defined as (<strong><em>p_fit — P-mean</em></strong>) and (<strong><em>n — p_fit</em></strong>). We can obtain the <em>p-value</em> by using one of many <a href="https://stattrek.com/online-calculator/f-distribution"><strong><em>F</em> distribution</strong> calculators</a> found online, with <strong><em>df_1 </em></strong>= (2–1) = 1, <strong><em>df_2 </em></strong>= (10–2) = 8, and <strong><em>F</em></strong> = 11.3. Given these values, the probability of observing a value of F that is greater than or equal to 11.3 is 0.00991. Since this <em>p-value</em> is less than our <em>alpha</em> of 0.05, we can say that our <strong>R²</strong> is significant.</p><h3>Extensions of Regression</h3><p>Now that we understand simple linear regression, we can easily extend that understanding to multiple and logistic regression.</p><p><strong><em>Multiple Regression</em></strong></p><p>Recall that the equation used for linear regression has two parameters (slope and y-intercept) and is defined by y = mx+b. For multiple regression, we simply add more variables to our equation. Since we are also estimating each of those additional variables, they will have a slope associated with them. Here is what a <em>n-level</em> multiple regression equation looks like:</p><p>1 Independent Variable: y = mx + b</p><p>2 Independent Variables: y = m_1x + m_2z + b</p><p>3 Independent Variables: y = m_1x + m_2z + m_3k + b</p><p><em>n</em> Independent Variables: y = m_1x + m_2z + m_3k … + m_<em>n{var} </em>+ b</p><p>Example: Consider our wombat example used in the linear regression section. In our research, we happened to also collect information on <strong>Wombat Age</strong>, so we want to see if adding that variable will be useful for our model. Now, our data looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*ABQxPwHQgM38FfOVEul3fw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uKkJUcjLCv_ydtkCXfyEbg.png" /></figure><p>Now instead of a 2D graph, we have a 3D one which also means that we are estimating three parameters: the y-intercept, slope associated with the x variable, and slope associated with the z variable (b, m_1, m_2, respectively). Our equation for finding <strong>Wombat Size</strong> is now:</p><p><strong>Wombat Size</strong> = m_1(<strong>Grass_Eaten</strong>) + m_2(<strong>Wombat_Age</strong>) + b</p><p>We follow the same process as in the linear regression section:</p><ol><li>Define a plane through the data, which can be defined by our equation y = m_1x + m_2z + b</li><li>Quantify how well that plane fits our actual data points using <strong>R² <em>adjusted R²</em></strong></li><li>Determine the statistical significance of <strong>R² </strong>and <strong><em>adjusted R²</em></strong></li></ol><p><strong><em>Adjusted R²</em></strong></p><p>As we add more variables to our equation, we need to be careful because those variables increase the likelihood that random chance will inflate our <strong>R²</strong> value and result in an overfitted model. To deal with this problem, we use a term called <strong><em>adjusted R², </em></strong>which adjusts for the number of terms used in the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*GyG49aCzN60UWP79AnQLIQ.png" /></figure><p>If we already know <strong>R²</strong>, it is easy to calculate <strong><em>adjusted R²</em></strong>. In the above equation, <em>n</em> is the sample size, and <em>k</em> is the number of independent regressors (slopes).</p><p>There is an important distinction in interpreting the <strong><em>adjusted R²</em></strong> versus regular <strong>R²</strong>. While <strong>R²</strong> assumes that all independent variables explain the variation in the dependent variable, <strong><em>adjusted R²</em></strong> tells you the percentage of variation explained by only the independent variables which actually affect the dependent variable. As such, the <strong><em>adjusted R²</em></strong> will decrease when you add useless variables and increase when you add useful variables but will always be less than or equal to <strong>R²</strong>.</p><p><strong><em>Logistic Regression</em></strong></p><p>Logistic regression is similar to linear regression in that they both try to predict some y-variable. In linear regression, that y-variable is continuous, while in logistic regression, it is categorical. As such, logistic regression is commonly used in classification-based machine-learning tasks.</p><p>To understand logistic regression, let’s continue with our wombat example; first, we must change our y-variable. The dependent variable will now be two categories: <strong>Overweight</strong> (OW) and <strong>Not Overweight</strong> (NOW).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bvEZL7y-j1yCm9bul-0wtg.png" /></figure><p>Next, instead of fitting a straight line to our data, we fit an S-shaped logistic function to the data. Unlike linear regression, our predicted y-value can only be between <strong>0</strong> and <strong>1</strong>.</p><p>General Case (Sigmoid function):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5U74xweaR-L-7GtWEBVQ-g.png" /><figcaption>The general case, with a Sigmoid curve between 0 (Category 1) and 1 (Category 2)</figcaption></figure><p>Our Example (NOW = 1.0 and OW = 0.0):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCjQJmiGWR78uOvfzBYtlA.png" /></figure><p>The curve represents the probability of being in one of the categories based on the value of the independent variable, <strong>x</strong> (<strong>Grass_Eaten</strong>, in our case). We can create a rule that states: “If the probability that a wombat is <strong>Overweight </strong>is ≥ 50%, classify that wombat as <strong>Overweight</strong>”. Just like in the linear regression example, we are predicting <strong>Weight_Class </strong>based on <strong>Grass_Eaten</strong>. However, because we are fitting a curve and not a straight line, we cannot calculate squared residuals and, therefore, can’t calculate <strong>R²</strong>. Instead, logistic regression generates the curve using maximum likelihood.</p><p>The goal of maximum likelihood is to fit the optimal curve to the data. To do this, we need to find a curve whose parameter values (mean and standard deviation, for example) maximize the likelihood that, using the curve, we can observe what is actually in our data. In short, we want our approximation (the curve) to most accurately represent our data. The details of maximum likelihood are out of scope for this blog post, but check out <a href="https://www.youtube.com/watch?v=XepXtl9YKwc">this video</a> if you want to learn more!</p><p>Logistic regression, much like its linear cousin, can be extended to have any number of independent variables. Unlike in linear regression, however, we can’t easily compare simple and complex models. In the Python Examples section, we’ll go into a more detailed example of logistic regression.</p><h3>Python Examples</h3><p><strong><em>Linear Regression</em></strong></p><p>Dataset: <a href="https://www.kaggle.com/datasets/unsdsn/world-happiness/download?datasetVersionNumber=2">World Happiness Report 2015–2019</a> (info page <a href="https://www.kaggle.com/datasets/unsdsn/world-happiness">here</a>)</p><p>We will be working with the <em>2019.csv </em>file from the above dataset. To start, import all required libraries and read in the file.</p><pre>import pandas as pd<br>import numpy as np<br>from sklearn.linear_model import LinearRegression<br>import seaborn as sns<br>import matplotlib.pyplot as plt</pre><pre># Read in the 2019.csv file<br>df = pd.read_csv(&#39;/content/sample_data/2019.csv&#39;) # NOTE: adjust your path as needed</pre><p>We’ll start off by looking at and exploring the data. We have a total of 156 rows and 9 columns with the following breakdown:</p><ul><li>8 numeric</li><li>1 string (Country or Region)</li></ul><p>There are no NA or null values in any of our columns; they are all complete.</p><pre>print(f&#39;Rows: {df.shape[0]}&#39;)<br>print(f&#39;Columns: {df.shape[1]}\n&#39;)<br>print(f&#39;Data Types:\n{df.dtypes}\n&#39;)<br>print(f&#39;Completeness % by column:\n{df.count() / df.shape[0] * 100}&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*6OSut6WQY5sd57Yfg1wMxA.png" /></figure><p>Now that we understand the structure of the data and know that there are no missing values, we can move forward with our analysis.</p><p>Our goal is to create a model to predict World Happiness Score. We’ll start by creating a linear regression model and then move on to a more complicated multiple regression model. In the end, we will be able to compare the two and decide which variables have the biggest impact on predicting the Score.</p><p>We’ll start by looking into the correlation between our variables. This will hopefully give us some insight and understanding into the relationships within this dataset.</p><pre>fig = plt.figure(figsize = (16, 9))<br>mask = np.triu(np.ones_like(df.corr(), dtype = bool)) # Mask the duplicate top half of the triangle<br>sns.heatmap(df.corr(), annot = True, mask = mask, vmin = -1, vmax = 1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ObDCg8JfUG-bGlzm0kupAA.png" /><figcaption>Correlation heatmap for our dataset</figcaption></figure><p>From the correlation heatmap above, we can see that <strong>Overall rank</strong> and <strong>Score</strong> and almost perfect correlated, meaning that they are co-linear. As such, they provide us with the exact same information about the data, so keeping both could skew our results. We will remove the <strong>Overall rank</strong> and re-run the correlation heatmap before moving further</p><pre>df = df.drop([&#39;Overall rank&#39;], axis = 1)<br><br>fig = plt.figure(figsize = (16, 9))<br>mask = np.triu(np.ones_like(df.corr(), dtype = bool)) # Mask the duplicate top half of the triangle<br>sns.heatmap(df.corr(), annot = True, mask = mask, vmin = -1, vmax = 1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a7rxOXJ2pO9jZjsJHoWChA.png" /><figcaption>Correlation heatmap after dropping the <strong>Overall rank</strong> variable</figcaption></figure><ul><li><strong>Score</strong> has a high correlation to <strong>GDP per capita</strong>, <strong>Social Support</strong>, and <strong>Healthy life expectancy</strong></li><li><strong>GDP per capita</strong> has a high correlation to <strong>Healthy life expectancy</strong> and <strong>Social Support</strong></li><li><strong>Social support</strong> has a high correlation to <strong>Healthy life expectancy</strong></li></ul><p>Logically, these findings make sense; people with more money, support, and higher life expectancy tend to be happier in general.</p><p>Next, we will run a few different linear regression models using the variables that are highly correlated with <strong>Score</strong> which we identified above.</p><pre># Linear Regression model for GDP Per Capita<br>gdp_lm = LinearRegression().fit(df[[&#39;GDP per capita&#39;]], df[&#39;Score&#39;])<br>print(f&quot;GDP per capita R2: {round(gdp_lm.score(df[[&#39;GDP per capita&#39;]], df[&#39;Score&#39;]), 2)}&quot;)<br>print(f&quot;GDP per capita equation: Score = {round(gdp_lm.coef_[0], 2)} * (GDP per capita) + {round(gdp_lm.intercept_, 2)}\n&quot;)<br><br># Linear Regression model for Social Support<br>social_lm = LinearRegression().fit(df[[&#39;Social support&#39;]], df[&#39;Score&#39;])<br>print(f&quot;Social support R2: {round(social_lm.score(df[[&#39;Social support&#39;]], df[&#39;Score&#39;]), 2)}&quot;)<br>print(f&quot;Social support equation: Score = {round(social_lm.coef_[0], 2)} * (Social support) + {round(social_lm.intercept_, 2)}\n&quot;)<br><br># Linear Regression model for Healthy Life Expectancy<br>life_lm = LinearRegression().fit(df[[&#39;Healthy life expectancy&#39;]], df[&#39;Score&#39;])<br>print(f&quot;Healthy life expectancy R2: {round(life_lm.score(df[[&#39;Healthy life expectancy&#39;]], df[&#39;Score&#39;]), 2)}&quot;)<br>print(f&quot;Healthy life expectancy equation: Score = {round(life_lm.coef_[0], 2)} * (Healthy life expectancy) + {round(life_lm.intercept_, 2)}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*-q8SS5N3cr_M9iMmjLRlrw.png" /></figure><p>We’ll use these models to get the predicted values and plot them against the actuals. We do this by adding a _Predicted column for each of our three models, which will give us the <strong>Score</strong> value our model would spit out for each actual value of x. Each of the plots below shows the original data, along with its corresponding model’s predicted regression line.</p><pre>df[&#39;GDP_Predicted&#39;] = gdp_lm.predict(df[[&#39;GDP per capita&#39;]])<br>df[&#39;Social_Predicted&#39;] = social_lm.predict(df[[&#39;Social support&#39;]])<br>df[&#39;Life_Predicted&#39;] = life_lm.predict(df[[&#39;Healthy life expectancy&#39;]])</pre><pre>fig, ax = plt.subplots(nrows = 1, ncols = 3, figsize=(25, 8))<br><br>fig, ax = plt.subplots(nrows = 1, ncols = 3, figsize=(25, 8))<br><br>sns.regplot(x = &#39;GDP per capita&#39;, y = &#39;GDP_Predicted&#39;, data = df, color = &#39;red&#39;, fit_reg = True, marker = &quot;&quot;, ax = ax[0])<br>sns.regplot(x = &#39;GDP per capita&#39;, y = &#39;Score&#39;, data = df, color = &#39;red&#39;, fit_reg = False, ax = ax[0]).set(title = &#39;GDP Per Capita vs Score with LM Regression Line&#39;)<br><br>sns.regplot(x = &#39;Social support&#39;, y = &#39;Social_Predicted&#39;, data = df, color = &#39;green&#39;, fit_reg = True, marker = &quot;&quot;, ax = ax[1])<br>sns.regplot(x = &#39;Social support&#39;, y = &#39;Score&#39;, data = df, color = &#39;green&#39;, fit_reg = False, ax = ax[1]).set(title = &#39;Social Support vs Score with LM Regression Line&#39;)<br><br>sns.regplot(x = &#39;Healthy life expectancy&#39;, y = &#39;Life_Predicted&#39;, data = df, color = &#39;blue&#39;, fit_reg = True, marker = &quot;&quot;, ax = ax[2])<br>sns.regplot(x = &#39;Healthy life expectancy&#39;, y = &#39;Score&#39;, data = df, color = &#39;blue&#39;, fit_reg = False, ax = ax[2]).set(title = &#39;Healthy Life Expectancy vs Score with LM Regression Line&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Z773RZk4u2_ktovl3nFOw.png" /></figure><p>Given the above, it seems like all three of our variables would do an equally good job at predicting future values of the <strong>Score</strong> variable. However, we are still missing information about the significance of R². To get this (and much more) useful information, we will use the <a href="https://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.OLS.html"><em>statsmodels</em></a><em> </em>package. While the syntax for running the code will be different, all of the under-the-hood calculations are the same as what we did above.</p><p>For all of our analysis going forward, we will use an <em>alpha </em>level of 0.05, which will be our cutoff for significance. As a reminder, if we obtain a <em>p-value</em> less than our <em>alpha </em>then that result can be considered to be significant.</p><pre>import statsmodels.api as sm<br><br>y = df[&#39;Score&#39;]<br><br># Linear Regression model for GDP Per Capita<br>x = df[[&#39;GDP per capita&#39;]]<br>x = sm.add_constant(x)<br>gdp_lm_2 = sm.OLS(y, x).fit()<br>print(f&#39;GDP Per Capita Model Summary:\n{gdp_lm_2.summary()}\n&#39;)<br><br># Linear Regression model for Social Support<br>x = df[[&#39;Social support&#39;]]<br>x = sm.add_constant(x)<br>social_lm_2 = sm.OLS(y, x).fit()<br>print(f&#39;Social Support Model Summary:\n{social_lm_2.summary()}\n&#39;)<br><br># Linear Regression model for Healthy Life Expectancy<br>x = df[[&#39;Healthy life expectancy&#39;]]<br>x = sm.add_constant(x)<br>life_lm_2 = sm.OLS(y, x).fit()<br>print(f&#39;Healthy life expectancy Model Summary:\n{life_lm_2.summary()}\n&#39;)</pre><p><strong>GDP Per Capita </strong>model summary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*eHnLQhKnxDi3J3x3Kwp_xw.png" /></figure><p><strong>Social Support </strong>model summary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*mjRNWM8OBKlfvmPgE2x63w.png" /></figure><p><strong>Healthy Life Expectancy </strong>model summary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*y8WbsFekz0BrgoyzYRYmJg.png" /></figure><p>The summary information reinforces our original conclusion — all three variables do an equally good job of predicting the outcome variable — because all three models have a very low <em>p-values </em>(P &gt; |t| in the above screenshots) associated with their R².</p><p><strong><em>Multiple Regression</em></strong></p><p>In our linear regression example, we came up with a somewhat unsatisfactory answer to our original question — which variable is the best predictor of <strong>Score</strong>? Since we got three equally good answers to our question, let’s see if we can get a better model if we include multiple independent variables in our model instead. We’ll continue on with the <a href="https://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.OLS.html"><em>statsmodels</em></a><em> </em>package since its summary function output is extremely useful in interpreting model results.</p><p>Reminder: Our model’s equation will take the form</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*oVaAO7hGmRgBbDjb4jiCbg.png" /></figure><p>where:</p><ul><li>y = <strong>Score</strong> variable</li><li>b₀ = y-intercept</li><li>b₁ = slope associated with <strong>GDP per capita</strong></li><li>x₁ = <strong>GDP per capita</strong> variable</li><li>b₂ = slope associated with <strong>Social support</strong></li><li>x₂ = <strong>Social support</strong> variable</li><li>b₃ = slope associated with <strong>Healthy life expectancy</strong></li><li>x₃ = <strong>Healthy life expectancy</strong></li></ul><pre>import statsmodels.api as sm<br><br>y = df[&#39;Score&#39;]<br><br># Multiple Regression model for GDP Per Capita + Social Support + Healthy Life Expectancy<br>x = df[[&#39;GDP per capita&#39;, &#39;Social support&#39;, &#39;Healthy life expectancy&#39;]]<br>x = sm.add_constant(x)<br>mr_model = sm.OLS(y, x).fit()<br>print(mr_model.summary())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*JvEQ583Uz5LA4HsEz1M2Lw.png" /></figure><p>Given the output, we can now put together the equation representing our model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dcZZo6SRr58TRT90LdRjTA.png" /></figure><p>Looking at the <em>p-values</em> (P &gt; |t| in the above screenshot) for our independent variables, they are all less than 0.05, so we can say that they are all significant and should all be kept. This model also has an adjusted R² of 0.721, meaning that it does a better job of predicting the <strong>Score</strong> variable than any of our previous models.</p><p>Although we could stop here and accept this as our final model, we should still explore using a model that includes all of our independent variables:</p><pre>y = df[&#39;Score&#39;]<br><br># Multiple Regression model for GDP Per Capita + Social Support + Healthy Life Expectancy + Freedom to make life choices + Generosity + Perceptions of corruption<br>x = df[[&#39;GDP per capita&#39;, &#39;Social support&#39;, &#39;Healthy life expectancy&#39;, &#39;Freedom to make life choices&#39;, &#39;Generosity&#39;, &#39;Perceptions of corruption&#39;]]<br>x = sm.add_constant(x)<br>mr_model2 = sm.OLS(y, x).fit()<br>print(mr_model2.summary())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*HMI7Dw-J2_jW41PwZQTY3g.png" /></figure><p>This model actually happens to be better than our previous model, with an adjusted R² of 0.77! Before moving on, however, note that the model summary tells us that given this model, <strong>Generosity </strong>and <strong>Perceptions of Corruption</strong> are not significant predictors at <em>alpha</em> = 0.05. So what happens if we re-run the above but remove the least significant predictor, <strong>Generosity</strong>?</p><pre>y = df[&#39;Score&#39;]<br><br># Multiple Regression model for GDP Per Capita + Social Support + Healthy Life Expectancy + Freedom to make life choices + Perceptions of corruption<br>x = df[[&#39;GDP per capita&#39;, &#39;Social support&#39;, &#39;Healthy life expectancy&#39;, &#39;Freedom to make life choices&#39;, &#39;Perceptions of corruption&#39;]]<br>x = sm.add_constant(x)<br>mr_model2 = sm.OLS(y, x).fit()<br>print(mr_model2.summary())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*AyV2IUEuI0W6YYPxvuxpSg.png" /></figure><p>We actually end up with an equally good but less complex model! Notice that when we took away <strong>Generosity</strong> as a predictor and re-ran the model without it, <strong>Perceptions of Corruption </strong>are now significant at <em>alpha</em> = 0.05.</p><p>Since all of our predictors are significant and our adjusted R² of 0.77 is the best of all the models we’ve run, we can finally state that the following multiple regression model does the best job at predicting <strong>Score</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*uEOmkc_L4ZGDiFaxOng7qg.png" /></figure><p><strong><em>Logistic Regression</em></strong></p><p>Now that we have a good grasp on simple linear regression and multiple linear regression let’s move on to logistic regression. Recall that logistic regression is similar to linear regression in that they both are trying to predict some y-variable. In linear regression, that y-variable is continuous, while in logistic regression, it is categorical.</p><p>Since our dataset doesn’t contain any categorical variables, we’ll simply create one by dividing the <strong>Score</strong> column into two categories: High and Low. Scores below the mean will be categorized as “Low,” and those above the mean will be categorized as “High.” Now the goal of our model has changed from predicting the exact values of <strong>Score</strong> to predicting the correct <strong>Score Category</strong>.</p><p>Just like with linear regression, logistic regression models can be single or multi-variate (meaning one or more predictors in the model). We’ll start with the most complex model, the one that includes all predictors.</p><pre>from sklearn.linear_model import LogisticRegression<br>from sklearn.metrics import classification_report, confusion_matrix<br><br># Create the categorical variable, Score Category, by separating the Score column into High and Low categories<br>df[&#39;Score Category&#39;] = pd.cut(df[&#39;Score&#39;],  bins = [np.min(df[&#39;Score&#39;]), np.mean(df[&#39;Score&#39;]), np.max(df[&#39;Score&#39;])], labels = [&#39;Low&#39;, &#39;High&#39;], include_lowest = True)<br><br># Logistic Regression model for GDP Per Capita + Social Support + Healthy Life Expectancy + Freedom to make life choices + Generosity + Perceptions of corruption<br>x = df[[&#39;GDP per capita&#39;, &#39;Social support&#39;, &#39;Healthy life expectancy&#39;, &#39;Freedom to make life choices&#39;, &#39;Generosity&#39;, &#39;Perceptions of corruption&#39;]]<br>y = df[&#39;Score Category&#39;]<br><br>lr_model = LogisticRegression().fit(x, y)</pre><p>Now we can extract a few important pieces of information that will tell us how well this model performed. First, we look at the predicted probabilities. These are the probabilities that the model has assigned of each row being classified as either “High” (0) or “Low” (1).</p><pre>print(lr_model.predict_proba(x))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/1*i6o8-GnUm4gUf4r1s8jwtg.png" /></figure><p>The output above is a numpy array, which can be read as follows: “The model predicts that row 1 of our original dataframe goes into the “High” category with a probability of 93.4%”. This likelihood is given for every row in our original dataframe.</p><p>We can also extract the actual classifications for each row output by the model:</p><pre>print(lr_model.predict(x))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*tvlada07-djxuQ_UsDmLPw.png" /></figure><p>As well as the overall accuracy score (number of observations classified correctly):</p><pre>print(round(lr_model.score(x, y), 2))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*WsBFGxgzcn7trxhNyG0ZSw.png" /></figure><p>To get more detail on the accuracy of the model, we can look at the confusion matrix in two ways:</p><pre>print(confusion_matrix(y, lr_model.predict(x)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/75/1*h5InFsuur1lPSnUSRhLAQA.png" /></figure><pre># Confusion Matrix Heatmap<br>cm = confusion_matrix(y, lr_model.predict(x))<br><br>fig, ax = plt.subplots(figsize = (16, 9))<br>ax.imshow(cm)<br>ax.grid(False)<br>ax.xaxis.set(ticks = (0, 1), ticklabels = (&#39;Predicted High&#39;, &#39;Predicted Low&#39;))<br>ax.yaxis.set(ticks = (0, 1), ticklabels = (&#39;Actual High&#39;, &#39;Actual Low&#39;))<br>ax.set_ylim(1.5, -0.5)<br><br>for i in range(2):<br>    for j in range(2):<br>        ax.text(j, i, cm[i, j], ha = &#39;center&#39;, va = &#39;center&#39;, color = &#39;black&#39;, size = 50)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*oPaDNLTTx9B55jsUGB1fGA.png" /></figure><p>Both outputs tell us the same thing:</p><ul><li>There were 67 rows correctly classified as High</li><li>There were 67 rows correctly classified as Low</li><li>There were 12 rows incorrectly classified as High</li><li>There were 10 rows incorrectly classified as Low</li></ul><p>Note that this is essentially a decomposition of our overall accuracy measure of 0.86 since (67+67) / (67+67+10+12) = 0.86.</p><p>Finally, we can look at a comprehensive summary of our model:</p><pre>print(classification_report(y, lr_model.predict(x)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*E2PQbtv5W9eMQShLNH33pg.png" /></figure><p>Just like with the multiple linear regression example, we can now investigate other models or accept this one as our final one. There are many tuning parameters that you are able to change, but going over all of them would be out of scope for this blog post. Please see here for documentation that goes into more detail for all available parameters:</p><ul><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html">Logistic Regression</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression">Simple &amp; Multiple Linear Regression</a></li></ul><h3>Blog 3 (Decision Trees) — Coming Soon!</h3><p>In this post, we did a deep dive into one of the most foundational mechanisms behind machine learning — regression. We learned the theoretical underpinnings behind regression and extended that understanding to multiple linear and logistic regression. Finally, we applied that knowledge in three practical Python examples. In our next post (out soon!), we’ll cover another foundational mechanisms, decision trees, following a similar format.</p><p>If you’d like to discuss this topic with us, you can comment on this post or connect with us on <a href="https://www.linkedin.com/company/99p-labs/">LinkedIn</a> or <a href="https://medium.com/99p-labs">Medium</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f991281932e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/99p-labs/an-overview-of-machine-learning-part-2-all-about-regression-2f991281932e">An Overview of Machine Learning — Part 2: All About Regression</a> was originally published in <a href="https://medium.com/99p-labs">99P Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Overview of Machine Learning — Part 1]]></title>
            <link>https://medium.com/99p-labs/an-overview-of-machine-learning-part-1-7784ba3b3540?source=rss-a36222816d6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7784ba3b3540</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Luka Brkljacic]]></dc:creator>
            <pubDate>Tue, 22 Nov 2022 19:06:06 GMT</pubDate>
            <atom:updated>2022-12-04T01:38:10.164Z</atom:updated>
            <content:encoded><![CDATA[<h3>An Overview of Machine Learning — Part 1</h3><p>Written by: Luka Brkljacic and Nithin Santhanam</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*f8GMDf5aab9UgGskD2N4zg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@santesson89?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Andrea De Santis</a> on <a href="https://unsplash.com/s/photos/machine-learning?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3><strong>Table of Contents</strong></h3><ol><li><a href="#8917"><strong>Introduction</strong></a></li><li><a href="#4ec0"><strong>Why Machine Learning?</strong></a></li><li><a href="#0f39"><strong>Machine Learning Process</strong></a></li><li><a href="#9ba1"><strong>Machine Learning Methodologies</strong></a><strong><br></strong><a href="#7a05"><strong> a. Supervised Learning<br></strong></a><strong> </strong><a href="#5cee"><strong>b. Unsupervised Learning<br></strong></a><strong> </strong><a href="#62a7"><strong>c. Reinforcement Learning</strong></a></li><li><a href="#843a"><strong>Machine Learning Approaches</strong></a><strong><br></strong><a href="#74c8"><strong>a. Supervised Learning</strong></a><strong><br></strong><a href="#6482"><strong>b. Unsupervised Learning<br></strong></a><strong> </strong><a href="#ad3e"><strong>c. Reinforcement Learning</strong></a></li><li><a href="#d21a"><strong>Choosing a Machine Learning Solution</strong></a><strong><br> </strong><a href="#dcc0"><strong>a. Cheat Sheet Guide</strong></a></li><li><a href="#0b0f"><strong>Example Problems Related to Our Team’s Work</strong></a></li><li><a href="#a740"><strong>Final Thoughts</strong></a></li><li><a href="#c457"><strong>Resources &amp; Further Reading</strong></a></li></ol><h3>Introduction</h3><p>At 99P Labs, we focus on various types of research and innovation that will help push the vision of future mobility forward. Our team’s specific focus is to gain a better understanding of mobility data. Our hypothesis is that understanding mobility data better will lead to the generation of insights, and those insights will improve decision-making. A big component of this data comes in the form of telematics, or vehicle data. Throughout our work, we have made frequent use of numerous machine learning (ML) concepts, which has led us to the creation of this post. As such, the goal of this blog post is to introduce readers to some of these important fundamental machine learning concepts. This is the first post in a series on machine learning. Each subsequent post will go into greater detail on the various subtopics.</p><h3>Why Machine Learning?</h3><p>As human beings, we tend to be pretty proud of our brains — and usually with good reason. Throughout millions of years of evolution, our brains have evolved to be excellent pattern-finding machines. From connecting the way the forest looks so that we can hunt specific prey to identifying sales trends in modern businesses, humans have always sought out patterns in our world (<a href="https://en.wikipedia.org/wiki/Apophenia">even when there objectively aren’t any there!</a>). However, with the advent of Big Data we reached a point where humans struggled to keep track of and process that huge amount of information; this is where ML came in. Machine learning is <a href="https://www.ibm.com/cloud/learn/machine-learning#toc-what-is-ma-qhM6PX35">defined as</a> “a branch of artificial intelligence (AI) and computer science which focuses on the use of data and algorithms to imitate the way that humans learn, gradually improving its accuracy.” Essentially, ML is simply a tool that enables humans to learn and find the same kinds of patterns we are used to finding, in a data context that is too large for our brain to accurately comprehend and process. ML is a tool that extends our ability to find solutions to complex problems.</p><p>As a general rule, ML requires lots of data to be able to provide useful and accurate insights. Because we know that ML is essentially an extension of our brain in finding patterns in data, then any problem with a small amount of data can’t magically be solved by throwing a ML algorithm at it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fO1jodFh0E6hl7Aru4etVg.png" /></figure><p>If you are trying to predict whether a patient will have a heart attack or not and you only have two prior patients, then the likelihood of accurately predicting the health outcome for a new patient based on that data is very low — you simply don’t have enough data to accurately determine a pattern among what you have already seen. In the visual below, given the data we have (left table), it would be impossible to accurately predict the health outcome for the new patient (right table).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PU3RkQ2BH-FMXrrJMy6sEw.png" /></figure><p>However, if you had 1,000 patients and their heart attack outcomes and saw that 90% of those who had a heart attack were Males between the ages of 40 and 65 with a poor diet, then when a new patient matching that description walked in you would be able to confidently say “based on my prior experience, your characteristics very closely match the heart attack group, therefore you are at a high risk of having a heart attack.” The above example is something doctors have been doing since the advent of modern medicine, but think about how ML can be useful in this context: If instead of having just three columns (Sex, Age, and Diet information) for those 1,000 patients imagine we had 200 different columns of information describing everything from their income and homeowner status to their genomic indicators. Finding a pattern among 200 different indicators would be impossible for any human to do, but relatively straightforward for a ML algorithm. In the example above, ML provides a potential solution to the issue of accurately predicting an elevated risk of heart attack.</p><h3>Machine Learning Process</h3><p>The nature of machine learning is such that not any one specific model is ideal for every situation. Each algorithm has its own strengths and weaknesses based on the methodology behind it. While there isn’t a one-size-fits-all approach, all ML algorithms follow a generalized process to problem solving.</p><ul><li>We have some data, <strong><em>X</em></strong>. Our goal is to predict some kind of <em>outcome </em>based on <em>features </em>contained in <strong><em>X. </em></strong><br>– The outcome could be either quantitative (like a stock price or # of products sold) or categorical (patient had/didn’t have a heart attack)<br>– The features are just the columns contained in the dataset that describe something about what we are trying to predict. Things like Age, Sex, Diet, Smoker/Non-Smoker, Ethnicity and Income are all features which could be used to predict someone’s risk of heart attack, for example.</li><li>We split the data contained in <strong><em>X</em></strong> into two sub-datasets called Training (<strong><em>Xtrain</em></strong>) and Testing (<strong><em>Xtest</em></strong>).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jyHHSRb8o2BnmXlNPxPHLQ.png" /></figure><ul><li>We use the training subset to “teach” the computer about all of the patterns and rules inherent in this dataset<br>– The way in which this is done results in the variety of different machine learnings methods but they all boil down to this basic principle — we are using some algorithm on the training data to understand rules and patterns within that data</li><li>Now that the computer has learned about the data in the training subset, we want to evaluate how well it has learned by applying that knowledge to the testing subset</li><li>If the way we taught the data using <strong><em>Xtrain</em></strong> was good, then we will also have good results on <strong><em>Xtest<br></em></strong>– However, just like with any kind of learning, we can run into two very big problems:<br>– Mismatch between <strong><em>Xtrain</em></strong> and <strong><em>Xtest</em></strong> — if you are taking an English Literature class and you study <em>Pride &amp; Prejudice</em> for 6 months but the final exam is on <em>A Tale of Two Cities</em>, it doesn’t matter how well you understand <em>Pride &amp; Prejudice</em> because that isn’t what the test is about. This is why it is important to properly separate<em> </em><strong><em>X</em></strong>into <strong><em>Xtrain</em></strong> and<strong> <em>Xtest</em></strong>.<br>– Overfitting <strong><em>Xtrain </em></strong>resulting in poor performance on <strong><em>Xtest</em></strong> — imagine you are still taking the same English Literature class mentioned above. This time, you know for sure that the test will be on <em>Pride &amp; Prejudice</em>, which you are reading. However, you decide to really focus on the first 5 chapters of the book and can answer any question about them; but the test happens to be on the whole book! Although you can do really well on the questions about the first 5 chapters, you won’t do very well on the rest of the test. This is why it’s important to ensure that you are not hyper-focused on <strong><em>Xtrain</em></strong>, as you could be too specialized to do well on a more generalized <strong><em>Xtest</em></strong></li><li>Resources:<br>– <a href="https://youtu.be/Gv9_4yMHFhI">A Gentle Introduction to Machine Learning</a><br>– <a href="https://youtu.be/EuBBz3bI-aA">Machine Learning Fundamentals: Bias and Variance</a></li></ul><h3>Machine Learning Methodologies</h3><p>Machine learning methodologies can be separated into a few different types. These types are defined in similar ways that we would define any problem-solving methodology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nnyqtFTWzg9kHiqU6OoMiw.png" /></figure><h3>1. Supervised Learning</h3><p>The first type is called supervised learning. This framework describes the case where an algorithm is trained on a “training data” containing a desired outcome variable that you wish to model. We are then able to use this trained model to predict that outcome variable on the “test data” and because we have the actual outcome variable values of that test data, we can then compare how accurately our model was able to predict the outcome.</p><p>Imagine that we are posed a problem of determining whether an image is a cat or a dog. Our goal is to develop a methodology for accurately being able to identify whether images are cats or dogs in some standardized, consistent manner. First we would do research about many dog and cat breeds that exist and understand both visually and behaviorally what makes one different from one another. We gradually amass answers to various questions about potential differentiating factors between the animals. How large do cats grow to be? Which animal is faster? What colors are commonly found in each animals coat? We even stop at pet stores to meet some of them and interact with them in person. However, after all this research, it is difficult for us to know how well we are able to differentiate between the two pets. In order to objectively test ourselves on the knowledge we’ve gained, we decide to take a quiz. The quiz contains 50 images of either dogs or cats and we have to determine which animal the image most closely resembles. For the final validation that we are seeking, we then look at the answer key on the last page of the quiz to evaluate our performance compared to the correct answers. This training and evaluation process is what’s known as supervised learning in the machine learning domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5N1Y_uE1bIEfehLC-jR2yQ.png" /></figure><h3>2. Unsupervised Learning</h3><p>The second type is known as unsupervised learning. This describes the case where an algorithm identifies unique differences in a dataset without being trained. Additionally, these problems don’t contain outcome values that you are attempting to predict, you are simply separating the data into groups or identifying several unique groups within the data that are defined by similar feature values.</p><p>Let’s say we’re given a problem of separating a group of animal pictures into groups. We aren’t familiar with any of these animals and we are tasked with simply separating them into groups that feel appropriate. We see that there’s a group of animals that have beaks and wings so we separate these animals into one group. We see another group of animals that seem to have fins so we separate these animals into another group. Lastly, we find another group of animals that has ears and we make these animals our last group. These 3 distinctions seemed to have cleanly partitioned the starting group of animals into reasonable groupings! That being said, we aren’t sure if these are logically correct groupings and it’s possible that there exists animals that have both ears and wings or fins and beaks. How would we decide which group to place these animals? The problem presented has not even given us a criteria on how we are expected to form these groups. This problem is an example of unsupervised learning. We don’t know the correct answers but we are identifying 3 groups to place these animals based on some differentiating traits relative to each other. In order to develop useful insights from the groups we created, we would need to go through a separate process of understanding the meaning and significance behind the groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sJOqfaNQgD1HXdymfjsQUA.png" /></figure><h3>3. Reinforcement Learning</h3><p>The third type is known as reinforcement learning, which is based around an agent (subject) interacting with the surrounding system (environment). It works by training the agent to make decisions without explicitly defining how to handle specific situations. This process is done by presenting an agent with different scenarios and using risk vs. reward systems as the learning mechanism. The agent can then create a decision-making process that learns to optimize the risk-reward decision. This learning method is almost exactly how reinforcement learning is done in real life psychology in a variety of contexts like pet training or instinct learning such as determining if an object is hot or cold. The agent does not have a high level view of how it should make decisions, it simply makes decisions based on its prior learnings and the current state of the environment.</p><p>In nature, when we observe ant systems, we can identify some extremely intelligent and nuanced behavior from groups of ants. However, when observing individual ants, we notice that their behavior is not very advanced. This occurs because ants are trained to react with other ants and their environment and so they make super simple decisions individually but in aggregate in an ant system, combinations of these individual ant’s behavior can create sophisticated systemic behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oH9UuJEynYTZwwZ_ZlD1cA.png" /></figure><h3>Machine Learning Approaches</h3><p>In addition to the different methodologies discussed above, it is useful to identify the ideal approach to a machine learning problem in order to obtain our solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AKFd4AnmxzAr8Pe-5vREag.png" /></figure><h3>Supervised Learning</h3><p>Within supervised learning, we can generally separate problems into Regression and Classification solutions.</p><p><strong>Regression</strong></p><ul><li><strong>Overview<br></strong>Regression is a statistical technique that identifies relationships between independent variables and numeric dependent variables. People often do regression analysis mentally without recognizing the process they’re using. For example, let’s say we go on a website and find that the price of 5 oranges is about $5. We then go to our local grocery store and see oranges priced at $1.50 per orange, and we want to determine difference in this pricing versus the prices you saw on the website earlier. Mentally we would create the equation: <br><em>Price of 5 Oranges = (1.50) x (Number of Oranges)</em><br>We then plug in the value we saw on the website: <em>(1.50) x (5 Oranges)</em> and we arrive at 5 Oranges at this grocery is $7.50! That’s a $2.50 overall price difference from the website!<br>Furthermore we also have a general equation that we mentally created that would determine the price for any amount of oranges we wanted to purchase in the form of:<br><em>Oranges = (Orange Price) x (Number of Oranges)</em><br>This methodology is commonly used in creating a best guess line or curve that weights features according to importance to project a corresponding outcome value. We can plot the equation on a graph as shown below to further understand how factors interact with one another and which are influential or insignificant.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*4qLk-YI-aI2nC24iC8wjyg.png" /></figure><ul><li><strong>Applications<br></strong>This technique is commonly used in number heavy disciplines such as finance, medicine, physics or chemistry related fields, etc. This is the case because in these fields, the most crucial variables to explore are numeric and have both statistical and logical correlation (relationship) with one another. Since most businesses revolve around supply and demand you use regression to find the right combination between production, inventory, and consumer demand to maximize revenue. Generally any problem where optimization of numeric variables is a focus can apply regression techniques.</li><li><strong>Examples</strong><br>Stock Market Forecasting, Demand Analysis, Tumor Growth over time, Weather Forecasting</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2rKMCEVdgybgHeU96GemFg.png" /></figure><p><strong>Classification</strong></p><ul><li><strong>Overview<br></strong>Classification is a form of supervised learning that identifies relationships between independent variables and categorical dependent variables. This approach aims to classify or label data by weighting features to determine which outcome label most accurately represents the context of the independent variables. Classification is defined as the process of recognition, understanding, and grouping of objects and ideas into preset categories a.k.a “sub-populations.” You would use this process to either predict categorical outcomes or predict the probability of a set of predetermined outcomes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*GLsavJxB-XvOnZdeYYHvxA.png" /></figure><ul><li><strong>Applications<br></strong>This approach is used when you have a good understanding of the domain space of your problem and you’re able to segment your outcome variable into different groups. These groups can take the form of binary outcomes such as yes/no, True/False, Win/Lose or they can take the form of multi-class outcomes like predicting which breed a dog is based on some known characteristics. The goal of this approach is to identify and recognize patterns, understand the interactions behind those patterns and then group data into categories using your understanding of these patterns.</li><li><strong>Examples<br></strong>Win-Loss sports betting, Fraud Detection, Image Recognition, Likelihood of event occurring, Diagnostics</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2zgTyDZtbsSoqwY0AnF8Rw.png" /></figure><h3>Unsupervised Learning</h3><p>Within unsupervised learning, we can separate problems into Clustering and Dimensionality Reduction solutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AJPyw1yRliECv2WmS9hZGw.png" /></figure><p><strong>Clustering</strong></p><ul><li><strong>Overview<br></strong>Clustering is an approach that aims to create unique groups within a dataset based on feature similarities. Clustering is often used to describe data and separate it into relatively unique, discrete groups. Since clustering is an unsupervised method, it is not evaluating whether any data is “correct”, “good” or “bad” at an absolute level. Clustering is evaluating similarity between values relative to other values in the data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*imNWEfwmTDGJh0nZQdC-8Q.png" /></figure><ul><li><strong>Applications<br></strong>Clustering is a very useful technique when you want to make uninformed groupings of data. Since clustering is an unsupervised technique, it is often used for exploratory purposes when dealing with unfamiliar data. It is very difficult (if not impossible) to control the result of clustering techniques; furthermore clustering has no ability to contextualize what it sees. For this reason, often you will see data scientists use clustering in order to identify patterns and then with these patterns they will aim to either find or create a dataset that will enable supervised learning techniques.</li><li><strong>Examples<br></strong>Uninformed Pattern Recognition, Outlier Detection, Curve Smoothening</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3I2ovu_uSSNGh6ET_WyKJA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y-TlMi_qisc3yJiu0L86og.png" /></figure><p><strong>Dimensionality Reduction</strong></p><ul><li><strong>Overview<br></strong>Simply put, dimensionality reduction is a technique to reduce the number of variables in a dataset while preserving as much information as possible. It aims to reduce a n-dimension feature space into feature space with a dimension &lt; n where all the features are unique and provide unique value. These techniques aim to simplify the modeling of complex problems, reduce processing required, eliminate redundancy and weigh truly unique features appropriately instead of having correlated independent variables unevenly influence model outputs. In essence, these methods reduce the number of variables in the data by trading off a decrease in accuracy for an increase in simplicity.</li><li><strong>Applications<br></strong>Dimensionality reduction is often used to simplify a problem both conceptually and logistically. Reducing dimensions allows you to identify the most important components of the data, enabling you to filter out noisy factors and data. Dimensionality reduction techniques are advantageous to data professionals working with massive datasets, performing data visualization and analyzing complex data. They also enable the process of data compression, allowing the data to take up less storage space as well as reducing computation times.</li><li><strong>Examples<br></strong>Projecting Data onto a subspace, Metric Creation, Noise Filtering</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CXrzd4WBM5WhCbTFZUEJyw.png" /></figure><h3>Reinforcement Learning</h3><ul><li><strong>Overview<br></strong>Reinforcement learning is a unique approach to machine learning systems. Supervised learning approaches problems from a Top-Down perspective, while Unsupervised learning approaches problems from a bottom up perspective. Reinforcement learning approaches problems with a combination of Top-Down and Bottom-Up perspectives. It’s a method of contextualizing a problem with properties of a system environment while not explicitly defining or even knowing the end result model behavior. The goal is to teach agents through a risk vs. reward system on how to make a series of decisions given some amount of factors and then allow the agent to interact with it’s environment to see what happens. The agent will attempt to optimize the reward system at every decision it needs to make and you may be surprised at results you find.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EG4a47NcLxNj6UHGAXzUrA.png" /></figure><ul><li><strong>Applications<br></strong>You would mainly use reinforcement learning when attempting to model a single agent behavior in its environment. Reinforcement learning could be appropriate in any problem where you understand aspects of an agent but you want to know how that agent interacts with its environment. Often, in this approach the goal is to use your knowledge of small parts to understand how the net interactions of various parts affect the whole from a big picture perspective. This could mean creating a model of a cell that is being attacked by a virus and then projecting how a patient’s immune system will be affected holistically. Reinforcement learning is useful when there is no correct way to perform a task or an expected outcome, yet there are strict rules in the environment or problem context that the agent must abide by. This approach can be very effective in opening your mind to possibilities that hadn’t occurred to you initially because it helps you understand the cascading effects of infinite environmental interactions.</li><li><strong>Examples<br></strong>Traffic Simulation, Ant Colony, System Behavior Found in Nature, General Simulation</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*9OPTOw-6sBuiGCNu8OMLzA.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLu56xVlZ40M&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLu56xVlZ40M&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FLu56xVlZ40M%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0b99fb07bf0cd453a83b017e26c65692/href">https://medium.com/media/0b99fb07bf0cd453a83b017e26c65692/href</a></iframe><h3>Choosing a Machine Learning Solution</h3><p><strong>Question to Ask Yourself to Identify the Ideal ML Solution:</strong></p><ul><li>Does the data contain an outcome variable?<br> — Identify which type of ML methodology type would be appropriate given the data provided: Supervised vs. Unsupervised techniques</li><li>Are you attempting to describe the state of the data? Or are you attempting to understand the data to enable future projections and predictions?<br> — Exploratory vs. Predictive Modeling</li><li>What data is available to you?<br>— Is there a way to enrich the data?<br> — If your problem falls into the predictive modeling category but you lack an outcome variable, is it possible to contextually add one?</li><li>What are the data types of the independent variables? What is the data type of the outcome variable if one exists?<br> — Identify which machine learning approach would be most appropriate</li></ul><p>We can convert the questions above into the decision tree flow diagram below to help visualize the process. This figure should be a good resource to refer to when approaching a new problem. It will help you think through the questions you need to ask and guide you in the right direction for choosing an appropriate ML solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HZn1eIH_eFVPpyGIRqX6vA.png" /></figure><h3><strong>Example Problems Related to Our Team’s Work</strong></h3><p>In order to further help in understanding some of the machine learning topics discussed, let’s walk through a few example problems in our research domain to determine how we would decide to approach the problems!</p><p><strong>Problem 1:</strong></p><p>Our company wants to understand where people commute to understand what potential vehicle modifications may be necessary. The hope is that we can improve the designs of our vehicles to accommodate the terrains found in those popular areas. Given Latitude and Longitude data on where we see the most trips taken, create a machine learning solution framework that would be appropriate for this goal.</p><ul><li><strong><em>Does the data contain an outcome variable?</em></strong><em><br></em>We do not have an expected outcome variable to predict. We are not even aware of the potential outcomes possible</li><li><strong><em>Are you attempting to describe the state of the data? Or are you attempting to understand the data to enable future projections and predictions?</em></strong><em><br></em>Problem is exploratory in nature; we just need to see the distinctly unique areas present in the data</li><li><strong><em>What data is available to you?</em></strong><em><br></em>Location data in the form of (Latitude, Longitude) is available to us</li><li><strong><em>Is there a way to enrich the data?</em></strong><em><br></em>It is potentially possible to enrich this data using reverse geocoding. Reverse geocoding is the process of using (Lat, Lon) pairs to retrieve information about the area where the coordinates reside. For example, if points within the dataset are located around 39.9612° N, 82.9988° W, this would be around the city of Columbus, Ohio. We can then retrieve any relevant information about Columbus like the population, Economic status, etc.</li><li><strong><em>If your problem type falls into the predictive modeling category, but you lack an outcome variable, is it possible to contextually add one?</em></strong><em><br></em>We could use this newly obtained information as another independent feature in our problem or we can potentially create a problem where one of our variables becomes the outcome variable</li><li><strong><em>What are the data types of the independent variables? What is the data type of the outcome variable if one exists?</em></strong><em><br></em>The independent variables, Latitude and Longitude, are numeric and represent points on the Earth. This type of data is conducive to getting the distance between points and methods revolving around relative proximity to other points</li></ul><p><strong>Potential Solution:</strong></p><ol><li>Using the vehicle location data, we want to identify high concentrated areas of vehicles because this would indicate that vehicles travel to these areas frequently.</li><li>In the current problem state we do not have an outcome variable, and our task is aimed at exploring the data we have in our attempt to describe it. Unsupervised Learning techniques are available to us.</li><li>We can use clustering techniques to determine groups of unique areas.</li><li>Then in those areas identify terrains.</li><li>Finally use the above findings to suggest potential vehicle modifications centered around terrain information to our design teams.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*jDiNwHsKcNCIrdr6hH0gkg.png" /></figure><p><strong>Problem 2:</strong></p><p>We are given vehicle trip data containing fuel consumption, distance and time data. We want to create a model of customer fuel efficiency. This model will be used to evaluate past and future data to give drivers efficiency scores based on how economically they are driving.</p><ul><li><strong><em>Does the data contain an outcome variable?</em></strong><em><br></em>The data contains an outcome variable. Supervised methodologies are available to us</li><li><strong><em>Are you attempting to describe the state of the data? Or are you attempting to understand the data to enable future projections and predictions?</em></strong><em><br></em>We want to use this model to create a generalized solution to evaluate gas consumption for driver data in the past and future or in other words we want to enable future projections and predictions</li><li><strong><em>What data is available to you?</em></strong><em><br></em>Fuel Consumption, Distance and Time data are available to us</li><li><strong><em>Is there a way to enrich the data?</em></strong><em><br></em>Data contains a desired outcome variable based on the problem defined. The independent variables have a logical connection to the outcome variable. Enrichments could be possible depending on external public data sources available like weather, census or reverse geocoding but initially may not be necessary. We could potentially enrich the data later on if our model performs poorly</li><li><strong><em>If your problem type falls into the predictive modeling category, but you lack an outcome variable, is it possible to contextually add one?</em></strong><em><br></em>Based on problem definition, at this time we do not need to add an outcome variable</li><li><strong><em>What are the data types of the independent variables? What is the data type of the outcome variable if one exists?</em></strong><em><br></em>All independent variables, Distance and Time, are numeric. The outcome variable, Fuel Consumption, is numeric.</li></ul><p><strong>Potential Solution:</strong></p><ol><li>Using vehicle trip data we want to determine expected fuel consumption for users given the distance and time they traveled in order to evaluate how drivers perform relative to their expectations.</li><li>In the current problem definition, we have a fuel consumption outcome variable that would be very indicative of driver performance regarding fuel efficiency.</li><li>Both our independent variables and outcome variable are numeric, with strong logical correlation leading us to believe that regression would probably be appropriate.</li><li>We will create a regression model to predict fuel consumption, given Distance traveled and Time as independent variables using the data available as training data.</li><li>The regression model will predict the expected fuel consumption vs the actual fuel consumption that a driver accrues over a trip.</li><li>Using the expected vs. actual we will take the difference between the 2 where negative values will indicate that a driver used less fuel than expected and positive values will indicate that a driver used more fuel than expected.</li><li>Using that result we can create a standardized metric based on the percentile their results fall into. This metric evaluates the driver based on what percentile of fuel efficient they fall into.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j5tyJD57onHGqQ9YPErC_A.png" /></figure><h3>Final Thoughts</h3><p>Machine learning can be an intimidating when first attempting to step into the space. There’s so many so many components and options that it becomes difficult to know where to begin when trying to find a solution to a problem. One goal of this blog post is to break down the general machine learning philosophy into logical paths on when to use certain techniques versus others. Generally speaking for any problem, you can significantly narrow down or even quickly arrive at a solution if you’re able to identify the methodologies available to you and the potential approaches given the data properties and your goals. It is also worthwhile to note that every problem is different, and context behind every problem matters. Additionally, it is possible to change the context in front of you to one that fits your problem more appropriately. This could mean reframing the problem, understanding your data better, adding more data sources, transforming your data or any other of the several ways to modify your problem. Our hope is that you can use the “Questions to Ask Yourself to Identify the Ideal ML Solution”, along with the diagram guide section to find the best ML solutions for a problem.</p><h3>Blog 2 — Coming Soon!</h3><p>If you’re interested in diving into more aspects of machine learning, this is the first of several blog posts that will be apart of our machine learning guide series. Post 2 will be out shortly and will cover some of the topics discussed in this post but in more detail.</p><p>If you’d like to discuss this topic with us, you can comment on this post or connect with us on <a href="https://www.linkedin.com/company/99p-labs/">LinkedIn</a>.</p><h3>Resources &amp; Further Reading</h3><p>99P DevPortal: <a href="https://developer.99plabs.io/home/">https://developer.99plabs.io/home/</a></p><p>Beyond Algorithms: <a href="https://www.routledge.com/Beyond-Algorithms-Delivering-AI-for-Business/Luke-Porter-Santhanam/p/book/9780367613266">https://www.routledge.com/Beyond-Algorithms-Delivering-AI-for-Business/Luke-Porter-Santhanam/p/book/9780367613266</a></p><p>StatQuest: <a href="https://statquest.org/">https://statquest.org/</a></p><p>3Blue1Brown: <a href="https://www.3blue1brown.com/#lessons">https://www.3blue1brown.com/#lessons</a></p><p>An Introduction to Statistical Learning e-book: <a href="https://hastie.su.domains/ISLR2/ISLRv2_website.pdf">https://hastie.su.domains/ISLR2/ISLRv2_website.pdf</a></p><p>The Elements of Statistical Learning e-book: <a href="https://hastie.su.domains/Papers/ESLII.pdf">https://hastie.su.domains/Papers/ESLII.pdf</a></p><p>Computer Age Statistical Inference e-book: <a href="https://hastie.su.domains/CASI_files/PDF/casi.pdf">https://hastie.su.domains/CASI_files/PDF/casi.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7784ba3b3540" width="1" height="1" alt=""><hr><p><a href="https://medium.com/99p-labs/an-overview-of-machine-learning-part-1-7784ba3b3540">An Overview of Machine Learning — Part 1</a> was originally published in <a href="https://medium.com/99p-labs">99P Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>