<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Geoffrey Hung on Medium]]></title>
        <description><![CDATA[Stories by Geoffrey Hung on Medium]]></description>
        <link>https://medium.com/@geoffreyhung?source=rss-cf2c9ab57bf7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*7z06yX275J_KY2QY.</url>
            <title>Stories by Geoffrey Hung on Medium</title>
            <link>https://medium.com/@geoffreyhung?source=rss-cf2c9ab57bf7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 06:10:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@geoffreyhung/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Getting the Monkey Off My Back: How I Stopped Babysitting Coding Agents]]></title>
            <link>https://medium.com/@geoffreyhung/getting-the-monkey-off-my-back-how-i-stopped-babysitting-coding-agents-41b6b620b145?source=rss-cf2c9ab57bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/41b6b620b145</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[workflow]]></category>
            <dc:creator><![CDATA[Geoffrey Hung]]></dc:creator>
            <pubDate>Fri, 23 Oct 2026 16:30:46 GMT</pubDate>
            <atom:updated>2026-10-23T16:30:46.584Z</atom:updated>
            <content:encoded><![CDATA[<p>TL;DR: My setup is on GitHub: <a href="https://github.com/G-Hung/oh-my-coding-harness">oh-my-coding-harness</a>. The harness tools are free, and you can adopt only what your project needs; There’s a copy-and-paste adoption prompt at the bottom.</p><p>— — — — — — — — — — — — — — — — — — — — — — — —</p><p>A year ago, AI could write code, but I still had to manage the implementation: review diffs, run tests, catch regressions, and keep nudging it along. Then coding agents got dramatically better. They could handle entire features, yet somehow I was still spending my afternoons reviewing thousands of lines of code.</p><p>Naturally, I went down the rabbit hole: skills, spec-driven development, context engineering, multi-agent orchestration, Ralph loops. Some useful ideas, but also plenty of new things to learn and maintain. I kept asking myself: <strong>Can I get out of the implementation loop without taking on a second job supervising agents?</strong></p><p>There’s an old management analogy about the monkey on your back. Someone brings you a problem, you say, “I’ll take a look,” and suddenly their problem becomes yours. Coding agents are surprisingly good at this: “All done! Here are 30 changed files and 4,000 lines of code for you to review.”</p><p>When did I become my agent’s human subagent? 🐒</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pB83PTHXAEtj1a4G" /></figure><p>I wanted a different arrangement: we agree on the design, the agent handles implementation and verification, and I come back to a working result. Not a cheerful “Done!” followed by three hours of cleanup.</p><p>So I built a lightweight, low-cost harness around my coding agents, with automated checks, persistent architectural context, independent reviews, and regular codebase sweeps. The goal isn’t to make agents perfect. It’s to make them responsible for catching and fixing their own mistakes.</p><h3>I only want to be involved at two points</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qzogVtknody2VLh4" /><figcaption>Orange: human attention. Green: agent work. Schematic, not a benchmark.</figcaption></figure><p>An agent that works for an hour without interrupting me is more useful than one that finishes in twenty minutes but needs help every five. <strong>The real bottleneck isn’t execution time. It’s human attention.</strong></p><p>I want to make two decisions: what we’re building and whether the result is good enough. Everything between those points should be the agent’s responsibility, unless it encounters a genuine design decision.</p><p>Most interruptions are predictable: wrong behavior, failing tests, broken integrations, forgotten design constraints, or bugs the agent should have caught. Every time I had to step in, I asked: <em>How could the agent have caught this without me?</em></p><p>That’s where the following five layers came from.</p><h3>Layer 1: Spend more human attention in Plan Mode</h3><p>Product decisions, architectural tradeoffs, and design constraints are worth discussing before implementation. The agent proposes an approach, challenges my assumptions, and we agree on what to build.</p><p>Even a dashboard filter raises design questions: Should its state live in the URL? What survives a refresh? Where should permissions be enforced? Which existing components should we reuse?</p><p>We also agree on what we’re <em>not</em> doing, which existing behaviors must remain intact, and how to verify success. I’d rather spend a few extra minutes here than discover, 4,000 lines later, that we built the wrong thing.</p><p>Once the direction is clear, implementation belongs to the agent. It only needs to come back when something materially changes the agreed design.</p><h3>Layer 2: Don’t make me run a command to discover a failure</h3><p>I got tired of asking whether the agent had run typecheck, regression tests, or even opened the app. These shouldn’t depend on reminders.</p><p>The harness adds checks at different stages: fast feedback during development, required checks before committing, and broader verification before handoff. For example, the TypeScript kit provides:</p><pre>npm run lint<br>npm run typecheck<br>npm run check:readme -- --staged<br>npm run check</pre><p>The first three run at pre-commit; pre-push runs npm run check. Project-specific builds, integration tests, E2E tests, and CI provide additional coverage.</p><p><strong>The important part is that failures go back to the agent, not to me.</strong> It reads the error, fixes the issue, and reruns the checks. There’s no reason to spend my attention catching errors TypeScript already knows about.</p><p>Of course, green checks don’t guarantee a working product. The agent also needs to launch the app, exercise affected flows in a browser, and inspect the UI, console, and API responses. Unit tests can pass while the actual user experience is broken.</p><h3>Layer 3: Skip the fancy memory system. Write better READMEs.</h3><p>Agents can read code, but they can’t always infer why it’s structured that way. Why was a module split? Why was a dependency avoided? Which interface must remain stable?</p><p>I keep those decisions close to the code. Submodules get short READMEs describing their responsibilities, boundaries, verification steps, and important design decisions.</p><p>I borrowed a tip from <a href="https://x.com/karpathy/status/2105819303471976479">Karpathy</a>: write in the spirit of <strong>ASD-STE100</strong>, the simplified English used in aerospace maintenance documentation. Short sentences, consistent terminology, minimal ambiguity.</p><p>But like any documentation, the challenge is keeping it current.</p><p>My solution is a small script that finds the nearest README for changed files. If the code changed but the README didn’t, it asks the agent whether the documented behavior or interface needs updating at pre-commit. The reviewer also checks whether changes respect documented module boundaries.</p><p>This isn’t about forcing documentation edits for every commit. It’s about preserving architectural reasoning so the next agent doesn’t need me to explain the decision history again.</p><h3>Layer 4: Fight magic with magic</h3><p>Tests catch known failure modes, but they won’t catch everything. Once implementation passes its checks, a separate reviewer agent examines the change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FUKrzkEMtCRRfO8k" /></figure><p>The implementer writes the code; the reviewer tries to torture it. Findings go back to the implementer, and fixes return to the <strong>same reviewer</strong> until they’re resolved. Otherwise, you get a committee of amnesiacs: each new reviewer finds something different, while nobody confirms whether the original problems were fixed.</p><p>A useful finding identifies what’s wrong, where it happens, and how to verify the fix. The main agent can delegate investigation or testing to subagents, but remains responsible for the outcome.</p><p>The review loop also provides a useful architectural signal. <strong>If a powerful coding agent needs five rounds of fixes for a seemingly simple feature, something deeper may be wrong.</strong> Perhaps the abstraction doesn’t fit, module boundaries are unclear, or the design introduces unnecessary coupling.</p><p>I don’t need to read thousands of lines to notice that friction. Repeated failures can point me toward structural problems worth investigating, even before I inspect the code.</p><p>The normal handoff is simple: what changed, which checks passed, what the reviewer found, and what was fixed. If I still catch a bug, I ask which missing test, rule, or design constraint would catch it next time.</p><h3>Layer 5: Defend the codebase, not just individual PRs</h3><p>Every PR can be reasonable, and the codebase can still become a mess.</p><p>One feature needs API retries, so the agent implements them. Another feature needs retries and creates a slightly different implementation. A third does the same. All three PRs pass review, but now the codebase has three retry mechanisms to maintain.</p><p>An individual reviewer might reasonably accept each change. The problem only becomes obvious when you examine them together.</p><p>That’s why I added a <strong>code-sweep</strong> as a regular task. It reviews recent changes across the repository, looking for duplicated logic, drifting module boundaries, inconsistent patterns, and architectural or security issues that become visible across PRs.</p><p>Together, these five layers protect individual tasks while keeping the codebase from drifting as changes accumulate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1AHNh6V-4ZDiE5gg" /></figure><h3>So do I still review every line of code?</h3><p>I used to think I had to. I’m responsible for the code, so naturally I should understand every function before merging it.</p><p>Then agents started generating code much faster than I could read it. A few thousand lines would arrive, and another task would finish while I was still reviewing the first. I’d become a very expensive single-threaded bottleneck.</p><p>The uncomfortable realization was that AI often writes better code than I would, even in areas I know well. Spending more time reading every line wasn’t necessarily producing better outcomes.</p><p>So for most noncritical, reversible changes, I focus on the plan and the actual result. I let automated checks and reviewers handle implementation details, then open the app in a development environment and verify what we agreed to build.</p><p>High-risk changes are different. Payments, security-sensitive logic, migrations, and core data models deserve closer inspection and step-by-step verification. <strong>The goal isn’t to eliminate human review. It’s to allocate human attention where it has the most value.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*0fWHUJKo3Xcg0PjP" /></figure><h3>Once one works, why not start another?</h3><p>Once one agent could reliably complete a substantial task without constant supervision, running multiple agents became practical.</p><p>Git worktrees give each task its own working directory and branch. I plan the first feature and send it off, then plan the next while the first runs. This only works because the agents can handle most implementation and review issues themselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P8vpyyHCrg1WLGgQ" /><figcaption>Schematic, not measured completion times.</figcaption></figure><p>Right now, five main agents is about my comfortable limit [If you count all their reviewers and subagents, I could argue I’m managing 50+ agents!] But five is roughly how many independent tasks I can comfortably track.</p><p>A substantial feature typically takes around 30–60 minutes to complete. By the time I’ve planned and launched the fifth, the first is often ready for inspection. On a good day, several features can move forward in a couple of hours without constantly pulling me back into implementation.</p><h3>Putting it all together</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*znjvC-XbOjbaHugX" /></figure><p>The setup has three parts: <strong>five layers of defense</strong> to catch mistakes and protect the architecture, <strong>worktrees</strong> to run independent tasks concurrently, and <strong>human judgment</strong> focused on planning, important tradeoffs, and final validation.</p><p>I still choose the direction and judge the results. What I’ve stopped doing is carrying every failed check, bug report, and implementation question between agents. That’s their work, not mine.</p><p>Most of the time, I barely notice the harness running. Checks execute automatically, reviewers follow their findings through to resolution, and agents fix issues without pulling me back into implementation.</p><p>I used to worry that my backlog was growing faster than my product. Now I’m starting to wonder where I’ll find enough meaningful ideas to keep the agents busy. But perhaps that’s the more interesting challenge: <strong>not how to build more features, but how to create more value with the time I’ve gained.</strong></p><h3>Want to give your agent a try?</h3><p>I’ve mostly built and tested this with OpenAI Codex, but the workflow isn’t tied to one product. Claude Code, Cursor, and other coding agents can adapt the same ideas.</p><p>Open Plan Mode (or the equivalent in your agent) and use this prompt:</p><pre>Read https://github.com/G-Hung/oh-my-coding-harness.<br>Inspect this repo and my project. Recommend useful workflows, guides, and checks to adopt or adapt, while preserving what already works. Follow the adoption guide and present one grouped plan before making changes. Leave the result uncommitted for review.</pre><p>As agents get smarter, I see less value in adding more skills just to tell them how to work. I’ve reduced my workflow to just two skills, backed by various self-contained guides that provide additional context. You’re free to modify both the skills and guides as needed, giving you maximum flexibility to adapt your workflow.</p><p>When a plan is ready to execute, run ship-cycle and let the agent handle implementation, testing, review, and fixes while you move on to something else:</p><pre>$ship-cycle execute</pre><p>After a productive day, run code-sweep to identify architectural drift and worthwhile refactors:</p><pre>$code-sweep</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41b6b620b145" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Dichotomy of Builder and Worker Mindsets]]></title>
            <link>https://medium.com/@geoffreyhung/the-dichotomy-of-builder-and-worker-mindsets-b4d5358c1748?source=rss-cf2c9ab57bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4d5358c1748</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[mindset]]></category>
            <category><![CDATA[side-project]]></category>
            <dc:creator><![CDATA[Geoffrey Hung]]></dc:creator>
            <pubDate>Fri, 27 Oct 2023 00:36:22 GMT</pubDate>
            <atom:updated>2023-10-27T07:10:44.687Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YSlumFkNznvIPFAE" /><figcaption>DALL.E-3: illustration of a seesaw balancing two figures: one as an architect (Builder) with blueprints, and the other as a craftsman (Worker) with tools, set against a backdrop of coding scripts and ChatGPT prompts.</figcaption></figure><p>The whole mini-series:</p><ol><li><a href="https://medium.com/@geoffreyhung/my-side-project-journey-building-an-llm-based-tool-dafe641f7830">My Side Project Journey: Building an LLM-based Tool</a></li><li><a href="https://medium.com/@geoffreyhung/navigating-the-subtle-reliability-quirks-of-chatgpt-9494276d8134">Navigating the Subtle Reliability Quirks of ChatGPT</a></li><li><a href="https://medium.com/@geoffreyhung/navigating-the-full-stack-llm-web-development-with-zero-frontend-knowledge-caaeb15c296f">Navigating the Full Stack LLM App Development with Zero Frontend Knowledge</a></li><li>The Dichotomy of Builder and Worker Mindsets</li></ol><h3>Background:</h3><p>As the narrative of my side project journey unfolds, each phase presents a plethora of lessons and reflections. From grappling with the reliability issues of ChatGPT to navigating the Stack LLM Web Development, the voyage has been a roller-coaster of trials, errors, and insights.</p><p>In this final piece of the series, I wish to explore the subtle but distinct divide between<strong> the Builder and Worker mindsets</strong>. These are the unseen forces — silent battles waged in my mind, not yielding tangible outputs like code or prompts, yet heavily influencing my productivity and decision-making.</p><p>I believe the most significant differences lie in how I perceive <strong>priority</strong> and <strong>ownership</strong>, and I’ve started to grasp why certain seemingly unreasonable decisions were made in some companies I worked for.</p><h3>Priority:</h3><blockquote>Being able to work on this project only during weekends, I’ve <strong>had to actively discern the </strong>core tasks and <strong>trim down my TODO list</strong>. This led me to ask myself different sets of questions, for example:</blockquote><blockquote><strong><em>Is the color not appealing?</em></strong></blockquote><p><strong>At work:</strong></p><ul><li>How do other products handle this?</li><li>What are the best practices and theories?</li><li>What options are available?</li></ul><p><strong>While building:</strong></p><ul><li>How much time can I allocate here?</li><li>Can this be fixed later? Does it matter at this stage?</li><li>What’s the bare minimum needed for user-friendliness?</li></ul><p><strong>In the end: </strong>Spent a few minutes adding some styles, as long as it’s clear, moved on.</p><blockquote><strong><em>Designing a new data schema?</em></strong></blockquote><p>At work:</p><ul><li>What are the best practices?</li><li>Will this data model serve other purposes?</li><li>Can it handle XX QPS?</li><li>How to go about migration/backup?</li></ul><p>While building:</p><ul><li>What’s the fallout of a poor design?</li><li>What would it cost to fix it later?</li></ul><p>In the end: The implications are significant, necessitating a thorough contemplation. [same conclusion but different rationales]</p><blockquote><strong><em>Prompt engineering?</em></strong></blockquote><p>At work: Ah, it doesn’t work?</p><ul><li>What are the best practices?</li><li>To meet the reliability requirement, we probably need to set up RAG to augment context, perhaps experiment with model fine-tuning or use LangChain to run a chain of calls for cross-validation.</li></ul><p>While building: Ah, it doesn’t work?</p><ul><li>Reliability requirement? It seems unattainable; can we modify the requirement?</li></ul><p>In the end: Abandoned weeks of code, shifted to a technically viable approach from scratch.</p><blockquote>There isn’t a clear-cut good or bad; it’s situational.</blockquote><blockquote>In worker mode, I tend to <strong>optimize given tasks</strong>, while in builder mode, I aim to <strong>deliver given the end goal</strong>.</blockquote><blockquote>The outputs from my worker mode are <strong>usually well-designed and professional but slower</strong>; my builder mode outputs are <strong>usually basic yet faster to deliver</strong>.</blockquote><p>It’s naive to deem one better than the other. Much like in soccer, everything is an opportunity for the midfielder, everything is a risk to the defender. Which one is correct? It hinges on the situation.</p><h3>Ownership:</h3><p>The second poignant lesson revolves around ownership. At work, unless things are critical, <strong>tasks can often be shared or offloaded</strong>, divided based on roles and responsibilities.</p><p>However, in a lean project team, there’s no one to share or offload tasks to. If you aim to launch it, you have to write the front-end code, even with zero knowledge about it. You have no boss to convince to add more headcounts to write the frond-end code</p><p>When the prompt fails again, it’s on you to devise a solution or abandon the project. You have no boss to explain why it doesn&#39;t work, explain everything to yourself and figure it out</p><h3>Mental conflict b/w two mindsets</h3><p>The initial phase of my side project was uncomfortable, with two conflicting mindsets clashing in my head. One fretted over <strong>the quality of work</strong> while the other stressed<strong> the limited weekend hours</strong> post the Costco grocery trip.</p><p>Over the past few months, whenever I hung out with friends, I frequently mentioned how tired I felt. This wasn’t a reference to physical fatigue, even though I hit the gym four times a week. Rather, it was a mental exhaustion stemming from the constant tug-of-war between these two mindsets.</p><h3>The balance</h3><p>Now, I’ve found some balance. The trick is to first focus on the overall product flow, identify the minimal necessary parts, and contemplate <strong>the simplest</strong> and <strong>most reliable</strong> implementation. Then, within that framework, strive to do as well as possible.</p><blockquote>Essentially: Identify the core problems, tackle just those, and nothing more.</blockquote><p>This way, both my builder and worker modes are content. The builder mode is satisfied with defining the minimum tasks for delivery, while the worker mode can indulge in optimizing within those defined bounds.</p><h3>Ending remarks</h3><p>In this mini-series, I’ve attempted to chronicle my journey and the lessons learned while developing my side project. If you’ve been kind enough to read through all the articles, you may have noticed that I’ve written extensively on ChatGPT reliability, less on web development, and even lesser on the mental conflict between the two mindsets.</p><p>However, I truly believe they are all equally important. The last one is probably even more critical than the other two combined, as it impacts my motivation and priorities</p><p>Other than writing down my journey, of course, I would love you to try out my side project, <a href="https://www.producthunt.com/posts/clarify-ai">Clarify AI</a> :P</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4d5358c1748" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Full Stack LLM Web Development with Zero Frontend Knowledge]]></title>
            <link>https://medium.com/@geoffreyhung/navigating-the-full-stack-llm-web-development-with-zero-frontend-knowledge-caaeb15c296f?source=rss-cf2c9ab57bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/caaeb15c296f</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[side-project]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Geoffrey Hung]]></dc:creator>
            <pubDate>Thu, 26 Oct 2023 15:31:36 GMT</pubDate>
            <atom:updated>2023-10-27T07:10:07.759Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JiY89YNBQ-lyFQbt" /><figcaption>DALL.E-3: one path filled with thorns and obstacles labeled ‘ChatGPT Challenges’ and the other, a smooth road with milestones labeled ‘LLM Web Development’.</figcaption></figure><p>The whole mini-series:</p><ol><li><a href="https://medium.com/@geoffreyhung/my-side-project-journey-building-an-llm-based-tool-dafe641f7830">My Side Project Journey: Building an LLM-based Tool</a></li><li><a href="https://medium.com/@geoffreyhung/navigating-the-subtle-reliability-quirks-of-chatgpt-9494276d8134">Navigating the Subtle Reliability Quirks of ChatGPT</a></li><li>Navigating the Full Stack LLM App Development with Zero Frontend Knowledge</li><li><a href="https://medium.com/@geoffreyhung/the-dichotomy-of-builder-and-worker-mindsets-b4d5358c1748">The Dichotomy of Builder and Worker Mindsets</a></li></ol><h3>Background</h3><p>As I continue to chronicle my journey of building my side project , <a href="https://www.producthunt.com/posts/clarify-ai">Clarify AI</a>— a tool aimed at transforming vague ideas into structured contexts using Language Models like ChatGPT. Last time, I shared my <a href="https://medium.com/@geoffreyhung/navigating-the-subtle-reliability-quirks-of-chatgpt-9494276d8134">nightmarish experience dealing with the reliability of ChatGPT</a>.</p><p>This time, I shifted my focus towards Full Stack LLM Web Development. In this domain, <strong>I don’t have many hard substances to offer since there’s already a plethora of great materials available out there. Nonetheless, what follows may serve as a sharing of my personal experiences.</strong></p><p>Compared to the reliability issues with ChatGPT, the coding aspect of building an LLM app turned out to be more straightforward, even enjoyable.</p><h3>Wrong workload estimation</h3><blockquote>Initially, I harbored <strong>the misconception that once the prompt prototype was ready, coding would be the most demanding and time-consuming part</strong>, estimating the effort ratio at around 8:2 or even 9:1. Reality, however, leaned more <strong>towards 5:5 or even 4:6. </strong>[translation: 50% more time spent on the project]</blockquote><p>The time spent on “prompt engineering” often felt like it was in a time warp. Many a night went by fine-tuning the wordings in instructions, running a few test cases, and revisiting the drawing board if things didn’t pan out. The cyclical nature of this process was starkly different from the more straightforward coding progression.</p><p><strong>This is why I stress again the caution against using free-form long text conversations unless it’s necessary in the previous article.</strong> The evaluation of prompts, coupled with the necessity to continually adjust messages based on API responses, can take a very long time.</p><p>Interacting with ChatGPT is forgiving since users can keep refining their dialogue to hone in on desired responses. However, if you use that in your product, the users may not want to refine that for you and get things back on track.</p><h3>Full stack web development experience</h3><p>Given a chance for a do-over, I’d dive into <a href="https://www.youtube.com/@codewithantonio">some good Next JS tutorials</a> on YouTube, follow along with one, and perhaps explore a couple more instead of <strong>just</strong> reading the docs [You still need to read!].</p><p>Trusting the popular choices is usually your best bet when you begin [Next JS, T3 stack, Vercel, Supabase], don’t over-research choosing b/w React, Vue, and Angular. The rest, I’d delegate to GPT-4…</p><p>Despite GPT-4’s awesome performance, <strong>I realized I could contribute to enhancing its utility</strong>. For instance, structuring code by breaking down the entire page into distinct conceptual components, annotated with comments, eased the process. Ideally, GPT-4 could tackle a component in one response (around 100–200 lines), enabling a piece-by-piece assembly and fine-tuning.</p><h3>The productivity equation has changed</h3><p>The traditional trade-off between <strong>delivery speed</strong> and <strong>code quality</strong> is redefined. With GPT-4 in the equation, <strong>a well-thought-out design accelerates the coding phase, rather than delaying it.</strong> A significant chunk of my time was invested in orchestrating the schema for data models throughout their lifecycle — understanding their content, managing changes, and planning for extensions.</p><blockquote>This paradigm nudges towards good practices: <strong>clear thinking, thorough documentation, and better structuring</strong>, over a mad dash to the deadline. Because the rush is meaningless compared to the generation speed of GPT-4.</blockquote><p>The emphasis on<strong> foundational knowledge</strong> has become more relevant. Guiding GPT-4 effectively requires a solid grasp of <strong>first principles</strong> across various domains, discerning what’s feasible or beneficial and why.</p><p>The landscape is shifting, diminishing the glitter of work experience and fancy tricks. In a past where all-star teams were a rare commodity and the best minds juggled multiple tasks, a change in the productivity equation can now amplify the output of the adept ones by <strong>2–3X</strong>.</p><h3>Tailwind CSS and complex interactions in React</h3><p>However, even with GPT-4 as a co-pilot, I still want more guidance in areas like Tailwind CSS and structuring complex interactions in React. Still no clear clue on how to tackle them effectively.</p><p>BTW, I just got the private Beta for v0.dev, it may be a game changer if it really delivers as expected, check it out: <a href="https://v0.dev/">v0 by Vercel</a></p><h3>Final thought</h3><p>I really try to think hard about what can I share in the web development area, but I can’t think of anything more important than</p><ol><li>paying $20 for ChatGPT Plus</li><li>watching some end2end YouTube projects to understand what a complete project looks like</li></ol><h3>Learning materials that I found SUPER helpful</h3><p><strong>Youtube:</strong></p><ul><li>e2e project examples: <a href="https://www.youtube.com/@codewithantonio">https://www.youtube.com/@codewithantonio</a></li><li>Some engineering practices: <a href="https://www.youtube.com/@WebDevCody">https://www.youtube.com/@WebDevCody</a></li></ul><p><strong>React:</strong></p><ul><li>Next JS examples: <a href="https://vercel.com/templates">https://vercel.com/templates</a></li><li>React Doc: <a href="https://react.dev/">https://react.dev/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=caaeb15c296f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Subtle Reliability Quirks of ChatGPT]]></title>
            <link>https://medium.com/@geoffreyhung/navigating-the-subtle-reliability-quirks-of-chatgpt-9494276d8134?source=rss-cf2c9ab57bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/9494276d8134</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[side-project]]></category>
            <category><![CDATA[prompt]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Geoffrey Hung]]></dc:creator>
            <pubDate>Thu, 26 Oct 2023 00:32:22 GMT</pubDate>
            <atom:updated>2023-10-27T07:09:44.216Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3RIrf4OzxFeXrKv_" /><figcaption>DALL.E-3: illustration of a roller coaster ride labeled ‘ChatGPT Reliability’, starting with an upward climb symbolizing initial success and then a sharp descent into challenges and quirks.</figcaption></figure><p>The whole mini-series:</p><ol><li><a href="https://medium.com/@geoffreyhung/my-side-project-journey-building-an-llm-based-tool-dafe641f7830">My Side Project Journey: Building an LLM-based Tool</a></li><li>Navigating the Subtle Reliability Quirks of ChatGPT</li><li><a href="https://medium.com/@geoffreyhung/navigating-the-full-stack-llm-web-development-with-zero-frontend-knowledge-caaeb15c296f">Navigating the Full Stack LLM App Development with Zero Frontend Knowledge</a></li><li><a href="https://medium.com/@geoffreyhung/the-dichotomy-of-builder-and-worker-mindsets-b4d5358c1748">The Dichotomy of Builder and Worker Mindsets</a></li></ol><h3>Background</h3><p>As I continue to chronicle my journey on building my side project, <a href="https://www.producthunt.com/posts/clarify-ai">Clarify AI</a>, a tool aimed at transforming vague ideas into structured contexts using Language Models like ChatGPT, the road has been both exhilarating and bumpy. If you’ve followed my previous article, you’d know how the endeavor began as a quest to overcome the hurdles of providing comprehensive context for more profound and meaningful conversations with ChatGPT.</p><p>However, as the project transitioned from ideation to actual development, a new set of challenges cropped up, predominantly revolving around the <strong>reliability of ChatGPT</strong>.</p><h3>The prompt that becomes my nightmare</h3><p>I came across an <a href="https://www.reddit.com/r/ChatGPT/comments/14d7pfz/become_god_like_prompt_engineer_with_this_one/">impressive online prompt — ask GPT to do recursive clarification</a>. After some trial tests on it, the results seemed promising for my test questions. This spurred me on to package it into a chat product, but this endeavor soon morphed into my <strong>ultimate nightmare</strong>.</p><p>The concept was straightforward. If I could get the tool to recursively clarify questions with me, capture my responses, and summarize them on the fly, wouldn’t that be akin to having a personal consultant at my side? Especially one well-versed in [ChatGPT on X topics].</p><p>However, only upon diving into the implementation did I fully grasp what <a href="https://www.youtube.com/watch?v=hQP11Hd1hew">Ilya Sutskever [Open AI Chief Scientist] meant in an interview</a> regarding the real-world impact and reliability of such models.</p><p>What seemed to work seamlessly in the POC phase presented <strong>unforeseen hurdles</strong> once packaged into a product. You really need to immerse yourself to <strong>slowly uncover the subtle behavioral patterns that can disrupt the flow</strong>.</p><h3>The reliability issues</h3><p>So, what are these subtle patterns that derailed my initial approach?</p><p>The following discussions are predicated on the assumption that we are using prompt engineering on GPT-3.5 as of the time of writing. Who knows what OpenAI will unveil next, potentially resolving these issues.</p><blockquote>Regrettably, my budget constraints didn’t allow for the use of GPT-4 API, and the current rate limits don’t quite align with production requirements, but they are instrumental in evaluating the potential of more advanced models.</blockquote><h3><strong>1. It’s Depth-First, Not Breadth-First</strong></h3><p>Current LLMs have a narrow focus. Once a conversation veers in a certain direction, it tends to stay on that trajectory, irrespective of the prompt.</p><p><strong>Example:</strong></p><blockquote>System: Aid the user in brainstorming meal plans with as MUCH DIVERSITY as possible.</blockquote><blockquote>User: My friend is visiting from Hong Kong, what should we have for dinner?</blockquote><blockquote>Assistant: Asian food is…….</blockquote><blockquote>User: What else?</blockquote><blockquote>Assistant: {more Asian foods} // diversity!?</blockquote><p><strong>Implications:</strong></p><ul><li>The initial conversation <strong><em>heavily influences the downstream direction</em></strong>, and there seems to be no reliable way to steer it back, <strong>regardless of the system prompt</strong>.</li><li>If you can’t control how users respond, this may lead to unexpected outcomes.</li></ul><p><strong>Why?</strong></p><p>The decoder’s mechanism is straightforward: for every next token, it <strong>looks backward to compute attention</strong> and generate.</p><h3><strong>2. A Simple ‘Hello World’ Can Disrupt Your Flow</strong></h3><p>Unlike a code editor that can discern variables, expressions, and syntax from plain text, to GPT, everything is text [token to be precise], plain and simple. This uniform treatment of input can lead to unpredictable responses.</p><p><strong>Example:</strong></p><blockquote>System: Count the number of words in the user’s response, and output the count ONLY. NOTHING ELSE. REPEAT, NOTHING ELSE.</blockquote><blockquote>User: Yo, what can you do?</blockquote><blockquote>Assistant: 5</blockquote><blockquote>User: Hello world</blockquote><blockquote>Assistant: Hello world, what can I do for you today? // Oops…</blockquote><p><strong>Implications:</strong></p><ul><li>Unanticipated user input can break your flow…</li><li>A solid fallback plan is essential for a smooth product experience.</li></ul><p><strong>Why?</strong></p><p>The rationale is similar, GPT treats everything as a token, making it challenging to set strict rules for it to follow.</p><h3><strong>3. Overlooking the Obvious</strong></h3><p>At times, LLMs <strong>overlook obvious details even when explicitly mentioned</strong>, and this becomes evident when that part is sent to the API.</p><p><strong>Example:</strong></p><blockquote>System: Summarize everything the user said, DO NOT MISS ANYTHING.</blockquote><blockquote>User: A</blockquote><blockquote>Assistant: A</blockquote><blockquote>User: B</blockquote><blockquote>Assistant: AB</blockquote><blockquote>User: C</blockquote><blockquote>Assistant: BC //?!?!?!?!?!?!?</blockquote><blockquote>User: Why BC? Miss something?</blockquote><blockquote>Assistant: Sorry for the oversight, ABC // Ah, so you know ABC but gave me BC?</blockquote><p><strong>Implications:</strong></p><ul><li>If your plan is to use one prompt to handle both summary and the next step, you’re likely to fail [such as the recursive prompt I linked initially].</li><li>What irks me the most is, it does read all the information but outputs only a part. Ask again, and it corrects itself. However, this randomness can potentially break your product flow.</li></ul><h3><strong>4. Long Conversations Are Tricky</strong></h3><p>Evaluating the quality of a single reply is easy — just review the last two messages (in and out). But what about a conversation spanning 10 messages? How do you determine the quality of responses, and if they go awry, what’s the remedy?</p><p><strong>Implications:</strong></p><p>- If you aim to maintain a certain structure in the responses, the probability of something going wrong increases with each interaction.</p><h3>Conclusions</h3><p>To encapsulate my learnings (assuming we’re solely relying on prompt engineering [no fine-tuning, no chain of calls], and aiming for higher reliability [following a certain structure, e.g., JSON, response format]):</p><ul><li><strong>Avoid the chat concept unless necessary.</strong> Restrict interactions to one input, one output, as the probability of errors escalates with longer conversations. Limiting to 2 messages is more manageable and easier to evaluate.</li><li>Free-form text responses may seem flexible, but may not fit every product’s needs.</li><li>ChatGPT excels in maintaining long, free-form conversations because users can steer the conversation back on track when it veers off course.</li><li>If you’re building a product and desire the user to follow certain flows, a long free-form chat format may become a nightmare, <strong>both in terms of API cost and prompt engineering</strong>.</li></ul><h3>How do I perceive ChatGPT API?</h3><p><strong>Consider the popular product features demanding reliable output structures; they mostly keep things simple:</strong></p><ul><li>GitHub Copilot: 1 code context request -&gt; 1 code snippet output.</li><li>Notion AI: 1 request -&gt; 1 summary.</li></ul><p><strong>Certain products might not suit the chat format (though I could be wrong):</strong></p><ul><li>Flight ticket chatbot: “Can you find me a ticket to Houston from SF in November?” vs. a few clicks on Google Flights.</li><li>Recipe chatbot: “What to cook tonight?” vs. Youtube cooking tutorials.</li><li>News chatbot: “What’s happening in the Middle East?” vs. a quick visit to Reddit.</li></ul><p><strong>We should view this as the emergence of new capabilities and employ them only when necessary, capabilities:</strong></p><p>1. Understanding language across the board.</p><p>2. Generating responses with a touch of “intelligence” (though expect some errors).</p><blockquote>Chat is merely a format, and we shouldn’t be confined by it because of ChatGPT’s success.</blockquote><h3>Final thought</h3><p>Reflecting on the time spent uncovering these nuances, I wonder, could this exploration have been sidestepped? Probably not. Until you experience the subtle patterns firsthand, it’s challenging to conclude what doesn’t work reliably for your solution [I define reliable as at least 90% alignment with expectations, a 50% awesome + 50% suck equation equals failure].</p><p>But if you are planning to do something similar to what I tried, I hope this gives you some light on the potential issues</p><blockquote>One last nugget I wish to share is a trick I used quite often: if you desire your output to be in markdown format, after you request that in the prompt, end your prompt with: <strong>“Your next line should start with #”</strong>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9494276d8134" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Side Project Journey: Building an LLM-based Tool]]></title>
            <link>https://medium.com/@geoffreyhung/my-side-project-journey-building-an-llm-based-tool-dafe641f7830?source=rss-cf2c9ab57bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/dafe641f7830</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[side-project]]></category>
            <dc:creator><![CDATA[Geoffrey Hung]]></dc:creator>
            <pubDate>Wed, 25 Oct 2023 15:31:35 GMT</pubDate>
            <atom:updated>2023-10-27T07:09:21.541Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*noSWdp8mgboob_kAS10NLA.png" /><figcaption>DALL.E-3: illustration of a progression bar nearing its completion with the text ‘LLM-based Tool’ above it and a person writing down notes next to it.</figcaption></figure><p>Over the past few months, my weekends have been occupied by my side project: <a href="https://www.producthunt.com/posts/clarify-ai">Clarify AI</a>, an LLM-based tool designed to turn vague ideas into structured contexts. Now as I near the finish line, I think it’s worth penning down some documentation to encapsulate what I’ve learned.</p><h3><strong>Problem</strong></h3><blockquote>The inception of my side project can be traced back to a problem I encountered while using ChatGPT: <strong><em>providing comprehensive context for deeper conversations was an uphill battle.</em></strong></blockquote><p>Take, for instance, brainstorming the design for my audience-targeting project. It’s only reasonable to fill ChatGPT with background information such as the business model, industry, data sources, scale, metrics, and algorithms involved to have a meaningful conversation.</p><p>But jotting down all these details is a tedious task. Even with the best of intentions, I always seemed to miss something.</p><p>But dude, it’s a chat. It should be assisting me, not the other way around. Why should I break a sweat crafting numerous details just to elicit some responses? It felt like I was onboarding my assistant anew every single time.</p><h3>Existing approaches</h3><p>After scouring the internet, I identified three main approaches to get better responses from LLM:</p><p>1. <strong>Prompt Engineering</strong>: This entails crafting better prompts using various techniques like CoT. While it may enhance the output quality, it doesn’t solve the context problem. <em>The onus is still on you to learn how to pose a good question and provide the relevant context.</em></p><p>2. <strong>RAG (Retriever-Augmented Generation)</strong>: This involves storing existing context in a vector database, retrieving it on demand to augment contexts. It’s a solid plan if you have the context (e.g., building a vector database for all the Jira tickets in my team), but I was at square one — <em>I didn’t have any context.</em></p><p>3. <strong>Fine-tuning OS Models</strong>: This method leverages existing contexts (private data) to train the model. It’s a tough nut to crack. Many startups are venturing down this path, but it’s fraught with uncertainties and is the priciest option, especially for individual use.</p><h3>Idea</h3><blockquote>This led me to wonder, could there be a <strong>systematic</strong> and <strong>efficient</strong> way to construct the contexts?</blockquote><p>- <strong>Systematic</strong>: It should be versatile enough to tackle a class of problems, not just one. I can reuse the process for many more problems.</p><p>- <strong>Efficient</strong>: I am the boss, I give the directions, and the machine does the legwork.</p><p>I initially set out to wrap up this project over weekends spanning 3–4 months, allocating 1–2 months to grasp the basics of CSS/React/Next JS since my frontend knowledge was nonexistent. However, unforeseen challenges emerged from the woodwork.</p><h3>More…….</h3><p>Here’s a sneak peek into the topics I’ll delve into in this mini-series, which I hope will serve as a valuable reference if you are using ChatGPT API to build something:</p><ol><li>My Side Project Journey: Building an LLM-based Tool</li><li><a href="https://medium.com/@geoffreyhung/navigating-the-subtle-reliability-quirks-of-chatgpt-9494276d8134">Navigating the Subtle Reliability Quirks of ChatGPT</a></li><li><a href="https://medium.com/@geoffreyhung/navigating-the-full-stack-llm-web-development-with-zero-frontend-knowledge-caaeb15c296f">Navigating the Full Stack LLM App Development with Zero Frontend Knowledge</a></li><li><a href="https://medium.com/@geoffreyhung/the-dichotomy-of-builder-and-worker-mindsets-b4d5358c1748">The Dichotomy of Builder and Worker Mindsets</a></li></ol><p>Stay tuned to see all the mistakes I made!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dafe641f7830" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python for Advanced Beginners]]></title>
            <link>https://medium.com/@geoffreyhung/python-for-advanced-beginners-cd39b42ef10?source=rss-cf2c9ab57bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd39b42ef10</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[techniques]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Geoffrey Hung]]></dc:creator>
            <pubDate>Fri, 21 Jan 2022 05:06:23 GMT</pubDate>
            <atom:updated>2022-01-21T20:14:48.995Z</atom:updated>
            <content:encoded><![CDATA[<h4>Some Python techniques I wish DS/ML projects should adopt more</h4><blockquote>The target audience are people who are in the data industry and need to write reusable Python code</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*caTlEJc6bbFvbbs6MPvShg.jpeg" /><figcaption>photo by <a href="https://unsplash.com/@hiteshchoudhary">hiteshchoudhary</a> on Unsplash</figcaption></figure><p>^ Just find a random photo as featured image…….</p><p>There are already many articles on Python, covering the topics such as data structures [eg: set, dict, list], data assessing techniques [eg: lst[:5], lst[::-1]], loop [eg: for, while], function, class, pandas, sklearn……</p><p>But I am not planning to reorganize them to make a new article. What I am planning to write here is some Python techniques that are <em>simple</em> and <em>super useful</em>, yet I don’t know why people don’t adopt that in their daily work.</p><p>These techniques need to fulfill the following requirements:</p><p><strong>Useful</strong>, something that we can really use in the work/project</p><p><strong>Simple</strong>, we can learn and use them within <em>5 mins</em></p><p><strong>Extensible</strong>, it can lead us to more interesting topics, not just a hack for a specific case</p><blockquote><a href="https://colab.research.google.com/drive/1nLtbLQAeiTgSjWOgIza0tGAG_cEe3rRV#scrollTo=RZ2i_yiywJRk">Colab</a></blockquote><blockquote><a href="https://github.com/G-Hung/python-article">GitHub</a></blockquote><h3>0. <strong>Code is read much more often than it is written</strong></h3><p>Since we are using Python, let’s start with index zero</p><p>The first thing isn’t about any technique but the mindset: <strong>Code is read much more often than it is written.</strong> Repeat after me: <strong>Code is read much more often than it is written!</strong></p><p>To make the future you &amp; your peers have a happy life, make the code readable. It is surprising to me that this mindset isn’t the norm and leads to many sad stories.</p><p>Code that is not readable is harder to change, and hence harder to estimate the workload when you want to make a “simple” change [eg: add a new feature]. Code quality isn’t nice to have, but it is necessary to keep productivity &amp; transfer knowledge for any meaningful project.</p><blockquote>Everything is a tradeoff, sometimes we need to hack around, but organized in a way that people can fix that easily in the future [eg: avoid hardcoded, isolate the hacking logic in a limited scope, add TODO comments in the code, write more comments to explain intentions]</blockquote><h3>1. Please, add type hints</h3><p>Let’s start with a simple example.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf88145c822bf4ea2bd54b0efc81d030/href">https://medium.com/media/bf88145c822bf4ea2bd54b0efc81d030/href</a></iframe><p>My first thought when I read the function is what are a and b. Since Python is a dynamically typed language, things will become “interesting” if the types change. It is also very annoying if I need to read through the code and documents to guess</p><p>Adding type hints is better, now when I use the function, I know what it needs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c39ad335b203240a82559879104e1de/href">https://medium.com/media/0c39ad335b203240a82559879104e1de/href</a></iframe><p>BTW, there is an attribute called __annotations__ that returns the annotations in a dictionary, in this case:</p><p>add.__annotations__ =&gt; {‘a’: int, ‘b’: int, ‘return’: int}</p><p>Using type hints can improve the readability a lot, let’s see another example</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1304efd9add92e8742b6dfe108232878/href">https://medium.com/media/1304efd9add92e8742b6dfe108232878/href</a></iframe><p>Last example about readability using type hints</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/09e0ef87b9dc834d04a59b4febec236a/href">https://medium.com/media/09e0ef87b9dc834d04a59b4febec236a/href</a></iframe><p>However, type hint does <strong>NOT</strong> validate the types, we may need to use other tools to help</p><pre>def add(a: int, b: int):<br>    print(a + b)</pre><pre>add(&#39;hello&#39;, &#39;world&#39;)<br># output str even we annotate it is int<br># &#39;helloworld&#39;</pre><p>Reference: <a href="https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html">https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html</a></p><h3>2. Use *args and **kwargs to write parameterized code</h3><p>It is often to see <strong>*args</strong> and <strong>**kwargs</strong> in Python, they are helpful when we want to pass all parameters from one place to another</p><blockquote>Be careful, don’t abuse them, it accepts all the inputs and most of the time it is not what you are looking for, especially if you have layers of abstraction, it is much more difficult to debug/trace</blockquote><p>Let’s see what it is doing</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/684f815882ff01de4835fae149f9964e/href">https://medium.com/media/684f815882ff01de4835fae149f9964e/href</a></iframe><p>We can use * and ** to feed the list and dictionary to function</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/699787187f2821d64e6efecdf3dccfaa/href">https://medium.com/media/699787187f2821d64e6efecdf3dccfaa/href</a></iframe><p>This pattern is often used in model training, we <strong>just change the configurations</strong>, not the model training code</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f898b94083de86d9e9faa5f6dc45b9c6/href">https://medium.com/media/f898b94083de86d9e9faa5f6dc45b9c6/href</a></iframe><p>Reference: <a href="https://realpython.com/python-kwargs-and-args/#unpacking-with-the-asterisk-operators">https://realpython.com/python-kwargs-and-args/#unpacking-with-the-asterisk-operators</a></p><h3>3. Enforce people to use keyword arguments</h3><p>Although we mentioned in point 2, we can use <strong>model(**params)</strong> pattern to pass the configuration, sometimes when we are experimenting with the code, we still prefer explicit arguments such as <strong>model(a=1, b=2, c=3)</strong></p><p>People are too lazy to type the keywords, although functions can treat them as positional arguments which is fine for the simple case, eg: <strong>add(1997, 2047)</strong> is pretty obvious</p><p>But for complex functions where you expect configuration, reading code such as <strong>train_model(10, False, True, 1234)</strong> in the repo is not fun. A simple trick is to add a * to enforce keyword arguments.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a6753fd0f8ec301b24ee45cf3d08463/href">https://medium.com/media/1a6753fd0f8ec301b24ee45cf3d08463/href</a></iframe><h3>4. Decorator for tedious and repeated logic</h3><p>Oftentimes we want to log the running time of certain functions such that we can debug/analyze the pipeline, this is how people handle it often</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9aa3f48415c5abb75c6d471793d8f0e1/href">https://medium.com/media/9aa3f48415c5abb75c6d471793d8f0e1/href</a></iframe><p>More “realistic” example, things become very annoying. I can guarantee you I will mistype some variables and measure the wrong things</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75482e705078b4a021d4b80a62887a05/href">https://medium.com/media/75482e705078b4a021d4b80a62887a05/href</a></iframe><p>We notice the logic is very similar and simple, just record the start time and calculate the duration after. If there is some magic such that it can <strong>run some snippet before &amp; after the function</strong>, it would be great!</p><p>Yeah, checkout <strong>decorator</strong> in Python</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f1ffabf34751a04ecf8fa9c583b0d55/href">https://medium.com/media/3f1ffabf34751a04ecf8fa9c583b0d55/href</a></iframe><p>We only need to focus on the key logic, not decorations</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0782fc2c6646d12fc2608f15554c0538/href">https://medium.com/media/0782fc2c6646d12fc2608f15554c0538/href</a></iframe><p>More examples in the colab:</p><ul><li><a href="https://colab.research.google.com/drive/1nLtbLQAeiTgSjWOgIza0tGAG_cEe3rRV#scrollTo=OLlnWoFJEgcj">decorator to log input dataframe shape</a></li><li><a href="https://colab.research.google.com/drive/1nLtbLQAeiTgSjWOgIza0tGAG_cEe3rRV#scrollTo=2DDJxGHS6Tn8">decorator to validate the function inputs</a> [just demo the functionalities, probably there are more robust solutions]</li></ul><blockquote>Again: don’t abuse, don’t include the side effects unless you know very clearly what you are doing</blockquote><p>Reference: <a href="https://realpython.com/primer-on-python-decorators/">https://realpython.com/primer-on-python-decorators/</a></p><h3>5. Mutable type is a big trap when they are function parameters</h3><p>There are some types that are mutable [eg: list, dict], easy to introduce the bug by accident.</p><p>Let’s say there is a function to process data, you set a list as default parameters, then somehow [when the business logic becomes complex it is possible] the function mutates the list</p><blockquote>WTF moment: If you run process_data() twice, the outputs are different!</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6433da46d9d7fa6019c8c5d51cff89b/href">https://medium.com/media/a6433da46d9d7fa6019c8c5d51cff89b/href</a></iframe><p>But the solution is simple if you are aware of that</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1e66c93097120d9b034ef1570843b51/href">https://medium.com/media/d1e66c93097120d9b034ef1570843b51/href</a></iframe><p>Another reason you should use <strong>flake8</strong> is that it can alert you about the mutable objects</p><h3>6. Use getattr to help configuration</h3><p>Recall we tried to group in dictionaries and feed them to the model. The model here is a simple print, but in reality, they are different estimators, eg: random forest, gradient boosting, etc</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c8a5ffb8ac40012c09a4c1e0ea39000/href">https://medium.com/media/6c8a5ffb8ac40012c09a4c1e0ea39000/href</a></iframe><p>Now we have two sets of dictionaries for random forest and gradient boosting respectively, the pattern may look like</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46b60544b0ce2b1ac2b0be1db38cc8d7/href">https://medium.com/media/46b60544b0ce2b1ac2b0be1db38cc8d7/href</a></iframe><p>What if we try to explore model types other than these two? Does it mean we need to change the model training code again and again? Well, you may try <strong>getattr</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ebe70ddf4da7b9432ab10646bead7b2/href">https://medium.com/media/0ebe70ddf4da7b9432ab10646bead7b2/href</a></iframe><p>The same technique could be applied to other areas, you can swap different optimizer, loss functions, or even model architecture in PyTorch without touching your training code</p><h3>7. pre-commit to automate &amp; debug as early as possible</h3><p>When we write code, it is really hard to keep the formatting right all the time. For example, some code above is not PEP-8 standard and I write that deliberately</p><p>In my earlier article, I have introduced a CI pipeline [Continuous Integration] to check the code before people can merge it. But it still has some issues:</p><p>1. the code is likely in PR/remote to trigger CI and capture these issues, this leads to some ugly commit messages such as `fix: format`</p><p>2. triggering CI pipeline needs time, it takes a few minutes</p><p>When we look at the development flow, CI takes effect on/after the pushing stage:</p><p>dev -&gt; commit -&gt; push -&gt; PR -&gt; merge</p><blockquote>Can we do something around the commit stage?</blockquote><blockquote>`pre-commit` <a href="https://pre-commit.com/">https://pre-commit.com/</a></blockquote><p>By adding <a href="https://github.com/G-Hung/python-article/blob/main/.pre-commit-config.yaml">.pre-commit-config.yaml</a> to your repo, it will trigger the runs when we commit. For example, it will run pytest, black, isort, detect aws credentials, flake8, etc <strong>BEFORE</strong> you commit the code successfully.</p><p>It means you can’t even commit the code if the checks fail, you have no chance to push the code to the repo</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AAi33sEjdj05U3PPH6Gv1Q.png" /><figcaption>What does pre-commit do when I commit the code</figcaption></figure><p>pre-commit fixes the styling issues automatically, this happens for free [don’t need to remember to run any script]</p><p><a href="https://github.com/G-Hung/python-article/compare/38d4439..8a94210">To see what pre-commit fix my code</a></p><h3>Wrap up</h3><p>Apparently, this article complements my <a href="https://towardsdatascience.com/from-jupyter-notebook-to-sc-582978d3c0c">previous article</a> and covers more tools/techniques for day-to-day uses.</p><blockquote>The real agenda is to advocate the use of scripts and not Notebooks unless we are doing one-off things/explorations</blockquote><p>With scripts, the learning curve is higher, but we can adopt the (new) tools for free. It forces the modularized code, enables code review, the life will be easier and more productive in the long run</p><p>With notebooks [for production use], it seems more intuitive, but it will be more and more difficult to keep the development speed</p><blockquote>I once spent a month refactoring a production notebook job: few functions, no asserts / unit tests, few comments, make every mistake this article mentioned, the most exciting part is: data retrieval, feature engineering, and model training are all in one big Notebook.</blockquote><p><strong>I beg everyone who is still using notebooks for production work</strong>, please at the very least use more functions and import them from different modular files, the notebook should only contain the main flow, not implementation details</p><p>Again: <strong>Code is read much more often than it is written!</strong> Human is good pattern recognizer, and the code should have patterns such that we can know/guess what to focus on</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd39b42ef10" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From cURL to Automated Workflow]]></title>
            <link>https://medium.com/usf-datascience/from-curl-to-automated-workflow-d79c7c108450?source=rss-cf2c9ab57bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d79c7c108450</guid>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Geoffrey Hung]]></dc:creator>
            <pubDate>Wed, 24 Feb 2021 02:01:43 GMT</pubDate>
            <atom:updated>2021-02-26T23:22:15.969Z</atom:updated>
            <content:encoded><![CDATA[<h4>Getting ARK fund data is not hard; handling the downloaded data is</h4><p>I am not a big fan of the stock market but still hear famous <strong>ARK funds </strong>recently [their bets on Tesla and Square!]! Frankly, I admit I am very curious about what they did, after all, who doesn’t like money.</p><p>Interestingly, <a href="https://ark-funds.com/investor-resources">ARK funds are transparent enough to show their holdings daily</a> and people can study what they are holding, but they only share the data snapshot of that day! [ie: what they are currently holding?]. To know what they held and how they changed the holdings weights, we need to have multiple snapshots.</p><p>In this article, I will talk you through my side project last weekend on scraping the data from the ARK site using the <strong>command line</strong>, pipelining the data to <strong>BigQuery</strong>, using <strong>dbt</strong> to do data quality testing, and lastly, automate and schedule the workflow using <strong>GitHub Actions</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*GGAo2dDwR2xxQpGeOegTIA.png" /><figcaption>High-level flow</figcaption></figure><p>We will explore some data engineering problems and <strong>WHY</strong> these tools are useful or even necessary if you don’t want to be a firefighter/human debugger all the time. Our goal is to leverage as many engineering tools as possible to create an automated workflow such that our limited life could be more meaningful</p><blockquote><a href="https://github.com/G-Hung/project-ark"><strong>&gt;&gt;&gt;TALK IS CHEAP&lt;&lt;&lt;</strong></a></blockquote><h3><strong>Functional Requirements</strong></h3><ol><li>Get the data</li><li>Store and access the data</li><li>Check data in (1)</li><li>Regularly repeat (1), (2), (3)</li></ol><h4><strong>Nice-to-have:</strong></h4><ol><li>Cost: Both time and $, ideally zero after development</li><li>Quality: Either alert me of the problem or I will simply use the data directly</li><li>Reliability: Don’t rely on the human or specific computer</li><li>Workflow: Monitor the whole process, alert me if things go wrong</li></ol><blockquote>Normal approach: this is not unheard of!</blockquote><ol><li>Get the data: Manually click the links and save them as CSV</li><li>Store and access the data: Store somewhere in share folder and Excel</li><li>Check data in (1): Eyeball check OR creating many pivot tables</li><li>Regularly repeat (1), (2), (3): Human</li></ol><blockquote>More advanced approach: use code</blockquote><ol><li>Get the data: request in Python OR curl in the command line</li><li>Store and access the data: store somewhere in the shared folder and Jupyter Notebook to access the data</li><li>Check data in (1): Many plots OR many cells with assert using Pandas</li><li>Regularly repeat (1), (2), (3): cron job</li></ol><blockquote>The drawbacks are obvious:</blockquote><ul><li>What if the people sick leave/computer is down and miss few days? [Data lost!]</li><li>Who knows where they store them exactly? [maybe people write in some doc/email/share folder? who knows]</li><li>Can we trust their checks? [Sorry I can’t, because I can’t even trust my checks]</li></ul><p>In short, these approaches are unreliable, there are endless possibilities to go wrong, some people think that’s the value of a data team, that they can have the capability to make the hypotheses about the potential data issues. But being able to do it doesn’t mean I need to do it “manually” and “repeatedly”.</p><blockquote>Recall what makes software eat the world: The marginal cost of software approaches zero. We should develop reliable solutions to automate as much as possible to reduce the marginal cost!</blockquote><h3>Get the data: command line</h3><p>Firstly, How can we get the data? Getting the data from the ARK site is no more than running the following command in the terminal</p><pre>curl <a href="https://ark-funds.com/wp-content/fundsiteliterature/csv/ARK_INNOVATION_ETF_ARKK_HOLDINGS.csv">https://ark-funds.com/wp-content/fundsiteliterature/csv/ARK_INNOVATION_ETF_ARKK_HOLDINGS.csv</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0-ron5MvaLMWPVhUiieLQA.png" /><figcaption>curl result</figcaption></figure><p>But seeing it in the terminal is not fun, we want to download it as CSV file! No problem, just add &gt; ARKK.csv</p><pre>curl <a href="https://ark-funds.com/wp-content/fundsiteliterature/csv/ARK_INNOVATION_ETF_ARKK_HOLDINGS.csv">https://ark-funds.com/wp-content/fundsiteliterature/csv/ARK_INNOVATION_ETF_ARKK_HOLDINGS.csv</a> &gt; ARKK.csv</pre><p>We open the CSV and see, hmmm what are these things……these remarks are not part of the data…….After I observed the patterns, we can simply remove the last 3 rows. What can I do? Maybe I can write a Python script to remove the last 3 rows [Seems to overkill]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*r6l2RSZXJ62TVlL3IVAHvg.png" /><figcaption>csv weird ending</figcaption></figure><p>For simple thing, maybe you can consider some command line, here I make a pipe</p><pre>curl <a href="https://ark-funds.com/wp-content/fundsiteliterature/csv/ARK_INNOVATION_ETF_ARKK_HOLDINGS.csv">https://ark-funds.com/wp-content/fundsiteliterature/csv/ARK_INNOVATION_ETF_ARKK_HOLDINGS.csv</a>| tail -r | sed ‘1,3d’ | tail -r &gt; ARKK.csv</pre><blockquote>tail -r | sed ‘1,3d’ | tail -r simply means:</blockquote><p>reverse the rows, delete the first 3 rows, reverse the rows</p><p>aka: remove the last 3 rows</p><p>Am I so smart to write this on my own? Surely not, google!</p><blockquote><a href="https://missing.csail.mit.edu/2020/shell-tools/">Nice materials to learn more about scripting</a></blockquote><p>With the extra for loop and some transformations, that’s the core part of <a href="https://github.com/G-Hung/project-ark/blob/main/scripts/collect_and_load.sh">collect_and_load.sh</a>! OK, now we have a script in hand, it will download the data to the folder when we run it. The next problem is where should we store the data such that we can access it later?</p><h3>Store &amp; Access the data: BigQuery</h3><p>Although people said data are just binary at last, different data formats require appropriate ways to store and access. Data formats could be table [like csv], object [like json], image [like png], text [like string], audio [like wav] etc etc.</p><p>For our case, the file format is CSV, it belongs to a huge family [structural data], which is the most typical data format and the standard way to query it is SQL [Basically appear in every DS interview]. So the problem is how can we set up the database such that we can upload and query the data?</p><blockquote>Aside:</blockquote><blockquote>For relational databases, there are two main types: OLAP (Online analytical processing) and OLTP (Online transaction processing).</blockquote><blockquote>OLAP is for analytics purpose, when people talk about data warehouse, they usually refer to OLAP. It can do aggregation queries in second-level for tons and tons of data. One of the reasons why it is so fast is that the data is stored in columnar format [eg: Parquet], ie: data in the same column are stored nearby, so some people also called them columnar database. The famous cloud solutions include: BigQuery [GCP], Athena, Redshift [AWS], Snowflake</blockquote><blockquote>OLTP is for transaction purpose, when people develop the applications [try to google some web application tutorials requiring database], they usually refer to OLTP. It can access and operate on a single row very fast, the data is stored in row format, ie: data in the same row are stored nearby, famous solutions include: MySQL, PostgresSQL, SQLAlchemy</blockquote><blockquote>You can checkout Data Engineering Lecture in <a href="https://stanford-cs329s.github.io/syllabus.html">https://stanford-cs329s.github.io/syllabus.html</a> for the overview</blockquote><p>The solution I picked was Google BigQuery, the key reasons I picked are:</p><ul><li>Serverless: Pay what I use, I don’t want to do any DevOps work for a side project and the free quota is enough</li><li>Scalability: Although it won’t be a case for this project, it is always nice to have a solution that can handle TBs+ level data</li><li>Console: query in the browser, no client software is needed</li><li>dbt: dbt supports BigQuery [Explain below]</li></ul><blockquote>I am not biased to GCP, AWS does equally good, checkout Athena / Redshift</blockquote><p>The steps are boring [It is a compliment because it means simplicity], basically:</p><ol><li>Go to GCP / BigQuery, create a project and dataset, use schema<a href="https://github.com/G-Hung/project-ark/blob/main/misc/bq_schema.txt">bq_schema.txt</a> and set date as partition field to create the table</li></ol><blockquote>Aside:</blockquote><blockquote>Partitioning is one of the easiest and most useful optimizations. Imagine we have a TB level data warehouse, we definitely don’t want to scan the whole dataset for every query [Because the pricing is based on how much data it processed]. Partitioning means to specify a field as “index”, when you specify the partitions in where-clause, it will ONLY scan the data in those partitions, hence reduce the scan size!</blockquote><p>2. Generate a service account credential and put it in the right location</p><p><a href="https://cloud.google.com/bigquery/docs/batch-loading-data#permissions">Required permission reference</a></p><p>3. Use <a href="https://github.com/G-Hung/project-ark/blob/main/scripts/bq_load.py">bq_load.py</a> to load data to BigQuery, we first remove the data [to avoid duplication] and load the data</p><p>Reference: <a href="https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv">https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y0zMowOyObJkfC8r0NL9fw.png" /><figcaption>Data is loaded to BQ yoho</figcaption></figure><p>More optimizations could be done for bigger data, but I don’t think they matter a lot for this project, just share some possibilities below:</p><p>a. Load to GCS first instead of loading to BigQuery directly. This optimization will ensure the data can be kept safely in Cloud Storage and be able to load bigger data to BigQuery. Local upload only allows 10MB max</p><p>b. Save the CSV as gzip/parquet. Just for convenience &amp; showcase because of small data</p><p>Great, now we use the script to get the data, store &amp; query them in BigQuery successfully. What’s next? I remember a conversation with my previous colleague [another team, non-tech people], I just told her: give you a number is easy, give you the right number is super hard!</p><p>Why hard? The data we use in schools are mostly cleaned but in reality, everything can happen. How can we have more confidence in the data quality?</p><blockquote>Disclaimer:</blockquote><blockquote>At most, we can only check the data fulfil some logics, we can’t auto correct the wrong data. Similar to testing in software, it definitely has problems if it can’t pass. But even it passes all tests, it doesn’t mean the program is correct</blockquote><h3>Check the data: dbt</h3><p>Nowadays, with more and more cloud solutions, storage and data access are hard to be the bottleneck for most companies, out-of-the-box solutions such as BigQuery (Redshift / Athena), Google Cloud Storage (S3), Apache Spark can store and process TB level data easily</p><p>Still, one of the most annoying things about DS is dirty data. It is similar to a disease, the earlier you find it, the easier you can cure it but it is hard to detect until some people use it.</p><blockquote>Let’s try a dummy example:</blockquote><blockquote>Suppose we want to join `users` information to `orders` table, can you spot the potential issues?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*y2XDmbvzxGJt8CP5mgP2gg.png" /><figcaption>Join two tables</figcaption></figure><blockquote>My answers below:</blockquote><p>orders:</p><ul><li>negative amount!?</li></ul><p>users:</p><ul><li>duplicated uid!?!?</li><li>NULL value!?!?!?</li></ul><p>join:</p><ul><li>since duplicated uid in table:users, it may expand the joined tables</li></ul><blockquote>If you can spot them all, nice!</blockquote><blockquote>Now imagine the tables become tens of columns and millions of rows, and I want you to join across multiple tables. What will you do?</blockquote><p>This is one of the most common errors because I will simply look at the schema and start to join them until it looks good. The queries are vulnerable to these little errors because they are not easy to enumerate and super annoying to examine even if you can list them all. When the bomb explodes, usually the downstream pipelines are polluted already :(</p><p>This pattern is similar to software, it is hard to imagine any mature programs without testing [Unit testing at least]. So is it possible to introduce the testing to the data world? Let me introduce you dbt (data build tool)!</p><blockquote>dbt is famous for doing data transformation in data warehouse, yet my favorite feature is data testing, I can use SQL to write unit tests! How?</blockquote><p>Use ARK CSV as an example, I want to assert few things:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sl7_NqwyLxWBrpjne5hLsg.png" /><figcaption>Example ARKK data</figcaption></figure><ol><li>Numeric fields should be positive</li><li>The sum of weights should be (almost) 100%</li><li>(date, fund, company) pair should be unique</li></ol><p><a href="https://github.com/G-Hung/project-ark/tree/main/ark-track-dbt/tests">Implemented test cases</a></p><pre>tests/raw.test:assert_positive_values.sql</pre><pre>-- If this fails: it means some reocrds in `shares`, `market_value`, `weight` are non-positive </pre><pre>SELECT date, fund, shares, market_value, weight<br>FROM {{ source(&#39;raw&#39;, &#39;test&#39;) }}<br>WHERE shares &lt;= 0 OR market_value &lt;= 0 OR weight &lt;= 0</pre><p>What does this query do? It will return records if either shares, market_value, or weight ≤ 0! [If data are normal, it returns no records]</p><p>That’s it, dbt will handle the rest, I will simply run:</p><pre>dbt test --data</pre><blockquote>Notice that I skip <a href="https://docs.getdbt.com/tutorial/setting-up/">the setup part</a>, dbt needs credentials and profiles.yml to connect to the data warehouse, under the hood, they compile our tests [turn it to queries] and run, PASS if no return else ERROR</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nfiMvYbxA3aee7xAuXUkuA.png" /><figcaption>dbt test output</figcaption></figure><p>Wooow TBH, I relieve a little bit when I see the tests are all passed, I know I can work on data that make some sense, but not wasting hours or days to know it is a data issue.</p><blockquote>I personally think this is a <strong>game-changing feature</strong> because we can adopt the testing practices from software engineering! Maybe not dbt at last, but something similar with low learning curve should be part of the data engineering tech stack</blockquote><p>On one hand, it encourages the team to write data tests instead of ad hoc queries for exploration. We can leverage what others developed and step on top. I can be less anxious when I need to review 200+ lines SQL and figure out what goes wrong.</p><p>On the other hand, it enables the new workflow to check data quality from upstream. With more iterations and better test coverage, data quality can be improved gradually because we are leveraging the team intelligence [don’t make the same mistake twice]</p><p>dbt itself is worth a series of articles, I only use one of the features, to learn more about dbt, try their tutorial: <a href="https://docs.getdbt.com/tutorial">https://docs.getdbt.com/tutorial</a></p><p>Below are my immature opinions on dbt:</p><ul><li>Integrate dbt testing to existing data warehouse project is straightforward, just start a dbt project, set the source.yml and you can start writing the tests. Don’t need to migrate the transformation (T in ELT) part to there yet! Strongly suggested for existing projects if your current data warehouse is ‘test-naked’</li><li>I think dbt transformation is great if your data size is small [that you can afford full tables generation]. But for partitioned data, I don’t think the materialization is mature enough for big data yet, feel free to correct me if you know how to precisely update one partition. I will suggest using it to manage the views as the start!</li></ul><blockquote>I am not biased to dbt, you should also checkout <a href="https://greatexpectations.io/">https://greatexpectations.io/</a></blockquote><h3>Automate the work: GitHub Actions</h3><p>To recap what we have achieved so far, we have the script to get the data, load &amp; query them in BigQuery successfully, include data testing using dbt. The things seem quite automated already except someone needs to run the command from time to time :(</p><p>I definitely don’t want to do this myself, instead, we will use GitHub Actions as our CI &amp; scheduler and make the workflow. Recall what we want it to do:</p><ol><li>At a specific time daily, kickstart the workflow</li><li>Using the script to download data and load it to BigQuery</li><li>Run dbt test</li><li>Save the downloaded data to our GitHub repo as well [optional]</li></ol><blockquote>With clear goals in mind, time to Google!</blockquote><p>Firstly, I found this <a href="https://github.com/tigger0jk/ark-invest-scraper/blob/main/.github/workflows/scrape.yml">nice GitHub repo</a> to download the data from the ARK site using GitHub Actions workflow, except it only downloads the latest data versus what we want to keep every date. But this is nice as our base template!</p><p>Then I want to load data to BigQuery, I definitely don’t want to allow random people to load data to my BigQuery, therefore some credentials things are needed. After some google, I found <a href="https://github.com/google-github-actions/setup-gcloud">google-github-actions repo</a> to set credentials in GitHub Actions</p><p>Next, we want to run dbt test in GitHub Actions and I found code snippet by others, like <a href="https://gist.github.com/troyharvey/d61ebe704395c925bf9448183e99af3e">this</a></p><p>With many trial-and-errors, I get <a href="https://github.com/G-Hung/project-ark/blob/main/.github/workflows/scrape.yml">my final script</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*iw0XOlXUggj287AWy03kaA.png" /><figcaption>Trial and error with GitHub Actions</figcaption></figure><h3>Code Structure:</h3><p>The number represents my development steps, one means my first step</p><pre>project-ark<br>├── README.md<br>├── ark-track-dbt           # dbt project folder            <strong>(3)</strong><br>│   ├── README.md<br>│   ├── analysis<br>│   ├── data<br>│   ├── dbt_modules<br>│   ├── dbt_project.yml     # Define project and profile to use<br>│   ├── macros<br>│   ├── models<br>│   │   └── source.yml      # Define source data and schema test<br>│   ├── profiles.yml        # Define BigQuery data source setting<br>│   ├── snapshots<br>│   └── tests               # Contain tests for data quality check<br>├── data                    # Contain processed data        <strong>(1)</strong><br>│   ├── 2021-02-19<br>│   │   ├── ARKF.csv<br>│   │   ├── ...<br>│   └── tmp<br>├── misc<br>│   └── bq_schema.txt       # Table schema in BigQuery      <strong>(2)</strong><br>├── requirements.txt        # Required Python Package for GitHub Actions<br>├── scripts<br>│   ├── bq_load.py          # script to load csv to BigQuery<strong>(2)</strong><br>│   └── collect_and_load.sh # script to extract &amp; load data <strong>(1)</strong><br>│── .github/workflows       # GitHub Action script          <strong>(4)</strong></pre><h3>Conclusion:</h3><p>That’s it, I have a workflow to run daily at 0030, 0830, 1630 to download data from the ARK site, load to BigQuery, do quality check using dbt, push the downloaded data to GitHub repo if any.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RsuCZEs7T-cXxkkJFAHMvg.png" /><figcaption>Scheduled workflow</figcaption></figure><p>We use GitHub Actions workflow as a scheduler and Google BigQuery as a database, all within free quota. And the most amazing part is, they are detached from me and my computer, I don’t need to concern the service reliability too much.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*0EP0RRd_DfedBsPT.png" /><figcaption>My attitude toward outsourcing service reliability to the cloud services-from &lt;The Hunger Game&gt;</figcaption></figure><p>If next time the data source isn’t ARK site anymore, of coz I need to modify some codes but the workflow is similar and scalable to much bigger data (as big as BigQuery can’t handle it OR you can’t afford the bill, I guess the later come first)</p><p>OK, the article is longer than I expected, let’s stop here, I hope you find this article useful!</p><p>you can star my <a href="https://github.com/G-Hung/project-ark">repo</a>!</p><p>OR add <a href="https://www.linkedin.com/in/geoffreyhung/">my LinkedIn</a> [Welcome but please leave a few words to indicate you are not a zombie]!</p><blockquote>Stay safe everyone! Remember to wear the mask!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d79c7c108450" width="1" height="1" alt=""><hr><p><a href="https://medium.com/usf-datascience/from-curl-to-automated-workflow-d79c7c108450">From cURL to Automated Workflow</a> was originally published in <a href="https://medium.com/usf-datascience">USF-Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Scripts To Prediction API]]></title>
            <link>https://medium.com/data-science/from-scripts-to-prediction-api-2372c95fb7c7?source=rss-cf2c9ab57bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/2372c95fb7c7</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Geoffrey Hung]]></dc:creator>
            <pubDate>Sun, 16 Aug 2020 17:11:01 GMT</pubDate>
            <atom:updated>2021-01-25T02:58:52.638Z</atom:updated>
            <content:encoded><![CDATA[<h4>submission.csv? No thanks!</h4><p>This is the continuation of my previous article:</p><p><a href="https://towardsdatascience.com/from-jupyter-notebook-to-sc-582978d3c0c">From Jupyter Notebook To Scripts</a></p><p>Last time we discussed how to convert Jupyter Notebook to scripts, together with all sorts of basic engineering practices such as CI, unit testing, package environment, configuration, logging, etc</p><p>Even with the script form, it still requires us to change the configuration and run the script, It is OK for Kaggle competitions because all you need is just the <strong>submission.csv</strong>, but you probably don’t want to sit behind the computer 24/7 and hit Run whenever users send you a prediction request 🙁</p><p>In this article, we will discuss how to utilize the models we have built last time and create prediction APIs to do model serving using FastAPI!</p><blockquote>For ML/DL folks, we are talking about <a href="https://fastapi.tiangolo.com/">FastAPI</a>, NOT <a href="https://www.fast.ai/">fast.ai</a> !!!</blockquote><h3>Background: FastAPI</h3><p>There are many frameworks in Python ecosystem for API, my original thought was to use Flask. But I am impressed by how simple and intuitive [and fast as the name suggests] FastAPI is and love to try it out in this mini-project!</p><p>“Rome wasn’t built in a day”, FastAPI has learned a lot from the previous frameworks such as Django, Flask, APIStar, I cannot explain better than the creator himself and <a href="https://fastapi.tiangolo.com/history-design-future/">this article</a> is great!</p><h3>The boring but necessary setup</h3><p>Things are all in <a href="https://github.com/G-Hung/model-productization_article">one repo</a> which is probably not a good practice, it should be different GitHub repo in the real use case, maybe I will <em>refactor</em> [professional way to say “clean my previous sxxt”] later!</p><p>*CS folks always say <em>single responsibility principle</em>, instead of saying “don’t put code with different functionalities together”, next time maybe you can say “we should follow single responsibility principle for this!”</p><p>First of all, let’s update requirements.txt with the new packages, as we mentioned last time, we should specify the <strong>exact version</strong> such that others can reproduce the work!</p><pre># for last article<br>pytest==6.0.1<br>pandas==1.0.1<br>Click==7.0<br>scikit-learn==0.22.1<br>black==19.10b0<br>isort==4.3.21<br>PyYAML==5.2</pre><pre># for FastAPI<br>fastapi==0.61.0<br>uvicorn==0.11.8<br>chardet==3.0.4</pre><p>After this, we need to install requirements.txt again in the conda env [because we have new packages]</p><pre><em># You can skip the line below if you have created conda env</em><br>conda create - name YOU_CHANGE_THIS python=3.7 -y</pre><pre>conda activate YOU_CHANGE_THIS</pre><pre>pip install –r requirements.txt</pre><h3>The game plan</h3><p>Let’s think about what is happening, we want to have API endpoint to do prediction, to be specific if users give <strong>us</strong> the input, we need to use the model to predict and return prediction.</p><p>Instead of having <strong>us</strong> [human] handle the incoming request, we just create an API server to wait for the requests, parse the inputs, do prediction, and return results. API is just the structural way to talk to our computer and ask for the service [prediction in this case]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GsXTWSGM4l1NsK7427Riyg.png" /><figcaption>What is happening in high-level</figcaption></figure><p>Below is the pseudocode:</p><pre><em># Load trained model</em><br>trained_model = load_model(model_path)</pre><pre><em># Let&#39;s create a API that can receive user request</em><br>api = CreateAPI()</pre><pre><em># If user send us the request to `predict` endpoint</em><br>when user sends request to `api`.`predict`:<br>    input = api[`predict`].get(input) # get input<br>    prediction = trained_model(input) # apply model<br>    return prediction                 # return prediction</pre><p>This is good for the happy flow! But we should <em>NEVER</em> trust the user, just ask yourself, will you ever read the user manual in your daily life?</p><p>For example, we expect {‘a’: 1, ‘b’: 2, ‘c’: 3} from the user but we may get:</p><ul><li>Wrong order {‘b’: 2, ‘a’: 1, ‘c’: 3}, or</li><li>Wrong key {‘a’: 1, ‘b’: 2, ‘d’: 3}, or</li><li>Missing key{‘a’: 1, ‘b’: 2}, or</li><li>Negative value {‘a’: -1, ‘b’: 2, ‘c’: 3}, or</li><li>Wrong type {‘a’: “HELLO WORLD”, ‘b’: 2, ‘c’: 3}, or</li><li>etc etc</li></ul><p>This is fatal to our API because our model doesn’t know how to respond to this. We need to introduce some input structures to protect us! Therefore, we should update our pseudocode!</p><pre><strong># Define input schema<br>input_schema = {......}</strong></pre><pre><em># Load trained model</em><br>trained_model = load_model(model_path)</pre><pre><em># Let&#39;s create a API that can receive user request</em><br>api = CreateAPI()</pre><pre><em># If user send us the request to `predict` endpoint</em><br>when user sends request to `api`.`predict`:<br>    input = api[`predict`].get(input) # get input<strong>           <br>    transformed_input = apply(input_schema, input)<br>    if not transformed_input.valid(): return Error<br></strong>    prediction = trained_model(<strong>transformed_input</strong>) # apply model<br>    return prediction                 # return prediction</pre><h3>The Code</h3><p>Looks good to me now! Let’s translate them using FastAPI part by part!</p><p><strong>Input schema</strong></p><p>It seems many lines but things are the same, as you can guess, we define a class called `Sample` which defines every predictor as <em>float</em> and <em>greater than [gt] zero!</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2051712d1e0c2dd18c6a4db8203c1fdc/href">https://medium.com/media/2051712d1e0c2dd18c6a4db8203c1fdc/href</a></iframe><p><strong>Load model</strong></p><p>Then we load the trained model, hmmm what is `Predictor`? it is just a custom class that wraps the model with different methods so we can call a method instead of implementing the logic in API server</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f39abc4879a591b5f9c2609cf0ae8edb/href">https://medium.com/media/f39abc4879a591b5f9c2609cf0ae8edb/href</a></iframe><p><strong>Create an API server</strong></p><p>Then we create the API using FastAPI……pseudocode is almost the code already</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b1fe7b47c26a23b3abdc9d3ca9804a4/href">https://medium.com/media/6b1fe7b47c26a23b3abdc9d3ca9804a4/href</a></iframe><p><strong>predict endpoint</strong></p><p>This looks complicated but they are very straightforward</p><p>Instead of saying <em>“when the user sends a request to `api`.`predict`”</em></p><p>We say: <em>“Hey, app, if people send “</em><strong><em>GET</em></strong><em> request” to `predict`, please run function predict_item, we expect the input follows the schema we defined in `Sample`”</em></p><p>What <em>predict_item</em> does is only transform input shape, feed to trained model and return the prediction, simple Python function</p><blockquote>If you want to know more about <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods">HTTP request methods</a></blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08ed05187fb021647d87c2c4b137eebf/href">https://medium.com/media/08ed05187fb021647d87c2c4b137eebf/href</a></iframe><p>But you may ask: hey! One line is missing!!! Where is the input validation? What if users provide the wrong data type/key or miss a field?</p><p>Well…….Remember we have defined `Sample` class for the input schema? Fast API <strong>automatically</strong> validates it for us according to the schema and we don’t need to care about that!!! This saves a lot of brainpower and many lines of code to build a robust and well-tested API!</p><h3>Try to use</h3><pre># At project root, we can run this<br># --reload is for development, API server autorefresh<br># when you change the code</pre><pre>uvicorn prediction_api.main:app --reload</pre><p>You should be able to see these, the API server is now running on “http://127.0.0.1:8000”!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*xs5rUO80zSvDT7o-skVubw.png" /></figure><p>There are different ways to experiment with the API, depending on your environment, you may use <em>requests in Python</em> or <em>cURL in the command line. </em>BTW there is a handy tool is called <a href="https://www.postman.com/">Postman</a>, try it out, it is a very intuitive and user-friendly tool for API!</p><p>We will use Python requests for the following examples, you can see them in <a href="https://github.com/G-Hung/model-productization_article/blob/master/notebook/prediction_API_test.ipynb">this Notebook</a> [sometimes Jupyter is helpful 😎]</p><blockquote>Example below uses a valid input: YEAH! 😍 We made it! The endpoint returns the prediction!!!</blockquote><pre>payload = {<br>    &quot;fixed_acidity&quot;: 10.5,<br>    &quot;volatile_acidity&quot;: 0.51,<br>    &quot;citric_acid&quot;: 0.64,<br>    &quot;residual_sugar&quot;: 2.4,<br>    &quot;chlorides&quot;: 0.107,<br>    &quot;free_sulfur_dioxide&quot;: 6.0,<br>    &quot;total_sulfur_dioxide&quot;: 15.0,<br>    &quot;density&quot;: 0.9973,<br>    &quot;pH&quot;: 3.09,<br>    &quot;sulphates&quot;: 0.66,<br>    &quot;alcohol&quot;: 11.8,<br>}</pre><pre>result = requests.get(&quot;<a href="http://127.0.0.1:8000/predict">http://127.0.0.1:8000/predict</a>&quot;, data = json.dumps(payload))</pre><pre>print(result.json())</pre><pre><strong>Output</strong><br>{&#39;prediction&#39;: 1, &#39;utc_ts&#39;: 1597537570, &#39;model&#39;: &#39;RandomForestClassifier&#39;}</pre><blockquote>Example below misses a field and FastAPI helps us to handle it according to our defined schema, I literally write nothing other than the schema class</blockquote><pre>payload = {<br>    &quot;volatile_acidity&quot;: 0.51,<br>    &quot;citric_acid&quot;: 0.64,<br>    &quot;residual_sugar&quot;: 2.4,<br>    &quot;chlorides&quot;: 0.107,<br>    &quot;free_sulfur_dioxide&quot;: 6.0,<br>    &quot;total_sulfur_dioxide&quot;: 15.0,<br>    &quot;density&quot;: 0.9973,<br>    &quot;pH&quot;: 3.09,<br>    &quot;sulphates&quot;: 0.66,<br>    &quot;alcohol&quot;: 11.8,<br>}</pre><pre>result = requests.get(&quot;<a href="http://127.0.0.1:8000/predict">http://127.0.0.1:8000/predict</a>&quot;, data = json.dumps(payload))</pre><pre>print(result.json())</pre><pre><strong>Output</strong><br>{&#39;detail&#39;: [{&#39;loc&#39;: [&#39;body&#39;, &#39;fixed_acidity&#39;], &#39;msg&#39;: &#39;field required&#39;, &#39;type&#39;: &#39;value_error.missing&#39;}]}</pre><p>Just for fun, I also implemented a <em>update_model</em> PUT API to swap the models, for example, originally we were using Random Forest, I updated it to Gradient Boosting☺️</p><pre>result = requests.put(&quot;<a href="http://127.0.0.1:8000/update_model">http://127.0.0.1:8000/update_model</a>&quot;)</pre><pre>print(result.json())</pre><pre><strong>Output</strong><br>{&#39;old_model&#39;: &#39;RandomForestClassifier&#39;, &#39;new_model&#39;: &#39;GradientBoostingClassifier&#39;, &#39;utc_ts&#39;: 1597537156}</pre><h3>Auto-generated Document</h3><p>One of the cool FastAPI features is auto-document, just go to <a href="http://127.0.0.1:8000/docs#/">http://127.0.0.1:8000/docs#/</a> and you will have the interactive and powerful API document out of the box! So intuitive that I don’t need to elaborate</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z9kx20rUhTZU6FhqZVtr1A.png" /></figure><h3>Revisit pytest</h3><p>I cannot emphasize enough about the importance of unit testing, it verifies the functions are doing what we expect them to do such that you will not break things accidentally!</p><p>But if I try to cover every test it will be too boring and lengthy. What I plan to do here is to share some areas I will brainlessly test &amp; some [probably useful] articles. Then I will talk about a pytest feature called parameterized unit test and some testing options in pytest. The easiest way to motivate yourself to learn unit testing is to try to refactor your previous code, the larger the better!</p><p><strong>Unit testing</strong></p><p>Whenever you found difficult to write/understand the unit tests, you probably need to review your code structure first. Below are 4 areas that I will consider brainlessly:</p><ol><li>Input data: dimension [eg: df.shape], type [eg: str], value range [eg: -/0/+]</li><li>Output data: dimension [eg: df.shape], type [eg: str], value range [eg: -/0/+]</li><li>Compare: output and expected result</li><li>After I debug, prevent it happens again</li></ol><p>For example, I focus quite a lot on output dimension, type, value range below. It seems simple but if you modify any output format, it will remind you what is the expected formats!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b684c57cad48b2cda68b3b8f3e4a1f89/href">https://medium.com/media/b684c57cad48b2cda68b3b8f3e4a1f89/href</a></iframe><p>Some articles FYR:</p><p><a href="https://towardsdatascience.com/unit-testing-for-data-scientists-dc5e0cd397fb">Unit Testing for Data Scientists</a></p><p><a href="https://medium.com/@keeper6928/how-to-unit-test-machine-learning-code-57cf6fd81765">How to unit test machine learning code</a> [deep learning]</p><p><strong>Parameterized unit test</strong></p><p>Suppose you have 100 mock data [annotate by D_i, i: 1..100] and you want to run the same unit test for each of them, how will you do?</p><blockquote><em>A brute force solution</em></blockquote><pre>def test_d1():<br>    assert some_operation(D_1)</pre><pre>def test_d2():<br>    assert some_operation(D_2)</pre><pre>def test_d3():<br>    assert some_operation(D_3)</pre><pre>......</pre><pre>def test_d100():<br>    assert some_operation(D_100)</pre><p>But if you need to modify `some_operation`, you need to modify it 100 times LOL………Although you can make it as a utility function, this makes the tests hard to read and very lengthy</p><blockquote><em>A better way maybe for-loop?</em></blockquote><pre>def test_d():<br>    for D in [D_1, D_2, D_3, ..., D_100]:<br>        assert some_operation(D)</pre><p>But you can’t know exactly which tests fail because these 100 tests are all in one test</p><blockquote><em>pytest offers us a feature called </em><strong><em>parametrize</em></strong></blockquote><pre><a href="http://twitter.com/pytest">@pytest</a>.mark.parametrize(&quot;test_object&quot;, [D_1, D_2, ..., D_100])<br>def test_d(test_object):<br>    assert some_operation(test_object)</pre><p><strong>Common pytest options</strong></p><blockquote><strong>pytest FOLDER</strong></blockquote><p>Last time we mentioned we can just run `<strong>pytest</strong>` in command line and pytest will find <strong>ALL</strong> the tests under the folder itself. But sometimes we may not want to run all the unit tests during development [maybe some tests take a long time but unrelated to your current tasks]</p><p>In that case, you can simply run <strong>pytest FOLDER</strong>, eg: `pytest ./scripts` or `pytest ./prediction_api` in the demo</p><blockquote><strong>parallel pytest</strong></blockquote><p>Sometimes your test cases are too heavy, it may be a good idea to run things in parallel! You can install <strong>pytest-xdist</strong> and replace pytest by py.test in your command, eg: py.test -n 4</p><blockquote><strong>pytest -v</strong></blockquote><p>This is personal taste, I prefer the verbose output and see green PASSED ✅ to start my day</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*sUDohl0QJYRXPYH4UI1Afw.png" /><figcaption>pytest</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*4DnxRvvL1ajmXBrkQWh-VQ.png" /><figcaption>pytest -v</figcaption></figure><blockquote>You can read more from the materials below:</blockquote><blockquote><a href="https://docs.pytest.org/en/stable/">https://docs.pytest.org/en/stable/</a></blockquote><blockquote><a href="https://www.guru99.com/pytest-tutorial.html#5">https://www.guru99.com/pytest-tutorial.html#5</a></blockquote><h4>At last, I hope you can enjoy <a href="https://www.youtube.com/watch?v=rR4n-0KYeKQ">this 1 min Youtube video</a> as I do 😆</h4><h3>Conclusions</h3><p>Yooo✋ we have created a prediction API that consumes our model, users can now send the request and get the prediction without a human sitting behind, this over-simplifies the reality [throughput, latency, model management, authentication, AB testing, etc] but this is the idea!</p><p>At least if your prototype is at this level, engineers are much happier to take over from this point and hence speed up the whole process, and you can show them you know something 😈</p><p>To wrap up, we:</p><pre>a. Update conda env [requirements.txt]<br>b. Brainstorm pseudocode and convert to code [FastAPI, uvicorn]<br>c. Utilize API [cURL, requests, Postman]<br>d. Talk about Auto-generated documents by FastAPI<br>e. Some pytest techniques [parallel, parameterized, -v]</pre><p>File tree below to show the development steps</p><pre>.<br>├── notebook<br>│   ├── prediction-of-quality-of-wine.ipynb<br>│   └── prediction_API_test.ipynb           [c] &lt;-consume API<br>├── prediction_api<br>│   ├── __init__.py<br>│   ├── api_utility.py                      [b] &lt;-wrap up methods<br>│   ├── main.py                             [b] &lt;-modify demo<br>│   ├── mock_data.py                        [e] &lt;-Unit test<br>│   ├── test_api_utility.py                 [e] &lt;-Unit test<br>│   └── test_main.py                        [e] &lt;-Unit test<br>├── requirements.txt                        [a] &lt;-FastAPI doc<br>.<br>.<br>.</pre><p><strong>BUT</strong> (again, bad news usually starts with BUT) they are still on my <strong>local</strong> computer.</p><p>Although we don’t need to sit behind and hit Run, the user requests cannot reach the API endpoints. Even if they do, it means I cannot close my Macbook, it means I cannot scale if there are many incoming prediction requests😱!!!</p><p>The way to escape from this hell, as we mentioned in the last article, is to either buy another computer OR rent a server from cloud providers such as AWS</p><p>But first, we also need to ensure the code is working fine there! How?</p><p>Short answer: Docker</p><p><strong>Aside:</strong></p><p>Although I haven’t tried, there is a startup called <a href="https://docs.cortex.dev/">Cortex</a> which focuses on open source machine learning API framework and they also <a href="https://towardsdatascience.com/why-we-switched-from-flask-to-fastapi-for-production-machine-learning-765aab9b3679">use FastAPI</a> under the hood!</p><p>By now, you should be able to understand their tutorial, in short, they solve many production-level problems behind the scene such as rolling update, DL model inference, integration with AWS, Autoscaling etc…… [These are <em>DevOps</em> concerns? Or maybe fancier term: <em>MLOps</em>]</p><p>But from user [aka you] perspective, they deploy the APIs using declarative yml [similar to how we configure model in the last article], have a predictor class [similar to our Predictor class], trainer.py [similar to train.py in the last article]</p><p>Writing the code is relatively easy but writing an article for the code is hard, if you found this article useful, you can leave some comments</p><p>OR you can star my <a href="https://github.com/G-Hung/model-productization_article">repo</a>!</p><p>OR <a href="https://www.linkedin.com/in/geoffreyhung/">my LinkedIn</a> [Welcome but please leave a few words to indicate you are not zombie]!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2372c95fb7c7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/from-scripts-to-prediction-api-2372c95fb7c7">From Scripts To Prediction API</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Jupyter Notebook To Scripts]]></title>
            <link>https://medium.com/data-science/from-jupyter-notebook-to-sc-582978d3c0c?source=rss-cf2c9ab57bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/582978d3c0c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Geoffrey Hung]]></dc:creator>
            <pubDate>Fri, 07 Aug 2020 02:31:00 GMT</pubDate>
            <atom:updated>2022-02-05T01:32:35.072Z</atom:updated>
            <content:encoded><![CDATA[<h4>Don’t play toy models; be ready to productionalize your works!</h4><blockquote>16 Aug: <a href="https://towardsdatascience.com/from-scripts-to-prediction-api-2372c95fb7c7">From Scripts To Prediction API</a></blockquote><blockquote>23 Feb: <a href="https://medium.com/@geoffreyhung/from-curl-to-automated-workflow-d79c7c108450">From cURL to Automated workflow</a></blockquote><blockquote>2022 Jan: <a href="https://medium.com/@geoffreyhung/python-for-advanced-beginners-cd39b42ef10">Python for advanced beginners</a></blockquote><p>As advanced beginners, we know quite a lot: EDA, ML concepts, model architectures etc…… We can write a big Jupyter Notebook, click “Restart &amp; Run All” and generate state-of-art results. OK, what next? How can others use/reproduce/scale your results?</p><p>In this article, I will show you how I convert a Jupyter Notebook from Kaggle on the Wine dataset to scripts, share some engineering practices that we can adapt easily and bring benefits right away.</p><p>For each of them, there is much more to dig in but we will only surface with the basics, show the benefits, and explain the thought process! The goal is to minimize the probability that the engineers roll their eyes when they talk to you.</p><h3>Jupyter Notebook</h3><p>Jupyter Notebook is nice for exploration and prototype but we cannot stay there forever. No engineers [Or no one] wants to read your code and rewrite it. Even you are fortunate enough to have engineering resources, it just slows the whole process down🙁</p><p>Somehow we need to adapt engineering practices, make it reproducible, automatable, and deployable! The first step is to convert the notebook to scripts and it is the main theme of this article!</p><p><a href="https://github.com/G-Hung/model-productization_article">&gt;&gt;&gt;GitHub repo&lt;&lt;&lt;</a></p><p>All credits of <a href="https://www.kaggle.com/vishalyo990/prediction-of-quality-of-wine">this great notebook</a> in repo go to the original creator!</p><h3>Scripts: ipynb -&gt; .py</h3><p>Why script? Why can’t we just use the interactive notebook when it is working fine? Let me propose some scenarios to justify why it may be a good idea to use scripts beyond exploration!</p><ol><li>If you want to run 20 big models in parallel, will you open 20 notebooks?</li><li>Suppose you are willing to open 20 notebooks, how will you configure the hyperparameters? Go to each notebook and find related cells?</li><li>Suppose you manually configure 20 notebooks, how will you analyze the results? Go to each notebook?</li><li>Suppose you can handle 20 notebooks for the first 3 steps, what if you have a great idea but you need to experiment with 200 sets of hyperparameters, are you willing to repeat steps 1–3?</li><li>Suppose you find a bug in the original notebook and need to rerun everything, are you willing to repeat steps 1–3?</li></ol><p>If you answer YES to all, then this article is NOT for you, bye and have a good day 😊</p><p>The way I structure the pipeline is to break it down into <strong>etl.py</strong>, <strong>train.py</strong>, and <strong>predict.py</strong>. Each script has its responsibility and tries to be as atomic as possible such that they can be developed by individual teammates as long as the responsibility is clear.</p><p><strong>etl.py</strong>: convert the raw data, do some transformation, create train.csv and test.csv under data folder [train_test_split]</p><p><strong>train.py</strong>: use data in train.csv, export a model to model folder [model.fit(X, y)]</p><p><strong>predict.py</strong>: use data in test.csv and trained model, export the predictions, and evaluate [model.predict(X, y)]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*nqbLGPriluEcI1IyXqv01Q.png" /><figcaption>Script relationship, each handles their own part</figcaption></figure><pre># File structure<br>.<br>├── README.md<br>├── __init__.py<br>├── autoformat.sh<br>├── data<br>│   ├── predict.csv<br>│   ├── test.csv<br>│   ├── train.csv<br>│   └── winequality.csv<br>├── log<br>│   ├── etl.log<br>│   ├── predict.log<br>│   └── train.log<br>├── model<br>│   └── model.pkl<br>├── notebook<br>│   └── prediction-of-quality-of-wine.ipynb<br>├── requirement.txt<br>└── scripts<br>    ├── config.yml<br>    ├── <strong>etl.py</strong><br>    ├── <strong>predict.py</strong><br>    ├── test_train.py<br>    ├── test_utility.py<br>    ├── <strong>train.py</strong><br>    └── utility.py</pre><p>If I mess up something, I will not try to figure out why [imagine state mutation in Jupyter because you random run the cells], I will just run the script again and see.</p><pre># Run in command line, as simple as this :)</pre><pre>$ python3 ./scripts/etl.py<br>$ python3 ./scripts/train.py<br>$ python3 ./scripts/predict.py</pre><p>Great, we can load and run the scripts in the command line! But as we mentioned in the scenarios above [№2], how can we configure the hyperparameters then ?_?</p><h3>Config file as input: click</h3><p>The principle is to try not to touch the code, I have created another config.yml to contain all the configurations and load them via <a href="https://click.palletsprojects.com/en/7.x/">click</a>. In this way, every new model is just another configuration, this ensures the reproductivity and avoids manually changing the code [and introduce bugs 🙁 ].</p><pre># config.yml example, I only use this to configure the scripts<br>etl:<br>  raw_data_file: &quot;data/winequality.csv&quot;<br>  processed_path: &quot;data&quot;<br>  test_size: 0.2<br>  random_state: 42</pre><pre>train:<br>  processed_train: &quot;data/train.csv&quot; <br>  ensemble_model: &quot;RandomForestClassifier&quot;<br>  model_config: {n_estimators: 300}<br>  model_path: &quot;model/model.pkl&quot;</pre><pre>predict:<br>  model_path: &quot;model/model.pkl&quot;<br>  processed_test: &quot;data/test.csv&quot;<br>  predicted_file: &quot;data/predict.csv&quot;<br>  export_result: True</pre><p>Inside the script, I created a utility function to parse the configuration file</p><pre>def parse_config(config_file):<br>    with open(config_file, &quot;rb&quot;) as f:<br>        config = yaml.safe_load(f)<br>    return config</pre><p>Then we can load the configurations to the scripts in an extensible way!</p><pre><a href="http://twitter.com/click">@click</a>.command()<br><a href="http://twitter.com/click">@click</a>.argument(&quot;config_file&quot;, type=str, default=<strong>&quot;scripts/config.yml&quot;</strong>)<br>def etl(config_file):<br>    config = <strong>parse_config</strong>(config_file)<br>    raw_data_file = config[&quot;etl&quot;][&quot;raw_data_file&quot;]<br>    processed_path = Path(config[&quot;etl&quot;][&quot;processed_path&quot;])</pre><pre>.......DO SOMETHING.......</pre><p>If we have 20 models with a similar architecture, we can just prepare 20 configurations, that’s it</p><h3>No print: logging</h3><p>We can now combine the script and configuration in the command line environment! How can I get the outputs? In the Jupyter world, I will print and see but it means I need to scan through a long long notebook, what if I have many models? Is there any concrete way to collect the outputs in a file?</p><p>Let print go, learn <a href="https://docs.python.org/3/howto/logging.html">logging</a>! Logging module in Python does what its name suggests, do logging in Python. We can define and log the things we are interested in, for example, which stage of the script [for debug], a summary of the metrics [for evaluation], I extracted a piece from model prediction and evaluation as an example</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aed8b646cabd7bd0364ef10dfb730d89/href">https://medium.com/media/aed8b646cabd7bd0364ef10dfb730d89/href</a></iframe><p>This piece of code will generate the log similar to the one below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d8b53664ee98faabd1329bc8466422f/href">https://medium.com/media/3d8b53664ee98faabd1329bc8466422f/href</a></iframe><p>Learn more about logging from this article! <a href="https://www.machinelearningplus.com/python/python-logging-guide/">https://www.machinelearningplus.com/python/python-logging-guide/</a></p><p><strong>Aside:</strong></p><p>Logging is great but still if we have 100 experiments, we will get 100 log files 🙁 Even they are concrete, we may not want to read them one by one, is there any better way to evaluate? Below are some potential solutions:</p><ul><li>Yet another script, utilize grep or regex, parse and extract the key information [but what if someone changes the format or spelling!?]</li><li>Yet another tool, checkout <a href="https://mlflow.org/docs/latest/projects.html">MLFlow</a> or <a href="https://github.com/VertaAI/modeldb">ModelDB</a>! [Logging is only one of the features of both products]</li></ul><h3>Reproducibility matters: Conda env</h3><p>Yo, things seem good now! I can push to GitHub and my friends/colleagues can clone and run in <strong>their </strong>computer 😊</p><p>Oh wait……..their computer……..what if we are using different versions? It is common because every package has many versions, eg: <a href="https://scikit-learn.org/stable/whats_new.html">sklearn</a>. The APIs / interfaces may change over time [and breaks the code 🙁 ], how can we ensure we are using the same package?</p><p>Short answer: Conda env, check out <a href="https://towardsdatascience.com/a-guide-to-conda-environments-bc6180fc533">Matt’s article</a></p><p>I list out some useful commands below, just enough for our needs</p><pre># In terminal</pre><pre># Create the environment<br>$ conda create — name YOU_CHANGE_THIS python=3.7 -y</pre><pre># Activate it<br>$ conda activate YOU_CHANGE_THIS</pre><pre># Install the needed package<br>$ pip install –r requirements.txt</pre><pre># # ## # ## # ## # ## # ## # ##<br># # # Do your development # # #<br># # ## # ## # ## # ## # ## # ##</pre><pre># Deactivate and back to normal environment<br>$ conda deactivate</pre><pre># This will remove the environment, do it when you finish the project?<br>$ conda remove –name YOU_CHANGE_THIS –all</pre><p>*in requirements.txt, you should always specify the version! If you don’t specify it, it will try to download the latest one, but your latest maybe not my latest and lose the whole point of using Conda env</p><pre># this is what I have inside requirements.txt<br>pytest==5.3.5<br>pandas==1.0.1<br>Click==7.0<br>scikit-learn==0.22.1<br>black==19.10b0<br>isort==4.3.21<br>PyYAML==5.2</pre><h3>Don’t break your previous code: pytest</h3><p>In the software / ML world, change is the constant, we change something whenever we work. But how can we ensure the new change doesn’t break the existing one [especially those you wrote a few months before]?</p><p>Unit tests are here to help 😊 We can write the test cases for each function to ensure they have the expected behavior. Unit tests cannot prove program correctness but they can safeguard the scenarios you listed!</p><p>For example, the code below asserts the type and output dimension are as expected. It seems silly and simple but if your load_data function becomes more complex, you will appreciate something safeguard you all the way [especially if you want to change something, don’t ask me how I know]!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/37d97b5f809f8694bea8f25312d88335/href">https://medium.com/media/37d97b5f809f8694bea8f25312d88335/href</a></iframe><p>To run the unit test, simply run <strong>pytest</strong> in the terminal and you should see the output like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*koXC4iwTNJW6gAvwL7RVVQ.png" /></figure><p>Cool, we can run pytest before we push to GitHub and try to maintain a workable program! But human isn’t reliable, what if we forget to run pytest before push and break the code!? How can we protect ourselves from this situation?</p><h3>If possible, don’t rely on human: Continuous Integration</h3><p>What if the program itself can run the unit tests automatically when we push/create pull request? Then it can stop the merge if it cannot pass the tests! If you heard of continuous integration, this is part of the feature! Popular products in the market are CircleCI / TravisCI / jenkins, but I am lazy, I don’t want to do any extra configuration work if not needed so we will use relatively new GitHub Action for that 😊</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ca497bdd0e3d6f15af8197311ab2359/href">https://medium.com/media/9ca497bdd0e3d6f15af8197311ab2359/href</a></iframe><p>In the past, I think the engineers are so smart, how can they remember the syntax and come up with a script like this [it is simple already] on their own? Don’t think in that way, most people just copy &amp; modify the template as I do [there is a template called <strong>Publish Python package</strong> under CI workflows in GitHub Actions]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5CKfEB6aWrt4qKxcorUFuA.png" /></figure><p>Now you see it! The scripts, package versions, and pytest work together to ensure the package versions are right and the program is workable! They will safeguard us when we develop incrementally, they will say NO if the code breaks the unit tests. If we find a bug, we can include that as another test case to ensure we don’t commit the same mistake twice!</p><p>Can you do the same in Jupyter Notebook with the same level of confidence?</p><p><strong>Code Style matters: black, isort</strong></p><p>Have you heard of <a href="https://www.python.org/dev/peps/pep-0008/">pep8</a>? It is the style convention to write Python, there are many (good) points but honestly, please don’t test me on that, my brain cannot remember them all!</p><p>I remember in one of my classes, my professor requires us to use <a href="https://pypi.org/project/pycodestyle/">pycodestyle</a> it is a tool to check the code style, then I got these messages and fix them one by one. For example:</p><pre>$ pycodestyle --first optparse.py<br>optparse.py:69:11: E401 multiple imports on one line<br>optparse.py:77:1: E302 expected 2 blank lines, found 1<br>optparse.py:88:5: E301 expected 1 blank line, found 0</pre><p>I think it is a good tool but kind of incomplete, if you can find my problems, why don’t you just correct that for me and save my time?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/807c4729b6e3d70a20d4ca049e166118/href">https://medium.com/media/807c4729b6e3d70a20d4ca049e166118/href</a></iframe><pre>Simply run this and the script will do the rest!</pre><pre># If you get permission error, you can try<br># chmod +rx autoformat.sh<br>$ ./autoformat.sh</pre><h3>Conclusions</h3><p>Things are good for <strong>local</strong> now! I can push them to GitHub and my friends can clone and run it in their computer</p><p>BUT! We know the important (bad) messages usually start with BUT/HOWEVER, eg: job application rejections 🙁</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KQzyaHtfx3XCm6l-xq9HXw.png" /></figure><p>BUT what if we want to productionalize the model ?_?</p><p>Q: Firstly, what it means by productionalize?</p><p>A: In a narrow sense, it means model consumption, I give you the input, you return me the output [maybe in JSON form], eg: web endpoint / API. We need to have a server [another computer] to host the models and endpoints, I don’t want to use <strong>my computer</strong> to host it or the whole product will rely on my poor <strong>Macbook</strong>🙁</p><p>I <strong>highlighted </strong>two words above and they are exactly the challenges</p><p>1.How to find another computer to host the models?</p><p>Short answer: Either buy a computer OR rent one from cloud provider [eg: AWS, Azure, GCP], then we can claim we know CLOUD COMPUTING 😊</p><p>2.What if another computer isn’t MacOS, how can I ensure my code is working there?</p><p>Short answer: Docker</p><p><strong>To wrap up, we:</strong></p><pre>a. start from a Jupyter Notebook<br>b. Code conversion [ipynb to .py]<br>c. Make the scripts configurable [Click]<br>d. Include logging [logging]<br>e. Make sure the local environment is the same [Conda env]<br>f. Include unit test and basic CI [pytest, GitHub Action]<br>g. Autoformat the script style [black, isort]</pre><p>I hope file tree structure with annotations can show you the code evolution, don’t be scared by many folders because they are developed step-by-step, if you understand the thought process, it is pretty straightforward!</p><pre>.<br>├── README.md<br>├── __init__.py<br>├── .github/workflows         [f]<br>├── autoformat.sh             [g]<br>├── data<br>│   ├── predict.csv           [b]<br>│   ├── test.csv              [b]<br>│   ├── train.csv             [b]<br>│   └── winequality.csv<br>├── log<br>│   ├── etl.log               [d]<br>│   ├── predict.log           [d]<br>│   └── train.log             [d]<br>├── model<br>│   └── model.pkl             [b]<br>├── notebook<br>│   └── prediction-of-quality-of-wine.ipynb [a]<br>├── requirement.txt           [e]<br>└── scripts<br>    ├── config.yml            [c]<br>    ├── etl.py                [b, c]<br>    ├── predict.py            [b, c]<br>    ├── test_train.py         [f]<br>    ├── test_utility.py       [f]<br>    ├── train.py              [b, c]<br>    └── utility.py</pre><p><strong>Something we mentioned but didn’t elaborate:</strong></p><p>-&gt; Databricks’s MLFlow / VertaAI’s ModelDB</p><p><strong>Something we can discuss more:</strong></p><ul><li>Model deployment in cloud [AWS]</li><li>Containerization [Docker]</li></ul><p>Now we have built a solid foundation, these scripts are in the form that we can reuse it further! Imagine how can you do this with Jupyter Notebook?</p><p>I understand what it feels like when someone said you should try ABC / XYZ but they don’t explain why or how to evolve from scratch, I hope this article can contribute a little bit :)</p><p>If this article is useful, you can leave some comments</p><p>OR you can star my <a href="https://github.com/G-Hung/model-productization_article">repo</a>!</p><p>OR <a href="https://www.linkedin.com/in/geoffreyhung/">my LinkedIn</a> [Welcome but please leave a few words to indicate you are not zombie]!</p><p>I learned most of these things from my internship @ <a href="https://www.manifold.ai/manifold">Manifold AI</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=582978d3c0c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/from-jupyter-notebook-to-sc-582978d3c0c">From Jupyter Notebook To Scripts</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to fill OPT starting date this year — Data Approach]]></title>
            <link>https://medium.com/@geoffreyhung/how-to-fill-opt-starting-date-this-year-data-approach-83b3198b2419?source=rss-cf2c9ab57bf7------2</link>
            <guid isPermaLink="false">https://medium.com/p/83b3198b2419</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[uncertainty]]></category>
            <category><![CDATA[opt]]></category>
            <category><![CDATA[covid-19-crisis]]></category>
            <category><![CDATA[international-student]]></category>
            <dc:creator><![CDATA[Geoffrey Hung]]></dc:creator>
            <pubDate>Fri, 03 Apr 2020 16:49:42 GMT</pubDate>
            <atom:updated>2020-04-03T17:52:12.351Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to fill OPT starting date this year — A Data-Driven Approach</h3><p><a href="https://www.linkedin.com/in/geoffreyhung/">Geoffrey Hung</a> and <a href="https://www.linkedin.com/in/drdata/">Nicholas Ross</a></p><h3>TL;DR</h3><p>Without any insider information, we expect that OPT extension processing will (probably) be once again backed up. Given this, we built some simple heuristics around what could happen to process time. Once again, we have <em>zero</em> knowledge, this is all guesses based on data and what we have seen before.</p><p><strong>Suggestions for international students:</strong></p><ol><li>Submit your I-765 application ASAP.</li><li>Fill your application carefully, don’t be one of the “Over 180 days”</li><li>No matter how you analyze this, prepare to submit NOW. See #1.</li><li>You may want to consider putting a start date out a bit from graduation in order to save OPT, but not that much. If anything it’s <em>probably</em> better to sacrifice a bit of OPT at the start in order to avoid potentially losing a job.</li><li>Don’t panic (it won’t help).</li></ol><h3>Overview</h3><p>International students on F-1 Visas need to apply for <strong>O</strong>ptional <strong>P</strong>ractical <strong>T</strong>raining (“OPT”). OPT allows students to work in a field directly related to their major area of study for one year (Masters and PhD STEM degrees are also eligible for an additional two-year extension). A student has a 5-month window to submit an OPT application starting from 90 days before degree completion to 60 days after degree completion. Based on<a href="https://egov.uscis.gov/processing-times/historic-pt"> last year’s data</a> [FY 2019, form I-765], the processing time was around <strong>4.5 months</strong>. Students who file then receive an Employment Authorization Document (“EAD”) that allows them to work in the United States for a set period. They file with the United States Citizenship and Immigration Service (“USCIS”).</p><p>Students also need to select a start date on their application. If a student graduates in May and selects a start date in August, then companies may not be interested in hiring them due to having to wait. If a student selects an early start date but then doesn’t have an offer, they lose some of their OPT time!</p><p>Up until 2019, processing times were nearly in line with the 90-day pre-graduation application window: a student could (conceivably) put in their application 90 days before graduation and be <em>nearly</em> assured that the application would be processed and they would be able to start right after graduation. This is assuming they filled their application out correctly and had no outstanding other issues.</p><p>For 2019 graduates, this relationship devolved as the average processing time went to 4.5 months (about 135 days). What this meant was that if a student graduated on 5/14/2019 and filled out their application on 2/14/2019 (90 days beforehand) and sent it in, then (assuming you were average) the earliest you could begin working was 6/28/2019, a full month and a half after graduation! In 2019 many international students had offers rescinded and a number of companies avoided making offers once hearing that a student was an international student due to the inability to have a firm start date.</p><p>This year, students are even in further jam. Not only is there an unknown processing time, but it is possible that things are lagging even more this year. For example, if we look at the current processing time for the Potomac servicing center we see that they anticipate an estimated time range of 1.5 to 5 months for F-1 students.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kZsxe2tfemlkjrlP" /></figure><p>However, looking at the above it says that the “Receipt date” for a case inquiry is 11/2/2019. Meaning that an applicant won’t be provided answers about their case unless it was filed before November (meaning that they are trailing nearly five months)!</p><p>The scariest part, for today’s international student is the unknown around this. It’s hard to know when to start looking for a job when, even if they give you an offer, you don’t know when you can start! If the current dates stand, you won’t even be able to ask for a case update till nearly 2 months after graduation.</p><h4>Even worse</h4><p>At the end of last calendar year (USCIS’s 2020Q1 FY) USCIS reported that they had cut into their backlog, and their average processing time had gone from 4.5 to 4.2 months. However, the calendar year 2020 is off to a rough start, with USCIS closing its office (at least March 18 — April 7) to slow the spread of Coronavirus. While we have zero inside information or knowledge, it’s conceivable (probable?) that this is slowing down application processing. We wanted to look and see if we could figure out how much this changes things.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*M2ax31ffK9HVY1nh.jpg" /><figcaption>2020: New Normal</figcaption></figure><p>We started by studying USCIS Form Status Statistics in<a href="https://www.uscis.gov/sites/default/files/USCIS/Resources/Reports%20and%20Studies/Immigration%20Forms%20Data/All%20Form%20Types/Quarterly_All_Forms_FY17Q4.pdf"> 2017</a>,<a href="https://www.uscis.gov/sites/default/files/USCIS/Resources/Reports%20and%20Studies/Immigration%20Forms%20Data/All%20Form%20Types/Quarterly_All_Forms_FY18Q4.pdf"> 2018</a> and<a href="https://www.uscis.gov/sites/default/files/USCIS/Resources/Reports%20and%20Studies/Immigration%20Forms%20Data/All%20Form%20Types/Quarterly_All_Forms_FY19Q4.pdf"> 2019</a> Fiscal Year and the relationships between pending and processed applications. Data from USCIS are cleaned and loaded to DataStudio [If you want to learn more about DataStudio, you may read Geoffrey’s article — <a href="https://medium.com/usf-msds/data-visualization-lazy-approach-1203416e3459">Data Visualization — Lazy Approach</a>].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7QH_CnAgnalatTpr" /></figure><p><a href="https://datastudio.google.com/reporting/c4602add-0aee-4691-af35-4eb72e7b5257"><strong>Find the data studio dashboard here</strong></a></p><p>A few important takeaways from this:</p><ol><li>Based on the most recent data there is a roughly five-month backlog in “pending” applications — which aligns with what was being reported on their status page.</li><li>The number processed moves around from a high of around 650K to a low of a bit shy of 300K. This means that there is at least some ability to allocate resources in the case of applications falling behind significantly.</li><li>Specifically, if a quarter has more applications, the processing capacity in the next quarter seems to increase to keep pending down to some threshold.</li><li>The form (I-765) used by international students isn’t tied to the graduation cycle, probably implying that there are quite a few non-students who also apply using this same form.</li><li>This may mean that the total number of applications may go down as people are less likely to immigrate during this time. That being said, we don’t know the breakdown of who fills out an i-765 (<a href="https://www.uscis.gov/sites/default/files/files/form/i-765instr.pdf">Who may file form i-765</a>), so it is tough to estimate the actual effect.</li><li>In the most recent quarter, it looks like they worked to pull down the pending numbers and it seems that a good rule of thumb is that they have the ability to process exactly 1 day of applications (meaning that if the office is closed for 1 day, they are backed up for one day).</li></ol><h3>Different Scenarios</h3><p>Given the above, we have undertaken some scenario analysis to try to understand what an international student should do. We will make some other assumptions: First, we assume that this is a FIFO queue and that processing an application occurs “instantly” once it is at the head of the queue. We are also assuming that the current queue is 4.5 months (18 weeks) long. Note that the most recent data said that the queue was 4.2 months, but last year it was 4.5 over the entire year (<a href="https://egov.uscis.gov/processing-times/historic-pt">Historical Processing times</a>). We are also assuming that all months are four weeks long exactly and hand-waving around any issues about dates ending on a weekend.</p><p><strong>Scenario #0: </strong>USCIS, when it closes, is able to continue all “standard” operations as if nothing is changed. In this case, the student will receive their EAD 6 weeks (1.5 months) after graduation, assuming that they file on time in mid-February for mid-May graduation.</p><p>The student would receive their EAD card in the first week of July.</p><p><strong>Scenario #1:</strong> USCIS is basically shutdown (only doing certain requests which do not include I-765 for F1 students) until the end of April (3/18–4/30 — call it 6 weeks for rounding). They cannot scale back up, and an applicant sent their application in on time (mid-February) for a Mid-may graduation.</p><p>At the time that the student filed they would be in a queue 18 weeks long. From mid-February until mid-March (4 weeks), the queue processes as normal, but at which point it is suspended. When it restarts at the start of May, the student still has 14 weeks in the queue. They will have two weeks until their graduation, at which point they have 12 weeks (3 months) left until they receive their EAD document.</p><p>The student would receive their EAD 3 months after graduation (mid-August).</p><p><strong>Scenario #2: </strong>USCIS is not fully shut down, but only handling half their current expected volume until the end of <strong>April</strong>, at which point they ramp up to their pre-COVID processing levels, but not beyond that.</p><p>At the time that the student filed they would be in a queue 18 weeks long and from mid-February until mid-March (4 weeks), the queue processes as normal, as in Scenario #1. At this point, the student has 14 weeks left in their queue, but the next 6 weeks (mid-March until the end of April) the queue is only processed at half-speed, resulting in only 3 weeks taken off their personal queue (11 weeks left). At the start of May, they have 11 weeks left in the queue.</p><p>The student would receive their EAD card 9 weeks after graduation, in the third week of July.</p><p><strong>Scenario #3: </strong>USCIS is not fully shut down, but only handling half their current expected volume until the end of <strong>May</strong>, at which point they ramp up to their pre-COVID processing levels, but not beyond that.</p><p>At the time that the student filed they would be in a queue 18 weeks long and from mid-February until mid-March (4 weeks), the queue processes as normal, as in Scenario #1 and #2. At this point, the student has 14 weeks left in their queue, but the next 10 weeks (mid-March until the end of May) the queue is only processed at half-speed, resulting in only 5 weeks taken off their personal queue (9 weeks left). At the <strong>end</strong> of May, they have 9 weeks left in the queue. Note that our calculation has already gone past graduation and the student still has 9 weeks in the queue).</p><p>The student would receive their EAD card in the second week of August.</p><p>Note that we went over the basic logic above. We are, once again, totally uninformed and making things up. You can feel free to play with the assumptions and logic.</p><h3>What are some other things to consider:</h3><p><strong>Filing:</strong> If you sent your application in after the closure (and USCIS is doing some sort of slow down), you will end up getting triple penalized: first because the queue is longer than the expected 90 days <em>and</em> because the queue will process more slowly <em>and</em> because they seem to work in a FIFO manner.</p><p><strong>The Lottery:</strong> If your end goal is to stay in the United States then it’s important to understand your timeline. For individuals looking to be part of the H1B lottery, note that this lottery happens in Feb/March/April and thus the goal is to be employed during this time at a company that will sponsor you.</p><p>In other words, a month on either end isn’t going to help your chances of staying in the United States unless that month is in the aforementioned H1B lottery period. There is no difference between your OPT expiring in August or September (all else being equal) because it won’t change the number of draws that you have in the lottery.</p><p>This means that your loss function is probably asymmetric: Gaining a month at the end of the period probably won’t matter (unless that month is April), but losing a job opportunity at the start because you set the day too late will. Why? Jobs are hard to get and the cost of losing an opportunity is really high.</p><p><strong>Mailing</strong>: Mailing service (so far) in the United States has been only minimally changed by the virus. If this becomes disrupted, applications may take longer from sending to arrival. Given this it is probably a good idea to spend some money to send your package in via FedEx, UPS or another courier who has both a tracking label and can guarantee a delivery date.</p><p><strong>Fill it out correctly: </strong>Filling out the form incorrectly / having an issue. In <a href="https://www.aila.org/File/DownloadEmbeddedFile/75234">another report</a>, the number of days required to file was reported. You can see in the table below the breakdown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*Egd7BdNsUrysLcJBW5iwhQ.png" /></figure><p>While we have no inside information, it’s our guess that at least some of the long tail is due to misfiling parts of the application. Make sure that your application is complete and accurate in order to avoid being placed in a slow bin. Once again, because this form handles not just OPT, but other forms of immigration, it’s tough to know how much of an effect this has on OPT.</p><h3>Summary</h3><p>Deciding when and how to fill out an OPT form is terrifying for international students. As people who are interested in this process, we wanted to take a look at the data. A few key points: (1) we are not experts, (2) we have no inside information and (3) we don’t know anyone who works at USCIS. Take everything we said with a few tablespoons of salt.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83b3198b2419" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>