<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by david wong on Medium]]></title>
        <description><![CDATA[Stories by david wong on Medium]]></description>
        <link>https://medium.com/@davidwong102?source=rss-d63c9b8b0879------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*mgy68ynKQPqszium.jpg</url>
            <title>Stories by david wong on Medium</title>
            <link>https://medium.com/@davidwong102?source=rss-d63c9b8b0879------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:06:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@davidwong102/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Truth About Data Companies: They’re Actually Research Companies]]></title>
            <link>https://medium.com/@davidwong102/the-truth-about-data-companies-theyre-actually-research-companies-739f35466cde?source=rss-d63c9b8b0879------2</link>
            <guid isPermaLink="false">https://medium.com/p/739f35466cde</guid>
            <dc:creator><![CDATA[david wong]]></dc:creator>
            <pubDate>Wed, 21 Oct 2026 02:08:31 GMT</pubDate>
            <atom:updated>2026-10-21T02:08:31.439Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The most expensive line item in AI training isn’t compute. It’s knowing what data to make.</em></p><p>!<a href="https://images/media-generation-cover-research-whiteboard-0-cdef11a3-fdef-4bd9-ab62-f27f96d5c781.webp">Cover: researchers discussing data distribution charts on a whiteboard in a modern office</a></p><p>Two companies both sell “AI training data.” One charges two dollars an hour for labeled rows. The other charges two hundred — and buyers line up, sign annual contracts, and renew.</p><p>Same industry. Same product category on paper. A hundred-to-one price gap.</p><p>The gap isn’t labor quality. It’s that the second company isn’t selling labor at all. It’s selling research.</p><h3>The factory and the lab</h3><p>Walk into a data factory and you see what you’d expect: rows of annotators, dashboards counting rows per hour, a price list by task type. The pitch is throughput. The moat is headcount and tooling. The product is interchangeable — if one factory won’t do it for $1.80 a row, another will.</p><p>Walk into a data lab and the room looks different. Fewer screens, more whiteboards. The whiteboards aren’t covered with schedules. They’re covered with distributions: where the model fails, which slices of the task are underrepresented, what the last eval run said. The pitch isn’t throughput. It’s a study design.</p><p>!<a href="https://images/media-generation-factory-vs-lab-0-d849e58f-5597-4169-baae-d287edc64d4b.webp">Split scene: a crowded annotation factory floor versus a small research team around charts and whiteboards</a></p><p>Here’s the uncomfortable truth the price list already knows: the labeling was never the hard part. The hard part is knowing what to label.</p><p>Anyone can hire a thousand annotators. Almost no one can tell you, before the money is spent, exactly which ten thousand examples will move your evals — and why. That “exactly which, and why” is a research question. It takes hypotheses, experiments, and the judgment to read an eval breakdown and say: the model doesn’t need more data, it needs different data, from this slice, in this proportion, with these edge cases deliberately overrepresented.</p><p>That’s not operations. That’s science.</p><h3>Three ways to tell them apart</h3><p><strong>1. Listen to the pitch.</strong> The factory talks about headcount, throughput, QA pass rates, price per row. The lab talks about your failure modes. The first meeting with a real data lab feels like a research review: they bring an analysis of where your model breaks, a hypothesis about what’s missing in your training mix, and a proposed experiment — before you’ve paid them anything. If the pitch could be delivered by an ops manager, it’s a factory. If it takes a research scientist to deliver it, it’s a lab.</p><p>!<a href="https://images/media-generation-eval-analysis-lab-0-2118f555-2833-4a23-93fe-39e52ad74706.webp">A researcher studying model evaluation charts and error heatmaps on large monitors</a></p><p><strong>2. Look at the org chart.</strong> Factories are pyramids: a thin layer of project managers over a wide base of annotators. Labs are inverted: researchers and domain experts on top, with annotation as the execution arm underneath. Ask who designed your dataset and you’ll get the answer immediately. “Our ops team” means factory. A name, a paper trail, a person who can defend the distribution choices against your evals — that’s a lab.</p><p><strong>3. Watch what they refuse.</strong> A factory never says no. Every task is a row count times a unit price. A lab says no constantly: no, more data won’t fix that — your eval is contaminated; no, we won’t label that slice — it’s already saturated; no, don’t buy the package — buy the experiment first. Refusal is the signature of judgment. You can’t fake it at scale, which is exactly why it’s the most reliable signal.</p><h3>What this changes for anyone buying data</h3><p><strong>1. Stop buying rows. Buy the study.</strong> The unit of value isn’t the labeled example — it’s the research behind which examples got labeled. When you evaluate a vendor, don’t ask for the price list first. Ask for their last three dataset designs: the hypothesis, the distribution, what the evals said afterward. A factory will show you throughput charts. A lab will show you before-and-after eval deltas. Buy from whoever shows you the deltas.</p><p>!<a href="https://images/media-generation-before-after-evals-0-45e84539-002f-437b-8625-59119717a43c.webp">A researcher’s hand pointing at a chart showing before-and-after evaluation improvements</a></p><p><strong>2. The interview question changes.</strong> “Can you label X?” is the wrong question — everyone says yes. The right question is: “Show me a time you told a client not to buy data.” Then: “Show me a time your dataset design was wrong, and how you knew.” You’re not hiring hands. You’re hiring the judgment to spend your data budget where it moves the needle.</p><p><strong>3. The market is already repricing.</strong> Look at where the money goes: the vendors being acquired, the ones raising at research-lab multiples, the ones serious labs sign multi-year deals with — they’re the ones with the whiteboards, not the ones with the biggest annotator headcount. Compute got commoditized. Labeling is getting commoditized. The scarce thing left is knowing what the model needs to learn next. That knowledge lives in research teams, and the price list is catching up.</p><h3>The bottom line</h3><p>Every serious AI lab has quietly learned the same lesson: data stopped being a supply-chain problem and became a research problem. The companies that figured that out early stopped selling annotation and started selling answers to the question “what should the model learn next.”</p><p>The factories will keep competing on price per row, and there’s nothing wrong with that — someone has to do the labeling. But the hundred-to-one gap isn’t a pricing inefficiency. It’s the market telling you, in the bluntest possible language, where the value moved.</p><p>It moved from hands to heads.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=739f35466cde" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Write One Answer, Earn Once. Write the Rubric, Earn Ten Times.]]></title>
            <link>https://medium.com/@davidwong102/write-one-answer-earn-once-write-the-rubric-earn-ten-times-a1ac44328819?source=rss-d63c9b8b0879------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1ac44328819</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[training-data]]></category>
            <dc:creator><![CDATA[david wong]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 20:53:22 GMT</pubDate>
            <atom:updated>2026-10-06T20:53:22.073Z</atom:updated>
            <content:encoded><![CDATA[<h4>AI Training Data</h4><h4>The same expert, two products, a ten-to-one price gap. The AI data market just told us where the value moved.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ndE3IJv4941gXhkPhu8hUA.png" /></figure><p>There’s a new line item on the AI data price list, and it pays like nothing else in the market.</p><p>Take the same domain expert. Ask them to write a reference answer: they earn one answer’s fee. Ask them to write the scoring rubric for that same question — the dimensions, the point values, what separates excellent from acceptable: they earn ten times as much.</p><p>This isn’t a joke about consultants overcharging. It’s the market telling you, in the bluntest possible language, where the value moved.</p><p>The product is called rubric data. In one sentence: it translates the squishy human judgment of “this is good” into dimensions and scores — an objective evaluation built out of subjective taste. We used to buy answers. Now we buy the criteria for judging them.</p><h3>The three acts</h3><p>Rubrics didn’t arrive all at once. They climbed a ladder, and each rung changed what the data was for.</p><p><strong>Act one was evaluation.</strong> Subjective questions have no answer key — a medical consultation, a legal memo, an essay. The only honest way to grade them is to hire experts to write down, in fine detail, what “good” looks like. In this era the rubric lived in the eval report: a measuring instrument, nothing more.</p><p><strong>Act two moved it into training.</strong> The rubric stopped being a document and became a reward signal — <em>rubric reward</em>, in the industry’s jargon. The model is no longer rewarded for resembling the reference answer. It’s rewarded for hitting each scoring point, one by one. That sounds like a subtle change. It isn’t. “Be like the answer” teaches mimicry; “hit these points” teaches the actual skill the points describe. The rubric went from grading the test to teaching the student.</p><p><strong>Act three is happening now.</strong> The industry is training models to write rubrics themselves, with human experts stepping back into the role of auditors. And that shift has already spawned the next product category: paired data for teaching a model to write a scoring standard. The judge is being automated; auditing the judge is the new scarce labor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ueo-_b3KgcpDhDblZvtaNQ.png" /></figure><h3>Three things this changes for anyone buying data</h3><p><strong>1. What you buy — and who can sell it.</strong> The expert used to write answers; now they write criteria. The bar is unambiguously higher: real judgment, deep domain knowledge, and the rare ability to decompose taste into atomic dimensions with defensible weights. Those people are more expensive and much scarcer. You’re not hiring hands anymore. You’re hiring judgment.</p><p><strong>2. How you accept delivery.</strong> We used to check right versus wrong. With rubric data, you check <em>consistency</em>: two experts, same rubric, same model response — do their scores agree? Inter-rater agreement isn’t a nice-to-have metric anymore. It’s showing up as a clause in the contract.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DY35FAfoqYOOphLXbYESbA.png" /></figure><p><strong>3. How long the data lives.</strong> A rubric gets gamed. Once a model learns to optimize against the test — Goodhart’s law with gradient descent — the rubric is dead and has to be rewritten. Rubric data isn’t inventory; it’s a consumable. Which is why the next category is already emerging: dynamic, evolving rubrics that rotate in real time, precisely because static ones get burned through.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u42ASpCoa7afkP_Iodpz_g.png" /></figure><h3>The bottom line</h3><p>The data market’s center of gravity is moving from content to criteria. The people who were valuable because they could write good answers are being replaced by the models themselves. The people who can write good rubrics for subjective domains — accurately, decomposing judgment into dimensions — can’t be replaced yet.</p><p>They’re the most top-tier expert talent in the market right now. And the price list knows it: ten to one.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1ac44328819" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Used to Buy Answers. Now We Buy the People Who Judge Them.]]></title>
            <link>https://medium.com/@davidwong102/we-used-to-buy-answers-now-we-buy-the-people-who-judge-them-01168f39294a?source=rss-d63c9b8b0879------2</link>
            <guid isPermaLink="false">https://medium.com/p/01168f39294a</guid>
            <dc:creator><![CDATA[david wong]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 21:40:43 GMT</pubDate>
            <atom:updated>2026-10-05T21:40:43.148Z</atom:updated>
            <content:encoded><![CDATA[<p>Scoring rubrics — not answers — have become the most expensive commodity in the LLM data market. Here’s how it happened, and what it changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZjHBUt8a3bvnal4LI1OaA.jpeg" /><figcaption>The new protagonist of the data market: the scoring rubric.</figcaption></figure><p>A domain expert writes one model answer: they get paid for one answer. The same expert writes one scoring rubric: they get paid ten times that. Same person, same hour, wildly different price — because the second artifact isn’t content. It’s a standard.</p><p>The industry has a name for it now: <strong>Rubric Data</strong>.</p><h3>The one-line definition</h3><p>A rubric takes the fuzzy human notion of “good” and turns it into dimensions — each with its own scale and weight. Clinical accuracy: 0–5. Bedside manner: 0–3. Safety: pass/fail. Add it up, and subjective judgment becomes something an engineer can build on, a reward function can optimize against, and two strangers can agree on.</p><p>That last property is the whole game. “Good” is an opinion. A rubric is an opinion with an API.</p><blockquote>We used to buy answers. Now we buy the people who judge the answers.</blockquote><p>From evaluation harness to training loop to self-generation.</p><h3>Three stages of evolution</h3><p><strong>1. Evaluation.</strong> Rubrics were born in eval harnesses. For subjective domains — medicine, law, code review — there is no single correct answer, so the only way to grade a model was to have experts write down, in advance, exactly what “good” looks like, dimension by dimension. Expensive, slow, and confined to the test set.</p><p><strong>2. Training.</strong> Then rubrics escaped the eval harness and walked into the training loop. Instead of rewarding a model for resembling a reference answer, teams began rewarding it for hitting the rubric’s scoring points — “Rubric Reward.” The optimization target stopped being <em>look like this answer</em> and became <em>satisfy these criteria</em>. For open-ended tasks, that’s a fundamentally better gradient.</p><p><strong>3. Self-generation.</strong> And now the industry is training models to write the rubrics themselves, with human experts stepping back into an auditing role. A new data category has appeared almost overnight: paired data of models drafting scoring standards and experts correcting them. The rubric — once the most human artifact in the pipeline — is becoming model-generated too.</p><p>One rubric outweighs a stack of answers — literally, on the price sheet.</p><h3>Three things this changes for anyone buying data</h3><p><strong>1. What you buy.</strong> Experts used to be paid to write answers. Now they’re paid to write rubrics. The skill bar is dramatically higher: it demands judgment, deep domain knowledge, and the rare ability to decompose “quality” into atomic dimensions with defensible weights. People who can do this are expensive — and scarce.</p><p><strong>2. How you accept delivery.</strong> Acceptance for rubric data isn’t about right versus wrong. It’s about consistency: give the same rubric and the same model response to two independent experts, and their scores should land close together. Inter-rater agreement becomes the acceptance test. Data QA teams built for answer-checking have to rebuild their playbooks.</p><p><strong>3. How long the data lives.</strong> Here’s the uncomfortable part: rubrics can be hacked. Once a model learns to game a rubric’s scoring points — optimizing for the letter of the criteria while missing their spirit — the rubric is dead and has to be rewritten. Rubric data isn’t inventory you stockpile. It’s a consumable. Expect an entire new category to emerge: dynamic, evolving rubrics that rotate before models can overfit to them.</p><p>The irreplaceable expert: producing the definition of good.</p><h3>The bottom line</h3><p>The people who only wrote answers are being replaced by the models they were training. That’s already happening.</p><p>But the people who can write precise, game-resistant rubrics in subjective domains — medicine, law, engineering judgment, taste — are becoming the most valuable human talent in the AI supply chain. They don’t produce content. They produce the definition of good.</p><p>In a market that used to pay for answers, the new protagonist is the judge.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01168f39294a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>