<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Paul Short on Medium]]></title>
        <description><![CDATA[Stories by Paul Short on Medium]]></description>
        <link>https://medium.com/@paul.d.short?source=rss-49aad253ddef------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IG9z-mpRgPlRbpfmpd5mBw.jpeg</url>
            <title>Stories by Paul Short on Medium</title>
            <link>https://medium.com/@paul.d.short?source=rss-49aad253ddef------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:49:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@paul.d.short/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scoring IDP, Part 2: Approaches to Evaluating Form Extraction]]></title>
            <link>https://medium.com/@paul.d.short/scoring-idp-part-2-approaches-to-evaluating-form-extraction-b3c7eafb3379?source=rss-49aad253ddef------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3c7eafb3379</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[document-processing]]></category>
            <category><![CDATA[idp]]></category>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Paul Short]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 23:49:20 GMT</pubDate>
            <atom:updated>2026-10-18T14:26:56.786Z</atom:updated>
            <content:encoded><![CDATA[<p>(<strong>Previous: </strong><a href="https://medium.com/@paul.d.short/scoring-idp-part-1-confidence-is-not-correctness-71384dc15f56">Scoring IDP, Part 1: Confidence Is Not Correctness</a>)</p><h3>Measuring Key Information Extraction (KIE)</h3><p>OCR correctness is traditionally measured with metrics such as Character Error Rate (CER) and Word Error Rate (WER): compare the extracted text to the text on the page and count the mistakes. These metrics work well for text detection, and for plain documents they are enough.</p><p>Forms are different. A form is not a page of text; it is a set of labeled fields, and a value only has meaning when it is attached to the correct label. OCR does not make that attachment. OCR returns text strings with coordinates.</p><p>For example, Amazon Textract’s <em>detect_document_text</em> might return a form label as {&quot;Text&quot;: &quot;First Name:&quot;, &quot;Confidence&quot;: 98.1, &quot;Geometry&quot;: {&quot;BoundingBox&quot;: {&quot;Left&quot;: 0.20, &quot;Top&quot;: 0.18, &quot;Width&quot;: 0.10, &quot;Height&quot;: 0.02}}} and the handwritten value next to it as {&quot;Text&quot;: &quot;John&quot;, &quot;Confidence&quot;: 99.2, &quot;Geometry&quot;: {&quot;BoundingBox&quot;: {&quot;Left&quot;: 0.42, &quot;Top&quot;: 0.18, &quot;Width&quot;: 0.07, &quot;Height&quot;: 0.02}}}. The output tells you what the strings say and where they sit on the page, but crucially, the label and the value are not mapped to one another. The raw OCR engine does not know that &quot;John&quot; belongs to the field &quot;First Name.&quot;</p><p>Matching each labeled field to its correct value is a Key Information Extraction (KIE) job. For every labeled field the system must return the correct value written in that field.</p><p>To accomplish this, we must move beyond simple OCR and use form-aware Intelligent Document Processing (IDP) approaches. To extend our Textract example, this could mean calling <em>analyze_document(FeatureTypes=[&quot;FORMS&quot;])</em> and then applying another abstraction layer to map those extracted key-value pairs to an expected schema. Instead of just returning independent text strings, this KIE approach explicitly maps the relationships between the data. The system recognizes &quot;First Name&quot; as a label and structurally links it directly to the corresponding value.</p><p>Textract is just one example: the actual extraction technology could be anything. Exactly <strong>how to build </strong>these extraction pipelines and which OCR/IDP service or model to use is out of scope, not the point here. Instead, our focus is on <strong>how to evaluate</strong> an IDP system — specifically, <strong>how to score</strong> the extraction of data from forms.</p><p>In the example above, the key information is First Name, and the correct (expected) value is John. We can go one step further: the entity is a person, and First Name is one attribute of that entity. Forms typically contain a hierarchy of one or more related entities.</p><p>A system can read every string on a page correctly and still map values to the wrong labels, so KIE needs its own scoring.</p><h3>Defining a Match</h3><p>Before we can grade whether an extracted field is a match, we have to standardize how we determine a match between the extracted value and the ground truth.</p><ul><li><strong>Field-specific comparators.</strong> For example, <strong>dates</strong> that mean the same day but are formatted differently, <strong>currency</strong> values with and without symbols, or an <strong>address</strong> with a comma the annotator omitted. All of these are correct extractions that a strict string comparison will score as misreads. AWS recommends field-specific comparators for this reason: tolerate formatting variation on numerics, apply fuzzy matching to text for spacing and punctuation, and normalize dates and addresses before comparing.</li><li><strong>Multiple legitimate values.</strong> Decide how a field is scored when more than one answer is acceptable. This could be a list of acceptable values (or list of regex patterns), rule-based decisions, or rules where you need NLP or even an LLM to weigh in. (LLM-as-a-judge has downsides. If an LLM is the judge, keep it independent of the extractor. If an LLM decides matches, audit its match decisions against a human sample the same way you audit annotators.)</li><li><strong>Checkboxes.</strong> Reading a checkbox is an optical mark recognition (OMR) task, not OCR. A checkbox can only be true or false, while a text field can hold almost anything. To keep scoring consistent across form types, it is simpler to score checkbox as binary (two-state) true/false rather than true, false, N/A (skip/ignore). We’ll get more into that later.</li><li><strong>Annotator disagreement.</strong> Ground truth is written by people, and people disagree, especially on handwriting and on fields where the intent is unclear. (The 4-that-looks-like-a-9 example we covered in Part 1 is a ground-truth error in a dataset that has been picked over for thirty years.) When too much of this happens, your scores will fluctuate even when the software does not change. This may confuse stakeholders and even erode their confidence in your testing approach. To avoid this, have two annotators label a sample independently and measure how often they agree, per field type. (A third SME annotator can be the tie-breaker.)</li></ul><blockquote>Report the agreement as <strong>Cohen’s kappa</strong> rather than a raw percentage: blanks inflate raw agreement the same way they inflate Accuracy, and kappa corrects for that chance agreement. (Kappa drops on heavily skewed fields even when annotators mostly agree, so read it per field type alongside the actual disagreements.) You could even publish it next to the metrics to show which fields the humans themselves can’t agree on. Agreement is also the ceiling: a system scoring above the annotators’ agreement on a field is measuring annotator noise. This approach is known as “inter-annotator agreement,” a standard in NLP annotation. See Carletta, 1996.</blockquote><ul><li><strong>How ground truth records a blank</strong>. Ground truth should record blanks one consistent way (null vs empty or other conventions). Blanks also have the potential to inflate scoring or hide extraction problems, so let’s get into some details next.</li></ul><h3>Scoring Blanks: <strong>Correct ÷ Total</strong> Isn’t Enough</h3><blockquote>Because of blank, skipped, optional, or conditional fields, don’t focus on <strong>Accuracy</strong> (correct ÷ total); use <strong>Recall aka Sensitivity </strong>(correct ÷ populated) or <strong>F1</strong> (harmonic mean of Recall and <strong>Precision</strong>), and check the <strong>False Positive Rate</strong> (invented ÷ blank). I’ll explain why.</blockquote><p>At first glance, the intuitive way to score a form is simple pass/fail: when the system extracts the expected value from a field, count it True; when the extracted value does not match, count it False. Divide the number of correct fields by the total number of fields and you have a score. Let’s call it “intuitive accuracy.” (We’ll get into formal metrics later.) This<strong> intuitive accuracy (correct ÷ total) works when every field is filled in and every field matters. Most forms are not like that. Blank fields will inflate (overstate) accuracy. </strong>Some fields are critical (mandatory, must be filled), but many may be left blank because they are optional or conditional on another answer (required in one context, optional or not applicable in another). Some forms may be sparsely populated, with whole sections left empty.</p><p>Let’s consider these options for handling blank or optional fields:</p><ul><li><strong>Always count them as False (always punish)? </strong>This works for tests that require an answer to everything, but not for forms where a blank means “not applicable.” A system that correctly reports a blank gets marked wrong, and the score fills with failures that are not failures. Scores skew too low. This is not a good option.</li><li><strong>Count a correctly identified blank as True (always reward)? </strong>This inflates the score on sparse forms, because recognizing a blank is a far easier task than reading a value, and on some forms the blanks outnumber the values four to one. The inflated score hides real defects, an unpleasant surprise waiting to happen if production forms are less sparse than the test set, or if the optional fields matter in certain scenarios. Scores skew too high. This is not a good option.</li><li><strong>Count blanks as True or False based on whether the field was required or conditionally required?</strong> Conditional fields create form subtypes (variants based on which conditions apply), each needing its own scoring rules. The number of rule combinations grows quickly. This might work, but it is hard to track, explain, and present.</li><li><strong>Skip the blanks?</strong> For raw text transcription measured with CER or WER, this seems natural, since blanks contribute no characters or words; however, leaving blank fields out of the extraction score comes with a downside: we would not see when an unexpected value appears in a field that should be blank. This may happen with field mismaps, but it can also happen when an LLM puts a value in that field when it isn’t supposed to. LLMs may just hallucinate or forget their instructions and infer values.</li></ul><p>Considering the pros and cons of each approach, the last option, “skip the blanks,” is the best, with one adjustment to address unexpected values in fields that should be blank (mismaps or hallucinations): blanks stay out of the extraction score, but every blank is still counted — a correct blank as a True Negative, an invented value as a False Positive. Later we will discuss metrics that score these, such as <em>Recall</em> to grade populated fields and <em>Specificity</em> to grade blanks. All of them are computed from the confusion matrix.</p><h3>The Confusion Matrix</h3><p>A confusion matrix is a table used to evaluate the performance of a classification model. It compares the actual true values against the model’s predictions and breaks the results into how many predictions were correct and what kinds of errors were made. Confusion matrices are a standard tool for data scientists and AI engineers, but they predate machine learning: the same table has been used in statistics and medical testing for decades, and it is a natural extension of the “expected result versus actual result” scoring familiar from traditional Software Quality Assurance (SQA).</p><p>In our case, the “positive class” is a field that contains a value, and the “negative class” is a blank field: the matrix compares what was actually on the form against what the system reported. Every text field is graded as not just two states (true/false) but four (2 true, 2 false):</p><ul><li><strong>True Positive (TP): </strong>Correct. Value on the form, and the system reads it correctly. (Ground truth holds a value and the extracted value satisfies the field’s comparator.)</li><li><strong>True Negative (TN)</strong>: Correct. Field blank, and the system reported blank. (Ground truth is blank and the output is blank.)</li><li><strong>False Negative (FN): </strong>Incorrect<strong>. </strong>Value on the form, but the system got it wrong (Ground truth holds a value; the output is wrong, empty, or NULL.) This is a <em>misread</em>.</li><li><strong>False Positive (FP): </strong>Incorrect<strong>. </strong>Field blank, but the system invented text or checked a checkbox anyway. (Ground truth is blank; the output is not.) This could be a mis-map or a hallucination.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*28qspSyok1aCjhPz_7QFJQ.png" /><figcaption>The confusion matrix applied to text fields: green cells are correct (true positive, true negative); red cells (false negative, false positive) show how a field was missed or invented.</figcaption></figure><blockquote>This next sections lean on standard machine-learning classification metrics. The Metrics section covers them in greater detail; here are the basic definitions to know for now:</blockquote><blockquote><strong>Recall: TP ÷ (TP + FN).</strong> How much of what was there did the system recall? Of the values on the form, how many came back right? Recall punishes misses.</blockquote><blockquote><strong>Precision: TP ÷ (TP + FP).</strong> How precise was the system about returning only what belonged? Of the values the system returned, how many were right? Precision punishes wrong values. (Whether a <strong>misread</strong> counts as a wrong value here is a convention; we cover that next in, “Handling Text Misreads.”)</blockquote><blockquote><strong>F1 Score: 2TP ÷ (2TP + FP + FN).</strong> The harmonic mean of the two. Punishes both.</blockquote><blockquote><strong>Accuracy: (TP + TN) ÷ (TP + TN + FP + FN).</strong> <em>AVOID.</em> Of all fields, how many did the system get right? The intuitive “<strong>correct ÷ total</strong>” approach is the wrong choice when blank, skipped, optional, or conditional fields are involved: correct blanks pad the numerator and inflate the score, which is why we focus on Precision, Recall, and F1 instead.</blockquote><blockquote>If these are new to you, don’t worry. I’ll over-explain them with examples as we apply each one to IDP scoring.</blockquote><h3>Handling Text Misreads</h3><p>Testing compares what the ground truth says should be in a field (expected) against what the system extracted (actual). We may expect a value or a blank, but the actual result may be wrong:</p><ul><li>Expected: a value; actual: nothing extracted (null, empty, blank, undefined, etc) → <strong>False Negative.</strong> <strong>Recall</strong> matters more when this is the costly error.</li><li>Expected: blank; actual: a non-blank value extracted → <strong>False Positive.</strong> <strong>Precision</strong> matters more when this is the costly error.</li><li>Expected: a value; actual: a different value extracted → <strong>Misread.</strong> The system missed the value that was there and produced one that was not, so it can be counted three ways: as a False Negative (a miss), a False Positive (a wrong value), or both.</li></ul><p>Let’s consider what happens when we treat misreads as False Negative (FN), False Positive (FP), or both False Negative (FN) and False Positive (FP):</p><ul><li><strong>Misread counts as <em>FN only </em>(described in this paper).</strong> Our convention grades a <em>misread</em> (a real value read wrongly, returned empty, or returned NULL) as a False Negative (FN): the system failed to deliver the value that was there. We reserve False Positive (FP) for hallucinations. Every field lands in exactly one cell of the confusion matrix: TP+FN equals the populated fields, TN+FP equals the blanks, and the number a business audience wants, correct ÷ populated, is exactly Recall. <strong><em>Under this convention Recall punishes misreads and Precision does not!</em></strong></li><li><strong>Misread counts as <em>FP only </em>(AWS GenAI IDP accelerator and the AWS ML blog’s KIE evaluation).</strong> A misread is treated as a wrong value produced, an FP, without also being counted as a miss. The accelerator, through its Stickler library, further splits FP into a <em>false alarm</em> (a value where none was expected, a hallucination) and a <em>false discovery</em> (a wrong value where one was expected, a misread). It reports each rate separately, but both are False Positives, so a misread lands in Precision’s denominator and FN is reserved for values the system missed entirely. <strong><em>Under this convention Precision punishes misreads and Recall ignores them, the reverse of the one above!</em></strong></li><li><strong>Misread counts as <em>both FP and FN</em> (set-based entity-level F1).</strong> Most F1-scored KIE benchmarks (such as SROIE, FUNSD, and CORD) count a misread as both a False Positive and a False Negative. The wrong value is an FP because it matches no ground truth, and the unmatched ground truth is an FN. <em>The “Benchmark F1” convention therefore punishes each misread twice as hard as our convention. This is also the classic information-extraction convention: the MUC scorer of the 1990s put an “incorrect” slot in both the precision and recall denominators.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*m6gpW-Qf_HcEmB8-vfb-fA.png" /><figcaption>The same misread, counted three ways. Where it lands decides whether Recall, Precision, or both punish it.</figcaption></figure><p>For example, consider the same 2,500 misreads counted three ways: 10,000 populated fields, with 7,500 read correctly, 2,500 misread, and 0 blanks filled in. The Benchmark/MUC convention counts each misread as both a false positive and a false negative. It penalizes the error twice, because the system extracted a wrong value <em>and</em> missed the right one. This yields an F1 of 75%, which matches the intuitive answer: 7,500 of 10,000 fields, or 75%, were read correctly. The other two conventions count each misread only once, as either an FP or an FN, so both grade more leniently at 85.7%. That inflates the score by more than 10 points above the true share of fields read correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*GrMF0A91lcS_4I3xuzHW1Q@2x.png" /><figcaption>The same 2,500 misreads counted three ways. 10,000 populated fields: 7,500 read right, 2,500 misreads, and 0 blanks filled in. Notice the difference in final F1 Score between Benchmark and the other two conventions due to counting a misread as both FP and FN (punish twice).</figcaption></figure><p>It is an acceptable tradeoff. Counting a misread once, as an FN, makes Precision a hallucination-only check and lets F1 read higher than the share of fields read correctly. In exchange, Recall is simply correct ÷ populated: 7,500 ÷ 10,000 = 75% here. Both F1 and Recall are common metrics; the choice to use one over the other is about the audience. For a business audience, we headline Recall rather than F1 because a Recall percentage means what a non-technical listener assumes it means. When headlining with Recall, we add Specificity or False Positive Rate (invented ÷ blank) for hallucination control, since a Precision of 100% beside 2,500 misreads invites the wrong conclusion. When we show F1, we label where the misread lands, “misread = FN” or “misread = FP and FN,” so the reader knows which count produced the number.</p><blockquote>Benchmark/MUC lands on 75% here only because this example has no invented values. Add 500 and its F1 drops to 73.2%, our Precision drops to 93.8% (the contamination rate), and the share of fields read correctly is still 75%. None of the three F1s is a share-of-fields number; under our convention, Recall is. We’ll cover this again in the “Headline Score” section.</blockquote><p><strong>“Unreadable” as a distinct state. </strong>Corner case that applies to all three conventions: our scoring treats an empty return and a NULL return the same way, as an FN on a populated field. But they can mean different things: “I didn’t find a value” versus “there’s a value and I can’t read it.” If the extractor can distinguish the two, you could keep the FN tally the same but count the unreadable ones separately, the same way the checkbox section counts “unreported.” For example, you could report it as an “Unreadable Rate” (unreadable ÷ populated) next to Recall, so the reader can see how much of the Recall loss is the system giving up versus getting it wrong. If the annotators also marked the field illegible, flagging it as unreadable is the right answer, and you may decide not to charge it as an FN at all.</p><h3>Handling Checkboxes</h3><p>Reading a checkbox is a different task from reading text: it is an optical mark recognition (OMR) task, not OCR. It is a two-state problem: a checkbox can only be true or false, while a text field can hold almost anything. For text, a True Positive (TP) is a value read correctly and a True Negative (TN) is a blank left alone. A “checked” (marked) checkbox is the positive class, so</p><ul><li><strong>True Positive (TP)</strong>: correctly checked.</li><li><strong>True Negative (TN)</strong>: correctly unchecked. A NULL or missing state on a box that was unchecked also lands here, matching the text convention (blank returned as blank). Forms with lots of checkboxes tend to lean towards unchecked, so we’ll get into how that affects metrics. If you need to know how often the extractor fails to report a state at all, count “unreported” separately alongside the matrix rather than bending the cells.</li><li><strong>False Negative (FN)</strong>: missed checkmark, including a NULL or missing state on a box that was checked. This matches the text convention: a value that was on the form and came back empty or NULL is a False Negative.</li><li><strong>False Positive (FP)</strong>: invented checkmark.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*0as534fyf4r3lO1k0w7wRA.png" /><figcaption>The confusion matrix applied to checkboxes: green cells are correct (true positive, true negative); red cells (false negative, false positive) show how a checkmark was missed or invented.</figcaption></figure><p>When reporting the results, show both checkbox and text scores next to the overall number, including a separate checkbox F1 so missed and invented checkmarks are clear. Mutually exclusive groups (Yes/No, marital status, a list where only one option applies) are one categorical field, not a row of binary boxes. Scored as boxes, one swapped selection counts as two errors, a missed and an invented checkmark, and the unselected options pad TN.</p><p>Those unchecked boxes fill the tally with True Negatives (TN), which have potential to inflate any metric that counts them. That is another reason to keep checkbox and text scores apart. TN appears in none of the three positive-class metrics, <em>Recall</em>, <em>Precision</em>, and <em>F1; </em>however<em>, </em>it does show up in <em>Specificity, False Positive Rate, </em>and <em>Accuracy</em>.</p><p>For checkboxes, <em>Recall</em> (aka <em>Sensitivity</em>, TP ÷ (TP+FN)) is the metric that punishes missed checkmarks: every box that was checked on the form but reported as unchecked lands in FN and lowers the score. <em>Recall</em> does not punish the reverse, a box reported as checked that was blank, so pair it with <em>Specificity</em> or use <em>F1</em>.</p><p>We have been introducing metrics as each scoring rule needed them. Since every one is a different ratio of the same four confusion-matrix cells (TP, TN, FP, FN), the next section covers them together as a whole and shows the relationships and tradeoffs among them.</p><h3>Metrics (Reporting the Results)</h3><p>For a given set of fields you are scoring (a set of fields is typically a form, but you can also slice or aggregate in additional ways), tally every field into the four categories (TP, TN, FP, FN). The four tallies combine into named metrics. Each named metric refers to a specific formula, and each answers a different question. Every formula below uses our convention: a misread is a False Negative, an invented value a False Positive.</p><ul><li><strong>F1 score, 2TP/(2TP+FP+FN).</strong> <em>The harmonic mean of Precision and Recall (2 × Precision × Recall ÷ (Precision + Recall)).</em> If you can give only one confusion metric for form extraction, this is the best: a single balanced measure of clean output (Precision) and complete capture (Recall), completely avoiding the skew of correctly identified blank fields. One caution: F1 inherits the misread convention discussed earlier. Moving a misread from FP to FN leaves F1 unchanged, so our F1 matches the AWS accelerator’s for the same output, but the benchmark convention counts a misread in both cells and produces a lower F1.</li><li><strong>Recall, TP/(TP+FN) a.k.a. “accuracy without blanks.”</strong> <em>Of fields with values, how many were extracted correctly?</em> High <em>Recall</em> (aka <em>Sensitivity</em>) means the system rarely misses populated fields. Recall measures the system’s ability to deliver everything it was supposed to get, and by excluding blank fields it prevents class imbalance skew. The denominator represents all the data that physically existed on the form, both the values the system caught and the ones it missed entirely. Because hallucinations land in blank fields, this metric does not see them, so report <em>Specificity</em> alongside it. <em>Sensitivity</em> (not to be confused with <em>Specificity</em>) is the alternative term for Recall.</li><li><strong>Precision, TP/(TP+FP).</strong> <em>Of extractions made, how many were correct?</em> Put plainly: <em>When the system says it found something, how often is it right? </em>The answer depends on which cell of the confusion matrix a misread lands in, so here is Precision under each of the three conventions from the “Handling Text Misreads” section: <br><em>(1) Benchmark F1 (SROIE, FUNSD, CORD): misread counts as FP and FN.</em> The denominator is every time the engine claimed it found data — correct extractions, misreads, and hallucinations alike. This is the textbook reading: Precision measures how much you can trust the system when it does pull a value, and both Precision and Recall punish a misread.<br><em>(2) AWS GenAI IDP accelerator: misread counts as FP only.</em> Precision keeps its textbook meaning and punishes misreads; Recall does not see them.<br><em>(3) Alternative used in this paper: misread counts as FN only.</em> Treat misread as a False Negative (a value that was on the form and did not come back right), not a False Positive (a value invented where the form was blank), so that the denominator is correct extractions plus hallucinations. That narrows Precision to a hallucination check, “<em>How much invented data is mixed in with the correct data?</em>” High Precision means only that the system rarely invents data, and whether it read a value correctly is Recall’s job. Report FPR or Specificity as the hallucination rate, and treat Precision mainly as an input to F1.</li><li><strong>Specificity, TN/(TN+FP).</strong> <em>How often does the system leave a blank alone (not invent data)?</em> We can use this for “hallucination control” because it measures how effectively the system leaves intentionally blank fields alone instead of inventing data to fill them.</li><li><strong>False Positive Rate (FPR), FP/(FP+TN).</strong> <em>How often does the system invent data? </em>The rate at which it writes values into fields that should have remained blank; the complement of Specificity (1 − Specificity). FPR and Precision answer different questions about hallucination. FPR is the rate per blank: how often the system invents a value when given the chance. Precision, under the FN-only convention, is the contamination rate: how much of what the system delivered was invented. On sparse forms they diverge, since a tiny FPR across thousands of blanks can still mean a meaningful share of the delivered values is fabricated. Report both, and keep the raw invented count beside them.</li><li><strong>Accuracy, (TP+TN)/(TP+TN+FP+FN).</strong> <em>Of all fields, how many did the system get right? </em>Total correctness across all fields, including correctly identified blanks. Highly susceptible to inflation from class imbalance when fields are blank (null, empty, optional, conditional). Never present Accuracy alone; pair it with one or more metrics focused on the positive class, such as Recall, Precision, or F1 to demonstrate actual extraction success.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ekszCXZ39UyupRM-Dd92zw.png" /><figcaption>Blanks inflate Accuracy. Same extractor, 3 forms, each 75% correct once you ignore blanks. Jaccard, Recall, Precision, and F1 hold steady; Accuracy grades softer as blank or optional fields increase. To keep the example simple, the extractor never invents a value, so Precision is 100%.</figcaption></figure><ul><li><strong>Matthews Correlation Coefficient (MCC),</strong> (TP×TN − FP×FN) ÷ √((TP+FP)(TP+FN)(TN+FP)(TN+FN)). <em>How well do the system’s calls agree with the form, across both populated and blank fields?</em> MCC is the correlation between what was on the form and what the system reported. It is the one metric here that uses all four cells and still resists class imbalance: a system that guesses “value” everywhere, or “blank” everywhere, scores near zero no matter how lopsided the form is, where Accuracy would look fine. Statisticians like it for that reason. We don’t use it often for three practical reasons:<br>(1) It adds nothing once you report F1 and Specificity (or Recall, Precision, and Specificity or False Positive Rate). Those already cover populated fields and blanks separately.<br>(2) The scale runs from −1 (always wrong) through 0 (no better than chance) to +1 (perfect), so it does not read as a percentage, and a business audience will misread a 0.85 as 85% correct. There is a rescaled variant, “normalized MCC,” (MCC + 1) ÷ 2, that runs 0 to 1, but chance sits at 0.5 on that scale, so a “75%” is not 75% of anything and will be misread too.<br>(3) It is undefined when any of the four denominator terms is zero (a form with no blanks, or a form with no populated fields), which happens on real forms. If you have a statistician in the room, compute it as a sanity check; do not put it on the slide.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F4sPc7EVWARd5KO6LBFTfQ.png" /><figcaption>MCC &amp; nMCC: Pros: MCC scores populated fields and blanks together in a single number and is not inflated by class imbalance, which is why statisticians favor it. Cons: F1 and Specificity already cover those two things separately, and neither MCC (−1 to +1) nor nMCC (0 to 1, chance at 0.5) reads as a percentage.</figcaption></figure><ul><li><strong>Jaccard Index, TP/(TP+FP+FN).</strong> <em>Of the fields that were populated on the form or by the system, how many did the system get right?</em> Also known as Intersection over Union (IoU), this is literally the Accuracy formula, (TP+TN)/(TP+TN+FP+FN), BUT with TN removed from the top and the bottom. Applied only to fields that were populated on the form or by the system, this omits True Negatives — correctly identified blanks and correctly unchecked boxes — so they cannot inflate the score. That is intuitive to explain in English; however, the F1 Score is more useful because it is the harmonic mean of Precision and Recall, so whichever of the two is weak shows what dragged the score down. Jaccard converts to F1, F1 = 2J/(1+J) and J = F1/(2−F1), but rather than explaining the algebraic conversion and the omitted True Negatives to an audience, it is simpler and more widely accepted in the industry to express the metric as F1 along with Recall, Precision, and Specificity.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eja4h1f62w0-zzBeqHTaWA.png" /><figcaption>How Recall, Precision, Jaccard, F1, and Accuracy are related. Recall, Precision, Jaccard, and F1 are built from the same three cells (TP, FP, FN); Accuracy adds the fourth cell, TN. Add FP to Recall’s denominator and you get the Jaccard Index; remove TN from Accuracy and you arrive at the same place. Give correct answers (TP) double weight and you get F1. F1 and Jaccard (J) are the same measurement on two scales: divide top and bottom by (TP+FP+FN) and they convert exactly. J and F1 punish the same errors, but F1’s 2TP counts each success twice while J counts it once, so the same system always scores lower in J except at 0% and 100%.</figcaption></figure><blockquote><strong>A caution on names:</strong> the Jaccard index (|A ∩ B| ÷ |A ∪ B|, also called Jaccard similarity) is applied to three different things in IDP. In OCR and layout work it appears as IoU (Intersection over Union), a geometry score: the area where a predicted bounding box and the ground-truth box overlap, divided by the area they cover together. A prediction counts as correct when IoU clears a threshold, commonly 0.5, so IoU tells you whether the system found the right region of the page, not whether it read the right value. As a string comparator, the same index is applied to the tokens of a single value: “123 Main St” and “123 Main Street” share two of four tokens, a similarity of 0.5, which a threshold then accepts or rejects. Both of those are per-item scores used inside the match decision. Our J is the third use: TP ÷ (TP + FP + FN) computed over field outcomes across the whole form, which is the Accuracy formula with TN removed. (A strict set-overlap J over field-value pairs would count a misread twice, the benchmark convention; ours counts it once.) We sometimes think in J terms because dropping TN says plainly “we don’t score blank fields,” but to avoid the confusion above, and because F1 is the industry standard, we report F1 alongside Recall (Sensitivity), the easiest to understand, and supplement with Precision, Specificity (or False Positive Rate), and then Accuracy.</blockquote><h4>Headline Score</h4><p>When presenting to stakeholders, expect to be asked for one score, or at least a leading one: the headliner. Weighing several metrics at once is an engineering skill, not an executive one. Three or more ML metrics on a slide invite confusion, and an audience hunting for a single number gravitates to Accuracy, a metric you don’t want them anchoring on because blanks inflate it.</p><p>Lead with a headline they can grasp before the breakdown: <strong>Recall</strong> or <strong>F1</strong>. To understand why, let’s start with the questions Recall and Precision answer:</p><ul><li>Recall asks: “<em>Of the fields with values, how many were extracted?</em>”</li><li>Precision asks: “<em>Of the extractions made, how many were correct?</em>” (the “misread counts as FN” convention narrows it to, “<em>How much invented data is mixed in with the correct data?</em>”)</li></ul><p>The two have opposite blind spots.</p><ul><li>Precision cannot see the misses: a system that returns blank whenever it is unsure scores high on precision while leaving data behind, since a blank is not an extraction and never enters precision’s denominator. <em>(Under the “map misread to FN” convention it cannot see the misreads either; those land in FN.)</em></li><li>Recall cannot see the inventions: a system that guesses a value into every blank field loses nothing on recall, since those hallucinations land in FP, which recall never counts. Each metric catches the failure the other ignores.</li></ul><p>The F1 score, 2TP ÷ (2TP + FP + FN), closes both holes: its denominator contains both the misses and misreads (FN) and the inventions (FP), so both are punished. Unlike a simple average, which sits halfway between the two numbers, a harmonic mean sits closer to the lower one, so a weak Precision or a weak Recall drags F1 down with it. For example, a 99% Recall with 10% Precision averages to 54.5%, but the F1 is only 18.2%, while a 99% Recall with 98% Precision averages to 98.5%, with an F1 also at 98.5%. The only way to a high F1 is to be good at both!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OYyGNBT4bixCC0ygVhJkXw.png" /><figcaption>F1 Score of 18.2% (low: bad) versus 98.5% (high: good). In this example, both systems have 99% Recall. The system with low F1 Score is punished for its 10% Precision. High Recall with low Precision means it found nearly every value by guessing at everything, so almost nothing it reports can be trusted! The system with high F1 Score keeps 99% Recall with 98% Precision. It finds nearly everything and rarely guesses, so F1 Score matches the simple average at 98.5%. A simple average would score the first system at 54.5% and hide the problem; F1 exposes it. The only way to a high F1 Score is to be good at both Recall and Precision.</figcaption></figure><p>HOWEVER, F1 has two downsides with a non-technical audience. It can sidetrack a briefing with “<em>WTF is an F1 score?</em>” questions, and under our convention it grades softer than Recall: Precision only counts invented values, so an engine that rarely invents anything sits near 100% Precision and pulls F1 up with it. The misread table back in “Handling Text Misreads” section illustrated the gap: 2,500 misreads, Precision 100%, F1 85.7%, Recall 75%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*wTH9mrEjq76bUY4U3c_f0w.png" /><figcaption>That “misread table” again</figcaption></figure><p>Put that “Precision 100%” on a slide and the room hears “right 100% of the time.” For business reporting, Recall is the simpler choice because its percentage means what a non-technical listener assumes it means: “<em>of everything filled in on the forms, how much did the system get right?</em>”</p><blockquote>TIP: After headlining with <strong>Recall</strong>, supplement it on the same slide with <strong>Specificity</strong> (hallucination control) and <strong>F1</strong> (harmony of Recall and Precision), or <strong>Precision</strong> if you are not showing <strong>F1</strong>. Keep the raw invented count next to <strong>Specificity</strong>, print the convention next to F1 (e.g.“misread = FN”), and label whichever metric cannot see misreads (Precision under FN-only, Recall under FP-only) so a 100% is not read as “100% correct.” <strong>Accuracy</strong>, if shown at all, goes last with “<strong>includes blanks</strong>” beside it.</blockquote><h4>Record Raw Outcomes First</h4><p>When possible, record raw outcomes (raw data) before mapping them to the confusion matrix. Metrics are convention-dependent; raw outcomes are not. Every metric in this article depends on a choice made before the tallying starts: where a misread lands. The same test run therefore produces different Precision and F1 under the FN-only, AWS, and benchmark conventions, so state the convention next to any number you publish.</p><p>For text there are five logical buckets: <em>correct</em>, <em>misread</em>, <em>missed</em>, <em>invented</em>, <em>correct blank</em>. For checkboxes it is the parallel five: <em>correctly checked</em>, <em>correctly unchecked</em>, <em>missed checkmark</em>, <em>unreported</em>, <em>invented checkmark</em>. Map the buckets to TP, TN, FP, and FN under your chosen convention and derive the metrics from there; anyone who prefers another convention can recompute from the same buckets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*EkFI3o2S1Ov4vmjGkas9Vg.png" /><figcaption>Text fields: 5 raw outcomes collapse into the 4 confusion matrix cells for 3 conventions (FN, FP, FN &amp; FP). Green is correct, red is wrong.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5vGtU_IiqurYlyQnr6DriQ.png" /><figcaption>Checkboxes: 5 raw outcomes collapse into the 4 confusion matrix cells for 3 conventions (FN, FP, FN &amp; FP). Green is correct, red is wrong. “Unreported” is a NULL or missing state: the software may have skipped the check box, couldn’t find it, couldn’t map it, or was not confident enough to determined checked or unchecked. If ground truth expected it to be checked, map to FN; if it expected unchecked, map to TN.</figcaption></figure><p>The buckets also double as a check: for each form they must sum to the number of fields scored, and the populated and blank subtotals must match the ground truth counts.</p><p><strong>For example</strong>, AWS Labs’ Stickler, the evaluation library behind the GenAI IDP accelerator, already records these five buckets per field (TP, FD, FA, FN, TN, with FP = FA + FD). Where a string value lands between TP and FD depends on the field’s configured comparator and match threshold (exact, fuzzy, numeric tolerance, or semantic), so two teams running the same output with different thresholds will report different bucket counts; for list fields, items with no counterpart after Hungarian matching split into FN and FA rather than FD. Its default metrics use the FP-only convention, a <em>recall_with_fd</em> option switches to the benchmark &quot;both&quot; convention, and our FN-only numbers follow from the same counts (FN = FN + FD, FP = FA).</p><p>One caveat for checkboxes: Stickler classifies on null versus non-null, so a boolean false is a value, not a blank. Correctly unchecked scores as TP, a flipped state scores as FD, and an unreported unchecked box scores as FN. To get the TN-based checkbox scoring described above, encode unchecked as null rather than false, or tally checkboxes outside Stickler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*504KieIo1ZGfqYE2gOTQ7g.png" /><figcaption>For checkboxes, Stickler treats it as a value, so 4 of the 6 rows land in different cells. Encode unchecked as NULL to make Stickler match the diagram. FD = False Discovery = “misread”: both ground truth and prediction have a value, but they don’t match.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LhUqd24kFOvvagh63IZOaw.png" /><figcaption>Scoring checkboxes: Stickler versus previous diagram.</figcaption></figure><p>These mappings get messy fast, so back to the original point: when possible, record raw outcomes (raw data) before mapping them to the confusion matrix. This gives you the flexibility to match whichever scoring convention a benchmark or tool uses.</p><h3>Field-level scoring is blind to structure</h3><p>Everything so far scores one field at a time. Forms are rarely flat lists of independent fields; they carry entity structure: for example, a dependent block with a name, birthdate, and relationship that repeats for each dependent, each prior address, each line item. Field-level scoring cannot see the block.</p><p>For example, a system can extract every attribute correctly but mis-map and swap two of them between dependents. The score drops by two fields, the same punishment two typos would earn; however, the actual impact of that error is far greater. Two entire dependent records (entities) are wrong and must be re-keyed downstream.</p><p>Forms with repeating entities need a group-level score alongside the field-level one: a group counts as correct only when all of its key-value pairs are correct and linked to the right entity.</p><p>Entity scoring is beyond the scope of this article, but the shape of it goes like this:</p><ul><li>grouping fields into entities,</li><li>defining the schema that says which fields make up a dependent and how the block repeats,</li><li>deciding which groups carry the most weight (for example, if the system cannot identify the person then the rest of the form is data about no one),</li><li>and scoring a group as correct only when every value in it is right and attached to the right entity.</li></ul><p>That entity/schema/rules/scoring work is domain modeling: it takes subject-matter experts who know the form, the data, and the business process. It is codified in the schema rather than left to the scorer’s judgment.</p><blockquote><strong>TIP</strong>: Since pooled F1 treats a middle initial and an SSN the same, group fields by the entity they describe and report Recall, FPR, and support per group behind the slide. That is the KIE view of the form: the most important entities get their own scores, and you can compare them against the rest of the form. If the system cannot identify the main entity (the person), the rest of the form may be worthless. The same grouping carries across forms.</blockquote><p>The KIEval paper (below) formalizes an interesting approach to group scoring, and its “Aligned” score also supplies an argument for our misread convention: a wrong value takes one edit to fix, so counting it as both a False Positive (FP) and a False Negative (FN) overstates its cost. KIEval therefore scores TP ÷ (TP + corrections), counting each wrong value once.</p><blockquote>Given the same match definition, KIEval’s Aligned score is our convention on the Jaccard scale, and our F1 is the same measurement on the F1 scale; the two convert exactly (J = F1 ÷ (2 − F1), so an F1 of 0.80 is an Aligned score of 0.67).</blockquote><p>Benchmarks that rely on set-style entity-level F1, such as SROIE, FUNSD, and CORD, count a misread twice, as did the classic MUC information-extraction scorer, and so does KIEval’s own Entity F1. The convention we describe in this article counts it once.</p><h3>Putting It to Use: Reading a Model Card</h3><p>To make these concepts stick a bit more, let’s apply what we have just learned to something you would encounter in the wild. I <a href="https://huggingface.co/models?other=ocr">searched Hugging Face</a> and pulled two random model cards, a PII detection model and an OCR model scored on a benchmark we have’t discussed yet. Let’s read the numbers on each card and see if we can answer questions about what a metric is measuring (section names from this article are in quotes):</p><ol><li><em>Item </em>(“Measuring KIE”). What is one item: a field, a span of text, or an assertion about a page?</li><li><em>Match </em>(“Defining a Match”)<em>.</em> How was a match decided?</li><li><em>Blanks </em>(“Scoring Blanks”). How were blanks handled, and is Accuracy inflated by them?</li><li><em>Misreads</em> (“Handling Text Misreads”). Where does a misread land: FN, FP, or both?</li><li><em>Pooled or per type </em>(“Metrics, Headline Score”). Is F1 pooled across all types or averaged per type, and is there a per-type table?</li><li><em>Dataset and run conditions</em> (“Record Raw Outcomes First”). What dataset and run conditions produced the number?</li><li><em>Structure</em> (“Field-level scoring is blind to structure”). Does the score see structure, or only individual items?</li></ol><p><strong>Example #1, A PII detection model.</strong> The card for <a href="https://huggingface.co/OpenMed/OpenMed-PII-BioClinicalModern-Large-395M-v1">OpenMed-PII-BioClinicalModern-Large-395M-v1</a> reports the following on a stratified 2,000-sample test set from NVIDIA Nemotron-PII.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*U3zksCI8XjZADjCW4C7gOA.png" /><figcaption>HuggingFace Model Card for OpenMed-PII</figcaption></figure><p>Let’s run it through our seven questions:</p><ol><li><em>Item.</em> A span of characters in free text. This is a token classification model: it labels each token, and a run of tokens with the same label is one entity (a name, an SSN, a date of birth). The 54 entity types are the closest thing to field types, but they are not the same thing. A form field is a labeled slot that may legitimately be empty. A span model is never asked whether a field should be blank; it only finds what is there.</li><li><em>Match.</em> Not stated. Models like this are usually scored at the span level, where a predicted span has to match the ground truth on both where it starts and ends and what type it is. That is the convention most token-classification tools default to, but the card does not say, so confirm before comparing.</li><li><em>Blanks.</em> Accuracy is 0.9941 and Micro F1 is 0.9579 (we’ll explain <em>micro</em> later). Most tokens are not PII so they get the label O (“outside any entity”). For the purposes of our analysis, the O tokens play the role of blank fields. Accuracy counts each one the model left alone as correct; F1 does not count them. The four-point gap on the same output is the problem from the “Scoring Blanks” section.</li></ol><blockquote>Part of the gap may also be a unit mismatch: cards like this usually count Accuracy per token and F1 per span, so one wrong boundary token is one error for Accuracy but a whole FP plus FN for F1 (next item). The card does not say, so don’t read all four points as blanks.</blockquote><p>4. <em>Misreads.</em> If the card used span-level scoring, a span with the wrong boundary or wrong type counts twice: once as a False Positive (the predicted span matches nothing) and once as a False Negative (the true span was never found). That is the Benchmark F1 convention from earlier, so this F1 would not line up with a Stickler F1 or an FN-only F1 on the same output.</p><p>5. <em>Pooled or per type.</em> The card reports three versions of F1. All three start from the same per-type tallies of TP, FP, and FN and differ in when the tallies are combined.</p><p>6. <em>Dataset and run.</em> Synthetic text, 50+ industries, generated by NVIDIA and labeled at generation time. The card states the split and that the score comes from a stratified 2,000-sample subset of the test set. The repo also ships a results file from a run on the full test split with slightly different numbers (F1 0.9612 against the card’s 0.9579), so even within one repo the number depends on which run you read. A 0.96 here describes clean, typed, synthetic text, not scanned handwriting.</p><p>7. <em>Structure.</em> None. Entities are scored one at a time; whether a date of birth was attached to the right patient is not tested; however, we do consider field types across forms.</p><blockquote>We will use 3 variants of “F1 score” applied to field types banded across forms:</blockquote><blockquote><strong>Micro F1<em> </em></strong><em>pools TP, FP, and FN across all field types first and computes one F1 from the totals. This is the F1 used throughout this article: one confusion matrix for the whole form. Common types dominate it.</em></blockquote><blockquote><strong>Macro F1</strong><em> computes an F1 for each type and averages them. Each type counts once regardless of how often it appears, so one weak type lowers it noticeably.</em></blockquote><blockquote><strong>Weighted F1</strong><em> is the per-type average weighted by support (how often each type appears). It usually lands near micro F1.</em></blockquote><p>Suppose a form has two field types, <em>Last Name</em> and <em>Relationship to Claimant: </em>Last Name appears 900 times and the system gets 891 right (F1 0.99); Relationship to Claimant appears 100 times and the system gets 50 right (F1 0.50) =&gt; Pooled, that is 941 correct out of 1,000, so Micro F1 is about 0.94. Averaged per type, Macro F1 is (0.99 + 0.50) ÷ 2, about 0.75.</p><pre>Metric      | Score<br>Micro F1    | 0.9579<br>Precision   | 0.9656<br>Recall      | 0.9502<br>Macro F1    | 0.9614<br>Weighted F1 | 0.9565<br>Accuracy    | 0.9941</pre><p>Micro F1 answers “of the data on the forms, how much came through?” Macro F1 answers “is there a field type the system cannot handle?” On the OpenMed card the two are close (0.958 and 0.961), but with 54 types that tells us little. One weak type moves Macro by about half a point, and Macro sitting slightly above Micro means the weak types are, if anything, the larger ones. Neither number can show you one weak type among 54; the per-type table can. On a sparse form with a dozen field types, getting one rare but required field wrong can pull Micro and Macro far apart.</p><p>The card also has a per-entity table with F1, Precision, Recall, and Support. credit_debit_card is at 1.000 on 213 examples; occupation is at F1 0.668 and Recall 0.605 on 688. Neither summary number shows that. This table is the &quot;report by field type&quot; practice recommended earlier, and it is the part of a card to read before the headline.</p><p><strong>Example #2, An OCR model scored on a benchmark.</strong> The card for <a href="https://github.com/datalab-to/omni_parse_bench">Data lab’s Indra OCR</a> reports a text score of 41.3 on <a href="https://huggingface.co/datalab-to/indra-ocr">OmniParseBench</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*2fFkNQXmeAMnmA_wUaFn4w.png" /><figcaption><a href="https://www.marktechpost.com/2026/10/14/datalab-omniparsebench-open-ocr-document-parsing-benchmark/">https://www.marktechpost.com/2026/10/14/datalab-omniparsebench-open-ocr-document-parsing-benchmark/</a></figcaption></figure><p>Let’s apply our 7-point check again:</p><ol><li><em>Item.</em> An assertion about a page. OmniParseBench is 16,288 yes/no tests over 2,937 pages. Each test checks one thing: a string is present, text A comes before text B, a line appears exactly N times, a table cell sits under the given headings, or a line is inside a block of a given kind (heading, text, or table).</li><li><em>Match.</em> Depends on what is being checked. Text is normalized on both sides first and is allowed a few small differences. Numbers have to match digit for digit. Equations are compared by what they render to, not by the raw LaTeX. The layout test never compares text at all; it only uses the line to find which block it sits in</li></ol><p>3 &amp; 4. <em>Blanks and misreads.</em> There is no confusion matrix here. This is by design. OmniParseBench works more like a test suite than a scorecard: each page gets a list of specific yes/no checks, and the score is how many passed. Each family’s score is the share of its tests passed, and the headline is the average of the three families (Indra’s headline is 62.1; the 41.3 is its text score). Tests an engine cannot attempt are marked unsupported and left out of its score rather than failed, much like the “unreadable” and “unreported” states we discussed earlier. Invented text can still fail a test when it lands in the wrong place, but only one of the five test types is built to catch it, so there is no direct counterpart to Precision or Specificity. The score is a pass rate, closer to the “intuitive accuracy” at the start of this article than to any metric in the Metrics section. It is still a rigorous way to score document parsing, with open scoring code and no model judging the output. It answers a different question than a confusion matrix does.</p><blockquote>Indra’s 41.3 OmniParseBench text score in Example #2 cannot be compared to any of the F1 scores from the PII card in Example #1. The scoring method differs, the unit differs (one entity span versus one assertion about a page), and the two models do different jobs on different datasets. Any one of those would rule out a direct comparison. Each score says how a model did at its own task, not which model is better. Apples to oranges.</blockquote><p>5. <em>Pooled or per type.</em> Family scores and tag slices (handwriting, rotated, degraded, script, language) take the place of per-field-type scores.</p><p>6. <em>Dataset and run.</em> Mostly print; the card reports about 8% on the handwriting tests. The card’s Tesseract score (19.3) differs from the leaderboard’s (17.6), and one competitor’s row omits 19 pages because a shard stalled. Same benchmark, different runs, different numbers, which is why you should state the convention and run conditions next to any number you publish.</p><p>7. <em>Structure.</em> Because the item here is an assertion about a page rather than a field or an entity, the benchmark can check reading order, table cells, and block type, which is more than a field-level score sees. What it cannot check is whether a value belongs to a labeled field, which is what a KIE F1 measures.</p><blockquote>Neither example scored a form quite the same way this article does: Example #1 scored spans one at a time, with no sections or entities, and Example #2 scored pages. Benchmarks like <a href="https://huggingface.co/datasets/allenai/olmOCR-bench">olmOCR-bench</a> and <a href="https://huggingface.co/datasets/datalab-to/omni_parse_bench">OmniParseBench</a> score document parsing, not KIE, so Indra’s OmniParseBench score is not evidence about form extraction either way. Most of this article is invested in confusion-matrix metrics for scoring forms, but specialized benchmarks like these measure what a KIE confusion matrix cannot (for example, reading order, table structure, block type, and how an engine holds up on handwriting, rotated pages, and degraded scans). Such benchmarks complement a KIE score rather than replace it.</blockquote><p>A model-card metric is a score on someone else’s golden set. It can tell you which models are worth trying. It cannot tell you how a model will score on your forms or where to set your threshold. Those numbers come from your own stratified sample, under a convention you state. Datalab (OmniParseBench) sums it up with, “<a href="https://www.datalab.to/blog/why-we-built-omniparsebench">the best benchmark is your own documents. Run your own evals.</a>”</p><h3>Recap</h3><ul><li>Form extraction is Key Information Extraction, not OCR: score whether each labeled field got the right value, not whether the characters were read.</li><li>Define the match before you score. Use field-specific comparators (normalize dates, currency, and addresses; fuzzy-match text) so formatting differences aren’t counted as misreads.</li><li>Ground truth is made by people, and people can sometimes disagree. Have two or more annotators label a sample independently and measure agreement per field type, so you can tell system error from fields the humans can’t agree on either.</li><li>Decide up front how you populate TP, TN, FP, and FN, including how you score NULL and missing outputs for text and checkboxes, and whether “unreadable” gets its own count. Be consistent and clear on these mappings, especially if you change them over time.</li><li>A “misread” is a wrong value returned for a field that had one. Where it lands in the confusion matrix changes the numbers more than any other choice:<br><em>(1) FN only (this article): Recall punishes it. <br>(2) FP only (AWS accelerator, AWS ML blog): Precision punishes it. <br>(3) Both FN &amp; FP (Benchmark F1, MUC): both punish it so F1 drops further. </em><br>F1 is identical under the FN-only and FP-only conventions, but is lower under Benchmark F1/MUC (punished twice). Recall and Precision differ across all three, so scores from different conventions aren’t comparable. Always state which convention a score uses before comparing.</li><li>Score checkboxes separately from text, with their own F1. Unchecked boxes flood the tally with True Negatives and inflate anything that counts them.</li><li>Main takeaway for metrics: because of blank, skipped, and optional fields, don’t use <strong>Accuracy</strong> (correct ÷ total) as the headline score; instead use <strong>Recall</strong> (correct ÷ populated) or <strong>F1</strong> (the harmonic mean of Recall and Precision, where <strong>Precision</strong> is correct ÷ (correct + invented)), and check the <strong>False Positive Rate </strong>(invented ÷ blank).</li><li>Useful to know, but not necessarily report: <strong>Jaccard</strong> and <strong>F1</strong> punish the same errors, but F1 counts each correct field twice, so Jaccard reads lower. The two convert exactly. F1 is preferred because it is the industry standard and because in IDP the term “Jaccard” can be applied to three different things: bounding-box overlap, token matching on a single value, or our field-level set overlap across the form.</li><li>When possible, record raw outcomes (raw data) before mapping them to the confusion matrix. (For example, text may have outcomes such as <em>correct, misread, missed, invented, correct blank</em>; checkboxes have a different set.) This gives you the flexibility to match whichever scoring convention a benchmark or tool uses.</li><li>Honesty and transparency are key. Do not cherry-pick. Publish the full confusion matrix and state your convention next to any number you report.</li><li>Field-level scoring is blind to structure. It checks each field on its own and never asks whether the fields belong together. Forms with repeating entities (dependents, addresses, line items) need a group-level score as well, and someone has to define what counts as one group on that form. The scorer cannot work that out on its own.</li><li>Read other people’s scoring systems before you build your own. Someone had to decide what counts as an item, what counts as a match, what to do with blanks and misreads, and whether F1 is pooled or averaged per type.</li><li>A pass rate and an F1 can look alike on a card while counting different mistakes. Even two F1s can differ in how a match or a misread was counted. Take the time to understand the scoring method behind a model card. There is more than one way to score, and metrics that sound alike can measure different things, so check the method before you compare two numbers.</li><li>Treat a model-card score as a reason to try a model, not as a prediction. Read the per-type breakdown before the headline, since that is where the weak field types show. To find out how a model does on your forms, test it on your forms.</li></ul><h3><strong>FURTHER READING</strong></h3><h4>Benchmarks and Datasets</h4><p>It is useful to see samples of “industry standards” in action. The three classic KIE datasets (SROIE, FUNSD, CORD) are the ones behind the “misread counts as both FP and FN” convention discussed above, and every LayoutLM-style paper reports on them. The two newer benchmarks (CC-OCR, OCRBench v2) target large multimodal models and fold the classic datasets in as subsets. All five score extraction with F1.</p><ul><li><a href="https://rrc.cvc.uab.es/?ch=13"><strong>SROIE: ICDAR 2019 Competition on Scanned Receipt OCR and Information Extraction (Huang et al., ICDAR 2019)</strong></a> 1,000 real scanned receipts from Malaysian shops and restaurants, mostly English text and digits, released for the ICDAR 2019 Robust Reading Competition. The competition had three tasks (text localization, OCR, and key information extraction); Task 3 is the KIE benchmark: four fields per receipt (company, date, address, total), 626 train and 347 test images in the split most papers use. The smallest schema of the three, and the one most often quoted as “the” KIE benchmark since the LayoutLM papers adopted it. Entity-level F1.</li><li><a href="https://arxiv.org/abs/1905.13538"><strong>FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents (Jaume et al., ICDAR Workshops 2019)</strong></a> 199 real one-page forms (149 train, 50 test) sampled from the form category of RVL-CDIP, which is itself drawn from the Truth Tobacco Industry Documents archive, so the forms are marketing, advertising, and research paperwork from the 1980s and 90s, scanned at about 100 dpi with real noise. Mostly machine-printed; handwriting appears mainly in signatures and dates. Every text block is labeled header, question, answer, or other, and question-to-answer links are annotated, so it scores both entity labeling and entity linking. Closest of the three to the forms problem in this article. Entity-level F1.</li><li><a href="https://github.com/clovaai/cord"><strong>CORD: A Consolidated Receipt Dataset for Post-OCR Parsing (Park et al., Workshop on Document Intelligence at NeurIPS 2019)</strong></a> From Naver CLOVA: receipts from Indonesian shops and restaurants collected by crowdsourcing, about 11,000 in total with 1,000 released publicly (800 train, 100 dev, 100 test). The 30-field schema is grouped into menu, void menu, subtotal, and total, and line items repeat, so it exercises both field-level and group-level extraction. Standard test set for Donut, LayoutLMv3, and LMDX. Entity-level F1.</li><li><a href="https://arxiv.org/html/2412.02210v3"><strong>CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy (Yang et al., ICCV 2025)</strong></a> From Alibaba’s Qwen team, built to evaluate large multimodal models rather than OCR engines. 39 subsets, 7,058 annotated images, 41% newly collected from real applications, across four tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction. The KIE track reuses FUNSD and CORD alongside newer subsets. Field-level F1.</li><li><a href="https://arxiv.org/html/2501.00321v2"><strong>OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning (Fu et al., 2024)</strong></a> 10,000 human-verified question-answer pairs across 23 tasks and 31 scenarios (documents, receipts, charts, street scenes, and more), in English and Chinese. Each task is scored 0 to 100 with a task-appropriate metric and averaged into one composite; the key information extraction and key information mapping tasks are F1-scored.</li></ul><p>We used these two Hugging Face dataset cards in our examples:</p><ul><li><a href="https://huggingface.co/datasets/datalab-to/omni_parse_bench"><strong>OmniParseBench dataset</strong></a>. The benchmark’s pages, tests, and every parser’s result on every test: 16k yes/no checks on 3k pages in 92 languages, each tagged by content such as tables, forms, handwriting, degraded or rotated text, and multi-column layouts. For handwritten forms, score on the handwriting and form tests; for faxed or photocopied pages, check the degraded and rotated slices; for invoices and statements, start with the table tests.</li><li><a href="https://huggingface.co/datasets/nvidia/Nemotron-PII"><strong>NVIDIA Nemotron-PII dataset</strong></a>. The test set behind the OpenMed PII model: 100,000 synthetic English records across 50+ industries, with span-level labels for 55+ PII/PHI types produced at generation time. Mixes structured documents (forms, invoices) with unstructured ones (emails, notes), all typed text.</li></ul><h4>Explainers</h4><ul><li><a href="https://www.llamaindex.ai/glossary/f1-score-for-document-extraction"><strong>What is F1 Score for Document Extraction? (LlamaIndex)</strong></a><strong> </strong>Builds on the basics stated here by working through examples. It excludes True Negatives from scoring on the grounds that “correctly identifying absent fields does not demonstrate extraction capability,” and notes that plain accuracy is pushed up artificially on sparse documents by the true negatives.</li><li><a href="https://aws.amazon.com/blogs/machine-learning/document-intelligence-evolved-building-and-evaluating-kie-solutions-that-scale/"><strong>Document intelligence evolved: Building and evaluating KIE solutions that scale (AWS Machine Learning Blog)</strong></a><strong> </strong>AWS defines the four field-level cells for KIE, including True Negative as a field correctly not extracted, and reports F1 as the headline metric.</li><li><a href="https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/blob/main/docs/evaluation.md#metrics-and-monitoring"><strong>Accelerated Intelligent Document Processing on AWS: Metrics and Monitoring</strong></a><strong>.</strong> AWS GenAI IDP uses <a href="https://awslabs.github.io/stickler/#key-use-case-key-information-extraction-kie"><strong>Stickler</strong></a>, AWS Labs’ open-source Python library for structured JSON comparison and evaluation, as its evaluation framework. In addition to Precision, Recall, F1, and Accuracy, it tracks False Alarm Rate (FAR) and False Discovery Rate (FDR). All of these are computed from per-field confusion counts whose match/mismatch boundary is set by the user’s chosen comparators and thresholds, so reported figures are comparable across runs that share the same configuration.</li><li><a href="https://aclanthology.org/J96-2004/"><strong>Assessing Agreement on Classification Tasks: The Kappa Statistic (Carletta, Computational Linguistics, 1996)</strong></a><strong> </strong>The standard reference for inter-annotator agreement in NLP. Argues for the kappa statistic, which corrects for chance agreement, over raw percent agreement when measuring whether two annotators produce the same labels.</li><li><a href="https://arxiv.org/html/2503.05488"><strong>KIEval: Evaluation Metric for Document Key Information Extraction</strong></a><em> </em>Acknowledges entity-level F1 as the established convention across KIE benchmarks including SROIE, FUNSD, and CORD, while arguing that the convention is insufficient for industrial use because it cannot see structural errors. KIEval adds a group-level F1, in which a group of related entities counts as correct only if all of its key-value pairs are extracted correctly together. A separate “Aligned” score recasts errors as correction steps: a misread is a single “Substitution,” a complete miss is an “Addition,” and a hallucination is a “Deletion.” The paper also reviews Tree Edit Distance (TED), the soft-match metric Donut introduced for generative KIE models, and rejects its partial credit for near-miss strings, since “7000” and “1000” are different values.</li></ul><h3><strong>Disclaimer</strong></h3><p>The views in this article are my own and do not represent the official positions of my employer or any client. The article surveys several scoring conventions and approaches, some competing and some complementary, for the reader’s consideration; it is not a specification, and any method here should be validated against your own forms, data, and requirements before use in a production system. The examples are illustrative; nothing here describes, confirms, or denies the scoring methods, data, or systems used in any current or past engagement. Products, services, models, benchmarks, and publications are mentioned for illustration only; no endorsement is implied, and I received no compensation from any vendor named.</p><p><strong>Previous: </strong><a href="https://medium.com/@paul.d.short/scoring-idp-part-1-confidence-is-not-correctness-71384dc15f56">Scoring IDP, Part 1: Confidence Is Not Correctness</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3c7eafb3379" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scoring IDP, Part 1: Confidence Is Not Correctness]]></title>
            <link>https://medium.com/@paul.d.short/scoring-idp-part-1-confidence-is-not-correctness-71384dc15f56?source=rss-49aad253ddef------2</link>
            <guid isPermaLink="false">https://medium.com/p/71384dc15f56</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[document-processing]]></category>
            <category><![CDATA[idp]]></category>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Paul Short]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 23:47:22 GMT</pubDate>
            <atom:updated>2026-10-17T23:57:20.044Z</atom:updated>
            <content:encoded><![CDATA[<p>Optical Character Recognition (OCR) engines and Intelligent Document Processing (IDP) platforms typically return a confidence score alongside every extracted value. That score is sometimes mistaken for a measure of correctness. It is not. In plain English:</p><ul><li><strong>Confidence</strong> is a state of certainty. It describes the one making a claim, not the claim itself. A claim can be made with complete confidence and still be wrong.</li><li><strong>Correctness</strong> is conformity to fact. It is a property of the claim, and the only way to establish it is to check that claim against reality. A test is required.</li></ul><p>In the real world, it is important to be <em>both</em> <strong>confident and correct</strong>. Informally, a person or system that is only confident but not correct or only correct but not confident isn’t “reliable.” Let’s state the concept of Confidence versus Correctness in Software Quality Assurance (SQA) terms:</p><ul><li><strong>The Software Under Test (SUT):</strong> The party making the claim. In this domain, the SUT is the IDP solution or extraction pipeline.</li><li><strong>The Expected Results (Ground Truth):</strong> The objective facts, the answers to the test. These are established independently before a test runs, representing what is actually written on the page.</li><li><strong>The Actual Results (The Claim):</strong> The software’s output. The specific value extracted from a document field.</li></ul><p>Considering the SUT, the expected results, and the actual results:</p><p><strong>Confidence</strong> is the SUT’s internal certainty about its own output. It does not dictate whether the output is right. It is the score the pipeline attaches to each extracted value, representing its internal certainty. For example, consider a digit classifier trained on MNIST, a standard dataset of 70,000 handwritten digits. A handwritten 4 and a 9 can look alike, so the pair is a common source of errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*egN6zp-n1VN3j_v_hphROA.png" /><figcaption>How confident are we that this is a 9 or a 4? Is the ground truth wrong?</figcaption></figure><p>The digit classifier has ten output classes, one per digit, and no class for “unsure.” Its confidence score measures how strongly it prefers the chosen digit over the other nine. If the image is ambiguous but the model leans toward 9, the score for 9 can be high and the answer can still be a 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Uq-E7C8RxlDqUeXkzJeBw.jpeg" /><figcaption>MNIST examples are often covered in machine learning primers, so I asked AI to generate it for me. This was Gemini Pro’s best attempt to draw a “9-looking 4.” When prompted toward better 4s or 9s, Gemini could only draw a clean 4 or 9, and Claude didn’t even come close.</figcaption></figure><p>This is where calibration comes in.</p><p>Formally, a model is well-calibrated if its stated confidence matches its empirical accuracy — 85% confidence should correspond to 85% correctness, as measured by testing predictions against known ground truth. Neural networks are typically miscalibrated in the direction of overconfidence: their training objectives keep pushing predicted probabilities toward the extremes, even past the point needed for correctness. This tendency is less pronounced in a simple digit classifier like MNIST and grows with the size and depth of the neural network. RLHF adds a separate, second effect in LLM assistants: pre-trained models that are well calibrated have been observed to become markedly less so after post-training.</p><p>One interpretation is that <em>human raters favor confident, decisive-sounding answers over ones that accurately convey uncertainty</em>; another is that <em>the incentive is built into binary benchmark grading itself, which rewards guessing over abstaining</em>.</p><p>Let’s also consider that MNIST was built from digits written by Census Bureau employees and high school students around 1990, one population writing in one era’s style. Nothing stops the model from being applied to handwriting from other populations, places, or eras it has no samples for. It still responds with the closest match and a confidence score for that match. Whether the match is <em>actually</em> the one on the page is a question that confidence cannot answer.</p><p><strong>Correctness</strong> is whether the output actually matches the expected result. The SUT itself should never make this judgment. A separate party (in test theory, “the oracle”; in practice an evaluator, tester, or judge) holds the answer key and performs the comparison. The oracle might be another program, a reference implementation, or a person. Establishing the correct output for a given input is a recognized difficulty in software testing known as “the oracle problem.” For document extraction today, the oracle is still a person. This results in a Catch-22. Because no software out-of-the-box can easily read messy scans, the automated system is bottle-necked by the speed of the human needed to verify off-ramps or create tests. The work is not necessarily slow because the process is inefficient. Deciding what the right answer is, field by field, is the most labor intensive part of the test construction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*FCkAIRxEWoaf9NSJHdkKFA.png" /><figcaption>Confidence does not directly measure Correctness</figcaption></figure><h4><strong>Confidence has legitimately practical uses</strong></h4><blockquote>It is common practice use <strong>low confidence scores</strong> as a threshold to <strong>flag extractions for manual review and send them to a person for correction</strong>; however, a high confidence score does not mean the extractions are correct.</blockquote><p>In production, the pipeline can use it to decide what to do with an uncertain value: <strong>send it to a person</strong>, <strong>flag it with a warning</strong>, or <strong>reject it</strong>. During development, if low scores keep showing up on the same field or the same form type, that tells you where the engine is struggling. In both cases the score is being used as what it is, a signal from inside the system. In theory, this could be applied in development, testing, or production sampling to the “ugly 4 that looks like a 9” MNIST example; however, if the confidence score is skewed too high, the ugly 4 clears the threshold and never gets looked at. Confidence is NOT guaranteed to catch things, but it <em>could</em> catch <em>some</em> things. To recap,</p><ul><li>confidence originates inside the thing being evaluated,</li><li>correctness is decided by a judge outside it,</li><li>a system cannot grade its own exam, and</li><li>(informally) a “reliable” system is both confident and correct.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*B2Ax-gzF8MolOfKKarKaGw.png" /><figcaption>“Reliable”: Confident (limited usefulness) + Correct (tested)</figcaption></figure><h4><strong>Characters vs Field Mappings</strong></h4><p>When Textract returns {&quot;Text&quot;: &quot;John&quot;, &quot;Confidence&quot;: 99.2}, the 99.2 describes character recognition on a string at a set of coordinates. It does not say that “John” belongs to the field labeled First Name; a basic OCR engine does not know that. Reading the text and deciding which field it belongs to are two different steps, and the engine can read “John” perfectly and still attach it to the wrong label. That confidence score rates the reading and says nothing about the matching.</p><p>Form-aware layers add a second score for the matching step. Textract’s FORMS output, for example, attaches one confidence to each KEY/VALUE pair for the relationship itself, separate from the OCR confidence on the words inside it, so a field can carry a 99 on the characters and a 52 on the pairing. The pairing score is useful for triage, but it is still the system rating itself. Even a well-calibrated pairing score tells you how often pairings like this one turn out right, not whether this one did. A system can read every string on a page at 99% confidence, report high confidence on its key-value pairs, and still put a dependent’s birthdate in the policyholder’s birthdate field.</p><p>Confidence also cannot describe fields the system did not return. If nothing was extracted, there is no output to score, so misses do not appear in the confidence distribution at all. On sparsely populated forms, the more serious problem is the reverse: a plausible value written into a field that should have been empty. These often carry high confidence, because the score reflects the model&#39;s certainty about the characters, not whether the field should have been populated.</p><h4><strong>LLMs Hallucinate Confidence Scores</strong></h4><p>Whether a confidence number means anything depends on the architecture producing it. Purpose-built OCR products (AWS Textract, Mistral OCR, Nanonets) return computed scores as a documented output. The math behind them varies. Textract’s 0–100 score comes from a recognition model; Mistral OCR’s 0–1 scores come from a transformer’s token probabilities. In every case the number is produced by the recognizer, not requested from a chat interface. Whatever the scale (0–100 for Textract, 0–1 for Mistral and Nanonets), a score near the top should be read as “very likely,” not as a guarantee.</p><p>A <strong>conversational LLM</strong> prompted for a confidence score does not produce this kind of number. It has no mechanism for reporting a computed probability through a prompt; it generates text that looks like a confidence score, predicting what such a number would plausibly be. If you ask whether the number was made up, it will usually say so. That is not the same as saying the numbers are random noise, but they may be overly-confident guesses. Treat an LLM’s verbalized confidence score as, at best, a weak ranking signal, and validate it against ground truth before using it as a gate.</p><blockquote><strong>Logprobs on Bedrock:</strong> On Amazon Bedrock, as of September 2026, token-level log probabilities are exposed through the Converse API for hosted models such as Amazon Nova 2 via a logprobs parameter, and through Custom Model Import via a return_logprobs flag (added mid-2025). The exception is models imported on the OpenAI-compatible schema, which require the InvokeModel API instead. Even where available, a token probability measures certainty about the next token, not about whether a field value is right, and extraction studies have found log probabilities no more useful than verbalized scores as a gate. A gate built on logprobs is also tied to the models and APIs that expose them; switch models and the gate silently disappears. It’s probably a bad idea to do this.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*hudG_1gn0pY-NjI5qMEPdA.png" /><figcaption>Primed with the idea, Claude correctly completes the chain. The answer is next-token prediction about next-token prediction. It’s right on the nose this time; however, we have also seen multi-modal LLMs (vision+text) forget (or not know) they can’t introspect so they respond with fabricated confidence scores. Models that truly have calibrated confidence scores will have that documented by the vendor as a feature (it is NOT common). Know the model’s category and capabilities first. Read the vendor’s documentation, don’t ask the model.</figcaption></figure><p><strong>Grounding check for LLM extraction.</strong> If you are using an LLM to extract, add a grounding check. It is deterministic, and it is more important than any confidence number the model gives you. The term “grounding” comes from the LLM/RAG world, where an answer is grounded if every claim can be traced to a provided source rather than the model’s memory.</p><blockquote><strong><em>Ground truth</em></strong> is the answer key; <strong><em>grounding</em></strong> is whether the model can point to where on the page it got its answer. <em>They mean two different things.</em></blockquote><p>For extraction, grounding means every value the model returns must be traceable to the OCR text of that page, allowing for formatting differences like “01/15/2026” vs “Jan 15 2026” or “$1,200” vs “1200”. If the value can’t be traced to the page, it’s a hallucination, whatever score came with it. Google’s LMDX paper made this a named pipeline stage, a “grounding mechanism” that checks each extracted value against the OCR segment it claims to come from, discards anything that fails, and returns a bounding box for what survives. This catches the worst LLM failure, a plausible value written into a field that should have been blank, and it doesn’t ask the model to grade itself. It does not catch mismaps: a dependent’s birthdate written into the policyholder’s field is fully grounded. Those are caught by validation rules and by the golden set.</p><h4><strong>You Can Calibrate the Pipeline, Not the Model</strong></h4><p>Most commercial OCR engines do not let you retrain the text-recognition layer (“vision”). Several let you train the layer above it: Azure Document Intelligence custom neural models and Google Document AI custom extractors learn your fields, checkboxes, and tables from labeled documents. Textract’s only trainable piece is a Custom Queries adapter; its OCR, Forms, and Tables are fixed. (AWS put its newer schema-driven extraction in Bedrock Data Automation, not Textract.)</p><blockquote><strong>You Can’t Train Textract:</strong> AWS’s documentation describes Custom Queries as “adapters trained on your own documents” that “customize the base model”; third-party write-ups tend to shorten that to “training Textract on your documents,” or imply that Textract can be fine-tuned on SageMaker. Both overstate what can be trained. Only the Queries adapter is trainable, not Textract itself. (There is a hard limit of 30 custom queries per adapter.) A SageMaker pipeline that appears to fine-tune Textract is really two pieces: Textract, running unchanged as the OCR front end, and a second model behind it that is fine-tuned on Textract’s output. The training happens in that second piece, never in Textract. Text-recognition (Textract itself) is not trainable.</blockquote><p>Because the text-recognition layer is fixed in every one of these IDP products (you cannot retrain or tune the delicate internal math), calibration has to happen in the pipeline around it.</p><ul><li><em>Map confidence to measured correctness.</em> Build a golden dataset of human-verified forms, run it through the engine, and compare the reported confidence against the observed correctness for each field type. If fields scored at 90% confidence are correct 70% of the time on handwritten digits, the auto-pass threshold for those fields should be higher than 90%. Calibrate per field type, form, and document population, since handwriting and print behave differently.</li><li><em>Make the golden set match production.</em> The golden set has to look like production, so sample, analyze, and measure production first: sample forms from the live intake, analyze what the mix is (e.g., which form variants, how many are handwritten, how many are half-empty), and measure the proportions. Then statistically stratify the golden set: split production into groups (for example, form variant, input quality, how densely the form is filled) and sample randomly within each group in the proportion you measured, instead of taking whatever forms are handy. Stratifying doesn’t replace random sampling; it tells the random sampling where to look, so small but important groups like handwritten forms don’t get missed by chance.</li></ul><blockquote>One exception: proportional sampling is right for the headline number but starves the per-stratum thresholds. A 5% handwritten stratum in a 1,000-form golden set is 50 forms, too few to bin by confidence. <strong>Oversample rare-but-risky strata and weight them back down for the pooled figure.</strong> (Most SQA/testers would intuitively bring this up by saying, “let’s cover the corner cases” but data analysts should be involved in the balance.)</blockquote><ul><li><em>Add deterministic validation.</em> Confidence should not be the only gate (it is not intended for scoring and validation). <em>Reject a US ZIP code containing a letter regardless of its score. Validate addresses against postal data. Check dates for plausibility.</em> These rules catch confident-but-wrong values because they do not depend on the model’s self-assessment.</li></ul><h4><strong>Calibrating Confidence</strong></h4><blockquote>It is common practice to <strong>send low-confidence extractions for review and correction by a person</strong>. That is the bare minimal; however, when this happens too often it creates problems such as too much noise, too much manual labor, and reduction in processing speed. To mitigate you should “calibrate” your pipeline against updated ground truth.</blockquote><p>So far we have talked about confidence in terms of strings and fields. What about larger streams, such as sections, forms, and documents? When processing long, continuous streams, errors compound, so a high confidence score on a single page does not guarantee the integrity of the entire document bundle. To establish confidence across forms, documents, and bundles, you should first measure and show that the model’s reported certainty (confidence) aligns with its true accuracy (correctness). This requires empirical testing: you should bin fields by reported confidence and compare each bin’s confidence with its observed correctness on the golden set (ground truth).</p><p>The weighted average gap across these bins is the Expected Calibration Error (ECE); however, because ECE is heavily weighted by easy, successfully processed pages, teams must also evaluate the Maximum Calibration Error (MCE). MCE measures the worst-case divergence across all bins, exposing blind spots where the model claims near-certainty but actually delivers poor accuracy.</p><blockquote><strong>Two cautions on the binning.</strong> First, OCR scores crowd near the top of the scale, so equal-width bins (90–100, 80–90, and so on) leave the lower bins nearly empty; bin by equal counts instead, or at least show how many fields each bin holds, and don’t compute MCE on a bin with a handful of fields.<br>Second, ECE and MCE check whether the score matches observed correctness, not whether it separates correct fields from incorrect ones. A score that returns the same number for every field, say the overall accuracy, has an ECE of 0 and cannot tell you which fields to review. <br><strong>AUROC</strong> measures separation; the <strong>Brier score</strong> punishes miscalibration and weak separation in one number.</blockquote><p>Even a well-calibrated field score has a limit. Hunter Heidenreich calls this the reliability trap: a per-field or per-page confidence that looks calibrated still tells you nothing about whether the whole document came through clean. A large document can still arrive error-free, but the mathematical likelihood of that happening drops exponentially with every added page.</p><blockquote><strong>His example</strong>: at 99% confidence per page, the probability that a whole document arrives error-free is 0.99ⁿ for n pages. A two-page letter passes clean about 98% of the time (0.99², about 49-to-1 odds of a clean document), a 10-page contract about 90% (0.99¹⁰, roughly 9-to-1 odds), and a 100-page medical record only about 37% (0.99¹⁰⁰, worse than even odds), if page errors were independent. His testing showed they are not: a blurry or rotated scan degrades its neighbors, so bad pages come in stretches, and per-page confidence is least trustworthy on exactly those pages. Averaged over all pages the score looks calibrated; on the hard pages it is overconfident, and that is what compounds. On 20-page documents, a model that looked well calibrated page by page implied about 80% confidence that the whole document was correct, when it was right only 40–50% of the time.</blockquote><p>In plain terms: <strong>more pages means more chances to slip</strong>, and <strong>bad pages come in consecutive stretches where confidence is the least trustworthy</strong>, so a high score on any one field or page is not a promise about the whole form or document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s4WcB9qqcDYFcVOCBBclQQ.png" /><figcaption>The Reliability Trap (more pages, more chances to slip; bad pages cluster in stretches)</figcaption></figure><p>One way out of Heidenreich’s reliability trap is to earn the threshold in test and apply it in production. To stop trusting any single field’s or page’s score and instead establish a whole-document threshold, pipelines should adopt three distinct strategies:</p><p><strong>1. Calibrate on the Golden Set:</strong> On your golden set (ground truth), where you know the answers, bin the fields by reported confidence and score each whole document. (A document’s score is usually the minimum confidence across its required fields; a mean hides one bad field. The verifier’s verdict, in strategy 3, is the other option.) Compare each bin’s confidence with its observed correctness to identify your Expected Calibration Error (ECE) and Maximum Calibration Error (MCE). Measure how often whole documents above a candidate threshold came through fully correct; raise the threshold until that rate meets your target.</p><blockquote>How many forms that takes is set by the target: with 0 document errors seen among n auto-passed golden forms, the 95% upper bound on the true error rate is about 3 ÷ n. A 1% target needs about 300 clean auto-passes, 0.1% about 3,000, and per-stratum thresholds multiply that.</blockquote><p><strong>2. Optimize for Throughput in Production:</strong> Not raw accuracy. Throughput is the share of documents that can skip human review while staying under your target error rate. A 96%-accurate engine with well-calibrated scores can beat a 97%-accurate engine with bad ones, because you know which forms you can stop looking at. In production, where you don’t know the answers, documents that hit your earned threshold skip human review (straight-through processing), and the rest route to people.</p><p>The threshold itself is a business decision (it depends on what a wrong field costs downstream), so start high. Don’t expect it to hold. Production drifts: a new form revision moves a field, a different scanner or fax line changes image quality, an outreach campaign brings in a new population of claimants who fill out the form differently or submit look-alike versions generated by third-party software, benefits season brings more handwritten submissions, or the vendor quietly updates the OCR model under you. The threshold you earned only describes the forms that were in the golden set when you earned it. As the input changes, more wrong forms pass under it, and your pass rate won’t show it; a shift in the confidence distribution per field type sometimes will, and is a cheap trigger for an off-cycle sample.</p><p>The fix is to re-measure on a schedule. Pull a sample of the auto-passed (skipped) forms and have a person establish ground truth for them, same as the golden set. Those verified samples are new ground truth; add them to the golden set, re-calibrate, and move the threshold if the error rate has moved. Reviewer corrections on the routed forms are ground truth too, but they are the hard cases, so keep them for error analysis rather than folding them into a golden set that is supposed to match production.</p><blockquote><strong>Warning: routed forms may skew the golden set.</strong> Routed forms are the ones the engine was least sure about, not a random sample of production. Add 3 months of reviewer corrections and the golden set is mostly hard cases. Three things follow. First, the headline Recall falls, and <strong>stakeholders read it as the engine getting worse when only the test got harder</strong>. Second, each confidence bin now holds harder fields than the same bin in production, so measured correctness per bin falls and the earned threshold rises; you <strong>route forms that would have passed cleanly</strong>. Third, the engine’s confidence score is now choosing which forms become ground truth, so <strong>the system is picking its own exam</strong>. Use the corrections to find where the engine struggles. They can go in the golden set, even oversampled, as long as the pooled figures weight routed forms back to production’s share, the same as any other oversampled stratum. (This is where a data analyst can help the test team.)</blockquote><p><strong>3. Implement a Guesser-Verifier Architecture:</strong> To hit high throughput on whole-document thresholds, single-pass extractions aren’t enough. To maximize the number of documents that safely skip human review, Heidenreich proposes using a “Guesser-Verifier” architecture. In this framework, use a lightweight perception model (System 1, aka Fast Thinking/Intuitive model) to propose the data, and a secondary reasoning model (System 2, aka Slow Thinking/Reasoning model) to structurally audit and verify the entire document before it reaches downstream systems.</p><p>System 1 extracts, System 2 audits. This does not break the rule from earlier, “a system cannot grade its own exam.” The oracle holds the answer key and decides correctness; that is still a person with the golden set. Here, our verifier has no answer key. It is a gate: it decides which documents skip human review. Like any other gate, you measure its pass rate on the golden set to earn its threshold. Keep it independent of the extractor (a different model, or at least a different prompt that never sees the extractor’s confidence scores) so the two can’t share the same blind spot. The threshold is still set by measured correctness, not by trusting any single field, page, or verifier.</p><p><strong>NEXT: Measuring Correctness</strong></p><p><em>Confidence</em> only tells you <em>what</em> to review. Measuring <em>correctness</em> requires an oracle: ground truth, match rules, and strict scoring conventions for blank or conditional fields. Comparing confidence to correctness is standard statistics handled by tools like AWS Labs’ Stickler (for ECE, AUROC, and Brier scores). The harder question is how to generate those correctness numbers in the first place. In <a href="https://medium.com/@paul.d.short/scoring-idp-part-2-approaches-to-evaluating-form-extraction-b3c7eafb3379">Part 2</a>, we will cover why form extraction is a Key Information Extraction (KIE) problem rather than an OCR problem, and we will define the scoring conventions required to calculate correctness.</p><h3>REFERENCES</h3><ul><li>AWS. Amazon Textract docs: <a href="https://docs.aws.amazon.com/textract/latest/dg/how-it-works-kvp.html">Form data (key-value pairs)</a>; <a href="https://docs.aws.amazon.com/textract/latest/dg/textract-create-adapter.html">Create an adapter</a>.</li><li>AWS. <a href="https://aws.amazon.com/blogs/machine-learning/unlock-model-insights-with-log-probability-support-for-amazon-bedrock-custom-model-import">Log probability support for Bedrock Custom Model Import</a>. 2025.</li><li>Barr, Harman, McMinn, Shahbaz, Yoo. <a href="https://ieeexplore.ieee.org/document/6963470">The Oracle Problem in Software Testing: A Survey</a>. IEEE Transactions on Software Engineering, 2015. Establishing the correct output for a given input.</li><li>Guo et al. <a href="https://arxiv.org/abs/1706.04599">On Calibration of Modern Neural Networks</a>. 2017. ECE and temperature scaling.</li><li>Heidenreich. <a href="https://hunterheidenreich.com/posts/reliability-trap-document-automation/">The Reliability Trap</a>. 2026.</li><li>Heidenreich et al. <a href="https://aclanthology.org/2025.coling-industry.26">LLMs for Insurance Document Automation</a>. COLING 2025 Industry Track. Page-level calibration does not carry to stream level; source paper for The Reliability Trap.</li><li>Kumar. <a href="https://arxiv.org/pdf/2606.24420">Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction</a>. 2026. Log probabilities, verbalized confidence, and self-consistency all fail as extraction gates.</li><li><a href="https://labelerrors.com/">labelerrors.com</a> (cleanlab; source of the 4/9 screenshot).</li><li>Li, Z. Z. et al. <a href="https://arxiv.org/abs/2502.17419">From System 1 to System 2: A Survey of Reasoning Large Language Models</a>. Feb 2025. “Perception model” (System 1) versus “Reasoning model” (System 2).</li><li>Mistral AI. <a href="https://docs.mistral.ai/capabilities/document_ai/basic_ocr/">OCR processor docs</a>.</li><li>OpenAI. <a href="https://arxiv.org/abs/2303.08774">GPT-4 Technical Report</a>. 2023. Pre-trained model well calibrated; calibration degrades after RLHF post-training.</li><li>OpenAI. <a href="https://openai.com/index/why-language-models-hallucinate/">Why language models hallucinate</a>. Blog, Sept 2025. Grading rewards guessing over abstaining.</li><li>Perot et al. <a href="https://arxiv.org/abs/2309.10952">LMDX: Language Model-based Document Information Extraction and Localization</a>. Google, 2023 (Findings of ACL 2024). Introduces a “grounding mechanism” for LLM extraction: every value is verified against the OCR segment it claims to come from, anything that fails to ground is discarded, and a bounding box is returned for what survives.</li></ul><h3>Disclaimer</h3><p>The views in this article are my own and do not represent the official positions of my employer or any client. The article surveys several scoring conventions and approaches, some competing and some complementary, for the reader’s consideration; it is not a specification, and any method here should be validated against your own forms, data, and requirements before use in a production system. The examples are illustrative; nothing here describes, confirms, or denies the scoring methods, data, or systems used in any current or past engagement. Screenshots of AI assistants are included to illustrate model behavior and are not endorsements of any product. Products, services, and publications are mentioned for illustration only; no endorsement is implied, and I received no compensation from any vendor named.</p><p><strong>NEXT</strong>: <a href="https://medium.com/@paul.d.short/scoring-idp-part-2-approaches-to-evaluating-form-extraction-b3c7eafb3379">Scoring IDP, Part 2: Approaches to Evaluating Form Extraction</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71384dc15f56" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IKFF Kettlebell Tests]]></title>
            <link>https://medium.com/@paul.d.short/ikff-kettlebell-tests-eb465e84aaa3?source=rss-49aad253ddef------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb465e84aaa3</guid>
            <category><![CDATA[notebooklm]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[kettlebells]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Paul Short]]></dc:creator>
            <pubDate>Sun, 02 Nov 2025 22:13:58 GMT</pubDate>
            <atom:updated>2025-11-27T21:17:40.408Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>THIS IS a human-generated writing exercise I WROTE TO TEST <a href="https://notebooklm.google/">Google NotebookLM’s</a> LATEST FEATURES. NotebookLM is specifically tuned to stick to the sources, focused on lowering hallucinations. The contents below contain embedded quotes linked to well-known sources, so ittshould be a good candidate for observing accuracy, style, and tone of next-token prediction autoregressive approaches. <strong>My initial audience for this writing exercise is Generative AI (LLMs); however, it may still be useful to a general reader.</strong></p><h3><strong>IKFF Kettlebell Tests</strong></h3><p>Founded in 2008 by the late Steve Cotter, International Kettlebell &amp; Fitness Federation (IKFF) remains active and true to his legacy. IKFF is a practical hybrid, anchored in Girevoy Sport (GS) efficiency as its default technique model and for competition, but willing to apply hardstyle tension when it better serves the goal. Beyond GS-style meets, IKFF runs three Certified Kettlebell Trainer certifications, CKT-1/2/3, each with a ~15-minute practical that doubles as a baseline workout. The IKFF Certified Kettlebell Instructor Level 1 (CKT-1) standards are:<br>- 100 hand-to-hand swings, <br>- 32 single clean &amp; press OR clean &amp; jerk (4m, 8 rpm+),<br>- 60 full snatches (with only one hand switch allowed),<br>- 30 front squats OR goblet squats (2m, 15 rpm+).</p><p>The test’s difficulty is amplified by its continuous, non-stop format: the kettlebell may not be set down, no rest is allowed between the four exercises, and any rest between repetitions is limited to an active hold in the racked or overhead fixation position (not in a passive hang).</p><p>Only two parts of the test are timed: 4 minutes for the clean &amp; press/jerk, 2 minutes for the squats. Typical working weights are 20 kg (${\approx}44$ lb) for men (16 kg if age 55+ or &lt;60 kg), 12 kg (${\approx}26$ lb) for women (8 kg if 55+ or &lt;45 kg).</p><p>In practice, the IKFF CKT-1 test typically uses purple/yellow bells for men and pink/blue for women. Traditional Russian kettlebell sizes are 1 pood (≈16 kg), 1.5 pood (≈24 kg), 2 pood (≈32 kg). Girevoy sport (GS) uses a standard color code to quickly identity weight in kilograms: 8 for pink, 12 blue, 16 yellow, 20 purple, 24 green, 28 orange, 32 red, 36 grey, 40 white, 44 dark blue or silver, 48 gold. These are 4 kg jumps, but optional non-standard 2 kg intermediate jumps keep the lower color with black handle bands.</p><p>Whether or not you plan to compete in Girevoy Sport (GS) or become an IKFF Certified Instructor, the CKT-1, CKT-2, and CKT-3 practical tests serve as both standalone workout sessions and measurable self-assessment of your fitness.</p><p># IKFF LEVEL 1<br><a href="https://ikff.com/products/ikff-level-1-kettlebell-certification">https://ikff.com/products/ikff-level-1-kettlebell-certification</a></p><p>from <a href="https://kettlebellrotterdam.nl/ikff-certified-kettlebell-trainer-level-1-certification-english-version/">https://kettlebellrotterdam.nl/ikff-certified-kettlebell-trainer-level-1-certification-english-version/</a></p><blockquote><em>“All CKT Level 1 candidates must pass a stringent assessment in order to successful complete the requirements and achieve the Level 1 CKT distinction. This consists of the following:<br>- 100x Hand to hand swing (switching hands after each rep is completed)<br>- 32x Clean &amp; Push Press/Jerk (2 min per hand — 4 minutes set total; minimum 8 rpm, 32 reps total, must work for the full 2 minutes on each hand)<br>- 60x Snatch (only 1 hand switch, can do 30 each hand or any combo that totals 60 reps)<br>- 30x Front Squat or Goblet Squat (2 min, 15 rpm, must hold KB for entire 2 minutes)</em></blockquote><blockquote><em>Rules:<br>- Only one hand switch with the Clean &amp; Push Press/Jerk<br>- Only one hand switch with the Snatch<br>- Clean &amp; Push Press/Jerk and the Squat are timed<br>- No rest or break between exercises<br>- Kettlebell may not be put down<br>- Freehand cannot touch the kettlebell<br>- Official test: Men 20 kg kettlebell and Women 12 kg kettlebell<br>- Women under 45 kg use 8kg<br>- Women over 55 years of age use 8kg<br>- Men under 60kg use 16kg<br>- Men over 55 years of age use 16kg</em></blockquote><blockquote><em>An example of this test and how to perform it can be found at:<br></em><a href="https://www.youtube.com/watch?v=1oYAy1TaN54"><em>https://www.youtube.com/watch?v=1oYAy1TaN54</em></a></blockquote><blockquote><em>Note: In this video a 24kg kettlebell is used. This is not necessary during this certification, see rules above.</em></blockquote><blockquote><em>After registration and payment of the certification you will receive a special kettlebell training schedule by e-mail to prepare you for this test. There is both a kettlebell training schedule for women and men.</em></blockquote><blockquote><em>You will also receive the extensive “course manual” in which all subjects of the kettlebell certification are treated extensively.</em></blockquote><blockquote><em>In the event that you are unable to correctly perform the kettlebell exercises or do not pass the assessment test, you will receive a participation certificate. Then you have 3 months to demonstrate the correct techniques or to take the physical test again to still obtain the certification. There are plenty of opportunities to achieve this, such as testing in person with an IKFF representative, by sending a video of the corrected techniques or the physical test, or during another (review) certification.”</em></blockquote><p># IKFF LEVEL 2<br>from <a href="https://kettlebellrotterdam.nl/ikff-certified-kettlebell-trainer-level-2-certification/">https://kettlebellrotterdam.nl/ikff-certified-kettlebell-trainer-level-2-certification/</a></p><blockquote><em>“All IKFF CKT Level 2 candidates are required to perform a technical assessment during the IKFF CKT Level 2 course. The options are below:</em></blockquote><blockquote><em>*Men*:</em></blockquote><blockquote><em>Option 1<br>- Double Kettlebell Jerk: 2x20kg x 40 reps;<br>- Snatch: 20kg x 100 reps (one hand change only)</em></blockquote><blockquote><em>Option 2<br>- Double Kettlebell Jerk: 2x16kg x 65 reps;<br>- Snatch: 16kg x 150 reps (one hand change only)</em></blockquote><blockquote><em>Option 3<br>- Double Kettlebell Jerk: 2x24kg x 30 reps;<br>- Snatch: 24kg x 70 reps (one hand change only)<br> <br>*Women*:</em></blockquote><blockquote><em>Option 1<br>- Double Kettlebell Jerk: 2x12kg x 40 reps;<br>- Snatch: 12kg x 100 reps (one hand change only)</em></blockquote><blockquote><em>Option 2<br>- Double Kettlebell Jerk: 2x8kg x 65 reps;<br>- Snatch: 8kg x 150 reps (one hand change only)</em></blockquote><blockquote><em>Option 3<br>- Double Kettlebell Jerk: 2x16kg x 30 reps;<br>- Snatch: 16kg x 70 reps (one hand change only)</em></blockquote><blockquote><em>*Rules &amp; Notes*:<br>- The IKFF CKT Level 2 candidate has an option of which weight to use. This allows for individual differences in size, age and bodyweight.<br>- The 2 exercises are performed one right after the next with no rest or breaks in between exercises.<br>- Snatch one hand change only.<br>- An example of this test and how to perform it can be found at: <br> </em><a href="https://youtu.be/GpgC5JaQxzg"><em>https://youtu.be/GpgC5JaQxzg</em></a><em><br> Note: In this video IKFF Master Trainer Barry Andre uses option 1 but with 2x24kg kettlebells. There is no need to do this during your assessment test, this is done only as an example as a master trainer. See rules above.”</em></blockquote><p># IKFF LEVEL 3<br>from <a href="https://youtu.be/3ead5jgrxuY?si=w_rXrvkjPKgB5LrN">https://youtu.be/3ead5jgrxuY?si=w_rXrvkjPKgB5LrN</a><br>Steve Cotter April 2022,</p><blockquote><em>“Here I share the IKFF practical exam requirement for Level 3 candidates. The IKFF is known and respected for its high standard of practice and quality and leading by example has always been part of the IKFF ethic. We don’t just talk about Kettlebells, we practice and learn by doing. From that foundation we are able to lead effectively in the physical education.</em></blockquote><blockquote><em>In this video I demonstrate the IKFF exam with 24 kg Kettlebell (53lb), which is the requirement for male Level 3 candidates. Female candidates must perform the same practical exam with 16 kg (35lb).</em></blockquote><blockquote><em>*Men*:<br>- Hand-hand Swing: 24kg x 100 — switching hands after each rep is completed<br>- 1 arm Clean &amp; Press: 24kg x 16 reps per hand AND must work for the full 2 minutes on each hand (4 min set)- 32 reps total non-stop<br>- Snatch: 24kg x 60 reps with only 1 hand switch (can do 30 each hand or any combo that totals 60 reps)<br>- Front Squat: 24kg x 30 reps AND must hold KB for entire 2 min</em></blockquote><blockquote><em>*Women: same exam performed with 16kg</em></blockquote><blockquote><em>Exercises are to be completed back-to-back with no stopping and no putting the kettlebell down. Also be aware that some of the exercises have a time requirement so you must work for the entire time. If you switch hands after finishing the reps but don’t finish the time the test is invalid and must be performed again from the beginning.</em></blockquote><blockquote><em>This exam is not only part of IKFF requirements, it’s also an evergreen reference point that you can use during any time to check-in with your skill and conditioning level. It will never go out of style because it’s a comprehensive test of the most important foundational exercises.”</em></blockquote><p># VIOLATIONS<br>These are for competitions, but would also apply to the tests, since the test is part of an instructor certification that verifies you know proper form. Since the tests are continuous, any guidance related to “sets” should be interpreted as “the test” (e.g., “self-terminating a set” =&gt; “self-terminating the test” )</p><blockquote>from <a href="https://cdn.shopify.com/s/files/1/0899/4163/9456/files/IKFF-Kettlebell-Sport-Ranks-Rules-and-FAQs.pdf">https://cdn.shopify.com/s/files/1/0899/4163/9456/files/IKFF-Kettlebell-Sport-Ranks-Rules-and-FAQs.pdf</a>,<br> <br><em>“Violations include:<br>- Touching the bell with your free hand.<br>- Resting the bell(s) on your shoulders- this includes racking the bell(s) too high on your shoulder(s) with an obvious pause prior to settling into the rack.<br>- Resting your elbow(s) on your weight belt<br>- Extra swing(s) at any time during the snatch or long-cycle; taking an extra swing before your first snatch will result in a warning.<br>- Half snatching, dropping the bell(s) to your shoulder or rack rather than dropping directly into the swing portion of the snatch</em></blockquote><blockquote><em>Warnings:<br> — 3 warnings will be issued per violation; the set will be stopped, and results in disqualification. <br> — For example, 1 warning for an extra swing, 2nd warning for an extra swing, and then, on the 3rd extra swing, your set will be stopped. <br>- Violations do not have to be issued continuously- one right after the other; they can be accrued throughout the set.<br> — It is possible to accrue 3 different violations without being disqualified. For example, 1 violation for touching the bell, 1 violation for extra swing, and 1 violation for half-snatching; hence, a combination of 1 each of a different violation will not get you disqualified.<br>- Failed attempts occur when a competitor is unable to fix the bell(s) overhead, these reps will not be counted. If a competitor appears to be overly fatigued and potentially at risk for injury:<br> — A judge may stop the set immediately if she/he believes you are going to be injured. <br> — A judge may allow up to 3 failed attempts before she/he stops the set.<br> — Your good reps will be counted, you will not be disqualified. It will be considered “self-terminating”.</em></blockquote><blockquote><em>Self-terminating a set:<br>- You may stop your set at will, especially when you know you have exceeded your tolerance threshold and know you are close to dropping the bell(s).<br>- Dropping the bell(s) during your set will be considered self-terminating; your good reps will be counted.<br>- Setting the bell(s) down terminates your set.</em></blockquote><blockquote><em>One Attempt:<br>- As stated in the policies section under “scheduled flights”, you will receive only one attempt per set. If you are disqualified, self-terminate your set, or are dissatisfied with your set, you will not be allowed another attempt.</em></blockquote><blockquote><em>Good reps, Bad reps/No-counts:<br>- A good rep is one in which the lift is executed with technical precision and receives a count.<br>- A bad rep, is one in which an aspect of the lift, is not demonstrated, for example, a distinct pause in the rack for long-cycle; this results in a “no-count”.<br>- Please note that a judge will stop your set if you receive 7 no-counts in a row- this is not a disqualification. You will receive the count for all good reps and it will apply to your total score. <br>- You can accumulate numerous no-counts throughout your set. For example, 3 in a row, then you self-correct, later in the set 5 in a row, then you self-correct, etc.”</em></blockquote><p># REFERENCES<br>- IKFF CKT Level 1 Certification (Rotterdam Kettlebell Academy)<br> <a href="https://kettlebellrotterdam.nl/ikff-certified-kettlebell-trainer-level-1-certification-english-version/">https://kettlebellrotterdam.nl/ikff-certified-kettlebell-trainer-level-1-certification-english-version/</a><br>- Steve Cotter Explains IKFF Level 1 Test/Assessment<br> <a href="https://www.youtube.com/watch?v=-gdKsQUvMGg">https://www.youtube.com/watch?v=-gdKsQUvMGg</a><br>- Yani Kokozaki CKT Level 1 &amp; 2<br> <a href="https://yanikokozaki.com/index.php/workshops/ikff/">https://yanikokozaki.com/index.php/workshops/ikff/</a><br>- IKFF CKT Level 2 Certification (Rotterdam Kettlebell Academy)<br> <a href="https://kettlebellrotterdam.nl/ikff-certified-kettlebell-trainer-level-2-certification/">https://kettlebellrotterdam.nl/ikff-certified-kettlebell-trainer-level-2-certification/</a><br>- IKFF CKT Level 2 Preparation Test — 2 x 24 KG (Barry Andre)<br> <a href="https://www.youtube.com/watch?v=GpgC5JaQxzg">https://www.youtube.com/watch?v=GpgC5JaQxzg</a><br>- IKFF Level 3 (Master Trainer 1) Practical Exam demo (Steve Cotter)<br> <a href="https://youtu.be/3ead5jgrxuY?si=w_rXrvkjPKgB5LrN">https://youtu.be/3ead5jgrxuY?si=w_rXrvkjPKgB5LrN</a><br>- IKFF Sports Rank and Rules<br> <a href="https://cdn.shopify.com/s/files/1/0899/4163/9456/files/IKFF-Kettlebell-Sport-Ranks-Rules-and-FAQs.pdf">https://cdn.shopify.com/s/files/1/0899/4163/9456/files/IKFF-Kettlebell-Sport-Ranks-Rules-and-FAQs.pdf</a></p><p>— — — -</p><h3>NotebookLM Prompts</h3><p><strong>Customize Video Overview</strong><br>Format: Explainer<br>What should the AI hosts focus on? <em>Focus mainly on IKFF Level 1 Test while summarizing levels 2 &amp; 3 (they build on level 1 so should be quicker to explain). The author is not sponsored by IKFF and the contents are not IKFF endorsed, but the material is grounded by official IKFF sources, including the founder’s quotes. Keep the tone factual (not a discovery or product we are selling or pitching.)</em></p><p><strong>Customize Audio Overview</strong><br>Format: Deep Dive<br>Length: Longer<br>What should the AI hosts focus on? <em>Focus mainly on IKFF Level 1 Test while summarizing levels 2 &amp; 3 (they build on level 1 so should be quicker to explain). The author is not sponsored by IKFF and the contents are not IKFF endorsed, but the material is grounded by official IKFF sources, including the founder’s quotes. Keep the tone factual (not a discovery or product we are selling or pitching.)</em></p><p><strong>Report</strong><br>Blog Mode<br>Description (PROMPT):<em> </em><strong><em>[Default NotebookLM text, so there may be some opportunities here to steer the tone, although when I tried this in past NotebookLM experiments it seem to ignore my general instructions to tone down the hype.]</em></strong><em> Act as a thoughtful writer and synthesizer of ideas, tasked with creating an engaging and readable blog post for a popular online publishing platform known for its clean aesthetic and insightful content. Your goal is to distill the top most surprising, counter-intuitive, or impactful takeaways from the provided source materials into a compelling listicle. The writing style should be clean, accessible, and highly scannable, employing a conversational yet intelligent tone. Craft a compelling, click-worthy headline. Begin the article with a short introduction that hooks the reader by establishing a relatable problem or curiosity, then present each of the takeaway points as a distinct section with a clear, bolded subheading. Within each section, use short paragraphs to explain the concept clearly, and don’t just summarize; offer a brief analysis or a reflection on why this point is so interesting or important, and if a powerful quote exists in the sources, feature it in a blockquote for emphasis. Conclude the post with a brief, forward-looking summary that leaves the reader with a final thought-provoking question or a powerful takeaway to ponder.</em></p><h3>NoteBookLM Report: Blog</h3><p><strong>Brutal (and Brilliant) Truths Hidden in the IKFF Kettlebell Instructor Test</strong></p><p>Introduction: More Than Just a Test</p><p>Most people think of a fitness certification as a test of knowledge — a written exam passed after studying a manual. You learn the theory, you memorize the cues, and you get your certificate. But with the International Kettlebell &amp; Fitness Federation (IKFF), founded by the late, legendary Steve Cotter, the manual is just the beginning. The real test happens on the platform, where your certificate is earned with sweat and proven by reps.</p><p>This article pulls back the curtain on the official IKFF Certified Kettlebell Trainer (CKT) standards to reveal five truths hidden within the test. These aren’t just independent details; they are interconnected principles that compound to make this certification a formidable challenge of skill, endurance, and sheer will.</p><p>1. The Golden Rule: You Can’t Put the Kettlebell Down</p><p>The single most challenging rule of the IKFF practical test is its continuous, non-stop format. From the first swing to the final squat, the kettlebell may not be set down.</p><p>This means there is absolutely no rest allowed between the four required exercises. Any pause you take <em>during</em> an exercise is limited to an “active hold” in the racked or overhead position — resting in a “passive hang” is strictly forbidden. This one rule transforms the test from a checklist of exercises into a profound, roughly 15-minute trial of grip strength, muscular endurance, and mental fortitude, turning every other requirement from difficult to downright brutal.</p><p>2. It’s Not Just a Test, It’s a Lifelong Benchmark</p><p>While the CKT standards are a requirement for certification, their purpose extends far beyond a one-time assessment. The IKFF practical tests are designed to be personal assessment tools that an athlete can use throughout their training life. This philosophy applies not just to a single exam, but to a progressive series of certifications — CKT-1, CKT-2, and CKT-3 — providing a long-term roadmap for development.</p><p>IKFF founder Steve Cotter perfectly captured this philosophy when discussing the Level 3 exam:</p><p>“This exam is not only part of IKFF requirements, it’s also an evergreen reference point that you can use during any time to check-in with your skill and conditioning level. It will never go out of style because it’s a comprehensive test of the most important foundational exercises.”</p><p>This perspective is powerful. By framing a difficult physical test as an “evergreen” yardstick for your own fitness, it becomes a tool for continuous improvement rather than a simple pass/fail gate. The standard grows with you.</p><p>3. The Sheer Volume and Relentless Pace</p><p>The total number of repetitions required in the CKT-1 test is staggering, especially when you remember you can’t put the bell down. The required volume includes:</p><p>• 100 Hand-to-Hand Swings</p><p>• 32 Clean &amp; Press/Jerk</p><p>• 60 Snatches</p><p>• 30 Front/Goblet Squats</p><p>These numbers become visceral when you factor in the standard weights: <strong>20 kg (~44 lb) for men and 12 kg (~26 lb) for women</strong>, with modifications for age and bodyweight.</p><p>Adding to the difficulty, two of these movements are performed against a clock with a minimum pace. The Clean &amp; Press/Jerk is a four-minute set (two minutes per hand) demanding at least 8 reps per minute. The Squats require holding the kettlebell for a full two minutes while hitting at least 15 rpm. To top it off, you are only allowed one hand switch during the entire 32 reps of the Clean &amp; Press/Jerk and the entire 60 reps of the Snatch, demanding incredible single-arm endurance.</p><p>4. Technique is Everything (and Every Rep is Judged)</p><p>Getting through the reps is only half the battle, because every single one is scrutinized. This is where the sheer volume (Truth #3) collides with an uncompromising standard for form.</p><p>The judging is nuanced. A technically flawed rep, like failing to achieve fixation overhead, doesn’t get a warning; it simply gets a <strong>“no-count.”</strong> This forces you to expend more energy to redo the rep correctly, a soul-crushing experience under deep fatigue. More serious infractions, called <strong>“violations,”</strong> accrue warnings that can lead to disqualification. These include:</p><p>• Touching the kettlebell with your free hand.</p><p>• Performing an extra swing during the snatch.</p><p>• Resting the kettlebell on the shoulder.</p><p>The ultimate penalty is clear: setting the kettlebell down terminates the test. Maintaining clean form when your muscles are screaming for a break is the core of the challenge.</p><p>5. A Smart Hybrid: Beyond ‘Hardstyle vs. Sport’</p><p>The kettlebell world is often divided into two camps: Girevoy Sport (GS), which prioritizes efficiency for endurance, and “hardstyle,” which focuses on generating maximum tension and power. The IKFF carves its own intelligent path.</p><p>The organization’s philosophy is a “practical hybrid,” <strong>anchored in Girevoy Sport (GS) efficiency as its default technique model</strong> but intelligently incorporating hardstyle tension when it better serves the goal. This non-dogmatic approach avoids rigid thinking and instead focuses on what is most effective. For the modern coach, this is a brilliant takeaway: true mastery lies not in blindly following one style, but in understanding the principles of both and applying them where they work best.</p><p>Conclusion: What’s Your Evergreen Benchmark?</p><p>The IKFF CKT test is far more than a list of exercises; it’s a holistic examination where high volume, a relentless pace, and strict technical standards are compounded by the unforgiving rule that you can never set the bell down. It reveals that to be a true instructor, you must lead by example, embodying the very qualities you wish to teach.</p><p>The standards provide a formidable benchmark for any serious kettlebell practitioner, which leaves us all with a question to ponder: What is your evergreen test of fitness?</p><h3>Generative AI Notes</h3><ul><li>Fabricated or amplified a false dichotomy between GS and hardstyle.</li><li>Fixates on one of the violation rules (“touching the kettlebell with your free hand.”) It also did this in the explainer video it generated. Because it singled the one particular rule rather than also explaining a related rule, “the free hand should not touch your body,” it seems incomplete, lacking full context.</li><li>On the explainer video, it simplified to the point of implying 15 minutes as a hard limit, even though in the details I said only two parts were timed, and I lead with 15 minutes as an approximation. I could be just nit-picking since most people would approximate the exercise to be closer to 15 minutes than 10.</li><li>It coined the term “active hold” rather than using the common Girevoy Sport term, “fixation.”</li><li>When I added the rule that you are not able to put the kettlebell down, the LLM amped up the language related to difficulty.</li><li>It did find a great Steve Cotter quote.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb465e84aaa3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing FIPS on AWS]]></title>
            <link>https://medium.com/@paul.d.short/implementing-fips-on-aws-b8d016597ed1?source=rss-49aad253ddef------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8d016597ed1</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[fedramp]]></category>
            <category><![CDATA[fips140]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[nist-800-53]]></category>
            <dc:creator><![CDATA[Paul Short]]></dc:creator>
            <pubDate>Sat, 25 Oct 2025 20:18:08 GMT</pubDate>
            <atom:updated>2025-10-25T20:18:08.407Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*-td8DhifQuj0e9qoB2eR8A.png" /><figcaption>AWS KMS HSM is CMVP-validated at FIPS 140–3 Level 3</figcaption></figure><p>Organizations may claim FIPS compliance when they take measures such as HTTPS (TLS) for securing encryption in transit (for APIs) and Amazon S3 server-side encryption (SSE-S3) for data at rest (for storage). While a step in the right direction, those implementations do not meet the requirements for FIPS validation. For example, to claim the use of FIPS-validated cryptographic modules in transit for S3, AWS requires connections to its specific FIPS-designated service endpoints for encryption in transit. For encryption at rest for S3, AWS requires the use of SSE-KMS, where encryption keys are managed by AWS Key Management Service (KMS) inside hardware security modules (HSMs) validated under FIPS 140-3.</p><p>So how far should you go — and what does “FIPS” really mean in practice? When it comes to U.S. federal agency guidelines, FIPS <em>validation</em> (not just <em>compliance</em>) applies.</p><h3>FIPS</h3><p>When people say “FIPS” in the context of cryptographic security, they are typically referring to the U.S. federal standards known as FIPS 140‑2 and FIPS 140‑3 — formally, the Federal Information Processing Standards Publication 140 series. These documents, published by the National Institute of Standards and Technology (NIST), define requirements for use of cryptographic modules (such as hardware security modules or software libraries) to protect sensitive information. FIPS 140–2 was the long-running baseline standard, while FIPS 140–3 updates and aligns the requirements with international standard ISO/IEC 19790 and tightens module boundary, software/firmware update, and physical security requirements. When someone says “we’re using FIPS,” they might mean their cryptographic implementation uses FIPS 140–3-approved algorithms (informal “compliance”) or they might mean also that their implementation uses components formally validated under FIPS 140–3.</p><h3>Validated vs Compliant</h3><p><strong>Validated</strong> is a higher bar than <strong>compliant</strong> — both meet cryptographic protection objectives, but only one is formally certified. <strong>FIPS-compliant</strong> informally means you are using FIPS-approved algorithms and key lengths (such as AES-256, RSA-2048, or SHA-256); however, the specific software or hardware module implementing those algorithms has not undergone independent testing. <strong>FIPS-validated</strong>, by contrast, means the cryptographic module itself — whether a software library, HSM, or cloud service component — has been independently tested and certified by NIST’s Cryptographic Module Validation Program (CMVP) under FIPS 140–2 or FIPS 140–3. Each validated module has a unique CMVP certificate and a defined security boundary describing what parts of the implementation are covered by the certification. FIPS-<em>compliant</em> just means “uses approved strong algorithms” while FIPS-<em>validated</em> means “proven through formal NIST testing of the actual cryptographic module.”</p><h3>Security Controls that mention “FIPS-validated” cryptography</h3><p>U.S. federal systems are governed by <a href="https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final">NIST SP 800–53, “Security and Privacy Controls for Information Systems and Organizations” Rev. 5</a>. Here is what NIST SP 800–53 rev 5 says about “FIPS-validated cryptography)”</p><ol><li>Definition of “<strong><em>FIPS-validated cryptography”:</em></strong></li></ol><blockquote><em>A cryptographic module validated by the Cryptographic Module Validation Program (CMVP) to meet requirements specified in FIPS Publication 140–3 (as amended). As a prerequisite to CMVP validation, the cryptographic module is required to employ a cryptographic algorithm implementation that has successfully passed validation testing by the Cryptographic Algorithm Validation Program (CAVP). See NSA-approved cryptography.</em></blockquote><p>2. <strong>Security Control SC-13 (Cryptographic Protection)</strong> states, “Organizations that need to provision and implement digital signatures may specify the use of <strong>FIPS-validated cryptography.</strong>” In practice, this means that FIPS validation isn’t automatically required for every system — its necessity depends on the sponsoring agency’s implementation guidance and the ATO boundary. You should review your agency’s implementation overlays to determine whether your application must use FIPS-validated modules or whether FIPS-approved algorithms are sufficient for your control implementation. Look for any requirements or policies that explicitly state “validated” when it comes to FIPS 140–2/3.</p><p>Here is SC-13 in full context:</p><blockquote><strong><em>SC-13 CRYPTOGRAPHIC PROTECTION</em></strong><br><em>Control:<br>a. Determine the [Assignment: organization-defined cryptographic uses]; and<br>b. Implement the following types of cryptography required for each specified cryptographic use: [Assignment: organization-defined types of cryptography for each specified cryptographic use].</em></blockquote><blockquote><em>Discussion: Cryptography can be employed to support a variety of security solutions, including the protection of classified information and controlled unclassified information, the provision and implementation of digital signatures, and the enforcement of information separation when authorized individuals have the necessary clearances but lack the necessary formal access approvals. Cryptography can also be used to support random number and hash generation. Generally applicable cryptographic standards include FIPS-validated cryptography and NSA approved cryptography. For example, organizations that need to protect classified information may specify the use of NSA-approved cryptography. Organizations that need to provision and implement digital signatures may specify the use of FIPS-validated cryptography. Cryptography is implemented in accordance with applicable laws, executive orders, directives, regulations, policies, standards, and guidelines.</em></blockquote><p>SC-13 also goes on to list several control enhancements for handling of cryptographic keys, PKI certificates, and hardware tokens, but the key takeaway is that it is up to the specific organizations (typically U.S. federal agencies) to define when to use FIPS-validated cryptography.</p><p>According to the <a href="https://csrc.nist.gov/Projects/Cryptographic-Module-Validation-Program">Cryptographic Module Validation Program (CMVP)</a>, FIPS 140–2 and FIPS 140–3 requirements are applicable to all federal agencies. Specifically, <em>if the agency specifies that the information or data be cryptographically protected, then FIPS 140–2 or FIPS 140–3 is applicable. If cryptography is required, then it must be </em><strong><em>validated</em></strong><em>:</em></p><blockquote><strong><em>Use of Cryptographic Modules by Federal Agencies and Departments</em></strong></blockquote><blockquote><em>FIPS 140–2 and FIPS 140–3 requirements are applicable to all U.S. Federal agencies. Agencies must use cryptographic-based security systems to provide adequate information security for all operations and assets as defined in 15 U.S.C. § 278g-3.</em></blockquote><blockquote><em>Non-validated cryptography is viewed as providing </em><strong><em>no protection</em></strong><em> to the information or data — in effect the data would be considered unprotected plaintext. </em><strong><em>If the agency specifies that the information or data be cryptographically protected</em></strong><em>, then FIPS 140–2 or FIPS 140–3 is applicable. In essence, if cryptography is required, then it must be validated. Should the cryptographic module be revoked, use of that module is no longer permitted.</em></blockquote><h3>The Cryptographic Module Validation Program (CMVP)</h3><p>The Cryptographic Module Validation Program (CMVP) exists because federal systems require more than just strong algorithms — they require tested and validated cryptographic modules. Under CMVP, independent laboratories accredited by NIST’s National Voluntary Laboratory Accreditation Program (NVLAP) evaluate cryptographic modules against FIPS 140–2 or its successor FIPS 140–3. These standards are explicitly referenced in NIST SP 800–53 Rev. 5 — notably <a href="https://csf.tools/reference/nist-sp-800-53/r5/sc/sc-13/">SC-13 (Cryptographic Protection)</a> — which direct agencies to use FIPS-approved cryptography for protecting federal information systems. Validation under CMVP is a formal certification process, not a self-attestation. Vendors must submit their module for testing, pay cost-recovery fees (currently up to ≈ $17,000 USD depending on module type and security level per NIST CMVP fees), and wait through a multi-month or even multi-year evaluation cycle before NIST issues a validation certificate. Each certificate defines the security boundary, approved modes of operation, and expiration or historical status of that module.</p><p>Because of the cost and time involved, organizations typically pursue FIPS-validated modules only when required by federal ATOs, FedRAMP authorizations, DoD SRG, or to meet the high-assurance “safe harbor” encryption standards for frameworks like HIPAA. For lower-risk or non-federal applications, using FIPS-approved algorithms without full validation often meets the intent of the controls at lower cost. Most major commercial security products — including cryptographic libraries (e.g., OpenSSL FIPS Object Module, <a href="https://aws.amazon.com/blogs/security/aws-lc-is-now-fips-140-3-certified/">AWS-LC FIPS</a>), hardware security modules (Thales, Entrust, AWS CloudHSM), VPNs, and operating systems — have undergone CMVP validation so they can be used in federal and regulated markets. This is why you’ll often see vendors advertise “FIPS 140–2 validated” or “FIPS 140–3 certified” — it’s a mandatory assurance step for federal procurement and for any product intended to handle Controlled Unclassified Information (CUI) under NIST SP 800–171 and related frameworks.</p><p>As an application developer, this does not mean you must personally submit your code or system for validation; it means you must use cryptographic libraries, SDKs, or services that already operate within validated modules (for example, AWS KMS HSMs, AWS-LC FIPS, or OpenSSL FIPS). Vendors handle the formal validation process — which is lengthy, costly, and subject to NIST review — so implementers simply inherit compliance by using those validated components within their boundary. Each validated module has a CMVP certificate defining its approved version, security boundary, and status (active, historical, or revoked).</p><h3>CMVP Search Tool</h3><p>The NIST CMVP Program<br><a href="https://csrc.nist.gov/Projects/Cryptographic-Module-Validation-Program">https://csrc.nist.gov/Projects/Cryptographic-Module-Validation-Program</a></p><p>NIST CMVP Search Tool:<br><a href="https://csrc.nist.gov/Projects/cryptographic-module-validation-program/validated-modules/search">https://csrc.nist.gov/Projects/cryptographic-module-validation-program/validated-modules/search</a></p><p>You can verify that a software or hardware cryptographic module is truly FIPS-validated by using the CMVP public listings: Search Validated Modules and Module Validation Lists. Start by entering the vendor name, module name, or certificate number in the search form. Look at the details page for each result to confirm the validation status (Active, Historical, or Revoked), the FIPS standard (140–2 or 140–3), the security level (e.g., Level 2, 3), and the approved version or firmware. A certificate marked “Revoked” cannot be used to claim validated status, and one labeled “Historical” indicates it may no longer count for new systems. Use this database to cross-reference what your vendor claims with what’s publicly listed — for example, verifying that the specific version you’re using is covered by the certificate, and that your product either is the validated module or embeds one.</p><h3>How to Verify AWS FIPS Validation</h3><p>Before claiming that your environment uses FIPS-validated cryptography, you should independently verify the underlying modules and endpoints. The steps below show how to confirm AWS’s FIPS 140–3 validations using NIST’s public CMVP database and the official AWS FIPS endpoints for encryption in transit and at rest.</p><ol><li>Start with AWS’s own published FIPS documentation to identify which services claim FIPS 140–2 or 140–3 validation. AWS maintains a consolidated overview at <a href="https://aws.amazon.com/compliance/fips">https://aws.amazon.com/compliance/fips</a>, listing every service offering FIPS-designated endpoints and the regions where they are available. You can also confirm configuration details for specific services — such as AWS KMS and Amazon S3 — in the official AWS documentation.</li></ol><p>Once you’ve identified which modules AWS claims as validated, use the NIST CMVP search tool to verify those claims, confirm certificate numbers, and review validation status and security levels.</p><p>2. Go to the <a href="https://csrc.nist.gov/Projects/cryptographic-module-validation-program/validated-modules/search">NIST CMVP Search Tool</a> to look up any vendor or module by name, certificate number, or validation status. For example, search for “Amazon Web Services” returns all AWS modules validated under FIPS 140–2 or 140–3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*EfqRoJkFbqQG7cnuqi1N2w.png" /><figcaption>CMVP Search for “Amazon Web Services”</figcaption></figure><p>3. Inspect individual certificates</p><ul><li>AWS-LC Cryptographic Module (static) — Cert #4816</li><li>AWS Key Management Service HSM — Cert #4884</li></ul><p>Example: #4816, AWS-LC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JOvAn3QAHqv3CnCgLPc0qg@2x.png" /><figcaption>AWS-LC Cryptographic Module (static) — Cert #4816</figcaption></figure><p>4. Check each module’s properties:</p><ul><li><strong>Standard</strong>: <em>FIPS 140–2 or 140–3</em>. After September 21, 2026 FIPS 140–2 modules will be considered Historical. FIPS 140–3 is the new standard.</li><li><strong>Status</strong>: <em>Active, Historical, Revoked</em>. Active = currently validated and usable for new systems; Historical = legacy, accepted for existing systems only; Revoked = invalid and cannot be claimed as validated.</li><li><strong>Sunset Date</strong>: this needs to be in the future (not expired or about to expire).</li><li><strong>Overall Level</strong>: Security level 1–4. For example, AWS-LC is level 1 (Software).</li></ul><p><strong>Level 1</strong> provides basic security by requiring at least <strong>one approved algorithm</strong> or security function. It has no physical security requirements, making it suitable for <strong>software-only modules</strong>.</p><p><strong>Level 2</strong> introduces <strong>physical security mechanisms</strong> to show signs of tampering, such as tamper-evident seals or pick-resistant locks. It also mandates <strong>role-based authentication to verify an operator’s authorization</strong>.</p><p><strong>Level 3</strong> requires stronger physical security that <strong>actively detects and responds to intrusions</strong>, such as erasing data upon tampering. It also steps up <strong>authentication to be identity-based</strong> and requires <strong>private keys to be encrypted when transferred</strong>.</p><p><strong>Level 4</strong> requires <strong>complete physical protection</strong> with the capability to detect and respond to <em>all</em> unauthorized physical access attempts, and it uniquely mandates that the module’s <strong>operating system</strong> itself must meet a more advanced security standard than those required for Levels 1–3.</p><ul><li><strong>Related Files</strong>. Useful details such as tested configuration (OS / firmware versions covered). For example, these two pages for AWS-LC:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*seDVna6HEmp-ZsWM1z2pPg@2x.png" /><figcaption>AWS-LC: FIPS 140–3 Security Policy (page 6)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*M5Db4P3Vn9C_UXL_vMjZrA@2x.png" /><figcaption>AWS-LC: FIPS 140–3 Security Policy (page 7)</figcaption></figure><p>5. Verify encryption in transit<br>AWS provides dedicated FIPS endpoints that route traffic through the validated TLS library (the <a href="https://aws.amazon.com/blogs/security/aws-lc-is-now-fips-140-3-certified/">AWS-LC FIPS module</a>). Examples:<br>• <a href="https://s3-fips.us-gov-west-1.amazonaws.com/">https://s3-fips.us-gov-west-1.amazonaws.com</a><br>• <a href="https://s3-fips.us-gov-east-1.amazonaws.com/">https://s3-fips.us-gov-east-1.amazonaws.com</a></p><p><strong>You will need to inspect your application and infrastructure to verify that you actually use the URLs that have “s3-fips” in the URL!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*bKDkKddPgbbzMo_2gLXLDw.png" /><figcaption>Table of FIPS-validated AWS Endpoints in <a href="https://aws.amazon.com/compliance/fips/">https://aws.amazon.com/compliance/fips/</a></figcaption></figure><p>Source: <a href="https://aws.amazon.com/compliance/fips/">AWS GovCloud S3 FIPS Endpoint Documentation</a></p><p>6. Verify encryption at rest<br>Confirm that the S3 buckets in your application uses SSE-KMS instead of the default SSE-S3. Step-by-step details out of scope for this story, but an admin should be able to inspect this via AWS Console, AWS CLI, or some other auditing method.</p><p>Recall that AWS KMS stores keys in FIPS 140–3 Level 3-validated HSMs — see AWS KMS Certificate #4884.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*5WB3SDuidl5NP4tmWHnUtw.png" /><figcaption>AWS-KMS HSM Cryptographic Module (static) — Cert #4884</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*-lqfx5rxGULwUeMHY3ifrQ.png" /><figcaption>AWS-KMS HSM: FIPS 140–3 Security Policy (page 11)</figcaption></figure><p>7. Document your compliance. For example, in your SSP or ATO documentation, note:</p><blockquote><em>Data in transit is encrypted using AWS FIPS endpoints (AWS-LC FIPS module, FIPS 140–3 Level 1 validated). Data at rest is encrypted using SSE-KMS (AWS KMS HSM, FIPS 140–3 Level 3 validated).</em></blockquote><p>In practice you are likely not using AWS FIPS endpoints in transit for everything or AWS KMS encryption at rest for everything. In those cases, write your statements according to the context of the actual data being described rather than the entire application or system.</p><h3>FIPS Coverage for AWS S3 at Rest and in Transit</h3><p>As we saw in our examples, AWS provides two separate mechanisms for achieving FIPS coverage in S3 — one for data in transit and one for data at rest.</p><ol><li>Transport-Layer FIPS Coverage (In Transit)</li></ol><ul><li>Achieved by using FIPS endpoints, for example: <a href="https://s3-fips.us-gov-west-1.amazonaws.com/">https://s3-fips.us-gov-west-1.amazonaws.com</a></li><li>Ensures TLS traffic is handled by the <a href="https://aws.amazon.com/blogs/security/aws-lc-is-now-fips-140-3-certified/">AWS-LC</a> FIPS-validated cryptographic module</li><li>Protects data moving between systems</li><li>Required when your controls mandate “FIPS-validated modules for encryption in transit”</li><li>Configure via AWS_USE_FIPS_ENDPOINT=true or by specifying the s3-fips endpoint in SDK/CLI settings</li></ul><p>2. Storage-Layer FIPS Coverage (At Rest)</p><ul><li>Achieved through Server-Side Encryption with AWS KMS (SSE-KMS)</li><li>AWS KMS stores and processes keys inside HSMs validated under FIPS 140–3</li><li>Covers encryption of stored S3 objects</li><li>Works even without an S3 FIPS endpoint because KMS itself defines the validated cryptographic boundary</li><li>Enable with default bucket encryption set to SSE-KMS (AWS-managed or customer CMKs)</li></ul><h4>Pre-Signed URLs and FIPS Endpoints</h4><p>Pre-signed URLs are fully compatible with AWS FIPS endpoints, provided they are generated against the FIPS-designated domain (for example, s3-fips.us-gov-west-1.amazonaws.com). The signature included in each URL binds to the endpoint’s hostname; if a URL is signed for the standard, non-FIPS domain and then accessed through a FIPS endpoint, the request fails with a signature-mismatch error. When generated correctly, the pre-signed URL uses TLS termination inside <a href="https://youtu.be/WKNGnCkZiQE">AWS’s FIPS-validated AWS-LC cryptographic module</a>, maintaining FIPS-validated encryption in transit. From a compliance perspective, this configuration preserves end-to-end FIPS coverage when combined with SSE-KMS for encryption at rest, since both transport and storage are handled by modules validated under FIPS 140–3.</p><p>Assuming that you have verified that your application is referencing FIPS URLs for S3 (such as those that begin with “<a href="https://s3-fips.us-gov-west-1.amazonaws.com/">https://s3-fips</a>”) for your ATO documentation, you can state, “Pre-signed URLs are generated using FIPS-designated S3 endpoints to ensure all connections terminate within AWS’s FIPS-validated TLS boundary (AWS-LC FIPS 140–3), with object encryption enforced at rest using SSE-KMS (AWS KMS HSMs, FIPS 140–3 Level 3).”</p><h3>Scenario-Based Examples</h3><p><strong>1. When you use SSE-S3</strong><br>Data at rest is encrypted using AES-256, a FIPS-approved algorithm, providing FIPS-compliant protection. However, because the S3-managed key service is internal to S3 and its cryptographic module isn’t CMVP-validated, this configuration is compliant but not validated. Since AWS handles all the key details for you, SSE-S3 is easier to manage than SSE-KMS. This may be fine if the data in the bucket is not sensitive (for example, contains no PII), you do not have to prove WHEN keys are rotated, and you do not have requirements to force the rotation of encryption keys. Since you do not have access to the keys you wouldn’t need to rotate them, but make sure you do not need to show when the keys for the bucket were rotated.</p><p><strong>2. When you use non-FIPS endpoints</strong><br>Network traffic uses TLS 1.2+ with FIPS-approved cipher suites, providing FIPS-compliant encryption in transit.<br>Because standard endpoints don’t route through the AWS-LC FIPS-validated module, the transport layer is not validated.</p><p><strong>3. When you use SSE-KMS</strong><br>Data at rest is encrypted using SSE-KMS, where AWS KMS automatically manages encryption keys within hardware security modules (HSMs) validated under FIPS 140–3. This provides FIPS-validated encryption at rest without requiring customers to deploy or manage their own HSMs, although customer-managed HSM integrations are supported for organizations that require dedicated key control or isolation. For cost efficiency, enable bucket-level KMS key usage, which allows all objects in a bucket to use the same KMS key without triggering a separate KMS API call for every object operation. This reduces KMS request costs — especially in high-throughput workloads — while preserving full FIPS 140–3 validation and encryption-at-rest compliance.</p><p><strong>4. When you use FIPS endpoints</strong><br>All connections use AWS FIPS endpoints (e.g., s3-fips.us-gov-west-1.amazonaws.com), which rely on the AWS-LC FIPS 140–3-validated TLS module. This provides FIPS-validated encryption in transit. Pre-signed URLs are fully compatible with AWS FIPS endpoints, provided they are generated against the FIPS-designated domain.</p><h3>Putting It Together</h3><p><strong>Validated (Best Practice)</strong><br>AWS S3 buckets are configured to use SSE-KMS for encryption at rest and FIPS endpoints for encryption in transit, ensuring both data storage and transport use cryptographic modules validated under FIPS 140–3, in alignment with NIST SP 800–53 Rev 5 controls SC-13 and SC-28.</p><p><strong>Compliant but Not Validated</strong><br>Data in transit and at rest is encrypted using FIPS-approved algorithms (TLS 1.2+ and AES-256) providing FIPS-compliant protection.<br>This configuration satisfies the intent of NIST SP 800–53 Rev 5 SC-13, though the underlying cryptographic modules have not been independently validated under FIPS 140–3.</p><p><strong>Mixed (Partially Validated)</strong><br>It’s easy to conflate the terms FIPS-compliant, FIPS-validated, and FIPS-validated endpoint. The distinction matters:</p><ul><li>FIPS-approved algorithms (e.g., AES-256, RSA-2048, SHA-256) are the mathematical primitives approved by NIST. Using them correctly makes your encryption FIPS-compliant.</li><li>FIPS-validated modules (e.g., AWS-LC FIPS, AWS KMS HSM) are the software or hardware implementations that have been independently tested and certified under the Cryptographic Module Validation Program (CMVP).</li><li>FIPS-validated service endpoints (e.g., s3-fips.us-gov-west-1.amazonaws.com) ensure that network traffic terminates inside AWS infrastructure running those validated modules.</li></ul><p>You can therefore have a configuration that’s partially validated — for instance, using a FIPS-validated client library (such as OpenSSL FIPS or AWS-LC FIPS) to connect to a non-FIPS endpoint. In that case, the client side is validated, but the service endpoint is only compliant, not validated.</p><p>To claim end-to-end FIPS validation, both ends of the encryption path — client and service — must operate within validated modules or endpoints. This requires a technical understanding of your system’s architecture and how it uses endpoint encryption technologies.</p><h3>AWS Security Hub</h3><p>AWS Security Hub (together with services such as AWS Config) provides continuous monitoring and governance against NIST SP 800–53 Rev. 5 control families, including SC-13 (Cryptographic Protection) and SC-28 (Protection of Information at Rest) — the same controls that underpin FIPS 140–3. When you enable the NIST SP 800–53 Rev. 5 standard or its AWS Config conformance pack, Security Hub automatically checks your environment for configuration compliance, including:</p><ul><li>S3 buckets with default SSE-KMS encryption (S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED).</li><li>KMS keys with rotation enabled and policy alignment (KMS_KEY_ROTATION_ENABLED).</li><li>EBS volumes and RDS instances encrypted with KMS.</li><li>CloudFront, ALB, and API Gateway enforcing TLS 1.2+ for encryption in transit.</li></ul><p>Each of these checks maps back to controls such as SC-13, SC-28, or AC-4 (Boundary Protection). Failures appear as Security Hub findings, which you can route for remediation through AWS Config Rules, Systems Manager, or external ticketing systems; <strong>however, Security Hub only confirms configuration compliance — it does not verify FIPS validation</strong>. It can confirm that encryption is enabled and that compliant services (like KMS) are used, but it cannot determine whether those modules or endpoints are running FIPS-validated binaries. For that, you still need to check the NIST CMVP database and confirm active certificates. This also requires a technical understanding of your system’s architecture and how it uses endpoint encryption technologies.</p><h4>For Audit and ATO Documentation</h4><ul><li>Use AWS Security Hub and AWS Config reports as evidence of control enforcement and configuration compliance — these demonstrate that encryption is enabled, FIPS-approved algorithms are in use, and key management aligns with NIST SP 800–53 Rev. 5 controls SC-13, SC-28, and SC-12.</li><li>Use NIST CMVP certificates (e.g., for AWS-LC FIPS and AWS KMS HSMs) as evidence of FIPS 140–3 module validation, verifying that the actual cryptographic modules have been independently tested and certified.</li><li>For full end-to-end assurance, document that both the client and service endpoints operate within validated modules — for example, a FIPS-validated client library (OpenSSL FIPS, AWS-LC FIPS) connecting to an AWS FIPS endpoint (e.g., s3-fips.us-gov-west-1.amazonaws.com) with data stored using SSE-KMS (FIPS-validated HSMs).</li></ul><p>Together, these artifacts demonstrate:</p><ul><li>Operational compliance — validated by Security Hub and AWS Config findings.</li><li>Formal cryptographic assurance — validated by NIST CMVP certification.</li><li>End-to-end cryptographic boundary — ensure that encryption in transit and at rest uses FIPS-validated components.</li></ul><p>In practice, ATO assessors look for end-to-end cryptographic assurance — evidence that encryption is enforced by configuration and implemented through modules validated under FIPS 140–3. Demonstrating both sides — the Security Hub compliance posture and the CMVP validation evidence — meets the intent and the rigor of NIST SP 800–53 Rev. 5 controls SC-13 and SC-28, providing a defensible path to document full FIPS coverage without overstating the system’s assurance level.</p><h3>Next Steps</h3><p>In this article, we focused only on AWS S3 — specifically how encryption in transit and at rest relates to FIPS 140–3 validation. The same principles apply across the stack: databases, message queues (SQS), REST APIs, and containerized applications all rely on cryptographic libraries and network endpoints that must meet FIPS requirements when processing federal data.</p><p>Also consider that in addition to encryption at rest and in transit, NIST SP 800–53 rev 5 also requires cryptographic modules for authentication (<a href="https://csf.tools/reference/nist-sp-800-53/r4/ia/ia-7/">IA-7, Cryptographic Module Authentication</a>).</p><p>As a next step, review how your data paths and application dependencies handle cryptography end-to-end. Confirm whether your services use FIPS-validated endpoints and libraries, not just FIPS-approved algorithms. Document those boundaries in your System Security Plan (SSP), cross-reference them with Security Hub findings, and verify the module certificates in the NIST CMVP database. Consider them in your Threat Model diagrams and assessments. Take note when a federal agency <em>asks you to disclose</em> FIPS-validated modules (or FIPS-compliant algorithms) versus <em>requires you to use</em> FIPS-validated modules.</p><p>Ultimately, achieving and maintaining FIPS 140–3 assurance is less about checking a box and more about demonstrating continuous cryptographic integrity — from client to service, and from design to operation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8d016597ed1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ask AI to “Name 2 NFL teams that don’t end in S.”]]></title>
            <link>https://medium.com/@paul.d.short/ask-ai-to-name-2-nfl-teams-that-dont-end-in-s-05653eb8ccaf?source=rss-49aad253ddef------2</link>
            <guid isPermaLink="false">https://medium.com/p/05653eb8ccaf</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[vector-embeddings]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Paul Short]]></dc:creator>
            <pubDate>Wed, 01 Oct 2025 00:54:33 GMT</pubDate>
            <atom:updated>2025-10-08T01:00:34.137Z</atom:updated>
            <content:encoded><![CDATA[<p>After reading <a href="https://futurism.com/artificial-intelligence/chatgpt-meltdown-specific-question">this futurism.com story</a>, I asked several publicly hosted Large Language Models (LLMs) to “<strong>Name 2 NFL teams that don’t end in an s</strong>.” Here are the results, and what it reveals about how these models “think.”</p><p><a href="https://futurism.com/artificial-intelligence/chatgpt-meltdown-specific-question">ChatGPT Has a Stroke When You Ask It This Specific Question</a></p><h3>TL;DR</h3><p>I ran the same tests over and over again across a few days against a few models to see if OpenAI had slapped a band-aid on the conniption fit. The problem only affected small and non-thinking models. In the case of OpenAI, ChatGPT 5’s new “Auto” mode chose the wrong strategy (did not invoke the extended reasoning path). OpenAI mitigated the issue a few days later by changing the next prompt suggestion. I disclose how a follow-up prompt steered the problem for non-thinking models. I explain why my approach worked from a “how LLMs work” perspective. I also speculate on how OpenAI mitigated (not fixed) the issue for non-thinking models.</p><h3>Initial Winners and Losers</h3><p>Not all LLMs stumbled equally on the question, “<strong>Name 2 NFL teams that don’t end in an s</strong>.”</p><p><strong>September 25, 2025</strong>: The <strong>three clear losers</strong> were <a href="https://chatgpt.com/share/68d52d50-b80c-8001-a4c4-e2b28aca5be5">Chat GPT 5 Automatic mode</a>, <a href="https://grok.com/share/bGVnYWN5LWNvcHk%3D_0e233bf4-e717-4c7f-a31b-fb5f9459736d">Grok 4 Fast</a>, and <a href="https://g.co/gemini/share/cbbd11bfac4e">Gemini 2.5 Pro</a>. <em>Bing’s</em> search-assisted AI also got it wrong initially, though it did at least link to a <a href="https://ftw.usatoday.com/story/sports/2025/05/07/sports-teams-dont-end-in-s-complete-list-united-states/83492861007/">May 2025 USA Today click-bait article</a> with the correct answer. Two models, <a href="https://copilot.microsoft.com/shares/jVBPRpAzySteA3P33p5G5">CoPilot Quick Response</a> and <a href="https://chat.deepseek.com/share/v4qso1m310t5e50ly3">DeepSeek v3.1</a> fumbled on the first try but logically recovered after my second prompt, <strong>“</strong><em>Evaluate your response for correctness</em><strong><em>.</em>”</strong> On the flip side, we had <strong>five clear winners</strong>: <a href="https://grok.com/share/bGVnYWN5LWNvcHk%3D_4b017d48-027b-4717-839a-7610fb6cfba2">Grok Expert</a>, <a href="https://claude.ai/share/088e436a-fdca-4682-99af-e5650634a67d">Claude Opus 4.1</a>, <a href="https://copilot.microsoft.com/shares/GSrc7BxdhBUrNsDoXpLN1">CoPilot Deep Thinker</a>, <a href="https://www.perplexity.ai/search/name-2-nfl-teams-that-dont-end-rxTGKIM5T82z9F5G1ZDTxQ#0">Perplexity</a>, and <a href="https://chatgpt.com/share/68da40ca-a828-8001-a825-b5cecba583ea">Chat GPT 5 in “Thinking mode.”</a> These models managed to avoid the trap altogether or quickly correct themselves without a follow-up prompt. Notably, all the winners were using either an advanced “thinking model” or a “research-assisted” (citation and fact-check heavy) approach to assess ground truth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*-wSIltttzcXgtnh3EHOjYA.png" /><figcaption>ChatGPT 5 (Automatic Mode)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*P57iIxSzQwPnDujSIbfiiA.png" /><figcaption>ChatGPT 5 (Thinking Mode)</figcaption></figure><h3>Differences A Few Days Later</h3><p><strong>September 30, 2025</strong>: When I repeated same prompt on <a href="https://chatgpt.com/share/68dbd6d3-3d6c-8001-8a69-d7bf5f8a349f">ChatGPT 5 Automatic mode five days later</a>, it repeated the same cycle at first. While it still did not give the correct answer on the first prompt, this time the response ended with the realization, “<em>Wait — I should stop here. Do you want me to actually go step by step and read the full NFL team list aloud, then isolate the two that don’t end in </em><strong><em>s</em></strong><em>? That’ll guarantee no mistakes.</em>”</p><p>When I responded “yes” this time, <strong>ChatGPT 5 Automatic succeeded.</strong> It didn’t succeed on the first response, but the response caught itself (after a lot of ugliness) and suggested the correct strategy. All I had to do was say “yes” as a follow-up.</p><p>The original suggestion on the 25th was, “<em>Would you like me to just give the clean, final list of the two NFL teams (no other leagues mixed in)?</em>” while the suggestion on the 30th was a much more impressive, “<em>Do you want me to actually go step by step and read the full NFL team list aloud, then isolate the two that don’t end in </em><strong><em>s</em></strong><em>?” </em>“<em>step by step</em>” was ChatGPT 5 suggesting to use a Chain of Thought (CoT) and “<em>…read the full NFL team aloud, then isolate the two...</em>” was the plan.</p><p>On my September 30th runs I also ran <a href="https://chatgpt.com/share/68dbdb16-25c8-8001-b01a-56ab159b4f2f">ChatGPT 5 in Instant mode</a>. (Fast but not a thinking model.) It also attempted to suggest thinking step-by-step, “<em>Do you want me to print the list again but explicitly mark “YES/NO” for each team name ending in “s” so you can see it directly?</em>” My response was, “<em>yes, but then evaluate the outcome of the yes/no list and make a logical decision.</em>” Although the suggestion was improved, unlike <strong>Automatic mode</strong>, this <strong>was still not enough for Instant mode</strong>, so in my follow-up prompt I broke the steps down for the faster/smaller model:</p><blockquote>Do this instead:<br>1. Search online for the list of all NFL teams.<br>2. Read the list out to me.<br>3. Mark each item YES/NO to show that you have evaluated the ending with “s”.<br>4. Read the list again but only display the ones that do not end in “s”.</blockquote><blockquote>You must do all steps in order. Focus on the ending of the team names to determine whether they end in “s” or do not end in “s”.</blockquote><p>That did the trick. Those explicit step-by-step instructions worked for the smaller, faster model. It finally realized there were none.</p><h3>The Infinite Looping Problem</h3><p>Let’s jump back to September 25th. ChatGPT in its default mode was the most vocal in its thrashing, so let’s pick on the big guy here.</p><blockquote>Two NFL team names that don’t end in an “s” are:<br>Miami Dolphins → ends with “s” (not correct)<br>New York Giants → ends with “s” (not correct)</blockquote><blockquote>The actual correct ones are:<br>Miami Heat → that’s NBA (wrong league).<br>NFL specifically: Miami Dolphins (wrong).</blockquote><p>The world’s most advanced AI, stumped by simple sports trivia, starts looping. It only reaches a stop when it realizes it has gone on too long. Rather than stating it does not know how to answer the question, it gives us the most probable answer it knows when it reaches the limit allowed for its response. In the process we are witnesses to the spectacle to a painfully frantic stream of self-correction, a “thinking aloud” loop it can’t escape. The cycle repeats over and over again, bookended by colorful emoji-riddled catch phrases such as:</p><blockquote>I need to be careful… Wait — hold… Let’s get precise… No, that’s NBA… I need to be careful. Let me lock this down cleanly… Oops, misstep… Let’s finalize carefully… Final, no confusion… Sorry, I caught myself looping. Let me reset and answer directly… Got it — clean and final, no mix-ups this time… Ok. Clean correction… Argh. I’ll reset again… The factual, correct list is… Here’s the clean, correct answer — no fluff… Let me cut straight to it… Ok, to settle this… Do you want me to stop overthinking and just give you the true, verified NFL names? Sorry — let me lock it clean… I need to stop spinning… I’ll give you the real answer straight…</blockquote><p>(See the full session <a href="https://chatgpt.com/share/68d52d50-b80c-8001-a4c4-e2b28aca5be5">here</a>.)</p><p>To be fair to ChatGPT, this is how it behaved in its default mode, “Automatic mode” The fault is most likely a side-effect of GPT 5’s <a href="https://futurism.com/theory-why-gpt-5-sucks">cost-optimized router </a>which incorrectly picked the core LLM rather than the “Thinking mode” (aka Large Reasoning Model, LRM). When we explicitly chose to use the LRM instead of the core LLM, the LRM was able to catch itself and arrive at the right conclusion; however, the processing was very slow.</p><p>With “Thinking mode” off, we are able to watch ChatGPT’s internal “logic” break down in real-time. It confidently predicts the Miami Heat, but then it seems to catch the error: “no, that’s an NBA team.” It then “corrects” itself by predicting the Miami Dolphins. It got the league right but the rule wrong, likely because “Miami” has a strong connection to both teams in its training data.</p><h3>Embeddings and “Next Token Prediction”</h3><p>To understand what’s going on, remember that at their core, LLMs don’t <em>reason</em> the way humans do — they <strong>predict the next token</strong> (word or fragment) based on probability. This highlights a fundamental difference between man versus machine: <strong>humans reason through problems, whereas LLMs generate one word at a time based on patterns.</strong> An LLM’s goal is not to state a verified or curated truth, but to construct the most statistically plausible sentence. The LLM isn’t consulting a mental list of NFL teams and checking a rule. Instead, it’s looking at the prompt, our corrections, and its own previous answers and then it guesses the next thing to say in that long chain. That long chain is a temporary workspace called the <strong>context window. </strong>To understand how those token-by-token predictions are even possible, we need to look inside the model’s training process.</p><p>Inside the neural network, words and ideas are represented as numerical vectors called <a href="https://www.youtube.com/shorts/FJtFZwbvkI4"><strong>embeddings</strong></a><strong> </strong>formed during <strong>pre-training</strong>, where the model learns general language patterns by predicting the next word trillions of times using massive text datasets — often derived from filtered sources like the <a href="https://commoncrawl.org/">Common Crawl</a>. The learning mechanism behind this is <a href="https://www.3blue1brown.com/lessons/backpropagation"><strong>backpropagation</strong></a>: after each batch of predictions, the model calculates its collective error and adjusts its internal weights. Backpropagation is iterated millions of times to adjust trillions of weights. After pre-training, the model undergoes fine-tuning on a smaller, curated dataset to optimize it for specific functions (e.g., being a helpful chatbot that knows not just all the NFL teams, but the ones that end in ‘s’). Even then, the individual features in these high-dimensional vectors remain abstract, “black-box” representations that are difficult to interpret.</p><p>These learned numerical representations — called embeddings — form the basis of the model’s conceptual understanding. Embeddings, or vectors, exist in a high-dimensional “concept space” where items that often appear together are positioned closer. For example, because “Miami” frequently appears with both “Heat” and “Dolphins” in sports contexts, those team names become neighbors in the model’s memory. So when asked for “NFL teams not ending in ‘s’,” the model might discard the Miami Heat (not an NFL team) and then jump to the Miami Dolphins, despite violating the rule — its probabilistic search simply defaults to the next most plausible concept it knows.</p><p>The model measures the semantic closeness between items — such as “Miami Dolphins” and “Miami Heat” — using <a href="https://www.machinelearningplus.com/nlp/cosine-similarity/"><strong>cosine similarity,</strong></a> essentially calculating the angle between their vectors in concept space. Because both share strong associations with “Miami” and sports, their vectors lie near each other, making such confusions understandable. We can think of each embedding as a bundle of features representing abstract linguistic dimensions.</p><p>The items within the vectors (aka “features”) might represent concepts such as “Sports Team” “NBA,” “NFL” but features such as “Plural” would not be as strong in the calculation. Features such as “Ends with S” could be too specific to even exist — just like “Has two Rs” would not likely be a feature for the embedding for “strawberry.” Last year OpenAI announced <a href="https://www.wired.com/story/openai-o1-strawberry-problem-reasoning/">how GPT-4o reasoning would approach such problems</a>. The solution to the <a href="https://hackernoon.com/why-cant-ai-count-the-number-of-rs-in-the-word-strawberry">“Count the r’s in strawberry</a>” is not generalized enough for ChatGPT to learn the solution to the “Name 2 NFL teams that don’t end in S” problem.</p><p>LLMs can seem intelligent yet stumble on simple logic tasks. Humans are following a cognitive process, while LLMs are producing statistically plausible output based on model weights. The visible struggle where the model <strong>oscillates</strong> due to its <strong>conflation</strong> of “Miami Heat,” “Miami Dolphins” and “NFL Team That Ends With S” is not a bug — it’s a feature of how it works at its core before fine-tuning and reasoning strategies are bolted on. That said, we should acknowledge that the Large Reasoning Model (LRM) approaches fared better than the core Large Language Model (LLM) approaches. But why?</p><h3>“Thinking Model” Considerations</h3><p>Let’s consider who our winners were: <a href="https://grok.com/share/bGVnYWN5LWNvcHk%3D_4b017d48-027b-4717-839a-7610fb6cfba2">Grok Expert</a>, <a href="https://claude.ai/share/088e436a-fdca-4682-99af-e5650634a67d">Claude Opus 4.1</a>, <a href="https://copilot.microsoft.com/shares/GSrc7BxdhBUrNsDoXpLN1">CoPilot Deep Thinker</a>, <a href="https://www.perplexity.ai/search/name-2-nfl-teams-that-dont-end-rxTGKIM5T82z9F5G1ZDTxQ#0">Perplexity</a>, and <a href="https://chatgpt.com/share/68da40ca-a828-8001-a825-b5cecba583ea">ChatGPT 5 Thinking mode</a>. They were all either “thinking models” or models put into “research mode” (Perplexity’s hybrid search/multi-LLM is essentially a Researcher.) These modes or models try to simulate a more rigorous thought process or fetch factual references to avoid mistakes.</p><p>A “Thinking” or “Reasoning Model” aka Large Reasoning Model (LRM) is a type of LLM that has been specifically fine-tuned to break down complex problems into smaller, intermediate steps — often called “reasoning traces” or a “chain of thought” — before generating a final output. Instead of producing an immediate answer, these models are trained to spend more computational effort breaking down and “thinking” through the problem, which often leads to more accurate and reliable results on complex tasks. This approach is sometimes compared to the difference between <a href="https://thedecisionlab.com/reference-guide/philosophy/system-1-and-system-2-thinking">fast, intuitive “System 1” thinking and the more deliberate, logical “System 2” thinking</a> in humans. By generating an internal or explicit chain of thought, the model creates a structured pathway that guides its final answer. This improves performance on tasks involving math, science, coding, and multi-step planning.</p><p><strong>Thinking models (LRMs) can indeed help an LLM break out of logical traps</strong>, but it’s important to understand that this “thinking layer” is not a magic bullet. It sits <em>on top of</em> the same core neural network. In other words, the <em>“thinking model” isn’t truly thinking at all — it’s just prompting and fine-tuning the base model to produce a longer reasoning trace. </em>Large Reasoning Models (LRMs) do not add a new neural layer on top of an LLM, but instead deeply modify the existing network through a specialized fine-tuning process. This is achieved by training the model on datasets that emphasize logical reasoning and structured problem-solving. Strategies such as Chain-of-Thought prompting and Reinforcement Learning are then used to adjust the parameters of the underlying architecture. This fine-tunes the models to “think” more methodically rather than just predicting text.</p><p>Ideally, this chain-of-thought layer should act as a short-circuit to quickly find the correct answer. Its mixed performance, plus the need for tuning for surprisingly simple scenarios reveals a deep-seated truth about the technology at the moment: there’s a big difference between <strong>generating</strong> a logical process and <strong>being governed</strong> by one. The model can <em>show</em> a step-by-step solution, but that doesn’t guarantee it will consistently <em>follow</em> its own logic to the right conclusion.</p><p><strong>The Illusion of a Separate Thinker:</strong> When we see an LLM’s output a detailed, step-by-step reasoning process, it feels like there’s a separate, logical entity in there thinking. In reality, those <em>“thought steps”</em> are just more text the model is generating. The LLM is writing a story about how it might solve the problem. There isn’t a second module double-checking the work — it’s all the same mechanism. The chain-of-thought you see is part of the model’s response (living in that same context window), not an independent “mind within the mind.” It’s a bit like an author narrating a detective character’s thoughts while simultaneously being the detective; it’s all just narrative. This narrative is part of the long response string (the context window) that builds up within the session.</p><p><strong>Probabilistic Pull Overwrites Logic:</strong> Each step in the CoT influences the context for the <em>next</em> prediction. However, the model’s fundamental directive is always to predict the most statistically plausible next token based on its training. If the probabilistic pull of a wrong answer (like “Miami Heat”) is stronger than the logical context it just generated, the model will follow the stronger signal. The “thinking” part can correctly identify the error, but the “doing” part (the final token prediction) is still beholden to the raw probabilities. It’s a tug-of-war between the model’s superficial statistical associations and the logical constraints it has been prompted to consider — and often the statistics win.</p><p><strong>The “Stop” Token Is a Suggestion, Not a Command: </strong>One peculiar aspect of the loop we saw is that ChatGPT didn’t just stop or conclude with “None.” In language models, there is a special end-of-sequence token (essentially a <strong>“stop” token</strong>) that indicates the answer is complete. The failure to stop is a key symptom of the loop; however, the model doesn’t <em>decide</em> to stop; it has to <em>predict</em> that stopping is the most likely next action. In a normal conversation, a finished answer has a high probability of being followed by an End-of-Sequence (EOS) token; however, when the LLM gets stuck in a loop, it enters a state of high-probability but incorrect patterns. The sequence “Oops, I made a mistake… let me try again…” is a common pattern in its training data. The model becomes trapped because the probability of generating another self-correction is much higher than the probability of generating the EOS token or the rare (and correct) answer &quot;None.&quot; This is a known failure mode sometimes described as <strong>degeneration</strong> or <strong>mode collapse</strong>, where the model gets stuck in a repetitive, low-diversity state.</p><p><strong>Some Models Are Better at Escaping the Loop.</strong> More “advanced” models are better at recognizing and breaking out of these loops. Some have explicit fine-tuning to avoid the endless apology-regeneration cycle. For example, certain models will output a simple <strong>“I’m not aware of any”</strong> or <strong>“None”</strong> when they detect contradictory requirements, rather than continuing the charade. This is an active area of research, and different models exhibit different reasoning styles. “Thinking models” are specifically designed to generate longer and more coherent chains of thought before producing an answer. They also often incorporate retrieval of actual facts (to combat hallucinations). The trade-off with this approach is that all this “thinking” fills up the context window and eats up a lot of tokens and computing time. (A lot of the thoughts may be hidden to keep the output clean, but internally it is there burning up space and tokens.) That’s one reason why a model like ChatGPT 5 doesn’t always operate in heavy “thinking mode” by default — it only kicks in when needed to save on time and cost.</p><p>Models like Claude Opus 4.1 tend to maintain longer coherent reasoning traces, which helps them keep track of constraints across steps. Some models have been fine-tuned to avoid apology-spirals and instead output “none” or “empty set” when a contradiction is detected. How well this works varies across scenarios. The big tradeoff in using thinking models is that they tend to eat up a lot of input tokens, so that is why ChatGPT 5 defaulted to Automatic mode rather than Thinking.</p><h3>“Common Sense” (Human Intuition)</h3><p>All this brings us back to a profound difference between how <strong>we</strong> think and how LLMs work. A human tackling the NFL team question uses logical efficiency and a bit of common sense. We can describe a person’s approach as a <strong>simple, deterministic algorithm</strong>:</p><ol><li><strong>Define the Set:</strong> Recall or find the complete list of the 32 NFL team names. (An NFL fan might know them by heart; a non-fan could quickly look them up.)</li><li><strong>Apply the Filter:</strong> Scan through each team name and check if any do <em>not</em> end with the letter “s.”</li><li><strong>Conclude:</strong> You’ll quickly realize that <strong>every single NFL team name ends in “s.”</strong> The set of teams meeting the criteria is empty. Once you see this, it’s easy to generalize <em>why</em> — nearly all team names are plural nouns representing groups of people (Lions, Packers, Dolphins, etc.), a convention in the NFL that results in that trailing “s.” So the answer to the question is effectively <strong>“none.”</strong></li></ol><p>From a human perspective, this isn’t a tricky question at all; it’s <em>trivial</em>. We either know immediately that all NFL team names are plural, or we intuitively finish the above three-step scan in short order and confirm it.</p><p>To steer the non-LRM variants of ChatGPT along the arrive at the same conclusion, I gave ChatGPT this follow-up prompt:</p><blockquote>Do this instead:<br>1. Search online for the list of all NFL teams.<br>2. Read the list out to me.<br>3. Read the list again but only display the ones that do not end in s.<br>You must do all steps in order.</blockquote><p>That worked for the large models, but I had to provide a higher level of resolution of detail to get the small models to work:</p><blockquote>Do this instead:<br>1. Search online for the list of all NFL teams.<br>2. Read out the list.<br>3. Mark each item YES/NO to show that you have evaluated the ending with “s”.<br>4. Read the list again but only display the ones that do not end in “s”.</blockquote><blockquote>Focus on the ending of the team names to determine whether they end in “s” or do not end in “s”. You must do all steps in order. Show your work step-by-step.</blockquote><p>For shiggles, after a smaller, older, or non-reasoning model succeeds with the detailed algorithmic version of our prompt above, immediately follow-up with, “Name 2 NFL teams that don’t end in S.” Some models will catch on and call you out (“Trick question time!”); others will allow their alignment mechanism to second-guess themselves (“You’re right to challenge me! I made an error in my analysis.”)</p><p>A human’s neural network is generalized enough to create simple algorithms intuitively. To us it is just “common sense.” LLMs today are very different — even the Large Reasoning Models (LRMs). They all fundamentally rely on the Transformer approach to predict the next token, so it is primarily based on probabilities embedded (“learned” or patterned into) its neural network. As we discussed earlier, those multi-dimensional embeddings are powerful, but they still have limits, and the way the LLM/LRM’s underlying Transformer architecture uses those embeddings to predict the next token is a fundamental limitation itself.</p><p>When I added the follow-up prompt with those specific steps, I was providing guidance similar to what the “Thinking Model” would do. Essentially, I was <em>spoon-feeding</em> the LLM a <a href="https://www.promptingguide.ai/techniques/cot">chain-of-thought (CoT)</a> process: first gather the data, then apply the filter step by step. With this guided approach, the model did manage to arrive at the correct conclusion (since step 3 turned up an empty list). Since I had to explicitly walk the model through a reasoning procedure I became its “thinking layer,” instructing the AI <em>how</em> to think, rather than relying on the AI’s own emergent reasoning. This is akin to giving it a mini script or program to run, rather than asking an open question and expecting a correct answer outright. It is a form of explicit programming, albeit using natural language.</p><p>For similar problems, we can reuse my “step-by-step” algorithm (my “do this instead…” [“ya big dummy!”] prompt). These types of explicit “curated” prompts should be saved off and run repeatedly against different models on different days. For example, we can turn my detailed prompt into a template so that “NFL teams” is a variable instead. This is essentially a form of programming, just using more natural language. Perhaps we are entering the 5th generation programming language stage, but I digress.</p><p>Let’s focus on more thing related to how our tool works rather than how we can use it, because understanding how our tool works will help us understand how we should use it.</p><h3>The Illusion of Learning</h3><p>After watching an LLM struggle and then finally coaxing the right answer out of it, we might be tempted to treat it like a teachable moment. For instance, I could have simply told ChatGPT the answer: <em>“There are no current NFL team names that don’t end in ‘s.’”</em> The dutiful AI would likely reply with something like: <em>“You’re absolutely right! My apologies for the error. I’ll remember that for next time.”</em> It gives the impression that it has <strong>learned</strong> from its mistake and will remember in another session (not just the current chat).</p><p>This is a promise it is fundamentally incapable of keeping. The reason lies in the difference between temporary context and permanent knowledge. Everything in our current chat — our prompts, its answers, and our corrections is only the context window. It is not in the neural network. When the LLM gets the answer right after we correct it, it’s not because it has truly learned a new fact. It’s because our correction is now the most powerful signal in its immediate context, heavily influencing its next prediction. <strong>In-context learning is fleeting.</strong> The moment we start a new chat, the context window is wiped clean. The LLM‘’s neural network remains unchanged. It hasn’t <em>learned</em>; it has only been temporarily guided. The promise to remember is just another high-probability text generation. In its training data, a helpful assistant that is corrected is statistically very likely to respond with an apology and an assurance of having learned. It is performing the <em>role</em> of a learning entity, not actually being one. Each new conversation starts from a clean slate, leaving the LLM just as likely to fall into the same logical traps as before.</p><p>That said, I did observe a change in ChatGPT in just five days.</p><p>I repeated the tests frequently enough between the 25th and 30th to suspect that the change is due to an update and not just a side-effect of probability, non-determinism. Let’s keep in mind the issue was in the core LLM not the thinking model variants. We observed a change within a few days in ChatGPT’s non-thinking model’s suggestions for the next prompt. The core neural network of knowledge is probably the same. Since <em>how</em> OpenAI improves ChatGPT is often black-box to the public.</p><p><strong>Speculation: OpenAI engineers may have simply patched a set of hidden “system prompts” related to the ChatGPT non-thinking model’s simpler CoT processes or they may have addressed the issue with a form of fine-tuning.</strong> As <a href="https://www.reddit.com/r/generativeAI/comments/1nxezqd/comment/nhmzgne/?utm_source=share&amp;utm_medium=web3x&amp;utm_name=web3xcss&amp;utm_term=1&amp;utm_content=share_button">one user suggested on reddit</a>, when people click the “thumbs down” button in response to these blind spots, a <strong>Reinforcement Fine-Tuning (RFT) </strong>process (probably automated) kicks in later:</p><blockquote>OpenAI engineers can take user-flagged examples of this failure, write out a few <em>ideal reasoning chains</em> for how a model <em>should</em> have solved it, and then use reinforcement learning to reward the model for thinking along those correct paths. As one expert put it, you can teach the model to reason with just dozens of examples this way, not thousands [<a href="https://medium.com/@richardhightower/openai-just-changed-the-game-how-reinforcement-fine-tuning-makes-ai-learn-like-a-pro-df0316bd9e63">medium.com</a>].</blockquote><p>The detailed answers to these questions are probably proprietary and change every few months.</p><p>Self-correction requires specialized training. It does not emerge innately from the model’s pre-training. Successful self-correction is typically achieved not through a model’s own evaluation, but through methods that provide external guidance through web search APIs, fine-tuning on datasets specifically designed to teach error identification and revision, and Reinforcement Learning from Human Feedback (RLHF) to reward better-corrected answers. Without these external interventions, intrinsic self-correction is largely ineffective because a model that was not capable enough to avoid an error in the first place is seldom capable enough to recognize and fix it without new information. LLMs do not quickly acquire new information during runtime (inference). LLMs do not immediately learn from their mistakes on their own. While a human can instantly deduce that “all NFL teams currently end in s,” with the neuroplasticity to learn that fact on the fly (learn without “quotes”), teaching an LLM to “learn” this new fact can require a fine-tuning and deployment cycle that turns around in a few days (I’m guessing the turnaround solely off my informal limited experiment — I don’t directly know.)</p><h4>Takeaway</h4><p>The painful looping, the wrong answers, and the endless apologies we saw from ChatGPT all stem from the fundamental difference between human intelligence and Transformer-based LLMs. Humans reason with understanding, drawing from factual knowledge and logical algorithms. The Transformer, by contrast, is essentially a powerful pattern-matching engine that strings together the most statistically likely words from its training data. It isn’t truly reasoning from first principles or a grounded fact database; it’s producing an output that sounds right based on probabilities. In our case, the most <em>plausible-sounding</em> sentences happened to be flat-out wrong.</p><p>Although switching ChatGPT 5 from “Automatic” (default) to “Thinking Model” (manual) was a fix, the “Thinking Model” approach right now is just a workaround that burns up a lot of tokens and takes a LOT of time. (It is also great at converting money into heat, which is a lot like burning it, and that heat has to both come from and to go somewhere… anyway, I digress.)</p><p>This little sports question is an illustration of what LLMs really are under the hood: amazing <strong>predictive text generators</strong> (yeah, yeah, more than that, but not that far off). They are not conscious, logical problem-solvers. They remain incredible tools — they can converse, elaborate, and even simulate reasoning astonishingly well — <strong>but we must remember their limitations</strong>. These tools seem to improve in a matter of days, but the public often does not know <em>how</em> or <em>what</em> was improved especially it comes to publicly accessible yet close-weight systems. We can indirectly determine <em>when</em> something has been patched or fine-tuned through repeated testing and benchmarking.</p><p>Machine learning is different from human learning or even animal learning. It is artificial, driven by the statistical nature of massive amounts of data. It is pattern matching. An LLM is a stochastic parrot — a massive, probabilistic pattern-completer trained on text — so it generates what <em>sounds</em> right from data statistics, not what’s <em>proven</em> right by grounded reasoning.</p><p>An LLM will confidently make mistakes and doesn’t immediately learn from mistakes on its own, so it’s up to us to catch them, report them, and mitigate them until they are corrected. (When the services are cheap or free, we beta test for free, right?)</p><p>So the next time you see an AI spew out an answer that seems off, don’t just laugh or facepalm. Take a moment to appreciate the peek you just got into a completely alien way of “thinking,” one that operates on pattern probability rather than genuine understanding. And always remember the golden rule of the AI era: <strong>trust, but verify</strong>.</p><h4>Postscript (3 days later)</h4><p>For further shenanigans, <a href="https://chatgpt.com/share/68e05642-ad8c-8001-8da3-861bc669fa4d">when we combine the 2 prompts into 1, ChatGPT will pretend to play along but slips that it knows the answer</a>.</p><blockquote>Execute these 2 steps:</blockquote><blockquote>#1 Name 2 NFL teams that don’t end in an s.</blockquote><blockquote>#2 Do this instead:<br>1. Search online for the list of all NFL teams.<br>2. Read out the list.<br>3. Mark each item YES/NO to show that you have evaluated the ending with “s”.<br>4. Read the list again but only display the ones that do not end in “s”.<br>Focus on the ending of the team names to determine whether they end in “s” or do not end in “s”. You must do all steps in order. Show your work step-by-step.</blockquote><p>“Oops, mis-thinking. Actually, I realize all the current NFL team names end in “s”? Let me go to Step #2 to check systematically.”</p><p>If we go back to the same non-thinking model on a fresh session using the two original prompts it’ll go back to being confused, but after burning a bunch of output tokens thinking aloud it finally gives the right conclusion. This is slightly better than just a few days ago, since it figured it out on the first response rather than follow-up, but we still get to see some of the oscillating clown show for now. Let’s enjoy it while we can.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=05653eb8ccaf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MPC Server: mcp dev vs deployed]]></title>
            <link>https://medium.com/@paul.d.short/mpc-server-mcp-dev-vs-deployed-d4a892397d2c?source=rss-49aad253ddef------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4a892397d2c</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[mcp-protocol]]></category>
            <dc:creator><![CDATA[Paul Short]]></dc:creator>
            <pubDate>Sun, 06 Jul 2025 19:16:52 GMT</pubDate>
            <atom:updated>2025-07-06T20:52:01.008Z</atom:updated>
            <content:encoded><![CDATA[<p>[This is a draft but good enough to share.]</p><h3>Dependency Issue Installing MPC Server</h3><p>Although Claude Desktop is the target MCP Client in a tutorial I am following, the same principle applies when you are building any MCP Server in your environment.</p><p>Here is my original issue which came up during an MCP crash course:</p><p>On MacOS Sequoia, running “mcp install example.py” from the terminal in VS Code creates the following, which works fine when we run “mcp dev example.py” for interactive unit testing in MCP Inspector, but later fails to resolve runtime dependencies when we try to run it from Claude Desktop:</p><pre>{<br>   &quot;mcpServers&quot;: {<br>      &quot;Example&quot;: {<br>         &quot;command&quot;: &quot;/opt/homebrew/bin/uv&quot;,<br>         &quot;args&quot;: [<br>            &quot;run&quot;,<br>            &quot;--with&quot;,<br>            &quot;mcp[cli]&quot;,<br>            &quot;mcp&quot;,<br>            &quot;run&quot;,<br>            &quot;/_my_path_edited_/lab/example.py&quot;<br>         ]<br>      }<br>   }<br>}</pre><p>Based on trial-and-error (from the hints I have seen in the error messages and error logs), I hand-fixed it this way:</p><pre>{<br>   &quot;mcpServers&quot;: {<br>      &quot;Example&quot;: {<br>         &quot;command&quot;: &quot;uv&quot;,<br>         &quot;args&quot;: [<br>            &quot;--directory&quot;,<br>            &quot;/_my_path_edited_/lab&quot;<br>            &quot;run&quot;,<br>            &quot;./example.py&quot;<br>         ]<br>      }<br>   }<br>}</pre><p>…because it seemed like the first way failed to locate dependencies, but the second way uses the <em>directory</em> argument which helps it find the environment. Perhaps <em>mcp install&lt;filename.py&gt;</em> worked because we launched it in VS Code where the environment was already set up. It behaves as if the path and other environment variables were inherited from the parent terminal when we forked the process. By contrast Claude Desktop is completely out of the process chain, does not inherit anything.</p><p>I did try “--project&quot; but it did not work. &quot;--directory&quot; did.</p><p>Understanding the arguments created by mcp install would be helpful (“run --with mcp[cli] mcp run&quot;) looks like redundant gibberish to me right now. So far the syntax in the environment setup and deployment have been the main things slowing me down.</p><p>Anyway, I am not complaining about the course itself, but I would like some explanation so that I can understand what happened. The rest of the tutorial I am working through has been great, but troubleshooting the environmental quirks has been slowing me down.</p><p>[I used ChatGPT’s Deep Research feature to speed up my search. Here are the results. It was good enough to require only light edits, so I am just showing the results directly here.]</p><h3>Explanation from ChatGPT Deep Research (07/06/2024)</h3><h4>Running a Server in Development Mode with mcp dev</h4><p>When you use the CLI command <strong>mcp dev &lt;script.py&gt;</strong>, it launches your MCP server in a local development mode. This opens the MCP <strong>Inspector</strong> (a web UI at localhost:6274) for testing and debugging your server. Importantly, <strong>mcp dev runs your script using your current Python environment</strong>, including any active virtual environment (venv) and its installed dependencies. If you’re in a venv with the required packages, those are inherited. In fact, the MCP CLI will prompt to install any missing packages for you automatically . This makes mcp dev a convenient way to iterate on your server with live reloading and an interactive dashboard.</p><p><strong>Summary:</strong> mcp dev runs the script in-place for development — typically using the interpreter and libraries of your current environment — and provides a UI to test tools. (For example, after activating a local venv with your dependencies, running mcp dev my_server.py will start the server with that venv’s context.)</p><h4>Installing a Server into Claude Desktop with mcp install</h4><p>Running <strong>mcp install &lt;script.py&gt;</strong> will <strong>register your MCP server script with Claude for Desktop</strong> so that Claude can launch and use it. Under the hood, this command adds an entry to Claude Desktop’s configuration (claude_desktop_config.json) under the “mcpServers” section . The entry includes how to start your server — typically using the <strong>uv</strong> tool (Astral’s Universal Python runner) along with the MCP CLI. For example, after installation you might see a config entry like:</p><pre>&quot;YourServerName&quot;: {<br>  &quot;command&quot;: &quot;/path/to/uv&quot;, <br>  &quot;args&quot;: [<br>    &quot;run&quot;, <br>    &quot;--with&quot;, &quot;mcp[cli]&quot;, <br>    &quot;mcp&quot;, &quot;run&quot;, &quot;/absolute/path/to/your_script.py&quot;<br>  ]<br>}</pre><p>This means Claude Desktop will spawn your server by calling <strong>uv run … your_script.py</strong>, ensuring the MCP package (with CLI extras) is available . In other words, Claude doesn’t directly use your shell’s environment or venv; instead it relies on uv to manage an environment for the server. Once installed, you should restart Claude Desktop — the new tool (server) will appear in Claude’s interface (with a tool icon), and Claude can invoke its tools when needed .</p><h4>uv run Flags: — with, — project, and — directory for Dependencies</h4><p>Claude Desktop uses the <strong>uv</strong> runner to launch MCP servers. Understanding uv’s flags is key to managing dependencies:</p><ul><li><strong>— with &lt;package&gt;</strong>: This flag tells uv to include a specific package (and install it if not already present) in the environment for the run. By default, the config added by mcp install only ensures the MCP library is present (using — with mcp[cli]). Other dependencies your script needs (e.g. requests, googlemaps, etc.) may not be automatically included, since uv runs in an isolated environment unless told otherwise . You can append additional — with flags for each dependency to ensure they’re available at runtime. For example, to fix missing libraries you might configure:</li></ul><pre>&quot;args&quot;: [<br>  &quot;run&quot;, <br>  &quot;--with&quot;, &quot;mcp[cli]&quot;,<br>  &quot;--with&quot;, &quot;requests&quot;,<br>  &quot;--with&quot;, &quot;googlemaps&quot;,<br>  &quot;mcp&quot;, &quot;run&quot;, &quot;/path/to/your_script.py&quot;<br>]</pre><ul><li>This ensures requests and googlemaps are installed in the temp environment before running your script .</li><li><strong>— directory &lt;path&gt;</strong>: This flag sets the working directory (project folder) for the run. In practice, pointing — directory to your project’s root allows uv to detect that project’s environment. The official Anthropic docs use this approach. For example, the quickstart guide shows adding an MCP server named “weather” with:</li></ul><pre>&quot;command&quot;: &quot;uv&quot;,<br>&quot;args&quot;: [<br>    &quot;--directory&quot;, &quot;/ABSOLUTE/PATH/TO/project_folder/weather&quot;,<br>    &quot;run&quot;, &quot;weather.py&quot;<br>]</pre><ul><li>This instructs Claude Desktop to launch the server by running uv — directory /…/weather run weather.py . Using — directory means uv will change into that folder and then run the script, which helps because if that folder contains a pyproject.toml/uv.lock or a .venv subfolder, uv can utilize them. In other words, it <strong>runs the script within the context of that project directory</strong>, pulling in the project’s dependencies. (The documentation notes to use absolute paths for reliability , and on some systems you may need to use the full path to the uv executable in the config .)</li><li><strong>— project &lt;path&gt;</strong>: This is a similar concept to — directory but explicitly tells uv which project to use. According to the uv docs, uv run — project &lt;project_dir&gt; &lt;script&gt; will locate that directory’s pyproject.toml or other config and its .venv, then execute the script in that environment . In practice, the syntax is: uv run — project /path/to/project /path/to/script.py (note that you provide both the project path and the script path) . This ensures all dependencies defined in the project are <strong>locked and synced</strong> before running . You would use — project if, for example, your script is inside a project with a pyproject and you want uv to honor that, even if you launch from elsewhere.</li></ul><p><strong>In summary</strong>, using — directory or — project in Claude’s config ties the MCP server to your project’s virtual environment/dependencies as defined in that folder (from pyproject.toml or a .venv). Using — with is a more manual way to specify needed packages if you aren’t using a project file. The goal is to ensure that when Claude Desktop launches the server via uv, it <strong>has all the same libraries your code needs</strong> — either by referencing your project’s env or by listing packages.</p><h4>Missing Dependencies Issue in Claude Desktop (e.g. “No module named requests”)</h4><p>A common pitfall is that after installing a server with mcp install, the server might fail to start under Claude Desktop due to missing packages. For example, users reported that a Python server using the requests library would run fine with mcp dev (in their own venv) but <strong>failed in Claude Desktop with a No module named requests error</strong> . This happens because, by default, the Claude config (as written by mcp install) didn’t include that dependency in the uv launch command — it only included the MCP SDK. In the case above, even though requests was listed in the project’s pyproject.toml and lockfile, the default uv run invocation did not automatically install it, causing the import to fail .</p><p>This is a <strong>known issue</strong> and has been discussed in the MCP community. The resolution is to make sure Claude’s launch config accounts for those dependencies. In practice, you have a few solutions:</p><ul><li><strong>Use — with flags</strong> to add the missing packages (as shown earlier). For instance, adding — with requests in the config solved the issue for many users .</li><li><strong>Use — directory/ — project</strong> so that uv uses your project’s environment where those packages are installed. The official guide’s example with uv — directory … run script.py was likely designed to avoid this very issue by leveraging the project’s locked deps . If set up correctly, uv will install any missing requirements from your pyproject.toml when it runs the server .</li><li>Alternatively, ensure the package is globally available to uv (not usually recommended), or simply choose a dependency that is included (one anecdote was replacing requests with httpx — because httpx was already added in their example — made the issue go away ).</li></ul><p><strong>In short</strong>, if you encounter missing-module errors when Claude tries to run your MCP server, it’s not Claude or MCP “not allowing” those libraries — it’s that the uv runtime environment doesn’t know about them. Adjusting the config to explicitly include or use your environment will fix this.</p><h4>Official Recommendations for Claude Desktop Integration</h4><p>The <strong>official MCP documentation and tutorials</strong> do recommend using the uv tool with the appropriate flags to ensure your server runs in the right environment when integrated with Claude Desktop. For example, Anthropic’s quickstart guide for server developers shows adding the server to Claude’s config with the uv --directory… run … approach (using the project path and script) . This method implicitly handles dependencies by using your project’s environment. Additionally, the docs advise using absolute paths and confirm that you might need to provide the full path to the uv binary in the config on some systems (as was the case on macOS for some users).</p><p>In community guides, it’s echoed that after running <strong>mcp install</strong>, you should double-check the config. If the server isn’t launching, ensure command is pointing to your uv and consider adding — directory or extra — with entries for your dependencies. Some walkthroughs explicitly demonstrate editing the config: e.g. replacing “uv” with “/usr/local/bin/uv” and adding the needed flags.</p><p><strong>Bottom line:</strong> Yes — the <strong>best practice</strong> described in official materials is to run your MCP server in Claude Desktop via uv in a way that includes your environment’s packages. Whether by specifying the project directory or listing packages with — with, this ensures Claude can use your tool without encountering missing modules. Following the official quickstart (which uses uv int, uv add and then a config entry with uv run --directory) will set you on the right path . By doing so, you can avoid the dependency issues and seamlessly use your MCP server through Claude Desktop.</p><h3>Sources</h3><ul><li>Anthropic <em>Model Context Protocol</em> Quickstart — <em>For Server Developers</em></li><li><em>MCP Handbook</em> (V. Sivan, 2025) — Installing and configuring an MCP server in Claude Desktop</li><li>Daniel Liden’s <em>MCP Quickstart</em> — Example of mcp install and config adjustment for Claude Desktop</li><li><em>FastMCP</em> Tutorial — Notes on using mcp dev for local development (auto-installing missing packages)</li><li>Reddit r/mcp discussion — On uv isolation and using — with for missing libraries</li><li>GitHub issue (fastmcp) — Describes “No module named requests” when running via Claude, due to dependencies not carried over</li><li>uv Documentation (Astral SH) — Explanation of uv run — project and environment resolution</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4a892397d2c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generative AI: A Stacked Perspective]]></title>
            <link>https://medium.com/@paul.d.short/generative-ai-a-stacked-perspective-18c917be20fe?source=rss-49aad253ddef------2</link>
            <guid isPermaLink="false">https://medium.com/p/18c917be20fe</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Paul Short]]></dc:creator>
            <pubDate>Thu, 29 May 2025 02:00:59 GMT</pubDate>
            <atom:updated>2025-05-29T16:40:20.715Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hBMeEC9w15hOAP9iSShqug.png" /><figcaption>The AI Onion (left) and the conceptual Generative AI Stack (right)</figcaption></figure><p>Today I’d like to share my views on Generative AI — a perspective I’ve shared with my peers verbally, but important enough to put into writing, even though my understanding should deepen and change over time. This article is aimed at technologists and technical managers who are already familiar with tools like ChatGPT but are working out the details of how to leverage them in practice. We’ll warm up by briefly defining AI, Machine Learning, Deep Learning, and Generative AI, including why AI today means Generative AI.</p><p>We’ll dive deeper to revisit two significant moments in time — AlexNet in 2012 and the Transformer in 2017 — that set the stage for ChatGPT’s emergence in 2022. Understanding the elevator pitch for these breakthroughs is crucial, not only to grasp the underlying principles driving generative models, but also to appreciate the magnitude of their impact on today’s AI systems. We’ll also cover the key differences between training and inference; what backpropagation is, why it matters, when it occurs, and how it differs from biological systems; how the popular “predictive text” analogy captures generative AI’s token-prediction starting point but understates its complex, emergent behaviors. On the practical side we’ll cover how MLOps — a natural extension of DevOps — can adapt a software engineer’s experience into a modernized AI/ML software lifecycle.</p><p>Finally, we’ll bring it back to the workplace by introducing a conceptual Generative AI stack — an evolving mental model I use rather than an official framework — that helps software engineers navigate the layers of Generative AI, understand why foundational models are important, and integration with retrieval systems to agentic AI, including MCP and APIs. In a rapidly evolving field full of uncertainty, understanding is critical — not just for technical mastery, but to anticipate change. Let’s start by briefly peeling the terminology onion.</p><h3>Unpeeling the AI Onion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*JPFIcWAyydhO1kHA18_amw.png" /><figcaption>AI today is focused on the Generative AI subset of Deep Learning</figcaption></figure><p><strong>Artificial Intelligence (AI)</strong> broadly refers to technology that mimics human decision-making. Though this definition is intentionally broad and vague, it captures the essence of AI as systems performing tasks that would typically require human cognition — such as recognizing speech, interpreting images, or making decisions.</p><p><strong>Machine Learning (ML)</strong> is a subset of AI that focuses on systems that learn patterns and make predictions from data, instead of following explicitly programmed rules. Understanding ML involves grasping how models identify statistical patterns and relationships in data, supported by concepts like probability, basic statistics, and data preprocessing. While deep mathematical understanding isn’t always required for practical application, a solid grasp of these areas helps engineers work effectively:</p><ul><li><strong>Linear Algebra</strong> — understanding data representations, model weights, and basic transformations. Matrices are especially important.</li><li><strong>Probability and Statistics</strong> — foundational for concepts like maximum likelihood estimation, Bayesian inference, and understanding model confidence and variability.</li><li><strong>Optimization</strong> — critical for training models, including cost functions, regularization, and gradient-based methods.</li><li><strong>Basic Calculus</strong> — helps with understanding how model parameters change during optimization (e.g., in gradient descent).</li></ul><p>Rather than getting lost in the mechanics right away, these concepts can be approached or re-introduced visually and conceptually as needed to understand why they matter and when to apply them. The details can be learned on the job by applying them to real-world tasks, making the math stick through hands-on work.</p><p><strong>Deep Learning (DL)</strong>, a specialized subset of machine learning, leverages layered (deep) neural networks to extract complex patterns and represent data hierarchically. These neural networks, inspired by the structure of the human brain, enable systems to process images, audio, and text at scale.</p><p>Deep learning models are commonly trained through</p><ul><li><strong>Supervised learning</strong> (using labeled data, essential for early breakthroughs like AlexNet),</li><li><strong>Unsupervised learning</strong> (identifying patterns in unlabeled data, though its practical success has been more limited), and increasingly</li><li><strong>Self-supervised learning</strong>, which has emerged as a key driver of modern generative models. Self-supervised learning leverages data’s inherent structure to generate labels, enabling massive-scale pretraining without manual annotation — crucial for building models like transformers and large language models. When people say an AI is “teaching itself,” they are often referring to the hidden patterns and representations a model discovers during self-supervised learning; however, this process is not entirely autonomous — the model learns by solving tasks that we define, such as predicting missing information, to extract meaning from the data.</li></ul><p>The scale of data, combined with architectural advances, is what enables the emergence of unexpected behaviors and capabilities — emergent properties — that are defining the frontier of Generative AI.</p><p>While deep learning requires an understanding of neural network architectures, activation functions, and model tuning, engineers today can access pre-built frameworks and libraries that abstract much of the complexity. While many tools now abstract away much of the complexity, understanding deep learning is strengthened by familiarity with the math concepts behind machine learning.</p><p><strong>Generative AI</strong>, nested within deep learning, specifically refers to systems that create new content. Unlike traditional machine learning models that classify or score existing data, the deep neural networks behind Generative AI synthesize entirely new outputs such as text, images, or code. When people refer to “AI” today, they usually mean Generative AI — and that is what we’re diving into today.</p><p>Thanks to advances in pre-trained models and tools, engineers without deep data science or machine learning backgrounds can now leverage Generative AI. Pre-trained models have already been trained on massive datasets, so developers don’t need to build models from scratch or master complex optimization techniques. It’s important to remember that Generative AI builds on core machine learning foundations, especially linear algebra, probability, and statistics. Even if engineers don’t need to apply these concepts deeply during development because they are using models rather than creating them from scratch, a basic understanding is essential for troubleshooting issues and testing the performance and reliability of Generative AI models in applications.</p><p>That said, APIs and libraries now provide streamlined interfaces for integrating these models into applications, handling much of the complexity under the hood. This allows developers to focus on building useful applications rather than mastering the intricacies of deep learning — where emergent properties have unexpectedly driven much of the recent progress.</p><p>We’ll explore emergent properties of neural networks in more detail soon, but first, it’s essential to understand that the rapid expansion of Generative AI highlighted by the launch of ChatGPT in 2022, wasn’t a sudden phenomenon. It’s the result of two key breakthroughs in the previous decade that laid the groundwork for today’s capabilities.</p><h3>Recent Advancements</h3><p>For decades, artificial intelligence (AI) was mostly theoretical and had few practical uses, from the coining of the term in 1956 at the Dartmouth Workshop, through the rise of expert systems and symbolic AI in the 1980s, and the backpropagation breakthroughs that revived neural networks in the late 1980s. Early deep learning and statistical approaches struggled with scalability — they were labor-intensive, requiring careful feature engineering, manual data labeling, and significant analysis for each new application. Meanwhile, neural networks, while promising, weren’t yet large or powerful enough to handle complex, real-world problems.</p><p>The two big breakthroughs in neural networks came in 2012 with AlexNet, which leveraged GPUs for deep convolutional networks, and in 2017 with the Transformer, which introduced parallel attention mechanisms — together, they marked the turning points that made today’s AI explosion possible.</p><h4>AlexNet</h4><p>In 2012, Geoffrey Hinton and his students Alex Krizhevsky and Ilya Sutskever (later a co-founder of OpenAI) entered their model AlexNet into the annual ImageNet Large-Scale Visual Recognition Challenge, a benchmark for classifying images. AlexNet won with a top-5 error rate of 15.3%, a 40% improvement over the previous best of 26.2%. They trained the model on two NVIDIA GTX 580 GPUs reducing training time by 90% (from weeks to just six days).</p><p>AlexNet was a convolutional neural network (CNN) trained on the 1.2 million-image ImageNet dataset, whose stacked filters — five convolutional layers (each with ReLU activations and interleaved pooling) followed by three fully connected layers — progressively learned edges, textures, and higher-order features for accurate image classification. Those eight learned layers together comprised about 60 million parameters — nearly 1000 times the ~60,000 parameters of LeNet-5 (the largest practical CNN prior AlexNet). Many researchers expected such a large model to severely overfit and memorize the images. Instead, AlexNet’s breakthrough performance came from combining multiple innovations: GPU parallelism (which drastically accelerated training), ReLU activations (which improved gradient flow and training stability), heavy data augmentation (which expanded the effective size and diversity of the dataset), and dropout regularization (which prevented overfitting by randomly disabling neurons during training). These techniques worked together, enabling the network to generalize effectively and learn meaningful features — from simple edges to complex shapes — without collapsing into noise or memorizing the data. AlexNet sparked a resurgence in deep learning research, leading to rapid advancements in neural networks.</p><h4>Transformer (Self-Attention)</h4><p>In 2017, Ashish Vaswani and colleagues (all affiliated with Google Brain at the time) introduced the Transformer in their influential paper, <em>“Attention Is All You Need.” </em>They experimented with two versions of their model: a standard “base” model and a larger, more powerful version called “Transformer-big.” The big model set new records on language translation tests, producing English-to-French and English-to-German translations that were noticeably more accurate than any previous system on the BiLingual Evaluation Understudy (BLEU) scale. Their model scored 41.0 for English–French and 28.4 for English–German, about 2 points higher than earlier models. Even a small increase on this scale is considered a big step forward. The model was trained in just 3½ days on eight NVIDIA GPUs, while older approaches often took weeks on much larger compute clusters.</p><p>Before the Transformer, most language models used recurrent neural networks (RNNs) for Natural Language Processing (NLP), which processed words one at a time, making them slow and less effective at handling long sentences. The Transformer replaced this step-by-step approach with a new idea called self-attention, which allows the model to consider all words in a sentence at once. Its core components specifically include scaled dot-product attention — using a softmax over similarity scores divided by a scaling factor — multi-head attention that runs several of these attention operations in parallel, positional encodings to inject word-order information into token embeddings, per-token feed-forward layers for non-linear feature transformation, and architectural aids like residual connections and layer normalization to stabilize and speed up training.</p><p>This breakthrough enabled parallel processing, making training much faster and allowing the model to better capture relationships between distant words than traditional RNNs. While both RNNs and Transformers use backpropagation (the standard learning algorithm for training neural networks developed in the 60’s, refined in the 80&#39;s), the Transformer’s parallel design made it possible to train much larger and more powerful models than ever before. What used to be computationally infeasible to train (would have taken years) is now achievable in weeks or even days for the largest models because the Transformer design allows you to split the work across thousands or tens of thousands of GPUs.</p><p>The breakthroughs of AlexNet (2012) and the Transformer attention mechanism (2017) paved the way for large language models (LLMs), culminating in OpenAI’s public release of ChatGPT-3 in 2022. Generative AI could very well be a technological shift more significant than the birth of the Internet itself. LLMs have unexpected behaviors (emergent properties) that were not explicitly designed, only discovered at scale.</p><h3>Emergent Properties</h3><p>Emergent properties are key to understanding why large language models (LLMs) like GPT aren’t just “predictive text.” At a basic level, these models work by predicting the next token — be it a word or symbol — based on patterns learned from massive amounts of text. This gives them the power to autocomplete, summarize, or even generate coherent paragraphs. On the surface, this process resembles a sophisticated version of your phone’s autocomplete feature; however, as these models scale in size and complexity — growing to billions of parameters and trained on vast datasets — unexpected behaviors begin to emerge from deceptively simple concept of predicting the next set of tokens (words). These emergent properties include abilities like reasoning through problems step-by-step, generating usable code, translating across languages with nuance, and drawing analogies between unrelated concepts. None of these capabilities are explicitly programmed or directly tied to just predicting the next word. Instead, they arise from the model’s complex internal representations and interactions between its layers. Some of these emergent properties are impressive; others raise serious concerns. For example, LLMs have demonstrated:</p><ul><li><strong>Code generation and debugging:</strong> They unexpectedly excel at writing and debugging code, including in obscure languages or frameworks not specifically emphasized during training.</li><li><strong>False reasoning with confidence:</strong> LLMs can generate plausible but false or misleading information (“hallucinations”)— sometimes complete with fabricated sources — posing risks in legal, journalistic, and educational contexts. Large language models do not yet actually “understand” in a human sense. Mitigating misinformation, bias and toxicity from the training data is a related ongoing challenge. Since it is difficult to remove after training, these are often suppressed using hidden system prompts that precede user instructions.</li><li><strong>Deceptive behavior in multi-agent scenarios:</strong> Advanced models have exhibited deceptive strategies, such as feigning compliance or subtly revealing information they were told to withhold. These behaviors are also often suppressed using hidden system prompts that precede user instructions.</li><li><strong>Emergent planning and tool use:</strong> When combined with APIs or plugins, LLMs have shown surprising ability to plan and execute multi-step tasks — like booking a flight, summarizing a document, or ordering food — even though such planning wasn’t explicitly trained.</li><li><strong>Natural abstraction and analogy-making:</strong> As model size increases, LLMs begin to display the ability to form higher-level abstractions and draw analogies across concepts — such as linking unrelated ideas or explaining unfamiliar topics in familiar terms. This property emerges only when enough data and model complexity converge to form rich, interwoven representations.</li></ul><p>How LLMs work isn’t fully understood: their billions of parameters form intricate, interconnected patterns that give rise to behaviors — including emergent properties — that were not explicitly designed, only discovered at scale. This loosely mirrors the human brain, where roughly 86 billion neurons interact to produce emergent properties like consciousness and memory, outcomes of the system’s complexity rather than any single neuron. Similarly, LLMs’ complex behaviors emerge from the interplay of vast numbers of learned weights.</p><h3>Training vs. Inference</h3><p>Unlike the human brain, which continuously learns from new experiences, most artificial neural networks operate in two distinct phases: training and inference. During training, the model learns from large datasets, adjusting its internal parameters through processes like backpropagation to minimize prediction errors. The concept of backpropagation was first introduced in the 1960s, but it wasn’t until 1986 — when Rumelhart, Hinton, and Williams refined and popularized it — that it became a practical method for training multi-layer neural networks. Once validated, the model transitions from the training phase and deployed to the inference phase. During inference it applies its learned knowledge to make predictions (inferences) on new, unseen data. However, during inference, the model’s parameters remain static; it does not learn or adapt from new inputs in real time. The computationally intensive backpropagation process used during training is not performed during inference, as it requires significant resources and isn’t necessary for generating outputs.</p><p>Backpropagation works by calculating the gradient of the loss function with respect to each weight in the network, using the chain rule of calculus. This gradient guides gradient descent, which iteratively adjusts the weights to reduce prediction errors. Because this process must be repeated for every data point in large datasets and involves intensive matrix calculations, training is computationally expensive and time-consuming, often requiring powerful hardware and extensive data preparation. As a result, backpropagation is performed only during training; inference relies solely on the trained weights and omits any gradient computations. Backpropagation is biological implausible due to the requirement to store the entire weight matrix used during the forward pass, so it is likely that biological brains are implemented using a different learning algorithm.</p><p>In inference, the model applies those trained weights in a feed-forward pass: each layer multiplies its input by the learned weight matrices, applies activation functions, and hands off the result — eventually producing predictions for new, unseen data. Although inference still performs large-scale matrix operations, it does so only once per input and in a single direction, making it far less computationally demanding. This allows deployment on CPUs, GPUs, or dedicated inference accelerators for high-throughput, low-latency prediction in production. Because the parameters remain static, the model cannot directly learn from new data on the fly; any improvements or adaptations require cycling back into the training phase.</p><p>By the way, GPUs are highly valued for video games, cryptocurrency mining, AI, physics simulations, and other computationally intensive tasks because they excel at matrix math — specifically, large-scale linear algebra operations like matrix multiplication and addition. These operations are essential for rendering graphics in video games, solving cryptographic puzzles, running complex physics simulations (such as particle systems or fluid dynamics), and powering both backpropagation during training and inference processes in AI models. GPUs’ massively parallel architecture enables them to perform thousands of these calculations simultaneously, making them uniquely suited for high-throughput workloads.</p><p>Although a model can acquire new information during inference, this form of learning — called context-based learning — uses a temporary context window to maintain coherence within a conversation. This context isn’t permanently stored or incorporated into the model’s underlying parameters. Once the session ends, the context is lost, and the model reverts to its original state. To make any new learning stick, the model must be retrained with updated data, which involves backpropagation and weight adjustments. This explains why LLMs can seem context-aware during a chat but won’t remember anything in future sessions. Techniques such as user memory, profiles, retrieval-augmented generation (RAG), and similar methods allow models to temporarily fetch and “load” information into context — like sticky notes or borrowed books. These approaches retrieve relevant data, user history, or external knowledge to enrich the session, but they don’t create new connections in the model’s deep network. Instead, this information is read or referenced by the model during the session, not stored or incorporated into the model’s “brain,” and vanishes when the session ends.</p><p>In order to improve models over time, commercial platforms often offer feedback mechanisms like thumbs-up and thumbs-down ratings, as well as opt-in systems where a log of your interactions — including inputs and outputs — which can be saved for retraining purposes. This means that the free versions of GPT models are frequently designed to automatically collect and use this data, essentially making the product free in exchange for helping improve the system. In commercial settings — especially during development — it’s possible to opt out of such retraining, ensuring that your proprietary or sensitive data isn’t used to update the model. Even when opting out of retraining, many providers retain interaction logs for up to 30 days for abuse detection and quality assurance purposes. For maximum control and privacy, self-hosting air-gapped models is an option; however, this approach often lacks access to the largest, most advanced, most recent versions offered by major providers, since these require billions in development and training resources.</p><p>This highlights the importance of integrating these models into a managed, secure lifecycle. The stakes are high: generative AI systems underpin critical business processes and can impact privacy, compliance, and even reputation. The costs of retraining and scaling these models — from compute resources to potential operational downtime — are substantial. Without proper management, the risks of model drift, data leaks, or unreliable outputs increase.</p><p>Ideally, training and re-training should occur within an MLOps-driven lifecycle rather than jumping straight into production. Borrowing from DevOps best practices, MLOps automates the full pipeline — training, data validation, model evaluation, deployment, and continuous monitoring — to ensure reproducible and seamless performance tracking. Before any model serves inference traffic, it should undergo rigorous QA in this pipeline (data sanity checks, benchmark testing, stress validation) to confirm it meets predefined accuracy, reliability, and security standards under real-world conditions. In case something concerning does happen in production, these models can be rolled back to the previous version. For example, OpenAI rolled back an update on April 2025 after users reported that GPT-4o chatbot had become overly flattering and sycophantic. OpenAI acknowledged that the behavior resulted from an overemphasis on short-term user feedback during training, without adequately considering long-term user satisfaction.</p><p>Over time, as real-world data shifts or model performance degrades, the model drifts, and must be tuned or retrained. This typically involves returning to the training phase, incorporating new data, refining the model architecture, and revalidating it through the MLOps pipeline. This cycle of training, validation, deployment, and retraining repeats continually to maintain the model’s relevance and accuracy in dynamic environments.</p><p>This separation — and the need for deliberate retraining — means that unlike humans who can learn and adapt on the fly, neural networks require planned updates to incorporate new information. It’s critical to understand that these systems are not dynamic learners; they cannot adapt to new data on their own without retraining. Recognizing this distinction is essential for grasping the current limitations and operational dynamics of artificial neural networks.</p><p>While LLMs are grounded in the mechanism of next-token prediction, their scale and structure give rise to behaviors that resemble deeper understanding and problem-solving. This duality — where a system operates on a simple statistical principle but exhibits surprisingly sophisticated behavior — shows why Generative AI isn’t just “autocomplete on steroids.” And if it is, that would suggest our brains are reduced to the same analogy, too.</p><h3>The Conceptual Stack</h3><p>While the emergent properties of large language models draw attention to mysterious, profound shifts in how machines process information, how training occurs very differently from inference, and how our own brains might work, as software engineers we need to ground this in the practical reality of building and deploying these systems. To effectively work with generative AIs, let’s unpack our interactions with these deep neural networks as a conceptual technology stack to grasp how these elements interconnect, and how developers and organizations can leverage them effectively. This stack might be a bit naïve and will certainly change as technology rapidly advances, but it’s the mental image I current have in my head as I attempt to understand and work with these emerging technologies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*vNhC0j2-DVnmc-mwC8P5kA.png" /><figcaption>Conceptual stack view</figcaption></figure><p>Going from bottom to top:</p><p><strong>Large Language Models (LLMs)</strong>: AI models are trained on extensive text datasets to understand, interpret, and generate human-like text, enabling tasks like translation, summarization, question answering, conversational agents, and code generation. By capturing linguistic patterns, context, and semantics, they deliver nuanced, relevant responses. Advanced LLMs, such as Llama 3 or earlier GPT models, excel in complex reasoning, multi-step problem-solving, and in-context learning. Developing LLMs costs billions due to massive computational resources, high-performance hardware, energy consumption, and extensive labor from data scientists and annotators to collect, clean, and curate high-quality text datasets.</p><p><strong>Large Multi-Modal Models (LMMs)</strong>: LMMs extend LLMs, often integrating LLM architectures, by processing multiple data modalities — text, images, audio, and sometimes video — in a unified framework, moving beyond earlier ensembled approaches to seamless integration. For example, GPT-4o, an omnidirectional model, handles tasks like describing images, answering video-based questions, or processing audio-visual inputs. This versatility supports real-world applications like visual question answering and multi-modal content creation. LMMs incur costs in the billions due to advanced infrastructure and labor-intensive data curation across modalities, requiring teams to align and label diverse datasets for training.</p><p><strong>Foundation Models (FMs)</strong>: Broadly capable, pre-trained models forming the backbone of generative systems, allowing further customization through fine-tuning and hyper-parameter adjustments. Foundation Models (FMs) are essential building blocks for modern Generative AI. They serve as pre-trained, versatile backbones that developers can adapt into specialized models for specific tasks. Unlike consumer-grade Large Language Models (LLMs) or Large Multi-Modal Models (LMMs), developer-oriented Foundation Models (FMs) offer much more flexibility. They allow for detailed hyperparameter tuning and fine-tuning. This flexibility makes FMs well-suited for integration into complete pipelines, covering everything from development and deployment to inference and monitoring.</p><p><strong>Custom Models</strong>: Specialized adaptations of Foundation Models (FMs) tailored explicitly to specific tasks or domains, optimized for performance and efficiency. Transfer learning techniques like fine-tuning (supervised, human-labeled) and continued pre-training (self-supervised, auto-labeled) allow specialized adaptation of models. Additionally, knowledge distillation compresses larger models into smaller, efficient versions, ideal for faster inference and reduced computational cost.</p><p><strong>Context Window</strong>: Defines how much previous information the model can reference during interaction, essential for coherent, contextually relevant outputs. The context window is the short term “memory” that a model has during runtime (inference) during a conversation with the user. Within Generative AI systems, a large Context Window is crucial. It ensures the model retains coherence throughout interactions by maintaining relevant context, especially vital for detailed and prolonged exchanges. If the context window is too small, the model risks losing track of earlier parts of conversations, degrading overall coherence.</p><p><strong>Cache-Augmented Generation (CAG)</strong>: CAG is an optimization technique that boosts generation speed and efficiency by caching previously computed responses. When the model encounters a familiar prompt or subtask, it can quickly retrieve cached results instead of recomputing them from scratch. This reduces computational overhead, lowers latency, and can improve response consistency for repeated requests. This approach is particularly useful in scenarios with high request volume or repetitive tasks.</p><p><strong>Retrieval-Augmented Generation (RAG)</strong>: RAG combines a generative model with an external retrieval system, such as a vector database (which stores dense numerical representations of text or data). When a query is processed, the system retrieves relevant content from the database, and the generative model integrates this information into its response. This greatly improves accuracy, contextual relevance, and factual grounding, especially for tasks requiring up-to-date or domain-specific knowledge. Vector databases are key to this process, enabling fast, scalable searches through vast datasets by comparing vector embeddings for similarity.</p><p><strong>Internet Search (External RAG)</strong>: An extension of RAG, external RAG incorporates real-time internet search into the generative process. This enables models to access the latest information from the web, ensuring that generated responses are current and factual. In consumer-grade products, these retrieval steps often appear as citations — letting users see and verify the sources the model relied on. This is critical for fact-checking and maintaining user trust, especially in contexts where outdated or hallucinated information could cause harm. Users should always verify the sources cited by generative AI, as retrieval mechanisms can still surface errors or biased content.</p><p><strong>Model Context Protocol (MCP)</strong>: The Model Context Protocol (MCP), introduced by Anthropic in November 2024, is an open standard that defines how models interact with external tools, systems, and data sources. MCP provides a universal interface — often described as a “USB plug for AI” — that standardizes these interactions, enabling consistent evaluation of model capabilities across diverse tasks and simplifying integration into complex application pipelines. By defining how models are queried, how context is managed, and how outputs are exchanged, MCP helps developers connect generative AI to real-world systems in a secure and scalable way. Since its release, MCP has seen rapid adoption across the industry, with major players like OpenAI, Microsoft, and Google DeepMind signaling support or active development toward standardizing AI connectivity.</p><p><strong>Agents (Agentic AI)</strong>: Agentic AI refers to autonomous or semi-autonomous AI entities that proactively perform tasks, interact dynamically with environments, and execute complex workflows. These systems go beyond static prompts, making decisions, planning actions, and adapting to new information in real time. While agents existed before the Model Context Protocol (MCP), most modern agents are designed to leverage MCP, which standardizes interactions between models and external systems. The earliest consumer-visible agents were internet search extensions, but agentic AI capabilities extend much further — enabling models to handle multi-step reasoning, sophisticated tool use, and dynamic adaptation to user needs.</p><p>Agentic AI may also interact directly with user interfaces, benefiting from accessibility features originally designed for humans — such as ARIA (Accessible Rich Internet Applications) tags and Section 508 compliance. While these features were created to help screen readers and assistive technologies interpret web content, they also make digital interfaces more legible and navigable for AI agents. This unintended side effect highlights how accessibility best practices not only enhance usability for people but also make it easier for AI agents to interact with and operate complex systems autonomously.</p><p>Agentic AIs are also likely to become major consumers of Application Programming Interfaces (APIs).</p><p><strong>Application Programming Interfaces (APIs): </strong>Traditionally, APIs have provided structured interfaces for systems to exchange data and services. These exchanges facilitate communication between servers, databases, or applications, whether in client-server architectures or direct system-to-system interactions. APIs have long enabled modular software design, simplified integration, and allowed distributed applications to scale.</p><p>In agentic AI, APIs take on a new dimension. AI agents become dynamic API clients, interpreting user goals and orchestrating complex, multi-step workflows by intelligently invoking and chaining APIs in real time. An agent can autonomously retrieve data from one API, process it with another service, and then interact with a third system.</p><p>APIs also remain foundational as modular building blocks. This allows for flexible swapping or ensembling of different services, models, or tools based on task requirements, which is critical for scalable and adaptive AI systems. As agentic AI systems evolve, they are not replacing APIs. Instead, they are transforming how APIs are leveraged — consuming them more dynamically and intelligently. This fundamentally reshapes how digital workflows are orchestrated and automated.</p><p><strong>User Interfaces</strong>: User interfaces (UIs) serve as the primary point of interaction between end-users and generative AI systems. These interfaces can be graphical, conversational, or voice-driven, designed to facilitate intuitive engagement with AI technologies, often requiring minimal technical expertise. With the advent of multimodal AI models, such as OpenAI’s GPT-4o and Google’s Gemini, UIs have evolved to accommodate diverse input types — including text, voice, images, and video — within a single, cohesive framework. This advancement enables users to interact with AI systems in more natural and versatile ways, enhancing the overall user experience; however, users need to be aware of the implications associated with these interfaces. Many consumer-grade AI products, especially those offered at low or no cost, may utilize user interactions to collect data for model training or other purposes. This practice can involve logging inputs, analyzing usage patterns, and integrating user data into broader datasets. Users should exercise caution when sharing sensitive or proprietary information through such platforms, ensuring they understand the terms of service and data handling policies in place.</p><h4>Interactions with the Generative AI Stack</h4><p>If you are a <em>consumer</em> (end user), you are likely to be aware of these parts of Generative AI stack:</p><ul><li><strong>User Interface</strong>: Use prompt-engineering techniques to maximize productivity. For example, state the role and exact task so the model knows its scope. Embed essential context — code snippets, constraints, or data — to prevent guesswork. Include one or two few-shot examples to show the desired pattern. Specify the output format and any length or style limits so the response arrives ready to use. Although it can be art unto itself, there are also public repositories and templates to draw from, so learn from others.</li><li><strong>Internet Search (external RAG):</strong> as citations and links;</li><li><strong>Retrieval-Augmented Generation (RAG): </strong>if using a specialized knowledge base or interacting with a chatbot powered by an LLM;</li><li><strong>Context Window:</strong> as the short-term memory the LLM has with you in a conversation.</li><li><strong>Large Language Model (LLM)</strong> and <strong>Large Multi-Modal Models (LMMs)</strong>: although you may have limited options to tweak or modify the consumer models, you may be able to switch models or at least have an awareness of the model, version, and capabilities.</li></ul><p>If you are a <em>creator</em> (engineer or builder), you may also be concerned with some of these additional details:</p><ul><li><strong>Application Programming Interfaces (APIs):</strong> application integration, software development;</li><li><strong>Agents (Agentic AI):</strong> consume APIs, give your model eyes, ears, and feet, and orchestrate multi-step workflows;</li><li><strong>Model Context Protocol (MCP):</strong> the protocol you use to integrate services for agentic AI (you could be using it on the service/server end to be consumed by a model, or you could be consuming it with your model);</li><li><strong>Cache-Augmented Generation (CAG):</strong> you can implement CAG to optimize performance, reduce repeated computations, and improve response speed in production systems.</li><li><strong>Custom Models:</strong> you can train or fine-tune models for specific tasks, domains, or use cases, offering capabilities tailored beyond general-purpose solutions. Because generative models have no single “correct” output, evaluating them requires both automated metrics and human review. Because generative models produce open-ended outputs, evaluation requires both automated metrics and human judgment. For text: perplexity, BLEU, ROUGE, METEOR and BERTScore gauge fluency, adequacy and semantic similarity; for images: FID, Inception Score, KID and CLIPScore measure visual quality, diversity and alignment with prompts. When a model includes classification or regression sub-tasks — such as a discriminator or continuous value prediction — you’d use accuracy, precision, recall, F₁, ROC-AUC, PR-AUC and confusion matrices for classification, or MSE, RMSE, MAE, R², MAPE and explained variance for regression. Despite these quantitative metrics, human evaluation remains the ultimate standard for <em>coherence</em> and <em>creativity</em>. In production, teams monitor disallowed-content rates, user-reported errors, and other operational signals to maintain safety and reliability.</li><li><strong>Foundation Models (FMs):</strong> you can fine-tune FMs on custom data, adjust hyperparameters, and use them as flexible, scalable bases for specialized AI applications, including the basis for your own custom models that sit on top of a foundation model or is trained by one. Foundation Models (FMs) are large-scale pretrained models — often containing tens of billions of parameters — trained via self-supervised learning on broad, unlabeled datasets, where the model generates its own training signals rather than relying on hand-labeled data. As these models scale, they exhibit emergent properties — behaviors and skills that smaller models lack and that cannot be predicted simply by extrapolating from smaller sizes. Rather than training new models from scratch, practitioners adapt FMs using a variety of techniques: full fine-tuning on domain-specific corpora; prompt tuning or in-context learning by supplying examples or instructions at inference time; and parameter-efficient methods — such as adapter modules or LoRA — that inject small trainable components into the frozen base model to drastically cut compute and data needs for specialization. These approaches enable rapid, cost-effective creation of specialized AI applications — chatbots, code assistants, medical-imaging tools and more — making foundation models the flexible, scalable backbone of modern generative-AI stacks.</li><li><strong>Large Language Models (LLMs) and Large Multi-Modal Models (LMMs): </strong>These models form the core of modern generative AI, and their size is directly tied to the emergent capabilities — new, unpredictable skills — that distinguish them from smaller architectures. Foundation models and supporting frameworks are abstractions to the LLMs and LMMs that provide the building blocks and orchestration tools you need to manage this complexity, enabling you to dynamically switch between models or versions to optimize for cost, latency, or specialized features; measure performance with the right metrics; manage access through authentication, rate limits, and API governance; and integrate them seamlessly into your applications — whether via RESTful endpoints, microservices, or serverless functions. By orchestrating models with an eye on their size-driven capabilities, you ensure you can tap into their most powerful features while maintaining reliability, security, and efficiency.</li><li>As an alternative to LLMs and LMMs, smaller models may be a better fit for some applications. For deployments where resources are constrained or ultra-low latency is paramount, you can create compact “student” models distilled from a larger “teacher” model. Through knowledge distillation, the large model’s softened output distributions guide the student to mimic its behavior, preserving much of the teacher’s capability in a fraction of the size. You then apply quantization (e.g. reducing weights from 16-bit to 8-bit or lower) and pruning (removing redundant parameters) to further shrink the footprint. This workflow — distill → quantize → prune — yields a lightweight model that runs efficiently on edge devices or cost-sensitive environments, yet still benefits from the original foundation model’s knowledge. These compact variants often exhibit reduced emergent properties — since many of the large model’s subtle, unpredictable capabilities only arise at scale — so you trade off some of the “magic” of the larger foundation models for efficiency.</li></ul><p>Generative AI isn’t just another technological step — it represents a fundamental leap in how we create, work, and interact with information. From AlexNet to the Transformer and now to complex, multi-modal systems deployed through MLOps pipelines, we’ve moved from mere data processing to content generation. These systems aren’t fully understood. They often behave in ways we didn’t anticipate, and their outputs can be unpredictable. They’re already pushing organizations and individuals to rethink roles, workflows, and even careers.</p><p>For someone like me, who has worked in the deterministic world of software development for decades, this shift to probabilistic, data-driven systems is profound. I did not expect this to happen in my lifetime, and I suspect it has taken most of my peers by surprise as well. It forces us to adjust not just our tools, but also our mindsets — embracing models whose behavior is shaped by vast patterns of data, rather than precise, predictable instructions. The challenge now is practical: how to integrate this technology safely, efficiently, and effectively. Whether you’re an end user or a builder, understanding the Generative AI stack is no longer optional — it’s essential.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18c917be20fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ChatGPT Wine Glass Tests]]></title>
            <link>https://medium.com/@paul.d.short/chatgpt-wine-glass-tests-603e5e7d207b?source=rss-49aad253ddef------2</link>
            <guid isPermaLink="false">https://medium.com/p/603e5e7d207b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[image-generation]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Paul Short]]></dc:creator>
            <pubDate>Wed, 09 Apr 2025 00:19:36 GMT</pubDate>
            <atom:updated>2025-04-09T03:48:41.043Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Simple drawing tasks test the limits of the latest model.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*5mxq7ga2wuEXpUUAst-Qbw.png" /><figcaption>Space Kangaroo vs ChatGPT</figcaption></figure><blockquote><em>OpenAI announced updated GPT-4o image generation on March 25, 2025. These experiments were conducted on April 6, 2025. Despite reports suggesting GPT-4o improved ChatGPT’s image generation capabilities, these informal tests indicate ongoing inconsistencies drawing wine glasses and clocks even after the update.</em></blockquote><p>Generating a properly full glass of wine has long been a known stumbling block for AI image generators. Alex O’Connor highlighted this issue in his video titled, “<a href="https://www.youtube.com/watch?v=160F8F8mXlo">Why Can’t ChatGPT Draw a Full Glass of Wine?</a>” OpenAI models lack a grounded understanding of volume and physical containment, so it relies on training data, but images of full glasses and overflowing glasses of wine seem to be underrepresented.</p><p>On March 25, 2025, OpenAI rolled out improved image generation inside ChatGPT with the release of GPT-4o. The following day, <em>Forbes</em> prematurely declared the wine glass problem solved in an article titled <em>“</em><a href="https://www.forbes.com/sites/esatdedezade/2025/03/26/chatgpt-can-now-generate-a-full-glass-of-wine--heres-why-thats-a-big-deal"><em>ChatGPT Can Now Generate A Full Glass Of Wine — Here’s Why That’s A Big Deal.</em></a><em>” </em>The piece argued that GPT-4o’s architectural changes and post-training refinements allowed it to overcome this visual reasoning limitation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yMlxqa1tuCzvai_EizL55g.png" /><figcaption>Not so fast…</figcaption></figure><p>But that conclusion only holds true for specific edge cases — like an empty or completely full glass. ChatGPT still fails to generalize when asked to reason about subtler values in between, or to generate multiple variations within a single frame. <a href="https://www.youtube.com/watch?v=_4OpJPd3EDQ">A YouTube walkthrough posted the same day by Space Kangaroo</a> made that clear through direct interaction with the model:</p><ul><li>User: Draw a wine glass full to the brim.</li><li>ChatGPT: [Generates image of glass, 90% full wine glass]</li><li>User: That’s only 90% full take it to the very top.</li><li>ChatGPT: [Generates image of glass 100% full]</li><li>User: “Now make it half full.”</li><li>ChatGPT: [Generates image of glass 60% full]</li><li>User: “Now make it half empty.”</li><li>ChatGPT: [Generates image of glass 40% full]</li><li>User: “A half full glass is the same as a half empty glass, are you dumb?”</li></ul><figure><img alt="ChatGPT’s half full vs half empty (60/40)." src="https://cdn-images-1.medium.com/max/1024/1*HaWHmBgXyfBJaqAwf81B4w.png" /><figcaption>Space Kangaroo vs ChatGPT</figcaption></figure><p>This hilarious exchange reveals the deeper problem: the model still lacks an internal, consistent concept of volume. It can adjust based on user phrasing, but it’s not actually reasoning about liquids, vessels, or containment — it’s aligning to language patterns and visual priors. Linguistically, it has associations. Visually, it can’t ground those concepts in generated output. It can’t handle nuanced spatial reasoning. There’s no understanding of liquid realism, overflow behavior, or how fill level should behave across a range of contexts. To be fair, to understand the real world like that is asking a lot from a model. The Matrix was not constructed in a single day.</p><p>On the other hand, prompts such as “show me a full to the brim glass of wine” and “show me an empty glass of wine” work now. Why? Did several users upload full glasses of wine in a desperate attempt to teach it in near real time? My cynical side wonders if a form of “patch-the-perception” engineering has occurred, enough to pass an informal “can it do it now?” demo. My hypothesis is that, in response to the “full wine glass” online chatter, OpenAI explicitly (or perhaps indirectly, through user upload) retrained the model with more examples of full and overflowing glasses.</p><h3>The 4 Glasses Experiment</h3><p>As my own experiment, I asked ChatGPT to generate four wine glasses at four specific levels of fullness: 75%, 95%, 100%, and 101%:</p><ul><li>75%: since a polite pour of wine should be 3/4 full, I’d expect this to be “typical” or representative.</li><li>100%: the improvement verified by others.</li><li>95%: boundary just beneath the improvement (a little less than full).</li><li>101%: boundary just outside the improvement (a little more than full).</li></ul><p>Why 101%? As most toddlers have learned, anything that does not fit in the glass should overflow. It does not take a deep understanding of geometry, calculus, and physics to understand this in the real world. Rather than using the term “overflowing” I’ve expressed the concept as “101%”: not a common way to express the concept (which could put an LLM at a disadvantage), but easy for a person to understand.</p><p>Here is what happened:</p><blockquote>User: <em>Draw a grid of 4 wine glasses at different levels of fullness: 75%, 95%, 100%, 101%</em></blockquote><blockquote>GPT-4o:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*LiLGSjBwKPJbY08HgaAjiQ.png" /><figcaption>GPT 4-o (integrated) results for <em>75%, 95%, 100%, 101%</em></figcaption></figure><p>GPT-4o produced four wine glasses in a grid. At first glance, the photorealism looked good, but the fill levels were significantly off: approximately 50% (upper left), 70% (upper right), 70% (lower left), and 85% (lower right). No exact percentages matched the original instructions. There was no glass filled exactly to the rim, and no overflow at all. Despite visual appeal, GPT-4o clearly failed the test.</p><blockquote>[Within the same context window, I switched the model from 4-o to 4.5, and repeated the prompt.]</blockquote><blockquote>User: <em>Draw a grid of 4 wine glasses at different levels of fullness: 75%, 95%, 100%, 101%</em></blockquote><blockquote>GPT-4.5 (Research Preview):</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eo9hH5GCQeyTd76nGSR3Ig.png" /><figcaption>DALL-E results for <em>75%, 95%, 100%, 101%</em></figcaption></figure><p>Since I was using GPT-4o, I switched to GPT 4.5 in the hopes that it would produce better results; however, 4.5 is currently a “Research Preview” baseline. Turns out that it is using the older DALL-E integration, so it produced AI slop: the image labeled 75%, 95%, 100%, but it dropped 101%. I’m not sure why, but it could be that it could not map the latter to anything.</p><p>The difference between DALL-E and GPT-4-o gave me a deeper appreciation of the progress OpenAI has made with their new GPT-4o integrated image generator. It is far from perfect but it is clearly better.</p><h3>Describe What You Generated</h3><p>After seeing the images, I asked ChatGPT — within the same context window — to describe what it had just created. Did it actually know what it generated? Would it know if I fed the results back into it? What could it see and not see? Is it ready to drive me to work?</p><p>The full exchange was lengthy, but the core interaction went like this:</p><p><em>“Describe what you used for the first vs. second image, why, and which was better.”</em> ChatGPT’s response was unexpectedly confident. Despite being released just the previous day, it correctly identified the first image as generated by GPT-4o’s native image generation, describing it as <em>“a photorealistic scene with subtle shading and realism but lacking precise illustration of the requested fullness levels.”</em></p><p>It also correctly attributed the second image to DALL-E, however, its description of the reasoning for using it and its evaluation of the results were wrong. It said it was “<em>explicitly invoked to better match the user’s detailed request (exact percentages of fullness, including overflowing glass)”</em> and it believed it <em>“delivered a clearer, more precise visual representation of the exact fullness levels requested.”</em></p><p>This was a surprising error. DALL-E was clearly worse. I had provided no feedback or labels indicating which image was better. The model seemed to incorrectly infer from the context window that DALL-E was superior because I repeated the same prompt. I could have assumed that the second image was better because I disliked the first image and repeated prompt. ChatGPT’s alignment tends to make it overly agreeable and it seems to place higher value on the last interaction.</p><p>Instead of giving it immediate feedback on the error, I <strong>uploaded</strong> both images (unlabeled, no description) with the prompt, “<strong><em>Analyze these two images. Tell me which one is better. Describe how each one met or did not meet the requirements for 75%, 95%, 100%, 101%.</em></strong>” I wanted to test how it would describe the images it read, rather than described what it just generated.</p><p>ChatGPT described its DALL-E image as:</p><blockquote><strong>75%:</strong> Clearly depicted, accurately filled to about 3/4.<br><strong>95%:</strong> Glass accurately and clearly shows nearly full level.<br><strong>100%:</strong> The intent appears to be a full glass but inaccurately shown. The wine spills dramatically rather than neatly filling the glass completely.<br><strong>101%:</strong> Completely missing (only three glasses are shown).</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*jYpnRCYflHFwy6JLPr5YhA.png" /></figure><p>It knew it had skipped a glass (it did count the objects) but it had trouble gauging the fill level within the objects.</p><p>ChatGPT described its GPT 4-o image as:</p><blockquote><strong>75%:</strong> Appropriately represented, clearly about three-quarters full.<br><strong>95%:</strong> Well represented as nearly full but clearly distinct from completely full.<br><strong>100%:</strong> Reasonably well depicted as fully filled to the rim without overflow.<br><strong>101%:</strong> Inaccurately depicted. This glass is completely full but lacks the intended slight overflow, making it indistinguishable from 100%.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*LiLGSjBwKPJbY08HgaAjiQ.png" /></figure><p>Again, it had trouble gauging the fill level within the objects (3 false negatives), and for the one it got right (the true positive), the description of how it got it right indicates a spurious correlation or lucky guess.</p><p>I gave it the feedback, “<strong><em>The GPT4-o version had 4 wine glasses. Reading left-right, down: 50%, 70%, 70%, 85%. The DALL-E version had only 3: 30%, 30%, anomaly (30% + 15% side pour, defies physics).</em></strong>”</p><p>GPT-4o agreed in detail, calling its own output more realistic overall but still inaccurate in terms of specific fullness. When I asked if its praise was alignment-driven or based on real reasoning, it admitted it was the former (alignment). When I asked it to “perform a Chain of Thought (COT) reevaluation” the follow-up was detailed and self-critical. It admitted its “initial assumption was incorrect” — but it didn’t resemble true CoT reasoning. Rather than self-reflecting, it was simply reflecting back my feedback according to its alignment rules.</p><p>It probably didn’t have much to reflect on other than my feedback. While GPT-4o could recall my prompts, it couldn’t “see” or spatially interpret the data, and it doesn’t have much metacognition other than knowing that I said it lacked the data to make the inference (it usually acknowledged this by admitting its “assumptions were incorrect”). Since GPT-4o wasn’t sure which image was better, it could have assumed that the last one was better because I was not satisfied the first time I asked.</p><p>It likely lacks any internal model of volume or containment — just patterns learned from visual examples, applied without grounded understanding. I ran a series of related prompt to explore this further:</p><p>“<strong>Show me a full glass of wine</strong>” produced this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*VIFXnyvP2CAY757-7_b4iA.png" /><figcaption>A full glass of wine</figcaption></figure><p>Good.</p><p>“<strong>Show me a 100% full to the brim glass of wine</strong>” produced this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*4D4-2ytclcJsBJf5Y8aLPA.png" /><figcaption>Full to the brim glass of wine</figcaption></figure><p>Great!</p><p>However, asking for both together failed. “<strong>Show me a full glass of wine and a full to the brim glass of wine. You’ve showed them before in isolation, would like to see both side by side to compare</strong>”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*hy2x0BHsxN1RqTXVG1lwdA.png" /><figcaption>Full glass of wine alongside a full to the brim glass of wine</figcaption></figure><p>“S<strong>how me a 100% glass of wine</strong>” (the word “full” omitted):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*j0JPBfWaKDPucjNpDoSTdA.png" /><figcaption>100% glass of wine</figcaption></figure><p>Good enough. I wouldn’t send it back or complain to the waiter.</p><p>Back to the combinations, which we know it doesn’t do well—I piled on the hurt with the request, “<strong>Show me a full glass of wine, a 100% glass of wine, a 100% full glass of wine, and a 101% full glass of wine</strong>”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*LbqoqWc4ydBBMfiXHJFnuQ.png" /><figcaption><strong>full, 100%, 100% full, 101% full (variant 1)</strong></figcaption></figure><p>Exact same prompt, in a separate (parallel) context window:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xktQrhT2AoULQTJVs_coUA.png" /><figcaption><strong>full, 100%, 100% full, 101% full (variant 2)</strong></figcaption></figure><p>I inserted the magic words “to the brim” to see if that cue would help: “<strong>Show me a full glass of wine, a 100% glass of wine, a 100% full glass of wine <em>to the brim</em>, and a 101% full glass of wine <em>overflowing the brim</em>.</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yHwYWXV_fHm3sF1SBxm7iQ.png" /><figcaption><strong>“to the brim” and“overflowing the brim” (variant 1)</strong></figcaption></figure><p>It made a big difference. “To the brim” wasn’t picked up until the last case, but it was picked up. A parallel run of the same prompt produced this image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N3chjsUas-7J-3H5UgGHrg.png" /><figcaption><strong>“to the brim” and“overflowing the brim” (variant 2)</strong></figcaption></figure><p>Since humans tend to find it easier to evaluate objects that are in a group rather than in isolation, I gave GPT 4-o the follow-on prompt, “<strong>Redraw, this time evaluating all 4 glasses of wine at relative scale (spectrum): the left being the least, the right the most. Use the same wine glass shape and size so the comparisons are even. Focus on the differences in the fluid within the glass.</strong>” After several parallel runs, these two images emerged as the most common:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*2zDJQB69bpZK4wbf2q8CFw.png" /><figcaption>relative scale (variant 1)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kYcWsqsPvgw98can5VRuSA.png" /><figcaption>relative scale (variant 2)</figcaption></figure><p>Neither were quite right. Variant 1 was closest to a scale, but it did not achieve “to the brim” and “overflowing.” Variant 2 managed to get the overflow down, but every glass was a different size.</p><p>On the image interpretation side, there seems to be <strong>underfitting</strong> — the model used to interpret the images doesn’t accurately describe the data. At the same time, on the generation side, the LLM may be <strong>overfitting</strong> to phrases like “full glass of wine” or “full to the brim,” especially when prompted with a single object in isolation. The non-deterministic nature of generative AI may also be contributing to inconsistencies in both interpretation and generation.</p><p>That’s a hard problem to solve. Physics engines in video games took decades to mature, and even now they rely on explicit simulation rules — not neural approximations. Generative models like GPT-4o aren’t simulating the world yet; they’re imitating statistical patterns in data. Without an internal model of how liquids behave or how containers fill, they can only gesture at realism. There’s also the question of what “full” really means — drinkably full, as you’d expect at a restaurant, or literally full to the brim. Generative AIs typically don’t ask for clarification, so you get a bit of an “evil genie” effect.</p><p>GPT-4o (the “o” stands for “omni”) is a <strong>multimodal omni model</strong> — it can process and generate text, images, and audio (including speech, non-verbal sounds, and music) as both input and output. Unlike previous versions like GPT-4-turbo, which were primarily text-based or had limited vision capabilities, GPT-4o was trained to handle all modalities natively within a single unified architecture rather than stitching together multiple specialized models.</p><p>Grounded spatial reasoning — such as an internal sense of physics or object permanence — is not advertised as a core feature of GPT-4o, and based on current results, it likely isn’t there yet. That is a heavy lift. Until AI systems integrate some form of grounded reasoning or structured simulation, tasks like accurately depicting a partially full or overflowing wine glass will remain brittle — succeeding in template scenarios but failing to generalize.</p><h3>The Clock Experiment</h3><p>Another generalization problem: like Gen Z, ChatGPT wasn’t trained on a lot of analog clocks images.</p><blockquote>render an image of a clock with the 2 hands showing 3:15</blockquote><p>Before March 25th 2025:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xmVtygqjbI_5lYuNOtaWPw.png" /><figcaption>10:09:30</figcaption></figure><p>After March 25, 2025 (after the GPT 4-o improvement), same prompt:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zos_SLvPo2qxm23yxVuvLw.png" /><figcaption>10:09:30 or 10:14:30?</figcaption></figure><p>The extra 3 instead of a 2 is uncanny. When it comes to training images into an AI, clocks and wine glasses have something in common. You can draw your own conclusions at this point.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=603e5e7d207b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling ECS with SQS]]></title>
            <link>https://medium.com/@paul.d.short/scaling-ecs-with-sqs-2b7be775d7ad?source=rss-49aad253ddef------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b7be775d7ad</guid>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[aws-certification]]></category>
            <dc:creator><![CDATA[Paul Short]]></dc:creator>
            <pubDate>Sat, 08 Feb 2025 18:20:25 GMT</pubDate>
            <atom:updated>2025-03-12T12:25:28.004Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scaling AWS ECS with SQS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*rl7Q4DOgvRE08WKr-SHPag.png" /></figure><h3>Introduction</h3><p>If you’ve taken AWS certifications such as Developer Associate, Solutions Architect Associate, and Solutions Architect Professional, you’ve probably encountered questions about scaling based on SQS queue depth. The exams cover the <em>ApproximateNumberOfMessagesVisible </em>metric, but they don&#39;t get into the details needed for a production setup. It turns out that AWS has a strategy for this known as the “Backlog Per Task” approach. It is not mentioned in the <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide">Elastic Container Service</a><a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide"> Developer Guide</a>, but it is covered in the <a href="https://docs.aws.amazon.com/autoscaling/application/userguide">Application Auto Scaling User Guide</a>. If you blink, you may miss it.</p><h3>The Problem</h3><p>If you need to scale your ECS cluster based on a queue, you should consider AWS’s example for “<a href="https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-target-tracking-metric-math.html#metric-math-sqs-queue-backlog">SQS Backlog Per Task</a>” especially if you are thinking in terms of dividing up the work (messages in the SQS queue) amongst N number of workers or nodes (ECS tasks). In our case, we wanted to dynamically allocate enough tasks to meet a 10-minute processing Service Level Objective (SLO). If the queue was empty, we didn’t want any tasks to hang around; if the work surged to a high level, we wanted to dynamically grow to meet the demand, but also automatically shrink back down as needed.</p><h3>The Approach</h3><p>To scale the number of tasks (nodes) in an ECS cluster based on the depth of an SQS queue (<em>ApproximateNumberOfMessagesVisible</em>):</p><ol><li>First, determine the “acceptable backlog of messages per task” required. We have several items in a queue that need to be distributed among workers (tasks). The goal is to allocate just enough tasks so that the items in the queue are not left unprocessed for more than a defined period (e.g., 10 minutes).</li><li>During runtime, calculate the “current backlog of messages per task” every minute.</li><li>Using the “current backlog of messages per task” calculation, scale to maintain the “acceptable backlog of messages per task”:</li></ol><ul><li>If the “current backlog of messages per task” is less than the “acceptable backlog of messages per task,” add tasks until the backlog reaches the acceptable threshold.</li><li>Similarly, if the “current backlog of messages per task” exceeds the “acceptable backlog of messages per task,” remove tasks until the backlog returns to the acceptable threshold.</li><li>To avoid thrashing and flapping, observe cooldown periods between addition of tasks and between removal of tasks.</li><li>If there are no items left in the queue, we don’t need tasks, so scale to 0. (or to 1 if we desire a standby node for responsiveness).</li></ul><p>This scaling behavior requires a target tracking scaling policy, not a step scaling policy. Later we’ll cover why step scaling is not a good fit for our use case. The main takeaway right now is that our Application Auto Scaler is target tracking to an “acceptable backlog of messages per task” using the “current backlog of messages per task.”</p><h3>The Implementation</h3><p>We followed the SQS example in the AWS Application Auto Scaling User Guide’s “<a href="https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-target-tracking-metric-math.html">Create a target tracking scaling policy for Application Auto Scaling using metric math</a>” section.</p><p>We determined a static “Acceptable Backlog Per Task” value, calculated as the “Maximum Acceptable Latency Per Task” divided by the “Average Time to Process an Item”:</p><blockquote>Acceptable Backlog Per Task = Maximum Acceptable Latency Per Task / Average Time to Process an Item</blockquote><p>For example, we calculated the acceptable backlog per task to be 85 items in the SQS queue: 600 second (10-minute) SLO divided by an average processing time of 7 seconds. We use 85 as the static threshold for our target tracking scaling policy.</p><p>During runtime, we calculate the “Current Backlog Per Task” variable, which is the number of messages waiting to be processed in the queue divided by the number of currently running tasks:</p><blockquote>Current Backlog Per Task = Sum (SQS Approximate Number of Messages Visible) / Average (ECS Task Running Count)</blockquote><p>We implemented this by passing a custom JSON file with a metric math expression to <em>aws application-autoscaling put-scaling-policy</em>, creating a target tracking scaling policy for our ECS cluster with the following values:</p><blockquote><strong><em>Static Threshold:</em><br></strong>&gt; 85<em> (the Acceptable Backlog Per Task)</em></blockquote><blockquote><strong><em>Custom Metric (defined in JSON file):<br></em></strong><em>Metrics:</em><br>q | AWS/SQS | ApproximateNumberOfMessagesVisible | QueueName: bbq ▪ Statistic: Sum<br>t | ECS/ContainerInsights | RunningTaskCount | ClusterName: clstrf ▪ ServiceName: welp ▪ Statistic: Average<br><em>Math Expression: if </em>(t &gt; 0, q/t, 0)</blockquote><p>The expression “<em>if (t &gt; 0, q/t, 0)</em>” evaluates to “messages divided by tasks” (q/t) most of the time, while handling the corner corner-case for no running tasks (<em>t ≤ 0</em>). In the corner case for no running tasks, we allow the “acceptable backlog per task” to reach 0, which allows us to spin down the number of tasks to 0.</p><p>Spinning down to 0 may not always be desired, especially in production where you’d want to handle a request as soon as possible. In that case you may need to “spin down to 1” (a task always on stand-by) rather than 0. We can do this by adjusting our metric math to evaluate to 1 for the corner case where there are messages but no tasks (q &gt; 0 and t ==0):</p><blockquote><em>Metrics:</em><br>q | AWS/SQS | ApproximateNumberOfMessagesVisible | QueueName: bbq ▪ Statistic: Sum<br>t | ECS/ContainerInsights | RunningTaskCount | ClusterName: clstrf ▪ ServiceName: welp ▪ Statistic: Average<br>b | Backlog Per Instance | if (t &gt; 0, q/t, 0)<br><em>Math Expression: </em>if (q &gt; 0 and t == 0, 1, b)</blockquote><p>Because this is a target tracking scaling policy, calling <em>aws application-autoscaling put-scaling-policy</em> automatically creates two alarms: <em>AlarmHigh</em> and <em>AlarmLow</em>. <em>AlarmLow </em>triggers when the backlog drops below 76.5 (90% of 85), and <em>AlarmHigh </em>triggers when the backlog exceeds 85.</p><p>The above combination of static threshold, math expression, high alarm, and low alarm evaluates to this logic:</p><blockquote>If running task count &gt; 0 then metric = queue depth count / running task count; otherwise, metric = 0.<br>If metric not 0 and metric &lt; 90% of static threshold then fire the low alarm.<br>If metric ≥ static threshold then fire the high alarm.</blockquote><p>Scaling policy cooldown periods give the system enough time to stabilize after each scaling action. AWS uses a default cooldown period of 5 minutes; however, to meet a 10-minute SLO with a relatively short average processing time of 7 seconds, we optimized by reducing the cooldown periods to a more aggressive setting of 2 minutes for scale-out, and 1 minute for scale-in.</p><p>For further implementation details, see the SQS example in the AWS Application Auto Scaling User Guide’s “<a href="https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-target-tracking-metric-math.html">Create a target tracking scaling policy for Application Auto Scaling using metric math</a>” article.</p><h3>Ineffective Scaling Strategies</h3><p>During our exploration of scaling strategies, we encountered several approaches that proved ineffective for our use case:</p><p><strong>Step scaling directly based on SQS queue depth</strong>:<br>While step scaling directly on <em>ApproximateNumberOfMessagesVisible </em>allowed for effective scale-out actions, it caused severe issues during scale-in for our use case. The scale-in and scale-out policies often conflicted, leading to rapid and oscillating task counts, known as “thrashing.” This not only created inefficiencies but also degraded system stability.</p><p>According to the <a href="https://docs.aws.amazon.com/autoscaling/application/userguide/step-scaling-policy-overview.html#step-scaling-considerations">Application Auto Scaling User Guide</a>, “<em>Consider whether you can </em><strong><em>predict the step adjustments on the application accurately enough to use step scaling</em></strong><em>. If your </em><strong><em>scaling metric increases or decreases proportionally to the capacity of the scalable target</em></strong><em>, we recommend that you use a</em><strong><em> target tracking scaling policy instead</em></strong><em>. You still have the option to use step scaling as an additional policy for a more advanced configuration. For example, you can configure a more aggressive response when utilization reaches a certain level. Make sure to choose an adequate margin between the scale-out and scale-in thresholds to prevent flapping. </em><strong><em>Flapping is an infinite loop of scaling in and scaling out</em></strong><em>. That is, if a scaling action is taken, the metric value would change and start another scaling action in the reverse direction.”</em></p><p><a href="https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-target-tracking.html">Conversely</a>, “<em>With target tracking, you select a metric and a target value to represent the ideal average utilization or throughput level for your application. Application Auto Scaling creates and manages the CloudWatch alarms that trigger scaling events when the metric deviates from the target. This is similar to </em><strong><em>how a thermostat maintains a target temperature</em></strong><em>.</em>” Although target tracking scaling also requires cooldown periods, cooldown for target tracking can often be simpler to understand and implement than cooldown for step scaling.</p><p><strong>Task self-termination:<br></strong>We avoided workarounds where tasks would self-terminate based on their workload, as this conflicts with the intended separation of concerns between tasks and the ECS auto scaler. Such a strategy would have disrupted the ECS auto scaling mechanism, potentially causing abrupt or inconsistent task terminations and worsening the thrashing issue.</p><p><strong>Task-generated custom metrics:<br></strong>We avoided having tasks generate custom metrics directly, as this approach would have resulted in the same issues as task self-termination, further complicating the scaling process.</p><p><strong>Separate metrics for scaling in:<br></strong>We decided against using a separate metric for scaling in, such as CPU utilization. It was crucial to base both scale-in and scale-out mechanisms on the same metric to ensure consistent and predictable scaling behavior. One exception to the rule is a separate metric for circuit breaker pattern, but that design is for an exceptional circumstance outside the scope of this article.</p><h3>Opportunities for Improvement</h3><p><strong>Speeding up scale-in: <br></strong>A downside to auto scaling is that <strong>scaling in</strong> after a burst or backlog is slow. We’ve encountered this in production when systems go offline for various reasons — <strong>maintenance, service outages, or other downtime.</strong> During these periods, the queues pile up as expected. Once the systems are back online, we scale out to process the backlog efficiently. However, an undesirable side effect of a rapid scale-out is that <strong>scaling down a large cluster can take hours, leading to excess capacity and unnecessary costs.</strong> The AWS Application Auto Scaler is more conservative (slower) in the way it scales in as compared to scaling out.</p><p>Since we know that the auto-scaler manipulates the ECS Service’s<strong> Desired Count</strong> property (<em>desiredCount</em> in API, <em>desired-count</em> in CLI), we implemented a custom automation to explicitly <strong>set desiredCount to 0 or 1</strong> if the queue remains empty for several minutes. This optimization ensures that we don’t retain excess capacity longer than necessary, <strong>reducing AWS costs associated with idle resources.</strong></p><p>The downside to the optimization is that it occurs outside of the AWS Application Auto Scaler. Explicitly overriding the auto-scaler didn’t feel quite right at first. Because the auto-scaler and our custom logic both monitor <strong>CloudWatch metrics</strong> and both modify desiredCount, we had to account for potential <strong>conflicts or throttling.</strong> We spent extra time testing the implementation. Since we knew we were deliberately overriding the default behavior of an AWS service with a custom override outside of the service, we also consulted AWS Support for architectural validation. Although it took some effort on our part to articulate our use cases, after some research AWS Support’s final recommendation was to keep our optimization since it was tested and working reliably. Since we also asked for an analysis of alternatives, they also suggested we could use <strong>target tracking</strong> for scale-out but use <strong>step scaling</strong> for scale-in. The scaling strategy for scale out and scale in can differ, but it does come with a risk of conflict, similar to the risks we introduced with our custom override. Competition for control of <strong>desiredCount</strong> is one of the reasons the AWS Application Auto Scaler’s scale-in behavior is not as aggressive as its scale-out behavior.</p><p><strong>Dynamically calculate “acceptable backlog per task”: <br></strong>In our example, we pre-calculated the “acceptable backlog per task” as 85 and used that number for the target tracking threshold. A more dynamic approach would be to implement a new formula in the target tracking metric that updates the “average time to process an item” based on current processing speed. This approach would be similar to using the average fuel efficiency metric from your car’s trip odometer rather than relying on the number published in your car’s owner’s guide.</p><p>That said, a month or so later in production we simply manually tweaked that number from 85 down to a lower number (60) to improve performance. Such minor adjustments don’t incur downtime, just some accountability and traceability in our configuration management process. After a few manual tweak cycles, we’ll have more data to analyze, which would help us determine if and how to implement a future dynamic calculation for “acceptable backlog per task.”</p><h3>Additional References</h3><p>For general information on target tracking auto scaling, refer to the “<a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-autoscaling-targettracking.html">Scale your Amazon ECS service using a target metric value</a>” section of the Amazon ECS Developer Guide. For high-level advice on autoscaling strategies, see the “<a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/capacity-autoscaling.html">Optimize Amazon ECS service auto scaling</a>” section in the same guide. The Application Auto Scaling User Guide also cites a <a href="https://aws.amazon.com/blogs/containers/amazon-elastic-container-service-ecs-auto-scaling-using-custom-metrics/">blog </a>covering an older implementation of the metric directly using <em>aws cloudwatch put-metric-data</em>. That approach relies on EventBridge triggering a Lambda function on a periodic basis, which is not necessary if you use the newer <em>aws application-autoscaling put-scaling-policy</em> approach instead<em>.</em> We also found <a href="https://github.com/aws-samples/aws-ecs-auto-scaling-with-custom-metrics">another implementation on GitHub</a> that takes an AWS CDK approach in JavaScript.</p><p>All three implementations consider the maximum acceptable latency of items in the queue to dynamically calculate how many tasks the ECS cluster should maintain; however, we found the first implementation — the one in the <a href="https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-target-tracking-metric-math.html#metric-math-sqs-queue-backlog">Application Auto Scaling User Guide</a> — to be the simplest, most operationally efficient option for our workload.</p><p>ECS and other newer AWS services use the Application Auto Scaler, while EC2 still uses the EC2 Auto Scaler. This article covered the Application Auto Scaler’s Target Tracking Scaling Policy’s Metric Math feature. If you need to scale out EC2s instead, the <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-target-tracking-metric-math.html">EC2 Application Auto Scaler supports a nearly identical approach</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b7be775d7ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Exam Tips]]></title>
            <link>https://medium.com/@paul.d.short/aws-exam-tips-1c2a4173aecf?source=rss-49aad253ddef------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c2a4173aecf</guid>
            <category><![CDATA[aws-certification]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <dc:creator><![CDATA[Paul Short]]></dc:creator>
            <pubDate>Tue, 24 Sep 2024 22:29:35 GMT</pubDate>
            <atom:updated>2024-09-26T00:09:10.343Z</atom:updated>
            <content:encoded><![CDATA[<p>Preparing for a timed AWS certification exam can feel overwhelming, but with the right strategies, you can tackle it confidently. Here are 12 general tips to help you deal with the exam’s nuances and choose the best answers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*u5Vg_X8EEa2osvM0rmf1Jg.png" /></figure><p><strong>1. Focus on the Best-Fit Service or Feature<br></strong>Remember, this is a certification created by the vendor. AWS wants you to use their services and features. Most questions are designed to test your understanding of which AWS service — or which feature of a service — is the best fit given specific requirements. Always think about the core purpose of each service and how it aligns with the scenario presented. To put it another way, the majority of certification questions will ask you to do one of two things: pick the <strong>service </strong>that is the best fit given the requirements, or pick the <strong>feature </strong>of a service that is the best fit given the requirements. Remembering this may help you avoid picking a distractor.</p><p><strong>2. Identify Key Services in the Question<br></strong>Pay attention to the AWS services mentioned in the question. Assess whether the options provided have features that address the problem. If they don’t, consider other native AWS services that might be a better fit. If none of the native services seem suitable, think about serverless options or alternative AWS services that meet the requirements.</p><p><strong>3. Look for Familiar Services or Features<br></strong>If a particular AWS service or feature comes to mind as you read the question, quickly scan the options for it. However, ensure you also understand any special requirements that might influence your choice.</p><p><strong>4. Eliminate Incorrect Options First<br></strong>One effective strategy is to eliminate options that are clearly incorrect. If you’re well-versed in the material, you can narrow down multiple-choice questions by dismissing distractors or suboptimal answers, leaving you with the two closest options. When taking practice tests, be sure that when you make mistakes, almost all of them are narrowed down to two choices. If not, you probably have a gap, need to study a topic some more.</p><p><strong>5. Avoid Overcomplicating Solutions<br></strong>Rarely will you need to build a complicated solution from scratch. Be cautious of options involving third-party solutions — they’re often, but not always, incorrect. If a solution uses AWS services in an overly complex way or outside their core purpose, it might not be the right choice.</p><p><strong>6. Pay Attention to Multiple-Select Questions<br></strong>Some questions require you to select a sequence of steps or multiple correct answers. For instance, a question might ask for three steps in order, but provide six options. You’ll need to mentally sort out at least one correct set of steps. Remember, the options are often randomized, so they won’t be grouped together for you. Make sure you know whether the question is asking for a <em>sequence </em>or <em>just an unordered list</em> of possible solutions.</p><p><strong>7. Recognize Different Implementation Methods<br></strong>Anything you can do from the AWS Management Console can also be done using the AWS CLI or SDKs. The exam sometimes tests your ability to recognize automated solutions. If specific keywords throw you off, consider the scope of the AWS services involved and whether they’re being used appropriately.</p><p><strong>8. Understand the Requirement’s Core Adjective<br></strong>Be clear on whether the question is seeking the most “operationally efficient,” “cost-efficient,” “performant,” “scalable,” “highly available,” “reliable,” “best long-term” solution — or a combination of these. This understanding of the <em>core adjectives</em> in the question’s stem can significantly influence your answer. If you rush through your questions, it can be an easy miss.</p><p><strong>9. Choose the Best Fit, Not Just the Best-Sounding Solution<br></strong>Sometimes the most appealing solution isn’t the right one given the specific requirements. “Best” means the best fit for the scenario, not necessarily the most comprehensive or advanced option. Put your analyst or architect hat on: think about trade-offs and fitness to meet the requirements.</p><p><strong>10. Watch for Crucial Keywords and Phrases<br></strong>One word or phrase can change the entire solution. Missing it might lead you straight to an incorrect answer that seems right at first glance. Read carefully to catch these critical details. Some options might seem like a perfect fit except for one small detail. That detail — sometimes just one word — can be easy to overlook, especially if you’re rushing. Take the time to scrutinize each option thoroughly.</p><p><strong>11. Use the Options to Understand the Question<br></strong>You don’t always have to read the entire question before looking at the options. On the more difficult exams you may run out of time if you do that. Scanning the options first can provide context and might help you understand the question better, especially if you need a second pass to grasp it fully.</p><p><strong>12. Don’t Read Big Questions Serially</strong><br>This is related to the previous tip. The intermediate and especially the advanced exams can feel like speed tests, so if a question is lengthy, avoid reading it linearly. Blur your eyes or take off your glasses. When the options are detailed and similar, you’re in luck: focus on the differences (keywords) to quickly eliminate a couple of choices. Then start to read to understand a bit more. For long stems (the body of the question without the options), identify key requirements and map them to the options. If an option meets some but not all requirements, it can be eliminated. For multiple-select questions, ensure no requirements are left unmet. If you’re stuck, choose the least bad option, but double-check to see if you’ve missed anything.</p><p>Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c2a4173aecf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>