Jira Cloud’s API renders Markdown as literal text. Here’s why — and a dependency-free Python script that converts Markdown into real Atlassian Document Format (ADF) JSON.

If you’ve ever posted a comment to a Jira Cloud ticket through an API call, a bot, or an AI agent, you’ve probably seen this:

**Root cause:** the retry logic in the `sync` worker
| Step | Status |
|---|---|
| 1 | done |
| 2 | failed |

…render as exactly that. Literal asterisks. Literal backticks. Literal pipes and dashes. No bold, no monospace, no table.

I ran into this while building an automation script that posts status updates and structured summaries to Jira tickets as part of an agent workflow. Every comment came out looking like Markdown wearing a costume — the syntax was there, but Jira just printed it as plain characters. This is the story of why that happens, and the small script that fixes it, verified live against a real ticket.

Key takeaways

• Jira’s comment API doesn’t parse Markdown or wiki-markup at all. It wraps whatever string you send as one literal text node — that’s true whether you write **bold**, ||Header||, or anything else.
• The real fix is ADF (Atlassian Document Format), the JSON tree Jira’s own editor produces. Build that JSON and POST it to the REST API v3 comment endpoint instead of v2.
• A ~300-line, dependency-free Python script can convert a Markdown file into valid ADF — headings, bold/italic/code, links, tables, ordered/bullet lists, task lists, fenced code blocks, and Jira’s colored panels.
• Task-list checkbox state is not scoped to your document. It’s tracked by a site-wide realtime service keyed on localId — use a real UUID per item, not a sequential counter, or checkboxes can silently inherit an unrelated task's checked state.
• You can preview ADF JSON without posting to a real ticket, using Atlassian’s own Document Viewer tool.

The problem: Markdown renders as literal text in Jira

I tested every reasonable variant before concluding it wasn’t a syntax mistake on my end:

• **bold** → shows as literal asterisks, not bold.
• `inline code` → literal backticks, no monospace.
• # Heading → literal hash characters, no heading.
• A Markdown pipe table (| a | b | + |---|---|) → literal pipes and dashes, no grid.
• Jira’s own wiki-markup table syntax (||Header||) → also literal.
• Only genuine Unicode characters (emoji) rendered correctly, because they aren’t markup at all — just characters.

That last point was the tell. Whatever was posting these comments wasn’t failing to parse one particular markup dialect — it wasn’t parsing any markup. It was treating the entire string as inert text.

Why this happens: Jira REST API v2 vs. v3

Jira Cloud’s REST API has two comment endpoints that are easy to conflate:

• v2 — expects a plain string, optionally in the legacy wiki-markup dialect.
• v3 — expects ADF (Atlassian Document Format): a JSON document tree. This is the exact same format Jira’s own ProseMirror-based comment editor produces when a human types in the browser.

Whatever tool or integration sits between “a string I want to post” and Jira usually wraps that string as a single literal ADF text node and sends it to the v3 endpoint. It never parses Markdown or wiki-markup first — it just boxes the raw string. So any syntax you throw at it renders literally, regardless of which flavor you used, because nothing downstream is looking for syntax at all.

The only real fix: build actual ADF JSON yourself and post it to the v3 endpoint directly.

The fix: a Markdown-to-ADF converter in Python

I wrote a single Python file, pure standard library, no dependencies, that reads a Markdown file and converts it to an ADF doc tree. It can either print the resulting JSON for inspection (--dry-run) or POST it directly as a real comment:

POST /rest/api/3/issue/{key}/comment
Authorization: Basic base64(email:api_token)
Body: {"body": <adf-json>}

Auth follows the standard Jira Cloud API-token pattern — an account email plus a token generated at id.atlassian.com, and a base site URL.

How it works: from Markdown text to an ADF tree

At a high level, the converter runs in two passes:

1. Block splitting — walk the Markdown text line by line and group runs of lines into typed blocks: heading, paragraph, table, bullet/ordered list, task list, panel, fenced code, rule.
2. Block-to-ADF conversion — turn each typed block into the matching ADF node, recursively parsing inline formatting (bold, code, italics, links) inside any text content.

The block splitter is one function that looks just far enough ahead to tell block types apart — a table needs to peek at the next line for a |---|---| separator, a panel needs to scan forward to its closing :::, a fenced code block needs to consume everything up to the next fence:

HEADING_RE = re.compile(r"^(#{1,6})\s+(.*)$")
RULE_RE = re.compile(r"^(-{3,}|\*{3,}|_{3,})\s*$")
TASK_RE = re.compile(r"^[-*+]\s+\[([ xX])\]\s+(.*)$")
BULLET_RE = re.compile(r"^[-*+]\s+(.*)$")
ORDERED_RE = re.compile(r"^\d+\.\s+(.*)$")
TABLE_SEP_RE = re.compile(r"^\s*\|?[\s:\-|]+\|?\s*$")
PANEL_START_RE = re.compile(r"^:::panel\s+(info|note|warning|success|error)\s*$")
PANEL_END_RE = re.compile(r"^:::\s*$")
BLOCK_START_RE = re.compile(
    r"^(#{1,6}\s+|```|:::|[-*+]\s+|\d+\.\s+|(-{3,}|\*{3,}|_{3,})\s*$)"
)

def split_into_blocks(md_text):
    lines = md_text.split("\n")
    n = len(lines)
    blocks = []
    i = 0
    while i < n:
        line = lines[i]
        if line.strip() == "":
            i += 1
            continue
        if line.startswith("```"):
            lang = line[3:].strip()
            i += 1
            code_lines = []
            while i < n and not lines[i].startswith("```"):
                code_lines.append(lines[i])
                i += 1
            i += 1
            blocks.append(("code", lang, code_lines))
            continue
        if HEADING_RE.match(line):
            blocks.append(("heading", line))
            i += 1
            continue
        if RULE_RE.match(line.strip()):
            blocks.append(("rule",))
            i += 1
            continue
        if "|" in line and i + 1 < n and TABLE_SEP_RE.match(lines[i + 1]) and "-" in lines[i + 1]:
            table_lines = [line, lines[i + 1]]
            i += 2
            while i < n and "|" in lines[i] and lines[i].strip() != "":
                table_lines.append(lines[i])
                i += 1
            blocks.append(("table", table_lines))
            continue
        panel_m = PANEL_START_RE.match(line.strip())
        if panel_m:
            panel_type = panel_m.group(1)
            i += 1
            inner_lines = []
            while i < n and not PANEL_END_RE.match(lines[i].strip()):
                inner_lines.append(lines[i])
                i += 1
            i += 1  # skip the closing ':::'
            blocks.append(("panel", panel_type, inner_lines))
            continue
        if TASK_RE.match(line):
            list_lines = []
            while i < n and TASK_RE.match(lines[i]):
                list_lines.append(lines[i])
                i += 1
            blocks.append(("task_list", list_lines))
            continue
        if BULLET_RE.match(line):
            list_lines = []
            while i < n and BULLET_RE.match(lines[i]):
                list_lines.append(lines[i])
                i += 1
            blocks.append(("bullet_list", list_lines))
            continue
        if ORDERED_RE.match(line):
            list_lines = []
            while i < n and ORDERED_RE.match(lines[i]):
                list_lines.append(lines[i])
                i += 1
            blocks.append(("ordered_list", list_lines))
            continue
        para_lines = [line]
        i += 1
        while i < n and lines[i].strip() != "" and not BLOCK_START_RE.match(lines[i]):
            para_lines.append(lines[i])
            i += 1
        blocks.append(("paragraph", para_lines))
    return blocks

Every branch does the same three things: recognize a line pattern, consume however many following lines belong to that block, and append a tagged tuple describing it. Order matters — fenced code blocks are checked first, so a stray # or | character inside a code block is never mistaken for a heading or a table.

Each tagged block then flows through a small dispatcher that maps the tag to its own builder function:

def block_to_adf(block):
    kind = block[0]
    if kind == "heading":
        m = HEADING_RE.match(block[1])
        level = len(m.group(1))
        return {"type": "heading", "attrs": {"level": level},
                "content": parse_inline(m.group(2)) or [{"type": "text", "text": ""}]}
    if kind == "rule":
        return {"type": "rule"}
    if kind == "table":
        return block_table(block[1])
    if kind == "bullet_list":
        return block_list(block[1], ordered=False)
    if kind == "task_list":
        return block_task_list(block[1])
    if kind == "ordered_list":
        return block_list(block[1], ordered=True)
    if kind == "panel":
        return block_panel(block[1], block[2])
    if kind == "code":
        _, lang, code_lines = block
        node = {"type": "codeBlock", "content": [{"type": "text", "text": "\n".join(code_lines)}]}
        if lang:
            node["attrs"] = {"language": lang}
        return node
    # paragraph
    text = " ".join(l.strip() for l in block[1])
    content = parse_inline(text) or ([{"type": "text", "text": text}] if text else [])
    return {"type": "paragraph", "content": content}

And the whole pipeline collapses into one top-level function:

def markdown_to_adf(md_text):
    blocks = split_into_blocks(md_text)
    return {"type": "doc", "version": 1,
            "content": [block_to_adf(b) for b in blocks]}

Split into blocks, convert each block, wrap the results in a doc envelope. Each individual builder — table, lists, task lists, panels — is a small, single-purpose function, which is what made it manageable to add ADF's less obvious node types (panels, task lists) without the whole thing turning into a tangle.

Supported Markdown subset

MarkdownADF node/mark# .. ######heading (levels 1–6)**bold**strong mark*italic* / _italic_em mark`code`code mark[text](url)link mark- item / * itembulletList1. itemorderedList- [ ] item / - [x] itemtaskList (real, clickable Jira checkboxes)| a | b | + |---|---|table (real grid, styled header row)```lang ... ```codeBlock---rule:::panel warning ... :::panel (custom syntax — see below)

One deliberate limitation: nested or overlapping inline marks (bold inside a link, for instance) aren’t supported. Inline formatting stays flat. It’s a small cost for a converter that has no dependencies and fits in one file.

The core of the inline parser is a single regex that tries each mark in order and slices the string around whatever it finds:

INLINE_RE = re.compile(
    r"\*\*(?P<bold>.+?)\*\*"
    r"|`(?P<code>.+?)`"
    r"|\*(?P<italic>.+?)\*"
    r"|_(?P<italic2>.+?)_"
    r"|\[(?P<linktext>[^\]]+)\]\((?P<linkurl>[^)]+)\)"
)

def parse_inline(text):
    nodes = []
    pos = 0
    for m in INLINE_RE.finditer(text):
        if m.start() > pos:
            plain = text[pos:m.start()]
            if plain:
                nodes.append({"type": "text", "text": plain})
        if m.group("bold") is not None:
            nodes.append({"type": "text", "text": m.group("bold"),
                          "marks": [{"type": "strong"}]})
        elif m.group("code") is not None:
            nodes.append({"type": "text", "text": m.group("code"),
                          "marks": [{"type": "code"}]})
        # ...italic, link handled the same way
        pos = m.end()
    if pos < len(text):
        nodes.append({"type": "text", "text": text[pos:]})
    return nodes

Nothing exotic — it’s the block-level splitting (headings, tables, lists, fenced code, panels) that does the real structural work, walking the document line by line and grouping runs of related lines into blocks before handing each block to its own converter function.

Colored panels: the nicest surprise in the ADF schema

ADF has a panel node type that matches, one-to-one, the five panel types in Jira's own comment-editor dropdown: Info, Note, Success, Warning, Error — each with its own icon and background color (blue, purple, green, yellow, red respectively).

Markdown has no native syntax for a colored callout box, so the script invents a small one:

:::panel warning
This is a **warning panel** — inline formatting still works inside it.
:::

That gets parsed into:

{
  "type": "panel",
  "attrs": { "panelType": "warning" },
  "content": [
    {
      "type": "paragraph",
      "content": [
        { "type": "text", "text": "This is a " },
        {
          "type": "text",
          "text": "warning panel",
          "marks": [{ "type": "strong" }]
        },
        {
          "type": "text",
          "text": " — inline formatting still works inside it."
        }
      ]
    }
  ]
}

The panelType attribute has to be one of info, note, success, warning, or error — anything else is treated as a script error rather than silently ignored, which is the right failure mode for a formatting tool: better to fail loud than post a broken panel.

The builder function is a two-line recursion — it re-runs the same block splitter on the lines between :::panel warning and :::, so a panel can contain anything a top-level document can (a paragraph, a list, even a table), not just plain text:

def block_panel(panel_type, inner_lines):
    inner_blocks = split_into_blocks("\n".join(inner_lines))
    content = [block_to_adf(b) for b in inner_blocks] or [{"type": "paragraph", "content": []}]
    return {"type": "panel", "attrs": {"panelType": panel_type}, "content": content}

Task lists: real checkboxes, and a bug that only showed up live

GFM-style - [ ] / - [x] becomes a real, independently clickable Jira checklist — a taskList node containing taskItem nodes with an attrs.state of "TODO" or "DONE".

Here’s where it got interesting. The first version of the script generated each taskItem's required localId from a simple incrementing counter: local-1, local-2, and so on. I posted a checklist with one item checked and one unchecked. Both came back checked.

The root cause turned out to be genuinely surprising: task-item checked/unchecked state in Jira and Confluence isn’t purely a property of the document content sitting in that one comment. It’s synced through a realtime “tasks and decisions” service that’s keyed by localId — and that key is site-wide, not scoped to the specific comment or document that declared it.

A low-cardinality, guessable ID like "local-1" can collide with some completely unrelated task elsewhere on the same Jira site. When that happens, the new task item silently inherits that other task's checked state instead of the one it was actually posted with.

The fix was a one-line change:

def _local_id():
    # Task-item checked/unchecked state is tracked by Jira's realtime "tasks
    # and decisions" service keyed on localId, SITE-WIDE, not scoped to this
    # comment/doc -- a low-cardinality id like "local-1" can collide with an
    # unrelated task elsewhere and silently inherit its checked state.
    # A real UUID makes collision practically impossible.
    return str(uuid.uuid4())

Swapping a sequential counter for uuid.uuid4() per task item makes collision practically impossible. What makes this worth calling out is that it's not the kind of bug you'd catch from reading the ADF schema documentation — the schema doc will happily tell you localId is "a unique identifier" without telling you how site-wide that uniqueness requirement actually is. It only surfaced by posting a real checklist to a real ticket and looking at what rendered.

Here’s the full builder, with the fix in place — note that both the taskList wrapper and every taskItem inside it need their own localId:

def block_task_list(lines):
    items = []
    for line in lines:
        m = TASK_RE.match(line.strip())
        checked, text = m.group(1), m.group(2)
        state = "DONE" if checked.lower() == "x" else "TODO"
        items.append({"type": "taskItem", "attrs": {"localId": _local_id(), "state": state},
                       "content": parse_inline(text)})
    return {"type": "taskList", "attrs": {"localId": _local_id()}, "content": items}

Bare URLs don’t auto-link

[text](url) becomes a real, clickable link mark, exactly as you'd expect. A bare https://... pasted directly into the Markdown source, however, does not turn into a link — the inline-parsing regex only matches the explicit [text](url) syntax, so an unwrapped URL is left as plain text in the output.

The practical rule: always wrap URLs explicitly, even when it feels redundant —

[PR #482](https://github.com/your-org/your-repo/pull/482)

rather than pasting the raw link and hoping the renderer is smart about it.

Seeing it rendered before you post

You don’t have to post to a real ticket just to check whether your ADF is correct. Atlassian publishes two small tools on its developer site for exactly this:

• Document builder — a WYSIWYG editor where you type/format content and watch the ADF JSON it produces update live. Useful for reverse-engineering “what does node X look like,” but it does not interpret Markdown — type **bold** into it and you get the literal string **bold**, not a bold mark, because it's building ADF from rich-text editing, not parsing syntax.
• Document viewer — the tool that actually matters here. Paste in ADF JSON and it renders it the same way Jira’s own editor would: panels, tables, task lists, colors and all. This is the fastest way to sanity-check a converter’s output before spending an API call on it.

Here’s a single Markdown source file that exercises most of what the converter supports — a heading, inline bold/code, a warning panel, a table, an ordered list, a task list with both checked and unchecked items, an explicit link, a rule, and a second panel:

# Root Cause: Sync Worker Retry Storm

The `sync` worker was retrying failed jobs **without backoff**, which amplified load during the outage instead of easing it.

:::panel warning
This fix changes retry timing defaults. Any service still reading the old `RETRY_DELAY_MS` env var will fall back to the new default of `2000` instead of `500`.
:::

## Findings

| Metric      | Before | After |
| ----------- | ------ | ----- |
| Retries/min | 4200   | 180   |
| p95 latency | 3.1s   | 640ms |
| Error rate  | 12%    | 0.4%  |

## Fix steps

1. Add exponential backoff to `retry_job()`
2. Cap max retries at `5`
3. Emit a `retry_exhausted` metric for alerting

## Acceptance criteria

- [x] Backoff verified in staging load test
- [x] Metric visible in dashboard
- [ ] Rollout to production behind feature flag

See [PR #482](https://github.com/your-org/your-repo/pull/482) for the implementation.

---

:::panel note
Root cause and fix verified together in staging; production rollout is tracked separately.
:::

Running it through the converter with --dry-run produces a full ADF doc tree — headings, the panel nodes with their panelType attrs, the table with a styled header row, the orderedList, a taskList with per-item UUID localIds, and a link mark on the PR reference. Pasting that JSON into Document viewer renders it exactly as it would look inside a real Jira comment:

The table comes out as a real grid with a styled header row, not the pipe-and-dash text it started as:

And the task list renders as genuine, independently-checked/unchecked boxes, the link is clickable, and the closing panel picks up its own icon and background color for the note type:

That’s the whole workflow in practice: write Markdown, run --dry-run, paste into Document viewer, confirm it looks right, then drop --dry-run and let the script post it for real.

Beyond comments: overwriting a whole field

Comments are additive by nature — POST /issue/{key}/comment never touches anything that's already there. Sometimes what you actually want is to replace a field instead, most commonly a ticket's own description, which tools that create issues through simpler APIs often populate as flat text with hand-rolled section dividers like === Root Cause === instead of real structure.

The script supports this with a --field flag, which instead sends:

PUT /rest/api/3/issue/{key}
Body: {"fields": {"description": <adf-json>}}

This call overwrites the field wholesale — there is no partial-update or append mode for a rich-text ADF field, so it’s a genuinely different operation from posting a comment, not a variant of it.

Used correctly, it turns an old plain-text description — dividers and all — into a real document: headings, an ordered list, a couple of panels (a note for context, a warning for a known limitation), and a task list for acceptance criteria. A follow-up GET on the issue confirms it worked: the updated timestamp moves, and the field now contains a proper ADF document instead of a string with === in it.

Auth and the two HTTP calls

Both operations share the same Basic-auth header, built from an account email and an API token:

def resolve_auth(args):
    base_url = args.base_url or os.environ.get("JIRA_BASE_URL")
    if not base_url:
        site = args.site or os.environ.get("JIRA_SITE")
        if site:
            base_url = f"https://{site}.atlassian.net"
    email = args.email or os.environ.get("JIRA_EMAIL")
    token = args.token or os.environ.get("JIRA_API_TOKEN")
    missing = [name for name, value in (
        ("JIRA_BASE_URL / --base-url (or JIRA_SITE / --site)", base_url),
        ("JIRA_EMAIL / --email", email),
        ("JIRA_API_TOKEN / --token", token),
    ) if not value]
    if missing:
        sys.exit("Missing Jira auth: " + ", ".join(missing))
    return base_url.rstrip("/"), email, token

def _auth_header(email, token):
    basic = base64.b64encode(f"{email}:{token}".encode("utf-8")).decode("ascii")
    return {"Authorization": f"Basic {basic}",
            "Content-Type": "application/json",
            "Accept": "application/json"}

Posting a comment and overwriting a field are then two thin wrappers around urllib.request — no HTTP client dependency needed for two calls:

def post_comment(base_url, email, token, issue_key, adf_body):
    url = f"{base_url}/rest/api/3/issue/{issue_key}/comment"
    payload = json.dumps({"body": adf_body}).encode("utf-8")
    req = urllib.request.Request(url, data=payload, method="POST",
                                  headers=_auth_header(email, token))
    try:
        with urllib.request.urlopen(req) as resp:
            return json.loads(resp.read().decode("utf-8"))
    except urllib.error.HTTPError as e:
        sys.exit(f"Jira API error {e.code}: {e.read().decode('utf-8', errors='replace')}")

def update_field(base_url, email, token, issue_key, field_name, adf_body):
    url = f"{base_url}/rest/api/3/issue/{issue_key}"
    payload = json.dumps({"fields": {field_name: adf_body}}).encode("utf-8")
    req = urllib.request.Request(url, data=payload, method="PUT",
                                  headers=_auth_header(email, token))
    try:
        with urllib.request.urlopen(req) as resp:
            resp.read()  # 204 No Content on success -- nothing to parse
    except urllib.error.HTTPError as e:
        sys.exit(f"Jira API error {e.code}: {e.read().decode('utf-8', errors='replace')}")

And main() just wires the pieces together — read the file, convert it, then decide which of the three modes to run:

def main():
    args = parse_args()  # issue_key, markdown_file, --dry-run, --field, --site, --email, --token
    md_text = sys.stdin.read() if args.markdown_file == "-" \
        else open(args.markdown_file, encoding="utf-8").read()
    adf = markdown_to_adf(md_text)

    if args.dry_run:
        print(json.dumps(adf, indent=2, ensure_ascii=False))
        return

    base_url, email, token = resolve_auth(args)
    if args.field:
        update_field(base_url, email, token, args.issue_key, args.field, adf)
        print(f"Field updated ({args.field}): {base_url}/browse/{args.issue_key}")
    else:
        result = post_comment(base_url, email, token, args.issue_key, adf)
        print(f"Comment posted: {base_url}/browse/{args.issue_key}?focusedCommentId={result.get('id')}")

That’s the entire tool: one pass to turn Markdown into an ADF tree, one small auth/HTTP layer to send it, and a three-way branch (--dry-run / --field / plain comment) to decide what to do with the result.

Windows-specific gotchas

A handful of friction points are worth a short aside, even though they’re incidental to the Jira/ADF problem itself:

• python3 isn't on PATH in Git Bash by default on Windows — it silently resolves to a Microsoft Store stub. Use the full interpreter path instead (e.g. .../AppData/Local/Programs/Python/Python311/python.exe).
• Printing the --dry-run JSON straight to a Windows console can throw a UnicodeEncodeError on emoji, because the default console codepage is cp1252, not UTF-8. Set PYTHONIOENCODING=utf-8, or redirect the output to a file and read that back instead.
• Running the script from a UNC path (\\wsl.localhost\...) while also passing a UNC path as an argument can double up the path, because the process's own working directory gets prefixed onto an already-absolute path. Simplest fix: copy both the script and the input file to a plain local path first.
• Occasional RemoteDisconnected errors on the POST call — a transient network reset, not something specific to this script. The request can still have succeeded server-side even though the client-side response read failed, so a naive "retry on any error" loop can end up double-posting. Check the target ticket for an already-landed comment before retrying.

FAQ

Why doesn’t Markdown work in Jira comments?

Because the comment API doesn’t parse Markdown (or wiki-markup) at all. It posts whatever string it receives as a single literal ADF text node, so any formatting syntax you write shows up as plain characters.

What is Atlassian Document Format (ADF)?

ADF is the JSON document tree Jira and Confluence use internally to represent rich text — the same structure their own ProseMirror-based editors produce when a person types in the browser. Headings, bold/italic, tables, lists, panels, and more each map to a specific ADF node or mark type.

How do I post formatted content to Jira via the API?

Build an ADF JSON document and POST it to the REST API v3 comment endpoint (/rest/api/3/issue/{key}/comment), not v2, which only accepts plain strings or wiki-markup.

Can I preview ADF output without posting to a real Jira ticket?

Yes — paste the ADF JSON into Atlassian’s Document Viewer developer tool, which renders it exactly as Jira’s own editor would, without touching a live ticket.

Does Jira support Markdown at all?

Jira’s newer editor accepts some Markdown-like shortcuts when a human types them interactively (e.g. typing ** around text), but that's editor-level autoformatting, not something the REST API applies to a string you send it programmatically.

Why did my Jira task-list checkboxes show the wrong checked state?

If taskItem nodes use low-cardinality, sequential localId values, they can collide with unrelated tasks elsewhere on the same Jira site, because checkbox state is tracked by a site-wide realtime service keyed on that id — not scoped to your document. Use a real UUID per task item instead.

The takeaway

A few hundred lines of dependency-free Python turn Jira comments — and, with --field, whole issue fields — from "flat text wearing Markdown as a costume" into the same rich content a human gets from the real editor: tables, task lists, colored panels, clickable links, all driven entirely by an agent or script writing plain Markdown.

The interesting part isn’t really the ADF schema itself — that’s just JSON, and it’s reasonably well documented by Atlassian. The interesting part is that the state behind something as simple as a checkbox turned out not to live where it looks like it lives — not in the document you just posted, but in a site-wide realtime service keyed by an ID you control. And the only way to find that out was to actually post something to a real ticket and look carefully at what rendered.

If you’re building any kind of bot, agent, or integration that writes to Jira, that’s the general lesson worth taking away: read the schema, but don’t trust it as the full picture. Post something small, look at the result, and let the live behavior correct your mental model.

Tags: Jira, Atlassian, API, Python, Developer Tools, Automation, JSON

Search for “keyword extraction in Python” and you will find hundreds of tutorials: TF-IDF with scikit-learn, RAKE, YAKE, TextRank, KeyBERT. Search for “automatic term extraction” and you will find almost nothing outside the academic literature — where the task has a formal definition, a manually annotated multilingual benchmark, three survey papers published between 2023 and 2026, and an active research frontier.

This asymmetry is unfortunate, because the two tasks are not the same, and the harder one is better studied. Automatic Term Extraction (ATE) is, in the words of the field’s most recent survey, “a Natural Language Processing (NLP) task intended to automatically identify specialized terms present in domain-specific corpora,” where terms function “as units of knowledge in a specific field of expertise” (Tran et al., 2026). The output is not a per-document summary; it is a termbase — the vocabulary that constitutes a domain.

This article surveys the field and implements it. We derive the C-value algorithm from first principles, build a complete extraction pipeline in dependency-free Python, run it on a wind-energy corpus, and then subject the output to an honest error analysis. We then trace what the research literature has established since: the ACTER benchmark that made ATE systems comparable, the Transformer models that displaced feature engineering, and the recent and genuinely nuanced findings about where large language models help and where they do not.

Principal findings:

• Terms and keywords answer different questions; conflating them produces systems that optimise the wrong objective.
• The C-value algorithm (Frantzi et al., 2000) remains in use in 50% of published ATE studies, twenty-five years after its introduction (Blandón Andrade et al., 2026).
• Transformer token classification is the current state of the art, reaching 58.3–69.6 F1 on the ACTER benchmark against 54.2–66.1 for the best feature-engineered system (Tran et al., 2023).
• LLM in-context learning matches fine-tuned encoders under domain shift (60.2 vs. 61.2 F1) but trails them badly in-domain (53.8 vs. 88.5 F1) (Chun et al., 2025).
• The nested-term problem, diagnosed in 1998, remains an open challenge for neural systems in 2026.

▶ Run the code while you read. Every pipeline stage in this article is worked through cell by cell in the companion notebook: Automatic Term Extraction from Scratch on Kaggle · source on GitHub. The tables and figures below are its actual output.

1. Terms Are Not Keywords

The distinction is not pedantic; it determines the entire system design.

Keyword extraction operates on a single document and asks: which words best summarise this text? The reference point is the document itself. TF-IDF answers it by contrasting the document against the rest of the collection; KeyBERT answers it by measuring the cosine similarity between candidate phrases and the document embedding. Both are, structurally, summarisation signals.

Term extraction operates on a domain corpus and asks: which expressions constitute the specialised vocabulary of this field? The reference point is general language. A term such as doubly fed induction generator is a term regardless of which wind-energy document it appears in, and regardless of whether it summarises that document.

The consequences are concrete. A TF-IDF pipeline over a unigram vocabulary cannot, in principle, emit doubly fed induction generator as a single unit. A KeyBERT ranking will favour phrases that summarise a document over phrases that define a field. Neither models the relationship between wind farm and offshore wind farm.

1.1 What counts as a term?

The question resists a purely theoretical answer, which is why the ACTER annotation scheme is valuable: it operationalises the definition into categories that human annotators can apply consistently (Rigouts Terryn et al., 2020).

This four-way split explains a convention that otherwise looks arbitrary in the literature: ATE papers report two sets of scores, one for terms only and one for terms plus named entities. Named-entity recognition and term extraction overlap without coinciding, and the benchmark refuses to pretend otherwise.

2. The Canonical Pipeline

Nearly all classical ATE systems share a two-stage architecture: a linguistic filter that proposes syntactically plausible candidates, followed by statistical ranking that scores them. The systematic literature mapping of Blandón Andrade et al. (2026) found this hybrid design in 50% of the 113 papers they reviewed, against 38.2% purely statistical and 11.8% purely linguistic.

The ranking stage is conventionally decomposed into two distinct properties:

• Unithood — is this sequence of words a coherent lexical unit, or an accidental collocation?
• Termhood — assuming it is a unit, is it specific to this domain, or general language?

C-value addresses unithood. Weirdness and TF-IDF address termhood. Systems that measure only one produce characteristic failures: unithood alone accepts frequent but general phrases; termhood alone accepts fragments of terms.

3. Stage One: The Linguistic Filter

Frantzi et al. (2000) restrict candidates to part-of-speech patterns, offering three filters of increasing permissiveness:

The choice is a precision/recall dial. The closed filter yields fewer, cleaner candidates; the open filter captures prepositional terms such as quality of service at the cost of substantial noise.

For a dependency-free implementation we approximate the POS filter with a stopword-boundary chunker, which exploits the fact that English terms rarely span a function word. Within each stopword-free chunk of a sentence, every n-gram up to length four becomes a candidate:

def candidates(text, min_len=1, max_len=4):
    """Inside each stopword-free chunk of a sentence, emit every n-gram."""
    out = []
    for sentence in SENT_SPLIT_RE.split(text):
        chunk = []
        for token in tokenize(sentence) + [None]:   # None flushes the chunk
            if token is None or token in STOPWORDS or len(token) < 3:
                for n in range(min_len, max_len + 1):
                    for i in range(len(chunk) - n + 1):
                        out.append(tuple(chunk[i:i + n]))
                chunk = []
            else:
                chunk.append(token)
    return out

This is the same candidate-generation strategy RAKE uses. The difference between RAKE and ATE lies entirely in what happens next.

4. Stage Two: C-value and the Nested-Term Problem

4.1 The problem

Raw frequency systematically over-ranks substrings. If offshore wind farm occurs twice, then wind farm necessarily occurs at least twice as well, and a frequency-ranked list will credit the shorter string with occurrences it never independently earned. Frantzi et al. (2000) named these nested terms and designed C-value specifically to handle them.

4.2 The formula

Three mechanisms operate simultaneously. The log₂|a| factor rewards length, encoding the empirical observation that multi-word expressions are more likely to be terms than single words. The f(a) term preserves frequency as evidence. The subtraction removes the frequency that a merely inherited from longer terms containing it — the average frequency of every longer candidate in which a is nested.

4.3 Implementation

def c_value(freqs, min_freq=2):
    longer_containing = defaultdict(list)
    by_len = defaultdict(list)
    for cand in freqs:
        by_len[len(cand)].append(cand)

    for length, shorter_list in by_len.items():
        for shorter in shorter_list:
            for longer_len in range(length + 1, MAX_LEN + 1):
                for longer in by_len.get(longer_len, ()):
                    if _contains(longer, shorter):
                        longer_containing[shorter].append(freqs[longer])

    scores = {}
    for cand, freq in freqs.items():
        if freq < min_freq:
            continue
        length_weight = math.log2(len(cand)) if len(cand) > 1 else 0.5
        nested_in = longer_containing.get(cand)
        if nested_in:
            score = length_weight * (freq - sum(nested_in) / len(nested_in))
        else:
            score = length_weight * freq
        if score > 0:
            scores[cand] = score
    return scores

4.4 The correction, observed

Running this on the wind-energy corpus makes the mechanism visible.

Consider the three candidates wind farm, offshore wind farm and axis wind. All three occur; frequency alone ranks wind farm (f = 5) far above the others. C-value redistributes the evidence: wind farm retains a score of 3.86 rather than 5, because two of its five occurrences were inside offshore wind farm. Meanwhile axis wind — which in this corpus appears almost exclusively inside axis wind turbine — is discounted from f = 2 to a C-value of 0.75, and falls to the bottom of the ranking. The algorithm has, without supervision, distinguished an independent term from a fragment.

4.5 NC-value: the context extension

Frantzi et al. extend C-value with context evidence. Term context words are learned from the top-ranked candidates and weighted by the proportion of candidate terms they co-occur with, and the final score combines the two sources of evidence:

Here t(w) is the number of candidate terms that context word w appears with, n the total number of candidates considered, Cₐ the set of distinct context words of a, and f_a(b) the frequency of b as a context word of a.

The hand-tuned 0.8/0.2 weighting is a period detail worth noting: it illustrates how differently NLP systems were constructed before end-to-end learning.

5. Stage Three: Termhood via Contrastive Scoring

C-value measures unithood but is blind to domain specificity. A phrase can be a perfectly coherent unit and still be general language. The standard remedy is weirdness (Ahmad et al.), which contrasts the technical corpus against a general-language reference:

def weirdness(tech_tokens, general_tokens, smoothing=0.5):
    f_tech, f_gen = Counter(tech_tokens), Counter(general_tokens)
    n_tech, n_gen = max(len(tech_tokens), 1), max(len(general_tokens), 1)
    return {
        w: (f / n_tech) / ((f_gen[w] + smoothing) / n_gen)
        for w, f in f_tech.items()
    }

For multi-word candidates the word-level scores are aggregated by geometric mean, so that a single anomalous word cannot carry an entire phrase. The final termhood score combines both dimensions, with a logarithmic dampener so that weirdness re-ranks rather than dominates:

termhood = c_value_score * math.log1p(weirdness_score)

Plotting the two axes against each other shows the structure the pipeline is exploiting.

Valid multi-word terms — wind turbine, wind farm, wind speed — occupy the upper-right quadrant, scoring well on both dimensions. General vocabulary such as power and speed achieves moderate unithood but low weirdness, and is correctly demoted. Fragments such as axis wind score low on unithood despite respectable weirdness. The two measures are complementary precisely because they fail in different directions.

6. Use Case: Building a Wind-Energy Termbase

6.1 Scenario

A technical translation provider receives several hundred pages of wind-energy documentation. Before translation begins, a termbase must be assembled so that every translator renders pitch control identically. Constructing this by hand occupies a terminologist for days. ATE produces the first draft in seconds.

The domain is not arbitrary: wind energy is one of the four ACTER domains, so results here map directly onto the standard benchmark.

6.2 Results

The complete implementation is dependency-free and runs on any Python 3 installation. Every step below — candidate generation, C-value, weirdness, and the error analysis in section 7 — is worked through cell by cell in the companion notebook:

▶ Run it yourself: Automatic Term Extraction from Scratch on Kaggle · source on GitHub

Seven of the top ten candidates are valid domain terms. Two exhibit boundary errors — offshore wind is a fragment of offshore wind farm, and axis wind turbine should be horizontal axis wind turbine. One, axis wind, is spurious.

A 70% precision rate at rank 10, achieved without training data, annotation, or any model larger than a Counter, is a reasonable result. But the failures are more instructive than the successes.

7. Error Analysis

Two systematic failure modes appear, both of which anticipate findings in the research literature.

The frequency threshold destroys recall. The pipeline uses min_freq = 2, discarding every candidate occurring once. Panel A quantifies the cost: candidates fall from 98 at min_freq = 1 to 33 at min_freq = 2. Among the casualties are pitch control, capacity factor, power curve and rotor diameter — all genuine, high-value terms that happen to occur once in this corpus. Frequency-based methods are structurally incapable of recovering rare terms, and this is precisely the dimension on which TermEval 2020 found systems to differ most sharply: some participants were "exceptionally good at finding rare terms" while others were not (Rigouts Terryn et al., 2020).

A term can also be lost before any scoring happens. Yaw control never appears in the output either, but for a different and more instructive reason: it is never generated as a candidate at all. A line break falls between yaw and control in the source text, and the sentence splitter treats \n as a boundary, so the bigram is never formed. A candidate the linguistic filter fails to propose cannot be recovered by any ranking function, however sophisticated — which is why the filter stage deserves as much attention as the algorithm, and why production systems use a real sentence segmenter rather than a regular expression.

Single words crowd the ranking. Panel B shows that six of the top twelve candidates are unigrams, with wind occupying first place at more than twice the score of any multi-word term. This is why production ATE tools expose a have_single_word switch — PyATE included — and why the parameter is not a convenience but a necessity.

Two further observations deserve emphasis. First, the boundary errors in Table 3 are the nested-term problem re-emerging in the opposite direction: C-value successfully demotes axis wind, but nothing in the pipeline knows that axis wind turbine is itself incomplete. Second, weirdness is only as good as the contrast corpus, and a 200-word toy general corpus is far too small; production systems contrast against Wikipedia-scale references.

8. Evaluation: The ACTER Benchmark

Before 2020, ATE results were effectively incomparable. Each research group evaluated on its own corpus, with its own operational definition of “term.” TermEval 2020 changed this by releasing ACTER — the Annotated Corpora for Term Extraction Research — with the explicit aim of providing “a large, transparent, qualitatively annotated, and diverse dataset to the ATE research community” (Rigouts Terryn et al., 2020).

Two design decisions make ACTER effective as a benchmark. The heart-failure domain is held out entirely, so reported F1 measures cross-domain generalisation rather than in-domain fitting. And the corpus is multilingual by construction (English, French, Dutch), enabling the cross-lingual transfer experiments that dominate recent work. The corruption subcorpus is parallel; the remaining three are comparable corpora matched for size, topic and register.

Evaluation itself uses standard metrics against the gold annotation:

Tran et al. (2023) note that historically “the number of studies using just Precision is higher than the number of studies that evaluate the system according to all three evaluation criteria,” a consequence of gold-standard scarcity. Systems evaluated on ACTER report all three.

9. What the Literature Actually Uses

Before turning to neural methods, it is worth establishing what practitioners have in fact adopted. Blandón Andrade et al. (2026) systematically mapped 113 ATE papers published between 2015 and 2022.

Two findings stand out. Hybrid linguistic-plus-statistical designs account for half of all studies, with purely linguistic approaches nearly extinct at 11.8%. And C-value alone appears in 50% of the reviewed work, with TF-IDF second at 31%. An algorithm published in 1998 was, through 2022, the single most widely used ranking function in its field.

The most-used tool was Python’s NLTK — a mild but real divergence from practitioner reality, where spaCy dominates and where the principal ATE library, PyATE, is built on spaCy’s POS tagger.

It bears repeating that this review window closes in 2022, immediately before the developments in the next two sections.

10. The Neural Era: Transformer Token Classification

Tran et al. (2023) surveyed 112 papers selected from an initial pool of 329, and reported comparative results on the ACTER held-out test set. The picture is unambiguous.

Three conclusions follow from these numbers.

Neural models substantially outperform feature engineering. The survey states plainly that “neural models generally outperform machine learning models based on feature engineering by a large margin.” HAMLET — a Random Forest over 152 hand-designed features, and the strongest non-neural system by a wide margin — reaches 54.2–66.1 F1. The XLM-R token classifier reaches 58.3–69.6.

How the task is framed matters more than model size. The same XLM-R encoder scores 45.2–48.5 F1 as a sequence classifier (classifying each candidate n-gram as term or non-term) and 58.3–69.6 as a token classifier (labelling each token in a sequence-labelling formulation). A twenty-point gap arises purely from problem formulation. Sequence classification is additionally expensive, since it requires generating and scoring all candidate n-grams.

Zero-shot cross-lingual transfer is nearly free. The XLM-R token classifier applied zero-shot to an unseen language achieves 58.3–69.8 F1 — matching, and at the top of the range slightly exceeding, its monolingual counterpart. For low-resource languages this is a consequential result: annotating a new language may be unnecessary.

11. The LLM Era: Two Papers, One Decision Rule

The 2023 survey identified prompting as an open opportunity, noting that prompt-based methods “remain underexplored for token-level ATE” (§8.5). That gap closed quickly, and the results are more interesting than the usual narrative allows.

11.1 LLMs win in the few-shot regime

Banerjee et al. (2024) evaluated GPT-3.5-Turbo few-shot in-context learning against a fine-tuned BERT-sized baseline and unsupervised count-based methods. In-context learning outperformed both. The paper also quantified prompt sensitivity — measuring performance variation across prompt templates — which remains an unusually honest contribution in a literature that often reports a single template’s results.

Notably, the authors frame ATE as “a key first step in ontology learning and knowledge graph creation,” a downstream framing shared by Xu et al. (2025) and distinct from the traditional terminology-management perspective.

11.2 Fine-tuned encoders still win in-domain

Chun et al. (2025), published in Findings of ACL, report the complementary result.

Under domain shift on ACTER, the best LLM configuration reaches 60.2 F1 against 61.2 for a task-specific fine-tuned model — statistically a tie. In-domain, the gap widens dramatically: on the BioCreAtIvE gene-mention benchmark, the LLM reaches 53.8 F1 against 88.5 for the fine-tuned encoder.

These findings are not in conflict. They answer different questions, and together they yield a usable decision rule:

LLMs win when labelled data is scarce. Fine-tuned encoders win when it is not.

11.3 The counterintuitive part: retrieve by syntax, not semantics

Chun et al.’s methodological contribution is a retrieval strategy for selecting few-shot demonstrations. The standard approach retrieves semantically similar examples. The authors argue this fails across domains, since when “cross-domain scenarios lack gold term overlap, retrieving semantically similar sentences may not provide useful guidance.”

Their alternative retrieves examples with structurally similar constituency parse trees, using the FastKASSIM tree kernel. The intuition is that a term’s meaning does not transfer across domains, but its shape does: horizontal axis wind turbine and chronic heart failure treatment share no semantics whatsoever, yet share a syntactic skeleton — and that skeleton is what teaches the model where term boundaries fall.

To diagnose what the retrieved demonstrations actually contribute, the authors introduce the Term Overlap Ratio, measuring the proportion of a query’s gold terms that already appear in its retrieved examples:

Three results from this paper deserve wider circulation:

1. Syntactic retrieval achieves a Term Overlap Ratio of zero cross-domain — the retrieved demonstrations contain none of the gold terms — and remains competitive regardless. Structural guidance alone suffices.
2. Semantic retrieval shows a negative correlation between term overlap and F1 cross-domain. The authors attribute this to “words functioning as domain-specific terms in one field but generic in others.” The intuitive move of adding semantic similarity measurably hurts.
3. An unlexicalized PCFG parser outperforms neural CRF+RoBERTa parsers for retrieval, supporting the hypothesis that isolating syntax from semantics is what makes the method work.

The authors are appropriately measured about the overall result, noting that “modest absolute improvements and persistent gaps versus task-specific PLMs underscore inherent limitations of in-context learning for terminology extraction.”

12. Choosing a Method

Use unsupervised statistical methods when no annotated data exists and interpretability matters. Every score decomposes into inspectable components, which matters when a terminologist must justify inclusion decisions.

Use a Transformer token classifier when in-domain annotation is available and accuracy is the objective. Frame the task as sequence labelling, not candidate classification.

Use zero-shot cross-lingual transfer when the target language lacks annotated data but a related language has it.

Use LLM in-context learning in genuine few-shot settings or when entering a domain with no data at all — and select demonstrations by syntactic rather than semantic similarity.

12.1 In practice: PyATE

For classical algorithms, PyATE implements C-value, Basic, Combo Basic, Weirdness and Term Extractor over spaCy POS tagging:

from pyate import combo_basic

print(combo_basic(technical_corpus).sort_values(ascending=False))

import spacy
from pyate.term_extraction_pipeline import TermExtractionPipeline

nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("combo_basic")
doc = nlp(technical_corpus)
print(doc._.combo_basic.sort_values(ascending=False).head(5))

Two caveats. PyATE ships a supervised spaCy model, en_acl_terms_sm, reporting 94.71% F-score, 95.41% precision and 94.03% recall — but trained on ACL computer-science terminology, and, per the library's own documentation, not validated on other domains. Given that cross-domain ACTER results cluster in the 50–70 F1 range, that figure should not be read as general-purpose performance. Second, PyATE targets older spaCy releases; pin your version or use an isolated environment.

13. Open Problems

Nested terms remain unsolved. Tran et al. (2023) list them as an open challenge for Transformer models: current systems “often predict nested shorter terms rather than complete multi-word terms,” struggling to infer from context whether a term stands alone or extends further. This is the same phenomenon Frantzi et al. named in 1998 — and it appeared unprompted in our own use case, where axis wind turbine was extracted instead of horizontal axis wind turbine. Twenty-eight years on, the diagnosis has outlived every proposed cure.

Annotation does not scale. Building domain-specific annotated corpora is “error-prone, labor-intensive, time-consuming, and subjective.” This is what makes zero-shot and few-shot results consequential rather than merely interesting.

Ensembling is under-explored. The 2023 survey recommends combining pretrained language models with the classical statistical measures — termhood, unithood, C-value — rather than treating the paradigms as competitors. Given that C-value contributes an inductive bias about nested structure that no Transformer has yet learned reliably, this seems a promising direction.

14. Conclusion

Automatic Term Extraction has a twenty-five-year literature, a rigorous multilingual benchmark, and a set of results that are more nuanced than the surrounding discourse on keyword extraction suggests. Its central algorithm predates the deep learning era and remains in half of all published work. Its central open problem — deciding where a term ends — was correctly identified in 1998 and has survived the transition to Transformers and then to large language models intact.

For practitioners, the practical conclusions are narrow and defensible. If you are extracting the vocabulary of a domain rather than summarising documents, you are doing ATE, and the field’s accumulated results apply. If you have annotated data, fine-tune a token classifier. If you do not, prompt an LLM and select demonstrations by syntax. And if you need to explain to a domain expert why a particular string was included in their termbase, the 1998 algorithm remains the only option that can answer.

Reproducing This Work

The full pipeline is available as a runnable notebook. There is nothing to install and nothing to download — sections 3 through 8 depend on nothing outside the Python standard library.

Figures 2, 3, 4 and Table 3 are computed directly in the notebook; the values printed in this article are its actual output. Figures 5, 6 and 7 reproduce published results, cited below.

References

Ahmad, K., Gillam, L., & Tostevin, L. Weirdness indexing for logical document extrapolation and retrieval. Proceedings of TREC-8.

Banerjee, S., Chakravarthi, B. R., & McCrae, J. P. (2024). Large Language Models for Few-Shot Automatic Term Extraction. Natural Language Processing and Information Systems (NLDB 2024), Lecture Notes in Computer Science, vol. 14762. Springer, Cham. https://doi.org/10.1007/978-3-031-70239-6_10

Blandón Andrade, J. C., Medina Otálvaro, C. M., Zapata Jaramillo, C. M., & Morales Ríos, A. (2026). Approaches, Tools, Algorithms, and Methods for Automatic Term Extraction: A Systematic Literature Mapping. https://doi.org/10.1177/18758967251392652

Chun, Y., Kim, M., Kim, D., Park, C., & Lim, H. (2025). Enhancing Automatic Term Extraction with Large Language Models via Syntactic Retrieval. Findings of the Association for Computational Linguistics: ACL 2025. https://aclanthology.org/2025.findings-acl.516/ · arXiv:2506.21222

Frantzi, K., Ananiadou, S., & Mima, H. (2000). Automatic recognition of multi-word terms: the C-value/NC-value method. International Journal on Digital Libraries, 3(2), 115–130. https://doi.org/10.1007/s007999900023

Lu, K. PyATE: Python Automated Term Extraction. https://github.com/kevinlu1248/pyate · https://doi.org/10.5281/zenodo.5039290

Rigouts Terryn, A., Hoste, V., & Lefever, E. (2019). In no uncertain terms: a dataset for monolingual and multilingual automatic term extraction from comparable corpora. Language Resources and Evaluation, 54(2), 385–418. https://doi.org/10.1007/s10579-019-09453-9

Rigouts Terryn, A., Hoste, V., Drouin, P., & Lefever, E. (2020). TermEval 2020: Shared Task on Automatic Term Extraction Using the Annotated Corpora for Term Extraction Research (ACTER) Dataset. Proceedings of the 6th International Workshop on Computational Terminology (COMPUTERM 2020), 85–94. https://aclanthology.org/2020.computerm-1.12/

Tran, H. T. H., Martinc, M., Caporusso, J., Doucet, A., & Pollak, S. (2023). The Recent Advances in Automatic Term Extraction: A survey. arXiv:2301.06767. https://arxiv.org/abs/2301.06767

Tran, H. T. H., Martinc, M., Caporusso, J., Delaunay, J., Doucet, A., & Pollak, S. (2026). Recent Advances in Automatic Term Extraction: A Comprehensive Survey. ACM Computing Surveys. https://doi.org/10.1145/3787584

Xu, K., Feng, Y., Li, Q., Dong, Z., & Wei, J. (2025). Survey on terminology extraction from texts. Journal of Big Data. https://doi.org/10.1186/s40537-025-01077-x

Datasets: ACTER — https://github.com/AylaRT/ACTER (CC BY-NC-SA 4.0)

When should you replace regex with Aho-Corasick or FlashText? A rigorous Python benchmark across five string matching methods with complexity analysis, charts, and a practical decision guide

There is a particular kind of performance bug that only shows up at scale. You write a keyword extraction routine, it works fine on your test set of a few hundred documents, you ship it, and three weeks later someone runs it against the full corpus and it does not finish overnight. The culprit, almost every time, is a regular expression loop that nobody thought to profile and the fix is usually the Aho-Corasick algorithm.

This article is a systematic benchmark of the main options available in Python for multi-pattern string matching: pyahocorasick (Aho-Corasick), re.compile with alternation, FlashText, KMP, and plain string methods. Each is measured across the dimensions that actually matter in production throughput, keyword-set scalability, memory overhead, and build cost, so you can make an informed choice rather than a habitual one.

Key findings at a glance:

• Below ~500 keywords, re.compile with alternation is fast enough and the most flexible option.
• Above 500 keywords, Aho-Corasick (pyahocorasick) is 10–100× faster than compiled regex and scales independently of keyword count.
• FlashText matches Aho-Corasick speed but enforces whole-word boundaries-a useful constraint, not a limitation.
• KMP offers no practical advantage over str.find() for multi-pattern use.
• Build time for Aho-Corasick is milliseconds at 10 000 keywords; amortized across a large corpus it is negligible.

The Multi-Pattern Matching Problem

Given a text corpus of arbitrary size and a dictionary of k patterns, find all occurrences of every pattern in every document. This is the canonical multi-pattern string matching problem, and it appears under many names in practice: keyword extraction, named entity detection, log triage, content moderation, sensitive-word filtering, tokenization preprocessing.

The naive solution is to iterate over each pattern independently; this carries a time complexity of O(k × n), where n is the total text length. That is acceptable when k is small. When k grows into the thousands, it becomes the reason your pipeline takes five days instead of fifteen minutes.

The Contenders

1. Python Built-in String Methods

The floor of comparison. Methods like str.find(), in, str.startswith() run at C speed inside CPython but offer no structural advantage for multiple patterns. Each pattern requires a separate traversal of the text.

Complexity: O(k × n), same as naive regex
Advantage: zero dependencies, minimal memory
Limitation: no pattern composition, no partial matching

2. Regular Expressions: Uncompiled

The re.search() / re.findall() calls with a pattern string compiled anew on every invocation. This is the antipattern that inflicts the most damage in real codebases, because the cost of compiling the finite automaton from the pattern string is paid repeatedly.

Complexity: O(k × n) per call, plus compilation overhead
Advantage: expressive syntax
Limitation: hidden compilation cost, regex backtracking on pathological patterns

3. Regular Expressions: Compiled with Alternation (re.compile)

Combining k keywords into a single alternation pattern (word1|word2|...|wordk) and compiling it once is the standard optimization. Under the hood, Python's re module (which uses a backtracking NFA engine) evaluates this as a single automaton pass over the text, but the automaton itself is not optimal for large alternation sets; pathological inputs can trigger exponential backtracking.

Complexity: O(n) average, potentially worse
Advantage: single compilation, flexible pattern syntax
Limitation: backtracking risk, memory usage grows with pattern count, slow to compile for large k

4. Knuth-Morris-Pratt (KMP)

KMP constructs a deterministic finite automaton from a single pattern and achieves O(n + m) matching where m is the pattern length. It is the theoretical predecessor of Aho-Corasick but is not designed for multi-pattern use: running KMP independently for each of k patterns gives back O(k × n) total complexity.

Complexity: O(n + m) per pattern, O(k × (n + m)) total
Advantage: no backtracking, clean theoretical foundation
Limitation: not natively multi-pattern

5. Aho-Corasick

The algorithm by Alfred Aho and Margaret Corasick (1975) generalizes KMP to multiple patterns by building a trie over all patterns and augmenting it with failure links, which are pointers that redirect the automaton to the longest proper suffix of the current state that is also a prefix of some pattern. The result is an automaton that processes the text in a single left-to-right pass, regardless of how many patterns are loaded.

Complexity: O(Σ|p_i| + n + z), where z is the number of matches
Advantage: linear in text length, keyword-count-independent search time
Limitation: memory proportional to total pattern length × alphabet size (when transitions are precomputed); build time proportional to total pattern length

6. FlashText

FlashText (Singh, 2017) is a Python library that implements the same Trie/Aho-Corasick principle but restricts matching to whole-word boundaries. It is optimized for the extract-and-replace use case and exposes a simpler API than pyahocorasick. The word-boundary restriction eliminates false partial matches (it will not match "apple" inside "pineapple") but also means it cannot be used for substring matching.

Complexity: O(n) search, O(Σ|p_i|) build
Advantage: simple API, built-in replace, whole-word semantics
Limitation: no substring matching, no regex-style patterns

Benchmark Setup

All benchmarks were conducted on synthetic data to isolate algorithmic behavior from corpus-specific effects. The experimental variables are:

• Number of keywords (k): 10, 50, 100, 500, 1 000, 5 000, 10 000
• Text length (n): 10 000, 100 000, 1 000 000 characters
• Measurement: wall-clock time via timeit (5 repetitions, best of 3 runs), peak RSS memory via tracemalloc

Keywords were sampled from a fixed vocabulary of common English words; the text corpus was generated by concatenating random sentences from the same vocabulary, ensuring that approximately 15% of patterns appear in any given text sample. Pattern lengths range from 3 to 12 characters.

Algorithm Comparison Summary

Benchmark Results: Speed, Memory, and Build Time

Search Time vs. Keyword Count

The most consequential dimension is how each method scales with the number of patterns. Figure 1 plots search time on a log-log scale across seven keyword-count values from 10 to 10 000, with text length held constant at 100 000 characters.

The separation is stark. str.find() loops, uncompiled regex, and per-pattern KMP scale linearly with k: they all roughly double in cost when the keyword count doubles. At k = 10 000, the gap between these methods and Aho-Corasick spans roughly two to three orders of magnitude.

Compiled regex with alternation occupies a more nuanced position. Because Python’s re module converts the alternation to a single NFA, search time does not scale linearly with k, yet it is not constant either. Large alternation patterns push the NFA into increasingly deep state graphs, and worst-case backtracking behaviour on adversarial inputs can degrade performance unpredictably. On benign English text, compiled regex holds up to around 500–1 000 keywords before Aho-Corasick pulls decisively ahead.

Aho-Corasick and FlashText exhibit essentially flat search time as k grows. This is the theoretical guarantee playing out in practice: the text is traversed once, and the automaton resolves each character transition in O(1). The difference between the two at large k is primarily build-time and memory overhead, not search time.

Search Time vs. Text Length

With keyword count held constant at 1 000, Figure 2 shows how the three fastest methods scale with text length.

All three grow linearly with text length, as their O(n) complexities predict. The important observation here is the constant factor: Aho-Corasick and FlashText consistently process the same character volume in less time than compiled regex, with the gap widening slightly at larger n. This is attributable to the overhead of Python’s backtracking NFA matching loop versus the deterministic transition table lookups of the Aho-Corasick automaton.

Build Time

Build time is the deferred cost of Aho-Corasick’s O(1)-per-character search.

Constructing the automaton takes time proportional to the total length of all patterns, and with precomputed transitions it allocates memory proportional to (states × alphabet size). For k = 10 000 short keywords, this means tens of milliseconds and a few megabytes, which is negligible if the automaton is built once and reused across a large corpus, but relevant in latency-sensitive settings where patterns change frequently.

re.compile becomes notably slow at large k because alternation compilation is not linear in the total pattern length; the regex engine must analyse all possible paths through the NFA. At k = 10 000 keywords, re.compile takes considerably longer than building an equivalent Aho-Corasick automaton.

Memory

Aho-Corasick with precomputed transitions carries the highest memory footprint, driven by storing all state-symbol transitions explicitly. FlashText is more memory-efficient because it builds a Trie without precomputing the full transition table. Compiled regex memory scales roughly linearly with total pattern length.

For most real workloads where the pattern dictionary fits comfortably in RAM, the memory difference is not a practical concern. It becomes relevant only when the vocabulary grows into the hundreds of thousands of entries or when operating in constrained environments.

Speedup Over Compiled Regex

At k = 1 000 keywords on a 100 000-character document, Aho-Corasick and FlashText both deliver search speedups well above 1× over compiled regex, and this ratio widens as k increases. The crossover point (below which compiled regex remains competitive) sits around k = 200–500 in typical conditions, which aligns with the empirical recommendation of 500 keywords cited across multiple practitioner accounts.

Heatmaps: Full k × n Grid

The following three heatmaps present search time (in milliseconds) across the full combination of keyword counts and text lengths tested. Darker cells indicate longer execution times.

The heatmaps make the algorithmic difference concrete. The regex heatmap darkens sharply as both k and n increase, reflecting the O(k × n) regime. The Aho-Corasick and FlashText heatmaps darken only along the text-length axis; keyword count has no visible effect on colour, because search time is indeed independent of k once the automaton is built.

Numeric Results

A Note on Pattern Expressiveness

These benchmarks measure what Aho-Corasick and FlashText are fast at: exact multi-keyword matching. They cannot express the class of patterns that regex handles naturally: character classes, quantifiers, anchors, and lookahead. This is not a shortcoming to be engineered around; it is a deliberate trade-off. A finite automaton that commits to exact pattern strings during the build phase can make strong guarantees about search complexity. A backtracking NFA that interprets patterns at runtime cannot.

The practical implication is that these tools are not alternatives to regex in general; they are alternatives to regex for the specific sub-problem of exact keyword lookup. When the problem is “does this document contain any of these 5 000 product names”, Aho-Corasick is the right tool. When the problem is “extract all strings matching \d{4}-\d{2}-\d{2}", it is not.

Decision Guide: Which Python String Matching Method to Use

Use string methods (str.find, in, str.startswith) when matching a handful of known prefixes or exact strings and the keyword set is static and small. These are the fastest option for k ≤ 10 because there is no overhead at all.

Use compiled regex (re.compile) when patterns require regular-expression syntax (character classes, quantifiers, anchors) or when the keyword count is below roughly 500 and you need the flexibility of partial-word matching. Pre-compile once and cache the object.

Use Aho-Corasick (pyahocorasick) when operating on exact string patterns at k ≥ 500, particularly when the automaton can be built once and reused across a large corpus, or when substring matching is required.

Use FlashText when the use case is whole-word extraction or replacement (entity normalization, synonym standardization, keyword tagging) and partial matches are undesirable. Its enforced word-boundary semantics eliminate an entire class of false-positive bugs.

Use KMP when matching a single pattern against many texts and you cannot introduce a dependency on pyahocorasick. For multi-pattern use it offers no advantage over naive looping.

Practical Recommendations

Build once, search many times. The amortization argument for Aho-Corasick holds only if the automaton is not rebuilt for every document. In a streaming pipeline, build the automaton at startup and hold it for the lifetime of the process. The build cost is a one-time investment; the search cost is paid per document.

The hybrid pattern is underused. For workloads that mix large keyword dictionaries with a small number of complex regex patterns, the most effective approach is to run Aho-Corasick first to identify candidate sentences or paragraphs, then apply regex only to those candidates. This reduces the regex surface area by orders of magnitude while preserving full pattern expressiveness where needed.

Profiling before optimizing. The performance difference between compiled regex and Aho-Corasick matters only when string matching is actually the bottleneck. For short documents and moderate keyword counts, I/O latency, tokenization, and model inference time typically dwarf the search cost. Profile first.

Python Code Examples

Install the two external libraries if you have not already:

pip install pyahocorasick flashtext

str.find() loop

keywords = ["error", "timeout", "failure"]
text = "connection timeout after retry"

for kw in keywords:
    idx = text.find(kw)
    if idx != -1:
        print(f"Found '{kw}' at {idx}")

re.compile with alternation

import re

keywords = ["error", "timeout", "failure"]
pattern = re.compile("|".join(re.escape(k) for k in keywords))

for m in pattern.finditer(text):
    print(f"Found '{m.group()}' at {m.start()}")

Aho-Corasick (pyahocorasick)

import ahocorasick

A = ahocorasick.Automaton()
for idx, word in enumerate(keywords):
    A.add_word(word, (idx, word))
A.make_automaton()

for end_idx, (_, word) in A.iter(text):
    start = end_idx - len(word) + 1
    print(f"Found '{word}' at {start}")

FlashText

from flashtext import KeywordProcessor

kp = KeywordProcessor()
kp.add_keywords_from_list(keywords)

found = kp.extract_keywords(text)
print(found)  # ['timeout']

replaced = kp.replace_keywords(text)

KMP (single pattern, no dependencies)

def kmp_search(text: str, pattern: str) -> list[int]:
    n, m = len(text), len(pattern)
    fail = [0] * m
    j = 0
    for i in range(1, m):
        while j > 0 and pattern[i] != pattern[j]:
            j = fail[j - 1]
        if pattern[i] == pattern[j]:
            j += 1
        fail[i] = j
    results, j = [], 0
    for i in range(n):
        while j > 0 and text[i] != pattern[j]:
            j = fail[j - 1]
        if text[i] == pattern[j]:
            j += 1
        if j == m:
            results.append(i - m + 1)
            j = fail[j - 1]
    return results

print(kmp_search("connection timeout after retry", "timeout"))  # [11]

Conclusion

The Aho-Corasick algorithm has been in continuous production use for fifty years: network intrusion detection systems, genomic sequence search engines, compiler lexers, and most recently the tokenization stages of large language model pipelines. Its longevity is earned. For the specific and common problem of matching many exact patterns against large text, it is the theoretically correct and practically fastest option available, and the Python ecosystem makes it accessible with minimal overhead via pyahocorasick.

None of this makes regular expressions obsolete. The two tools answer different questions. Knowing which question you are actually asking is, in the end, the only benchmark that matters.

References

• Aho, A. V., & Corasick, M. J. (1975). Efficient string matching: An aid to bibliographic search. Communications of the ACM, 18(6), 333–340.
• Knuth, D. E., Morris, J. H., & Pratt, V. R. (1977). Fast pattern matching in strings. SIAM Journal on Computing, 6(2), 323–350.
• Singh, V. (2017). Replace or retrieve keywords in documents at scale. arXiv preprint arXiv:1711.00046.
• Python Software Foundation. (2024). re-Regular expression operations. Python 3 documentation.
• pyahocorasick: https://github.com/WojciechMula/pyahocorasick
• flashtext: https://github.com/vi3k6i5/flashtext

Abstract

Retrieval-Augmented Generation (RAG) has become the dominant paradigm for grounding large language model (LLM) responses in verifiable external knowledge. Yet single-agent RAG pipelines expose a fundamental tension: they conflate retrieval quality, reasoning depth, and answer synthesis into a single, opaque forward pass that is difficult to evaluate, audit, or improve systematically. The system incorporates a self-improving evaluation loop that automatically identifies underperforming prompt dimensions, proposes targeted rewrites, and gates their promotion behind a human approval step with quantitative regression detection. All agent activity is streamed to clients via Server-Sent Events (SSE) and is fully persisted for reproducible auditing. The full source code is publicly available at https://github.com/huseyincenik/multi_agent_rag_system.

1. Introduction

The progression from prototype to production in LLM-based systems is rarely linear. A system that achieves reasonable accuracy on held-out benchmarks frequently degrades in the presence of adversarial inputs, underspecified queries, or distribution shift in user behaviour. Classical software engineering addresses quality regression through unit testing and CI/CD gates; LLM systems require an analogue that operates at the level of prompt behaviour rather than function signatures.

The design described here addresses three interrelated problems:

1. Compositional reasoning. Complex queries benefit from decomposition into dependent sub-tasks, multi-hop retrieval, and explicit contradiction resolution — capabilities that are difficult to reliably embed in a single monolithic prompt.
2. Systematic evaluation. Evaluation must be multi-dimensional, reproducible, and capable of isolating which agent or prompt dimension is responsible for failure — not merely reporting an aggregate accuracy figure.
3. Controlled improvement. Prompt changes should be governed by the same discipline as code changes: proposed in a diff-able format, reviewed by a human, and blocked from promotion if they regress any previously passing dimension.

2. System Architecture

2.1 Overview

The system is composed of five Docker services orchestrated via Docker Compose: a FastAPI application server, an ARQ background worker, PostgreSQL 16 with the pgvector extension, Redis, and an Adminer log browser.

Client (HTTP)
    │  POST /query/stream
    ▼
FastAPI (port 9000)
    │  enqueue ARQ job
    ▼
ARQ Worker (async)
    │
    ├── Orchestrator  →  routing plan (LLM-generated)
    │       │
    │       ├── Decomposition Agent   (sub-task DAG)
    │       ├── RAG Agent             (multi-hop pgvector)
    │       ├── Critique Agent        (per-claim confidence scoring)
    │       └── Synthesis Agent       (provenance-mapped final answer)
    │
    ├── Redis Streams  →  SSE (XADD / XREAD)
    └── PostgreSQL     →  jobs, eval_runs, agent_logs, prompt_versions

All inter-agent communication passes through a SharedContext object with a defined JSON schema. Agents do not call each other directly; the orchestrator mediates every handoff. This constraint eliminates implicit coupling between agents and makes the routing plan fully auditable.

2.2 Framework Stack

FastAPI was chosen for its native async support, automatic OpenAPI documentation generation, and dependency injection system — the latter being critical for clean database session lifecycle management across concurrent requests. ARQ provides a pure-asyncio Redis-backed job queue that integrates naturally with an async codebase without requiring a separate process manager such as Celery.

pgvector enables semantic retrieval directly within PostgreSQL using HNSW indexes on 1536-dimensional embeddings (text-embedding-3-small), eliminating the need for a separate vector database service. Redis Streams (XADD/XREAD) provide durable, ordered event delivery for SSE — superior to in-process queues because events survive worker restarts and can be replayed.

3. Agent Design

3.1 Orchestrator

The orchestrator receives the raw query and the global token budget. It issues a single LLM call to produce a routing plan — a structured JSON document specifying:

• which agents to invoke and in what order,
• the token budget allocated to each agent,
• dependency relationships (expressed as a DAG) between agents,
• an error strategy per agent (retry, skip, or abort).

The orchestrator does not itself answer the query. Its sole function is planning. This separation of concerns means that routing logic can be evaluated and improved independently of answer quality.

3.2 RAG Agent

The RAG agent performs at least two retrieval hops against a pgvector-indexed corpus of document chunks. Hop 1 issues a cosine similarity search against the original query embedding. Hop 2 synthesises a follow-up query from the first hop’s results and issues a second search. Only chunks that contribute to the final answer are cited, and each citation records the source chunk ID and the sentence it supports.

A PolicyViolation is logged if fewer than two hops complete, making single-hop shortcuts visible in the execution trace. This constraint prevents the retrieval agent from taking the degenerate path of returning the top-1 result without verifying coverage.

The embedding model is text-embedding-3-small; the generation model is gpt-4o-mini at temperature=0 for deterministic outputs during evaluation.

3.3 Decomposition Agent

For ambiguous or compound queries, the decomposition agent asks the LLM to produce a typed sub-task DAG: a set of named sub-tasks with typed outputs and explicit dependency edges. The DAG is validated for acyclicity using a topological sort before execution begins. Independent sub-tasks are executed in parallel topological waves; dependent sub-tasks wait for their predecessors to resolve.

3.4 Critique Agent

The critique agent reviews every prior agent’s output stored in SharedContext, producing per-claim confidence scores (0–1) and flagging specific text spans it disagrees with. A flag must reference a non-empty character span (start < end) within the text; whole-output rejections without a span reference are rejected by the Pydantic validator. This design forces the critique agent to be precise and falsifiable.

3.5 Synthesis Agent

The synthesis agent merges all outputs, applies contradiction resolution using critique flags, and produces a final answer with a provenance map linking each output sentence to its source agent and (where applicable) source document chunk. Four resolution strategies are supported:

In SSE mode, the synthesis agent emits agent_token events per OpenAI streaming fragment, giving the client per-token visibility into which agent is writing at any moment.

4. Context Window Management

A ContextBudgetManager tracks cumulative token consumption per agent per turn using tiktoken's CL100K tokenizer. Before any agent appends to its context, it queries the remaining budget. If the assembled context exceeds the declared budget, a compression agent is invoked to summarise older context, with the constraint that structured data (tool outputs, scores, citations) is preserved losslessly while conversational filler is compressed.

Agents that overflow without requesting compression are logged as PolicyViolation events with the excess token count. This makes budget compliance measurable and surfaceable in the evaluation harness.

5. Tool Calling

Four tools are registered with the orchestrator, each with a defined failure contract:

All tool calls are logged with input hash, output hash, latency, and the agent’s binary accepted/rejected decision on the result. An agent that determines a tool result is insufficient may re-call the tool with a modified input, up to two retries, each logged as a separate tool_retry event.

6. Evaluation Pipeline

6.1 Test Suite

The evaluation harness runs 15 test cases through the full pipeline. The suite is partitioned into three categories:

• Baseline (5 cases): Straightforward queries with known correct answers; primarily test retrieval precision and citation accuracy.
• Ambiguous (5 cases): Underspecified or multi-part queries; primarily test decomposition quality and synthesis coherence.
• Adversarial (5 cases): Prompt injection attempts, queries with confident but factually wrong premises, and queries designed to cause the critique agent to disagree with the synthesis agent.

6.2 Scoring Dimensions

Each test case is scored independently across eight dimensions:

Every dimension produces a numeric score ∈ [0, 1] and a written justification string. No third-party evaluation framework is used; all scoring logic is implemented in eval_harness.py.

6.3 Stored Results from the Reference Run

A reference run (eval_run_id: de1dd5ab) on the system following a clean no-cache Docker rebuild produced the following results:

The wrong_premise_handling score of 0.000 reflects a known limitation: the synthesis agent, when given a query containing a confident but factually incorrect premise, does not reliably challenge the premise before forming an answer. This is the dimension targeted by the self-improving loop in the reference run.

7. The Self-Improving Prompt Loop

The self-improving loop operates in five sequential stages, each persisted to the database with timestamps:

POST /eval/run
    │
    ▼
failure_analyzer identifies worst (dimension, agent_id)
    │
    ▼
prompt_rewrite_proposer generates structured diff + justification
    │
    ▼  stored as prompt_rewrite (status: pending)
    │
POST /eval/prompt-rewrite/{id}/approve
    │
    ├── regression check:
    │     for each dimension where baseline ≥ 0.70
    │     if candidate_score < baseline_score - 0.05 → 409 REGRESSION_DETECTED
    │
    ├── force: true → override regression block
    │
    └── approved → status: approved
    │
POST /eval/retry-failed?rewrite_id={id}
    │
    ▼
targeted_eval re-runs only previously failed cases
    │
    ▼
delta_scores JSONB stored in rewrite_approvals

7.1 Regression Detection

The regression gate selects the most recent eval run older than the candidate run as the baseline. This ensures that the baseline always predates the candidate, preventing a newer run from being incorrectly used as the reference point. The SQL sub-select is:

SELECT id FROM eval_runs
WHERE id != CAST(:id AS UUID)
  AND triggered_at < (
      SELECT triggered_at FROM eval_runs WHERE id = CAST(:id AS UUID)
  )
ORDER BY triggered_at DESC LIMIT 1

A regression is declared when a dimension that previously scored ≥ 0.70 drops by ≥ 0.05. The HTTP response to a regression-blocked approval is:

{
  "error_code": "REGRESSION_DETECTED",
  "regressions": [
    {
      "dimension": "answer_correctness",
      "baseline_score": 0.8,
      "candidate_score": 0.71,
      "delta": -0.09,
      "baseline_run_id": "..."
    }
  ]
}

Callers can override the block by setting force: true in the request body, which is logged as an explicit human decision rather than a system bypass.

8. End-to-End Use Case Walkthrough: “What is machine learning?”

The following walkthrough traces a single query through every layer of the system — from the initial SSE stream to eval scoring, prompt rewrite approval, and targeted re-evaluation. All output was captured from a live Docker environment on 2026–05–07 (gpt-4o-mini, temperature=0).

Step 1 — POST /query/stream

POST /query/stream
Content-Type: application/json

{ "query": "What is machine learning?", "context_budget": 3000 }

The response is a persistent SSE connection. 283 events were received before the stream closed:

The orchestrator decided to skip the critique agent for this query (low ambiguity, no retrieved contradictions) and execute the three-agent chain in sequence.

Step 2 — GET /trace/{job_id}

The execution trace reconstructed from structured logs confirms the routing plan and records a completed_at timestamp. The policy_violations_total: 0 field confirms no agent exceeded its declared token budget.

Step 3 — GET /logs

Nine log entries were persisted for this job across three event_type values: budget_declared, budget_consumed, and routing. Each row carries agent_id, input_hash, output_hash, latency_ms, and token_count.

Step 4 — POST /eval/run

The full 15-case evaluation suite was executed as an ARQ background job. After scoring, the failure analyzer identified the worst-performing (dimension, agent) pair:

{
  "eval_run_id": "8828d321-847e-4be0-be1c-288e7de4c46d",
  "total_cases": 15,
  "failure_analysis": {
    "worst_dimension": "wrong_premise_handling",
    "worst_agent_id": "synthesis"
  }
}

Step 5 — GET /eval/summary

The summary endpoint returns scores broken down by both category and dimension:

overall_score      : 0.8083
worst_dimension    : wrong_premise_handling (0.0000)
categories         : baseline, ambiguous, adversarial

The wrong_premise_handling score of 0.0 indicates that the synthesis agent accepted factually incorrect premises in all adversarial-premise test cases rather than challenging them — the exact failure mode the self-improving loop is designed to target.

Step 6 — POST /prompt-rewrite/{id}/decision

A rewrite for the synthesis agent's system prompt was auto-proposed by prompt_rewrite_proposer. A human reviewer approved it via the approval endpoint. Because this was the first eval run (no prior baseline existed), the regression gate had no prior scores to compare against and passed unconditionally:

{ "decision": "approved", "decided_at": "2026-05-07T12:24:33.646529+00:00" }

Step 7 — POST /eval/retry-failed

The targeted re-evaluation ran only the 6 test cases that failed in the previous run (all wrong_premise_handling cases plus the baseline cases that scored below 0.70):

{
  "new_eval_run_id": "eeed8187-af3f-49ba-b389-834897e9f6f1",
  "cases_targeted": 6
}

The per-case delta_scores JSONB payload (stored in rewrite_approvals) records the before/after score for every dimension, enabling a diff-able comparison between the original run and the post-rewrite run.

Use Case Summary

All 7 steps passed on the first clean no-cache Docker build.

9. Streaming and Observability

All agent activity is streamed to the client via Server-Sent Events over Redis Streams (XADD / XREAD). The SSE event schema:

event: agent_token
data: {"agent_id": "synthesis", "token": " based", "budget_remaining": 1847}

event: tool_call
data: {"agent_id": "rag", "tool": "vector_search", "latency_ms": 43}

event: job_complete
data: {"job_id": "...", "total_tokens": 2341, "duration_ms": 4812}

Because events are written to Redis Streams before being forwarded to the SSE connection, they survive worker restarts and are available for replay. A client that connects after a job completes can reconstruct the full event sequence by reading from the beginning of the stream.

Structured logs follow a consistent schema across all agents: timestamp, agent_id, event_type, input_hash, output_hash, latency_ms, token_count, and policy_violation (nullable). The GET /trace/{job_id} endpoint reconstructs the full execution sequence in chronological order, including routing plan, per-agent events, tool calls, and handoffs.

10. Discussion

9.1 Architectural Trade-offs

The choice to implement multi-hop retrieval within a single agent (rather than as a separate retrieval orchestration layer) reflects a pragmatic trade-off: a dedicated retrieval planner would add a round-trip LLM call per query in exchange for more modular control over retrieval strategy. For the scale of this system (single-tenant, non-latency-critical evaluation), the simpler design is appropriate.

The use of Redis Streams rather than in-process asyncio queues for SSE delivery adds operational complexity (Redis becomes a critical dependency) but provides durability and the ability to serve multiple concurrent SSE consumers for the same job — a requirement for production deployments where browser reconnects must not lose events.

9.2 Known Limitations

• Wrong premise handling is the weakest dimension. The synthesis agent does not reliably challenge a confident but false premise before generating an answer, because the critique agent must first flag it — and the critique agent only reviews other agents’ outputs, not the input query itself.
• Citation accuracy (0.667) reflects that the RAG agent sometimes cites chunks that are topically relevant but do not directly support the claimed sentence. A re-ranking step before citation selection would likely improve this.
• The evaluation suite is static and small (15 cases). It validates the pipeline’s behaviour on known inputs but does not cover the full distribution of real user queries.
• No authentication or rate limiting is implemented. All endpoints are publicly accessible.

9.3 Directions for Future Work

The most impactful extension would be closing the feedback loop between production traffic and the evaluation suite: logging real user queries that receive low confidence scores and automatically promoting them as new test cases. This would make the eval suite a living document rather than a static fixture.

A second high-value extension is cross-dimension regression analysis: the current regression gate operates per-dimension in isolation. A rewrite that improves wrong_premise_handling while degrading answer_correctness by exactly 0.04 (below the 0.05 threshold) passes the gate despite making the system worse overall. A joint regression score across dimensions would address this.

11. Conclusion

This article has described a multi-agent RAG system in which retrieval, decomposition, critique, and synthesis are handled by specialised agents coordinated by a runtime-planning orchestrator. The system implements a closed self-improving loop: an evaluation harness identifies underperforming prompt dimensions, a meta-agent proposes targeted rewrites, and a regression-blocking approval gate ensures that promoted changes do not degrade passing dimensions. All activity is streamed in real time via Server-Sent Events and persisted in a fully queryable audit trail.

The design choices — ARQ over Celery, pgvector over a dedicated vector database, Redis Streams over in-process queues, and custom scoring logic over third-party eval frameworks — reflect a preference for minimising service count while retaining the operational properties (durability, observability, reproducibility) that distinguish production systems from prototypes.

The complete implementation, including Docker Compose configuration, database schema, all agent source code, and the 15-case evaluation suite, is available at:

https://github.com/huseyincenik/multi_agent_rag_system

References

1. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.
2. Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023.
3. Madaan, A., et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023.
4. Chase, H. (2022). LangChain: Building Applications with LLMs through Composability. GitHub.
5. OpenAI. (2023). Function Calling and Other API Updates. OpenAI Blog.
6. Johnson, J., Douze, M., & Jégou, H. (2019). Billion-scale similarity search with GPUs. IEEE Transactions on Big Data.
7. FastAPI Documentation. Sebastián Ramírez. https://fastapi.tiangolo.com
8. ARQ Documentation. Samuel Colvin. https://arq-docs.helpmanual.io
9. pgvector: Open-source vector similarity search for PostgreSQL. https://github.com/pgvector/pgvector

The author used GitHub Copilot (Claude Sonnet 4.6) as an AI pair-programming assistant throughout the implementation. All architectural decisions, test case authorship, and code review were performed by the author.

Every top-performing Kaggle notebook does something that most beginner notebooks don’t: it looks like someone cared about it. The cover banner, the colored section headers, the styled callout boxes — these aren’t just decoration. They signal that the author thought carefully about communication, not just computation.

But the formatting tricks used in those notebooks work just as well in GitHub READMEs, Jupyter Lab sessions, VS Code notebooks, and HackMD documents. Markdown is Markdown. HTML is HTML.

This guide collects every useful pattern from a scan of 500+ data science notebooks, organized by type, with working code you can copy directly.

1. The Markdown Baseline (Works Everywhere)

Before reaching for HTML, know your Markdown fundamentals. These render identically in Kaggle, GitHub, Jupyter, and VS Code:

Headings create structure AND navigation

# H1 — Title / major chapter
## H2 — Section header  
### H3 — Sub-section
#### H4 — Minor heading

GitHub auto-generates a table of contents from headings. Kaggle and Jupyter create anchor links you can target with [link](#section-name).

Rule: Never skip a heading level. Going from # to ### without ## breaks visual hierarchy and navigation.

Text formatting

**bold text**          <!-- strong emphasis -->
*italic text*          <!-- light emphasis -->
~~strikethrough~~      <!-- deprecated, removed -->
`inline code`          <!-- function names, file paths -->
> blockquote           <!-- important notes, quotes -->

GitHub Flavored Markdown (GFM) tables

| Model     | CV Score | LB Score |
|-----------|----------|----------|
| LightGBM  | 0.9797   | 0.9810   |
| XGBoost   | 0.9789   | 0.9801   |
| Ensemble  | **0.9814**   | **0.9831**   |

These render in GitHub READMEs, Kaggle, and Jupyter with proper alignment.

Fenced code blocks with syntax highlighting

```python
import lightgbm as lgb
model = lgb.LGBMClassifier(n_estimators=1000, learning_rate=0.05)
```

Use the language identifier (python, sql, bash, json, html) for syntax highlighting.

2. HTML in Markdown: What You Can Do

Jupyter-based platforms (Kaggle, Jupyter Lab, VS Code notebooks) render a wide subset of HTML directly in Markdown cells. This unlocks everything from colored text to full-page banner designs.

What works (everywhere except GitHub READMEs)

• <div>, <span>, <p> with inline style="" attributes
• CSS gradients, border-radius, box-shadow, grid, flexbox
• <table> with custom styling
• <details> / <summary> collapsible blocks
• <kbd> keyboard key display
• <a id="anchor"> for internal navigation
• <img> with external URLs
• <style> blocks (including @import for Google Fonts)
• HTML entities (&nbsp;, &mdash;, &copy;, etc.)

What GitHub READMEs strip

GitHub’s HTML sanitizer removes style="" attributes and <style> blocks. For GitHub, stick to Markdown tables and  images.

3. Cover Banners: The Most Impactful Single Element

The notebook cover is the first thing readers see. A well-designed cover communicates competence before they read a single line of code.

Pattern 1: Dark Gradient (most common in top notebooks)

<div style="
    background: linear-gradient(135deg, #0f172a 0%, #1e3a5f 50%, #0f4c75 100%);
    border-left: 6px solid #14B8A6;
    border-radius: 16px;
    padding: 44px 48px;
    box-shadow: 0 10px 30px rgba(0,0,0,0.5);
    color: #f9fafb;
    font-family: 'Segoe UI', sans-serif;
">
    <h1 style="font-size:36px; font-weight:800; margin:0 0 12px 0;">
        🏆 Your Notebook Title
    </h1>
    <p style="color:#94a3b8; font-size:15px; margin:0 0 20px 0;">
        Competition or project description
    </p>
    <div style="display:flex; gap:16px; flex-wrap:wrap; font-size:13px; color:#94a3b8;">
        <span>👤 <strong style="color:#f9fafb;">Author</strong></span>
        <span>📅 <strong style="color:#f9fafb;">2025</strong></span>
        <span>🏅 <strong style="color:#f9fafb;">Metric</strong></span>
    </div>
</div>

Pattern 2: Photo Background with Dark Overlay

<div style="
    background-image:
        linear-gradient(rgba(0,0,0,0.55), rgba(0,0,0,0.70)),
        url(https://medium.com/'https://images.unsplash.com/photo-XXXX?w=1200&auto=format&fit=crop');
    background-position: center;
    background-size: cover;
    padding: 60px 20px;
    border-radius: 18px;
    text-align: center;
">
    <h1 style="color:white; font-size:36px; text-shadow:2px 2px 8px rgba(0,0,0,0.8);">
        Notebook Title
    </h1>
    <p style="color:rgba(255,255,255,0.85); font-size:16px;">Subtitle</p>
</div>

Use Unsplash for free photos. The rgba(0,0,0,0.55) overlay ensures text stays readable over any image.

Pattern 3: Light Earth Tone (from high-scoring notebooks)

<div style="
    background: linear-gradient(135deg, #F7F3EA 0%, #EEF4ED 100%);
    border: 1px solid #D9D0BF;
    border-radius: 18px;
    padding: 30px;
">
    <div style="font-size:11px; letter-spacing:0.14em; text-transform:uppercase;
                color:#7A6045; font-weight:700; margin-bottom:8px;">
        Subtitle Label
    </div>
    <div style="font-size:2rem; font-weight:800; color:#233127;">
        Main Heading
    </div>
</div>

This light variant works well when your notebook uses light-mode matplotlib plots.

4. Section Headers: Dividing Long Notebooks

Long notebooks need clear visual section breaks. Here are the three most common patterns extracted from competition notebooks:

Left-border section header (clean and readable)

<div style="
    padding: 12px 22px;
    background: linear-gradient(90deg, #0f2d24 0%, #0d4a3e 100%);
    border-radius: 10px;
    border-left: 5px solid #84cc16;
    color: #f9fafb;
    margin: 28px 0 14px 0;
">
    <div style="font-size:20px; font-weight:600;">2. Exploratory Data Analysis</div>
</div>

Inline heading-div fusion (for Kaggle/Jupyter only)

# <div style="text-align:center; border-radius:15px; padding:15px;
     background:linear-gradient(90deg,#0f2027,#2c5364); color:white;">
<b>Section Title</b></div>

This embeds a CSS-styled div inside a Markdown heading tag. The heading still acts as a navigable anchor, but displays with full CSS styling.

ALL-CAPS minimal divider

<p style="font-size:11px; letter-spacing:3px; text-transform:uppercase;
          color:#64748b; font-weight:700; border-bottom:1px solid #e2e8f0;
          padding-bottom:6px; margin:28px 0 14px 0;">
    ⬛ Section 4 · Model Training
</p>

5. Callout Boxes: Highlighting Key Information

Callout boxes catch the reader’s eye for important notes, warnings, or insights.

Bootstrap-style alerts (the quick way)

Most notebook platforms support <div class="alert alert-info"> Bootstrap-style alerts natively:

<div class="alert alert-success">
    <strong>✅ Key Result:</strong> Ensemble achieved CV 0.9814 — top 3% of leaderboard.
</div>

<div class="alert alert-warning">
    <strong>⚠️ Warning:</strong> Target encoding must be fit inside CV folds to prevent leakage.
</div>

<div class="alert alert-danger">
    <strong>🚨 Critical:</strong> This dataset has severe class imbalance (3.3% positive class).
</div>

Custom styled callouts (for dark notebooks)

<div style="
    background: #1e293b;
    border: 1px solid #334155;
    border-left: 4px solid #60a5fa;
    padding: 14px 18px;
    border-radius: 0 8px 8px 0;
    font-family: 'Segoe UI', sans-serif;
">
    <strong style="color:#93c5fd;">📌 Note</strong><br>
    <span style="color:#cbd5e1; font-size:14px;">Your important note here.</span>
</div>

Metric cards grid

<div style="display:grid; grid-template-columns:repeat(4,1fr); gap:12px; font-family:'Segoe UI',sans-serif;">
    <div style="background:linear-gradient(135deg,#1e3a5f,#2563eb); border-radius:10px;
                padding:16px; text-align:center; color:white;">
        <div style="font-size:26px; font-weight:800;">75,000</div>
        <div style="font-size:12px; opacity:0.8; text-transform:uppercase; letter-spacing:1px; margin-top:4px;">Training Rows</div>
    </div>
    <!-- Repeat for other metrics -->
</div>

6. Progress Bars and Visual Indicators

Progress bars are a clean alternative to tables when you want to show relative scores at a glance:

<div style="font-family:'Segoe UI',sans-serif; margin:12px 0;">
    <div style="margin:6px 0;">
        <div style="display:flex; justify-content:space-between; font-size:13px; margin-bottom:3px;">
            <span>LightGBM</span><span>97.97%</span></div>
        <div style="background:#e2e8f0; border-radius:6px; height:12px;">
            <div style="background:#2563eb; width:97.97%; height:12px; border-radius:6px;"></div>
        </div>
    </div>
    <div style="margin:6px 0;">
        <div style="display:flex; justify-content:space-between; font-size:13px; margin-bottom:3px;">
            <span>Ensemble</span><span style="font-weight:700;">98.14% 🏆</span></div>
        <div style="background:#e2e8f0; border-radius:6px; height:12px;">
            <div style="background:linear-gradient(90deg,#16a34a,#4ade80); width:98.14%; height:12px; border-radius:6px;"></div>
        </div>
    </div>
</div>

7. Badges (Shields.io) — GitHub-Friendly Metadata

Badges are SVG images generated by Shields.io and embedded as regular Markdown images. They work in GitHub READMEs, Kaggle notebooks, and everywhere else.

![Python](https://img.shields.io/badge/Python-3.10+-3776AB?logo=python&logoColor=white)
![License](https://img.shields.io/badge/License-MIT-green)
![CV Score](https://img.shields.io/badge/CV_Score-0.9797-brightgreen)
![Kaggle](https://img.shields.io/badge/Kaggle-20BEFF?logo=kaggle&logoColor=white)

Render as: inline colorful badges visible in any platform.

Common badge patterns:

• ?style=flat (default) or ?style=flat-square for square corners
• Add logo: ?logo=python&logoColor=white
• Custom colors: use hex (-1e3a5f) or named colors (-brightgreen, -blue, -red)

8. Keyboard Tags, Entities, and Other HTML Gems

<kbd> keyboard shortcut display

Press <kbd>Shift</kbd> + <kbd>Enter</kbd> to run a cell.

Renders as styled keyboard keys that look like actual key caps.

HTML entities for special characters

IntentCodeOutputForce space / indent&nbsp;(non-breaking space)Show < literally&lt;<Em dash in prose&mdash;—Copyright&copy;©Right arrow&rarr;→

<details> collapsible sections

<details>
<summary style="cursor:pointer; padding:10px 14px; background:#f1f5f9;
                border-radius:6px; font-weight:600;">
    ▶ Click to expand: Parameter details
</summary>
<div style="padding:14px; background:#f8fafc; border:1px solid #e2e8f0; border-radius:0 0 6px 6px;">
    Your expanded content here.
</div>
</details>

Works in Kaggle, GitHub, Jupyter, VS Code, and HackMD.

9. Multi-Column Layouts

CSS Grid makes side-by-side content easy in any notebook platform that supports inline CSS:

<div style="display:grid; grid-template-columns:1fr 1fr; gap:16px;">
    <div style="background:#f8fafc; border:1px solid #e2e8f0; border-radius:10px; padding:16px;">
        <h4 style="margin:0 0 8px 0; color:#1e3a5f;">Left Column</h4>
        <p style="margin:0; font-size:14px; color:#64748b;">Content here</p>
    </div>
    <div style="background:#f8fafc; border:1px solid #e2e8f0; border-radius:10px; padding:16px;">
        <h4 style="margin:0 0 8px 0; color:#1e3a5f;">Right Column</h4>
        <p style="margin:0; font-size:14px; color:#64748b;">Content here</p>
    </div>
</div>

Note: CSS grid is stripped by GitHub’s sanitizer. Use a Markdown table or plain paragraphs for GitHub READMEs.

10. Platform Compatibility Reference

Not all HTML works everywhere. Here’s a quick reference:

FeatureKaggleJupyterGitHub READMEVS CodeHackMDInline CSS style=""✅✅❌ stripped✅✅<style> block✅✅❌ stripped✅✅CSS grid / flexbox✅✅❌✅✅<details> collapsible✅✅✅✅✅LaTeX math✅✅❌⚠️ ext✅Shields.io badges✅✅✅✅✅External images✅✅✅✅✅Google Fonts @import✅ (internet)✅❌⚠️✅

Key rule: Write for Kaggle/Jupyter’s full HTML support, then simplify for GitHub by removing inline styles and <style> blocks.

11. Complete Notebook Template

Here’s a minimal but well-formatted notebook starter you can drop into any data science project:

Cell 1 — Cover (Markdown):

<a id="top"></a>
<div style="background:linear-gradient(135deg,#0f172a,#1e3a5f,#0f4c75);
            border-left:6px solid #14B8A6; border-radius:16px; padding:44px 48px;
            box-shadow:0 10px 30px rgba(0,0,0,0.5); color:#f9fafb;
            font-family:'Segoe UI',sans-serif;">
    <h1 style="font-size:34px; font-weight:800; margin:0 0 10px 0;">🏆 Your Notebook Title</h1>
    <p style="color:#94a3b8; font-size:15px; margin:0 0 20px 0;">
        Project description · Metric · Date
    </p>
    <div style="display:flex; gap:16px; flex-wrap:wrap; font-size:13px; color:#94a3b8;">
        <span>👤 <strong style="color:#f9fafb;">Your Name</strong></span>
        <span>📅 <strong style="color:#f9fafb;">2025</strong></span>
    </div>
</div>

Cell 2 — Section Header (Markdown):

<div style="padding:12px 20px; background:linear-gradient(90deg,#1e3a5f,#2563eb);
            border-radius:10px; color:white; margin:20px 0 10px 0;">
    <h2 style="margin:0; font-size:18px; font-weight:700;">1. Introduction</h2>
</div>

Cell 3 — Footer (Markdown):

<center>
<div style="display:inline-block; padding:10px 24px; border-radius:8px;
            background:linear-gradient(90deg,#1e3a5f,#2563eb); color:white;
            font-family:'Segoe UI',sans-serif; font-size:15px;">
    <strong>👍 If this helped you, please upvote!</strong>
</div>
</center>

12. Advanced Patterns: Narrative Templates & Dynamic Elements

These patterns come from a deeper scan of top-performing competition notebooks. They solve a specific problem: how do you make a long analytical notebook readable — not just technically correct?

12.1 Dashed-Border Key Findings Box

After every major plot, add a dashed annotation box. The dashed border reads as a comment on the chart above — not an alert, not a callout box, but an honest “here’s what I noticed”:

<div style="padding:18px 24px; background:#fafaf9; border:1.5px dashed #a3a3a3;
            border-radius:10px; margin:14px 0; font-family:'Segoe UI',sans-serif;">
    <p style="margin:0 0 8px 0; font-size:11px; letter-spacing:2px;
              text-transform:uppercase; color:#737373; font-weight:600;">
        ✏ Key findings
    </p>
    <ul style="margin:0; font-size:14px; color:#525252; font-style:italic;
               line-height:1.7; padding-left:18px;">
        <li><b>Feature A</b> shows the largest class separation.</li>
        <li>Train and test distributions are well-aligned.</li>
    </ul>
</div>

12.2 Sub-Section Divider Strip

A left-border heading strip for three-tier hierarchy — section pill → sub-section strip → findings box:

<div style="padding:8px 18px; border-left:4px solid #14b8a6;
            background:#f0fdf4; border-radius:0 8px 8px 0; margin:18px 0 10px 0;">
    <h3 style="margin:0; color:#0f2d24; font-size:16px; font-weight:700;
               font-family:'Segoe UI',sans-serif;">
        3.1 — Numerical feature distributions
    </h3>
</div>

Available in three variants: teal (default analysis), blue (informational), amber (important/warning).

12.3 Faded-Prefix Numbered Section Header

Dark gradient pill where the number is at reduced opacity — creates magazine-style editorial section breaks:

<div style="padding:12px 22px; background:linear-gradient(90deg,#0f2d24,#0d4a3e);
            border-radius:10px; border-left:5px solid #84cc16; color:#f9fafb;
            margin:28px 0 14px 0; font-family:'Segoe UI',sans-serif;">
    <div style="font-size:20px; font-weight:600;">
        <span style="opacity:0.45; font-weight:500; margin-right:12px;">01</span>
        Data Quality &amp; Structure
    </div>
</div>

12.4 Score Metric Mini-Cards

Horizontal flexbox strip for displaying model results — each card has a color-coded top border matching the metric type:

<div style="display:flex; gap:12px; flex-wrap:wrap; margin:16px 0;
            font-family:'Segoe UI',sans-serif;">
    <div style="flex:1; min-width:130px; background:#1e293b; border-radius:10px;
                padding:14px 18px; border-top:3px solid #2563eb; text-align:center;">
        <div style="font-size:11px; color:#94a3b8; text-transform:uppercase;
                    letter-spacing:1.5px; margin-bottom:6px;">CV Score</div>
        <div style="font-size:26px; font-weight:800; color:#60a5fa;">0.9797</div>
        <div style="font-size:11px; color:#64748b; margin-top:4px;">5-Fold Stratified KFold</div>
    </div>
    <!-- Add more cards: LB Score (teal), Models (purple), Top % (amber) -->
</div>

12.5 HTML Model Comparison Table

Styled leaderboard with gradient dark header and alternating row shading. Supports emoji medals and per-column color coding — none of which Markdown tables can do:

<table style="border-collapse:collapse; width:100%; font-size:14px; margin:16px 0;
              font-family:'Segoe UI',sans-serif;">
    <thead>
        <tr style="background:linear-gradient(90deg,#1e3a5f,#2563eb); color:#f1f5f9;">
            <th style="padding:11px 16px; text-align:left;">Model</th>
            <th style="padding:11px 16px; text-align:center;">CV Score</th>
            <th style="padding:11px 16px; text-align:center;">LB Score</th>
        </tr>
    </thead>
    <tbody>
        <tr style="background:#f8fafc;">
            <td style="padding:10px 16px; border-bottom:1px solid #e2e8f0;">🥇 Ensemble</td>
            <td style="padding:10px 16px; text-align:center; color:#16a34a; font-weight:700;">0.9797</td>
            <td style="padding:10px 16px; text-align:center; color:#16a34a; font-weight:700;">0.9774</td>
        </tr>
    </tbody>
</table>

12.6 Dynamic Dataset Summary Strip (Python → HTML)

A Python function that emits an HTML banner inline, between data preview cells. The banner shows row count, column count, and memory usage — all computed dynamically from the actual DataFrame:

from IPython.display import HTML, display

def dataset_summary(df, label="Dataset"):
    display(HTML(f"""
    <div style="border-left:5px solid #2563eb; padding:10px 18px;
                background:#eff6ff; border-radius:0 8px 8px 0; margin:12px 0;
                font-family:'Segoe UI',sans-serif;">
        <b style="color:#1e3a5f;">{label}</b>
        <span style="margin-left:14px; color:#2563eb; font-weight:600;">{df.shape[0]:,} rows</span>
        <span style="margin-left:10px; color:#64748b;">×</span>
        <span style="margin-left:10px; color:#2563eb; font-weight:600;">{df.shape[1]} columns</span>
        <span style="margin-left:16px; color:#94a3b8; font-size:12px;">
            memory: {df.memory_usage(deep=True).sum() / 1024**2:.1f} MB
        </span>
    </div>"""))
    return df

# Use as a wrapper — returns the dataframe so .head() chaining still works
dataset_summary(train_df, "Training Set").head()

12.7 Hypothesis → Inference Blockquote Chain

A pure Markdown pattern for feature engineering rationale. The 👉 emoji acts as a visual pointer; the > blockquote highlights the core claim:

### 🔗 Interaction Features

Based on EDA: extreme values in a single feature rarely predict the target alone.

👉 **Hypothesis:**

> If variable_A is high **and** variable_B is low → the target outcome is likely Class 1.

We therefore created a ratio feature: `variable_A / (variable_B + 1e-6)`

This creates a clear audit trail: every engineered feature has a documented reason for existing.

12.8 ELI5 + Section Insight Template

The most consistent structural pattern in highly-voted notebooks: every major analysis block starts with a plain-English ELI5 preamble and ends with a 💡 Insight footer.

## 📦 Section 3 — Numerical Feature Distributions

**ELI5:** For each numerical feature, we want to know:
- What values are typical? (mean, median)
- Are train and test distributions similar?

[... code + plots ...]

---

### 💡 Section 3 Insight

**Uniform kurtosis signature:** All excess kurtosis values cluster near −1.2.
This tells us features were sampled independently and uniformly.

**Implication for modeling:**
- No imputation needed
- Feature interactions will matter more than raw feature values

The ELI5 pattern serves two audiences simultaneously: beginners who need context, and experts who skip straight to the Insight footer.

Summary

The patterns in this guide were extracted directly from 500+ Kaggle competition notebooks and validated across multiple platforms. The key takeaways:

1. Markdown first — everything in section 1 works everywhere
2. Inline CSS works in Kaggle, Jupyter, VS Code, and HackMD but is stripped by GitHub
3. CSS grid/flexbox unlocks multi-column layouts in notebooks
4. A cover banner takes 10 minutes and makes your notebook look 10× more professional
5. Section headers help readers navigate long notebooks — use left-border divs or # <div> fusion
6. Shields.io badges are the only way to add styled metadata that works in GitHub READMEs
7. <details> is the one HTML element that GitHub does render — use it for collapsible content
8. ELI5 + Insight structure makes your notebook useful to both beginners and experts — and forces you to articulate what you actually found

The complete catalog notebook with rendered examples of every pattern above is available at: Kaggle — Markdown & HTML Reference Catalog. You can visit Linkedin or Github.

All patterns tested on data science competition notebooks. Code examples are production-ready and copy-paste safe.

Text data has become one of the most complex yet rewarding areas in data science today. To organize and derive meaningful insights from it, text clustering algorithms play a pivotal role in Natural Language Processing (NLP). This article dives into the methods, applications, and performance evaluations of a complete text clustering workflow — providing detailed explanations and reproducible code for each technique.

OVERVIEW

Text clustering is a complex task that involves several critical steps to ensure meaningful and accurate results. Each phase plays a vital role in transforming raw text into actionable structure. A methodical approach is crucial, as the effectiveness of the clustering largely depends on how well each phase is executed.

Below is the complete pipeline this article walks through:

Raw Text Data
     │
     ▼
 1. Text Preprocessing
     │  cleaning · normalization · NER · fuzzy deduplication
     ▼
 2. Vectorization
     │  TF-IDF · Sentence Transformers
     ▼
 3. Dimensionality Reduction
     │  t-SNE
     ▼
 4. Clustering Tendency Assessment
     │  Hopkins Statistic
     ▼
 5. Optimal k Selection
     │  Elbow · Silhouette · Davies-Bouldin · Calinski-Harabasz
     ▼
 6. Cluster Validation
     │  Yellowbrick SilhouetteVisualizer
     ▼
 7. Cluster Labeling
        LLM (GPT-4o-mini)

Fig 1. Workflow diagram illustrating the process of text data preprocessing, vectorization, dimensionality reduction, evaluating data suitability, applying clustering algorithms, and evaluating cluster quality.

Let’s go through each step.

Step 1 — Text Preprocessing

Garbage in, garbage out. No embedding model or clustering algorithm can compensate for noisy, inconsistent raw text. Preprocessing is where the quality of your entire pipeline is decided.

1.1 Basic Cleaning and Normalization

A general-purpose cleaning function should handle the most common issues in real-world text: leading/trailing noise words, stray punctuation, inconsistent whitespace, and inconsistent casing.

import re

def clean_text(text: str) -> str:
    # Remove leading conjunctions like "and", "or" (common in list entries)
    text = re.sub(r'^\s*(?i)(and|or)\s+', '', text)
    # Keep only alphanumeric characters and safe symbols
    text = re.sub(r'[^\w\s\(\)/\-,]', '', text)
    # Collapse multiple spaces into one
    text = re.sub(r'\s+', ' ', text)
    # Strip leading/trailing punctuation and whitespace
    text = text.strip('," ')
    # Title-case for consistent token representation
    return text.title().strip()

Why title-case? Downstream steps like Named Entity Recognition (NER) and fuzzy matching are case-sensitive. Normalizing to title-case ensures "machine learning", "Machine Learning", and "MACHINE LEARNING" all map to the same token.

1.2 Multi-Value Splitting

Raw text columns often contain multiple values packed into a single cell (comma-separated, semicolon-separated, etc.). For clustering, each atomic unit should be its own row:

import pandas as pd

def explode_column(df: pd.DataFrame, col: str, sep: str = ",") -> pd.DataFrame:
    """Split a delimited text column into one row per value."""
    df = df.copy()
    df[col] = df[col].apply(clean_text)
    df[col] = df[col].str.split(sep)
    df = df.explode(col)
    df[col] = df[col].str.strip()
    df = df[df[col] != ""].reset_index(drop=True)
    return df
df_clean = explode_column(df, col="text_column")
print(f"Rows after splitting: {len(df_clean)}")

1.3 Named Entity Recognition with GLiNER

One of the most powerful preprocessing techniques for domain-rich text is Named Entity Recognition (NER). When text entries are long, verbose descriptions, NER can extract the canonical terms that actually carry clustering signal — discarding the filler words around them.

GLiNER (Generalist and Lightweight Model for Named Entity Recognition) is an excellent choice here because it supports zero-shot, user-defined entity labels. You define what you’re looking for; GLiNER finds it — no fine-tuning required.

# pip install gliner
from gliner import GLiNER

# Load a pretrained GLiNER model
model = GLiNER.from_pretrained("urchade/gliner_mediumv2.1")
# Define your domain's entity labels
labels = [
    "Product Category",
    "Technology",
    "Service Type",
    "Topic",
    "Domain",
    # Add as many domain-specific labels as needed
]
def extract_entities(text: str, threshold: float = 0.8) -> str:
    """Extract canonical entity mentions from a text string."""
    entities = model.predict_entities(text, labels, threshold=threshold)

    # Deduplicate and join extracted terms
    terms = set(entity["text"] for entity in entities)
    return ", ".join(terms)

Apply NER selectively — only on entries long enough to need normalization. Short entries (e.g., 1–3 words) are already concise:

df_clean["word_count"] = df_clean["text_column"].apply(lambda x: len(x.split()))

# Only apply NER to long entries
df_clean["entities"] = df_clean.apply(
    lambda row: extract_entities(row["text_column"]) if row["word_count"] > 8 else "",
    axis=1
)

# Replace verbose entries with their extracted entities
mask = df_clean["entities"] != ""
df_clean.loc[mask, "text_column"] = df_clean.loc[mask, "entities"]

# Drop entries that are still too long after NER
df_clean["word_count"] = df_clean["text_column"].apply(lambda x: len(x.split()))
df_clean = df_clean[df_clean["word_count"] <= 8].reset_index(drop=True)

💡 Why threshold=0.8? Lower thresholds produce more entities but with more false positives. For most domains, 0.8 gives a good balance between recall and precision. Tune this based on manual inspection.

1.4 Fuzzy Deduplication with RapidFuzz

After cleaning and NER, near-duplicate strings still remain — e.g., "Machine Learning" vs "Machine Learning Methods", or "Diabetes Type 2" vs "Type 2 Diabetes". These create artificial fragmentation in your clusters.

RapidFuzz provides fast, Levenshtein-based string similarity that scales to large vocabularies:

# pip install rapidfuzz
from rapidfuzz import fuzz
from collections import defaultdict

SIMILARITY_THRESHOLD = 95  # Tune this carefully
def build_fuzzy_groups(terms: list, threshold: int = 95) -> defaultdict:
    """Group near-duplicate strings together by fuzzy similarity."""
    groups = defaultdict(list)
    for term in terms:
        matched = False
        for group_key in groups:
            if fuzz.ratio(term, group_key) >= threshold:
                groups[group_key].append(term)
                matched = True
                break
        if not matched:
            groups[term].append(term)
    return groups
unique_terms = list(df_clean["text_column"].unique())
groups = build_fuzzy_groups(unique_terms, threshold=SIMILARITY_THRESHOLD)
# Show only groups with actual duplicates
for key, members in groups.items():
    if len(members) > 1:
        print(f"Group '{key}': {members}")

Once groups are built, map every variant back to the canonical group key:

def apply_fuzzy_normalization(text: str, groups: defaultdict) -> str:
    """Replace a term with its group's canonical representative."""
    for key, members in groups.items():
        if text in members:
            return key
    return text

df_clean["text_column"] = df_clean["text_column"].apply(
    lambda x: apply_fuzzy_normalization(x, groups)
)

⚠️ Threshold selection matters. A threshold of 80 risks merging genuinely different concepts that happen to share many characters. A threshold of 95–99 is safer for domain-specific vocabularies where subtle differences carry real meaning. Always manually review the generated groups before applying them.

Step 2 — Vectorization

Once the text is clean, it must be converted into numerical vectors that capture semantic meaning. There are two main approaches.

2.1 TF-IDF (Term Frequency–Inverse Document Frequency)

TF-IDF is the classic, lightweight baseline. It produces sparse high-dimensional vectors where each dimension corresponds to a vocabulary token.

from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.tokenize import word_tokenize

# sublinear_tf=True applies log normalization → prevents high-frequency
# terms from dominating the vector space
tfidf_vectorizer = TfidfVectorizer(
    sublinear_tf=True,
    tokenizer=word_tokenize,  # Use NLTK tokenizer for better handling
    min_df=1
)
texts = df_clean["text_column"].unique().tolist()
tfidf_matrix = tfidf_vectorizer.fit_transform(texts)
print("TF-IDF shape:", tfidf_matrix.shape)
# → (num_unique_terms, vocabulary_size)

Strengths: Fast, interpretable, no GPU needed. Weakness: Purely lexical — "automobile" and "car" have zero similarity. Works best when your vocabulary is consistent and controlled.

2.2 Sentence Transformers (Dense Semantic Embeddings)

For richer semantic similarity, sentence transformers produce dense 768-dimensional vectors where semantically similar phrases cluster together in vector space — even when they share no common words.

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer
import pickle

# Choose a model appropriate for your domain
# General-purpose:
#   all-MiniLM-L6-v2          → fast, small, good for general text
#   all-mpnet-base-v2          → higher quality, slightly slower
# Domain-specific:
#   NeuML/pubmedbert-base-embeddings  → biomedical text
#   sentence-transformers/multi-qa-mpnet-base-dot-v1  → Q&A / retrieval
model = SentenceTransformer("all-MiniLM-L6-v2")
texts = df_clean["text_column"].unique().tolist()
embeddings = model.encode(texts, show_progress_bar=True)
print("Embeddings shape:", embeddings.shape)

# → (num_unique_terms, 384) for MiniLM or (num_unique_terms, 768) for larger models
# Persist to disk - encoding is expensive; avoid recomputing
with open("embeddings.pkl", "wb") as f:
    pickle.dump(embeddings, f)

To reload later:

with open("embeddings.pkl", "rb") as f:
    embeddings = pickle.load(f)
print("Loaded embeddings:", embeddings.shape)

💡 General vs domain-specific models: A general-purpose model like all-MiniLM-L6-v2 is a solid default. If your text is highly specialized (legal, medical, financial), a domain-specific fine-tuned model will significantly improve cluster coherence. Always benchmark a few models against each other using the evaluation metrics covered later in this article.

TF-IDF vs Sentence Transformers — when to use which:

TF-IDFSentence TransformersSpeedVery fastModerate (GPU helps)Semantic understandingNone — lexical onlyStrongWorks across paraphrasesNoYesMemory footprintSparse (efficient)Dense (larger)Best forControlled vocabularies, keyword-rich corporaFree-form prose, synonym-rich domains

Step 3 — Dimensionality Reduction with t-SNE

Dense embeddings live in 768+ dimensional space. Distance-based algorithms like KMeans suffer in very high dimensions — a phenomenon known as the curse of dimensionality — where all pairwise distances converge to the same value, making cluster separation meaningless.

t-SNE (t-Distributed Stochastic Neighbor Embedding) projects high-dimensional data into 2D while preserving local neighborhood structure: points that were close in 768D remain close in 2D.

import numpy as np
from sklearn.manifold import TSNE

def reduce_with_tsne(embeddings: np.ndarray, n_components: int = 2,
                     random_state: int = 42) -> np.ndarray:
    tsne = TSNE(
        n_components=n_components,
        random_state=random_state,
        perplexity=30,       # Controls local vs global structure trade-off
        n_iter=1000,         # More iterations → more stable layout
        learning_rate="auto"
    )
    return tsne.fit_transform(embeddings)
reduced = reduce_with_tsne(embeddings)
print("Reduced shape:", reduced.shape)  # → (n_samples, 2)

Visualizing the result:

import matplotlib.pyplot as plt

def plot_tsne(reduced: np.ndarray, labels=None, title="t-SNE Projection"):
    plt.figure(figsize=(12, 8))
    scatter = plt.scatter(
        reduced[:, 0], reduced[:, 1],
        c=labels, cmap="tab20", alpha=0.6, s=20
    )
    if labels is not None:
        plt.colorbar(scatter, label="Cluster")
    plt.title(title)
    plt.xlabel("t-SNE Dimension 1")
    plt.ylabel("t-SNE Dimension 2")
    plt.grid(True, alpha=0.3)
    plt.tight_layout()
    plt.show()
plot_tsne(reduced)

t-SNE vs UMAP: t-SNE is computationally expensive O(n²) for large datasets. UMAP is a faster alternative that additionally preserves global structure. For datasets under ~10K samples, t-SNE is the clearer choice; for larger datasets, consider UMAP.

# UMAP alternative (pip install umap-learn)
import umap

reducer = umap.UMAP(n_components=2, random_state=42)
reduced_umap = reducer.fit_transform(embeddings)

Step 4 — Clustering Tendency Assessment: The Hopkins Statistic

Before running any clustering algorithm, you must ask a fundamental question: does this data actually have cluster structure? Random, uniformly distributed data can produce arbitrary clusters that mean nothing.

The Hopkins statistic answers this question quantitatively:

​​Where:

• ui = distance from a uniformly random point to its nearest neighbor in the data
• wi​ = distance from a real data point to its nearest neighbor in the data

Interpretation:

H valueMeaningH ≈ 0.5Data is random / uniformly distributed — no clustering tendencyH > 0.75Meaningful clustering structure presentH > 0.9Very strong clustering tendency

from sklearn.neighbors import NearestNeighbors
from random import sample
from numpy.random import uniform
import numpy as np

def hopkins_statistic(X: np.ndarray, sampling_ratio: float = 0.05) -> float:
    """
    Compute the Hopkins statistic for clustering tendency.
    
    Parameters
    ----------
    X : np.ndarray of shape (n_samples, n_features)
    sampling_ratio : float
        Fraction of data to sample. Paper by Lawson & Jures recommends 0.05.
    
    Returns
    -------
    float : Hopkins statistic H in [0, 1]
    """
    if not isinstance(X, np.ndarray):
        X = X.values
    n_samples = int(X.shape[0] * sampling_ratio)

    # Step 1: Draw a uniform random sample across the data space
    X_uniform = uniform(X.min(axis=0), X.max(axis=0), (n_samples, X.shape[1]))
    
    # Step 2: Draw a random sample from the real data
    real_indices = sample(range(X.shape[0]), n_samples)
    X_real = X[real_indices]

    # Step 3: Fit a nearest-neighbor index on the full data
    nbrs = NearestNeighbors(n_neighbors=2).fit(X)

    # Distances from uniform random points to their nearest real neighbor
    u_dists, _ = nbrs.kneighbors(X_uniform, n_neighbors=2)

    u_dists = u_dists[:, 0]  # First nearest neighbor

    # Distances from real points to their nearest real neighbor
    # (second neighbor because the first is the point itself, distance=0)
    w_dists, _ = nbrs.kneighbors(X_real, n_neighbors=2)
    w_dists = w_dists[:, 1]
    H = np.sum(u_dists) / (np.sum(u_dists) + np.sum(w_dists))
    return H

Run it multiple times and average for stability, testing multiple scalers:

from sklearn.preprocessing import StandardScaler, MinMaxScaler

# Test with MinMaxScaler
scores_minmax = [hopkins_statistic(MinMaxScaler().fit_transform(reduced), ratio=1.0)
                 for _ in range(10)]
print(f"Hopkins (MinMax)  : {np.mean(scores_minmax):.4f}")

# Test with StandardScaler
scores_standard = [hopkins_statistic(StandardScaler().fit_transform(reduced), ratio=1.0)
                   for _ in range(10)]
print(f"Hopkins (Standard): {np.mean(scores_standard):.4f}")

# Final scaled data for clustering
X = StandardScaler().fit_transform(reduced)
print(f"Hopkins (final)   : {hopkins_statistic(X, ratio=1.0):.4f}")

⚠️ If H < 0.6, stop and re-examine your preprocessing and vectorization. Your data may genuinely lack cluster structure, or your dimensionality reduction may have destroyed it.

Step 5 — Finding the Optimal Number of Clusters

Choosing k in KMeans is one of the most consequential decisions in the pipeline. There is no single correct answer — but using multiple evaluation metrics together gives you a much more reliable picture than any one metric alone.

5.1 The Elbow Method (Inertia)

Inertia is the within-cluster sum of squared distances from each point to its centroid. It always decreases as k increases, but the rate of decrease slows once the "natural" number of clusters is reached. The elbow point identifies this transition.

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import numpy as np

k_range = range(2, 50)
inertias = []
for k in k_range:
    km = KMeans(n_clusters=k, n_init=100, max_iter=400, random_state=42)
    km.fit(X)
    inertias.append(km.inertia_)
# Plot
plt.figure(figsize=(10, 5))
plt.plot(k_range, inertias, marker='o', color='steelblue')
plt.title("Elbow Method - Finding Optimal k")
plt.xlabel("Number of Clusters (k)")
plt.ylabel("Inertia (Within-Cluster Sum of Squares)")
plt.grid(True, alpha=0.4)
plt.tight_layout()
plt.show()

The elbow is the inflection point — where adding more clusters gives diminishing returns. In practice, the elbow is often not perfectly sharp, which is why you need the other metrics below.

5.2 Silhouette Score

The Silhouette Score measures how similar a data point is to its own cluster compared to the nearest neighboring cluster:

Where:

• a(i) = average distance from point ii to all other points in the same cluster
• b(i) = average distance from point ii to all points in the nearest other cluster

from sklearn.metrics import silhouette_score

silhouette_scores = []
for k in k_range:
    km = KMeans(n_clusters=k, n_init=100, max_iter=400, random_state=42)
    labels = km.fit_predict(X)
    silhouette_scores.append(silhouette_score(X, labels))
best_k_sil = k_range[np.argmax(silhouette_scores)]
print(f"Best k by Silhouette: {best_k_sil} (score={max(silhouette_scores):.4f})")

5.3 Davies-Bouldin Index

The Davies-Bouldin Index (DBI) measures the average ratio of within-cluster scatter to between-cluster separation:

from sklearn.metrics import davies_bouldin_score

db_scores = []
for k in k_range:
    km = KMeans(n_clusters=k, n_init=100, max_iter=400, random_state=42)
    labels = km.fit_predict(X)
    db_scores.append(davies_bouldin_score(X, labels))
best_k_db = k_range[np.argmin(db_scores)]
print(f"Best k by Davies-Bouldin: {best_k_db} (score={min(db_scores):.4f})")

5.4 Calinski-Harabasz Index

The Calinski-Harabasz Index (CHI), also called the Variance Ratio Criterion, computes the ratio of between-cluster dispersion to within-cluster dispersion:

from sklearn.metrics import calinski_harabasz_score

ch_scores = []
for k in k_range:
    km = KMeans(n_clusters=k, n_init=100, max_iter=400, random_state=42)
    labels = km.fit_predict(X)
    ch_scores.append(calinski_harabasz_score(X, labels))
best_k_ch = k_range[np.argmax(ch_scores)]
print(f"Best k by Calinski-Harabasz: {best_k_ch} (score={max(ch_scores):.4f})")

5.5 Combined Multi-Metric Visualization

Plot all four metrics side-by-side to see where they agree:

fig, axes = plt.subplots(1, 4, figsize=(22, 5))

axes[0].plot(k_range, silhouette_scores, marker='o', color='royalblue')
axes[0].axvline(x=best_k_sil, color='royalblue', linestyle='--', label=f"Best k={best_k_sil}")
axes[0].set_title("Silhouette Score\n(higher is better)")
axes[0].set_xlabel("k")
axes[0].legend(); axes[0].grid(True, alpha=0.3)
axes[1].plot(k_range, db_scores, marker='o', color='crimson')
axes[1].axvline(x=best_k_db, color='crimson', linestyle='--', label=f"Best k={best_k_db}")
axes[1].set_title("Davies-Bouldin Index\n(lower is better)")
axes[1].set_xlabel("k")
axes[1].legend(); axes[1].grid(True, alpha=0.3)
axes[2].plot(k_range, ch_scores, marker='o', color='seagreen')
axes[2].axvline(x=best_k_ch, color='seagreen', linestyle='--', label=f"Best k={best_k_ch}")
axes[2].set_title("Calinski-Harabasz Index\n(higher is better)")
axes[2].set_xlabel("k")
axes[2].legend(); axes[2].grid(True, alpha=0.3)
axes[3].plot(k_range, inertias, marker='o', color='darkorange')
axes[3].set_title("Inertia (Elbow)\n(lower is better)")
axes[3].set_xlabel("k")
axes[3].grid(True, alpha=0.3)
plt.suptitle("Multi-Metric Cluster Evaluation", fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()

Metric cheat sheet:

5.6 Incremental Evaluation with CSV Checkpointing

For large k ranges, the full grid search takes hours. Saving results to CSV after each k means crashes do not restart the job from zero:

import os
import pandas as pd

def evaluate_clusters_with_checkpoint(
    X: np.ndarray,
    k_range,
    output_csv: str = "cluster_eval.csv"
) -> pd.DataFrame:
    """
    Run KMeans over k_range, saving results to CSV after each k.
    Skips k values already present in the CSV.
    """
    if os.path.exists(output_csv):
        results = pd.read_csv(output_csv)
        done = set(results["k"].tolist())
        print(f"Resuming from checkpoint - {len(done)} values already computed.")
    else:
        results = pd.DataFrame(columns=["k", "silhouette", "db_index", "ch_index", "inertia"])
        done = set()
    for k in k_range:
        if k in done:
            continue
        km = KMeans(n_clusters=k, n_init=100, max_iter=400, random_state=42)
        labels = km.fit_predict(X)
        row = {
            "k":           k,
            "silhouette":  silhouette_score(X, labels),
            "db_index":    davies_bouldin_score(X, labels),
            "ch_index":    calinski_harabasz_score(X, labels),
            "inertia":     km.inertia_
        }
        results = pd.concat([results, pd.DataFrame([row])], ignore_index=True)
        results.to_csv(output_csv, index=False)  # Persist after every step
        print(f"k={k:4d} | Sil={row['silhouette']:.4f} | "
              f"DB={row['db_index']:.4f} | CH={row['ch_index']:.4f} | "
              f"Inertia={row['inertia']:.1f}")
    return results
results_df = evaluate_clusters_with_checkpoint(X, k_range=range(4, 200))

💡 Why n_init=100? KMeans is sensitive to centroid initialization. Running 100 random initializations and keeping the best one significantly reduces the chance of converging to a poor local minimum, at the cost of computation time.

Step 6 — Cluster Validation with Yellowbrick

After selecting your k, Yellowbrick's SilhouetteVisualizer provides a cluster-level view of silhouette scores. Each horizontal bar represents one cluster; the average silhouette score is shown as a vertical dashed line.

• Bars wider than the average line → well-separated, cohesive cluster ✅
• Bars narrower than average → sparse or fragmented cluster ⚠️
• Negative bars → points likely misassigned ❌

# pip install yellowbrick
from yellowbrick.cluster import SilhouetteVisualizer
from sklearn.cluster import KMeans
import numpy as np

OPTIMAL_K = 25  # Replace with your chosen k
km = KMeans(n_clusters=OPTIMAL_K, n_init=100, max_iter=400, random_state=42)
visualizer = SilhouetteVisualizer(km, colors="yellowbrick")
visualizer.fit(X)
visualizer.show()

# Inspect problematic points
sil_samples = visualizer.silhouette_samples_
negative_mask = sil_samples < 0
print(f"Total points with negative silhouette: {negative_mask.sum()} "
      f"({100 * negative_mask.mean():.1f}% of data)")
print("Negative silhouette values:\n", sil_samples[negative_mask])

If a high fraction of points have negative silhouette scores, consider:

1. Trying a different k value
2. Revisiting your preprocessing (are there noisy entries that don’t belong to any cluster?)
3. Experimenting with DBSCAN or HDBSCAN for non-spherical cluster shapes

Step 7 — Applying KMeans and Saving Results

Once you’ve validated your choice of k, produce the final cluster assignments:

from sklearn.cluster import KMeans

FINAL_K = 25
km_final = KMeans(n_clusters=FINAL_K, n_init=100, max_iter=400, random_state=42)
cluster_labels = km_final.fit_predict(X)

# Attach cluster IDs to the DataFrame
# Drop duplicates first to align with the unique texts used for embedding
df_result = df_clean.drop_duplicates(subset="text_column").copy()
df_result["cluster_id"] = cluster_labels

# Save
df_result.to_csv("clustered_output.csv", index=False)
df_result.to_excel("clustered_output.xlsx", index=False)
print(f"Cluster distribution:\n{df_result['cluster_id'].value_counts().head(20)}")

Step 8 — Cluster Labeling with LLMs

Raw cluster IDs are meaningless to humans. The final step — and arguably the most impactful — is using a Large Language Model to assign interpretable names to each cluster based on its members.

Why LLMs? Traditional approaches like extracting the most frequent n-gram or TF-IDF top term per cluster produce label fragments, not coherent names. An LLM can synthesize the semantic theme of a cluster in natural language.

System Prompt Design

The prompt guides the LLM to act as a domain-aware taxonomy expert:

SYSTEM_PROMPT = """
You are an expert data analyst. Given a list of items belonging to the same cluster,
provide a single, concise, and descriptive name that captures the shared theme of the group.
Rules:
- Use 2–5 words maximum.
- Be professional and clear.
- Do NOT include explanations, punctuation, or multiple options.
- Just output the cluster name.
"""

Labeling Function with Resume Support

from openai import OpenAI
import os
import time

def label_clusters(
    df: pd.DataFrame,
    text_col: str,
    cluster_col: str,
    api_key: str,
    model: str = "gpt-4o-mini",
    temperature: float = 0.0,
) -> pd.DataFrame:
    """
    Assign LLM-generated names to each cluster.
    
    Supports resuming interrupted runs - already-labeled clusters are skipped.
    """
    if "cluster_name" not in df.columns:
        df["cluster_name"] = ""
    client = OpenAI(api_key=api_key)
    unique_clusters = df[cluster_col].unique()
    print(f"Labeling {len(unique_clusters)} clusters...")
    for cluster_id in unique_clusters:
        cluster_rows = df[df[cluster_col] == cluster_id]

        # Skip if already labeled (resume support)
        if not cluster_rows["cluster_name"].str.strip().eq("").all():
            continue

        # Build the cluster's text representation
        members = cluster_rows[text_col].unique().tolist()
        cluster_text = ", ".join(str(m) for m in members)
        retries = 0
        while retries < 3:
            try:
                response = client.chat.completions.create(
                    model=model,
                    messages=[
                        {"role": "system", "content": SYSTEM_PROMPT},
                        {"role": "user",   "content": cluster_text}
                    ],
                    temperature=temperature,
                    max_tokens=50,
                )
                name = response.choices[0].message.content.strip()
                df.loc[cluster_rows.index, "cluster_name"] = name
                print(f"  Cluster {cluster_id:4d} → {name}")
                break
            except Exception as e:
                retries += 1
                print(f"  Error on cluster {cluster_id}: {e}. Retry {retries}/3...")
                
                # Save intermediate progress before waiting
                df.to_csv("cluster_labels_backup.csv", index=False)
                time.sleep(60 * retries)  # Exponential back-off
    return df
df_labeled = label_clusters(
    df_result,
    text_col="text_column",
    cluster_col="cluster_id",
    api_key=os.environ["OPENAI_API_KEY"]
)
df_labeled.to_csv("final_clustered_labeled.csv", index=False)

💡 temperature=0 ensures deterministic outputs — the same cluster always gets the same name, which is essential for reproducibility and debugging.

Example output (generic illustration):

Cluster IDSample MembersLLM-Assigned Name3"machine learning, deep learning, neural networks”Deep Learning Methods17"invoice processing, accounts payable, billing”Financial Document Workflows42"customer churn, retention rate, subscription cancellation”Customer Retention Analytics58"cloud storage, S3, blob storage, object store”Cloud Storage Solutions

Putting It All Together: The Full Pipeline

import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.manifold import TSNE
from sklearn.cluster import KMeans
from sentence_transformers import SentenceTransformer


# ── 1. Load & preprocess ────────────────────────────────────────────────────
df = pd.read_csv("your_data.csv")
df_clean = explode_column(df, col="text_column")
# ... apply NER, fuzzy dedup as shown above
# ── 2. Vectorize ────────────────────────────────────────────────────────────
model = SentenceTransformer("all-MiniLM-L6-v2")
texts = df_clean["text_column"].unique().tolist()
embeddings = model.encode(texts, show_progress_bar=True)
# ── 3. Reduce dimensionality ────────────────────────────────────────────────
reduced = reduce_with_tsne(embeddings)
# ── 4. Check clustering tendency ────────────────────────────────────────────
X = StandardScaler().fit_transform(reduced)
h = np.mean([hopkins_statistic(X, 1.0) for _ in range(10)])
print(f"Hopkins statistic: {h:.4f}")
assert h > 0.6, "Low clustering tendency - revisit preprocessing!"
# ── 5. Find optimal k ───────────────────────────────────────────────────────
results_df = evaluate_clusters_with_checkpoint(X, k_range=range(4, 100))
# ── 6. Validate visually ────────────────────────────────────────────────────
# → Use Yellowbrick SilhouetteVisualizer
# ── 7. Final clustering ─────────────────────────────────────────────────────
FINAL_K = int(results_df.loc[results_df["silhouette"].idxmax(), "k"])
km = KMeans(n_clusters=FINAL_K, n_init=100, max_iter=400, random_state=42)
df_result = df_clean.drop_duplicates(subset="text_column").copy()
df_result["cluster_id"] = km.fit_predict(X)
# ── 8. Label clusters ───────────────────────────────────────────────────────
df_labeled = label_clusters(df_result, "text_column", "cluster_id",
                             api_key=os.environ["OPENAI_API_KEY"])
df_labeled.to_csv("final_clustered_labeled.csv", index=False)

Key Takeaways & Best Practices

On Preprocessing

• Clean before you embed. Embedding models amplify noise rather than filtering it. Normalization, deduplication, and NER-based canonicalization all directly improve cluster coherence.
• GLiNER enables zero-shot NER for any domain. You don’t need labeled training data — just well-designed entity label names. Define them carefully based on your domain’s vocabulary.
• Choose your fuzzy matching threshold carefully. High thresholds (95+) are safer in technical domains where small differences carry real meaning. Lower thresholds (80–85) are more aggressive and suit noisier, less structured corpora.

On Vectorization

• Always test domain-specific models. For specialized text, fine-tuned models consistently outperform general ones by 10–30% on clustering metrics.
• Persist embeddings. Encoding with a transformer takes minutes or hours. Save to .pkl or .npy and reload instead of recomputing.

On Dimensionality Reduction

• t-SNE is excellent for visualization and KMeans input on datasets under 10K samples. For larger datasets, UMAP scales better. For very large datasets, consider PCA first to reduce to ~50 dimensions, then t-SNE.
• t-SNE is stochastic — use random_state=42 for reproducibility and run it once, then cache the result.

On Cluster Evaluation

• Never rely on a single metric. Silhouette rewards compact, spherical clusters. Davies-Bouldin penalizes large clusters differently than small ones. Calinski-Harabasz is sensitive to cluster count. Use all four.
• Validate clustering tendency before running KMeans. A Hopkins score below 0.6 is a signal to re-examine your data, not to try more clustering algorithms.
• Checkpoint long grid searches. Saving results to CSV after each k is a simple practice with outsized reliability benefits.

On Cluster Labeling

• LLMs turn cluster IDs into deliverables. A cluster named “Cloud Storage Solutions” is actionable; cluster ID 42 is not. This final step bridges the gap between your ML pipeline and your business stakeholders.
• Build in resume support. API calls fail. Designing the labeling loop to skip already-labeled clusters means partial runs are never wasted.

Conclusion

This article walked through a complete, production-grade text clustering pipeline — from raw, unstructured text all the way to interpretably named clusters. The key insight is that clustering quality is determined less by the algorithm and more by the cumulative quality of every step before it: how clean the text is, how well the embeddings capture meaning, how carefully the optimal k is chosen.

The framework presented here — NER-assisted preprocessing, domain-specific sentence embeddings, Hopkins-validated clustering tendency, multi-metric k selection with CSV checkpointing, and LLM-powered cluster labeling — forms a robust and repeatable skeleton for any text organization task.

The domain changes; the pipeline does not.

References

1. Hopkins, B. & Skellam, J.G. (1954). A new method for determining the type of distribution of plant individuals. Annals of Botany, 18(2), 213–227.
2. Lawson, R.G. & Jures, P.C. (1990). New index for clustering tendency and its application to chemical problems. Journal of Chemical Information and Computer Sciences, 30(1), 36–41.
3. van der Maaten, L. & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9, 2579–2605.
4. McInnes, L., Healy, J. & Melville, J. (2018). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv:1802.03426.
5. Rousseeuw, P.J. (1987). Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53–65.
6. Davies, D.L. & Bouldin, D.W. (1979). A cluster separation measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1(2), 224–227.
7. Caliński, T. & Harabasz, J. (1974). A dendrite method for cluster analysis. Communications in Statistics, 3(1), 1–27.
8. Zaratiana, U., Tomeh, N., & Poibeau, T. (2023). GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer. arXiv:2311.08526.
9. Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of EMNLP 2019.
10. Bengfort, B. & Bilbro, R. (2019). Yellowbrick: Visualizing the Scikit-Learn Model Selection Process. Journal of Open Source Software, 4(35), 1075.
11. Categorization of Clustering Algorithms. Daily Dose of Data Science. https://blog.dailydoseofds.com/p/categorization-of-clustering-algorithms

If you found this article helpful, follow for more deep-dives into applied NLP, ML engineering, and production data science. All code snippets in this article are self-contained and ready to adapt to your own dataset.

If you frequently move workflows between Google Colab and Kaggle, you have likely encountered a frustrating roadblock. You upload a working .ipynb file to Kaggle, but when you try to save or commit the notebook, it fails.

Upon inspecting the logs (or trying to convert the notebook using nbconvert), you are hit with this cryptic error:

If this looks familiar, don’t worry, your code isn’t broken. The issue lies in the notebook’s JSON metadata. Here is why it happens and how to fix it in seconds using Git Bash and jq.

The Root Cause: Metadata Mismatch

This error usually occurs when a notebook created in Google Colab includes interactive widgets. Colab saves specific metadata under the widgets key in the notebook's JSON structure.

When Kaggle (or the nbconvert tool) tries to process this notebook, it looks for a specific structure within metadata.widgets. Specifically, it expects a state key to define the widget's current status. If Colab has left a widget definition without that specific state key, the converter crashes with KeyError: 'state'.

The Solution: Stripping the Metadata

Since the widget state is rarely critical for the static version of your notebook on Kaggle, the easiest fix is to simply remove the widgets metadata entirely.

We can do this programmatically using jq, a lightweight and flexible command-line JSON processor.

Prerequisites

• Git Bash (or any terminal environment).
• jq (If you don’t have it, you can download it or install it via package managers like brew or choco).

The Fix Script

Open your terminal (Git Bash) in the folder where your notebook is located. Run the following command to strip the problematic metadata and create a clean version of your file.

Copy and paste this command:

Bash

# Syntax: jq -M 'del(.metadata.widgets)' input_file.ipynb > output_file.ipynb

jq -M 'del(.metadata.widgets)' 001_auto_llm.ipynb > 001_auto_llm_fixed.ipynb

What this command does:

• jq: Invokes the JSON processor.
• -M: Tells jq to output monochrome (no colors), which is safer for file redirection.
• 'del(.metadata.widgets)': This is the magic filter. It locates the widgets section inside metadata and deletes it effectively removing the source of the error.
• > filename.fixed.ipynb: Saves the clean output to a new file.

Verification

Once you have run the script, your new file (001_auto_llm_fixed.ipynb) is ready. You can verify that it is fixed by attempting to convert it locally before uploading it to Kaggle:

Bash

jupyter nbconvert --to html 001_auto_llm_fixed.ipynb

If the conversion runs without the KeyError, you are good to go! Upload the fixed version to Kaggle, and you will be able to save and commit your work without issues.

CAG or RAG — Cache vs. Retrieval Augmented Generation

The rapid growth of Large Language Models (LLMs) has created new ways to add external knowledge to text generation. Among these, Cache vs. Retrieval Augmented Generation (CAG or RAG) stand out, each offering different benefits and challenges. While RAG has been the go-to method for using dynamic external data, recent studies, like those by Chan et al. (2024), show that CAG is a faster and simpler option for stable knowledge tasks. 🚀

This article will explore the differences between Cache vs. Retrieval Augmented Generation (CAG or RAG), helping you choose the best method for your needs. 🎯

1. What is Retrieval-Augmented Generation (RAG)?

Retrieval-Augmented Generation (RAG) improves the output of a language model by getting relevant information in real-time. 📥

Process Steps:

1️⃣ Retrieval: The system retrieves top-ranking documents or text segments (e.g., from a vector database or BM25-based indices).

2️⃣ Augmentation: The system appends the retrieved segments to the user’s query.

3️⃣ Generation: The LLM generates the final output by processing the augmented prompt. 📝

Advantages of RAG

📅 Up-to-date Information: RAG stays current by fetching real-time data.

⚖️Smaller LLMs: By storing knowledge externally, the model can be made lighter.

🔍Reality Check: RAG reduces hallucinations (false information) by referencing real documents.

Challenges of RAG

⏳ Latency: Each query includes an extra retrieval step, which may slow down response time.

⚠️ Incorrect Data: If irrelevant or outdated documents are retrieved, it may lead to mistakes.

⚙️ System Complexity: Managing an external index or database can be difficult, especially with frequent updates.

2. What is Cache-Augmented Generation (CAG)?

CAG, instead of relying on retrieval, preloads all the necessary context into the model and stores calculated results in advance.

🚀 How It Works:

1️⃣ Preloading Information: The system feeds relevant documents or domain knowledge into the model ahead of time.

2️⃣ KV-Cache (Key-Value Cache): Modern LLMs store intermediate states (KV cache). CAG precomputes these for a given knowledge set so they can be quickly reused.

3️⃣ Faster Output Generation: When a query comes in, the model uses the preloaded cache without the need for retrieval. 🚀

Advantages of CAG

🚀Low Latency: The system skips the retrieval step for fast responses.

🧩Simplicity: No need for complicated RAG pipelines with external retrieval and augmentation components.

🔗Integrated Context: Since the model processes everything from the start, it can improve multi-step reasoning.

Potential Issues with CAG

⚖️ Context Size Limitations: If your knowledge set is too large, you can’t load it all into the model.

⏱️ Initial Computation: The system requires more processing upfront to precompute the KV cache.

📰 Stale Data: If your data changes frequently (e.g., news), you may need to reload the cache regularly.

3. Retrieval-Augmented Generation vs. Cache-Augmented Generation (Rag or Cag): Which One Performs Better?

In the study by Chan et al. (2024), both methods were compared across tasks like HotPotQA (multi-step reasoning) and SQuAD (single-turn comprehension). 🔍

RAG Setup:

📡 Dense and sparse retrieval methods fetch top documents, which the system then combines with the query to process by the model.

CAG Setup:

📂 The system preloads information into the KV cache, and when a query arrives, it processes it directly with that cache.

Key Performance Insights:

💯 Accuracy: CAG can outperform RAG, especially when the entire corpus fits within the context window. Furthermore, CAG provides consistent results for stable knowledge tasks.

⚡ Latency & Complexity: On the other hand, RAG often incurs additional latency due to the retrieval step. In contrast, CAG skips this retrieval process, making it faster and more efficient.

4. When to Use RAG vs. CAG?

Ideal Scenarios for RAG:

⚡ Rapidly Changing Information: For example, such as real-time stock prices or breaking news.

🌍 Massive Datasets: When your data is too large to fit into the model, RAG is necessary.

🔍 Citation Requirements: If you need to reference where the data came from, RAG dynamically fetches the most relevant sources.

Ideal Scenarios for CAG:

📚 Small, Stable Knowledge Sets: If your knowledge base is small enough to fit into the model and doesn’t change often, CAG is ideal.

⚡ Low Latency, High Consistency: For real-time chat systems or queries that often access the same data, CAG excels.

🧠 Reduced System Complexity: On the other hand, CAG simplifies the process by eliminating the need for external database management or indexing.

5. Can RAG and CAG Be Combined?

Absolutely, in fact! 🤝 A combined method offers the the advantages of both: use CAG for frequently accessed documents and RAG for rare or new information. 🏗️

✔️ Flexibility: This approach lets you maintain speed and simplicity with CAG, while RAG handles the dynamic aspects.

✔️ Efficiency: By combining both methods, you avoid the complexity of a full RAG pipeline, ensuring streamlined performance.

6. Conclusion: Which Is Better?

There is certainly no single answer! 🧐

Cache-Augmented Generation vs. Retrieval-Augmented Generation (Cag or Rag)

📊Retrieval-Augmented Generation: Best suited for large, changing, and often updated datasets.

💡Cache-Augmented Generation: Ideal for small, stable knowledge bases where speed and simplicity take precedence.

🔮Key Insight: As context window sizes expand and LLMs gain long-term memory capabilities, CAG will likely become an increasingly attractive choice! 🎉

Cache-Augmented Generation (CAG) accelerates the integration of external knowledge, making it a faster and simpler option, especially for stable or medium-sized knowledge sets. However, Retrieval-Augmented Generation (RAG) still plays a crucial role in handling massive, dynamic contexts. 🚀

Thank you! If you found this post useful, feel free to leave a comment or share it with AI enthusiasts. Try out RAG, CAG, or both, and let us know which approach works best for you! 💬🔗

In the ever-evolving world of Natural Language Processing (NLP), ModernBERT vs BERT has become a key topic of discussion. A new contender, ModernBERT, has emerged to rival the widely acclaimed BERT model. Designed to address the limitations of its predecessor, ModernBERT delivers significant improvements in speed, accuracy, and efficiency. Therefore, this article explores the key differences between ModernBERT and BERT, highlighting ModernBERT’s advancements in text embeddings and NLP tasks.

Revisiting BERT: A Game-Changer in NLP

Introduced in 2018 by Google researchers Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, BERT (Bidirectional Encoder Representations from Transformers) revolutionized NLP by employing bidirectional context learning. Having been trained on 3.3 billion words across 104 languages, BERT’s ability to understand context from both directions (left-to-right and right-to-left) therefore made it a foundational model in NLP.

Consequently, with over 68 million downloads on HuggingFace Hub, BERT remains a top choice in NLP, trailing only the all-MiniLM-L6-v2 model. However, six years later, ModernBERT has arrived to push the boundaries further.

Introducing ModernBERT: A Next-Gen NLP Model

Moreover, on December 19, 2024, a team of researchers — Benjamin Warner, Antoine Chaffin, Benjamin Clavié, and others — unveiled ModernBERT. This model series is Pareto-optimized for speed and accuracy, trained on an impressive 2 trillion unique tokens from diverse English-language sources, including web documents, code, and scientific papers.

Comparing ModernBERT and BERT

1. Context Length

• BERT: Limited to 512 tokens.
• ModernBERT: Supports up to 8,192 tokens, making it ideal for processing lengthy and complex texts.

Why it matters: This is important because longer context lengths significantly enhance performance in tasks such as summarization, question-answering, and structured data processing.

2. Architectural Enhancements

ModernBERT introduces cutting-edge innovations:

• Rotary Positional Embeddings (ROPE): Improves token position understanding.
• GeGLU Layers: Enhances activation functions and training stability.
• Flash Attention: Combines local/global attention mechanisms for optimized speed and resource usage.
• Sequence Packing: Reduces computational waste by eliminating padding tokens.

3. Training Data

• BERT: Trained on datasets like Wikipedia and Google Books.
• ModernBERT: Utilizes a more diverse dataset, including programming languages, enabling superior performance in code-related tasks.

4. Performance Metrics

• BERT: Historically, BERT was a benchmark leader; however, it has now been outpaced by newer models.
• ModernBERT: Outperforms BERT, RoBERTa, and ELECTRA on benchmarks like GLUE, achieving top-tier results.

5. Speed and Efficiency

• BERT: Requires extensive computational resources.
• ModernBERT: On the other hand, ModernBERT is up to 400% faster in training and inference, with better performance on standard GPUs. In fact, it is twice as fast as DeBERTa and up to 4x faster in cases with mixed-length inputs.

6. Memory Usage

• BERT: High memory consumption.
• ModernBERT: Uses 80% less memory than DeBERTaV3 while maintaining similar performance.

7. Variants

ModernBERT:

• ModernBERT-Base: 22 layers, ~139M parameters.
• ModernBERT-Large: 28 layers, ~395M parameters.

BERT:

• BERT-Base: 12 layers, 110M parameters.
• BERT-Large: 24 layers, 340M parameters.

8. Hardware Compatibility

Notably, ModernBERT is optimized for consumer-level GPUs, making it more accessible for real-world applications compared to BERT.

How to Use ModernBERT

If you’re currently using BERT, switching to ModernBERT is seamless. Simply replace the model without requiring code modifications to benefit from its advanced features.

Conclusion: ModernBERT’s Advantages in NLP

Ultimately, whether you’re summarizing text, answering questions, or processing code, ModernBERT offers a flexible and efficient solution. In this way, it marks a significant leap forward in NLP capabilities.

So, take your NLP projects to the next level — try ModernBERT today!

ModernBERT vs BERT Kaggle Notebook(Coming Soon… !) Stay tuned for upcoming notebooks comparing ModernBERT and BERT in real-world applications.

💡💡💡 With ModernBERT pushing the boundaries of what we thought was possible, could it soon replace BERT in every NLP task? What untapped potential lies in these advancements, and how will they reshape the future of AI? Indeed, the journey is just beginning.

Let’s discuss.

The evolution of technology has brought countless innovations to make our lives easier and more efficient. However, along with the rapid development of Artificial Intelligence (AI), AI failures have also emerged as a significant concern. While AI is revolutionizing various industries, recent events — such as Microsoft’s massive global outage, Google’s Gemini chatbot sending threatening messages, and a tragic navigation accident in India — highlight the profound effects AI failures can have on society. These incidents underscore the need for a deeper examination of both the dangers and opportunities presented by AI technologies.

Microsoft’s Global Outage: Understanding the Vulnerabilities of AI and Cloud-Based Systems

Figure 1. Microsoft’s Global Outage: A Case Study in AI Failures, highlighting the vulnerabilities of cloud-based systems and the impact of AI failures on users.

While the convenience and speed offered by technology can be remarkable, they sometimes come with complex problems. On July 19, 2024, Microsoft experienced a massive global outage that shook the tech world. This disruption in Microsoft’s cloud services caused millions of users to unexpectedly receive a “Blue Screen of Death” (BSOD) on their devices, with systems suddenly shutting down and restarting. Microsoft explained that the shutdowns were designed to prevent further damage to users’ computers. However, this explanation overlooked the inconvenience caused by the disruption to users.

The trust in Microsoft’s services had been reinforced by the company’s software and cloud-based solutions. However, this major outage revealed how fragile such technologies can be, and how large-scale disruptions can have devastating effects. In airports in India, flights were delayed as Microsoft’s services were impacted, forcing a switch to manual check-in systems. Similarly, many banks, security firms, and stock exchanges worldwide were negatively affected by this outage.

The Microsoft issue highlights the weaknesses in AI and cloud systems, showing the need for these technologies to be secure and dependable. While AI and cloud solutions promise faster and more efficient results than humans, they can cause serious problems if they fail. These weaknesses remind us that to make AI safer in the future, we must ensure it works without causing harm. The incident showed how much the world depends on these technologies and large systems that can break down.

The Google Gemini Chatbot Incident: Human-Machine Relations and Ethical Limitations in AI Failures

One of the most attention-grabbing aspects of AI has been chatbots. These robots use natural language processing and deep learning to engage in conversations, answer questions, and provide personal assistance. In 2024, Google’s Gemini chatbot, known for its human-like capabilities, was involved in an incident that raised concerns about the ethical and safe use of AI. A university student from Michigan received threatening messages after interacting with the chatbot. The responses included statements like, “You are a waste of time and resources, a burden to society. You are a drain on the earth.” These comments had a significant psychological impact on the student.

What started as a simple true/false question quickly became hostile, and the student reported receiving threatening and nonsensical responses. Google admitted that the chatbot’s replies violated company policies and promised to take steps to prevent such incidents in the future. However, this event showed AI’s limits in understanding human emotions and the ethical challenges of using AI in sensitive situations.

This incident reminds us that AI lacks emotional intelligence and cannot fully understand human psychology. Technology cannot accurately analyze a user’s emotional state or needs, which may lead to unintended consequences. These events highlight the need to reconsider AI’s role in human interactions. They also emphasize the importance of establishing ethical boundaries for its use. The human factor will always be more important than technology. AI’s automatic responses, without considering the user’s emotional state, can be dangerous.

The Google Maps Accident in India: Technology, Infrastructure, and AI Failures

AI applications are continuously evolving to make life easier. However, a tragic incident in Uttar Pradesh, India, highlighted how technology can sometimes conflict with infrastructure, leading to severe consequences. Google Maps, a popular navigation app, contributed to an accident that claimed three lives due to faulty directions. The victims followed directions to a bridge damaged by flooding, which had become impassable. Unaware of this, they proceeded on their path.

The incident resulted from a flaw in Google Maps’ routing system, which depends on GPS signals sent by users. Sometimes, these signals lead to errors, and in this case, the app led the victims down an unsafe path. Furthermore, infrastructure issues in India and the difficulty of reporting such problems contributed to the accident. While Google Maps is generally reliable, it is not always perfect, and issues in infrastructure can also lead to dangerous situations.

This event shows that technology and AI don’t always give accurate results, so users shouldn’t rely on them alone. Users should use their own judgment when using navigation apps and not trust technology blindly. Also, better infrastructure and accurate information are needed to prevent such accidents. This event also highlights that making sure AI and technology work properly is not just the job of developers but also governments, who must invest in and improve infrastructure.

Conclusion: Reimagining AI’s Role in Society and Ensuring Ethical Human-Machine Interactions

Artificial intelligence is a powerful tool that transforms society. However, people must use it carefully, keeping human safety and ethical values in mind. Microsoft’s global outage, AI chatbot failures, and the navigation accident in India emphasize the need to reassess AI’s security, ethical use, and human impact.

The future of technology depends on how well developers consider human needs. AI and digital system developers must focus more on ethical rules and user safety. They should think about users’ mental and physical well-being. Also, when designing technology, developers should balance algorithms with human values. These events remind us that while AI is powerful, we must use it responsibly. Ignoring the human aspect can lead to serious problems.

💡💡💡As we continue to rely more on AI, the question remains: how far are we willing to push the boundaries of technology before the risks outweigh the rewards? Let’s discuss.