A few weeks ago, I published a VSCode/Cursor extension on a whim. It detects and displays the project’s logo in your editor’s status bar and explorer panel. Not a world-changing technology, but a small quality-of-life widget that I wanted but never had the time (or honestly, the TypeScript skills) to build.

I’m not a frontend developer. I’m not a TypeScript developer. I’m a data scientist and AI engineer who spends most of his time with Python, modeling domain knowledge to data, creating pipelines, and in recent times, mostly building LLM-powered systems. Stuff like that were firmly in someone else’s jurisdiction (of the kind of thing you’d open a ticket for, wait a sprint, and then accept a someone else’s version of what you actually wanted). So I let go.

A Timeline of Concessions

There’s a history here of which the LLM revolution is just the visible tip. Code has been slowly, methodically stolen from us, and we’ve been in denial about it the whole time.

It started harmless, with books. O’Reilly manuals, thick as bricks, that encoded the knowledge of the craft. You didn’t need to know everything if you had the right shelf. Knowledge moved from the heads of experts into something accessible, transferable.

Then came blogs. Stack Overflow. The knowledge became searchable. You didn’t even need to read the book anymore — you needed to know how to use Google. The expert’s thought process, compressed into a forum thread, where you can also inflict vengeance for bad answers by other mere humans. Stealing someone else’s algo was common practice.

Then came ChatGPT. Now a real tectonic plate shifted. The loop

problem → search → read → adapt → paste→ refactor

collapsed into a single conversation. “How do I do X in TypeScript?” became a dialogue, not a scavenger hunt.

GitHub Copilot and similar tools took the next slice. Now you didn’t even need to finish your thought and the code already met you halfway (or even just right after you wrote the function name). The boilerplate and the obvious implementation just magically conjured itself in front of you.

And then Cursor arrived. And then Claude Code. And the agentic workflows. We know where we are today. That’s when the jurisdiction didn’t just shrink — it collapsed.

So Now What’s New?

Each handover felt uncomfortable. Each one was met with some version of: “But you need to understand the code you’re writing.” And each time, that was partially true, until it wasn’t. Until the next layer arrived, and the previous anxiety looked odd. If you have co-workers that insist they know every piece of code they wrote by heart, they are either lying or just terribly unproductive.

What’s left in your jurisdiction today is: the what and the why. Not the How. The domain knowledge. The product sense. The architectural judgment. The ability to look at a system and ask: what problem are we actually solving here?

That’s not a small thing. That’s everything!

The Hardest Part Isn’t the Code

The good thing is that’s not new. But still, it seems like letting go of the code is psychologically hard.

For engineers especially, coding is identity. The ability to sit down, understand a system at the function level, and craft a solution from scratch, is what the industry trained us to value. It’s what interviews test. It’s what the senior title rewards. It’s what separates the “real engineers” from everyone else. We are not that far from the day that even the first part of tackling a problem — which is simply understanding it — will be taken out of our jurisdiction as well.

Watching an AI write your code while you describe what you want feels, the first few times, like cheating. Like you’re not really building it and someone will find out. Imposter syndrome from your first post-graduation job is back to hunt you down. People used to (and still) chuckle apologetically when they vibe-coded something, a word that used to be almost derogatory. This term is very 2025, and it died way quicker than expected. If you know people who are still using it to tease their colleagues, safely tell them they are boomers both in humor and in terms of AI technology adoption.

But here’s the thing: nobody finds out, because there’s nothing to find out. The thing works. The users don’t care how it was built. The code is reviewed, the tests pass, the logic is sound. The question of whether you personally typed the curly braces is a question that was already becoming irrelevant. We’ve just finally crossed the threshold where it’s obvious.

We stopped being coders a long time ago. When we started copy-pasting from Stack Overflow, we became assemblers. When Copilot arrived, we became directors. Now, with agentic workflows, we’re becoming something new again. And the transition, as always, is uncomfortable for the people who built their identity on the previous layer.

Don’t get me wrong — the code your agent produces is your code and your responsibility. You still need to make sure good tests are included and that it works. You need to make sure there are no potential secutirty disasters (or just normal disasters because you didn’t even have a quick glance we quick glances are important). This all depends on your product and how critial its functioning is. If you wrote software for Artemis II is one thing, if it’s a scraper that is doom to fail anyway sometimes — it is another.

What a Data Scientist Can Build Now

Let me be concrete about what changed for me.

Before, building a VSCode extension required: learning the extension API, understanding the TypeScript ecosystem, knowing how to structure a VS Code project, debugging the activation events, figuring out the publish workflow, handling the manifest, the icons, the marketplace metadata. It required, at minimum, a weekend of reading documentation and probably a few frustrated Stack Overflow tabs.

Now? I described what I wanted. I approved a plan. I reviewed the components at a high level. I made decisions about behavior and the implementation happened around those decisions. I still read the code and I still caught issues. I definitely made architectural calls. For now, we still have that layer (but don’t get used to it).

The Problem That Proved the Point

I needed this extension because my project count exploded. Two years ago I was juggling two, maybe three projects at once — the normal load of a data scientist with a roadmap and a backlog.

But now, every spark of an idea could become a POC in hours. Every POC could reach production in days. The pipeline from I wonder if we could… to here’s a working demo compressed from weeks to an afternoon. And all in parallel. Long-term initiatives I’d been pushing forward in spare moments started moving fast. New ideas stopped dying on the whiteboard and I didn’t need “to ask time” for stuff or get a nod of approval from Product or Leadership. I went from two projects to five, then ten. Dispatch work and move on to the next one.

And that’s when I needed the extension and not as a toy project. Because I was switching between so many repos, so many tech stacks, so many contexts, that I genuinely needed a visual anchor to tell me where I was.

The 100x Operator

There’s a new archetype emerging. Not the 10x engineer. The 100x operator.

The 100x operator isn’t the best coder in the room. They might not be a coder at all, in the traditional sense. What they are is: deeply fluent in the problem domain, architecturally opinionated, and most importantly: AI-aware. They know how to break a problem into components. They know how to review what the AI produces, when to trust it and when to push back. They know how to chain tools, manage context, and maintain quality without drowning in implementation.

So as Elsa says, let it go. You are no longer paid to write code. You are paid to solve problems at a much higher level.

Let It Go: You’re Not Getting Paid To Write Code Anymore was originally published in Healthee on Medium, where people are continuing the conversation by highlighting and responding to this story.

Become a Powerhouse Developer with Claude Code: Rules, Skills and Agents

Automating the Dev Loop: Linear Tickets, GitHub Branches, and CI Monitoring with Claude Code

Every developer knows the friction of starting new work. You have an idea, but before writing a single line of code, there’s overhead: create a ticket, format a branch name, remember which project it belongs to, push and wait for CI, check logs when it fails. These micro-tasks aren’t hard, but they accumulate into cognitive load that fragments your focus.

Claude Code can integrate directly with Linear and GitHub to handle this overhead automatically. This article walks through a production workflow covering ticket creation, branch management, CI monitoring, code review handling, and multi-repository coordination.

The Problem

You’re deep in a debugging session. You’ve finally understood the root cause: a hash calculation that differs between two services. The fix is clear in your mind. But before you can write it, you need to: open Linear, create a ticket, think of a good name, assign it to yourself, copy the ticket ID, switch to terminal, construct a branch name following your team’s convention, run git checkout, and finally… wait, what was the fix again?

This is the real cost of process overhead. It’s not the time spent on each step (seconds, really). It’s the cognitive interruption. Every context switch forces your brain to serialize its current state, handle the interruption, then deserialize back to where you were. Some of that state inevitably gets lost.

The individual steps exist for good reasons — ticketing creates accountability, branch naming enables CI automation, status tracking helps teams coordinate. The problem isn’t the process. It’s that humans are doing mechanical work that machines should handle.

When you encounter a bug or need to implement a feature, the ideal workflow is: describe the problem, then work on the problem. Everything else — ticket creation, branch management, CI monitoring, work logging — should happen automatically in the background while you stay focused on the actual engineering challenge.

That’s what this setup achieves. You tell Claude what you’re working on. Claude handles the ceremony. You stay in flow.

Part 1: Linear Integration via MCP

The Model Context Protocol (MCP) is an open standard that lets AI assistants communicate with external services through a unified interface. Linear provides an official MCP server, which means Claude Code can create tickets, update statuses, and query issues directly through conversation.

Setting it up takes one command:

claude mcp add linear-server https://mcp.linear.app/sse — transport sse

Once connected, Claude gains access to Linear’s full API surface — creating issues, listing them, updating statuses, managing projects. But raw API access isn’t enough. You need to teach Claude *how* you want to use it.

Creating a Skill for Ticket Creation

Skills are workflow definitions written in markdown. They tell Claude what steps to follow when a particular situation arises. Think of them as runbooks that Claude executes on your behalf.

For ticket creation, you want Claude to understand your team structure, infer the right project from context, suggest a good ticket name, and then automatically check out the corresponding branch. Here’s a simplified version of what that skill looks like:

— -

name: new-ticket

description: Creates a Linear ticket before starting work on a feature or fix.

— -

# Create New Linear Ticket

## Instructions

1. **Analyze Context**: Look at the current conversation, working directory,

and recent commits to understand what work is planned.

2. **Suggest Ticket Name**: Generate a concise, descriptive title.

3. **Create Ticket** using Linear MCP:

- Team: Data Science

- Assignee: me

- Status: In Progress

4. **Auto-checkout branch**: Use the `gitBranchName` field from Linear’s response:

```bash

git checkout -b <gitBranchName>

```

The magic here is that Linear’s API returns a pre-formatted branch name based on the ticket. No need to manually construct `username/ticket-id-slug` patterns — it comes from the source of truth.

With this skill in place, starting new work becomes conversational:

User: I need to fix the hash mismatch bug in the docling processor

Claude: Based on our discussion, I suggest:

**Ticket Name**: Fix docling hash mismatch between driller and predictor

**Project**: Driller EOC

**Team**: Data Science

Should I create this ticket?

User: yes

Claude: Created DAT-1517: “Fix docling hash mismatch…”

URL: https://linear.app/healthee/issue/DAT-1517

Switched to branch: yoavvollansky/dat-1517-fix-docling-hash-mismatch

From problem description to tracked ticket to ready-to-code branch in seconds, without leaving your editor.

Part 2: Automatic CI Monitoring

Pushing code and waiting for CI is another flow-breaker. You push, switch to GitHub, wait for the workflow to appear, refresh to see status, and if it fails, dig through logs to find the problem. This back-and-forth can happen multiple times per feature.

Claude Code can use the GitHub CLI (`gh`) to monitor CI status as part of the development loop. The key is telling Claude that CI passing is part of task completion, not a separate concern.

Add this rule to your project’s `CLAUDE.md` file:

After any push to a branch, check GitHub Actions workflow status using

`gh run list` or `gh run view`. If tests fail, analyze the failure logs

and fix the tests. Verify tests pass before considering the task complete.

This single rule changes Claude’s behavior significantly. After pushing, Claude doesn’t just report “pushed successfully” — it follows through to verify the work actually builds and passes tests. If something fails, Claude examines the logs and attempts to fix the issue before reporting completion.

Here’s what that looks like in practice:

Claude: Pushed changes to yoavvollansky/dat-1517-fix-docling-hash-mismatch

Let me check CI status…

$ gh run list — branch yoavvollansky/dat-1517-fix-docling-hash-mismatch — limit 1

STATUS NAME WORKFLOW BRANCH

X CI ci.yml yoavvollansky/dat-1517-fix-docling-hash-mismatch

CI failed. Let me check the logs…

$ gh run view 12345678 — log-failed

Test failure in test_hash_calculation:

AssertionError: Expected hash ‘abc123’ but got ‘def456’

The test is using a hardcoded expected hash. I’ll update it to match the

new hash calculation logic…

[fixes test]

Pushed fix. Checking CI again…

$ gh run list — branch yoavvollansky/dat-1517-fix-docling-hash-mismatch — limit 1

STATUS NAME WORKFLOW BRANCH

✓ CI ci.yml yoavvollansky/dat-1517-fix-docling-hash-mismatch

CI passing. Task complete.

This loop continues until CI passes or Claude encounters something that genuinely requires human judgment. The result is that “done” actually means done — code pushed, tests passing, ready for review.

Part 3: Agents vs Skills

Both skills and agents extend Claude Code’s capabilities, but they serve fundamentally different purposes. Understanding when to use each is key to building effective workflows.

Skills: Context-Aware Workflows

Skills run in the same conversation context as your interaction with Claude. This means a skill can reference anything discussed earlier in the conversation — it has full awareness of what you’ve been working on.

Skills are ideal for:

- User-invocable workflows like `/new-ticket` or `/deploy`

- Tasks that need to understand the current conversation

- Simple tool orchestration where the steps are predictable

The structure is straightforward — a markdown file with frontmatter and instructions:

— -

name: my-skill

description: When to trigger this skill

— -

# Instructions for Claude to follow

…

…

When you invoke a skill, it’s like giving Claude a detailed runbook to follow while maintaining full awareness of your conversation history.

Agents: Autonomous Subprocesses

Agents are different. They spawn as separate subprocesses with their own context, tools, and execution environment. An agent might run in the background, gathering information from multiple sources, and return a single consolidated result.

Agents excel at:

- Autonomous multi-step research

- Tasks that need to query external sources (GitHub, Linear, etc.)

- Background operations that shouldn’t block your main workflow

- Work that benefits from isolation from the main conversation

The structure is similar to skills but includes additional configuration:

— -

name: my-agent

description: When Claude should spawn this agent

model: sonnet

— -

# Agent’s system prompt and instructions

The key difference is that agents don’t automatically see your conversation history. They gather their own context from the sources you configure them to query.

When to Use Which

The decision comes down to where the context lives.

Skills:

- Context: Full conversation history

- Execution: Inline (same process)

- Background: No

- Tool access: All tools

- Best for: Workflows needing chat context

Agents:

- Context: Isolated (unless explicitly shared)

- Execution: Subprocess

- Background: Yes (can run async)

- Tool access: Defined subset per agent type

- Best for: Autonomous research/operations

A practical example: ticket creation works well as a skill because Claude needs to understand what you’ve been discussing to suggest the right ticket name. Work logging works better as an agent because it gathers context from external sources (GitHub PRs, Linear issues, git history) rather than from the conversation.

Backing Rules

Here’s something that’s easy to miss: agents need triggers. An agent definition sitting in `~/.claude/agents/` won’t do anything if Claude doesn’t know when to invoke it.

Every agent should have a corresponding rule in your `CLAUDE.md` that tells Claude when to spawn it:

# Rule 28: CI Monitoring

After any push, check GitHub Actions using `gh run list`. If tests fail,

analyze logs and fix. Use the github-ops agent for complex CI investigations.

# Rule 31: Work Logging

At the end of a work session or after completing significant tasks, use the

daily-work-logger agent to document accomplishments across all repositories.

Without these backing rules, you have agents that exist but never run. The rule defines *when*, the agent defines *how*.

Part 4: Multi-Repository Workflows

Modern software rarely lives in a single repository. A feature might touch an API service and its infrastructure configuration. A bug fix might require coordinated changes to a client library and the server that consumes it. When work spans multiple repos, the coordination overhead multiplies.

The Challenge

Consider a typical cross-repo change: you need to add a parameter to a service’s API (repo A) and update the client that calls it (repo B). The changes are coupled — deploying one without the other breaks things. You need two tickets, two branches, two sets of commits, and careful deployment ordering.

Traditional approach: two terminal windows, two sets of commits, manual tracking of which is deployed, and hoping you don’t forget the deployment order.

Claude’s Cross-Repo Awareness

By configuring agents to search across your entire GitHub organization rather than just the current directory, you can maintain visibility across all your work. The key is using GitHub’s search API rather than local git commands:

Search ALL PRs you authored across the entire org

gh search prs — author=”@me” — created=”>$(date -v-7d +%Y-%m-%d)” — json repository,title,state,url

Get your activity across all repos

gh api “/users/$(gh api user — jq ‘.login’)/events” — jq ‘.[] | select(.type==”PushEvent”)’

When Claude uses these commands, it surfaces work regardless of which directory you’re currently in. A work logging agent configured this way can report:

Found activity in 3 repositories today:

**GetHealthee/Driller** (2 PRs)

- #241: Add hash debug logging — MERGED

- #242: Pass use_cache parameter — IN REVIEW

**GetHealthee/infrastructure** (2 PRs)

- #1345: Update predictor image to v168 — MERGED

- #1346: Add use_cache to predictor schema — MERGED

**GetHealthee/api-gateway** (1 PR)

- #89: Add cache-control headers — IN REVIEW

This visibility is valuable. You can see your full day’s work in one place, not scattered across repo-specific git logs.

Coordinated Tickets

Linear’s issue linking helps manage cross-repo dependencies. When you describe work that spans repositories, Claude can create linked tickets with proper dependency relationships:

User: I need to add a use_cache parameter — it needs changes in both the

infrastructure predictor and the driller client

Claude: This spans two repos. I’ll create linked tickets:

Created DAT-1521: “Add use_cache parameter to predictor” (Infrastructure)

Created DAT-1522: “Pass use_cache to SageMaker calls” (Driller)

→ Linked as “blocked by DAT-1521”

Checking out branches…

In /infrastructure: user/dat-1521-add-use-cache-to-predictor

In /driller: user/dat-1522-pass-use-cache-to-sagemaker

The “blocked by” relationship encodes the deployment order directly in your ticket system. Anyone looking at DAT-1522 can see it depends on DAT-1521 being completed first.

IDE Integration

IDEs like Cursor amplify multi-repo workflows by letting you work with multiple repositories in a single workspace. With several repo folders open together, you get:

- Visual git state: Branch names, uncommitted changes, and sync status for all repos at once in the Source Control panel

- Unified search: Find references across service boundaries without switching contexts

- Split editing: Modify the client call in one pane, the server handler in another, seeing both simultaneously

This visual awareness complements Claude’s API-level awareness. You handle the code changes with full visual context; Claude handles the ticket tracking, CI monitoring, and work logging across all repos simultaneously.

The Deployment Pattern

For coordinated changes across services, a consistent pattern emerges:

1. Create linked tickets with explicit dependencies

2. Implement server-side first (the thing being called)

3. CI passes → merge and deploy server

4. Implement client-side (the thing doing the calling)

5. CI passes → merge and deploy client

Claude can track this progression and prompt you when dependencies resolve:

Claude: DAT-1521 (infrastructure) is merged and deployed.

DAT-1522 (driller) is now unblocked. Ready to merge?

Consolidated Work Logging

You’ve done impressive work across multiple repositories with automated contributions. How do you keep track of it all? How do you clearly communicate progress to your manager? With so much AI-generated code flying around, even you can lose the thread. That’s where daily-work-logger comes in — an agent that ties everything together by generating a single, structured log of your work across all repositories, organized by day and project:

# Weekly Work Log — Week 04, 2026

## Monday, January 20, 2026

### Project: Infrastructure (SageMaker)

- 11:28 **MERGED: Added use_cache parameter** (DAT-1521)

- PR #1346, Files: predictor.py

### Project: Driller

- 11:45 **IN REVIEW: Pass use_cache to SageMaker** (DAT-1522)

- PR #242, Files: docling_utils.py

- Blocked by: DAT-1521 ✓ (now resolved)

One log, multiple repos, automatic correlation via Linear ticket IDs in commit messages. At the end of a week, you have a complete record of what was accomplished without manual tracking.

Part 5: Handling Code Reviews

The development loop doesn’t end when CI passes. There’s still the review gauntlet: automated bots (Cursor Bot, GitHub Copilot), team leads, or — occasionally — actual humans who read your code and have opinions about it.

Review comments create another context-switching problem. You’ve moved on to the next task. A notification arrives: “3 new comments on your PR.” Now you need to context-switch back, remember what the code was doing, parse each comment, decide if it’s valid, implement fixes or craft responses, and push again. For a PR with multiple review rounds, this ping-pong can stretch across days.

The Review Analysis Skill

A skill for handling code reviews fits naturally here because it needs full conversation context — you might have discussed the implementation approach earlier, and that context informs whether a reviewer’s suggestion makes sense.

The `check-git-comments` skill analyzes PR feedback with a critical eye:

— -

name: check-git-comments

description: Analyzes GitHub PR comments and suggests responses or fixes.

— -

# Analyze PR Review Comments

## Instructions

1. **Fetch comments** using `gh pr view — comments` or the GitHub API

2. **For each comment**, analyze:

- Is the feedback technically valid?

- Does it conflict with requirements discussed earlier?

- Is it a style preference vs. a correctness issue?

3. **Categorize and respond**:

- Valid fix needed → suggest the code change

- Valid but already addressed → point to the relevant code

- Disagree with feedback → draft a respectful response explaining why

4. **Present all suggestions** for user approval before taking action

The key here is *critical analysis*, not blind compliance. Automated review bots are useful but imperfect. They flag potential issues without understanding intent. A good review handler distinguishes between:

- Actionable feedback: “This function doesn’t handle the empty array case” → implement the fix

- Style preferences: “Consider using `forEach` instead of `for`” → acknowledge or push back based on team conventions

- False positives: “Unused variable `config`” when it’s actually used in a dynamic import → explain why the bot is wrong

The Workflow in Practice

After pushing a PR, the review cycle might look like this:

[Cursor Bot leaves 4 comments on PR #242]

User: Check the comments on my PR

Claude: Analyzing comments on PR #242…

**Comment 1** (line 45): “Consider adding error handling for null response”

→ Valid. The SageMaker endpoint can return null on timeout. I’ll add a check.

**Comment 2** (line 67): “Magic number 30 should be a constant”

→ Valid style improvement. I’ll extract to CACHE_TTL_SECONDS.

**Comment 3** (line 89): “Unused import: hashlib”

→ False positive. hashlib is used in compute_hash() on line 142. I’ll reply

explaining this.

**Comment 4** (line 112): “Function is too long (>50 lines)”

→ Disagree. This function handles a complete workflow and splitting it would

obscure the logic. Recommend responding with rationale.

Proposed actions:

1. Fix null response handling (Comment 1)

2. Extract magic number to constant (Comment 2)

3. Reply to Comment 3: “hashlib is used on line 142 in compute_hash()”

4. Reply to Comment 4: “Keeping as single function for readability — the steps

are sequential and tightly coupled”

Proceed with these changes and responses?

User: yes

Claude: [implements fixes, posts responses, pushes]

Changes pushed. Responded to comments 3 and 4.

The developer stays focused. They review Claude’s analysis, approve or adjust, and move on. No need to mentally re-enter the code context just to handle review feedback.

Human Reviewers

The same skill works for human reviewers, but the tone shifts. Bot feedback tends to be mechanical and can be addressed efficiently. Human feedback often carries implicit context — a senior engineer’s “consider whether this scales” might be a genuine question or a polite way of saying “this doesn’t scale, fix it.”

Claude can parse the comment, but the judgment call remains with you:

**Comment from @tech-lead** (line 200): “Have we considered the memory

implications of caching these objects?”

Analysis: This appears to be a architectural concern rather than a specific

bug. The current implementation caches parsed documents in memory without

eviction policy.

Options:

A) Add LRU eviction with configurable max size

B) Move to Redis for distributed caching

C) Respond asking for clarification on expected scale

Recommendation: Start with © to understand requirements before implementing.

Draft response: “Good question — currently scoped for single-instance deployment

with ~100 concurrent documents. If we need to scale beyond that, I’d suggest

moving to Redis. What’s the expected load for this service?”

You review, adjust the tone if needed, and approve. The skill handles the mechanical parts — fetching comments, correlating them to code, drafting responses — while you make the judgment calls.

Why a Skill, Not an Agent

Review handling works better as a skill than an agent because:

1. Conversation context matters: You might have discussed why you chose a particular approach. That context helps evaluate whether a reviewer’s alternative suggestion makes sense.

2. Approval is required: You don’t want an agent autonomously responding to your tech lead’s comments. Every response and code change needs human sign-off.

3. It’s interactive: The back-and-forth (“what about comment 3?” / “actually, let’s push back on that one”) benefits from the conversational flow.

Agents work well for autonomous background tasks. Review handling is inherently collaborative — Claude analyzes and suggests, you decide and approve.

Setup Checklist

Getting this workflow running requires a few one-time setup steps:

1. Linear MCP

claude mcp add linear-server https://mcp.linear.app/sse — transport sse

2. GitHub CLI (authenticate if not already)

gh auth login

3. Skills directory (for user-invocable workflows)

mkdir -p ~/.claude/skills/new-ticket

mkdir -p ~/.claude/skills/check-git-comments

Then Create SKILL.md files with your workflows

4. Agents directory (for autonomous operations)

mkdir -p ~/.claude/agents

Create agent .md files

5. Project rules (`CLAUDE.md` in your repo root)

# CI Monitoring

After any push, check GitHub Actions…

# Ticket Creation

Before starting work on main/DEV branch, create a Linear ticket…

etc.

The Result

With these pieces in place, the development loop compresses significantly. Starting new work no longer requires leaving your editor or manually coordinating across tools:

User: I need to add caching to the API

Claude: [creates Linear ticket via MCP]

Created DAT-1520: “Add caching to API”

Switched to branch: user/dat-1520-add-caching-to-api

[writes code]

Pushed. Checking CI…

CI failed: test_cache_invalidation

Fixing…

[fixes test, pushes]

CI passing. DAT-1520 ready for review.

The mechanical overhead disappears. Tickets get created, branches get named correctly, CI gets monitored, work gets logged — all automatically, while you focus on the actual problem you’re solving.

That’s the goal: not to remove process, but to automate it so thoroughly that proper hygiene becomes the path of least resistance.

A Gentle Introduction for Becoming a Powerhouse Developer with Claude Code: Rules, Skills and… was originally published in Healthee on Medium, where people are continuing the conversation by highlighting and responding to this story.

Envision a project management system that integrates data from tools like Linear, Slack, OpsGenie, Sentry, HubSpot, Jira, AWS CloudTrail, Prometheus, and even your code on GitHub. By leveraging LLMs such as OpenAI’s GPT-4 or Lllama-3 you can analyze and synthesize information from these diverse sources to provide real-time, actionable updates — and even suggest solutions — based on this data. This kind of sophisticated integration enhances decision-making and streamlines workflows. As an example, let’s explore a tiny “tip of the iceberg” example and see how to automate daily or weekly task reporting, making this process effortless and effective.

At Healthee, we love using Linear. It’s flexible, it’s versatile, and it makes project management as smooth as a well-oiled machine. If you manage projects, I can confidently assure you that Linear is a game-changer since it keeps tasks organized, deadlines manageable, insights clear and our sanity intact.

But one thing that we enjoy even more than efficient project management is automating stuff. Imagine you walk into the office (or roll out of bed if you’re working from home), and your manager has already seen a beautifully summarized update of your daily in-progress tasks in Slack. With the joined forces of Linear, OpenAI, and Slack, you can automate this process and keep your manager informed effortlessly. Here’s how to do it.

Step 1: Setting Up Linear

First, you’ll need a Linear account and an API key. If you’re not working with Linear yet, now’s a good time to start. Generate an API Key by navigating to your settings in Linear and creating it.

Step 2: Fetching Tasks from Linear

We will write a Python script to fetch your in-progress tasks from Linear. You can choose which kinds of tasks to include — just in-progress, in review, or others based on your preference. You can get Linear data using a GraphQL query. Here’s the code:

import requests

LINEAR_API_KEY = 'your_linear_api_key'
LINEAR_API_URL = 'https://api.linear.app/graphql'
ALLOWED_STATE_NAMES = {"Todo", "In Progress", "Backlog", "Done"}

def fetch_linear_user_id():
    query = """
    query {
      viewer {
        id
      }
    }
    """
    headers = {
        'Authorization': LINEAR_API_KEY,
        'Content-Type': 'application/json'
    }
    response = requests.post(LINEAR_API_URL, json={'query': query}, headers=headers)
    if response.status_code == 200:
        return response.json()['data']['viewer']['id']
    else:
        raise Exception(f"Failed to fetch user ID: {response.status_code} {response.text}")

def fetch_tasks(user_id, state_names=["Todo", "In Progress"]):
    invalid_state_names = [state for state in state_names if state not in ALLOWED_STATE_NAMES]
    if invalid_state_names:
        raise ValueError(
            f"Invalid state names: {', '.join(invalid_state_names)}. Allowed state names are: {', '.join(ALLOWED_STATE_NAMES)}")

    query = """
    query($assigneeId: ID!, $stateNames: [String!]!) {
      issues(filter: {state: {name: {in: $stateNames}}, assignee: {id: {eq: $assigneeId}}}) {
        nodes {
          title
          description
          state {
            name
          }
          labels {
            nodes {
              name
            }
          }
          projectMilestone {
            name
          }
          project {
            name
          }
          comments {
            nodes {
              body
            }
          }
        }
      }
    }
    """
    variables = {"assigneeId": user_id, "stateNames": state_names}
    headers = {
        'Authorization': LINEAR_API_KEY,
        'Content-Type': 'application/json'
    }
    response = requests.post(LINEAR_API_URL, json={'query': query, 'variables': variables}, headers=headers)
    if response.status_code == 200:
        return response.json()['data']['issues']['nodes']
    else:
        raise Exception(f"Failed to fetch tasks: {response.status_code} {response.text}")

user_id = fetch_linear_user_id()
tasks = fetch_tasks(user_id, ["Todo", "In Progress", "In Review"])

Step 3: Getting Insights from Tasks with OpenAI

Now comes the exciting part: harnessing the power of Large Language Models (LLMs) like OpenAI’s GPT-4 to summarize our tasks. LLMs can take your verbose, detailed task descriptions and distill them into concise, easy-to-read summaries. It can tell you which projects they belong to, how urgent they are, and what labels are attached to them if that’s what you’d like. It’s like having your very own J.A.R.V.I.S. from the Avengers, minus the British accent.

To get started, we’ll send the task details to OpenAI and receive a succinct summary in return. Here’s how you can set up the OpenAI API call:

from openai import OpenAI

openai.api_key = 'your_openai_api_key'

def summarize_tasks(tasks):
    client = penAI(api_key=OPENAI_API_KEY)
    prompt = PROMPT + tasks

    response = client.chat.completions.create(
        messages=[
            {"role": "system", "content": prompt}
        ],
        model="gpt-4o",
        temperature=0.7
    )

    summary = response.choices[0].message.content
    return summary

Crafting Effective Prompts

When using OpenAI, the quality of your results largely depends on how well you craft your prompts. This is known as prompt engineering, and it’s an essential skill to get the most out of AI. A well-crafted prompt can guide the AI to produce precise, relevant, and high-quality responses, while a poorly constructed prompt can lead to vague or off-target results.

Tips for Effective Prompt Engineering

1. Be Specific: The more specific your prompt, the better the AI can understand and respond to your needs. Specify what you want the summary to focus on, such as key actions, deadlines, or pending items.
2. Provide Context: Giving the AI some context can help it generate more accurate summaries. For example, if you’re summarizing tasks for a project, mention the project name and any relevant details.
3. Use Examples: Including examples in your prompt can show the AI exactly what kind of response you’re looking for. This can be especially helpful for complex tasks or detailed summaries.
4. Iterate and Refine: Prompt engineering is often an iterative process. You may need to refine your prompts based on the AI’s initial responses to get the desired output.

I know, you must be thinking “why isn’t he just giving us the bloody prompt?” (with a British accent of course). well, here it is:

PROMPT = """ Im going to tell you my tasks for today as coming from the Linear app API. Please summarize them into bullet points in the following way:
    - Tasks that are "In Progress" should get a bullet point each. Say what they are, which project they belong to, and what the milestone is. Also, mention any labels and comments if exist very briefly. For example, if the task is "Fix the bug", the project is "Driller", the milestone is "v1.0", and the label is "Critical", you should write:
    - Fix the bug in Driller project, v1.0 milestone, Critical". You won't mentioned any comments, because there are none. 
    - All tasks that are "Todo" should be summarized in a single bullet point and mentioned briefly. 
    - Don't be very formal, it is a slack message. 
    - When mentioning a milestone, just mention it in brackets. For example, "Driller project (scoping)". Where "scoping" is the milestone. No need to mention the word "milestone".
    - if two tasks are in the same project and milestone, you can mention them in the same bullet point.
    - don't explicitly mention a label. If the label is "Improvement", you can start the bullet point like "Improving... this and that". If the label is "Feature" you can start the bullet point like "Adding a feature...". If the label is "bug", you can start the bullet point like "Fixing a bug...".
    - arrange groups of tasks in the same project and milestone together (as in following bullet points). 
    
    - IT IS VERY IMPORTANT to only use present tense (as in, "doing", "fixing" etc) to tasks that are "In Progress" only!
    - Tasks that are "Todo" should be mentioned all in one bullet point (the last one) starting with "Pending Todo tasks:..." and then mentioning the tasks in a comma separated list. Just mention the task without anything else (no project, no milestone, no labels, no comments).
    - If there are "In Review" tasks, you must mention them in a single bullet point at the end, starting with "Tasks currently in review:..." and then mentioning the tasks in a comma separated list. Just mention the task without anything else (no project, no milestone, no labels, no comments).
    - Symmaries comments if there are any. Don't just write them but make sense of them as a whole and summarize them.
    
    - The bullet point should be the round one.

These are my tasks in JSON format: 

Step 4: Configuring a Slack App

Before we can post messages to Slack, you need to configure a Slack app and get a bot token. Here’s how you can do it:

1. Create a Slack App: Go to the Slack API and create a new app.
2. Add OAuth Scopes: Under “OAuth & Permissions”, add the following scopes: chat:write, channels:history.
3. Install the App: Install the app to your workspace to get a bot token.
4. Get the Bot Token: Copy the bot token, which you’ll use to authenticate API requests.
5. Make sure you get the channel ID from slack (search it somewhere in channel settings
6. If you’re of the jester kind, search for your manager’s user ID as well.

Now, here’s where the fun comes in. You can really confuse people by naming the Slack app after yourself and using the same profile picture as your avatar. Imagine the bewilderment on everyone’s faces when they see “you” posting updates at 6:30 AM. You digital doppelgänger will be working while you still turn in your bed.

But of course, I would never trick my boss like that. Ever. Really.

Step 5: Slack SDK

from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError

SLACK_BOT_TOKEN = 'your_slack_bot_token'
CHANNEL_ID = 'your_channel_id'
BOSS_USER_ID = 'your_boss_user_id'

client = WebClient(token=SLACK_BOT_TOKEN)

def fetch_boss_message_ts(channel_id, boss_user_id, days_back=1):
    try:
        response = client.conversations_history(
            channel=channel_id,
            limit=100
        )
        messages = response['messages']
        for message in messages:
            if message.get('user') == boss_user_id and "What did everybody do today?" in message.get('text', ''):
                return message['ts']
    except SlackApiError as e:
        print(f"Error fetching messages: {e.response['error']}")
        return None

def markdown_to_block_kit(markdown_text):
    block_kit = [
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": markdown_text.strip()
            }
        }
    ]
    return block_kit

def post_update_to_thread(markdown_text, channel_id, thread_ts):
    blocks = markdown_to_block_kit(markdown_text)
    try:
        response = client.chat_postMessage(
            channel=channel_id,
            text=markdown_text,  # Fallback text for notifications
            blocks=blocks,
            thread_ts=thread_ts
        )
        print("Message posted successfully.")
    except SlackApiError as e:
        print(f"Error posting message: {e.response['error']}")
        return None
    return response

# Example usage
boss_message_ts = fetch_boss_message_ts(CHANNEL_ID, BOSS_USER_ID)
if boss_message_ts:
    markdown_text = f"""
    *Daily Task Summary*

    {task_summary}
    """
    post_update_to_thread(markdown_text, CHANNEL_ID, boss_message_ts)

Here you’re not only posting an update in the channel but also posting an answer to a specific thread where you were asked about their daily tasks. (Yes, I know the question string is quite specific, but not if your manager is using automation of their own to ask the team about their daily updates!) When AI takes over we’re going to be nothing but faded automated replicas of one another conversing with each other, so let’s start now.

Take it Further

This capillary of a pipeline can be enhanced in so many ways. For example, you can ask another LLM agent to further analyze the fetched tickets and suggest ways to tackle them. You can check other Linear boards and see if your task is related to other tasks that took place in other so you know who to talk to or look at other Slack channels to see if a related topic was mentioned recently. You can analyze past issues or logs that are related to your ticket with a slightly more enhanced architecture (such as RAG or chunking). You can even simply summarize important discussions that took place in your channel last week (or month) and remind everyone what was discussed.

The Last Step

(which you’ll do on your own)

Finally, you would want to either run this code on a server with cron or take everything to the next level and have your bot detecting activity on Slack and respond accordingly (not that difficult to implement but an overkill for this example).

Jokes aside

At Healthee, we believe that serious builders need a reliable set of tools. We go by this approach to create custom integrations and solutions that streamline our workflows and enhance our productivity. Naturally, we do this fundamentally with traditional tools and practices for analytics and monitoring (for things like infra, CI/CD, BI etc.), but when you throw AI and LLMs into the pot you you really can craft your own “homemade” solutions. This kind of automation not only keep us informed but also empowers us to make data-driven decisions swiftly.

While the idea of effortlessly informing your manager with automated Slack updates is amusing, the real potential of this setup lies in its ability to keep teams consistently informed and create sophisticated monitoring and alerting systems. By combining various data with event-based sources while leveraging the reasoning capabilities of an LLM like OpenAI’s GPT-4, teams can stay up-to-date and responsive to crucial developments.

So what’s next? Imagine integrating data from common project management tools like Linear together with other developer tools, your code base as well as analytical data all into an LLM-powered system. This setup can automatically generate personalized updates and alerts in real-time for you to act upon. But not just that: if a critical bug is reported in a project milestone, the system can instantly notify relevant team members, summarize the issue, and suggest potential fixes or actions based on historical data, best practices, and internet searches. All about that and other kinds of magic in a future article ⭐️

How to Generate Useful Project Management Insights With AI and LLMs (While You Sleep) was originally published in Healthee on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the realm of co-pays, deductibles, and plans,
We navigate the maze with our healthcare demands.
“PPO, HMO,” we rap, understanding the flow,
Choosing our coverage, as we grow.

“Pre-existing conditions?” No longer a scare,
Reforms have made healthcare more fair.
With HSAs and FSAs, saving the day,
For our medical bills, we’ve found a way.

Annual check-ups, preventive is key,
Vaccines, screenings, all copay-free.
Wellness visits, keeping us prime,
Insurance covers, every time.

MRI, X-rays, the imaging dance,
Approval needed, it’s not left to chance.
High-tech scans, in the diagnostic quest,
Ensuring our treatment, is nothing but the best.

In-network, out, the terms we know,
EPOs and POS plans, in the insurance show.
Caps on out-of-pocket, a protective measure,
In the rap of healthcare, it’s all our treasure.

Who knew SEO optimization would one day serve as a perfect excuse to publish this poem.

This post explores how leveraging Pydantic for data extraction from any source significantly streamlines the evaluation process, ensuring data scheme alignment between predictions and ground truth data.

Motivation

It’s well-known that the landscape of Language Learning Models (LLMs) is rapidly evolving, capable of executing many non-trivial tasks and doing the heavy lifting with just a prompt in English. While creating a silly song is fun, the importance of assessing an output’s accuracy becomes critical when it involves complex data extraction and processing.

The rationale for evaluating an LLM’s output is straightforward. LLMs are not yet in a hallucination-free stage of development. Data-driven processes relying on the output of your LLM pipeline are likely to produce unreliable outputs if you don’t methodically measure accuracy across a sizable dataset.

Evaluation Pipeline

As data scientists, our approach to an LLM problem wouldn’t differ much from how we assess any ML model’s output. With quality prompts, evaluating the model becomes almost the sole necessity (well, that plus a considerable amount of MLOps around building robust evaluation and inference pipelines). If you are well-funded or prefer not to deal with the hassle of open-source models, you might opt for an off-the-shelf LLM like OpenAI’s GPT-4 or its less proficient alternatives. In such cases, the model comes ready-made, and the main evaluation focuses on the hyperparameters, which, in an LLM context, essentially means the prompts you crafted. This scenario also offers a free evaluation of how well-engineered your pipeline is, since your inference pipeline will likely be similar, if not simpler, than the one running the evaluation.

Implementing an evaluation pipeline in the classical sense also involves developing metrics that measure your “model’s” success. But before delving into metrics, how can you tame the beast that is an LLM to output data in your desired format? And how can you ensure your ground truth data is as streamlined as possible into your evaluation pipeline? (Assuming you have ground truth data, right? Otherwise, you can stop reading this post right now as you’re entering the “finger in the wind” realm, not data science. The outcomes can be unpredictably erroneous for you and your LLM.)

Pydantic: The Cornerstone for Data Scheme Consistency

Suppose you’re dealing with heaps of PDFs that have undergone text and table extraction. Now, you wish to extract meaningful data from them in a structured manner. Here, Pydantic, a Python data validation and settings management library, plays a crucial role. Its ability to enforce type hints at runtime and provide user-friendly errors offers an incredible shortcut for both validating schema and streamlining OpenAI’s LLM evaluation pipelines.

Pydantic’s core feature — enforcing type hints at runtime — ensures data adherence to a predefined schema. This feature is invaluable for integrating with LLMs for several reasons:

• Function Input Validation: By using a Pydantic model as a function argument for an LLM, you ensure that the output data strictly follows the specified schema. This preemptive measure eliminates common sources of errors caused by schema mismatches, thereby enhancing your entire pipeline’s reliability.
• Scheme Alignment for Predictions and Ground Truth Data: Utilizing the same Pydantic model for extracting and validating ground truth data ensures that both your “true” and “predicted” data match. This alignment is crucial for accurate evaluation, facilitating like-for-like comparisons without additional data wrangling.

Data Consolidation

It’s essential to allow some flexibility in your schema to accommodate common scenarios where ground truth data might not be clean or in the expected format. For instance, if your focus is health insurance and your data strangely models patient coverage facts as strings, whereas the LLM retrieves them as booleans (as you rightly designed), you’d either have to include these string permutations in your prompts (a risky workaround) or allow the LLM to do its job, performing post-processing on your ground truth data before scoring. To accommodate both booleans and strings, your schema needs flexibility.

Also, consider that your inferred data might include elements of interest not present in the ground truth data. In such cases, remember to exclude these elements from the evaluation process. This means your process should retain those attributes, but your metrics should ignore them.

In-Scheme LLM Instructions

An important part of this entire mechanism is to utilize the way PyDantic is built to give implicit instructions (in a sense) to your LLM via functions. The descriptions and docstrings you provide for the scheme you are working with will tell the LLM when to utilize your scheme to return a structured JSON answer. For example, in our specific healthcare domain example where I extract benefit data from an SBC plan, the main scheme would look like this:

The main scheme uses two other schemes itself, which look like this:

As you can see, both the docstring and the description of each field tell the LLM how to associate the data it extracts (as instructed in the prompts) with the data structure it should output eventually.

Additionally, it is important to design your PyDantic scheme such that it would be flexible to validate both your true data and the data you get from the LLM.

Having inference in mind, you would sometimes like to include fields in your scheme that you would like to extract, but that are not in your true data. In that case, you will have no choice but to implement filtering to the “true” and “pred” data so you can compare the exact same structures.

In other words, your evaluation should be accurate which might make it “narrower” than the data you eventually decide to extract. Is it cheating? Will the evaluation score represent reality? That Depends on what you ask from your LLM (how much inference data is different than the evaluated data), how much the evaluation score looks promising, and how much human-in-the-loop you expect to include in your inference.

Metrics

Evaluating LLM outputs against ground truth data involves a nuanced comparison of predicted and actual values, taking into account the specific aspects of interest. Like any ML evaluation, choosing your metrics wisely is paramount.

One approach could be a “DictWise” comparison of the schema, checking the values of your true schema against your predictions. You might allow for some leniency by enabling strings in the schema to be measured with their sub-metric (such as fuzzy matching or some distance algorithms). Alternatively, developing a method to vectorize your schema — considering normalization and possibly using an embedding model for strings — could offer another solution. In this case, you should consider utilizing embeddings in order to vectorize the textual data.

Here’s a straightforward implementation of the DictWise metric:

def compare_dicts(
        dict1: Dict, dict2: Dict, compare_mode: str = "loose", schema=None, return_individual_scores=False
) -> tuple[float | int, dict[Any, float | int]] | float | int:
    """
    Compares two dictionaries based on a specified mode and schema, returning both the total score and individual key scores.
    Compare mode can be "loose" or "strict": strict mode requires all keys and values to match, while loose mode allows
    for some differences in values and missing keys and is calculating the score by Levenshtein distance.
    LIMITATION: this metric does not penalize for extra keys in the dictionary (false positives)
    """
    if schema:
        keys_to_compare = {field for field in schema.__fields__}
    else:
        keys_to_compare = set(dict1.keys()) | set(dict2.keys())

    total_keys = len(keys_to_compare)
    total_score = 0.0
    individual_scores = {}

    for key in keys_to_compare:
        in_dict1, in_dict2 = key in dict1, key in dict2
        if not in_dict1 or not in_dict2:
            individual_scores[key] = 0.0
            continue

        val1, val2 = dict1.get(key), dict2.get(key)
        if compare_mode == "loose":
            if isinstance(val1, str) and isinstance(val2, str):
                distance = Levenshtein.distance(val1, val2)
                max_len = max(len(val1), len(val2))
                score = 1 - (distance / max_len) if max_len > 0 else 1
            else:
                score = float(val1 == val2)
        elif compare_mode == "strict":
            score = float(val1 == val2)

        total_score += score
        individual_scores[key] = score

    normalized_total_score = total_score / total_keys if total_keys > 0 else 0
    if return_individual_scores:
        return normalized_total_score, individual_scores
    return normalized_total_score

Scoring

I will not get much into scoring in this post, but your approach to this will heavily depend on your business logic and use case. For example, you might assign different weights to various items depends on their significance or to formulate some non-linear calculation method. Combining several metrics to produce a comprehensive score is also an option. You can also just leave it vanila.

Practical Implementation

To effectively implement this approach, follow these steps:

1. Define Your Pydantic Model: Clearly outline the data schema of interest, covering all relevant fields and types. This model serves as your blueprint for data extraction and LLM input/output processing.
2. Data Extraction: Utilize the Pydantic model for extracting ground truth data from your sources, ensuring data validation and alignment with your schema.
3. LLM Integration: Maintain schema consistency by passing inputs through the Pydantic model when interfacing with the LLM. This ensures direct comparability between the LLM’s outputs and your ground truth data.
4. Evaluation Metrics: Design your evaluation metrics around the schema defined by your Pydantic model, ensuring relevance and alignment with the data’s characteristics.

Takeaways

Integrating Pydantic into LLM evaluation pipelines provides a robust framework for ensuring data consistency and reliability. By employing Pydantic models for data extraction and processing, researchers and developers can ground their evaluations in a solid, error-resistant foundation. This methodology not only streamlines the evaluation process but also improves the accuracy and relevance of outcomes, advancing the development of more effective and dependable LLMs.

More about metrics in depth and a detailed explanation of an evaluation pipeline in following posts!

Enhance LLM Evaluation Pipelines using PyDantic was originally published in Healthee on Medium, where people are continuing the conversation by highlighting and responding to this story.

Let your search algorithm rise through the ranks 🪄

There are countless metrics that can be employed to gauge the precision of a search engine. Some notable examples include the commonly used precision, recall, and F1 scores, along with more situation-specific metrics such as MRR, nDCG, and Kendall-Tau. In reality, search engines are evaluated using a mix of these metrics, tailored and adjusted according to the specific use case.

But what if our data in reality is structured differently from the typical output of a classic search engine? What if we follow a case-specific methodology to designate the order of results as significant or relevant that is different from the usual and simple list of results?

This is precisely the scenario for us at Healthee, where the search service results had to adhere to a “bucketed” structure, rather than just a list. The exact position of a search result item isn’t as critical as its association within a group of results that appear together and their location as a group. Essentially, the importance lies in a search result having a certain rank or being within the same “league” of results.

Happy users lead to thriving companies, and these users are usually those who receive relevant search results when looking up stuff on your website or app. But who determines the relevance of these results? Your search algorithm might be complex and it could involve one or more of many methods (such as a closed lookup table with fuzzy logic or lists with synonyms for simple sets as well as similarity searches, knowledge graphs, and other machine learning models for more advanced cases). Regardless of the method used, an independent evaluation method for your search algorithm is essential. This is typically achieved by conducting regular tests using a regularly maintained gold set. You would probably want to trigger this test following a significant data update or at regular time intervals if your data expands steadily over time.

To illustrate, let’s consider a real-world example as it arises in the Healthee App, which aids users in navigating the healthcare system and obtaining information about their coverage for a variety of medical services.

Imagine a user who wants to find out if they’re covered for an MRI, a service consumed by millions daily. The user enters a search query (simply “MRI” in this instance) and receives a list of results. At this point, our search algorithm acts as a lone sailor at sea, solely reliant on previously acquired skills. However, during the optimization of our search model (whether through machine learning or manual parameter adjustment), we aim to ensure search efficacy.

In our specific healthcare context, it is doctors who are entrusted with the responsibility of maintaining an accurate gold set that is used to evaluate the search algorithm. Naturally, there are other methods to evaluate searches such as user signals, but this approach is based on the knowledge of a domain expert angle rather than in-app events.

Back to our results. We searched for “MRI” and got the following:

Looks good, doesn’t it? All results are with ‘mri’, so.. probably?

Well, the thing is we don’t really know. Sailor alone at sea, remember? But now let’s help our brave seaman and grant him a precious life vest: the ability to know the real order of the results.

The testing phase plays out precisely like this: our gold set acts as a handy guide, allowing us to understand the correct sequence of results. So, how did we do? The outcome was not exceptional, but not disastrous either; we completely overlooked our initial search result, yet the second to fourth results were correctly ordered and relatively high. The fifth and sixth ones were roughly where they were anticipated to be.

The good news is that there are already great metrics to measure these results’ accuracy, as mentioned above. We can set a score according to those metrics and evaluate how good our search algorithm is.

Yet, here lies the challenge: what if we aren’t overly concerned about the precise placement of each search result? After all, perhaps “advanced imaging — mri — outpatient hospital” and “advanced imaging — mri — freestanding radiology facility” carry equal importance, and a swap in their positions wouldn’t be catastrophic. This dilemma is precisely what our doctors encountered when creating the gold set. If they insist on accuracy linked directly to the index, they might impose an overly rigid rule, which could potentially undermine the search algorithm rather than enhance it.

So, instead of constructing an index-based gold set, we opted for a ranking-based one.

Let’s take a glance at the actual gold set we’ve assembled for MRI test results:

What we see in our ranking-based gold set is the following:

1. The 1st ranking bucket contains only one result.
2. The 2nd bucket contains three different results.
3. The 3rd bucket contains two results.

As we have stipulated earlier, the function of the buckets is to eliminate the difference between cases where items from the same rank swap places in the search result. If such a swap occurred, we would consider it acceptable and the accuracy of the test would be the same as in the case the exact gold set was returned.

Let’s explore an example. Assume that the so-called “perfect set” yields a certain score. The following set would produce the exact same score:

And even the following one would:

But this returned result will come up with a lower score:

And the next one with even a lower one:

Just to be clear, the search engine itself does not return any rankings. Instead, it simply generates a list of search results based on its own logic. The numbers on the right are merely for our own comprehension of the search engine’s performance. Naturally, these rankings are known to the testing mechanism, and shortly we will dive into how that mechanism operates and how it utilizes the rankings predetermined by our doctors.

We mentioned the word score before several times. How does this score come to be? What principles lie at the base of such scoring? Answering this question is key to designing a test that will work well with the kind of data we have in our hands (namely, lists of ranked items).

Building on NDCG

We have previously touched upon existing algorithms that measure test results mostly based on the index value of the returned result item. One of them, nDCG (normalized discounted cumulative gain), was initially contemplated for our specific scenario but encountered an obstacle when addressing the ranking concern — specifically when our domain experts started bucketing the source of truth data. Before we could implement it, some modifications would be necessary.

How can we measure the accuracy of the returned result if the gold set is in buckets? We would have to make an important assumption about the distribution of the results: to assume that their rank structure mirrors that of the gold set. Yes, even though there is nothing inherently “ranky” about our search engine.

So if our gold set is

we would extrapolate from it the “assumed ranks” and will apply them to the set of results that the search engine provides, which are devoid of any ranking. Therefore, if our test results look like the following, they would be assigned a set of assumed ranks:

Now that we have the ability to compare apples to apples, we can use a measuring algorithm and create our metric. For that, we can build on nDCG and re-engineer it to our purpose so ranks are taken into account.

The function for calculating the gain is

Let’s break this down:

In the numerator, we are raising 2 to the power of a number. That number is the lesser value between the true rank and the assumed rank. In the denominator, we’re taking the base-2 logarithm of the greater value between the two. This is done to prevent attributing more significance to the swapping of a higher rank from the gold set with a lower rank from the assumed ranks, or vice versa. In other words, it wouldn’t make a difference to us whether we “swapped up” or “swapped down” — the gain will be the same either way. We add 2 to avert a denominator that’s equal to zero.

We also need to mention that our doctors are very passionate about the Olympics games. When they set the rankings they did it according to the medals system: gold medal is for first place, silver for the second, etc. However our algorithm actually interprets higher numbers as higher ranks, therefore we need to perform an inversion prior to any calculation:

Also, note that rankings are 1-indexed and not 0-indexed.

Next, what happens during calculation (once we have inferred the assumed ranks) is an iteration through all the search results — which are also in the true relevant set — and calculating the gain each result is contributing to the final score

where RDCG stands for Ranked Discounted Cumulative Gain, accumulating gain from each ith out of N results.

(Eagle-eyed readers may spot a potential hiccup, as we’re only making comparisons between known entities — i.e., items that are in the gold set. We’ll address this shortly.)

“Can I get a discount, please?”

By this point, we’ve developed a clear understanding of the “Ranked” part of the algorithm quite well, as well as the “Cumulative” and “Gain” parts. Let’s focus on “Discounted” for a moment. This component of the algorithm means that the further down we move in our ranking list, the lesser we want the gain to contribute to the accuracy score. This is because a rank at the 15th position is of much less concern to us compared to the 1st or 2nd rank. This principle manifests in two ways: firstly, higher ranks contribute more to RDCG; this is the numerator of the formula. Secondly, a discrepancy between higher ranks is more critical than a discrepancy between lower ranks, because we care more about results that are returned at the top of the list; this is the denominator — specifically the log.

Let’s explore this concept with some numerical examples (keeping in mind that we’ve had to invert the ranks):

When true rank is 3 and predicted rank is 4:

And when true rank is 3 and predicted rank is 5:

So we see that the bigger the gap, the lower the gain.

Additionally, when true rank is 4 and predicted rank is 5:

That is, when the gap is 1 rank just like in the first example, but at higher ranks altogether, then the gain is higher too.

A crucial step we need to take to compare different queries and their corresponding search results is normalization. Since the gain is simply a result of score addition and is not limited by an upper bound, if we have a big set it is most likely that it will yield a bigger score compared to a smaller set. How do we perform this normalization (i.e., bind the score to a range between 0 and 1)? To do that, we will first calculate the score of the perfect set and normalize it accordingly. Thus:

Where NRDCG is Normalized RDCG. Now we have a score that would be interpretable across different queries as well as easier to interpret across time.

“Hope of ill gain is the beginning of loss.” — Democritus

Let’s consider the following scenario. We have two sets of search results returned from our engine and wish to measure the accuracy. One set is the perfect set where all results are in the best possible order (or more accurately — one of them), and we know it will give us the maximal RDCG. The other set is far from perfect as it returns many false positives interweaved into the perfect set, keeping “good results” in order. What do you think will happen in terms of scoring?

The “perfect” set:

Set with “infiltrators”:

One thing that immediately sticks out is that our results set has grown. How does this affect the scoring? (The ranks above are actually not in the structure the algorithm will assume about the test results, we just want to see the “infiltrator” items). The assumed ranks will still get the rank structure of the true set as it “transforms” it (in Sklearn terms).

So our assumed ranks will look like this:

Congratulations — we have a new rank! Our assumed ranks are indeed reflected in the resulting rank distribution, but they represent only part of it. Because the list grew, some of the items are now out of bounds of the gold set length. Those out-of-bounds items are assigned a new rank (the lowest rank possible) so that when the algorithm iterates over the list it will get the lowest gain when calculated for those items.

But this is not a loss, exactly, is it? This simply leads to a lower gain. This is why the implementation of the algorithm also allows to control for and penalize two different kinds of situations: one is when “infiltrators” are gettings into the ranks (pun definitely intended) as above, namely, having false positives; and another is when items from the gold set don’t appear at all in the search results, namely, false negatives. We are also able to control whether either of the false positives or false negatives is penalized in proportion to the rank gap or getting a fixed maximal penalty. Penalizing results are not part of nDCG, so this control elevates our algorithm to NRDCGL — Normalized Ranked Discounted Cumulative Gain and Loss.

Allowing for loss is introducing a new phenomenon to our scoring — it can become negative. While a negative score in itself is not a problem (especially if we measure individual queries in a “set of sets” across time), it can make a complete test of queries less interpretable if you want to average the entire test scores to get a total score between 0 and 1. In order to facilitate a meta-aggregation of a complete search result set accuracy scores (i.e, of many queries and many search results that are associated with those) there is also an implementation of the algorithm that does not allow for negative scores:

One more thing to clarify about loss here is that it is not the same as in the sense of loss function. There isn’t any optimization or an objective function to measure deviation from, but simply a dynamic rule-based approach to penalties.

The disadvantage of NRDCGL is that if there are too many types of differences in the search results, then we might not be able to tell what factor contributed to the increase or decrease of the score. It is kind of like having too many cooks — you will not find the culprit that ruined the dish. As mentioned above, it is always a good idea to combine several metrics when assessing non-deterministic models, especially such that rely on frequently changing data.

In a future article, we will go beyond the algorithmic layer and see how to drive a complete test with an annotated dataset and create a quantitative report on Monday.com (or an analytics platform of your choice). The global testing process will reflect the advantages of RaDiCuGLoss by showing how changes made to the data or the algorithm (that manifest themselves in a limited change in the output of the search algorithm), can help us monitor performance.

RaDiCuGaLoss is publicly available here. You’re most welcome to contribute! ❤️

Kudos to Amit BenDor and EladL for their great help in developing this algorithm 🙌

RaDiCuGLoss! was originally published in Healthee on Medium, where people are continuing the conversation by highlighting and responding to this story.