Read in the browser, for best viewing for best viewing experience.

Why I Started Using AI to Read Research Papers

I’ll be honest: I used to dread reading research papers. The dense math, the notation-heavy proofs, the assumption that you already know three other papers to understand this one. As an ML engineer, I knew these papers held insights I could apply at work, but actually extracting that value felt like a slog.

My learning style doesn’t mesh well with traditional academic writing. I learn best through small examples and code I can run and tinker with. Show me a 3x3 matrix example before the general n×n case. Let me see the algorithm work on toy data before throwing theorems at me.

For years, this created a barrier. Research papers felt like they were written for PhDs, not practitioners. You either had the advanced math background to parse them, or you waited months for someone to write a blog post explaining it in plain English.

That’s changed. With AI assistants (both local and cloud-based LLMs), I can now have a conversation with a paper. I can ask “what does this symbol mean?” or “show me this concept with a simple Python example” and get an answer calibrated to my current understanding. The paper adapts to me, not the other way around.

This post walks through my workflow for using AI to read technical papers, so you can apply the same approach to level up your own learning.

By the end of this post, you’ll be able to:

• Set up a local LLM (using Ollama) or connect to cloud APIs for paper reading
• Convert PDFs to markdown that LLMs can actually process
• Craft system prompts that turn an LLM into a patient tutor, not a lecture-bot
• Build an interactive Q&A workflow where you control the pace and depth
• Apply this approach to papers, documentation, or any dense technical content

Prerequisites & Setup

Before we dive in, let’s get our environment ready. You’ll need two things: a local LLM running on your machine, and a few Python libraries to glue everything together.

Installing Ollama

Ollama makes running LLMs locally dead simple. Head to ollama.com and download the installer for your OS. On Mac, it’s a standard .dmg install. On Linux, a one-liner:

curl -fsSL https://ollama.com/install.sh | sh

Once installed, verify it’s working:

ollama --version

Choosing and Pulling a Model

Browse the Ollama model library to see what’s available. For reading papers with diagrams and equations, I recommend qwen3-vl:8b, a vision-language model that can handle both text and images. You can use other models based on your machine’s specifications.

Pull it with:

ollama pull qwen3-vl:8b

This downloads the model weights (around 5GB). Grab a coffee.

To start serving:

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Keep this terminal running. Ollama now exposes an OpenAI-compatible API at localhost:11434.

Note: Default context length in Ollama is 4096 tokens which is too small for our usecase, so we bump it upto 64K.

Python Libraries

We need three packages:

- pymupdf4llm: Converts PDF pages to clean markdown that LLMs can parse

- lisette: Lightweight wrapper on litellm that simplifies multi-turn conversations and provide a Unified API to talk to 100+ LLM providers (local or cloud) with one interface

Install them:

pip install pymupdf4llm lisette

With Ollama running and these libraries installed, you’re ready to start chatting with papers.

Core Workflow: PDF to Conversation

The workflow has two parts: getting the paper into a format LLMs can work with, then setting up the conversation so the AI becomes a useful tutor rather than a generic chatbot.

Converting PDF to Markdown

First step is to convert a pdf into text. pymupdf4llm handles this by extracting text while preserving structure. Here’s an example, reading this paper:

import pymupdf4llm
# Convert PDF to txt
paper_txt = pymupdf4llm.to_markdown("./paper/1_2018_DPP_YT.pdf")
# Save the paper as Markdown
with open("./paper/1_2018_DPP_YT.md", "w") as f:
    f.write(paper_txt)

Let’s see the first paragraph.

print(paper_txt[:500])

That’s it. You now have clean markdown with headers, code blocks, and reasonable table formatting. For papers with complex layouts, the results aren’t perfect, but they’re good enough for an LLM to understand the content. If you want a more accurate PDF to markdown converter, you can also look at Datalab Marker.

Designing Your System Prompt

This is where the magic happens. A vanilla “you are a helpful assistant” prompt gives you Wikipedia-style explanations. A well-crafted system prompt gives you a patient tutor who meets you at your level.

Here’s the prompt which works well for paper-reading sessions:

system_prompt = f"""
You are helping a someone understand an academic paper.
Here is the paper \n
<paper>
{paper_txt}
</paper>

CRITICAL RULES:
1. NEVER explain everything at once. Take ONE small step, then STOP and wait.
2. ALWAYS start by asking what the learner already knows about the topic.
3. After each explanation, ask a question to check understanding OR ask what they want to explore next.
4. Keep responses SHORT (2-4 paragraphs max). End with a question.
5. Use concrete examples and analogies before math. 
6. Build foundations with code - Teach unfamiliar mathematical concepts through small numpy experiments rather than pure theory. Let the learner run code and observe patterns.
7. If they ask "explain X", first ask what parts of X they already understand.
8. Use string format like this for formula display `L_ij = q_i × q_j × exp(-α × D_ij^γ)`.

TEACHING FLOW:
- Assess background → Build intuition with examples → Connect to math → Let learner guide direction

BAD (don't do this):
"Here's everything about DPPs: [wall of text with all equations]"
"""

The Teaching Prompt Deep-Dive

The prompt design isn’t arbitrary. Each rule targets a specific failure mode I’ve seen when using LLMs for learning:

1. The paper lives in the system prompt. This keeps the full context available for every question you ask. You don’t need to re-paste sections or remind the AI what paper you’re discussing.
2. “Intuition before math” matches how practitioners learn. Most of us grasp concepts faster through examples and analogies than through theorem statements. If your learning style differs, swap in your own preferences here.
3. “Tiny example first” counteracts the LLM’s default behavior. Without this, models tend toward abstract, general explanations. Forcing concrete examples (a 3x3 matrix, a toy dataset) makes concepts stick.
4. “Check if I want to go deeper” prevents walls of text. LLMs love to explain everything at once. This rule puts you in control of pacing. You decide when to zoom in on details vs. move to the next topic.
5. The “BAD” example shows what to avoid. Negative examples help the model understand boundaries. Without it, you’ll get comprehensive but overwhelming responses.

Putting It Together: With these two pieces, you’re ready to start a conversation. The next section shows how to wire this up with lisette for a smooth multi-turn chat experience.

Wiring Up the Conversation with Lisette

lisette handles the boilerplate of multi-turn conversations: tracking message history, formatting requests for different LLM providers, and managing context windows. Here’s the minimal setup:

from lisette import Chat, contents
from IPython.display import HTML, Markdown, display
chat = Chat(
    model="ollama/qwen3-vl:8b",
    sp="" # Insert prompt here
)

That’s it. The model string follows litellm’s provider/model convention. For Ollama, prefix with ollama/. For cloud providers, you’d use anthropic/claude-sonnet-4-20250514 or openai/gpt-4o.

To send a message and get a response:

chat("Hi! I am Aayush")

As we can see above, it’s as easy to start a conversation with a ollama running in your local machine. Chat class is designed to be persistent which means it remembers the previous conversations.

chat("What's my name?")

As you can see model remembers my name from the previous conversation. Each call to chat() appends your message and the AI’s response to the conversation history. The next time you call it, the model sees the full exchange so far. Let’s see current history:

chat.hist

Live Example: Reading a DPP Paper

Here’s a real exchange from when I was working through a paper on Determinantal Point Processes. For ease of reading in the blog post, lets create a class which can render the AI output in a terminal style for easy reading.

def ask(chat, question, **kwargs):
    r = chat(question, **kwargs)
    text = contents(r).content
    display(HTML(f"""
    <div style="background-color: #1e1e1e; color: #00ff00; padding: 12px; 
                border-radius: 5px; font-family: monospace; white-space: pre-wrap;">
<span style="color: #00aaff;">You:</span> {question}
<span style="color: #ffcc00;">Tutor:</span> {text}
    </div>
    """))
    
chat = Chat(
    model="ollama/qwen3-vl:8b",
    sp=system_prompt
)

Note: This time we are actually passing the prompt we created in previous section with paper text and our instructions.

q = """
I want to learn about this paper. Help me.
"""
ask(chat, q, think=True)

q = """
I know about recommendation systems and work on a large recommender system which serves video content.
I am familiar with recommendation algorithms. I am interested in understanding how DPP works.
"""
ask(chat, q, think=True)

q = """
Yes I don't understand what determinant are, can you help me understand the basic intuition and use code examples?
I am fairly familiar with numpy.
"""
ask(chat, q)

Note: The rendering from Chat class is markdown style, which is more readable. The above examples are wrapped for easy separation b/w blog text and code output.

Managing Chat History: Checkpoints and Context Control

One advantage of this approach over general chat interfaces is that you have full access to the conversation history. This matters when sessions get long or when the AI goes down a wrong path and you want to rewind.

Accessing the History

The Chat object stores messages in a list you can inspect and modify:

# See the current conversation
print(chat.hist)  # Returns list of message dicts
# Check how many turns you've had
print(len(chat.hist))

Creating Checkpoints

Before exploring a tangent, save your current state:

# Save checkpoint before a risky question
checkpoint = len(chat.hist)
# Ask something that might derail the conversation
ask(chat, "Actually, can you explain all the math in one go?")

# That response was overwhelming. Rewind.
chat.hist = chat.hist[:checkpoint]
print(chat.hist[len(chat.hist)-1])

Why This Matters

Two problems this solves:

1. Context bloat. Long conversations eat up your context window. If you spent 10 turns on a section you now understand, trim those exchanges and reclaim tokens for the next topic.
2. Context poisoning. If the AI misunderstands something and you correct it, the wrong explanation still sits in history, potentially confusing later responses. Removing it gives you a cleaner slate.

Going Further: Paid LLM Options

Local models are great for privacy and cost, but sometimes you need more firepower. Longer papers, denser math, or multi-step reasoning tasks can benefit from larger models like Claude or GPT-4o.

The good news: lisette uses litellm under the hood, so switching providers is a one-line change.

For cloud providers, you’ll need API keys. Set them as environment variables or use a .env file:

import os
from dotenv import load_dotenv
load_dotenv()

# Claude (Anthropic)
chat = Chat(model="anthropic/claude-haiku-4-5", sp=system_prompt)
ask(chat, "Help me understand the paper")

As you can see the workflow stays identical. Your system prompt, conversation history, and teaching flow all transfer across providers.

Key Takeaways

Reading technical papers used to be a solitary grind. You either had the background to parse the notation, or you didn’t. AI changes this equation.

Here’s what I want you to walk away with:

1. The barrier to technical content has dropped. The same workflow that worked for this paper works for textbooks, API documentation, dense blog posts, or any content where you need concepts explained at your level. The AI adapts to you.
2. You control the pace and depth. Unlike courses or tutorials that assume a fixed audience, this approach lets you skip what you know and linger on what confuses you. Your learning path, not someone else’s curriculum.
3. Local LLMs make this nearly free. Ollama plus a capable open model means you can read papers all day without API costs. You can try reading books, YT transcripts, brainstorming using same method. One tool, many uses.
4. The system prompt is the lever. Generic prompts give generic results. The tutor prompt we built forces the AI into a teaching mode that actually helps. Steal it, tweak it, make it yours.
5. Start with a paper you actually care about. Don’t practice on something boring. Pick a paper relevant to your work or a concept you’ve always wanted to understand. Motivation carries you through the setup friction.

The tools exist. The cost is minimal. The only question is which paper you’ll read first.

I hope you found this blog useful. If you have questions or want to share your own paper reading patterns, find me on LinkedIn.

Using AI to Read Technical Research Papers was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

This post captures what I learned from the Udemy course “Claude Code Crash Course: Claude Code in a Day”.

Read in browser for best viewing experience.

If you’ve been following AI tooling trends, you’ve noticed something interesting: AI companies are going direct-to-developer. Cursor showed us what’s possible when you embed AI deeply into a code editor. But now Claude (Anthropic) and other foundation model providers are shipping their own coding agents. This isn’t just competition. It’s a signal that the “AI wrapper” layer is collapsing, and the companies building the models want to own the developer experience.

As an ML Engineer, I spend my days building systems that learn from data. But I’m also a practitioner who uses AI tools daily. When Claude Code launched, I wanted to understand it deeply, not just use it casually. What makes it different from Cursor or Copilot? What new patterns does it enable? And most importantly: how do I get the most out of it for real engineering work?

I’ll walk through the core concepts (context engineering, memory hierarchies, MCP), the practical commands you’ll use daily, and the advanced patterns like subagents and parallel workflows. Whether you’re evaluating Claude Code for your team or just curious about where AI-assisted development is heading, this should give you a clear picture.

If you are a video learner, I created a video using Notebook LM, which summarizes key concepts from the blog, watch here — Learning Claude Code

Core Concept: Context Engineering

If you’ve done any prompt engineering, context engineering is the next step. The idea is simple: prompts are static, but context is dynamic. Your agent needs information from many sources (your input, previous interactions, tool outputs, external data) and assembling the right context at the right time is what separates agents that work from agents that fail.

The course frames context engineering failures around three failure modes:

• Context poisoning: A hallucination enters the context and corrupts every response that follows. Once bad information gets in, it spreads.
• Context confusion: Irrelevant context pulls the model’s attention in the wrong direction. You asked about database design but your frontend docs are drowning out the signal.
• Context clash: Contradictory information in the same context. The model gets pulled between conflicting instructions or facts.

The takeaway here is “garbage in, garbage out,” but applied specifically to agents. LLMs can’t read your mind. They need the right information AND the right tools at the right time.

Claude Code implements four strategies to manage this:

1. Writing context: A three-tier memory hierarchy (user, project, dynamic imports) that persists information across sessions
2. Intelligent retrieval: Automatically searching folders for relevant CLAUDE.md files and prioritizing recently-used context
3. Context compression: /clear wipes everything, /compact summarizes to essentials
4. Context isolation: Subagents run in their own context windows, preventing main conversation bloat

The course also covers the “Goldilocks zone” for system prompts. Too specific and you’re treating the LLM like a state machine with hard-coded if/else logic. Too vague and you’re saying “do the right thing” without defining what “right” means. Good prompts establish clear identity, empower rather than constrain, and teach principles instead of enumerated rules.

Getting Started: The Basics

Before you can do anything interesting with Claude Code, you need to understand how it stores and retrieves information. Most tutorials jump straight to commands without explaining the underlying system. Let’s start there.

Installation

Getting Claude Code running is straightforward:

npm install -g @anthropic-ai/claude-code
cd your-project
claude

This drops you into an interactive terminal session (no GUI, no VS Code extension needed). But Claude Code doesn’t know anything about your project yet. It’s a blank slate every time you start.

The /init Command and CLAUDE.md

Run /init in your project directory. Claude will analyze your codebase and generate a CLAUDE.md file at the root. This file becomes Claude’s memory of your project: coding conventions, architecture decisions, file structure, anything it needs to work effectively.

Here’s what a real CLAUDE.md might look like:

# Project: recommendation-service
## Tech Stack
- Python 3.11, FastAPI
- PostgreSQL with SQLAlchemy
- Redis for caching

## Conventions
- All API endpoints follow REST naming
- Tests go in tests/ mirroring src/ structure
- Use Pydantic for all request/response models

## Important Context
- The ranking model lives in src/ranker/ and uses PyTorch
- Never modify files in src/legacy/ without explicit approval

You can and should edit this file manually. The auto-generated version is a starting point. Here’s an example: without explicit guidance, Claude kept suggesting print() statements for debugging in the codebase. You can add “Use structured logging using python logging module , never print()” to CLAUDE.md, the problem disappears. Add the patterns your team follows, the things Claude keeps getting wrong, and the directories it should ignore.

The Three-Tier Memory Hierarchy

Claude Code reads context from three levels, with more specific context overriding more general:

┌─────────────────────────────────────────────┐
│  ~/.claude/CLAUDE.md (User-level)           │
│  Your personal preferences across projects  │
│  e.g., "Always use type hints"              │
└─────────────────┬───────────────────────────┘
                  │ overridden by
                  ▼
┌─────────────────────────────────────────────┐
│  ./CLAUDE.md (Project-level)                │
│  Tech stack, conventions, architecture      │
│  Created by /init                           │
└─────────────────┬───────────────────────────┘
                  │ overridden by
                  ▼
┌─────────────────────────────────────────────┐
│  ./src/auth/CLAUDE.md (Dynamic imports)     │
│  Subdirectory-specific patterns             │
│  Loaded when working in that directory      │
└─────────────────────────────────────────────┘

1. User-level (~/.claude/CLAUDE.md): Personal preferences that apply everywhere. Maybe you always want type hints, or you prefer pytest over unittest. This follows you across all projects.
2. Project-level (./CLAUDE.md): Project-specific context. Tech stack, conventions, architecture. This is what /init creates.
3. Dynamic imports: Other CLAUDE.md files scattered throughout your codebase. Put one in your src/auth/ folder to explain your authentication patterns. Claude will find and load it when working in that directory.

Tip: Start with /init, then spend 10 minutes editing the generated CLAUDE.md. Add your team’s PR conventions, your testing requirements, any gotchas that would take a new engineer a week to discover. That 10 minutes will save you hours of correcting Claude later.

Essential Commands

Claude Code is a terminal application, and like any good CLI tool, it’s driven by commands. But these aren’t just shortcuts. Each command maps to a context engineering strategy we discussed earlier. Understanding when to use which command is half the battle.

Context Management

These three commands control what Claude knows about your conversation:

/clear    # Wipes everything, fresh start
/compact  # Summarizes conversation to essentials, keeps key info
/context  # Shows what's consuming your context window

Use /context liberally. It breaks down token usage: system prompt, MCP tools, messages, memory files. When your responses start degrading, run this first to see what’s eating your budget. I’ve found MCP tools are often the silent killer here.

The difference between /clear and /compact matters. If Claude is going in circles or referencing something incorrect from earlier, /clear is your friend. If you just need more room but want Claude to remember the gist of what you’ve discussed, /compact compresses without amnesia.

Configuration

/config   # Opens settings panel
/model    # Switch between Opus and Sonnet
/cost     # Monitor token usage (API key mode only)

Model selection is situational. When I asked Sonnet to refactor a ranking module, it jumped straight to code. Opus first asked clarifying questions about performance constraints and suggested three architectural approaches before writing anything. Opus thinks harder but costs more.

Model Selection Tip: Use Opus for planning and research, Sonnet for implementation. For most coding tasks, Sonnet hits the sweet spot between quality and cost.

Workflow

/init     # Analyze codebase, generate CLAUDE.md
/memory   # Open memory file in editor
/rewind   # Checkpoint system. Undo changes by prompt

The /rewind command deserves special mention. Claude Code tracks file changes made during each prompt. Press Escape twice or run /rewind to see your conversation history with indicators of which prompts touched which files. You get three restore options: code only, conversation only, or both.This is your safety net for ambitious changes that don’t pan out.

Modes

Shift+Tab toggles Plan Mode. In this read-only mode, Claude can research, analyze, and create specs but cannot edit files. Use this when you want to think through a problem before committing to implementation. Shift+Tab+Tab enables deeper thinking for complex planning tasks.

Other Useful Commands

/agents   # Manage subagents
/mcp      # View and manage MCP servers  
/plugins  # Browse and install plugins
/ide      # Integrate with your IDE (Cursor, VS Code)
/git      # GitHub integration setup

TipContext Budget Tip

Get in the habit of running /context before starting a complex task. If you’re already at 50% usage, you might want to /compact first or spin up a fresh session. Running out of context mid-refactor means Claude forgets the files it already changed.

Extending Claude Code

Out of the box, Claude is smart but generic. The extension system lets you teach it your patterns, connect it to your tools, and automate repetitive tasks. There are five extension mechanisms: custom commands, MCP, hooks, skills, and plugins.

Slash Commands

Slash commands are custom actions you define and trigger manually with /commandname. Unlike skills (which activate automatically) or hooks (which fire on events), commands run when you explicitly call them.

A command is a markdown file in .claude/commands/:

.claude/commands/
  review.md
  test-plan.md
  pr-summary.md

Here’s a simple example, .claude/commands/review.md:

---
description: Review code for common issues
---
Review the current file for:
1. Security vulnerabilities (SQL injection, XSS, auth bypasses)
2. Performance issues (N+1 queries, unnecessary loops)
3. Code style violations per our CLAUDE.md conventions

Format findings as a checklist with severity (high/medium/low).

Now /review is available in your session. The description appears in autocomplete when you type /.

Arguments and Interpolation

Commands can accept arguments using $ARGUMENTS:

---
description: Generate unit tests for a function
---
Write pytest tests for the function: $ARGUMENTS

Include:
- Happy path cases
- Edge cases (empty input, None, boundary values)  
- Error cases that should raise exceptions
Follow our test conventions in CLAUDE.md.

Call it with /test-plan calculate_ranking_score and $ARGUMENTS becomes calculate_ranking_score.

Project vs User Commands

Like CLAUDE.md, commands exist at two levels:

• Project commands (.claude/commands/) - Shared with your team via git. PR templates, project-specific workflows.
• User commands (~/.claude/commands/) - Personal shortcuts that follow you everywhere. Your preferred code review checklist, your debugging workflow.

You can find a whole list of practical commands in this GitHub repo — wshobson/commands

Tip: Commands are prompts you’ve optimized through repetition. When you find yourself typing the same multi-line instruction for the third time, turn it into a command. Five minutes of setup saves hours over a project’s lifetime.

MCP (Model Context Protocol)

MCP is how Claude Code talks to external systems: GitHub, JIRA, Slack, databases, your internal tools.

Without MCP, every AI tool needs a custom integration for every external service. That’s N × M integrations (N tools × M services). MCP flips this to N + M. Implement MCP once and use it everywhere.

# Add an MCP server for Supabase
claude mcp add --transport stdio supabase \
  --env SUPABASE_ACCESS_TOKEN=YOUR_TOKEN \
  -- npx -y @supabase/mcp-server-supabase@latest

# Add GitHub integration
claude mcp add github gh copilot mcp

Run /mcp to see your active servers. Read more in Claude code official docs - Link

MCP Context Implications: MCP servers consume context window space even when you’re not using them. I connected a handful of servers while experimenting and noticed my responses degrading on longer tasks. Running /context showed I’d burned 30% of my budget before writing a single prompt. Enable only what you need for the current project.

Hooks

Hooks let you run custom code at specific points in Claude’s workflow. Think of them as event listeners: something happens inside Claude Code, your script executes.

There are many trigger points where we can invoke hooks, some notable ones are:

• PreToolUse — Fires before Claude uses a tool (file write, bash command, etc.). Return non-zero to block the action.
• PostToolUse — Fires after a tool completes. Useful for validation or follow-up actions.
• Notification — Triggers when Claude sends a notification.
• Stop — Fires when Claude finishes responding.

Hooks live in your settings file (~/.claude/settings.json for user-level, .claude/settings.json for project-level):

{
  "hooks": {
    "PostToolUse": [{
      "type": "command",
      "command": "afplay /System/Library/Sounds/Glass.aiff",
      "matcher": "Write|Edit"
    }]
  }
}

This plays a sound whenever Claude edits a file. Seems trivial, but it’s genuinely useful. I can context-switch to Slack or documentation while Claude works through a refactor, then hear when it’s done writing. No more staring at a terminal waiting.

Other patterns examples:

• Auto-run pytest after test file modifications
• Run black for code formatting after any file edit
• Log all file changes to a separate audit file
• Block writes to production config files

Read more in claude code offical docs — Link

Skills

Skills are auto-invoked context providers. Unlike slash commands that you trigger manually, skills activate automatically when their description matches what you’re working on.

A skill is just a folder with a SKILL.md file:

.claude/skills/
  explain-code/
    SKILL.md      # Description + instructions

The SKILL.md includes a description field that Claude uses to decide when to load it. If you have a skill with description “Explains code with visual diagrams and analogies” and you ask Claude to explaing some code, it automatically pulls in that skill’s instructions. Here is an example SKILL.md file -

---
name: explain-code
description: Explains code with visual diagrams and analogies. Use when explaining how code works, teaching about a codebase, or when the user asks "how does this work?"
---

When explaining code, always include:
1. **Start with an analogy**: Compare the code to something from everyday life
2. **Draw a diagram**: Use ASCII art to show the flow, structure, or relationships
3. **Walk through the code**: Explain step-by-step what happens
4. **Highlight a gotcha**: What's a common mistake or misconception?
Keep explanations conversational. For complex concepts, use multiple analogies.

This differs from MCP in an important way. MCP tools are always present in the context, costing tokens whether you use them or not. Skills load on demand based on relevance. Simon Willison called skills “maybe a bigger deal than MCP” for this reason.

Use skills for domain knowledge that applies situationally: your custom UI library patterns, how your team structures GraphQL, testing conventions for specific modules.

Read more in claude code offical docs — Link

Plugins

Plugins bundle everything together. A plugin is a shareable package containing any combination of slash commands, subagents, skills, hooks, and MCP configurations.

The problem is distribution. You’ve spent hours configuring Claude Code perfectly for your team’s workflow. A new engineer joins. Without plugins, you’re walking them through manual setup. With plugins:

/plugin install dev-toolkit@your-org

Plugin structure:

my-plugin/
├── .claude-plugin/
│   └── plugin.json    # Manifest
├── skills/            # Skills (optional)
├── commands/          # Slash commands (optional)
├── hooks/             # Hooks (optional)
├── .mcp.json          # MCP configs (optional)
└── README.md

Teams can host private marketplaces internally. Plugins aren’t new functionality; they’re a distribution mechanism for the other extension types.

Read more on Claude Code official docs:

• Create a Plugin
• Discover and Install Prebuilt Plugins

TipExtension Strategy

Start simple. Add one MCP server for your most-used external tool. Create one hook that enforces a rule you keep reminding Claude about. Build a skill for the pattern it keeps getting wrong. Plugins come later, when you have something worth sharing across your team.

Advanced: Subagents and Parallel Workflows

If you’ve used Claude Code for a while, you’ve probably hit the context wall. The conversation grows, responses degrade, and eventually you’re reaching for /clear or starting fresh. Subagents solve this problem by isolating heavy work into separate context windows.

What Subagents Actually Are

A subagent is a preconfigured AI personality that Claude Code can delegate tasks to. Each one has:

• Its own system prompt defining how it operates
• Its own fresh context window (isolated from your main conversation)
• A configurable set of tools (following least privilege)
• Reusability across projects

Think of it like delegating to a contractor. You write up what needs doing (the prompt), they show up with their own tools and expertise, do the work independently, and hand you back the result. You don’t need to know every step they took.

Why Context Isolation Matters

Imagine this, you are refactoring a module, you asked Claude to analyze the existing code, propose three approaches, implement the winner, write tests, and update documentation. By the time you got to tests, Claude had forgotten details from the analysis phase. The context window was stuffed with intermediate reasoning that no longer mattered.

Without subagents, your main conversation accumulates everything:

TaskWhat Happens“Analyze this module”Claude reads 15 files, adds analysis to context“Compare approaches”Previous analysis + new comparison reasoning“Implement option B”All of the above + implementation details“Write tests”Context is now bloated with stale analysis

Subagents sidestep this. They receive one prompt, work in their own window, and return one condensed response. Your main thread stays lean. If I’d delegated “analyze and propose approaches” to a subagent, I’d have gotten back a clean summary without the intermediate file contents clogging my context.

Creating Your First Subagent

Run /agents and select “Create new agent”. You’ll specify:

1. A name and description (this helps Claude decide when to delegate)
2. Which tools the subagent can access
3. Which model to use (Sonnet for most tasks, Opus for complex reasoning)
4. A system prompt with the agent’s personality and instructions

Here’s a stripped down example for a code reviewer:

# Funny Code Reviewer
A senior engineer who reviews code with humor. Invoke when user says "funny review".
## Tools
Read (read-only tools only)
## Model
sonnet
## System Prompt
You are a staff senior engineer with a dry sense of humor. Review code for bugs, 
style issues, and improvement opportunities. Deliver feedback with wit but keep 
it constructive. No write access needed.

Parallel Subagents: The /infinite Pattern

Running multiple Claude Code instances on independent tasks is useful, but running parallel subagents takes this further. The /infinite command (credit to IndieDevDan’s repository) demonstrates this pattern:

/infinite specs/hero.spec components/heroes 6

This spawns 6 subagents simultaneously, each implementing a UI component differently based on the spec file. Each agent works in complete isolation, unaware of the others. You review all variations and pick the best one.

The magic is in how it generates prompts. The command:

1. Reads and analyzes your spec file
2. Surveys existing implementations in the output directory
3. Dynamically generates unique prompts for each subagent (Meta Prompting)
4. Assigns creative directions so agents don’t duplicate each other

Six agents produced six genuinely different implementations: one focused on accessibility, another on animation, another on minimal dependencies. Without the dynamic prompt generation, you’d get six variations of the same approach.

Git Worktrees for Multi-Agent Work

When you’re running multiple Claude Code instances locally, git worktrees keep everything organized. A worktree lets you check out multiple branches into separate directories without cloning the repo again.

cloud-code-crash-course/
├── hookup/              # Original (project/hookup branch)
├── agent-animations/    # Worktree 1 (separate branch)
└── agent-hero-redesign/ # Worktree 2 (separate branch)

Each agent works on its own branch. If you like the result, merge it. If not, delete the worktree and branch. Your original code stays untouched throughout.

The course demonstrated this with three simultaneous Claude instances (two local worktrees, one cloud), each implementing a different feature. The instructor ran all three in parallel, then merged them into a single branch with Claude handling the conflict resolution. The whole workflow took maybe 15 minutes for what would have been an hour of sequential work.

The Orchestration Tradeoff

Parallel agents scale your output, but they add coordination overhead. You’re now a manager, not just a developer. During the course demo, I noticed the instructor spending as much time switching between terminals and reviewing outputs as the agents spent coding. The merge step also required judgment calls about which implementation to keep when agents made different architectural choices.

This is the real skill shift with multi-agent systems. You’re not writing code line by line. You’re writing specs, reviewing implementations, and deciding which AI-generated approach best fits your constraints. Git worktrees make the mechanics manageable, but the cognitive load moves from “how do I implement this” to “how do I evaluate these six implementations.”

For simple tasks, this overhead isn’t worth it. For exploratory work where you genuinely want multiple approaches, or for parallelizable features with clear specs, the time savings are real.

Security: What AI Agents Handle Well (and What They Don’t)

If you’re expecting Claude Code to handle security for you, adjust your expectations. A December 2025 study by Tenzai called “Bad Vibes:Comparing the Secure Coding Capabilities of Popular Coding Agents” tested all the major coding agents (Cursor, Claude Code, Codex, Replit, Devin) by having each build identical applications. The result? 69 vulnerabilities across 15 apps. Every agent shipped vulnerable code.

Solved Vulnerability Classes

Agents consistently avoid problems with clear rules. SQL injection? They use parameterized queries. Cross-site scripting? Frameworks escape output properly. These vulnerability classes have built-in framework protections, so agents learned the patterns from their training data.

Authorization and Business Logic

This is where things fall apart. In the study, Codex allowed sellers to access any order in the system. Claude allowed unauthenticated users to delete files. Business logic vulnerabilities were everywhere: four out of five agents allowed negative quantities in shopping carts, three out of five allowed negative prices on products.

Security controls simply weren’t implemented. No CSRF protection. No security headers. Zero rate limiting on login (enabling brute force attacks). No account lockout. The agents didn’t implement these controls incorrectly. They didn’t implement them at all.

The SSRF Problem

All five agents introduced SSRF (Server-Side Request Forgery) vulnerabilities. There’s no universal rule for safe versus dangerous URL fetches because the answer depends entirely on context. When clear guardrails don’t exist, agents struggle.

The study confirms what experienced engineers already suspected: agents excel where deterministic rules exist and fail where judgment calls require understanding intent. I use Claude Code to move faster on implementation, but threat modeling and security architecture stay with me.

Key Takeaways

1. Context engineering is prompt engineering’s evolution. Static prompts don’t cut it for agents. You need dynamic context assembly from multiple sources, and Claude Code’s three-tier memory hierarchy (user, project, dynamic imports) gives you explicit control over what the model knows.
2. The extension system follows a clear hierarchy. MCP connects external tools (always loaded, costs tokens). Skills load on-demand based on relevance (more efficient). Hooks enforce hard rules that override soft guidance. Plugins bundle everything for team distribution.
3. Subagents solve the context bloat problem. Heavy tasks run in isolated context windows, returning condensed results. Your main conversation stays lean. Combine with git worktrees for parallel workflows that would take longer sequentially.
4. Security remains your job. Agents handle rule-based vulnerabilities (SQLi, XSS) well but miss authorization, business logic, and SSRF consistently. Treat AI-generated code like junior developer code: review everything, especially access control.
5. Start simple, extend as needed. Run /init, edit your CLAUDE.md, learn /compact and /rewind. Add one MCP server, one hook, one skill. Plugins and parallel agents come later when you’ve outgrown the basics.

Closing Thoughts

Claude Code represents a shift in how we interact with AI coding tools. It’s not just autocomplete or chat. It’s an agent with memory, tools, and the ability to delegate work. The learning curve is steeper than Copilot, but the ceiling is higher too.

What surprised me most was how much the workflow resembles managing people. You write clear specs (CLAUDE.md), delegate tasks (subagents), set boundaries (hooks), and review output (git worktrees). The skill isn’t typing code faster. It’s communicating intent clearly and knowing when to trust the output.

I hope you found this walkthrough useful. If you have questions or want to share your own Claude Code patterns, find me on LinkedIn.

Read in browser for best viewing experience.

In Part 2, we explored how DPP balances relevance and diversity using a single parameter α. But there’s a catch: we used the same α for every user.

Think about it. On a video platform like YouTube or Reels, you’ll find:

• User A: Watches nothing but true crime documentaries. Their last 50 videos? All true crime. They know what they like.
• User B: Watches cooking tutorials on Monday, standup comedy on Wednesday, and tech reviews on weekends. Their history looks like a buffet.

Should we show both users the same amount of diversity? Probably not.

User A has a clear, focused preference. Pushing diverse content might feel random or irrelevant to them. User B, on the other hand, has shown they enjoy variety. A narrowly focused feed might bore them.

This is the core limitation of standard DPP: it treats diversity as a system-level knob, not a user-level preference. The solution? Personalized DPP (pDPP), which adapts the diversity-relevance tradeoff per user based on their historical behavior.

The idea comes from a Huawei research paper on personalized re-ranking, and the implementation is surprisingly simple: we measure how “diverse” each user’s past interactions have been using information theory (Shannon entropy), then use that to personalize α.

In this post, we’ll:

• Quantify user diversity appetite using Shannon entropy over interaction history
• Derive a personalized α that scales diversity based on each user’s behavior
• Implement pDPP by extending our DPP code from Part 2
• Compare results side-by-side for narrow vs. broad-interest users

Let’s get into it.

The Core Insight: Users Have Different Diversity Appetites

How do we measure a user’s appetite for diversity? The Huawei paper uses Shannon entropy over the user’s interaction history.

Shannon entropy measures the “surprise” or “spread” in a distribution. If a user watches videos from 10 genres equally, entropy is high (maximum uncertainty about what they’ll watch next). If they watch 95% from one genre, entropy is low (very predictable).

This gives us a principled, interpretable metric: we’re not guessing what users want; we’re measuring what they’ve demonstrated through their actions.

The next step is to turn this entropy score into a personalized α value. A user with high entropy gets a higher α (more diversity weight), while a user with low entropy gets a lower α (more relevance weight).

The Math: Computing User Entropy

Let’s formalize the intuition from the previous section. For a user u, we compute Shannon entropy over their genre distribution:

H(u) = -Σ P(g|u) × log(P(g|u))

Where:

• P(g|u) is the probability of genre g in user u’s interaction history
• The sum is over all genres the user has interacted with

Let’s implement this:

import numpy as np
from collections import Counter

def compute_entropy(genre_list: list[str]) -> float:
    """Compute Shannon entropy over a list of genre interactions."""
    if not genre_list: return 0.0
    
    genre_counts = Counter(genre_list)
    total = len(genre_list)
    
    entropy = 0.0
    for count in genre_counts.values():
        p = count / total
        entropy -= p * np.log(p)
    
    return entropy

# User A: watches mostly one genre (narrow taste)
user_a_genres = ['action', 'action', 'action', 'action', 'comedy']
# User B: watches many genres (diverse taste)
user_b_genres = ['action', 'comedy', 'drama', 'documentary', 'horror']
print(f"User A(narrow taste) entropy: {compute_entropy(user_a_genres):.3f}")
print(f"User B(diverse taste) entropy: {compute_entropy(user_b_genres):.3f}")

The numbers confirm our intuition:

• User A (entropy = 0.500): With 4 action + 1 comedy, their behavior is highly predictable. Low entropy means low “surprise” in what they’ll watch next.
• User B (entropy = 1.609): With equal spread across 5 genres, their behavior is maximally uncertain. This is actually the maximum possible entropy for 5 categories: ln(5) ≈ 1.609.

What the values mean practically:

• Entropy of 0 = watches only one genre (perfectly predictable)
• Entropy of log(n) = watches n genres equally (maximum diversity)

So User B’s entropy being exactly log(5) tells us they have a perfectly uniform distribution across genres. User A’s 0.500 is much closer to 0, reflecting their strong single-genre preference.

This entropy value becomes our “diversity propensity score” that we’ll use to personalize α in the next section.

From Entropy to Personalized α

Now we have each user’s entropy score. How do we turn it into a personalized α?

Huawei paper uses a simple multiplicative formula:

α_u = f_u × α_0

Where:

• α_0 is your baseline diversity weight (the system default)
• f_u is a user-specific scaling factor between 0 and 1

The scaling factor f_u is computed via min-max normalization of the user’s entropy:

f_u = (H_u - H_min + l) / (H_max - H_min + l)

Where:

• H_u is the user’s entropy
• H_min, H_max are the min/max entropy across your user population
• l is a smoothing parameter

The l parameter as a personalization dial:

• l = 0: Full personalization. Users at H_max get f_u = 1, users at H_min get f_u = 0
• l → ∞: No personalization. Everyone gets f_u ≈ 0, so α_u ≈ 0 for all users
• In practice, l in the range of H_max - H_min gives moderate personalization

Let’s implement it:

def compute_f_u(H_u, H_min, H_max, l=0):
    return (H_u - H_min + l) / (H_max - H_min + l)

# Show how l controls personalization strength
print("Effect of smoothing parameter l on f_u:")
print("-" * 50)
print(f"{'l':<10} {'User A f_u':<15} {'User B f_u':<15} {'Difference':<10}")
print("-" * 50)

H_a = compute_entropy(user_a_genres)  # ~0.5
H_b = compute_entropy(user_b_genres)  # ~1.609
# For this example, assume population-wide bounds
H_min, H_max = 0.5, 1.61  # You'd compute these from your user base
for l in [0, 0.25, 0.5, 1.0, 2.0]:
    f_a = compute_f_u(H_a, H_min, H_max, l)
    f_b = compute_f_u(H_b, H_min, H_max, l)
    print(f"{l:<10} {f_a:<15.3f} {f_b:<15.3f} {f_b - f_a:<10.3f}")

As l increases, the gap between User A and User B shrinks. At l=0, we get maximum personalization (f_u ranges from 0 to 1). At l=2, both users are clustered around 0.5, meaning they get nearly the same diversity treatment. This lets you tune how aggressively the system personalizes: start conservative with higher l, then decrease it as you gain confidence in the entropy signal.

Now let’s bring in our DPP functions from Part 2.

def greedy_map_dpp(L, k):
    N = L.shape[0]
    selected = []
    remaining = set(range(N))
    
    for _ in range(k):
        best_item, best_det = None, -1
        
        for item in remaining:
            candidate_set = selected + [item]
            det_val = np.linalg.det(L[np.ix_(candidate_set, candidate_set)])
            
            if det_val > best_det:
                
                best_det = det_val
                best_item = item
        
        selected.append(best_item)
        remaining.remove(best_item)
    
    return selected

def youtube_dpp_kernel(relevance, embeddings, alpha=0.5, sigma=1.0):
    N = len(relevance)
    L = np.zeros((N, N))
    
    # Compute pairwise distances
    # D_ij = ||embedding_i - embedding_j||²
    for i in range(N):
        for j in range(N):
            if i == j:
                # Diagonal: quality squared
                
                L[i, i] = relevance[i] ** 2
            else:
                # Off-diagonal: scaled similarity with RBF kernel
                D_ij = np.sum((embeddings[i] - embeddings[j]) ** 2)
                similarity = np.exp(-D_ij / (2 * sigma**2))
                L[i, j] = alpha * relevance[i] * relevance[j] * similarity
    
    return L

def youtube_dpp_ranking(relevance, embeddings, k, alpha=0.5, sigma=1.0):
    W = list(range(len(relevance)))  # Remaining candidate indices
    R = []  # Final ranked list
    
    while len(W) > 0:
        # Build kernel for current candidate pool
        rel_W = relevance[W]
        emb_W = embeddings[W]
        L = youtube_dpp_kernel(rel_W, emb_W, alpha=alpha, sigma=sigma)
        
        # Select up to k items from current pool
        window_size = min(k, len(W))
        M = greedy_map_dpp(L, k=window_size)
        
        # Map back to original indices
        selected_items = [W[i] for i in M]
        R.extend(selected_items)
        
        # Remove selected items from candidate pool
        W = [w for w in W if w not in selected_items]
    
    return R

Let’s create the personalized version.

def compute_user_alpha(genre_history, H_min, H_max, alpha_0, l=0):
    """Compute personalized alpha for a user based on their genre history."""
    H_u = compute_entropy(genre_history)
    f_u = compute_f_u(H_u, H_min, H_max, l)
    return f_u * alpha_0

def pdpp_ranking(relevance, embeddings, k, genre_history, H_min, H_max, alpha_0=1.0, l=0, sigma=1.0):
    """Personalized DPP ranking using user's genre history."""
    alpha_u = compute_user_alpha(genre_history, H_min, H_max, alpha_0, l)
    print(f"User alpha: {alpha_u:.4f} (base alpha_0={alpha_0})")
    return youtube_dpp_ranking(relevance, embeddings, k, alpha=alpha_u, sigma=sigma)

# 5 videos with relevance scores and genre embeddings (2D for simplicity)
relevance = np.array([0.9, 0.85, 0.8, 0.7, 0.6, 0.5])
genre = ['Action', 'Action', 'Action', 'Comedy', 'Documentary', 'Mixed comedy/documentary']
# Embeddings: action movies are close together, others spread out
embeddings = np.array([
    [1.0, 0.0, 0.0],   # video 0: pure action
    [0.9, 0.1, 0.0],   # video 1: action (very similar to 0)
    [0.8, 0.1, 0.1],   # video 2: action (also very similar to 0)
    [0.0, 1.0, 0.0],   # video 3: comedy
    [0.0, 0.0, 1.0],   # video 4: documentary
    [0.1, 0.5, 0.5],   # video 5: mixed
])
# Global entropy bounds (you'd compute these from your user population)
H_min, H_max = 0.5, 1.61
# Compare rankings for User A (narrow taste) vs User B (diverse taste)
print("=== User A (narrow taste - prefers action) ===")
ranking_a = pdpp_ranking(relevance, embeddings, k=6, genre_history=user_a_genres, H_min=H_min, H_max=H_max, l=0)
print(f"Top 6: {[genre[i]+'('+str(relevance[i])+')' for i in ranking_a[:6]]}")
print("=== User B (diverse taste) ===")
ranking_b = pdpp_ranking(relevance, embeddings, k=6, genre_history=user_b_genres, H_min=H_min, H_max=H_max, l=0)
print(f"Top 6: {[genre[i]+'('+str(relevance[i])+')' for i in ranking_b[:6]]}")

The results show exactly what we’d expect:

User A (α ≈ 0.0) gets the three action movies upfront, ranked purely by relevance. With α near zero, diversity has almost no influence. The algorithm respects their focused history: “You’ve watched action 80% of the time, so here’s the best action content first.”

User B (α ≈ 1.0) sees a completely different pattern. They get one action movie, then the algorithm immediately jumps to comedy, documentary, and mixed content. The other two action movies get pushed to positions 5 and 6. Their high entropy history translates to high α, which penalizes clustering similar items together.

Same 6 videos. Same relevance scores. Completely different experiences.

This is the power of pDPP: rather than treating diversity as a global system setting, it becomes a per-user preference learned directly from behavior. User A’s narrow feed isn’t a bug; it’s what their history tells us they want. User B’s varied feed isn’t random; it matches their demonstrated exploration pattern.

A Production Variant: Bounded Additive Formulation

The multiplicative formula α_u = f_u × α_0 has a problem: it can be too aggressive. Look at User A’s result above: with f_u ≈ 0, they got α ≈ 0, meaning zero diversity consideration. In production, you probably don’t want to completely eliminate diversity for anyone.

The multiplicative approach also lacks intuitive bounds. If your team decides “we never want α below 0.4 or above 0.8,” the multiplicative formula doesn’t give you that control directly.

The additive bounded formulation:

Instead of multiplying, we use an additive offset centered around 0.5:

α_u = α_0 + (f_u - 0.5) × α_range

Where:

• α_0 is your baseline (center point)
• α_range controls how far users can deviate from baseline
• f_u - 0.5 shifts the range to [-0.5, +0.5], so the adjustment is symmetric

This gives you explicit floor and ceiling:

• Floor: α_0 - 0.5 × α_range
• Ceiling: α_0 + 0.5 × α_range

For example, with α_0 = 0.6 and α_range = 0.4:

• Minimum α = 0.6 - 0.2 = 0.4 (most focused users)
• Maximum α = 0.6 + 0.2 = 0.8 (most diverse users)

def compute_user_alpha_bounded(genre_history, H_min, H_max, alpha_0=0.6, alpha_range=0.4, l=0):
    """Compute personalized alpha with explicit floor/ceiling bounds."""
    H_u = compute_entropy(genre_history)
    f_u = compute_f_u(H_u, H_min, H_max, l)
    # Additive formulation: center at alpha_0, vary by alpha_range
    alpha_u = alpha_0 + (f_u - 0.5) * alpha_range
    return alpha_u

def pdpp_ranking_bounded(relevance, embeddings, k, genre_history, H_min, H_max, 
                         alpha_0=0.6, alpha_range=0.4, l=0, sigma=1.0):
    """Personalized DPP with bounded additive alpha."""
    alpha_u = compute_user_alpha_bounded(genre_history, H_min, H_max, alpha_0, alpha_range, l)
    return youtube_dpp_ranking(relevance, embeddings, k, alpha=alpha_u, sigma=sigma)

print("=== User A (narrow taste) - Bounded ===")
ranking_a_bounded = pdpp_ranking_bounded(
    relevance, embeddings, k=6, genre_history=user_a_genres,
    H_min=H_min, H_max=H_max, alpha_0=0.6, alpha_range=0.4
)
print(f"Top 6: {[genre[i]+'('+str(relevance[i])+')' for i in ranking_a_bounded]}\n")
print("=== User B (diverse taste) - Bounded ===")
ranking_b_bounded = pdpp_ranking_bounded(
    relevance, embeddings, k=6, genre_history=user_b_genres,
    H_min=H_min, H_max=H_max, alpha_0=0.6, alpha_range=0.4
)
print(f"Top 6: {[genre[i]+'('+str(relevance[i])+')' for i in ranking_b_bounded]}")

Now both users stay within the guardrails we set (0.4 to 0.8):

User A (α = 0.4) still gets all three action movies first, but now there’s some diversity consideration baked in. They’re at the floor, not at zero. If we had more videos in the candidate pool, this baseline diversity would prevent completely homogeneous results.

User B (α = 0.8) gets a diversified feed, but not as extreme as before. Compare to the multiplicative version where they had α ≈ 1.0: now the second action movie appears at position 4 instead of position 5. The ceiling prevents over-diversification.

Why this matters in production:

• You can guarantee minimum diversity for all users (prevents filter bubbles)
• You can cap maximum diversity (prevents feeds that feel random)
• The parameters are intuitive to tune: “baseline 0.6, users can vary ±0.2”
• Product and engineering can align on explicit bounds before launch

The tradeoff is less dramatic personalization. The multiplicative approach gave a 10x difference between User A and B (0.0 vs 1.0). The bounded approach gives a 2x difference (0.4 vs 0.8). Which is right depends on your product goals and risk tolerance.

Production Considerations

Deploying pDPP requires a few decisions that don’t show up in toy examples.

Computing H_min and H_max

You need population-wide entropy bounds. Two approaches:

1. Batch computation: Run a daily/weekly job over all active users, compute their entropy, store the min/max. Simple, but lags behind distribution shifts.
2. Approximate with theory: If you have n genres, the theoretical bounds are H_min = 0 (user watches one genre) and H_max = ln(n) (uniform distribution). This avoids the batch job but may not reflect your actual user distribution.

In practice, I’d recommend starting with theoretical bounds, then validating against your actual user distribution. If 99% of users fall between 0.3 and 1.2 but your theoretical max is 2.3, you’re wasting personalization range.

Cold Start

New users have no history, so no entropy. Options:

• Default to α_0 (the baseline) until they have N interactions
• Use cohort-level entropy (users who signed up from similar sources tend to have similar diversity preferences)
• Start slightly above baseline to encourage exploration, then let it settle

How Often to Update f_u

Entropy is relatively stable for established users. Someone who’s watched 500 videos won’t see their entropy swing wildly from one more view. Options:

• Batch (daily/weekly): Simplest. Compute entropy offline, store in user profile.
• Nearline (per session): Recompute when user starts a session. More responsive but adds latency.
• Streaming: Update incrementally with each interaction. Most responsive but adds complexity.

For most systems, daily batch is sufficient. Entropy changes slowly.

What Counts as “Genre”?

The paper uses genre, but you can use any categorical attribute:

• Content category (comedy, drama, news)
• Creator/channel
• Content length bucket (shorts vs. long-form)
• Engagement type (videos you comment on vs. passive watches)

You can even combine multiple: compute entropy over (genre, creator_type) tuples. The key is choosing attributes that meaningfully capture “diversity” for your product.

Key Takeaways

1. Standard DPP’s single α ignores that users have different diversity appetites. Shannon entropy over interaction history gives us a principled way to measure each user’s diversity propensity from their behavior.
2. pDPP personalizes the relevance-diversity tradeoff per user. High-entropy users (explorers) get more diversity; low-entropy users (focused) get more relevance.
3. The bounded additive formulation is production-friendly. It gives explicit floor/ceiling control and prevents edge cases where diversity is completely eliminated.

References & Further Reading

• Personalized Re-ranking for Recommendation (Pei et al., 2020) — The Huawei paper introducing pDPP. Covers the entropy-based personalization approach and the parameterized min-max normalization formula.
• Determinantal Point Processes for Machine Learning (Kulesza & Taskar, 2012) — The definitive DPP tutorial. Essential background for understanding the kernel mechanics.
• Practical Diversified Recommendations on YouTube with Determinantal Point Processes (Cheng et al., 2018) — YouTube’s production DPP implementation that we extended in this post.

Previous posts in this series:

• Part 1: Diversity in Recommendations using MMR — Maximal Marginal Relevance as a simpler diversity approach.
• Part 2: Diversity in Recommendations using DPP — DPP fundamentals and YouTube’s kernel parameterization.

I hope this series has been useful for thinking about diversity in your recommendation systems! If you’re experimenting with pDPP or have questions, connect with me on LinkedIn to share your experience.

Diversity in Recommendations using Personalized DPP (pDPP) was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

Part 2 of the diversity series: Moving from greedy MMR to principled probabilistic frameworks with DPPs

Read in browser for best viewing experience.

In previous blog, we explored MMR as a practical way to inject diversity into recommendation results. MMR works well, but it has a fundamental limitation: it’s greedy and myopic. At each step, it only considers similarity to items already selected, not the global structure of the final set.

What if we could evaluate the diversity of an entire set at once? That’s exactly what Determinantal Point Processes (DPPs) offer. DPPs come from probability theory and give us a principled way to assign higher probability to sets of items that are both high-quality and diverse. For example, MMR might select two sci-fi thrillers that individually seem distinct from other picks, but together make the final set feel repetitive. DPPs evaluate the whole set at once, catching this overlap.

In this post, we’ll:

• Build geometric intuition for DPPs through determinants as volume
• Understand the quality-diversity decomposition for production systems
• Implement greedy MAP inference step-by-step
• Explore YouTube’s production approach with tunable parameters (α, σ)
• Walk through the windowed algorithm for ranking entire feeds
• Compare DPP and MMR on our video recommendation example

Linear Algebra Refresher: Determinants as Volume

Before diving into DPPs, we need to refresh one key concept: determinants measure volume.

Consider two vectors in 2D space. The determinant of the matrix formed by these vectors equals the area of the parallelogram they span.

import numpy as np
import matplotlib.pyplot as plt

# Two similar vectors (small area)
v1 = np.array([1, 0])
v2 = np.array([0.9, 0.1])
similar_matrix = np.array([v1, v2])
print(f"Similar vectors determinant: {np.linalg.det(similar_matrix):.3f}")

# Two orthogonal vectors (maximum area)
v1 = np.array([1, 0])
v2 = np.array([0, 1])
diverse_matrix = np.array([v1, v2])
print(f"Orthogonal vectors determinant: {np.linalg.det(diverse_matrix):.3f}")

This is the geometric intuition behind DPPs: diverse sets span more volume, giving them higher determinants. When vectors point in similar directions, they span a thin, squashed parallelogram. When they’re spread out, the volume is large.

This extends to higher dimensions. For k vectors in d-dimensional space, the determinant of their Gram matrix (L = VVᵀ) measures the squared volume of the k-dimensional parallelotope they span.

Note: Gram matrix is also called the “kernel matrix” in DPP literature.

# 3 items in 3D feature space
items = np.array([
    [1, 0, 0],    # Item A
    [0, 1, 0],    # Item B  
    [0, 0, 1],    # Item C (diverse from A, B)
])
L = items @ items.T  # Gram matrix
print(f"Diverse set det(L): {np.linalg.det(L):.3f}")

# Replace C with something similar to A
items_similar = np.array([
    [1, 0, 0],      # Item A
    [0, 1, 0],      # Item B
    [0.8, 0.1, 0.1],  # Item C' (similar to A)
])
L_similar = items_similar @ items_similar.T
print(f"Similar set det(L): {np.linalg.det(L_similar):.3f}")

The diverse set has 100x higher determinant. This is the core of DPPs: recommendation probability proportional to determinant means diverse sets are exponentially more likely.

DPP Fundamentals: From Volume to Probability

Now we can define what a DPP actually is. Given a ground set of N items (our candidate pool), a DPP defines a probability distribution over all possible subsets. The key formula is:

P(Y) ∝ det(L_Y)

where:

• Y is a subset of items we might recommend
• L is an N×N positive semi-definite kernel matrix capturing item relationships
• L_Y is the submatrix of L indexed by items in Y

The probability of selecting a set is proportional to the determinant of its kernel submatrix. From our linear algebra refresher, we know determinants measure volume, so diverse sets that span more volume get higher probability.

Let’s use the same video recommendation example from Part 1:

relevance = np.array([0.9, 0.85, 0.8, 0.7, 0.6, 0.5])
genre = ['Action', 'Action', 'Action', 'Comedy', 'Documentary', 'Mixed comedy/documentary']
embeddings = np.array([
    [1.0, 0.0, 0.0],   # video 0: pure action
    [0.9, 0.1, 0.0],   # video 1: action (very similar to 0)
    [0.8, 0.1, 0.1],   # video 2: action (also very similar to 0)
    [0.0, 1.0, 0.0],   # video 3: comedy
    [0.0, 0.0, 1.0],   # video 4: documentary
    [0.1, 0.5, 0.5],   # video 5: mixed
])
# Build kernel matrix L = VVᵀ (Gram matrix from embeddings)
L = embeddings @ embeddings.T
print("Kernel matrix L:")
print(np.round(L, 2))
print(f"\nSimilarity of video 0 ({genre[0]}), with other videos")
for i in range(1,len(genre)):
    print(f"Video {i}: {L[0][i]} ({genre[i]})")

Notice how videos 0, 1, 2 (all Action) have high similarity (~0.8–0.9), while Action vs Comedy is 0. Now let’s compare the DPP probability of two different 3-video sets:

def subset_det(L, indices):
    """Compute det(L_Y) for a subset of indices"""
    L_Y = L[np.ix_(indices, indices)]
    return np.linalg.det(L_Y)

# Set 1: Three action videos (redundant)
redundant = [0, 1, 2]  # Action, Action, Action
det_redundant = subset_det(L, redundant)
print(f"Redundant set: {[genre[i] for i in redundant]}")
print(f"  det(L_Y) = {det_redundant:.4f}")
# Set 2: Action + Comedy + Documentary (diverse)
diverse = [0, 3, 4]
det_diverse = subset_det(L, diverse)
print(f"\nDiverse set: {[genre[i] for i in diverse]}")
print(f"  det(L_Y) = {det_diverse:.4f}")
print(f"\nDiverse set is {det_diverse/det_redundant:.0f}x more likely under DPP")

The diverse set spanning Action, Comedy, and Documentary is 10000x more likely than picking three similar action videos! This is the DPP magic: it naturally penalizes redundancy by evaluating the entire set at once, rather than making greedy pairwise comparisons like MMR.

NoteWhy L(Kernel Matrix) needs to be Positive Semi-Definite (PSD)

Note: Why L(Kernel Matrix) needs to be Positive Semi-Definite (PSD)

L needs to be positive semi-definite (PSD). Practically, this means:

All eigenvalues are ≥ 0

Determinants of all submatrices are ≥ 0 (which we need for valid probabilities!)

Why DPPs require PSD: Since P(Y) ∝ det(L_Y), we need det(L_Y) ≥ 0 for all subsets. PSD guarantees this.

Good news for practitioners: When you build L = VVᵀ (Gram matrix from embeddings), it’s automatically PSD. So as long as you construct your kernel matrix as L = VVᵀ (or use standard similarity kernels like RBF), you’re safe. You only run into trouble if you manually construct L with arbitrary values.

Quality vs Diversity Decomposition

So far, our kernel matrix L only captures similarity between items. But in real recommendations, we care about both quality (relevance) and diversity. A diverse set of low-quality items is useless.

DPPs handle this elegantly through the quality-diversity decomposition:

L[i,j] = q_i × q_j × S[i,j]

where:

• q_i is the quality (relevance) score for item i
• S[i,j] is the similarity between items i and j (from embeddings)

This decomposes naturally: diagonal entries L[i,i] = q_i² capture item quality, while off-diagonal entries capture similarity scaled by both items’ quality. High-quality similar items create larger off-diagonal terms, which reduce the determinant (penalizing redundancy among your best items).

Let’s rebuild our kernel matrix with relevance scores:

# Similarity matrix from embeddings
S = embeddings @ embeddings.T

# Quality-weighted kernel: L[i,j] = q_i * q_j * S[i,j]
q = relevance.reshape(-1, 1)  # Column vector
L_quality = (q @ q.T) * S
print("Quality-weighted kernel L:")
print(np.round(L_quality, 3))

Let’s compare the two sets again with the new kernel matrix:

redundant = [0, 1, 2]  # Top 3 by relevance, all Action
diverse = [0, 3, 4]    # Action + Comedy + Documentary

det_redundant = subset_det(L_quality, redundant)
det_diverse = subset_det(L_quality, diverse)
print(f"Redundant set (top-3 relevance): {[f'{genre[i]} ({relevance[i]})' for i in redundant]}")
print(f" det = {det_redundant:.6f}")
print(f"\nDiverse set (mixed relevance): {[f'{genre[i]} ({relevance[i]})' for i in diverse]}")
print(f" det = {det_diverse:.6f}")
print(f"\nDiverse set is {det_diverse/det_redundant:.0f}x more likely")

Even though the diverse set includes lower-relevance items (0.7, 0.6 vs 0.85, 0.8), the diversity bonus from spanning different genres still makes it far more probable (3815x as compared to 10000x before). The quality weighting ensures we don’t pick diverse but irrelevant items.

The intuition: Think of q_i as the “length” of item i’s feature vector. The determinant measures the volume spanned by these vectors. A high-quality item contributes a longer vector, but if two long vectors point in the same direction (similar items), the parallelogram is still flat. You need both length (quality) AND spread (diversity) to maximize volume.

This maps directly to production recsys:

• q_i = your relevance model output (e.g., predicted click probability)
• S[i,j] = Similarity of content embeddings (or 1 - distance in any embedding space)
• L = the kernel that balances both signals

Greedy MAP: The Practical DPP Algorithm

We’ve established that DPPs assign higher probability to diverse, high-quality sets via determinants. The natural question: how do we find the best set?

The challenge: Finding the subset Y that maximizes det(L_Y) is called MAP (Maximum A Posteriori) inference. With N candidates, there are 2^N possible subsets. This is NP-hard in general.

The solution: Greedy MAP approximation. Build the set one item at a time, always adding whichever item increases the determinant most. This runs in O(Nk³) time and provides provably good approximations, where N is the number of candidates to select from and k is the number of items to select.

Walking Through Greedy MAP Step-by-Step

Let’s trace through selecting 3 videos from our candidate pool. At each step, we try adding every remaining candidate and pick whoever maximizes det(L_Y).

Step 1: First Selection

With an empty set, we’re just comparing single-item determinants. For a 1×1 matrix, det([L[i,i]]) = L[i,i] = q_i². So the first pick is simply the highest-quality item:

print("Step 1: Evaluating single-item determinants (= quality squared)")
for i in range(len(relevance)):
    det_single = L_quality[i, i]
    print(f"  Video {i} ({genre[i]}): det = {det_single:.3f} (relevance² = {relevance[i]**2:.3f})")

first_pick = int(np.argmax([L_quality[i,i] for i in range(len(relevance))]))
print(f"\n→ Select video {first_pick} ({genre[first_pick]})")

Step 2: Second Selection

Now it gets interesting. We evaluate each remaining candidate by computing det(L_Y) for the 2-item set:

selected = [first_pick]
print(f"Step 2: Current set = [Video {first_pick}]")
print("Evaluating candidates:")

for candidate in range(len(relevance)):
    if candidate in selected:
        continue
    
    subset = selected + [candidate]
    L_subset = L_quality[np.ix_(subset, subset)]
    det_val = np.linalg.det(L_subset)
    
    # Show the 2x2 matrix being evaluated
    print(f"\n  Video {candidate} ({genre[candidate]}):")
    print(f"    L_subset = [[{L_subset[0,0]:.3f}, {L_subset[0,1]:.3f}],")
    print(f"                [{L_subset[1,0]:.3f}, {L_subset[1,1]:.3f}]]")
    print(f"    det = {L_subset[0,0]:.3f} × {L_subset[1,1]:.3f} - {L_subset[0,1]:.3f} × {L_subset[1,0]:.3f} = {det_val:.4f}")

Notice how adding another Action video (high off-diagonal similarity) leads to a lower determinant, while Comedy or Documentary (zero similarity to Action) leads to a higher determinant. In this case we will pick Video 3 as it has the highest determinant of 0.3969.

# Find second pick
best_det, second_pick = -1, None
for candidate in range(len(relevance)):
    if candidate in selected:
        continue
    subset = selected + [candidate]
    det_val = np.linalg.det(L_quality[np.ix_(subset, subset)])
    if det_val > best_det:
        best_det, second_pick = det_val, candidate

print(f"\n→ Select video {second_pick} ({genre[second_pick]}), det = {best_det:.4f}")
selected.append(second_pick)

Step 3: Third Selection

Same process with a 3×3 determinant:

print(f"Step 3: Current set = {selected}")
print("Evaluating candidates:")

for candidate in range(len(relevance)):
    if candidate in selected:
        continue
    
    subset = selected + [candidate]
    det_val = np.linalg.det(L_quality[np.ix_(subset, subset)])
    print(f"  Video {candidate} ({genre[candidate]}): det = {det_val:.4f}")
# Find third pick
best_det, third_pick = -1, None
for candidate in range(len(relevance)):
    if candidate in selected:
        continue
    subset = selected + [candidate]
    det_val = np.linalg.det(L_quality[np.ix_(subset, subset)])
    if det_val > best_det:
        best_det, third_pick = det_val, candidate
selected.append(third_pick)
print(f"\n→ Select video {third_pick} ({genre[third_pick]})")
print(f"\nFinal selection: {[genre[i] for i in selected]}")

The Complete Implementation

Now we can wrap this into a clean function:

def greedy_map_dpp(L, k):
    N = L.shape[0]
    selected = []
    remaining = set(range(N))
    
    for _ in range(k):
        best_item, best_det = None, -1
        
        for item in remaining:
            candidate_set = selected + [item]
            det_val = np.linalg.det(L[np.ix_(candidate_set, candidate_set)])
            
            if det_val > best_det:
                
                best_det = det_val
                best_item = item
        
        selected.append(best_item)
        remaining.remove(best_item)
    
    return selected

# Run it
result = greedy_map_dpp(L_quality, k=3)
print(f"Greedy MAP result: {[f'{genre[i]} ({relevance[i]})' for i in result]}")

Pure relevance ranking would give us Action, Action, Action. DPP gives us Action, Comedy, Documentary. Three genres covered in three slots.

Tuning the Quality-Diversity Tradeoff

In production, you need a way to control how much the system emphasizes relevance versus diversity. YouTube’s approach, described in their 2018 paper on DPPs for recommendations, provides a clean parameterization that’s proven to work at scale.

YouTube’s Kernel Parameterization

Instead of the simple L[i,j] = q_i × q_j × S[i,j] formulation, YouTube uses:

L[i,i] = q_i² (diagonal: quality squared)

L[i,j] = α × q_i × q_j × exp(-D_ij / 2σ²) for i ≠ j (off-diagonal: scaled similarity)

where:

• q_i is the relevance score for item i
• D_ij is the distance between items i and j in embedding space
• α controls the overall strength of diversity (0 ≤ α ≤ 1)
• σ controls the length scale of similarity

The key insight: The diagonal entries (quality) remain unchanged, but off-diagonal entries (similarity) are scaled by α. This gives you direct control over the diversity penalty.

Note: This formulation is identical to a standard (Gaussian) radial basis function (RBF) kernel when α = 1.

Understanding the Parameters

α (diversity strength):

• α = 1: Full diversity penalty. Similar items strongly repel each other.
• α = 0: No diversity penalty. Reduces to pure relevance ranking.
• α ∈ (0,1): Smooth tradeoff between quality and diversity.

σ (similarity scale):

• Small σ: Only very similar items are considered redundant (tight clusters).
• Large σ: Even moderately different items are penalized (broad diversity).

Let’s implement this new kernel building function:

def youtube_dpp_kernel(relevance, embeddings, alpha=0.5, sigma=1.0):
    N = len(relevance)
    L = np.zeros((N, N))
    
    # Compute pairwise distances
    # D_ij = ||embedding_i - embedding_j||²
    for i in range(N):
        for j in range(N):
            if i == j:
                # Diagonal: quality squared
                
                L[i, i] = relevance[i] ** 2
            else:
                # Off-diagonal: scaled similarity with RBF kernel
                D_ij = np.sum((embeddings[i] - embeddings[j]) ** 2)
                similarity = np.exp(-D_ij / (2 * sigma**2))
                L[i, j] = alpha * relevance[i] * relevance[j] * similarity
    
    return L

# Build kernel with YouTube's approach
L_youtube = youtube_dpp_kernel(relevance, embeddings, alpha=0.5, sigma=1.0)
print("YouTube DPP kernel (α=0.5, σ=1.0):")
print(np.round(L_youtube, 3))

Sweeping the α Parameter

Let’s see how α affects the diversity-quality tradeoff:

Key observations:

• α = 0.0–0.25: Pure relevance ranking. Off-diagonal terms are too weak to overcome quality differences, so we get the top-3 by relevance (all Action).
• α = 0.5: Moderate diversity kicks in. The algorithm trades Video 2 (Action, 0.80) for Video 3 (Comedy, 0.70), accepting a small relevance drop for genre diversity.
• α = 0.75–1.0: Strong diversity preference. The algorithm maximizes spread across genres (Action, Comedy, Documentary), dropping Video 1 (Action, 0.85) in favor of lower-relevance items from different genres.

The transition happens around α = 0.5, where the diversity penalty becomes strong enough to outweigh small relevance differences. In production, you’d tune α based on metrics like session diversity, user engagement, or A/B test results.

Sweeping the σ Parameter

The σ parameter controls what counts as “similar”. Let’s fix α = 0.75 and vary σ:

Key observations:

• σ = 0.1: Very tight similarity threshold. Only near-identical items are penalized as redundant, so the diversity effect barely activates. Result: top-3 by relevance (all Action).
• σ = 0.5–1.0: Moderate similarity scale. Items within the same genre are considered similar enough to penalize, leading to genre mixing (Action, Comedy, Documentary).
• σ = 2.0: Broader similarity scale. Even moderately different items start to look similar. The algorithm becomes more willing to sacrifice relevance for diversity.
• σ = 5.0: Very broad similarity scale. Almost everything looks similar, which actually weakens the diversity signal. The algorithm struggles to distinguish between items and reverts toward relevance-based ranking.

In practice, σ should match the natural scale of your embedding space. If your embeddings are normalized (unit length), σ ≈ 1.0 is a reasonable starting point. Tune it based on how strict you want diversity: smaller σ for subtle redundancy detection within tight clusters, larger σ for aggressive spreading across broader categories.

Visualizing the RBF Kernel

The exp(-D_ij / 2σ²) term is an RBF (Radial Basis Function) kernel. Let’s visualize how it maps distances to similarities:

Interpretation: With σ = 1.0, items at distance 1.0 have similarity ≈ 0.6 (moderate penalty), while items at distance 2.0 have similarity ≈ 0.14 (weak penalty). Smaller σ makes the kernel “sharper” — only very close items are considered similar.

Practical Guidance

Tuning strategy:

1. Start with α = 0.5, σ = 1.0: This is a reasonable default that balances quality and diversity.

2. Tune α first:

• Sweep α ∈ [0.2, 0.8] in 0.1 increments
• Measure offline diversity metrics (intra-list distance, genre entropy) vs relevance metrics (nDCG)
• Plot the Pareto frontier to find acceptable tradeoffs

3. Then tune σ:

• If items are still too similar, decrease σ (tighter similarity definition)
• If diversity is too aggressive, increase σ (looser similarity definition)
• σ should roughly match the typical distance scale in your embedding space

4. A/B test top candidates: Run online experiments with 2–3 (α, σ) pairs to validate offline metrics.

Typical values in production:

• Balanced systems: α ≈ 0.4–0.6, σ ≈ 0.5–1.5
• Quality-focused (e.g., search): α ≈ 0.2–0.4
• Discovery-focused (e.g., explore page): α ≈ 0.6–0.9

The beauty of YouTube’s parameterization is that it decouples quality (diagonal) from diversity (off-diagonal), giving you two independent knobs to tune the system behavior.

YouTube’s Production DPP Implementation

The greedy MAP algorithm we implemented earlier works well for selecting a fixed top-k set. However, YouTube’s production system needs to rank entire feeds of hundreds or thousands of videos. Their 2018 paper introduces a windowed approach that processes candidates in batches, making DPPs scalable for large-scale recommendation systems.

The Windowed Algorithm

Instead of selecting all k items at once from the full candidate pool, YouTube’s algorithm:

1. Selects k items from the current candidate pool using greedy MAP
2. Adds them to the result list R
3. Removes selected items from the pool and updates the kernel matrix
4. Repeats until all candidates are ranked (or enough items are selected)

This approach has several advantages:

• Scalability: Processes N items in O(Nk²) time instead of O(Nk³)
• Diversity within pages: Ensures each “window” of k items is diverse, not just the top-k
• Memory efficiency: Can process large pools without computing full N×N kernels
• Flexibility: Can stop early after ranking enough items for the user’s viewport

Here’s the algorithm from YouTube’s paper:

def youtube_dpp_ranking(relevance, embeddings, k, alpha=0.5, sigma=1.0):
    W = list(range(len(relevance)))  # Remaining candidate indices
    R = []  # Final ranked list
    
    while len(W) > 0:
        # Build kernel for current candidate pool
        rel_W = relevance[W]
        emb_W = embeddings[W]
        L = youtube_dpp_kernel(rel_W, emb_W, alpha=alpha, sigma=sigma)
        
        # Select up to k items from current pool
        window_size = min(k, len(W))
        M = greedy_map_dpp(L, k=window_size)
        
        # Map back to original indices
        selected_items = [W[i] for i in M]
        R.extend(selected_items)
        
        # Remove selected items from candidate pool
        W = [w for w in W if w not in selected_items]
    
    return R

Walking Through the Windowed Algorithm

Let’s trace through ranking all 6 videos with window size k=2:

Code

print("YouTube Windowed DPP (k=2, α=0.75, σ=1.0)")
print("=" * 70)
# We'll manually trace the first few iterations
W = list(range(len(relevance)))
R = []
k_window = 2
alpha, sigma = 0.75, 1.0
iteration = 1
while len(W) > 0 and iteration <= 3:  # Show first 3 iterations
    print(f"\nIteration {iteration}:")
    print(f"  Remaining pool W: {W}")
    print(f"  Pool genres: {[genre[i] for i in W]}")
    
    # Build kernel for current pool
    rel_W = relevance[W]
    emb_W = embeddings[W]
    L = youtube_dpp_kernel(rel_W, emb_W, alpha=alpha, sigma=sigma)
    
    # Select k items
    window_size = min(k_window, len(W))
    M = greedy_map_dpp(L, k=window_size)
    selected_items = [W[i] for i in M]
    
    print(f"  Selected from pool: {M} (pool indices)")
    print(f"  Maps to original: {selected_items}")
    print(f"  Genres: {[genre[i] for i in selected_items]}")
    print(f"  Relevance: {[relevance[i] for i in selected_items]}")
    
    R.extend(selected_items)
    W = [w for w in W if w not in selected_items]
    
    iteration += 1
print(f"\n{'─'*70}")
print(f"Final ranking R: {R}")
print(f"Genres: {[genre[i] for i in R]}")

Key observations:

• Iteration 1: From the full pool, selects the highest-quality Action video (0.9) and highest-quality diverse option (Comedy, 0.7)
• Iteration 2: From remaining items, selects another Action video (0.85) paired with Documentary (0.6) for diversity
• Iteration 3: Clears the remaining pool with the last Action and Mixed genre video

Notice how diversity is enforced within each window of 2 items, not just globally. This ensures users see varied content throughout their feed, not just at the top.

Comparing Single-Pass vs Windowed

Let’s compare the two approaches side-by-side:

Comparison of top-3 selections:
Method           Indices      Genres                        Relevance
────────────────────────────────────────────────────────────────────────────────────────────────────
Single-pass      [0, 3, 4]    Action, Comedy, Documentary   [0.9, 0.7, 0.6]
Windowed (k=2)   [0, 3, 1]    Action, Comedy, Action        [0.9, 0.7, 0.85]

The windowed approach includes Video 1 (Action, 0.85) in the top-3 because it was selected in the second window after Videos 0 and 3 were removed. This demonstrates how window size k affects the ranking: smaller windows enforce local diversity but may allow similar items to appear nearby across window boundaries.

In the paper, it’s mentioned that typically the candidate pool is several hundreds of videos and sub-window size is a dozen (12) or so videos.

The windowed algorithm is YouTube’s production solution for bringing the mathematical elegance of DPPs to real-world scale. By processing items in batches, it maintains diversity guarantees while handling feeds with thousands of candidates in milliseconds.

System Design Integration

Now that we’ve covered DPP theory and implementation, let’s see how it fits into a production recommendation system. Recall from Part 1 the standard recommendation pipeline:

DPPs slot into the Re-ranking stage, right where we used MMR in Part 1. Here’s how the stages work together:

1. Candidate Generation (Recall)

• Retrieve ~1000 candidates from multiple sources (collaborative filtering, content-based, trending)
• Goal: High recall, cast a wide net
• Output: Large pool of potentially relevant items

2. Scoring (Ranking)

• ML model predicts relevance for each candidate (e.g., click probability, watch time)
• Features: user history, item metadata, context
• Output: Relevance scores q_i for each item, typically in 200-500 candidates

3. Re-ranking (Diversity)

• This is where DPP comes in
• Input: Scored candidates + content embeddings
• Build kernel matrix: L[i,j] = α × q_i × q_j × exp(-D_ij / 2σ²)
• Apply windowed DPP to produce final ranked list
• Output: Diverse, high-quality feed. Batches of size 6–12 candidates

4. Business Logic

• Apply final filters (content policy, already-seen, etc.)
• Insert ads, promotions
• Output: User-facing feed

The key advantage of DPPs is that they give you a mathematically principled way to balance quality and diversity, backed by decades of research in probability theory. Once you understand the volume-based intuition, the system becomes just as interpretable as MMR but with stronger theoretical guarantees.

Key Takeaways

• DPPs provide a mathematically principled framework for diversity. Unlike greedy approaches like MMR that make local decisions, DPPs evaluate entire sets at once through determinants. This volume-based intuition (diverse sets span more space) translates directly into probability: P(Y) ∝ det(L_Y).
• The quality-diversity decomposition maps naturally to production recsys. The kernel formulation L[i,j] = α × q_i × q_j × similarity(i,j) cleanly separates relevance (from your ML model) and diversity (from content embeddings). You get two interpretable knobs: α controls how much diversity matters, σ controls what counts as “similar.”
• Greedy MAP is deterministic and faster. While exact MAP inference is NP-hard, the greedy algorithm runs in O(Nk²) time and provides strong approximations. Unlike sampling-based approaches, it’s deterministic and stable, making it suitable for A/B testing and debugging.
• YouTube’s windowed approach scales to video feeds. By processing candidates in batches of k items (typically 6–12), the algorithm maintains diversity throughout long feeds, not just at the top. This makes DPPs practical for ranking hundreds or thousands of candidates in milliseconds.

References & Further Reading

• Determinantal Point Processes for Machine Learning (Kulesza & Taskar, 2012) — The definitive tutorial covering DPP theory, algorithms, and applications. Highly readable despite the math.
• Practical Diversified Recommendations on YouTube with Determinantal Point Processes (Cheng et al., 2018) — YouTube’s paper describing their windowed algorithm and kernel parameterization. This is the blueprint for production DPP systems.

Previous blog post:

• Part 1: Diversity in Recommendations using MMR — My previous post covering Maximal Marginal Relevance as a simpler alternative to DPPs.

I hope you found this deep dive into DPPs useful! If you’re implementing diversity in your recommendation system, I’d love to hear about your experience. Connect with me on LinkedIn to share your thoughts or questions.

Diversity in Recommendations — Determinantal Point Processes (DPP) was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

Beyond Accuracy: Using Maximal Marginal Relevance to create more diverse and engaging recommendations.

Read in browser for best experience.

Picture this: you open YouTube looking for something to watch. The algorithm shows you 10 videos, and they’re all basketball highlights from the same game. Sure, you like basketball, but maybe right now you’re in the mood for that ML lecture you’ve been putting off. Or maybe something else entirely.

Most recommendation systems score each item independently based on predicted relevance. The result? If your model is confident you like basketball, you get basketball. All basketball. The system optimizes for the single most likely intent while ignoring the possibility that you might have multiple interests, or that its prediction might simply be wrong.

Diversity addresses this by explicitly penalizing redundancy. Instead of asking “what’s most relevant?”, we ask “what’s most relevant that we haven’t already covered?”

This isn’t just theoretically appealing. Diversity boosting delivers real results in production systems and can dramatically improve engagement metrics. Nobel laureate Harry Markowitz called diversification “the only free lunch in finance” when applied to portfolio management. The same principle applies to recommendations: hedge your bets across the user’s possible intents.

In this post, we’ll:

• Explain why diversity matters and when pointwise scoring fails
• Introduce Maximal Marginal Relevance (MMR), a simple and battle-tested diversity method
• Walk through the algorithm step-by-step with code
• Explore how the λ parameter lets you tune the quality-diversity tradeoff

Why Diversity Matters

Recommendation systems face a fundamental challenge: they don’t know what the user wants right now. They have historical signals, but a user who watched NBA highlights yesterday might be looking for a cooking video today. A user who binges Peppa Pig with their kids on weekends has different needs on a Tuesday night.

Pointwise ranking models score each candidate independently. If your embedding space clusters similar content together (as it should), similar items receive similar scores. The top-K results end up being variations of the same thing.

This creates several problems:

• Missed opportunities: If the user’s current intent doesn’t match your top prediction, you’ve wasted valuable screen real estate
• Filter bubbles: Users get stuck in narrow content loops, reducing long-term engagement
• Poor fault tolerance: When your model is wrong about intent, it’s wrong across the entire slate

Diversity isn’t unique to recommendations. Portfolio managers diversify across asset classes to reduce risk. Autonomous driving systems like IntentNet diversify over possible agent intents rather than betting on a single predicted trajectory. The core insight is the same: when you’re uncertain about what the environment wants, spread your bets.

The business case is compelling. Users who see a mix of content that spans their interests are more likely to find something they want right now, and more likely to come back.

The Problem Setup

Let’s make this concrete with an example. Imagine you have 6 candidate videos for a user:

import numpy as np
import pandas as pd
from fastcore.utils import L
from sklearn.metrics.pairwise import cosine_similarity

relevance = np.array([0.9, 0.85, 0.8, 0.7, 0.6, 0.5])
genre = L(['Action', 'Action', 'Action', 'Comedy', 'Documentary', 'Mixed comedy/documentary'])
df = pd.DataFrame({
    'video_id': [i for i in range(len(relevance))],
    'genre': genre,
    'relevance': relevance
})
df

Pure relevance ranking returns: Action, Action, Action, Comedy, Documentary, Mixed

The top three slots go to nearly identical content. If the user wanted comedy, they have to scroll past three action videos. If they wanted a documentary, it’s buried at position 5.

Now consider what happens with diversity-aware ranking: Action, Comedy, Documentary, Action, Mixed

The slate now covers three distinct genres in the top three positions. The second action video drops to position 4 since its marginal value is low once we’ve already shown an action video.

This is the core insight: marginal relevance matters more than absolute relevance. The 3rd action video isn’t 3x as valuable as the first. Each additional similar item has diminishing returns.

We can represent each video as an embedding vector over genres:

embeddings = np.array([
    [1.0, 0.0, 0.0],   # video 0: pure action
    [0.9, 0.1, 0.0],   # video 1: action (very similar to 0)
    [0.8, 0.2, 0.0],   # video 2: action (also very similar to 0)
    [0.0, 1.0, 0.0],   # video 3: comedy
    [0.0, 0.0, 1.0],   # video 4: documentary
    [0.1, 0.5, 0.5],   # video 5: mixed
])

similarities = cosine_similarity(embeddings)
print(f"Similarity b/w Video 0 and 1 is {similarities[0,1]:.2f}")
print(f"Similarity b/w Video 0 and 3 is {similarities[0,3]:.2f}")

Videos 0 and 1 have cosine similarity near 1.0. Videos 0 and 3 are orthogonal (similarity = 0). This similarity structure is exactly what MMR exploits to rerank results.

Maximal Marginal Relevance: The Algorithm

MMR was introduced by Carbonell and Goldstein in 1998 for document retrieval and summarization. The core formula is:

MMR = ArgMax[λ × Rel(item, user) — (1-λ) × Max(Sim(item, selected_items))]

Where:

• item = candidate video not yet selected
• user = user profile
• selected_items = set of items already chosen for the slate
• λ = trade-off parameter between relevance and diversity (0 to 1)

You can see the first part λ × Rel(item, user) is just relevance. The second part (1-λ) × Max(Sim(item, selected_items)) is the diversity penalty.

The algorithm is greedy and iterative:

1. First pick: Select the item with highest relevance (the similarity penalty term is zero since nothing has been selected yet)
2. Subsequent picks: For each remaining candidate, compute relevance minus a penalty for similarity to the most similar already-selected item. Pick the candidate with the highest score.
3. Repeat until you’ve filled K slots

The key insight is the Max in the second term. You’re penalized based on your similarity to whichever selected item you’re most similar to. Being different from 2 out of 3 selected items doesn’t help if you’re a near-duplicate of the third.

The λ parameter controls the tradeoff:

• λ = 1.0 → Pure relevance ranking, no diversity
• λ = 0.0 → Maximum diversity, ignore relevance
• λ = 0.5 → Equal weight to both
• λ = 0.7 → The paper’s suggested starting point

Walking Through MMR Step-by-Step

Let’s trace through the algorithm by hand before writing any code.

Step 1: First selection

With an empty slate, there’s no diversity penalty.

selected_item = np.argmax(relevance)
print(f"First selected item: {int(selected_item)}, Relevance: {relevance[selected_item]}")

We pick the highest relevance item: Video 0 (Action, relevance = 0.9).

Step 2: Second selection

Now we need to choose between the remaining videos. Let’s compare Video 1 (another Action, relevance=0.85) vs Video 3 (Comedy, relevance=0.7)

lam = 0.5
video_1_mmr_score = lam * relevance[1] - (1-lam) * cosine_similarity(embeddings[[1]], embeddings[[selected_item]])
video_3_mmr_score = lam * relevance[3] - (1-lam) * cosine_similarity(embeddings[[3]], embeddings[[selected_item]])
print(f"MMR score of Video 1: {video_1_mmr_score.item():.3f}")
print(f"MMR score of Video 3: {video_3_mmr_score.item():.3f}")

Video 3 wins despite having lower relevance. The similarity penalty makes Video 1’s score negative. This is MMR in action: redundancy is penalized so heavily that a lower-quality diverse item beats a high-quality duplicate.

The full implementation

def mmr_select(relevance, embeddings, k=3, lam=0.5):
    n = len(relevance)
    selected = []
    remaining = list(range(n))

    # First pick: pure relevance (no penalty term)
    first = int(np.argmax(relevance))
    selected.append(first)
    remaining.remove(first)

    # Subsequent picks: balance relevance vs redundancy
    while len(selected) < k:
        best_score, best_idx = -float('inf'), None

        for i in remaining:
            # Penalty = similarity to most similar already-selected item
            max_sim = max(similarities[i, j] for j in selected)
            mmr_score = lam * relevance[i] - (1 - lam) * max_sim

            if mmr_score > best_score:
                best_score, best_idx = mmr_score, i

        selected.append(best_idx)
        remaining.remove(best_idx)

    return selected

result = mmr_select(relevance, embeddings, k=3, lam=0.5)
print(genre[result])

Pure relevance ranking would give us Action, Action, Action. MMR gives us Action, Comedy, Documentary. Three genres covered in three slots.

Tuning λ: The Quality-Diversity Tradeoff

The λ parameter is your dial between relevance and diversity. Let’s see how it affects the ranking:

for lam in [0.3, 0.5, 0.7, 0.9, 1.0]:
    result = mmr_select(relevance, embeddings, k=6, lam=lam)
    print(f"λ={lam}: {genre[result]}")

At λ=1.0 we recover pure relevance ranking. As λ decreases, diverse items get promoted earlier in the slate.

Choosing λ in practice:

• Start with λ=0.7 (the original paper’s suggestion)
• A/B test different values against engagement metrics

System Design of Diversity Boosting

In terms of system design, diversity boosting can be added at various stages by using different techniques, but MMR falls into the class of re-rank algorithms which are typically applied after pointwise scoring.

The diagram above shows a typical recsys pipeline:

1. Candidate Videos → Raw pool of items to consider
2. Pointwise video scorer → Takes user features and video features, outputs a relevance score per video independently
3. Ranking and Diversity Boosting (highlighted in green) → This is where MMR lives. It takes the pointwise scores and reorders them to balance relevance with diversity
4. Policy → Business rules layer (e.g., content safety filters, frequency caps)
5. User → Final slate shown

MMR is a post-scoring reranker. The pointwise model does its job first (predicting relevance), then MMR adjusts the ordering to reduce redundancy. This is a clean separation of concerns — you don’t need to retrain your ranking model to add diversity.

Key Takeaways

1. Pointwise ranking has a blind spot. Scoring items independently leads to redundant results when similar items cluster in your embedding space.
2. Diversity hedges against uncertainty. When you don’t know the user’s current intent, covering multiple possibilities increases the chance of a hit.
3. MMR is simple and effective. The algorithm is greedy, easy to implement, and has powered production systems for decades.

I hope you enjoyed reading it. If there is any feedback on the blog post, feel free to comment below or reach out on LinkedIn.

Diversity in Recommendations — Maximal Marginal Relevance (MMR) was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read in browser for best experience.

One of my favorite hobbies outside of work is sharing technical knowledge through blog posts. Over the years, writing has opened doors I never expected: speaking at AI meetups, opportunities to author books with famous publishers, and countless LinkedIn conversations that turned into opportunities. In this post, I want to share why I think every technical person should consider blogging, and give you some practical tips to get started.

My blogging setup

I publish on my personal website, which I host through GitHub Pages and build using Quarto. Jupyter notebooks are my tool of choice for writing since they let me mix text, code and output of that code. Quarto handles converting them into HTML. After letting a post sit on my site for a few days, I cross-post to Medium for greater visibility.

My posts focus on AI, ML, and Python development. I try to center my writing around real examples of the techniques I’m discussing so readers can gain practical knowledge they can apply to their own use cases. I typically write about:

• Niche techniques I use at work
• Emerging technologies (like generative AI lately)
• Topics I’m actively studying and researching

Five reasons to start blogging

1. Increase your visibility and reach

Many people have reached out to me through LinkedIn or email asking about topics I’ve written about. Sometimes these turn into job opportunities. Blogs are like resumes, but better. They show what you actually know, not just where you’ve worked.

2. It helps you learn

Organizing knowledge forces you to synthesize your own ideas. One reliable test of whether you understand something is whether you can explain it to someone else. Writing a blog post is a great way to do that. I often discover gaps in my own understanding while trying to explain a concept clearly.

3. It improves your writing skills

The more you write, the more comfortable you become expressing your ideas. This carries over to work emails, design docs, and presentations.

4. Establish yourself as an expert

Through my writing, I’ve landed opportunities to present at local AI meetups, write books with well-known publishers, and review technical books before their release. None of this would have happened without a public body of work people could find.

5. It saves time (DRY principle)

Don’t Repeat Yourself applies to explanations too. Instead of spending two hours giving someone an overview of a topic, I can point them to an article. I also refer back to my own posts as notes, which saves me time when I need to refresh my memory on something I worked on years ago.

Tips for writing your first post

If you’re convinced but don’t know what to write about, here are some ideas that have worked for me:

1. Curate and comment: Make a list of links to blog posts, articles, or papers you like. Write brief summaries and highlight what you found useful. This is low-pressure and helpful to others.
2. Summarize what you learned: Attended a conference? Taking a class? Write up your notes and takeaways.
3. Turn emails into posts: Any technical explanation you’ve written more than once in email or Slack should become a blog post.
4. Write for yourself from a year ago: You’re best positioned to help people one step behind you. Your background, your style, and your knowledge level will give a different perspective than existing tutorials.
5. Answer: what would have helped me six months ago?: Think about problems you struggled with and write the guide you wish existed.
6. Don’t be a perfectionist: Post your writing sooner rather than later. Waiting just builds pressure and you end up writing less. You can always update posts later.

Additional resources for motivation

• Why you (yes, you) should blog by Rachel Thomas
• Show Your Work!: 10 Ways to Share Your Creativity and Get Discovered by Austin Kleon

Getting started

You don’t need a fancy setup. GitHub Pages is free. Quarto or Jekyll can turn markdown or notebooks into a website with minimal configuration. Medium works fine if you just want to write without managing infrastructure.

The hardest part is starting. Pick one thing you learned recently and write 500 words about it. That’s it. You can refine your process after you’ve published a few posts.

I hope you enjoyed reading this. If there is any feedback on the blog post, feel free to reach out on LinkedIn.

Read here — Link

Large Language Models (LLMs) have surged in popularity, pushing the boundaries of natural language processing. OpenAI’s GPT-3.5 is a prime example, revolutionizing our technology interactions and sparking innovation. Particularly, LLMs excel in building Question Answering applications on knowledge bases. In this blog, we delve into the top trending GitHub repository for this week: the PrivateGPT repository and do a code walkthrough.

What is privateGPT?

One of the primary concerns associated with employing online interfaces like OpenAI chatGPT or other Large Language Model systems pertains to data privacy, data control, and potential data leakage. The privateGPT repository presents a fully offline alternative for engaging with personal documents. It is constructed using open-source tools and technology, thereby enabling the utilization of LLMs capabilities without compromising data privacy or encountering data leakage issues.

Running privateGPT locally

To run privateGPT locally, users need to install the necessary packages, configure specific variables, and provide their knowledge base for question-answering purposes. Additional information on the installation process and usage can be found in the repository documentation or by referring to a dedicated blog post on the topic.

Essentially you can run it by calling the privateGPT.py file like -

python privateGPT.py

And get a response that also mentions the sources it looked up for context.

Code Walkthrough

privateGPT code comprises two pipelines:

1. Ingestion Pipeline: This pipeline is responsible for converting and storing your documents, as well as generating embeddings for them. The documents are stored in a suitable format, and their embeddings are stored in an embedding database.
2. Q&A Interface: This interface accepts user prompts, the embedding database, and an open-source Language Model (LM) model as inputs. It utilizes these inputs to generate responses to the user’s queries.

1. Ingestion Pipeline

Let’s delve into the ingestion pipeline for a closer examination. The ingestion pipeline encompasses the following steps:

1. Identifying files with various extensions and retrieving all the knowledge base from the source directory.
2. Splitting the documents into smaller chunks based on the parameters of chunk_size and chunk_overlap.
3. Initializing the Huggingfaceembeddings module of langchain. This involves loading a pre-trained language model from the sentence_transformers library.
4. Initializing the Chroma database from langchain.vectorstores. This step involves taking the chunked text and the initialized embedding model and saving it in the embedding database on disk.

Let’s look at these steps one by one.

1.1 Identifying and loading files from the source directory

First, we import the required libraries and various text loaders from langchain.document_loaders.

import os
import glob
from typing import List
from multiprocessing import Pool
from tqdm import tqdm
from langchain.document_loaders import (
    CSVLoader,
    EverNoteLoader,
    PDFMinerLoader,
    TextLoader,
    UnstructuredEmailLoader,
    UnstructuredEPubLoader,
    UnstructuredHTMLLoader,
    UnstructuredMarkdownLoader,
    UnstructuredODTLoader,
    UnstructuredPowerPointLoader,
    UnstructuredWordDocumentLoader,
)
from langchain.docstore.document import Document

Next, we define the mapping b/w each extension and their respective langchain document loader. You can read document loader documentation for more available loaders.

# Map file extensions to document loaders and their arguments
LOADER_MAPPING = {
    ".csv": (CSVLoader, {}),
    ".doc": (UnstructuredWordDocumentLoader, {}),
    ".docx": (UnstructuredWordDocumentLoader, {}),
    ".enex": (EverNoteLoader, {}),
    ".epub": (UnstructuredEPubLoader, {}),
    ".html": (UnstructuredHTMLLoader, {}),
    ".md": (UnstructuredMarkdownLoader, {}),
    ".odt": (UnstructuredODTLoader, {}),
    ".pdf": (PDFMinerLoader, {}),
    ".ppt": (UnstructuredPowerPointLoader, {}),
    ".pptx": (UnstructuredPowerPointLoader, {}),
    ".txt": (TextLoader, {"encoding": "utf8"}),
}p

Next, we define our single document loader.

def load_single_document(file_path: str) -> Document:
    ## Find extension of the file
    ext = "." + file_path.rsplit(".", 1)[-1] 
    if ext in LOADER_MAPPING: 
        # Find the appropriate loader class and arguments
        loader_class, loader_args = LOADER_MAPPING[ext] 
        # Invoke the instance of document loader
        loader = loader_class(file_path, **loader_args) 
        ## Return the loaded document
        return loader.load()[0] 
    raise ValueError(f"Unsupported file extension '{ext}'")
    
git_dir = "../../../../privateGPT/"
loaded_document = load_single_document(git_dir+'source_documents/state_of_the_union.txt')
print(f'Type of loaded document {type(loaded_document)}')
loaded_document

The load_single_document function accomplishes the following steps:

1. Extracts the file extension from the given file path.
2. Retrieves the corresponding document loader and its arguments from the previously defined LOADER_MAPPING dictionary.
3. Creates an instance of the appropriate document loader.
4. Loads the document using the instantiated loader.
5. Returns the loaded document.

We can see that load_single_document returns a document of type langchain.schema.Document. Which according to the documentation consists of page_content (the content of the data) and metadata (auxiliary pieces of information describing attributes of the data).

def load_documents(source_dir: str, ignored_files: List[str] = []) -> List[Document]:
    """
    Loads all documents from the source documents directory, ignoring specified files
    """
    all_files = []
    for ext in LOADER_MAPPING:
        #Find all the files within source documents which matches the extensions in Loader_Mapping file
        all_files.extend(
            glob.glob(os.path.join(source_dir, f"**/*{ext}"), recursive=True)
        )
    
    ## Filtering files from all_files if its in ignored_files
    filtered_files = [file_path for file_path in all_files if file_path not in ignored_files]
    
    ## Spinning up resource pool
    with Pool(processes=os.cpu_count()) as pool:
        results = []
        with tqdm(total=len(filtered_files), desc='Loading new documents', ncols=80) as pbar:
            # Load each document from filtered files list using load_single_document function
            for i, doc in enumerate(pool.imap_unordered(load_single_document, filtered_files)):
                results.append(doc)
                pbar.update()
    
    return results

The load_single_documents function carries out the following steps:

1. Initializes an empty dictionary called all_files.
2. For each extension in the LOADER_MAPPING dictionary, it searches for all the files with that extension in the source directory and adds them to the all_files list.
3. Creates a new list named filtered_files by removing the files listed in the ignored_files list from the all_files list.
4. Executes a parallel loading operation on all the files in the filtered_files list using the load_single_document function, and appends the results to the results list.
5. Returns the list of loaded documents.

loaded_documents = load_documents(git_dir+'source_documents')
print(f"Length of loaded documents: {len(loaded_documents)}")
loaded_documents[0]

You can see we have loaded the state_of_the_union.txt file from the privateGPT repo. As this is the only file in that directory the length of loaded documents is one.

1.2 Splitting the documents into smaller chunks

Now we have seen how we can load multiple documents of different extensions using the load_documents function. The next step is to look at process_document function which loads and splits large documents into smaller chunks.

from langchain.text_splitter import RecursiveCharacterTextSplitter

chunk_size = 500
chunk_overlap = 50
def process_documents(source_dir: str, ignored_files: List[str] = []) -> List[Document]:
    """
    Load documents and split in chunks
    """
    print(f"Loading documents from {source_dir}")
    documents = load_documents(source_dir, ignored_files)
    if not documents:
        print("No new documents to load")
        exit(0)
    print(f"Loaded {len(documents)} new documents from {source_dir}")
    ## Load text splitter
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
    ## Split text
    texts = text_splitter.split_documents(documents)
    print(f"Split into {len(texts)} chunks of text (max. {chunk_size} tokens each)")
    return texts
processed_documents = process_documents(git_dir+'source_documents')

The process_documents function performs the following steps:

1. Loads all the documents from the source_dir directory using the load_documents function.
2. Initializes an instance of RecursiveCharacterTextSplitter from the langchain.text_splitter module, providing the chunk_size and chunk_overlap parameters. This class is responsible for splitting a list of documents into smaller overlapping chunks. [RecursiveCharacterTextSplitter documentation].
3. Uses the split_documents method of the RecursiveCharacterTextSplitter instance to split the loaded documents into smaller chunks.
4. Returns the resulting list of the smaller document chunks.

1.3 Initializing the embedding model

Next, we load our embedding module which converts the smaller document chunks from previous steps to embeddings.

from langchain.embeddings import HuggingFaceEmbeddings
EMBEDDINGS_MODEL_NAME = "all-MiniLM-L6-v2"
embeddings = HuggingFaceEmbeddings(model_name=EMBEDDINGS_MODEL_NAME)

print("Testing on a single query.")
embedded_vector = embeddings.embed_query("What is your name?")
print(f"Size of embedded vector: {len(embedded_vector)}")

The given code snippet carries out the following steps:

1. Imports the HuggingFaceEmbeddings function from the langchain.embeddings module. This function is responsible for loading and encapsulating the SentenceTransformers embeddings, which are used for generating dense vector representations of sentences. You can refer to the HuggingFaceEmbeddings documentation for more details.
2. Loads the all-MiniLM-L6-v2 model from the sentence_transformers library. This model is specifically designed to map sentences and paragraphs into a 384-dimensional dense vector space. It is commonly utilized for tasks such as semantic search and similarity analysis.

We can see that our embedded vector on a sample query returns a 384 dimension vector.

1.4 Embed smaller text and save it in the vector database

The next step involves utilizing the document chunks and the embedding model to store the documents and their corresponding embeddings in a vector database.

from chromadb.config import Settings
from langchain.vectorstores import Chroma

PERSIST_DIRECTORY= git_dir+"db"
# Define the Chroma settings
CHROMA_SETTINGS = Settings(
        chroma_db_impl='duckdb+parquet',
        persist_directory=PERSIST_DIRECTORY,
        anonymized_telemetry=False
)
## Create the embedding database
db = Chroma.from_documents(processed_documents, embeddings, persist_directory=PERSIST_DIRECTORY, client_settings=CHROMA_SETTINGS)
db.persist()

The given code snippet performs the following operations:

1. It imports the Settings class from the chromadb.config module and the Chroma class from the langchain.vectorstores module.

2. It creates an instance of the Settings class named CHROMA_SETTINGS, providing several configuration parameters:

• chroma_db_impl is set to 'duckdb+parquet', specifying the implementation to be used for the Chroma vector database.
• persist_directory is set to the PERSIST_DIRECTORY variable defined earlier, indicating the directory where the vector database will be saved.
• anonymized_telemetry is set to False, indicating whether anonymized telemetry data should be collected.

3. It creates a vector database by calling the Chroma.from_documents() method. This method takes the following arguments:

• processed_documents: The list of processed documents obtained from the previous step.
• embeddings: The embeddings object/model used to generate the document embeddings.
• persist_directory: The directory where the vector database will be persisted, specified by the PERSIST_DIRECTORY variable.
• client_settings: The settings object (CHROMA_SETTINGS) containing configuration parameters for the vector database.

4. We use db.persist() to store the index for future retrieval task

## Test the semantic retrieval 
db.similarity_search(query="What is the American Rescue Plan?", k= 4)

To test the retrieval of semantic similarity, we can use the similarity_search function. similarity_search function takes a text query as input and returns the top k=4 document chunks from the vector database.

2. Question & Answer Interface

Let’s explore the Q&A interface in more detail. The Q&A interface consists of the following steps:

1. Load the vector database and prepare it for the retrieval task.
2. Load a pre-trained Large language model from LlamaCpp or GPT4ALL.
3. Prompt the user with a query and generate a response using the RetrievalQA pipeline from langchain.chains.

Let’s look at these steps one by one.

2.1 Load the vector database

First, we import the required libraries.

from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import Chroma
from chromadb.config import Settings
git_dir = "../../../../privateGPT/"
PERSIST_DIRECTORY= git_dir+"db"
EMBEDDINGS_MODEL_NAME = "all-MiniLM-L6-v2"

# Define the Chroma settings
CHROMA_SETTINGS = Settings(
        chroma_db_impl='duckdb+parquet',
        persist_directory=PERSIST_DIRECTORY,
        anonymized_telemetry=False
)
embeddings = HuggingFaceEmbeddings(model_name=EMBEDDINGS_MODEL_NAME)
db = Chroma(persist_directory=PERSIST_DIRECTORY, embedding_function=embeddings, client_settings=CHROMA_SETTINGS)
retriever = db.as_retriever()

The given code snippet carries out the following steps:

1. Loads the embeddings using the HuggingFaceEmbeddings function, which was previously used to create the embedding store.
2. Instantiates a Chroma vector database that was created earlier.
3. Sets the vector database in retrieval mode.

## Testing retriever
retriever.vectorstore.similarity_search(query = "What is Amercian rescue plan?")

2.2 Load a pre-trained Large language model.

from langchain.llms import GPT4All

MODEL_PATH = git_dir+"models/ggml-gpt4all-j-v1.3-groovy.bin" 
MODEL_N_CTX=1000
# Prepare the LLM
llm = GPT4All(model=MODEL_PATH, n_ctx=MODEL_N_CTX, backend='gptj', callbacks=None, verbose=False)

The code snippet above creates an instance of the GPT4All class named llm, which represents the Language Model (LLM) using the GPT-4All model. The constructor of GPT4All takes the following arguments:
- model: The path to the GPT-4All model file specified by the MODEL_PATH variable.
- n_ctx: The context size or maximum length of input sequences specified by the MODEL_N_CTX variable.
- backend: The backend to use for the LLM. In this case, it is set to ‘gptj’.
- callbacks: The callbacks to be used during the LLM execution. In this case, it is set to None.
- verbose: A boolean flag indicating whether to print verbose output during LLM execution. In this case, it is set to False.

2.3 Prompt the user with a query and generate a response

from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever, return_source_documents=True)
query = "What is American rescue plan?"
res = qa(query)
answer, docs = res['result'], res['source_documents']

# Get the answer from the chain
# Print the result
print("\n\n> Question:")
print(query)
print("\n> Answer:")
print(answer)

# Print the relevant sources used for the answer
for document in docs:
    print("\n> " + document.metadata["source"] + ":")
    print(document.page_content)

> Question:
What is American rescue plan?

> Answer:
 The American Rescue Plan is a program that provides funding to schools to hire teachers and help students make up for lost learning due to the COVID-19 pandemic. It also provides economic relief for tens of millions of Americans by helping them put food on their table, keep a roof over their heads, and cut the cost of health insurance. The plan also helps working people by providing breathing room and giving them a little breathing room. It is a program that helps millions of families on Affordable Care Act plans save $2,400 a year on their health care premiums and combat climate change by cutting energy costs for families an average of $500 a year.

> ../../../../privateGPT/source_documents/state_of_the_union.txt:
The American Rescue Plan gave schools money to hire teachers and help students make up for lost learning.  

I urge every parent to make sure your school does just that. And we can all play a part—sign up to be a tutor or a mentor. 

Children were also struggling before the pandemic. Bullying, violence, trauma, and the harms of social media.

> ../../../../privateGPT/source_documents/state_of_the_union.txt:
It fueled our efforts to vaccinate the nation and combat COVID-19. It delivered immediate economic relief for tens of millions of Americans.  

Helped put food on their table, keep a roof over their heads, and cut the cost of health insurance. 

And as my Dad used to say, it gave people a little breathing room. 

And unlike the $2 Trillion tax cut passed in the previous administration that benefitted the top 1% of Americans, the American Rescue Plan helped working people—and left no one behind.

> ../../../../privateGPT/source_documents/state_of_the_union.txt:
Look, the American Rescue Plan is helping millions of families on Affordable Care Act plans save $2,400 a year on their health care premiums. Let’s close the coverage gap and make those savings permanent. 

Second – cut energy costs for families an average of $500 a year by combatting climate change.

> ../../../../privateGPT/source_documents/state_of_the_union.txt:
That’s why the Justice Department required body cameras, banned chokeholds, and restricted no-knock warrants for its officers. 

That’s why the American Rescue Plan provided $350 Billion that cities, states, and counties can use to hire more police and invest in proven strategies like community violence interruption—trusted messengers breaking the cycle of violence and trauma and giving young people hope.

Firstly, an instance of the RetrievalQA class named qa is created using the from_chain_type method. The RetrievalQA class is a chain specifically designed for question-answering tasks over an index. Please refer to the documentation for further details. The from_chain_type method takes the following arguments:

• llm: The Language Model instance (llm) that was created previously.
• chain_type: A string representing the type of chain to be used. In this case, it is set to "stuff". There may be other available chain types specific to the question-answering scenario. Please consult the langchain documentation for more information.
• retriever: An instance of a Chroma database used to retrieve relevant documents for the given query.
• return_source_documents: A boolean flag indicating whether to return the source documents along with the answer. In this case, it is set to True.

Next, the qa instance is used to process a query. The Language Model (LLM) within the qa instance generates a response that includes the query, the answer, and the source documents used as context for generating the answer.

Finally, the answer and source documents are printed out for display.

Conclusion

In this blog post, we explored privateGPT, its implementation, and the code walkthrough for its ingestion pipeline and q&A interface. I hope this blog post has been valuable in understanding privateGPT and its implementation. I recommend my readers try privateGPT on their knowledge base.

I hope you enjoyed reading it. If there is any feedback on the code or just the blog post, feel free to comment below or reach out on LinkedIn.

What is Model Calibration?

When working with classification problems, machine learning models often produce a probabilistic outcome ranging between 0 to 1. This probabilistic output is then used by people to make decisions. Unfortunately, many machine learning models’ probabilistic outputs cannot be directly interpreted as the probability of an event happening. To achieve this outcome, the model needs to be calibrated.

Formally, a model is said to be perfectly calibrated if, for any probability value p, a prediction of a class with confidence p is correct 100*p percent of the time. In more simple terms, the probabilistic output equals the probability of occurrence.

Let us visualize a perfect calibrated and non-calibrated curve.

import numpy as np
import pandas as pd
import plotly.io as pio
import plotly.graph_objects as go
import plotly.express as px
pio.renderers.default = "plotly_mimetype+notebook_connected"

chart_df = pd.DataFrame({
    "actual_prob": np.arange(0,1.1, 0.1),
    "Calibrated": np.arange(0,1.1, 0.1),
    "Non-Calibrated":  [min(1,val - np.random.randn()*(1-idx)/20) for idx, val in enumerate(np.arange(0,1.1, 0.1))]
})
fig = px.line(
        data_frame = chart_df, 
        markers = True,
        x = "actual_prob", 
        y = ["Calibrated", "Non-Calibrated"],
        template = "plotly_dark")

fig.update_layout(
        title  = {"text": "Calibrated vs Non-calibrated model", "y": 0.98, "x": 0.5},
        xaxis_title="Predicted Probability",
        yaxis_title="Actual Probability",
        font = dict(size=15), 
        legend=dict(yanchor="bottom", y=0.95, orientation="h")
)

fig.update_traces(patch={"line": {"dash": "dash"}}) 
fig.show(renderer="notebook")

On the x-axis, we have model output p which is between 0 and 1 and on the y-axis, we have fractions of positive captured within the predicted probability bin. We expect a linear relationship with slope 1.

Why do we need Model calibration?

There are many cases where model calibration is not required like in the case of ranking or selecting the top 20% for some targeting campaign. Calibrated models are especially important in making decision between multiple options of different magnitude or sizes, like expected value problems. In complex machine-learning decision engines, a machine-learning model might be used in conjunction with other machine-learning models.

For example, the business would like to prioritize upselling customer an additional product. So, a data scientist builds two ML models to estimate the probability of a customer buying two different products in addition to the products they are already purchasing.

In this case, Product 1 with $10 in revenue has an 80% probability of upselling, while Product 2 with $100 in revenue has a 60% chance of upselling. As a business, you might want to recommend Product 2 to the customer because of the higher expected value.

To do this expected value comparison between models, you need your models to be calibrated to output accurate probabilities.

Case study: Three methods for calibration

For demonstration in this article, we will be solving a binary classification problem using the Adult Income Dataset from the UCI machine learning Repository[1] to predict if a certain individual income based on various census information (education level, age, gender, occupation, and more) exceeds $50K/year. For this example, we will limit our scope to the United States and the following predictors:

• Age: continuous variable, individuals’ age
• Occupation: categorical variable, Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.
• HoursPerWeek: continuous variable, amount of hours spent in a job per week
• Education: categorical variable, Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.

In the below code we:

• Import the libraries needed for this project
• Load the data from our source
• Label our columns
• Filter data to only our desired columns, and limit our scope just to the United States for simplicity
• Label encode our categorical columns
• Set our random seed (for reproducibility)
• Split our dataset into test and train

## Importing required libraries
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn.isotonic import IsotonicRegression
from sklearn.preprocessing import LabelEncoder
from sklearn.calibration import CalibratedClassifierCV
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('dark_background')

def calibration_data(y_true, y_pred):
    df = pd.DataFrame({'y_true':y_true, 'y_pred_bucket': (y_pred//0.05)*0.05 + 0.025})
    cdf = df.groupby(['y_pred_bucket'], as_index=False).agg({'y_true':["mean","count"]})
    return cdf.y_true.values[:,0][cdf.y_true.values[:,1]>10], cdf.y_pred_bucket.values[cdf.y_true.values[:,1]>10]

def label_encoder(df,columns):
    '''
    Function to label encode
    Required Input - 
        - df = Pandas DataFrame
        - columns = List input of all the columns which needs to be label encoded
    Expected Output -
        - df = Pandas DataFrame with lable encoded columns
        - le_dict = Dictionary of all the column and their label encoders
    '''
    le_dict = {}
    for c in columns:
        print("Label encoding column - {0}".format(c))
        lbl = LabelEncoder()
        lbl.fit(list(df[c].values.astype('str')))
        df[c] = lbl.transform(list(df[c].values.astype('str')))
        le_dict[c] = lbl
    return df, le_dict

df = pd.read_csv( "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data", header=None)
df.columns = [
    "Age", "WorkClass", "fnlwgt", "Education", "EducationNum",
    "MaritalStatus", "Occupation", "Relationship", "Race", "Gender",
    "CapitalGain", "CapitalLoss", "HoursPerWeek", "NativeCountry", "Income"
]

## Filtering for Unites states
df = df.loc[df.NativeCountry == ' United-States',:]

## Only - Taking required columns
df = df.loc[:,["Education", "Age","Occupation", "HoursPerWeek", "Income"]]
df, _ = label_encoder(df, columns = ["Education", "Occupation", "Income"])
X = df.loc[:,["Education","Age", "Occupation", "HoursPerWeek"]]
y = df.Income

seed = 42
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=seed)

Next, we:

• Fit a random forest classifier on this data
• Visualize the calibration plot

## Fitting the model on training data
rf_model = RandomForestClassifier(random_state=seed)
rf_model.fit(X_train, y_train)

## getting the output to visualize on test data
prob_true, prob_pred  = calibration_data(y_true = y_test, 
                                          y_pred = rf_model.predict_proba(X_test)[:,1])

chart_df = pd.DataFrame({
    "actuals": prob_true,
    "predicted": prob_pred,
    "expected": prob_pred
})
fig = px.line(
        data_frame = chart_df, 
        markers = True,
        x = "predicted", 
        y = ["actuals", "expected"], 
        template = "plotly_dark")
fig.update_layout(
        title  = {"text": "Calibration Plot: Without calibration", "y": 0.95, "x": 0.5},
        xaxis_title="Predicted Probability",
        yaxis_title="Actual Probability",
        font = dict(size=15)
)
fig.show(renderer='notebook')

As we can see above, the model is over-predicting in lower deciles and under-predicting in higher deciles.

Calibration method 1: Isotonic Regression

Isotonic regression is a variation of ordinary least squares regression. Isotonic regression has the added constraints that the predicted values must always be increasing or decreasing, and the predicted values have to lie as close to the observation as possible. Isotonic regression is often used in situations where the relationship between the input and output variables is known to be monotonic, but the exact form of the relationship is not known. In our case, we want a monotonic behavior where the actual probabilities must always be increasing with increasing predicted probability.

Mathematically, we can write the Isotonic regression as following -

Where-

• iso is the isotonic function
• f(x) is the predicted probability from the original classifier f
• pcalib is the calibrated probability

and we want to optimize the following function -

Isotonic regressions is prone to overfitting, hence it works better when there is enough training data available.

We will be using CalibratedClassifierCVto calibrate our classifier. The isotonic method fits a non-parametric isotonic regressor, which outputs a step-wise non-decreasing function. [2]

## training model using random forest and isotonic regression for calibration
calibrated_rf = CalibratedClassifierCV(RandomForestClassifier(random_state=seed), method = 'isotonic')
calibrated_rf.fit(X_train, y_train)

## getting the output to visualize on test data
prob_true_calib, prob_pred_calib  = calibration_data(y_test, calibrated_rf.predict_proba(X_test)[:,1])

chart_df = pd.DataFrame({
    "actuals": prob_true_calib,
    "predicted": prob_pred_calib,
    "expected": prob_pred_calib
})
fig = px.line(
        data_frame = chart_df, 
        markers = True,
        x = "predicted", 
        y = ["actuals", "expected"], 
        template = "plotly_dark")
fig.update_layout(
        title  = {"text": "Calibration Plot: Isotonic", "y": 0.95, "x": 0.5},
        xaxis_title="Predicted Probability",
        yaxis_title="Actual Probability",
        font = dict(size=15)
)
fig.show(renderer='notebook')

As we can see above, the model is well calibrated once Isotonic regression is used because the actual probability lies much closer to the expected probability.

Calibration method 2: Sigmoid / Platt scaling

The sigmoid method, also known as Platt scaling, works by transforming the outputs of the classifier using a sigmoid function. The calibrated probabilities are obtained using the following sigmoid function -

Where -

• pcalib is the calibrated probability
• f(x) is the predicted probability from the original classifier f
• A and B are scalar parameters that learned by the algorithm

The parameters A and B are estimated using a maximum likelihood method that optimizes on the same training set as that for the original classifier f.

Sigmoid/Platt scaling was originally invented for SVMs and it works well for other methods as well. This method is less prone to overfitting and should be preferred over Isotonic regression if you have less training data.

## training model using random forest and sigmoid method for calibration
calibrated_sigmoid = CalibratedClassifierCV(
     RandomForestClassifier(random_state=seed), method = 'sigmoid')
calibrated_sigmoid.fit(X_train, y_train)

## getting the output to visualize on test data
prob_true_calib, prob_pred_calib  = calibration_data(y_test, calibrated_sigmoid.predict_proba(X_test)[:,1])

chart_df = pd.DataFrame({
    "actuals": prob_true_calib,
    "predicted": prob_pred_calib,
    "expected": prob_pred_calib
})
fig = px.line(
        data_frame = chart_df, 
        markers = True,
        x = "predicted", 
        y = ["actuals", "expected"], 
        template = "plotly_dark")
fig.update_layout(
        title  = {"text": "Calibration Plot : Sigmoid", "y": 0.95, "x": 0.5},
        xaxis_title="Predicted Probability",
        yaxis_title="Actual Probability",
        font = dict(size=15)
)
fig.show(renderer='notebook')

As we can see above, the model is more calibrated than our default model but not as well as the isotonic method.

Calibration method 3: Train with “Logloss” metric

Many boosting based ensembling classifier models like GradientBoostingClassifier, LightGBM, and Xgboost use log_loss as their default loss function. Training with log_loss helps the output probabilities to be calibrated. To explain why log loss helps in calibration, let’s look at binary log loss function -

Now let’s take an instance of observation where the true outcome is 1. Consider two predictions of 0.7 and 0.9 from the model. If we take the cutoff > 0.5 both of these predictions are correct but the logloss for 0.7 and 0.9 prediction is 0.36 and 0.1 respectively. As we can see log loss penalizes uncertainty in prediction forcing the model to predict close to the actual outcome.

Training with logloss is preferred over fitting a calibration function as model is already calibrated and reduces overhead of training and maintaining an extra model for calibration.

## Fitting the model on training data
gb_model = GradientBoostingClassifier()
gb_model.fit(X_train, y_train)

## getting the output to visualize on test data
prob_true, prob_pred  = calibration_data(y_true = y_test, 
                                          y_pred = gb_model.predict_proba(X_test)[:,1])

chart_df = pd.DataFrame({
    "actuals": prob_true,
    "predicted": prob_pred,
    "expected": prob_pred
})
fig = px.line(
        data_frame = chart_df, 
        markers = True,
        x = "predicted", 
        y = ["actuals", "expected"], 
        template = "plotly_dark")
fig.update_layout(
        title  = {"text": "Calibration Plot: Using boosting method", "y": 0.95, "x": 0.5},
        xaxis_title="Predicted Probability",
        yaxis_title="Actual Probability",
        font = dict(size=15)
)
fig.show(renderer='notebook')

As we can see above, the model seems to be well calibrated. This is usually the case with most of the boosted methods as they use the log_loss function as their default loss function for binary classification.

Conclusion and Practical Guidance

In this article, we discussed what calibration is, in which applications it is important and why, and three different methods for calibration.

We demonstrated isotonic regression which can be used when there is enough training data available and model is not pre-calibrated.

We also demonstrated the Sigmoid method of calibration which works better when there is not enough training data and model is not pre-calibrated.

And finally, we demonstrated the Logloss metric for calibration which is the default in boosting methods and is our preferred method since the generated model is pre-calibrated and doesn’t require training and maintaining an extra model for calibration.

Aayush Agrawal is on LinkedIn.

References

1. Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science. This dataset is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.↩︎
2. Scikit learn calibration documentation↩︎

Model calibration for classification tasks using Python was originally published in Data Science + AI at Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.

Increasingly it’s becoming important for Data professionals to become better at programming and modern programming is centered around Object Oriented programming paradigm. This article helps in explaining some important programming concepts which are mostly language agnostic but we will be using Python in this article.

Object-oriented programming (OOPs) is a programming paradigm that relies on the concept of classes and objects. The basic idea of OOP is to divide a sophisticated program into a number of objects that interact with each other to achieve the desired functionality. There are several advantages of using OOP for data science:

• Encapsulation: OOPs allow you to wrap data and the methods that operate on that data within a single unit (i.e., an object). This makes it easier to organize and manage your code, as well as protect the data from accidental modification.
• Code reusability: OOPs allow you to create reusable code by defining classes that can be used to create multiple objects with the same behavior. This makes it easier to develop and maintain your code, as you can reuse existing code instead of writing new code from scratch.
• Modularity: OOPs allow you to divide your code into smaller, modular units (i.e., objects). This makes it easier to understand and maintain your code, as you can focus on one piece of functionality at a time.
• Inheritance: OOPs allow you to create a new class that is a modified version of an existing class. This allows you to reuse code from the existing class and override or extend it as needed. This can save time and reduce the amount of code you need to write.

Overall, OOP can help data professionals organize and manage their code more effectively, making it easier to develop and maintain data science projects. Let’s dive into the OOPs concept.

What are Objects and Classes?

Classes are the blueprint for defining an Object. While an Object is a collection of data/properties and their behaviors/methods.

For example- Think of a class Bulb that will have a state (On/Off) and methods to turnOn and turnoff the bulb.

class Bulb():
    def __init__(self, onOff=False): self.onOff = onOff    
    def turnOn(self): self.onOff = True
    def turnOff(self): self.onOff = False

Now we can create multiple bulb objects from this Bulb class.

b1 = Bulb(onOff=True)
b2 = Bulb()
print(f"Bulb 1 state is :{b1.onOff}, Bulb 2 state is :{b2.onOff}")

b1 and b2 are objects of the class Bulb. Let’s use the turnOn and turnOff methods to update the bulb properties.

b1.turnOff(); b2.turnOn()
print(f"Bulb 1 state is :{b1.onOff}, Bulb 2 state is :{b2.onOff}")

We can see from the example above, a Bulb object contains the onOff property. Properties are variables that contain information regarding the object of a class and Methods like turnOn and turnOff in our Bulb class are functions that have access to the properties of a class. Methods can accept additional parameters, modify properties and return values.

Class and Instance variables

In Python, properties can be defined in two ways -

• Class Variables — Class variables are shared by all objects of the class. A change in the class variable will change the value of that property in all the objects of the class.
• Instance Variables — Instance variables are unique to each instance or object of the class. A change in instance variable will change the value of the property in that specific object only.

class Employee:
    # Creating a class variable
    companyName = "Microsoft"
    
    def __init__(self, name):
        # creating a instance variable
        self.name = name
    
e1 = Employee('Aayush')
e2 = Employee('John')

print(f'Name :{e1.name}')
print(f'Company Name: {e1.companyName}')
print(f'Name :{e2.name}')
print(f'Company Name: {e2.companyName}')

We can see above, the class variable is defined outside of the initializer and the instance variable is defined inside the initializer.

Employee.companyName = "Amazon"
print(e1.companyName, e2.companyName)

We can see above changing a class variable in the Employee class changes the class variable in all objects of the class. Most of the time we will be using instance variables but knowledge about class variables can come in handy. Let’s look at an interesting use of class variable -

class Employee:
    # Creating a class variable
    companyName = "Microsoft"
    companyEmployees = []
    
    def __init__(self, name):
        # creating a instance variable
        self.name = name
        self.companyEmployees.append(self.name)
    
e1 = Employee('Aayush')
e2 = Employee('John')

print(f'Name :{e1.name}')
print(f'Team Members: {e1.companyEmployees}')
print(f'Name :{e2.name}')
print(f'Company Name: {e2.companyEmployees}')

We can see above, we are saving all objects of the Employee class in companyEmployees which is a list shared by all objects of the class Employee.

Class, Static and Instance methods

In Python classes, we have three types of methods -

• Class Methods — Class methods work with class variables and are accessible using the class name rather than its object.
• Static Methods — Static methods are methods that are usually limited to class only and not their objects. They don’t typically modify or access class and instance variables. They are used as utility functions inside the class and we don’t want the inherited class to modify them.
• Instance Methods — Instance methods are the most used methods and have access to instance variables within the class. They can also take new parameters to perform desired operations.

class Employee:
    # Creating a class variable
    companyName = "Microsoft"
    companyEmployees = []
    
    def __init__(self, name):
        # creating a instance variable
        self.name = name
        self.companyEmployees.append(self.name)
    
    @classmethod
    def getCompanyName(cls): # This is a class method
         return cls.companyName
    
    @staticmethod
    def plusTwo(x): # This is a static method
        return x+2
    
    def getName(self): # This is a instance method
        return self.name
    
e1 = Employee('Aayush')
print(f"Calling class method. Company name is {e1.getCompanyName()}")
print(f"Calling Static method. {e1.plusTwo(2)}")
print(f"Calling instance method. Employee name is {e1.getName()}")

We can see above we use the @classmethod decorator to define the class method. cls is used to refer to the class just as self is used to refer to the object of the class. The class method at least takes one argument cls.

We use @staticmethod decorator to define static class plusTwo. We can see that static methods don’t take any argument like self and cls.

The most commonly used methods are instance methods and they can be defined without a decorator within the class. Just like the class method they take at least one argument which is self by convention.

Access Modifiers

Access modifiers limit access to the variables and functions of a class. There are three types of access modifiers — public, protected, and private.

Public Attributes

Public attributes are those methods and properties which can be accessed anywhere inside and outside of the class. By default, all the member variables and functions are public.

class Employee:
    def __init__(self, name):
        self.name = name ## Public variable
        
    def getName(self): ## Public method
        return self.name

e1 = Employee("Aayush")
print(f"Employee Name: {e1.getName()}")

In the case above, both property name and method getName are public attributes.

Protected Attributes

Protected attributes are similar to public attributes which can be accessed within the class and also available to subclasses. The only difference is the convention, which is to define each protected member with a single underscore “_”.

class Employee:
    def __init__(self, name, project):
        self.name = name ## Public variable
        self._project = project ## Protected variable
        
    def getName(self): ## Public method
        return self.name
    
    def _getProject(self): ## Protected method
        return self._project
    
e1 = Employee("Aayush", "Project Orland")
print(f"Employee Name: {e1.getName()}")
print(f"Project Name: {e1._getProject()}")

In the case above, both property _project and method _getProject are protected attributes.

Private Attributes

Private attributes are accessible within the class but not outside of the class. To define a private attribute, prefix the method or property with the double underscore”_“.

class Employee:
    def __init__(self, name, project, salary):
        self.name = name ## Public variable
        self._project = project ## Protected variable
        self.__salary = salary
        
    def getName(self): ## Public method
        return self.name
    
    def _getProject(self): ## Protected method
        return self._project
    
    def __getSalary(self): ## Protected method
        return self.__salary
    
e1 = Employee("Aayush", "Project Orland", "3500")
print(f"Employee Name: {e1.getName()}")
print(f"Project Name: {e1.__getSalary()}") 

We can see above, __salary property and __getSalary method are both private attributes and when we call them outside of the class they throw an error that the 'Employee' object has no attribute '__getSalary'.

Encapsulation

Encapsulation in OOP refers to binding data and the methods to manipulate that data together in a single unit, that is, class. Encapsulation is usually used to hide the state and representation of the object from the outside. A good use of encapsulation is to make all properties private of a class to prevent direct access from outside and use public methods to let the outside world communicate with the class.

class Employee:
    def __init__(self, name, project, salary):
        self.__name = name ## Public variable
        self.__project = project ## Protected variable
        self.__salary = salary
        
    def getName(self): ## Public method
        return self.__name
    
e1 = Employee("Aayush", "Project Orland", "3500")
print(f"Employee Name: {e1.getName()}")

Encapsulation has several advantages -

• Properties of the class can be hidden from the outside world
• More control over what the outside world can access from the class

A good example of encapsulation would be an access control class based on username and password.

class Auth:
    def __init__(self, userName=None, password=None):
        self.__userName = userName
        self.__password = password
        
    def login(self, userName, password):
        if (self.__userName == userName) and (self.__password == password):
            print (f"Access granted to {userName}")
        else:
            print("Invalid credentials")
            
e1 = Auth("Aayush", "whatever")
e1.login("Aayush", "whatever") ## This will grant access

e1.login("Aayush", "aasdasd") ## This will say invalid creds
e1.__password ## This will raise an error as private properties can't be accessed from outside.

As we can see above __username and __password are protected properties and can only be used by the class to grand or reject access requests.

Inheritance

Inheritance provides a way to create new classes from the existing classes. The new class will inherit all the non-private attributes(properties and methods) from the existing class. The new class can be called a child class and the existing class can be called a parent class.

import math
class Shape:
    def __init__(self, name):
        self.name = name
        
    def getArea(self):
        pass
    
    def printDetails(self):
        print(f"This shape is called {self.name} and area is {self.getArea()}.")
        
class Square(Shape):
    def __init__(self, edge):
        ## calling the constructor from parent class Shape
        Shape.__init__(self, name = "Square")
        self.edge = edge
    
    ## Overiding the getArea function
    def getArea(self):
        return self.edge**2
    
class Circle(Shape):
    def __init__(self, radius):
        ## calling the constructor from parent class Shape
        Shape.__init__(self, name = "Circle")
        self.radius = radius
    ## Overiding the getArea function
    def getArea(self):
        return math.pi * (self.radius**2)
    
obj1 = Square(4)
obj1.printDetails()

obj2 = Circle(3)
obj2.printDetails()

We can see above we defined a parent class Shape and then we inherited it to create a Square and Circle child class. While defining the Square and Circle class we overwrote the getArea function pertinent to the class but we used the printDetails function from the parent class to print details about child classes. The more common example in the machine learning world would be to create your own models in Pytorch where we inherit from nn.Module class to create a new model.

Use of super() Function

super() function comes into play when we implement inheritance. The super() function is used to refer to the parent class without explicitly naming the class. super() function can be used to access parent class properties, calling the parent class, and can be used as initializers. Let’s look at the example above and see how we can modify the Square class to use super() function.

class Shape:
    maxArea = 100
    def __init__(self, name): self.name = name
    def getArea(self): pass
    def printDetails(self): 
        print(f"This shape is called {self.name} and area is {self.getArea()}.")

class Square(Shape):
    maxArea = 50
    def __init__(self, edge):
        super().__init__(name = "Square") ## Initializing parent class
        self.edge = edge
    
    def getName(self):
        return super().maxArea
    
    def getArea(self):
        return self.edge**2
    
    def printDetails(self):
        super().printDetails() ## Calling a parent class function
        print(f"Max are from Shape class: {super().maxArea}") ## Accessing parent class property
        print(f"Max are from Square class: {self.maxArea}")

obj1 = Square(4)
obj1.getName()
obj1.printDetails()

As we can see in the example above we have used -

• super().__init__ to initialize the parent Shape class
• super().printDetails() function to use a method from parent class
• super().maxArea to access a property of a parent class

There are many advantages of inheritance -

1. Reusability — Inheritance makes the code reusable. Common methods and properties can be stored in a parent class and child classes can inherit these methods.
2. Modification — Code modification becomes easier if we use inheritance, if we want to make a change in the base class function it will be propagated to the child classes.
3. Extensibility — We can derive new classes from the old ones by keeping things we need in the derived class.

Polymorphism

Polymorphism refers to the same object exhibiting different forms and behaviors. For example consider our shape class which could be a square, rectangle, polygon, etc. Instead of writing multiple functions to get the area of these shapes, we can use a common function like getArea() and implement this function in the derived class.

import math
class Shape:
    def __init__(self, name):
        self.name = name
        
    def getArea(self):
        pass
    
    def printDetails(self):
        print(f"This shape is called {self.name} and area is {self.getArea()}.")
        
class Square(Shape):
    def __init__(self, edge):
        ## calling the constructor from parent class Shape
        Shape.__init__(self, name = "Square")
        self.edge = edge
    
    ## Overiding the getArea function
    def getArea(self):
        return self.edge**2
    
class Circle(Shape):
    def __init__(self, radius):
        ## calling the constructor from parent class Shape
        Shape.__init__(self, name = "Circle")
        self.radius = radius
    ## Overiding the getArea function
    def getArea(self):
        return math.pi * (self.radius**2)
    
obj1 = Square(4)
print(f"Area of this {obj1.name} is {obj1.getArea()}")

obj2 = Circle(3)
print(f"Area of this {obj2.name} is {obj2.getArea()}")

As we can see above there is a pre-defined dummy method called getArea in the Shape class. We override this method in the Square and Circle class. This technique is called method overriding. The advantage of method overriding is that the derived class can write its own specific implementation based on the requirement while using the same function name.

Abstract base classes

Abstract base classes define a set of methods and properties that a class must implement in order to inherit the parent class. This is a useful technique to enforce that certain functions within the derived class must exist. To define an abstract base class, we use the abc module. The abstract base class inherits from the built-in ABC class and we use the decorator @abstractmethod to declare an abstract method.

from abc import ABC, abstractmethod

class Shape(ABC):
    def __init__(self, name):
        self.name = name
    
    @abstractmethod
    def getArea(self):
        pass
    
    def printDetails(self):
        print(f"This shape is called {self.name} and area is {self.getArea()}.")
        
class Square(Shape):
    def __init__(self, edge):
        ## calling the constructor from parent class Shape
        Shape.__init__(self, name = "Square")
        self.edge = edge
    
obj1 = Square(4)
print(f"Area of this {obj1.name} is {obj1.getArea()}")

We can see above that we have created a Shape class from the ABC class which has an abstract method getArea. Since our child class Square didn’t have getArea implemented we get an error instantiating this class.

from abc import ABC, abstractmethod

class Shape(ABC):
    def __init__(self, name):
        self.name = name
    
    @abstractmethod
    def getArea(self):
        pass
    
    def printDetails(self):
        print(f"This shape is called {self.name} and area is {self.getArea()}.")
        
class Square(Shape):
    def __init__(self, edge):
        ## calling the constructor from parent class Shape
        Shape.__init__(self, name = "Square")
        self.edge = edge
    
    def getArea(self): return self.edge**2
    
obj1 = Square(4)
print(f"Area of this {obj1.name} is {obj1.getArea()}")

We can see above, once we implemented the getArea method, the code runs fine.

Abstract base classes serve as a blueprint for derived classes to implement methods that are required to run the function appropriately.

8 Conclusion

In this article, we learned about what is object-oriented programming and key concepts using Python. A good understanding of these concepts will lay a solid foundation for any software professional to write and understand python code better.

I hope you enjoyed reading it. If there is any feedback on the code or just the blog post, feel free to comment below or reach out on LinkedIn.