What AI agents change about Jupyter notebooks, and what the IDE vs. CLI debate misses about data work.

Here is a Jupyter session:

In [1]: import pandas as pd
In [2]: sales = pd.read_csv("sales.csv")
In [3]: sales.groupby("region")["revenue"].sum().plot.bar()
Out[3]: <Axes: xlabel='region'>

And here is the same work with an AI agent:

You:   Load sales.csv and chart revenue by region.
Agent: Done. West leads, but 14% of rows have no region.
       Should I drop them or group them as "Unknown"?

The rhythm is identical: a prompt, a response, and state that carries into the next turn. The difference shows up the next morning. You can re-run the first. You can only re-read the second.

The resemblance is no accident. Notebooks have always been conversations with a computer. Mathematica introduced its notebook interface in 1988. IPython brought numbered In and Out prompts to Python in 2001 and a browser-based notebook in 2011, and Project Jupyter, named for Julia, Python, and R, grew out of it in 2014. From its early days, Jupyter’s goal was the computational narrative: people make sense of the world through stories, so code and results have to be woven into a story for a particular audience before they’re truly useful [1].

What’s new is a third participant. Agents in Colab, Databricks, Deepnote, and Hex now carry much of that conversation on their own, writing cells, running them, and reacting to what comes back. And notebooks aren’t fading as this happens: GitHub repositories containing Jupyter notebooks grew about 75% in a year, with notebook usage nearly doubling in AI-tagged projects [2].

Most commentary on AI coding tools asks where you should steer an agent from: the IDE or the terminal. It’s a fair question, but data work raises a more consequential one. When the agent is done, what do you keep? This post argues for a simple division of labor. The conversation, whether it happens in a chat panel or a terminal, is the scratchpad. The notebook is the commit.

The debate so far: steering from the editor or the shell

Here’s the IDE vs. CLI debate in one table:

In shorthand, IDE AI is coding assistance and CLI AI is workflow automation. One augments the developer; the other acts on the developer’s behalf.

Much of the choice comes down to taste. IDE agents inherit rich visual review, with inline diffs, navigation, and a debugger, while terminal agents trade that visual feedback for speed and scriptability [3]. What matters more than the environment is verification: agents have made changes cheap to produce but no cheaper to check [4]. One useful way to frame the split is flow versus intent, and for UI-heavy work the IDE keeps an edge for a mundane reason: it’s easier to drag a screenshot into the agent [5].

That screenshot detail points at something the debate usually skips: the more visual the work, the more your tools need to show you things. Data work is about as visual as work gets.

The same split is already happening inside the notebook

Look closely at today’s notebook products and you’ll find the IDE vs. CLI divide replayed in miniature.

The IDE-style layer is cell-level help. Genie Code, the successor to Databricks Assistant, offers autocomplete, error diagnosis, and suggested fixes you can accept and run [6][7]. In Colab, you can describe a change in plain English and have it locate and rewrite the relevant code [8]. You stay in charge of each edit, as you would with a pair programmer.

The CLI-style layer is agent mode. Give Colab’s Data Science Agent a dataset and a goal, and it builds a working notebook [9]. Genie Code’s agent plans an approach, looks up relevant tables, runs cells, reads their outputs, and fixes its own errors [10]. Hex and Deepnote ship notebook agents of their own [11][12].

Then there’s the literal terminal. Google’s Colab MCP Server lets agents like Claude Code and Gemini CLI create and run Colab notebooks [13]. The Hex CLI and Databricks’ installable agent skills bring those platforms to coding agents such as Claude Code, Codex, and Cursor [14][15]. Marimo pair gives terminal agents a live notebook kernel [16], and Jupyter AI 3 connects Claude, Codex, Gemini, and others to JupyterLab through the Agent Client Protocol [17].

Adapted to the notebook, the comparison looks like this:

Cell assistants are coding assistance. Notebook agents are analysis automation.

Notice which way the defaults are moving. Genie Code has a Chat mode and an Agent mode, and Agent is the default [18]. Jupyter AI 3.0 moved its magic commands out of the default install into an optional package, on the reasoning that agents can now edit notebooks directly [19]. In the notebook world, autonomy is becoming the starting point.

Control vs. autonomy: why data raises the stakes

In an IDE, the AI behaves like a pair programmer and you approve each change. In a terminal, it behaves more like automation. Developers who value step-by-step precision gravitate to the first, while those facing large codebases, repetitive operations, or infrastructure work gravitate to the second.

Data work complicates the trade-off because of how things fail. When an autonomous agent breaks application code, something usually complains: a test, a type checker, a build. When an autonomous agent botches an analysis, nothing complains. A join that duplicates rows, a filter applied in the wrong order, or a train/test split that leaks information all produce a clean chart and a plausible number.

The platforms do add brakes. Genie Code asks for approval before running code, though you can switch it to auto-approve [10], and Jupyter AI’s agents request permission before writing files or running commands [17]. But approving an execution isn’t the same as validating a result. And notebooks had a reproducibility problem long before agents showed up. In a 2019 analysis of roughly 1.4 million GitHub notebooks, only about 24% ran without errors and about 4% reproduced their original results, with hidden state and out-of-order execution among the leading causes [20]. An agent can manufacture hidden state far faster than a person can.

So the real question isn’t whether agents should act autonomously in notebooks. They already do. It’s what has to be true of their work before you believe it.

Chat is the scratchpad, the notebook is the commit

Good notebook practice has long held that you should record your explorations, dead ends included, because a notebook captures code automatically but not the reasoning behind it [21]. That advice has always pulled against the notebook’s other job: a detailed lab record eventually has to be trimmed into a concise story for a wider audience.

Agents resolve that tension almost by accident, because agent-driven work naturally produces two artifacts. The conversation holds the questions, the instructions, the false starts, and the agent’s reasoning. The notebook holds what survived. That suggests a clean division of labor: treat the conversation as the scratchpad and the notebook as the commit.

The tools are already converging on that shape. Marimo pair separates the two by design: a terminal agent experiments freely in a scratchpad attached to the live kernel, and only the steps it chooses to keep, and that Marimo accepts, become cells in a notebook someone can re-run and review later [16]. A conversation in Hex’s Threads interface can be saved as a notebook project so the data team can extend or audit it [22], and the Hex CLI offers the terminal version of the same move: build the analysis locally with the coding agent you prefer, then publish it to Hex as a versioned project [14]. Through Colab’s MCP Server, agents can produce complete, runnable notebooks rather than loose code snippets [23]. Genie Code’s full-page experience starts from the chat and opens notebooks and files beside it as tabs [24]. And Jupyter AI stores chats as ordinary files in your workspace, so the scratchpad can sit right next to the commit [25].

Why should the notebook, and not the transcript, be the artifact of record? A transcript tells you what the agent said. A notebook can show you, on demand, what the data does. That only holds if the commit is honest. When a terminal agent edits an .ipynb file directly, as Claude Code does with its cell-level notebook tool, it can read the outputs saved in the file [26] but can’t tell whether those outputs are stale, because it isn’t looking at a running kernel [27]. Committing a notebook whose outputs nobody regenerated is like merging code on the strength of last week’s green checkmark. Kernel-connected agents narrow that gap, but the only real proof is a clean run from the top.

Commit size matters too. Agents do far better work when they build a notebook one step at a time, loading data, checking it, and fixing problems as they go, than when they write the whole thing at once and run it at the end, which tends to produce something tidy-looking that doesn’t hold up [28]. Small commits beat big ones, for agents as much as for people.

What a good notebook commit looks like

If the notebook is the commit, it deserves the standards we already apply to commits. Several are long-standing notebook best practices [21]; the agent era just makes them non-negotiable.

A minimal setup takes care of the mechanical parts:

# Keep outputs out of Git diffs (publish executed copies separately if reviewers need them)
nbstripout --install

# Give agents and reviewers a plain-text view of each notebook
jupytext --set-formats ipynb,py:percent analysis.ipynb

# CI gate: every committed notebook must run top to bottom in a fresh kernel
pytest --nbmake notebooks/*.ipynb

If rendered outputs matter to your reviewers, keep them, just not inside the source diff. Deepnote’s VS Code extension, for example, can write outputs to separate snapshot files instead of the main notebook [30].

Where the terminal fits, and whether the skeptic is right

There’s a blunt counterargument to the terminal enthusiasm: AI companies only moved to the CLI because they don’t have an IDE of their own and don’t want to depend on Microsoft or anyone else, and a strong IDE like IntelliJ still beats the terminal for real coding. The first half probably contains some truth, since platform independence is a real incentive. And for building a large library, a mature IDE is formidable.

But look at what data platforms actually use the terminal for. It isn’t replacing the notebook. It moves work between the conversation, the commit, and production: building an analysis locally and publishing it to Hex, pushing agent compute off your laptop and into Colab, installing Databricks skills into the agents you already run. Jupyter AI even stopped focusing on its own agent, Jupyternaut, in order to connect the agents developers already use [31]. In data work, the terminal is the plumbing, not the destination.

As for the IDE, the bottleneck in most analysis isn’t typing code, which agents now do quickly. It’s judging outputs. Is that distribution plausible? Did that join behave? Does this chart say what we think it says? That judgment needs a canvas that shows code, data, and results together, no matter which tool wrote the code.

A hybrid workflow: converse, commit, ship

The familiar three-phase model of terminal, then IDE, then finalization adapts neatly to data work.

Phase 1: Converse. Start in a chat panel or terminal session connected to live data. Ask questions, let the agent explore, and chase dead ends while they’re cheap. Look at every chart as it appears and push back early. Nothing here needs to be tidy. This is the scratchpad.

Phase 2: Commit. Promote what survived into a clean notebook, one meaningful step per cell, with Markdown explaining why each step exists. Add assertions for the data expectations you relied on. Restart the kernel, run everything from the top, and read the outputs yourself, not just the code. Keep the conversation file alongside it if it explains decisions a future reader will wonder about.

Phase 3: Ship. Hand the committed notebook to a terminal agent to extract reusable logic into modules, add tests, parameterize runs, and wire everything into CI or a scheduler. Review those diffs in your IDE, then commit, test, and push with full human oversight.

Final thoughts

For nearly four decades, the notebook has been a conversation between a person and a kernel, recorded as it happens. Agents now join that conversation as a third voice, and a very fast one.

That makes it tempting to treat the chat as the new notebook. It isn’t. Chats are wonderful for thinking out loud, but you can’t re-run a transcript. In the agent era, the notebook’s job gets narrower and more important: hold only what survived, in a form anyone can execute and question.

Chat tools help you explore faster. Terminal and IDE tools help you ship. The notebook is where exploration becomes something you can re-run, review, and trust.

The future of data tooling may not be IDE vs. CLI, or even chat vs. notebook. It may be a conversation that ends in a commit.

References

1. Fernando Pérez and Brian Granger, Project Jupyter: Computational Narratives as the Engine of Collaborative Data Science (2015)
2. The New Stack, JetBrains kills Kotlin Notebook months after Microsoft’s Polyglot exit. But Jupyter is doing just fine. (June 2026)
3. Sonar, CLI vs. IDE: How to choose the best environment (July 2026)
4. The New Stack, CLI or IDE? Build in verification first (August 2026, sponsored by Sonar)
5. Hacker News, Ask HN: Do you prefer AI coding in an IDE or CLI? And why?
6. Databricks docs, Get coding help from Genie Code
7. Databricks docs, Databricks Assistant FAQ
8. Google Developers Blog, The new AI-first Google Colab is now available to everyone (June 2025)
9. Google Developers Blog, Data Science Agent in Colab (March 2025)
10. Azure Databricks docs, Use Genie Code for data science
11. Hex, Introducing the Notebook Agent
12. Deepnote, Deepnote on GitHub
13. Google Developers Blog, Announcing the Colab MCP Server (March 2026)
14. Hex, Introducing the Hex CLI (April 2026)
15. Databricks docs, Agent skills for AI coding assistants
16. marimo, Implementing marimo pair: letting agents use notebooks as a tool (July 2026)
17. Jupyter AI, PyPI project page
18. Databricks docs, Use Genie Code
19. Jupyter AI, Release notes
20. Pimentel et al., A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks (MSR 2019)
21. Rule et al., Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks (PLOS Computational Biology, 2019)
22. Hex, Fall 2025 launch
23. InfoQ, Coverage of the Colab MCP Server (April 2026)
24. Databricks docs, Genie Code
25. Jupyter AI docs, Getting started
26. Claude Code docs, Tools reference
27. Kanaries, Can Claude Code analyze Jupyter notebooks for data science?
28. Talk Python To Me, Episode #555: marimo pair (July 2026)
29. Google, Customize your Gemini agent in Colab (April 2026)
30. Deepnote, Snapshot mode in the VS Code extension (January 2026)
31. Jupyter AI, v3.0 release plan

Paste Mermaid source, see the diagram render as you type, and click Share.
You get a permanent page plus a direct `.svg` URL that works in Markdown:

```md
```

There is no account to create. Pastes can be public or unlisted, and public is the default. The shared page also lets you copy the source, copy an embed, or download the SVG.

Why I built it

Mermaid is convenient because diagrams live as text next to the ideas they
describe. The awkward part is sharing one outside the repository or tool where it was written.

I wanted a workflow as small as this:

Example: https://mermaidbin.com/LNze5LO5WP1tqwdB-e4UUG

A screenshot loses the source. A Mermaid code block depends on the destination supporting Mermaid. An online editor is useful, but often the thing I actually want is a URL I can paste into a README, issue, chat, or document.

MermaidBin is an attempt to make that last step boring.

The interesting implementation detail

The browser and the server solve different parts of the problem.

The browser renders Mermaid immediately. This gives fast feedback while typing and means sharing does not wait for a headless browser. When a paste is created, the API stores its source in PostgreSQL and returns the permanent page.

A separate worker then renders the durable SVG using Mermaid CLI and Chromium.
The worker sanitizes the output and stores it with the paste. The public API
process never needs Chromium, and the stable `.svg` URL can be cached and embedded independently of the JavaScript viewer.

From polling to LISTEN/NOTIFY

The first version of the worker repeatedly checked PostgreSQL for pending
renders. It worked, but the user-visible result was a “Preparing SVG” state for up to the polling interval. Reducing the interval made the UI faster at the cost of constant database queries.

The current design uses PostgreSQL `LISTEN`/`NOTIFY`. The transaction that creates a paste also sends a notification. PostgreSQL delivers it after commit, and the worker wakes immediately.

I did not treat notifications as a durable queue. They can be missed while a
worker is disconnected. PostgreSQL remains the source of truth, and the worker still performs a reconciliation scan every 30 seconds. It also reclaims jobs that have been stuck in `rendering` for five minutes.

That combination gave me the behavior I wanted:

- Low latency in the normal case;
- No 250 ms database polling loop;
- Recovery after a worker restart or broken LISTEN connection;
- A simple path to multiple workers with `FOR UPDATE SKIP LOCKED`.

At much higher volume I would probably use a dedicated queue, but for this product PostgreSQL is already present and the simpler operational model is valuable.

Rendering untrusted diagrams

Running Chromium on user-controlled source deserves care. The renderer is isolated from the web process and runs in a non-root container with a read-only root filesystem, dropped capabilities, a default seccomp profile, and only a small writable temporary volume.

Before rendering, it rejects external resource URLs. Mermaid runs in strict
mode with HTML labels disabled. The resulting SVG goes through `DOMPurify`, with scripts, event handlers, and `foreignObject` removed. Rendering has a timeout, and output is capped at 2 MiB.

This is defense in depth rather than a claim that rendering arbitrary input is risk-free. Keeping Chromium outside the API container limits what a renderer failure can affect and makes that boundary easier to reason about.

A container bug that only appeared in the worker

One of the more useful lessons came from Chromium itself. SVG rendering worked locally but failed in the hardened container with:

```text
chrome_crashpad_handler: — database is required
```

The worker has a read-only root filesystem. Chromium tried to initialize
Crashpad before it had a writable profile location and exited before rendering.
The fix was to put `HOME`, the Chromium profile, config, cache, and Crashpad state inside the worker’s per-render temporary directory, and to disable crash reporting. I also disabled `/dev/shm` usage because its small container default is another common source of Chromium failures.

The test suite now runs the real PostgreSQL → worker → Chromium → SVG path in Docker with the same read-only-root constraint as production. Mocking the renderer would never have found this.

What is running?

MermaidBin currently runs on a regional Google Kubernetes Engine:

- two small Node/Express API replicas;
- one isolated SVG renderer worker;
- Cloud SQL for PostgreSQL;
- Cloud SQL Auth Proxy sidecars;
- Cloud CDN and Cloud Armor in front of the service;
- Prometheus-compatible API and worker metrics.

SVG is stored in PostgreSQL for now. That keeps render state and output together and is simple at the current scale. PostgreSQL can handle values much larger than my application limits, but every diagram is capped at 200 KiB of source and 2 MiB of SVG. If storage volume becomes meaningful, moving SVG bodies to object storage is an obvious next step.

Tradeoffs and unfinished parts

This is a deliberately small product, and there are real limitations:

- “unlisted” means link-only, not private;
- there are no accounts or diagram revisions;
- expired rows are currently cleaned up lazily;
- the schema is bootstrapped by the app rather than versioned migrations;
- the fallback application rate limiter is per replica;
- the worker needs richer retry and dead-letter controls before large scale;
- browser-level visual regression coverage is still on the list.

I would rather expose those boundaries than pretend a small service is a fully mature platform.

Try it

MermaidBin is live at mermaidbin.com.

I would especially value feedback on:

- whether the paste → share → embed workflow feels obvious;
- whether public-by-default is the right default;
- which Mermaid diagram types fail or render surprisingly;
- whether the direct SVG URL is useful in the tools where you write.

If you try it, please avoid pasting secrets or sensitive diagrams. Public
pastes are listed, and unlisted links are still accessible to anyone who has
the URL.

Followed GKE’s Serve Gemma with GPUs and vLLM tutorial for `google/gemma-4-E2B-it`, but deployed on a single `nvidia-l4` GPU (`g2-standard-8`) instead of the tutorial’s `nvidia-rtx-pro-6000`.

Symptom

Chat completions took 20–25+ seconds even for short prompts:

$ curl http://127.0.0.1:8000/v1/chat/completions -X POST -H "Content-Type: application/json" -d '{
 "model": "google/gemma-4-E2B-it",
 "messages": [{"role": "user", "content": "Why is the sky blue?"}],
 "chat_template_kwargs": {"enable_thinking": true},
 "skip_special_tokens": false
}' | python -m json.tool
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 5902 100 5643 100 259 239 10 0:00:25 0:00:23 0:00:02 1237

`kubectl logs -f -l app=gemma-server` while that request was running showed the engine steadily reporting only ~45–50 tokens/s generation throughput, with a single request in flight at a time — no queueing or batching contention:

(APIServer pid=1) INFO 07–17 05:33:47 [async_llm.py:420] Added request chatcmpl-bfed389dac8894c3–8a84daa5.
(APIServer pid=1) INFO 07–17 05:33:48 [loggers.py:259] Engine 000: Avg prompt throughput: 2.2 tokens/s, Avg generation throughput: 16.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 07–17 05:33:58 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 49.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 07–17 05:34:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 49.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO: 127.0.0.1:33494 - "POST /v1/chat/completions HTTP/1.1" 200 OK

So the delay wasn’t a stall — it was genuinely slow decode. The response’s `usage` block confirmed the math: 1142 completion tokens at ~49 tok/s ≈ the observed ~23s.

"usage": {
 "prompt_tokens": 22,
 "total_tokens": 1164,
 "completion_tokens": 1142,
 "prompt_tokens_details": null
}

Root cause

Two compounding factors:

1. chat_template_kwargs: {“enable_thinking”: true}` was roughly doubling output length by generating a full reasoning trace (visible in the response’s `”reasoning”` field) before the visible answer. Turning it off cut completion tokens from 1142 to 569 for the same prompt.

2. Gemma-4-E2B-it’s “E2B” name is misleading. Despite ~2B *effective* compute, the actual weight table is much bigger — confirmed straight from the vLLM startup log:

(EngineCore pid=73) INFO 07–11 23:30:56 [gpu_model_runner.py:4601] Model loading took 9.79 GiB memory and 118.751103 seconds
(EngineCore pid=73) INFO 07–11 23:32:41 [gpu_worker.py:456] Available KV cache memory: 9.67 GiB
(EngineCore pid=73) INFO 07–11 23:32:41 [kv_cache_utils.py:1316] GPU KV cache size: 337,968 tokens

9.79 GiB in bf16 is what you’d expect from a much larger model — this is a MatFormer/elastic architecture with large per-layer embedding tables. Autoregressive decode is memory-bandwidth-bound, and the L4’s ~300 GB/s bandwidth is the real ceiling here — nowhere near what an RTX PRO 6000 or A100 offers. This is why the tutorial’s stock bf16 config didn’t need quantization: it was written for faster hardware.

Confirming the node’s actual GPU:

$ kubectl get node gke-gemma4-gpu-pool-1-e2a2b531–6jdu -o jsonpath='{.metadata.labels}' | python3 -m json.tool | grep -i accelerator
 "cloud.google.com/gke-accelerator": "nvidia-l4",

Fixes applied

- `enable_thinking: false` (client-side) — cuts output length ~2x when the reasoning trace isn’t needed.
- `‑‑quantization=fp8` — native FP8 tensor cores on L4 (Ada Lovelace); shrank weights and raised throughput ~47 → ~74 tok/s (**+57%**) with no observed quality loss.
- `‑‑enable-auto-tool-choice` — was missing from the initial deploy; without it, `‑‑tool-call-parser=gemma4` is configured but never engaged, so tool/function calling silently doesn’t activate.
- `‑‑max-model-len=32768` (was `8192`) — matches the tutorial’s intended context window; the previous cap was silently truncating `max_tokens`.
- `‑‑gpu-memory-utilization=0.95` (was `0.90`) — grows the KV cache pool to safely support the larger context at the existing `‑‑max-num-seqs=16`, aided by memory freed from FP8.

Applying FP8 quantization

Patched directly onto the live Deployment to test quickly, before folding it into the checked-in manifest:

kubectl patch deployment vllm-gemma-deployment - type=json \
 -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": " - quantization=fp8"}]'

Startup logs confirmed it took effect cleanly, with no errors:

(APIServer pid=1) INFO 07–17 05:55:45 [utils.py:299] non-default args: {…, 'quantization': 'fp8', …}
(EngineCore pid=174) INFO 07–17 05:57:34 [__init__.py:255] Selected CutlassFP8ScaledMMLinearKernel for Fp8OnlineLinearMethod
(EngineCore pid=174) INFO 07–17 05:59:34 [gpu_model_runner.py:4601] Model loading took 8.02 GiB memory and 119.515078 seconds

Weight memory dropped from 9.79 GiB → 8.02 GiB (not a full 2x, since large embedding tables stay in bf16 and only linear/attention projections get quantized).

Folding all changes into the deployment manifest

Final diff against the manifest (`vllm-4-e2b-it.yaml`), reviewed with `kubectl diff -f` before applying:

diff

- - enable-log-requests
 - - enable-chunked-prefill
 - - enable-prefix-caching
+ - - enable-auto-tool-choice
 - - generation-config=auto
 - - tool-call-parser=gemma4
 - - dtype=bfloat16
 - - max-num-seqs=16
- - - max-model-len=8192
- - - gpu-memory-utilization=0.90
+ - - max-model-len=32768
+ - - gpu-memory-utilization=0.95
 - - reasoning-parser=gemma4
 - - trust-remote-code
 - - quantization=fp8

kubectl apply -f vllm-4-e2b-it.yaml
kubectl rollout status deployment/vllm-gemma-deployment - timeout=300s

deployment.apps/vllm-gemma-deployment configured
service/llm-service unchanged
deployment "vllm-gemma-deployment" successfully rolled out

Result

Benchmarked with the same prompt before and after, `enable_thinking: false` in both cases:

- Original (bf16, `‑‑max-model-len=8192`) — 569 completion tokens in ~12s ≈ **47 tok/s**
- + FP8 quantization — 592 completion tokens in ~8s ≈ **74 tok/s**
- + all changes (FP8, 32K context, tool-choice), 2nd request / steady state — 517 completion tokens in ~7s ≈ **74 tok/s**

~57% faster generation, correct tool-calling behavior, and 4x the usable context window — all on cheaper L4 hardware than the tutorial assumes, no GPU upgrade needed.

Takeaway

vLLM’s per-step `Avg generation throughput` log line is the single most useful diagnostic — it separates “the model is slow” from “the request is queued/stuck.” And model name conventions (like “E2B”) can undersell actual memory footprint; check `Model loading took X GiB` in the startup logs before assuming a small model should be fast on modest hardware.

A minimal CUDA kernel. The `__global__` keyword marks a function that runs on the GPU but is launched from the CPU. Each thread reads its own index from `threadIdx.x` and prints it.

#include <stdio.h>

__global__ void roll_call() {
 const int threadIndex = threadIdx.x;
 printf("Thread %d here!\n", threadIndex);
}

Launching the kernel with 1 thread block containing 10 threads. cudaDeviceSynchronize() makes the CPU wait until the GPU finishes — without it, the program could exit before any output appears.

int main() {
 roll_call<<<1, 10>>>();
 cudaDeviceSynchronize();
 return 0;
}

Compile with NVIDIA’s `nvcc` compiler:

$ nvcc roll_call.cu -o roll_call.o

Each of the 10 threads announces itself:

$ ./roll_call.o
Thread 0 here!
Thread 1 here!
Thread 2 here!
Thread 3 here!
Thread 4 here!
Thread 5 here!
Thread 6 here!
Thread 7 here!
Thread 8 here!
Thread 9 here!

Reading and Writing Data in CUDA Kernels

A kernel that increments one array element per thread. Each thread uses its index to pick “its” element, so all 10 elements are updated in parallel.

__global__ void array_increment(int* in) {
 const int threadIndex = threadIdx.x;
 in[threadIndex] = in[threadIndex] + 1;
}

First, allocate and initialize the array on the host (CPU) side with plain `malloc`:

const int arraySize = 10;
// Allocate host memory for the input array.
//
// The amount of memory allocated is equal to
// the length of the array times the size of
// an integer, or 10 * 4 = 40 bytes.
int* array = (int*)malloc(arraySize * sizeof(int));
// Initialize the input array with values 0, 10, 20, 30, …
for (int i = 0; i < arraySize; i++) {
 array[i] = i * 10;
}

The GPU can’t read host memory directly, so we allocate a matching buffer in GPU (device) memory with cudaMalloc. The `d_` prefix is a common convention for device pointers.

// Allocate GPU memory for the input array
int* d_array;
cudaMalloc(&d_array, arraySize * sizeof(int));

Copy the data from host to device. The last argument tells CUDA which direction the copy goes.

cudaMemcpy(
 d_array, // The destination memory address
 array, // The source memory address
 arraySize * sizeof(int), // The number of bytes to copy
 cudaMemcpyHostToDevice // The direction of the copy
);

Run the kernel on the device data, then copy the results back to the host:

array_increment<<<1, arraySize>>>(d_array);
cudaMemcpy(
 array, // The destination memory address
 d_array, // The source memory address
 arraySize * sizeof(int), // The number of bytes to copy
 cudaMemcpyDeviceToHost // The direction of the copy
);

The full program: allocate → copy to GPU → run kernel → copy back → free. This allocate/copy/compute/copy-back pattern is the backbone of almost every CUDA program.

void printArray(int* array, int arraySize) {
 printf("[");
 for (int i = 0; i < arraySize; i++) {
   printf("%d", array[i]);
   if (i < arraySize - 1) {
    printf(", ");
   }
 }
 printf("]\n");
}

int main() {
 const int arraySize = 10;
 // Allocate host memory for the input array
 int* array = (int*)malloc(arraySize * sizeof(int));
 // Initialize the input array
 for (int i = 0; i < arraySize; i++) {
  array[i] = i * 10;
 }
 printf("Before: ");
 printArray(array, arraySize);
// Allocate GPU memory for the input array
 int* d_array;
 cudaMalloc((void**)&d_array, arraySize * sizeof(int));
// Copy the input array from host memory to GPU memory
 cudaMemcpy(d_array, array, arraySize * sizeof(int), cudaMemcpyHostToDevice);
array_increment<<<1, arraySize>>>(d_array);
// Copy the result array from GPU memory back to host memory
 cudaMemcpy(array, d_array, arraySize * sizeof(int), cudaMemcpyDeviceToHost);
 printf("After: ");
 printArray(array, arraySize);
// Free the host and GPU memory
 free(array);
 cudaFree(d_array);
 return 0;
}

Every element was incremented by 1 — each by a different thread, all at once:

$ ./array_increment
Before: [0, 10, 20, 30, 40, 50, 60, 70, 80, 90]
After: [1, 11, 21, 31, 41, 51, 61, 71, 81, 91]

Thread Blocks

Thread blocks can be 2D (or 3D), which maps naturally onto matrices. Here each thread moves one element to its transposed position, computing a flat input index from `(x, y)` and a flat output index from the swapped `(y, x)`.

__global__ void transpose_matrix(int* in, int* out) {
 // Select an element from the input matrix by using the
 // threadIdx x and y values. blockDim.x contains the number
 // of rows in the 2D Thread Block.
 const int threadIndex = threadIdx.x + threadIdx.y * blockDim.x;
// Select the corresponding position in the output matrix.
 // blockDim.y contains the number of columns in the 2D Thread Block.
 const int outIdx = threadIdx.y + threadIdx.x * blockDim.y;
out[outIdx] = in[threadIndex];
}

A `dim3` struct defines the 2D block shape — one thread per matrix element:

int main() {
 // Allocate host & GPU memory; copy the input array to GPU memory
 // …
dim3 numThreadsPerBlock(rows, cols);
 transpose_matrix<<<1, numThreadsPerBlock>>>(d_in_matrix, d_out_matrix);
// Release host and GPU memory
 // …
}

Running Multiple Thread Blocks

A single thread block is capped at 1024 threads. Asking for 1025 fails — and because kernel launches are asynchronous, you need `cudaGetLastError()` to actually see the error.

int main() {
 roll_call_kernel<<<1, 1025>>>();
 cudaError_t err = cudaGetLastError();
 if (err != cudaSuccess) {
  printf("Kernel launch error: %s\n", cudaGetErrorString(err));
  exit(-1);
 }
 cudaDeviceSynchronize();
 return 0;
}

Kernel launch error: invalid configuration argument

Streaming Multiprocessors

Which physical core is running my thread? This kernel uses a line of inline PTX assembly to read the special `%smid` register — the ID of the Streaming Multiprocessor (SM) executing the thread.

__global__ void sm_roll_call() {
 const int threadIndex = threadIdx.x;
uint streamingMultiprocessorId;
 asm("mov.u32 %0, %smid;" : "=r"(streamingMultiprocessorId));
printf("Thread %d running on SM %d!\n", threadIndex, streamingMultiprocessorId);
}
int main() {
 sm_roll_call<<<1, 5>>>();
 cudaDeviceSynchronize();
 return 0;
}

With one block, all threads land on the same SM — a block never spans multiple SMs:

$ ./sm_roll_call
Thread 0 running on SM 0!
Thread 1 running on SM 0!
Thread 2 running on SM 0!
Thread 3 running on SM 0!
Thread 4 running on SM 0!

Launching 4 blocks instead:

int main() {
 // Launch 4 thread blocks with 2 threads per block
 sm_roll_call<<<4, 2>>>();
 cudaDeviceSynchronize();
 return 0;
}

Now the blocks are distributed across different SMs and run truly in parallel:

Thread 0 running on SM 6!
Thread 1 running on SM 6!
Thread 0 running on SM 4!
Thread 1 running on SM 4!
Thread 0 running on SM 0!
Thread 1 running on SM 0!
Thread 0 running on SM 2!
Thread 1 running on SM 2!

Warps

Threads don’t execute individually — the hardware groups them into *warps* of 32 that execute in lockstep. This kernel also reads the `%warpid` and `%laneid` registers to show each thread’s warp and its position (lane) within it.

__global__ void warp_roll_call() {
 const int threadIndex = threadIdx.x;
uint streamingMultiprocessorId;
 asm("mov.u32 %0, %smid;" : "=r"(streamingMultiprocessorId));
uint warpId;
 asm volatile("mov.u32 %0, %warpid;" : "=r"(warpId));
uint laneId;
 asm volatile("mov.u32 %0, %laneid;" : "=r"(laneId));
printf("SM: %d | Warp: %d | Lane: %d | Thread %d - Here!\n",
 streamingMultiprocessorId, warpId, laneId, threadIndex);
}

With 2 threads per block, each block occupies a single (mostly empty) warp:

SM: 6 | Warp: 0 | Lane: 0 | Thread 0 - Here!
SM: 6 | Warp: 0 | Lane: 1 | Thread 1 - Here!
SM: 4 | Warp: 0 | Lane: 0 | Thread 0 - Here!
SM: 4 | Warp: 0 | Lane: 1 | Thread 1 - Here!
SM: 2 | Warp: 0 | Lane: 0 | Thread 0 - Here!
SM: 2 | Warp: 0 | Lane: 1 | Thread 1 - Here!
SM: 0 | Warp: 0 | Lane: 0 | Thread 0 - Here!
SM: 0 | Warp: 0 | Lane: 1 | Thread 1 - Here!

With more than 32 threads in a block, the block is split into multiple warps: threads 0–31 fill warp 0, threads 32+ spill into warp 1, and lane IDs restart at 0 within each warp.

SM: 0 | Warp: 1 | Lane: 0 | Thread 32 - Here!
SM: 0 | Warp: 1 | Lane: 1 | Thread 33 - Here!
SM: 0 | Warp: 1 | Lane: 2 | Thread 34 - Here!
…
SM: 0 | Warp: 0 | Lane: 30 | Thread 30 - Here!
SM: 0 | Warp: 0 | Lane: 31 | Thread 31 - Here!

Global Memory

A deliberately naive dot product: a single thread loops over both vectors in global memory — the largest but slowest memory on the GPU. Every `a[i]` and `b[i]` read goes out to global memory, making this a useful baseline for profiling.

__global__ void dot_product(float* a, float* b, float* c, int size) {
 // Compute the dot product of a and b, then store the result in c
 float sum = 0;
 for (int i = 0; i < size; i++) {
 sum += a[i] * b[i];
 }
 *c = sum;
}

Launched with a single thread (`<<<1, 1>>>`) — no parallelism at all, on purpose:

int main() {
 const int vectorLength = 16;
// Allocate memory for the input arrays and the output array
 // …
dot_product<<<1, 1>>>(d_a, d_b, d_c, vectorLength);
// Copy the result back to host memory
 // …
}

Profile it with Nsight Compute to see where the memory traffic goes:

sudo ncu - set=full -f -o dot_product ./dot_product

Local Memory & Registers

In the generated PTX assembly, scalar variables like `sum` live in registers — the fastest storage on the GPU. This instruction initializes register `%f4` to `0.0f`:

mov.f32 %f4, 0f00000000;

Shared Memory

Shared memory and L1 cache come out of the same physical pool on the SM. `cudaFuncSetAttribute` lets you hint how to split that pool for a specific kernel:

cudaFuncSetAttribute(
 my_kernel,
 cudaFuncAttributePreferredSharedMemoryCarveout,
 20 // Use 20% of combined L1/Shared Memory for Shared Memory
);
my_kernel<<<num_blocks, num_threads>>>();

Thread Synchronization

A first parallel softmax. Each thread is responsible for a slice of the output (`start_idx` to `end_idx`), but the max and the sum of exponentials are computed *redundantly* — every thread loops over the entire input. Correct, but wasteful.

__global__ void softmax_kernel(float* input, float* output, int size) {
 // Number of threads in the Thread Block.
 // Assumes that the Thread Block is one-dimensional.
 int num_threads = blockDim.x;
// Each thread will compute the softmax of
 // num_elements_per_thread elements
 int num_elements_per_thread = size / num_threads;
int thread_index = threadIdx.x;
// This thread will compute the softmax of elements from
 // start_idx to end_idx in the input array
 int start_idx = thread_index * num_elements_per_thread;
 int end_idx = min(size, start_idx + num_elements_per_thread);
// Loop over the input array to find the maximum value
 float max_val = 0.0f;
 for (int i = 0; i < size; i++) {
 if (input[i] > max_val) {
 max_val = input[i];
 }
 }
// Loop over the input array to compute the sum of the exponentials
 float sum_exp = 0.0f;
 for (int i = 0; i < size; i++) {
 sum_exp += expf(input[i] - max_val);
 }
// Store the softmax result in the output array
 for (int i = start_idx; i < end_idx; i++) {
 output[i] = expf(input[i] - max_val) / sum_exp;
 }
}

Example: Softmax with Shared Memory

The optimized version: each thread computes the max and exponential sum only over *its own slice*, publishes the partial result to fast on-chip shared memory (`__shared__`), and then `__syncthreads()` acts as a barrier so no thread reads the shared arrays before every thread has written to them. The partial results are then combined into the global max and sum.

__global__ void softmax_kernel_smem(float* input, float* output, int size) {
 // Number of threads in the Thread Block.
 // Assumes that the Thread Block is one-dimensional.
 int num_threads = blockDim.x;
// Each thread will compute the softmax of
 // num_elements_per_thread elements
 int num_elements_per_thread = size / num_threads;
int thread_index = threadIdx.x;
// This thread will compute the softmax of elements from
 // start_idx to end_idx in the input array
 int start_idx = thread_index * num_elements_per_thread;
 int end_idx = min(size, start_idx + num_elements_per_thread);
// Array in shared memory to store the local max values
 __shared__ float shared_max_val[NUM_THREADS];
float max_val = 0.0f;
 for (int i = start_idx; i < end_idx; i++) {
 if (input[i] > max_val) {
 max_val = input[i];
 }
 }
 shared_max_val[thread_index] = max_val;
// Wait for all threads to finish writing their
 // local max values to shared memory
 __syncthreads();
for (int i = 0; i < num_threads; i++) {
 if (shared_max_val[i] > max_val) {
 max_val = shared_max_val[i];
 }
 }
// Array in shared memory to store the local sums of
 // the exponentials
 __shared__ float shared_sum_exp[NUM_THREADS];
float sum_exp = 0.0f;
 for (int i = start_idx; i < end_idx; i++) {
 sum_exp += expf(input[i] - max_val);
 }
 shared_sum_exp[thread_index] = sum_exp;
// Wait for all threads to finish writing their
 // local sums
 __syncthreads();
for (int i = 0; i < num_threads; i++) {
 sum_exp += shared_sum_exp[i];
 }
// Compute softmax
 for (int i = start_idx; i < end_idx; i++) {
 output[i] = expf(input[i] - max_val) / sum_exp;
 }
}

Writing Custom PyTorch Kernels — pybind11

pybind11 is the bridge that exposes C++ functions to Python. Install it first:

sudo apt-get install python3-pybind11

Kernels can’t be launched directly from Python, so we wrap the launch in a plain C++ “launcher” function:

void roll_call_launcher() {
 roll_call_kernel<<<1, 5>>>();
 cudaDeviceSynchronize();
}

The binding file declares the launcher and registers it as a Python function. `PYBIND11_MODULE` generates the module boilerplate — after this, Python can call `example_kernels.roll_call()`.

#include <torch/extension.h>
// Declare the roll_call_launcher function
void roll_call_launcher();
// Write the C++ function that we will call from Python
void roll_call_binding() {
 roll_call_launcher();
}
PYBIND11_MODULE(example_kernels, m) {
 m.def(
 "roll_call", // Name of the Python function to create
 &roll_call_binding, // Corresponding C++ function to call
 "Launches the roll_call kernel" // Docstring
 );
}

PyTorch’s `CUDAExtension` handles the tricky part — compiling `.cu` files with `nvcc` and `.cpp` files with the host compiler, then linking them into one Python module:

from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
__version__ = "0.0.1"

# Define the C++ extension modules
ext_modules = [
 CUDAExtension('example_kernels', [
 'csrc/roll_call/roll_call_binding.cpp',
 'csrc/roll_call/roll_call.cu',
 ])
]
setup(
 name="cuda_basics",
 version=__version__,
 ext_modules=ext_modules,
 cmdclass={"build_ext": BuildExtension},
)

Once built, the CUDA kernel is callable like any Python function:

$ python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import example_kernels
>>> example_kernels.roll_call()
Thread 0 here!
Thread 1 here!
Thread 2 here!
Thread 3 here!
Thread 4 here!

Best practice: use the `TORCH_EXTENSION_NAME` macro instead of hardcoding the module name, so the name in `setup.py` and the binding can never drift apart:

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
 m.def(
 "roll_call", // Name of the Python function to create
 &roll_call_binding, // Corresponding C++ function to call
 "Launches the roll_call kernel" // Docstring
 );
}

Using PyTorch in C++

To operate on real tensors, the launcher accepts a `torch::Tensor` and extracts the raw GPU pointer with `data_ptr<int>()` — the kernel itself doesn’t change at all:

#include <torch/extension.h>
__global__ void array_increment(int* in) {
 const int threadIndex = threadIdx.x;
 in[threadIndex] = in[threadIndex] + 1;
}
void array_increment_launcher(torch::Tensor& in, int arraySize) {
 array_increment<<<1, arraySize>>>(in.data_ptr<int>());
}

The binding now passes the tensor through from Python to the launcher:

#include <torch/extension.h>
void array_increment_launcher(torch::Tensor& in, int arraySize);
// Write the C++ function that we will call from Python
void array_increment_binding(torch::Tensor& in, int arraySize) {
 array_increment_launcher(in, arraySize);
}
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
 m.def(
 "array_increment", // Name of the Python function to create
 &array_increment_binding, // Corresponding C++ function to call
 "Launches the array_increment kernel" // Docstring
 );
}

The kernel modifies the PyTorch tensor in place, directly in GPU memory — no copies:

>>> import torch
>>> import array_increment_kernel
>>> input = torch.tensor([1, 2, 3, 4, 5], dtype=torch.int).cuda()
>>> array_increment_kernel.array_increment(input, 5)
>>> input
tensor([2, 3, 4, 5, 6], device='cuda:0', dtype=torch.int32)

Dealing with Data Types

There’s a catch: `data_ptr<int>()` hardcodes the type. Pass a float tensor and PyTorch throws immediately:

>>> input = torch.tensor([1, 2, 3, 4, 5], dtype=torch.float).cuda()
>>> array_increment_kernel.array_increment(input, 5)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
RuntimeError: expected scalar type Int but found Float

Step one of the fix: make the kernel a C++ template so it works for any element type:

template <typename T>
__global__ void array_increment(T* in) {
 const int threadIndex = threadIdx.x;
 in[threadIndex] = in[threadIndex] + 1;
}

Step two: `AT_DISPATCH_FLOATING_TYPES` inspects the tensor’s dtype at runtime and instantiates the template with the matching type, exposed inside the lambda as `scalar_t`:

void array_increment_launcher(torch::Tensor& in, int arraySize) {
 AT_DISPATCH_FLOATING_TYPES(in.scalar_type(), "array_increment_launcher", [&]() {
 array_increment<<<1, arraySize>>>(in.data_ptr<scalar_t>());
 });
}

The same binding now handles `float32` and `float64` tensors transparently:

>>> import torch, array_increment_kernel
>>> input_f32 = torch.tensor([1, 2, 3, 4, 5], dtype=torch.float32).cuda()
>>> array_increment_kernel.array_increment(input_f32, 5)
>>> input_f32
tensor([2., 3., 4., 5., 6.], device='cuda:0')
>>> input_f64 = torch.tensor([1, 2, 3, 4, 5], dtype=torch.float64).cuda()
>>> array_increment_kernel.array_increment(input_f64, 5)
>>> input_f64
tensor([2., 3., 4., 5., 6.], device='cuda:0', dtype=torch.float64)

References:

https://tinkerd.net/blog/machine-learning/cuda-basics/

The Problem with Container Startup

Containers are a great way to package and deploy applications.
However, starting a container can be slow, especially for large images.
This is because the container runtime needs to download and extract the entire image before it can start the container.
This process can take a long time, especially for large images with many layers. In this blog I explained how I speed up Colab container to < 1 second startup time.

OverlayBD to the Rescue

OverlayBD is a technology that can significantly improve container startup time.
It does this by converting the container image into a block device format that can be mounted efficiently by the container runtime.
Instead of extracting every layer sequentially, the image is presented as a virtual block device.
This allows for lazy loading of the container image, which means that only the necessary parts of the image are loaded into memory when they are needed.

How it Works

OverlayBD uses the Linux TCMU (Target Core Module Userspace) subsystem to present the container image as a virtual block device.
When the container runtime starts a container, it mounts the virtual block device as the root filesystem.
When the container accesses a file, the corresponding data is read from the virtual block device and loaded into memory.
This process is transparent to the container and the application running inside it.

Architecture

The OverlayBD architecture consists of two main components:

* **overlaybd-tcmu**: This component is responsible for presenting the container image as a TCMU block device.
It reads data from the container image and serves it to the kernel when requested.
* **overlaybd-snapshotter**: This component is a containerd proxy plugin that manages the snapshots of the container image.
It works with `overlaybd-tcmu` to provide the container runtime with a mountable filesystem.

Here is a high level diagram of the architecture:

```
containerd
└── overlaybd snapshotter (proxy plugin)
├── overlaybd-snapshotter (manages snapshots, serves gRPC)
└── overlaybd-tcmu (presents layers as TCMU block devices)
├── reads from /opt/overlaybd/fuse_cache/ (if FUSE cache active)
│ ├── SSD tier: /opt/overlayd/registry_cache/
│ └── GCS tier: /mnt/gcsfuse_cache/ (optional)
├── or reads from /opt/overlaybd/registry_cache/ (direct, default)
└── target_core_user (kernel module)
```

Optimizing Performance with Caching

OverlayBD provides several caching mechanisms to optimize performance.

Warm Cache

A warm cache contains all the data that is needed to start an application.
By creating a warm cache, you can significantly reduce the startup time of your application.
To create a warm cache, you first need to profile the startup of your application.
This can be done by running the `overlaybd-deploy profile-startup` command.
This command will clear the cache and run a container to record which data blocks are accessed during its initial startup.
This will generate a `startup-profile.json` file.

Once you have the startup profile, you can use it to pre warm the local cache by running the `overlaybd-deploy manage-cache warm` command.
This command will read the `startup-profile.json` file and download all the required blobs into the local SSD cache.
After this step, subsequent container starts will be much faster as they will read from the local SSD instead of the network.

You can then snapshot the warm cache and deploy it to other machines. This can be done by running the `overlaybd-deploy manage-cache snapshot` command.

FUSE Multi-Tier Cache

OverlayBD also provides a FUSE passthrough filesystem that sits in front of the SSD cache.
This adds optional GCS fallback and memory pinning of hot blobs.
OverlayBD’s TCMU reads from the FUSE mount instead of the SSD cache directly.

Reads check SSD first, then fall through to GCS.
Blobs served from GCS are asynchronously promoted (copied) to SSD for future reads.

To enable the FUSE cache, you can run the `overlaybd-deploy fuse-cache start` command.

Kernel I/O Tuning

OverlayBD also applies kernel level I/O tunings to optimize for cache workloads.
These tunings include increasing the block device read ahead value and adjusting the virtual memory dirty page ratio.
These tunings help to improve the performance of the cache and reduce the number of I/O operations.

Getting Started

The `overlaybd-deploy` package provides a CLI to install and configure OverlayBD. Check the following repo: https://github.com/codeexec/overlaybd-deploy

Installation

You can install the `overlaybd-deploy` package from PyPI:

```bash
pip install overlaybd-deploy
```

Or you can install it from the git repository:

```bash
git clone https://github.com/containerd/overlaybd.git
cd overlaybd-deploy
pip install -e “.[dev]”
```

Quick Start

Once you have installed the `overlaybd-deploy` package, you can use the following commands to get started:

```bash
# 1. Install OverlayBD (idempotent)
sudo overlaybd-deploy install

# 2. Set up credentials (pick one)
sudo overlaybd-deploy setup-credentials oauth2 # temporary (1 hour)
sudo overlaybd-deploy setup-credentials service-account /path/to/key.json # permanent

# 3. Pull and run
export GOOGLE_CLOUD_PROJECT=${GOOGLE_CLOUD_PROJECT:-$(gcloud config get-value project)}
sudo overlaybd-deploy pull-image
sudo ctr run — snapshotter overlaybd — rm \
“us-docker.pkg.dev/${GOOGLE_CLOUD_PROJECT}/colab-optimized/runtime:latest_obd” \
test /bin/echo “hello from overlaybd”
```

For more information, please refer to the [git repository](https://github.com/containerd/overlaybd).

The Benefits of OverlayBD

The main benefit of using OverlayBD is faster container startup time.
For example, with OverlayBD, a 27GB container image can start in as little as 170 milliseconds with a warm cache.
With lazy loading, the same image can be pulled in 2.2 seconds and started in 5.6 seconds.
This is a significant improvement compared to the traditional container startup process, which can take several minutes for an image of this size.

Another benefit of OverlayBD is that it allows for lazy loading of the container image.
This means that you don’t have to wait for the entire image to be downloaded before you can start the container.
This is especially useful in development environments, where you may want to quickly test a change to your application.

Conclusion

OverlayBD is a powerful technology that can significantly improve container startup time.
It is a great choice for anyone who is working with large container images and wants to improve the performance of their containerized applications.

How do you optimize infrastructure for Prediction and Runtimes when the container itself is a monolith? It requires moving beyond simple size checks and performing a forensic analysis of the boot process.

In this post, we are going to perform an autopsy on the public Google Colab runtime image. We will use tools like docker inspect, ncdu, and strace to look inside the layers, visualize the disk budget, and debunk common performance myths. By tracing the exact system calls during startup, we will identify what actually causes latency—distinguishing between necessary overhead and invisible bloat like unused Jupyter extensions.

Analyzing Docker image:

The following commands allow you to get an understanding of what is contained in each image, in this case we are analyzing the Public Colab image:

DOCKER_IMAGE=us-docker.pkg.dev/colab-images/public/runtime

docker manifest inspect $DOCKER_IMAGE
docker inspect $DOCKER_IMAGE
docker history $DOCKER_IMAGE
docker history --format '{{.Size}}\t{{.ID}}\t{{.CreatedAt}}\t{{.CreatedBy}}' $DOCKER_IMAGE --no-trunc | sort -rh
alias dive="docker run -ti --rm  -v /var/run/docker.sock:/var/run/docker.sock docker.io/wagoodman/dive"
dive $DOCKER_IMAGE

DOCKER_IMAGE=us-docker.pkg.dev/colab-images/public/runtime
docker manifest inspect $DOCKER_IMAGE
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 12111,
         "digest": "sha256:d954b05cd315ef5db8a6aac71487f28cce49df9a27a8c1bdff3dea94638edc24",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:06bc74963df5e8cef1bed9f2bcd54818fc4b6b675b61ed4c3247e67c53914459",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}

This output indicates that the tag is an OCI Image Index (a “manifest list”), which acts as a wrapper grouping two distinct items:

1. The Runnable Image (linux/amd64): This is the actual container image that executes your code.
2. The Metadata Artifact (unknown/unknown): This is a non-runnable attachment—likely a SLSA provenance attestation or signature.

The takeaway: Modern registries use this structure to bundle security verification data alongside the software itself. Docker clients are smart enough to pull the runnable image (Item 1) while security tools can inspect the metadata (Item 2) to verify the build integrity.

DIGEST="sha256:06bc74963df5e8cef1bed9f2bcd54818fc4b6b675b61ed4c3247e67c53914459"
REPO="us-docker.pkg.dev/colab-images/public/runtime"

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Accept: application/vnd.oci.image.manifest.v1+json" \
     "https://us-docker.pkg.dev/v2/colab-images/public/runtime/manifests/$DIGEST"
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:679c6533dc9666962a1371d3dd633f6defe7c11de372db6941f9854dbe35eca9",
    "size": 167
  },
  "layers": [
    {
      "mediaType": "application/vnd.in-toto+json",
      "digest": "sha256:93646c93355e912fcfe5f7feb944c8b9f7c0cbdbc6421c0b30aeb2d5ac53ecce",
      "size": 1369,
      "annotations": {
        "in-toto.io/predicate-type": "https://slsa.dev/provenance/v0.2"
      }
    }
  ]
}

“When we inspected the ‘unknown’ artifact attached to the Docker image, we found a SLSA Provenance Attestation.

Think of this as a digital, tamper-proof receipt for the container image. It is not code that runs; it is a signed document that proves:

1. Where the image was built (e.g., Google Cloud Build).
2. When it was built.
3. What source code (Git commit hash) was used to build it.

By attaching this receipt directly to the image in the registry (using the OCI Image Index), Google allows security tools to automatically verify the image’s origin before allowing it to run in production.”. If you want to see the actual “receipt” (the build details, repo source, and builder ID), you can inspect the blob in the layers section:

LAYER_DIGEST="sha256:93646c93355e912fcfe5f7feb944c8b9f7c0cbdbc6421c0b30aeb2d5ac53ecce"
curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)"      "https://us-docker.pkg.dev/v2/colab-images/public/runtime/blobs/$LAYER_DIGEST" | python -m json.to
ol
{
    "_type": "https://in-toto.io/Statement/v0.1",
    "predicateType": "https://slsa.dev/provenance/v0.2",
    "subject": [
        {
            "name": "pkg:docker/us-docker.pkg.dev/colab-images/private/public-runtime@release-colab-external_20251217-060051_RC00?platform=linux%2Famd64",
            "digest": {
                "sha256": "d954b05cd315ef5db8a6aac71487f28cce49df9a27a8c1bdff3dea94638edc24"
            }
        },
        {
            "name": "pkg:docker/us-docker.pkg.dev/colab-images/private/public-runtime@staging?platform=linux%2Famd64",
            "digest": {
                "sha256": "d954b05cd315ef5db8a6aac71487f28cce49df9a27a8c1bdff3dea94638edc24"
            }
        }
    ],
    "predicate": {
        "builder": {
            "id": ""
        },
        "buildType": "https://mobyproject.org/buildkit@v1",
        "materials": [
            {
                "uri": "pkg:docker/us-docker.pkg.dev/colab-images/private/base-runtime@release-colab-external_20251217-060051_RC00?platform=linux%2Famd64",
                "digest": {
                    "sha256": "bde98c047e4fd7b39325a5abb632e68b0a8d90dab5f11ca30ac3d65d8a9da407"
                }
            }
        ],
        "invocation": {
            "configSource": {
                "entryPoint": "public-image.Dockerfile"
            },
            "parameters": {
                "frontend": "dockerfile.v0",
                "locals": [
                    {
                        "name": "context"
                    },
                    {
                        "name": "dockerfile"
                    }
                ]
            },
            "environment": {
                "platform": "linux/amd64"
            }
        },
        "metadata": {
            "buildInvocationID": "2bhz6smjgkfn76jkv05ayr9j5",
            "buildStartedOn": "2025-12-17T14:42:17.943253663Z",
            "buildFinishedOn": "2025-12-17T14:45:18.025930694Z",
            "completeness": {
                "parameters": false,
                "environment": true,
                "materials": false
            },
            "reproducible": false,
            "https://mobyproject.org/buildkit@v1#metadata": {}
        }
    }
}

Inspecting the runnable image reveals it is an Ubuntu 22.04 environment heavily optimized for NVIDIA GPUs (CUDA 12.5). Unlike typical microservices, this is a ‘fat’ image with over 60 layers. This architectural choice trades download size for speed: by pre-baking gigabytes of data science libraries into the layers, the runtime allows users to start training models seconds after booting, rather than waiting for dependencies to install.”

docker inspect $DOCKER_IMAGE
[
    {
        "Id": "sha256:d5458217b23549186bb7d0b5e67f4eef0bd82ca12f84c3a341e7a46aa148b02a",
        "RepoTags": [
            "us-docker.pkg.dev/colab-images/public/runtime:latest"
        ],
        "RepoDigests": [
            "us-docker.pkg.dev/colab-images/public/runtime@sha256:53b9c1a98f563d1dc4eab3f1b54ff43c8a82531f893e17d8818aa7671b36c6fd"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2025-10-24T14:11:20.743187819Z",
        "DockerVersion": "20.10.24",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "8080/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin",
                "NVARCH=x86_64",
                "NVIDIA_REQUIRE_CUDA=cuda>=12.5 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551",
                "NV_CUDA_CUDART_VERSION=12.5.82-1",
                "CUDA_VERSION=12.5.1",
                "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64",
                "NVIDIA_VISIBLE_DEVICES=all",
                "NVIDIA_DRIVER_CAPABILITIES=compute,utility",
                "NV_CUDA_LIB_VERSION=12.5.1-1",
                "NV_NVTX_VERSION=12.5.82-1",
                "NV_LIBNPP_VERSION=12.3.0.159-1",
                "NV_LIBNPP_PACKAGE=libnpp-12-5=12.3.0.159-1",
                "NV_LIBCUSPARSE_VERSION=12.5.1.3-1",
                "NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-5",
                "NV_LIBCUBLAS_VERSION=12.5.3.2-1",
                "NV_LIBCUBLAS_PACKAGE=libcublas-12-5=12.5.3.2-1",
                "NV_LIBNCCL_PACKAGE_NAME=libnccl2",
                "NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1",
                "NCCL_VERSION=2.22.3-1",
                "NV_LIBNCCL_PACKAGE=libnccl2=2.22.3-1+cuda12.5",
                "NVIDIA_PRODUCT_NAME=CUDA",
                "NV_CUDA_CUDART_DEV_VERSION=12.5.82-1",
                "NV_NVML_DEV_VERSION=12.5.82-1",
                "NV_LIBCUSPARSE_DEV_VERSION=12.5.1.3-1",
                "NV_LIBNPP_DEV_VERSION=12.3.0.159-1",
                "NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-5=12.3.0.159-1",
                "NV_LIBCUBLAS_DEV_VERSION=12.5.3.2-1",
                "NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-5",
                "NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-5=12.5.3.2-1",
                "NV_CUDA_NSIGHT_COMPUTE_VERSION=12.5.1-1",
                "NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-5=12.5.1-1",
                "NV_NVPROF_VERSION=12.5.82-1",
                "NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-5=12.5.82-1",
                "NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev",
                "NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1",
                "NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.22.3-1+cuda12.5",
                "LIBRARY_PATH=/usr/local/cuda/lib64/stubs",
                "NV_CUDNN_VERSION=9.2.1.18-1",
                "NV_CUDNN_PACKAGE_NAME=libcudnn9-cuda-12",
                "NV_CUDNN_PACKAGE=libcudnn9-cuda-12=9.2.1.18-1",
                "NV_CUDNN_PACKAGE_DEV=libcudnn9-dev-cuda-12=9.2.1.18-1",
                "PYTHONPATH=/env/python",
                "TF_FORCE_GPU_ALLOW_GROWTH=true",
                "CLOUDSDK_CONFIG=/content/.config",
                "CLOUDSDK_PYTHON=python3",
                "DEBIAN_FRONTEND=noninteractive",
                "LANG=en_US.UTF-8",
                "LANGUAGE=en_US",
                "LC_ALL=en_US.UTF-8",
                "PYTHONUTF8=1",
                "LAST_FORCED_REBUILD=20250623",
                "COLAB_JUPYTER_IP=127.0.0.1",
                "COLAB_DEBUG_ADAPTER_MUX_PATH=/usr/local/bin/dap_multiplexer",
                "COLAB_LANGUAGE_SERVER_PROXY=/usr/colab/bin/language_service",
                "COLAB_WARMUP_DEFAULTS=1",
                "TCLLIBPATH=/usr/share/tcltk/tcllib1.20",
                "UV_BUILD_CONSTRAINT=",
                "UV_CONSTRAINT=",
                "UV_INSTALL_DIR=/usr/local/bin",
                "UV_SYSTEM_PYTHON=true",
                "KMP_LISTEN_PORT=6000",
                "COLAB_KERNEL_MANAGER_PROXY_PORT=6000",
                "KMP_TARGET_PORT=9000",
                "COLAB_FILE_HANDLER_ADDR=localhost:3453",
                "VM_GCE_METADATA_HOST=169.254.169.254",
                "ENV=/root/.bashrc",
                "SHELL=/bin/bash",
                "COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00",
                "COLAB_TPU_1VM=",
                "COLAB_JUPYTER_ALLOW_ORIGIN_PAT=https://colab\\.(sandbox|research)\\.google\\.com",
                "COLAB_HUMAN_READABLE_NODE_LOGS=1"
            ],
            "Cmd": null,
            "Image": "sha256:bc1532477588f8bece50515c5fc9d8e499338b84923945959fb29184a888daf2",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/datalab/run.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.nvidia.cudnn.version": "9.2.1.18-1",
                "maintainer": "Google Colaboratory Team <colaboratory-team@google.com>",
                "org.opencontainers.image.ref.name": "ubuntu",
                "org.opencontainers.image.version": "22.04"
            },
            "Shell": [
                "/bin/bash",
                "-o",
                "pipefail",
                "-c"
            ]
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 41045136128,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/50143af3eb6b3901afb5e990f12a930a1199b312f9a58fa9eef152ed1ee0051d/diff:/var/lib/docker/overlay2/29048d08e14128cd2c3d1df7e79da94c980ed9f271599bf550334d47719d64fc/diff:/var/lib/docker/overlay2/d65a5b55f9d12a621ebaf574e35e91dc89cd09ab8e95ddce866a1ef1ff09c756/diff:/var/lib/docker/overlay2/d240cba4a175d90bfa09f2046e9b61a91b7e93c6b20659ba61c6d38a15177c16/diff:/var/lib/docker/overlay2/3e0c7e8bb2ac66a9cdce9578f48fdb2321520cebcebebed2816c730ba2eae3d8/diff:/var/lib/docker/overlay2/ac502588bd77aadee70544c044fee4a01c9b14d613e699f99c12a35c73255b80/diff:/var/lib/docker/overlay2/f99246d6897594b3bd61be3ddb1bf1988174561ad2c36bfef85ee4a5e733d58a/diff:/var/lib/docker/overlay2/b3beb6128cda61259defeb13d8c0e4cdd57425636e0ffbdd1651770b09e2ae13/diff:/var/lib/docker/overlay2/90c55f10385c33ac193d2d15890378ee5acf7a2c98cceb4140a36fbbb84be484/diff:/var/lib/docker/overlay2/c1b33a768c602682239ca8f5049df33708eec5bf1d93ef084b7d4f93b87dc734/diff:/var/lib/docker/overlay2/a903633614cc2b7bd6d544d6a6135b546c0460dee418f889838c54ec69607a70/diff:/var/lib/docker/overlay2/7ad0e8e9ffdd8ddaed3862362441471aaf069d06a59cf52e9465b74934e2ddf3/diff:/var/lib/docker/overlay2/e3bde1f23dc61bdb02ae4c47cf986f625f090ff82d595d4a6821ab8619676e9a/diff:/var/lib/docker/overlay2/b471212efb1b0faa7a84f7356270646660e0201882bbaba8440a28ad5896cf49/diff:/var/lib/docker/overlay2/16f36e1113634bb9e88a334622cf05b34347bb64631a9d488b28a9d6951efc01/diff:/var/lib/docker/overlay2/ca0a8e66388ffe8261edf7768abf440e7748d60e97e8df8ffd3bc75d6c7b3327/diff:/var/lib/docker/overlay2/eedbad0a5761a8388c5927f4e6aa580b73c8fc6493821584c894d81c1549f440/diff:/var/lib/docker/overlay2/bc5ded093a5cd6e44c6242dcd9f415ee8719bfd4428780c648b575e169e3176f/diff:/var/lib/docker/overlay2/c44e543209ecf7dffa54f200d4d34ca2455b844d1ce2d7ed2a9dd2ce0f49c2e4/diff:/var/lib/docker/overlay2/33131410c62cece58f3f06e7e22ff15b8644279f3dc718add5e135682a19478a/diff:/var/lib/docker/overlay2/b431d1e048891ea61d809e3f9097e2278641c730d1dc27a0903af9e19b524eff/diff:/var/lib/docker/overlay2/7efca97d820c4e09bc399344ecfaa3413e982a83391f2cc2599631c37b08aa79/diff:/var/lib/docker/overlay2/38c57b7866c2be3f3872542d122a677df5badf63915036da2fa762f3de950128/diff:/var/lib/docker/overlay2/4a1827eee6c582d92ab3e73f1f1ae80a044736bd972021065eb943a23752052d/diff:/var/lib/docker/overlay2/10854d88a7a9a9116f85dee92b0b11664a8ef1ec5dcdb74cbfdfef991859560f/diff:/var/lib/docker/overlay2/0ca10bffb45d8aec05802848fb1dde61e2812acd69422e2b8f685342823762d3/diff:/var/lib/docker/overlay2/8ec560583309f1a618f1eb2c786cf38caad92eb9feb4fb865bc3fc797ed0f50e/diff:/var/lib/docker/overlay2/71b888aeb067c03ef910f2fe1cf98735b0b8ba03777681e806ed30cdb8d54554/diff:/var/lib/docker/overlay2/b8035c9e683749440f038f49cbdcf46dd5de2873b2484d3021c460e808ea0836/diff:/var/lib/docker/overlay2/98ee8667be22a23edaa31c6a17049b5e7354ce34fd8d12e53f86c5e1f9bf28bf/diff:/var/lib/docker/overlay2/5376b20d9ec992ee336b06c9356654dae8ae50b1d330f1082d315f3b8e8bed4d/diff:/var/lib/docker/overlay2/c5637244899ef86d1b91655a090c8faa3486a33f40e26ee6ea4f510da0beb311/diff:/var/lib/docker/overlay2/9ba3e7c6d8ba9d09b7300612d2de4d6b526b5589b5698be02b938eb38d909c0b/diff:/var/lib/docker/overlay2/df942cce1fa03d57fc12460d19f59e1d71fb5c6d7e91b808027f65b9aad6dc1a/diff:/var/lib/docker/overlay2/4ae2f7575ae1a7df51f75eabffc3a2b92bca3a352fbaef8b31ccc42bae0ade11/diff:/var/lib/docker/overlay2/e6a6af12d09868dbc97201314341c864711354ea49752b457ce86545a760453c/diff:/var/lib/docker/overlay2/701b2cdef3fd972636871b8a9872e3bba014b111918b3bd535a887a8a2ee079c/diff:/var/lib/docker/overlay2/440fb4d50c760ec0dac1a9d1eb3bf2317aafa6f83daed06f8e251b8f8a5c6e83/diff:/var/lib/docker/overlay2/20109eabbe2be55b3e0833c674e705d3dfa0d4f97737ecb74077f11a67a79508/diff:/var/lib/docker/overlay2/4264cf569e4101b8142d323ef55e1a4846081096cd0181b5fce2b3e8d05b4ef9/diff:/var/lib/docker/overlay2/b523c2c980147cfbe1cf8303e35be7a90fd45a8f4050de988926703c77ba3e74/diff:/var/lib/docker/overlay2/c25b759a4dbe0e21f3eec29631aa1af59ce121b29a3b482cd77dc9d623c27c84/diff:/var/lib/docker/overlay2/e82c21fa732f432857a02180057c78df9a0b3d65fc4c85a18d66d21447fef991/diff:/var/lib/docker/overlay2/373771e39758ac1244fa00319a2286f741b3ae6d90a102dfcab093185ef06f93/diff:/var/lib/docker/overlay2/8b3c49056cc46285693e989fbb137184f3a0122c3b48768000b61f7e36474c73/diff:/var/lib/docker/overlay2/db8d787efacb0cc66603e61ed086fd6bd715c1e479fe2bfb81bc080f11763748/diff:/var/lib/docker/overlay2/105af54dc4b24834eec5377d5186cd61826889b68cf74c76942178e3dc0d2815/diff:/var/lib/docker/overlay2/35ef4bf9be99de2eadb915844b680f7bd938f6a7269c8c21fee721395ab2779a/diff:/var/lib/docker/overlay2/11da47f03b8323d2b3bb63ac17fc5a7fcfdaf0fafebb26c5c786f900854f374a/diff:/var/lib/docker/overlay2/35f04ebae6e7de070aa3242fb673b7b781b7455765e30299c8ef4e535843b927/diff:/var/lib/docker/overlay2/7cdc8f72b68a222e9e6a00c6c7af0d2e83d1463e591d88aa925a7d196ae4b290/diff:/var/lib/docker/overlay2/4e9f1a06e3e1b897874a9c4416f14b8f7dc1a091ba548553fb0ff7b8d7efbf02/diff:/var/lib/docker/overlay2/9b963bd29c9ac65c7270b7b2ecc04105421516d6af24dfb201f668d41b91f80a/diff:/var/lib/docker/overlay2/93ecbf0b3f58b9deb748c2b09be7a46c3cb28969f6e4b0d6ecc7cda4c8876c54/diff:/var/lib/docker/overlay2/4a82e1544d4cf0e3a85adc652e92f3b91d306b4caf5a7b4f93310f45abd56957/diff:/var/lib/docker/overlay2/c139193edf9f0c99db1dd8a5806040939cc47fb98142a76c195f73bba987c01e/diff:/var/lib/docker/overlay2/95c253a72d4ec01a9d87a8a00fd0f9155fa8d2d30099f4495b61abf1db6453c7/diff:/var/lib/docker/overlay2/2b04239c3402988d10bfeb227e229f1357176c5980228be0861418d8c72cf007/diff:/var/lib/docker/overlay2/9ba1ac5e083d1d34cf5c85b4700dd7f4bc36af7ccc71ed8e12f03d11ceacbd62/diff:/var/lib/docker/overlay2/6b1ee7711b7964b53ffeff9d87a14f3fb6e232c897872ce00a04f38369fbdae8/diff:/var/lib/docker/overlay2/02c389eed2a2e3ce53a7a72a4dc8870d112fb3cfe7d5a6e440239bc39f73dc06/diff",
                "MergedDir": "/var/lib/docker/overlay2/b25799bdae7a1d3ee1c8fc36146ef64a85d87fcf07ef1223a4792628d0af98bb/merged",
                "UpperDir": "/var/lib/docker/overlay2/b25799bdae7a1d3ee1c8fc36146ef64a85d87fcf07ef1223a4792628d0af98bb/diff",
                "WorkDir": "/var/lib/docker/overlay2/b25799bdae7a1d3ee1c8fc36146ef64a85d87fcf07ef1223a4792628d0af98bb/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:931b7ff0cb6f494b27d31a4cbec3efe62ac54676add9c7469560302f1541ecaf",
                "sha256:db406658b54593a161c3de8c38f321420ed569ae94d7e16b1a6b7ce4d46bfba0",
                "sha256:c35c14c72813fbfa41e48dee6d70ccf43984e3943eea24e6e7bc7be423056f42",
                "sha256:888b66dc5a210906050f1550c3caa0b69b9b48d0adc022431f50d8fcf34a6dda",
                "sha256:563e7c97ed4d3ed6e70bb03b82738fe7131ca91147b502659423b46f3fa7dfab",
                "sha256:1bdf3319743984171af50d4f825439769df5a9aac393cc716bff7793ecc4fe89",
                "sha256:31633e4421ebd1030235b44510c716a76ea2edad37fd07a9ce646af331becf8c",
                "sha256:a6ec53dbbbce285cd5df125037e2426ade00cd4cc9aaab88f67a54848e5d56ea",
                "sha256:22f1a6014bd5e4c2ae5c127142860e65ea08406b74c0a4c468c384d396aa30cb",
                "sha256:55234d4baee80a9a386e05bf6f2eab6acb91a2f13b1502ea702537b00303233b",
                "sha256:d0059c7387112aa7b80113a77956702d498be28dc0cac94ccbe1194da3a2f494",
                "sha256:28e4bd83ae2754e3ff412bef2ca1fff2a580748074ee5c261bb28abc4f55f6c8",
                "sha256:4dc90ba6ab7a3e335066c918d640af38e5e1e4a583c8f1bfe6698a14cffce978",
                "sha256:3cde843408435ec7690f9a8076ecce15b919cbee4cacf6fc7b0365ff9626c6ed",
                "sha256:208731d38664f139be3c632953e87d8c426c6ba2b84bcecdecb55cd1d455376d",
                "sha256:379ab108fa86ba4af88261e1fa4e713b3b7ea1bd61c629bac67360026f6655c5",
                "sha256:ce0bb5285a19e9d595607d4547661984f034f7d700bb246a0404491495dad9c9",
                "sha256:c64cde5c8a6945933aafdce17c88a28f88f6845009c81d2ec075050f9182a9d4",
                "sha256:5fd664040aca6c6665fef21fc1a1242d0888838af268b79426e8dfb3855f088a",
                "sha256:4924e38231fc6d412b7221573e1fa2d0fd72e72deebd96c7ccb761b7da2cc8a8",
                "sha256:7fa654400d7ec0ab172f0a4a63879ffc18228b8170d30328be8be89672277332",
                "sha256:ffa32dbee57a89aa4283d4d5f30db582c434f532ebf22e6cb71555b71ceef265",
                "sha256:a8d92e6daccc9480d0d9211b30e97a322218995114c2f7f6551041171ac6aa77",
                "sha256:1dbd9b7e5fdda476816400b61ff7e89b23e4f08b325cc1e115d89a9bac839220",
                "sha256:c15d66af5ecc888961a383237e3969298e8ee4e029101be54ed2f28e45382261",
                "sha256:7f31386006660fde268705a6b11967364d9dad5725d782e1cb58a80456ccfb42",
                "sha256:731f5501ca58f6a0f9d62896e1e504f3358913e3e58f9ff8d96cc8e619b3f119",
                "sha256:a21b50e553cec8f091b86fbac8a2449fcde61ef60be56c8c26a837488d015731",
                "sha256:2e77443fe3e15d9d64a7105160e652ceb6a60f2962b20c28dea91e215532c6ac",
                "sha256:4d15737a6435c13328a2218d99d5e7fe74d06471490f90831fee9b546f3e0955",
                "sha256:af638a2a8290a8718c8b6e84ad949c7dc1be0fc504bd01741c6471c73df5c791",
                "sha256:b6feceb87de8f8c40ad8f72d677ba6d9ff25f208305bf5afa06ab5dc92c03d5f",
                "sha256:c4f457784b49a96278d80f6d3b59057470cea2eab33367649418d1127b56f728",
                "sha256:c230a2be23e4118c85eb86cf9370bf4982e051d0a05dbb3d28be9ef801ac6dda",
                "sha256:de3703b62e9e9549d5a1a4ef3fb125d257c390aacabe406a17f191dfcd8cc512",
                "sha256:1337f63226f634fa86c9dac03f21f75552ba096399e76ee8d80bcb2216cbb7e2",
                "sha256:845a1ee022e56c55a4ec4f94b5bce82e974ffc2cfa0bad7b046e7fa10d8ca8f0",
                "sha256:7f7df2718427941de1f6bb8cf7f8d96b37ebe668a42241502a755eefc2606eb0",
                "sha256:b3b2cd19a67bd706cad52272557b1c34e28ee7625aebe9e6854107789bd70590",
                "sha256:32336cde1d5adc41f09d77f21d6c1f9f2a008734c63d5b33aeadbaebd238b55f",
                "sha256:d15d5df36a16147777c90c3e84dc3fc492f213b2d11c9278a97bb724d7eb97d6",
                "sha256:b76b797623429d19b92beb93e2b5ab2d8df499937414927082f2c1ab48826c4a",
                "sha256:8db1f8803ae6467d9c93c478264fa4641f14aa44f68458041e5c480c2382a316",
                "sha256:b4e5b507afead7b2cd45ec66dbf653cceffe6da8ff2663879d46b561e504bbf1",
                "sha256:56663e6473d3d0b0e9ba9d831f864cc20f2141b4e58f7637bc1330cd3880d684",
                "sha256:068d81b73ae54936d3d19e7fb78376db81d6fa8fef2de0559a2bae6add81b9e0",
                "sha256:5a05938580404d3cff28c54af75469a9f4d52f7595488c7c4aa1d7d9b0a22ce6",
                "sha256:d5e899031dccede8cc815c3b60f6d417aadbe80d2fe971eaacf5812d95a5a957",
                "sha256:dd2ea2973c2c20014c4e8574312436d78834790605c67c6f566ded312e2033d9",
                "sha256:39e1272faa4613408b3a5193c7256bc4bdc728bab256c98a8c19782e1721d757",
                "sha256:54c6cab3c1b04371cae357a3e775143929a42a0ee83bbf6c2d2601ddcfc7eb36",
                "sha256:2cb24a4965f7fd91569353dc200c64faadc7d4bb112c5c92c89a0f284bec7b9a",
                "sha256:c9d749c7b9e70905e0f3df1385ccd7a957b292c324ef2ee29333d96d62e58ec0",
                "sha256:abc7716586408a8a39078a0fb498b04cf4d4b858fec6ec04556fc2e59098bfc4",
                "sha256:055ad36cbc628cae0ddfe8e278f260424265ce3b6c3960227559d7d62bb6a268",
                "sha256:d28ae9da1731e3b12e717382ed338edbceb8e5bddc7c8c37021189c5f2c02e51",
                "sha256:1eed00f06fabbea913284a0a0d050873b18e648c80e43e3e62686d7e49f847d8",
                "sha256:06c331f4e52aeaee5e427b2bfaded102492302ad9f15cccdc6aa3330df5d13ff",
                "sha256:0d75d682e239dcadac98c1712f596fb81e35ec1ff9ac0a4af6e92438cf07938d",
                "sha256:fab02472979b1c342afe6d2585591b753e91241e4ddeb2ff664a754413ed97b3",
                "sha256:70f56a036c9da06865f3f1cad3fc0ac285ad1578413010b3b07224bb53813a2c",
                "sha256:f86c1788bb90d3fb68827f0ebf16cad50556510762a0255c2d59d9e56fd948b2"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

When analyzing a bloated container, docker inspect is your X-ray machine. Here is how to read the three most critical sections for optimization, using a 38GB deep-learning image as our case study:

1. Size: The "Cold Start" Tax

• What it is: The cumulative size of the image on disk (in our example: 41045136128 bytes, or ~38.2 GB).
• Why it matters: This number directly correlates to your cold-start latency. Every gigabyte represents network time required to pull the image and I/O time to unpack it.

2. RootFS (Layers): The Build Efficiency Log

• What it is: A JSON array of SHA256 hashes, where each hash represents a unique layer in the filesystem.
• Why it matters: This tracks the history of your filesystem changes. A long list (like the 60+ layers in the example) indicates a `Dockerfile` with too many separate instructions.

3. GraphDriver (Overlay2): The Runtime Assembly

• What it is: This section reveals how the overlay2 driver stacks your image.
• LowerDir: The stack of read-only image layers (the immutable base).
• UpperDir: The thin, writable layer where the running container stores changes.
• MergedDir: The unified view the application actually sees.
• Why it matters: This exposes the Copy-on-Write (CoW) mechanism. If your application frequently modifies files that exist in the huge LowerDir stack, the driver must copy them to the UpperDir first, causing I/O overhead.

Checking Docker history

docker history --format '{{.Size}}\t{{.ID}}\t{{.CreatedAt}}\t{{.CreatedBy}}' $DOCKER_IMAGE --no-trunc | sort -rh
20.2GB  <missing>       2025-10-24T13:52:13Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c export REQUIREMENTS_FILE=requirements3.txt &&     if [[ "${COLAB_TPU_1VM}" == "1" ]]; then         export REQUIREMENTS_FILE=requirements3_tpu.txt;      fi &&     uv pip install -U         --no-cache         $(grep '^numpy==' /backend-container/containers/${REQUIREMENTS_FILE}) &&     echo 'uv install for pre-required packages done.' &&     uv pip install         --no-cache         --reinstall-package six six==1.16.0 &&     echo 'uv install exceptional packages for Ubuntu 22.04 done.' &&     uv pip install -U         --no-cache         -r /backend-container/containers/${REQUIREMENTS_FILE} &&     C_INCLUDE_PATH=/usr/include/gdal CPLUS_INCLUDE_PATH=/usr/include/gdal         uv pip install --no-build-isolation         --no-cache         GDAL=="$(gdal-config --version)" &&     if [[ "${COLAB_TPU_1VM}" != "1" ]]; then         mkdir -p /etc/OpenCL/vendors &&         echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd &&         mkdir /tmp/lightgbm &&         pushd /tmp/lightgbm &&         LIGHTGBM_VERSION="$(uv pip show lightgbm 2>/dev/null | grep Version: | sed 's/.*Version: //')" &&         git clone --recursive --quiet --branch=v${LIGHTGBM_VERSION} https://github.com/microsoft/LightGBM &&         pushd LightGBM &&         ./build-python.sh bdist_wheel --gpu             --opencl-library="/usr/local/cuda/lib64/libOpenCL.so"             --opencl-include-dir="/usr/local/cuda/include/" &&         uv pip install --no-cache             --system             --no-build-isolation             dist/lightgbm-${LIGHTGBM_VERSION}-py3-none-linux_x86_64.whl &&         popd &&         popd &&         rm -Rf /tmp/lightgbm &&         uv pip install             --no-cache             --no-build-isolation             --reinstall-package tensorflow tensorflow[and-cuda]==2.19.0;         uv pip install             --index-url  https://download.pytorch.org/whl/cu126             --no-cache             --no-build-isolation             --reinstall-package torch torch==2.8.0             --reinstall-package torchaudio torchaudio==2.8.0             --reinstall-package torchvision torchvision==0.23.0;     else         uv pip install             --no-cache             --no-build-isolation             --reinstall-package libtpu libtpu==0.0.17;     fi &&     python3 -m spacy download en_core_web_sm &&     echo 'py3 spacy en_core_web_sm download done' &&     rm -rf /root/.cache/{pip,pip-tools,uv} /*whl &&     cp /backend-container/containers/${REQUIREMENTS_FILE} /etc/ &&     cp /backend-container/containers/requirements.core.in /etc/ &&     cp /backend-container/containers/requirements.user.in /etc/ &&     cp /backend-container/containers/requirements.in /etc/
5.02GB  <missing>       2025-10-24T13:18:54Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c apt-get install --no-install-recommends -y -qq         apt-utils         bc         build-essential         ca-certificates         clang         clinfo         cmake         coinor-libipopt1v5         coinor-libipopt-dev         curl         dirmngr         dnsutils         ffmpeg         file         fio         fonts-humor-sans         fonts-liberation         fuse         gfortran         gh         gir1.2-gtk-4.0         git         git-lfs         google-perftools         graphviz         intel-mkl         jq         kmod         less         libapt-pkg-dev         libasound2-dev         libatlas-base-dev         libavdevice58         libblocksruntime-dev         libboost-all-dev         libcairo2-dev         libcurl4         libcurl4-openssl-dev         libegl1         libfontconfig1         libfreetype6-dev         libgdal-dev         libgirepository1.0-dev         libgit2-dev         libgl1         libgles2         libglib2.0-0         libglvnd0         libgl1-mesa-glx         libhdf5-dev         libicu-dev         libjack-dev         liblapack-dev         libncurses5-dev         libopenblas-dev         libopencv-dev         libopengl0         libopenmpi-dev         libpng-dev         libprotobuf-java         libpython3-dev         libsm6         libsndfile-dev         libsndfile1         libssl-dev         libturbojpeg         libudunits2-0         libudunits2-dev         libwebpdemux2         libxext6         libxft-dev         libxml2-dev         libxrender1         libxslt1.1         libzmq3-dev         libzmq5         linux-headers-generic         locales         lsof         man         manpages         manpages-dev         manpages-posix         manpages-posix-dev         moreutils         net-tools         nvidia-opencl-dev         openjdk-11-jdk-headless         openssh-client         p7zip-full         pigz         pkgconf         protobuf-compiler         psmisc         python3.12         python3.12-dev         python3.12-tk         r-base         r-base-core         r-base-dev         r-recommended         rename         ripgrep         rsync         socat         sudo         tcllib         tcl-dev         tesseract-ocr         tk-dev         tmux         unrar         unzip         vim         xvfb         zip &&     locale-gen en_US.UTF-8 &&     update-locale LANG=en_US.UTF-8 &&     mkdir -p /tools &&     update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 1 &&     update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12 2 &&     sed -i -e '1,1s/python3/python3.12/' /usr/bin/add-apt-repository
4.87GB  <missing>       2024-07-10T23:20:17Z    RUN |1 TARGETARCH=amd64 /bin/sh -c apt-get update && apt-get install -y --no-install-recommends     cuda-cudart-dev-12-5=${NV_CUDA_CUDART_DEV_VERSION}     cuda-command-line-tools-12-5=${NV_CUDA_LIB_VERSION}     cuda-minimal-build-12-5=${NV_CUDA_LIB_VERSION}     cuda-libraries-dev-12-5=${NV_CUDA_LIB_VERSION}     cuda-nvml-dev-12-5=${NV_NVML_DEV_VERSION}     ${NV_NVPROF_DEV_PACKAGE}     ${NV_LIBNPP_DEV_PACKAGE}     libcusparse-dev-12-5=${NV_LIBCUSPARSE_DEV_VERSION}     ${NV_LIBCUBLAS_DEV_PACKAGE}     ${NV_LIBNCCL_DEV_PACKAGE}     ${NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE}     && rm -rf /var/lib/apt/lists/* # buildkit
2.05GB  <missing>       2024-07-10T22:50:58Z    RUN |1 TARGETARCH=amd64 /bin/sh -c apt-get update && apt-get install -y --no-install-recommends     cuda-libraries-12-5=${NV_CUDA_LIB_VERSION}     ${NV_LIBNPP_PACKAGE}     cuda-nvtx-12-5=${NV_NVTX_VERSION}     libcusparse-12-5=${NV_LIBCUSPARSE_VERSION}     ${NV_LIBCUBLAS_PACKAGE}     ${NV_LIBNCCL_PACKAGE}     && rm -rf /var/lib/apt/lists/* # buildkit
2.02GB  <missing>       2025-10-24T13:27:21Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c julia -e 'using Pkg; Pkg.add(["IJulia", "CSV", "DataFrames", "Makie", "Plots"]); precompile;'
1.49GB  <missing>       2025-10-24T13:31:34Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c julia -e 'using Pkg; Pkg.add(["Adapt", "Lux", "Reactant"]); precompile;' || true
1.3GB   <missing>       2025-10-24T13:21:09Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c wget -nv https://julialang-s3.julialang.org/bin/linux/x64/1.11/julia-1.11.5-linux-x86_64.tar.gz -O /tmp/julia.tar.gz &&     tar -xf /tmp/julia.tar.gz -C /usr/local --strip-components 1
895MB   <missing>       2025-10-24T13:37:33Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c wget -nv https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.zip &&     unzip -qq google-cloud-sdk.zip -d tools &&     rm google-cloud-sdk.zip &&     tools/google-cloud-sdk/install.sh --usage-reporting=false         --path-update=false --bash-completion=false &&     tools/google-cloud-sdk/bin/gcloud -q components update         gcloud core bq gsutil compute preview alpha beta &&     tools/google-cloud-sdk/bin/gcloud config set component_manager/disable_update_check true &&     tools/google-cloud-sdk/bin/gcloud config set compute/gce_metadata_read_timeout_sec 0 &&     touch -t 204001010000.00 /content/.config/gce &&     rm -rf tools/google-cloud-sdk/.install
874MB   <missing>       2024-07-10T23:22:21Z    RUN |1 TARGETARCH=amd64 /bin/sh -c apt-get update && apt-get install -y --no-install-recommends     ${NV_CUDNN_PACKAGE}     ${NV_CUDNN_PACKAGE_DEV}     && apt-mark hold ${NV_CUDNN_PACKAGE_NAME}     && rm -rf /var/lib/apt/lists/* # buildkit
715MB   <missing>       2025-10-24T13:54:40Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c ipython profile create default &&     jupyter server --generate-config &&     python3 -m ipykernel install &&     R -e 'IRkernel::installspec(user = FALSE)' &&     julia -e 'using IJulia; IJulia.installkernel("julia", specname = "julia", env=Dict("JULIA_NUM_THREADS"=>"auto"))' &&     mv /root/.local/share/jupyter/kernels/julia-1.11 /usr/local/share/jupyter/kernels/julia &&     sed -i 's/julia 1.11.5/julia/' '/usr/local/share/jupyter/kernels/julia/kernel.json' &&     mkdir -p /tools/node &&     wget -nv https://nodejs.org/dist/v20.19.0/node-v20.19.0-linux-x64.tar.gz -O node.tar.gz &&     tar xzf node.tar.gz -C /tools/node --strip-components=1 &&     rm node.tar.gz &&     npm install -g npm@v10.8.2 &&     mkdir -p /datalab/web &&     cd /backend-container/sources &&     mv * /datalab/web &&     cd /datalab/web &&     /tools/node/bin/npm ci &&     unzip /backend-container/pyright.zip -d pyright &&     npm install -g @google/gemini-cli &&     mkdir /var/colab &&     cd / && unzip /backend-container/colabtools.zip && cd /colabtools &&     python setup.py sdist &&     uv pip install         --no-cache         /colabtools/dist/google_colab-1.0.0.tar.gz &&     jupyter server extension enable --py nbclassic &&     jupyter server extension enable --py google.colab &&     jupyter server extension enable --system --py google.colab._serverextension &&     python3 -m pytest &&     python3 /backend-container/containers/update_kernelspecs.py &&     uv pip uninstall --no-cache --system jedi &&     echo 'uninstalling problematic jedi done' &&     rm -rf /tmp/* /backend-container /colabtools &&     cd /
625MB   <missing>       2025-10-24T13:36:19Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c if [[ "${COLAB_TPU_1VM}" != "1" ]]; then         julia -e 'env = "/root/.julia/environments/v1.11";                   mkpath(env);                   write("$env/LocalPreferences.toml", "[CUDA_Runtime_jll]\nversion = \"'${CUDA_VERSION}'\"\nlocal = \"true\"");                   using Pkg;                   Pkg.add("CUDA_Runtime_jll");                   pkg"add CUDA"; using CUDA; CUDA.precompile_runtime()';     fi
317MB   <missing>       2025-10-24T13:20:21Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c apt-get update &&     echo 'local({options(repos = c(CRAN = "https://cran.rstudio.com"))})' >>/etc/R/Rprofile.site &&     apt-get install --no-install-recommends -y -qq         r-cran-devtools         r-cran-tidyverse
211MB   <missing>       2025-10-24T13:12:48Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c apt-get update &&     rm -f /etc/dpkg/dpkg.cfg.d/excludes &&     apt-get install --no-install-recommends -y -qq         software-properties-common wget gnupg2
152MB   <missing>       2024-07-10T22:46:29Z    RUN |1 TARGETARCH=amd64 /bin/sh -c apt-get update && apt-get install -y --no-install-recommends     cuda-cudart-12-5=${NV_CUDA_CUDART_VERSION}     cuda-compat-12-5     && rm -rf /var/lib/apt/lists/* # buildkit
77.9MB  <missing>       2024-06-27T20:10:12Z    /bin/sh -c #(nop) ADD file:d5da92199726e42da09a6f75a778befb607fe3f79e4afaf7ef5188329b26b386 in / 
63.5MB  <missing>       2025-10-24T13:19:06Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c curl -LsSf https://astral.sh/uv/install.sh | sh &&     uv pip install         --no-cache         --requirement /backend-container/containers/requirements.constraints
56.8MB  <missing>       2025-10-24T13:37:52Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c chmod +w -R /content/sample_data &&     cd /content/sample_data &&     curl -OL https://dl.google.com/mlcc/mledu-datasets/california_housing_train.csv &&     curl -OL https://dl.google.com/mlcc/mledu-datasets/california_housing_test.csv &&     curl -OL https://dl.google.com/mlcc/mledu-datasets/mnist_train_small.csv &&     curl -OL https://dl.google.com/mlcc/mledu-datasets/mnist_test.csv
26.6MB  <missing>       2025-10-24T13:19:44Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c tar -xf python-apt.tar.xz &&     pushd python-apt &&     python3 setup.py build &&     python3 setup.py install &&     popd
22.8MB  <missing>       2025-10-24T13:13:20Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c if [[ "${COLAB_TPU_1VM}" != "1" ]]; then         /backend-container/containers/retrying.sh add-apt-repository --yes             'ppa:graphics-drivers/ppa';     fi &&     /backend-container/containers/retrying.sh add-apt-repository --yes         'ppa:deadsnakes/ppa' &&     wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc |         tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc &&     /backend-container/containers/retrying.sh add-apt-repository --yes         "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/" &&     wget -qO- https://r2u.stat.illinois.edu/ubuntu/dirk_eddelbuettel_pubkey.asc |         tee -a /etc/apt/trusted.gpg.d/r2u_key.asc &&     /backend-container/containers/retrying.sh add-apt-repository --enable-source --yes         "deb [arch=amd64] https://r2u.stat.illinois.edu/ubuntu $(lsb_release -cs) main" &&     /backend-container/containers/retrying.sh add-apt-repository --yes         'ppa:ubuntugis/ppa' &&     out=$(mktemp) && wget -nv -O$out https://cli.github.com/packages/githubcli-archive-keyring.gpg &&     cat "${out}" | tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null &&     chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg &&     mkdir -p -m 755 /etc/apt/sources.list.d &&     echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" |         tee /etc/apt/sources.list.d/github-cli.list > /dev/null &&     apt-get update
15.3MB  <missing>       2025-10-24T13:54:44Z    /bin/bash -o pipefail -c #(nop) COPY file:485df831b8044b58daeb052e392b59e6e916f125eb1996a1ece6d2a064d3022c in /usr/colab/bin/language_service 
11MB    <missing>       2025-10-24T13:54:45Z    /bin/bash -o pipefail -c #(nop) COPY file:f63757945088dc9d43194565a0aa4e503f6f1ebecf804a7ebc42fbbfed62de34 in /usr/colab/bin/kernel_manager_proxy 
10.6MB  <missing>       2024-07-10T22:46:16Z    RUN |1 TARGETARCH=amd64 /bin/sh -c apt-get update && apt-get install -y --no-install-recommends     gnupg2 curl ca-certificates &&     curl -fsSLO https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/${NVARCH}/cuda-keyring_1.1-1_all.deb &&     dpkg -i cuda-keyring_1.1-1_all.deb &&     apt-get purge --autoremove -y curl     && rm -rf /var/lib/apt/lists/* # buildkit
4.57MB  <missing>       2025-10-24T13:52:26Z    /bin/bash -o pipefail -c #(nop) COPY file:015df91e83e2a1774fb5f70ac47c706c45b497353955f145a51bb901600c216b in /backend-container/colabtools.zip 
4.06MB  <missing>       2025-10-24T13:54:43Z    /bin/bash -o pipefail -c #(nop) COPY file:7b7829f883a105cb7d559e3786779ba996ff367be034ebc4504b630575f54e0d in /usr/local/bin/dap_multiplexer 
2.33MB  <missing>       2025-10-24T13:52:27Z    /bin/bash -o pipefail -c #(nop) COPY file:f078e82601068dfd08be60c63e861d1d2c019a509babd74f4e66781a807c3ab1 in /backend-container/pyright.zip 
1.99MB  <missing>       2025-10-24T13:19:01Z    /bin/bash -o pipefail -c #(nop) COPY file:778a79bd2c394874556d5f16bcc884387d073873501cb3ec4de41cee066f2033 in /usr/grte/v5/lib64/locale/locale-archive 
1.12MB  <missing>       2025-10-24T13:20:46Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c R -e 'install.packages(c("repr", "IRdisplay", "IRkernel"), type = "source")' &&     R -e 'stopifnot(length(grep("agpl", installed.packages(), ignore.case = TRUE)) == 0)'
458kB   <missing>       2025-10-24T13:52:26Z    /bin/bash -o pipefail -c #(nop) COPY dir:aa10accd20deb0021f05ca24f11f6608d901b4786081d520b035fc57ab8714a7 in /backend-container/sources 
390kB   <missing>       2024-07-10T23:20:17Z    RUN |1 TARGETARCH=amd64 /bin/sh -c apt-mark hold ${NV_LIBCUBLAS_DEV_PACKAGE_NAME} ${NV_LIBNCCL_DEV_PACKAGE_NAME} # buildkit
346kB   <missing>       2025-10-24T13:19:06Z    /bin/bash -o pipefail -c #(nop) COPY file:c93c3fb181b5ffadff510f270f3593aa992de3d8fd9a2fc2b4270b3eeb10963d in python-apt.tar.xz 
263kB   <missing>       2024-07-10T22:50:58Z    RUN |1 TARGETARCH=amd64 /bin/sh -c apt-mark hold ${NV_LIBCUBLAS_PACKAGE_NAME} ${NV_LIBNCCL_PACKAGE_NAME} # buildkit
179kB   <missing>       2025-10-24T13:52:24Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c pushd /usr/local/lib/python3.12/dist-packages/IPython/ &&     patch -p1 < /backend-container/containers/py311_tokenize_backport.patch &&     popd
164kB   <missing>       2025-10-24T13:52:26Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c {       echo '/usr/local/lib' ;     } > /etc/ld.so.conf.d/colab.conf &&     ldconfig
98.9kB  <missing>       2025-10-24T13:37:52Z    /bin/bash -o pipefail -c #(nop) COPY multi:7a15c3b78e0c65540051dee5ebddc031890b49b14225711e8f14edb27eb5c1f4 in /backend-container/containers/ 
26.5kB  <missing>       2025-10-24T13:52:23Z    /bin/bash -o pipefail -c #(nop) COPY file:06c7afae6e572613d4643243e1583f34175d3e4d9e8303fa3d369968a63cf9f6 in /usr/local/lib/python3.12/dist-packages/IPython/lib/py311_tokenize.py 
18kB    <missing>       2025-10-24T13:37:53Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c python3 /backend-container/containers/require_pinned_requirements.py       /backend-container/containers/requirements.core.in &&     export REQUIREMENTS_FILE=requirements3.txt &&     if [[ "${COLAB_TPU_1VM}" == "1" ]]; then         export REQUIREMENTS_FILE=requirements3_tpu.txt;      fi &&     echo "Requirements file: ${REQUIREMENTS_FILE}" &&     if [[ "${COLAB_PIP_COMPILE}" == "1" ]]; then         rm /backend-container/containers/${REQUIREMENTS_FILE} &&         if [[ "${COLAB_TPU_1VM}" == "1" ]]; then             uv pip compile                 --verbose                 --upgrade                 --no-cache                 --find-links=https://storage.googleapis.com/jax-releases/libtpu_releases.html                 --find-links=https://storage.googleapis.com/libtpu-releases/index.html                 --find-links=https://storage.googleapis.com/libtpu-wheels/index.html                 --find-links=https://download.pytorch.org/whl/torch_stable.html                 --emit-find-links                 --no-emit-package pip                 --no-emit-package setuptools                 --output-file /backend-container/containers/${REQUIREMENTS_FILE}                 /backend-container/containers/requirements.core.in                 /backend-container/containers/requirements.user.in                 /backend-container/containers/requirements.tpu.in;         else             uv pip compile                 --verbose                 --upgrade                 --no-cache                 --emit-find-links                 --no-emit-package pip                 --no-emit-package setuptools                 --output-file /backend-container/containers/${REQUIREMENTS_FILE}                 /backend-container/containers/requirements.core.in                 /backend-container/containers/requirements.user.in                 /backend-container/containers/requirements.in;         fi;     fi
17.3kB  <missing>       2024-07-10T22:46:29Z    COPY NGC-DL-CONTAINER-LICENSE / # buildkit
8.69kB  <missing>       2025-10-24T13:54:43Z    /bin/bash -o pipefail -c #(nop) COPY file:3682207aec072d6574dcad95ab0452f77465b8a9dd4d6cf73b8e747d798420db in /usr/local/bin/ 
6.16kB  <missing>       2025-10-24T13:54:43Z    /bin/bash -o pipefail -c #(nop) COPY file:3d0156b109771b650783f7d676023b8a0ae245443f3e84eb6b66bc131e2de248 in /usr/local/bin/ 
3.71kB  <missing>       2025-10-24T13:52:27Z    /bin/bash -o pipefail -c #(nop) COPY file:41ea9712e0d68c47118b36f6871e8998a2fd8f2705903c47ac2ef5cc75e59ca8 in /datalab/ 
3.06kB  <missing>       2024-07-10T22:50:58Z    COPY entrypoint.d/ /opt/nvidia/entrypoint.d/ # buildkit
2.95kB  <missing>       2025-10-24T13:52:23Z    /bin/bash -o pipefail -c #(nop) COPY file:ee705e8c64dea950825cecdc80dfabe46c25bd5c990af203948223ced59bb07e in /backend-container/containers/py311_tokenize_backport.patch 
2.66kB  <missing>       2025-10-24T13:37:36Z    /bin/bash -o pipefail -c #(nop) COPY dir:fad3446edbc5f9c2127e7c88d5016d30439c738bb221f63c6f52a35704ed5131 in /content/sample_data 
2.53kB  <missing>       2024-07-10T22:50:58Z    COPY nvidia_entrypoint.sh /opt/nvidia/ # buildkit
2.09kB  <missing>       2025-10-24T13:52:27Z    /bin/bash -o pipefail -c #(nop) COPY file:1f6e83fcc0f425bb1f6648ba682eb4d96b71b5536408501af87a63ecaf28b1a0 in /etc/ipython/ipython_config.py 
1.89kB  <missing>       2025-10-24T13:52:26Z    /bin/bash -o pipefail -c #(nop) COPY file:986599674cfa20d7412b36c784c21062024f3ed221414f3ff2d2b2a10d7e7179 in /etc/jupyter/ 
1.48kB  <missing>       2025-10-24T13:52:27Z    /bin/bash -o pipefail -c #(nop) COPY file:72f07be42222b55d55791e2615a6b52f6885fa86312f4e35dfa696c8ea4e0373 in /backend-container/containers/ 
1.45kB  <missing>       2025-10-24T13:54:45Z    /bin/bash -o pipefail -c #(nop) COPY file:449cb0cd152ea14ab688051363372bd5b75b623b9629b10d3692b3744b2bb8d7 in /usr/local/colab/bin/ 
1.36kB  <missing>       2025-10-24T13:52:27Z    /bin/bash -o pipefail -c #(nop) COPY file:fd538c60ab58911429ba14debbafd873951a903e2588786d1d5915af82d09071 in /backend-container/ 
1.34kB  <missing>       2025-10-24T13:54:43Z    /bin/bash -o pipefail -c #(nop) COPY file:d138c03ba730a9cbc1aab19322ceaa59f11d8ab13eb534403ccd534f17c557c6 in /usr/local/bin/colab_diagnose_me 
1.16kB  <missing>       2025-06-24T13:14:34Z    /bin/bash -o pipefail -c #(nop) COPY file:a00c06c317a595406ea6277f7f769df8e4b71da12f63dde99a0649aa1354cbe9 in /backend-container/containers/ 
1.07kB  <missing>       2025-10-24T13:54:44Z    /bin/bash -o pipefail -c #(nop) COPY file:970e55d4a3689467c605712adb7b084bc02d0a401e69398c5ae0b2c82ed80ffe in /root/.bashrc 
357B    <missing>       2025-10-24T13:37:52Z    /bin/bash -o pipefail -c #(nop) COPY file:043943da3ac8b166378a4342b9835110d59de9a9e121beee7eedfd85cea612ff in /etc/ 
254B    <missing>       2025-10-24T13:54:44Z    /bin/bash -o pipefail -c #(nop) COPY file:c6467618ded482efe5080c32135c0d350ad44d10165b40aa7b0f540f30c1dd44 in /root/.tmux.conf 
195B    <missing>       2025-10-24T13:19:01Z    /bin/bash -o pipefail -c #(nop) COPY file:aa34963a155c947ab427480ebf40370b0efb8437fd0faa79a26b9bb17810c91d in /backend-container/containers/ 
162B    <missing>       2025-10-24T13:19:01Z    /bin/bash -o pipefail -c #(nop) COPY file:03c28878188f6da017439637bf91d5cfead17de2d7f2b96b7091a3b73aabfb29 in /backend-container/containers/ 
155B    <missing>       2025-10-24T13:37:36Z    /bin/bash -o pipefail -c #(nop) COPY file:7c86ae5c29200bbcfe8eb1e66a782ffd59e61baae7b8207bb3de317f3762b6ae in /usr/local/bin/python 
155B    <missing>       2025-10-24T13:37:36Z    /bin/bash -o pipefail -c #(nop) COPY file:7c86ae5c29200bbcfe8eb1e66a782ffd59e61baae7b8207bb3de317f3762b6ae in /usr/local/bin/pip 
150B    <missing>       2025-10-24T13:37:35Z    /bin/bash -o pipefail -c #(nop) COPY file:5c02dad9e444ea87f4386bf9299ab71e1f28f3ff5681c815b8a9f9da3341d619 in /etc/apt/apt.conf.d/ 
57B     <missing>       2025-10-24T13:19:45Z    /bin/bash -o pipefail -c #(nop) COPY file:82290c1ae07d98579bac01b736d1c5f486fe538210bf4002fd51b2ad975b2054 in /usr/local/bin/R 
46B     <missing>       2024-07-10T22:46:29Z    RUN |1 TARGETARCH=amd64 /bin/sh -c echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf     && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf # buildkit
0B      sha256:d5458217b23549186bb7d0b5e67f4eef0bd82ca12f84c3a341e7a46aa148b02a 2025-10-24T14:11:20Z    /bin/bash -o pipefail -c /usr/local/bin/colab_diagnose_me
0B      <missing>       2025-10-24T14:11:19Z    /bin/bash -o pipefail -c #(nop)  ENV COLAB_JUPYTER_ALLOW_ORIGIN_PAT=https://colab\.(sandbox|research)\.google\.com
0B      <missing>       2025-10-24T14:11:19Z    /bin/bash -o pipefail -c #(nop)  ENV COLAB_HUMAN_READABLE_NODE_LOGS=1
0B      <missing>       2025-10-24T13:54:46Z    /bin/bash -o pipefail -c #(nop)  ENV SHELL=/bin/bash
0B      <missing>       2025-10-24T13:54:46Z    /bin/bash -o pipefail -c #(nop)  ENV COLAB_TPU_1VM=
0B      <missing>       2025-10-24T13:54:46Z    /bin/bash -o pipefail -c #(nop)  ENV COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00
0B      <missing>       2025-10-24T13:54:46Z    /bin/bash -o pipefail -c #(nop)  ENTRYPOINT ["/datalab/run.sh"]
0B      <missing>       2025-10-24T13:54:45Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c [[ $(find /usr/lib/jvm -maxdepth 1 -mindepth 1 -type d |wc -l) = 1 ]]
0B      <missing>       2025-10-24T13:54:45Z    /bin/bash -o pipefail -c #(nop)  ENV ENV=/root/.bashrc
0B      <missing>       2025-10-24T13:54:44Z    /bin/bash -o pipefail -c #(nop)  ENV VM_GCE_METADATA_HOST=169.254.169.254
0B      <missing>       2025-10-24T13:54:44Z    /bin/bash -o pipefail -c #(nop)  ENV KMP_TARGET_PORT=9000
0B      <missing>       2025-10-24T13:54:44Z    /bin/bash -o pipefail -c #(nop)  ENV KMP_LISTEN_PORT=6000
0B      <missing>       2025-10-24T13:54:44Z    /bin/bash -o pipefail -c #(nop)  ENV COLAB_KERNEL_MANAGER_PROXY_PORT=6000
0B      <missing>       2025-10-24T13:54:44Z    /bin/bash -o pipefail -c #(nop)  ENV COLAB_FILE_HANDLER_ADDR=localhost:3453
0B      <missing>       2025-10-24T13:54:43Z    /bin/bash -o pipefail -c #(nop)  ENV UV_CONSTRAINT=
0B      <missing>       2025-10-24T13:54:43Z    /bin/bash -o pipefail -c #(nop)  ENV UV_BUILD_CONSTRAINT=
0B      <missing>       2025-10-24T13:19:02Z    /bin/bash -o pipefail -c #(nop)  ENV UV_SYSTEM_PYTHON=true
0B      <missing>       2025-10-24T13:19:01Z    |2 COLAB_PIP_COMPILE= COLAB_RELEASE_TAG=release-colab-external_20251024-060052_RC00 /bin/bash -o pipefail -c apt-get autoremove -y &&     apt-get clean &&     rm -rf /usr/local/src/*
0B      <missing>       2025-10-24T13:19:01Z    /bin/bash -o pipefail -c #(nop)  ENV UV_INSTALL_DIR=/usr/local/bin
0B      <missing>       2025-10-24T13:19:01Z    /bin/bash -o pipefail -c #(nop)  ENV UV_CONSTRAINT=/backend-container/containers/requirements.constraints
0B      <missing>       2025-10-24T13:19:01Z    /bin/bash -o pipefail -c #(nop)  ENV UV_BUILD_CONSTRAINT=/backend-container/containers/build.constraints
0B      <missing>       2025-10-24T13:19:01Z    /bin/bash -o pipefail -c #(nop)  ENV TCLLIBPATH=/usr/share/tcltk/tcllib1.20
0B      <missing>       2025-06-24T13:14:34Z    /bin/sh -c #(nop)  ENV COLAB_WARMUP_DEFAULTS=1
0B      <missing>       2025-06-24T13:14:34Z    /bin/sh -c #(nop)  ENV COLAB_LANGUAGE_SERVER_PROXY=/usr/colab/bin/language_service
0B      <missing>       2025-06-24T13:14:34Z    /bin/sh -c #(nop)  ENV COLAB_JUPYTER_IP=127.0.0.1
0B      <missing>       2025-06-24T13:14:34Z    /bin/sh -c #(nop)  ENV COLAB_DEBUG_ADAPTER_MUX_PATH=/usr/local/bin/dap_multiplexer
0B      <missing>       2025-06-24T13:14:34Z    /bin/bash -o pipefail -c #(nop)  SHELL [/bin/bash -o pipefail -c]
0B      <missing>       2025-06-24T13:14:33Z    /bin/sh -c #(nop)  ENV LAST_FORCED_REBUILD=20250623
0B      <missing>       2025-03-26T13:13:50Z    /bin/sh -c #(nop)  ENV PYTHONUTF8=1
0B      <missing>       2025-01-28T14:13:14Z    /bin/sh -c #(nop)  EXPOSE 8080
0B      <missing>       2025-01-28T14:13:14Z    /bin/sh -c #(nop)  ENV TF_FORCE_GPU_ALLOW_GROWTH=true
0B      <missing>       2025-01-28T14:13:14Z    /bin/sh -c #(nop)  ENV PYTHONPATH=/env/python
0B      <missing>       2025-01-28T14:13:14Z    /bin/sh -c #(nop)  ENV PATH=/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin
0B      <missing>       2025-01-28T14:13:14Z    /bin/sh -c #(nop)  ENV LC_ALL=en_US.UTF-8
0B      <missing>       2025-01-28T14:13:14Z    /bin/sh -c #(nop)  ENV LANGUAGE=en_US
0B      <missing>       2025-01-28T14:13:14Z    /bin/sh -c #(nop)  ENV LANG=en_US.UTF-8
0B      <missing>       2025-01-28T14:13:14Z    /bin/sh -c #(nop)  ENV DEBIAN_FRONTEND=noninteractive
0B      <missing>       2025-01-28T14:13:14Z    /bin/sh -c #(nop)  ENV CLOUDSDK_PYTHON=python3
0B      <missing>       2025-01-28T14:13:14Z    /bin/sh -c #(nop)  ENV CLOUDSDK_CONFIG=/content/.config
0B      <missing>       2025-01-28T14:13:13Z    /bin/sh -c #(nop)  LABEL maintainer=Google Colaboratory Team <colaboratory-team@google.com>
0B      <missing>       2025-01-28T14:13:13Z    /bin/sh -c #(nop)  ARG COLAB_TPU_1VM
0B      <missing>       2025-01-28T14:13:13Z    /bin/sh -c #(nop)  ARG COLAB_RELEASE_TAG
0B      <missing>       2025-01-28T14:13:13Z    /bin/sh -c #(nop)  ARG COLAB_PIP_COMPILE
0B      <missing>       2024-07-10T23:22:21Z    LABEL maintainer=NVIDIA CORPORATION <cudatools@nvidia.com>
0B      <missing>       2024-07-10T23:22:21Z    LABEL com.nvidia.cudnn.version=9.2.1.18-1
0B      <missing>       2024-07-10T23:22:21Z    ENV NV_CUDNN_VERSION=9.2.1.18-1
0B      <missing>       2024-07-10T23:22:21Z    ENV NV_CUDNN_PACKAGE_NAME=libcudnn9-cuda-12
0B      <missing>       2024-07-10T23:22:21Z    ENV NV_CUDNN_PACKAGE_DEV=libcudnn9-dev-cuda-12=9.2.1.18-1
0B      <missing>       2024-07-10T23:22:21Z    ENV NV_CUDNN_PACKAGE=libcudnn9-cuda-12=9.2.1.18-1
0B      <missing>       2024-07-10T23:22:21Z    ARG TARGETARCH
0B      <missing>       2024-07-10T23:20:17Z    LABEL maintainer=NVIDIA CORPORATION <cudatools@nvidia.com>
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_NVPROF_VERSION=12.5.82-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-5=12.5.82-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_NVML_DEV_VERSION=12.5.82-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_LIBNPP_DEV_VERSION=12.3.0.159-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-5=12.3.0.159-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.22.3-1+cuda12.5
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_LIBCUSPARSE_DEV_VERSION=12.5.1.3-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_LIBCUBLAS_DEV_VERSION=12.5.3.2-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-5
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-5=12.5.3.2-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_CUDA_NSIGHT_COMPUTE_VERSION=12.5.1-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-5=12.5.1-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_CUDA_LIB_VERSION=12.5.1-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NV_CUDA_CUDART_DEV_VERSION=12.5.82-1
0B      <missing>       2024-07-10T23:20:17Z    ENV NCCL_VERSION=2.22.3-1
0B      <missing>       2024-07-10T23:20:17Z    ENV LIBRARY_PATH=/usr/local/cuda/lib64/stubs
0B      <missing>       2024-07-10T23:20:17Z    ARG TARGETARCH
0B      <missing>       2024-07-10T22:50:58Z    LABEL maintainer=NVIDIA CORPORATION <cudatools@nvidia.com>
0B      <missing>       2024-07-10T22:50:58Z    ENV NV_NVTX_VERSION=12.5.82-1
0B      <missing>       2024-07-10T22:50:58Z    ENV NV_LIBNPP_VERSION=12.3.0.159-1
0B      <missing>       2024-07-10T22:50:58Z    ENV NV_LIBNPP_PACKAGE=libnpp-12-5=12.3.0.159-1
0B      <missing>       2024-07-10T22:50:58Z    ENV NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1
0B      <missing>       2024-07-10T22:50:58Z    ENV NV_LIBNCCL_PACKAGE_NAME=libnccl2
0B      <missing>       2024-07-10T22:50:58Z    ENV NV_LIBNCCL_PACKAGE=libnccl2=2.22.3-1+cuda12.5
0B      <missing>       2024-07-10T22:50:58Z    ENV NV_LIBCUSPARSE_VERSION=12.5.1.3-1
0B      <missing>       2024-07-10T22:50:58Z    ENV NV_LIBCUBLAS_VERSION=12.5.3.2-1
0B      <missing>       2024-07-10T22:50:58Z    ENV NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-5
0B      <missing>       2024-07-10T22:50:58Z    ENV NV_LIBCUBLAS_PACKAGE=libcublas-12-5=12.5.3.2-1
0B      <missing>       2024-07-10T22:50:58Z    ENV NV_CUDA_LIB_VERSION=12.5.1-1
0B      <missing>       2024-07-10T22:50:58Z    ENV NVIDIA_PRODUCT_NAME=CUDA
0B      <missing>       2024-07-10T22:50:58Z    ENV NCCL_VERSION=2.22.3-1
0B      <missing>       2024-07-10T22:50:58Z    ENTRYPOINT ["/opt/nvidia/nvidia_entrypoint.sh"]
0B      <missing>       2024-07-10T22:50:58Z    ARG TARGETARCH
0B      <missing>       2024-07-10T22:46:29Z    ENV PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
0B      <missing>       2024-07-10T22:46:29Z    ENV NVIDIA_VISIBLE_DEVICES=all
0B      <missing>       2024-07-10T22:46:29Z    ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
0B      <missing>       2024-07-10T22:46:29Z    ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
0B      <missing>       2024-07-10T22:46:16Z    LABEL maintainer=NVIDIA CORPORATION <cudatools@nvidia.com>
0B      <missing>       2024-07-10T22:46:16Z    ENV NV_CUDA_CUDART_VERSION=12.5.82-1
0B      <missing>       2024-07-10T22:46:16Z    ENV NVIDIA_REQUIRE_CUDA=cuda>=12.5 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551
0B      <missing>       2024-07-10T22:46:16Z    ENV NVARCH=x86_64
0B      <missing>       2024-07-10T22:46:16Z    ENV CUDA_VERSION=12.5.1
0B      <missing>       2024-07-10T22:46:16Z    ARG TARGETARCH
0B      <missing>       2024-06-27T20:10:12Z    /bin/sh -c #(nop)  CMD ["/bin/bash"]
0B      <missing>       2024-06-27T20:10:10Z    /bin/sh -c #(nop)  LABEL org.opencontainers.image.version=22.04
0B      <missing>       2024-06-27T20:10:10Z    /bin/sh -c #(nop)  LABEL org.opencontainers.image.ref.name=ubuntu
0B      <missing>       2024-06-27T20:10:10Z    /bin/sh -c #(nop)  ARG RELEASE
0B      <missing>       2024-06-27T20:10:10Z    /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH

Here is the breakdown of the “disk budget”:

• The ML Core (20.2 GB): The single largest layer. It contains the entire Python data science ecosystem (TensorFlow, PyTorch, GDAL), installed via uv for speed, with custom GPU-compiled extensions (like LightGBM).
• The “GPU Tax” (~7 GB): Split between runtime libraries (2GB) and development headers (4.9GB), allowing users to compile custom CUDA kernels inside the notebook.
• The System OS (5 GB): A heavy apt-get layer that turns the container into a full development machine, installing compilers (clang, gcc), generic tools (ffmpeg, git), and other languages (R, Java).
• The Julia Stack (~5 GB): A surprising amount of space is dedicated to the Julia language, including the runtime (1.3GB) and pre-installed plotting/ML libraries (3.5GB).

When you start a Colab Runtime you can see the specifics via Terminal

df -H
Filesystem      Size  Used Avail Use% Mounted on
overlay         121G   42G   80G  35% /
tmpfs            68M     0   68M   0% /dev
shm             6.2G     0  6.2G   0% /dev/shm
/dev/root       2.1G  1.3G  787M  62% /usr/sbin/docker-init
tmpfs           6.9G   50k  6.9G   1% /var/colab
/dev/sda1        80G   44G   36G  56% /kaggle/input
tmpfs           6.9G     0  6.9G   0% /proc/acpi
tmpfs           6.9G     0  6.9G   0% /proc/scsi
tmpfs           6.9G     0  6.9G   0% /sys/firmware

You can use `ncdu` tool to get more insights of disk space:

apt-get update && apt-get install -y ncdu
ncdu /

ncdu 1.15.1 ~ Use the arrow keys to navigate, press ? for help                                                                 
--- / -------------------------------------------------------------------------------------------------------------------------
   30.7 GiB [######### ] /usr                                                                                                  
    4.3 GiB [#         ] /root 
    1.7 GiB [          ] /tools
    1.2 GiB [          ] /opt
  135.2 MiB [          ] /var  
   54.3 MiB [          ] /content
   50.7 MiB [          ] /datalab
   20.0 MiB [          ] /python-apt
    3.4 MiB [          ] /etc  
  340.0 KiB [          ]  python-apt.tar.xz
   84.0 KiB [          ] /run 
   40.0 KiB [          ] /tmp
   20.0 KiB [          ]  NGC-DL-CONTAINER-LICENSE
    8.0 KiB [          ] /kaggle
    8.0 KiB [          ]  cuda-keyring_1.1-1_all.deb
e   4.0 KiB [          ] /srv    
e   4.0 KiB [          ] /mnt  
e   4.0 KiB [          ] /media
e   4.0 KiB [          ] /home
e   4.0 KiB [          ] /boot
    0.0   B [          ] /sys
.   0.0   B [          ] /proc
    0.0   B [          ] /dev
@   0.0   B [          ]  libx32
@   0.0   B [          ]  lib64
@   0.0   B [          ]  lib32
@   0.0   B [          ]  sbin
@   0.0   B [          ]  lib
@   0.0   B [          ]  bin
    0.0   B [          ]  .dockerenv

 Total disk usage:  38.3 GiB  Apparent size:  37.9 GiB  Items: 520162 

ncdu 1.15.1 ~ Use the arrow keys to navigate, press ? for help                                                                 
--- /usr ----------------------------------------------------------------------------------------------------------------------
                         /..                                                                                                   
   24.3 GiB [##########] /local
    4.9 GiB [#         ] /lib  
  436.3 MiB [          ] /lib64-nvidia
  361.1 MiB [          ] /share
  342.2 MiB [          ] /bin
  254.0 MiB [          ] /include
  140.1 MiB [          ] /src
   28.9 MiB [          ] /colab
    8.6 MiB [          ] /sbin
    1.9 MiB [          ] /grte
    1.6 MiB [          ] /libexec
   20.0 KiB [          ] /etc
    4.0 KiB [          ] /lib64 
e   4.0 KiB [          ] /libx32
e   4.0 KiB [          ] /lib32  
e   4.0 KiB [          ] /games

 Total disk usage:  30.7 GiB  Apparent size:  30.1 GiB  Items: 316027

Analyzing the `dist-packages` folder.

ncdu 1.15.1 ~ Use the arrow keys to navigate, press ? for help                                                                 
--- /usr/local/lib/python3.12/dist-packages -----------------------------------------------------------------------------------
                         /..
    3.6 GiB [##########] /nvidia                                                                                               
    2.1 GiB [#####     ] /libcugraph
    1.7 GiB [####      ] /tensorflow
    1.6 GiB [####      ] /torch
  926.7 MiB [##        ] /libcudf
  682.7 MiB [#         ] /libcuml
  591.3 MiB [#         ] /triton
  455.3 MiB [#         ] /pyspark
  370.2 MiB [#         ] /jax_plugins
  305.4 MiB [          ] /jaxlib
  207.5 MiB [          ] /xgboost
  196.8 MiB [          ] /gradio
  186.3 MiB [          ] /cupy
  162.2 MiB [          ] /google
  128.3 MiB [          ] /llvmlite
  126.7 MiB [          ] /pyarrow
  118.7 MiB [          ] /cuda
  116.5 MiB [          ] /spacy
  113.9 MiB [          ] /panel
  106.3 MiB [          ] /polars
  100.6 MiB [          ] /music21
   98.3 MiB [          ] /opencv_python.libs
   98.3 MiB [          ] /opencv_contrib_python.libs
   97.3 MiB [          ] /scipy
   94.6 MiB [          ] /plotly
   91.6 MiB [          ] /googleapiclient
   86.2 MiB [          ] /libucx
   79.8 MiB [          ] /cv2
   76.9 MiB [          ] /pyogrio.libs
   76.6 MiB [          ] /cmake
   76.2 MiB [          ] /imageio_ffmpeg
 Total disk usage:  17.6 GiB  Apparent size:  17.2 GiB  Items: 187914

In the case of Colab Enterprise Runtimes, the image container is baked into the OS, hence uses disk space into the boot disk to prevent delays during startup time. The boot disk is a persistent SSD PD that improves startup time performance, compared to just Standard PD.

If you want to see what exactly happens when the container starts you can use `strace`, in this case using the entrypoint `/datalab/run.sh` we analyze the files read and write:

docker run --name colab-startup-strace --entrypoint /bin/bash us-docker.pkg.dev/colab-images/public/runtime:latest -c "apt-get update && apt-get install -y strace && strace -f -T -e trace=openat,execve -o /tmp/startup_trace.txt /datalab/run.sh"

To extract a copy of file to local server use:

docker cp colab-startup-strace:/tmp/startup_trace.txt ./startup_trace.txt

Now you can start identify potential delays during Container startup:

[2025-12-29T00:03:54.333Z]  INFO: app/776 on 9e3e2d1d9f84: app: starting with settings: {"contentDir":"/content","datalabRoot":"","jupyterArgs":["--debug","--ServerApp.allow_origin_pat=\"https://colab\\.(sandbox|research)\\.google\\.com\"","--ServerApp.allow_origin=","--ServerApp.allow_credentials=True","--ip=127.0.0.1"],"nextJupyterPort":9000,"serverHost":"","serverPort":8080,"languageServerProxy":"/usr/colab/bin/language_service","debugAdapterMultiplexerPath":"/usr/local/bin/dap_multiplexer","kernelManagerProxyPort":6000,"fileHandlerAddr":"localhost:3453"}
[2025-12-29T00:03:54.355Z]  INFO: app/776 on 9e3e2d1d9f84: Launching Jupyter server at 9000
[2025-12-29T00:03:54.356Z]  INFO: app/776 on 9e3e2d1d9f84: Using jupyter server address 127.0.0.1
[2025-12-29T00:03:54.366Z]  INFO: app/776 on 9e3e2d1d9f84: Jupyter process started with pid 864 and args ["server","--debug","--ServerApp.allow_origin_pat=\"https://colab\\.(sandbox|research)\\.google\\.com\"","--ServerApp.allow_origin=","--ServerApp.allow_credentials=True","--ip=127.0.0.1","--port=9000","--FileContentsManager.root_dir=/","--FileContentsManager.allow_hidden=True","--ServerApp.log_format=\"|%(levelname)s|%(message)s\"","--ServerApp.iopub_data_rate_limit=1e10","--MappingKernelManager.root_dir=/content"]
[2025-12-29T00:03:54.391Z]  INFO: app/776 on 9e3e2d1d9f84: Starting server at http://localhost:8080
[2025-12-29T00:03:54.448Z]  INFO: app/776 on 9e3e2d1d9f84:
    /usr/local/bin/dap_multiplexer: DAP multiplexer listening on /tmp/debugger_2fez5h8cl0
    
2025-12-29 00:03:55.663793: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1766966635.962661     839 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1766966636.054792     839 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1766966636.774572     839 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1766966636.774997     839 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1766966636.775086     839 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1766966636.775147     839 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
2025-12-29 00:03:56.848269: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2025-12-29T00:03:57.067Z]  WARN: app/776 on 9e3e2d1d9f84:   warn( (type=jupyter)
[2025-12-29T00:03:57.069Z]  WARN: app/776 on 9e3e2d1d9f84:   warn( (type=jupyter)
[2025-12-29T00:03:57.069Z]  WARN: app/776 on 9e3e2d1d9f84: ServerApp.iopub_data_rate_limit config is deprecated in 2.0. Use ZMQChannelsWebsocketConnection.iopub_data_rate_limit. (type=jupyter)
[2025-12-29T00:03:57.877Z]  WARN: app/776 on 9e3e2d1d9f84:      /etc/jupyter/jupyter_server_config.json (type=jupyter)
[2025-12-29T00:03:57.879Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_server_config.d/google.colab._serverextension.json (type=jupyter)
[2025-12-29T00:03:57.879Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_server_config.d/ipyparallel.json (type=jupyter)
[2025-12-29T00:03:57.879Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json (type=jupyter)
[2025-12-29T00:03:57.879Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_server_config.d/jupytext.json (type=jupyter)
[2025-12-29T00:03:57.880Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_server_config.d/nbclassic.json (type=jupyter)
[2025-12-29T00:03:57.880Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_server_config.d/notebook_shim.json (type=jupyter)
[2025-12-29T00:03:57.880Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_server_config.d/panel-client-jupyter.json (type=jupyter)
[2025-12-29T00:03:57.880Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_server_config.json (type=jupyter)
[2025-12-29T00:03:57.893Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/etc/jupyter/jupyter_server_config.d/google.colab.json (type=jupyter)
[2025-12-29T00:03:57.893Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/etc/jupyter/jupyter_server_config.d/nbclassic.json (type=jupyter)
[2025-12-29T00:03:57.893Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/etc/jupyter/jupyter_server_config.json (type=jupyter)
[2025-12-29T00:03:57.897Z]  WARN: app/776 on 9e3e2d1d9f84:      /root/.local/etc/jupyter/jupyter_server_config.json (type=jupyter)
[2025-12-29T00:03:57.898Z]  WARN: app/776 on 9e3e2d1d9f84:      /root/.jupyter/jupyter_server_config.json (type=jupyter)
[2025-12-29T00:03:57.904Z]  WARN: app/776 on 9e3e2d1d9f84: /usr/local/lib/python3.12/dist-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The `version` property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://events.jupyter.org/jupyter_server/contents_service/v1 (type=jupyter)
[2025-12-29T00:03:57.904Z]  WARN: app/776 on 9e3e2d1d9f84:   validate_schema(_schema) (type=jupyter)
[2025-12-29T00:03:57.908Z]  WARN: app/776 on 9e3e2d1d9f84: /usr/local/lib/python3.12/dist-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The `version` property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://events.jupyter.org/jupyter_server/gateway_client/v1 (type=jupyter)
[2025-12-29T00:03:57.908Z]  WARN: app/776 on 9e3e2d1d9f84:   validate_schema(_schema) (type=jupyter)
[2025-12-29T00:03:57.913Z]  WARN: app/776 on 9e3e2d1d9f84: /usr/local/lib/python3.12/dist-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The `version` property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://events.jupyter.org/jupyter_server/kernel_actions/v1 (type=jupyter)
[2025-12-29T00:03:57.913Z]  WARN: app/776 on 9e3e2d1d9f84:   validate_schema(_schema) (type=jupyter)
[2025-12-29T00:03:58.142Z]  INFO: app/776 on 9e3e2d1d9f84: Extension package ipyparallel took 0.2273s to import (type=jupyter)
[2025-12-29T00:03:58.142Z]  WARN: app/776 on 9e3e2d1d9f84: A `_jupyter_server_extension_points` function was not found in ipyparallel. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server. (type=jupyter)
[2025-12-29T00:03:58.740Z]  INFO: app/776 on 9e3e2d1d9f84: Extension package jupyterlab_jupytext took 0.5650s to import (type=jupyter)
[2025-12-29T00:03:58.764Z]  WARN: app/776 on 9e3e2d1d9f84: A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server. (type=jupyter)
[2025-12-29T00:04:00.872Z]  INFO: app/776 on 9e3e2d1d9f84: Extension package panel.io.jupyter_server_extension took 2.1041s to import (type=jupyter)
[2025-12-29T00:04:00.872Z]  INFO: app/776 on 9e3e2d1d9f84: google.colab  (type=jupyter)
[2025-12-29T00:04:00.872Z]  INFO: app/776 on 9e3e2d1d9f84: google.colab._serverextension  (type=jupyter)
[2025-12-29T00:04:00.873Z]  INFO: app/776 on 9e3e2d1d9f84: ipyparallel  (type=jupyter)
[2025-12-29T00:04:00.884Z]  INFO: app/776 on 9e3e2d1d9f84: jupyter_server_terminals  (type=jupyter)
[2025-12-29T00:04:00.885Z]  INFO: app/776 on 9e3e2d1d9f84: jupyterlab_jupytext  (type=jupyter)
[2025-12-29T00:04:00.896Z]  INFO: app/776 on 9e3e2d1d9f84: nbclassic  (type=jupyter)
[2025-12-29T00:04:00.902Z]  INFO: app/776 on 9e3e2d1d9f84: Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret (type=jupyter)
[2025-12-29T00:04:00.909Z]  WARN: app/776 on 9e3e2d1d9f84:      /root/.jupyter/jupyter_notebook_config.json (type=jupyter)
[2025-12-29T00:04:00.910Z]  WARN: app/776 on 9e3e2d1d9f84:      /etc/jupyter/jupyter_notebook_config.json (type=jupyter)
[2025-12-29T00:04:00.912Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_notebook_config.d/ipyparallel.json (type=jupyter)
[2025-12-29T00:04:00.912Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_notebook_config.d/jupytext.json (type=jupyter)
[2025-12-29T00:04:00.912Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json (type=jupyter)
[2025-12-29T00:04:00.912Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/local/etc/jupyter/jupyter_notebook_config.json (type=jupyter)
[2025-12-29T00:04:00.918Z]  WARN: app/776 on 9e3e2d1d9f84:      /usr/etc/jupyter/jupyter_notebook_config.json (type=jupyter)
[2025-12-29T00:04:00.918Z]  WARN: app/776 on 9e3e2d1d9f84:      /root/.local/etc/jupyter/jupyter_notebook_config.json (type=jupyter)
[2025-12-29T00:04:00.919Z]  WARN: app/776 on 9e3e2d1d9f84:      /root/.jupyter/jupyter_notebook_config.json (type=jupyter)
[2025-12-29T00:04:00.919Z]  INFO: app/776 on 9e3e2d1d9f84: notebook_shim  (type=jupyter)
[2025-12-29T00:04:00.920Z]  INFO: app/776 on 9e3e2d1d9f84: panel.io.jupyter_server_extension  (type=jupyter)
[2025-12-29T00:04:01.504Z]  WARN: app/776 on 9e3e2d1d9f84: Customizing authentication via ServerApp.login_handler_class=<class 'google.colab._login_handler.ColabLoginHandler'> is deprecated in Jupyter Server 2.0. Use ServerApp.identity_provider_class. Falling back on legacy authentication. (type=jupyter)
[2025-12-29T00:04:01.601Z]  INFO: app/776 on 9e3e2d1d9f84: notebook_shim  (type=jupyter)
[2025-12-29T00:04:01.601Z]  INFO: app/776 on 9e3e2d1d9f84: google.colab  (type=jupyter)
[2025-12-29T00:04:01.616Z]  INFO: app/776 on 9e3e2d1d9f84: google.colab server extension initialized on 9e3e2d1d9f84. (type=jupyter)
[2025-12-29T00:04:01.616Z]  INFO: app/776 on 9e3e2d1d9f84: google.colab._serverextension  (type=jupyter)
[2025-12-29T00:04:01.626Z]  INFO: app/776 on 9e3e2d1d9f84: Loading IPython parallel extension (type=jupyter)
[2025-12-29T00:04:01.627Z]  INFO: app/776 on 9e3e2d1d9f84: ipyparallel  (type=jupyter)
[2025-12-29T00:04:01.629Z]  INFO: app/776 on 9e3e2d1d9f84: jupyter_server_terminals  (type=jupyter)
[2025-12-29T00:04:01.630Z]  INFO: app/776 on 9e3e2d1d9f84: [Jupytext Server Extension] Deriving an AsyncTextFileContentsManager from AsyncLargeFileManager (type=jupyter)
[2025-12-29T00:04:01.631Z]  INFO: app/776 on 9e3e2d1d9f84: jupyterlab_jupytext  (type=jupyter)
[2025-12-29T00:04:01.664Z]  INFO: app/776 on 9e3e2d1d9f84: nbclassic  (type=jupyter)
[2025-12-29T00:04:01.665Z]  INFO: app/776 on 9e3e2d1d9f84: panel.io.jupyter_server_extension  (type=jupyter)
[2025-12-29T00:04:01.666Z]  INFO: app/776 on 9e3e2d1d9f84: Serving notebooks from local directory: / (type=jupyter)
[2025-12-29T00:04:01.666Z]  INFO: app/776 on 9e3e2d1d9f84: Jupyter Server 2.14.0 is running at: (type=jupyter)
[2025-12-29T00:04:01.667Z]  INFO: app/776 on 9e3e2d1d9f84: http://127.0.0.1:9000/?token=e66132c0813a6756c1eaefd4249be89237d2a07455cc8289 (type=jupyter)
[2025-12-29T00:04:01.667Z]  INFO: app/776 on 9e3e2d1d9f84:     http://127.0.0.1:9000/?token=e66132c0813a6756c1eaefd4249be89237d2a07455cc8289 (type=jupyter)
[2025-12-29T00:04:01.667Z]  INFO: app/776 on 9e3e2d1d9f84: Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). (type=jupyter)
[2025-12-29T00:04:01.674Z] ERROR: app/776 on 9e3e2d1d9f84:  (type=jupyter)
[2025-12-29T00:04:01.675Z]  WARN: app/776 on 9e3e2d1d9f84:     To access the server, open this file in a browser: (type=jupyter)
[2025-12-29T00:04:01.675Z]  WARN: app/776 on 9e3e2d1d9f84:         file:///root/.local/share/jupyter/runtime/jpserver-864-open.html (type=jupyter)
[2025-12-29T00:04:01.675Z]  WARN: app/776 on 9e3e2d1d9f84:     Or copy and paste one of these URLs: (type=jupyter)
[2025-12-29T00:04:01.675Z]  WARN: app/776 on 9e3e2d1d9f84:         http://127.0.0.1:9000/?token=e66132c0813a6756c1eaefd4249be89237d2a07455cc8289 (type=jupyter)
[2025-12-29T00:04:01.675Z]  WARN: app/776 on 9e3e2d1d9f84:         http://127.0.0.1:9000/?token=e66132c0813a6756c1eaefd4249be89237d2a07455cc8289 (type=jupyter)

During the boot-up process you see the following:

The panel Extension (2.1s)

The log explicitly states: Extension package panel.io.jupyter_server_extension took 2.1041s to import. Panel is a high-level dashboarding library. To initialize, it likely imports Bokeh, Param, and potentially parts of Pandas or Matplotlib. If you are not strictly using Panel for dashboards in every single session, this is expensive "bloat" to pay on every startup.

The Duplicate Registration Errors (CUDA)

The logs Unable to register cuDNN factory... when one has already been registered indicate that your TensorFlow/XLA binary might be statically linking libraries that are also being dynamically loaded (or linked twice). This forces the CPU to verify symbols and handle errors, contributing to the 2.4s gap before TensorFlow is ready.

Check for Python open files

Count successes (Found the file) vs Failures (File not found)
echo "Total python opens: $(grep "openat" ./startup_trace.txt | grep ".py" | wc -l)"
echo "Failed opens (ENOENT): $(grep "openat" ./startup_trace.txt | grep ".py" | grep "ENOENT" | wc -l)"
Total python opens: 16559
Failed opens (ENOENT): 3408

The Data:

• Total Filesystem Lookups: 16,559
• Startup Time: ~7.3s
• The Culprit: A combination of heavy extensions (Panel) and an inefficient sys.path caused the interpreter to hammer the filesystem with thousands of failed lookups per second.

The Fix:

1. Disable Unused Extensions: removing panel cut the dependency graph.
2. Optimize PYTHONPATH: (Optional) Reordering the path so frequently used library directories come first reduces the 'seek time' for imports."

To dig further on panel

 python3 -X importtime -c "import panel.io.jupyter_server_extension" 2> import_log.txt  

import time:   1096259 |    1118432 |                rfc3987_syntax.syntax_helpers
import time:       233 |    1118664 |              rfc3987_syntax
...
import time:       748 |    1209474 |            jsonschema._format
import time:       224 |    1223863 |          jsonschema

Here is where your 2.2 seconds went:

1. rfc3987_syntax: 1.1s (Regex compilation overhead)
2. panel logic: 0.5s (Layout engine init)
3. bokeh logic: 0.4s (Plotting engine init)
4. numpy: 0.07s (Surprisingly fast!)

The recommendation here would be disabling panel extension.

The “IO Myth”

Common wisdom says “Docker is slow because opening thousands of files is expensive.”

Your Data: “False. We opened 3,536 TensorFlow files in just 0.22 seconds. The filesystem is blazing fast.”

The delay isn’t opening the files; it’s running them.

• TensorFlow:
• Time to find/open 3,536 files: 0.2s
• Time to initialize (from previous logs): ~2.4s
• Verdict: 90% of the time is spent on CPU/Linking, not Disk.
• Panel:
• Time to find/open 170 files: 0.008s
• Time to initialize: 2.2s
• Verdict: 99% of the time is CPU (specifically that rfc3987 regex compilation we found earlier).

| Library | Files Opened | Time to Open (I/O) | Time to Load (CPU) | Bottleneck |
| :--- | :--- | :--- | :--- | :--- |
| **TensorFlow** | 3,536 | 0.23s | ~2.40s | CPU / Linking |
| **Panel** | 170 | 0.008s | ~2.10s | CPU (Regex) |
| **Scipy** | 787 | 0.14s | ~0.50s | Mixed |

You can use this AWK script to measure library startup time and files opened. https://gist.github.com/gogasca/409259cc7efd75c0741894b243885bd5

awk -f library_import_time.awk startup_trace.txt | sort -t'|' -k2 -rn
tensorflow|0.233495|3536
keras|0.072873|985
scipy|0.065992|787
jax|0.035728|451
stdlib:__pycache__|0.032074|455
numpy|0.027703|441
IPython|0.019341|359

Checking extensions:

You could pass: `COLAB_WARMUP_DEFAULTS=0` and disable panel extension:

docker exec colab-startup-strace ls /usr/local/etc/jupyter/jupyter_server_config.d/
google.colab._serverextension.json
ipyparallel.json
jupyter_server_terminals.json
jupytext.json
nbclassic.json
notebook_shim.json
panel-client-jupyter.json
(base) jupyter@instance-20251214-215021:~$ docker exec colab-startup-strace cat /usr/local/etc/jupyter/jupyter_server_config.d/panel-client-jupyter.json
{
  "ServerApp": {
    "jpserver_extensions": {
      "panel.io.jupyter_server_extension": true
    }
  }
}
(base) jupyter@instance-20251214-215021:~$ docker exec colab-startup-strace cat /usr/local/etc/jupyter/jupyter_server_config.d/ipyparallel.json
{
  "ServerApp": {
    "jpserver_extensions": {
      "ipyparallel": true
    }
  }
}

Disable panel

# Option A: Remove the config only (Keeps library, disables auto-load)
RUN rm /usr/local/etc/jupyter/jupyter_server_config.d/panel-client-jupyter.json

# Option B: Uninstall completely (Best for disk space + speed)
RUN pip uninstall -y panel

Optimizing large-scale ML containers requires looking past the file size and understanding the runtime behavior. Our analysis of the Colab image revealed that high latency isn’t always a result of disk I/O, but often stems from CPU-bound tasks during initialization.

By using strace and forensic profiling, we uncovered three critical insights:

1. The “Cold Start” Tax is Complex: It’s not just pulling bytes; it’s the cost of verifying OCI layers, overlay filesystem overhead, and the initialization of massive libraries.
2. The I/O Myth: Contrary to popular belief, opening thousands of files is fast. The real bottleneck lies in processing those files specifically, CPU-intensive tasks like regex compilation in the panel extension or symbol linking in TensorFlow.
3. Invisible Bloat: A single unused extension can add seconds to your boot time. In our case, the panel dashboarding library was adding over 2 seconds to startup, regardless of whether the user needed it.

Optimizing your compute infrastructure isn’t just about faster SSDs; it’s about curating your dependencies and validating that every second of startup time delivers value to the user.

Coming Next: Now that we understand how to analyze the current state of a massive image, the natural next step is re-architecture. In our next post, we will explore how to simplify Docker image creation, reduce the attack surface and size, and leverage advanced container layering strategies to build lighter, faster Runtimes.

References:

• How to reduce the Nvidia Docker image size
• How to accelerate the startup of a large Docker image?
• Startup Containers in Lightning Speed with Lazy Image Distribution on Containerd

Evals for notebooks is an interesting topic. Not only does it require you to execute Python code and compare results, but you also need an underlying infrastructure that provides the following:

1. Same Runtime environment: Python version, pip libraries
2. End-User credentials
3. Code Execution

Currently, Colab Enterprise already supports all of these. However, for #3, if you want to execute code, you typically require connecting to the UI. Fortunately, I developed a new dedicated API that allows users to execute code by starting a kernel and passing a blob of text to execute, the same way a user would via the Colab UI. The advantage is that this API interacts directly with the Jupyter Kernel and supports long-running operations (e.g., cells that take a long time to execute).

During this discovery, I struggled to find an external dedicated sample for creating a Runtime Template and Runtime. I found REST, gcloud, and a few Python references, so I decided to just show a minimal example. Please take a look at the following:

1. Define Cloud environment and library imports

from google.cloud import aiplatform_v1
import os

PROJECT_ID = os.environ.get('GOOGLE_CLOUD_PROJECT')
LOCATION = os.environ.get('GOOGLE_CLOUD_REGION')
API_ENDPOINT = f"{LOCATION}-aiplatform.googleapis.com"
PARENT = f"projects/{PROJECT_ID}/locations/{LOCATION}"

# Initialize the client once globally
notebook_service_client = aiplatform_v1.NotebookServiceClient(client_options = {
    "api_endpoint": API_ENDPOINT,
})

2. Create a Runtime Template

def create_notebook_runtime_template(client: aiplatform_v1.NotebookServiceClient):
    notebook_runtime_template = aiplatform_v1.NotebookRuntimeTemplate()
    notebook_runtime_template.display_name = "eval"
    notebook_runtime_template.machine_spec = aiplatform_v1.MachineSpec(machine_type="e2-standard-4")
    notebook_runtime_template.network_spec = aiplatform_v1.NetworkSpec(
        network=f"projects/{PROJECT_ID}/global/networks/default",
        subnetwork=f"projects/{PROJECT_ID}/regions/{LOCATION}/subnetworks/default"
    )
    notebook_runtime_template.euc_config.euc_disabled = True
    notebook_runtime_template.idle_shutdown_config.idle_timeout = "10800s"
    request = aiplatform_v1.CreateNotebookRuntimeTemplateRequest(
        parent=PARENT,
        notebook_runtime_template=notebook_runtime_template,
    )
    print("Creating Runtime Template, waiting for operation to complete...")
    try:
        operation = client.create_notebook_runtime_template(request=request)
        runtime_template_ref = operation.result()
        print(f"Runtime Template: {runtime_template_ref}")
        print(f"Operation status: {operation.done()}")
        return runtime_template_ref.name
    except Exception as e:
        print(f"Error creating Notebook Runtime Template: {e}")

3. Create a new Runtime

def assign_runtime(runtime_template_ref: str, client: aiplatform_v1.NotebookServiceClient):
    if not runtime_template_ref:
        print("Cannot assign runtime: Notebook Runtime Template reference is missing.")
        return

    notebook_runtime = aiplatform_v1.NotebookRuntime()
    notebook_runtime.runtime_user = "gascagonzalo@gmail.com"
    notebook_runtime.display_name = "my-eval-runtime"

    request = aiplatform_v1.AssignNotebookRuntimeRequest(
        parent=PARENT,
        notebook_runtime_template=runtime_template_ref,
        notebook_runtime=notebook_runtime,
    )

    print("Creating Runtime, waiting for operation to complete...")
    try:
        operation = client.assign_notebook_runtime(request=request)
        response = operation.result()
        print(response)
        print(f"Operation status: {operation.done()}")
    except Exception as e:
        print(f"Error assigning Notebook Runtime: {e}")

Running example:

runtime_template_ref = create_notebook_runtime_template(notebook_service_client)
assign_runtime(runtime_template_ref, notebook_service_client)

One of the things I would love to do is to abstract this even more and integrate runtime template and runtime creation and code execution:

from google.colab import compute

runtime = compute.Runtime.assign("my-runtime", default=True)
p = runtime.exec("python", "-c", "print(123)", timeout=5)
print(p.stdout.read())

# Long running cell
p = runtime.exec("bash", "-c", "for i in {1..10}; do date +%T; sleep 1; done", timeout=15)
for line in p.stdout:
    print(line, end="")

runtime.stop()

This can simplify the developer experience, but also allow provide users to define parameters to templates if they want to do so.

from google.colab import compute

runtime_template_ref = compute.RuntimeTemplate.create("my-runtime-template", 
   machine_type="g2-standard-4", 
   idle_timeout="10800s", 
   euc=False, 
   python_version="3.12")

Would you be interested in using this new interface and a programatic code execution functionality?

Vertex Runtime Platform (VRP) is our new GKE-based system running in Google Managed Tenant Projects. This infrastructure enables customization options and faster default runtimes allocation. VRP maintains full compatibility with all Colab Enterprise features, including Idle Shutdown, End User Credentials and Library preservation.

This is available in private preview for external customers.

Please send me an email so I can whitelist your project and start testing!

https://medium.com/google-colab/colab-updated-to-python-3-11-00197d6172b1 Colab Enterprise team also has made Python 3.11 available in Private Preview for default Runtimes. If you are interested send me an email and I will whitelist your project.

Note: This is not supported by GCP, if you encounter any issues ping me here or via StackOverflow.

1. Install JupyterLab in your local workstation.

pip install jupyter-lab

2. Create a new Jupyter config file `jupyter_notebook_config.py`

Modify PROXY_URL with your Proxy URL. This can be obtained by clicking Open JupyterLab in Google Cloud Console.

"""VScode support for Workbench Instances."""
import datetime
import subprocess
import hashlib
import json

PROXY_URL = "https://<Your endpoint>-dot-us-central1.notebooks.googleusercontent.com"

c.NotebookApp.open_browser = False
c.ServerApp.token = ""
c.ServerApp.password = ""
c.ServerApp.port = 8080
c.ServerApp.allow_remote_access = True
c.ServerApp.disable_check_xsrf = True

def get_gcloud_token():
  """Helper method to get an OAuth token from gcloud."""
  p = subprocess.run(
      ["gcloud", "auth", "print-access-token"],
      capture_output=True,
      check=True,
      encoding="UTF-8",
  )
  bearer_token = p.stdout.strip()
  return bearer_token


c.GatewayClient.auth_scheme = "Bearer"
c.GatewayClient.auth_token = get_gcloud_token()
c.GatewayClient.headers = json.dumps({"Origin": PROXY_URL, "Cookie": "_xsrf=XSRF", "X-XSRFToken": "XSRF"})
c.GatewayClient.validate_cert = False
c.GatewayClient.url = PROXY_URL
c.Application.log_level = 0

3. Start JupyterLab locally

jupyter lab --port-retries=0 --ip 0.0.0.0 --allow-root --config=jupyter_notebook_config.py --debug

4. Start VSCode with JupyterLab extension pre-installed and connnect to the local Jupyter server. (`http://127.0.0.1:8080`). Local Jupyter server via Gateway client will connect to remote server and you should be able to access remote Vertex instance.

5. Verify you are connected to remote instance.

The next step is to add support to refresh token via `JUPYTER_GATEWAY_TOKEN_RENEWER_CLASS` Reference: https://github.com/jupyter-server/jupyter_server/blob/main/jupyter_server/gateway/gateway_client.py#L51

For Colab Enterprise, you can use same configuration. (Solved here: https://github.com/jupyter-server/jupyter_server/pull/1440)

Video here: https://www.youtube.com/watch?v=8o0Jo6KfuAU

Troubleshooting

1. Create a new Virtual environment
2. Install latest version of JupyterLab and Jupyter Server (jupyter_server)
3. Check Websockets connections are not blocked or additional libraries are not overwritting the original version in JupyterLab.