Your RAG stack is 90% unnecessary. Here’s the receipts, reproducible on a public dataset.

Vector database. Orchestrator. Embedding service. API keys everywhere. Three containers to answer questions about your own documents.

That’s the architecture we all normalized for Retrieval-Augmented Generation. And in 2026, for a large class of use cases, it’s obsolete.

BigQuery’s AI functions turn the LLM into a SQL operators, the same way JOIN or WHERE are operators. Embeddings, vector search, and structured generation all run where your data already lives, with your existing IAM, audit logs, and cost controls. And no, it's not a Gemini-only walled garden: you can bring Claude, Llama, Mistral, or an open-source model from Hugging Face.

In this article, I’ll build the pipeline twice, on a real public dataset you can query right now:

1. The one-shot version (GA): batch-embed a corpus, search it, generate grounded answers
2. The living version (Preview): a corpus that keeps itself embedded as data arrives, because production data doesn’t hold still

Then, I’ll show you exactly where this approach breaks and when Python remains the right answer.

Everything below runs on bigquery-public-data.bbc_news.fulltext. Open a BigQuery tab or the Google Cloud Data Agent Kit plugin in your IDE and follow along.

The stack we all normalized

A “standard” RAG architecture in most tutorials looks like this:

1. An embedding service: a hosted API, or a model endpoint you deploy and babysit
2. A vector database: one more SaaS, one more billing account, one more sync pipeline to keep consistent with your source of truth
3. An orchestrator: LangChain, LlamaIndex, or hand-rolled glue code
4. An LLM API: with its rate limits and quota management
5. Deployment: typically two or three containers to build, ship, and patch

Count the moving parts: five systems, at least three API keys, and (the part nobody talks about) your data leaves the warehouse. Every governance rule, every row-level security policy, every audit requirement you carefully built in BigQuery? Gone the moment you export chunks to an external vector store.

We built this architecture because we had no choice. That premise is no longer true.

What changed: the LLM is now a SQL operator

The BigQuery AI.* function family reached general availability. Three capabilities matter for RAG:

• AI.GENERATE_EMBEDDING: batch-generate embeddings from any table
• VECTOR_SEARCH: a native GoogleSQL function for semantic retrieval, with optional IVF or TreeAH indexes (the latter built on Google's ScaNN algorithm)
• AI.GENERATE: a scalar, row-wise generation function with structured output via output_schema, usable anywhere a value fits: SELECT, WHERE, even JOIN conditions

No API key in your code. Authentication goes through a BigQuery connection, a managed service account you grant a role once. There’s even a default connection now, which removes the setup ceremony entirely.

Not just Gemini

A common objection: “so I’m locked into Gemini?” No, and this deserves precision because it’s widely misunderstood.

BigQuery’s AI functions come in two flavors:

• General-purpose functions (AI.GENERATE_TEXT, AI.GENERATE_TABLE, AI.GENERATE_EMBEDDING with a MODEL): you have full control over the model choice. Supported remote models include the Gemini family, but also Anthropic Claude, Meta Llama, and Mistral AI as managed model-as-a-service endpoints, plus open-source models from Hugging Face or the Agent Platform (ex Vertex AI) Model Garden. For open models, the CREATE MODEL statement can even auto-deploy the model for you. Same SQL, your model.
• Managed functions (AI.IF, AI.CLASSIFY, AI.SCORE, and the scalar AI.GENERATE with an endpoint argument): these are Gemini-powered convenience functions where BigQuery handles model selection and prompt engineering for you.

Want your embeddings from a self-hosted multilingual E5 or Qwen model for compliance reasons, and your generation from Claude?

Both start with one CREATE MODEL statement. For embeddings, that's the whole switch: nothing else in the pipeline moves. For generation, one honest caveat: the typed output_schema used in Step 3 is a feature of the managed AI.GENERATE (Gemini).

To generate with Claude, Llama, or Mistral, you swap to AI.GENERATE_TEXT over your model (or AI.GENERATE_TABLE when you still want structured columns). Same warehouse, same governance, your model. The generation call is the one line you adapt, not the architecture.

Let’s build.

Path 1: the one-shot pipeline (GA)

This is the pattern for batch and analytical workloads: a corpus you embed once (or on a schedule), then query.

Step 0. One-time setup

Before the model, one prerequisite that most warehouse-native tutorials skip: the connection. It’s the managed service account BigQuery uses to call the models, and you set it up once.

The fast path, if you have the BigQuery Admin role, is the default connection: just write WITH CONNECTION DEFAULT (as below) and BigQuery provisions it and grants its service account the right roles on first use. Nothing else to do.

The explicit path, if you’d rather own your connection (this is the us.my_connection referenced in Path 2), is two commands: create a Cloud resource connection, then grant its service account the Agent Platform User role (roles/aiplatform.user, formerly "Vertex AI User"):

PROJECT_ID=my-poject-id

# 1. Create a Cloud resource connection (its location must match your dataset)
bq mk --connection --location=US --project_id=$PROJECT_ID \
  --connection_type=CLOUD_RESOURCE my_connection

# 2. Grant the connection's service account the Agent Platform User role
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$(bq show --format=prettyjson \
    --connection $PROJECT_ID.US.my_connection | jq -r .cloudResource.serviceAccountId)" \
  --role=roles/aiplatform.user

Grant the role in the project where you create the model (when the endpoint is a model name) or in the project named in the endpoint URL (when you pass a full URL). That’s the whole auth setup: one role, no keys.

Now the embedding model, using the default connection:

CREATE OR REPLACE MODEL `rag_demo_us.emb_model`
  REMOTE WITH CONNECTION DEFAULT
  OPTIONS (ENDPOINT = 'gemini-embedding-001');

That’s the entire “embedding service deployment”. No container, no endpoint to scale, no key to rotate. (Swap the endpoint for an Agent Platform URL if you deployed an open-source model.)

Step 1. Embed the corpus

We’ll embed BBC News articles. One statement:

CREATE OR REPLACE TABLE `rag_demo_us.news_emb` AS
SELECT *
FROM AI.GENERATE_EMBEDDING(
  MODEL `rag_demo_us.emb_model`,
  (
    SELECT title, body AS content
    FROM `bigquery-public-data.bbc_news.fulltext`
    WHERE category = 'tech'
  ),
  STRUCT('RETRIEVAL_DOCUMENT' AS task_type)
);

Two details tutorials often miss:

• The input query must expose a column named content (alias it if needed). Including a title column improves embedding quality with RETRIEVAL_DOCUMENT.
• task_type matters: RETRIEVAL_DOCUMENT for the corpus, RETRIEVAL_QUERY for the user's question. Asymmetric embeddings measurably improve retrieval.

The output table carries your passthrough columns plus three generated ones: embedding (the vector), statistics (a JSON with token_count and truncated), and status.

Note the names: the newer AI.GENERATE_EMBEDDING returns embedding, not the ml_generate_embedding_result you'll see in older ML.GENERATE_EMBEDDING tutorials. Mixing up the two column layouts is a classic copy-paste failure.

Pitfall #1: silent partial failures. A batch embedding job can succeed while individual rows fail (quota pressure, service hiccups). BigQuery reports this per row in the status column; an empty status means success. Always check it (WHERE status = ''). If parallel batches hit RESOURCE EXHAUSTED errors, Google ships remote-inference retry scripts and a Dataform package specifically to iterate until every row is processed. It's documented, because it happens.

Pitfall #1bis: silent truncation. Look at statistics: each row reports its token_count and a truncated flag. A BBC article sits comfortably around a couple hundred tokens, but feed the function documents longer than the model's input limit and it will embed a truncated version without failing. If your corpus has long documents, monitor LAX_BOOL(statistics.truncated) and chunk upstream.

Step 2. (Optional) Index it

For small corpora, skip this: VECTOR_SEARCH falls back to brute-force exact search. At scale:

CREATE OR REPLACE VECTOR INDEX news_idx
ON `rag_demo_us.news_emb`(embedding)
OPTIONS (index_type = 'IVF', distance_type = 'COSINE');

Know before you’re surprised: index maintenance is fully managed and asynchronous, new rows remain searchable immediately (BigQuery brute-forces the not-yet-indexed tail), tables under 10 MB don’t get a populated index, and indexes aren’t supported on the Standard edition.

Step 3. Retrieval + generation, one query

Embed the question, retrieve the top 5 articles, ground the prompt, generate a typed answer:

SELECT
  AI.GENERATE(
    (
      'Answer using only this context:\n',
      STRING_AGG(base.content, '\n---\n'),
      '\nQuestion: what were the major browser security stories?'
    ),
    endpoint      => 'gemini-2.5-flash',
    output_schema => 'answer STRING, confidence FLOAT64, sources ARRAY<STRING>'
  ).*
FROM VECTOR_SEARCH(
  TABLE `rag_demo_us.news_emb`, 'embedding',
  (
    SELECT embedding
    FROM AI.GENERATE_EMBEDDING(
      MODEL `rag_demo_us.emb_model`,
      (SELECT 'major browser security stories' AS content),
      STRUCT('RETRIEVAL_QUERY' AS task_type)
    )
  ),
  top_k => 5
);

Count the core pipeline: a dozen lines. No LangChain. No Pinecone. No deployment. And output_schema returns typed columns (answer, confidence, sources), not a JSON blob to parse and pray over.

But here’s the honest limitation of Path 1, and it’s a big one.

The one-shot pattern has a production problem: data doesn’t hold still. New articles arrive, descriptions get edited, tickets get updated. With Path 1, every insert or update means re-running the embedding statement, detecting deltas, handling retries, monitoring staleness.

Path 2: the living corpus (Preview)

This is exactly what autonomous embedding generation solves. You declare the embedding as a generated column, and BigQuery maintains it:

CREATE TABLE `rag_demo_us.news_live` (
  title STRING,
  body  STRING,
  body_embedding STRUCT<result ARRAY<FLOAT64>, status STRING>
    GENERATED ALWAYS AS (
      AI.EMBED(
        body,
        connection_id => 'us.my_connection',
        endpoint      => 'gemini-embedding-001')
    ) STORED OPTIONS (asynchronous = TRUE)
);

Now just insert data, from anywhere:

INSERT INTO `rag_demo_us.news_live` (title, body)
SELECT title, body
FROM `bigquery-public-data.bbc_news.fulltext`
WHERE category = 'tech';

Read that DDL again, because the design is elegant:

• GENERATED ALWAYS AS AI.EMBED(...): the embedding is table metadata, not pipeline code. Insert or update the source column, and BigQuery generates or refreshes the vector in the background.
• asynchronous = TRUE: your INSERTs don't block on model calls. Embedding jobs run on background slots (you'll see them prefixed gc_ in your job history).
• The column is a STRUCT<result, status>: failures are visible per row, not swallowed.

A word on the model choice: I use gemini-embedding-001 here because it's GA and tops the MTEB multilingual leaderboard. The canonical AI.EMBED doc example uses text-embedding-005, which is also perfectly valid; I'm picking the more recent, higher-quality option. One consequence to plan for: its default output is 3072 dimensions versus 768 for the text-embedding models, which raises storage and index cost. If that matters, request a smaller dimension via AI.EMBED's model_params (the model is Matryoshka-trained, so it truncates cleanly to 768 or 256).

And retrieval gets simpler, because the new AI.SEARCH function knows which model the table uses and embeds your query string automatically:

SELECT base.title, base.body, distance
FROM AI.SEARCH(
  TABLE `rag_demo_us.news_live`, 'body',
  'major browser security stories',
  top_k => 5
);

No manual query embedding. No model bookkeeping. You search text against a text column, and the warehouse handles the vector plumbing. Vector indexes integrate too: create one on the source column and index training starts once 80% of rows have embeddings.

This is the version that matters for anything that lives in production. The one-shot pattern is a demo; a corpus that keeps itself embedded is an architecture.

One label, though: this is Preview (Pre-GA terms, subject to change). Also worth knowing before you commit: one auto-generated embedding column per table, batch your inserts rather than trickling frequent DML (concurrent DML can delay generation), and copies, clones, and snapshots carry the data but not the generation config. Per the latest release notes, you can enable it on new or existing tables via CREATE TABLE or ALTER TABLE. As always, the BigQuery release notes are the only authoritative source for status; check them before you build.

What you inherit for free

This is the part that should matter most to data teams, and it’s the least discussed:

• Row-level and column-level security apply to VECTOR_SEARCH. If a user can't see a row, they can't retrieve it as RAG context. Try implementing that correctly across an external vector DB sync.
• IAM is the only auth model. No API keys in environment variables, no rotation runbooks for the pipeline itself.
• Audit logs, cost attribution, quotas: all the boring, load-bearing infrastructure your platform team already operates. You can even trace per-job Agent Platform charges in Cloud Billing via the bigquery_job_id_prefix label.
• No sync pipeline. The embeddings live next to the source rows. There is no “vector store drifted from the warehouse” incident, because there is no second store. With autonomous embeddings, there isn’t even a refresh job to own.

The best infrastructure is the one you don’t deploy.

Where this breaks

I promised receipts, not religion:

1. Cost is per token, and Gemini 2.5 charges for thinking. Generation on Gemini 2.5 models bills the thinking tokens on top of output. For classification-style tasks, set the thinking budget to 0 via model_params: lower latency, lower bill. Ignore this and batch jobs get expensive quietly. AI.COUNT_TOKENS exists to estimate before you run.

2. Row count surprises on complex queries. With JOINs or ORDER BY ... LIMIT, the number of rows actually sent to the model can differ from what you expect. Google's own recommendation: materialize your input to a table first, then run inference on it. Do it.

3. Quotas are real. Embedding throughput is governed by tokens-per-minute quotas (a few million by default). Fine for most workloads; a “let’s embed the whole data lake tonight” job needs planning (or Provisioned Throughput via request_type => 'dedicated'), not enthusiasm.

4. VECTOR_SEARCH bills bytes scanned. On-demand pricing charges for base table, index, and query scans. Use STORING columns in your index to avoid expensive joins back to the base table, and pre-filter with WHERE clauses on stored columns.

5. Prompt versioning doesn’t exist. Your prompt is a string inside a SQL query. No registry, no A/B testing, no evaluation harness. For a production assistant iterating on prompts weekly, this gap is real, and it’s where a thin Python layer (or dbt, with prompts as versioned config) earns its keep.

GA or Preview? Read the label

As of this writing:

AI.GENERATE and AI.GENERATE_TABLE reached GA in late January 2026; the partner models (Claude, Llama, Mistral) went GA in the first half of 2025. Statuses change; release notes don't lie.

So… is SQL all you need?

Three takeaways:

1. For batch and analytical RAG: yes, seriously consider it. Document classification, semantic deduplication, enrichment pipelines, internal Q&A over governed data: the warehouse-native approach wins on governance, simplicity, and total cost of ownership. And you keep your model choice.

2. For a living corpus, autonomous embeddings change the calculus. The hard part of production RAG was never the search; it was keeping vectors in sync with reality. Declaring the embedding as a generated column deletes that entire problem class. It’s Preview today, but it’s clearly where the platform is going.

3. The skill isn’t the syntax; it’s knowing where the line sits. Sub-second user-facing chat still wants a serving layer. Prompt lifecycle still wants versioning. Cost per token still wants monitoring. These constraints didn’t disappear; they moved.

I’m putting this to the test on the BBC News corpus: cost per 1,000 documents, end-to-end latency, retrieval recall with and without an index, and how fast autonomous embeddings actually converge after a bulk insert. That’s the follow-up article. If there’s a specific scenario you want benchmarked, tell me.

If you enjoyed this, I write about Data, AI, BigQuery, Cloud Run and a lot of cool stuffs.

References:

• Generative AI overview (model support matrix)
• CREATE MODEL for partner and open LLMs
• The AI.GENERATE function
• The AI.GENERATE_EMBEDDING function
• Autonomous embedding generation
• The AI.SEARCH function
• Search functions (VECTOR_SEARCH)
• Manage vector indexes
• BigQuery release notes

SQL Is All You Need: RAG without a Vector Database in a Dozen Lines of BigQuery was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’ve ever wished you could go from docker compose up on your laptop to a fully deployed multi-service application on Google Cloud Run with a single command — this article is for you.

docker compose up is a wonderful command for local development. But it stops at your machine. Now imagine replacing docker with gcloud run and getting the same stack running in the cloud, production-shaped, with caveats we’ll cover on the section 5. That's exactly what gcloud run compose up makes possible.

It also breaks in three specific, undocumented ways, which we’ll cover in 5 so you don’t hit them blind.

In this article, we’ll deploy a real-world app to Cloud Run using a single compose.yaml — no Kubernetes YAML, no manual GPU configuration. Along the way, we'll cover the key adaptation points, the gotchas nobody documents, and what actually happens under the hood.

1. Prerequisites and Getting Started

This article assumes you’re comfortable with Docker Compose and have a basic understanding of Google Cloud. You don’t need to know Cloud Run internals — that’s what we’re here to learn.

What you need before starting:

• Google Cloud SDK 564.0.0+ — required for gcloud run compose up. This feature’s launch stage is still evolving, so check the official docs before adopting in production.
• A GCP project with billing enabled and the following APIs activated:

gcloud services enable run.googleapis.com \
    artifactregistry.googleapis.com \
    storage.googleapis.com

• Docker Desktop 4.40+ with Model Runner enabled, if you want to test locally first:

docker desktop enable model-runner --tcp 12434

• Authenticate locally with gcloud auth application-default login(ADC). The credentials need BigQuery Data Viewer and BigQuery Job User roles .

Clone the repo to follow along:

git clone https://github.com/bricefotzo/gcp-release-notes.git
cd gcp-release-notes

💡 The patterns in this article apply to any multi-service compose.yaml. The GCP Release Notes Navigator is just the vehicle — adapt what you learn here to your own stack.

2. The Use Case: A Multi-Service Cloud Run Stack with an LLM

The app we’re deploying is a multi-service stack with three components:

• A Streamlit frontend: search and filter interface
• A FastAPI backend: queries BigQuery and calls the LLM
• Open Weights LLM (ai/llama3.2 ): served by Docker Model Runner for AI-powered summaries

💡 Why Docker Model Runner instead of Ollama?

Both expose an OpenAI-compatible API, so your code is identical either way. The difference is at deploy time: Docker Model Runner uses the models: top-level key in compose.yaml, which gcloud run compose translates directly into a GPU-backed Cloud Run service. With Ollama, you'd need to manage the GPU container yourself. Same dev experience, very different deployment story.

The local architecture looks like this:

After gcloud run compose up, the Cloud Run deployment looks like this:

The key shift: locally, the backend calls the LLM at http://model-runner.docker.internal:12434. On Cloud Run, the LLM becomes a separate service with its own HTTPS URL - and LLM_URL is automatically rewritten to point to it.

You don’t need to understand the app’s internals to follow this tutorial. What matters is that it’s a real multi-service stack defined in a single compose.yaml — the patterns here apply to your own projects.

3. The compose.yaml for gcloud run compose up, Annotated

The entire orchestration lives in a single file. Here it is in full — we’ll break it down right after:

services:
  # ── Streamlit frontend
  front:
    build:
      context: ./frontend
    ports:
      - "8090:8080"
    environment:
      - BACKEND_URL=http://back:8000
    depends_on:
      back:
        condition: service_healthy
  # ── FastAPI backend (port 8000)
  back:
    build:
      context: ./backend
    ports:
      - "8000:8000"
    # Docker Model Runner injects LLM_URL and LLM_MODEL automatically
    models:
      - llm
    environment:
      - PROJECT_ID=serial-techos
      - DATA_PROJECT_ID=bigquery-public-data
      - DATASET_ID=google_cloud_release_notes
      - TABLE_ID=release_notes
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
# ── Docker Model Runner ─────────────────────────────────────────
models:
  llm:
    model: ai/llama3.2
    context_size: 32768
    x-google-cloudrun:
      environment:
        PORT: 12434
        MODEL_RUNNER_PORT: 12434
      run-model:
        inference-container: docker/model-runner

💡 Everything prefixed with x- is a Compose extension. Docker Compose ignores it locally — it's metadata for Cloud Run only. This is what makes the same file work in both environments.

3.1 Frontend — Streamlit

The Streamlit frontend is straightforward, it builds from ./frontend, exposes port 8080 and depends on the backend being healthy.

environment:
  - BACKEND_URL=http://back:8000
depends_on:
  back:
    condition: service_healthy

• BACKEND_URL=http://back:8000 — On Cloud Run, front and back run as sidecars in the same service, sharing a network namespace. This means http://back:8000 resolves via localhost, exactly as it does locally. No change required.
• depends_on — Translated to a Cloud Run container dependency: {"front":["back"]}. Cloud Run won't route traffic to front until back is healthy.

3.2 Backend — FastAPI

models:
  - llm
environment:
      - PROJECT_ID=serial-techos
      - DATA_PROJECT_ID=bigquery-public-data
      - DATASET_ID=google_cloud_release_notes
      - TABLE_ID=release_notes
healthcheck:
  test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
  interval: 10s
  timeout: 5s
  retries: 5
  start_period: 30s

• models: - llm — This is the key that wires back to the LLM service. On Cloud Run, LLM_URL is automatically injected with the HTTPS URL of the deployed llm service. Locally, Docker Model Runner handles this injection directly.
• environment — The variables declared here are injected as Cloud Run environment variables.

For local development, if you chose to use this use case(github), I use ADC for authentication. For this, I use a compose.override.yaml where I define volume mount for authentication in the local container.

services:
  back:
    volumes:
      - ${ADC_PATH:-./adc.json}:/tmp/adc.json:ro
    environment:
      - GOOGLE_APPLICATION_CREDENTIALS=/tmp/adc.json

This second YAML is taken into account when you run docker compose up so that your back container is authenticated using ADC but the command gcloud run compose up ignores it.

3.3 The models: top-level key

models:
  llm:
    model: ai/llama3.2
    x-google-cloudrun:
      environment:
        PORT: 12434
        MODEL_RUNNER_PORT: 12434
      run-model:
        inference-container: docker/model-runner

This is where the local-to-cloud translation happens:

• models: is a top-level Compose key for Docker Model Runner. It declares the LLM as a named dependency, not a traditional service container.
• ai/llama3.2— the model pulled from Docker AI Hub.
• x-google-cloudrun: — the Cloud Run-specific configuration:
• PORT and MODEL_RUNNER_PORT: the port on which the model runner listens
• inference-container: docker/model-runner: the container image used to serve the model on Cloud Run.

On Cloud Run, this block is translated into a standalone Cloud Run service named llm, running docker/model-runner with an NVIDIA L4 GPU attached. No GPU configuration on your side — one key in the YAML, one service in the cloud.

4. Deploying with gcloud run compose up: One Command, Two Services

Before running the command, make sure everything is in place:

✅ APIs enabled (Cloud Run, Artifact Registry, Cloud Storage)

✅ Docker images build locally (docker compose build)

✅ Authenticated with gcloud auth application-default login

Then:

export PROJECT_ID=your-project-id
gcloud run compose up \
  --project=$PROJECT_ID \
  --region=europe-west1

That’s it. The command builds your images, pushes them to Artifact Registry, and creates Cloud Run services — all from the compose.yaml.

4.1 What gets created

This is the most surprising part: you defined three components in your compose file, but Cloud Run creates two services, not three.

Service 1: gcp-release-notes — front and back are co-located as sidecars in a single Cloud Run service. This is the Cloud Run multi-container pattern. The relevant part of the generated Knative manifest:

Two things to notice here. front is the ingress container (it has ports.name: http1). And LLM_URL has been automatically rewritten from the local Docker Model Runner address to the HTTPS URL of the deployed llm service.

Service 2: llm — The models: key is translated into a standalone Cloud Run service with a real GPU:

4 vCPUs, 16 GB RAM, one NVIDIA L4 GPU — all provisioned automatically from the models: block in your compose file. CPU throttling is disabled because GPU workloads require consistent CPU availability.

💡 About the L4 — and why you don’t get to pick.

Cloud Run’s models: translation provisions an NVIDIA L4 (24 GB VRAM) by default. It’s the minimum-cost inference GPU on Cloud Run, optimized for 7B–14B parameter models at FP16 or INT8/FP8. Plenty for llama3.2 or gemma3:4b. Not enough for a 70B model, that needs ~35 GB VRAM even at 4-bit and exceeds the L4.

Cloud Run also supports GPUs for bigger models, but gcloud run compose updoesn’t currently expose a GPU-type selector in the models: block. You just get an L4. If you need the Blackwell, you’ll have to fall back to deploying the LLM service manually with gcloud run deploy --gpu-type.

💡 Both services are tagged with managed-by: runcompose. Useful if you need to identify compose-managed resources in a project with other Cloud Run services.

4.2 ⚠️ Models are not pulled at deployment time

The docker/model-runner container starts empty — no model is loaded. After deployment, you must pull the model manually:

curl -X POST https://llm-xxx.europe-west1.run.app/models/create \
  -H "Content-Type: application/json" \
  -d '{"from": "ai/llama3.2"}'

🚨 Do this right after deployment. Until this call is made, any inference request from the backend will hang and eventually time out.

5. gcloud run compose up — Real pitfalls

gcloud run compose is a powerful abstraction, but it's still a young tool. Here are the real issues encountered deploying this project.

⚠️ Pitfall 1 — Why is LLM_URL injected without the /v1 suffix on Cloud Run?

Cloud Run injects LLM_URL pointing to the llm service, but the URL ends at the domain (https://llm-xxx.run.app) without the /v1 path required by the OpenAI-compatible API.

Locally, Docker Model Runner sets LLM_URL to http://model-runner.docker.internal:12434/engines/v1 — full path included. On Cloud Run, the injected value is just the base URL. Calling it directly returns 404 on every inference endpoint.

Fix — detect the environment at runtime:

# backend/src/ai.py
from openai import OpenAI

LLM_URL = os.environ.get("LLM_URL", "")
# Cloud Run injects the base URL without path suffix.
# Locally, Docker Model Runner provides the full path.
LLM_ENDPOINT = f"{LLM_URL}/v1" if "run.app" in LLM_URL else LLM_URL
client = OpenAI(base_url=LLM_ENDPOINT, api_key="not-needed")

⚠️ Pitfall 2— Why doesn’t gcloud run compose up redeploy my llm service?

If you change the model name or configuration under the models: key and re-run gcloud run compose up, the llm service is not updated. The command reports success, which gives the misleading impression that a new revision was created. In reality, the service is untouched.

Changing the context_size for example, has no workaround at the time of writing. If you’ve found one, please drop it in the comments.

⚠️ Pitfall 3 — Why do my pulled models disappear after a Cloud Run restart?

Model data is stored in the container’s local filesystem. If the llm container crashes or is evicted — for example due to a timeout on a large inference request — the pulled model is lost and the pull must be repeated.

A persistent volume would solve this, but since the llm service is generated entirely from the models: key (with no explicit service spec), there is currently no way to attach one via gcloud run compose. This remains an open limitation at the time of writing.

💡 In practice: Pitfalls 2 and 3 are related. The models: key is the most "magic" part of the compose translation — and magic comes with trade-offs. Expect this area to mature in future SDK releases.

⚠️ Pitfall 4 — Heads up: services default to `max_instances=1`

Both Cloud Run services created by gcloud run compose upare deployed with a maximum of one instance. This is a documented default of the Compose deployment path, not a bug — but it’s a silent ceiling. The first deploy works, the demo looks great, then the second concurrent user hits a cold start while the first is mid-request and you start chasing ghost latency.

This applies to both the gcp-realease-notes web service and the `llm` GPU service. For the LLM, this is doubly painful: with max_instances=1, you also can’t survive an instance eviction (see Pitfall 3).

Fix — raise it explicitly after deployment:

# Update the multi-container service
gcloud run services update gcp-release-notes \
 --max-instances=5 \
 --region=europe-west1

# Update the LLM service
gcloud run services update llm \
 --max-instances=2 \
 --region=europe-west1

Keep llm modest — GPU quotas are tight and each instance is an L4. Two is usually enough for a demo or small internal tool.

💡 Nothing in your `compose.yaml` controls this today. Until Compose exposes a `max_instances` knob, the post-deploy `gcloud run services update` is the only path.

Conclusion

gcloud run compose up turns a standard compose.yaml into a fully deployed Cloud Run application. Here's what to remember:

The compose.yaml is the single source of truth. The same file runs locally with Docker Desktop and deploys to Cloud Run with one command. No Dockerfile changes, no Kubernetes manifests.

Two services, not three. front and back are co-deployed as sidecars in a single multi-container service. The LLM becomes its own GPU-backed service.

The models: key is the bridge. It connects Docker Model Runner locally and triggers automatic GPU provisioning on Cloud Run. This is what makes the deployment seamless — and why Docker Model Runner matters over Ollama for this workflow.

Service-to-service communication just works. http://back:8000 resolves via localhost in the sidecar setup. LLM_URL is auto-rewritten to the HTTPS URL of the llm service.

The tool is still young. Model pulls, redeployment of the llm service, and ephemeral model storage are real limitations today. Check section 5 before deploying to production.

Teardown

If a Googler from the Cloud Run team is reading, please implement this command so that we can clean up easily:

gcloud run compose down \
  --project=$PROJECT_ID \
  --region=europe-west1

By the way, for now, you can use the label managed-by: runcompose that all the services created by gcloud run compose up hold.

Run this command:

# Delete them all
gcloud run services list \
  --project=$PROJECT_ID \
  --region=europe-west1 \
  --filter="metadata.labels.managed-by=runcompose" \
  --format="value(metadata.name)" | \
  xargs -I {} gcloud run services delete {} \
    --project=$PROJECT_ID \
    --region=europe-west1 \
    --quiet

Source code

The full source code for the GCP Release Notes Navigator is available on GitHub: github.com/bricefotzo/gcp-release-notes

If you found this article helpful, follow me on Medium for more content on Google Cloud, data engineering, and AI applications.

`gcloud run compose up`: Deploy a Multi-Service GPU Stack to Cloud Run from Docker Compose was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my previous article, we explored the fundamental concepts and key services that make up Google Cloud Platform. We covered everything from the core organizational structure to the essential services across compute, storage, database, networking, and AI/ML categories.

Now it’s time to roll up our sleeves and get our hands dirty with actual implementation.

I hope you’re full of energy and willing to master cloud computing, because it will be straightforward! Let’s channel that energy into creating your first Google Cloud project!

We’ll use Nginx, one of the most popular web servers in the world.
It’s lightweight, fast, and perfect for serving static content or acting as a reverse proxy.
Many major websites (Netflix, Airbnb, etc.) rely on it.

Step 1: Create a Google Cloud Account

Every journey begins with a single step, and your cloud journey starts with creating an account:

Visit cloud.google.com

1. Click “Get started for free”
2. Sign in with your existing Google account or create a new one
3. Complete the verification process, including payment information

Don’t worry about the payment information — Google provides $300 in free credits for new users, valid for 90 days. This is perfect for exploring services without immediate financial commitment.

Pro Tip: Even though you’re starting with free credits, treat them like real money. This builds good habits that will serve you well when working with actual budgets. Track your spending from day one.

Step 2: Set Up Your First Project

Projects are the fundamental organizational units in Google Cloud. They serve as isolated containers for your resources, APIs, and billing settings.

Once your account is active:

Go to the Google Cloud Console

1. Look for the “Select a project” dropdown near the top left
2. Click “New Project”
3. Give your project a descriptive name (e.g., “learning-gcp-basics” or “mycompany-website-dev”)

4. Click “Create”

When naming projects, I’ve found that following a consistent convention helps immensely as your cloud footprint grows. Consider patterns like:

• [purpose]-[environment]-[optional-identifier]
• [department]-[application]-[environment]

For example: marketing-website-dev or datascience-forecasting-test.

Step 3: Enable APIs for Your Services

Here’s an aspect of Google Cloud that often surprises newcomers: services don’t work until you explicitly enable their APIs. This design choice is actually a clever security feature — it prevents accidental usage and billing for services you never intended to use.

To enable an API:

1. Navigate to “APIs & Services” > “Library” in the Console
2. Search for the API you need (e.g., “Compute Engine API” for VMs)
3. Click on the API and then click “Enable”

For beginners, I recommend starting with:

• Compute Engine API (for virtual machines)
• Cloud Storage API (for object storage)
• Cloud Logging API (for monitoring)

These form the backbone of many cloud solutions and provide a good starting point for exploration.

I recall starting GCP implementation following tutorials, this point was so obvious for some that it was not mentioned and I struggled a lot with not working commands just because of services that were not enabled.

Step 4: Set Up the Google Cloud CLI

While the web console is user-friendly and great for learning, the command line will eventually become your best friend for efficiency and automation. The Google Cloud CLI (gcloud) gives you powerful control over your resources directly from your terminal.

1. Download and install the Google Cloud SDK for your operating system
2. Open your terminal and run the initialization command:

gcloud init

3. Follow the prompts to authenticate and select your project

Why using CLI over the console? CLI commands can fit into one script that you launch once to automate all your provisioning, update or delete where for each resource that you create in console it’s a lot of clicks. It’s more user friendly for beginners but way less reproducible and automation ready.

Step 5: Set Up IAM and User Access

Even if you’re just starting out solo, understanding Identity and Access Management (IAM) is crucial — especially if you plan to collaborate with others or follow security best practices.

To add a user to your project:

1. Navigate to “IAM & Admin” > “IAM” in the Console
2. Click “Add”
3. Enter the user’s email address(yours)
4. Select appropriate roles based on what they need to do

For the sake of the tutorial, select the Owner role for your account so that you can do a lot of things.

5. Click “Save”

Google Cloud follows the principle of least privilege: grant users only the permissions they need to perform their specific tasks — nothing more.

After an incident with a junior that accidentally deleted our VMs(because of me that provided admin Compute Engine permissions temporarily)I now follow these IAM best practices religiously:

1. Always use purpose-specific roles instead of broad admin access
2. Set time limits on temporarily permissions using IAM Conditions
3. Regularly audit who has what access to your projects
4. Use service accounts for programmatic access rather than personal accounts

These practices have saved me from numerous potential disasters in professional settings.

Step 6: Configure Default Settings

Spending a few minutes configuring defaults will save you countless hours of repetitive typing and parameter setting:

# Set default compute region and zone
gcloud config set compute/region us-central1
gcloud config set compute/zone us-central1-a

Region Selection Tip: Choose a region close to your target users to minimize latency. For learning purposes, any region works, but for production workloads, geography matters significantly.

You can verify your configuration anytime with:

gcloud config list

I’ve found that taking the time to set up these defaults, along with creating shell aliases for common commands, dramatically increases productivity when working with GCP daily. For example, setting the region prevent you from using the argument — region for the commands that need it if you have the same for all your resources.

Step 7: Create Your First Virtual Machine

Now for the exciting part — spinning up your first server in the cloud:

gcloud compute instances create my-first-instance \
  --machine-type=e2-micro \
  --image-family=debian-11 \
  --image-project=debian-cloud \
  --boot-disk-size=20GB

Let’s break down the VM creation command parameters:

- machine-type=e2-micro: This is one of Google Cloud’s economical machine types, featuring 2 shared vCPUs and 1 GB of memory. It’s perfect for learning and light workloads.

- image-family=debian-11 and image-project=debian-cloud: These parameters specify the operating system. We’re using Debian 11 (also known as “Bullseye”), which is a stable, well-supported Linux distribution.

- boot-disk-size=20GB: This sets the size of your main storage disk. You can adjust this based on your needs — remember, you pay for provisioned space regardless of usage.

Here is the output:

To connect to your newly created VM:

gcloud compute ssh my-first-instance

You’re now connected to a computer that exists only in Google’s data centers! This is cloud computing made tangible.

Let’s test some basic commands on your new VM:

# Check the system information
uname -a
# Check available disk space
df -h

Let’s deploy our first website on internet manually, using a VM:

# Install a simple web server
sudo apt update && sudo apt install -y nginx
# Start the web server
sudo systemctl start nginx

Now your nginx web server is available:

curl localhost:80

Here is the result:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

You’ve now deployed this web page on your VM. We can access it using curl in the VM but can we access it outside of the VM?

Accessing the VM from the console

Go back to the Google Cloud Console and navigate to the VM instances tab of the Compute Engine service:

You’ll see the list of your VMs that looks like this:

When your VM is created, it will have both an internal and external IP address:
- The internal IP is used for communication within your Google Cloud network
- The external IP allows access from the internet (once properly configured).

Copy the External IP of your VM and paste it into a page of your browser:

Unfortunately, our web page isn’t accessible due to a connection timeout.

But why?

Google Cloud Platform implements the principle of least privilege across all its services, including VM instances. By default, new VMs are not accessible from the internet — a security-first approach that protects your resources.

Let’s fix it!

Now, let’s configure our web server to be accessible from the internet using its IP address and port.

Let’s make our web server available on the global internet so that anyone can reach it !

To see your web server in action, you’ll need to allow HTTP traffic:

1. Go to the Google Cloud Console
2. Navigate to “VPC Network” > “Firewall”
3. Click “Create Firewall Rule”
4. Name your rule (e.g., “allow-http”)

5. Set direction of traffic to Ingress and Action to Allow

6. Set target tags to “http-server”

7. Add source IP range: 0.0.0.0/0

“0.0.0.0/0” means “allow from any IP address” and that this is not recommended for production environments

8. Select “Specified protocols and ports” and enter “tcp:80”

9. Click “Create”

Then tag your instance:

gcloud compute instances add-tags my-first-instance --tags=http-server

Now you can access your web server by navigating to your VM’s external IP address in your browser.

Conclusion

We just deployed our first web server on Google Cloud Platform after creating a Virtual Machine using the service Compute Engine in a newly created project. We saw that Google implement the least privilege principle everywhere: APIs are disable dy default and VMs are not connected to internet. To enable the internet traffic we created a Firewall very easily with the service VPC Network.

So we’ve used 3 services : IAM to handle our own permissions, Compute Engine to create a VM and deploy a server and VPC Network to control our network traffic.

Stay tuned for the next article in this series, where we’ll explore more advanced Google Cloud Platform features and implementations.

Until then, happy clouding!

Introduction

Tech is moving fast, AI is moving even faster! However, most challenges companies face are still fundamental ones like:

• Building and deploying reliable apps
• Collecting, processing and analyzing internal/external data (big or not)
• Taking advantage of data with Machine Learning/advanced analysis
• And so on…

One skill that companies consistently demand today is cloud computing expertise for building, deploying, and scaling applications. Among the major cloud providers, Google Cloud Platform (GCP) stands out with its robust infrastructure, advanced data analytics capabilities, and cutting-edge machine learning services — all built on the same infrastructure that powers Google’s own products.

In this comprehensive guide, we’ll explore the fundamental concepts of Google Cloud Platform, its core services, and how to get started with your first GCP project. Whether you’re a developer, architect, or IT professional looking to expand your cloud expertise, this article will provide you with the foundation you need to navigate the Google Cloud ecosystem confidently.

What is Google Cloud Platform?

Google Cloud Platform is a suite of cloud computing services that runs on the same infrastructure that Google uses internally for its end-user products, such as Google Search, Gmail, and YouTube. GCP offers a wide range of services including computing, storage, networking, big data, machine learning, and the Internet of Things (IoT).

Key Benefits of Google Cloud Platform

• Global Infrastructure: GCP operates in 35+ regions and 106+ zones worldwide, enabling you to deploy applications close to your users.
• Security by Design: Security is built into Google’s infrastructure from the ground up, with layered security measures and encryption by default.
• Innovative Technology: Access to cutting-edge technologies like TensorFlow, BigQuery, and Spanner that were developed at Google.
• Pay-as-you-go Pricing: Only pay for the resources you use, with per-second billing for many services.

Core Google Cloud Concepts

Before diving into specific services, let’s understand some fundamental concepts that are central to working with Google Cloud Platform.

Projects, Organizations, and Resource Hierarchy

In Google Cloud, resources are organized hierarchically:

1. Organization: The root node of the hierarchy, typically representing your company.
2. Folders: Optional containers for grouping projects (e.g., departments or teams).
3. Projects: The basic organizational entity in Google Cloud. Each project is a separate compartment with its own APIs, billing, and access controls.
4. Resources: Individual services and components (VMs, databases, etc.) that live within projects.

This hierarchy allows for logical organization and enables granular access control and policy management.

Identity and Access Management (IAM)

Google Cloud IAM is the system that controls who can do what on which resources:

• Who: Google accounts, service accounts, Google groups, or domains
• What: Permissions defined in roles
• Which: Google Cloud resources

IAM uses a policy that binds members to roles on resources. The three types of roles are:

1. Predefined roles: Curated collections of permissions managed by Google
2. Basic roles: Legacy roles (Owner, Editor, Viewer) with broad permissions
3. Custom roles: User-defined sets of permissions tailored to specific needs

Understanding IAM is crucial for maintaining security in your cloud environment.

Regions and Zones

Google Cloud resources are hosted in multiple locations worldwide, divided into:

• Regions: Geographic areas containing zones (e.g., us-central1, europe-west1)
• Zones: Deployment areas within a region that are independent of other zones

When deploying resources, consider factors like latency, availability requirements, and data residency regulations in choosing the appropriate region and zone.

Virtual Private Cloud (VPC)

A Virtual Private Cloud is a virtual version of a physical network that provides:

• Global networking: Single VPC can span multiple regions
• Software-defined networking: No physical networking equipment to manage
• Private connectivity: Resources can communicate without public IP addresses
• Firewall rules: Control traffic flow to and from your resources

VPCs are the foundation of network security and isolation in Google Cloud.

Interacting with Google Cloud Platform

Google Cloud provides multiple ways to interact with and manage your cloud resources, each with its own advantages depending on your needs and workflow.

Google Cloud Console

The Cloud Console is a web-based graphical user interface for managing your Google Cloud resources:

• User-friendly: Intuitive interface with search, navigation, and dashboards
• Visual management: Graphical representation of resources and their relationships
• Integrated tools: Built-in Cloud Shell, monitoring dashboards, and logging
• No installation required: Access from any browser

The Console is ideal for beginners, visualizing complex architectures, and for tasks that benefit from a graphical interface.

Google Cloud SDK and Command Line Interface (CLI)

The Google Cloud SDK includes command-line tools for managing Google Cloud resources:

• gcloud: Main CLI tool for most Google Cloud services
• gsutil: Specialized tool for Cloud Storage
• bq: Command-line tool for BigQuery

The CLI offers several advantages:

• Automation and scripting capabilities
• Faster execution of repetitive tasks
• Integration with CI/CD pipelines
• Precise control over resource configurations

# Example of creating a VM instance using gcloud CLI
gcloud compute instances create my-vm \
  --machine-type=e2-medium \
  --zone=us-central1-a \
  --image-family=debian-11 \
  --image-project=debian-cloud

Cloud Shell

Cloud Shell is an online development and operations environment accessible from your browser:

• Browser-based command line with pre-installed Google Cloud SDK
• 5GB of persistent disk storage
• Pre-authorized access to your Google Cloud resources
• Built-in code editor and web preview functionality

Cloud Shell is perfect for when you need CLI access without installing anything locally.

Google Cloud APIs

For programmatic access, Google Cloud offers comprehensive REST APIs:

• Available for all Google Cloud services
• Client libraries in multiple languages (Python, Java, Go, Node.js, etc.)
• Authentication through service accounts or user credentials
• Fine-grained control over resources

# Example of using the Python client library to list Compute Engine instances
from google.cloud import compute_v1

def list_instances(project_id, zone):
    instance_client = compute_v1.InstancesClient()
    instances = instance_client.list(project=project_id, zone=zone)
    for instance in instances:
        print(f"Instance: {instance.name}")
list_instances("my-project", "us-central1-a")

Infrastructure as Code Tools

Google Cloud supports several infrastructure as code (IaC) tools:

• Terraform: Open-source tool with strong Google Cloud support
• Pulumi: Modern IaC tool using familiar programming languages
• Ansible, Chef, Puppet: Configuration management tools with Google Cloud support

These tools allow you to define your infrastructure in code, enabling version control, repeatability, and consistency across environments.

# Example Terraform code to create a Cloud Storage bucket
provider "google" {
  project = "my-project"
  region  = "us-central1"
}
resource "google_storage_bucket" "static_website" {
  name          = "my-static-website-bucket"
  location      = "US"
  force_destroy = true
  website {
    main_page_suffix = "index.html"
    not_found_page   = "404.html"
  }
}

Mobile App

The Google Cloud mobile app allows you to monitor and manage your resources on the go:

• View resource status and performance metrics
• Receive alerts and notifications
• Manage incidents and take basic actions
• Monitor billing and budget alerts

The app is available for both Android and iOS devices.

Essential Google Cloud Services

Now that we’ve covered the core concepts, let’s explore the essential services offered by Google Cloud Platform. These services are categorized into several main areas:

Compute Services

Compute Engine

Google’s Infrastructure as a Service (IaaS) offering provides virtual machines (VMs) with complete control over the operating system, configurations, and installed software. Key features include:

• Custom machine types to optimize cost and performance
• Preemptible VMs for cost savings on interruptible workloads
• Live migration during host maintenance events
• Per-second billing with sustained use discounts

Google Kubernetes Engine (GKE)

GKE is a managed Kubernetes service that simplifies container orchestration:

• Automated cluster management and scaling
• Multi-cluster support and multi-region deployments
• Integration with Google Cloud services
• Auto-upgrade and auto-repair capabilities

Cloud Run

Cloud Run is a fully managed compute platform for deploying and scaling containerized applications:

• Serverless: only pay for the resources you use
• Built on Knative, an open-source serverless platform
• Supports any language, any library, any binary
• Automatic scaling from zero to handle traffic spikes

App Engine

App Engine is a Platform as a Service (PaaS) for building scalable web applications and APIs:

• Standard environment with preconfigured runtimes
• Flexible environment for custom runtime support
• Automatic scaling and high availability
• Built-in services like Memcache, Task Queues, and Cron jobs

Storage Services

Cloud Storage

Object storage for storing and accessing unstructured data:

• Multiple storage classes (Standard, Nearline, Coldline, Archive)
• Global edge caching with Cloud CDN integration
• Strong consistency and durability guarantees
• Lifecycle management policies

# Create a storage bucket with gcloud
gcloud storage buckets create gs://my-unique-bucket-name \
  --location=us-central1 \
  --default-storage-class=STANDARD

Persistent Disk and Local SSD

Block storage volumes for use with Compute Engine and GKE:

• Persistent Disks: durable network storage
• Local SSDs: high-performance, temporary local storage
• Snapshots for point-in-time backups
• Regional persistent disks for high availability

Filestore

Managed file storage for applications that require a file system interface:

• NFSv3 compatible
• High-performance for file-based workloads
• Fully managed and scalable
• Integrated with GKE and Compute Engine

Database Services

Cloud SQL

Managed relational database service for MySQL, PostgreSQL, and SQL Server:

• Automated backups, replication, and failover
• Automatic storage increases
• Point-in-time recovery
• Private connectivity through VPC

Cloud Spanner

Globally distributed, horizontally scalable relational database with strong consistency:

• 99.999% availability SLA
• Combines the benefits of relational structure with horizontal scale
• Automatic sharding and replication
• Strong consistency across regions and continents

Firestore and Datastore

Flexible, scalable NoSQL database for mobile, web, and server development:

• Real-time updates and offline support (Firestore)
• Automatic scaling with no downtime
• ACID transactions
• Powerful querying capabilities

Bigtable

Petabyte-scale, wide-column NoSQL database service:

• Designed for high throughput and low latency
• Seamless scaling without downtime
• Open-source HBase API compatibility
• Ideal for time-series data, IoT, and analytical workloads

Memorystore

Fully managed in-memory data store services compatible with Redis and Memcached:

• Sub-millisecond latency
• High availability configuration options
• Automatic failover
• Scalable performance

Networking Services

Virtual Private Cloud (VPC)

Software-defined networking with global reach:

• Global private network for your GCP resources
• Subnet creation and IP address management
• VPC Network Peering for connectivity between VPCs
• Shared VPC for multi-project networking

Cloud Load Balancing

Distribute traffic across instances and regions:

• Global and regional load balancing
• HTTP(S), TCP/SSL, and UDP load balancing
• Autoscaling and health checks
• Seamless integration with Cloud CDN

Cloud CDN

Content delivery network built on Google’s global edge network:

• Accelerate content delivery
• Reduce serving costs
• Offload origin servers
• Support for HTTP/2 and QUIC

Cloud DNS

Scalable, reliable, and managed authoritative DNS service:

• 100% uptime SLA
• Global anycast name servers
• Low-latency DNS lookups
• Integrated with Cloud Console and gcloud CLI

Big Data and Analytics

BigQuery

Serverless, highly scalable data warehouse with built-in machine learning:

• Petabyte-scale querying
• Separation of storage and compute
• Real-time analytics
• Built-in ML capabilities with BigQuery ML

-- Example BigQuery SQL query
SELECT 
  date,
  COUNT(DISTINCT user_id) AS daily_active_users
FROM 
  `project_id.dataset.events_table`
WHERE 
  date BETWEEN '2023-01-01' AND '2023-01-31'
GROUP BY 
  date
ORDER BY 
  date

Dataflow

Unified stream and batch data processing service:

• Based on Apache Beam
• Horizontal autoscaling
• Built-in integration with other Google Cloud services
• Real-time analytics and ETL

Pub/Sub

Messaging service for event ingestion and delivery:

• At-least-once message delivery
• Push and pull delivery models
• Global by default
• Supports millions of messages per second

Dataproc

Managed Hadoop and Spark service:

• Quick cluster creation
• Integration with other Google Cloud services
• Custom machine types
• Per-second billing

Artificial Intelligence and Machine Learning

Vertex AI

Unified platform for building, deploying, and scaling ML models:

• Agent builder, Model registry, Notebooks, etc…
• Custom training for maximum flexibility
• MLOps tools for deployment and monitoring
• Feature Store for feature management

Conclusion

In this introductory guide, we’ve covered the fundamental concepts of Google Cloud Platform, from its organizational structure and core services to the various ways you can interact with the platform. These building blocks form the foundation for any cloud journey(Data Engineer, Data Scientist, Developer, etc…), regardless of your specific use case or industry.

I wrote this article because of the growing demand of cloud computing skills and students/juniors that are asking me a lot where to start their cloud journey to learn efficiently. I hope this can help have a first overview before diving into more specific areas.

See you for the next ones on Cloud Computing with GCP.

If you have questions, feel free to reach on LinkedIn.

Set up a local spark cluster in 2 commands using Docker Compose

Are you a developer, data scientist, or student looking to harness the power of Apache Spark without the complexity of a full-scale cluster?

This tutorial will guide you through creating a local Spark cluster using Docker, streamlining your development process and reducing overhead.

Why use Docker for a local Spark Cluster?

• Develop and test locally: Run Spark jobs on your machine without a full-scale cluster.
• Simulate production environments: Mirror real-world settings for consistent development and deployment.
• Learn and experiment: Set up a functional Spark environment to practice big data processing techniques.

By leveraging Docker volumes, we’ll ensure your data and configurations persist, making your local cluster a robust tool for continuous development and testing.

Let’s dive in and discover how Docker can make working with Spark easier and more efficient!

Prerequisites

Before we dive into setting up our local Spark cluster using Docker, ensure you have the following prerequisites :

• Docker: you need Docker installed and running on your machine. You can download it from Docker’s official site.
• Basic knowledge of Docker and Spark: Familiarity with Docker commands and a basic understanding of Apache Spark will help you follow this tutorial more effectively, even if running the commands will just work.

Why use a non-root container?

Non-root container images add an extra layer of security and are generally recommended for production environments. However, because they run as a non-root user, privileged tasks are typically off-limits.

Configuration

Custom Docker image

Create a new file named Dockerfile with the following content:

# Use the bitnami Spark image as it comes pre-configured with necessary Spark components
FROM bitnami/spark:latest
# Add a user to run the application
RUN useradd -ms /bin/bash spark
# Create directories for data and jobs
RUN mkdir -p /data/inputs /data/outputs /jobs
# Set the ownership of the directories to the spark user
RUN chown -R spark:spark /opt/bitnami/spark /data /jobs
# Set the user to run the application
USER spark

We create and set up a new user named spark to run the application to ensure that the container runs as a non-root user.

It’s a good practice to run containers as non-root users to enhance security and reduce the risk of privilege escalation attacks. For example, if a container is compromised, the attacker will have limited access to the host system.

Start the cluster

Docker Compose is a tool for defining and running multi-container Docker applications. It uses a YAML file to configure the application’s services, networks, and volumes, making it easier to manage and scale your application.

Environment variables Customizable environment variables

• SPARK_MODE: Spark cluster mode to run (can be master or worker).
• SPARK_MASTER_URL: Url where the worker can find the master. Only needed when spark mode is worker.

Create a new file named docker-compose.yml with the following content:

services:
  spark-master:
    build:
      context: .
    container_name: spark-master
    hostname: spark-master
    networks:
      - spark-network
    ports:
      - "8080:8080"
      - "7077:7077"
    volumes:
      - spark-inputs:/data/inputs
      - ./data/outputs:/data/outputs
      - ./jobs:/jobs
    environment:
      - SPARK_MODE=master
  spark-worker:
    build:
      context: .
    hostname: spark-worker
    networks:
      - spark-network
    environment:
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_MODE=worker
    volumes:
      - spark-inputs:/data/inputs
      - ./data/outputs:/data/outputs
      - ./jobs:/jobs  
networks:
  spark-network:
    driver: bridge
volumes:
  spark-inputs:

This configuration file defines two services: spark-master and spark-worker, each running the custom-spark image. The spark-master service runs in master mode, while the spark-worker service runs in worker mode and connects to the spark-master service.

To start the Spark cluster using Docker Compose, run the following command:

docker-compose up -d

This command starts the Spark cluster using the configuration defined in the docker-compose.yml file.

To start only the master, you can use the command:

docker-compose up -d spark-master

To start only the workers, the number of workers you want to start, you can use the command:

docker-compose up -d --scale spark-worker=2

This command starts two worker nodes in the Spark cluster.

Running a Spark Job

• Prepare folder structure

Create the following directories to store the inputs, outputs, and job scripts:

mkdir -p ./data/inputs ./data/outputs ./jobs

• Create the script

We’ll create a simple PySpark script that reads the Iris dataset, performs a basic data manipulation, and writes the results back to the spark-data volume.

Create a new file named spark_job.py with the following content:

from pyspark.sql import SparkSession  
# Initialize Spark session 
spark = SparkSession.builder.appName("IrisDataProcessing").getOrCreate()  
# Read the dataset 
df = spark.read.csv("/data/inputs/iris.csv", inferSchema=True, header=False)  
# Rename columns 
columns = ["sepal_length", "sepal_width", "petal_length", "petal_width", "species"] 
df = df.toDF(*columns)  
# Perform a basic data manipulation: calculate average sepal length by species 
avg_sepal_length = df.groupBy("species").avg("sepal_length")  
# Write the results back to the Docker volume 
avg_sepal_length.write.csv("/data/outputs/avg_sepal_length_by_species")  
# Stop the Spark session 
spark.stop()

• Getting the data

Let’s download the Iris dataset from the UCI Machine Learning Repository or any other source you prefer.

wget https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data -O data/inputs/iris.csv

• Copy the data and the script to the cluster

Input data: Copy the Iris dataset to the Spark master container.

docker cp data/inputs/iris.csv spark-master:/data/inputs/iris.csv

That’s the output you should see:

PySpark script: we don’t need to copy the script to the master container, we can just move it to the jobs folder that is already mounted to the master container.

cp spark_job.py .jobs/spark_job.py

Verifying Data Accessibility To verify that the data is accessible, you can start an interactive session within the Spark master container:

docker exec -it spark-master ls /data/inputs /jobs

Inside the container, list the contents of the /data/inputs and /jobs directory to ensure the dataset is present.

You should see this output:

Job Execution

Execute the script using the Spark master container:

docker exec -it spark-master spark-submit \
  --master spark://spark-master:7077 \
  --deploy-mode client /jobs/spark_job.py

This command runs the PySpark script on the Spark cluster, reading the dataset from the volume, processing it, and writing the results back to the volume.

The output should look like this:

Once the job is finished, you can check the results:

docker exec -it spark-master ls /data/outputs/avg_sepal_length_by_species

You should see the output files in the avg_sepal_length_by_species directory.

The output files contain the results of the Spark job, showing the average sepal length by species. You can display the contents of the output files to view the results:

docker exec -it spark-master cat /data/outputs/avg_sepal_length_by_species/part-00000-9b270837-872e-4a34-bb5b-4cb6d52044be-c000.csv

Warning: The output file name may vary depending on the Spark job execution. Replace part-00000–9b270837–872e-4a34-bb5b-4cb6d52044be-c000.csv with the actual file name in your output directory.

Tips and Tricks

• Alias the spark-submit command: To simplify running Spark jobs, you can create an alias for the spark-submit command. For example, you can add the following line to your .bashrc or .bash_profile file:

alias spark-submit='docker exec -it spark-master spark-submit --master spark://spark-master:7077 --deploy-mode client'

With your cluster up and running, if you run the command:

spark-submit /jobs/spark_job.py

It will execute the Spark job on the cluster without needing to specify the master URL each time.

Stopping the Cluster

To stop the Spark cluster, you can use the following commands:

docker-compose down

Setting up a local Spark cluster using Docker can streamline your development process, enabling you to test Spark jobs, simulate production environments, and experiment with Spark features in a controlled environment.

By using Docker volumes, you can ensure that your data and configurations persist across restarts, making your local cluster a robust tool for continuous development and testing.

This approach is beneficial not only for Data Engineers but also for Data Scientists who work with Scala or PySpark. It allows for incremental development without heavily utilizing resources from a real cluster on any cloud platform, as it was the case in my previous job.

You can find the entire code there.

Let’s connect!

5 minutes to create your own Spark Cluster: A Step-by-Step guide was originally published in Data Engineer Things on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ensuring your Python project works smoothly, especially the complex ones, means making sure all dependencies are present with the correct versions. Most projects use a requirements.txt file for this.

In large or complex projects, I’ve often faced issues with this file — especially when updating from another file, dealing with multi-projects in a single repo, or managing conflicts.

In this article, I’ll introduce you to reqs-check, a tool designed to help simplify dependency management. We’ll explore its benefits, how to use it, and how to integrate it into your development workflow, including pre-commit hooks and CI/CD pipelines.

Why use reqs-check?

In complex projects, dependency issues can arise from multiple sources:

• Inconsistency between requirements files of different branches/workspaces of the same project
• Duplicate packages with different versions
• Missing packages across different environments
• Inconsistent package versions causing unexpected behaviors

reqs-check helps you identify and resolve these issues efficiently, ensuring smoother development and deployment processes.

Advantages of reqs-check

1. Easy comparison: Quickly compare multiple dependencies files to spot differences and inconsistencies.
2. Duplicate detection: Identify and resolve duplicate packages within your dependencies files.
3. Automation ready: Easily integrate with pre-commit hooks and CI/CD pipelines for automated checks.
4. Extensible: Future plans include support for additional file formats, security checks, and detailed reporting.

Installing reqs-check

Getting started with reqs-check is simple. You can install it using pip:

pip install reqs-check

How to use reqs-check?

Basic usage

reqs-check provides a command-line interface for checking and comparing dependencies files. Here's a breakdown of its usage:

usage: reqs-check [-h] [--only-diff] [--find-duplicates] F [F ...]

Positional arguments:

• F: Paths to the dependencies files

Options:

• -h, --help: Show this help message and exit
• --only-diff: Only show packages with differing versions
• --find-duplicates: Find duplicate packages in the dependencies files

Basic Commands

reqs-check provides several useful commands to help manage your dependencies:

1. Compare dependencies files: Compare multiple dependencies files to find differences.

reqs-check requirements1.txt requirements2.txt

2. Show only differing packages: Display only the packages with differing versions between files.

reqs-check --only-diff requirements1.txt requirements2.txt

3. Find duplicate packages: Identify duplicate packages within your dependencies files.

reqs-check --find-duplicates requirements.txt 

Examples

Let’s dive into a practical example. Consider two files:

requirements1.txt

pandas==1.1.5
numpy==1.19.5
scipy~=1.5.4
matplotlib==3.3.4

requirements2.txt

pandas==1.1.5
numpy==1.19.3
matplotlib==3.2.1
requests>=2.24.0
scipy==1.5.4
numpy==1.19
requests

To compare these files, you would run:

reqs-check requirements1.txt requirements2.txt

To focus only on the packages with differing versions, use:

reqs-check --only-diff requirements1.txt requirements2.txt

And to find duplicates within the files:

reqs-check --find-duplicates requirements1.txt requirements2.txt

As you can see, we can look for duplicates in many files at time.

Integrating reqs-check into your workflow

Pre-commit Hooks

Pre-commit hooks can catch issues before they make it into your version control system. Here’s how you can integrate reqs-check into your pre-commit workflow.

Let’s take this requirements.txt file as an example:

pandas==1.1.5
numpy==1.19.3
matplotlib==3.2.1
scipy==1.5.4
numpy
requests>=2.24.0

The package numpy is listed twice there and we’ll try to commit it into our repository.

Let’s move in a git repository.

1. Create the file dev/requirements.txt and add the content of the example
2. Create a .pre-commit-config.yaml file in your project root and add this:

repos:
-   repo: local
    hooks:
    -   id: reqs-check
        name: reqs-check
        entry: reqs-check --find-duplicates
        language: system
        files: ^.*requirements.*\.txt$

3. Install pre-commit and set up the hook:

pip install pre-commit

4. Add the changes to git:

git add .pre-commit-config.yaml dev/requirements.txt

5. Commit the changes

git commit -m "Add the dependencies"

7. Fix and resolve

Let’s fix it, removing the non versioned occurrence of numpy at the line 5

pandas==1.1.5
numpy==1.19.3
matplotlib==3.2.1
scipy==1.5.4
requests>=2.24.0

Add the changes and commit again:

git add dev/requirements.txt
git commit -m "Add the dependencies"

There is the result:

CI/CD Integration

Automating dependency checks in your CI/CD pipeline ensures consistent and reliable deployments. Here’s an example using GitHub Actions:

1. Create a workflow file .github/workflows/dependency-check.yml:

name: Dependency Check

on: [push, pull_request]
jobs:
  dependency-check:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v2
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: '3.x'
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install reqs-check
    - name: Run reqs-check
      run: reqs-check --find-duplicates requirements.txt

This workflow will run reqs-check on every push or pull request, ensuring that any issues with dependencies are caught early.

Future Developments

The current version of reqs-check focuses on comparing and cleaning up dependencies files, but we have exciting plans for the future:

• Linting for best practices: Ensure your dependencies files follow best practices.
• Support for additional formats: Including Pipfile and pyproject.toml.
• Security checks: Identify and address vulnerabilities in your dependencies.
• Detailed reports: Generate comprehensive reports in formats like JSON and HTML.
• CI/CD Integration: Enhanced support for various CI/CD systems.

Conclusion

Managing dependencies is a critical part of any Python project, and reqs-check provides a quite great toolset to make this task easier and more efficient. By integrating reqs-check into your workflow, you can ensure consistency, avoid conflicts, and maintain a healthier codebase.

Try out reqs-check today and see how it can simplify your dependency management process!

Don’t hesitate to provide some feedbacks on this package and how helpful it can be!

About me

You can follow me, contact me, ask questions or whatever I can help you for:

• GitHub
• Linkedin
• Medium

reqs-check: The Python package I wish I had known about earlier was originally published in Towards Dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

While useful, there are times when you need to add files to a running container without stopping it or rebuilding the image. Let’s explore alternative methods for transferring files into your Docker container.

This guide is ideal for any Data Guy or Software Engineers that want to develop in a container or try and update code with various data without stoping their container.

Let’s go!

Prerequisites

Before starting, make sure you have:

• Basic understanding of Docker (this tutorial can help)
• Docker installed on your machine

Getting Started

Here are 3 methods to copy files into a running container without using the COPY command in your Dockerfile:

1. Using docker cpcommand
2. Using bind mounts
3. Using wget or curl

Preparations

Before diving into the methods, let’s do some preparation:

1. Download this csv data that we’ll use and save it to your workspace
2. Run a FastAPI container where we’ll push our data and check their presence using a specific endpoint.

Pull the Docker image for tutorial

docker pull ghcr.io/bricefotzo/docker-init-example:main

Run the container

docker run -d --name my-api -p 8000:8000 ghcr.io/bricefotzo/docker-init-example:main

Navigating to the URL localhost:8000 in your browser, you’ll see this:

The one we’ll use to check data is the second(/tree).

You can try this endpoint to check what is inside the folder of the app: app.

As you can see, listing the directory of the project(/app), we have 4 files and 1 folder. In the following methods, we’ll use different methods to check data availability.

Let’s start transferring files !

Method 1: Using docker cp Command

With my-apicontainer running, we’ll add data in it without stopping it or rebuilding the image.

The docker cp command allows you to copy files or directories between the host and a container's filesystem, even after the container is running.

How to use docker cp:

Usage:

docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-

docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH

Copy a file from host to container:

docker cp ./iris.csv my-api:/app
#Successfully copied 6.66kB to my-api:/app

Copy a directory from host to container:

docker cp ./data my-api:/app 
#Successfully copied 12.8kB to my-api:/app

As you can see, 2 items were added: iris.csv and data.

Checking in /app/data, we have 2 files: iris.csv and iris2.csv.

Copy a file from container to host:

Let’s copy the file main.py into our host and let’s check what is inside.

docker cp my-api:/app/main.py ./
#Successfully copied 2.56kB to /Users/<username>/workspace/docker-init-example/data/./

When to use docker cp :

• When you need to update files in a running container without rebuilding the image.
• For temporary changes or debugging purposes.

Method 2: Using Bind Mounts

Bind mounts allow you to mount a directory from the host filesystem into the container. This method provides a direct mapping between the host and container, so changes in the host directory are immediately reflected in the container and vice versa.

How to use bind mounts:

Create a workspace on our host and add some files:

mkdir api-data
touch api-data/data1.csv
touch api-data/data2.csv

Start another API container with volume mounts:

docker run -d --name my-api2 -v ./api-data:/app/data/api -p 8001:8000 ghcr.io/bricefotzo/docker-init-example:main

The container is named my-api2 and the volume bind is done between the local folder api-data that we just created and the container folder /app/data/api.

Note: The port binding is changed to avoid conflict with the previous container. You can clean up the previous one and continue using the same container name and port if you want.

Let’s just add another file into the local directory api-data and check the content of /app/data/api.

touch api-data/data3.csv

Let’s check:

When to use bind mounts:

• For development environments where you want to reflect code changes in real-time without rebuilding the image.
• When you need persistent storage that directly maps to the host filesystem.

Method 3: Using wget or curl

Sometimes, it’s more efficient to download files directly from the internet into your container using tools like wgetor curl.

How to use wget or curl:

Download a file using docker exec and wget:

docker exec -it my-api2 /bin/sh -c "wget https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data -O /app/data/api/iris.csv"

Output:

--2024-06-12 18:29:52--  https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data
Resolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.10.252
Connecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.10.252|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified
Saving to: '/app/data/api/iris.csv'

/app/data/api/iris.csv                         [ <=>                                                                                    ]   4.44K  --.-KB/s    in 0.001s  

2024-06-12 18:29:54 (3.36 MB/s) - '/app/data/api/iris.csv' saved [4551]

Download a file using docker exec and curl:

docker exec -it my-api2 /bin/sh -c "curl -o /app/data/api/iris2.csv https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"

Output:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4551    0  4551    0     0   8538      0 --:--:-- --:--:-- --:--:--  8586

Then let’s check if the data have been downloaded in the target folder.

When to use wget or curl:

• When you need to download files from a remote server directly into the container.
• For small files or configuration scripts needed at runtime.

Conclusion

While the COPY command in a Dockerfile is useful for static file transfers during image build time, these alternative methods offer greater flexibility for various scenarios:

• docker cp for on-the-fly file transfers.
• Bind mounts for real-time development and persistent storage.
• wget or curl for downloading files directly into the container.

By utilizing these methods, you can manage file transfers more effectively in your Docker workflows, making your tasks as a data engineer more efficient and streamlined.

Let’s connect to continue the discussion!

How to copy files into a Docker container without using the COPY command was originally published in Towards Dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Imagine having docker containers running, and you want to update or restart it without losing the data you have inside. How would you manage that? This is a common challenge many faces when first diving into Docker. However, Docker volumes offer an elegant solution to this problem by ensuring your data persists beyond the lifecycle of a container.

In this article, we’ll explore what Docker volumes are, why they are indispensable, and how you can use them effectively to manage your data. Whether you’re new to Docker or looking to deepen your understanding, this guide will provide you with practical insights and examples.

Table of content

• Why using Docker volumes?
• Creating and managing volumes
• Using volumes with containers
• Cleaning up volumes

What makes Docker volumes essential?

Docker volumes are designed to persist data created by and used within Docker containers. Unlike a container’s writable layer, data in volumes exists outside the container lifecycle, ensuring it remains intact even if the container is deleted.

Why use Docker volumes?

1. Data persistence: Volumes ensure your data is not lost when containers are removed. This persistence is critical for applications that need to retain state or logs across container restarts.
2. Multi-Container sharing: Volumes can be shared among multiple containers, making them ideal for scenarios where data needs to be accessed by more than one container.
3. Space efficiency: Since volumes exist outside the container’s writable layer, they do not inflate the size of your Docker images, leading to more efficient storage management.
4. Lifecycle independence: Volumes have their own lifecycle, separate from the containers that use them, which adds flexibility in managing your data.

Creating and managing volumes

Creating a volume

To create a new volume, you use the docker volume create command:

docker volume create my_volume

This command creates a volume named my_volume.

Listing volumes

To see all available volumes, you can list them with:

docker volume ls

The result looks like this:

Inspect volumes

Let’s inspect our created volume named my_volume . The command to run is:

docker volume inspect my_volume

We have various information about the volume, especially the path where to find the volume data on the local machine.

Using volumes with containers

Running a container with a volume

When running a container, you can mount a volume to a specific directory within the container using the -v flag:

docker run -d --name my_container -v my_volume:/data nginx

This example runs a Nginx container and mounts my_volume to the /data directory inside the container.

You can check if the container created has the volume my_container with:

docker inspect -f "{{ .Mounts }}" my_container

Sharing volumes between containers

You can share volumes between containers, which is useful for applications requiring shared storage:

docker run -d --name my_container_2 --volumes-from my_container nginx

Here, a new nginx container my_container_2 is run, attaching the volumes from an existing container named my_container. Let’s check it with:

docker inspect -f "{{ .Mounts }}" my_container_2

Mounting many volumes in a container

You can mount many volumes to a single container like this:

docker run -d --name my_container_3 -v volume1:/data1 -v volume2:/data2 nginx

Let’s check:

We have in fact 2 volumes for the container my_container_3 .

Mounting a volume as Read-Only

For scenarios where you want to prevent modifications to the data, you can mount a volume with read-only access:

docker run -d -v my_volume:/data:ro nginx

This mounts my_volume in read-only mode at /data using the keyword ro. By default, the keyword rw for read-write is the one used.

Cleaning up volumes

Deleting a specific volume

To remove a volume, you use:

docker volume rm my_volume

Running a container with temporary volumes

In some cases, you might want to use volumes temporarily and delete them once the container stops:

docker run --rm -v temp_volume1:/data1 -v temp_volume2:/data2 nginx

The --rm flag ensures that temp_volume1 and temp_volume2 are removed after the container exits.

Pruning unused volumes

To clean up all unused volumes and free up space:

docker volume prune

This command removes all volumes that are not currently in use by any container.

Conclusion

Docker volumes are an essential feature for any Docker user, providing a robust solution for data persistence, sharing, and efficient storage management. By mastering the use of Docker volumes, you can ensure your data is safely managed and accessible, even as your containers come and go. With the practical examples and commands provided, you’re now equipped to leverage Docker volumes to their full potential in your projects.

Let’s connect to continue the discussion! 😉

Introduction

Working with RSS feeds can be challenging, especially when integrating different libraries and tools. Recently, I encountered issues while trying to run RSS extraction code using Langchain and Newspaper3k. This article walks through the debugging process and how I resolved the errors.

Initial Setup and Encountered Errors

I started by setting up my environment and writing a simple script to extract RSS feeds. Despite installing newspaper3k, I encountered the following error:

Error processing entry https://github.com/google-deepmind/penzai, exception: newspaper package not found, please install it with `pip install newspaper3k`

Using Newspaper3k

I decided to test Newspaper3k independently with a simple script:

import newspaper

cnn_paper = newspaper.build('http://cnn.com')

for article in cnn_paper.articles:
    print(article.url)

This led to another error:

Traceback (most recent call last):
  File "D:\workspace\rss\try.py", line 1, in <module>
    import newspaper
  File "D:\workspace\rss\.venv\lib\site-packages\newspaper\__init__.py", line 10, in <module>
    from .api import (build, build_article, fulltext, hot, languages,
  File "D:\workspace\rss\.venv\lib\site-packages\newspaper\api.py", line 14, in <module>
    from .article import Article
  File "D:\workspace\rss\.venv\lib\site-packages\newspaper\article.py", line 15, in <module>
    from . import network
  File "D:\workspace\rss\.venv\lib\site-packages\newspaper\network.py", line 14, in <module>
    from .configuration import Configuration
  File "D:\workspace\rss\.venv\lib\site-packages\newspaper\configuration.py", line 15, in <module>
    from .parsers import Parser
  File "D:\workspace\rss\.venv\lib\site-packages\newspaper\parsers.py", line 12, in <module>
    import lxml.html.clean
  File "D:\workspace\rss\.venv\lib\site-packages\lxml\html\clean.py", line 18, in <module>
    raise ImportError(
ImportError: lxml.html.clean module is now a separate project lxml_html_clean.
Install lxml[html_clean] or lxml_html_clean directly.

Resolving the errors

Step 1: Install Missing Dependencies

The error suggested installing lxml_html_clean, which I did using the following command:

pip install lxml_html_clean

Step 2: Verify Installation

After installation, I reran the Newspaper3k script, which worked without issues:

import newspaper

cnn_paper = newspaper.build('http://cnn.com')
for article in cnn_paper.articles:
    print(article.url)

Step 3: Retry Langchain Code

With Newspaper3k working, I retried the original Langchain code. This time, it executed successfully without any errors.

Conclusion

Debugging integration issues can be time-consuming, but systematically isolating and testing components helps identify and resolve problems effectively. Ensuring all dependencies are correctly installed and up-to-date is crucial for smooth execution. I hope this walkthrough helps others facing similar issues with RSS extraction and library integrations.