I landed in Göttingen with a suitcase and a phone. My German was close to zero. I had to do very real things in German: activate a SIM card, ask for a seat on the train, show my ticket to an inspector, and sit across from a landlord and tell about myself.

I reached several tools, especially Google’s tools. It is built from three pillars, Gemini, NotebookLM, and Google Cloud. It works not only for a language learning but for any subject. And you can set it up tonight.

“AI is my sparring partner. The street is the real match.”

The one rule that makes AI actually work

Most people use AI and still do not learn. They read a perfect answer, nod, and forget it. The fix is a loop:

PREPARE with AI  ->  PRACTICE in the real world  ->  FEEDBACK  ->  (repeat)

Here is the story that taught me this.
Activating a SIM card.
I could not explain my problem to the worker. First try, fail. Second, fail. By the fifth try I finally said it clearly and it worked. No grammar book corrected me. The situation did. Every confused look and every “wie bitte?” was a signal telling my brain what to change.

In my field we call that reinforcement learning: you act, you get a reward signal, you update. Your brain does the same thing with a real conversation. So the job of AI is not to replace the conversation. It is to get you ready for it, and to help you dissect it afterwards.

“Every ‘wie bitte?’ is training data.”

Now the toolchain.

Pillar 1: Capture reality, transcription with Google tools

You learn fastest from your own words and your own life. That means turning speech and the real world into text you can work with. Google gives you three tiers, from zero effort to full power.

Tier 1, zero setup: Google Docs Voice Typing and the Recorder app

Google Docs Voice Typing is free and instant. Open a Doc, go to Tools > Voice typing, pick German (Deutschland), and speak. It writes what it hears. Speak German for a minute and read back what actually came out. Where the text is wrong, either you mispronounced it or you used the wrong word. That is your first free feedback.

OpenAI’s ChatGPT also could be used normally.

On mobile phone the Recorder app transcribes as you record and lets you search inside your recordings. Record yourself or a real conversation you are allowed to record, and you get text you can paste into Gemini.

Tier 2, upload and ask: Gemini transcribes audio directly

Gemini is multimodal. In the Gemini app or in Google AI Studio you can upload an audio file and simply say: “Transcribe this German audio, then list every grammar mistake I made.” One step, transcript plus correction.

Other options: turboscribe.ai and elevenlabs.

Tier 3, power user: Google Cloud Speech-to-Text with Chirp

When you want accuracy and volume, use Cloud Speech-to-Text. Its universal Chirp model is trained on millions of hours across 120+ languages, German included, and Google gives you 60 minutes of transcription free every month, with no expiry. Async transcription handles files up to 480 minutes.

A minimal Python call looks like this:

# pip install google-cloud-speech
from google.cloud import speech_v2 as speech
from google.cloud.speech_v2.types import cloud_speech as t
client = speech.SpeechClient()
with open("meine_aufnahme.wav", "rb") as f:
    audio_bytes = f.read()
config = t.RecognitionConfig(
    auto_decoding_config=t.AutoDetectDecodingConfig(),
    language_codes=["de-DE"],
    model="chirp_2",
)
req = t.RecognizeRequest(
    # use a regional recognizer, e.g. europe-west4, see the docs
    recognizer="projects/PROJECT_ID/locations/europe-west4/recognizers/_",
    config=config, content=audio_bytes,
)
for result in client.recognize(request=req).results:
    print(result.alternatives[0].transcript)

Not a coder? You never touch this. Tier 1 and Tier 2 are enough for 95% of learners. The point is: your voice becomes text, and text is something AI can teach you from.

Pillar 2: Gemini, your always-on tutor

Gemini is the daily driver. Five features carry the whole workflow.

1. Gemini Live: talk out loud, hands free

Open the Gemini app, tap Live, and just have a conversation in German. This is your zero-stakes rehearsal before a real one. My go-to opener:

“Let’s talk in German at B1 level about apartment hunting. Ask me questions like a landlord would. Correct me gently as we go, and keep it slow.”

2. Gems: a saved tutor that knows you

A Gem is a custom Gemini you configure once and reuse forever. Open Gemini, expand the left menu, click Gems > New Gem, paste instructions, and (the killer part) connect it to a Google Drive folder so it reads your notes live. Update a Doc, and your tutor sees the change instantly.

Here is the instruction I use. Copy it into a Gem:

You are "Mein Deutschcoach", my personal German tutor. My level is B1.2, goal C1.
I am an English-fluent software engineer living in Germany.
Every session:
- Explain in clear English, keep all German examples correct (a o u umlauts and ss).
- Correct every German mistake I make, briefly, and keep a running error list.
- Teach new words in small batches of 5, then quiz me before moving on.
- Prefer real, everyday phrasing over textbook German.
- When I paste a transcript of my own speech, give me: a corrected version, a better
  B2 version, a table of my mistakes, and 5 drills targeting exactly those mistakes.
Read my linked Drive folder for my current vocabulary and my error list.
Start by asking what we work on today and how many minutes I have.

3. Gemini inside Google Docs: fix your writing in place

Every real German text I must write, an email to a housing platform, a message to a landlord, I draft myself first, then in the Doc I use Help me write and Help me refine to polish it and, more importantly, ask it to explain each change so I learn. You never leave the document.

4. Canvas: build a living study sheet

Canvas is an editable side-by-side workspace. Ask Gemini to “create a one-page B1 cheat sheet for two-way prepositions” in Canvas, then keep refining it in place instead of copying answers around. It can also build small interactive apps, a flashcard quiz for your words, for example.

5. Deep Research: understand any topic properly

Deep Research runs many searches and returns a cited report. I use it for grammar deep dives (“give me a complete, cited guide to German cases with common mistakes”), and for life admin (“how does the Anmeldung and SCHUFA process work for a new resident”). Same feature, any subject.

“I prepare with Google. I practice with Germans. I learn from the reaction.”

Bonus for builders: Google AI Studio (free)

If you want to go further, Google AI Studio (aistudio.google.com) is a free playground for the Gemini API. Set a system instruction, upload text/image/audio/video, test prompts, and click Get code to export a working Python or JavaScript snippet. I used it to prototype a tiny script that takes my daily voice note and returns a corrected version plus flashcards. Free tier note: on the free plan your prompts can be used to improve Google’s products, so keep anything sensitive out of it or use a billed project.

Pillar 3: NotebookLM, turn your material into a podcast, a video, and slides

This is the tool that surprises everyone. NotebookLM reads sources you give it, your lesson PDFs, your notes, a website, a YouTube link, up to 50 sources, and turns them into study material grounded in those sources with citations.

How to start: go to notebooklm.google.com, create a notebook, and add sources (upload files, paste a link, or import Google Docs and Slides straight from Drive). Then generate:

Output What you get My use Audio Overview a podcast-style discussion, now in 50+ languages, with interactive mode a German podcast of this week’s lesson for the train Video Overview a narrated slide presentation a visual recap of a grammar topic Mind Map the structure of the material see how a chapter connects Study guide / Quiz / Flashcards ready revision material self-test before class

Audio Overview is also built into the Gemini app now, powered by the free Deep Research model (free accounts get a monthly limit).

The language trick: set the Audio Overview output language to German for immersion, then generate it again in English to check you understood. Customization prompt I paste:

Speak slow, clear German at B1 level. Topic: this week's lesson.
Explain the grammar simply, use everyday examples (asking for a seat on a train,
a doctor visit, an apartment viewing), and repeat the key vocabulary twice.

The everyday Google apps that teach you all day

The big three do the heavy lifting. These four do the quiet, constant work.

• Google Lens turns the physical world into a lesson. Point it at a poster, a form, a product label, and read it in German with the translation next to it. Context makes words stick.
• Google Keep is my “overheard German” notebook. When I hear a phrase I do not know, I dump a quick voice note. At night I paste the list into Gemini: “Decode these phrases I heard today, explain the meaning and when to use them.”
• Google Calendar holds the loop together: a daily 10-minute review reminder and one recurring “speak to a real person today” nudge.

Store it once, let everything read it: Drive and Cloud

The system dies if it lives in 40 browser tabs. Put everything in one Google Drive folder: your lesson notes, your recordings, your running error list. Then:

• Your Gemini Gem reads that folder live.
• NotebookLM imports those Docs directly as sources.
• Google Cloud Storage holds longer audio for Speech-to-Text.
• Everything syncs across your phone and laptop automatically.

One source of truth, read by every tool. That is the whole architecture.

My weekly loop

When Action Tool Every lesson Record it, transcribe, turn into a clean note Recorder / Cloud STT + Gemini 3x a week Speak 1 minute, transcribe, get corrected Docs Voice Typing + Gem Daily 10-minute review + one real German interaction Calendar + the street Weekly This week’s notes become a German podcast + quiz NotebookLM Before a real event Roleplay it (landlord, phone call) Gemini Live

This is not only for language

Swap “German” for “Machine Learning” and nothing else changes. For my Master’s I drop research papers into NotebookLM and get an audio overview to hear the argument on a walk. I use Deep Research to map a topic before I read deeply. I keep a Drive folder per course that my Gemini Gem tutors me from. I transcribe my own spoken explanation of a concept and ask Gemini where my understanding has gaps.

The method is universal: capture your input, let AI structure and quiz you, produce something, then test it against reality. Language just gives you the fastest, most honest feedback, because a real person reacts in one second.

Start today: a 30-minute setup

1. Create one Google Drive folder called Deutsch (or your subject).
2. Build a Gemini Gem with the instruction above and connect it to that folder.
3. Open NotebookLM, create a notebook, add one document, and generate an Audio Overview.
4. Install Google Translate, download German offline, and pin the Phrasebook.
5. Put a Recorder / Voice Memo shortcut on your home screen.
6. Add two Calendar reminders: daily review, and “speak to one real person”.

Then do the only step that matters: go outside and use it. Order in German. Ask a question you already know the answer to. Get a little bit into trouble. Come back, transcribe what went wrong, and let the tools help you fix it.

“Do not let AI make you passive. Prepare with it, then go collect real feedback. That is where learning actually happens.”

If this helped, I share more about building, learning, and moving countries on YouTube at @inneovation and on LinkedIn at /in/jepbarov. Viel Erfolg.

#AI #Gemini #NotebookLM #GoogleCloud #GoogleAI #LanguageLearning #LearnGerman #DeutschLernen #GoogleDeveloperExpert #GDE #DAAD #StudyInGermany #Productivity #LifelongLearning #EdTech #AItools #Multilingual #DigitalNomad #SelfLearning #GenerativeAI

Sources and further reading

• NotebookLM features and 50+ language Audio Overviews: DigitalOcean guide, TechRadar on languages
• Audio Overview in Gemini: TechRadar
• Gemini Gems, Canvas, Deep Research: Google Workspace practical guide, Fresh van Root
• Google AI Studio free API: CodeForGeek, reskilll 2026 guide
• Cloud Speech-to-Text, Chirp, free tier: Chirp 3 docs, Speech-to-Text pricing
• Google Translate features: Google Translate Help, offline, AllTechNerd camera/voice/offline

Back in 2024, we had a pretty simple problem.

We needed a proper Knowledge Base.

Not just a folder full of Markdown files. Not another collection of random documents. We needed something where we could organize courses, documentation, homework, videos and user access in a way that actually made sense.

That was when I seriously discovered BookStack.
And the interesting part is that I still use it today.

BookStack: simple, but structured

For our Erkin Ugur environment, BookStack turned out to be a surprisingly good fit. Especially for courses.

When we launch a new course, we can create a dedicated user group, add people manually or through automated registration, and then control access to the relevant content.

The structure itself is also a very natural fit for education.

Book → Chapter → Page.

A book becomes a course. A chapter becomes a section. A page becomes a lesson.

Inside those pages, we can use Markdown, HTML, images, videos and additional materials. We can also rely on external storage when we don’t want to keep everything inside the Knowledge Base itself.

It is simple.
But it is structured.
And that combination is exactly what I like about it.

You read the material. You complete a homework task. You add something. You come back months later and you can still understand where everything is. For me, that is one of the most important jobs of a Knowledge Base.

It should not only help you find information today.
It should preserve the knowledge and context you built yesterday.

Wiki.js: flexibility with a price tag

We also used Wiki.js for another type of problem.

And Wiki.js is a great project.

In one of our use cases, we needed a more architectural approach and the ability to work with the system through GraphQL. For that particular scenario, Wiki.js gave us the flexibility we needed.

But flexibility comes with a price.

There are a lot of configuration options and a lot of things you can customize in Wiki.js. That is great when you actually need them.
But sometimes you don’t want to configure a hundred things.

Sometimes you just want to install a Knowledge Base, create a structure, add users and start documenting.
This is where BookStack wins for me.
It may not be the most flexible Knowledge Base system out there.
But it is incredibly easy to turn it into a working tool.

So, which one?

So if I had to choose today, my recommendation would be pretty simple.

If you need a clean and straightforward Knowledge Base for documentation, courses, internal materials or learning programs, I would definitely give BookStack a try.

If you need deeper customization, a more complex architecture, integrations and a stronger API-driven approach, and you have the time to work through a much larger configuration surface, Wiki.js might be the better choice.

Both are great open source projects.
They are just built around slightly different priorities.

A third path: staying inside Google

But there’s a third path I’ve been experimenting with, and it’s worth mentioning.

Not everyone wants to run a server, patch a container, or think about backups. Sometimes the honest answer is: don’t self-host anything at all.

This year I spent some time with NotebookLM Enterprise, and it’s a genuinely different animal from BookStack or Wiki.js. You’re not building a book with chapters and pages. You’re feeding it source material, PDFs, docs, transcripts, and it becomes a layer that answers questions grounded in exactly those sources, with citations back to where the answer came from. Google wraps it with IAM and VPC Service Controls, so a course’s materials can stay scoped to the people who should actually see them, the same way a BookStack user group would.

It’s not a replacement for a structured course library. There’s no Book → Chapter → Page hierarchy, no homework tracking, no sense of “this page belongs after that page.” What it’s good at is a different job entirely: someone asks a question, and the answer comes back grounded in your material instead of the model guessing.

For plain internal documentation that a team edits together, Google Sites still does the job it always did. It’s not exciting, and it won’t feel like a Knowledge Base in the BookStack sense, but if all you need is pages, some structure, and zero maintenance, it’s there and it’s free with a Workspace account.

So that’s the third option, and it’s really a different question than “BookStack or Wiki.js.” It’s “do I want to own this, or do I want Google to own it.” If you don’t want to think about updates, backups, or a database at all, and you’re fine with your content living inside Google’s ecosystem, that’s a completely reasonable choice. If you want the Book → Chapter → Page structure, the course-shaped access control, and the ability to move your content wherever you want, self-hosting is still the way I’d go.

Self-hosting without babysitting a server

And if you do want to self-host, but you don’t want to manage a VM either, there’s a middle ground worth knowing about.

Both BookStack and Wiki.js run fine as containers, and Google Cloud gives you two reasonable ways to run them without babysitting a server.

Option 1: Cloud Run, the quick way. Push the image to Artifact Registry, point Cloud Run at it, and let Google handle scaling. The catch is that Cloud Run expects a stateless container, and BookStack or Wiki.js are not stateless on their own, they need a database and somewhere to keep uploaded files. The fix is to split those pieces out: a Cloud SQL instance for MySQL or MariaDB, and either Cloud Storage through an S3-compatible bucket or a mounted volume for uploads. Roughly:

gcloud sql instances create bookstack-db --database-version=MYSQL_8_0 --tier=db-f1-micro --region=us-central1
gcloud artifacts repositories create kb-images --repository-format=docker --location=us-central1
docker build -t us-central1-docker.pkg.dev/PROJECT_ID/kb-images/bookstack .
docker push us-central1-docker.pkg.dev/PROJECT_ID/kb-images/bookstack
gcloud run deploy bookstack \
  --image us-central1-docker.pkg.dev/PROJECT_ID/kb-images/bookstack \
  --add-cloudsql-instances PROJECT_ID:us-central1:bookstack-db \
  --region us-central1

That’s a working setup for a small team, and you’re not managing an operating system at all.

Option 2: GKE Autopilot, if you outgrow that, or you’d rather have persistent volumes, background jobs and more control from day one. You still don’t manage nodes, Google provisions and scales them for you, but you get a real Kubernetes deployment with a persistent volume for uploads instead of juggling external storage:

gcloud container clusters create-auto kb-cluster --region us-central1
gcloud container clusters get-credentials kb-cluster --region us-central1
kubectl apply -f bookstack-deployment.yaml

Neither approach is complicated once you’ve done it once, and both keep you out of the business of patching an operating system, which is honestly most of what self-hosting used to mean.

Closing thoughts

For me, BookStack became one of those rare tools that I didn’t have to constantly rethink.

I installed it, structured the content and started using it.
And honestly, that might be one of the best compliments you can give to a piece of software.

By the way, for my personal knowledge base on my local machine, I use Obsidian. But that’s a story for another post.

BookStack: https://github.com/bookstackapp/bookstack
Wiki.js: https://github.com/requarks/wiki

#opensource #knowledgebase #bookstack #wikijs #documentation #selfhosted #opensourceprojects #devtools #elearning #markdown #googlecloud #gke #cloudrun

In March I hit the free tier limit on 11 Labs.

This is not a dramatic event — it happens to everyone who actually uses a product — but the moment I started looking for alternatives, I found something that bothered me more than the paywall. I had been using 11 Labs to transcribe voice messages, lecture recordings, and meeting notes. The quality was genuinely good: fast, accurate, handles background noise reasonably well.

When the limit came up, I moved to TurboScribe, which offered three free files per day, up to thirty minutes each. I noticed pretty quickly that opening an incognito tab reset the counter. So I uploaded a few test recordings, got the transcripts, and closed the incognito window.

—

Watch the Video on the GDG Ashgabat’s Build With AI Sessions:

And then it hit me: I had no idea where those files went.

I could not delete them. I did not even know which server they were sitting on. Those happened to be harmless test recordings, but the question stayed with me — what if they had been a client call? A medical consultation? Something confidential?

This is not a TurboScribe-specific problem. It is how every cloud transcription service works by design. TurboScribe’s privacy policy states they may retain your audio for up to thirty days. ElevenLabs’ terms include language about using content to improve their services. These are not hidden clauses — they are just things people scroll past.

For most casual use, this is a tolerable trade-off. For anything sensitive — legal, medical, journalistic, commercial — it is not.

The question I kept coming back to was:
- Does the transcription actually need to happen on their servers?
- Is there a technical reason the audio has to leave my machine?

The answer is that there is no technical reason at all.

OpenAI released Whisper as open source in 2022, and the model quality is on par with what the commercial services offer — because many of them are running Whisper or a derivative under the hood anyway. What was missing was a way to actually use it without spending an afternoon fighting CUDA dependencies and Python environment hell.

So I built one.

The Stack

The core of the project is faster-whisper, a reimplementation of OpenAI’s Whisper by SYSTRAN.

The key difference from the original is INT8 quantisation: the model weights are stored in a lower-precision format, which reduces memory usage and speeds up inference on CPU by roughly two to four times without any meaningful loss in transcription accuracy.

The large-v3-turbo model weighs about 1.5 GB on disk and uses around 6 GB of RAM at runtime. It is the sweet spot between quality and speed. Before audio reaches Whisper, it goes through FFmpeg for format conversion.

Whisper expects 16kHz mono WAV. In practice, your files will be MP3 voice messages, M4A recordings from your phone, MP4 exports from Zoom — FFmpeg normalises all of that in seconds. This is what makes the tool actually useful rather than just a demo: you do not have to think about format compatibility.

The third significant component is Silero VAD, a neural network for Voice Activity Detection.

Whisper has a known tendency to hallucinate — to “transcribe” words from silence. Running VAD before transcription eliminates most of that, and as a side effect, significantly reduces processing time on recordings with long quiet stretches.

For speaker identification, the tool uses SpeechBrain with ECAPA-TDNN voice embeddings and agglomerative clustering.

The result is a transcript with speaker labels: who said what, with timestamps. This runs entirely locally.

How I Structured the Code

I went through a few iterations before landing on something I was happy with. The first version was a single monolithic file — everything crammed together. It worked, but it was a nightmare to maintain.

The version I ended up with separates concerns properly.
The business logic lives in a core/ package:

• engine.py handles model loading and transcription
• diarize.py handles speaker detection
• format.py handles timestamps and segment grouping
• audio.py wraps FFmpeg

None of these modules know anything about HTTP or the web interface.
The transcription engine looks like this:

# core/engine.py
_cache = {}
def get_model(model_name):
    if model_name not in _cache:
        from faster_whisper import WhisperModel
        _cache[model_name] = WhisperModel(
            model_name,
            device="cpu",
            compute_type="int8",
            cpu_threads=8
        )
    return _cache[model_name]

def transcribe(audio_path,
               model_name="large-v3-turbo",
               language=None,
               on_progress=None):
    model = get_model(model_name)
    segments_iter, info = model.transcribe(
        audio_path,
        language=language,
        vad_filter=True,
        vad_parameters={"min_silence_duration_ms": 500},
        beam_size=5,
    )
    segments = []
    for s in segments_iter:
        segments.append({
            "start": s.start,
            "end": s.end,
            "text": s.text.strip()
        })
        if on_progress:
            on_progress(s.end, len(segments))
    return segments, info.language

A few things worth noting.

The model is loaded lazily — it only gets pulled into memory when somebody actually requests a transcription, not when the server starts.

That matters because the model takes several seconds to load and consumes a substantial amount of RAM, so you do not want that happening at import time.

The on_progress callback is how the web interface gets live progress updates: every time Whisper finishes a segment, the callback fires with the current timestamp and segment count, which the API then exposes to the frontend. The web layer is a FastAPI application.

When a file comes in, it gets written to a temp file and submitted to a ThreadPoolExecutor for background processing. The client gets back a job ID immediately and polls for status:

# api.py
_executor = ThreadPoolExecutor(max_workers=2)
@app.post("/v1/transcribe")
async def submit(file: UploadFile,
                 language="auto",
                 model="large-v3-turbo"):
    tmp = save_to_temp(file)
    job_id = str(uuid4())
    JOBS[job_id] = {
        "status": "queued",
        ...
    }
    _executor.submit(
        _run,
        job_id,
        tmp.name,
        language,
        model,
        ...
    )
    return {
        "job_id": job_id,
        "status": "queued"
    }

The frontend is plain HTML, CSS, and JavaScript. No React. No build step.

A file drop zone, language and model selectors, a progress bar that smoothly interpolates between backend updates, and an editable text area for the result.

You can record audio directly from your microphone as well, with a live waveform visualisation.

When transcription finishes, a synthesised chime plays — a small touch, but surprisingly useful when you have queued several files and walked away. The whole thing mounts as a static directory on the FastAPI app:

app.mount("/", StaticFiles(directory="static", html=True), name="static")

API routes are registered before the static mount, so:

• /v1/transcribe
• /v1/jobs/{id}

are handled by FastAPI whilst everything else falls through to index.html.

I also kept the command-line tools as thin wrappers over the same core/ package. If you prefer working in the terminal:

python transcribe.py -i meeting.mp3
python transcribe.py -i ~/recordings/ -l en
python diarize.py -i interview.mp3 -s 2

What It Actually Performs Like

On an AMD Ryzen 7 PRO with 14 GB of RAM — a mid-range machine — the large-v3-turbo model processes audio at roughly 2x real time.

A ten-minute recording takes about five minutes. An hour-long meeting takes about thirty minutes. This is not instant.

Cloud services with GPU backends return results in seconds. If you need sub-minute turnaround, local CPU transcription will not match that.

But for the actual use cases that matter — transcribing yesterday’s meeting, processing a batch of interview recordings, converting voice messages to text — 2x real time is perfectly workable.

You queue the files, do something else, and come back to finished transcripts. If you need speed over accuracy, the smaller models are an option.

• medium runs at roughly 4x real time
• small runs at roughly 7x real time

For English-language content with clear audio, medium will often give you results close to large-v3-turbo at significantly higher speed.

Deploying on Your Own Server

Local transcription covers the laptop use case.
But what if you want to transcribe from your phone? Or share access with a small team whilst keeping data off third-party infrastructure?

The concept is straightforward:
You run Edgescribe on a VPS you control, put it behind Nginx with HTTPS, and add an API key.

The result is functionally identical to TurboScribe:

• open a URL
• upload a file
• get a transcript

Except the server is yours, the data stays on your infrastructure, and there are no usage limits. For a single user or small household, a Hetzner CPX41 (8 vCPU, 16 GB RAM) is a comfortable setup.

The 16 GB RAM figure is important: the model needs around 6 GB, and you want headroom for the OS and concurrent requests.

The Dockerfile is straightforward:

FROM python:3.11-slim
RUN apt-get update && apt-get install -y ffmpeg
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["python", "api.py", "--host", "0.0.0.0"]

docker compose up -d

For HTTPS and domain access, the Nginx configuration needs two things that are not immediately obvious if you have not done this before.

First, client_max_body_size needs to be large enough for audio files - I set it to 500 MB.

Second, proxy_read_timeout needs to be much longer than the default 60 seconds, because transcription requests take minutes, not milliseconds.

Without this, Nginx will kill the connection before the transcription finishes.

server {
    listen 443 ssl;
    server_name transcribe.yourdomain.com;
    client_max_body_size 500M;
    location / {
        proxy_pass http://localhost:8000;
        proxy_read_timeout 3600;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

certbot --nginx -d transcribe.yourdomain.com

Let’s Encrypt via Certbot provides a free TLS certificate.

For access control, the API accepts an --api-key flag or reads from the EDGESCRIBE_API_KEY environment variable.

Every request then requires an X-API-Key header.

What you end up with is your own transcription endpoint, reachable from anywhere, using only your infrastructure.

You know exactly where the data is.
You can delete it.
You control the retention policy.
If the service shuts down, your transcription capability does not disappear.

Why This Matters

The argument for keeping audio on your own machine is not primarily about paranoia towards tech companies.

It is about professional responsibility and basic data hygiene.

If you work in a field where confidentiality matters — legal, medical, financial, journalistic — uploading sensitive recordings to a third-party service is not a neutral choice. It is a decision with liability implications that most people make by default because the SaaS option is frictionless and the local option requires setup.

We are at a point where consumer hardware is genuinely capable of running state-of-the-art speech recognition offline, at accuracy levels that match what the paid services offer.

The gap between local and cloud is mostly about latency, not quality.
For transcription — an inherently asynchronous task — the latency gap matters less than people assume.
Your voice recordings contain conversations you had in confidence.
The question of where those recordings are processed and stored is worth at least as much attention as the question of whether the transcript is accurate.

With the right setup, you do not have to choose between quality and control. You can have both.

Edgescribe is open source.
Repository: github.com/mikebionic/edgescribe
Stack: faster-whisper (SYSTRAN), FastAPI, SpeechBrain, Silero VAD, FFmpeg.
Runs on Python 3.9+, Linux, macOS, Windows (WSL2). No GPU required.

Автор не является юристом или официальным представителем каких-либо государственных органов. Информация в статье отражает личный опыт на момент прохождения процедуры и может не соответствовать актуальным требованиям. За официальной информацией обращайтесь напрямую в учреждения.

Зачем это нужно?

При подаче на магистратуру от вас потребуют легализованные (сертифицированные) копии документов об образовании: диплом, транскрипт (выписка с оценками), аттестат о среднем образовании.
Без печатей трёх инстанций — нотариата, Министерства юстиции и МИДа — ваши документы не примут.

Мой транскрипт был на английском языке. Нотариальное заверение в делается только на государственном языке, и это добавило целый дополнительный этап.

Что вам понадобится

Прежде чем начинать — соберите всё заранее:

Паспорт — государственный паспорт (оригинал, обязательно при себе на каждом этапе)
Документы — диплом, транскрипт, аттестат (оригиналы)
Копии — по 1–2 копии каждого документа
Банковская карта — для оплаты в МИДе (только безналичный расчёт)
Время — минимум 4–5 рабочих дней, комфортно — неделя

Важно: Все действия должен выполнять лично владелец документов. Нельзя отправить кого-то вместо себя.

Маршрут: 5 этапов за 4–7 дней

[Университет] → [Нотариат] → [Минюст] → [МИД] → [Перевод]
  прошивка +     заверение    печать      финал    при необх.
  перевод        документов   Минюста     легализ.

Примерный таймлайн

День 1–2  │ Университет — перевод транскрипта и прошивка
День 3    │ Нотариат — заверение документов
День 4    │ Минюст — сдать документы
День 5    │ Минюст — забрать с печатью (утром) → сразу в МИД
День 6    │ МИД — забрать готовые документы (утром до обеда)

Этап 1. Университет — прошивка и верификация

Цель: получить прошитый транскрипт с печатью вуза и официальный перевод на государственный язык

Если ваш транскрипт на иностранном языке (как у меня — на английском):

1. Переведите транскрипт на государственный язык — это обязательное требование для нотариального заверения
2. Отнесите перевод в деканат — отдел образования должен сверить перевод с оригиналом, поставить штамп и подпись декана
3. Убедитесь, что транскрипт прошит — документ должен быть прошит университетом до похода к нотариусу

На выходе: прошитый оригинал транскрипта + верифицированный перевод с печатью вуза.

Этап 2. Нотариат — заверение документов

Цель: нотариально заверить все документы.

Порядок действий:

1. Приходите с оригиналами и копиями, берёте талон электронной очереди
2. Нотариус сверяет копии с оригиналами и ставит печати

Оплата: нотариальные услуги платные.

Лайфхак

Если нужно вдруг идти по срочным делам, перед выходом можно взять талон и вернуться к нужному времени.

Этап 3. Министерство юстиции — легализация

Цель: получить печать Министерства юстиции.

Где: здание Министерства юстиции (Adalat ministrligi), проспект Арчабил. Удобный подъезд со стороны проспекта Чандыбиль.

Порядок действий:

1. Предъявите паспорт на входе и пройдите в отдел легализации
2. Предоставьте нотариально заверенные документы
3. Документы заберут на один рабочий день
4. На следующий день утром приходите и забираете документы с печатью Минюста

Оплата: бесплатно.

Тайминг: забрав документы утром, сразу езжайте в МИД.

Этап 4. МИД — финальная сертификация

Цель: получить финальную печать Министерства иностранных дел.

Где: здание МИД.

Порядок действий:

1. Предъявите паспорт на входе, возьмите электронный талон
2. Предоставьте легализованные документы + оригиналы + паспорт
3. Оплатите обработку документов — банковской картой
4. Документы заберут на один рабочий день
5. На следующий день утром до обеда приходите с паспортом и забираете готовые сертифицированные документы

Этап 5. Перевод (при необходимости)

После получения всех печатей — ваши документы полностью легализованы. Если приёмная комиссия требует документы на определённом языке (например, английском):

1. Переведите легализованные страницы (включая страницы с печатями) на нужный язык
2. Соберите всё в единый PDF-файл
3. Отправляйте в университет

Чеклист перед стартом

• [ ] Оригиналы всех документов об образовании собраны
• [ ] Сделаны копии (минимум по 1 шт. каждого)
• [ ] Паспорт при себе
• [ ] Банковская карта
• [ ] Перевод транскрипта готов (если документы на иностранном языке)
• [ ] Транскрипт прошит университетом

Словарь терминов

Adalat ministrligi — Министерство юстиции — Ministry of Justice
Daşary işler ministrligi — МИД — Ministry of Foreign Affairs
Notarius — Нотариат / нотариус — Notary
Kanunlaşdyrma — Легализация — Legalization
Tassyklama — Заверение / верификация — Certification

Этот гайд основан на личном опыте прохождения процедуры легализации документов. Процедуры могут меняться — уточняйте актуальные требования в учреждениях.

#Образование #Легализация #Документы #Магистратура #Учёба #ЛичныйОпыт

I recently published a podcast interview on YouTube. It’s a long one — about 75 minutes. A deep, meaningful conversation with the head of Uzbekistan’s largest IT community.
Here’s the video link: https://youtube.com/shorts/tFBFzx5j94s?feature=share

The content is great. But let’s be honest — nobody is watching a full 75-minute video on Instagram.

What I needed were short clips. 30 to 60 second Reels. The kind that grab attention instantly and make people curious enough to check out the full episode.

Normally, this would mean opening a video editor, scrolling through the entire timeline, manually selecting moments, exporting clips one by one. For 15–20 clips, that’s easily 2–3 hours of work.

This time, it took me less than 10 minutes.

Here’s exactly what I did.

The Setup

You only need three things:

1. A transcript with timestamps (YouTube captions work fine)
2. ffmpeg (a free command-line tool)
3. An AI assistant

No Premiere Pro. No DaVinci. No timeline editing. Just text and a terminal.

Step 1: Get the Transcript

Everything starts with the transcript. Mine looked something like this:

Chapter 4: How to build the first team with zero budget  
4:38  
So, we started as a very small movement. There were eight of us.  
4:46  
At first we gathered 30 volunteers, and eight stayed...

Each line has a timestamp. That’s all the AI needs. It doesn’t actually watch the video — it understands everything from text.

Step 2: Let AI Do the Heavy Lifting

I fed the full transcript into an AI assistant and gave it a simple instruction:

“Find short 30–60 second moments that would work well as Instagram Reels and make people want to watch the full episode.”

Within seconds, it scanned the entire conversation and suggested clips with exact timestamps and hooks.

At first, it gave me solid but pretty standard moments — interesting, but not very scroll-stopping.

So I pushed it:

“These aren’t strong enough. Find more emotional, surprising, or funny moments.”

That’s when things got good.

Instead of just “information,” it started picking moments with:

• Humor
• Conflict
• Personal stories
• Unexpected twists

For example:

• A story where someone tries to save a drowning person but grabs a ball instead
• Copying another brand’s logo on purpose so people confuse them
• Turning down a higher-paying job abroad to stay with the community
• A strict rule: miss once — you’re out
• A girl solving a family restriction by bringing her brother along

That’s when it clicked:

Emotion beats information. Every time.

Step 3: Generate the Clips Automatically

For each selected moment, the AI generated an ffmpeg command like this:

ffmpeg -y -ss 00:04:38 -to 00:05:50 \
  -i podcast.mp4 \
  -c:v libx264 -c:a aac \
  -avoid_negative_ts make_zero \
  clip_01.mp4

Even better — it didn’t run them one by one. It launched all the cuts in parallel.

So instead of waiting 10–15 minutes, everything rendered in about 2–3 minutes.

Step 4: Build a Trailer Mix

Then I asked for a short trailer — a fast-paced mix of the best moments.

The process was simple:

1. Cut small 5–8 second segments
2. Combine them into one video

The result: a 60-second highlight reel that feels dynamic and engaging.

Final Output

In one session, without opening any editing software, I got:

• 18 short clips (ready for Reels)
• 2 trailer videos
• 10 caption ideas with hashtags

Total time spent: about 10 minutes Rendering time: around 3 minutes

What I Learned

1. The transcript is everything No timestamps — no workflow. This is the foundation.

2. AI doesn’t need video Text is enough. It understands structure, tone, and storytelling surprisingly well.

3. The first result is rarely the best Iteration matters. The second or third pass is where quality jumps.

4. Emotion > information Stats don’t stop the scroll. Stories do.

5. ffmpeg is insanely powerful It replaces hours of manual editing with a few commands — and AI can handle it easily.

Try It Yourself

All you need:

• A video
• A timestamped transcript
• ffmpeg installed
• Any AI assistant

Give it the transcript. Ask for clips. Refine the results. Export.

That’s it.
What used to take hours is now just a conversation.

Try Yourself: https://coopcinema.onrender.com/

You know that moment when you’re on a call with a friend, you both open the same YouTube video, and someone says “okay, 3… 2… 1… play!” — and you’re still 4 seconds apart the entire time?

That was my pain. A stupid, small, annoying pain. And it turned into a weekend project.

The Problem

I live far from some of my closest friends. We like watching stuff together — movie trailers, conference talks, random YouTube rabbit holes at 2 AM. Every existing solution had a catch:

• Teleparty / Watch2Gether — browser extensions, account walls, or limited to Netflix/YouTube
• Discord screen share — kills the video quality, eats bandwidth, and the person sharing can’t resize their window without everyone suffering
• “Ready? 3… 2… 1…” — never actually works

I wanted something dead simple. Open a link, paste a YouTube URL, and you’re watching together. No installs, no accounts, no extensions.

The Idea

What if the server did almost nothing?

Seriously. What if the server was just a dumb relay — a WebSocket hub that receives a JSON message from one client and broadcasts it to everyone else in the room? No video processing. No transcoding. No storage. No authentication. Just: “hey, someone pressed play at timestamp 42.5 seconds — pass it on.”

The clients handle everything. Each browser loads the video independently (YouTube iframe, Vimeo embed, local file, whatever), and they just keep each other in sync through tiny JSON messages.

That’s it. That’s the architecture.

One Weekend, Go + Vanilla JS

The backend is about 200 lines of Go. A WebSocket hub manages rooms. Clients connect, join a room, and every message they send gets broadcast to everyone else in that room (except themselves). The server doesn’t know or care what a “play” or “pause” means. It just relays.

func (h *Hub) Broadcast(msg models.Message, sender *models.Client) {
    h.mu.RLock()
    defer h.mu.RUnlock()
    room := h.rooms[sender.RoomCode]
    for client := range room {
        if client != sender {
            client.Send <- msg
        }
    }
}

The frontend is vanilla JavaScript. No React, no Vue, no build step. One HTML file, one JS file, one CSS file. The JS handles:

• Detecting what kind of URL you pasted (YouTube? Vimeo? Twitch? Direct .mp4 link?)
• Loading the right player (YouTube IFrame API, Vimeo SDK, Twitch Embed, or a plain <video> tag)
• Intercepting play/pause/seek events and sending them over WebSocket
• Receiving those events from peers and applying them locally

The sync logic is surprisingly simple. When someone presses play, the client sends:

{
  "type": "play",
  "timestamp": 42.5,
  "sentAt": 1706000000000
}

The receiver checks: am I more than 0.5 seconds off from 42.5? If yes, seek first. Then play. The sentAt field lets receivers estimate network latency and adjust the seek target by half the round-trip time. It's not perfect, but it's close enough that you don't notice.

Beyond the Basics

Once the core sync worked, the fun features came fast:

Chat with toast popups — a slide-in chat sidebar with notification toasts that stack up when the chat is closed. Click a toast, chat opens. No message missed.

Emoji reactions — click an emoji and it floats up over the video with your name underneath it. Everyone sees it. Stupid simple, genuinely fun during a tense movie scene.

Theater Fullscreen — YouTube’s native fullscreen hides everything — no chat, no reactions, no toasts. So I built a custom fullscreen that uses the Fullscreen API on a wrapper div containing the video, reactions, chat, and controls. Everything stays visible.

Buffering sync — if one person’s connection stutters, everyone pauses automatically. When they’re ready, everyone resumes. No one misses a scene because their friend has slower internet.

Host mode — the room creator can enable host mode, where only the host’s playback controls sync. Everyone else just watches. Click someone’s badge to transfer host.

DevFest Ashgabat 2025

I used Co-op Cinema as the demo project for my talk at DevFest Ashgabat 2025 — “Code. Build. Deploy. Repeat. GitHub Pipelines & Cloud Deployments for Go.”

The talk was about CI/CD pipelines with GitHub Actions and deploying Go apps to production. Co-op Cinema was the app we built the pipeline around — from feature branches to automated builds to cloud deployment. About 690 people attended.

You can watch the full session on YouTube or check the slides.

The irony of demonstrating a deployment pipeline for a watch party app — during a live event where hundreds of people were watching together in the same room — was not lost on me.

Deploying It (for Free)

The whole thing is Dockerized. One command:

cp .env.example .env
docker compose up -d

For public access, I deployed to Render — free tier, Docker runtime, auto-deploys from GitHub on every push. The render.yaml blueprint means you can fork the repo and deploy in two clicks.

The Go binary is ~10MB. The Docker image is ~15MB. The server uses almost no memory or CPU because it does almost nothing. Perfect for a free tier.

What I Learned

The dumbest server is sometimes the best server. A stateless WebSocket relay with zero business logic handles every video source, every player API, and every edge case — because the clients do all the work. The server will never need updating when I add a new video source.

Vanilla JS is fine. No framework, no build step, no node_modules. The entire frontend is one HTML file, one JS file, one CSS file. It loads instantly. It works everywhere. Adding features means editing a function, not configuring a pipeline.

Sync is about tolerance, not precision. You don’t need millisecond-perfect sync for a watch party. Half-second tolerance, debounced events, and a latency offset get you 95% of the way there. The remaining 5% doesn’t matter because humans can’t tell.

Try It

The project is open source. Fork it, deploy it, watch something with someone who’s far away.

GitHub: coopcinema

Or just grab Docker and run it locally in 10 seconds:

git clone https://github.com/mikebionic/coopcinema.git
cd coopcinema
cp .env.example .env
docker compose up -d

Open localhost:8080, create a room, share the link. That's it.

Built with Go, gorilla/websocket, and vanilla JavaScript. No frameworks were harmed in the making of this project.

AI Didn’t Want to Give Me Exact MIDI Notes — It Didn’t Know It Was Dealing With a Musician :D

Video Demo | Try It Yourself

There’s a moment every developer knows — the moment you discover something in the browser that makes you think “wait, you can do THAT?” For me, that moment was the Web Audio API.

I wanted to make music. Not by loading .mp3 files. Not by embedding SoundCloud players. I wanted the browser itself to synthesize sound from nothing — oscillators, noise, waveforms, frequencies. Real sound generation. No libraries, no frameworks, no external audio files.

The result is WebChestra — a pure JavaScript, browser-based music mixer inspired by Incredibox and Sprunki Beats. Click emoji characters, layer instruments, build a beat. Zero dependencies.

The Pain: Web Audio API Is Powerful but Unforgiving

Let me be honest about the journey. The Web Audio API is incredibly capable — but it’s also incredibly low-level. There’s no playNote("C3") function. There's no createKickDrum(). You get raw oscillators, gain nodes, and biquad filters. You are the synthesizer.

Pain Point #1: Everything Is Frequencies

Want a bass note? You need to know that G2 is 98.00 Hz, C3 is 130.81 Hz, and D2 is 73.42 Hz. There’s no shortcut. You’re working with the physics of sound:

A4 = 440 Hz (standard tuning reference)
Each octave = 2x the frequency
Each semitone = multiply by 2^(1/12) ~ 1.0595

I manually mapped out every note frequency for both the bass (triangle wave, 73–196 Hz range) and lead synth (square wave, 392–880 Hz range). Nine notes each, painstakingly chosen to harmonize together.

Pain Point #2: Drums Don’t Exist — You Build Them

There’s no drum sample in the Web Audio API. A kick drum? That’s a sine wave oscillator that pitch-bends from 150 Hz down to 55 Hz in 100 milliseconds, passed through a lowpass filter at 500 Hz. A snare? White noise (literally Math.random() * 2 - 1 pumped into an audio buffer) filtered through a 10 kHz bandpass filter. Hi-hat? Same white noise, but through a 5 kHz highpass filter to get that sizzle.

// This is how you make a kick drum from nothing:
osc.frequency.setValueAtTime(150, audioContext.currentTime);
osc.frequency.exponentialRampToValueAtTime(55, currentTime + 0.1);

Every sound is built from first principles.

Pain Point #3: Timing Is Harder Than You Think

I used setInterval() running at 200ms per beat (effectively 300 BPM). It works — but JavaScript's event loop doesn't guarantee precise timing. Over long sessions, you can hear subtle drift. The "proper" solution is scheduling notes ahead of time using audioContext.currentTime, but for a fun experiment, setInterval gets the job done.

Pain Point #4: Volume Balancing Is Trial and Error

The lead synth at full volume obliterates everything else. I ended up with the bass at gain 0.4 and the lead at 0.05 — an 8x difference. There’s no mixing console. You tweak numbers, reload, listen, repeat. Over and over.

The Architecture: One File, Zero Dependencies

The entire project lives in a single index.html file. HTML structure, CSS styling, JavaScript logic — all together. Pragmatic? Yes. Production-ready? No. But it means you can literally download one file, open it in a browser, and make music.

Five Instruments, Two Sound Engines

Melodic instruments (Bass and Lead) use oscillator nodes with predefined frequency arrays and 64-beat pattern loops:

• Bass: Triangle wave — warm, rounded, sits underneath everything
• Lead: Square wave — bright, hollow, chiptune-like, cuts through the mix

Percussion (Kick, Snare, Hi-Hat) use generated white noise buffers filtered through biquad filters, running on an 8-beat loop:

• Kick: Sine wave with pitch bend + lowpass filter
• Snare: White noise + bandpass filter at 10 kHz
• Hi-Hat: White noise + highpass filter at 5 kHz

The pattern system encodes notes as arrays of integers — 0 means silence, 1–9 maps to a specific note index. Drums and melodic lines sync through modulo operations (currentBeat % 8 for drums, currentBeat % 64 for melodic).

Where AI Fell Short (and Where Being a Musician Mattered)

Here’s the part I enjoyed the most. When I tried to get AI assistance with the musical note frequencies and patterns, it was reluctant to provide exact frequency values for specific notes. It wanted to generalize, abstract, hand-wave.

But I’m a musician. I know what G2 sounds like. I know the difference between a triangle wave bass and a square wave bass. I know that a snare needs that bandpass filter crack, not a lowpass mush.

The AI didn’t know it was dealing with someone who would actually listen to the output and know if it was wrong.

So I did what any stubborn musician-developer would do: I looked up the frequency tables myself, tuned by ear, iterated on patterns until the groove felt right, and balanced volumes until the mix breathed.

The lesson: AI is a tool, not a producer. It can scaffold code, but it can’t tell you if the beat slaps.

What Could Be Better

I’m honest about the rough edges. Here’s what I’d improve:

• Proper audio scheduling — Replace setInterval with Web Audio API clock-based scheduling for sample-accurate timing
• Tempo control — Currently hardcoded at 300 BPM, should be adjustable
• Volume sliders — Per-instrument mixing instead of hardcoded gain values
• MIDI input support — navigator.requestMIDIAccess() would let you plug in a keyboard
• Pattern editor UI — Visual grid to edit beat patterns instead of changing code
• Code modularization — Break the monolithic HTML file into proper modules
• More waveforms and effects — Reverb, delay, distortion using Web Audio’s ConvolverNode and WaveShaperNode
• Mobile optimization — Touch interactions and responsive layout tweaks
• Save/load compositions — Export your mix as JSON or even render to WAV using OfflineAudioContext

Try It

WebChestra — Live Demo
Watch the Video Demo
Source Code on GitHub

No installs. No sign-ups. Just open your browser, click some emojis, and make a beat.

Built with nothing but curiosity, a browser, and stubbornness.

Stop reading theory. Start building. This is the fastest path from zero to a working Kubernetes cluster with a real application running.

What You’ll Build

• Self-hosted Kubernetes cluster using kubeadm
• Control plane + worker nodes
• Real FastAPI application in Docker
• Deployments with replicas and autoscaling

Time to complete: 30–45 minutes

Infrastructure Requirements

Control Plane (Master):

• Ubuntu Linux
• 6 GB RAM, 4 CPU

Worker Nodes:

• Ubuntu Linux
• 4 GB RAM, 2 CPU

All Nodes Must Have:

• Unique hostname
• Network connectivity
• Swap disabled
• 2+ CPU cores

Step 1: Disable Swap (All Nodes)

Kubernetes refuses to run with swap enabled.

sudo swapoff -a

Make it permanent by removing swap entries from /etc/fstab.

Step 2: Install Kubernetes Tools (All Nodes)

Install dependencies:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

Add Kubernetes signing key:

sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

Add Kubernetes repository:

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' | \
  sudo tee /etc/apt/sources.list.d/kubernetes.list

Install kubeadm, kubelet, kubectl:

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl enable --now kubelet

Step 3: Initialize Control Plane (Master Node Only)

Check your network interface:

ip route show

Initialize the cluster:

sudo kubeadm init

Configure kubectl for your user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Save the kubeadm join command printed at the end. You'll need it for worker nodes.

Step 4: Install Pod Network (Master Node)

Without this, pods cannot communicate.

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

Verify:

kubectl get nodes
kubectl get pods -A

Step 5: Join Worker Nodes

On each worker node, run the kubeadm join command from Step 3:

sudo kubeadm join <master-ip>:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

Verify from master:

kubectl get nodes

Step 6: Build Your Application

Create main.py:

from fastapi import FastAPI, Request
import random
import socket

app = FastAPI()

def get_server_ip():
    try:
        hostname = socket.gethostname()
        return socket.gethostbyname(hostname)
    except:
        return "Unknown"

@app.get("/")
async def index():
    return {
        "message": "Salam Dünýä!",
        "server_ip": get_server_ip()
    }

@app.get("/ping")
async def ping():
    return {"response": "pong"}

@app.get("/client")
async def client_info(request: Request):
    return {
        "client_ip": request.client.host,
        "headers": dict(request.headers)
    }

@app.get("/random")
async def random_number():
    return {"number": random.randint(1, 1_000_000)}

Create requirements.txt:

fastapi
uvicorn[standard]

Create Dockerfile:

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Build and push:

docker build -t yourregistry.com/yourapp:latest .
docker push yourregistry.com/yourapp:latest

Replace yourregistry.com with Docker Hub or your private registry.

Step 7: Deploy to Kubernetes

Simple Deployment (1 pod)

Create deployment-simple.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
spec:
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp-container
        image: yourregistry.com/yourapp:latest
        ports:
        - containerPort: 8000

Apply:

kubectl apply -f deployment-simple.yaml

Step 8: Deployment with Replicas

Create deployment-replicas.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment-replicas
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp-replicas
  template:
    metadata:
      labels:
        app: myapp-replicas
    spec:
      containers:
      - name: myapp-container
        image: yourregistry.com/yourapp:latest
        ports:
        - containerPort: 8000

Apply:

kubectl apply -f deployment-replicas.yaml

Step 9: Autoscaling (HPA)

Create deployment-autoscaling.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment-autoscaling
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp-autoscaling
  template:
    metadata:
      labels:
        app: myapp-autoscaling
    spec:
      containers:
      - name: myapp-container
        image: yourregistry.com/yourapp:latest
        ports:
        - containerPort: 8000
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-autoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app-deployment-autoscaling
  minReplicas: 2
  maxReplicas: 6
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Apply:

kubectl apply -f deployment-autoscaling.yaml

Check autoscaling status:

kubectl get hpa

Step 10: Access Your App

Port forwarding (quick test):

kubectl get pods
kubectl port-forward pod/<pod-name> 6789:8000

Open browser: http://localhost:6789

Essential Commands Cheatsheet

Cluster Status

kubectl get nodes
kubectl get pods -A
kubectl cluster-info

Deployments

kubectl get deployments
kubectl describe deployment <name>
kubectl delete deployment <name>

Pods

kubectl get pods
kubectl logs <pod-name>
kubectl describe pod <pod-name>
kubectl exec -it <pod-name> -- /bin/bash

Autoscaling

kubectl get hpa
kubectl describe hpa <name>

Cleanup

kubectl delete deployment --all
kubectl delete hpa --all

Reset Cluster

sudo kubeadm reset

Port Reference

Component Port Kubernetes API 6443 etcd 2379–2380 Kubelet API 10250 kube-scheduler 10259 kube-controller 10257

Check port availability:

nc 127.0.0.1 6443 -zv -w 2

Troubleshooting

Pods not starting?

kubectl describe pod <pod-name>
kubectl logs <pod-name>

Network issues?

kubectl get pods -n kube-system
# Check Calico pods are running

Node not joining?

# On worker node:
sudo kubeadm reset
# Then rejoin with kubeadm join command

Forgot join command?

# On master:
kubeadm token create --print-join-command

No abstractions. No magic. Just real Kubernetes.

Once you understand this manual setup, managed services like EKS, GKE, and AKS become trivial.

Next Steps

• Set up Services and LoadBalancers
• Configure Ingress for routing
• Add persistent storage with PVs/PVCs
• Implement monitoring with Prometheus
• Set up logging with ELK stack
• Learn Helm for package management

Bookmark This Guide

This is your reference for Kubernetes essentials. Keep it handy, and refer back whenever you need to spin up a cluster or deploy an application.

Happy clustering! 🚀

If you found this useful, follow for more practical DevOps guides!

This guide teaches Backend developers and DevOps engineers the essential database knowledge they need. It is not a guide on becoming a Database Administrator (DBA). The focus is on practical knowledge: how to run a database that ensures data integrity, maintains high performance, and survives disasters.

I’ve consolidated all the necessary knowledge, tips, commands, and scripts to enable you to use this powerful database confidently and effectively.

Save this article as a reference to consult whenever you encounter a problem.

At a Glance

PostgreSQL is a relational database that guarantees ACID properties (Atomicity, Consistency, Isolation, Durability). Here’s how it works internally:

The Write Process

INSERT statement
    ↓
Write-Ahead Log (WAL) buffer
    ↓
WAL synced to disk (this is the commitment point)
    ↓
Data written to shared buffers (in-memory cache)
    ↓
Background writer flushes to data files
    ↓
Data persisted

Key insight: PostgreSQL writes to the transaction log FIRST, before writing to data files. If it crashes, it can replay the log and recover everything. This one of the reasons why PostgreSQL is reliable.

Installation and Initial Setup

I’ll start from basics and will go into more advanced information, to be completely buckled up!

Install PostgreSQL

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install postgresql postgresql-contrib postgresql-client

# macOS
brew install postgresql

# Verify installation
psql --version
# Output: psql (PostgreSQL) 16.0

# Check if running
sudo systemctl status postgresql

# Output: ● postgresql.service - PostgreSQL RDBMS
#         Loaded: loaded (/lib/systemd/system/postgresql.service)
#         Active: active (running)

Connect to PostgreSQL

# Connect as postgres superuser
sudo -u postgres psql

# Or if you have sudo access without password
psql -U postgres -h localhost

# Verify you're connected
postgres=# \conninfo

# Output: You are connected to database "postgres" as user "postgres" via socket in "/var/run/postgresql" at port 5432.

Basic PostgreSQL Commands

# List all databases
\l

# Connect to database
\c database_name

# List all tables
\dt

# List all schemas
\dn

# Show table structure
\d table_name

# Get help
\h CREATE TABLE

# Exit
\q

Initial Configuration: postgresql.conf

The postgresql.conf file is where PostgreSQL behavior is configured. Find it:

# Find location
sudo -u postgres psql -c "SHOW config_file"
# Output: /etc/postgresql/16/main/postgresql.conf

# View important settings
sudo -u postgres psql -c "SHOW port"
# Output: 5432

sudo -u postgres psql -c "SHOW data_directory"
# Output: /var/lib/postgresql/16/main

Essential Configuration for Production

Edit /etc/postgresql/16/main/postgresql.conf:

# Network settings
listen_addresses = '*'           # Listen on all interfaces (restrict in production with IP)
port = 5432                      # Standard PostgreSQL port

# Memory configuration
shared_buffers = 256MB           # 25% of system RAM for dedicated server
effective_cache_size = 1GB       # 50-75% of total RAM
work_mem = 16MB                  # Per operation memory
maintenance_work_mem = 64MB      # For maintenance operations

# Checkpoint configuration (critical for performance)
checkpoint_timeout = 15min       # Force checkpoint every 15 minutes
checkpoint_completion_target = 0.9  # Spread checkpoint over 90% of interval
max_wal_size = 4GB              # Size of WAL before forcing checkpoint
min_wal_size = 1GB              # Don't delete WAL below this

# Write-Ahead Logging (WAL) configuration
wal_level = replica             # Enable replication (critical for backup/failover)
max_wal_senders = 10            # Max replication connections
wal_keep_size = 1GB             # Keep WAL for replication
wal_compression = on            # Compress WAL

# Connections
max_connections = 200           # Max client connections
superuser_reserved_connections = 10  # Reserve for superuser

# Logging
logging_collector = on          # Enable log file collection
log_directory = 'log'           # Relative to data_directory
log_filename = 'postgresql-%a.log'
log_truncate_on_rotation = on   # Overwrite old logs

# Log slow queries (anything over 1s)
log_min_duration_statement = 1000  # Milliseconds
log_statement = 'all'           # Log all statements (use 'mod' for less noise)
log_connections = on
log_disconnections = on

# Performance
synchronous_commit = on         # Wait for WAL write (safer but slower)
fsync = on                      # Always fsync to disk (never disable in production)

After editing, reload configuration:

# Reload without restart (some settings require restart)
sudo -u postgres psql -c "SELECT pg_reload_conf()"
# Output: pg_reload_conf
#         ---------------
#         t

# Or restart PostgreSQL (safer for critical changes)
sudo systemctl restart postgresql

# Verify settings applied
sudo -u postgres psql -c "SHOW shared_buffers"
# Output: 256MB

User and Database Management

Create Database and User

# Connect as superuser
sudo -u postgres psql

# Create database
CREATE DATABASE production_app;

# Create user with password
CREATE USER app_user WITH ENCRYPTED PASSWORD 'secure_password_123';

# Grant privileges to user
GRANT CONNECT ON DATABASE production_app TO app_user;
GRANT USAGE ON SCHEMA public TO app_user;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO app_user;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO app_user;

# Give default privileges for future tables
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO app_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO app_user;

# Verify permissions
\du
# Lists all roles and their attributes
# Exit
\q

Connection String for Application

# PostgreSQL connection string (DSN)
postgresql://app_user:secure_password_123@localhost:5432/production_app

# Test connection from command line
psql postgresql://app_user:secure_password_123@localhost:5432/production_app -c "SELECT 1"

# Output: ?column?
#         --------
#                1

Creating Tables and Indexes

Create a Schema with Best Practices

-- Connect as app_user
\c production_app app_user

-- Create table
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL,
    username VARCHAR(100) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    is_active BOOLEAN DEFAULT true
);

-- Add indexes for common queries
CREATE UNIQUE INDEX idx_users_email ON users(email);
CREATE UNIQUE INDEX idx_users_username ON users(username);
CREATE INDEX idx_users_created_at ON users(created_at);
CREATE INDEX idx_users_is_active ON users(is_active);

-- View table structure
\d users
-- View indexes
\di users*

-- Check index size
SELECT 
    schemaname,
    tablename,
    indexname,
    pg_size_pretty(pg_relation_size(indexrelid)) as size
FROM pg_indexes
WHERE tablename = 'users'
ORDER BY pg_relation_size(indexrelid) DESC;

Add Constraints

-- Add constraint to ensure valid email
ALTER TABLE users 
ADD CONSTRAINT check_valid_email 
CHECK (email ~* '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}$');

-- Add foreign key to another table
CREATE TABLE posts (
    id SERIAL PRIMARY KEY,
    user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    title VARCHAR(255) NOT NULL,
    content TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- View constraints
\d posts

Essential Queries for Monitoring

Check Database Health

-- Current connections
SELECT 
    datname,
    count(*) as connections,
    max(extract(epoch from (now() - query_start))) as longest_query_seconds
FROM pg_stat_activity
GROUP BY datname
ORDER BY connections DESC;

-- Long-running queries (over 5 minutes)
SELECT 
    pid,
    usename,
    query_start,
    extract(epoch from (now() - query_start)) as duration_seconds,
    query
FROM pg_stat_activity
WHERE query_start < now() - interval '5 minutes'
ORDER BY query_start;
-- Kill long-running query (DANGER: use carefully)
SELECT pg_terminate_backend(pid);

Check Database Size

-- Total database size
SELECT 
    datname,
    pg_size_pretty(pg_database_size(datname)) as size
FROM pg_database
WHERE datname NOT IN ('template0', 'template1')
ORDER BY pg_database_size(datname) DESC;

-- Size per table
SELECT 
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
FROM pg_tables
WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 20;

-- Unused indexes (wasting space and slowing writes)
SELECT 
    schemaname,
    tablename,
    indexname,
    idx_scan,
    pg_size_pretty(pg_relation_size(indexrelid)) as size
FROM pg_stat_user_indexes
WHERE idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC;

Check Slow Queries

First, ensure logging is configured:

# Check log file location
sudo -u postgres psql -c "SHOW log_directory"
# Output: log

# View recent logs
sudo tail -f /var/lib/postgresql/16/main/log/postgresql-*.log | grep "duration:"

# Output example:
# LOG:  duration: 2534.123 ms  statement: SELECT * FROM large_table;

Find and optimize slow queries:

-- Queries taking longest (requires pg_stat_statements extension)
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

SELECT 
    query,
    calls,
    total_time,
    mean_time,
    max_time
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;
-- Clear stats
SELECT pg_stat_statements_reset();

Analyze Query Performance

-- EXPLAIN shows query plan WITHOUT executing
EXPLAIN SELECT * FROM users WHERE email = 'test@example.com';

-- Output:
-- QUERY PLAN
-- ─────────────────────────────────────────────
--  Index Scan using idx_users_email on users
--    Index Cond: (email = 'test@example.com')

-- EXPLAIN ANALYZE executes and shows actual performance
EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'test@example.com';

-- Output includes actual timing and row counts
-- ─────────────────────────────────────────────
--  Index Scan using idx_users_email on users
--    Index Cond: (email = 'test@example.com')
--    Planning Time: 0.123 ms
--    Execution Time: 0.456 ms

-- Complex query analysis
EXPLAIN (ANALYZE, BUFFERS) 
SELECT u.username, COUNT(p.id) as posts
FROM users u
LEFT JOIN posts p ON u.id = p.user_id
GROUP BY u.id
ORDER BY posts DESC;

Write-Ahead Log (WAL): The Safety Net

WAL is PostgreSQL’s transaction log. It’s what makes PostgreSQL reliable.

Understanding WAL

# Check WAL directory
ls -lah /var/lib/postgresql/16/main/pg_wal/

# Output:
# -rw------- 1 postgres postgres 16M Nov 15 10:23 000000010000000000000001
# -rw------- 1 postgres postgres 16M Nov 15 10:25 000000010000000000000002
# -rw------- 1 postgres postgres 16M Nov 15 10:27 000000010000000000000003
# Each WAL file is 16MB
# Files are named sequentially
# Older files are deleted after checkpoint (unless archiving is enabled)

Configure WAL Archiving (for backups)

# Create archive directory
sudo mkdir -p /backup/wal_archive
sudo chown postgres:postgres /backup/wal_archive
sudo chmod 700 /backup/wal_archive

# Edit postgresql.conf
sudo nano /etc/postgresql/16/main/postgresql.conf

# Add these lines:
# wal_level = replica
# archive_mode = on
# archive_command = 'cp %p /backup/wal_archive/%f'
# archive_timeout = 300  # Force archiving every 5 minutes

After editing:

# Reload configuration
sudo -u postgres psql -c "SELECT pg_reload_conf()"

# Force a checkpoint to start archiving
sudo -u postgres psql -c "CHECKPOINT"

# Verify WAL files are being archived
ls -la /backup/wal_archive/

# Output: WAL files copied here

Monitor WAL

-- Check WAL level
SHOW wal_level;
-- Output: replica

-- Check WAL stats
SELECT 
    count(*) as wal_files,
    pg_size_pretty(sum(size)) as total_size
FROM (
    SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), '0/0')::bigint as size
) as t;

-- Current WAL position
SELECT pg_current_wal_lsn();
-- Output: 0/17AB3F0

Backup Strategies

To avoid worst scenarios, it’s required to setup backup strategy. Here are some approaches we can use.

Strategy 1: pg_dump (Best for small-medium databases)

# Simple backup of one database
pg_dump -U postgres -d production_app -F c -f backup.dump

# Backup with compression
pg_dump -U postgres -d production_app \
    -F c \
    -f production_app_$(date +%Y%m%d_%H%M%S).dump

# Backup all databases
pg_dumpall -U postgres -f all_databases.sql

# Custom format (recommended for large DBs)
pg_dump -U postgres -d production_app \
    -F c \
    --verbose \
    -f backup.dump

# Backup with jobs (parallel, faster)
pg_dump -U postgres -d production_app \
    -F d \
    -j 4 \
    -f backup_directory/

# Show backup size
ls -lh backup.dump

Strategy 2: Restore from Backup

# Restore from custom format dump
pg_restore -U postgres -d production_app -F c backup.dump

# List what's in the dump before restoring
pg_restore -l backup.dump | head -20

# Restore specific table
pg_restore -U postgres -d production_app -F c \
    -t users backup.dump

# Restore with progress
pg_restore -U postgres -d production_app \
    -F c -j 4 --verbose backup.dump

Strategy 3: Continuous Archiving + Point-in-Time Recovery

This is the most reliable backup strategy:

# 1. Full backup
pg_basebackup -h localhost -U postgres -D /backup/base_backup -Fp -Pv

# 2. WAL archiving runs continuously (configured in postgresql.conf)
# 3. To restore to specific point in time:

# Stop PostgreSQL
sudo systemctl stop postgresql

# Move current data directory
sudo mv /var/lib/postgresql/16/main /var/lib/postgresql/16/main.old

# Restore base backup
sudo cp -r /backup/base_backup /var/lib/postgresql/16/main
sudo chown -R postgres:postgres /var/lib/postgresql/16/main

# Create recovery.signal to enable recovery
sudo touch /var/lib/postgresql/16/main/recovery.signal

# Configure recovery settings
sudo nano /var/lib/postgresql/16/main/postgresql.conf

# Add:
# restore_command = 'cp /backup/wal_archive/%f %p'
# recovery_target_timeline = 'latest'
# recovery_target_time = '2024-01-15 14:30:00'

# Start PostgreSQL (it will replay WAL)
sudo systemctl start postgresql
# Verify recovery
sudo -u postgres psql -c "SELECT now()"

Strategy 4: Automated Backup Script

#!/bin/bash
# backup.sh

BACKUP_DIR="/backup/postgresql"
DB_USER="postgres"
RETENTION_DAYS=30
# Create backup directory
mkdir -p $BACKUP_DIR
# Full backup daily at 2 AM
BACKUP_FILE="$BACKUP_DIR/db_$(date +%Y%m%d_%H%M%S).dump"
echo "Starting backup..."
pg_dump -U $DB_USER -d production_app -F c -f $BACKUP_FILE
if [ $? -eq 0 ]; then
    echo "✓ Backup successful: $BACKUP_FILE"
    
    # Verify backup
    pg_restore -U $DB_USER -d production_app -F c -l $BACKUP_FILE > /dev/null
    if [ $? -eq 0 ]; then
        echo "✓ Backup verified"
    fi
else
    echo "✗ Backup failed"
    exit 1
fi
# Remove old backups (older than RETENTION_DAYS)
find $BACKUP_DIR -type f -name "*.dump" -mtime +$RETENTION_DAYS -delete
echo "✓ Cleanup completed"
# Optional: Copy to remote storage
# scp $BACKUP_FILE remote_server:/remote/backup/

Schedule it:

# Add to crontab
crontab -e

# Add line (runs daily at 2 AM)
0 2 * * * /home/postgres/backup.sh >> /var/log/pg_backup.log 2>&1

# List cron jobs
crontab -l

How to Set Up Streaming Replication

Primary Server (Production):

# Edit postgresql.conf
sudo nano /etc/postgresql/16/main/postgresql.conf

# Add:
wal_level = replica
max_wal_senders = 10
wal_keep_size = 1GB
hot_standby = on

# Edit pg_hba.conf for replication user
sudo nano /etc/postgresql/16/main/pg_hba.conf
# Add (allow replica connection):
host    replication    replica_user    192.168.1.0/24    md5

# Create replication user
sudo -u postgres psql -c "CREATE USER replica_user WITH REPLICATION ENCRYPTED PASSWORD 'replica_password';"

# Restart
sudo systemctl restart postgresql

Replica Server (Standby):

# Stop replica if running
sudo systemctl stop postgresql

# Clear old data directory
sudo rm -rf /var/lib/postgresql/16/main/*

# Take base backup from primary
pg_basebackup -h 192.168.1.100 -U replica_user -D /var/lib/postgresql/16/main -Fp -Pv

# Create standby.signal (tells PostgreSQL this is standby)
sudo touch /var/lib/postgresql/16/main/standby.signal

# Configure replica postgresql.conf
sudo nano /var/lib/postgresql/16/main/postgresql.conf

# Add:
primary_conninfo = 'host=192.168.1.100 port=5432 user=replica_user password=replica_password'
hot_standby = on

# Set correct permissions
sudo chown -R postgres:postgres /var/lib/postgresql/16/main
sudo chmod 700 /var/lib/postgresql/16/main

# Start replica
sudo systemctl start postgresql

# Verify replication is working
sudo -u postgres psql -c "SELECT * FROM pg_stat_replication;"
# On replica:
sudo -u postgres psql -c "SELECT * FROM pg_last_wal_receive_lsn();"

Monitor Replication

-- On Primary: Check replica lag
SELECT 
    client_addr,
    state,
    sent_lsn,
    write_lsn,
    flush_lsn,
    replay_lsn,
    sync_state
FROM pg_stat_replication;

-- On Replica: Check replication status
SELECT 
    pg_last_wal_receive_lsn() as receive_lsn,
    pg_last_wal_replay_lsn() as replay_lsn,
    pg_is_wal_replay_paused() as is_paused;

-- Calculate replication lag (in bytes)
SELECT 
    pg_wal_lsn_diff(sent_lsn, replay_lsn) as lag_bytes
FROM pg_stat_replication;

Promote Replica to Primary

If primary fails:

# On replica
sudo -u postgres psql -c "SELECT pg_promote();"

# Verify it's now primary
sudo -u postgres psql -c "SELECT pg_is_in_recovery();"
# Output: f (false = it's primary now)

Some Performance Tuning and log analyzing

# Enable query logging
sudo -u postgres psql -c "ALTER SYSTEM SET log_min_duration_statement = 1000;"
sudo -u postgres psql -c "SELECT pg_reload_conf();"

# View logs
tail -f /var/lib/postgresql/16/main/log/postgresql-*.log | grep "duration:"

# Parse and analyze
grep "duration:" /var/lib/postgresql/16/main/log/postgresql-*.log | \
    awk '{print $NF}' | sort -rn | head -20

If needed, we can optimize common issues:

-- 1. Add missing indexes
-- Query to find queries without indexes:
SELECT query FROM pg_stat_statements WHERE query LIKE '%WHERE%' LIMIT 5;

-- Add index after analyzing with EXPLAIN
CREATE INDEX idx_posts_user_id_created_at ON posts(user_id, created_at DESC);

-- 2. Analyze table statistics
ANALYZE users;
ANALYZE posts;

-- 3. Vacuum (reclaim space, update statistics)
VACUUM ANALYZE;

-- 4. Check for table bloat
SELECT 
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as total_size,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename) - pg_relation_size(schemaname||'.'||tablename)) as indexes_size
FROM pg_tables
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 10;

-- 5. Enable auto-vacuum (maintenance)
SHOW autovacuum;
ALTER SYSTEM SET autovacuum = on;
SELECT pg_reload_conf();

Monitoring and Alerting: Key metrics to monitor

# 1. Check active connections
sudo -u postgres psql -c "SELECT count(*) FROM pg_stat_activity;"

# 2. Check cache hit ratio (should be > 99%)
sudo -u postgres psql << EOF
SELECT 
    sum(heap_blks_read) as heap_read,
    sum(heap_blks_hit) as heap_hit,
    sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) as ratio
FROM pg_statio_user_tables;
EOF

# 3. Check disk usage
df -h /var/lib/postgresql

# 4. Check transaction ID age (prevent wraparound)
sudo -u postgres psql -c "SELECT datname, age(datfrozenxid) FROM pg_database;"

# 5. Check replication lag (if using replication)
sudo -u postgres psql -c "SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), '0/0')/1024/1024 as lag_mb;"

Create Monitoring Alert Script

#!/bin/bash
# pg_monitor.sh

DB_USER="postgres"
DB_NAME="production_app"
WARNING_CONNECTIONS=150
CRITICAL_CONNECTIONS=180
# Check connections
CONNECTIONS=$(sudo -u $DB_USER psql -d $DB_NAME -t -c "SELECT count(*) FROM pg_stat_activity;")
if [ "$CONNECTIONS" -gt "$CRITICAL_CONNECTIONS" ]; then
    echo "CRITICAL: PostgreSQL connections: $CONNECTIONS/$MAX_CONNECTIONS"
    # Send alert (email, PagerDuty, Slack)
elif [ "$CONNECTIONS" -gt "$WARNING_CONNECTIONS" ]; then
    echo "WARNING: PostgreSQL connections: $CONNECTIONS"
fi
# Check cache hit ratio
CACHE_HIT=$(sudo -u $DB_USER psql -d $DB_NAME -t -c "SELECT round(sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) * 100, 2) FROM pg_statio_user_tables;")
if (( $(echo "$CACHE_HIT < 95" | bc -l) )); then
    echo "WARNING: Low cache hit ratio: $CACHE_HIT%"
fi
# Check disk usage
DISK_USED=$(df /var/lib/postgresql | awk 'NR==2 {print $5}' | sed 's/%//')
if [ "$DISK_USED" -gt 80 ]; then
    echo "CRITICAL: Disk usage: $DISK_USED%"
fi

To schedule monitoring I use crontab:

# Add to crontab (check every 5 minutes)
*/5 * * * * /home/postgres/pg_monitor.sh >> /var/log/pg_monitor.log 2>&1

Disaster Recovery Checklist

Before Disaster

# 1. Verify backups work
pg_restore -U postgres -d test_db -F c /backup/database.dump

# 2. Document recovery procedure
# Create recovery_procedure.txt with all steps

# 3. Test failover
# Promote replica to primary

# 4. Document connection strings
# Primary: postgresql://user:pass@primary-ip:5432/db
# Replica: postgresql://user:pass@replica-ip:5432/db

# 5. Monitor backup age
find /backup -name "*.dump" -type f -mtime +1 -exec ls -la {} \;

When Disaster Strikes

# 1. Assess the damage
sudo systemctl status postgresql
sudo journalctl -n 50 -u postgresql

# 2. Check data directory integrity
sudo -u postgres pg_controldata /var/lib/postgresql/16/main | head -20

# 3. Try recovery first (don't restore unless necessary)
sudo systemctl restart postgresql
sudo tail -f /var/lib/postgresql/16/main/log/postgresql-*.log

# 4. If corrupted, restore from backup
# Follow "Strategy 3: Continuous Archiving" section above

# 5. Verify data after recovery
sudo -u postgres psql -c "SELECT count(*) FROM users;"

Common Commands Quick Reference

# Connection & Authentication
psql -U postgres -h localhost -d postgres
psql postgresql://user:password@host:5432/dbname

# Database Operations
\l                    # List databases
\c dbname            # Connect to database
CREATE DATABASE name;
DROP DATABASE name;

# Table Operations
\dt                  # List tables
\d tablename         # Show table structure
\di                  # List indexes
\dn                  # List schemas

# User Management
\du                  # List users
CREATE USER name WITH PASSWORD 'pass';
DROP USER name;
GRANT ALL ON DATABASE db TO user;

# Backup & Restore
pg_dump -U user -d db -F c -f backup.dump
pg_restore -U user -d db -F c backup.dump
pg_basebackup -U user -D /path -Fp -Pv

# Monitoring
SELECT * FROM pg_stat_activity;
SELECT datname, pg_size_pretty(pg_database_size(datname)) FROM pg_database;
EXPLAIN ANALYZE SELECT ...;

# Performance
VACUUM ANALYZE;
REINDEX TABLE tablename;
ALTER TABLE tablename SET (fillfactor=80);

# Replication
SELECT * FROM pg_stat_replication;
SELECT pg_promote();

Production Deployment Checklist

☐ postgresql.conf tuned (shared_buffers, wal_level, etc.)
☐ pg_hba.conf restricted (only necessary IPs)
☐ Users created with appropriate privileges
☐ Backup strategy implemented and tested
☐ WAL archiving configured
☐ Monitoring and alerting in place
☐ Replication configured (if high availability needed)
☐ Connection pooling configured (in application)
☐ Slow query logging enabled
☐ Regular VACUUM ANALYZE scheduled
☐ Cache hit ratio verified (>99%)
☐ Disk space monitored
☐ Recovery procedure documented
☐ Disaster recovery tested
☐ pg_stat_statements extension installed
☐ SSL/TLS configured for remote connections

You now understand:

• How PostgreSQL writes data reliably (WAL)
• How to back up data safely (pg_dump, pg_basebackup)
• How to recover from disaster (point-in-time recovery)
• How to scale without downtime (replication)
• How to optimize performance (indexes, monitoring)

This isn’t DBA-level knowledge. This is DevOps knowledge. Enough to run PostgreSQL safely in production.

The Blind Spot

Are you always shure the system is healthy?
If not, what would be your approach checking it? SSH into a server and check disk space & logs?

This is the reality for engineers without proper monitoring. You’re flying blind. You don’t know if your application is slow because of database queries or network latency. You don’t know if you’re approaching capacity. You discover outages from customer complaints on Twitter.

Then one day you realize: what if you could see everything?

Not logs. Logs tell you what happened after it already broke. You need metrics — real-time data about your system’s behavior. Response times. Error rates. Memory usage. Database connections. Every number that matters.

This is what Prometheus does.

Prometheus is a time-series database that scrapes metrics from your applications, stores them efficiently, and lets you query them to understand your system’s behavior. Unlike traditional monitoring that pulls from agents, Prometheus works differently: your applications push their metrics in a standard format, and Prometheus periodically pulls them.

By the end of this article, you’ll have a production-capable monitoring system that shows you exactly what’s happening inside your applications.

Why Prometheus? Why Now?

Before Prometheus, monitoring was complicated. Nagios. Zabbix. Complex agent installations. Steep learning curves. Proprietary formats.

Prometheus changed the game with four radical simplifications:

1. Simple text format — Metrics are readable human-friendly text, not binary protocols
2. Pull-based, not push — No complex agent infrastructure, just HTTP endpoints
3. Powerful query language — PromQL lets you ask complex questions about your data
4. Designed for microservices — Built to handle ephemeral containers and dynamic infrastructure

Here’s the key insight: Prometheus doesn’t complicate things. It simplifies them.

The Mental Model

Understanding Prometheus requires understanding three concepts:

1. Metrics

A metric is a measurement. Response time. Error count. Memory usage. Queue depth. Every number from your system is a metric.

Prometheus has four metric types:

• Counter — Only goes up (errors, requests processed)
• Gauge — Goes up and down (memory usage, active connections)
• Histogram — Tracks distribution (response times)
• Summary — Like histogram, calculates percentiles

2. Labels

A metric alone isn’t enough. You need context. The same metric means different things on different servers.

That’s where labels come in. They’re key-value pairs that add context:

http_requests_total{method="GET", path="/api/users", status="200"} 1534

This single line tells you: “1534 GET requests to /api/users that returned status 200.”

Labels let you slice your data any way you want. Query errors by endpoint. Compare response times across servers. Drill down to specific problems.

3. Scraping

Prometheus doesn’t wait for data. It periodically connects to your application at /metrics endpoint and pulls all metrics. This is called "scraping."

Your application exposes metrics. Prometheus scrapes them every 15 seconds (or whatever you configure). Prometheus stores them. You query them.

This is the pull model. Simple. Reliable. Resilient.

Building a Go Microservice with Metrics

Let’s write some Go!
Most of my articles are connected with this brilliant language.

I’ll do a simple API service that tracks:

• How many requests it receives
• How long requests take
• What errors occur
• Custom business metrics

Create main.go:

package main

import (
 "fmt"
 "github.com/prometheus/client_golang/prometheus"
 "github.com/prometheus/client_golang/prometheus/promauto"
 "github.com/prometheus/client_golang/prometheus/promhttp"
 "log"
 "math/rand"
 "net/http"
 "time"
)

var (
 // Counter: total requests received
 httpRequestsTotal = promauto.NewCounterVec(
  prometheus.CounterOpts{
   Name: "http_requests_total",
   Help: "Total number of HTTP requests",
  },
  []string{"method", "endpoint", "status"},
 )

 // Histogram: request duration in seconds
 httpRequestDuration = promauto.NewHistogramVec(
  prometheus.HistogramOpts{
   Name:    "http_request_duration_seconds",
   Help:    "HTTP request latency in seconds",
   Buckets: prometheus.DefBuckets,
  },
  []string{"method", "endpoint"},
 )

 // Gauge: active connections
 activeConnections = promauto.NewGauge(
  prometheus.GaugeOpts{
   Name: "active_connections",
   Help: "Number of active connections",
  },
 )

 // Counter: database errors
 databaseErrors = promauto.NewCounterVec(
  prometheus.CounterOpts{
   Name: "database_errors_total",
   Help: "Total number of database errors",
  },
  []string{"operation"},
 )

 // Gauge: items in processing queue
 queueDepth = promauto.NewGauge(
  prometheus.GaugeOpts{
   Name: "queue_depth",
   Help: "Number of items in processing queue",
  },
 )
)

func metricsMiddleware(next http.Handler) http.Handler {
 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
  start := time.Now()
  activeConnections.Inc()
  defer activeConnections.Dec()

  wrapped := &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}
  next.ServeHTTP(wrapped, r)

  // Record metrics
  duration := time.Since(start).Seconds()
  status := fmt.Sprintf("%d", wrapped.statusCode)
  httpRequestsTotal.WithLabelValues(
   r.Method,
   r.URL.Path,
   status,
  ).Inc()
  httpRequestDuration.WithLabelValues(
   r.Method,
   r.URL.Path,
  ).Observe(duration)
 })
}

type responseWriter struct {
 http.ResponseWriter
 statusCode int
}

func (rw *responseWriter) WriteHeader(code int) {
 rw.statusCode = code
 rw.ResponseWriter.WriteHeader(code)
}

func simulateDatabaseOperation() error {
 // 10% chance of error
 if rand.Float32() < 0.1 {
  databaseErrors.WithLabelValues("query").Inc()
  return fmt.Errorf("database error")
 }
 return nil
}

func handleGetUsers(w http.ResponseWriter, r *http.Request) {
 // Simulate processing time (10-100ms)
 delay := time.Duration(rand.Intn(90)+10) * time.Millisecond
 time.Sleep(delay)
 if err := simulateDatabaseOperation(); err != nil {
  http.Error(w, "Database error", http.StatusInternalServerError)
  return
 }
 w.Header().Set("Content-Type", "application/json")
 fmt.Fprintf(w, `{"users": [{"id": 1, "name": "Alice"}]}`)
}

func handleGetUser(w http.ResponseWriter, r *http.Request) {
 delay := time.Duration(rand.Intn(50)+5) * time.Millisecond
 time.Sleep(delay)
 if err := simulateDatabaseOperation(); err != nil {
  http.Error(w, "Database error", http.StatusInternalServerError)
  return
 }
 w.Header().Set("Content-Type", "application/json")
 fmt.Fprintf(w, `{"id": 1, "name": "Alice"}`)
}

func handleCreateUser(w http.ResponseWriter, r *http.Request) {
 delay := time.Duration(rand.Intn(200)+50) * time.Millisecond
 time.Sleep(delay)
 if err := simulateDatabaseOperation(); err != nil {
  http.Error(w, "Database error", http.StatusInternalServerError)
  return
 }
 w.WriteHeader(http.StatusCreated)
 fmt.Fprintf(w, `{"id": 123}`)
}

// We will Simulate background job that processes queue
func simulateBackgroundJob() {
 ticker := time.NewTicker(5 * time.Second)
 for range ticker.C {
  // Random queue depth between 0-50
  depth := rand.Intn(51)
  queueDepth.Set(float64(depth))
 }
}
func main() {
 go simulateBackgroundJob()

 mux := http.NewServeMux()
 mux.HandleFunc("/api/users", handleGetUsers)
 mux.HandleFunc("/api/users/1", handleGetUser)
 mux.HandleFunc("/api/users/create", handleCreateUser)

 handler := metricsMiddleware(mux)

 // Expose Prometheus metrics at /metrics
 http.Handle("/metrics", promhttp.Handler())
 http.HandleFunc("/", handler.ServeHTTP)
 log.Println("Service started on :8080")
 log.Println("Metrics available at http://localhost:8080/metrics")
 log.Fatal(http.ListenAndServe(":8080", nil))
}

To run this, you need the Prometheus client library:

go mod init user-service
go get github.com/prometheus/client_golang/...
go run main.go

Now visit http://localhost:8080/metrics and you'll see raw Prometheus format:

# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{endpoint="/api/users",method="GET",status="200"} 42

# HELP http_request_duration_seconds HTTP request latency in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{endpoint="/api/users",method="GET",le="0.005"} 5
http_request_duration_seconds_bucket{endpoint="/api/users",method="GET",le="0.01"} 12
...
# HELP active_connections Number of active connections
# TYPE active_connections gauge
active_connections 0

# HELP database_errors_total Total number of database errors
# TYPE database_errors_total counter
database_errors_total{operation="query"} 3

# HELP queue_depth Number of items in processing queue
# TYPE queue_depth gauge
queue_depth 17

This is Prometheus format. Your application is now instrumented.

Installing and Configuring Prometheus

Now that your application exposes metrics, you need Prometheus to scrape them.

Installation

# Download Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xvfz prometheus-2.48.0.linux-amd64.tar.gz
cd prometheus-2.48.0.linux-amd64

Or using Docker:

docker run -d -p 9090:9090 prom/prometheus

Configuration

Create prometheus.yml:

# Global settings
global:
  scrape_interval: 15s        # How often to scrape metrics
  evaluation_interval: 15s    # How often to evaluate alert rules
  external_labels:
    monitor: 'production'     # Label all metrics with this

# Alert manager configuration (we'll use this later)
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'localhost:9093'

# Alert rules
rule_files:
  - 'alerts.yml'

# Scrape configurations
scrape_configs:

  # Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Our user service
  - job_name: 'user-service'
    static_configs:
      - targets: ['localhost:8080']

    # Override global scrape interval for this job
    scrape_interval: 5s
    scrape_timeout: 5s

    # Add custom labels to all metrics from this job
    metric_relabel_configs:
      - source_labels: [__name__]
        target_label: service
        replacement: 'user-service'

Key concepts here:

• scrape_interval — How often Prometheus pulls metrics (15 seconds by default)
• evaluation_interval — How often alert rules are checked
• job_name — Logical name for this target
• targets — What to scrape (IP:port)
• metric_relabel_configs — Transform metrics as they're scraped

Run Prometheus

# If installed locally
./prometheus --config.file=prometheus.yml

# If using Docker
docker run -d -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Visit http://localhost:9090 and you'll see the Prometheus UI.

PromQL: Asking Questions About Your Data

Prometheus is useless if you can’t query it. That’s where PromQL comes in — a query language designed for time-series data.

Let’s explore it using the Prometheus UI. Click on the “Graph” tab and enter queries.

Simple Queries

What’s the current queue depth?

queue_depth

Returns: 17

How many total requests have we received?

http_requests_total

Returns all metrics with this name, broken down by labels.

How many successful GET requests to /api/users?

http_requests_total{method="GET", endpoint="/api/users", status="200"}

Ranges and Aggregation

Average response time over the last 5 minutes?

avg(rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]))

This looks complex, but it’s calculating: total seconds / total count = average.

Request rate (requests per second) over last minute?

rate(http_requests_total[1m])

rate() calculates per-second rate of change. Over 1 minute, it shows requests/sec.

Error rate for /api/users endpoint?

rate(http_requests_total{endpoint="/api/users", status="5xx"}[5m]) / rate(http_requests_total{endpoint="/api/users"}[5m])

Errors divided by total = error rate. Multiply by 100 for percentage.

Aggregation Across Labels

Total requests across all endpoints?

sum(rate(http_requests_total[5m]))

By method?

sum(rate(http_requests_total[5m])) by (method)

By endpoint?

sum(rate(http_requests_total[5m])) by (endpoint)

Top 3 slowest endpoints?

topk(3, avg(http_request_duration_seconds) by (endpoint))

Comparing Time Periods

How much traffic increase compared to 1 hour ago?

rate(http_requests_total[1m]) / rate(http_requests_total offset 1h [1m])

The offset modifier lets you reference past data.

PromQL Operators

• + - * / — Arithmetic
• == != > < >= <= — Comparison
• and or unless — Logical
• bool — Convert comparison to 0/1

Which endpoints are slower than 100ms?

(http_request_duration_seconds > 0.1) * bool

The beauty of PromQL is that it’s designed for the questions you actually ask: “Is this broken? How much traffic? Compare to yesterday? Which thing is slowest?”

Prometheus from Zero: Metrics Collection and Alerting

How to see inside your applications and know when things break before your users complain

The Blind Spot

It’s 3 PM on a Wednesday. Your boss walks by: “Is the system healthy?”

Your answer: “…I think so? Let me SSH into a server and check disk space.”

This is the reality for engineers without proper monitoring. You’re flying blind. You don’t know if your application is slow because of database queries or network latency. You don’t know if you’re approaching capacity. You discover outages from customer complaints on Twitter.

Then one day you realize: what if you could see everything?

Not logs. Logs tell you what happened after it already broke. You need metrics — real-time data about your system’s behavior. Response times. Error rates. Memory usage. Database connections. Every number that matters.

This is what Prometheus does.

Prometheus is a time-series database that scrapes metrics from your applications, stores them efficiently, and lets you query them to understand your system’s behavior. Unlike traditional monitoring that pulls from agents, Prometheus works differently: your applications push their metrics in a standard format, and Prometheus periodically pulls them.

By the end of this article, you’ll have a production-capable monitoring system that shows you exactly what’s happening inside your applications.

Why Prometheus? Why Now?

Before Prometheus, monitoring was complicated. Nagios. Zabbix. Complex agent installations. Steep learning curves. Proprietary formats.

Prometheus changed the game with four radical simplifications:

1. Simple text format — Metrics are readable human-friendly text, not binary protocols
2. Pull-based, not push — No complex agent infrastructure, just HTTP endpoints
3. Powerful query language — PromQL lets you ask complex questions about your data
4. Designed for microservices — Built to handle ephemeral containers and dynamic infrastructure

Here’s the key insight: Prometheus doesn’t complicate things. It simplifies them.

The Mental Model

Understanding Prometheus requires understanding three concepts:

1. Metrics

A metric is a measurement. Response time. Error count. Memory usage. Queue depth. Every number from your system is a metric.

Prometheus has four metric types:

• Counter — Only goes up (errors, requests processed)
• Gauge — Goes up and down (memory usage, active connections)
• Histogram — Tracks distribution (response times)
• Summary — Like histogram, calculates percentiles

2. Labels

A metric alone isn’t enough. You need context. The same metric means different things on different servers.

That’s where labels come in. They’re key-value pairs that add context:

http_requests_total{method="GET", path="/api/users", status="200"} 1534

This single line tells you: “1534 GET requests to /api/users that returned status 200.”

Labels let you slice your data any way you want. Query errors by endpoint. Compare response times across servers. Drill down to specific problems.

3. Scraping

Prometheus doesn’t wait for data. It periodically connects to your application at /metrics endpoint and pulls all metrics. This is called "scraping."

Your application exposes metrics. Prometheus scrapes them every 15 seconds (or whatever you configure). Prometheus stores them. You query them.

This is the pull model. Simple. Reliable. Resilient.

Building a Go Microservice with Metrics

Let’s build something real. A simple API service that tracks:

• How many requests it receives
• How long requests take
• What errors occur
• Custom business metrics

Create main.go:

package main

import (
	"fmt"
	"log"
	"math/rand"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
	// Counter: total requests received
	httpRequestsTotal = promauto.NewCounterVec(
		prometheus.CounterOpts{
			Name: "http_requests_total",
			Help: "Total number of HTTP requests",
		},
		[]string{"method", "endpoint", "status"},
	)

	// Histogram: request duration in seconds
	httpRequestDuration = promauto.NewHistogramVec(
		prometheus.HistogramOpts{
			Name:    "http_request_duration_seconds",
			Help:    "HTTP request latency in seconds",
			Buckets: prometheus.DefBuckets,
		},
		[]string{"method", "endpoint"},
	)

	// Gauge: active connections
	activeConnections = promauto.NewGauge(
		prometheus.GaugeOpts{
			Name: "active_connections",
			Help: "Number of active connections",
		},
	)

	// Counter: database errors
	databaseErrors = promauto.NewCounterVec(
		prometheus.CounterOpts{
			Name: "database_errors_total",
			Help: "Total number of database errors",
		},
		[]string{"operation"},
	)

	// Gauge: items in processing queue
	queueDepth = promauto.NewGauge(
		prometheus.GaugeOpts{
			Name: "queue_depth",
			Help: "Number of items in processing queue",
		},
	)
)

// Middleware to track request metrics
func metricsMiddleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()
		activeConnections.Inc()
		defer activeConnections.Dec()

		// Create a response writer wrapper to capture status code
		wrapped := &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}

		next.ServeHTTP(wrapped, r)

		// Record metrics
		duration := time.Since(start).Seconds()
		status := fmt.Sprintf("%d", wrapped.statusCode)

		httpRequestsTotal.WithLabelValues(
			r.Method,
			r.URL.Path,
			status,
		).Inc()

		httpRequestDuration.WithLabelValues(
			r.Method,
			r.URL.Path,
		).Observe(duration)
	})
}

type responseWriter struct {
	http.ResponseWriter
	statusCode int
}

func (rw *responseWriter) WriteHeader(code int) {
	rw.statusCode = code
	rw.ResponseWriter.WriteHeader(code)
}

// Simulate database operation
func simulateDatabaseOperation() error {
	// 10% chance of error
	if rand.Float32() < 0.1 {
		databaseErrors.WithLabelValues("query").Inc()
		return fmt.Errorf("database error")
	}
	return nil
}

// Handlers
func handleGetUsers(w http.ResponseWriter, r *http.Request) {
	// Simulate processing time (10-100ms)
	delay := time.Duration(rand.Intn(90)+10) * time.Millisecond
	time.Sleep(delay)

	if err := simulateDatabaseOperation(); err != nil {
		http.Error(w, "Database error", http.StatusInternalServerError)
		return
	}

	w.Header().Set("Content-Type", "application/json")
	fmt.Fprintf(w, `{"users": [{"id": 1, "name": "Alice"}]}`)
}

func handleGetUser(w http.ResponseWriter, r *http.Request) {
	delay := time.Duration(rand.Intn(50)+5) * time.Millisecond
	time.Sleep(delay)

	if err := simulateDatabaseOperation(); err != nil {
		http.Error(w, "Database error", http.StatusInternalServerError)
		return
	}

	w.Header().Set("Content-Type", "application/json")
	fmt.Fprintf(w, `{"id": 1, "name": "Alice"}`)
}

func handleCreateUser(w http.ResponseWriter, r *http.Request) {
	delay := time.Duration(rand.Intn(200)+50) * time.Millisecond
	time.Sleep(delay)

	if err := simulateDatabaseOperation(); err != nil {
		http.Error(w, "Database error", http.StatusInternalServerError)
		return
	}

	w.WriteHeader(http.StatusCreated)
	fmt.Fprintf(w, `{"id": 123}`)
}

// Simulate background job that processes queue
func simulateBackgroundJob() {
	ticker := time.NewTicker(5 * time.Second)
	for range ticker.C {
		// Random queue depth between 0-50
		depth := rand.Intn(51)
		queueDepth.Set(float64(depth))
	}
}

func main() {
	// Start background job
	go simulateBackgroundJob()

	// Setup routes with metrics middleware
	mux := http.NewServeMux()
	mux.HandleFunc("/api/users", handleGetUsers)
	mux.HandleFunc("/api/users/1", handleGetUser)
	mux.HandleFunc("/api/users/create", handleCreateUser)

	// Wrap all handlers with metrics middleware
	handler := metricsMiddleware(mux)

	// Expose Prometheus metrics at /metrics
	http.Handle("/metrics", promhttp.Handler())
	http.HandleFunc("/", handler.ServeHTTP)

	log.Println("Service started on :8080")
	log.Println("Metrics available at http://localhost:8080/metrics")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

To run this, you need the Prometheus client library:

go mod init user-service
go get github.com/prometheus/client_golang/...
go run main.go

Now visit http://localhost:8080/metrics and you'll see raw Prometheus format:

# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{endpoint="/api/users",method="GET",status="200"} 42

# HELP http_request_duration_seconds HTTP request latency in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{endpoint="/api/users",method="GET",le="0.005"} 5
http_request_duration_seconds_bucket{endpoint="/api/users",method="GET",le="0.01"} 12
...

# HELP active_connections Number of active connections
# TYPE active_connections gauge
active_connections 0

# HELP database_errors_total Total number of database errors
# TYPE database_errors_total counter
database_errors_total{operation="query"} 3

# HELP queue_depth Number of items in processing queue
# TYPE queue_depth gauge
queue_depth 17

This is Prometheus format. Your application is now instrumented.

Installing and Configuring Prometheus

Now that your application exposes metrics, you need Prometheus to scrape them.

Installation

# Download Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xvfz prometheus-2.48.0.linux-amd64.tar.gz
cd prometheus-2.48.0.linux-amd64

Or using Docker:

docker run -d -p 9090:9090 prom/prometheus

Configuration

Create prometheus.yml:

# Global settings
global:
  scrape_interval: 15s        # How often to scrape metrics
  evaluation_interval: 15s    # How often to evaluate alert rules
  external_labels:
    monitor: 'production'     # Label all metrics with this

# Alert manager configuration (we'll use this later)
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'localhost:9093'

# Alert rules
rule_files:
  - 'alerts.yml'

# Scrape configurations
scrape_configs:
  # Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Our user service
  - job_name: 'user-service'
    static_configs:
      - targets: ['localhost:8080']
    # Override global scrape interval for this job
    scrape_interval: 5s
    scrape_timeout: 5s
    # Add custom labels to all metrics from this job
    metric_relabel_configs:
      - source_labels: [__name__]
        target_label: service
        replacement: 'user-service'

Key concepts here:

• scrape_interval — How often Prometheus pulls metrics (15 seconds by default)
• evaluation_interval — How often alert rules are checked
• job_name — Logical name for this target
• targets — What to scrape (IP:port)
• metric_relabel_configs — Transform metrics as they're scraped

Run Prometheus

# If installed locally
./prometheus --config.file=prometheus.yml

# If using Docker
docker run -d -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Visit http://localhost:9090 and you'll see the Prometheus UI.

PromQL: Asking Questions About Your Data

Prometheus is useless if you can’t query it. That’s where PromQL comes in — a query language designed for time-series data.

Let’s explore it using the Prometheus UI. Click on the “Graph” tab and enter queries.

Simple Queries

What’s the current queue depth?

queue_depth

Returns: 17

How many total requests have we received?

http_requests_total

Returns all metrics with this name, broken down by labels.

How many successful GET requests to /api/users?

http_requests_total{method="GET", endpoint="/api/users", status="200"}

Ranges and Aggregation

Average response time over the last 5 minutes?

avg(rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]))

This looks complex, but it’s calculating: total seconds / total count = average.

Request rate (requests per second) over last minute?

rate(http_requests_total[1m])

rate() calculates per-second rate of change. Over 1 minute, it shows requests/sec.

Error rate for /api/users endpoint?

rate(http_requests_total{endpoint="/api/users", status="5xx"}[5m]) / rate(http_requests_total{endpoint="/api/users"}[5m])

Errors divided by total = error rate. Multiply by 100 for percentage.

Aggregation Across Labels

Total requests across all endpoints?

sum(rate(http_requests_total[5m]))

By method?

sum(rate(http_requests_total[5m])) by (method)

By endpoint?

sum(rate(http_requests_total[5m])) by (endpoint)

Top 3 slowest endpoints?

topk(3, avg(http_request_duration_seconds) by (endpoint))

Comparing Time Periods

How much traffic increase compared to 1 hour ago?

rate(http_requests_total[1m]) / rate(http_requests_total offset 1h [1m])

The offset modifier lets you reference past data.

PromQL Operators

• + - * / — Arithmetic
• == != > < >= <= — Comparison
• and or unless — Logical
• bool — Convert comparison to 0/1

Which endpoints are slower than 100ms?

(http_request_duration_seconds > 0.1) * bool

The beauty of PromQL is that it’s designed for the questions you actually ask: “Is this broken? How much traffic? Compare to yesterday? Which thing is slowest?”

Creating Dashboards

Queries are powerful, but you don’t want to type them every time. Dashboards let you visualize multiple metrics at once.

In the Prometheus UI, click “+” and create a new dashboard. Add graphs for:

1. Request Rate

• sum(rate(http_requests_total[1m])) by (endpoint)

1. Error Rate

• sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100

1. Response Time (P95)

• histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

1. Active Connections

• active_connections

1. Database Errors

• rate(database_errors_total[5m])

1. Queue Depth

• queue_depth

Pro tip: Save your dashboard. Then share it with the team. Everyone can now see the same metrics without knowing PromQL.

Alerting: Know When Things Break

Metrics are great, but monitoring a dashboard 24/7 isn’t realistic. You need alerts that notify you when things go wrong.

Prometheus uses an “Alertmanager” for routing alerts, but let’s start with the alert rules themselves.

Create alerts.yml:

groups:
  - name: application_alerts
    interval: 30s
    rules:
      # Alert if error rate > 5% for 5 minutes
      - alert: HighErrorRate
        expr: |
          (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }} for {{ $labels.job }}"

      # Alert if p95 response time > 1 second
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency detected"
          description: "P95 latency is {{ $value | humanizeDuration }}"

      # Alert if database errors > 5 per minute
      - alert: DatabaseErrors
        expr: |
          rate(database_errors_total[1m]) > 0.05
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Database errors detected"
          description: "{{ $value | humanize }} database errors per second"

      # Alert if queue depth > 100
      - alert: QueueBacklog
        expr: |
          queue_depth > 100
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Queue backlog detected"
          description: "Queue depth is {{ $value }}"

      # Alert if service is down (no metrics for 2 minutes)
      - alert: ServiceDown
        expr: |
          up{job="user-service"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Service is down"
          description: "user-service has been unreachable for 2 minutes"

Add this to your prometheus.yml:

rule_files:
  - 'alerts.yml'

Restart Prometheus. Visit the Alerts tab. You’ll see your alert rules and their current state (firing, pending, or inactive).

Understanding Alert Rules

Each alert has:

• expr — PromQL expression that evaluates to true when alert should fire
• for — How long condition must be true before alerting (prevents flapping)
• labels — Metadata about the alert (severity, team, etc.)
• annotations — Human-readable description and summary

The for: 5m prevents false alerts from brief spikes. The alert only fires if the condition persists for 5 minutes.

Alertmanager: Where Alerts Go

Prometheus evaluates alert rules, but where do the alerts go? Alertmanager routes them.

Install Alertmanager

wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xvfz alertmanager-0.26.0.linux-amd64.tar.gz
cd alertmanager-0.26.0.linux-amd64

Create alertmanager.yml:

global:
  # Email configuration
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_auth_username: 'your-email@gmail.com'
  smtp_auth_password: 'your-app-password'
  smtp_require_tls: true

# Templates for alert messages
templates:
  - '/etc/alertmanager/templates/*.tmpl'

# Routes define how alerts are processed
route:
  # Default receiver for all alerts
  receiver: 'team-email'
  
  # Group alerts by these labels
  group_by: ['alertname', 'cluster', 'service']
  
  # Wait this long to group alerts before sending
  group_wait: 30s
  
  # Wait this long after a new alert joins group
  group_interval: 5m
  
  # How long to wait before repeating notification
  repeat_interval: 4h
  
  # Sub-routes: specific handling for certain alerts
  routes:
    # Critical alerts go to ops team immediately
    - match:
        severity: critical
      receiver: 'ops-pagerduty'
      group_wait: 10s
      repeat_interval: 1h
    
    # Warnings can be batched
    - match:
        severity: warning
      receiver: 'team-email'
      group_wait: 2m

# Receivers define where alerts are sent
receivers:
  - name: 'team-email'
    email_configs:
      - to: 'team@company.com'

  - name: 'ops-pagerduty'
    pagerduty_configs:
      - service_key: 'your-pagerduty-key'

Run Alertmanager:

./alertmanager --config.file=alertmanager.yml

Now when Prometheus fires an alert, it sends to Alertmanager, which routes it according to your rules.

Alert Routing Logic

Alertmanager’s power is in routing:

• Group similar alerts so you get one notification instead of 100
• Send critical alerts immediately to PagerDuty
• Send warnings via email (batched)
• Prevent alert spam with repeat intervals
• Match alerts based on labels and route accordingly

This is how professional teams handle alerts. You’re not overwhelmed with notifications. Critical issues wake you up. Warnings are batched.

Production Best Practices

1. Use Recording Rules

Instead of calculating the same queries repeatedly, pre-calculate them:

groups:
  - name: recording_rules
    interval: 1m
    rules:
      # Record error rate every minute
      - record: http:error_rate:1m
        expr: sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m]))

      # Record p95 latency every minute
      - record: http:p95_latency:1m
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[1m]))

Then reference these in alerts:

- alert: HighErrorRate
  expr: http:error_rate:1m > 0.05

Recording rules make queries faster and alerts more efficient.

2. Retention Strategy

Prometheus stores metrics in memory. Large environments need data retention limits:

global:
  scrape_interval: 15s
  # Keep data for 30 days
  # Adjust based on storage available

Run Prometheus with:

./prometheus --storage.tsdb.retention.time=30d

3. Storage Considerations

Prometheus stores roughly 1–2 bytes per sample. With 1000 metrics scraping every 15 seconds:

• Per day: ~172 MB
• Per month: ~5 GB
• Per year: ~60 GB

For production, consider:

• Running on fast SSD
• Using Prometheus remote storage for long-term retention
• Archiving old data regularly

4. High Availability

Run two independent Prometheus servers scraping the same targets. If one fails, you still have data. Tools like Thanos can deduplicate metrics from both.

5. Naming Conventions

Follow these conventions so metrics are discoverable:

<namespace>_<subsystem>_<name>_<unit>

http_request_duration_seconds
database_connections_active
memory_usage_bytes
api_calls_total

• No underscores between words
• Include units
• Use suffixes: _total, _count, _sum, _bucket, _seconds

6. Documentation

Every metric needs documentation:

http_requests_total = promauto.NewCounterVec(
	prometheus.CounterOpts{
		Name: "http_requests_total",
		Help: "Total number of HTTP requests processed",
	},
	[]string{"method", "endpoint", "status"},
)

The Help field is critical. A year later, someone needs to understand what this metric means.

The Observability Stack

Prometheus is part of a larger observability picture:

┌─────────────────────────────────────────┐
│        Observability Stack              │
├─────────────────────────────────────────┤
│ Metrics (Prometheus)   - What happened? │
│ Logs (ELK/Loki)        - Why it happened│
│ Traces (Jaeger)        - How it happened│
└─────────────────────────────────────────┘

Prometheus gives you what happened: error rates, latencies, resource usage. Logs tell you the story. Traces show the path a request took.

Together, they’re unstoppable for debugging.

From Blindness to Visibility

Remember the question “Is the system healthy?”

With Prometheus, your answer is now:

- Error rate: 0.1% (green)
- P95 latency: 45ms (green)
- Database errors: 0 per minute (green)
- Queue depth: 2 items (green)
- Last check: 30 seconds ago

Specific. Quantified. Real-time. You’re no longer flying blind.

When something breaks, you see it immediately. You have data to explain what went wrong. You can correlate the outage with what changed.

This is observability.

Practical Troubleshooting

“My metrics aren’t appearing”

Check:

# Can Prometheus reach your service?
curl http://localhost:8080/metrics

# Is the job in Prometheus UI?
# Targets tab -> check if "up" is 1 for your job

If metrics exist but aren’t showing, you’re likely looking at a scrape timing issue.

“Alerts aren’t firing”

# Check prometheus.yml syntax
./prometheus --config.file=prometheus.yml --dry-run

# Look at Alerts tab in Prometheus UI
# Is the expression evaluating?
# Is the condition actually true?

Test your alert expression manually in the Graph tab first.

“Alertmanager not receiving alerts”

# Check alertmanager.yml syntax
./amtool config routes

# Look at Alertmanager UI at http://localhost:9093
# Check Alertmanager logs

Verify your Prometheus configuration points to Alertmanager.

Your First Week: A Learning Path

Day 1: Install Prometheus, run the demo service, view metrics at /metrics

Day 2: Configure Prometheus, query basic metrics in UI (request count, latency)

Day 3: Learn PromQL: error rates, percentiles, aggregations

Day 4: Create dashboards visualizing your most important metrics

Day 5: Write alert rules for critical scenarios (errors, latency, down)

Day 6: Install Alertmanager, route alerts to email/Slack/PagerDuty

Day 7: Apply to your actual application, iterate on metrics and alerts

By week two, you’ll have visibility into your production systems you’ve never had before.

The Philosophy: Monitoring as Code

Here’s what changed: you stopped monitoring after deployment. You started building monitoring into your system from day one.

Every line of code that matters gets a metric. Every critical scenario gets an alert. Your monitoring grows with your application.

Version control your alert rules. Code review them. Deploy them. This is monitoring as code.

Alerting: Know When Things Break

Metrics are great, but monitoring a dashboard 24/7 isn’t realistic. You need alerts that notify you when things go wrong.

Prometheus uses an “Alertmanager” for routing alerts, but let’s start with the alert rules themselves.

Create alerts.yml:

groups:
  - name: application_alerts
    interval: 30s
    rules:
      # Alert if error rate > 5% for 5 minutes
      - alert: HighErrorRate
        expr: |
          (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }} for {{ $labels.job }}"
      
      # Alert if p95 response time > 1 second
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency detected"
          description: "P95 latency is {{ $value | humanizeDuration }}"
      # Alert if database errors > 5 per minute
      - alert: DatabaseErrors
        expr: |
          rate(database_errors_total[1m]) > 0.05
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Database errors detected"
          description: "{{ $value | humanize }} database errors per second"
      # Alert if queue depth > 100
      - alert: QueueBacklog
        expr: |
          queue_depth > 100
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Queue backlog detected"
          description: "Queue depth is {{ $value }}"
      # Alert if service is down (no metrics for 2 minutes)
      - alert: ServiceDown
        expr: |
          up{job="user-service"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Service is down"
          description: "user-service has been unreachable for 2 minutes"

Add this to your prometheus.yml:

rule_files:
  - 'alerts.yml'

Restart Prometheus. Visit the Alerts tab. You’ll see your alert rules and their current state (firing, pending, or inactive).

Understanding Alert Rules

Each alert has:

• expr — PromQL expression that evaluates to true when alert should fire
• for — How long condition must be true before alerting (prevents flapping)
• labels — Metadata about the alert (severity, team, etc.)
• annotations — Human-readable description and summary

The for: 5m prevents false alerts from brief spikes. The alert only fires if the condition persists for 5 minutes.

Alertmanager: Where Alerts Go

Prometheus evaluates alert rules, but where do the alerts go? Alertmanager routes them.

Install Alertmanager

wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xvfz alertmanager-0.26.0.linux-amd64.tar.gz
cd alertmanager-0.26.0.linux-amd64

Create alertmanager.yml:

global:
  # Email configuration
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_auth_username: 'your-email@gmail.com'
  smtp_auth_password: 'your-app-password'
  smtp_require_tls: true

# Templates for alert messages
templates:
  - '/etc/alertmanager/templates/*.tmpl'
# Routes define how alerts are processed
route:
  # Default receiver for all alerts
  receiver: 'team-email'
  
  # Group alerts by these labels
  group_by: ['alertname', 'cluster', 'service']
  
  # Wait this long to group alerts before sending
  group_wait: 30s
  
  # Wait this long after a new alert joins group
  group_interval: 5m
  
  # How long to wait before repeating notification
  repeat_interval: 4h
  
  # Sub-routes: specific handling for certain alerts
  routes:
    # Critical alerts go to ops team immediately
    - match:
        severity: critical
      receiver: 'ops-pagerduty'
      group_wait: 10s
      repeat_interval: 1h
    
    # Warnings can be batched
    - match:
        severity: warning
      receiver: 'team-email'
      group_wait: 2m
# Receivers define where alerts are sent
receivers:
  - name: 'team-email'
    email_configs:
      - to: 'team@company.com'
  - name: 'ops-pagerduty'
    pagerduty_configs:
      - service_key: 'your-pagerduty-key'

Run Alertmanager:

./alertmanager --config.file=alertmanager.yml

Now when Prometheus fires an alert, it sends to Alertmanager, which routes it according to your rules.

Alert Routing Logic

Alertmanager’s power is in routing:

• Group similar alerts so you get one notification instead of 100
• Send critical alerts immediately to PagerDuty
• Send warnings via email (batched)
• Prevent alert spam with repeat intervals
• Match alerts based on labels and route accordingly

This is how professional teams handle alerts. You’re not overwhelmed with notifications. Critical issues wake you up. Warnings are batched.

Production Best Practices

1. Use Recording Rules

Instead of calculating the same queries repeatedly, pre-calculate them:

groups:
  - name: recording_rules
    interval: 1m
    rules:
      # Record error rate every minute
      - record: http:error_rate:1m
        expr: sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m]))
      # Record p95 latency every minute
      - record: http:p95_latency:1m
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[1m]))

Then reference these in alerts:

- alert: HighErrorRate
  expr: http:error_rate:1m > 0.05

Recording rules make queries faster and alerts more efficient.

2. Retention Strategy

Prometheus stores metrics in memory. Large environments need data retention limits:

global:
  scrape_interval: 15s
  # Keep data for 30 days
  # Adjust based on storage available

Run Prometheus with:

./prometheus --storage.tsdb.retention.time=30d

3. Storage Considerations

Prometheus stores roughly 1–2 bytes per sample. With 1000 metrics scraping every 15 seconds:

• Per day: ~172 MB
• Per month: ~5 GB
• Per year: ~60 GB

For production, consider:

• Running on fast SSD
• Using Prometheus remote storage for long-term retention
• Archiving old data regularly

4. High Availability

Run two independent Prometheus servers scraping the same targets. If one fails, you still have data. Tools like Thanos can deduplicate metrics from both.

5. Naming Conventions

Follow these conventions so metrics are discoverable:

<namespace>_<subsystem>_<name>_<unit>

http_request_duration_seconds
database_connections_active
memory_usage_bytes
api_calls_total

• No underscores between words
• Include units
• Use suffixes: _total, _count, _sum, _bucket, _seconds

6. Documentation

Every metric needs documentation:

http_requests_total = promauto.NewCounterVec(
	prometheus.CounterOpts{
		Name: "http_requests_total",
		Help: "Total number of HTTP requests processed",
	},
	[]string{"method", "endpoint", "status"},
)

The Help field is critical. A year later, someone needs to understand what this metric means.

The Observability Stack

Prometheus is part of a larger observability picture:

┌─────────────────────────────────────────┐
│        Observability Stack              │
├─────────────────────────────────────────┤
│ Metrics (Prometheus)   - What happened? │
│ Logs (ELK/Loki)        - Why it happened│
│ Traces (Jaeger)        - How it happened│
└─────────────────────────────────────────┘

Prometheus gives you what happened: error rates, latencies, resource usage. Logs tell you the story. Traces show the path a request took.

Together, they’re unstoppable for debugging.

From Blindness to Visibility

Remember the question “Is the system healthy?”

With Prometheus, your answer is now:

- Error rate: 0.1% (green)
- P95 latency: 45ms (green)
- Database errors: 0 per minute (green)
- Queue depth: 2 items (green)
- Last check: 30 seconds ago

Specific. Quantified. Real-time. You’re no longer flying blind.

When something breaks, you see it immediately. You have data to explain what went wrong. You can correlate the outage with what changed.

This is observability.

Practical Troubleshooting

“My metrics aren’t appearing”

Check:

# Can Prometheus reach your service?
curl http://localhost:8080/metrics
# Is the job in Prometheus UI?
# Targets tab -> check if "up" is 1 for your job

If metrics exist but aren’t showing, you’re likely looking at a scrape timing issue.

“Alerts aren’t firing”

# Check prometheus.yml syntax
./prometheus --config.file=prometheus.yml --dry-run

# Look at Alerts tab in Prometheus UI
# Is the expression evaluating?
# Is the condition actually true?

Test your alert expression manually in the Graph tab first.

“Alertmanager not receiving alerts”

# Check alertmanager.yml syntax
./amtool config routes

# Look at Alertmanager UI at http://localhost:9093
# Check Alertmanager logs

Verify your Prometheus configuration points to Alertmanager.

A Learning Path to get up and running!

1: Install Prometheus, run the demo service, view metrics at /metrics

2: Configure Prometheus, query basic metrics in UI (request count, latency)

3: Learn PromQL: error rates, percentiles, aggregations

4: Create dashboards visualizing your most important metrics

5: Write alert rules for critical scenarios (errors, latency, down)

6: Install Alertmanager, route alerts to email/Slack/PagerDuty

7: Apply to your actual application, iterate on metrics and alerts

Are you ready to change the way to monitor the logs? Clap this post and share with friends!