Whitespace can change. Attributes can appear in a different order. Empty elements can be written differently. Encoding and namespace details can also affect the raw representation.

For humans, these differences may be irrelevant.

For a computer calculating a hash, comparing documents, or verifying a digital signature, they can be significant.

That’s where XML canonicalization, commonly called C14N, comes in.

Canonicalization is not just formatting

A useful way to think about it is:

A formatter makes XML easier for humans to read. A canonicalizer makes XML deterministic for machines.

The W3C standardized XML canonicalization specifically to provide a defined way of producing a canonical representation of XML.

This becomes particularly interesting in an ingestion pipeline.

Canonicalization as a pipeline component

Imagine an ingestion system receiving XML from different sources:

             XML input
                 │
                 ▼
        ┌─────────────────┐
        │ Parse / Validate│
        └────────┬────────┘
                 │
                 ▼
        ┌─────────────────┐
        │ Canonicalization│
        │      (C14N)     │
        └────────┬────────┘
                 │
                 ▼
        ┌─────────────────┐
        │ Hash / Compare / │
        │ Sign / Identify  │
        └────────┬────────┘
                 │
                 ▼
          Store / Process

The important idea is that canonicalization creates a deterministic representation before operations that depend on the representation itself.

For example, an ingestion system might use the canonical XML to calculate a fingerprint:

XML
 ↓
C14N
 ↓
SHA-256
 ↓
content fingerprint

This is particularly useful in ingestion systems for:

• Deduplication — recognize equivalent XML representations.
• Integrity verification — detect meaningful changes.
• Digital signatures — ensure formatting changes don’t invalidate the signed XML.
• Change detection — distinguish serialization differences from actual data changes.
• Reproducible processing — normalize input before downstream operations.

Let’s see it in Python

You can run the following example directly in Google Colab.

Python’s xml.etree.ElementTree provides XML canonicalization through canonicalize() in modern Python versions.

1. Create two XML documents

Let’s start with two XML documents that contain the same attributes, but serialize those attributes differently:

import xml.etree.ElementTree as ET

xml_a = '<product name="Widget" id="123"></product>'
xml_b = '<product id="123" name="Widget"></product>'

The XML represents the same element and attributes, but the attribute order is different.

At the byte level, they are clearly different:

print(xml_a.encode() == xml_b.encode())

Output:

False

Now let’s calculate a hash directly from the original XML strings:

import hashlib

hash_a = hashlib.sha256(xml_a.encode()).hexdigest()
hash_b = hashlib.sha256(xml_b.encode()).hexdigest()

print(hash_a)
print(hash_b)

The hashes are different:

f312802c117a6b84f8e6b65c9b8a5adf8148b00ab5985b5ca5938e1a985a8f99
d01eb593bfc5662ab9ab89744e4a6fcafe08f7cc31f40d41eff6a3b3a7965e30

Look closely ☝️ — these are two different fingerprints.

That’s expected.

The computer is hashing the serialized bytes, not trying to understand that both XML documents represent the same structure.

This is where canonicalization can help.

Instead of hashing the original XML directly, we first canonicalize both documents:

canonical_a = ET.canonicalize(xml_data=xml_a)
canonical_b = ET.canonicalize(xml_data=xml_b)

print(canonical_a)
print(canonical_b)

print(canonical_a == canonical_b)

Output:

True

Now both XML documents have the same canonical representation.

We can then hash the canonical representation, rather than the original serialization:

canonical_hash_a = hashlib.sha256(
    canonical_a.encode("utf-8")
).hexdigest()

canonical_hash_b = hashlib.sha256(
    canonical_b.encode("utf-8")
).hexdigest()

print(canonical_hash_a)
print(canonical_hash_b)

print(canonical_hash_a == canonical_hash_b)

Output:

True

The important distinction is:

Original XML A → → SHA-256 → → Fingerprint A
Original XML B → → SHA-256 → → Fingerprint B

                        ↓

                 Different hashes

But when canonicalization is appropriate:

XML A → → C14N → → Canonical A → → SHA-256 → → Fingerprint
XML B → → C14N → →C anonical B → → SHA-256 → → Same fingerprint

So the core lesson is:

Hashing the original XML can produce different fingerprints for different serializations. Canonicalizing first can produce the same fingerprint when the differences are ones that C14N is designed to normalize.

And one important caveat:

C14N is not a general-purpose “make equivalent XML identical” operation. It follows precisely defined normalization rules and preserves information that may be meaningful, including text-node content. For example, whitespace inside an XML element can affect the canonical result.

2. Canonicalize the XML

Now let’s introduce the pipeline component:

canonical_a = ET.canonicalize(xml_data=xml_a)
canonical_b = ET.canonicalize(xml_data=xml_b)

print(canonical_a)
print(canonical_b)

The canonicalizer produces a deterministic representation.

Now compare them:

print(canonical_a == canonical_b)

The result is:

True

That’s the important part.

Although the original XML strings were serialized differently, canonicalization gives us an equivalent canonical representation.

3. Hash the canonical representation

Now we can use that canonical representation as the input to our fingerprinting step:

canonical_hash_a = hashlib.sha256(
    canonical_a.encode("utf-8")
).hexdigest()

canonical_hash_b = hashlib.sha256(
    canonical_b.encode("utf-8")
).hexdigest()

print(canonical_hash_a)
print(canonical_hash_b)

print(canonical_hash_a == canonical_hash_b)

The final result is:

d01eb593bfc5662ab9ab89744e4a6fcafe08f7cc31f40d41eff6a3b3a7965e30
d01eb593bfc5662ab9ab89744e4a6fcafe08f7cc31f40d41eff6a3b3a7965e30
True

Our pipeline is now:

Different XML serializations
          │
          ▼
      Canonicalize
          │
          ▼
 Same canonical representation
          │
          ▼
       SHA-256
          │
          ▼
    Same fingerprint

This is a much better foundation for applications that need to reason about XML equivalence rather than XML formatting.

Why would an ingestion system care?

Imagine an ingestion service receiving thousands of XML documents.

Without normalization, the system might encounter:

Document A → hash X
Document B → hash Y

even though both documents represent the same XML structure.

Depending on the application’s requirements, that can complicate:

• deduplication;
• integrity checks;
• change detection;
• digital signatures;
• reproducible processing.

Canonicalization gives the pipeline a standardized point at which the representation becomes deterministic.

However, it is important to understand what C14N doesn’t do.

It does not magically determine that two XML documents are semantically equivalent in every possible sense.

It applies a defined set of canonicalization rules.

For example, two documents could contain different values:

<price>100</price>

versus:

<price>101</price>

Canonicalization will not make those equal — and it shouldn’t.

Canonicalization is about normalizing representation, not changing meaning.

The mental model

A simple way to remember it:

Pretty printing
     ↓
"Make it nice for humans"



Canonicalization
     ↓
"Make its representation deterministic for machines"

And in an ingestion architecture:

             INPUT
               │
               ▼
        Parse / Validate
               │
               ▼
        ┌──────────────┐
        │     C14N     │  ← deterministic representation
        └──────┬───────┘
               │
          ┌────┴─────┐
          ▼          ▼
        Hash       Sign
          │          │
          └────┬─────┘
               ▼
            Storage

That makes canonicalization less mysterious. ❤️

It isn’t simply an XML formatter.

It is a normalization component that can sit between XML ingestion and operations that depend on a stable representation.

Note ¹

Why C14N?

Because C14N is a numeronym.

It comes from the word:

Canonicalization

Take the first and last letters:

• C — first letter
• 14 — number of letters between the first and last letters
• N — last letter

So:

Canonicalization → C + 14 + N → C14N

It’s the same naming pattern as:

• i18n → internationalization
• l10n → localization
• a11y → accessibility

In XML security, you’ll often see C14N used as shorthand for XML Canonicalization, especially when discussing the W3C canonicalization algorithms.

A nice little detail for your tutorial would be:

Why C14N? It’s a numeronym: Canonicalization has 14 letters between its first and last letters, giving us C14N.

See also: https://medium.com/jungletronics/using-gemini-chat-on-collab-2626fb035176

Disclosure: This tutorial is an educational exploration of XML canonicalization and ingestion pipelines. The examples are intentionally simplified and designed for experimentation in Google Colab; they should not be treated as production-ready security or data-processing guidance without appropriate validation for the target environment.

XML Canonicalization: The Normalization Step Your Ingestion Pipeline May Need was originally published in Jungletronics on Medium, where people are continuing the conversation by highlighting and responding to this story.

The bug that motivates this tutorial

A Python pipeline ran fine on a laptop. Inside Docker, the same code failed to reach a database:

No route to host

The credentials were right, the VPN was up, the database was online. The cause turned out to be two numbers:

• the database lived at 172.22.1.10 on the company network;
• Docker had created the project network as 172.22.0.0/16.

If you can explain why those two lines clash, you understand IP masks. By the end of this tutorial, you will.

1. An IP address is just 32 bits

An IPv4 address such as 172.22.1.10 is a 32-bit number, written as four octets (8 bits each) in decimal:

172      .22       .1        .10
10101100  00010110  00000001  00001010

Every octet goes from 0 to 255, because that’s what 8 bits can hold.

2. Network part and host part

An address has two parts, like a street address:

• the network part: which street;
• the host part: which house on that street.

The mask says where the street ends and the house number begins.

3. The mask, two ways of writing it

The mask is also 32 bits: a run of 1s (network part) followed by 0s (host part).

/16  →  11111111.11111111.00000000.00000000  →  255.255.0.0
/24  →  11111111.11111111.11111111.00000000  →  255.255.255.0

• Dotted notation: 255.255.0.0
• CIDR notation: /16, meaning "the first 16 bits are the network"

They mean the same thing. CIDR (Classless Inter-Domain Routing) is shorter and is what Docker, cloud providers and most tools use.

4. Reading 172.22.0.0/16

/16 fixes the first 16 bits, which is exactly the first two octets:

172.22 . x.x
└─────┘  └──┘
network  host (free)

So the network 172.22.0.0/16 covers every address from 172.22.0.0 to 172.22.255.255.

The first address names the network itself and the last one is for broadcast, so neither goes to a machine.

Quick rule: a network with prefix /n has 2^(32 − n) addresses.

A smaller prefix means a bigger network. Every step down doubles the size: /23 is twice a /24.

5. Private address ranges

Three ranges are reserved for private networks (RFC 1918). They never appear on the public internet, so any company, home or Docker install can reuse them:

Why does 172.16.0.0/12 stop at 172.31? With /12, the network part is the first octet plus the first 4 bits of the second octet:

16 = 0001 0000
31 = 0001 1111
     └──┘ fixed    └──┘ free → 16 values: 16, 17, ..., 31

This is the root of the bug. Everyone reuses the same private ranges. Your company may use 172.22.x.x for servers, and Docker, on the same machine, may pick 172.22.x.x for its own network. Nothing stops them from colliding.

6. How a machine decides where to send a packet

Every machine, and every container, has a routing table. Here is roughly what a container on the network 172.22.0.0/16 has:

$ ip route
default via 172.22.0.1 dev eth0          ← anything else: send to the gateway
172.22.0.0/16 dev eth0 scope link        ← this network: deliver directly

To send a packet, the machine picks the most specific entry that contains the destination. This is the longest prefix match: /16 beats default, which is /0.

Now send a packet to the database at 172.22.1.10:

1. Does 172.22.1.10 fall inside 172.22.0.0/16? Yes.
2. So the container treats it as a neighbor and tries to deliver it directly on its local network, without the gateway.
3. It asks “who has 172.22.1.10?” on the Docker network (an ARP request). The real database is on the company network, outside Docker, so nobody answers.
4. After a few tries, the kernel gives up with EHOSTUNREACH: No route to host.

Overlapping (broken):
container 172.22.0.2 ── "172.22.1.10 is local" ──► asks the Docker network ──► silence ✘

Not overlapping (works):
container 192.168.250.2 ── "not local" ──► gateway ──► host ──► VPN ──► database ✔

This also explains why the code worked outside Docker: the laptop’s own routing table had no 172.22.0.0/16 entry, so the packet went to the VPN as expected.

7. The fix

Pin the Docker network to a range that can’t overlap your company’s:

# docker-compose.yml
networks:
  default:
    ipam:
      config:
        - subnet: 192.168.250.0/24

192.168.250.0/24 gives 254 usable addresses, far more than a few containers need, and sits well away from 172.16.0.0/12.

To stop Docker from picking risky ranges for all new networks, set its default pools in /etc/docker/daemon.json (or Docker Desktop → Settings → Docker Engine), then restart Docker:

{
  "default-address-pools": [
    { "base": "10.200.0.0/16", "size": 24 }
  ]
}

Check with your network team first that the range you choose isn’t used anywhere in your company or VPN.

8. Check for overlaps yourself

What subnets is Docker using right now?

docker network ls -q | xargs docker network inspect \
  --format '{{.Name}}: {{range .IPAM.Config}}{{.Subnet}}{{end}}'

Does an address fall inside a network? Python answers without any binary math:

import ipaddress

docker_net = ipaddress.ip_network("172.22.0.0/16")
database   = ipaddress.ip_address("172.22.1.10")
print(database in docker_net)                                            # True  → trouble
print(docker_net.num_addresses)                                          # 65536
print(docker_net.overlaps(ipaddress.ip_network("172.22.1.0/24")))        # True
print(ipaddress.ip_network("192.168.250.0/24").overlaps(docker_net))     # False → safe

A three-line habit before choosing a subnet: list the ranges your company and VPN use, and test every one with .overlaps().

Practice

1. How many addresses are in 10.0.0.0/8?
2. What are the first and last addresses of 192.168.250.0/24?
3. Is 172.30.5.9 inside 172.16.0.0/12?
4. Docker created a network 172.20.0.0/16. Can a container reach a server at 172.22.1.10?
5. What is the mask of /20 in dotted notation?

Answers

1. 2^(32−8) = 2²⁴ = 16,777,216.
2. First (network) 192.168.250.0, last (broadcast) 192.168.250.255. Hosts use .1 to .254.
3. Yes. /12 covers 172.16 to 172.31, and 30 is in that range.
4. Yes. 172.20.0.0/16 only covers 172.20.x.x, so 172.22.1.10 is not local and goes to the gateway.
5. 20 ones: 11111111.11111111.11110000.00000000 = 255.255.240.0.

Key takeaways

• An IP address is 32 bits. The mask splits it into a network part and a host part.
• /n means the first n bits are the network. The network holds 2^(32 − n) addresses.
• 172.22.0.0/16 means everything from 172.22.0.0 to 172.22.255.255.
• Private ranges are shared by everyone, so overlaps between Docker, VPNs and corporate networks are common.
• Routing picks the most specific match. If a destination falls inside a local network, the packet never reaches the gateway.
• “Works outside Docker, fails inside” is a networking clue. Check for subnet overlap first.

Disclosure: This article is an educational and/or technical exploration based on concepts, experiments, and practical experience. Code samples, configurations, architectures, and examples may be simplified, adapted, or illustrative. No institutional source code, real production data, credentials, confidential information, or sensitive internal infrastructure details are reproduced.

What Does 172.22.0.0/16 Actually Mean? IP Masks Explained with a Real Docker Bug was originally published in Jungletronics on Medium, where people are continuing the conversation by highlighting and responding to this story.

The challenge

In Brazil, every sale between businesses and consumers is backed by an electronic invoice (NF-e / NFC-e): a signed XML document with a fixed, public schema. For each line item, the invoice carries two fields that should say the same thing:

• the commercial description, typed freely by the seller: CERV PILS LT 350ML (pilsner beer, 350 ml can), REFRI COLA PET 2L (2 L cola soda), PARAF SEXT ZB 1/4X2 (zinc-plated hex bolt), KIT PRESENTE C/3 (gift kit, 3 items);
• the tax classification code (NCM, the Mercosur Common Nomenclature, built on top of the international Harmonized System): 8 hierarchical digits that determine how the product is taxed.

They don’t always agree. Descriptions are abbreviated, ambiguous and vary from seller to seller. The code is sometimes just wrong, by accident or not. Checking this by hand across millions of items a month is impossible.

The project question was:

Can a small language model (SLM), running in a controlled environment, read the description, suggest the correct code with a confidence level and flag when the declared code looks inconsistent?

“Small” is a requirement, not a detail. For sensitive tax data, the model should run inside the controlled environment where the data resides, the cost per inference must be low, and answers must come in batch, over millions of rows.

This tutorial covers the whole project, in two halves:

1. Data engineering: turning a raw extraction into a standardized, pseudonymized, validated and verifiable dataset (Parquet + ZSTD).
2. Modeling: from that dataset to an SLM that classifies, gives a confidence and flags inconsistencies.

The first half has been implemented and validated. The second half describes the design of a possible next phase.

Architecture overview

Part 1: Data engineering

1. Contract first, code second

Before writing a single line of transformation, the output file was described as a contract: the name, type and requirement level of every field. Each field also records where it came from in the source, which gives end-to-end lineage.

from dataclasses import dataclass
import polars as pl

ERROR, WARNING, FREE = "ERROR", "WARNING", "FREE"
@dataclass(frozen=True)
class Field:
    number: int
    name: str
    dtype: pl.DataType
    required: bool
    null_severity: str   # what happens if the value is null
    source: str          # source expression (lineage)
FIELDS = (
    Field(1,  "invoice_hash",  pl.String,         True, ERROR,   "HMAC(invoice access key)"),
    Field(7,  "xProd",         pl.String,         True, ERROR,   "item description"),
    Field(11, "qCom",          pl.Decimal(15, 4), True, ERROR,   "commercial quantity"),
    Field(15, "declared_ncm",  pl.String,         True, WARNING, "NCM stated on the invoice"),
    # ...
)

(xProd and qCom are the official field names in the invoice XML schema.)

The severity matrix separates three situations:

It looks like bureaucracy. In practice, this contract ended endless discussions: any “can this field be empty?” question has a written, versioned, tested answer.

Three sources, three roles:

One lesson: the extraction SQL is not the truth about types. When checking the dictionary, a year-month field that the script produced as 'YYYY/MM' was stored as 'YYYYMM', and a numeric field arrived as text. Trust the database dictionary; use the SQL to understand lineage.

2. Pseudonymization: the HMAC that protected nothing

Identifiers (invoice key, issuer tax ID) cannot go into the dataset. The standard solution is an HMAC with a secret key: the same identifier always produces the same hash, so you can still group by it, but without the key nobody can get back to the original.

-- Oracle: STANDARD_HASH takes no key; HMAC comes from DBMS_CRYPTO
RAWTOHEX(DBMS_CRYPTO.MAC(UTL_RAW.CAST_TO_RAW(access_key),
                         DBMS_CRYPTO.HMAC_SH256, :secret_key))

During a review of the extraction script, this showed up:

DBMS_CRYPTO.MAC(..., <weak_placeholder_key>)

A deliberately weak key would have only 256 possible values. Anyone who knows a single real identifier (a publicly printed invoice key, for instance) tries all 256 in milliseconds, finds the key and re-identifies the entire dataset:

import hmac, hashlib

def recover_key(known_identifier: str, published_hash: str) -> bytes | None:
    for k in range(256):
        key = bytes([k])
        h = hmac.new(key, known_identifier.encode(), hashlib.sha256).hexdigest().upper()
        if h == published_hash:
            return key
    return None

In this case, the real key had been stripped from the shared script, and the value was just a placeholder. The rule still stands: an HMAC key must be at least 32 random bytes, live in a vault, and never appear in code, scripts or logs. A hash with a weak key is pseudonymization in name only.

3. Exact conversion: silent rounding

This was the most subtle trap in the project. Monetary values arrive from Oracle as NUMBER. The obvious approach is to read them with an explicit schema:

# DON'T do this with money
df = pl.DataFrame(rows, schema={"Prod_value": pl.Decimal(15, 2), ...}, orient="row")

With an explicit schema, Polars rounds decimals and nulls out incompatible values without warning. 12.345 becomes 12.35, and no error shows up. In a tax dataset, that is unacceptable.

The solution has three parts.

a) Read without imposing types. With fetch_decimals=True, the driver returns NUMBER as Python Decimal, and Polars infers the type from the values themselves:

import oracledb

oracledb.defaults.fetch_decimals = True

with connection.cursor() as cur:
    cur.arraysize = 10_000
    cur.execute(sql)
    names = [d.name for d in cur.description]
    batches = []
    while rows := cur.fetchmany():
        batches.append(pl.DataFrame(rows, schema=names, orient="row", infer_schema_length=None))
# batches may infer different scales: supertyping only widens, never rounds
df = pl.concat(batches, how="vertical_relaxed")

b) Convert, then prove the conversion was exact. Cast to the contract type, cast back to the source type and compare. If anything changed, the value does not go into the file:

def exact_cast(name, source, target):
    original = pl.col(name)
    converted = original.cast(target, strict=False)
    failed = original.is_not_null() & (
        converted.is_null() | (converted.cast(source) != original)
    )
    exact = pl.when(failed).then(None).otherwise(converted).cast(target)
    return exact.alias(name), failed.alias(f"conversion:{name}")

The final file holds either the exact source value or null, never an approximation. Every null produced by a conversion is flagged and becomes an ERROR in the report.

c) Distrust regex. To validate numeric text, \d looks like the natural choice. But in Polars' regex engine (Rust), \d matches Unicode digits from any script, including Arabic-Indic ٣٤٥. Use [0-9]:

YEAR_MONTH = r"^[0-9]{4}(0[1-9]|1[0-2])$"
INTEGER    = r"^(0|[1-9][0-9]*)$"   # no sign, spaces or leading zeros

For the same reason, code fields (NCM, tax operation codes, barcodes) are always text. If one arrives as a number, the run stops: 01012100 cast to an integer loses its leading zero and becomes a different code.

4. Validate, never fix

The validator applies about 36 rules: nulls by severity, domains ({0, 1}), formats (8-digit NCM, 64-character uppercase hex hashes), consistency (the document type matches the file) and duplicates.

The golden rule: identify and count, never fix or drop. A dataset “cleaned” by hidden heuristics is worse than a dirty one that is documented. The results go into a report:

{
  "status": "OK",
  "records": 1000000,
  "records_with_error": 0,
  "records_with_warning": 24107,
  "findings": [
    {"field": "tax_status_code", "rule": "null", "severity": "WARNING", "count": 4938}
  ],
  "duplicate_item_hash": 0
}

(Illustrative numbers.)

Duplicates deserve attention. A repeated invoice_hash is legitimate: one invoice has many items. A repeated item_hash (invoice + item number), however, means a JOIN in the extraction multiplied rows. That is a classic, silent bug.

The process exit code also carries meaning: 0 is OK, 1 means some records have ERRORs (the file was written), and 2 means a structural failure (no file was written). An orchestrator knows what to do without parsing logs.

5. Parquet with ZSTD 18, atomic writes and proof

def write(df: pl.DataFrame, path: Path) -> None:
    tmp = path.with_suffix(path.suffix + ".tmp")
    df.write_parquet(tmp, compression="zstd", compression_level=18, statistics=True)
    os.replace(tmp, path)   # atomic: either the old file or the complete new one

After writing, the pipeline reopens the file and proves four things:

1. the schema matches the contract;
2. the content read back is identical to what was written (read_back.equals(expected));
3. every column in every row group uses ZSTD;
4. validating the file read back gives exactly the same result as validating before the write.

A little-known detail: the compression level is not stored in Parquet metadata, only the codec. There is no way to prove, by reading the file, that it was compressed at level 18. The report says so honestly: “codec verified: ZSTD; level configured: 18”.

Why level 18? ZSTD 18 compresses much more slowly than the default level (3), but decompresses at the same speed. For a file written once and read many times, that is a good trade.

6. Estimating the size before having the data

Before touching real data, we had to say how much space the files would take. The estimate started from synthetic data, column by column:

import pyarrow.parquet as pq

meta = pq.ParquetFile("sample.parquet").metadata
rg = meta.row_group(0)
for i in range(rg.num_columns):
    c = rg.column(i)
    print(f"{c.path_in_schema:28s} {c.total_compressed_size / meta.num_rows:6.1f} B/row")

item_hash                      33.0 B/row   ← HMAC: random, does not compress
invoice_hash                    8.2 B/row   ← repeats within an invoice
xProd                           2.0 B/row   ← synthetic: far too repetitive
...
total                          78   B/row

Three takeaways:

• Hashes set the floor. A hex-encoded HMAC-SHA256 is pure entropy: 64 characters compress to about 32 bytes. The hash columns alone accounted for more than half of the file.
• Synthetic data¹ lies about entropy. I revised the estimate upward (95 to 120 B/row), assuming real descriptions would be more varied than synthetic ones.
• Real data surprised in the other direction. The real file came in at about 68 B/row, below even the synthetic one. Real commercial descriptions are far more repetitive than they look (the same product sold thousands of times with the same text), and ZSTD 18 takes full advantage of that.

Moral: estimate from the structure (the hash floor was right), and measure on a real sample as soon as possible. With 5% of the data, the extrapolation error is small.

7. The bug that wasn’t in the code: “No route to host”

The pipeline runs in Docker. On the first run against real data, the database connection failed. Credentials, VPN and database were all fine, and outside the container the connection worked.

The cause: the database was reachable through a private network’s 172.x.x.x range. Docker automatically picks subnets for its internal networks between 172.17 and 172.31. When the project network landed on a subnet that covered the database address, the container started treating the database as a "neighbor" on the internal network. Packets never left for the VPN.

container 172.x.0.2 ──► "the database is a neighbor, deliver locally" ──► nobody answers ✘

What made it hard to diagnose:

• the failure was intermittent, because the subnet depends on how many networks Docker has already created, including for other projects;
• the message (“No route to host”) points at the VPN or a firewall.

The fix is a fixed subnet, far from the corporate network:

networks:
  default:
    ipam:
      config:
        - subnet: 192.x.x.x/xx

192.168.x.x is not public because the entire 192.168.0.0/16 range is reserved for private networks (RFC 1918). These addresses are not directly routable on the public Internet.

So a subnet such as:

192.168.x.x/xx

represents a private/internal address range, not a public IP network.

For example, a Docker network may be configured with a private IPv4 subnet from the RFC 1918 address space. The exact subnet is intentionally omitted.

A better diagnosis came with it. For security, the log does not print the driver message (which contains host and service), but it now walks the exception chain down to the OSError and shows its type and errno, with a hint:

def network_cause(error: BaseException) -> str:
    cause = error.__cause__ or error.__context__
    while cause is not None and not isinstance(cause, OSError):
        cause = cause.__cause__ or cause.__context__
    if cause is None:
        return ""
    hint = {errno.EHOSTUNREACH: "no route: VPN or Docker subnet overlap",
            errno.ENETUNREACH: "network unreachable"}.get(cause.errno, "")
    return f" | cause: {type(cause).__name__} errno={cause.errno} {hint}"

A heuristic worth keeping: works outside Docker, fails inside points to Docker networking, not to your code.

Part 2: From Parquet to SLM

With a trustworthy dataset, the part that gets the most attention begins, and this is where the engineering in Part 1 pays off.

8. The label is noisy, and that is the core problem

The temptation is to train on xProd → declared_ncm. But the declared NCM is exactly what we want to audit: some of it is wrong. Training a model to imitate the declared label teaches it to repeat the mistakes.

The strategy:

1. Gold set: items whose code was reviewed by experts. The dataset has reviewed_ncm and reviewed_ncm_confidence fields for exactly this. It is small but reliable, and is used for evaluation and final fine-tuning.
2. Silver set: declared labels filtered by consensus. If the same normalized description appears with the same code across many different issuers, the label is probably right.
3. Drop from training descriptions with conflicting labels. They go to analysis, not to learning.

norm = (pl.col("xProd").str.to_uppercase()
        .str.replace_all(r"[^A-Z0-9 ]", " ")
        .str.replace_all(r"\s+", " ").str.strip_chars())

silver = (
    df.with_columns(desc=norm)
      .group_by("desc", "declared_ncm")
      .agg(issuers=pl.col("issuer_hash").n_unique(), n=pl.len())
      .with_columns(share=pl.col("n") / pl.col("n").sum().over("desc"))
      .filter((pl.col("issuers") >= 5) & (pl.col("share") >= 0.9))
)

9. Split by issuer, or the model memorizes

Every issuer has its own “dialect” of descriptions. If the same issuer appears in both training and test, the model memorizes its catalog, and the metric looks great and lies. The split is done by group, using the issuer hash. This is where pseudonymization shows its value: you can group without knowing who the issuer is.

from sklearn.model_selection import GroupShuffleSplit

gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(X, y, groups=issuers))

Deduplicate before splitting as well: identical descriptions must not land on both sides.

10. Start with an embarrassingly strong baseline

Before any LLM, a classic baseline. With character n-grams, it handles abbreviations (CERV, REFRI, LT) well and is often surprisingly strong:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import SGDClassifier
from sklearn.pipeline import make_pipeline

baseline = make_pipeline(
    TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5), min_df=3, sublinear_tf=True),
    SGDClassifier(loss="log_loss", alpha=1e-6, max_iter=20),
)
baseline.fit(train_desc, train_ncm)

The SLM has to beat this baseline to justify its cost. If it doesn’t, the honest answer is to ship the baseline.

11. Evaluate along the hierarchy

NCM is hierarchical: chapter (2 digits) → heading (4) → subheading (6) → item (8). Mistaking 22030000 (beer) for 22029900 (another beverage) is far less serious than mistaking it for a code in the machinery chapter. Evaluation measures accuracy at every level:

def hierarchical_accuracy(y_true, y_pred):
    return {level: sum(t[:n] == p[:n] for t, p in zip(y_true, y_pred)) / len(y_true)
            for level, n in [("chapter", 2), ("heading", 4), ("subheading", 6), ("item", 8)]}

Also report top-3 accuracy at item level and macro-F1 per chapter. Micro-accuracy hides the rare chapters, which tend to be exactly the ones with the highest tax risk.

12. SLM architecture: retrieve, then decide

There are roughly 10,000 NCM codes. Asking a small model to “generate” a code freely leads to codes that don’t exist. The architecture splits the work in two steps:

description ──► retriever (kNN on n-grams / embeddings) ──► top-10 candidates
                                                               │
                                        LoRA-tuned SLM ◄───────┘
                                               │
                              scores each candidate (log-prob)
                                               │
                           probability distribution over the 10

a) Candidate retrieval. Nearest neighbors among labeled descriptions, plus the official nomenclature descriptions:

from sklearn.neighbors import NearestNeighbors

vectorizer = TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5), min_df=2)
V = vectorizer.fit_transform(reference_desc)
knn = NearestNeighbors(n_neighbors=50, metric="cosine").fit(V)
def candidates(desc: str, k: int = 10) -> list[str]:
    _, idx = knn.kneighbors(vectorizer.transform([desc]))
    seen = []
    for i in idx[0]:
        if reference_ncm[i] not in seen:
            seen.append(reference_ncm[i])
        if len(seen) == k:
            break
    return seen

The metric that matters here is recall@10: if the right code is not among the candidates, the SLM has no way to get it right.

b) LoRA fine-tuning. An instruction model with 0.5 to 3 billion parameters, trained to choose among the candidates:

from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

def to_chat(ex):
    return {"messages": [
        {"role": "system", "content": "Classify the product into its NCM code. Answer with the 8 digits only."},
        {"role": "user", "content": f"Description: {ex['xProd']}\nUnit: {ex['uCom']}\n"
                                    f"Candidates: {', '.join(ex['candidates'])}"},
        {"role": "assistant", "content": ex["target_ncm"]},
    ]}
trainer = SFTTrainer(
    model="<1-to-3B-instruct-model>",
    train_dataset=train_dataset.map(to_chat),
    args=SFTConfig(output_dir="slm-ncm", num_train_epochs=2,
                   per_device_train_batch_size=16, learning_rate=2e-4),
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                           task_type="CAUSAL_LM"),
)
trainer.train()

One training detail that matters: shuffle the candidate order and make sure the right code is not always first in the list. Otherwise the model learns to copy a position instead of understanding the description.

c) Score instead of generate. Instead of letting the model generate free text, compute the log-probability of each candidate and apply a softmax across them. The answer is always a valid code, and it comes with a probability distribution:

import torch

@torch.no_grad()
def score(model, tok, messages, candidates):
    prompt = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
    logps = []
    for c in candidates:
        target = tok(c, add_special_tokens=False, return_tensors="pt").input_ids
        ids = torch.cat([prompt, target], dim=1)
        logits = model(ids).logits[0, prompt.shape[1] - 1 : -1]
        logps.append(torch.log_softmax(logits, -1).gather(1, target[0].unsqueeze(1)).sum())
    return dict(zip(candidates, torch.softmax(torch.stack(logps), 0).tolist()))

13. Confidence that means something

A model that says “95% confident” should be right about 95% of the times it says so. Neural networks tend to be overconfident. The fix is calibration with temperature scaling, fitted on a held-out set, and the check is ECE (expected calibration error):

import numpy as np

def ece(confidences, correct, n_bins=10):
    confidences, correct = np.asarray(confidences), np.asarray(correct)
    bins = np.linspace(0, 1, n_bins + 1)
    total = 0.0
    for lo, hi in zip(bins[:-1], bins[1:]):
        m = (confidences > lo) & (confidences <= hi)
        if m.any():
            total += m.mean() * abs(correct[m].mean() - confidences[m].mean())
    return total

The model output uses the same scale as the dataset: confidence 0, 1 or 2 (low, medium, high), with bands set from the calibration set rather than picked by eye.

14. Flag inconsistencies without crying wolf

The end product is not “the model thinks the code is X”. It is a triage queue for human auditors, ranked by evidence:

def inconsistency(probs: dict[str, float], declared: str, threshold: float = 0.85) -> dict | None:
    suggested, p_sug = max(probs.items(), key=lambda kv: kv[1])
    p_decl = probs.get(declared, 0.0)   # declared code not among candidates: strong evidence against it
    if suggested == declared or p_sug < threshold:
        return None
    return {
        "suggested": suggested,
        "confidence": p_sug,
        "p_declared": p_decl,
        "level": next(lvl for lvl, k in [("chapter", 2), ("heading", 4), ("subheading", 6), ("item", 8)]
                      if suggested[:k] != declared[:k]),
    }

High-confidence chapter-level inconsistencies go to the top of the queue. Item-level ones are often legitimate ambiguity in the nomenclature.

The metric that decides whether the system is useful is not accuracy. It is queue precision: out of every 100 flagged items, how many the auditor confirms as real inconsistencies. Measuring it takes a pilot with human review, and the threshold is tuned from it.

15. Going to production without the data leaving home

• Quantization (4 or 8 bits) to run on a modest GPU or even on CPU.
• Batch inference: Parquet in, Parquet out, with suggested_ncm, confidence and inconsistency columns. Same format, same contract, same validation as Part 1.
• Fully on-premise: the model goes to the data, not the other way around.
• Versioning: every run records the model, retriever and threshold versions, along with the SHA-256 of the input file.

Lessons from the whole project

1. Contract before code. Half the discussions disappear once the format is written, versioned and tested.
2. Never round silently. Convert, cast back and compare. If it changed, it’s an error, not an “adjustment”.
3. Validate, don’t fix. Count, classify and document. Hidden cleaning destroys trust in a dataset.
4. Prove what you wrote. Read the file back, compare and revalidate.
5. Pseudonymization is only as strong as its key. An HMAC is only as strong as its key.
6. Synthetic data tests the code, not reality. Measure on a real sample as early as possible.
7. “Works outside Docker” is a networking clue.
8. The declared label is the suspect, not the answer key. Train on reviewed or consensus labels.
9. Split by group, or the metric lies.
10. Baseline first; SLM only if it wins.
11. Confidence without calibration is just a number.
12. The product is the triage queue, not the model. Measure precision with humans in the loop.

Note ¹

The idea is that synthetic data often looks more repetitive than real-world data, so it can make compression and storage estimates appear artificially optimistic.

“Synthetic data lies about entropy”

Entropy here means, roughly, how varied and unpredictable the data is.

For example, synthetic descriptions might look like:

"PRODUCT 000001"
"PRODUCT 000002"
"PRODUCT 000003"

They have a lot of repetition and predictable structure, so compression algorithms can compress them very efficiently.

Real descriptions might look more like:

"Type 1 White Rice 5kg"
"Premium Parboiled Rice 5kg"
"T1 Long-Grain Rice 5kg"

There is considerably more variation. That generally means higher entropy and potentially worse compression.

So:

Synthetic data can lie about entropy because it may underestimate the variability of the real dataset.

That’s why revising the estimate from 95 B/row to 120 B/row is a conservative planning assumption: you’re allowing approximately 26% more storage per row rather than assuming that the compression observed on synthetic data will represent production data.

The important nuance is that 95 → 120 B/row is an estimate, not a measured compression ratio. Once real data is available, the actual Parquet/ZSTD output should replace the estimate.

Generally speaking:

Higher entropy → more variability/unpredictability → less efficient compression.

And conversely:

Lower entropy → more repetition/predictability → more efficient compression.

Disclosure: This article is an educational reconstruction of a data-engineering problem. Code samples are simplified and adapted, examples are illustrative, and no institutional source code, real fiscal data, credentials, internal infrastructure identifiers or confidential documentation are reproduced.

Code samples are simplified and names have been adapted. No real data is shown. Volume and compression figures are given in bytes per row so as not to reveal the actual size of the database.

From Invoice to Small Language Model: A Pipeline That Never Rounds was originally published in Jungletronics on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Have you ever opened a JavaScript file only to find your code tunneling sideways off the right side of the screen?

You aren’t alone.

For years, managing complex sequences of asynchronous operations often meant nesting one callback inside another, creating a structure that developers commonly refer to as Callback Hell or the Pyramid of Doom.

The problem isn’t that callbacks are inherently bad. Callbacks are a valid mechanism for responding to asynchronous events. The difficulty appears when multiple operations depend on the results of previous operations and each new step introduces another level of nesting.

JavaScript executes application code on a single main thread in the browser, but the browser itself can handle asynchronous operations such as network requests and timers outside the JavaScript execution stack.

If a long-running JavaScript computation runs synchronously on the main thread, it can block the user interface. Network requests, however, do not normally block the JavaScript thread while the browser waits for the response.

To build responsive web applications, we therefore need to understand how synchronous JavaScript execution interacts with asynchronous APIs, Promises, and the event loop.

In this tutorial, we’ll follow the evolution from nested callbacks to Promise chains and finally to async/await, using a simple CodePen experiment to observe the different programming styles.

Here are the clean, simplified steps to get your specific HTML layout and JavaScript code running perfectly inside CodePen for your tutorial’s practice section.

Step 1: Open CodePen

Go to CodePen.io and click “Start Coding” (or “Create > New Pen”).

Step 2: Set Up the HTML

Copy and paste this code block directly into the HTML panel (left or top box depending on your layout):

html

<div class="container">
  <h1>Developer Fetch Comparison</h1>
  <p>Open your browser console (F12) to see the logs!</p>
  
  <!-- Button 1: Modern Approach -->
  <button id="btn-async" class="btn btn-modern">
    Method A: Async / Await
  </button>
  
  <!-- Button 2: Traditional Approach -->
  <button id="btn-then" class="btn btn-traditional">
    Method B: Promises (.then)
  </button>
</div>

Use code with caution.

Step 3: Set Up the JavaScript

Copy and paste this exact logic into the JS panel to hook up the API triggers [finance]:

javascript

// ========================================================
// METHOD A: THE MODERN APPROACH (ASYNC / AWAIT)
// ========================================================
document.getElementById('btn-async').addEventListener('click', async () => {
  console.clear(); // Clears console so you only see this run
  console.log("🛰️ [Async/Await] Initiating background network lane...");
  
  try {
    // Execution pauses here until the server responds
    const response = await fetch('https://jsonplaceholder.typicode.com/users/1');
    
    // Execution pauses here until bytes turn into an interactive object
    const dataPackage = await response.json();
    
    console.log("📦 [Async/Await] Success! Payload received:", dataPackage.name);
  } catch (error) {
    console.error("❌ [Async/Await] Safe catch triggered:", error.message);
  }
});

// ========================================================
// METHOD B: THE TRADITIONAL APPROACH (.THEN / .CATCH)
// ========================================================
document.getElementById('btn-then').addEventListener('click', () => {
  console.clear(); // Clears console so you only see this run
  console.log("🛰️ [.then()] Registering network event hooks...");
  
  // fetch() runs instantly and returns a placeholder Promise object
  fetch('https://jsonplaceholder.typicode.com/users/1')
    .then((response) => {
      // Passes the next promise down the chain
      return response.json(); 
    })
    .then((dataPackage) => {
      // Executed much later when data is fully processed
      console.log("📦 [.then()] Success! Payload received:", dataPackage.name);
    })
    .catch((error) => {
      // Catches errors originating from any point in the chain above
      console.error("❌ [.then()] Catch block triggered:", error.message);
    });
});

Use code with caution.

Step 4: Add Minimal Styling (Optional CSS Panel)

Drop this into your CSS panel to center the buttons and align them beautifully on a modern dark theme:

css

body { font-family: sans-serif; background: #1e1e1e; color: #ffffff; padding: 40px; text-align: center; }
.container { max-width: 600px; margin: 0 auto; }
button { padding: 14px 24px; margin: 12px; font-size: 16px; border: none; border-radius: 6px; cursor: pointer; font-weight: bold; transition: background 0.2s; }
.btn-modern { background-color: #2ecc71; color: #fff; }
.btn-traditional { background-color: #3498db; color: #fff; }
button:hover { filter: brightness(0.9); }

Use code with caution.

Step 5: Execute the Code

1. Press F12 (or right-click -> Inspect -> Console) to open your native web browser console.
2. Click Method A: Async / Await to see the clean, modern response lifecycle fire.
3. Click Method B: Promises (.then) to see the exact same result using traditional downward event chain hooks.

Here is a brief, developer-level explanation of what is happening under the hood for each output:

Output 1: Async / Await

"🛰️ [Async/Await] Initiating background network lane..."
"📦 [Async/Await] Success! Payload received:" "Leanne Graham"

Use code with caution.

• The Mechanics: The code executes sequentially down the line like a book. The await keyword tells JavaScript to pause execution right there inside that event block, freeze the line, and wait for the remote data package to return.
• The Result: The moment the network finishes streaming, the code wakes back up instantly, moves to the next line, reads the user’s name parameter, and prints it out.

Output 2: Promises (.then())

"🛰️ [.then()] Registering network event hooks..."
"📦 [.then()] Success! Payload received:" "Leanne Graham"

Use code with caution.

• The Mechanics: The code executes all at once without stopping. When fetch() runs, it immediately sets up a background listener hook (.then()) and forgets about it, allowing JavaScript to instantly move on.
• The Result: The second log doesn’t print immediately. It sits in a separate network callback queue until the server responds. Once the data arrives minutes or milliseconds later, the browser fires the registered .then() hook and pushes the payload out to your console logs.

The Key Takeaway

Both methods accomplish the exact same thing: they keep the browser window interactive and non-blocking while waiting for an external server. The only difference is syntax structure — async/await lets you read the process linearly from top-to-bottom, while .then() uses callback event handlers chained together.

Now that you have verified both outputs work perfectly in your CodePen console, would you like to read my latest article, Demystifying CORS: Decoding Security Headers on Massive Infrastructure, and learn how to diagnose cross-origin issues, configure server-side headers, and manage trusted domains at scale?

Read the article on Medium →

Thank You for Reading!

Thank you for diving into the evolution of asynchronous JavaScript with me! Moving from the messy nesting of Callback Hell to the clean predictability of async/await is a major milestone for any developer. Keep practicing, keep your code clean, and happy hacking!

Both of your tutorials — CORS Policies and Demystifying Callback Hell — are now complete with code, explanations, and custom thumbnails. Would you like to compile them into a single reference document or start brainstorming a new development topic?

Disclosure: This article is an educational and/or technical exploration based on concepts, experiments, and practical experience. Code samples, configurations, architectures, and examples may be simplified, adapted, or illustrative. No institutional source code, real production data, credentials, confidential information, or sensitive internal infrastructure details are reproduced.

Demystifying Callback Hell was originally published in Jungletronics on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ever spent hours staring at a cryptic Blocked by CORS policy error in your browser console? You aren't alone. Cross-Origin Resource Sharing (CORS) is the browser's digital security guard, designed to keep sensitive web data safe from unauthorized domains.

While a simple wildcard (*) works for a basic public API, massive enterprise infrastructure demands a more sophisticated approach. In this tutorial, we will strip away the confusion behind cross-origin headers, see how huge systems manage trusted sites at scale without hardcoding thousands of domains, and give you the tools to uncover exactly how your company's network gateway validates incoming origins.

1. The Wildcard (*)

This is the open-door policy. The server does not check the identity or origin of the incoming request. It simply attaches a universal stamp of approval to every single response payload.

• The Header: Access-Control-Allow-Origin: *
• Best For: Public APIs. Ideal for government financial transparency portals, open-source mock servers, or public weather data feeds.
• The Verdict: Simple to operate and generally lightweight, but inappropriate when browser access must be restricted to specific origins, particularly for applications involving private user data or credentialed requests.

2. Dynamic Origin Validation

This is a customized approach used by many modern web platforms. When a request arrives, the server evaluates the incoming origin against an approved policy and determines whether that origin should receive CORS access.

• The Header:

Access-Control-Allow-Origin: https://trusted-client.example
Vary: Origin

• Origin (The server dynamically mirrors the requesting domain only if it passes the security check).
• Best For: Private SaaS Applications. Ideal for paid software platforms where clients manage and whitelist their own target domains via an admin dashboard.
• The Verdict: More restrictive and precise, but it introduces policy-management complexity. The approved origins may come from static configuration, application logic, a cache, a database, or another policy mechanism, depending on the architecture.

3. The Network Gateway

This is the institutional approach used at a massive architectural scale. The responsibility for evaluating CORS rules is completely removed from the developer’s application code and moved to the edge of the network infrastructure.

• The Header: Injected dynamically by routing layers like Kong, Apigee, AWS API Gateway, or NGINX.
• Best For: Enterprise Infrastructure. Essential for large financial institutions, public utility systems, or expansive microservice environments.
• The Advantage: Centralized, consistent, and scalable. It can standardize CORS policies across hundreds of independent services while reducing duplicated security logic in application code.

Part 4: Playing Detective — Uncovering Your Infrastructure Strategy

Working with public finance data on a massive infrastructure scale means security and auditability are critical. As enterprise environments grow, maintaining origin policies across many applications and services can become increasingly complex.

When you get back to your internal machine, you can play detective right from your browser or terminal to investigate how your infrastructure handles CORS. Here are three technical paths to uncover where the decision is being made:

Hint 1 → observe the HTTP behavior
Hint 2 → investigate the infrastructure
Hint 3 → confirm in the code where the decision is implemented

1˚ Hint

Inspect the “Origin Reflection” (The Terminal Test)

One of the simplest ways to investigate a server’s CORS behavior is to send a request with a specific Origin header and inspect how the server responds.

This does not reveal whether the server uses a database, cache, configuration file, or gateway policy internally. It only shows the CORS behavior that is visible at the HTTP response level.

Open your terminal or command prompt and run this curl command against an API endpoint you are authorized to test:

curl -i \
  -H "Origin: https://fake-evil-domain.com" \
  https://your-company-api.com

Important: curl does not enforce CORS. It simply sends the Origin header and lets you inspect the server's response. CORS enforcement happens in the browser, which determines whether JavaScript is allowed to access the cross-origin response.

Now inspect the response headers.

Scenario A: The response contains:

Access-Control-Allow-Origin: https://fake-evil-domain.com

• Diagnosis: The server reflected the supplied origin. This is consistent with dynamic CORS handling, but the response alone does not prove that the origin was validated against an allowlist — or even that a database was involved.

Scenario B: The response contains:

Access-Control-Allow-Origin: *

• Diagnosis: The endpoint is allowing cross-origin access for any origin under the CORS policy represented by *. This is commonly appropriate for genuinely public, non-credentialed APIs, but it is not appropriate when the API needs to restrict browser access to specific origins.

Scenario C: The response does not contain an Access-Control-Allow-Origin header for the supplied origin.

• Diagnosis: The response does not grant CORS access to that origin. The HTTP request may still have reached and been processed by the server; the absence of the CORS header means that a browser will not make the response available to calling JavaScript.

Scenario D: The server returns an HTTP error or the connection fails.

• Diagnosis: This indicates a server, network, routing, authentication, or other HTTP-level problem. It should not automatically be classified as a CORS rejection. Check the HTTP status and response before drawing conclusions about the CORS policy.

A useful follow-up test

To investigate whether the behavior changes for an approved origin, repeat the request with a domain that you are authorized to test:

curl -i \
  -H "Origin: https://trusted-client.example" \
  https://your-company-api.com

Compare the two responses.

If the server returns a specific origin only for the trusted domain and omits Access-Control-Allow-Origin for the untrusted one, that is evidence of origin-dependent CORS behavior. However, the test still cannot tell you whether the decision is implemented in application code, a database lookup, a cache, an API gateway, a reverse proxy, or another policy mechanism.

2˚ Hint

Look for Gateway Infrastructure (API Management)

In large enterprise environments, CORS policies may be managed centrally at the API gateway or edge infrastructure layer rather than implemented independently in every backend application.

When reviewing architecture documentation, infrastructure-as-code repositories, or DevOps configuration, look for:

• API Gateways / API Management: Kong, Apigee, AWS API Gateway, or similar platforms. Look for CORS policies, allowed origins, methods, and headers.
• Reverse Proxies / Load Balancers: NGINX, HAProxy, F5 BIG-IP, or similar infrastructure. Depending on the configuration, these layers may add, modify, or conditionally return CORS headers.
• Infrastructure as Code: Terraform, Kubernetes manifests, Helm charts, or gateway policies. Search for terms such as:

cors
allowedOrigins
allow-origin
Access-Control-Allow-Origin
Origin

The objective is to establish where the CORS decision is being made.

For example, the architecture might look like:

Browser → API Gateway / Reverse Proxy → Backend

Or:

Browser → API Gateway → Backend → CORS logic

The presence of a gateway does not prove that the gateway owns the CORS policy. Inspect the configuration or trace the request flow to determine which layer is actually responsible.

If the application configuration looks correct but the response differs from what the application appears to generate, an upstream gateway, reverse proxy, or load balancer may be adding or modifying the response headers.

3˚ Hint

Search the Codebase for Architectural Footprints

If you have authorized access to your team’s Git repositories, search the codebase for configuration, middleware, or policy definitions related to CORS.

You can use Ctrl + Shift + F in VS Code or tools such as grep from the terminal to search across the project.

Look for indicators such as:

• Configuration: Search for keys and environment variables such as allowedOrigins, corsOptions, CORS_WHITELIST, ALLOWED_ORIGINS, or similar names.
• CORS Middleware: Look for framework-specific CORS configuration or middleware. Depending on the technology stack, this might appear in application startup code, middleware registration, route configuration, or security modules.
• Origin Validation: Search for code that reads the incoming Origin header and compares it against an approved list or policy:

const origin = request.headers.origin;

if (allowedOrigins.includes(origin)) {
  // Apply the appropriate CORS response headers
}

• Dynamic Policy Lookup: In some architectures, approved origins may be retrieved from a database, cache, configuration service, or another policy store:

const client = await policyStore.findByOrigin(origin);

What you are trying to establish

The objective is to identify where the CORS decision is implemented and what data it depends on.

For example, you might discover a simple static configuration:

ALLOWED_ORIGINS=https://app.example.com,https://portal.example.com

Or you may find application logic that evaluates the incoming origin dynamically.

You may also find no CORS logic in the application at all. That is useful evidence too: it suggests that the policy may be implemented upstream at an API gateway, reverse proxy, load balancer, or another infrastructure layer.

The key is to distinguish between what the HTTP response reveals and what the source code or infrastructure configuration actually confirms.

Part 5: Practice — Test CORS Behavior

Here are the quick, simplified steps to set up this test directly inside your current CodePen setup. This allows you to simulate how your browser reacts to the three different server responses in real time.

Step 1: Update Your HTML

Add three new test buttons inside your existing HTML container in CodePen:

html

<h3>Part 5: Test CORS Behavior</h3>

<button id="btn-wildcard">Test CORS Access</button>
<button id="btn-reflection">Test CORS-Enabled Endpoint</button>
<button id="btn-blocked">Test CORS Access Not Granted</button>

Use code with caution.

Step 2: Update Your JavaScript

Paste this code at the bottom of your JS panel in CodePen.

Instead of using a real corporate server, this example uses public endpoints to demonstrate different observable browser outcomes.

These endpoints are useful for experimentation, but they do not reproduce the internal architecture of an enterprise gateway. The browser only sees the HTTP behavior exposed by the remote server; it cannot reveal whether a database, cache, configuration store, gateway, or application code produced that behavior.

javascript

// 1. Simulating Strategy 1: The Global Wildcard
document.getElementById('btn-wildcard').addEventListener('click', async () => {
  console.clear();

  try {
    const response = await fetch('https://jsonplaceholder.typicode.com/users/1');
    const data = await response.json();

    console.log(
      "✅ WILDCARD SUCCESS: The browser was allowed to read the cross-origin response.",
      data.name
    );
  } catch (err) {
    console.error("❌ Failed:", err.message);
  }
});


// 2. Simulating Strategy 2: Dynamic CORS Handling
document.getElementById('btn-reflection').addEventListener('click', async () => {
  console.clear();

  try {
    const response = await fetch('https://httpbin.org/get');

    const data = await response.json();

    console.log(
      "🔄 DYNAMIC CORS: The request succeeded and the browser exposed the response."
    );

    console.log("Origin observed by the endpoint:", data.headers?.Origin || "Not exposed");
  } catch (err) {
    console.error("❌ Failed:", err.message);
  }
});


// 3. Simulating Strategy 3: CORS Access Not Granted
document.getElementById('btn-blocked').addEventListener('click', async () => {
  console.clear();

  console.log(
    "🛰️ Testing a cross-origin resource that does not grant browser access..."
  );

  try {
    await fetch('https://google.com');

    console.log(
      "⚠️ The request completed, but this does not necessarily mean CORS access was granted."
    );
  } catch (err) {
    console.log(
      "❌ DIAGNOSIS: The browser blocked JavaScript from reading the cross-origin response."
    );

    console.error(
      "Browser error message:",
      err.message
    );
  }
});

Use code with caution.

This test demonstrates that the browser can access a cross-origin response when the remote server provides the appropriate CORS headers. The response alone does not reveal whether those headers were generated from a database, configuration file, cache, or another policy mechanism.

Step 3: Run the Test

1. Open your browser console (F12).
2. Click Test CORS Access → The console should show whether the browser was able to read the cross-origin response.
3. Click Test CORS Access Not Granted → Depending on the endpoint and browser, the console may show a CORS-related error.

This demonstrates the browser-side symptom that occurs when a cross-origin response does not grant JavaScript access. It does not prove that a firewall, whitelist, or gateway rejected the underlying HTTP request.

Part 6:

The Ultimate CORS Troubleshooting Checklist

If you, reader, hit a wall with a CORS error, use this sequential step-by-step checklist to isolate and resolve the issue quickly:

1. Identify the Core Issue

• Check the Console: Does the log explicitly mention a CORS policy violation, or is it a generic network error?
• Developer Tip: If the backend server drops entirely or has a broken DNS record, the browser will output Failed to fetch. This can sometimes look like a CORS error, but it is actually a standard infrastructure outage.

2. Inspect the Request Headers

• Open your browser’s Developer Tools (F12) and navigate to the Network tab.
• Click on the failing request and locate the automatic Origin request header.
• Verify your environment details: Did the protocol change (e.g., http to https) or the port shift (e.g., localhost:3000 to localhost:8080)? The browser treats any structural difference as a completely distinct origin.

3. Verify the Server Response

• Look closely at the Response Headers sent back by your API.
• Is the Access-Control-Allow-Origin header present?
• If yes, does its value match the exact string sent in your request’s Origin header?
• If no, the response does not grant CORS access to the requesting origin. The request may still have reached and been processed by the server.

4. Check for Preflight Handshakes

• Did your browser fire an extra OPTIONS network request right before your actual GET or POST request?
• Requests that are not considered CORS-safelisted — for example, many requests using custom headers or certain content types — may trigger a preflight OPTIONS request.

5. Audit the Infrastructure Layer

• If your application’s source code looks flawless but requests are still being blocked, move up the network pipeline:
• Does the API Gateway (e.g., Kong, Apigee) have a global CORS plugin running that is overriding your local configuration?
• Is an edge proxy server or load balancer (e.g., NGINX, HAProxy) configured with strict routing maps that filter out unfamiliar external traffic?

Thank You for Reading!

Thank you so much for taking the time to read this guide. I hope it helps demystify the inner workings of async JavaScript and complex CORS policies for you. Learning by doing is the best way to grow as a developer, so keep testing, keep querying your APIs, and happy coding!

If you found this helpful or discovered your company’s network strategy using these steps, let me know in the comments below!

Note ¹ Enterprise-Focused

• Title: 🛠️ Advanced Insight: Mitigating Information Disclosure
• Why it works: It sounds like an official security compliance document, which fits perfectly with massive infrastructure contexts.

🔒 Pro-Tip: Infrastructure Hardening & Information Disclosure

When auditing your network infrastructure via the terminal, you might notice headers like Server: example-gateway or X-Powered-By: Express in the server's response. While this looks like harmless metadata, in large enterprise or public finance environments, exposing your exact backend technology is considered a security risk known as Information Disclosure.

Attackers can use techniques such as banner grabbing and service fingerprinting to collect implementation details from exposed systems. A disclosed product and version can then be correlated with public vulnerability information, helping attackers prioritize further investigation.

How the Infrastructure Team Protects the Headers

To mitigate this without breaking any client or partner integrations, DevOps and Security teams apply server hardening techniques at the gateway layer:

1. Disabling Server Tokens (Basic)

In the core configuration file (example-gateway.conf), teams add a directive to hide the specific version numbers from headers and default error pages:

example-gateway

# Inside the http, server, or location block
server_tokens off;

Use code with caution.

• Result: The response no longer exposes the specific server version through the standard NGINX Server header. The exact header value depends on the server and proxy configuration.

2. Header Removal (Advanced)

If the architecture requires the `Server` header to be removed entirely, that is a separate edge-layer configuration task and depends on the proxy or gateway in use.

The exact mechanism varies by platform. For example, `server_tokens off;` in NGINX hides version information, but it does not remove the `Server` header itself.

The important distinction is that reducing version disclosure and completely removing a server header are two different configuration goals.

In most standards-compliant HTTP clients, changing or removing the Server metadata does not affect how the application processes the response. However, compatibility should always be verified when changing infrastructure behavior, especially in legacy or tightly integrated environments.

Here is an illustrative terminal command and a simplified response for the Technical Bonus section:

bash

curl -I \
  -H "Origin: https://example-attacker.test" \
  https://example-api.test

Example response:

HTTP/2 200
server: example-gateway
content-type: application/json

illustrative terminal command

Use code with caution.

🔍 What this output proves:

1. The server returned an HTTP response: The endpoint is reachable and returned `HTTP/2 200`. This alone does not establish how the application processed the request.
2. CORS access was not granted in this response: No `Access-Control-Allow-Origin` header is present. This observation alone does not reveal whether the decision came from a whitelist, application logic, gateway policy, or another mechanism.
3. Information Disclosure: The response exposes a server or gateway identifier. Such metadata can contribute to fingerprinting and may be correlated with publicly known technologies or vulnerabilities.

SubNote ².⚠️:

Server Fingerprinting

Exposing the server technology through response headers can provide additional information for server fingerprinting. Hiding unnecessary implementation details is therefore a reasonable hardening measure, although it should not be treated as a substitute for patching, secure configuration, authentication, authorization, or other primary security controls.

Security Disclosure

The examples in this article are illustrative and intended for educational purposes. They do not reproduce real institutional infrastructure, internal endpoints, credentials, production configurations, or confidential security findings.

Any domains, headers, responses, and command examples shown are synthetic or adapted for demonstration. Security testing should only be performed against systems for which you have explicit authorization.

Demystifying CORS: Decoding Security Headers on Massive Infrastructure was originally published in Jungletronics on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is an excellent question, and it gets to one of Django’s core design philosophies.

The short answer is:

Django is an application framework, not a web server.

The longer explanation is much more interesting.

The job of Django

Django’s responsibility is to generate dynamic responses.

For example:

Browser
   │
   ▼
GET /products/42
   │
   ▼
Django
   │
Queries database
Processes business logic
Renders template
   │
   ▼
HTML

That’s where Django shines.

The job of static files

Static files are different.

logo.png
style.css
app.js
video.mp4
font.ttf

These files never change while being served.

Serving them is simply:

Client
   │
GET /static/logo.png
   │
Read file from disk
   │
Send bytes

No Python code is needed.

Why not let Django do it?

It can.

In development, when DEBUG=True, Django already serves static files.

That’s why:

python manage.py runserver

works perfectly.

But this implementation is intentionally simple and not optimized for production.

Suppose your homepage contains:

• HTML
• CSS
• 8 JavaScript files
• 25 in
• 6 fonts

One page load becomes roughly:

1 HTML request
40 static requests
--------------
41 requests

Now imagine:

10,000 visitors

That’s approximately:

410,000 requests

If Django handled every one of those, every request would:

Browser
 ↓
Gunicorn
 ↓
Python interpreter
 ↓
Django
 ↓
StaticFileHandler
 ↓
Read disk
 ↓
Return file

Every request wakes up Python, allocates memory, and occupies a worker process.

Nginx is built for exactly this

For the same request:

Browser
 ↓
Nginx
 ↓
Read file
 ↓
Send file

Django is never involved.

No Python.

No ORM.

No middleware.

No templates.

Just raw file transfer.

A nice analogy

Imagine a restaurant.

You

Steak

The chef cooks it.

But imagine asking the chef:

“Can I have another napkin?”

Should the chef stop cooking to hand you a napkin?

Of course not.

A waiter handles that.

Static files are the “napkins” of a web application.

Nginx is the waiter.

But modern apps have lots of images…

Exactly.

Against of:

• logos
• product images
• icons
• PDFs
• we
• CSS bundles
• JavaScript
• web fonts

Django typically doesn’t serve these directly in production.

Instead, it tells the web server where they live:

/static/

For user-uploaded files:

/media/

Nginx usually serves those too.

Then what is WSGI?

Think of WSGI as an agreement between Python applications and web servers.

Browser
    │
    ▼
Nginx
    │
    ▼
Gunicorn
    │
    ▼
WSGI
    │
    ▼
Django

WSGI defines how the web server passes an HTTP request to your Python application and how the application returns a response.

And ASGI?

ASGI is the modern successor to WSGI.

It supports:

• asynchronous code
• WebSockets
• Server-Sent Events
• long-lived connections
• HTTP/2-friendly patterns

The architecture is similar:

Browser
     │
     ▼
Nginx
     │
     ▼
Uvicorn
     │
     ▼
ASGI
     │
     ▼
Django

ASGI expands what kinds of connections your application can handle, but it still doesn’t aim to be a high-performance static file server.

Then why does WhiteNoise exist?

Many deployments don’t use Nginx. For example, on platforms like Render, Railway, or Fly.io, you may only control your Python process.

WhiteNoise lets your Django application serve static files efficiently enough for many production use cases by adding optimized static-file handling inside the Python stack.

Browser
     │
     ▼
Gunicorn
     │
     ▼
WhiteNoise
     │
     ▼
Static file

If the request is for /static/..., WhiteNoise serves it directly without passing it through Django's URL routing or view logic. It's not as specialized as Nginx, but it's simple, reliable, and widely used for small to medium-sized applications.

The common product

Large applications often use this separation of responsibilities:

              Browser
                 │
                 ▼
              Nginx
          ┌──────┴──────┐
          │             │
          ▼             ▼
      /static/*     Gunicorn/Uvicorn
          │             │
          ▼             ▼
    CSS, JS, Images    Django
                           │
                           ▼
                      PostgreSQL

This design lets each component do what it’s best at:

• Nginx: fast delivery of static files, TLS termination, caching, compression, and reverse proxying.
• Gunicorn/Uvicorn: manage Python worker processes and communicate via WSGI or ASGI.
• DJ: business logic, authentication, templates, APIs, and database interaction.

Keeping these concerns separate is one of the reasons Django deployments scale well from small personal projects to large production systems.

Why django does not serve static files? was originally published in Jungletronics on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ubuntu’s Ext4 filesystem rarely needs defragmentation because it is designed to minimize fragmentation automatically. This procedure is mainly useful for traditional HDDs. Avoid defragmenting SSDs, as it causes unnecessary wear.

1. Install the Defragmentation Tool

sudo apt update && sudo apt install e2fsprogs

2. Check Fragmentation Level

Analyze the root filesystem:

sudo e4defrag -c /

Interpretation:

• Score < 30: Good condition
• Score > 30: Defragmentation may help

3. Defragment the Filesystem

sudo e4defrag /

This reorganizes fragmented files and may improve HDD performance.

4. Clean Package Cache

Remove downloaded package files that are no longer needed:

sudo apt clean

5. Remove Unused Dependencies

Delete orphaned packages left behind by software removals:

sudo apt autoremove

Summary

Commands used:

sudo apt update && sudo apt install e2fsprogs
sudo e4defrag -c /
sudo e4defrag /
sudo apt clean
sudo apt autoremove

For most Ubuntu systems, especially those running on SSDs, regular package cleanup provides more benefit than filesystem defragmentation.

Thank you!

🚀 Ubuntu Running Slower? This 5-Minute Cleanup Might Surprise You was originally published in Jungletronics on Medium, where people are continuing the conversation by highlighting and responding to this story.

No episódio anterior, construímos uma base Docker reutilizável para projetos Rails. O objetivo foi separar a aplicação da infraestrutura: Ruby, Rails e PostgreSQL ficam encapsulados nos containers, enquanto o código permanece no projeto local.

Agora vamos aproveitar essa mesma base para outra preocupação fundamental do desenvolvimento: testes automatizados.

Cucumber e RSpec cumprem papéis diferentes, mas complementares:

• Cucumber descreve o comportamento esperado da aplicação em linguagem natural, usando Gherkin;
• RSpec verifica o comportamento interno do código Ruby/Rails.

A ideia é trabalhar de fora para dentro e de dentro para fora:

                COMPORTAMENTO
                      │
                      ▼
                  CUCUMBER
             "O que o usuário espera?"
                      │
                      ▼
                   RSpec
             "O código está correto?"
                      │
                      ▼
                   RAILS
                      │
                      ▼
                  POSTGRESQL

A vantagem é que não precisamos mais preparar toda a infraestrutura antes de começar.

Nosso projeto já possui Ruby, Rails, PostgreSQL e Docker configurados. Vamos apenas adicionar a camada de testes.

0. Partindo da base existente

Estamos assumindo que você terminou o tutorial anterior e está na raiz de um projeto Rails:

cd ~/Documents/projects/rails/meu-projeto

Confirme que os containers estão funcionando:

docker compose ps

E confirme o Rails:

docker compose run --rm app \
  bundle exec rails -v

Se o PostgreSQL ainda não estiver iniciado:

docker compose up -d

A partir daqui, todos os comandos que anteriormente eram executados diretamente com rails, bundle, rspec ou cucumber serão executados dentro do container app.

Por exemplo:

docker compose run --rm app \
  bundle install

em vez de:

bundle install

Essa é a principal adaptação do tutorial antigo.

2. Adicionando Cucumber

Agora acrescentamos o Cucumber ao mesmo projeto.

No Gemfile:

group :development, :test do
  gem "cucumber-rails", require: false
  gem "database_cleaner-active_record"
  gem "capybara"
  gem "selenium-webdriver"
end

Instale novamente:

docker compose run --rm app \
  bundle install

Inicialize o Cucumber:

docker compose run --rm app \
  bundle exec rails generate cucumber:install

O Rails/Cucumber criará a estrutura:

features/
├── step_definitions/
└── support/

Teste:

docker compose run --rm app \
  bundle exec cucumber

Neste momento ainda não temos cenários:

0 scenarios
0 steps

Mas a infraestrutura de testes está funcionando.

A principal mudança em relação ao artigo original

Antes tínhamos:

rails generate scaffold ...
rails db:migrate
rspec
cucumber

Agora temos:

docker compose run --rm app \
  bundle exec rails generate scaffold ...

e:

docker compose run --rm app \
  bundle exec rails db:migrate

e:

docker compose run --rm app \
  bundle exec rspec

e:

docker compose run --rm app \
  bundle exec cucumber

O Rails continua sendo Rails.

A única diferença é onde ele está sendo executado.

ANTES

Ubuntu
 ├── Ruby
 ├── Rails
 ├── Bundler
 ├── PostgreSQL
 └── Cucumber/RSpec

AGORA
Ubuntu
 └── Docker
      │
      ├── app
      │    ├── Ruby
      │    ├── Rails
      │    ├── Bundler
      │    ├── RSpec
      │    └── Cucumber
      │
      └── db
           └── PostgreSQL

Isso é justamente a abstração que conseguimos no tutorial anterior.

Vamos fazer uma pequena melhoria adicional

Em vez de repetir:

docker compose run --rm app bundle exec ...

em cada comando do artigo, podemos criar uma seção de atalhos para testes depois que a infraestrutura estiver funcionando.

Por exemplo:

docker compose run --rm app bundle exec rspec

docker compose run --rm app bundle exec cucumber

ou, se você preferir entrar no container:

docker compose exec app bash

e então trabalhar normalmente:

bundle exec rspec
bundle exec cucumber

Isso deixa o tutorial muito mais parecido com o artigo original, sem abandonar a infraestrutura Docker.

A documentação atual do artigo original pode ser preservada na parte conceitual — especialmente a ideia de Cucumber como especificação de comportamento e RSpec como validação interna — mas a preparação do ambiente deve ser completamente substituída pela base Docker que acabamos de construir.

Perfeito. Vamos recriar o tutorial novo a partir daqui, mas já corrigindo algumas coisas do artigo antigo para o ambiente atual Rails + PostgreSQL + Docker + Cucumber 11 + RSpec.

A principal mudança será esta:

Tudo que antes era executado diretamente no computador agora será executado dentro do container app.

Começamos assumindo que você está na raiz do projeto.

1. Criando a aplicação inicial

Entre no container:

docker compose exec app bash

A partir daqui, todos os comandos abaixo são executados dentro do container.

Gerador + banco

Vamos criar o CRUD inicial de tarefas:

rails generate scaffold tarefa nota:text feito:boolean --skip-test
rails db:migrate

O scaffold cria automaticamente:

• model Tarefa;
• controller;
• views;
• migration;
• nada de testes básicos (criaremos manualmente);
• rotas REST.

A estrutura principal será:

app/
├── controllers/
│   └── tarefas_controller.rb
├── models/
│   └── tarefa.rb
└── views/
    └── tarefas/

E a migration criará a tabela tarefas no PostgreSQL.

2. Configurando as rotas

Abra:

config/routes.rb

Deixe assim:

Rails.application.routes.draw do
  resources :tarefas
  root "tarefas#index"
end

Temos agora duas coisas importantes:

resources :tarefas

cria as rotas CRUD da aplicação.

E:

root "tarefas#index"

define a página inicial.

Portanto:

/

aponta para:

TarefasController#index

3. Preparando a página inicial

Vamos substituir a view gerada automaticamente pelo scaffold.

Abra:

app/views/tarefas/index.html.erb

e substitua seu conteúdo por:

<h1>Listando Tarefas</h1>

<table class="table">
  <thead>
    <tr>
      <th>Feito</th>
      <th>Nota</th>
      <th colspan="3"></th>
    </tr>
  </thead>
  <tbody>
    <% @tarefas.each do |tarefa| %>
      <tr>
        <td><%= tarefa.feito? ? "true" : "false" %></td>
        <td><%= tarefa.nota %></td>
        <td><%= link_to "Mostrar", tarefa %></td>
        <td><%= link_to "Editar", edit_tarefa_path(tarefa) %></td>
        <td>
          <%= link_to "Apagar", tarefa,
              method: :delete,
              data: { confirm: "Está certo?" } %>
        </td>
      </tr>
    <% end %>
  </tbody>
</table>
<br>
<%= link_to "Nova Tarefa", new_tarefa_path, class: "btn btn-primary" %>

Por enquanto temos uma única lista.

É exatamente isso que queremos.

O objetivo deste episódio será modificar esse comportamento utilizando TDD.

4. Criando alguns dados

Opcionalmente, podemos criar algumas tarefas para visualizar a aplicação.

Abra:

db/seeds.rb

e acrescente:

Tarefa.create!(nota: "Estudar Ruby", feito: false)
Tarefa.create!(nota: "Lavar o carro", feito: true)

Execute:

rails db:seed

Podemos verificar:

rails db:migrate:status

5. Iniciando o servidor

Como estamos dentro do container, nao precisamos executar:

rails server

Abra no navegador:

http://127.0.0.1:3000

Você deverá encontrar algo semelhante a:

Listando Tarefas

Feito    Nota
false    Estudar Ruby
true     Lavar o carro

Temos agora nosso ponto de partida.

6. O novo requisito

Imagine que o cliente agora faça um novo pedido:

A lista de tarefas deve ser dividida em duas seções: tarefas concluídas e tarefas não concluídas.

Em vez de simplesmente alterar a view, vamos implementar essa mudança utilizando TDD.

A ideia será:

REQUISITO
    ↓
CUCUMBER
    ↓
TESTE FALHA
    ↓
IMPLEMENTAÇÃO
    ↓
TESTE PASSA
    ↓
RSPEC
    ↓
VALIDAÇÃO INTERNA

É aqui que começam a trabalhar juntos os dois protagonistas deste episódio:

Cucumber — descreve o comportamento.

RSpec — verifica a implementação.

7. Especificando o requisito

Antes de escrever código, vamos transformar o pedido do cliente em comportamento observável.

Exibição de tarefas concluídas

Ao visitar a página principal:

• devo ver a seção Tarefas Concluídas;
• se existir uma tarefa concluída chamada "Lavar o carro", ela deve aparecer nessa seção;
• se existir uma tarefa não concluída chamada "Estudar Ruby", ela não deve aparecer na seção de concluídas.

Podemos resumir:

Exibição de tarefas concluídas

Ao visitar a página principal
    eu devo ver a seção tarefas concluídas
    Teste 1:
      criar uma tarefa concluída chamada "Lavar o carro"
      eu devo ver "Lavar o carro" na seção concluídas
    Teste 2:
      criar uma tarefa não concluída chamada "Estudar Ruby"
      eu não devo ver "Estudar Ruby" na seção concluídas

Observe que ainda não estamos falando de HTML, CSS, controller ou ActiveRecord.

Estamos falando apenas de comportamento.

Essa é uma das ideias mais interessantes do Cucumber.

8. Transformando o requisito em passos

Agora podemos transformar nossa linguagem de negócio em ações que futuramente serão implementadas.

visitar_a_pagina_principal
criar_uma_tarefa_concluida_anotada(nota)
criar_uma_tarefa_nao_concluida_anotada(nota)
eu_devo_ver_a_secao_tarefas_concluidas
eu_devo_ver_na_secao_concluidas_a_tarefa(nota)
eu_nao_devo_ver_na_secao_concluidas_a_tarefa(nota)

Ainda não precisamos saber como cada função será implementada.

Primeiro definimos o que queremos que aconteça.

9. Criando nossa primeira Feature

Dentro do container:

mkdir -p features
touch features/tarefas.feature

Abra:

features/tarefas.feature

e escreva:

Feature: Exibicao de Tarefas Concluidas
  A pagina principal do app deve exibir corretamente as tarefas concluidas
  e garantir que tarefas nao concluidas nao aparecam na secao especificada.

Background:
    Given eu estou na pagina principal do app
  Scenario: Exibir uma tarefa concluida na secao "tarefas-concluidas"
    Given existe uma tarefa concluida chamada "Lavar o carro"
    When eu envio essa tarefa concluida
    Then eu devo ver "Lavar o carro" na secao "tarefas-concluidas"
  Scenario: Garantir que uma tarefa nao concluida nao apareca na secao "tarefas-concluidas"
    Given existe uma tarefa nao concluida chamada "Estudar Ruby"
    When eu envio essa tarefa nao concluida
    Then eu devo ver "Estudar Ruby" na secao "tarefas-nao-concluidas"
  Scenario: Exibir somente tarefas concluidas na secao "tarefas-concluidas"
    Given existe uma tarefa concluida chamada "Lavar o carro"
    And existe uma tarefa nao concluida chamada "Estudar Ruby"
    When eu envio ambas as tarefas
    Then eu devo ver "Lavar o carro" na secao "tarefas-concluidas"
    And eu nao devo ver "Estudar Ruby" na secao "tarefas-concluidas"

Por que não usamos acentos?

Aqui estamos deliberadamente usando:

concluida
nao
pagina
secao

em vez de:

concluída
não
página
seção

Não é uma exigência do Cucumber, mas simplifica a correspondência exata entre os textos da feature e as definições de passos.

Na interface do usuário, naturalmente, podemos usar a acentuação correta.

Acrescente no:

features/support/env.rb

# features/support/env.rb
ENV['RAILS_ENV'] ||= 'test'

require File.expand_path('../../../config/environment', __FILE__)

require 'cucumber/rails'

Capybara.default_driver = :rack_test
Capybara.javascript_driver = :selenium_chrome_headless

ActionController::Base.allow_rescue = false

# Limpa o banco entre cenários
begin
  DatabaseCleaner.strategy = :transaction
rescue NameError
  raise "Add database_cleaner-active_record to your Gemfile"
end

Cucumber::Rails::Database.autorun_database_cleaner = true

10. Instalando o RSpec

No Gemfile, acrescente:

group :development, :test do
  gem "rspec-rails"
end

Depois instale a dependência:

bundle install

Agora inicialize o RSpec:

bundle exec rails generate rspec:install

Isso cria a estrutura básica:

spec/
├── spec_helper.rb
└── rails_helper.rb

Podemos verificar:

bundle exec rspec

Inicialmente, é normal não haver exemplos:

No examples found.

Finished in ...
0 examples, 0 failures

O RSpec está instalado e funcionando.

Como o RSpec já foi instalado e configurado agora, precisamos apenas gerar o binstub para executá-lo de forma consistente dentro do projeto.

Ainda dentro do container:

bundle binstub rspec-core

Isso cria o executável:

bin/rspec

Verifique:

ls -l bin/rspec

A partir daqui, podemos executar os testes usando:

bin/rspec

Verifique:

chmod +x bin/rspec

E execute:

bin/rspec

O resultado esperado é semelhante a:

No examples found.

Finished in 0.00... seconds
0 examples, 0 failures

Está tudo certo.

Ainda não temos testes RSpec.

11. Um pequeno atalho

Como vamos executar RSpec muitas vezes, podemos criar um alias.

Dentro do container:

echo 'alias br="bin/rspec"' >> ~/.bashrc
source ~/.bashrc

Agora:

br

é equivalente a:

bin/rspec

E podemos executar:

br spec/system/tarefas_spec.rb

ou:

br spec/system/tarefas_spec.rb -fd

Isso deixa o fluxo bastante confortável.

12. Criando o suporte para nossa DSL

Crie:

mkdir -p spec/support
touch spec/support/tarefas_steps.rb

O arquivo:

spec/support/tarefas_steps.rb

terá inicialmente:

module TarefasSteps

  # Arrange

  def visitar_a_pagina_principal
    visit root_path
  end
  def criar_uma_tarefa_concluida_anotada(nota)
    Tarefa.create!(nota: nota, feito: true)
  end
  def criar_uma_tarefa_nao_concluida_anotada(nota)
    Tarefa.create!(nota: nota, feito: false)
  end

  # Act & Assert

  def eu_devo_ver_a_secao_tarefas_concluidas
    expect(page).to have_css(".tarefas-concluidas")
  end
  def eu_devo_ver_na_secao_concluidas_a_tarefa(nota)
    within(".tarefas-concluidas") do
      expect(page).to have_content(nota)
    end
  end
  def eu_nao_devo_ver_na_secao_concluidas_a_tarefa(nota)
    within(".tarefas-concluidas") do
      expect(page).not_to have_content(nota)
    end
  end
end

A ideia é criar uma pequena DSL (Domain Specific Language) para nossos testes.

Em vez de escrever:

visit root_path

podemos escrever:

visitar_a_pagina_principal

Em vez de:

Tarefa.create!(nota: "Lavar o carro", feito: true)

podemos escrever:

criar_uma_tarefa_concluida_anotada("Lavar o carro")

A DSL funciona como uma ponte entre a linguagem técnica e a linguagem utilizada na especificação.

13. Configurando o RSpec para carregar a DSL

Abra:

spec/rails_helper.rb

Adicione (ou comente):

Rails.root.glob("spec/support/**/*.rb").sort_by(&:to_s).each do |file|
  require file
end

E configure o módulo:

RSpec.configure do |config|
  config.include TarefasSteps, type: :system
  config.include TarefasSteps, type: :request
end

Assim os métodos definidos em TarefasSteps estarão disponíveis nos testes RSpec.

14. Configurando o Cucumber

Como já estamos usando Cucumber, precisamos das dependências de teste.

No Gemfile:

group :test do
  gem "cucumber-rails", require: false
  gem "database_cleaner-active_record"
  gem "capybara"
  gem "selenium-webdriver"
end

Depois:

bundle install

Agora gere a estrutura do Cucumber:

rails generate cucumber:install

Se o gerador perguntar se deve sobrescrever um arquivo que já criamos, preserve nossa configuração respondendo n.

15. Primeiro teste: vamos fazer quebrar

Agora vem uma parte importante do TDD.

Execute:

bin/cucumber

Neste momento, esperamos que os testes falhem.

3 scenarios (3 undefined)
14 steps (14 undefined)
0m0.118s

Por quê?

Nossa view ainda possui apenas:

<tbody>
  <% @tarefas.each do |tarefa| %>

Não existem ainda:

tarefas-concluidas
tarefas-nao-concluidas

Portanto o teste deverá procurar uma seção que ainda não existe.

Esse é exatamente o comportamento que queremos no ciclo TDD:

🔴 RED
Teste escrito
     ↓
Teste falha

🟢 GREEN
Implementação mínima
     ↓
Teste passa


🔵 REFACTOR
Melhorar o código
     ↓
Testes continuam passando

16. Implementando a funcionalidade

Agora sim vamos alterar a view.

Substitua o conteúdo de:

app/views/tarefas/index.html.erb

por:

<h1>Listando Tarefas</h1>

<h2>Tarefas Concluídas</h2>
<section id="tarefas-concluidas" class="tarefas-concluidas">
  <table class="table">
    <thead>
      <tr>
        <th>Feito</th>
        <th>Nota</th>
        <th colspan="3"></th>
      </tr>
    </thead>
    <tbody>
      <% @tarefas.where(feito: true).each do |tarefa| %>
        <tr>
          <td>true</td>
          <td><%= tarefa.nota %></td>
          <td><%= link_to "Mostrar", tarefa %></td>
          <td><%= link_to "Editar", edit_tarefa_path(tarefa) %></td>
          <td>
            <%= link_to "Apagar", tarefa,
                method: :delete,
                data: { confirm: "Está certo?" } %>
          </td>
        </tr>
      <% end %>
    </tbody>
  </table>
</section>
<h2>Tarefas Não Concluídas</h2>
<section id="tarefas-nao-concluidas" class="tarefas-nao-concluidas">
  <table class="table">
    <thead>
      <tr>
        <th>Feito</th>
        <th>Nota</th>
        <th colspan="3"></th>
      </tr>
    </thead>
    <tbody>
      <% @tarefas.where(feito: false).each do |tarefa| %>
        <tr>
          <td>false</td>
          <td><%= tarefa.nota %></td>
          <td><%= link_to "Mostrar", tarefa %></td>
          <td><%= link_to "Editar", edit_tarefa_path(tarefa) %></td>
          <td>
            <%= link_to "Apagar", tarefa,
                method: :delete,
                data: { confirm: "Está certo?" } %>
          </td>
        </tr>
      <% end %>
    </tbody>
  </table>
</section>
<br>
<%= link_to "Nova Tarefa", new_tarefa_path, class: "btn btn-primary" %>

A implementação é deliberadamente simples.

A regra é:

@tarefas.where(feito: true)

para concluídas.

E:

@tarefas.where(feito: false)

para não concluídas.

17. Rodando novamente o Cucumber

Agora:

bin/cucumber

Erro de novo.

o Cucumber encontrou o .feature, mas não encontrou os step definitions.

A mensagem 3 scenarios (3 undefined) confirma que o problema ainda não é a view nem o Rails.

1. Confira se o arquivo existe

Dentro do container:

ls -la features/step_definitions

Você deve ter:

tarefas_steps.rb

Se não existir:

mkdir -p features/step_definitions
touch features/step_definitions/tarefas_steps.rb

2. Coloque este conteúdo no arquivo

features/step_definitions/tarefas_steps.rb:

# features/step_definitions/tarefas_steps.rb

Given("eu estou na pagina principal do app") do
  visit "/"
end

Given("existe uma tarefa concluida chamada {string}") do |nome|
  @tarefa_concluida = Tarefa.create!(nota: nome, feito: true)
end

Given("existe uma tarefa nao concluida chamada {string}") do |nome|
  @tarefa_nao_concluida = Tarefa.create!(nota: nome, feito: false)
end

When("eu envio essa tarefa concluida") do
  # Se já existe no banco, apenas recarrega a página
  visit "/"
end

When("eu envio essa tarefa nao concluida") do
  # Apenas recarrega para refletir a lista
  visit "/"
end

When("eu envio ambas as tarefas") do
  visit "/"
end

Then("eu devo ver {string} na secao {string}") do |texto, secao|
  within(:xpath, "//*[contains(@class,'#{secao.parameterize}') or contains(@id,'#{secao.parameterize}')]") do
    expect(page).to have_content(texto)
  end
end

Then("eu nao devo ver {string} na secao {string}") do |texto, secao|
  within(:xpath, "//*[contains(@class,'#{secao.parameterize}') or contains(@id,'#{secao.parameterize}')]") do
    expect(page).not_to have_content(texto)
  end
end

3. Rode novamente

bin/cucumber

Agora o resultado não deve mais ser undefined.

O resultado esperado:

3 scenarios (3 passed)
14 steps (14 passed)

🎉

Temos nosso primeiro ciclo TDD completo:

REQUISITO
   ↓
FEATURE
   ↓
TESTE FALHANDO
   ↓
IMPLEMENTAÇÃO
   ↓
TESTE PASSANDO

E o mais interessante:

não escrevemos o teste para validar a implementação.

Escrevemos o teste para validar o comportamento que o usuário pediu.

18. Agora entra o RSpec

O Cucumber respondeu:

O comportamento esperado está funcionando?

Agora queremos outra pergunta:

A implementação interna também está correta?

Vamos criar um system spec:

rails generate rspec:system tarefas

Isso criará:

spec/system/tarefas_spec.rb

Substitua o conteúdo por:

require "rails_helper"

RSpec.describe "Exibição de tarefas concluídas", type: :system do
  before do
    driven_by(:rack_test)
  end
  context "Ao visitar a página principal" do
    before do
      visitar_a_pagina_principal
    end
    it "eu devo ver a seção tarefas concluídas" do
      eu_devo_ver_a_secao_tarefas_concluidas
    end
    context "Teste 1: criar uma tarefa concluída anotada" do
      let!(:tarefa_concluida) do
        criar_uma_tarefa_concluida_anotada("Lavar o carro")
      end
      before do
        visitar_a_pagina_principal
      end
      it "eu devo ver na seção concluídas a tarefa" do
        eu_devo_ver_na_secao_concluidas_a_tarefa("Lavar o carro")
      end
    end
    context "Teste 2: criar uma tarefa não concluída anotada" do
      let!(:tarefa_nao_concluida) do
        criar_uma_tarefa_nao_concluida_anotada("Estudar Ruby")
      end
      before do
        visitar_a_pagina_principal
      end
      it "eu não devo ver na seção concluídas a tarefa" do
        eu_nao_devo_ver_na_secao_concluidas_a_tarefa("Estudar Ruby")
      end
    end
  end
end

Execute:

br spec/system/tarefas_spec.rb -fd

Esperamos:

Exibição de tarefas concluídas
  Ao visitar a página principal
    eu devo ver a seção tarefas concluídas
    Teste 1: criar uma tarefa concluída anotada
      eu devo ver na seção concluídas a tarefa
    Teste 2: criar uma tarefa não concluída anotada
      eu não devo ver na seção concluídas a tarefa

3 examples, 0 failures

19. O que acabamos de construir?

Temos agora duas perspectivas para o mesmo requisito.

Cucumber

Expressa o comportamento:

Then eu devo ver "Lavar o carro" na secao "tarefas-concluidas"

RSpec

Verifica o comportamento através da aplicação:

expect(page).to have_content("Lavar o carro")

Podemos enxergar assim:

REQUISITO
                 │
                 ▼
        ┌─────────────────┐
        │    CUCUMBER     │
        │                 │
        │ O que o usuário │
        │ espera?         │
        └────────┬────────┘
                 │
                 ▼
             APLICAÇÃO
                 │
                 ▼
        ┌─────────────────┐
        │     RSpec       │
        │                 │
        │ A implementação │
        │ funciona?       │
        └─────────────────┘

Cucumber fala. RSpec garante.

E agora tudo isso está rodando sobre a mesma infraestrutura que construímos no episódio anterior:

Docker
  │
  ├── app
  │    ├── Ruby
  │    ├── Rails
  │    ├── Cucumber
  │    └── RSpec
  │
  └── db
       └── PostgreSQL

Isso deixa a série muito mais coerente: o primeiro episódio constrói a infraestrutura reproduzível; este episódio usa essa infraestrutura para desenvolver com testes.

Obrigado por ler!
Agradeço por você dedicar um tempo para explorar este guia — seu apoio significa muito. Desejo muito sucesso no seu projeto!

Cucumber + RSpec: Testes que Falam e Testes que Garantem was originally published in Jungletronics on Medium, where people are continuing the conversation by highlighting and responding to this story.

Criar uma aplicação Rails pode parecer simples até chegarmos à infraestrutura: versão do Ruby, Bundler, PostgreSQL, dependências nativas, permissões, portas, volumes e configuração do banco.

A proposta deste tutorial é justamente tirar essa complexidade do caminho.

Até aqui, construímos uma infraestrutura na qual o sistema operacional precisa apenas do Docker. Ruby, Rails, PostgreSQL, gems e suas dependências ficam isolados nos containers.

A abstração conquistada é simples:

SEU COMPUTADOR
      │
      │ Docker
      ▼
┌──────────────────────────────┐
│          Aplicação           │
│     Ruby + Rails + Gems      │
└──────────────┬───────────────┘
               │
               │ rede Docker
               ▼
┌──────────────────────────────┐
│         PostgreSQL           │
│          Banco de dados      │
└──────────────────────────────┘

Você desenvolve no seu projeto normalmente. O Docker cuida da infraestrutura.

Isso significa que, para começar outro projeto Rails, não precisamos repetir toda a preparação do ambiente. A mesma estrutura pode ser reutilizada, alterando apenas as configurações específicas da aplicação.

E há outro detalhe importante: o código continua no seu computador. Os containers fornecem o ambiente de execução, mas o projeto permanece acessível pelo editor, pelo Git e pelas ferramentas que você já utiliza.

O resultado é uma separação clara:

Seu computador fornece o Docker.
O projeto fornece a aplicação.
Os containers fornecem o ambiente.

O objetivo aqui não é construir uma infraestrutura sofisticada. É chegar ao ponto em que Rails + PostgreSQL simplesmente funcionem, de forma reproduzível, isolada e fácil de entender.

No final, incluímos uma nota técnica avaliando as decisões adotadas e os pontos que poderiam ser aprimorados em um ambiente mais robusto ou de produção.

Resultado final

Ubuntu
 └── Docker Engine
      ├── Rails 8.1.x + Ruby 3.4.x
      └── image: postgres:17

Neste cenário, Ruby, Rails e PostgreSQL são executados dentro dos containers, sem necessidade de instalação dessas ferramentas diretamente no Ubuntu.

E, na sua configuração final, teremos três boas medidas de segurança: Rails sem root, remoção de capabilities + no-new-privileges no app, e PostgreSQL sem porta publicada no host.

1. Criar o projeto

Escolha um diretório para seus projetos Rails:

mkdir -p ~/Documents/projects/rails
cd ~/Documents/projects/rails

Crie um diretório para a aplicação:

mkdir meu-projeto
cd meu-projeto

Crie os arquivos necessários:

touch compose.yml .env Dockerfile
code .

Neste tutorial, meu-projeto é apenas um nome de exemplo. Substitua pelo nome da sua aplicação.

Como os três se relacionam:

Pode pensar assim:

.env
  │
  │ fornece valores
  ▼
compose.yml
  │
  │ define os containers
  ▼
Dockerfile
  │
  │ constrói a imagem do app
  ▼
┌─────────────────────────┐
│ Docker                  │
│                         │
│  ┌───────────────────┐  │
│  │ app               │  │
│  │ Rails 8.1.3.1     │  │
│  │ Ruby 3.4.10       │  │
│  │ usuário: app      │  │
│  └─────────┬─────────┘  │
│            │            │
│          db:5432        │
│            │            │
│  ┌─────────▼─────────┐  │
│  │ PostgreSQL 17     │  │
│  │ volume persistente│  │
│  └───────────────────┘  │
└─────────────────────────┘

2. Credenciais no .env

Como o rails new já cria o .gitignore da aplicação, não precisamos criar esse arquivo manualmente. Basta garantir que o .env esteja listado nele.

Crie ou edite o .env:

UID=1000
GID=1000

APP_PORT=3000
DB_PORT=5432

POSTGRES_DB=app_development
POSTGRES_USER=app
POSTGRES_PASSWORD=app_dev

DATABASE_TEST_NAME=app_test

1. env — configurações e variáveis

É onde ficam os valores que podem variar conforme o ambiente, sem precisar alterar o compose.yml.

No seu caso:

• UID / GID → definem o usuário/grupo Linux usado pelo container da aplicação.
• APP_PORT=3000 → porta da aplicação Rails.
• DB_PORT=5432 → porta padrão do PostgreSQL; atualmente não está sendo publicada no host, apesar de estar definida aqui.
• POSTGRES_DB → nome do banco.
• POSTGRES_USER → usuário do PostgreSQL.
• POSTGRES_PASSWORD → senha do PostgreSQL.
• DATABASE_TEST_NAME → nome previsto para o banco de testes.

Importante: o .env não deveria ser versionado no Git quando contém senhas reais. Normalmente fica no .gitignore.

3. compose.yml

services:

  app:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        UID: ${UID:-1000}
        GID: ${GID:-1000}

    working_dir: /app

    volumes:
      - .:/app
      - bundle:/usr/local/bundle

    ports:
      - "127.0.0.1:${APP_PORT:-3000}:3000"

    environment:
      DATABASE_HOST: db
      DATABASE_NAME: ${POSTGRES_DB}
      DATABASE_USER: ${POSTGRES_USER}
      DATABASE_PASSWORD: ${POSTGRES_PASSWORD}

    depends_on:
      - db

    # Hardening
    security_opt:
      - no-new-privileges:true

    cap_drop:
      - ALL

    stdin_open: true
    tty: true

  db:
    image: postgres:17

    environment:
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}

    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  bundle:
  postgres_data:

2. compose.yml — orquestra os containers

É o arquivo que diz ao Docker como montar e executar o ambiente.

Você tem dois serviços:

app

É o container da aplicação Rails.

Ele:

• constrói a imagem usando o Dockerfile;
• monta seu projeto em /app;
• mantém as gems no volume bundle;
• publica Rails somente em:

127.0.0.1:3000

• encontra o PostgreSQL pelo nome:

db

• recebe as credenciais do .env;
• inicia depois do serviço db;
• possui alguns controles de segurança:

security_opt:
  - no-new-privileges:true

cap_drop:
  - ALL

Ou seja, você já colocou uma camada de hardening do container da aplicação.

db

É o container do PostgreSQL 17.

Ele:

• usa a imagem oficial postgres:17;
• recebe banco, usuário e senha do .env;
• grava os dados permanentemente no volume:

postgres_data:/var/lib/postgresql/data

E um ponto importante da configuração atual:

db:
  ...
  volumes:
    - postgres_data:/var/lib/postgresql/data

Não existe ports: no db.

Portanto, o PostgreSQL não está exposto diretamente no host. Ele fica acessível para o app pela rede interna do Docker através de:

db:5432

Por essas configuracoes:

# Hardening
    security_opt:
      - no-new-privileges:true

    cap_drop:
      - ALL

    stdin_open: true
    tty: true

O processo Rails roda como app, sem capabilities e sem possibilidade de ganhar novos privilégios:

Para entender o usuário do container e o alinhamento de UID/GID com o host, consulte a Nota ¹.

Como seu compose.yml já está pronto, precisamos apenas colocar de volta o Dockerfile bootstrap, aquele que existe antes do rails new:

Dockerfile

FROM ruby:3.4.10-bookworm

ARG UID=1000
ARG GID=1000

RUN apt-get update -qq && \
    apt-get install -y --no-install-recommends \
    build-essential \
    libpq-dev \
    libyaml-dev \
    git \
    curl && \
    rm -rf /var/lib/apt/lists/*

# Cria o grupo e usuário da aplicação
RUN groupadd --gid ${GID} app && \
    useradd --uid ${UID} \
             --gid ${GID} \
             --create-home \
             --shell /bin/bash \
             app

WORKDIR /app

RUN gem install bundler -v 2.6.9
RUN gem install rails -v 8.1.3.1

# Permissões
RUN chown -R app:app /app /usr/local/bundle

USER app

# A aplicação será montada pelo docker compose em /app.
# As gems serão instaladas no volume bundle.
CMD ["bin/rails", "server", "-b", "0.0.0.0"]

Depois:

docker compose build app

O Dockerfile define como criar o ambiente onde o Rails vai executar.

A sequência é basicamente:

Ruby 3.4.10
      ↓
Debian Bookworm
      ↓
instala ferramentas/dependências
      ↓
cria usuário "app"
      ↓
instala Bundler
      ↓
instala Rails 8.1.3.1
      ↓
ajusta permissões
      ↓
USER app
      ↓
inicia Rails

Um ponto importante de segurança está aqui:

USER app

Isso faz com que o processo principal do Rails não execute como root.

E aqui:

RUN groupadd --gid ${GID} app
useradd --uid ${UID} ...

você ainda alinha o UID/GID do usuário dentro do container com o usuário do ambiente Linux.

4. Criar o Rails

O UID e o GID já estão definidos no .env:

UID=1000
GID=1000

Crie a aplicação Rails dentro do container:

docker compose run --rm app \
  rails new . \
  --database=postgresql \
  --skip-test

Esse comando cria a aplicação Rails já configurada para usar PostgreSQL.

Por que o .?
O ponto indica o diretório atual. Como o projeto está montado no container em /app, o Rails cria os arquivos diretamente nesse diretório — que, por sua vez, corresponde à pasta do projeto no computador.

Durante a execução, o Rails poderá perguntar se deve sobrescrever arquivos que já existem.

Quando perguntar sobre o Dockerfile, responda n (não).

Por enquanto, vamos manter o nosso Dockerfile como está, pois ele foi preparado especificamente para este ambiente Docker. Não precisamos substituí-lo pelo Dockerfile gerado automaticamente pelo Rails.

5. Configurar PostgreSQL

O Rails deve utilizar PostgreSQL, portanto o Gemfile deve conter:

gem "pg", "~> 1.5"

e não deve depender de:

gem "sqlite3"

database.yml

O Rails gera um database.yml bastante completo, incluindo configurações voltadas a diferentes cenários de produção.

Como nosso objetivo neste tutorial é construir um ambiente de desenvolvimento simples com Docker Compose, vamos utilizar uma configuração menor e baseada em variáveis de ambiente.

A principal diferença é que o Rails precisa saber que o PostgreSQL está em outro container. No Docker Compose, o serviço é chamado db, portanto esse será o nosso host:

Rails → db:5432 → PostgreSQL

Substitua o conteúdo de config/database.yml por:

default: &default
  adapter: postgresql
  encoding: unicode
  host: <%= ENV.fetch("DATABASE_HOST", "db") %>
  username: <%= ENV.fetch("DATABASE_USER", "app") %>
  password: <%= ENV.fetch("DATABASE_PASSWORD", "app_dev") %>
  pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5) %>

development:
  <<: *default
  database: <%= ENV.fetch("DATABASE_NAME", "app_development") %>
test:
  <<: *default
  database: <%= ENV.fetch("DATABASE_TEST_NAME", "app_test") %>
production:
  <<: *default
  database: <%= ENV.fetch("DATABASE_NAME", "app_production") %>

Dessa forma, o Rails não depende de valores específicos gravados no código. As configurações são fornecidas pelo ambiente e podem ser alteradas sem modificar a aplicação.

Os valores vêm do ambiente definido no .env.

6. Instalar as gems

Depois que o Rails foi criado e o Gemfile está configurado:

docker compose run --rm app bundle install

As gems serão instaladas no volume Docker:

bundle:/usr/local/bundle

Assim, elas não precisam ser reinstaladas a cada recriação do container.

7. Criar o banco

Com as gems instaladas e o PostgreSQL em execução, podemos criar os bancos utilizados pelo Rails:

docker compose run --rm app \
  bundle exec rails db:create

O Rails utiliza as configurações do database.yml para se conectar ao serviço db e criar os bancos de desenvolvimento e testes.

Se os bancos já existirem, o Rails apenas informará:

Database 'app_development' already exists
Database 'app_test' already exists

Isso significa que a conexão está funcionando e que os bancos já estão disponíveis.

8. Subir a aplicação

Agora:

docker compose up --build

A aplicação estará disponível em:

http://localhost:3000

Se quiser executar em segundo plano:

docker compose up -d --build

E conferir:

docker compose ps

Pronto. 🎯

Observe que não definimos nomes de containers, nomes de projeto ou credenciais diretamente no Dockerfile. O ambiente é configurado pelo .env, enquanto o nome da instância do Compose pode ser escolhido com -p.

Por exemplo:

docker compose -p meu-projeto up -d

ou simplesmente:

docker compose up -d

Isso reforça justamente a ideia que queremos transmitir: Rails + PostgreSQL + Docker, mas sem amarrar a infraestrutura ao nome de uma aplicação específica.

O que você ganha com essa arquitetura

O projeto fica independente da instalação Ruby da máquina:

┌─────────────────────────────────────┐
│              Ubuntu                 │
│                                     │
│  Docker Engine                      │
│       │                             │
│       ├── Rails / Ruby              │
│       │      │                      │
│       │      └── /app ← projeto     │
│       │                             │
│       └── PostgreSQL                │
│              │                      │
│              └── postgres_data      │
└─────────────────────────────────────┘

Na máquina host você não precisa de:

Ruby
Rails
Bundler
PostgreSQL

Você precisa basicamente de:

Docker
Git
VS Code

Comandos que você realmente precisa lembrar

No dia a dia:

export UID=$(id -u)
export GID=$(id -g)

docker compose up

Para entrar no container:

docker compose exec app bash

Para Rails:

docker compose run --rm app bundle exec rails ...

significa:

“Crie temporariamente um container baseado no serviço app, entre nele, execute bundle exec rails db:migrate e depois remova o container."

Mas você não precisa decorar isso

Se o seu objetivo é trabalhar normalmente dentro do container, existe uma forma mais confortável:

docker compose exec app bash

Aí você entra no container:

root@container:/app#

E pode executar normalmente:

bundle exec rails db:migrate
bundle exec rails console
bundle exec rails routes

Ou simplesmente:

rails db:migrate

Para parar:

docker compose down

Para apagar também os dados do PostgreSQL:

docker compose down -v

⚠️ Esse último apaga o volume do banco.

A ideia central

O tutorial inteiro pode ser resumido em quatro decisões:

1. Ruby/Rails dentro do Docker.

2. PostgreSQL em outro container.

3. Credenciais no .env, não no código.

4. Projeto montado como volume, para editar normalmente pelo VS Code.

Foi isso que construímos — só que, durante a montagem, fomos resolvendo cada detalhe individualmente.

Note ¹

Quando um diretório do host é montado no container, alinhar UID/GID entre host e container evita a maior parte dos problemas de permissões

Entendendo esses conceitos :

UID 
GID 
/etc/passwd 
usuário do host 
usuário do container 
permissões de volume

Esses conceitos parecem complicados porque Docker mistura dois mundos Linux: o seu computador e o container. A chave para entender tudo é esta:

Linux não identifica usuários pelo nome. Identifica principalmente pelo número UID.

1. UID — User ID

É o número que identifica um usuário no Linux.

Por exemplo, no seu Ubuntu:

id

pode retornar:

uid=1000(j3) gid=1000(j3) groups=1000(j3),...

Aqui:

j3  → nome
1000 → UID

Para o Linux, o que realmente importa é:

UID 1000

O nome j3 é uma associação feita pelo sistema.

2. GID — Group ID

É a mesma ideia, mas para grupos.

Por exemplo:

gid=1000(j3)

Significa:

grupo: j3
GID:   1000

Um arquivo Linux pode ter:

UID → dono
GID → grupo

Por exemplo:

-rw-r--r-- 1000 1000 Gemfile

Significa:

dono  = UID 1000
grupo = GID 1000

3. /etc/passwd

Agora vem a parte que explica:

I have no name!

O arquivo:

cat /etc/passwd

contém o cadastro dos usuários conhecidos por aquele sistema Linux.

Por exemplo:

rails:x:1000:1000::/home/rails:/bin/bash

Isso diz:

nome       = rails
UID        = 1000
GID        = 1000
home       = /home/rails
shell      = /bin/bash

Imagine que o processo dentro do container esteja rodando como:

UID 1000

mas /etc/passwd não tenha ninguém com UID 1000.

O Linux sabe:

“Existe alguém com UID 1000.”

Mas não sabe:

“Qual é o nome dessa pessoa?”

Por isso:

I have no name!

4. Usuário do host

Seu host é o Ubuntu onde você está trabalhando.

Por exemplo:

Ubuntu
└── usuário: j3
    └── UID: 1000

Quando você cria:

touch teste.txt

o arquivo pertence ao seu UID:

UID 1000

5. Usuário do container

O container possui seu próprio ambiente Linux.

Ele também pode ter:

rails
UID 1000

Perceba algo importante:

O rails do container não precisa ser a mesma pessoa que j3 do Ubuntu.

São ambientes diferentes.

Mas ambos podem ter:

UID 1000

E isso é justamente o que queremos aproveitar.

HOST                         CONTAINER

 j3                           rails
 UID 1000                     UID 1000
    │                             │
    └──────── mesmo número ───────┘

6. Permissões de volume

Aqui está a razão de toda essa preocupação.

Imagine que seu projeto esteja no Ubuntu:

/home/j3/projeto

e o Docker monte essa pasta dentro do container:

volumes:
  - .:/app

Agora temos:

Ubuntu                         Container

/home/j3/projeto  ──────────→  /app
       │                          │
       │                          │
       └── arquivos               └── mesmos arquivos

O container não recebe uma cópia independente.

Ele está vendo os mesmos arquivos do host.

Imagine que:

Gemfile
dono = UID 1000

Se dentro do container você estiver como:

root
UID 0

o root normalmente consegue alterar o arquivo.

Mas se criar arquivos como outro UID, você pode acabar com uma bagunça de permissões.

E é aí que entra o nosso rails

Queremos:

HOST

j3
UID 1000
   │
   │
   ▼
Docker
   │
   ▼
CONTAINER
rails
UID 1000

Assim, quando o Rails dentro do container criar:

tmp/
log/
db/
algum_arquivo.rb

esses arquivos terão:

UID 1000

No host, o Ubuntu reconhece esse UID como seu usuário j3.

Resultado:

menos problemas de permissão.

E agora fica fácil entender o seu erro

Você estava com:

root@d390de9a0bad:/app#

Isso significa:

root
UID 0

E o Git encontrou arquivos em /app pertencentes a outro UID.

Então ele disse:

detected dubious ownership

Quando apareceu:

I have no name!@d390de9a0bad:/app$

era outra situação:

UID 1000
       ↓
não existe esse UID no /etc/passwd do container
       ↓
I have no name!

Portanto, nossa solução de criar:

rails
UID 1000
GID 1000

resolve duas coisas ao mesmo tempo:

HOST                  CONTAINER

j3 / UID 1000        rails / UID 1000
               │                     │
               └─────────┬───────────┘
                         │
                    /app (volume)
                         │
                         ▼
                  mesmas permissões

A frase para guardar

Nome é para humanos. UID/GID são para o Linux.

E no Docker:

Quando um diretório do host é montado no container, alinhar UID/GID entre host e container evita a maior parte dos problemas de permissões.

Note ²

Um princípio importante do projeto:

O nome da aplicação, o nome do banco e o nome do projeto Docker Compose são coisas diferentes.

Configuração genérica do ambiente

Crie um arquivo .env na raiz do projeto:

UID=1000
GID=1000

APP_PORT=3000
DB_PORT=5432
POSTGRES_DB=app_development
POSTGRES_USER=app
POSTGRES_PASSWORD=app_dev

O .env não deve conter tarefas nem meu-projeto. Assim, o mesmo Dockerfile e compose.yml podem ser reutilizados em qualquer aplicação Rails.

Nome do projeto Docker Compose

O nome do projeto é definido no momento em que você sobe o ambiente:

docker compose -p meu-projeto up -d

Por exemplo, outro projeto poderia usar:

docker compose -p minha-api up -d

Isso apenas muda os nomes dos recursos criados pelo Docker:

meu-projeto-app-1
meu-projeto-db-1

ou:

minha-api-app-1
minha-api-db-1

O .env continua igual.

Por que separar as coisas?

                 PROJETO DOCKER
                       │
             -p meu-projeto
                       │
          ┌────────────┴────────────┐
          │                         │
        app                        db
          │                         │
     Rails :3000              PostgreSQL :5432
          │                         │
          └──────────┬──────────────┘
                     │
                   .env
                     │
          configuração do ambiente

Essa separação torna o ambiente reproduzível e reutilizável.

O Dockerfile define como construir.
O compose.yml define como executar.
O .env define as configurações.
O -p define a identidade daquela instância do Compose.

Nota técnica

Nota: 9,0/10

A solução está tecnicamente bem estruturada e adequada para um tutorial introdutório de Rails + PostgreSQL + Docker.

Pontos fortes:

• Ruby/Rails executados dentro do container, sem dependências instaladas diretamente no Ubuntu.
• PostgreSQL isolado em container próprio.
• Comunicação entre Rails e PostgreSQL pelo nome do serviço db.
• Persistência do PostgreSQL por volume nomeado.
• Persistência do código por bind mount.
• Usuário não-root dentro do container.
• UID/GID configuráveis para evitar problemas de permissões no Linux.
• Bundler com volume próprio e permissões corrigidas.
• Portas configuráveis pelo .env.
• Configuração do banco desacoplada do nome da aplicação.
• Estrutura reproduzível com Dockerfile + compose.yml + .env.

Pontos que merecem atenção:

• postgres:17 é uma dependência externa que pode mudar de comportamento entre versões menores; para um ambiente de produção, seria recomendável fixar uma versão mais específica.
• O .env deve obrigatoriamente ficar fora do Git.
• depends_on garante ordem de inicialização, mas não garante que o PostgreSQL esteja pronto para aceitar conexões. Para um tutorial simples isso é aceitável; em um ambiente mais robusto, entraria um healthcheck.
• gem install bundler durante o build torna a versão do Bundler dependente do momento em que a imagem é construída. Para máxima reprodutibilidade, pode-se fixar a versão.
• A imagem poderia ser otimizada posteriormente, mas isso não é necessário para o objetivo didático.

Veredito

Para desenvolvimento local e finalidade didática: 9/10.

O mais importante é que o tutorial não está apenas “funcionando”: ele está ensinando alguns problemas reais de Docker em Linux — permissões, UID/GID, volumes, usuário não-root e comunicação entre containers.

🚀 NOVO EPISÓDIO

Cucumber + RSpec: testes que falam e testes que garantem.

Do comportamento descrito em linguagem simples ao teste automatizado de verdade.

Neste episódio, avançamos na nossa infraestrutura Rails + PostgreSQL + Docker e começamos a transformar requisitos em testes.

🧪 Cucumber → especificação que fala.
✅ RSpec → teste que garante.

Do “o sistema deve fazer” ao “o sistema realmente faz”.

Se a intenção é zerar o Docker localmente, removendo containers, imagens, volumes, redes não utilizadas e cache, use esta sequência.

Atenção: isso remove também volumes, portanto pode apagar bancos PostgreSQL persistidos localmente.

1. Parar e remover todos os containers

docker rm -f $(docker ps -aq) 2>/dev/null

2. Remover todas as imagens

docker rmi -f $(docker images -aq) 2>/dev/null

3. Remover todos os volumes

docker volume rm $(docker volume ls -q) 2>/dev/null

4. Remover redes criadas pelo usuário

docker network prune -f

5. Limpar absolutamente tudo que restou

docker system prune -a --volumes -f

6. Conferir

docker ps -a
docker images
docker volume ls
docker network ls

Você deve ficar basicamente com as redes padrão:

bridge
host
none

e sem containers, imagens ou volumes.

Rails + PostgreSQL + Docker was originally published in Jungletronics on Medium, where people are continuing the conversation by highlighting and responding to this story.

O Categoric¹ é uma aplicação moderna construída para transformar uma ideia em um sistema real, seguro e organizado. Por trás de cada tela existe um fluxo de comunicação que conecta o navegador, o Nginx, o Docker, o Gunicorn, o Flask e o PostgreSQL.

Neste tutorial, você vai acompanhar esse caminho como se estivesse explorando os bastidores de uma aplicação: entender como uma requisição entra, como os serviços conversam entre si e como os dados são processados até a resposta voltar para o usuário. Mais do que conhecer o Categoric, você verá como uma arquitetura profissional é construída, protegida e documentada para continuar compreensível no futuro.

O fluxo completo funciona assim:

Copiar

Cliente/Navegador
        │
        │  Requisição HTTP pela porta 80
        ▼
Docker
        │
        │  Rede interna dos containers
        ▼
Nginx, Proxy reverso
        │
        │  Encaminha para o serviço web na porta 8000
        ▼
Gunicorn
        │
        │  Executa a aplicação Python através do protocolo WSGI
        ▼
Flask
        │
        │  Processa rotas, regras de negócio e autenticação
        ▼
PostgreSQL
        │
        │  Consulta ou grava dados na porta interna 5432
        ▼
Flask → Gunicorn → Nginx → Cliente

1. Cliente envia a requisição

O usuário acessa uma página ou executa uma ação no navegador. O navegador envia uma requisição HTTP, por exemplo:

GET /dashboard HTTP/1.1
Host: exemplo.com

A requisição chega ao servidor pela porta pública 80. O cliente não acessa diretamente o Flask, o Gunicorn ou o PostgreSQL.

2. Docker fornece o ambiente

O Docker mantém os componentes isolados em containers e conecta os serviços por uma rede interna. Neste projeto, o Docker não é exatamente uma etapa de processamento da requisição; ele fornece o ambiente e a comunicação entre:

categoric-nginx-1
categoric-web-1
categoric-db-1

Somente o container do Nginx possui a porta 80 publicada no host. As portas 8000 e 5432 permanecem internas à rede Docker.

3. Nginx recebe a requisição

O Nginx atua como proxy reverso. Ele recebe a requisição externa, pode aplicar regras de rota, servir arquivos estáticos e encaminhar requisições dinâmicas para o container web.

O encaminhamento ocorre internamente, de forma semelhante a:

Nginx:80 → categoric-web-1:8000

O Nginx também pode adicionar informações como:

X-Real-IP
X-Forwarded-For
X-Forwarded-Proto
Host

Esses cabeçalhos permitem que a aplicação saiba o host acessado, o protocolo original e o endereço do cliente.

4. Gunicorn recebe a chamada WSGI

O Gunicorn é o servidor de aplicação. Ele recebe a solicitação encaminhada pelo Nginx na porta interna 8000 e distribui o trabalho entre seus workers.

A estrutura atual é:

Gunicorn master
├── worker 1
└── worker 2

O processo principal supervisiona os workers. Um worker recebe a requisição e chama a aplicação Flask por meio do protocolo WSGI.

5. Flask processa a lógica

O Flask interpreta:

• método HTTP, como GET ou POST;
• caminho, como /login ou /dashboard;
• parâmetros;
• cookies e sessão;
• cabeçalhos;
• regras de autenticação;
• validações;
• regras de negócio.

Por exemplo:

GET /dashboard
        │
        ▼
Rota Flask /dashboard
        │
        ▼
Consulta dados no PostgreSQL
        │
        ▼
Renderiza página ou retorna JSON

Se a requisição exigir dados, o Flask abre ou reutiliza uma conexão com o PostgreSQL usando a rede interna do Docker.

6. PostgreSQL responde à aplicação

O PostgreSQL escuta na porta interna 5432. Ele não deve ficar publicado diretamente para a Internet.

A comunicação ocorre assim:

categoric-web-1 → categoric-db-1:5432

O banco processa consultas, insere dados ou atualiza registros e devolve o resultado ao Flask. Depois que a aplicação termina a operação, a conexão pode ser devolvida ao pool ou encerrada conforme a configuração utilizada.

7. A resposta retorna pelo caminho inverso

Depois que o Flask produz a resposta, o fluxo volta:

Flask
  ↓
Gunicorn
  ↓
Nginx
  ↓
Cliente

Exemplo de resposta:

HTTP/1.1 200 OK
Content-Type: text/html

O Nginx então entrega a resposta ao navegador. Em caso de erro, o status pode ser, por exemplo:

404, rota não encontrada
500, erro interno da aplicação
502, Nginx não conseguiu falar com o Gunicorn
503, serviço indisponível

Resumo das responsabilidades

Note ¹

Category — Ecossistema de Aprendizagem Ética, iniciativa que reúne livro, plataforma digital, cursos, palestras e tutoria acadêmica em uma experiência contínua de aprendizagem. O objetivo é ensinar a Reforma Tributária do Consumo e demonstrar, na prática, como a Inteligência Artificial pode ser aplicada de forma ética, transparente e responsável na produção e disseminação do conhecimento

Por Trás do Categoric: O Caminho Secreto de Cada Requisição was originally published in Jungletronics on Medium, where people are continuing the conversation by highlighting and responding to this story.