Why the World Runs on Power Laws: A Data Science Guide to Heavy Tails

Generated by AI

Understanding PDFs, CDFs, and cCDFs in Continuous Distributions

When we work with data, we often ask:

What is the probability of seeing a value in this range? or What are the chances the outcome is less than a certain number?

To answer these, we need to understand three key tools of continuous probability distributions:

• The Probability Density Function (PDF)
• The Cumulative Distribution Function (CDF)
• The Complementary CDF (cCDF)

1. Probability Density Function (PDF)

From “3-Minute Data Science”

The PDF, usually written as p(x), describes the shape of likelihoods.
The probability that the random variable X falls between two numbers a and b is the area under the curve.

Rules of PDFs:
1. p(x)≥0 → no negative probabilities.
2. The total area under the curve is 1.

2. Cumulative Distribution Function (CDF)

The CDF, written as F(x), answers:

What is the probability that X is less than or equal to some value x?

From “3-Minute Data Science”

• At very small x, the CDF is close to 0.
• As x grows large, the CDF approaches 1

If we take the derivative of the CDF, you get back the PDF:

3. Complementary CDF (cCDF)

Sometimes, we want the opposite question:

What is the probability that X is greater than some value x?

That’s the cCDF:

Example:
If F(70)=0.8, that means 80% of values are below 70.
So, 20% are above 70 → F(70)=0.2.

So, recapping the PDF, CDF and CCDF, we can tell that

• PDF → tells us the shape of likelihoods.
• CDF → probability that X is less than or equal to a value.
• cCDF → probability that X is greater than a value.

Continuous Distributions: Gaussian vs. Exponential

When we talk about continuous random variables, two of the most common distributions we’ll encounter are the Gaussian (Normal) and the Exponential. Both describe uncertainty, but in very different ways.

Gaussian Distribution (Normal Distribution)

The Gaussian is the famous bell curve.

μ = mean (the center of the curve)
σ = standard deviation (how wide or narrow the curve is)

Generated by Python

Generated by Python

PDF: The bell-shaped curve. The shaded area under it gives the probability of being in a certain interval.

CDF: The S-shaped curve. It shows the accumulated probability as we move to the right.

• Starts near 0 (far left).
• Crosses 0.5 at the mean.
• Approaches 1 as we move far right.

The Gaussian is symmetric, smooth, and describes “spread around an average.”

Exponential Distribution

The Exponential distribution models the time (or distance) between random events.

λ = rate parameter (higher λ means events happen more quickly).

Think about waiting times — how long until the next bus arrives, or the time between radioactive decays.

Generated by Python

Generated by Python

Generated by Python

PDF (left graph): Starts high at x=0, then decays quickly.

• Most probability is near zero → short waiting times are common.
• Longer waits are less likely.
• The curve starts high at x=0.
• Suppose buses arrive on average every 5 minutes. Then λ=1/5=0.2.
  At x=1 minute, the PDF is high → high chance bus arrives soon.
  At x=20 minutes, the PDF is near zero → very small chance we’ll wait that long.
• Then it decays quickly as x increases.
  This means

CDF (middle graph): Rises quickly at first, then flattens near 1.

• Shows that most of the probability is concentrated early on.
• Starts at 0 and rises quickly toward 1.
• At each x, it shows the probability the event has happened by time x.
• Example:
• At x=5, let’s say F(5)=0.63.
  That means there’s a 63% chance the bus has already arrived within 5 minutes. At x=15, let’s say F(15)=0.95. So there’s a 95% chance the bus has come by then.

cCDF (right graph): This is Pr(X>x), the probability of waiting longer than x.

• This is Pr(X>x): the probability the wait is longer than x.
• Example:
  At x=5, CCDF might be 0.37 → 37% chance we’re still waiting after 5 minutes.
  At x=15, CCDF might be 0.05 → only a 5% chance we’re still waiting.

Exponential distribution recap

• PDF = how likely different waiting times are.
• CDF = probability the event happens before a given time.
• CCDF = probability the wait lasts longer than a given time.

Discrete Distributions Explained

Perfect — now we’ve moved from continuous distributions to discrete distributions. Not all random variables can take on infinitely many values like height or weight. Sometimes, they can only take specific, separate values.

Example: The outcome of rolling a die 🎲 → you only get {1, 2, 3, 4, 5, 6}.

A discrete random variable Xi can only take values from a set of distinct points (e.g., integers).

For discrete variables, instead of a PDF, we use a Probability Mass Function (PMF).

This gives the probability that the variable takes a specific value x.

Rules:

1. p(x)≥0 → no negative probabilities.
2. The probabilities add up to 1:

Example: For a fair 6-sided die,

Generated by Python

Just like for continuous variables, we can define a CDF:

This means:

Add up all probabilities for values less than or equal to xxx.
The result is the probability the outcome is at most xxx.

Generated by Python

Empirical Distributions

So far, we’ve looked at theoretical distributions (Gaussian, Exponential, etc.). But in practice, we often work with empirical distributions — those we build directly from observed data.

Generated by Python

Here’s the first empirical graph - Heights of Males.

• The data forms a bell-shaped curve (Gaussian-like).
• Most men are around 175 cm, with fewer very short or very tall.
• Classic Normal distribution in real life.

Generated by Python

Here’s the second empirical graph — Speeds of Cars. Another example of a Normal distribution, this time shaped by traffic rules and human driving behavior.

• Distribution also looks like a bell curve (Gaussian-like).
• Most cars drive near the average speed of 70 km/h.
• Very few are extremely slow or extremely fast.

Let’s now look at some other economic data. For example, the population

Generated by Python

Linear Scale:

• Here, the x-axis is population in regular numbers.
• The y-axis is the percentage of cities in each range.
• Result: we see a huge spike for small cities, and then the curve almost touches zero. The majority of the cities has very small population. But more cities are with very large population.

Generated by Python

Log-Log Scale:

• Both axes are logarithmic.
• That means instead of showing population as 10k, 20k, 30k, … we show 10,000 → 100,000 → 1,000,000 → 10,000,000.
• On this scale, the big cities finally show up, and the tail of the curve looks like a straight line.
• A straight line on a log-log plot means the data follows a power-law distribution:
  Many small, few huge.
• If we plot city populations on a normal (linear) scale, you mostly see the small cities.
• If we plot on a log-log scale, we uncover the hidden pattern — a power law.

Why Does the Power Law Matter in Real Life?

The power-law pattern — “many small, few huge” — shows up all over the world, not just in city populations. Understanding it helps us see why things are often unequal but predictable.

Cities:
Most cities are small towns.
A few mega-cities dominate the world’s population (Tokyo, Delhi, Shanghai).

Wealth Distribution:
Many people have modest savings.
A few people are billionaires.
Same long tail idea — inequality isn’t random, it follows a power law.

Internet & Social Media:
Most websites get very little traffic.
A few (Google, YouTube, Facebook) get the majority.

Music, Books, Videos:
Most songs/books get few listeners/readers.
A few hits go viral and get millions of streams.
That’s why we keep hearing the same top hits everywhere.

So, Power-law distributions reveal a kind of hidden order:

• They explain why success, size, and attention are unevenly distributed.
• They remind us that the world is not flat or equal — it’s heavy-tailed.
• And they show us why looking at data on log-log scale matters: it uncovers the real structure.

Gaussian distributions explain “ordinary variation” (like heights or car speeds). But power laws explain “extraordinary inequality” (cities, wealth, internet, culture).

Generated by Python

Power law

A power law says:

• x = the value (like city population)
  Example: x=1000000 means a city with 1 million people.
• α = the exponent (controls how heavy the tail is). It controls how quickly the number of cities falls as they get bigger. Example: if α=2.5, it means big cities are rare, but not impossible. A smaller α means fatter tails → more big cities.
• x_min​ = the smallest value where the power law holds. In our example, the smallest city size where we say the power law hold. Let’s say, we only consider cities with at least 10,000 people.
• C = just a math trick: it makes sure all probabilities add up to 1. We can think of it as a scaling factor.

When x is small (a city with 20,000 people), p(x) is relatively large → there are lots of such small cities.

Get Arman Malkhasyan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

When x is big (a city with 10 million people), p(x) is very small → very few such mega-cities exist.

Example:

Suppose α=2 and x_min⁡=10,000.

• A city of size 20,000:

• A city of size 200,000:

Now let’s compare them:

That means a city of 20k people is about 100 times more likely than a city of 200k people.

So:

• Most cities are small towns.
• Some are medium-sized.
• A few are mega-cities — rare, but still exist.
• The power law captures exactly how the “rarity” grows with city size.

Generated by Python

Here’s the city population power-law example:

• The blue line shows the power-law curve (α=2, x_min⁡=20k)
• The red points mark example cities:
  20k → common (high probability)
  200k → about 100× less likely.
  2M → about 10,000× less likely
  20M → extremely rare, but still possible

In human words:

• Small towns are everywhere.
• Mid-sized cities exist but are fewer.
• Mega-cities are rare — but not impossible.

Normalization

The next step is normalization. Why Do We Need Normalization?

When we define a probability distribution, the total probability must equal 1. For example, if you add up all probabilities of dice outcomes (1–6), we get 1. For continuous distributions, this means the area under the curve = 1.

Step 1: Start with the power law

We want:

Step 2: Plug into the equation

Factor out C:

Step 3: Solve the integral

Here k = — α

Turn the improper integral into a limit

Evaluate the limit (why we need α > 1)

If α>1, then 1−α<0. Therefore b^(1−α)→0 as b→∞. So

Here there can be raised a question. Why really α>1?

Case α > 1:

• Then 1−α < 0
• So as x→∞, x^(1−α) → 0
• The area under the curve is finite. This means the distribution is valid: total probability sums to 1.

Case α = 1

• Then the integral becomes:

• As x→∞, ln⁡(x)→∞
• The area is infinite. Not normalizable → can’t be a probability distribution.

Case α < 1

• Then 1−α > 0
• So as x→∞, x^(1−α)→∞
• The integral blows up even faster. Again, infinite area → invalid as a probability distribution.

So, we require α>1 to make the power law a valid probability distribution.

Generated by Python

• Blue (α = 0.5): Decays very slowly → the tail is so heavy that the area under it is infinite.
• Orange (α = 1.0): Decays like 1/x1/x1/x → slower than needed, the area still diverges.
• Green (α = 2.0): Decays much faster → the total area converges to a finite value (valid distribution).

This picture shows why α must be greater than 1. Only then does the curve shrink fast enough for the infinite tail not to consume infinite probability.

Step 4: Rearrange for C

Intuition with our example, Cities:

Let’s say:
α=2
x_min​=20,000

Then:

So the final formula is:

We can conclude that:

• At x=20000, probability density is 20000/(20000^2)=1/20000
• At x=200000, it’s 20000/(200000^2)​ → 100 × smaller.

The constant C ensures that if we add up the probabilities of all possible city sizes, they sum to exactly 1. So, the constant C is like a balancing weight that makes sure our probability curve is valid.

Final Formula (PDF)

So the power-law probability density function (PDF) is:

Comparing Distributions: Poisson, Exponential, and Power Law

Generated by Python

• Black (Poisson): counts of discrete events (e.g., number of emails in an hour). Peaks at λ=5 then declines.
  Used for counts of events in a fixed time interval.
  Parameter: λ = average number of events.

• Green (Exponential): used for waiting times until the next event. Decays very fast (long waits are rare).
  Example: waiting time until the next bus.
  Parameter: λ = rate (how often events happen).

• Blue (Power Law)
  Used for heavy-tailed phenomena (many small, few huge).
  Example: city populations, wealth, website traffic.
  Parameter: α = exponent (controls heaviness of tail).

• Red (Mixed Exponential–Power Law). A hybrid model when real-world data looks partly exponential, partly power law.
  Example: systems where small values behave exponentially, but big values follow a heavy tail.
  Parameters: λ (exponential part) and α\alphaα (power law part).

Overall:

• Poisson = discrete counts.
• Exponential = fast decay, short waits.
• Power law = slow decay, heavy tail.
• Mixed = compromise between the two.

Power Law CDF: Probability that X ≤ x

We have already discovered Power Law PDF which tells how likely is it to see a value exactly around size x and now diving deeper into CDF.

Definition:

Plug in the PDF:

After solving, we get:

As x → x_min​, F(x)=0 (probability below the minimum is zero).
As x→∞, F(x)→1 (eventually, we’ve included everything).

So, CDF answers:
“What fraction of cities are smaller than 1M people?”

Power Law cCDF: Probability that X>x

Definition:

Plugging in:

As x grows, the probability of seeing something bigger shrinks like a power law.
On a log-log plot, this looks like a straight line.

So, cCDF answers:
“What fraction of cities are bigger than 1M people?”

Key Difference

• CDF: fraction smaller than a threshold.
• cCDF: fraction larger than a threshold.

Both contain the same information, but:

• CDF is nice for small values (near 0).
• cCDF is nice for the tail (large values).

Exponential vs Power Law

Generated by Python

Power laws (solid lines): now much steeper for higher α
α=1.5: shallowest slope (yellow).
α=2.5: steeper (blue).
α=3.5: steepest (green).

Exponentials (dashed): bend downward rapidly — they vanish much faster.

This makes the difference crystal clear:
Power laws = straight lines on log–log, with slope −(α−1).
Exponentials = curves, dropping off super fast.

Spotting a Power Law in the Wild

On a log-log scale, the difference between distributions jumps right out at you. Exponentials bend down quickly, while power laws fall much more slowly — as neat straight lines.

Why does this matter? Because in the real world, we often collect data without knowing what distribution it follows. Think about it:

• Are earthquake magnitudes exponentially distributed, or do they follow a power law?
• Are city populations Gaussian (like heights), or heavy-tailed?
• Does the number of views on YouTube videos decay quickly, or does it have a fat-tailed pattern where a few videos dominate?

The easiest way to find out isn’t fancy math. It’s this:
Plot the data on a log-log scale and look at the tail.

• If the tail curves downward, the distribution is exponential-like (exponential, Gaussian, or Poisson).
• If the tail is a straight line, that’s the unmistakable signature of a power law.

This straight-line test has become one of the simplest and most powerful tools for data scientists, physicists, and network researchers. It’s how we tell whether big, rare events (mega-cities, viral hits, financial crashes, massive earthquakes) are truly unexpected — or if they’re just what we should expect in a power-law world.

Conclusion

Power-law distributions reveal a world that is very different from the smooth bell curves of Gaussian statistics. They remind us that:

• Most things are small, but a few are massive.
• Straight lines on a log-log plot are the telltale signature of this hidden order.
• They show up everywhere: from city populations and wealth distribution, to internet traffic, music, books, and earthquakes.

Unlike exponential or Gaussian models, power laws do not dismiss extreme events as vanishingly rare. Instead, they tell us that big — mega-cities, billionaires, viral videos, huge earthquakes — is not only possible but inevitable.

In case of any questions, feel free to reach out to me on LinkedIn.