Carbon-Aware Inference, Real Wins at Off-Peak

Shift AI workloads to cleaner grid hours to cut CO₂ and costs — without breaking your SLAs.

Shift AI inference to low-carbon hours using grid-aware scheduling to cut emissions and cloud costs — without hurting latency or accuracy.

Get Bhagya Rana’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Most teams squeeze inference for milliseconds and dollars. Few squeeze it for carbon. Yet the grid’s “clean” and “dirty” hours swing every single day. If you route flexible work into cleaner windows, you can drop emissions and spend — often with zero user-visible change. Let’s be real: this is one of the rare climate wins that also ships business value.

Why “carbon-aware” inference now

AI isn’t just training anymore; inference is the long tail. Every embed, rerank, summary, and fraud check piles up — 24×7. Meanwhile, electricity cleanliness changes hourly with wind, solar, and demand. That variability is a feature, not a bug. If your system can wait a little (minutes or hours) or move a little (to a cleaner region), you can ride the grid when it’s greener.

The idea in one line

Shift what’s flexible. Protect what’s not.
That’s it. You classify traffic, then steer the flexible slice toward lower-carbon hours (time shifting) and sometimes lower-carbon regions (space shifting). You keep strict-SLA requests on their fastest path. Everyone wins.

What’s actually flexible?

Think in latency classes instead of one big “inference” bucket:

• Real-time (hard SLO): search ranking, payments risk, chat responses. No shifting — optimize hardware efficiency instead.
• Near-real-time (soft SLO, 1–5 min): analytics enrichments, feature stores, async notifications. Small shifts possible.
• Batch/async (hours): nightly embeddings, bulk summarization, report generation, ML evaluations. Big shifts possible.

You might be wondering, “Will users notice?” If you shift the right classes and set graceful fallbacks, they won’t. The trick is defining SLOs that reflect human experience, not just server timers.

Three levers that matter

1. Time shifting: Queue flexible jobs into low-carbon windows (e.g., 00:00–06:00 local or during daytime solar peaks).
2. Space shifting: Prefer regions with cleaner fuel mix when latency budgets allow. Keep data residency and egress in mind.
3. Model/precision shifting: When the grid is “dirty,” run smaller models, quantized weights, or cached answers. When it’s clean, backfill with your best model.

What to feed your scheduler

You don’t need magic, just better inputs:

• Carbon intensity now + forecast (grams CO₂/kWh).
• Your queue depth and SLOs per class.
• Energy-linked cost signals (spot price, preemptible/low-priority capacity).
• Hardware efficiency (tokens/Joule, images/Joule), so you can choose the most efficient accelerator per job.

A minimal scheduling loop (Python)

Conceptual example — drop this behind a feature flag and ship a pilot.

import time
from datetime import datetime, timedelta

CARBON_HIGH = 500      # gCO2/kWh threshold for "dirty"
CARBON_LOW = 300       # gCO2/kWh threshold for "clean enough"

def carbon_forecast(region, horizon_hours=6):
    # Returns [(ts, g_per_kwh), ...] — stub for your provider or internal feed
    ...

def next_clean_slot(region, max_wait=timedelta(hours=6)):
    fc = carbon_forecast(region)
    now = datetime.utcnow()
    for ts, g in fc:
        if g <= CARBON_LOW and ts - now <= max_wait:
            return ts
    return None

def run_inference(job, fast_path=False):
    # Your actual inference call (model/router/etc.)
    ...

def enqueue(job):
    region = job.get("region", "auto")
    class_slo = job.get("class", "batch")  # "realtime" | "near_rt" | "batch"
    fast_path = (class_slo == "realtime")

    if fast_path:
        return run_inference(job, fast_path=True)

    # Soft/Batch path
    slot = next_clean_slot(region)
    if slot:
        delay = (slot - datetime.utcnow()).total_seconds()
        if delay > 0:
            time.sleep(min(delay, 60 * 60))  # naive sleeper; swap for your queue scheduler
        return run_inference(job)

    # Fallback if no clean slot soon and SLO is tightening
    return run_inference(job)

What’s happening:

• Real-time stays on the fast path.
• Near-real-time and batch look ahead for a clean window; if found within the allowed wait, they shift.
• If the grid stays dirty and your SLO is approaching, you fail “open” and run anyway. Reliability first.

Tip: In production, replace sleep with your orchestrator (queues/cron/workflows). Add circuit breakers: if queues exceed a threshold or forecast feeds fail, you default to normal compute.

A composite case study (how teams see gains)

A product org split traffic into three classes and shifted ~35% of daily inferences into low-carbon windows (late-night wind + midday solar). Over a month, they saw 20–30% lower grams CO₂e per flexible request, and 8–15% infrastructure cost reduction from better use of discounted capacity. Latency for real-time APIs didn’t budge, because they never touched that lane. These are typical pilot numbers when the classification is done well.

Guardrails and gotchas

• Data residency & egress: If you space-shift, keep legal boundaries intact. Sometimes “cleaner region” is a non-starter — then stick to time shifting.
• Fairness across time zones: Don’t penalize users in “dirty hours” regions; shift only non-user-facing work or use cached answers to equalize.
• Staleness: For embeddings/summaries, cap max age. Backfill with high-quality runs when the grid is clean.
• Forecast drift: Carbon forecasts aren’t perfect. Use bands (clean/neutral/dirty) instead of single thresholds, and keep a fail-open policy.
• Rebound effect: Savings can tempt teams to do more inference. Set budgets on grams CO₂e, not just dollars.

Measuring what matters

Define outcomes before you ship:

• gCO₂e per request by class (flexible vs real-time).
• % of flexible work shifted (time and/or space).
• SLO adherence (p95 latency, error rate).
• Cost per 1k requests and tokens/Joule (or images/Joule).

Shadow-deploy the scheduler for a week: make decisions, log them, but don’t enforce. If the graphs look good — shift for real.

A simple rollout plan

1. Tag traffic with latency classes (start with 2–3).
2. Wire a carbon feed and build a 24-hour calendar of clean/dirty blocks.
3. Pilot on one batch job (e.g., nightly embeddings). Measure carbon + cost + SLOs.
4. Expand to near-real-time enrichments with tight fallbacks.
5. Optional: add space shifting where legal and beneficial.
6. Close the loop: review weekly, tighten thresholds, and tune models for “dirty-hour” efficiency.

Quick win: degrade gracefully when the grid is dirty

When intensity is high and you’re forced to run, you can still reduce impact:

• Prefer lower-precision or distilled models.
• Cap concurrency to the most efficient accelerator types.
• Serve cached answers and schedule high-fidelity recompute into the next clean window.
• Log a “carbon debt” and pay it down later with backfills that replace low-fidelity outputs.

Bottom line

Carbon-aware inference is not a moonshot. It’s a routing problem with business upside. Classify your traffic, read the grid, and steer the work that can move. Start tiny, prove it in a week, then scale. If you’ve already shaved milliseconds and dollars, this is your next lever.

CTA: What would you shift first — embeddings, summaries, or evaluations? Drop a comment with your stack, and I’ll suggest a low-risk pilot plan you can run this week.