One threshold turned six push-ups into a count of twenty

Building a rep counter on MediaPipe pose landmarks, with every design decision settled by a measurement rather than a preference

Originally published at condados.ai/blog/rep-counting-jitter-filters-state-machine

Counting repetitions from a pose model is four stages and about 125 lines of logic. Getting it right is four decisions, and I made each one with a measurement instead of a preference. The two that matter most are free: read the joint angle from world landmarks, because the projected one moves 43.6 degrees while the elbow holds still, and use two thresholds instead of one, because a single threshold turned a clip of six push-ups into a count of twenty. The filter everyone argues about is third in size, and the one that wins is not the usual choice.

The naive version is short enough to write from memory. Take a pose model, compute the elbow angle, count the times it crosses a line:

def naive_count(angles, threshold=110.0):
    reps, phase = 0, "up"
    for angle in angles:
        if angle < threshold and phase == "up":
            phase = "down"
        elif angle > threshold and phase == "down":
            phase = "up"
            reps += 1
    return reps

It works on the first video you try. Then the complaints arrive: counts that climb while the user is resting, half repetitions registering as whole ones, the same repetition counted twice.

The post this is drawn from has that widget running live on your own camera, with the naive counter beside the correct one so you can watch them part: condados.ai/blog/rep-counting-jitter-filters-state-machine. Everything below is how it got there.

Every number here is measured on one clip, named once so it can be checked: Interval Push-ups by Taco Fleur, CC BY-SA 4.0 on Wikimedia Commons [4]. 51.4 s at 640×480 and 30 fps, a backlit silhouette in profile, six push-ups with a plank held between them. Six is the hand-counted truth, established from the video before any of this was run, because the pipeline under test cannot be the source of its own ground truth.

The jitter floor you cannot filter away

Worth knowing before any filter decision, because it sets the floor. Decode one frame, then hand MediaPipe’s Pose Landmarker [1] that same array of pixels sixty times.

The landmarker gives 33 body points per frame from the BlazePose family [3]. In IMAGE mode, the landmarks come back bit for bit identical across all three published model sizes. In VIDEO mode they do not:

• lite: 0.0000 px in IMAGE mode, 0.418 px RMS in VIDEO mode on identical pixels (max 0.995)
• full: 0.0000 px, and 0.340 px RMS (max 0.650)
• heavy: 0.0000 px, and 0.207 px RMS (max 0.536)

Per-landmark RMS deviation about each landmark’s own mean, 60 repeats of one 640 px frame, nine conditions across three model sizes and three resolutions.

A photograph, fed in repeatedly, produces landmarks that wander by a third of a pixel. Nothing about the scene changed, so this is not the sensor, the lighting or the encoder. VIDEO mode crops each frame from the region of interest it derived from the previous one, so its own output is part of its next input and the loop never settles. That is the design that makes the mode fast enough to run at frame rate, and the cost is usually invisible. The heavy model sits 2.0 times lower than the lite one, so capacity moves the floor and does not remove it.

Two smaller traps in the same call. The GPU delegate is not guaranteed and createFromOptions rejects when it is missing instead of falling back, so ask for GPU and then for CPU in a loop. And VIDEO mode needs strictly increasing timestamps in milliseconds, supplied by you: two frames delivered inside the same millisecond break the internal tracking, and offline you take the timestamp from the frame index and the declared frame rate rather than from the wall clock. There is no smoothing option to turn on. The smooth_landmarks flag people remember belongs to the retired mp.solutions.pose API. Anything built on the current one filters its own signal or does not filter.

Decision 1: the angle comes from world landmarks

The counter reads one scalar per frame, the joint angle. An angle computed on image coordinates is the angle of the limb’s shadow on the sensor, so turning the person changes the shape of the shadow while the elbow holds still.

This can be checked exactly. Take a real skeleton from a real frame, rotate the whole thing about its own spine as a rigid body, and reproject it through a pinhole. Rotating a rigid body cannot change any angle inside it, so the world angle is constant by construction, and every degree the projected angle moves is projection.

At an elbow bent to 45 degrees, turning the torso through 120 degrees moves the projected angle by 43.6 degrees. At 90 degrees of bend, it moves 41.5 degrees, and the effect shrinks as the arm straightens: 28.2 degrees at 135, and 13.3 degrees at 170.

Worked example: a curl at the bottom of its range, someone turning to check a mirror.

The state machine below ends up with two thresholds at 80 and 150 degrees, so 150 − 80 = 70 degrees of protection against noise.

The projected angle moves 43.6 degrees at an elbow held at 45 degrees.

Fraction of the gap consumed: 43.6 / 70 = 0.62, so 62 percent.

What it means: that gap clears the 3.48 degrees of resting jitter measured further down by a factor of 20. The same gap has less than a factor of two of margin against a person turning while they exercise, and a rotation is not noise, so no filter removes it. Read the angle from world landmarks and the term disappears.

Two traps live in the three lines that feed the angle. Normalized landmarks are x / width and y / height, so on a 16:9 frame one unit of x is 1.78 times as many pixels as one unit of y, and an angle taken straight from the normalized values is the angle of a horizontally stretched picture of the arm. Multiply back into pixels first. And the z you get from landmarks is not metric: MediaPipe returns a second set,worldLandmarks, in metres and centred on the hips [3]. Use those.

The empirical half of this corrected my hypothesis, and the correction is the useful part. Binning every frame of every clip by how far the shoulders are turned from square put the worst disagreement at the smallest turn, which is backwards. A push-up filmed from the side has the torso fully turned and the arm swinging in a plane parallel to the sensor, so the projection is faithful. What distorts a projected angle is how much of the limb points at the camera, not how far the person turned. Re-binned on the arm’s own tilt out of the image plane, over 23,059 frames, the rank correlation with the 2D-versus-3D disagreement is +0.376 for tilt against −0.393 for torso yaw, my first guess. The tilt correlation is positive inside every one of the seven clips taken separately, from +0.14 to +0.80.

Decision 2: the noise has memory, and that changes the filter

Filters are usually chosen against white noise, because white noise is what is easy to generate. Every still segment across three clips, meaning every stretch where the limb is locked out and the angle is flat, gives 232 seconds of residual once a straight line is removed from each segment.

Correlated by 0.694 from one frame to the next, and still outside the white-noise band at five frames. The memory runs out somewhere around a quarter of a second.

This matters for a practical reason. A low-pass filter separates signal from noise by frequency, and noise with that much memory has most of its energy at low frequencies, which is where the movement lives too. The same filter that removes white noise cheaply has to eat into the signal to remove this.

The effect is measurable. Take the same residual samples and shuffle them: the distribution is untouched, and the memory is gone, which is the signal a white-noise benchmark assumes. Across 65 filter configurations, filters leave 1.78 times more jitter behind on the real residual than on the shuffled version, with a range of 1.07 to 2.21. A benchmark run on simulated white noise reports a filter that is 78 percent better than the one you deploy.

What an exponential moving average costs

The filter people reach for first is one line: y[n] = α·x[n] + (1−α)·y[n−1]. Each output is a blend of the newest reading and where the filter already was, so a small α means the filter mostly ignores the newest reading. That buys smoothness by being slow to believe anything, which is the same thing as being late.

There are two ways to quantify how late, and across the settings swept here they differ by a factor of 2.4 to 3.3. That matters because people quote one while meaning the other. Group delay at DC is the mean delay of the impulse response, which is (1−α)/α samples. The step response is 1−(1−α)ⁿ, so the time to arrive at 90 percent of a change is ln(0.1)/ln(1−α) samples.

Worked example: the elbow reaches the top of a push-up, at 30 fps.

Take α = 0.25, the best EMA setting found in the sweep below.

Group delay: (1 − 0.25) / 0.25 = 3.0 samples, and 3.0 / 30 = 0.100 s, so 100 ms.

Rise time: ln(0.1) / ln(0.75) = −2.303 / −0.288 = 8.0 samples, and 8.0 / 30 = 0.267 s, so 267 ms.

What it means: the arm reaches the top of the push-up, and the filtered angle does not report having arrived for another quarter of a second. Quoting the 100 ms figure while the user waits 267 ms for the counter to tick is a 2.7× error on the number that decides whether the app feels responsive.

The comparison, and the winner

The 1 euro filter of Casiez, Roussel and Vogel [2] raises its cutoff with the estimated speed of the signal: heavy smoothing while the arm is still, and it gets out of the way while the arm moves.

Both axes were measured on the condition where each one matters, and neither was simulated. Jitter is the RMS of each filter’s output over the real residual above, where the true value is a constant, so anything the filter emits is error. Lag is measured against a zero-phase reference, the same low-pass run forwards and then backwards so its phase cancels, on a real descent from the push-up clip.

My first version of the lag measurement used an ideal step and had to be thrown out. A step is 120 degrees inside one frame, a speed-adaptive filter opens its cutoff all the way for it, and the step reports 2.53 times the lag of a real descent. That would have credited the 1 euro filter with a movement no arm can make.

The 1 euro filter wins where a real-time app has to live. At a 100 ms budget, it leaves 0.819 degrees against the best EMA’s 1.464, which is 44.1 percent less jitter, and the gap closes to 5.1 percent by 400 ms. At 50 ms the advantage is 23.0 percent, and at 200 ms it is 31.3 percent. My own line is that a counter reporting a repetition 400 ms after it happened is not one I would ship, which puts the region where the two filters tie outside the part of that figure I would use.

The constant-velocity Kalman filter tracks the EMA closely enough to make its extra state hard to justify. Below 300 ms, it never beats the EMA by more than 1.8 percent, and it is 4.5 percent worse at a 200 ms budget. It does pull ahead at 400 ms, by 10.8 percent, in the same region where the EMA and the 1 euro filter converge and where the delay is long enough to notice. Forsyth and Ponce set out the constant-velocity model and where it is appropriate; a repetition reverses its angular rate at the top and the bottom of every rep, which is where a constant-velocity model is at its worst. One clip is not a proof of mechanism, so treat that connection as the likely reason rather than a demonstrated one.

Get Luís Condados’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Two details a from-memory rewrite of the 1 euro filter tends to lose, both of which still produce a plausible-looking output: the derivative is taken against the previous filtered value, and the sampling frequency is re-estimated from consecutive timestamps so the filter behaves under a dropped frame.

class OneEuroFilter {
  constructor(minCutoff = 0.05, beta = 0.005, freq = 30, dCutoff = 1) {
    Object.assign(this, { minCutoff, beta, freq, dCutoff });
    this.x = null; this.dx = 0; this.tPrev = null;
  }
  alpha(cutoff) {
    const te = 1 / this.freq;
    return 1 / (1 + (1 / (2 * Math.PI * cutoff)) / te);
  }
  filter(value, t) {
    if (this.tPrev !== null && t > this.tPrev) {
      this.freq = 1 / (t - this.tPrev);
    }
    this.tPrev = t;
    const rate = this.x === null ? 0 : (value - this.x) * this.freq;
    const a = this.alpha(this.dCutoff);
    this.dx = a * rate + (1 - a) * this.dx;
    const cut = this.minCutoff + this.beta * Math.abs(this.dx);
    const b = this.alpha(cut);
    this.x = this.x === null ? value : b * value + (1 - b) * this.x;
    return this.x;
  }
}

Check any port you write. The authors publish a groundTruth.csv for this purpose, and the Python side of the companion repository replays it: agreement to 4 × 10⁻⁶, where the residual is the CSV's six-significant-figure printing accumulating through a recursive filter. The tuning order is the authors': set beta to zero and lower the cutoff until slow-motion jitter is gone, then raise beta until fast motion stops lagging.

Decision 3 and 4: the state machine

This is where counting is won or lost, and it is the cheapest stage to get right.

class RepCounter {
  constructor({ flexBelow = 80, extendAbove = 150, confirmS = 0.1,
                minVisibility = 0.5 } = {}) {
    Object.assign(this,
      { flexBelow, extendAbove, confirmS, minVisibility });
    this.phase = 'extended';
    this.count = 0;
    this.candidate = null;
    this.candidateSince = 0;
  }
  // Feed one frame. Returns true if a repetition completed on it.
  update(angle, visibility, t) {
    if (!Number.isFinite(angle)) return false;
    // hold through an occlusion
    if (visibility < this.minVisibility) return false;
    let wants = null;
    if (this.phase === 'extended' && angle < this.flexBelow) {
      wants = 'flexed';
    } else if (this.phase === 'flexed' && angle > this.extendAbove) {
      wants = 'extended';
    }
    // Confirmation, not lockout: the angle has to stay past the line.
    if (wants === null || wants !== this.candidate) {
      this.candidate = wants;
      this.candidateSince = t;
      return false;
    }
    if (t - this.candidateSince < this.confirmS) return false;
    this.candidate = null;
    if (wants === 'flexed') { this.phase = 'flexed'; return false; }
    this.phase = 'extended';
    this.count += 1;
    // the only edge that counts a rep
    return true;
  }
}

Four rules are doing the work, and they are not equally important.

Two thresholds instead of one. Entering the flexed phase needs the angle below 80 degrees; leaving it needs the angle above 150. Noise has to be larger than the gap to move the machine at all.

Both bounds, in that order. A repetition is counted on the flexed-to-extended transition, so a movement that never gets below the lower bound never enters the flexed phase and has no transition to complete. That is what rejects a half repetition, and it follows from the first rule rather than being an extra one.

A time requirement on a phase change. There are two of these in the wild and they do not do the same job. A lockout refuses transitions for a while after the last one; at the instant a spike arrives, the machine has been in its current phase for far longer than the lockout, so the spike flips the phase unopposed and the lockout only delays the return. A confirmation instead requires the angle to stay past the threshold before the phase changes at all, which is what a spike cannot do. An ablation table that conflates the two credits the wrong rule.

A confidence gate. Frames where the weakest landmark in the chain is below a visibility cut are dropped, and the phase is held. An angle measured during an occlusion is not a measurement.

What the first rule is worth

A counter with a single threshold is not wrong everywhere, which is why it passes review and ships. On the push-up clip, with the threshold at 115 degrees, halfway down a 120-degree swing, it reports six out of six. So does every other configuration I tested. Correct execution does not break a rep counter.

The clip contains the failing condition anyway, in its gaps. Between repetitions, the subject holds a plank with the arm locked out. Taking that strictly, above 160 degrees, leaves 4.0 seconds of this clip, resting at a mean of 162.5 degrees with 3.48 degrees RMS around it. The looser criterion used for the noise profile earlier keeps 40.5 seconds of the same clip and gives 1.57 degrees, so the 3.48 figure describes the top of the range instead of the clip as a whole.

Now walk a single threshold through that resting band.

Twenty reported against a truth of six, from 39 crossings of a line the signal is sitting on. At 160 degrees it is 21 from 41 crossings. Move the line 6 degrees to 164, and the same counter reports 3, undercounting instead. Hysteresis with a lower bound at 80 degrees reports 6 at a well-placed 115 and never exceeds 3 anywhere in the band.

Worked example: the plank between push-ups, 4.0 s of it.

The arm rests at 162.5 degrees with 3.48 degrees RMS of noise on it. Put the threshold at 158.

Distance from the resting value to the line: 162.5 − 158 = 4.5 degrees, which is 4.5 / 3.48 = 1.29 standard deviations.

For a Gaussian, a value that far out or further arrives about 10 percent of the time. Over 120 frames of plank that is roughly 12 excursions past the line, and the measurement counts 39 crossings, because the noise is correlated and arrives in runs instead of as independent draws.

Now open the two thresholds to 80 and 150. The gap is 150 − 80 = 70 degrees, which is 70 / 3.48 = 20 standard deviations. Nothing in this clip is going to cross that.

What it means: the failure is not that noise exists, it is where the line was put relative to it. The single-threshold counter needs the signal to spend no time near its threshold, and a person resting between sets spends all of it there.

The rule that follows: place the thresholds where the signal does not rest, and put as much distance between them as the movement allows. Tuning a threshold to “roughly where the arm is at the top” puts it in the worst possible place.

The four errors, ranked

On this clip: a badly placed threshold costs 14 repetitions, a projected angle costs up to 43.6 degrees of apparent movement, the filter choice costs 0.645 degrees of residual jitter at a 100 ms lag budget, and the landmarker’s own floor costs a third of a pixel. The two at the top are free to fix. The one that gets the attention is third.

Before you ship

Five checks, each of which catches something this measured.

1. Feed the landmarker one frozen frame and watch the output. If it moves, you have found your jitter floor, and no amount of lighting work goes below it.
2. Log the timestamps you pass to detectForVideo. A repeat or a step backwards means the tracker is degrading and nothing downstream will tell you.
3. Hold a limb still on the threshold for thirty seconds. The count must not move. This is the single most informative test in the list and it takes half a minute.
4. Do half repetitions on purpose. They must not count. If they do, the machine is triggering on one bound instead of two.
5. Turn side-on while holding a joint at a fixed angle. If the reported angle moves, you are reading image coordinates somewhere.

Limitations

One movement, one camera, a handful of clips. Push-ups in profile, plus two press clips for the pooled noise. Nothing here says how the numbers move for a squat at four metres, where the landmarks are worse to begin with.

The counting comparison has one labelled clip. Six repetitions, hand-counted twice. The threshold table is a walk across one clip’s resting band, and the shape of that band is a property of that clip. The direction of the effect follows from the mechanism; the size of it is one measurement.

Three of the conditions people complain about are missing. A subject deliberately holding still for half a minute, a set that mixes full and partial repetitions, and a limb held at a fixed angle while the torso rotates. These are apparatus, not exercise content, so found video does not contain them. The still condition is substituted by the plank holds and by the frozen-frame experiment. The partial-repetition case is argued from the state machine’s structure and demonstrated in the tests, and it is not measured on video here.

The rotation experiment is geometry plus a correlation. The rigid-body half is exact and proves what projection does. It does not prove that MediaPipe’s estimated world landmarks stay stable through a real turn. The clip-based correlation of +0.376 supports it and is not the same claim.

The widget runs a different model from the measurements. It loads the lite bundle so the download stays small, while every number here was measured on full. From the frozen-frame table that costs it 0.08 px more jitter, 0.418 against 0.340.

Where the numbers come from

Every figure above is a replay of a saved landmark track from that one clip, on MediaPipe 1.0.1 and the full float16 bundle, CPU only. The full reproducibility block, meaning the hardware, the OS, the pinned versions, the exact commands, and what is excluded from each measurement, is on the canonical post: condados.ai.

Code, models, and every saved artifact: github.com/CondadosAI/pose-rep-counting. The notebook notebooks/pose_rep_counting.ipynb has a cell for every Python snippet above.

The version with the live widget

This is the short form. The original carries the counter running on your own camera with the naive version beside it, the full 65-configuration filter sweep, the derivation of the joint angle, and the reasoning on how the companion repository factors the same state machine as a table of rules: condados.ai/blog/rep-counting-jitter-filters-state-machine

Related, and the same mistake at the other end of a pipeline: pupil diameter from a webcam, where a network asked for millimetres with no scale reference produces a per-person offset. This post is about a threshold with no reference for where the signal rests.

References

[1] Google. MediaPipe Pose Landmarker. Google AI Edge documentation, mediapipe 1.0.1. developers.google.com/edge/mediapipe/solutions/vision/pose_landmarker

[2] Casiez, G., Roussel, N. and Vogel, D. 1 € Filter: A Simple Speed-Based Low-Pass Filter for Noisy Input in Interactive Systems. Proceedings of CHI ’12, pp. 2527–2530, 2012. doi.org/10.1145/2207676.2208639

[3] Grishchenko, I., Bazarevsky, V., Zanfir, A., Bazavan, E. G., Zanfir, M., Yee, R., Raveendran, K., Zhdanovich, M., Grundmann, M. and Sminchisescu, C. BlazePose GHUM Holistic: Real-time 3D Human Landmarks and Pose Estimation. arXiv:2206.11678, 2022. arxiv.org/abs/2206.11678

[4] Fleur, T. Interval Push-ups. Wikimedia Commons, CC BY-SA 4.0. commons.wikimedia.org/wiki/File:Interval_Push-ups.webm

Oppenheim and Schafer’s Discrete-Time Signal Processing (ch. 5, Transform Analysis of Linear Time-Invariant Systems) is where group delay is defined, and Forsyth and Ponce’s Computer Vision: A Modern Approach (ch. 11, Tracking) covers the constant-velocity model. The full numbered list is on the canonical post.