Teaching a Camera to Be a Fitness Trainer

How I built ShapeForm, an app that checks exercise form from a video, and why the hardest part was making its feedback worth trusting.

Have you ever been working out and wondered whether your form was actually right? I have, and I could never tell. So I built ShapeForm, a free website that watches you exercise through your camera, counts your reps and tells you what to fix.

Getting the computer to find a person in a video took a few weeks. Getting it to give fair, trustworthy feedback took much longer. This article covers the whole journey, from the research papers I read to the website I ended up building.

Where It Started

I enjoy building things in Python, but most of my projects lived on a screen. I wanted to make something that could help with a real-life activity, and exercise felt like a natural fit, since checking my own form was a problem I had already run into.

Before the program could judge anything, it had to find the person in the video. For this, I used MediaPipe, a free tool from Google that does something called pose estimation. A video is really just a long series of still pictures called frames. Pose estimation looks at each frame and marks where the body’s joints are, such as the shoulders, elbows, hips, knees and ankles. Connect those points with lines, and you get a simple skeleton. The first version of ShapeForm found those points, drew a moving stick figure over the video, and showed a message about the person’s form.

Seeing that stick figure move with me for the first time was exciting. Then I started paying attention to the messages. They could change several times during a single squat, and sometimes the program sounded completely sure of itself while its advice was wrong. That was when I realized that finding a person in a video and understanding what that person is doing are two very different problems.

One Picture Is Not Enough

Think about what happens during a squat. You start standing, lower yourself down, and then stand back up. If I showed you a single photo from the middle of that movement, you could not tell whether the person went low enough or kept their back steady the whole way through. My first approach had exactly this weakness. It judged each frame on its own, with no idea what came before or after, which helps explain why the messages kept jumping around.

So I changed the question I was asking. Could the program watch one complete repetition, or rep, from start to finish and judge the whole movement? This is where machine learning came in. A machine learning model is a program that figures out patterns by studying lots of examples. To learn what a good squat looks like, it needs many squat videos, and each one needs a label, which is a note describing what was done well or badly. A collection of examples like this is called a dataset. Before writing much more code, I wanted to see how researchers had approached the same problem.

Learning From People Who Tried First

The research papers were not easy reading. Some of the diagrams and equations went over my head the first time through. What helped was focusing on the problem each paper was trying to solve. Even when I could not follow every detail, I could usually understand what the researchers were worried about, and that gave me ideas I could test in Python. Three papers shaped the project the most.

1. Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition, 2018

The title is a mouthful, but the main idea is simple. The researchers treated the body as a graph, which here just means a set of points joined by lines, like joints joined by bones. Your knee is connected to your hip and your ankle, and the way those three move together says much more than the position of the knee on its own. I tried the paper’s general approach and compared it with a simpler method that reads the body’s movement in order, one moment after another. The lesson I kept was to look at how body parts move together before judging any single joint.

2. 3D Pose Based Feedback for Physical Exercises, 2022

This paper was the closest to what I wanted to build. A lot of work in this area focuses on recognizing which exercise someone is doing. This team focused on finding the mistakes inside the exercise, which was exactly my goal. Their dataset, called EC3D, records exercises in 3D, which meant I could turn the same squat around and look at it from different camera positions. That became important later. Their system also shows a corrected 3D version of each movement, which ShapeForm does not attempt. The idea I took from them was simpler. Good feedback should describe how a movement was performed.

3. What to Say and When to Say It, 2024

This paper looked at a problem I had not thought much about yet, which is how to coach someone while they are in the middle of exercising. A good coach has to decide what to say, and also when to say it. The exercise videos connected to this research became a big part of my experiments. The paper also reminded me what the end goal was. A program that prints “incorrect” has only done half the job, because the person in front of the camera needs advice they can understand and act on.

Each paper gave me something specific to test. Turning those ideas into working Python code, and finding out which ones actually helped, was up to me.

The Problems I Did Not Expect

Testing those ideas meant working with real data, and the first obstacle was surprisingly ordinary. I had to download it. The exercise video collection was far bigger than the storage space I had for training, so I wrote Python code that downloaded it one section at a time, kept only the videos I needed, and picked up where it left off if a download stopped partway. It was slow, unglamorous work, and without it there would have been nothing for the program to learn from.

Then I found a more serious problem in the labels. Some video descriptions were being counted as mistakes even when they described correct form. A description that mentioned shoulder-width feet, for example, was being treated as a warning, even though a shoulder-width stance is usually exactly what you want. Other details showed up twice under slightly different names, so the program saw them as two separate things. This matters more than it sounds. If the answers you give a program are confused, it can become very good at learning the confusion.

I sorted the descriptions into useful groups, merged the duplicates, and stopped the program from treating unknown information as if it had a definite answer. That work taught me something I had not expected. Cleaning up and understanding the data could matter just as much as improving the model itself.

Camera angles caused a third problem. A camera facing you can show how far apart your knees are, while a camera beside you is much better at showing how far forward you lean. Most of the exercise videos were filmed from the front, so I used the 3D recordings from EC3D to study the same exercise from other angles. This is also why the finished website tells you to film most exercises from the side.

All of this happened alongside school. Some progress came from writing new code. Just as often, it came from staring at a graph or a result that made no sense and working out why.

The Coach Who Criticized Everything

Once the data was in better shape, I hit the most frustrating moment of the project. I tested ShapeForm on a small set of demonstration videos, and three squats that were labelled correct were all marked wrong. The program was working in every technical sense. It was also giving exactly the kind of feedback that makes someone stop trusting an app.

By this point I had tried several ways of analyzing movement. The biggest version, which I called v5, had three parts working together.

• Rules I wrote myself, such as checking how far the hips dropped.
• A BiGRU, a type of model that reads a movement one moment at a time, both forwards and backwards, so it can use what happened earlier and later in the rep.
• A 3D CNN, a heavier model that looks for patterns across several moments of movement at once.

The most complicated version was not consistently the most useful one, so the final design ended up smaller. It leaves out the 3D CNN, looks at each complete rep, and combines the patterns learned from videos with simple measurements, like how far the hips dropped and how much the body leaned. Even then, the program was too eager to point out problems. So I changed its decision settings, which control how sure the program has to be before it calls something a mistake. Weaker warning signs were now treated more cautiously, and only a possible mistake that was serious enough would count as a wrong rep.

To measure whether this helped, I used a score called balanced accuracy. Normal accuracy can be misleading. Imagine a test with 90 good squats and 10 bad ones. A lazy program that calls every squat “good” would score 90% accuracy while missing every single mistake. Balanced accuracy gives equal weight to getting the good examples right and getting the bad examples right, so that same lazy program would score only 50%, which is what you would get by guessing.

With the new decision settings, balanced accuracy for squats rose from 57% to 80%. The program accepted far more of the correct squats, but it also missed some mistakes it used to catch. The chart below compares the three versions. Removing the 3D CNN on its own lowered the squat and lunge scores, and the big jump came from changing the decision settings.

Get Dayemhumayun’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The balance between accepting good reps and catching bad ones ended up being the main lesson of the project. A coach who criticizes everything gets ignored, and a coach who stays quiet about real problems is no help either. A good one has to land somewhere in between.

Turning It Into a Website

Up to this point, ShapeForm only ran on my laptop. I would start a Python script, point it at a video and read the results in a terminal window. That was fine for testing, but nobody else could use it. I wanted someone to be able to open a link, stand in front of their camera and get feedback without installing anything. To build the website I used Flask, a Python tool for making web pages. My trained models and the rest of my Python code could stay as they were, and Flask gave them a front door that anyone with a browser could walk through.

The biggest design decision was about privacy. Originally, MediaPipe ran on my server, which is the computer that runs the website, so the whole video had to be sent there first. I did not like the idea of people’s workout videos travelling across the internet, so I moved the body-tracking step into the browser itself. Now MediaPipe finds 33 points on the body using the person’s own device, and only those points, written as numbers, are sent to the server. The models on the server judge the rep and send the feedback back. The video itself never leaves the device, and you do not need an account to use the site.

Building the site exposed bugs I had never seen in testing. The worst one involved rep counting. When the program spotted a form mistake, it sometimes threw that rep away completely, so a set of ten squats with a few bad ones could show up as seven. A coach should do the opposite, since a bad rep still counts and it is the rep you most want feedback on. I fixed it by splitting the work in two, so one part of the code counts the rep and a separate part grades it.

Putting the site online brought one more surprise. Everything ran perfectly on my laptop, and then the live version refused to start. The website could not find the trained model files, because the folder locations I had written only made sense on my own computer. Once I changed the code to look for the files based on where the program itself was stored, the site finally came to life.

What ShapeForm Can Do Today

You can try ShapeForm at dayemriyasat.pythonanywhere.com. Nothing needs to be installed, and it works on a phone as long as you prop it up so your whole body stays in frame. In numbers, it covers 5 exercises, tracks 33 body points in every frame and knows how to spot 16 named form mistakes.

Analysing a Lift

Using the site takes four steps, which match the “How it works” section on the home page.

1. Pick your lift. ShapeForm covers squats, lunges, push-ups, deadlifts and bicep curls, and each one loads its own trained model.
2. Frame yourself. You can use your webcam for live feedback, upload a video you already recorded, or check a single photo. Squats, lunges, push-ups and deadlifts work best filmed from the side, while bicep curls work best facing the camera so both arms are visible.
3. Move. The site counts your reps and shows corrections about a second after a mistake happens. To count reps, the program keeps track of which stage of the movement you are in, such as standing, lowering, at the bottom or coming back up. One full trip through those stages is one rep, so a slow squat and a fast squat are counted the same way, while half reps and bounces are ignored.
4. Review. When you press “Finish & save,” the set is saved with its rep count, the number of reps done cleanly and the mistakes that came up.

During a set, each rep gets a score out of 100, along with specific things to work on, such as a squat that was too shallow or a forward lean. A confidence meter on the screen shows how sure the program is about what it sees. When confidence drops below about 60%, the cause is usually the camera setup. If the camera cannot see enough of your body, the site shows “position unclear” instead of guessing, and stepping back, lighting yourself from the front and using a plain background usually fixes it.

The photo mode works a little differently. You upload one picture, and the site shows it side by side with an analysed version, along with measurements such as knee angle, hip angle, squat depth, stance width and how far your back leans.

The Dashboard

The dashboard, called “Your training,” adds up everything you have done. At the top it shows your total time, reps, accuracy, number of sessions and your streak of training days. Below that are a chart of reps per day that switches between week, month and year, a 14-day grid that fills in on the days you trained, a breakdown of reps and accuracy for each exercise, and a list of your most common mistakes under “What to work on next.” Every past set is listed newest first and can be filtered by exercise. Sessions are linked to an anonymous cookie, which is a small note the website stores in your browser, instead of your name, and you can clear your history whenever you want.

The Exercise Library

A warning on its own does not help much if you do not know what to change, so I wanted every piece of feedback to connect back to an explanation. The exercise library has a page for each lift, with a difficulty level, the muscles it works, where to put the camera, how to set up and how to perform a clean rep. The squat page, for example, covers the four mistakes ShapeForm looks for, which are leaning backward, leaning forward, a stance that is too narrow and a stance that is too wide. Each mistake comes with why it matters and how to fix it. Leaning backward is marked as high risk, because it shifts the load onto the lower spine.

The site also has an FAQ, a privacy note that explains exactly what data leaves your device, and a reminder on every page that ShapeForm gives general feedback. It cannot tell if something hurts, and it does not replace a coach or physiotherapist.

How Well It Works

Before getting to the numbers, one detail is worth being upfront about. The upgraded analysis I described earlier covers squats, push-ups and lunges. Deadlifts and bicep curls still use the earlier form-checking approach, and the photo option is limited, since a single picture cannot answer every question about a movement.

The final test results were strongest for squats at 80% balanced accuracy and lunges at 85%. Push-ups were weaker at 64%, partly because the data had very few examples of push-ups done correctly, so the program had less to learn from. These numbers describe how often the program made the right call on labelled test videos. They are a grade for the program, and they say nothing about whether any one person’s squat was “80% perfect.”

What It Still Cannot Do

ShapeForm is still a prototype, and a few gaps stand out. The live webcam mode works, but it needs much more testing with different cameras, rooms and lighting. Push-ups also need far more examples of correct form before the program can judge them as well as it judges squats and lunges. I would also like ShapeForm to get better at saying “I cannot tell from this video” when the camera is moving or the body is hard to see. Giving no answer can be better than giving a wrong one with confidence.

What I Learned

When I started, I wanted to know whether a camera could recognize exercise form. Looking back, that was the easier question. The harder one was whether the feedback would actually help the person doing the exercise.

ShapeForm taught me to question a result even when it looks good, and to test ideas against each other before assuming the bigger model will win. It taught me to be honest about what still does not work. It also showed me why research papers are worth the effort. They gave me starting points, and I still had to turn those ideas into Python code, test them, build a website around them and deal with all the messy parts in between. I am proud that ShapeForm has moved well beyond counting reps. I am just as interested in the problems it has not solved yet, and that is probably the biggest change in how I think about building things.

TL;DR

I built ShapeForm, a free website that watches you exercise through your camera, counts your reps and gives feedback on each one, while the video stays on your own device. It also saves your sessions to a dashboard and explains every mistake in an exercise library. Three research papers gave me ideas to test, and most of the real work went into downloading huge amounts of data, cleaning up confusing labels, handling different camera angles, and fixing a “coach” that criticized almost every squat. Changing how the program decided what counted as a mistake raised squat balanced accuracy from 57% to 80%. The biggest lesson was that feedback is only useful if people can trust it, and sometimes the most trustworthy answer is “I cannot tell.”

- Dayem R.

Sources and Acknowledgments

• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. Sijie Yan, Yuanjun Xiong and Dahua Lin, 2018.
• 3D Pose Based Feedback for Physical Exercises. Ziyi Zhao and colleagues, 2022.
• What to Say and When to Say It. Sunny Panchal and colleagues, 2024.
• Exercise-Corrector project case study, October 2026. All test results and figures in this article come from this case study.