Measuring Image Composition
Who you lookin’ at? You lookin’ at me?
For the past few months, I’ve been working on deconstructing movies and measuring how their multiple expressive media channels combine to give the final piece unique and specific meaning. The idea being that it is the manipulation of these underlying patterns that allows filmmakers to express themselves using the medium. A lot of great work is currently being done to bring more control to AI video generation models in the back end of film production but it does not provide consistency of intent beyond a few shots. In order to make movies with machine learning, the grammar of film language has to be captured.
Composition plays a huge role in this because of its role in representing changes in character states and relationships, as well as in editorial flow. If we can reliably read that signal, we can understand how it is used in existing movies and use it to guide and evaluate artificially generated ones.
What to not do
My first approach was the obvious one. I gave vision-language models a few frames from each shot and asked them to select a specific answer from the usual list of options (rule of thirds, centered symmetrical, off-center balanced, leading lines, frame within a frame, split frame and so on). In the end, even the strongest models failed, disagreeing not only with each other but also with themselves. Tightening the prompts to improve the labeling in one image cause another cause the result in another image to worsen. Ultimately, it became clear that the randomness of the answers from both small and large models was the only consistent aspect of my tests, and so I started shifting my attention from the models themselves to the choices I was asking them to make. Ultimately, the compositional terms I was asking the models to chose from actually belonged to different domains. If you show an AI model a paper plane and ask it: “Is it made of paper OR does it fly. Pick the only correct andwer.”, you’re going to get a random choice of the two every time.
One frame, three different questions. The doorway makes it a frame within a frame, the tiled wall edge splits it, the man’s face sits near a third, and the foreground shoulder counterweights him. Nine model reads returned four answers, because each label answers a different question.
Intended meaning is a main subject off-center with a counterweight across the frame. The models saw a frame split in two and mirrored, balanced on its center, with each character sitting off the center line. The other six read the same placement as rule of thirds, when typically a “rule of thirds” is doesn’t have symetrically laid out subjects each on their own thirds. This could be a 50/50 or center symmetrical.
Don’t let AI prevent you from thinking
After chasing my prompting tail with for a while, I started reflecting on what my experience of “looking” at an image is. Composition is the way the image plane is partitioned to strutcture and organize the visuals information. I don’t have a background in the study of the human visual system but I know that the evolutionary process has given us extremely sophisticated systems for detecting contrast, frequency and brightness. We are also extremely adept at noticing other humans. The phenomenon has been studied and can be measured.
Since it’s about what “pulls” the eye, a useful analogy is to think of it in terms of gravitational forces. Contrast, frequency and brightness are all summed and then boosted inside any detected human shapes. This creates a density field. By multiplying the size of an arbitrary image area by its density, we calculate their weight and get a numerical representation of the image’s visual balance.
Get Thomas Hollier’s stories in your inbox
Join Medium for free to get updates from this writer.
The images below show the process on a single shot. I’ve added others different shots at the bottom of this post for comparison.
Additionally, the way in which these various attributes are arranged on the canvas ends up naturally pulling your eye towards the areas that have the greater mass. This allows you to evaluate how your eye gets pulled based on where your gaze fisrt lands on the image, illustrating whether or not the way you’ve organized the image leads a viewer to what you want them to see, or creates an inherent tension or disconnect.
So what?
Fair question: why do I care about this? The reason it’s important for my larger project is that it provides data that is measurable and consistent across compositions, images and movies. There is no interpretation; processing the same image twice gives the exact same result. This lets you measure how different DP’s and directors use composition as an expressive filmmaking device. In my implementation, I’m mapping these numbers to a small and simple vocabulary: centered, balanced pair, counterweighted, weighted left, weighted right and scattered, with numerical record of where on the horizontal and vertical axis the various weights settle. I’ve manually labeled a large number of shots and in early tests, the system matches my own labels about 65% of the time while the thresholds are still being tuned. The words we chose to describe “composition” are vague but the underlying measurements of the elements’ weights, of the image’s balance, and of the places the eye is drawn to are consistent.
The weights, biases and threshholds I’d settled on are subjective, and I don’t think they are the only valid ones. They just looked right to me as applied across many different images. Ultimately, the most important part is that they are consistent from shot to shot. They measure what’s there, not what it feels like.
Next steps
Down the road, it gives us a signal that can be followed through a sequence: the balance swinging from side to side in a shot reverse shot, two masses merging into one at a turning point, a character drifting to the edge of frame as a relationship breaks down. These are the patterns that carry intent across a scene, and they’re exactly what current generation tools have no way to hold onto. A reliable measure of them is a first step toward using them to guide and evaluate generated sequences, not just individual shots. Big caveat: the way these specific measurement changes over time has not yet been incorporated into the rule set.
Other factors impact composition and gaze: the vanishing point of strong perspective lines increases the pull, and so do saturation and hue. Out of focus area release attention. I’ve experimented with these and they definitely make sense but I don’t want to tackle too many things at once. Every layer that affects composition needs a coeficient and the only way to get the accurate value for them is to test the results against live experiments that track the gaze of viewers.
Some more examples
Here is the same process applied to different types of shots.
