Building a Synthetic Media Engine

Synthetic media generated by AI

Introduction

For a while now, I’ve been seeing reels and other types of content where generative AI has taken center stage in creating surreal situations — from talking objects with hilarious stories to realistic-looking characters and scenarios.

There are plenty of websites offering this technology as a service. But as a good engineer, I can’t just use these services and call it a day. I need to be able to build my own lab and understand how this technology actually works.

AI is expensive. The hardware required to run these models, along with the energy consumption, is not exactly within reach for the average home user. However, everything can be scaled down.

I don’t need to produce studio-quality results. I just want to find out whether I can generate something similar, even if the quality is significantly lower.

And the good news is: yes, I can. Mission accomplished! 🚀

Disclaimer

Some of you may already have another term in mind when reading about this technology — a term that, because of its history, carries quite a bit of negative baggage. I’m talking about deepfakes.

Technically speaking, yes, some of what we are doing could fall under that term, especially when we use reference images, even when the result is a surreal situation where it is obvious that the content is AI-generated.

The negative connotation largely comes from the way the technology became known to the general public, particularly through non-consensual adult content using the likeness of celebrities and other public figures. But not all generative content carries that context. As mentioned above, clearly AI-generated surreal or humorous videos could technically fall under the same umbrella as well.

However, precisely because of that historical baggage, I will avoid using the term throughout this series and refer to this technology as synthetic media instead.

That said, the goal of this lab is to democratize knowledge and give people the tools to understand how this technology works. What each person does with that knowledge is ultimately their own responsibility.

Personally, I live by a very simple rule:

Don’t do to others what you wouldn’t want done to you

Process workflow

The goal is to start with a reference image, a descriptive prompt, a voice sample, and a text script, and use them to produce a video in which the described scene takes place, with the character speaking the provided script.

We will split the process into several independent components, connected together as a pipeline. The output of one component becomes the input of the next until we eventually reach the desired result: a synthetic video with a generated voice and synchronized lip movements.

Workflow

1. Reference image + prompt → Synthetic image with preserved identity
2. Generated image + prompt → Synthetic video (without audio)
3. Voice sample + text → Synthetic voice
4. Generated video + synthetic voice → Lip-synced video

Notice that step 3 can run in parallel with steps 1 and 2. Once both branches are complete, their outputs are passed to step 4, which generates the final artifact.

Technologies Involved

Personally, I always work with a Linux environment for this kind of thing, but it is not strictly necessary. What will be the core of the entire AI stack, however, is Python, which has become the undisputed leader in this space.

Get Sergio Perez Rodriguez’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

For the generative environment, we are going to use ComfyUI. It seems to be where the community is heading, and if there is one thing my 15 years of experience have taught me, it is this: don’t always go for the prettiest or most powerful option; follow the technical trend.

Otherwise, you may end up facing limitations around community support, forums, third-party development, integrations, and so on.

The only independent component will be voice synthesis. For this, I found XTTS more useful than trying to bring everything under ComfyUI. The reason will become clear later, when we turn each component into a stateless microservice.

But that’s a spoiler… we’ll get there later!

Index

I’ll add the links here as each article is published. The final index will be:

1. Part 0 — Setting up the Environment
2. Part 1 — Identity-Preserved Image Generation
3. (WiP) Part 2 — Bringing Images to Life
4. (WiP) Part 3 — Generating Synthetic Voices
5. (WiP) Part 4 — Making Characters Speak
6. (WiP) Part 5 — Final Architecture

Conclusions

Building a Synthetic Media Engine turned out to be much more interesting than simply getting each individual model to work.

What started as a collection of experiments inside a single environment evolved into four independent capabilities: identity-preserved image generation, image-to-video generation, synthetic voice generation, and lip sync. Each one brought its own models, dependencies, limitations and, inevitably, its own share of things that broke along the way.

One of the main lessons from the lab is that the interesting engineering problem is not necessarily the generation itself. Getting a model to produce an impressive result is only the first step. Making the different capabilities composable, reproducible and resilient is where the system starts becoming an actual engine.

This is also why the final architecture deliberately separates the components. Each capability can evolve independently, expose a simple interface, and be orchestrated asynchronously through queues and object storage. The models may change, the workflows may change, and new capabilities can be added without having to rebuild the entire system around them.

There is still plenty of room for improvement. Generation quality, consistency, latency, GPU utilization, model lifecycle management and cost are all areas that would require considerably more work before calling this a production-ready platform.

But that was never really the point of the lab.

The goal was to understand the path from generative AI experiments to an engineered media pipeline — and, more importantly, to see where the hard problems actually are.

And if you’ve made it all the way to the end of the series, you now know what it takes.