This week my co-authors, Matthew Loper, Michael J. Black, Javier Romero, Gerard Pons-Moll, and I received the PAMI Mark Everingham Prize for the SMPL Model at the ICCV 2025 Conference in Hawaii. I can’t express enough what an honor this is! My sincere gratitude goes out to the PAMI committee for their recognition of our work.

The award, coincidentally, also marks SMPL’s ten-year anniversary and enduring impact on research and industry. But, for me, it has been an absolute pleasure seeing all the innovation, ingenuity and so many novel ideas being built on top of SMPL over the past years. For the uninitiated 😉 What is SMPL?

SMPL

SMPL in numbers

• 100+ companies that use SMPL (AI, robotics, health tech, apparel, gaming, ..you name it!)
• 5000+ citations (just for the vanilla “SMPL Model” — not counting citations for the numerous fantastic variants, MANO, SMPL-H, SMPL-X, SMIL, STAR, SUPR, to name just the major few)
• countless commercial products, opensource repos & academic projects

👆 This is just a sneak peek of what the SMPL body model has meant for the world in these past 10 years since we first published it.

When we first published SMPL in 2015, our goal was simple (literally): to create a realistic, parametric, differentiable model of the human body that could be easily integrated into 3D vision and graphics applications. At that time, we could not have imagined how far SMPL would go — powering research across universities, enabling animation pipelines in industry, and forming the backbone of countless startups and open-source projects in 3D human modeling. Over the past decade, SMPL has become the most cited and adopted human body model in computer vision, machine learning, robotics and AI.

But this is all thanks to the community that was ready and eager to build on all this with us. Whether it was for motion capture, clothing and fit, automation, biomechanics, expressive faces, articulated hands, or training robots to dance, play sports or even give massages: the applications have been boundless. And it is still the community of brilliant technologists and researchers that helps SMPL continue to evolve as a foundation for next-gen digital humans, and motion understanding. I’m deeply grateful to my co-authors, collaborators, and to the many researchers and developers who have carried this work forward.

Here’s to the next decade of building more human-centered technology — grounded in research, and inspired by creativity.

— Naureen Mahmood, CEO, Meshcapade

Further reading :)

• The Story of SMPL
• Building Human Foundation Models with SMPL
• The Ontology of Being Human

10 Years of SMPL: Receiving the Everingham Prize at ICCV 2025 🎉 was originally published in Meshcapade on Medium, where people are continuing the conversation by highlighting and responding to this story.

The latest and greatest version of MoCapade is LIVE: our best-in-class markerless motion capture ⚡

This is a culmination of years of work on accurate 3D motion from monocular video, and there’s still SO much more to come! I have so much to share about the launch and how it came to be but at the same I continue to be so proud of the work we’ve achieved along the way. So first, I’d like to share a bit of history.

We launched the very first version of our 3D Motion from Video product, called MoCapade 1.0, at GDC 2024.

This was a “first look” release: a proof of concept after years of development, our holy grail 🏆 in computer vision. Estimating body shape and accurate motion from a single view is one of the longest researched problems in computer vision and one of the most sought after. Behind the scenes, there is a decades-long history of science publications and research from many key members of Meshcapade that went into the development of our product. From parametric body models, body shape estimation methods, motion extraction methods, pose probability models, camera estimation and tracking models, and finally, tokenized representations of body, shape and motion are all a few of the decades of development just from our small team at Meshcapade. But in spite of all this history, even we stand on the shoulders of giants who’ve moved mountains within the field of computer vision, and without whom the progress we’ve made today would not have been possible.

At GDC 2024, we received some great feedback and support from you all, our amazing community members — in person and online — on what your pain points are, what you’d like to work on improving and how you’d like to use this product in your animation pipelines.

Armed with this feedback, we continued building and launching improvements throughout the year. In just 5 months after GDC, we released MoCapade 2.0 in August 2024. This was the first ever commercial markerless motion capture solution to estimate the 3D human body shape, 3D body motion and the 3D trajectory of the camera, while also using a discrete token-based pose representation of 3D pose [based on our work https://tokenhmr.is.tue.mpg.de/].

Today, with the continuous effort on building and perfecting our motion-from-video process (also coincidently, another 5 months later 😉) we are unveiling our latest and greatest progress with MoCapade 3.0.

Here is a look at all the new features coming together with MoCapade 3.0:

• 👯 Multi-person capture
• 🎥 Export estimated 3D camera trajectories
• 🙌 Detailed hand articulation and gestures
• ⏫ New 3D & video export options: GLB, MP4 and SMPL

All of this, just from a single video. Any video. Even moving handheld cameras, phones and awkward angles!

Introducing multi-person capture for MoCapade

We’re excited to introduce the first version of multi-person capture for MoCapade. Now, you can process videos containing up to four people! This feature is available for Pro account users.

To start using multi-person mode, simply choose MoCapade at the start screen, upload your video and select “Multiple people” on the refinement options screen.

Export the 3D camera

This was one of the most often requested features from 3D animation artists as well as AI and robotics researchers.

With our new GLB file format, we now support exporting the estimated 3D camera and its motion trajectory. The exported camera shows the estimated motion in world coordinates together with the estimated motion of the people in the scenes.

Detailed hands and gestures

We’re so excited that this release brings you even more detailed 3D motion with better hand and foot positioning. Higher accuracy in people interacting with objects and each other — these are some of the improvements you can expect to see here. 🥳

At last: the SMPL format

But along with that, another one of the most significant updates with this release is the creation of the new .SMPL file format. SMPL, our parametric 3D body model, is already the standard representation for 3D human shape & motion widely used across academia and industry. We made it freely available for academia and it has now also become an essential representation for humans in 3D for industries, including apparel, medicine, robotics, motion analysis and AI.

But using and sharing the research models typically used in academia was never easy for most commercial users. So making SMPL into an easily shareable format has been another long-held ambition for @Meshcapade. To be able to do that, we had to make sure that all the different versions of SMPL could be represented as well as the usability of this format was seamless and cross-platform. We’ve made sure that this new format and the respective SMPL codec library remain compatible with the existing versions of SMPL & SMPLX family, but can also accommodate newer models in developments in academia for the realistic skeleton and musculature representations with SMPL.

➡️ You can now easily export avatars from our platform in our new “SMPL” format, compactly saving your avatar and motion data. You can even drag and drop these files back into our platform at any time to view the file in 3D or export them in other formats.

➡️ You can also use our brand new smplcodec Python library to integrate with your own code and take advantage of the open-source SMPL model ecosystem!

GitHub - Meshcapade/smplcodec: A simple SMPL library

SMPL codec for the Unreal’s Live Link & Nvidia’s Omniverse

With the new smplcodec description, one huge benefit is streaming of the human body shape and pose parameters for collaborative environments, like NVidia’s Omniverse. Since SMPL uses linear blend skinning (LBS), the rigging and blend shape information is retained in the USD format. The SMPL model representation encodes the complete 3D body shape and motion of any person with just 100 numbers. This is why academic projects, like VR Kitchen and GRADE, are already building towards this idea.

To support further research and development with SMPL for academia and industry, we are now bringing the SMPL standard together with the SMPL codec library directly into live animation pipelines with Unreal and Omniverse integrations. You can find our current plugins here: https://me.meshcapade.com/plugins

Compatibility with Unreal, Roblox & Fortnite

We’re also releasing tools and tutorials to help import your motions directly into Unreal Engine, Roblox and Fortnite!

Welcoming new investors

And last, but not at all the least, with this launch we’re delighted to welcome our new key investors, HV Capital, LBBW VC, Luc Vincent & Jeff Dean joining us on our adventure!

We’re pushing the boundaries of 3D embodiment for digital characters. Seamless 3D motion capture from video has been one of the biggest milestones on our journey.

We are all immensely grateful for all of the support and encouragement we’ve received from our investors, our customers and the amazing community of creatives and researchers who have been rooting for us!

❤️ Stay tuned for more exciting updates as we continue to innovate and enhance your experience with Meshcapade.

Introducing MoCapade 3.0 was originally published in Meshcapade on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the beginning, there was only mocap!

10 years ago Matthew Loper, Michael J. Black, and I worked on a project that we called “MoSh: Motion and Shape Capture from sparse markers”. Today, this work has been awarded the Test-of-Time award by ACM SIGGRAPH at the #2024SiggraphAsia in Tokyo.

Side-note: Yes, indeed, we intentionally chose the acronym mosh to depict movement/dancing, even if it was #hardmetal kind of dancing. Coming up with cool, pun-filled paper titles that fit nicely to the subject matter is one of Michael’s special superpowers honed over decades of paper writing 😄

I couldn’t make it to Tokyo for the award ceremony, but it goes without saying what an incredible honor it is to have had such brilliant co-authors like Matt & Michael and to receive the recognition from ACM SIGGRAPH. That said, it’s also a delightful happenstance that we are receiving this prize in 2024:

⚡ 2014 was the year when we published #MoSh — capturing true to life 3D motion from any motion capture system.

⚡ 2024 is the year when Meshcapade launched #MoCapade — capturing true to life 3D motion from any video!

Over the past ten years, all the work that has been built using MoSh not just in academia but also in the industry has been phenomenal.

- MoSh is what led to Meshcapade’s motion from video solution.
- MoSh is what led to Meshcapade’s motion from text solution.
- MoSh is what led to Meshcapade’s 3D generative motion models that we’re just starting to release.

#MoSh is the engine that has helped us lay the groundwork for the tons of real and synthetic 3D human motion data we’ve captured and generated for training our best-in-class #AI models. The Test-of-Time award is a testament to the academic significance, but the work is a cornerstone in many more ways beyond academia as well.

Fun fact #️⃣ 1:
Did you know, Michael J. Black is one of only two researchers in the world to have received all 3 major Test-of-Time awards in #ComputerVision? With MoSh, his work in #ComputerGraphics is gaining recognition for excellence as well. I sometimes forget what a privilege it is to be working with someone as extraordinary, yet as humble as he is.

Fun fact #️⃣ 2:
When we started Meshcapade, we knew MoSh would be central to much of what we were building. In fact, I was also finishing up a publication that was a follow up to MoSh, called #AMASS, where we used #MoSh to unify all publicly available motion capture datasets under the standard #SMPL 3D human body format.

So, one of the very first names that the Meshcapade founders (Talha Zaman, Michael J. Black, and I) agreed on, was MoshCraft.

- Mosh: from MoSh
- Craft: from StarCraft (2 of 3 Meshcapade founders happen to be huge StarCraft II fans 😉)

Some of Meshcapade’s earliest business documents even featured MoshCraft in the header. Unfortunately, we later found out that the name was already in use. And so the search for the name continued until we landed on “Meshcapade”.

MoSh wins Test-of-Time award was originally published in Meshcapade on Medium, where people are continuing the conversation by highlighting and responding to this story.

Is it empathy? Is it embodiment? Is it an ability to appreciate the existence of one another?

The model that you see above, was not created by anyone by hand. It’s created entirely by a machine learning model. But it looks like a human, moves like a human. The methods we trained learned to understand what a human is, and generate humans doing different things. It keeps learning and getting better at its understanding of humans as we keep training it with more data about us.

Will we be able to train it to have empathy?
Will we be able to train it to understand and appreciate the world?
Yes, I believe we will.

Classic Star Trek portrayed a future that is built on optimism. A lot of people like to say that kind of future can never happen, the only possibility is a dystopian world. I strongly disagree.

I think our future with AI will be what we make it.

So I want us to make it the Star Trek future. They have embodied AI agents that can interact and behave like real people in virtual spaces and in the real world. But that’s not the hard part any more. The embodied AI agents in this world also have an understanding of human intentions, good and bad. They’re built with an understanding of the world, the people in it, our emotions, and an appreciation of everything around them. The technologies we create will embody our own values.

That said, we do not have to be naive about the implications of AI. Everything that can be digitized, will be. And it will be used to train AIs. But you’ll find people everywhere who strive to not just build these models, but to keep ahead of how they could be misused. We have the tools to build the safeguards. We have the people committed to it. People who want to create a better future. People who don’t back down and will keep trying new ways to do things right.

A future built on AI is an absolute. The only variable is the people who will build it. And to build new innovations that are used for good, we’ll need people with optimism to build them.

Whatever it means to be human, doesn’t have to end with us.

What makes us human? was originally published in Meshcapade on Medium, where people are continuing the conversation by highlighting and responding to this story.

“You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.” — Buckminster Fuller

Last month at a talk at Stanford our Chief Scientist, Dr. Michael Black, unveiled our vision of building the world’s first Human Foundation Agent (HFA). A foundation model built on the template of real people — so that it can see us, understand us, and with that understanding it can interact with us like a real person. We are building an autonomous virtual AI agent that can play, dance, learn and teach, just like us. And we’re building this using our 3D human parametric model, SMPL, as a foundation.

When we first created SMPL, the goal was to be able to take the physical reality of humans in the real world, regardless of whatever modality we capture that reality in, and convert it into a compressed digitized format.

With SMPL we were building the 1s & 0s of humans for computers to understand us. With the HFA we are building on these 1s & 0s to teach computers to be like us.

1. SMPL is a compression algorithm.

For all the applications we use, the 3D attributes of a human are always separated into many different parts: 3D shape, motion capture, body measurements, faces, hands, feet, emotions. So every field of research, industry, and even a different team within the same company in each field has to make their own “one part” representation of the human. The assumption here is that since they are only dealing with maybe just the body motion, or just the body shape, or just the hands or just the faces, they don’t have to train a model for the full human. And then they’re stuck with that. They can’t make that representation encode any of the other details about the real, full human. In movies, where they do have to recreate the full human body, motion, faces, expressions and behavior — they have to rely on artists manually, and painstakingly creating this over hours of work. Different artists covering the body shape, different artists covering the skeletal structure, others doing the motion, expressions. It goes on. But that’s obviously a very limited representation because faces or hands or body motion are not independent add-ons to the human-being. Humans are not modular and our bodies don’t function modularly.

SMPL replaces ALL of these separate representations of humans with a single representation. It encodes body shape, posture, motion, faces, expressions, soft tissue deformations, behavior, together with articulation for the body, hands and feet AND all of it in a format that’s differentiable so that it can be fit to any form of input or easily used to train neural networks.

And it does so with just 100 numbers. Every movement, every emotion, every body shape variation you can imagine for any person can be encoded in just 100 numbers. That’s less than 1KB of data. This is incredibly low dimensionality for training any AI model, be it a virtual AI agent or a robot that need to interact with or move like humans.

→ It inverts the 3D reality of humans into a form that computers can “read”.

2. SMPL is the control signal for diffusion models.

Video diffusion models do not have an understanding of humans. They actually don’t really need it — unless you want to add some form of control over the motion of the humans generated in a video. Video diffusion models encode information into a “latent” space without needing to understand what the latent space encoding means. The latent space is just a compression of all the information available to the diffusion model. It doesn’t have to differentiate between the kinds of information encoded, it all just needs to have a statistically coherent structure that the training process can learn from, and it has to be extremely compact. But that means, there is no way to separate and control the representation of what is a human in this statistical hodgepodge.

SMPL is a statistical distribution, and the most compact representation of humans. Using SMPL in the encoding gives us the control over the encoding so we can drive the generated video with human motion.

→ It’s like having a joystick to control your character in the game-world of video diffusion models.

3. SMPL is the label for all human behavior.

One of the major obstacles to training AI models to understand human behvior is simply that we don’t have enough words to describe the things we do. Everything in an AI model needs to have a “label” for the computer to create a semantic understanding that can be generated using text, image or video input. For image and video models, text has been a sufficient “label” to help computers learn to generate different kinds of pictures and videos. But when it comes to precise control over the things we humans do, human language just doesn’t have enough labels.

Quick, describe your exact pose at this moment in words. If you tell this description to a friend over text, will they be able to replicate your exact pose? I doubt it.

We don’t need words to describe a lot of what we do because we are able to simply observe, understand and recreate the pose and movements using our own mental model of human embodiment and pose. At Meshcapade, we are solving the data & labelling problem for AI models using SMPL. We can embed complete human behaviors into AI training just using SMPL’s compact representation of humans.

→ The 100 parameters of SMPL give us the “language” of human behavior for training AI agents.

4. SMPL is the “canvas” for humans in 3D space.

What’s even more brilliant about SMPL is that it also enables 3D human information to be encoded on a consistent canvas. Consider pixels in an image — we always know pixel 0,0 will be the top left corner of an image (or bottom left, depending on your image renderer).

For AI and machine learning, SMPL very quickly became the de-facto standard for representing 3D humans because anyone who needs to work with 3D human data — whether it’s 3D scans, 2D images, motion capture — they need the 3D “canvas”. This is the one consistent representation of a human in 3D. In an image, a computer needs to understand the 2D grid representation on which point (0,0) is the top left of the canvas, (20,10) is 20 pixels from the left, 10 pixels down, and so on. With that, the computer can analyze the pixels across millions of images and correlate them with other pixels or even with text. But 3D is far more complex. There are no consistent rows and columns. Before any of this data can be used for AI training, the computer needs to learn a consistent way to identify that point (0,0,0) is, say, the tip of right index finger of a human and point (1,0,0) in the same 3D point cloud is the edge of a table. And so on.

If you have no way to create a consistency across all the possible 3D representations of the world, you’ll end up having to do this over and over again for each new 3D scan or motion data that you get. This is why SMPL became so essential for training new models, whether it’s neural nets or transformers. SMPL can be simply fit to any 3D representation: point clouds, discretized volumes, implicit surfaces, other graphics primitives (NURBS, CAD, etc.) or even to images of people in different scenes, and it will recreate the human in 3D from such data. This gives us a consistent way to represent their own data onto a known 3D canvas, which is the SMPL model. The field of human pose and shape estimation cannot move forward without a consistent 3D representation for humans. This is why SMPL became an essential cornerstone for this.

SMPL also defines a volume in addition to the surface. All implicit shape representations of humans use this to canonicalize 3D space by unposing it with SMPL — very useful! So SMPL does more than provide a consistent canvas (surface).

→ SMPL provides a consistent representation for 3D space on and inside the human.

5. SMPL is a 3D graphics primitive.

During grad school I learned about the primitives coded into 3D graphics programs. Planes, cubes, spheres, cones, cylinders — and of course the Utah teapot. This always struck me as such a cool little detail about how we create all of 3D graphics. Yes, it is triangles all the way down. But, if you turn complex forms into an equation, then you can simply create a “primitive” for it in Maya (or blender).

With SMPL, we had done just that. We created a 3D primitive for representing humans in 3D for graphics engines. This is just like cylinders, cubes and spheres that we make inside Maya or Blender. Let’s say you create a sphere inside Maya. There is an equation built into Maya’s API that describes the 3D world coordinates for the center of the sphere, the radius and the points on the surface. You can change the sphere in Maya by changing the radius or the center. In the same way, SMPL lets you create a realistic human body in 3D with just a simple equation. Just like the sphere equation, the parameters of the SMPL equation let you create different human body shapes and poses right inside a 3D environment.

→ It’s like any other equation for objects in 3D engines. It just creates a human instead of a sphere or a cube.

Further reading:

These are some of the papers from just this year’s CVPR 2024 that use SMPL for 3D human pose estimation, motion generation, hand-object interaction, and even training humanoid robots.

I’ll link the X links for the additional explainer in tweet form:

• SMPL & 3D human understanding — ChatPose: Chatting about 3D Human Pose: https://x.com/Michael_J_Black/status/1803436127359213824
• SMPL & robot motion — HumanPlus: Humanoid Shadowing and Imitation from Humans: https://x.com/zipengfu/status/1801283564350411020
• SMPL & 3D pose & shape estimation — TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation: https://x.com/Michael_J_Black/status/1798602004610342928
• SMPL & 3D motion control — Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation: https://x.com/MathisPetrovich/status/1802530924552466888
• SMPL & 3D motion generation — WANDR: https://x.com/Michael_J_Black/status/1796509572842258602
• SMPL & detailed facial expressions — SMIRK:3D Facial Expressions through Analysis-by-Neural-Synthesis: https://x.com/DanecekRadek/status/1802761423585722706
• SMPL & hand-object interaction — HOLD: https://x.com/Michael_J_Black/status/1791423005635031265
• SMPL & inference of 3D human muscle & tissue — HIT: Human Implicit Tissues: https://x.com/Michael_J_Black/status/1802056037186601162

Also, check out Meshcapade’s Science publications page which contains a lot more publications (past and present) from our team about topics related to humans in 3D: https://meshcapade.com/science

Building Human Foundation Models with SMPL was originally published in Meshcapade on Medium, where people are continuing the conversation by highlighting and responding to this story.

Playing with wires.

Growing up, I was always the electrician at home. People who’ve grown up in Pakistan will know — there is no such thing as good wiring anywhere in Pakistan.

No, seriously. There just isn’t. We had short circuits all the time. The first time we got a microwave, we couldn’t have it running at the same time as any air conditioner in the house. The electricity grid for most (all) cities is also in shambles. Electricity gets cut off on a daily schedule for “load-shedding”, but whenever it got turned back on by the city — you better have your important electronics unplugged or else the power supply might get fried due to an electric surge. Surge protectors were a luxury — still true for most towns.

As a kid with no concept of the cost of things, this always felt like a great opportunity to open up a new electronic device to see if I could fix it. Whenever things broke down, I’d be excited about it.

By grad school, I was the one fixing CPUs and laptops, mostly upgrading them, for other students in the girls’ dorms. The most impressive and intricate devices I upgraded back in the day were macbooks, no surprise there. They’re almost works of art. But, honestly, I felt privileged to have had the opportunity to play with electronics so freely from a young age so they didn’t scare me. My mother encouraged it. She’s a fixer herself. She doesn’t do much electronics, but she’ll build you a designer shelf with her own hands in less than a day!

Robots can fold laundry.

The first time I attended SIGGRAPH was as a Student Volunteer in 2010. It was during my Masters at the VizLab department at Texas A&M and I was super excited that I could attend (thanks to SIGGRAPH’s awesome student volunteer program!). One of the jobs I had to do as a student volunteer was to help the folks demoing their new projects in the “Emerging Technologies” program. The one team who’s demo I was part of that I’ll always remember was these very nice guys from Japan who had made a laundry folding robot. They called it FOLD-E, a nice little homage to WALL-E that had just recently come out. I thought it was all brilliant — the name, the robot.

It was a prototype of a robot using computer vision and motors to solve a fairly complex problem — clothes are very deformable. It worked well. You had to lay the shirts in a very specific way. Then you had to have a very specific lighting. They brought their own lights. The robot would take a moment to analyze the clothes, then start folding. The robot worked, but very slowly. Sometimes it got confused and crumpled up the shirt so you had to set it right again. It could only work on t-shirts and pants. They told me they were working on programming it to fold dress shirts next. I got to demo this robot to a number of folks, and I got to troubleshoot it quite a few times whenever it got stuck. It was a good learning moment for me to realise how slowly robots will actually evolve with the technologies we had at the time.

It’s 2024 now, and we still don’t have a laundry folding robot. Turns out, folding laundry has always been a much more complex problem than we anticipated.

God bless the Arduino!

I never had actual electrical engineering education. It wasn’t until grad school at Texas A&M where I got a real taste of soldering electronic circuits for LEDs, sensors, motors, actuators, etc. and then controlling them through code — thanks to the mighty little Arduino! This was during the one short course at the VizLab Department at A&M, newly introduced by the phenomenal Phillip Galanter. Funnily enough, the course was called Generative Computing — would have been a very popular course these days, what with the hype for generative methods nowadays. But back then it was just a handful of people in the class. Once I got the hang of how to work with sensors, how to program them to respond to certain inputs, and how to control motors and other end effectors with code — I made Heidi.

Heidi.

Heidi was a shy little robot. In a crowded space, she’d keep her head tucked in so that nobody could see her. She was shy, but very curious about people, so she’d perk up when she saw someone approaching. She had mismatched eyes, one green, one yellow. She was ticklish on her ears so if you scratched either of the ears she’d start giggling. She was sensitive about her nose though. “Do not touch my nose”, said a sign next to her. People would still do it and she’d immediately duck her head back in. If you kept bugging her like that, she’d duck her head in and roll away from you.

I was also a bit too shy back then to talk about things I built. It was interesting to me, and I knew the things I build were super awesome. But only to me. So I didn’t tell people much about Heidi back then — I just quietly brought her to our annual VizLab studio demos. But since then, whenever I’ve told people about Heidi, they’ve loved the story. Young and old, tech / non-tech, and especially groups of younger girls at our Open House sessions in Germany or at conferences. It makes them realise they can also build their own little robots. It makes them realise they can make any kind of electronics. So that’s why I tell people more often about Heidi now.

Someday, I’d like to go back to building more robots with emotions, behaviours and personalities. When I joined Max Planck Institute to work at Michael Black’s lab as a scientific engineer, I told Michael that if I ever did a PhD, I’d want to do it with Cynthia Breazel. I want to build robots with emotions and interactions like people. I think Michael took that as a challenge. He is now my co-founder at Meshcapade and we’re now starting to make interactive behaviors for robots that live inside the computer for now. With a little bit more time and a few more people at Meshcapade, we’ll also finally bring these robots into the physical world!

Further reading:

• SMPL as robot: https://perceiving-systems.blog/en/news/smpl-as-robot
• The Heidi Project: https://howdweknows.blogspot.com/2010/05/final-project.html
• FOLD-E: https://techcrunch.com/2009/09/10/fold-e-clothes-folding-robot-demoed-at-siggraph/

Happy little robots was originally published in Meshcapade on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the summer of 2014 I was the one scheduled to give a talk at our weekly department meeting. I used to dread these meetings. I often got away without having to present because a PhD student or PostDoc would want the spot to practice their presentation for a rapidly approaching paper submission deadline. I was happy being an engineer with no rapidly approaching deadlines. It wasn’t because I didn’t have things to talk about. I had tons. I was always on 3 different projects at any given time! I just really hated public speaking (maybe I still do. Can you tell? 🙂). Michael, who was our department director, insisted that everyone working on a project should have a chance to talk about their work.

Well, it finally happened. This time I couldn’t get out of it. Darn it.

The dreaded moment arrived and I talked about the beginnings of an idea that Matt Loper, Michael and I had been noodling around in our heads. A standard 3D body mesh with a standard skeleton with blend weights and pose corrective blendshapes. All learned from 3D scans. The novelty was in learning the deformations on vertices instead of triangles, as done in older models like SCAPE and BlendSCAPE (our lab’s own, very heavily used parametric body model). Making this switch to vertices meant this model could finally work with standard 3D blend weight rigging used in all graphics engines. The second important thing was to make the deformations linear. Making them linear meant the model could finally work with standard 3D blendshape deformations used in all graphics engines.

We called this concept the “character compiler”. This was the first time we talked to our group about the idea that evolved into what we now call SMPL. We wanted to train a 3D shape space of humans and the deformations due to motion, make it differentiable and also make it compatible with standard graphics so that we could animate it easily in Maya or create a game-compatible rig for Unity and Unreal.

My talk was just focused on the 3D character creation aspect. How 3D rigs are used in games and how blend weights work, how blendshapes work. To show our team how the eventual SMPL rig would be used in a 3D engine, I rigged a couple of our 3D body meshes from our older BlendSCAPE model. Fun fact, I remember one of the 3D meshes was Eric Rachlin, co-founder of Body Labs, the ever willing participant for cool projects! I loaded up some motions onto these 3D rigged meshes using Mixamo. I put them in a dummy game level in Unity to show everyone how a 3D animation with SCAPE-like models would work. “3D Eric” could run around, climb up stairs, jump off with a somersault from great heights and walk through walls. Only his blend weights were a bit off.

In total, it took Matt just around 2 months to train the core model. Yes, Matt is really quite brilliant. But it wasn’t a straight line. We were aiming for a SIGGRAPH submission. For the uninitiated, a SIGGRAPH submission means missing Christmas and working in a sad, cold lab through December. The first version of the model, which we still called “character compiler”, was completed by early January 2015. Matt, Michael and I worked on it through the new year doing and re-doing all the math, testing, running evaluations, putting together all the paper sections, creating the videos. It was a race. SIGGRAPH submissions are no walk in the park!

But, then came the lows. Matt realized the results weren’t good enough yet when testing the new model with a new dataset. With one week to go until the submission, we decided to pull out of SIGGRAPH. Matt took a weekend to be depressed. Another week to think it over. By mid-February, we had a new model, performing better than anything before! Now we had more time to try even more things. We pulled in Javier Romero for even more meticulous evaluation and testing. We also brought in Gerard-Pons Moll, to learn soft-tissue dynamics from his earlier work, DYNA, so that SMPL models not only had pose-dependent soft-tissue motion, but also tertiary soft-tissue movement — you know, jiggliness!

At this point, Michael had a brilliant idea: let’s lean in on the fact that we’re trying to make this model as “simple” as possible for 3D graphics engines. We were literally adding a machine learning component into the super-standard 3D graphics equation that had been used for inside Maya, Blender and other graphics engines for years. So we decided to give the model a new name: SIMPLE — Skinned Multi-Person Linear model. We all loved the idea and we high-fived each other for such a perfect name to describe what we wanted. It stuck for just about a couple of days. We couldn’t easily find each other’s emails about this project using “SIMPLE” as a search term anymore. It was too common a word.

So, SIMPLE then became SMPL just to make it easier to search :)

With the new name and incredible performance, we decided to submit it to SIGGRAPH Asia. Out of 5 reviewers, only 2 voted it as a clear accept. Others were on the fence. We managed to convince them with the rebuttal, and so SMPL finally made it to the ACM Transactions on Graphics (TOG), the foremost peer-reviewed journal in the field of graphics.

And now, as I write my little history of the model, SMPL has moved to the top 3 most cited publications in the history of all of ACM TOG publications! https://dl.acm.org/action/doSearch?fillQuickSearch=false&SeriesKey=tog&sortBy=cited

This may not mean much to most , but it’s a pretty big deal to me. ACM publications include SIGGRAPH, SIGGRAPH Asia and many other top-rated 3D graphics journals, publishing research since 1982 from giants in the 3D graphics field and companies like Disney, Pixar, Weta and Epic.

When we started working on the character-compiler project, little did we know we’d be starting a whole new movement of 3D technologies built around our work for 3D human understanding!

If you’d like to read more about the paper, you can find the project website here: https://smpl.is.tue.mpg.de/

If you want to see the latest capabilities built on SMPL for commercial use, try out the Meshcapade platform here: http://meshcapade.me/

There are also a few newer versions of SMPL, you can find them here:

MANO & SMPL+H: http://mano.is.tue.mpg.de/

SMPL-X: https://smpl-x.is.tue.mpg.de/

STAR: https://star.is.tue.mpg.de/

SUPR: https://supr.is.tue.mpg.de/

The story of SMPL was originally published in Meshcapade on Medium, where people are continuing the conversation by highlighting and responding to this story.

As we wrapped up at GDC 2024 a few weeks ago, I started receiving ever more requests from people wanting to know the background of Meshcapade, how we got started and my thoughts and vision around it. So I decided I’d bring everyone along on the journey of Meshcapade with a blog series to start sharing the stories about the past, our vision, and my thoughts as we move forward.

For the uninitiated, Meshcapade is a tech startup building foundation models to enable 3D digital humans for AI, gaming, VR, fashion & healthcare. Meshcapade was founded by Talha Zaman (CTO), Michael Black (Chief Scientist) and me, Naureen Mahmood (CEO), in the quaint little, fairytale town of Tübingen in 2018. Now, we’re a stellar team of 26 (and growing), spread all across the globe. In our quest for helping computers understand the world, we’ve spent years building a whole suite of features that help machines perceive how humans look, move, talk, interact. The latest features we were super excited to showcase at our first ever GDC expo this year were text-to-motion and markerless mocap. See our promo for GDC here.

Before getting into the hows and whys — let me start off with something I’ve been thinking about a lot lately. In the movies, especially in sci-fi, there’s often a futuristic vision of having machines that interact with us on an almost human level. Why, yes! In some distant future, of course we’ll have robots that will talk to us and have thoughts and personalities. Even the disembodied thinking machines of our sci-fi worlds, Jarvis from the Avengers or HAL from 2001: Space Odyssey — it seems perfectly logical for them to exist in a somewhat distant future.

But for a machine that can do that, there are so many modalities of “human-ness” that need to be learned, and pretty much all of them have a lot to do with Perception. The ability to understand the world through vision, touch, sound. Even just to interact with each other, we humans rely on our senses to see and to hear each other, to understand our motions, behaviors, expressions, emotions, gestures and our words.

When we started Meshcapade, what drove us all was an ambition, a dream, to solve these perception problems. At our core, we are computer vision scientists. But that enthusiasm very easily expanded into computer graphics, machine learning and AI — all the tools available to us to solve a century old problem of helping machines understand and interact with the world around them!

When I first came to Germany 11 years ago, it was to join the Perceiving Systems Department at the Max Planck Institute for Intelligent Systems (MPI-IS) in Tübingen, led by Michael Black. My best friend in Pakistan, a brilliant humanities major, asked me what I’ll be doing there. I told her I’d be working on computer vision. She said, “What’s that?”. I started telling her about how to detect different kinds of features in images, I excitedly told her how one can track corners and edges in a photo or videos, how to use these to perceive motion and so on. Of course, I was making no sense. She wasn’t asking how, she was asking what I would do. When she still looked puzzled after my spiel, I had to think for a bit.

Then, I sighed and told her what it was really all about, “I just want to teach robots to see!” That answer gave her immense joy. Since that day, every time she mentioned me to someone, that’s what she told them :)

So, that’s precisely what I did at Max Planck — with many, many amazing people. And now with Meshcapade, we’re getting to bring all the groundbreaking research we’ve worked on to real users everywhere.

And yes, our tech is indeed used to train robots to see. And so much more too!

Michael, who is the brains behind many adventures in computer vision, was also the mastermind behind Meshcapade. It was a crazy idea to me at the time he suggested it. There was no doubt in my mind that there should be a startup that takes over the fantastic work we’d all been doing at Max Planck, to continue it, to make it bigger and better so that everyone, from big tech to the indie creators could use it.

The vision was clear. The potential was clear. There was no doubt in my mind about how important it was. I just never imagined myself taking the reins. I knew very few examples of women founders / CEOs, and zero examples of women founders who looked like me, wore a headscarf. But between Michael and Talha, the two convinced me. Their main argument was, if not you, who else?

So we dove in. I was the jack of all trades, but I can also be the brute force implementor with a slight inability to understand the word impossible. That’s essentially how we managed to launch one of the very first AI startups in the tiny town of Tübingen in Germany (and now Tübingen even has a whole AI startup ecosystem, CyberValley). Michael became our Chief Scientist and builder of grand ideas. Talha Zaman, the meticulous software engineer and avid gamer (it’s frustrating how much better he is at every game), was the missing piece to bring it all together from big ideas and dirty research code to real, elegantly designed software where all the moving parts flow seamlessly together.

I think Talha could write a book on how badly researchers write code. I might get my own chapter in it.

An Unexpected Party was originally published in Meshcapade on Medium, where people are continuing the conversation by highlighting and responding to this story.

For professional Blender users and if you prefer to go to the repo and README directly, follow this way.

Step 1: Acquiring the Plugin

First things first, let’s get you set up. You’ll find the plugin along with detailed documentation on our github. Don’t worry about tech jargon — we’ve got you covered with installation instructions for Windows, iOS, and Linux.

Step 2: Avatars, Your Way

Now, let’s talk avatars. We’ve got a couple of avenues to bring them into your Blender universe — be it from a bunch of measurements or a simple 2D image. Pay a visit to meshcapade.me, our interactive avatar creation platform. There, you can craft up to 5 avatars daily without spending a dime. For those looking to level up, grab some credits and enjoy crafting even more avatars.

Step 3: Perfect Poses with a Twist

Your avatars are in Blender — now let’s make them move and groove. Pose, dance, or strike a pose — it’s all within reach. Work your Blender magic on your Meshcapade avatar, then hit the “Calculate Pose Correctives’’ button. Planning a dynamic animation sequence? No problem! For animations, use the “Calculate Pose Correctives for Entire Sequence” option.

Step 4: Expressions That Speak Volumes

Let’s make those avatars emote! SMPLX and SUPR bodies are equipped with facial expression support. You’ll find 6 pre-set expressions, a “Random Facial Expression” button for a twist of spontaneity, and a “Reset” button to return to the default look. For the enthusiasts, dive into finer control by tweaking shape keys under “Object Data Properties.” It’s your chance to let avatars wear their emotions on their sleeves!

Step 5: The Hidden Powers

Curious about avatars’ secrets? Click the “Modify Metadata” button to play around with gender and SMPL versions. A word of caution — tinkering with this might lead to unexpected results. Body monsters incoming!

And don’t miss out on the nifty tools. For the data scientists and machine learning wizards, the “Write Pose” buttons are your gateway to poses in SMPL format.

******************************************************

Bonus Step for our SMPL License Customers

If you already have a SMPL model license for commercial or academic use, check out the following features that are only available for you. Just shoot a message to sales@meshcapade.com, and they’ll set you up.

Interested in getting a SMPL R&D or commercial license for your business? Check out our website.

Bonus — Step 6: Shape, Pose, Repeat

Ready to shake things up? With our bonus data folder, our licensed customers can unlock even more avatar potential. Adjust the avatar’s height, refine their weight — it’s like a tailor shop for avatars. The joint locations update automatically. Want to dive deeper? Utilize Shape blend shapes to fine-tune avatars’ forms. Remember to tap the “Update Joint Locations” button afterward.

But that’s not all! Load in poses with .npz files like a pro. Don’t forget to select the right up-axis in the import options.

Dive into the world of avatars and transform your Blender scenes into captivating spectacles. The Meshcapade Blender plugin is your key to unleashing the magic. Let your creativity flow and your avatars shine!

Embrace Avatars in Blender with Meshcapade Me Plugin was originally published in Meshcapade on Medium, where people are continuing the conversation by highlighting and responding to this story.

Big day!

I’m incredibly excited to announce that we have raised a $6M seed round led by the amazing Diana Kimball Berlin at Matrix and joined by some brilliant tech minds, including Nate Mitchell (Co-Founder of Oculus), Matthew Ball (Author of “The Metaverse”), William O’Farrell (Co-Founder of BodyLabs), Andrew Hamel (Former VP Machine Learning at Amazon), Gary Bradski (CSO at OpenCV), CLO 3D (provider of leading 3D clothing software), and others.

We launched Meshcapade in the spring of 2018, and until now had been chugging along on a slightly different tune than most tech startups. We started off with customers already at the door. It was great. It was scary. We had to hit the ground running, and give ourselves a crash course on building a business that could immediately serve the customers we had right away, while also working constantly towards the bigger goal of what we envision our technology can do. Along the way, we also brought on a few amazing angel investors who have continually helped us with their advice, time and unwavering support because they also understood the vision and jumped in wholeheartedly.

So, what are we building?

Today, everything we’ve achieved with AI is just the tip of the iceberg. Despite the breathtaking progress in large language models (LLMs) and generative AI, today’s AI systems still struggle to understand how we humans move, interact with the world, and express ourselves. Yet, the most important pieces for all real world applications will always be the humans who interact with them.

As we look towards the future, it is clear that for us to effectively use AI in our daily lives, it needs to understand how we look and behave. This is where Meshcapade and our SMPL Standard for 3D Humans step in, aiming to provide the “digital human layer” of the internet. From clothing to gaming, from VR to healthcare, Meshcapade’s technology provides the foundation for all applications to transport the physical reality of people — talking, moving, interacting with the world — into the digital space.

Our mission is to make technology more human. Since 2018, we have been building the “digital human layer” of the internet that supports all business applications that must “see” or generate humans. Starting with the patented SMPL body model, we built the standard representation for all AI, graphics and computer vision needs. Through easy to use APIs and web-based apps, we have been building instant avatar solutions that allow users to convert images, videos, body scans, motion capture data, text, and measurements into 3D digital humans.

We are creating the class-leading foundation models for the analysis and synthesis of 3D human motion. Our platform takes an image or video and generates a precise 3D model of the body shape and pose, making it possible for businesses to see and generate accurate 3D motion. This is all part of a broader goal: To provide the 3D digital human foundation for every industry.

Everything we’ve built on the Meshcapade platform is powered by the SMPL Standard family of body, face, hand, and foot models, the only commercially available solutions for 3D humans that use machine learning to represent accurate body shape and motion. Trained on over one million 3D and 4D body scans, the SMPL Standard is unparalleled in its accuracy and compatible with all game engines.

Since we launched in 2018, we’ve been growing rapidly, becoming a trusted provider for the SMPL Standard and for 3D human solutions for some of the world’s largest businesses including six of the top ten NASDAQ companies. We serve over 100 customers in many different verticals, ranging from Microsoft to Zalando. It’s a testament to the importance and value of the vision that we started with, and a clear indication of the dire need for the “digital human layer” in today’s business landscape.

Give it a try

You can try out our instant 3D avatars at me.meshcapade.com.

Next stop: foundation models for analysis and synthesis

Talha, Michael and I embarked on this journey together and knew one thing with absolute certainty: there are no absolute certainties in AI. It is an extremely exciting field which has yet to yield many more amazing results. The other thing we all knew for sure was that we will continue to learn and be excited about where this space is going. With Michael bringing in his pioneering work in computer vision and 3D humans, my background in computer graphics and the seminal work on creating the SMPL model together with Michael, and Talha with his years of software engineering expertise that would be critical to building the platform we envisioned, we set off.

It has been an exciting journey so far and with this investment round we are aiming for mankind’s next giant leap towards making foundation models for the analysis and synthesis of 3D human motion and appearance.

Thanks for joining the ride!

First off, I want to express how thankful I am to our customers, all of whom have helped us grow from a small trio with a vision to where we are today. As we look towards the future, the human element in AI will become more critical than ever.

Secondly, and most importantly, I am grateful to and immensely proud of the team we’ve built together at Meshcapade. They are the ones who have taken our vision and made it their own, and are able to keep surprising us founders constantly with newer, better ideas everyday. I can’t wait to see what lies ahead for Meshcapade and am excited to continue building the future of 3D digital humans with all of you.

Want to join the adventure?

We are a team of passionate creators from a variety of backgrounds, seeking to change how people generate, think about, and make use of digital human avatars: https://meshcapade.com/careers

You can find more information about Meshcapade at meshcapade.com, read all about the team at https://meshcapade.com/about-us.

For questions, please feel free to reach out at info@meshcapade.com.

Meshcapade’s seed round was originally published in Meshcapade on Medium, where people are continuing the conversation by highlighting and responding to this story.