Last week, I managed to check off quite a few tasks. While Oscar and Philip worked on solving and retargeting the remaining animations, I was able to cut and integrate them into the project.

A major milestone was successfully linking the puppets to the AI. Now, they can engage in conversations on stage and play the corresponding animations that match their dialogue. Seeing this system come to life has been one of the most satisfying moments of the project!

I also finished modeling and texturing the Mephisto puppet, which was another big goal of mine.

Additionally, I continued refining the physics, specifically trying to implement limb grabbing. However, I still need to make improvements. Right now, there are some issues when grabbing a puppet with a second hand, so that’s something I’ll be fixing in the coming days.

With just one week left, we still have plenty of work ahead of us. It’s going to be a sleepless final push, but I can’t wait to see how everything comes together in the end!

Semester: P5
Group 3 THE BOYS: Aaron Höhn, Philip Owor, Oscar Pohl
Author of the story: Aaron Höhn
Date: 29.01.2025

The Final Stretch: One Last Week to Go was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

While Oscar began the long and detailed job of cleaning up all the motion capture data, I switched gears to finalize the user tutorial. This feature is now fully functional: a floating hand hovers over the tutorial puppet and shows the user exactly what to do, step by step. First, it demonstrates how to grab the puppet, then it points to where the puppet needs to be placed. Once the tutorial is complete, additional puppets appear, ready for the user to interact with.

I also got started implementing some of the cleaned animations that Oscar worked on and added the finished Charlotte puppet of Philip to the project. Additionally I was playing around with cloth physics and added it to Goethe´s coat and Charlotte´s skirt. It’s great to see everything coming to life in the project!

What’s Next?

My next big task is to create a system that detects which puppets are in the theater and links that information to our AI. This will allow the AI to take over, syncing animations with conversations in a seamless and dynamic way. If things go smoothly and I have time left, I’m planning to start working on adding a Mephisto puppet to the project as well.

Semester: P5
Group 3 THE BOYS: Aaron Höhn, Philip Owor, Oscar Pohl
Author of the story: Aaron Höhn
Date: 22.01.2025

15.01.-22.01. A productive Week was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Polishing the Motion Capture

Of course, raw motion capture data is never perfect. Sometimes the trackers don’t work properly or get blocked, which can lead to weird glitches like clipping or jittery movements. That’s where Oscar came in. He spent several days in the studio cleaning up all 80 animations (yes, 80!). Thanks to his hard work, the animations look amazing now.

While Oscar was busy fixing the data, Aaron and Phillip didn’t waste any time. They started retargeting the animations to the puppet rigs and even got some of them integrated into the project.

AI Dialogues: Conversations Come to Life

We also hit a big milestone this week by getting our AI dialogues up and running. We can now simulate full-blown conversations, complete with emotional undertones. Check out this snippet of a chat between two versions of Goethe:

What makes this feature even cooler is that the AI also handles the emotions, which we’re linking directly to the animations. So when a character gets excited, sad, or frustrated, you’ll see it in their movements too.

New Features and Final Touches

On top of all that, we finished the tutorial this week. It’s super straightforward for the user: a floating puppet with a hovering animated hand gestures for you to pick up the puppet and place it in the theater. Once you do, more puppets appear so you can interact with them. It’s simple, but it sets the stage perfectly.

We also wrapped up the Charlotte puppet and added her to the project. Plus, we gave her skirt and Goethe’s coat cloth physics, so their movements look way more realistic now.

Oh, and the theater curtains? They’re finally animated! We hit a bit of a snag because Blender doesn’t export shape keys in a Unity-friendly way, but we found a workaround and got it done.

What’s Next?

Next week, we’re tackling the big one: syncing the AI dialogues with the animations. This means connecting the emotions in the conversations with matching movements for the puppets. It’s a key part of making everything feel natural and immersive.

Semester: P5
Group 3 THE BOYS: Aaron Höhn, Philip Owor, Oscar Pohl
Author of the story: Aaron Höhn
Date: 22.01.2025

Weekly Project Update: Our Most Productive Week Yet! was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

First, I revisited the Goethe puppet because I wasn’t at all satisfied with the face. Now, I’ve given the puppet a face I’m finally happy with.

Additionally, I integrated the new Goethe puppet into our project and implemented ragdoll physics for it. Initially, the physics didn’t work at all as intended. Every time I picked it up, the limbs stretched like rubber bands and flung around my workspace.

With some adjustments to the settings and finalizing the scripts, I was able to resolve these issues.

On Thursday, we’ll be recording our animations. I’m really looking forward to integrating AI with our puppets and animations together with my team. Although this will be a very big and challenging task for us.

Semester: P5
Group 3 THE BOYS: Aaron Höhn, Philip Owor, Oscar Pohl
Author of the story: Aaron Höhn
Date: 15.01.2025

08.01.-15.01. Puppet Physics was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Even over the festive season, our team managed to make some exciting strides in our project, and we’re eager to share the progress with you.

This week, we added the functionality to grab our puppets. Alongside this, we enhanced the state machine so that after releasing the puppet, they seamlessly transition back to their previous actions. The next big step here will be integrating ragdoll physics into this system, but first, we need to finalize our new puppet models.

Our work on the puppet models has also taken a significant leap forward. We created new concept art and drawings. This effort has made it much easier to achieve accurate and fitting proportions for our puppets.

Admittedly, modeling human figures was a daunting challenge for us as we’re not experts in this field. Our initial results were far from satisfactory, which was disheartening at times.

However, through perseverance and practice, we achieved results that meet our standards and serve as a solid foundation for further development.

We’ve completed a first model for the Goethe puppet and have started texturing it. Simultaneously, we’ve begun work on modeling the Charlotte puppet.

Another exciting milestone is the successful implementation of the ChatGPT interface in Unity. This marks the beginning of an innovative feature that will generate dynamic puppet conversations. At present, we’re experimenting with executing and fine-tuning the outputs to ensure smooth and engaging interactions.

As we move into the new year, our focus will be on:

1. Finalizing and rigging the puppet models, including the addition of ragdoll physics.
2. Perfecting the character texturing and modeling process.
3. Refining the ChatGPT integration to create realistic and contextually appropriate dialogues.

Stay tuned for more updates!

Semester: P5
Group 3 THE BOYS: Aaron Höhn, Philip Owor, Oscar Pohl
Author of the story: Aaron Höhn
Date: 01.01.2025

Holiday Progress and New Milestones was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

A City of Culture

Shortly after our arrival we went through the center of the city. Along the way we saw multiple museums and cultural monuments with pictures of influential people. The streets were clean and all the people we encountered were academically dressed. At the center we saw the statue of Goethe and Schiller after which we proceeded to the Goethe National Museum and Wohnhaus.

Immersing in Goethe’s World

Our exploration began at the Goethe National Museum, where we were introduced to the life and works of Johann Wolfgang von Goethe. Visiting the Goethehaus was a captivating experience. We were grouped and tasked to walk through the exhibit and learn about Goethe. One of the activities was the Goethe-Apparat which was a Virtual interactive experience of Goethe’s Study Room on a curved screen with Virtual Reality controllers.

In the exhibition, we explored Goethe’s collections of artifacts and discoveries. The collections were divided in themes of art, archeology, literature and his memories. Equipped with audio guides we gained a deeper understanding of his life’s work.

Later we explored the house itself. The creaky floors and the labyrinthine layout of the rooms drew us in. Each room offered glimpses into Goethe’s multifaceted interests, from art to literature, with numerous artifacts that spoke volumes about his life. What was a special feature of the rooms were the individual colors that gave Goethe different feelings. His study was painted green since that was his favorite color.

A highlight was the innovative Goethe Live 3D project, which allowed us to virtually traverse the house. Despite some technological challenges, the thrill of experiencing Goethe’s world in such an interactive way was exhilarating. We felt like we were stepping back in time, surrounded by Goethe’s thoughts and belongings.

The Enchantment of Schloss Belvedere

Our visit to Schloss Belvedere added another layer to our journey. Although we didn’t find extensive information on Goethe’s Theory of the Urpflanze as we had hoped, the castle and its gardens were breathtaking. Walking through the expansive grounds under a clear blue sky was a moment of pure joy. The unexpected sounds of live opera music drifting through the air made the experience feel almost cinematic. Surrounded by vines and we felt a deep connection to nature, reminiscent of childhood days spent exploring outdoors.

Surprising Discoveries

On our last day, we delved deeper into Goethe’s personal history. We were particularly intrigued to learn that he kept a lock of hair from Charlotte Buff, a detail that humanized the literary giant for us. It was mindblowing to realize that something so intimate had survived for so long. Additionally, discovering Goethe’s passion for collecting fossils did strike interest in some of the group members. It served as a reminder that behind his poetic works, Goethe was a curious individual fascinated by the natural world around him.

Creative Collaboration

The culmination of our trip was the Design Sprint at the Bauhaus Museum, where we brainstormed ideas for a Mixed Reality Experience centered on Goethe. This collaborative effort allowed us to reflect on our favorite moments from the trip. One idea discussed involved a virtual experience in one of Goethe’s rooms, where participants could explore his desk and drawers, revealing minerals and fossils. It was an opportunity to merge creativity with technology, celebrating the legacy of a man who was both a poet and a scientist.

Conclusion

Our journey to Weimar was not just a visit to a historical city; it was a discovery that enriched our understanding of art, literature, and nature. We learned about Goethe not just as a figure of the past but as a person whose interests and passions continue to resonate today. The beauty of Weimar, combined with our shared experiences and reflections, created lasting memories that will undoubtedly influence our future endeavors. As we departed, it became clear that Weimar had left an indelible mark on our hearts and minds, inviting us to carry its spirit into our own lives.

Semester: P5
Authors of the story: Aaron Höhn, Owor.Philip, Oscar Pohl
Date: 29.10.2024

Discovering Weimar: A Journey Through Culture and Nature was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.