We ran into a few bugs but all in all we’re done with the project. Sometimes it was tough but I’m very glad to have worked with Aaron Höhn and Oscar Pohl on this project. Their support really helped me learn new things, cloth sims, motion capture, more about 3D Modelling and Texturing! And the fact that half of the animations in the project were captured and retargetted by me and the curtain simulation is also in the final build actually shows me that I can make a meaningful impact on a project.

There’s other achievments like making the backdrops and sketching out the front and side references for our characters but these here are easily by biggest achievements this semester.

Semester: P5
Author of the story: Owor.Philip

Date: 05/02/2025

Project Delivered! was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’re reading this for the first time or have been keeping up with the blog posts up until this point, I feel it is incumbent on me to show you where we started and in relation to where we are now. So let’s start at the beginning:

We are Aaron, Oscar and Philip. The Boys, Students in Group 3 of the 5th Semester at Darmstadt University of Applied Sciences studying Augmented and Virtual Reality Design and our topic for this semester was Johann von Goethe.

Research, Ideation and Concept Phase

We traveled to Weimar, spent one night there and worked with Klassikstiftung Weimar at the Goethe National Museum and did some research.

Last year we spent more time concepting and developing than researching and ideating but this time after the trip, it was the other way around. We came up with so many ideas.

But in the end, we focused on what our target group would want which is primarily Museum goers usually older and rather than making a game, we decided to go with an experience, the puppet theater that Goethe grew up playing with! So our concept is a Mixed Reality puppet theater that you can put 2 puppets in that will act in a short play using AI dialogue which we developed for the Quest 3. Here’s our prototype video.

Development Phase

The Development Phase consisted of a few design sprints. The first was setting up the project, the second was creating the visuals and the interactions and the third was creatiung the puppet physics, motion capture animations and getting the Quest 3 to detect the room for our application. The AI dialogue and text to speech development were taking place simultaneously throughout the project. Then there was polishing and user testing. We held daily Discord meetings to support eachother throughout the development process and each bug or problem we ran into, by putting our minds together we were able to solve many problems and even learn a lot from eachother!

We learned a lot more stuff as well and we believe we were more or less on schedule. We are proud of our work and excited to show it during our final presentation.

The Final Week of Polishing and Delivery

So this last week we ran into some problems with the application not creating a build because of my stellar cloth simulation. Apparently the file type known as Alembic isn’t buildable on Unity but Aaron and I were hard at work trying to figure out how to make this build for the project. So we explored multiple solutions for this problem.

In the end, after putting our heads together and combining brain power we built the application to the quest with the curtains and it all worked perfectly. In a nutshell, this involved creating keyframes for the cloth simulation and then exporting the simulation with these keyframes as an FBX (a file type that Unity accepts) and then Aaron Höhn went in Unity and cut the curtain animation I made into “Open” and “Close” and using code made the curtains close when both the puppets are in the theater and the play is generating and made them open when the play is finished generating.

On top of that, we added text-to-speech package. However, last week it didn’t work at all but after adding a TTS package from Meta we were able to hear the puppets speak in the play according to their lines. There were a few bugs that needed squashing which Aaron managed to take care of and after retargetting the last of our mocap animations left over from when Oscar was sick, we basically were done and created a Build of the project with much of the functionality we intended to have. Or that’s what we thought until we found out that the theater is super laggy in the scene. Remember what I said in last week’s post, right at the end of the development phase, how everything seems to go wrong? Oscar and Aaron by combining brain power again managed to squash the bug, it was as simple as pressing a button actually. There were some Meta project settings that needed to be applied and because they weren’t the performance was horrendous.

We also are nearly done with our presentation and we even made a poster for our project.

So after this, we will be presenting our project but we still won’t be done yet, we’ll need to capture footage for our trailer which Philip Owor will then edit and then we’ll be done. There are unfortunately some stuff we were not able to add to the project, for example the animated eyebrows, being able to throw puppets and so on. But all in all, we still did it. Which is why I w can confidently say: We made a Goethe Puppet Theater in Mixed Reality and it was outrageously fun! The Group spirit was always very chill and we were always there to support eachother. We learned a lot here, creating our own assets, models and textures and even animations! And we are proud to show our project to the world.

Semester: P5
Group 3 THE BOYS: Aaron Höhn, Philip Owor, Oscar Pohl
Author of the story: Owor.Philip

Date: 05/02/2025

We Made a Mixed Reality Goethe Puppet Theater was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Have you ever been in a project and right at the most critical part of the development phase, everything goes wrong?

Your colleague who is critical to the dev process falls ill?

Through a series of unfortunate and regrettable mistakes, Windows needs to reinstall and wipe the data from your storage?

Well… If only the only problem was that my PC needed to reinstall windows. No, when we merged our changes to the main branch, the main branch overwrote some of our progress for the last few days. Thankfully, we recovered our progress but the funny thing is there always seems to be something that goes wrong right at the most critical parts of development. But I’ve noticed that these setbacks are great opportunities where we can learn to overcome our doubts and come up with solutions.

Personally, I was worried that all my data for my Trailer Making Elective was completely and utterly wiped out but in fact it remained intact. Finding this out after 24 hours spent fixing my Windows PC was truly was the Silver Lining. In the midst of “it’s so over”, I found within me an invincible “we’re so back”

Semester: P5
Author of the story: Owor.Philip

Date: 29/01/2025

Setbacks was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

So, pretty much 7 days left to go and we made a lot of progress. Good news first. This past week we cleaned our mocap animations, retargetted them in Blender, imported them into AND edited them in Unity so we could create that 2 different animations under the same emotion but for talking and listening. This entire process went like so:

Oscar: Cleaned the all mocap animations

Philip: Retargetted all animations in Blender

Aaron: Imported into Unity and edited the animations talking and listening

What would have taken 12 hours for one person to do, having split the tasks of this entire workflow among us, we managed to do it in less than 4 hours together.

Then Aaron Höhn went on to connected the AI to the puppets so that they would act out the animations based on their lines. He worked also on the UI so that the puppet’s lines would be displayed while they speak. He also made the Mephisto 3D Model and it looks Devilishly Handsome!

Philip Owor created the backdrops for the puppet theater from the references we already had. They needed to be transparent since they will be in the theater so he had to remove the backgrounds of the theater, separate them into different parts meaning Foreground, Midground and Background. However, one backdrop was missing a sky so Philip generated one and edited the color balance to make it thematically homogenous to the backdrop that would be in front of it.

We found that the puppet theater curtains animations Philip created were clipping through the theater walls so a new animation had to be created and reimported in Blender. Phillip is still in the process of creating the drapes at the top of the puppet theater.

On top of that Oscar Pohl took over 3D modeling the puppet theater for me while I continued to work on the curtains. We did run into an issue where we didn’t know what the puppet theater looked like from the side even though we had our dimensions. However, in a surprising twist of events Oscar managed to dig up a 360° photo of what the puppet theater model looked from the side. What a stroke of luck that he decided to take as many photos as possible while we were in Weimar.

And so combining this image reference with the front view we already had, Oscar go to finishing the 3D model and even textured it.

So all in all, we have the puppets actually talking to each other inside the puppet theater with the right animations for when they are either talking or listening and Aaron created a standalone build on the Quest 3 with the apk running smoothly all on it!

The curtains for the puppet theater still need to be done for that authentic Goethe Puppet Theater feel. In the mean time work on the final presentation is now underway and of course we do run into the occasional bug.

As for the bad news. Oscar fell sick and so is unable to work on the project for the time being and Philip’s PC needed to reinstall Windows losing the apps and software he was using to add to the project but luckily keeping the all my actual project files intact, so minimal damage had occurred.

Semester: P5
Group 3 THE BOYS: Aaron Höhn, Philip Owor, Oscar Pohl
Author of the story: Owor.Philip

Date: 29/01/2025

The Final Week was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Aaron got to work implementing the ragdoll effect. There were some… problems with the puppets having noodle limbs when they became ragdolls but Aaron had to set up the max joint rotation so they wouldn’t go bonkers.

I worked on the Puppet Theater using the proper dimensions and reference photos to create a better base mesh and creating the curtain animations as well as finishing up Charlotte’s skirt textures.

Honestly while Aaron was teaching me everything about substance painter, we did go over a lot of information so when we all had somewhat of a break for a few days and came back, most of what Aaron taught me was a little hazy and finishing the skirt textures was quite difficult but I managed to push through and with his help, I was able to make the skirt textures in a way that works well. As far as what we have planned, we will finally capture our animations right after our P5 lecture on Thursday.

Currently our next steps are to capture the animations for the puppets, all the different emotes, walks and idles. Finish the puppet theater as well as creating two more puppets, connecting the AI to the puppets then integrating the text to speech, transition between animation and ragdoll and animate the eyebrows.

We are a bit off schedule but at the current speed at which things are going, we believe we will hit our goals in good time.

Semester: P5
Group 3 THE BOYS: Aaron Höhn, Philip Owor, Oscar Pohl
Author of the story: Owor.Philip

Date: 15/01/2025

Time to get Serious was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Semester: P5
Author of the story: Owor.Philip

Date: 15/01/2025

Thoughts on the next few weeks… was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Directly after our P5 coaching on the first week back at uni, we motion captured our animations that we will have in our project. 78 animations! I’ll spit it out, I’m not so good at improv. In fact, I was a bit nervous at first to step into the mocap suit for this but in the end I’m very glad I did. Motion capturing all the different emotions allowed me to loosen up a little and capturing some specific emotions was kind of cathartic for me. We even had some pizza afterwards!

I then finished up the curtain animation for the puppet theater, imported it into Unity and helped Aaron retarget the animations we captured to the puppets. There was some pretty good synergy, after he showed me how to retarget using the Rokoko Blender add-on, I started retargeting the animations one by one and then sending them to him to import into the Unity project.

Also the Charlotte Puppet Textures are finally done. I couldn’t have done it in this short amount of time without Aaron Höhn and Oscar Pohl’s help.

At first I thought I wasn’t doing much in the project but as I write this now, I see that I have accomplished and am accomplishing a lot along side Oscar and Aaron and the speed at which we’re working, I think we can make the deadline.

Semester: P5
Author of the story: Owor.Philip

Date: 23/01/2025

Even as I was learning from Aaron, I was also able to help him fix the facial features of Goethe’s puppet which looked spectacular lmao.

We imagined this could be used as a meme:

“How it started vs How it’s going”

While researching the puppet theater, we were unable to get the actual measurements of the theater. All accounts of the measurements online were not agreed upon until we reached out to KSW which gladly provided us with the actual measurements. So now we can make the theater size as accurate as possible. We also got some different backdrops for the theater so we can make our scenes dynamic.

Overall I’m really grateful for the daily discord meetings with the group, being able to sit down with them and let them guide me through the entire process truly helped diffuse my doubts and I made something I am genuinely proud of.

Semester: P5
Author of the story: Owor.Philip

Date: 08/01/2025

BEHOLD! Charlotte Buff as a Tin Puppet! was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

However, I felt it was incumbent on me to create the 3D model for Charlotte, so after I created the sketches I took up responsibility for modelling her and thankfully I have Aaron Höhn and Oscar Pohl to Guide me in the 3D modelling process because I was getting a bit frustrated at times with it but the more I did it, the more I realized that, being a skill, 3D modeling can improve and I was able to step back from my doubts and see myself improve bit by bit.

And while I may not model as fast as Aaron can (yet) I have made enough progress that I can be proud of. Like look at the topology on her Corset! Normally my topology sucks but somehow all the stars aligned and I accidently made it not suck!

Semester: P5
Author of the story: Owor.Philip

Date: 01/01/2025

From Digital Sketching to 3D Model was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

With a little hesitation but a committment to personal growth, I took up the responsibility to create the reference photos for the puppets (the front and side Orthographic views) to 3D model off of. Cool! I just need to get to the required competence level for this. Here’s the thing though, the last big drawing I did with my tablet was of a mountain range over 2 years ago and since then it’s been gathering dust sort of. But now it’s time to take it out and reacquaint myself with digital art. Luckily after 2 years at uni, I’ve made enough friends to turn to for help. So with a little bit of guidance and focusing solely on the immediate task, I know I can make it work.

Semester: P5
Author of the story: Owor.Philip

Date: 25/12/2024

Creating the Puppet Reference Photos was originally published in P3 & P5 blog on Medium, where people are continuing the conversation by highlighting and responding to this story.