<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kinetix on Medium]]></title>
        <description><![CDATA[Stories by Kinetix on Medium]]></description>
        <link>https://medium.com/@kinetixtech?source=rss-9c8fa0f9d89b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*b_TjWSrYvjC1tIMA7peF7A.png</url>
            <title>Stories by Kinetix on Medium</title>
            <link>https://medium.com/@kinetixtech?source=rss-9c8fa0f9d89b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:58:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kinetixtech/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An overview of text-conditioned human motion synthesis]]></title>
            <link>https://medium.com/@kinetixtech/introduction-to-text-condition-human-motion-using-ai-3dc8ffb7ec1b?source=rss-9c8fa0f9d89b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3dc8ffb7ec1b</guid>
            <dc:creator><![CDATA[Kinetix]]></dc:creator>
            <pubDate>Fri, 28 Apr 2023 09:38:33 GMT</pubDate>
            <atom:updated>2023-04-28T11:20:29.213Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Did you know that one could generate animations with text prompts? This article is an introduction to the topic of human motion synthesis from text, providing an overview of the different families of text-to-motion methods.</em></p><p><em>Authored by </em><a href="https://www.linkedin.com/in/thomas-daniel-0b7923127/"><em>Thomas Daniel</em></a><em> (Deep Learning R&amp;D Engineer, Kinetix)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w8_6-zeyeTW74bEdhuQH1w.png" /><figcaption>Text2Emote Feature Coming Soon</figcaption></figure><h3>Introduction to human motion generation from text</h3><p>The last few years have seen significant advances in generative artificial intelligence, especially with text-to-image generative models such as DALL-E 2 [1] and Stable Diffusion [2]. Text-to-image models are now capable of creating high-quality images from a textual description or “prompt”. They leverage the power of diffusion models for generative tasks and transformers for processing texts. Diffusion models have become the first choice for image synthesis from text since the seminal work of A. Nichol <em>et al.</em> on GLIDE [3], thanks to their ability to produce diverse and photorealistic images while avoiding the difficulties encountered with GANs (Generative Adversarial Networks) during training.</p><p>The recent breakthroughs and the success of text-to-image technology with the general public stimulated research in other domains, like human motion synthesis. In this article, we will present an overview of text-conditioned human motion synthesis, that is, the task of animating a humanoid character based on a prompt provided by the user. Researchers in AI for 3D animation were already designing text-to-motion methods before the rise of diffusion models for image synthesis, motivated by potential applications in the gaming, film, and robotics industries. Nonetheless, contributions in this field have increased the pace lately and sometimes borrow ideas from text-to-image methods. Although sharing similar purposes, text-to-motion and text-to-image differ as they do not have the same amount of annotated data. Indeed, datasets containing animations with textual descriptions are much smaller than text-to-image datasets, which partially explains why text-to-motion solutions cannot yet achieve the same level of quality and robustness as text-to-image methods. For instance, Stable Diffusion was trained on billions of image-caption pairs from the LAION-5B dataset [4], while one of today’s most extensive text-to-motion datasets called HumanML3D [5] contains 14,616 motion clips and 44,970 captions. In addition, human motion synthesis requires manipulating spatiotemporal data and satisfying visual and physical constraints related to motion plausibility and naturalness.</p><p>We will first introduce a few definitions of text-conditioned human motion synthesis, interesting tasks, datasets, and evaluation metrics. We will proceed to present different families of text-to-motion methods, ranging from early ones based on a search in a database of motion clips to the most recent ones using deep learning. Finally, we will illustrate a few limitations of art in this field and provide perspectives on the future of human motion synthesis</p><h3>Text-to-motion basics</h3><h4>Text-to-motion and related tasks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*XylZ-qjUxEdBcQloU_ufSA.png" /><figcaption>Text-to-motion examples, from TEMOS [20]</figcaption></figure><p>Conditional human motion synthesis consists in building a model that can produce an animation of a humanoid character using contextual information or a control signal, referred to as the <em>condition</em>. The conditioning input can be, for example, an action label, a text, music, or an initial motion. This article focuses on text conditioning, where the model must infer a plausible motion reflecting the semantic content of a prompt written by a user.</p><p>Action conditioning (action-to-motion) and text conditioning are different. Action-to-motion methods like Action2Motion [6] or ACTOR [7] produce short animations corresponding to the input action label (generally a simple verb), whereas text-to-motion methods try to consider free-form texts with more detailed descriptions involving actions, styles, and temporal constraints. Usually, text-to-motion methods also address the action-to-motion task.</p><p>A text-to-motion method is qualified as deterministic when it always produces the same animation for a given prompt. Stochastic methods are generally preferred as they can create various animations from a given prompt.</p><p>Some text-to-motion methods can be applied to other text-driven tasks, namely motion editing, and in-betweening. In text-based motion editing, an initial motion clip is edited by modifying the movements of some parts of the body via a prompt. Text-conditioned motion in-betweening is another task consisting in generating a transition between two motion clips, taking into account the textual description. For example, we can ask the character to raise his hands while walking between its starting and final locations defined by the previous and future motion clips.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JUG1BDf1zheURSYYZEmDCw.png" /><figcaption>Editing applications, from MDM [25]</figcaption></figure><h4>Datasets</h4><p>KIT Motion-Language dataset [8], created in 2016, contains 3,911 animations (11 hours) from the larger AMASS dataset with 6,353 textual descriptions indicating the action type, direction, and sometimes the speed.</p><p>BABEL dataset [9] (2021) contains 43 hours of motion capture from AMASS dataset, with different levels of textual descriptions (28k sequence labels and 63k frame labels) and 250 action categories. In 2022, [5] released the HumanML3D dataset, made of more than 14k motion sequences (29 hours) from AMASS and HumanAct12, with almost 45k textual descriptions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eZNLYjCGQwEWlPfzx8f1jQ.png" /><figcaption>BABEL dataset [9]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-bp9nztTmHr2CH0Dg2JzhA.png" /><figcaption>HumanML3D dataset [5]</figcaption></figure><h4>Evaluation metrics</h4><p>Evaluating a text-to-motion model is difficult, like any other type of generative model, since motion realism and correspondence with the prompt depend on many visual details that are subjective and hard to formulate explicitly. Different evaluation metrics have been proposed to quantify the performances of text-to-motion models according to various criteria. Motion quality can be assessed by calculating the Fréchet distance between the distributions of the real and generated motion features, like the <em>FID (Fréchet Inception Distance)</em> metric commonly used for image synthesis but replacing the Inception model with a motion feature extractor. The relevance of generated motions to the prompts can be quantified by different condition-matching metrics, such as the <em>R-Precision</em> and <em>Multimodal Distance </em>as defined in [5]<em>. </em>Both metrics require text and motion encoders to extract similar features for matched text-motion pairs. Multimodal Distance is the average distance between the features of a text and those of the resulting motion. R-Precision, in our context, is obtained by considering 32 prompts for each motion (including 31 mismatched prompts and the true motion description) and ranking them from the closest to the farthest in the feature space, to check whether the true prompt is retrieved in the top 1/2/3 (top-k accuracy). For action-to-motion, condition matching is evaluated via the accuracy of an action recognition model. Other common metrics measure the variability of the generated animations in general (<em>Diversity</em>) or for a given prompt (<em>MultiModality</em>).</p><h3>Review of text-to-motion methods</h3><h4>Early methods</h4><p>In 2009, a method generated animations from natural language texts using a database of motion clips for different characters. The work, presented in two papers [10, 11], was motivated by movie production needs for simple animation systems that would provide mockups and previsualizations. The input text is analyzed to search for relevant motion clips and schedule them according to temporal constraints. Contrary to modern methods, this technique does not create new animations but combines pre-existing motion clips.</p><p>With the development of deep learning and more specifically generative AI, new methods were proposed in 2017/2018. The first deep learning-based methods addressed the text-to-motion task as a machine translation problem, using a sequence-to-sequence RNN approach [12, 13, 14]. In particular, Text2Action [14] uses a GAN based on the sequence-to-sequence model. Another example of a text-to-motion method using GANs can be found in[15].</p><h4>Cross-modal latent spaces</h4><p>Synthesizing motions from texts can also be achieved by constructing a shared latent space for both texts and motions. The idea is to train a motion autoencoder by constraining the encoder part to give embeddings (or representations) that are similar to the text embeddings returned by a text encoder. In this way, a given motion clip and its textual description have similar representations in the joint cross-modal latent space. At inference time, the input prompt is encoded into a latent representation by the text encoder, and the motion decoder returns the corresponding motion from this latent representation.</p><p>Joint Language-to-Pose (JL2P) [16] implicitly pushes text and motion embeddings closer together by combining reconstruction losses from both embeddings, using the smooth L1 loss (Huber loss). In Ghosh et. al [17], additional loss terms are considered, namely the Huber loss between text and motion embeddings to penalize their distance explicitly, and an adversarial loss to encourage motion plausibility. MotionCLIP [18] chooses the L2 reconstruction loss to train a transformer autoencoder and aligns motion embeddings with CLIP [19] text and image embeddings using the cosine distance. More recently, TEMOS [20] used a transformer variational autoencoder (VAE) to build a probabilistic method for producing diverse animations from a given prompt. The original JL2P reconstruction losses are combined with the classical VAE regularization term, Kullback-Leibler (KL) divergences between the distributions of the text embedding and the motion embedding, and a smooth L1 loss between the sampled text and motion embeddings. A more sophisticated version of TEMOS, TEACH [21], was released later to generate longer animations containing multiple actions. [22] is another method based on a cross-modal latent space, which covers the motion-to-text task.</p><p>Let us return for a moment to MotionCLIP [18]. The authors underline that available datasets like KIT and BABEL were not large enough to represent the wide variety of possible human motions. Hence, they had the idea of leveraging the knowledge-rich latent space of CLIP (Contrastive Language-Image Pretraining) [19], a famous visual-textual embedding model trained in a contrastive fashion to align embeddings of images and texts that are semantically similar. By aligning its motion latent space with CLIP’s text-image joint latent space, MotionCLIP achieved impressive results with prompts containing cultural references or describing actions not represented in the training set, such as specific movements from Olympic sports. The continuity and disentanglement of MotionCLIP’s latent space inherited from CLIP enable motion editing operations such as motion composition and interpolation, and its semantic structure was relevant for action recognition.</p><h4>Diffusion models</h4><p>Late 2022 saw the growing influence of diffusion models in the human motion synthesis domain. MotionDiffuse [23], FLAME (Free-form LAnguage-based Motion synthesis &amp; Editing) [24], and MDM (human Motion Diffusion Model) [25] are three different methods applying diffusion on raw motion sequences, replacing the usual U-Net architecture by a transformer that is better suited to handle temporal data of variable length. These methods took advantage of the HumanML3D dataset, recently created by another method called T2M [5] and based on a temporal VAE.</p><p>Inspired by the GLIDE text-to-image model, FLAME’s diffusion model predicts the noise in the diffused motion and learns the reverse-process variance to reduce the number of diffusion steps required to get high-quality results. MotionDiffuse predicts the added noise too but sets the reverse-process variance as a constant for the sake of simplicity. MDM follows the choice made in DALL-E 2 to predict the unnoised motion directly and regularizes the network with kinematic losses including a contact loss to mitigate foot sliding. Finally, MLD (Motion Latent Diffusion) [26] uses a latent diffusion model as introduced in [2] for Stable Diffusion, which consists in applying diffusion in a latent space to significantly reduce the computation time during training and inference. MLD trains a transformer-based motion VAE to build the latent space, and a transformer-based diffusion model with long skip connections to predict the noise. Following the practices of famous text-to-image diffusion models, FLAME, MDM and MLD use classifier-free guidance to amplify the effect of the prompt and boost motion quality, at the cost of a little decrease in diversity. In diffusion-based approaches, conditioning information is obtained by encoding the prompt into token embeddings and injected into the diffusion model via a cross-attention mechanism or concatenation with the diffused motion data. Experiments conducted by MDM and MLD suggest that concatenation leads to better results than cross-attention.</p><h4>Approaches based on vector quantization</h4><p>Early 2023, T2M-GPT (Text2Motion Generative Pretrained Transformer) [27] competed with state-of-the-art diffusion methods with a vector-quantized variational autoencoder (VQ-VAE) coupled with a GPT-like model. The VQ-VAE provides a discrete representation for the motion, which is mapped to a sequence of indices referring to elements of a learned codebook. T2M-GPT predicts a sequence of codebook indices autoregressively from CLIP text embeddings, which is then transformed into an animation by the decoder of the VQ-VAE. The idea of applying vector quantization to create discrete motion tokens and formulating text-to-motion as an autoregressive next-token prediction task was also explored in TM2T [28], a method considering both the text-to-motion task and the motion-to-text task.</p><p>Contrary to most methods that set the motion length in advance, T2M-GPT implicitly learns the duration of a motion sequence via an “end” token. This interesting feature is also proposed by T2M [5] where a specific module predicts the motion length given the prompt.</p><h3>Perspectives for the future of text-to-motion methods</h3><h4>Dataset size and motion representation</h4><p>Despite the recent releases of BABEL (2021) and HumanML3D (2022) datasets, the dataset size remains a major limiting factor for high-quality text-to-motion solutions. Larger datasets would enable increasing the capacity of the AI model used for human motion synthesis, expanding the scope of motions generated, and increasing the level of details of a prompt in terms of movements, style, and chronology.</p><p>The choice of motion representation is a crucial topic. Some methods rely on a redundant representation of the motion, proposed in [5] and used in the original version of the HumanML3D dataset. This data format is obtained by concatenating information about the skeleton’s root joint (angular velocity around the vertical axis, linear velocity in the horizontal plane, and height), the local joints positions, velocities and rotations in root space, and binary labels for contacts between the ground and foot joints (heel and toe for each foot). As argued in the MLD paper, this redundant motion representation is <em>“quite suited to neural models, particularly VAEs.”</em> Methods using this data format usually post-process the output to get the global 3D positions of the 22 joints (SMPL skeleton without hands), resulting in a simplified animation based on a stick figure. To get a more realistic animation with the SMPL body model, a time-consuming optimization procedure is run to fit the SMPL pose parameters to the given 3D joints’ positions. On the contrary, working directly on the joints’ rotations and the root’s translation enables getting clean pose parameters that can produce realistic animations with the SMPL body model. A choice made by some methods like FLAME, TEMOS, and TEACH, among others. The 3D rotations are written using the 6D continuous representation introduced in [29], which is more suitable for learning purposes. There are other possible data formats (root translation and root-relative 3D joint positions), and future works will provide a better insight into their respective advantages and drawbacks for human motion synthesis.</p><h4>Toward more sophisticated models</h4><p>Beyond the questions of available data and their format, we can wonder what are the most promising approaches for building new text-to-motion methods. Diffusion models have shown promising results, but the recent T2M-GPT method [27] gave <em>“comparable or even better performances than concurrent diffusion-based approaches”</em> suggesting that <em>“the performance gain of the diffusion-based approaches might not be that significant.”</em> Other methods relying on a cross-modal latent space have great potential for future improvements. The coexistence of competing ways to address the text-to-motion task is very exciting for the future.</p><p>Furthermore, there is room for improvement in several areas, such as enhancing the capacity to generate long animations, mixing different conditioning modalities in a unified generative model, and increasing motion realism and compliance with the prompt. The current text-to-motion methods usually fail to produce high-quality sequences lasting more than 10 seconds. Longer animations generated by diffusion-based methods are noisy. Concerning data, the motion clips in HumanML3D dataset last from 2 to 10 seconds, with an average duration of 7.1 seconds. In addition, most methods can hardly synthesize animations with multiple actions whose order comply with the chronology described in the prompt. TEACH [21] and MultiAct [30] are two methods that address the problem of temporal action composition. They show how to generate long-term human motions given a sequence of descriptions or action labels and desired durations. In a recent paper introducing a method called DoubleTake [31], 10-minute-long animations are generated in a zero-shot manner using the pre-trained MDM text-to-motion model that was initially designed for 10-second animations. The authors also manage to produce two-person animations in a few-shot setting by leveraging MDM as a generative prior.</p><p>Concerning the combination of different conditioning modalities, two methods called MoFusion [32, 33] already propose models generating animations from text or music. They use the AIST++ dataset for music-conditioned choreography synthesis. One of them [32] also gives an innovative trick to introduce additional loss terms during the training of a diffusion model to increase motion realism, providing an alternative to how MDM applies kinematic losses. According to them, their methodology could benefit future diffusion-based methods by adding physics-based soft constraints. Another method called PhysDiff [34] enforces physical constraints during the sampling process of a pre-trained diffusion model, thanks to a physics-based motion projection module trained by reinforcement learning for motion imitation in a physics simulator.</p><p>Finally, a recent paper [35] enhanced the performances of motion diffusion models by integrating a retrieval mechanism in the denoising process. This method, called ReMoDiffuse (Retrieval-augmented Motion Diffusion model), guides and refines the generative process by incorporating information from retrieved motion sequences of an auxiliary database.</p><h3>Conclusion</h3><p>Generative AI has opened the door to new possibilities in the creative industries. Generative AI tools in 3D computer animation could help animators give life to virtual characters. In the gaming industry, users in virtual worlds could benefit from generative models to control their avatars. However, text-to-motion methods still have a long way to go before being able to create a short animated movie from a film script. Many exciting challenges remain for researchers in this field, and collecting more textually-annotated animations is essential for the future of text-conditioned human motion synthesis.</p><p>At Kinetix, we are convinced that generative AI can revolutionize 3D animation. Our R&amp;D team has recently started work on conditional human motion synthesis to develop our skills and contribute to research efforts in this field. Our knowledge and expertise in AI motion capture and related topics will help us address some challenges, such as improving motion realism and compliance with the conditioning input. We are currently developing our text-to-motion technology, with the ambition of other conditioning modalities in the future. For this purpose, we are building new datasets for generative tasks, thanks to our 3D animators and external collaborations.</p><h3>What we are Building</h3><h4>Upcoming Kinetix Generative AI Feature: Text2Emote</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F8JFsHOxV2rk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D8JFsHOxV2rk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F8JFsHOxV2rk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0442c2d71d9cf4550857f6146110e579/href">https://medium.com/media/0442c2d71d9cf4550857f6146110e579/href</a></iframe><p><strong>With Different Prompts:</strong></p><ol><li>DJ Prompt</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fb2-lw9GwO0Y%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Db2-lw9GwO0Y&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fb2-lw9GwO0Y%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/eda1385cdfcb61ac80fe2ec1863548d5/href">https://medium.com/media/eda1385cdfcb61ac80fe2ec1863548d5/href</a></iframe><p>2. Angry Prompt</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FtmDHiO2_9-o%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DtmDHiO2_9-o&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FtmDHiO2_9-o%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/859b5b85c5dbdab63ace0a3cec1ef48f/href">https://medium.com/media/859b5b85c5dbdab63ace0a3cec1ef48f/href</a></iframe><p>3. Hello Prompt</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F6yNPI71N680%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6yNPI71N680&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F6yNPI71N680%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4775c548a12eb58824ae46f93f6004ee/href">https://medium.com/media/4775c548a12eb58824ae46f93f6004ee/href</a></iframe><h4>Acknowledgments</h4><p><em>Many thanks to Aditya Prabhugaunkar and Thomas Rossi for reviewing this article.</em></p><h4>References</h4><p>[1] <em>[DALL-E 2]</em> Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., &amp; Chen, M. (2022). Hierarchical Text-Conditional Image Generation with CLIP Latents. <em>ArXiv, abs/2204.06125</em></p><p>[2] <em>[Stable Diffusion]</em> Rombach, R., Blattmann, A., Lorenz, D., Esser, P., &amp; Ommer, B. (2021). High-Resolution Image Synthesis with Latent Diffusion Models. <em>2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</em>, 10674–10685.</p><p>[3] <em>[GLIDE]</em> Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., &amp; Chen, M. (2021). GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. <em>International Conference on Machine Learning</em>.</p><p>[4] <em>[Training set for Stable Diffusion]</em> Baio, A. (2022). Exploring 12 Million of the 2.3 Billion Images Used to Train Stable Diffusion’s Image Generator. <a href="http://waxy.org/"><em>Waxy.org</em></a><em>.</em></p><p>[5] <em>[T2M and HumanML3D]</em> Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., &amp; Cheng, L. (2022). Generating Diverse and Natural 3D Human Motions from Text. <em>2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</em>, 5142–5151.</p><p>[6] <em>[Action2Motion]</em> Guo, C., Zuo, X., Wang, S., Zou, S., Sun, Q., Deng, A., Gong, M., &amp; Cheng, L. (2020). Action2Motion: Conditioned Generation of 3D Human Motions. <em>Proceedings of the 28th ACM International Conference on Multimedia.</em></p><p>[7] <em>[ACTOR]</em> Petrovich, M., Black, M.J., &amp; Varol, G. (2021). Action-Conditioned 3D Human Motion Synthesis with Transformer VAE. <em>2021 IEEE/CVF International Conference on Computer Vision (ICCV)</em>, 10965–10975.</p><p>[8] <em>[KIT]</em> Plappert, M., Mandery, C., &amp; Asfour, T. (2016). The KIT Motion-Language Dataset. <em>Big data, 4 (4)</em>, 236–252.</p><p>[9] <em>[BABEL]</em> Punnakkal, A.R., Chandrasekaran, A., Athanasiou, N., Quiros-Ramirez, A., Systems, M.J., &amp; Konstanz, U. (2021). BABEL: Bodies, Action and Behavior with English Labels. <em>2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</em>, 722–731.</p><p>[10] Oshita, M. (2010). Generating animation from natural language texts and semantic analysis for motion search and scheduling. <em>The Visual Computer, 26</em>, 339–352.</p><p>[11] Oshita, M. (2009). Generating Animation from Natural Language Texts and Framework of Motion Database. <em>2009 International Conference on CyberWorlds</em>, 146–153.</p><p>[12] Plappert, M., Mandery, C., &amp; Asfour, T. (2017). Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks. <em>Robotics Auton. Syst., 109</em>, 13–26.</p><p>[13] Lin, A.S., Wu, L., Corona, R., Tai, K.W., Huang, Q., &amp; Mooney, R.J. (2018). Generating Animated Videos of Human Activities from Natural Language Descriptions.</p><p>[14] <em>[Text2Action]</em> Ahn, H., Ha, T., Choi, Y., Yoo, H., &amp; Oh, S. (2017). Text2Action: Generative Adversarial Synthesis from Language to Action. <em>2018 IEEE International Conference on Robotics and Automation (ICRA)</em>, 1–5.</p><p>[15] Lin, X., &amp; Amer, M.R. (2018). Human Motion Modeling using DVGANs. <em>ArXiv, abs/1804.10652</em>.</p><p>[16] <em>[JL2P]</em> Ahuja, C., &amp; Morency, L. (2019). Language2Pose: Natural Language Grounded Pose Forecasting. <em>2019 International Conference on 3D Vision (3DV)</em>, 719–728.</p><p>[17] Ghosh, A., Cheema, N., Oguz, C., Theobalt, C., &amp; Slusallek, P. (2021). Synthesis of Compositional Animations from Textual Descriptions. <em>2021 IEEE/CVF International Conference on Computer Vision (ICCV)</em>, 1376–1386.</p><p>[18] <em>[MotionCLIP]</em> Tevet, G., Gordon, B., Hertz, A., Bermano, A.H., &amp; Cohen-Or, D. (2022). MotionCLIP: Exposing Human Motion Generation to CLIP Space. <em>European Conference on Computer Vision</em>.</p><p>[19] <em>[CLIP]</em> Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., &amp; Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. <em>International Conference on Machine Learning</em>.</p><p>[20] <em>[TEMOS]</em> Petrovich, M., Black, M.J., &amp; Varol, G. (2022). TEMOS: Generating diverse human motions from textual descriptions. <em>ArXiv, abs/2204.14109</em>.</p><p>[21] <em>[TEACH]</em> Athanasiou, N., Petrovich, M., Black, M.J., &amp; Varol, G. (2022). TEACH: Temporal Action Composition for 3D Humans. <em>ArXiv, abs/2209.04066</em>.</p><p>[22] Yamada, T., Matsunaga, H., &amp; Ogata, T. (2018). Paired Recurrent Autoencoders for Bidirectional Translation Between Robot Actions and Linguistic Descriptions. <em>IEEE Robotics and Automation Letters, 3</em>, 3441–3448.</p><p>[23] <em>[MotionDiffuse]</em> Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., &amp; Liu, Z. (2022). MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model. <em>ArXiv, abs/2208.15001</em>.</p><p>[24] <em>[FLAME]</em> Kim, J., Kim, J., &amp; Choi, S. (2022). FLAME: Free-form Language-based Motion Synthesis &amp; Editing. <em>ArXiv, abs/2209.00349</em>.</p><p>[25] <em>[MDM]</em> Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., &amp; Bermano, A.H. (2022). Human Motion Diffusion Model. <em>ArXiv, abs/2209.14916.</em></p><p>[26] <em>[MLD]</em> Chen, X., Jiang, B., Liu, W., Huang, Z., Fu, B., Chen, T., Yu, J., &amp; Yu, G. (2022). Executing your Commands via Motion Diffusion in Latent Space. <em>ArXiv, abs/2212.04048</em>.</p><p>[27] <em>[T2M-GPT]</em> Zhang, J., Zhang, Y., Cun, X., Huang, S., Zhang, Y., Zhao, H., Lu, H., &amp; Shen, X. (2023). T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations. <em>ArXiv, abs/2301.06052</em>.</p><p>[28] <em>[TM2T]</em> Guo, C., Xuo, X., Wang, S., &amp; Cheng, L. (2022). TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts. <em>European Conference on Computer Vision</em>.</p><p>[29] <em>[6D continuous representation for rotations]</em> Zhou, Y., Barnes, C., Lu, J., Yang, J., &amp; Li, H. (2018). On the Continuity of Rotation Representations in Neural Networks. <em>2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</em>, 5738–5746.</p><p>[30] <em>[MultiAct]</em> Lee, T.J., Moon, G., &amp; Lee, K.M. (2022). MultiAct: Long-Term 3D Human Motion Generation from Multiple Action Labels. <em>ArXiv, abs/2212.05897</em>.</p><p>[31] <em>[PriorMDM / DoubleTake]</em> Shafir, Y., Tevet, G., Kapon, R., &amp; Bermano, A.H. (2023). Human Motion Diffusion as a Generative Prior. <em>ArXiv, abs/2303.01418</em>.</p><p>[32] <em>[MoFusion 1]</em> Dabral, R., Mughal, M.H., Golyanik, V., &amp; Theobalt, C. (2022). MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis. <em>ArXiv, abs/2212.04495</em>.</p><p>[33] <em>[MoFusion 2]</em> Ma, J., Bai, S., &amp; Zhou, C. (2022). Pretrained Diffusion Models for Unified Human Motion Synthesis. <em>ArXiv, abs/2212.02837</em>.</p><p>[34] <em>[PhysDiff]</em> Yuan, Y., Song, J., Iqbal, U., Vahdat, A., &amp; Kautz, J. (2022). PhysDiff: Physics-Guided Human Motion Diffusion Model. <em>ArXiv, abs/2212.02500</em>.</p><p>[35] <em>[ReMoDiffuse] </em>Zhang, M., Guo, X., Pan, L., Cai, Z., Hong, F., Li, H., Yang, L., Liu, Z. (2023) ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model. <em>ArXiv, abs/2304.01116.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3dc8ffb7ec1b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Tools for Virtual Identity in the Metaverse]]></title>
            <link>https://medium.com/@kinetixtech/ai-tools-for-virtual-identity-in-the-metaverse-ea1316ecca7?source=rss-9c8fa0f9d89b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea1316ecca7</guid>
            <category><![CDATA[gaming]]></category>
            <category><![CDATA[metaverse]]></category>
            <category><![CDATA[artficial-intelligence]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Kinetix]]></dc:creator>
            <pubDate>Wed, 15 Mar 2023 11:33:59 GMT</pubDate>
            <atom:updated>2023-03-30T14:06:10.150Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9O6X80nbv9KJ7WXDDXVDMw.png" /></figure><h3><strong>Introduction</strong></h3><p>Since the dawn of social media, individuals are concerned about their online identity and how to design their social media to visualize them in the real world. Individuals carefully design their online identities across different applications.</p><p><strong><em>What is Virtual Identity?</em></strong> Virtual Identity is the visual presentation of an individual in the virtual space.</p><p>With Games, Virtual Worlds &amp; Metaverse, virtual identity is a very close topic for users. Every platform is helping users forge their virtual identity.</p><p>In this mapping, we have examined <strong>12 AI tools</strong> helping users develop their own Virtual Identities. We analyzed these AI tools based on Creation, Integration &amp; Monetization:</p><ul><li>Ready Player Me</li><li>Didimo</li><li>Anything World</li><li>Kinetix</li><li>Inworld</li><li>Voicemod</li><li>Reallusion</li><li>VRSTL</li><li>Emoshape</li><li>Plask</li><li>Reblium</li><li>Radical</li><li>VROID</li><li>Avaturn (In3D)</li><li>Union Avatars</li></ul><blockquote><strong>Do you want to be part of this mapping? </strong>Fill out this form &amp; we will add you to the mapping: <a href="https://kinetixtech.typeform.com/virtualidentity"><strong>https://kinetixtech.typeform.com/virtualidentity</strong></a></blockquote><h3><strong>In-depth explanation:</strong></h3><h4><strong>Ready Player Me</strong></h4><p>Ready Player Me is <strong>a cross-game avatar platform for the Metaverse</strong>. It lets you create a 3D avatar with a selfie and use it in 3,000+ compatible apps and games. In total, Ready Player Me has over 500,000 registered made on their <a href="https://readyplayer.me/apps">platform</a>.</p><p>The company is based in Estonia and recently closed a <a href="https://techcrunch.com/2022/08/23/ready-player-me-a-platform-to-build-dynamic-cross-game-avatars-for-virtual-worlds-raises-56m-led-by-a16z/">$56M series B funding round</a> led by the prestigious VC firm Andreessen Horowitz (a16z).</p><p><strong><em>Creation</em></strong></p><p>There are two components to Ready Player Me:</p><p><strong>The first component</strong>: The Avatar creator product. A web-based avatar builder that anyone can use. The idea of this avatar builder is that everyone can easily create a custom avatar based on a simple selfie or photo:</p><ol><li>Take a selfie or pick from a selection of pre-made avatars.</li><li>Customize your avatar and choose from 300+ options.</li><li>Export your avatar and use it in one of the 6500+ supported apps and games to explore Metaverse-like environments.</li></ol><p>Ready Player Me developed a patent-pending AI technology for predicting realistic face shapes from a single 2D photo. The tech uses a proprietary database of 20,000 realistic face scans captured with their professional 3D scanners.</p><p>Ready Player Me introduced an AI texture creation tool at the start of February 2023, empowering avatar creators to customize their virtual outfits with text commands, delivering instant results for their RPM characters.</p><p>Important to note is that Ready Player Me is also actively looking at turning their avatars or avatar cosmetics into NFTs.</p><p><strong>The second component:</strong> The Ready Player Me SDK (software development kit) allows developers to integrate RPM, including the avatar creation, into their application.</p><p><strong><em>Integration</em></strong></p><p>Ready Player Me has spent seven years building the perfect avatar system, with a strong focus on integration and compatibility.</p><p>glTF and VRM are the two most popular avatar standards in Metaverse environments. According to the<em> CTO Rainer Selvet</em>,<a href="https://thenewstack.io/ready-player-me-and-the-challenges-of-3d-interoperability/"> Ready Player Me has been building on top of the glTF standard</a> since the early days.</p><p>RPM is actively looking into newer emerging standards in addition to glTF like USD (universal scene decryption), an open standard developed by Pixar.</p><p>To integrate with the most prominent players, Ready Player Me has made multiple SDKs:</p><ul><li>RPM for Unity</li><li>RPM for Unreal Engine</li><li>RPM for Web apps</li><li>RPM for iOS applications</li><li><a href="https://docs.readyplayer.me/ready-player-me/">RPM for Android applications.</a></li></ul><p>They even went a step further and recently launched their avatar API. With the help of this API, developers can fully customize the RPM avatars to be fully compatible with their Metaverse environments.</p><p><strong><em>Monetization</em></strong></p><p>Both the avatar creation tool and professional SDKs are free to use. Right now, <a href="https://venturebeat.com/games/ready-player-me-raises-56m-for-metaverse-avatars/">Ready Player Me wants a wide range of users</a> to be integrated into many Metaverse worlds, games, and virtual environments.</p><p>The ultimate goal of RPM is to become the largest platform for avatar systems-as-a-service. How they will monetize is not clear.</p><h4><strong>Didimo</strong></h4><p>The <a href="https://www.didimo.co/">Didimo</a> platform enables the rapid, mass creation of digital avatars. Users can create lifelike avatars by transforming 2D facial images into 3D models. The company is dedicated to elevating the Metaverse experience through the infusion of real emotions and personalization.</p><p>In October 2022, Didimo secured $7.15 million in a Series A funding round led by Armilar Venture Partners, aimed at boosting the growth of their cloud-based platform. The company was founded and led by Veronica Orvalho.</p><p><strong><em>Creation</em></strong></p><p>The name “Didimo” is derived from the Greek word for “twin,” reflecting the company’s goal of creating highly realistic avatars. Didimo utilizes a proprietary modular pipeline to streamline the various processes in delivering these avatars. The platform works:</p><ul><li>The user provides a sing front-facing image, selfie, or scan.</li><li>The Didimo tech creates a 3D mesh based on the user&#39;s face</li><li>The Didimo pipeline automatically creates a rigging system</li><li>The rig allows for manipulation of the face and expressions</li><li>The pipeline delivers a high-fidelity 3D model ready to use.</li></ul><p>Didimo can deliver professional results in a matter of minutes. Their AI and pipeline solutions are patented.</p><p><strong><em>Integration</em></strong></p><p>Didimo offers multiple integration solutions, including a B2B platform-as-a-service that provides businesses, developers, and content creators with the tools necessary to generate realistic avatars. The avatars can be seamlessly integrated into games, websites, VR/AR experiences, etc.</p><p>Didimo can be integrated with leading industry tools such as Unity, Unreal, WebGL, Oculus, iOS, and Android via its custom API. The company is also actively developing multiple SDKs, with their <a href="https://github.com/didimoinc/didimo-digital-human-unity-sdk?utm_source=dev-portal&amp;utm_medium=dev-portal&amp;utm_campaign=ALL">Unity SDK</a> already available, allowing developers to create digital humans from a simple selfie.</p><p><strong><em>Monetization</em></strong></p><p>Didimo uses a pricing model based on a ‘point system.’ Points are credits that developers can use to create avatars. In total, they offer four packages:</p><ul><li>A trial package that allows users to test the platform and generate their samples for free.</li><li>A starter package that costs $100, this package can create up to 35 face models and 30 full-body characters.</li><li>An advance access package of $350, this package gives developers access to ARKit poses and lets you create 173 face models and 152 full-body characters.</li><li>Finally, they also have a custom pricing solution for enterprise contracts that want to integrate large volumes of realistic avatars into their platform or environment.</li></ul><p>Packages can be purchased on the customer portal of Didimo and are one-time payments. Once you have reached your limit, you can re-purchase a package.</p><h4><strong>Anything World</strong></h4><p>Anything World is a cutting-edge web-based 3D animation platform that brings static models to life through dynamic experiences. The company was founded by Gordon Midwood and Sebastian Hofer, both accomplished developers with over 20 years of experience at industry giants such as Google, Facebook, PlayStation, and Apple. Based in London, Anything World secured <a href="https://anything.world/showcase">$7.5 million in funding</a> from Warner Music Group in late 2022.</p><p><strong><em>Creation</em></strong></p><p>Anything World streamlines the animation process for users and to easily animate static 3D models with a click. This advanced technology enables developers to quickly and cost-effectively create interactive 3D experiences, including metaverse worlds, games, unique filters, 3D content, and more.</p><p>One key feature that sets Anything World apart from other animation platforms is its ability to create 3D animations using voice commands. Allowing users to auto-rig and animate static 3D models with a few simple voice commands. Simplifying the animation process and offering an additional level of creative freedom.</p><p><strong><em>Integration</em></strong></p><p>Anything World boasts a community of 7,500 creators building interactive experiences on the platform. The company has also integrated with leading 3D library platforms, such as Sketchfab and Turbosquid, providing users access to over 500,000 3D models for their creations.</p><p>The platform also offers seamless integrations with industry-leading software programs, Unity and Unreal Engine, allowing experiences created on Anything World to be easily integrated into these programs. Additionally, Anything World integrates with Roblox, a widely popular user-generated content game with over 60 million monthly active players. Several mini-games within Roblox have been created using Anything World.</p><p><strong><em>Monetization</em></strong></p><p>Anything World operates on a Software-as-a-Service (SaaS) model by providing individual creators a free version, including one seat and limited access. Professional studios and enterprises can access the full range of features and capabilities by subscribing to a monthly plan tailored to their size and usage needs.</p><p>It is also important to note that Anything World incorporates a bandwidth fee for monetized games or experiences created on the platform. This means that creators will be charged a fee based on the number of API calls or time spent in the application. The exact bandwidth fee is undisclosed.</p><h4><strong>Kinetix</strong></h4><p>Kinetix is a no-code 3D creation tool powered by artificial intelligence. They democratize access to 3D animations for all creators by developing a user-friendly platform that enables user-generated content. The company is Paris based and was <a href="https://whitepaper.kinetix.tech/project/summary">founded in 2020 by Yassine Tahi and Henri Mirande</a>.</p><p>With the help of Kinetix, everyone can now produce animations for video games, virtual worlds, virtual concerts, and more! Powered by user-generated content, the platform opens new ways of self-expression in the Metaverse.</p><p>In May 2022, Kinetix received $11 million in seed funding. The financing round was led by Adam Ghobarah, founder of Top Harvest Captial and former longtime executive at Google. The funding round also saw the participation of multiple virtual worlds like The Sandbox and ZEPETO.</p><p><strong><em>Creation</em></strong></p><p>The Kinetix platform allows users to freely create, edit and mint high-quality emote NFTs thanks to their powerful AI and no code tools. Today the platform has over 30,000 users with +15,000 monthly active users. Below is an overview of how to use the tool:</p><ul><li>Users have three options for creating an emote: upload a video clip, select from Kinetix’s library, or use an online URL.</li><li>Kinetix’s AI will automatically recognize dance moves in video clips and convert them into animated avatar emotes. Important to note that each emote will have certain traits based on the characteristics of the emote (f.e. speed, stamina, spin, strength).</li><li>Emotes can be edited to the user’s desired style by mixing and matching animations.</li><li>Showcase finished emotes on Kinetix’s NFT marketplace, designed for 3D animations and minted on the Polygon blockchain. Users can also download the 3D source file directly (GLB format).</li><li>Integrate emotes into worlds partnered with Kinetix for creative potential (more about this in the integration part).</li></ul><p>Kinetix’s goal is to provide users with countless possibilities to create custom emotes that will boost their self-expression within virtual environments and metaverses.</p><p><strong><em>Integration</em></strong></p><p>Kinetix is dedicated to expanding its reach by partnering with virtual worlds and environments, providing its users with endless opportunities to showcase their custom-made emotes. Currently, Kinetix is collaborating with over 20 industry leaders and pioneers, with more partnerships in the works. They focus on virtual worlds, creation studios, IP holders, organizations, and technology providers.</p><p>Some of Kinetix&#39;s most notable integrations include popular metaverse worlds such as The Sandbox, Decentraland, Zepeto, and Next Dancer, as well as a custom dancing game on Roblox.</p><p>To further aid developers, Kinetix is developing a Unity SDK that enables the integration of its core features into other virtual environments. With this SDK, developers can easily add a custom emote wheel or a store of hundreds of emote, bringing new benefits to users. The SDK is currently in development, and developers can sign up for the beta program to test it out.</p><p><strong><em>Monetization</em></strong></p><p>Kinetix offers a free online creation tool to design custom emotes, including NFT emotes. However, they have monetization strategies in place to generate revenue.</p><p>First, Kinetix offers premium NFT emotes created in partnership with brands or developers, with perks such as reduced minting fees, higher royalties, exclusive emotes, and more.</p><p>Second, Kinetix aims to create a creator-friendly NFT marketplace, eliminating trading fees and giving 90% of the sale price back to the owner (with a 5% royalty for both Kinetix and the creator).</p><p>These two monetization methods are currently the focus of Kinetix. The ultimate goal is to attract a large user base and provide a unique emote creation experience.</p><h4><strong>Inworld</strong></h4><p><a href="https://inworld.ai/">Inworld</a> is a cutting-edge web-based platform that empowers users to create realistic, AI-powered games and entertainment characters. The platform offers advanced NPC behavior and unscripted dialogue, bringing unprecedented depth and intelligence to metaverse characters.</p><p>The Inworld team boasts a highly experienced and accomplished group of AI experts with over a decade of experience in the field. One of the founders and CTO, Michael Ermolenko, is a leading AI researcher and former employee of Google.</p><p>The startup successfully closed its Series A funding round last year, raising $70 million from some of the top venture capital funds in the industry, including Bitkraft, Microsoft, and LG. Inworld was also exclusively selected as one of the companies to participate in Disney’s 2022 Metaverse and Web3 Acceleration Program.</p><p><strong><em>Creation</em></strong></p><p>Inworld offers a web-based platform that allows users to create AI-powered virtual characters, or “avatar brains.” Users can train their avatars to possess specific abilities, behaviors, emotions, and speech patterns. The platform can give characters unique personalities, moods, and characteristics. Users provide text input into the tool, and the system does the rest.</p><p><strong><em>Integration</em></strong></p><p>Once users have “trained” their avatars, they can test them in virtual reality using the Inworld Oculus companion app. Afterward, the characters can be integrated into games and apps with the help of Unity and Unreal Engine SDKs. Inworld has a fully-operational plugin system for these platforms, making it easy for developers to experiment with their creations.</p><p>If you’re interested in experiencing Inworld’s capabilities firsthand, the company recommends trying the arcade function on its website. This feature allows users to interact with the multiple AI chatbots that Inworld has created, giving a glimpse of the avatars created with the platform.</p><p><strong><em>Monetization</em></strong></p><p>Inworld operates on a Software-as-a-Service (SaaS) business model, providing a free version with unlimited character creation and customization options and 120 minutes of free API time. Additionally, they offer three paid monthly subscriptions tailored to professional users based on API usage and integration needs.</p><p>Inworld is in the Beta phase, and the official release date is not announced. It is expected to be released later this year.</p><h4><strong><em>Voicemod</em></strong></h4><p>Voicemod is a renowned audio avatar company that offers a real-time AI voice changer, enabling users to add voice-changing effects to their avatars, characters, or other virtual identities. The company, founded by three brothers, is based in Valencia, Spain, and has raised €7.1 million in funding led by Bitkraft ventures. The software has been downloaded over <a href="https://www.voicemod.net/blog/">10 million times</a>.</p><p><strong><em>Creation</em></strong></p><p>Voicemod leverages artificial intelligence and digital signal processing to enable users to manipulate their voices in real time. The startup has gained widespread popularity among streamers, YouTubers, and gamers who use it to experiment with alternate personas.</p><p>Users can download the Voicemod software (available on both Windows and Mac) and experiment with voice changes. The platform offers a free soundboard that allows users to change their voices by adjusting various metrics such as pitch, tone, and tremble. The tool lets users create voice skins and avatars that can be integrated into metaverse environments.</p><p>In recent updates, Voicemod has introduced a Text-To-Song music generator. This feature enables users to input their text, select an avatar, and the generator will create a custom song.</p><p><strong><em>Integration</em></strong></p><p>Voicemod is a widely popular voice changer that offers a variety of integrations with popular platforms, making it a convenient tool for streamers. It can be used with Twitch, TikTok, and Omegle and is also compatible with online meeting platforms like Zoom, Google Meet, and Discord. Voicemod can be natively used in many games, including World of Warcraft, Minecraft, Fortnite, Valorante, and League of Legends.</p><p>Furthermore, developers can easily integrate Voicemod into their software, metaverse environments, applications, or games thanks to Voicemod SDK’s seamless API integration. It also allows apps and web pages to use Voicemod.</p><p><strong><em>Monetization</em></strong></p><p>Voicemod offers two versions of its voice changer: a free version that includes voice-changing options and a paid version called Voicemod Pro, which provides access to premium features such as additional voices and voice-changing options. The Pro version costs $45 for a lifetime subscription. Additionally, the Voicemod SDK for professional integrations is available at no cost.</p><h4><strong><em>Reallusion</em></strong></h4><p><a href="https://www.reallusion.com/about/default.html">Reallusion</a> is a platform for digital human character creation and animation. Use cases are for Reallusion characters to populate across Media and entertainment, Metaverse, digital twin factories, Architectural visualization, and AI simulations. The US-based company was founded in 2000 by Charles Chen.</p><p><strong><em>Creation</em></strong></p><p>Reallusion is a company that specializes in creating software for animation and character design. One of their flagship products is iClone, a real-time 3D animation software that helps users produce professional animations.</p><p>Reallusion’s Character Creator is a full-character solution that allows designers to generate, import, and customize stylized or realistic avatars. One of the features of this product is the HEADSHOT AI-powered creator plug-in. Generates 3D real-time digital humans from a single photo using intelligent texture blending and head mesh creation. The generated digital doubles are fully rigged for voice lip sync, facial expressions, and full body animation, making it a powerful tool for character animation and design.</p><p><strong><em>Integration</em></strong></p><p>Reallusion products have many plug-ins to enhance the creator experience. These include Blender character animation, unreal live link, metahuman live link, omniverse, Iray, and popcorn FX for the iclone integrations.</p><p><strong><em>Monetization</em></strong></p><p>Reallusion has different models that can be used to license its products. They work with perpetual licensing models. Each product is purchased stand-alone as well as certain features or libraries which have to be purchased separately. The most optimal model that can be requested is the enterprise license which will include a perpetual royalty-free license and full IP ownership over derivative content.</p><h4><strong>VRSTL</strong></h4><p>VRSTL aims to be the ultimate solution for 3D and Avatar Interoperability. It offers a convenient, all-in-one tool for brands and creators to create digital product collections, customize avatars, optimize 3D files for various distributions using an AI-powered optimization algorithm, and auto-rig them for web2 and web3 gaming. It is closely tied to Threedium (both were founded by Mike Charalambous).</p><p><strong><em>Creation</em></strong></p><p>There are three parts to the VRSTL creation process; <strong>Build</strong>, <strong>Customize,</strong> and <strong>optimize</strong>.</p><p><strong>Builder</strong>: The VRSTl builder can be used by creators to create 3D wearables and customize avatars. The builder is web-based.</p><p><strong>Optimizer</strong>: Optimizer can convert any 3D assets to industry-standard 3D formats with their proprietary AI algorithm. It can optimize 3D assets based on custom inputs like polygon number, texture size, merge, etc.</p><p><strong>Animator</strong>: The animator can auto-rig avatars for any specific metaverse. They can then download any animation from hundreds of animations in the asset library.</p><p><strong><em>Integration</em></strong></p><p>VRSTL supports multiple 3D environment standards like Unreal Engine, unity, Decentraland, Roblox, Zepeto, Sandbox, Spatial, and OVR. This makes it easy for creators to make a 3D asset compatible with multiple metaverses.</p><p>They allow creators to mint NFTs linked to their 3D assets. Owners can store their wearables in their wallets (e.g., Metamask) and connect to virtual worlds with their digital assets. They have integrated with the Ethereum and Polygon blockchains.</p><p><strong><em>Monetization</em></strong></p><p>No information on monetization is available yet. It would seem most likely that they would use a licensing model.</p><h4><strong>Emoshape</strong></h4><p><a href="https://emoshape.com/">Emoshape</a> has several different technologies focused on adding emotions to AI. This is in the form of emotional voice recognition, natural language generation, or emotional voices/ facial expressions by NPCs</p><p><strong><em>Creation</em></strong></p><p><strong>SER </strong>(speech emotion recognition) is a cloud service that utilizes state-of-the-art emotion recognition deep learning frameworks pulled from the human voice. Emoshape SER can measure directly from someone’s voice whether they are angry, happy, fearful, or neutral state of mind.</p><p><strong>Emotion synthesis </strong>emoshape EPU (emotion processing unit) can generate any emotional state and allows the AI models to emotionally understand a situation, emotional reasoning, and decision-making. These emotions then get transferred to a natural language generation system.</p><p><strong>Empathy in synthetic voices </strong>Emoshape has a proprietary AI voice offering. Ti’s advanced emotion synthesis can dynamically layer emotion and empathy into any voice application.</p><p>They use all this technology and package it up in their <strong>MetaSoul </strong>offering. This is where AI will be put into the avatar&#39;s NPCs making for unique and dynamic in-game interactions. As well as allowing the NPCs to learn and adapt based on interactions they have had with the players of the game.</p><p><strong><em>Integration</em></strong></p><p>Emoshape offers cloud API services for their EPU. Developers can develop games around the EPU cloud using Unity and Unreal engine four existing plugins. For IoT, automotive, and robotics, the full SDK comes with EPU full functionalities, and GUI precompiled for macOS, Windows, RPI, and Ubuntu and can be cross-compiled for other platforms like Android and IOS.</p><p><strong><em>Monetization</em></strong></p><p>Access to Emoshapes EPU cloud instances comes for 18$ a month for 1 Unit(AI persona). The price<a href="https://emoshape.com/shop/"> per unit</a> is lower depending on the number of units ordered. (use table)</p><p>Emoshape EPU 2 and 3 SDK license is sold for $7000 per unit.</p><h4><strong>Plask</strong></h4><p><a href="https://plask.ai/">Plask</a> streamlines the animation process for character movement through its automated web solution, enabling the creation of top-notch animations at a reduced cost compared to traditional methods.</p><p>In October 2021, Plask secured $2.56 million in funding from investors led by South Korean venture capital firm Smilegate Investment to enhance its AI-powered animation software. The start-up is based in Los Angeles, United States.</p><p><strong><em>Creation</em></strong></p><p>Plask leverages “pose extraction” and advanced deep learning technology to effortlessly produce animated character movements. With its browser-based tool creators can upload dance videos and convert the movements into animations without using costly mocap suits.</p><p>The resulting motions are converted into GLB files which can be easily modified within the tool or downloaded for integration into games, virtual environments, and other applications.</p><p><strong><em>Integration</em></strong></p><p>Plask is a B2B-focused company that offers a highly customizable API to integrate its AI motion technology into games, enterprise, and educational applications. The API delivers motion capture data in JSON format, including animations of each humanoid joint, which can be directly applied to your 3D characters.</p><p>The JSON files are compatible with leading software such as Unity, Maya, Unreal, and 3Ds. Additionally, Plask provides a Motion SDK enabling clients to effortlessly add characters and apply motion using mobile video footage.</p><p>According to our research, Plask is less focused on integrating with Metaverse or virtual environments like Sandbox and Decentraland.</p><p>Their solution is more focused on integrating their API with various players in the market.</p><p><strong><em>Monetization</em></strong></p><p>Plask offers its tool through both a freemium and subscription model. In the free version, users are granted a limited number of credits for motion extraction from video captures in various formats such as FBX, GLB, MP4, MOV, and WEBM files. The premium subscription, at $50 per month, provides unlimited motion extractions with faster processing and the ability to capture multiple people, unlike the one-person limit in the free version.</p><p>For the API and Mobile SDK, Plask utilizes a usage-based pricing model, charging per frame processed by their GPU. This allows users to pay only for what they use. Plask also recognizes the importance of education and offers a 50% discount to academic institutions incorporating the Plask software and API into their curriculum.</p><h4><strong>Reblium</strong></h4><p>Reblium is a web-based avatar creation platform focused on creating high-fidelity virtual influencers. The company is Dutch based founded by Mao Lin and Adam Tati, who have decades of experience in the movie, gaming, and animation industry.</p><p>Reblium is one of the few exclusive start-ups participating in the Outlier Venture and Farfetch accelerators. They have not raised any funding.</p><p><strong><em>Creation</em></strong></p><p>Reblium has developed a cutting-edge web-based avatar creation tool that empowers users to create unique avatars by uploading photos or selecting from a wide range of pre-made avatars in their digital library. Although currently in beta, the tool is set to be launched to the public in Q1 2023. Some of the features of this tool include ultra high-fidelity avatars, a vast library of digital fashion items, detailed make-up and tattoo options, and full IP-RIGHTS to the avatar created.</p><p>This last feature is especially noteworthy, as Reblium aims to provide brands and companies with the ability to create their virtual influencers &amp; VTUBERS for advertising purposes. The Reblium platform also enables users to record videos of their avatars.</p><p><strong><em>Integration</em></strong></p><p>Reblium is actively exploring a range of integration options to expand the capabilities of its avatar creation tool. These include integration with the Apple AR kit, allowing users to use their avatars as filters on their phones or in virtual meetings.</p><p>Additionally, Reblium is exploring integration with MOCAP suits, enabling professionals to animate their avatars for use in movies, games, and metaverse worlds. The company is also working with B2B partners such as Sephora, a leading cosmetics brand, to integrate its avatar creation tool into the Sephora metaverse, showcasing the latest make-up trends.</p><p><strong><em>Monetization</em></strong></p><p>Reblium will introduce a freemium model for its web-based avatar creation tool, allowing users to explore the platform and create their avatars at no cost. To access premium features and enterprise solutions, the company will offer custom pricing through a Software-as-a-Service (SaaS) model.</p><p>The specific pricing details are not announced. Furthermore, Reblium’s business strategy also includes providing white-label solutions for industry leaders looking to showcase their products and services on high-fidelity avatars, such as fashion and cosmetics. These white-label solutions will be a big part of their business model.</p><h4><strong><em>Radical</em></strong></h4><p>Radical uses advanced deep learning techniques to enable a (mostly) cloud-based 3D motion capture, animation, and human visualization solution. They can transform 2D video into 3D animation with motion capture from a single camera angle. Radical was founded by Gavan Gravesen and has raised ($1.6 million to date).</p><p><strong><em>Creation</em></strong></p><p>Radical uses a proprietary configuration of computer vision, deep learning, and anthropometry to enable human motion capture from 2D content. Radical makes motion capture is possible from a single camera angle. Their unique cloud rendering techniques allow for real-time motion capture streaming.</p><p><strong><em>Integration</em></strong></p><p>Radical can either be used with their FBX output out-of-the-box, or developers can use the other plugins and add-ons to connect animations to 3D environments. Integrations include Autodesk, Unreal Engine, Unity, Blender webGL, and iClone7.</p><p><strong><em>Monetization</em></strong></p><p>Radical has different price tiers for its core motion capture offering; creator ($8/mo), producer ($28/mo), and professional ($78/mo). These all give a certain amount of monthly credits, and users get charged once these credits are exceeded on the Pay-as-you-go rate. For access to real-time motion capture, there is no fixed rate. Can be inquired for by getting in touch.</p><h4>VROID</h4><p><strong><em>Creation</em></strong></p><p>VRoid is a 3D project created by Pixiv. Inc. They have the philosophy of “Make Creativities More Enjoyable”. With their VRoid studio app, users can create characters as easily as draw pictures. On top of that, the software gives the user total freedom to express originality, avatar customizations, and detailing. VRoid was launched in 2018 and is very big in Japan and China.</p><p><strong><em>Integration</em></strong></p><p>The VRoid studio software is used to create hand-drawn custom characters. Once a character is created you can use the VRoid Hub to show off your creations. Other users can also download your creation and integrate this with one of the many partner apps. The VRoid characters are compatible with a number of popular applications such as VRChat, VRDancer, FumiFumi, and Virtu. The VRoid studio and hub are very popular in Japan and China, the online platform has over 800,000 monthly visitors.</p><p>VRoid also has a Unity SDK. With UnitySDK developers can easily integrate VRoid characters into their virtual environments. They also provide an avatar management system for creating multiplayer games with VRoid characters. There are also API possibilities.</p><p>Last year, VRoid closed a partnership with Viverse, the HTC metaverse platform. This brings the VRoid anime character style to the virtual metaverse.</p><p>Next to VRoid studio and the VRoid hub they also offer VRoid Mobile, an avatar camera app to create 3D characters based on your pictures. Next to that they also have VRoid wear these are real-life items like clothing and other accessories in collaboration with VRoid characters.</p><p><strong><em>Monetization</em></strong></p><p>The VRoid studio, hub, and camera app are free to use for anyone. It’s very easy to set up, as you just need to download the software. All models that are created with the software are free to use. You can even commercialize the characters that you’ve created.</p><p>When you want to integrate the VRoid SDK or API as a professional, some of the features are not free. You need to contact the VRoid team to get custom pricing based on your case.</p><h4>Avaturn (In3D)</h4><p><strong><em>Creation</em></strong></p><p>In3D turns people into realistic full-body 3D avatars with the help of your mobile phone camera. In 2023, In3D rebranded to Avaturn, a platform that allows users to create and customize their selfie avatars. Avaturn wants to partner up with various metaverse players and platforms, so users can easily integrate their selfie avatars into metaverse environments. The company was founded by Dmitry Ulyanov, Vadim Lebedev, Ekaterina Ulianova, and Sergei Sherman.</p><p><strong><em>Integration</em></strong></p><p>The avatars created with Avaturn are fully rigged, 3D avatars that are very easy to animate. On top of that, the avatars are compatible with most major software platforms, such as Unity, Unreal, Android, IOS, and more. All avatars that are created via the software are .glb files.</p><p>Avaturn currently has a full SDK integration for professionals. They also offer a developer program that allows professionals to experiment with the API and customization.</p><p>Avaturn recently partnered with echo3D. Echo3D is a cloud platform for 3D asset management and provides developers with the infrastructure to build AR and VR games.</p><p><strong><em>Monetization</em></strong></p><p>Avaturn is currently in a beta phase and users can create their own avatar for free. For more professional usage, we recommend contacting the Avaturn team directly, as no pricing models are yet available to the public.</p><h4>Union Avatars</h4><p><strong><em>Creation</em></strong></p><p>Union Avatars is an avatar creator tool to enhance the virtual world and metaverse experiences. Their tool allows users to create customized avatars from different styles by uploading a picture, like the realistic or voxel. Within minutes the avatar is ready to be used in virtual environments. The Barcelona-based company was founded in 2020 and has raised $1.5M to date.</p><p><strong><em>Integration</em></strong></p><p>Union Avatars has SDK integration toolkits and APIs for developers that are ready to deploy to mobile, desktop, VR, and AR-based applications. Their tools are compatible with android, IOS, windows, unity, and Unreal engine.</p><p><strong><em>Monetization</em></strong></p><p>The union avatar toolkit can currently be used for free upon request.</p><p><em>This article is a collaboration between </em><a href="https://kinetix.tech/"><strong><em>Kinetix</em></strong></a><em> and </em><a href="https://www.metanomics.be/"><strong><em>Metavision</em></strong></a><em>.</em></p><p><strong>About Metavision:</strong></p><p>MetaVision is a VC firm with a focus on web3 and metaverse founded by entrepreneurs. MetaVision invests in early-stage startups with big ambitions and a vision for the future.<br><strong><em>LinkedIn:</em></strong><em> </em><a href="https://www.linkedin.com/company/metavision-international/"><em>https://www.linkedin.com/company/metavision-international/</em></a><em><br></em><strong><em>Twitter: </em></strong><a href="https://twitter.com/metavision_vc"><em>https://twitter.com/metavision_vc</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea1316ecca7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating with Polygon for interoperable Emotes!]]></title>
            <link>https://medium.com/@kinetixtech/integrating-with-polygon-for-interoperable-emotes-2924a76d8d23?source=rss-9c8fa0f9d89b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2924a76d8d23</guid>
            <category><![CDATA[web3]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[kinetix]]></category>
            <category><![CDATA[polygon]]></category>
            <category><![CDATA[polygon-network]]></category>
            <dc:creator><![CDATA[Kinetix]]></dc:creator>
            <pubDate>Mon, 16 Jan 2023 15:54:16 GMT</pubDate>
            <atom:updated>2023-01-18T08:33:06.031Z</atom:updated>
            <content:encoded><![CDATA[<h4>Kinetix enables its community of thousands to mint Emotes as NFTs. This article explains how it drives utilities by facilitating Emotes integration within games and virtual worlds and why we chose Polygon as a Blockchain partner.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*om10GaM5ZG9VWxWhyn7X1w.png" /></figure><p>Kinetix’s vision is to help drive mass adoption of the Metaverse by facilitating self-expression through emotes. Through AI and no-code tools, we are empowering our users to create emotes that represent them.</p><p>Kinetix’s vision is to help drive mass adoption of the Metaverse by facilitating self-expression through emotes. Through AI and no-code tools, we are empowering our users to create emotes that represent them.</p><p>Reaching interoperability is our biggest target. For that, we made a healthy choice of making our Emotes compatible with every ownership format. Indeed, Kinetix Emotes can be saved as assets to be played in Web2 environments but can also be turned into Web3 assets. Our white paper explains everything about Emotes formats and utilities in further detail, you can read it <a href="https://whitepaper.kinetix.tech/project/summary">here</a>.</p><p>Providing every user with a great minting process has been a challenge for us since most users need to become more familiar with Web3. For that, we considered many factors to select a safe blockchain that facilitates interoperability and accessibility. In our evaluation process, Polygon appeared to be the best technological bridge allowing the integration of Emotes within Web3 games and virtual worlds. Polygon is on the same page as us as they provide services lowering barriers to entry in Web3.</p><p>Here are the main reasons why we chose Polygon:</p><h4><strong>Accessible Minting Price &amp; Low Gas Fees</strong></h4><p>With this blockchain, we are able to provide our users with a minting price of $1 for User-Generated Emote NFTs. This really low price invites everyone to create and mint their first assets with Kinetix. We detail how to create and mint those Emotes <a href="https://whitepaper.kinetix.tech/platform/create/user-generated">here.</a></p><p>The average Gas Price (Gwei) is about <a href="https://dune.com/KARTOD/Polygon-vs-Ethereum-gasfees">1000 times lower on Polygon than on Ethereum.</a> The significantly lower Gas Prices lower the barrier of entry for Metaverse users and significantly lower the Minting prices for NFTs.</p><h4><strong>Battle-tested network</strong></h4><p>With over tens of thousands of <a href="https://polygon.technology/ecosystem">decentralized apps (or dApps)</a> being built on the network, we know Polygon is used by many developers as recognized as a safe and secure environment. This scalability ensures that this solution will be able to cope with very high user demand. Guaranteeing our users with a stable and safe environment is one of our main concern that Polygon blockchain perfectly meet.</p><h4><strong>Compatibility with multiple games &amp; virtual worlds</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SwWEWA9fSuh4mVKpFmMl2w.png" /><figcaption>Polygon Ecosystem</figcaption></figure><p>Major Web3 gaming projects and marketplace are migrating to the Polygon network for scalability. One of the largest Gaming NFT Marketplaces, MagicEden, decided to migrate to Polygon in 2022 while The Sandbox &amp; Decentraland are already established their ecosystems on the Polygon Network. At Kinetix, we want to build a truly interoperable asset that can be used across different Metaverse ecosystems which will be enabled with our upcoming SDK.</p><h4><strong>Fast Transaction Speeds for optimal user experience</strong></h4><p>Polygon blockchain being a Layer-2 blockchain can clock in <a href="https://www.blockchain-council.org/blockchain/solana-vs-polygon-vs-ethereum/#:~:text=Polygon">65,000 transactions</a> per minute and fulfills its major functionality of enabling a multi-chain Ethereum Ecosystem. In comparison, Ethereum clocks in only 15–20 transactions per second. Transaction speed is one of the major factors from a developer’s point of view so that users can have a seamless experience.</p><p>It perfectly matches our creation process criteria. Indeed, we put our efforts into allowing everyone to create Emotes that can be added to games in only minutes. Fast transactions allow owners to directly import their newly minted Emotes in games when connecting their wallets.</p><h4><strong>Carbon Neutral Blockchain</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CBgXGrG038i4jYYT2aj-uQ.jpeg" /><figcaption>Carbon Neutral Blockchain</figcaption></figure><p>In mid-April, Polygon released its <a href="https://polygon.technology/blog/polygon-is-going-carbon-negative-in-2022-with-a-20-million-pledge">Green Manifesto</a> to be a sustainable network and has pledged to work on a series of community initiatives that will reduce the environmental impact of Blockchains. Within 2 months, Polygon achieved its goal of carbon neutrality by participating in a series of initiatives that will allow them to offset its carbon consumption. It is very important for us to select partners sharing a responsible approach to corporate social responsibility.</p><h4>Conclusion</h4><p>This article was drafted by <a href="https://www.linkedin.com/in/arthurjacq/">Arthur Jacq</a> (Head of Marketing at Kinetix) &amp; <a href="https://www.linkedin.com/in/prabhugaunkar/">Aditya Prabhugaunkar</a> (Marketing Manager at Kinetix).</p><p>Minting Emotes with Kinetix is as easy as creating them. We are proud to collaborate with Polygon as we judge their solution as the best blockchain to allow thousands of new users to enter the Metaverse with Emote NFTs. Emote NFT allow more creators to benefit from the positive benefits of Web3. With Kinetix, creators decide if they want to become sole owners of their emotes to use them in-game and in virtual worlds and trade them with other users. Thanks to blockchain we are opening a new form of monetization for the Creators of tomorrow. What’s your view on the blockchain? Do you think facilitating the emergence of Web3 models benefits the Creator Economy?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2924a76d8d23" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Human Body Estimation, Why good academic Metrics don’t necessarily mean good animations for…]]></title>
            <link>https://medium.com/@kinetixtech/human-body-estimation-why-good-academic-metrics-dont-necessarily-mean-good-animations-for-79212e3dbbd3?source=rss-9c8fa0f9d89b------2</link>
            <guid isPermaLink="false">https://medium.com/p/79212e3dbbd3</guid>
            <category><![CDATA[avatar]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[motion-capture]]></category>
            <category><![CDATA[research-and-development]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Kinetix]]></dc:creator>
            <pubDate>Thu, 22 Dec 2022 11:22:29 GMT</pubDate>
            <atom:updated>2022-12-22T11:22:29.717Z</atom:updated>
            <content:encoded><![CDATA[<h3>Human Body Estimation, Why good academic Metrics don’t necessarily mean good animations for end-users</h3><h4>Understanding why common evaluation metrics are not the final criteria to achieve good results for users. Challenging beliefs for better perception of 3D animations by non-academic.</h4><p><em>The article was written by </em><a href="https://www.linkedin.com/in/luc-jiang-819278153/"><em>Luc Jiang</em></a><em>, R&amp;D Engineer in Deep Learning at </em><a href="https://kinetix.tech/"><em>Kinetix</em></a></p><figure><img alt="human vs 3D skeleton vs avatar. A visual representation of how Body Estimation works" src="https://cdn-images-1.medium.com/max/720/1*9INUzo_4Gz6zEq5rNSpwAQ.png" /></figure><h3><strong>Introducing Challenges of human body estimation in computer vision</strong></h3><p>Monocular 3D human pose reconstruction is a challenging task of computer vision. This task consists of placing a human 3D body in the correct position and posture according to a single point-of-view image or video. The goal is similar to motion capture, which generally involves a sensor body suit, multiple cameras, and calibration of the environment. Such machine learning technology generalizes and simplifies motion capture to a single-view camera removing all body sensors and calibration; thus generating a 3D animation from just a video. The subject is currently studied by numerous research labs and institutes globally, providing complex machine-learning solutions.</p><p>As we are only interested in recreating a human body&#39;s pose and not the person&#39;s complete identity, we can assimilate the human pose as a 3D skeleton with articulated joints (wrists, knees, etc.). A pose is thus represented by the 3D rotations of the skeleton joints and the pelvis&#39; 3D positions, considered the skeleton&#39;s root joint.</p><p>The main machine learning paradigm represents an animation as a sequence of skeleton positions. By taking video as an input (which is a sequence of images/frames), we match the skeleton’s pose to every frame. Working with a human skeleton gives a strong assumption on what the algorithm has to learn and greatly restrains the problem compared to generating the vertices’ position of a body mesh.</p><p>The most popular skeleton used in research is the SMPL-Body [1] skeleton which comprises 22 joints and comes with a mesh parametrized by a shape vector representing the body shape. An extended version exists, SMPL-X [5], which includes hands, a total of 54 joints on the skeleton, and facial expressions.</p><figure><img alt="4  SMPL Models showing the 22 joints and mesh. The 4 models are: Template Mesh, Shape Blend Shapes, Pose Blend Shapes and Final Mesh" src="https://cdn-images-1.medium.com/max/1024/1*sMGKma9hZtgpEKY4M2zwGA.png" /><figcaption><em>The SMPL Model </em>[1]</figcaption></figure><h3><strong>Commonly used Evaluation Metrics in Body Estimation</strong></h3><p>In Deep Learning, neural network models are trained by minimizing a loss function that reflects on one or multiple metrics. Motion capture task is easily adaptable to deep learning as we can train models with a regression loss on the skeleton joints, so minimizing the error between predicted values and ground truth. The main metric used is the Mean Per Joint Position Error (MPJPE) and, as its name indicates, it corresponds to the average distance between the joints of the predicted poses and the joints of the ground truth.</p><figure><img alt="MPJPE formula" src="https://cdn-images-1.medium.com/max/867/1*LMW5AyLk_U8aMjiae7mEiA.png" /></figure><p>The 3D position of each joint can be deduced from the 3D rotations and the parameters of the skeleton (bones’ lengths, joints hierarchy). This metric is easy to understand and interpret as it represents the mean error, usually in millimeters, of predicted joints.</p><p>The predicted root position and orientation of the skeleton, the pelvis joint for SMPL, have a substantial impact on the metric as a correct pose prediction in the wrong global position in space would cause this metric to soar. By removing the pelvis position and rotation from the other joints when computing the MPJPE, we can directly evaluate the body pose from the other joints. This operation is more commonly called Procrustes Alignment, thus giving this metric the name PA-MPJPE. It is generally lower than the MPJPE.</p><figure><img alt="Comparison Table of human body estimation methods by PhysCap [2]" src="https://cdn-images-1.medium.com/max/765/1*bI0oscH1AITEo-EBrO7Tiw.png" /><figcaption><em>Comparison Table of human body estimation methods by PhysCap </em>[2]</figcaption></figure><p>More than joints, we can compare the vertices error (MVE, mean vertices error) of the meshes of the SMPL skeleton [1], as the mesh is parametrized by shape. Some papers focus on the reconstruction of the mesh, for instance, on human-scene interaction.</p><p>We can also cite the PCK metric, the Percentage of Correct Keypoints. Given a threshold on the Euclidean error of the predicted joint, we can classify the prediction of each joint as correctly predicted or not. The PCK corresponds to the percentage of correctly predicted joints. That means for a threshold of 10mm, the PCK (10mm) corresponds to the proportion of joints predicted to be at less than 10mm from the ground truth. That shifts the view on the problem as a classification one and allows us to compute classification-related metrics such as accuracy or AUC (area under the curve) metric.</p><h3><strong>Why Good metrics don’t necessarily make good animations</strong></h3><p>Those metrics try to make models predict poses as close as real animation seen in the dataset during training. Thus, results highly depend on the quality of the animation, which is not always perfect and mostly comes from real motion capture with a bit of noise.</p><p>Some persistent problems are often visible in the resulting animation.</p><p>Jitter is one of the most obvious ones. It can be easily mitigated by filtering and smoothing algorithms but at the cost of precision and dynamism of the movement.</p><p>Interpenetration and self contact is also an issue rarely handled in animation generation. A cross-armed human with arms interpenetrating in each other would look glitchy. It is also the case with interpenetration between the feet and the floor or hands and objects.</p><figure><img alt="A man doing a pose and the avatar doing a similar pose. This gif shows an example of how to arm penetration works" src="https://cdn-images-1.medium.com/max/600/1*1AtaxDHKHXYvgL_kcY_m6g.gif" /><figcaption><em>Example of interpenetration</em></figcaption></figure><p>Foot skating is a major issue that needs to be solved to make animations look realistic. It is visible and troubling for every animation, whether it is for sports, dancing, or fighting scene.</p><p>The video below shows an animation of jumping jacks from MOVI datasets. On the left is the ground truth of the dataset, and on the right, it is the same animation with the grounding algorithm of Kinetix. This technic locks the feet on the ground and at the same level to avoid foot sliding and penetration. The MPJPE on the left is null as it is the ground truth but the right one is modified. Therefore, the MPJPE is not null, and the animation feels more realistic regarding the feet’ contact with the ground.</p><figure><img alt="2 Kinetix avatars doing jumping jack exercises to show improved ground results of feet" src="https://cdn-images-1.medium.com/max/600/1*J37tffkaAEZsdOlleWgv2Q.gif" /><figcaption><em>Grounding results by Kinetix</em></figcaption></figure><p>Finally, realism in animation is about the posture of the body, how natural the movement is, that is proper to humans. We can call this an overall consistency in the animation where we can feel that the dynamism is plausible in real life. Certain aspects are subjective and hard to evaluate for their realism.</p><h3><strong>Understanding Past Academic Research on body estimation</strong></h3><p>Academic research papers in 3D human estimation primarily focus on solving a precise issue in realistic animation. While the Applied R&amp;D team needs to create a complete pipeline that addresses most of these issues.</p><p>In academic research, we can find SimPoe’s [6] research paper from Carnegie Mellon University and Facebook Reality Labs. It gives a foot skating and penetration solution using reinforcement learning controllers in a physics simulation. SimPoe first runs a classic body estimator and refines the resulting animation within a simulator using character controllers to match the body dynamics.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_0BtkjOrwX4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_0BtkjOrwX4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_0BtkjOrwX4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2924401ed42b0dfe98eb7f35bc2cd385/href">https://medium.com/media/2924401ed42b0dfe98eb7f35bc2cd385/href</a></iframe><p>Moreover, some research papers improve specific points given some assumptions on the input and tackle specific cases. For instance, GLAMR [3], from NVIDIA and Carnegie Mellon University, improves the body estimation of obstructed characters by considering that the camera is moving. It also deals with leg and foot motion by predicting the body trajectory using the body pose estimation. It achieves astounding results with the global position estimation, tracking, and mesh reconstruction of multiple characters from a monocular and dynamic camera estimation.</p><figure><img alt="GLAMR [3] pipeline above and a video presenting their results below" src="https://cdn-images-1.medium.com/max/1024/1*AV-vHbqg5qtKbpauC1VwrA.png" /><figcaption><em>GLAMR [3] pipeline above and a video presenting their results below</em></figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FwpObDXcYueo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DwpObDXcYueo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FwpObDXcYueo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/592be2376ec3e4059f4f9f4dbdb3eead/href">https://medium.com/media/592be2376ec3e4059f4f9f4dbdb3eead/href</a></iframe><p>This paper presents significant results in trajectory prediction and improved leg motion consistency in a global environment.</p><p>We can also cite <strong>On Self-Contact and Human Pose</strong> [4] from Max Planck Institute and ETH Zürich**, whic**h focuses on improving self-contact and self-penetration. This paper proposes new self-contact datasets and a regressor to correct mesh contact called TUCH. It first predicts the body pose and shape using the Smpl-X [5] skeleton and then detects and optimizes self-contact to create real contact on the resulting mesh.</p><figure><img alt="TUCH [4] results on self-contact" src="https://cdn-images-1.medium.com/max/743/1*69K650kRZaUToN__vNHt2A.png" /><figcaption><em>TUCH [4] results on self-contact</em></figcaption></figure><h3><strong>Conclusion</strong></h3><p>Academic research mainly focuses on improving some specific aspects and resolving precise issues on the subject. Academic papers deal with important points on body poses and dynamism plausibility providing valuable solutions to these matters. However, for applied research, we mainly focus on having good results in most facets of the problem, taking the computation time into account, the realism of output animation, and tackling the main issue for users. To achieve such a goal and create a complete pipeline, we should not rely on a simple machine learning model because main 3D body estimation metrics are not necessarily realistic metrics, but gather and unify these solutions. Developing a one-size-fits-all Artificial Intelligence is the challenge Kinetix tries to solve to democratize access to 3D animations.</p><p>With the rise in animation demands from video games and virtual worlds, there is a need to build better Machine Learning Models to ensure end users appreciate the results. With the adoption of 3D animations by a larger audience of non-academic researchers, new questions arise. Will the end goal still be to make animations more realistic science-wise or the focus will shift toward developing algorithms for better users’ perception? Which non-academic metrics will emerge as new standards in the industry? Will we observe the development of new algorithms not focused on realistic renderings but on providing smarter customization tools?</p><h3>References</h3><p>[1] Loper, Matthew, et al. “SMPL: A skinned multi-person linear model.” <em>ACM transactions on graphics (TOG)</em> 34.6 (2015): 1–16.</p><p>[2] Shimada, Soshi, et al. “Physcap: Physically plausible monocular 3d motion capture in real-time.” <em>ACM Transactions on Graphics (ToG)</em> 39.6 (2020): 1–16.</p><p>[3] Yuan, Ye, et al. “GLAMR: Global occlusion-aware human mesh recovery with dynamic cameras.” <em>Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</em>. 2022.</p><p>[4] Muller, Lea, et al. “On self-contact and human pose.” <em>Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</em>. 2021.</p><p>[5] Pavlakos, Georgios, et al. “Expressive body capture: 3d hands, face, and body from a single image.” <em>Proceedings of the IEEE/CVF conference on computer vision and pattern recognition</em>. 2019.</p><p>[6] Yuan, Ye, et al. “Simpoe: Simulated character control for 3d human pose estimation.” <em>Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</em> . 2021.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79212e3dbbd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>