<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Julia Schäfer on Medium]]></title>
        <description><![CDATA[Stories by Julia Schäfer on Medium]]></description>
        <link>https://medium.com/@julia.schafer?source=rss-bc4e90d11a0e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*gReJIQtRiK8Gmh5RTVoEJw.jpeg</url>
            <title>Stories by Julia Schäfer on Medium</title>
            <link>https://medium.com/@julia.schafer?source=rss-bc4e90d11a0e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:05:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@julia.schafer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Live Letters: Strike a pose, get a letter]]></title>
            <link>https://medium.com/live-letters/live-letters-strike-a-pose-get-a-letter-e7bcadcc8cd7?source=rss-bc4e90d11a0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7bcadcc8cd7</guid>
            <category><![CDATA[movement-research]]></category>
            <category><![CDATA[typography]]></category>
            <category><![CDATA[live]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[posenet]]></category>
            <dc:creator><![CDATA[Julia Schäfer]]></dc:creator>
            <pubDate>Mon, 11 May 2020 17:54:50 GMT</pubDate>
            <atom:updated>2020-09-17T14:11:05.839Z</atom:updated>
            <content:encoded><![CDATA[<h4>A speculative machine learning tool for expressive text</h4><blockquote>“I’m a frustrated calligrapher. I got caught up in the revival of italic handwriting as a schoolboy in England but could not get the pen to go where I wanted it to. I had to resort to more laborious ways of making letters.” <br><em>— </em><a href="https://www.aiga.org/medalist-matthewcarter"><em>Matthew Carter</em></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*14XE861YGPLpYqgl_i-phQ.jpeg" /><figcaption>Collecting poses</figcaption></figure><p>The speed and shortness of online text exacerbate misunderstanding in modern communication. When writing a message over a keyboard we might be missing out on certain subtleties such as sarcasm and anger. I am interested in speculating on creating a writing tool that supports the fluid and expressive text. I started by asking: Can we imagine a form of writing, which is neither done by hand, nor by using a sort of keyboard? When communicating with text can the letters change while one is writing and reading? These, in addition to others, are just a few of the questions that will be addressed in this article.</p><h3>Activating the Screen</h3><p>Since we spend most of our time sitting in front of a screen we should also think about how new tools could be introduced to activate various gestures as well as body movement. The idea is to create a performative tool that translates poses to letters. While performing live in front of the screen, the letters are predicted and displayed. The interface could be used as a messaging tool or as a way to encrypt movement in a performance setting. Live Letters is my attempt at making this tool.</p><p>Video communication tools have been pushed into every aspect of our lives because of self-isolation restrictions imposed in reaction to COVID-19. Body language is challenging to read over a screen; speaking gets interrupted and meaning gets blunted. People often have to clarify reactions by using chat in addition to video and voice. Physical and dimensional visual cues normally help to decode a message but are lost to the flatness of the screen. In the Live Letters project, I investigate a way to integrate more expression into writing through the body using poses as a foundation for inscription.</p><p>Today, the intersection of bodies and technology is widespread and seamless. These experiments, as well as the work I was focusing on in the last two years, try to examine and disrupt this relationship by recontextualizing it through the lens of design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EulfywZdSZEBlS49it85MQ.jpeg" /><figcaption>(left) “The Tudor Pattern Book“, MS. Ashmole from 1504, folio 45c, Bodleian Library, Oxford, UK, (upper right) historiated initials by Francis Barlow from Edward Benlowes Theophelia, (lower right) Jan Christian Bierpfaff, “<em>Alphabet of organic type”</em> from Libellus Novus Elementorum Latinorum, mid 17th century</figcaption></figure><p>In a written book, human initials or drop caps are mostly at the beginning of a chapter and might show a scene or a recognizable person that relates to the text, in and around the shape of the letter. Sometimes illustrated initials contrast with the text: they might show irreverent or humorous figures and scenes. <a href="https://en.wikipedia.org/wiki/Historiated_initial">Historiated initials,</a> first encountered in <a href="https://www.bl.uk/catalogues/illuminatedmanuscripts/GlossI.asp#INSULAR">Insular</a> illuminated manuscripts, were popular in the first half of the eighth century. These initials, formed by human figures, inspired this research to investigate how the body can be reintroduced as an essential part of transmitting a message.</p><p>The basis of my interest in expressive typography is built on a project titled “Eyes as a Tool” that I completed during my third semester at Yale in 2019. The goal was to use eye-tracking as a way to implement the invisible tracking of looking as a way to bring human features back into a mechanical typeface such as Helvetica.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rOAJLiVGf8tQ0eB5Nadwpg.jpeg" /><figcaption>Results of eye-tracking heatmap experiment imagining letterforms</figcaption></figure><p>Live Letters can be broken into three areas: expression, interpolation, and prediction.</p><h3>Expression</h3><p>The indication of emotion through typography is what I mean by expression here. By exploring both input- as well as the output letterforms through various experiments I try to investigate their expressive potentials. How do the tools we use influence how we read a text? What if the act of inscription is more fluid?</p><h4>Output expression</h4><p>Emoticons were invented to respond to the need to express one&#39;s emotional state and offer another range of “tone” and feeling through texting that portrays specific emotions to support the shortcoming of solely text-based messages. However, emojis help to clarify the writer’s intent. The first generation of emoticons was generated using ASCII text. Several combinations of characters were used to create a variety of emoticons. Sometimes users attempt to express intense emoticons by putting several characters directly after each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*40iQvlJYalKFfwA84Re7IA.png" /><figcaption>Emoticons printed in 1881 in the U.S. magazine Puck, 30 March 1881</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LeCrexoKXU2fQ4g968qvvQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5yuzAkOkOVptGoK8OLb66Q.png" /><figcaption>(left) Emoji based corporate identity for “Les Urbaines” in 2018 by <a href="https://www.p-b.li/">Eurostandart</a>, (right) <a href="http://www.emojidick.com/">Emoji Dick</a> by Fred Benenson</figcaption></figure><p>Another form of expression is, of course, our own handwriting. Ten years ago, the typographer Hans Eduard Meier was prompted to come up with a new so-called “basis font” teaching writing in Swiss schools. Instead of teaching cursive or a more modern san serif lettering, the government decided that there should be one typeface. They wanted it to be very readable and easy to teach. Unsurprisingly, this led to a heated discussion about the loss of expressive handwriting and the fear that the computer keyboard would replace the pen and paper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*NHBVJeEaXkZM4a_-cW6w9w.png" /><figcaption><a href="https://www.basisschrift.ch/schriftdateien">“Basisschrift</a>“ by Hans Eduard Meier</figcaption></figure><h4>Input Expression</h4><p>Similar to the human initials in manuscripts, the input could be tied to an expression of the body whereas the output is a letter. This expressive input could open up new possibilities to achieve a more expressive text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TP1MwrmNG61V4wTRwKxwgA.jpeg" /><figcaption>Illustration by Dace Krēgere</figcaption></figure><p>While in the 16th century, illuminated book initials were far-reaching and trendy, yoga and instruction videos might be today&#39;s equivalent. I started to fantasize: might yoga videos, for example, be a tool to send encrypted messages through body movement? What if we could send hidden messages over commercial platforms?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HczbO1GIjAtP1AKF" /><figcaption>A message “discovered” in an online yoga session.</figcaption></figure><h3>Interpolation</h3><p>Interpolation is defined as the calculation of the value of a function between the values that are already known.</p><h4>Interpolation in 2D</h4><p>Historically, large font families were created by manually calculating the stroke weights between given cuts (for example: Regular and Bold were used to create a Medium cut) and then manually adjust the results to look right. Today, these families are created using digital drawing tools that allow designers to automate the process of interpolation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z8fW8A5Tf72ntCPWv9VJaQ.png" /><figcaption>2-dimensional interpolations: Black and red/yellow/green/blue dots correspond to the interpolated point and neighboring samples, respectively. Their heights above the ground correspond to their values.</figcaption></figure><p>Invented in 2016, variable fonts have become a very powerful tool in a designer’s kit. It allows virtually infinite fonts to be packaged within a single font file. Variable Fonts use vectors and bezier curves to produce a continuous path between two weights.</p><p>Adam Forbes and I created an interactive tool using variable fonts that transcribe voice into text and captures the speaker’s volume as weight; whispers are thin while shouting is so bold it’s almost illegible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*s1R_z4cFOa-bK8CyDgB6XQ.gif" /><figcaption>Voice volume input, variable font as output.</figcaption></figure><p>However, these tools have only explored interpolation between letters. Can you also show what lies between two different letters? Using a tool like Adobe After Effects one can morph between letters and styles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*z327Lhqp_lS0CuPTfbSIPQ.gif" /><figcaption>Letter interpolation via After Effects.</figcaption></figure><p>So how is interpolation different using machine learning? Using an ML model trained on a human-labeled dataset of letters we do not need to rely on mechanically determined interpolations but can recreate a human-like understanding of the essence of an “A”, “B”, etc.</p><p>In the following GAN experiment, I show how interpolation is created using a text to image software from Runway ML called AttnGan. My input text was “The letter A”, “The letter B,” etc. in order to see what a visual interpretation would look like. The output images suggesting letterforms are not informed by the interpretation of the labeled images in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kPrOR7ytOmjO5uKCAtty8w.jpeg" /><figcaption>Describing letters to an AttnGan in Runway ML</figcaption></figure><p>The created images revealed some similarities like the color palette, a suggested spatial dimension as well as the irregular overlaying pattern in the background. It was interesting that some letters actually resembled the input letter, for example, the “A” or the “X”. The artifacts in the output are mysterious, yet resemble house interiors, furniture, or rooms.</p><h4>Interpolation in 3D</h4><p>Live Letters expands interpolation through three-dimensional, human interaction that is time-based.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zu3N9EDMjtbWtiKg52Ysxw.png" /><figcaption>3-dimensional interpolations: Black and red/yellow/green/blue dots correspond to the interpolated point and neighboring samples, respectively. Their heights above the ground correspond to their values.</figcaption></figure><p>The interpolation that is captured through the movement here is seamless and fluid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*HVV8qq47vDc5zVXcZuODiw.gif" /><figcaption>Experiment with movement and continuous overlay</figcaption></figure><p>In the prototype below, the input is expressive through the real-life movement of the body whether the output letter remains fixed. There is no spatial expression achieved in the interpolation of the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*khrilDUw4Cq0h6ETOjla5Q.gif" /><figcaption>Prototype for the Live Letter interface</figcaption></figure><h3>Interpretation or Prediction</h3><p>In order to predict the outcome for a letter through a machine learning model, the lighting, background, and clothing can falsify the expected results and confuse the interpretation. When wearing a face mask, for example, one can fool the AI of Face unlock ID to access a phone. For Live Letters, it would be even more interesting to make use of costumes, materials, or different lighting situations in order to disrupt the outputs from the software.</p><p>The spiritual writing Ouija board shows similar qualities by completely unconscious decision-making processes. The Ouija board, also known as a spirit board or talking board, is a flat board marked with the letters of the alphabet, the numbers 0–9, the words “yes”, “no”. The board is accompanied by a “planchette,” a teardrop-shaped device, usually with a small window in the body, used to maneuver about the board. The idea was that two or more people would sit around the board, place their fingertips on the planchette, pose a question, and watch, dumbfounded, as the planchette moved from letter to letter, spelling out the answers seemingly of its own accord.<br>Could those qualities of creating a message out of an unconscious stream of movements also lead to interesting results with a machine learning model?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*J6gPgCH98_l2j0ZQag9vvA.png" /><figcaption>The makers of the first talking board asked the board what makers of the first talking board asked the board what they should call it; the name “Ouija” came through and, when they asked what that meant, the board replied, “Good luck.”</figcaption></figure><h3>Methods</h3><p>In the five experiments leading up the prototype for a live lettering tool, I looked into <a href="https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5">Pose Estimation</a> as well as <a href="https://teachablemachine.withgoogle.com/">Teachable Machine</a>. Both of these tools let you train your own model to recognize poses. I trained my own dataset on poses I made in front of a webcam. These then allowed me to pose and have the machine predict what letter I performed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hVmL7KhNXd6XXGFO" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6lUkMgX79sU7lL1yDt9sxg.png" /><figcaption>Confidence prediction through Teachable machine</figcaption></figure><p>Another experiment I attempted was using a pose classifier and in p5.js in order to overlay the letter predicted directly over the live image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sC5Uy3NRonV6mzzz" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*So5Atl2g49Zht-dV" /><figcaption>Input movement, output letters</figcaption></figure><p>The third and most complex way to go about it would be to create a keyframe detection model via Tensorflow and Google Collab. This method would entail the following steps:</p><ol><li>Create a dataset with 26 letters by capturing images + one set with no fixed letter poses.</li><li>Pass images through a model to get keyframes.</li><li>Train a network to get the output as letters.</li><li>Load images into Keras in order to resize them to 28x28 px.</li><li>Deploy the model in realtime.</li></ol><p>An existing image classification model called <a href="https://www.tensorflow.org/tutorials/keras/classification">Fashion MNIST</a> could be used by formatting the data to replace it with the existing Fashion MNIST dataset. In the Fashion MNIST dataset, the outcome predicted is a string label; a single character. However, for the body letter dataset, we would need to predict a number (0,1,2,3, ….).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aRyXivQtJ7naCpueE_CeEA.png" /><figcaption>The letter of each pose and the letter predicted by the model.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eaJ9X-Tbe1RnV73Z" /></figure><h4>Results</h4><p>The most exciting results were where the morphing and the interpolation were visible and expressed such as in the After Effects example as well as in the Machine Learning text to image AttnGan because the shapes expressed were unexpected and new. It seems that there is a huge potential by including interpolation in the visual output that has a spatial dimension as well.</p><h3>Conclusion</h3><p>In expression, interpolation, and prediction I outlined each step leading up to the Live Letter prototype which uses real-time pose estimation to output text. Tools like this make communication feel less and leave space for more emotional input<em> — </em>which is especially urgent when life is spent over video chat tools.</p><p>In a future prototype, I would like both the input and output to be expressive and spatial; A messaging tool that is as emotive as an illuminated manuscript’s initials and as fluid as a choreographed dance piece.</p><h4>References</h4><p>TensorFlow Basic image classification: <br><a href="https://www.tensorflow.org/tutorials/keras/classification">https://www.tensorflow.org/tutorials/keras/classification</a></p><p>Fashion MNIST<br><a href="https://github.com/zalandoresearch/fashion-mnist">https://github.com/zalandoresearch/fashion-mnist</a></p><p>Open Pose Estimation<br><a href="https://nanonets.com/blog/human-pose-estimation-2d-guide/">https://nanonets.com/blog/human-pose-estimation-2d-guide/</a><br><a href="https://medium.com/pixel-wise/real-time-pose-estimation-in-webcam-using-openpose-python-2-3-opencv-91af0372c31c">https://medium.com/pixel-wise/real-time-pose-estimation-in-webcam-using-openpose-python-2-3-opencv-91af0372c31c</a></p><p>Open Pose<br><a href="https://storage.googleapis.com/tfjs-models/demos/posenet/camera.html">https://storage.googleapis.com/tfjs-models/demos/posenet/camera.html</a><br><a href="https://ml5js.org/reference/api-PoseNet/">https://ml5js.org/reference/api-PoseNet/</a><br><a href="https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5">https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5</a></p><p>Interpolation in Type Design<br><a href="https://www.underware.nl/case-studies/">https://www.underware.nl/case-studies/</a><br><a href="http://www.eyemagazine.com/feature/article/from-punch-cutters-to-number-crunchers">http://www.eyemagazine.com/feature/article/from-punch-cutters-to-number-crunchers</a></p><p>Human Alphabet <br><a href="https://www.slideshare.net/sotos1/human-alphabets-2/75">https://www.slideshare.net/sotos1/human-alphabets-2/75</a><br><a href="http://press.emcimprint.english.ucsb.edu/the-making-of-a-broadside-ballad/jeremy-chow-ballad-as-body">http://press.emcimprint.english.ucsb.edu/the-making-of-a-broadside-ballad/jeremy-chow-ballad-as-body</a></p><p>Research in expression emotions over a keyboard<a href="https://link.springer.com/article/10.1007/s11042-018-6753-3"><br>https://link.springer.com/article/10.1007/s11042-018-6753-3</a></p><h4>Appendix</h4><p>Thank you</p><p>– Dana Karwas, Director of the <a href="https://ccam.yale.edu/">Center for Collaborative Arts and Media </a>(CCAM) at Yale University and a Critic at the Yale School of Architecture <br>– Douglas Duhaime, Full Stack Developer at Yale University’s <a href="https://dhlab.yale.edu/">Digital Humanities Lab</a> for your advice and support. <br>– Nina Stössinger, Senior Typeface Designer at <a href="https://frerejones.com/">Frere-Jones Type</a> and Type Design teacher at the Yale School of Art</p><p><a href="https://www.yourtypeface.com/">Your Typeface</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7bcadcc8cd7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/live-letters/live-letters-strike-a-pose-get-a-letter-e7bcadcc8cd7">Live Letters: Strike a pose, get a letter</a> was originally published in <a href="https://medium.com/live-letters">Live Letters</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>