<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gideon Mann on Medium]]></title>
        <description><![CDATA[Stories by Gideon Mann on Medium]]></description>
        <link>https://medium.com/@gideonmann?source=rss-62074d7853e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*MwWKtj_zD2OgIKmymQvaUA.jpeg</url>
            <title>Stories by Gideon Mann on Medium</title>
            <link>https://medium.com/@gideonmann?source=rss-62074d7853e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:55:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gideonmann/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The 3D Ecosystem]]></title>
            <link>https://medium.com/@gideonmann/the-3d-ecosystem-704b777c376e?source=rss-62074d7853e------2</link>
            <guid isPermaLink="false">https://medium.com/p/704b777c376e</guid>
            <category><![CDATA[metaverse]]></category>
            <category><![CDATA[3d-printing]]></category>
            <dc:creator><![CDATA[Gideon Mann]]></dc:creator>
            <pubDate>Wed, 10 Nov 2021 00:28:22 GMT</pubDate>
            <atom:updated>2021-11-10T00:28:22.404Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Facebook’s change in name to “Meta” has focused a lot of attention on the idea of a “metaverse”, but co-existing with the metaverse is a set of technologies around “3D”. These 3D technologies contribute to the metaverse but also have a distinct set of purposes and use.</blockquote><h4>3D in the metaverse</h4><p>One of the key aspects of the metaverse is that it is a “digital twin” of the real world, existing in a 3D world, or perhaps VR or AR. No matter where, these 3D worlds require a set of technology needed for these experiences. Chief among these technologies is computer graphics (e.g. ray-tracing) which is required to render 3D objects. For AR, computer vision is also critical in order to be register and track surfaces in the real-world, since in order to understand how to embed objects in the world, it is critical to understand the underlying geometries of the world. This is true for complex 3D AR realizations, but it’s also true for a simple snap-chat filter that puts a dog tongue on someone’s face: you have to be able to find and track the face in real-time. Holograms are another technology that also relies on 3D objects. <a href="https://lookingglassfactory.com/">Looking glass factory</a> is one example of a desktop holographic interface (the entry level device is $300). It’s just another end-point for 3D images and render them on a holographic display on your desk. Google’s telepresence project <a href="https://www.engadget.com/google-project-starline-191228699.html">Starline</a> is another example of how light field technology can realize immersive 3D spaces.</p><p>Another part of the metaverse concept is that objects can be transferred between the spaces. What is interesting about 3D is that they already are able to commute between existing VR and AR spaces. To give a small, but illustrative example: a child can create a 3D design in <a href="https://www.tinkercad.com/">TinkerCAD</a> (a 3D platform for children), render it as a <a href="https://blog.tinkercad.com/2013/08/15/tinkercraft">Minecraft object</a>, import it into that platform and play inside of that design. In more sophisticated platforms, this happens in larger ways. The Unreal rending engine enables creation of incredibly realistic 3D <a href="https://www.unrealengine.com/en-US/digital-humans">human models</a> that can then be imported into the Epic platform. Sites like <a href="http://kitbash3d.com/">kitbash3d.com</a> stock inventories of 3d objects that can then be used across movies and games. Of course, these objects are lack dynamics, they’re static constructs (3D geometries, color etc) where you’d expect more from a realized metaverse object (e.g. physical dynamics — like the movement of an opening door).</p><h4>3D Object Creation is getting easier, but it’s still pretty hard!</h4><p>One of the main roadblocks to compelling digital environments is the cost to create 3D content. <strong>Authoring</strong> a 3D model from scratch requires artistic talent, computer expertise (to manipulate the in-progress piece of art), and compute power, though the availability of powerful graphics cards and the decreasing cost of compute has dramatically decreased the cost of building 3D models. As one example, <a href="https://www.blender.org/">Blender</a> (for 3D illustration) is a massively popular open-source 3D authoring tool which can run on a mid-price desktop. As a consequence, what used to take a render farm as recently as 20 years ago, can now be done locally and freely. Unity’s acquisition of Weta (the digital effects studio) speaks to still how central authoring 3D content is. Here is Unity SVP Marc Whitten explaining the rationale behind the acquisition (<a href="https://techcrunch.com/2021/11/09/unity-is-buying-peter-jacksons-weta-digital-for-over-1-6b/">link</a>):</p><blockquote>“Think back 10 years ago,” he told me. “Think about 2D, and just how many photos were being created. There was probably a lot! But project forward 10 years… and it’s just extraordinarily greater now. I think we’re in the same place for 3D.”</blockquote><blockquote>“The one difference, though” he notes, “is that 10 years ago, I … kind of thought I could take a photo. I can sort of do something with 2D. But that’s actually almost not-true — because every time I click the shutter button on my iPhone, it’s probably running 5 million lines of super advanced code to turn me into [a photographer]. But in 3D, I actually personally can’t model something today. What we need to do over the next 10 years is take that same approach; to take this incredibly deep technology and make sure that it works effortlessly.”</blockquote><p>The other side of this is 3D objects created from an engineering (and less artistic) point of view. Premium CAD (computer-aided design) software can thousands of dollar per year, but there are emerging low-cost or free 3D authoring tools enabled by the emergence of powerful and inexpensive graphics cards and cheap cloud compute has lowered the barrier to authoring 3D objects. Mid-tier authoring tool, like Fusion 360 from AutoCAD, costs $500/yr. But there is now the emergence of freemium software. OnShape, as an example allows you to create content as long as it is public. SketchUp allows you to create within some limitations. These low-cost tools are getting rapidly better, and its getting easier to create new 3D content.</p><p>Of course, there is another way to create 3D objects: <strong>scanning</strong> which builds a 3D representation of the world by photographs. Long range LiDAR, as used in autonomous vehicles, is still expensive, but the iPhone 12 has a short range LiDAR cameras and as a consequence it is very inexpensive to generate fairly faithful 3d scans, as suggested in this <a href="https://twitter.com/albn/status/1322614235843731457?s=20">tweet</a>. Even ignoring LiDAR, depth imaging software, enabled by machine learning, has improved greatly so that you can estimate a near-real time depth map from an embedded piece of software (e.g the <a href="https://www.luxonis.com/">Luxonis</a> open-source AI drivern camera package). Already these scanners are being used, e.g. in real estate where 3D scans are being generated for homes that enable someone to virtually walk through a home (<a href="https://matterport.com/industries/real-estate">Matterport</a> as one example).</p><h3>An Introvert’s 3D</h3><p>The metaverse presupposes a social world that is the center for 3D artifacts, but 3D technology is also used to create objects in the physical world, outside of a social context.</p><p><a href="https://medium.com/u/54699731aa0c">Venkatesh Rao</a> (@vgr) has been writing and talking about the emerging REALIST stack for engineering, which describes a set of technologies/disciplines from tensor programming to lithium batteries that form a new stack for building machines. The “A” in the stack is “Additive” for additive manufacturing technologies (e.g. 3D printing FFM).</p><p>Additive manufacturing technologies are powerful — fast to deliver custom prototypes and the technology changes the way design happens, as they enable cheap and fast realization of 3D parts. The innovation in the space has been amazing. The <a href="https://reprap.org/wiki/RepRap">RepRap</a> printer platform is an open hardware platform for building 3D printers. 3d printer parts are now cheap and ubiquitous (e.g. stepper motors). There is wild experimentation with various filament and extruder types like metal, ceramic, wood, and chocolate. Even “slicers” (e.g. the PrusaSlicer) that convert a 3D representation into “gcode” (the physical movement of an extruder feed or CNC cutter) have been massively developed. The ease of use of the REALIST stack has gone down so much, that it is no surprise that former artists are picking up the space (e.g. <a href="https://www.instagram.com/odd_jayy/?hl=en">odd_jayy</a>) and able to produce novel pieces. In fact, it feels like the skill set needed now to do interesting work in robotics isn’t engineering, but art &amp; design.</p><p>It’s not just additive technology though, it’s also CNC machines, laser cutters. Once you have a 3D model, you can realize or instantiate the objects this in virtual worlds (VR, AR, hologram) and physical worlds (FFM, cnc, laser cutters).</p><p>What may be surprising is that in order to build anything interesting, you still need to go beyond the 3D printer paradigm. Typically, you need some small hardware pieces (e.g. a <a href="https://www.thingiverse.com/thing:555242">flexible thread</a>), active components (motors, servos), and compute (Arduino, rasperry pi). At that point, you can build an incredible variety of small robotic applications, like a 3D printer itself or a <a href="https://www.youbionic.com/">bionic arm</a>. It feels like 3D printing hasn’t reached its final form.</p><p>As a side point, usually VR and AR are thought of as ends in of themself, but the ability to use AR to make a visual product that is consumed in another way is another (asocial) angle. While Filmmakers have used computer graphics for a long time, film makers can now utilize augmented reality environments to film (e.g.<a href="https://www.youtube.com/watch?v=gUnxzVOs3rk"> ILM’s Mandalorian stage</a>). This has the potential to radically decrease filming costs — see <a href="http://lighttwist.io/">lighttwist.io</a> as a company working on bringing this technology to the masses. Here the AR environment is being used not as a place to interact socially, but as a tool to create another 2D visual object.</p><h4>Asocial replication has some missing pieces</h4><p>Social (e.g. metaverse) is also not the only way that 3D objects can be distributed. The acquisition of <a href="https://sketchfab.com/">Sketchfab</a>, a 3D content viewing and sharing repository, by Epic Games is one example of the importance of distribution of 3D models. Beyond that, are repositories of 3D images like <a href="https://www.thingiverse.com/">Thingiverse</a> and <a href="https://grabcad.com/">grabcad</a> which host millions of CAD designs for free.</p><p>These repositories are great because there was nothing like them before, but from a utility perspective, they are bare bones. It’s impossible to find anything — you can only search by the keywords authors assign. It’s not even clear what search would mean for 3d shapes. The selection of content is also random, according to what amateurs have chosen to scan and upload. On top of this, it is incredibly challenging to take existing 3D objects and repurpose them, especially for CAD applications. It should be straight-forward to take an existing 3D cad drawing of a hinge, embed it into a 3D image and allow for physical dynamics of a swinging door, but there is no straight-forward way to do this no.</p><p>It feels, in a very real sense, like what the web was like in the late 1990s — full of amazing content but predominantly inaccessible and incomplete. A social metaverse is an imagined future for 3D from Facebook. I’m curious to see what the future for 3D from Google would look like.</p><h3>Altogether</h3><p>Altogether, 3D is a significant emerging technology theme that is starting to have significant commercial impact. Hard to imagine it won’t have less in the coming years. Already technology like computer graphics (for rendering), computer vision (for registering existing 3D surfaces), additive manufacturing (3D printers, slicers), LiDAR (for 3D scanning), lightfield (for holograms) and tools like for authoring (TinkerCAD, Blender) and browsing (SketchFab, thingiverse, grabcad) have made a connected ecosystem to support 3D objects.</p><p>When there is technology for searching among 3D models beyond SEO keywords and for adaptation of existing models, we’ll have reached yet another enormous set of opportunities for 3D that will lift all of the pieces of the ecosystem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=704b777c376e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Book Review : “Practicing” by Glenn Kurtz]]></title>
            <link>https://medium.com/@gideonmann/book-review-practicing-by-glenn-kurtz-3965df23b6cd?source=rss-62074d7853e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3965df23b6cd</guid>
            <dc:creator><![CDATA[Gideon Mann]]></dc:creator>
            <pubDate>Wed, 27 Oct 2021 00:45:55 GMT</pubDate>
            <atom:updated>2021-10-27T00:45:55.236Z</atom:updated>
            <content:encoded><![CDATA[<h3>Book Review : “Practicing” by Glenn Kurtz</h3><p>Kurtz writes “Practicing” as a reflection on the part of his life he spent devoted to learning classical guitar. He was an immensely talented child, playing guitar seriously from an early age, going to the Boston Conservatory for college and immersing himself in study and practice. Then, after briefly trying to make it as a working musician, he finds himself lost, having forgotten why he plays music. He puts the guitar in a closet in his early twenties. Now, after a number of careers and many years of living, he is writing the book as he is opening the closet once again and trying to take the guitar out.</p><p>The emotional center of the book is Kurtz’s time in college, where he reached the pinnacle of his ambitions for himself. He measures himself constantly against “Segovia”, quietly racing towards this virtuoso as a goalpost in the distance. His goal isn’t fame, exactly. It is mastery, somehow, and transcendence over an instrument he bitterly complains isn’t as complete as the piano or the violin. Transcendence over himself and his limitations. He feels a constant dissatisfaction with himself, with what he is able to do on his instrument, and burns against the constant feeling that his work is never enough. He is trying to pursue a perfection that he believes he can achieve if he practices enough, that he believes is even possible.</p><p>At the end of the book, the guitar is out again and Kurtz is trying to practice and to find another meaning in this work. He struggles with the realization that he will never regain the peak of his performance and despairs at having lost a decade to not practicing, to self-abnegation. He despairs at having given up in his desire to be more.</p><p>But … what would have been enough for him? Was there ever a level of mastery of success that would have felt like enough? This question is left hanging as is the question of where he should go now as he sits in a place where he judges his work a failure.</p><p>Most books are written by the winners. This isn’t one of them. The book grapples with the feeling of immense and cosmic failure even with something as simple as a block of wood and six strings. Perhaps with this, Kurtz is showing so beautifully what practice really feels like.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3965df23b6cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is SuperRare Forever?]]></title>
            <link>https://medium.com/@gideonmann/is-superrare-forever-9d633c3453c7?source=rss-62074d7853e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d633c3453c7</guid>
            <dc:creator><![CDATA[Gideon Mann]]></dc:creator>
            <pubDate>Sat, 13 Mar 2021 06:24:03 GMT</pubDate>
            <atom:updated>2021-03-13T06:24:03.037Z</atom:updated>
            <content:encoded><![CDATA[<p>A number of years ago, my wife and I went to the Armory Show — a venue for the exhibition and sale of modern art. Aside from a few pieces from Rafael Lozano-Hemmer, which were lovely, there were almost no pieces of work that had technology. Almost all of the modern art could have been created using materials that were decades old. In what sense was this art modern? It was confusing to me, until I heard a conversation between a prospective buyer and an exhibitor around a particular piece, where they were discussing how to store the piece and how to preserve it.</p><p>One of the surprising things about technology is how short a lifespan it has. Machines aren’t built to last. The most long-lived have a lifespan measured in decades. It is remarkable when we see cars from the 40’s or 30’s. Mechanical parts get worn down from use. Metal rusts. The only way to preserve machines are not to use them. This shorten lifespan is a problem even for software, that has no physical form to decay because in the case of software, the machinery needed to run the software also has to be maintained. Computer programs from the 80s can barely be run anymore, because the hardware needed to run them doesn’t exist anymore.</p><p>In the context of purchasing art, one reason why technology isn’t front-and-center in modern art may simply be the inherent complexities (or futilities) of preserving it. Ancient Greek statues are still around, and still magnificent, thousands of years later. It’s unlikely that a mechanical and computer piece will last that long.</p><p>So, what then of NFT art?</p><p>On the one hand, they are digital and their provenance tracking and the display has to be preserved just as with anything digital. On the other hand, as digital pieces, they can exist indefinitely. Not only is the ownership unambiguous and the authentication immediate, but unlike a painting, NFTs never change. They will never get damaged. They will never discolor with age. They are almost the inverse of the Buddhist sand mandala. As long as our human capacity for computation exists, they can be owned and exchanged. Perhaps, this is part of the reason that valuations are so high for NFTs . Unlike all other collectibles, buyers see a good that will last forever,</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d633c3453c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[José Clemente Orozco Farías]]></title>
            <link>https://medium.com/@gideonmann/jos%C3%A9-clemente-orozco-far%C3%ADas-5354eda9a628?source=rss-62074d7853e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5354eda9a628</guid>
            <dc:creator><![CDATA[Gideon Mann]]></dc:creator>
            <pubDate>Sat, 23 Jan 2021 06:40:43 GMT</pubDate>
            <atom:updated>2021-01-23T06:40:43.830Z</atom:updated>
            <content:encoded><![CDATA[<p>“Clemente,” as we called him, was an almost mythical figure at camp. Giggling, almost madly, he was always halfway out of frame and not in focus. He hated to be the center of attention. His style was New Hampshire hermit. He drove a Rent-a-Wreck with a trunk of battered antiques, scavenged iron parts, demented treasures. He was like a gnome who plays tricks at night: swapping out the eggs in the carton for clementines or switching a daisy in a pitcher with a carrot — sweetly reminding the world of the absurd possibility of joy.</p><p>Always mischievous, never malevolent, so kind to anyone lucky enough to wander through his cluttered studio. He was endlessly patient, helping kids through tie-dye after tie-dye, the rubber bands, the dead swirl. It was barely art, but I’m sure he thought that the soggy, blurry, truly lazy tee-shirts were a gateway drug, a way to nurture the happiness and creativity and mischief he found in art. The last time I saw him, he showed me the handmade books he was making, an evolution of the lithographs he lovingly carved.</p><p>Today, I heard with shock that he died this past January. It’s hard to believe that someone made of magic could ever die. As he did for so many others, he let me enter into his world of possibility, at a moment I really needed it, and I miss him terribly.</p><p>Rest in peace.</p><p><a href="http://gda.com/detalle-de-la-noticia/?article=4271964">http://gda.com/detalle-de-la-noticia/?article=4271964</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5354eda9a628" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI in Synthetic Digital Spaces]]></title>
            <link>https://medium.com/@gideonmann/ai-in-synthetic-digital-spaces-5f10c49c6e73?source=rss-62074d7853e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f10c49c6e73</guid>
            <dc:creator><![CDATA[Gideon Mann]]></dc:creator>
            <pubDate>Sun, 19 Apr 2020 03:21:52 GMT</pubDate>
            <atom:updated>2020-04-19T03:21:52.434Z</atom:updated>
            <content:encoded><![CDATA[<h3>AI for Synthetic Digital Spaces</h3><p>Since the pandemic, I’ve been playing a lot of Minecraft with my kids. It’s nice to have a place to hang out with them and build digital legos together. Also interesting to think about what a shared digital space might look like.</p><p>This article <a href="https://www.washingtonpost.com/video-games/2020/04/17/fortnite-metaverse-new-internet/">https://www.washingtonpost.com/video-games/2020/04/17/fortnite-metaverse-new-internet/</a> highlights the appeal of digital shared spaces especially during the pandemic. The highlights these challenges with synthetic spaces: (1) distributed systems able to handle thousands of people in the same area, (2) rendering and graphics engines, (3) identity that can cross walled garden platforms, (4) the ways in which players can engage with the platform to create new aspects. Clearly more work to do on pushing through these technologies.</p><p>With respect to AI, there are also some things that might be worthwile in shared digital spaces.</p><ul><li><strong>Alignment between digital and physical experience</strong>. As one example, you could image the need for computer vision “Meat puppet” like-products where a camera transfers physical movement to a digital avatar. With these systems, the user doesn’t have to direct his digital avatar to wink, he just winks and its transmitted. Another example is haptic force feedback from the virtual environment to the client. Being able to feel in these dgital worlds would enable a deeper immersive experience, and a richer experience.</li><li><strong>Cheap creation of seemingly real environments</strong>. Synthetic generative models allow for the creation of rich digital spaces, with 3D geometry and even evolving stories (a la AI Dungeon 2). Presumably this could reduce the cost of creating immersive and compelling synthetic worlds</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f10c49c6e73" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open Role — Architect: Data Manufacturing]]></title>
            <link>https://medium.com/@gideonmann/open-role-architect-data-manufacturing-a6280b0688b9?source=rss-62074d7853e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6280b0688b9</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Gideon Mann]]></dc:creator>
            <pubDate>Tue, 05 Nov 2019 19:23:42 GMT</pubDate>
            <atom:updated>2019-11-05T19:25:16.782Z</atom:updated>
            <content:encoded><![CDATA[<h3>Open Role — Architect: Data Manufacturing</h3><p>I’ve slowly come to realize how central the manufacturing of data is to Bloomberg’s mission of providing decision makers with a critical edge. Certainly, when I started here in 2014, I vaguely understood that data — like fundamental company data or breaking news — had some impact on the company and our clients. I was also dimly aware that there was some process in which we found sources of relevant data and made them accessible to our clients. But, it took a long time before I fully realized the complexity and scope of this process. From where I now sit, the impact of this manufacturing process, which converts raw incoming materials into something more easily digestible, is enormous.</p><p>One aspect of this manufacturing process is the <em>conversion</em> of data from an unstructured or semi-structured form into a structured form. As I described in <a href="https://drive.google.com/file/d/183r_3X3yzDhEWM43e2pOoS_w3kUxJbaR/view">my EMNLP 2018 keynote</a>, the extraction of data from text and tables (which <a href="https://medium.com/@gideonmann/de-rendering-latex-tables-267f7d2ac54b">I love dearly</a>) is a complex scientific challenge in its own right. However, deploying machine learning systems requires configuration per task, and, as the breadth of domains in which we operate is large, this configuration cost is non-trivial. Beyond that, in many cases, existing state-of-the-art methods do not reach the quality bar required in the financial domain. These challenging domains require human intervention to remediate and to perform quality assurance.</p><p>However, even when the incoming data is structured, there can still be significant challenges. For example, incoming structures must be mapped to internal data schemas. Often, incoming structured data can be noisy and may occasionally change in format, therefore requiring monitoring and remediation. Normalization of incoming structured data also sometimes requires the application of complex accounting rules (e.g., recalculating reported income with respect to GAAP standards). Often, human intervention is also needed to reconfigure past streams and move them to a new data schema, or to clean past data streams that haven’t historically been addressed sufficiently.</p><p>This process is distinguished from a mere collection of data pipelines. Data manufacturing is a different sort of work. Unlike a data pipeline, data manufacturing requires a repeated human and machine process of configuring and maintaining data ingestion flows. This central process of human quality control is a central aspect that distinguishes this data manufacturing process from other automated data processing pipelines. As such, Bloomberg’s <a href="https://www.bloomberg.com/company/global-data/">Global Data</a> team operates an enormous data manufacturing operation.</p><p>Therefore, it is with great pleasure that I’m happy to announce that my Data Science team in the Office of the CTO at Bloomberg has <a href="https://careers.bloomberg.com/job/detail/78953">an opening for a very senior data science architect</a> to work directly in this area. My hope is that, together, we will push the state of the art in understanding what a data manufacturing operation should look like and bring increased transparency to the global capital markets by delivering cleaner data — even faster — to our clients.</p><p><em>If you’re interested, please reach out to Jen Carberry jcarberry7@bloomberg.net.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6280b0688b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[De-rendering LaTex Tables]]></title>
            <link>https://medium.com/@gideonmann/de-rendering-latex-tables-267f7d2ac54b?source=rss-62074d7853e------2</link>
            <guid isPermaLink="false">https://medium.com/p/267f7d2ac54b</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Gideon Mann]]></dc:creator>
            <pubDate>Mon, 23 Sep 2019 21:44:55 GMT</pubDate>
            <atom:updated>2019-09-24T00:14:05.630Z</atom:updated>
            <content:encoded><![CDATA[<h3>De-rendering LaTeX Tables</h3><p>At Bloomberg, one key challenge our Global Data team faces is the normalization of incoming data. Their goal is to transform raw incoming data into clean, normalized data that can be used to power financial analytics for our users. For example, publicly-traded companies’ quarterly and annual reports, which are filed with the SEC, include data like quarterly net income and loss. This information is extracted and placed into a database where customers can graph, screen and test against this ground truth.</p><p>One of the most complicated settings for this normalization occurs when the data is textual. Given textual data, there has been significant work in NLP — in information extraction in particular– to normalize semantic content from ambiguously-worded text. One example of this is extracting CEO succession relationships from free text. However, in some cases, the incoming data isn’t in a clean format like XML where the structure can be transparently read. Often, it comes in PDFs or images where even extracting the exact words and the page structure is not straight-forward.</p><p>PDFs, in particular, are of tremendous business interest as they are the medium for many regulatory filings, especially outside of the United States. Unlike other problems in machine learning, where the goal is to recover the original human intent from some observations, the problem here is subtle and different. Instead of a human process, there is a computational process which takes some of the markup language and renders it as an image. For example, scientific documents are written using the <a href="https://www.latex-project.org/">LaTeX typesetting system</a>, which is first interpreted by a LaTeX compiler and then rendered as a PDF. Typically, the original LaTeX commands are lost during this transformation. However, this LaTeX is crucial for understanding the document contents, as it contains content about the structural makeup of the document.</p><p>Over the past few years, there has been a line of work dedicated to the problem of “de-rendering.” Interestingly, <a href="https://www.overleaf.com/learn/latex/Articles/LaTeX_is_More_Powerful_than_you_Think_-_Computing_the_Fibonacci_Numbers_and_Turing_Completeness">LaTeX is Turing complete</a>. So, in a sense, this de-rendering is almost a form of program synthesis — figuring out how to derive a program that meets a set of behaviors.</p><p>One of the seminal pieces of work looking at this problem was “<a href="https://arxiv.org/pdf/1609.04938.pdf">Image-to-Markup Generation using Coarse-to-Fine Attention</a>” [Deng et al. 2017]. In that early work, the authors demonstrated how to de-render LaTeX equations from their images. The proposed model combines a convolutional neural network (CNN) that integrates information across the image pane, and then has a series of Bi-LSTMs that scan across rows of the compressed feature representation to emit LaTeX tokens. This model has been surprisingly effective and has been widely replicated — for example, a team of Weights &amp; Biases students reimplemented it (here is an online demo <a href="http://www.wand.com/articles/image-to-latex">www.wand.com/articles/image-to-latex</a>). There’s also a video that shows the attention during processing (<a href="http://lstm.seas.harvard.edu/latex/">lstm.seas.harvard.edu/latex/</a>)</p><p>At <a href="https://icdar2019.org/">ICDAR 2019</a> in Sydney this week, I’ll be presenting recent work led by <a href="https://github.com/da03">Yuntian Deng</a> of <a href="https://nlp.seas.harvard.edu/">Harvard NLP</a>, which was supported by David Rosenberg and I during his summer internship with the Data Science team in the Office of the CTO at Bloomberg. It scales up this work to the harder problem of LaTeX table de-rendering. LaTeX tables are typically more complex than equations. First off, they’re larger and frequently contain multiple sub-equations within them. Second, the rendering is non-local — one command (e.g., a row formatted as centered instead of right-aligned) can alter multiple parts of the image. As such, these represent an interesting increase in complexity over equations. In our paper, “<a href="https://drive.google.com/open?id=12gxFrRKUMPULTR28dvzhoNS0IOqVVPFg">Challenges in end-to-end neural scientific table recognition,</a>”we provide a corpus of around a half a million LaTeX tables extracted from arXiv, and demonstrate experimental results on applying the equation model to this table corpus.</p><p>It is heartening to see that the equation model performs adequately, suggesting that it is a reasonable baseline for future work. But, there is a significant gap in the accuracy of the de-rendered LaTeX tables and ample room for future work. We hope this corpus will provide a mechanism for others to develop more sophisticated algorithms.</p><p>Moreover, what is most surprising is simply how difficult this problem is. Certainly, there is an ample amount of training data, as we have access to many hundreds of thousands of tables, but approaches which have been very successful for, say, image recognition, do not work as well in these documents.</p><p>Again, we’re merely talking about recovering the structure of tables. The larger problem of recovering the structure of whole PDF documents that have been rendered from LaTeX still looms unsolved. We hope that, with more attention to this problem, the community will develop methods that scale to whole documents with high precision, enabling increased transparency and liquidity across the capital markets.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=267f7d2ac54b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Augmented Programming]]></title>
            <link>https://medium.com/@gideonmann/augmented-programming-d916c4d14688?source=rss-62074d7853e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d916c4d14688</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Gideon Mann]]></dc:creator>
            <pubDate>Thu, 22 Aug 2019 15:29:57 GMT</pubDate>
            <atom:updated>2019-08-22T16:57:55.475Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the past decade, deep learning research has led to significant advances in perceptual tasks, such as object detection, face recognition, and speech recognition. In each of these use cases, raw real-world inputs have to be mapped into a normalized representation. Deep learning has also started to lead to significant advances in natural language processing, for example, where contextual embeddings enable multi-task transfer learning. A newer, emerging field for machine learning is in its application within the production, deployment, and maintenance of software. Research is increasingly identifying areas where machine learning sits alongside traditional software engineering tools and the human workflow to deliver on the promise of augmented programming.</p><p>There are many areas where machine learning is having a gradual impact on the software production process. One holy grail in this effort is program synthesis, where a program can be generated directly from natural language, explanations, or examples. Here, one of the most interesting avenues is the automatic creation of GUIs from sketches. Program-assisted code completion has also gotten significantly better, inspired by work related to speech recognition and machine translation — in particular, a technology called “language modeling.” With increasing research, bug fixing and identification using machine learning by means of static analysis of code have also steadily improved, often by means of learning what bugs have been fixed in the past. However, with a few notable exceptions, automated program synthesis still remains far from matching or augmenting human capabilities.</p><p>However, there are a few areas where machine learning techniques seem poised to vastly improve how software is produced. One comes in the compilation process. With modern compilers, humans specify the output, but the compiler specifies instruction ordering — and sometimes extensive rewriting. In turn, this can significantly impact the code’s performance characteristics. The typical optimizations that compilers use to search for equivalent and more efficient programs are hand-designed. But, over the past half-decade, researchers have identified areas, specifically in loop-free code, where MCMC sampling can find optimizations that significantly outperform what traditional compilers generate. Other work in auto-tuning compiler optimization suggests a whole host of areas where heuristic approaches in compiler design will be improved by machine learning mechanisms.</p><p>Decompilation has also seen a recent flowering of research work. Decompilation, or reverse engineering, is a critical step in many security and malware detection processes, where low-level assembly code is translated into a high-level programming language. The key step is to imbue the resulting high-level language with the semantics of the low-level program. Techniques borrowed from machine translation have started to be applied usefully to infer variable names for code where the original variable names have been scrambled.</p><p>While testing is always a chore for developers, it is crucial to exposing faults and vulnerabilities in software applications. One approach to software testing is fuzz testing, where a wide variety of inputs are sent to a program with the hope of exposing behavior that results in a crash or other aberrant program behavior. In grey-box fuzzing, the fuzzer executes inputs and then instruments and monitors internal program execution in order to select new examples to test. Here, experiments at Bloomberg and elsewhere have demonstrated how both reinforcement learning and deep learning can be applied effectively to build models of program behavior and build ever-more efficient fuzzers.</p><p>These three applications (compiler optimization, decompilation, and testing) are all areas developers typically ignore and generally struggle with. Even though machine learning models understand very little about how the program actually executes, they may still be able to extend the reach of what humans can do with software by leveraging extensive computation. While machine learning is still very far from being able to assume full responsibility for software production, it will increasingly be a tool that is integral to its efficient deployment.</p><p><em>This article was originally published on </em><a href="https://opendatascience.com/what-is-augmented-programming/"><em>opendatascience.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d916c4d14688" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My new favorite Bug]]></title>
            <link>https://medium.com/@gideonmann/my-new-favorite-bug-de03960793b7?source=rss-62074d7853e------2</link>
            <guid isPermaLink="false">https://medium.com/p/de03960793b7</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[debugging]]></category>
            <dc:creator><![CDATA[Gideon Mann]]></dc:creator>
            <pubDate>Thu, 18 Apr 2019 12:46:24 GMT</pubDate>
            <atom:updated>2019-04-18T23:58:40.193Z</atom:updated>
            <content:encoded><![CDATA[<p>I confess. I love debugging, maybe even more than I love programming. It is amazingly frustrating, a tiny piece of discovery and building, and often a way for me to reliably get into a flow state. Bizarre to say, but it is often very relaxing. Ok, maybe I’m exaggerating, but I think the joys of debugging are underrated.</p><p>Recently, I found a new favorite bug.</p><p>For the past little while, I’ve been working on <a href="https://medium.com/@gideonmann/hot-fuzzing-7ea2fcf54736">fuzzing</a>. The problem of fuzzing is to find vulnerabilities in a binary by testing a wide variety of inputs. The goal is to pick inputs to test carefully so as to exercise as much different program behavior as possible. Typically, in “grey-box fuzzers”, the fuzzer is able to see program trace information (the code path executed by an particular input) to help guide its choice in what inputs to choose.</p><p>In my set-up, I use AFL (American Fuzzy Lop) and a large test set of binaries (<a href="https://github.com/trailofbits/cb-multios">https://github.com/trailofbits/cb-multios</a>). One of the key pieces of AFL is how it instruments a binary to collect trace information. Here is the description from the <a href="http://lcamtuf.coredump.cx/afl/technical_details.txt">technical report</a>:</p><pre>The instrumentation injected into compiled programs captures branch (edge) coverage, along with coarse branch-taken hit counts. The code injected at branch points is essentially equivalent to:<br><br>  cur_location = &lt;COMPILE_TIME_RANDOM&gt;;<br>  shared_mem[cur_location ^ prev_location]++; <br>  prev_location = cur_location &gt;&gt; 1;</pre><p>Essentially, at each point AFL wants to capture trace information, at compile time it generates a new random number to represent this code location. Ok, first, this is really cool because in machine learning you typically don’t get to fool around with the compile-time process. Second, it’s elegant. The randomization of the constants is important here because the shared memory space is of a constant size and this is a clean randomized algorithm to minimize collisions in the stored information. It’s also nice that the instrumentation step can be achieved without intention from the original programmer.</p><p>There are a few ways AFL achieves this injection. (1) It can augment the binary with assembly directly around each branch point by inserting assembly code. (2) It can use an emulator (QEMU) to capture trace information for binaries you can’t recompile. (3) It can do things in a “principled” way by adding a pass in the LLVM and adding assembly language. One of the advantages of method (3), going through LLVM, is that LLVM can produce a very clean intermediate representation (called bitcode). On top of that, LLVM feels a little cleaner. Instead of just dumping a <a href="https://github.com/mirrorer/afl/blob/master/afl-as.h">pre-canned code chunk</a> like this into the assembly code:</p><pre>&quot;\n&quot;<br>&quot;/* --- AFL TRAMPOLINE (64-BIT) --- */\n&quot;<br>&quot;\n&quot;<br>&quot;.align 4\n&quot;<br>&quot;\n&quot;<br>&quot;leaq -(128+24)(%%rsp), %%rsp\n&quot;<br>&quot;movq %%rdx,  0(%%rsp)\n&quot;<br>&quot;movq %%rcx,  8(%%rsp)\n&quot;<br>&quot;movq %%rax, 16(%%rsp)\n&quot;<br>&quot;movq $0x%08x, %%rcx\n&quot;<br>&quot;call __afl_maybe_log\n&quot;<br>&quot;movq 16(%%rsp), %%rax\n&quot;<br>&quot;movq  8(%%rsp), %%rcx\n&quot;<br>&quot;movq  0(%%rsp), %%rdx\n&quot;<br>&quot;leaq (128+24)(%%rsp), %%rsp\n&quot;<br>&quot;\n&quot;<br>&quot;/* --- END --- */\n&quot;<br>&quot;\n&quot;;</pre><p>the AFL llvm pass has a fairly high level language that you can describe the same process. In the <a href="https://github.com/mirrorer/afl/blob/master/llvm_mode/afl-llvm-pass.so.cc">llvm code</a>:</p><pre>/* Make up cur_loc */       <br>unsigned int cur_loc = AFL_R(MAP_SIZE);       <br>ConstantInt *CurLoc = ConstantInt::get(Int32Ty, cur_loc);       </pre><pre>/* Load prev_loc */       <br>LoadInst *PrevLoc = IRB.CreateLoad(AFLPrevLoc);      <br>PrevLoc-&gt;setMetadata(M.getMDKindID(&quot;nosanitize&quot;), MDNode::get(C, None));      <br>Value *PrevLocCasted = IRB.CreateZExt(PrevLoc, IRB.getInt32Ty());</pre><pre>....</pre><p>Much prettier! Here you’re creating assembly instructions that are then injected explicitly by the compiler. Maybe it’s just that I know very little assembly, but this feels like what a real programmer would be doing.</p><p>Now the binaries in cb-multios come with their own Makefile and they have a fairly large and involved process to build them. In order to build them with this instrumentation, you need to reconfigure the CC compiler to go through the AFL wrapper around the compiler. There’s one more wrinkle, the cb-multios binaries are typically built with clang, and don’t produce LLVM output, and for various reasons, I want to get the LLVM bitcode output from the compiled binaries. Ok, so in order to be able to extract LLVM from the resulting binaries, you need to change the compiler to <a href="https://github.com/travitch/whole-program-llvm">wllvm</a>,/ from the description:</p><pre>WLLVM provides python-based compiler wrappers that work in two steps. The wrappers first invoke the compiler as normal. Then, for each object file, they call a bitcode compiler to produce LLVM bitcode. The wrappers also store the location of the generated bitcode file in a dedicated section of the object file. When object files are linked together, the contents of the dedicated sections are concatenated (so we don’t lose the locations of any of the constituent bitcode files). After the build completes, one can use a WLLVM utility to read the contents of the dedicated section and link all of the bitcode into a single whole-program bitcode file. This utility works for both executable and native libraries.</pre><p>Ok! This is great, now to compile the binaries, all that is needed is to create a chain, where the overarching compiler is set to afl which then calls wllvm to create all of the binaries. After this point, I could extract the bitcode and find the compile time constants.</p><p>So then the bug.</p><p>When I ran the executables, the traces from the program reported didn’t match what I could pick out of the LLVM bitcode. That is, the compile-time constants I could find (manually) from the disassembled bitcode were completely different than the compile-time constants that were evidenced from the program traces. Now, a lot of crazy stuff happens in the compilation phase, but this shouldn’t happen.</p><p>My first thought was to blame <a href="https://twitter.com/siddkaramcheti">Sidd</a> who had written a lot of the glue code, but quickly realized, as I should have known from the start, his code was blameless.</p><p>Eventually, I had the idea to compare the actual object file (assembly code) and the LLVM bitcode. I could see where the bitcode was storing the cur_loc and where the assembly code was storing the cur_loc, and the numbers didn’t match! My mental model is that compilation proceeds by code -&gt; bitcode (LLVM) -&gt; assembly, so there should be no differences. Weird. So, then, I took the exact numbers as generated from one run of the regular AFL instrumentation and inserted them exactly into the binary. Progress! I was able to then extract the trace information I expected from the binary. I was also able to compare two modes of compilation: (A) the files generated from the random number generator and (B) the files compiled with the exact sequence generated by the random number generator. While the llvm passes were identical, the object files were different, but only in the constants they had.</p><p>At this point, I’m feeling pretty great about myself, because now I had a route to finishing this task — I could just generate a ton of “random” numbers, like ten thousand, and stick them into the binary and be done with this bug. But it was nagging at me. Then I realized what had happened. A careful reader might also have figured out at this point as well.</p><p>The wllvm compiler does two passes (as noted above): one to generate the real object files and one to generate the llvm files. Since compilation is deterministic, the bitcode should be what then generated into the object. However, in our case, compilation isn’t deterministic, and at each run, a new set of compile time constants are generated. I added a call to srand() and set the random seed and the bug disappeared.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de03960793b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I have a few questions about Hogwarts]]></title>
            <link>https://medium.com/@gideonmann/i-have-a-few-questions-about-hogwarts-89601364bba4?source=rss-62074d7853e------2</link>
            <guid isPermaLink="false">https://medium.com/p/89601364bba4</guid>
            <category><![CDATA[harry-potter]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Gideon Mann]]></dc:creator>
            <pubDate>Sun, 13 Jan 2019 23:51:54 GMT</pubDate>
            <atom:updated>2019-01-13T23:51:54.836Z</atom:updated>
            <content:encoded><![CDATA[<p>My kids and girlfriend and I were watching “Harry Potter and the Sorcerer’s Stone” over the weekend, and now I have a few questions.</p><p>According to my son, Muggles are not allowed at Hogwarts. This is rock-solid from a sociological perspective — wouldn’t want the secret that there are magicians to get out. But there are a number of troubling consequences from this key point about the world.</p><p>In general, Hogwarts seems like a fairly normal place, technologically speaking. Moving staircases aside, we can assume that their built environment is fairly standard, for example they have standard toilets and whatnot, because otherwise … gross. So who cleans the toilets? A magician cleans the toilets? That must be a horrible job, even with magic plungers. You go to wizard school for god who knows long and at the end this is what you’re doing. I don’t think even Hufflepuffs (which the kids insist would be my school) would want to do this.</p><p>But assuming there’s a fix to the number one and two problems, there are some other technology-related questions. Where does the train to Hogwarts come from? Who built it? Magicians must have built it since there are no Muggles at Hogwarts. Ok, sure, but how did they figure out how a train works? Did they just copy an existing train? Do they need a new train spell? Is a magician driving the train? Did he have to go to conductor school?</p><p>For that matter, do the kids learn anything at this school beyond magic stuff? Like… history? Or english? Or math? Or take arts — Hogwarts is truly beautiful. Who designed it? A magician? How did the spell know what they wanted? Who knew that the brown brick should be matched with red and that the wall sconces shouldn’t be too big. Someone had to go through years of art education to figure this out.</p><p>Without a broad education, it would be hard for them to reason about the world, and know the consequences of their actions which, as magicians, are enormous. I worry about super powerful beings with the ability to shape the world through their craftsmanship don’t have enough knowledge about the world to understand their consequences.</p><p>At the heart of it, I’ve always felt that computer programming is pretty much magic, and one of the enduring lessons about computer programming in industry is that most software requires more than just the programmer to be wonderful. Artists, UX designers, product managers, subject matter experts are crucial in designing the right thing. So a world populated with only magicians, seems to me, extremely limited.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89601364bba4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>