<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sam Hawes on Medium]]></title>
        <description><![CDATA[Stories by Sam Hawes on Medium]]></description>
        <link>https://medium.com/@samhawes?source=rss-c97e8c01416a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*aVRF8E-GWLomroegedvahg.jpeg</url>
            <title>Stories by Sam Hawes on Medium</title>
            <link>https://medium.com/@samhawes?source=rss-c97e8c01416a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 04:06:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@samhawes/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I’ve migrated to Substack]]></title>
            <link>https://samhawes.medium.com/ive-migrated-to-substack-cef223f808cc?source=rss-c97e8c01416a------2</link>
            <guid isPermaLink="false">https://medium.com/p/cef223f808cc</guid>
            <dc:creator><![CDATA[Sam Hawes]]></dc:creator>
            <pubDate>Thu, 11 May 2023 22:26:19 GMT</pubDate>
            <atom:updated>2023-05-11T22:26:19.096Z</atom:updated>
            <content:encoded><![CDATA[<p>Check it out here: <a href="https://samhawes.substack.com/">https://samhawes.substack.com/</a></p><p>Based on a few comments, it sounds like publishing on Medium was very outmoded, like having an @<a href="http://aol.com/">aol.com</a> email address. Plus Substack gives me the option to one day enable a subscription paywall, but for now everything is free.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cef223f808cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In 10 years, will OpenAI’s story look more like Google’s or Netscape’s?]]></title>
            <link>https://samhawes.medium.com/in-10-years-will-openais-story-look-more-like-google-s-or-netscape-s-2b018e51124c?source=rss-c97e8c01416a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b018e51124c</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sam Hawes]]></dc:creator>
            <pubDate>Wed, 26 Apr 2023 23:06:40 GMT</pubDate>
            <atom:updated>2023-04-27T19:49:19.926Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A0KOLWWd-6_jv0zXSEIdbA.png" /></figure><p>Let’s imagine a tech columnist in 2033 writing a retrospective summary of the early days of the AI revolution. Here I’m assuming journalism still exists as a vocation and hasn’t been taken over by AI, both in the sense of “people are still paid to write articles” and “all of humanity hasn’t been enslaved to toil in the silicon mines for their robot overlords”. OpenAI, having already built the fastest growing product in the history of the world (ChatGPT, if you haven’t heard), will be a featured part of AI’s early years. OpenAI’s path over the next decade is useful to contemplate, not because it’s fun to try to predict winners, but because what role it ultimately plays will be determined by how AI evolves as a technology, and how people want to use it:</p><p>1/ <strong>Technology overhead: Will the creation of new AI models always require massive amounts of computational power and expertise?</strong> Though it’s tempting to assume that costs always fall, <a href="https://en.wikipedia.org/wiki/Moore%27s_law">Moore’s Law</a> doesn’t exist for all use cases. Why hasn’t there been a serious challenger to Google Search in the last 20 years (at least until Bing’s recent ChatGPT integration)? Google’s entrenched position certainly plays a role, but it’s also really, really hard and expensive to create a product that indexes the entire internet. That’s as true today as it was 20 years ago. It may be that AI models with the breadth of ChatGPT–and the safeguards necessary for wide adoption–will forever be hard to build, so OpenAI may create a barrier to entry where their tech is so superior that it locks out smaller entrants.</p><p>2/ <strong>Star Trek vs Star Wars: What kind of robot friend are you looking for? </strong>Will the best user experience for AI be a platform that can do anything you want, or lots of separate helpers, each built for a different task? Science fiction, for better or worse, can be a useful way to test our assumptions of how we’d face a new technology. On Star Trek, the ship computer can answer any question, control any part of the ship, or act as a problem-solving partner when dealing with the life-threatening emergencies the crew of the Enterprise regularly find themselves in. On the other hand, the droids of Star Wars each have their assigned tasks: manual laborers, translators, cleaners, and whatever it is that R2-D2 does.</p><p>If adding more and more capabilities into a single platform subsequently makes the AI more useful–it can do many things at once, draw connections between different parts of your life, and so on–then maybe we’re headed for a Star Trek future. If a tailored experience is better, where each task is accomplished by an app designed for that function, maybe our future looks more like Star Wars, though hopefully <a href="https://www.youtube.com/watch?v=Z_OjTojCNm0">with less robot complaining</a>.</p><p>So keeping these two questions in mind, let’s see where this takes us. Is OpenAI the equivalent of Google circa 2000, in the early stages of developing a platform that changes computing and leads to near-universal adoption? Or is the best comparison Netscape in the 90’s, pioneering a new way for humans to interact with computers that others will ultimately execute–the spark that lights a computing revolution that blazes across the industry?</p><p><strong>OpenAI’s path to Google-like dominance</strong></p><p>ChatGPT is so unlike anything the world had seen that it left many veteran tech commentators <a href="https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html">deeply unsettled</a>. Like Ben Thompson, who <a href="https://stratechery.com/2023/chatgpt-learns-computing/">wrote</a> that “for reasons that are hard to explain, I feel like I have crossed the Rubicon” and “it was the most profound interaction I’ve ever had with a computer.” Though ChatGPT was built with the same technological underpinnings that have powered AI models for years, OpenAI’s insight was to focus the model on optimizing for human-like conversation. As a result, <a href="https://medium.com/@samhawes/did-this-come-out-of-nowhere-how-the-inner-workings-of-chatgpt-portend-an-ai-revolution-ab0287b3ed71">as I wrote last week</a>, OpenAI has seemingly (and perhaps inadvertently) discovered something new about human language.</p><p>Building this capability wasn’t easy, and the amount of effort that went into it is staggering. According to OpenAI, training Chat GPT-3 cost over $3M in computing resources. It ran the model on 285,000 processor cores and 10,000 graphics cards, equivalent to about 800 petaflops of processing power. And that is just the cost of the infrastructure overhead; OpenAI employs some of the very best, and most expensive, AI engineers in the world.</p><p>Of course, Google also employs lots of top-tier talent, and compute power isn’t an issue for them. But Google has been working on Bard for years, and its <a href="https://arstechnica.com/information-technology/2023/04/clash-of-the-ai-titans-chatgpt-vs-bard-in-a-showdown-of-wits-and-wisdom/">capabilities pale in comparison to GPT-4</a>. And OpenAI is continuing to outpace them by further pushing ChatGPT into a do-everything Star Trek-style platform. Its API is being integrated into <a href="https://gizmodo.com/chatgpt-gpt4-snapchat-instacart-duolingo-bing-ai-1850229116/slides/10">dozens of products</a>. <a href="https://mashable.com/article/expedia-kayak-openai-chatgpt-plugins">Plugins are coming</a> that would allow ChatGPT to overcome its immediate shortcomings, like the fact that it’s bad at performing calculations (no problem, just <a href="https://www.wolfram.com/wolfram-plugin-chatgpt/">integrate Wolfram Alpha</a>), can’t connect to the internet to get up-to-date info (<a href="https://openai.com/blog/chatgpt-plugins#browsing">here’s a plugin</a> that let’s it browse the web), and is so selfish it won’t even order food for you (<a href="https://www.instacart.com/company/updates/instacart-chatgpt/">time to give it access to your Instacart account</a>).</p><p>It’s not hard to get your head swimming as you consider the implications of ChatGPT with access to dozens, or maybe hundreds, of different parts of your life. The more an AI knows about you, the better it can do its job. Imagine a personal AI assistant that knows that your best friend’s birthday is coming up, so it coordinates with the assistants of the other people in your WhatsApp friend group, finds a time in Google Calendar when everyone is free, books a reservation through your OpenTable account, and arranges Ubers ahead of time for everyone.</p><p>Or, my favorite example: How could a common AI platform help a kid through the earliest part of their formal education all the way through high school and college? Imagine an AI tutor that has learned, over the years, what you’re good at, what you need help with, and how you like to learn. It would connect to your school’s student/teacher platform so it would understand the syllabi and track performance. It would integrate with third-party visualization tools to help students better understand historical timelines, how circular motion works, or the structure of organic molecules. Since your AI tutor would imbibe everything you produced, for years, it could help you draw connections between subjects, refer to past notes, and learn from previous successes and failures. It would help you plan assignments, collaborate with your classmates, create problem sets from scratch, and yes, even help you with essays (I think that the current hand-wringing over ChatGPT doing students’ homework for them is distracting from the transformative and positive impact this tech could have in the classroom).</p><p>If the marginal utility of an AI keeps increasing as its knowledge about the world (and its user) increases–and we are able to overcome the obvious privacy and abuse concerns inherent in centralized power–then we are likely in only the earliest stages of OpenAI becoming the most powerful company in the world.</p><p><strong>OpenAI as Netscape: A pioneer</strong></p><p>There’s an important insight in the rise of ChatGPT that doesn’t get sufficient air time: The way we’ve been interacting with computers for the past 30 years is clunky and lame, and asking questions in normal language, and getting back a response that’s equally easy to parse, is the real secret sauce here.</p><p>In the 90’s, Netscape (and its predecessor, Mozilla) introduced people to the Internet in a way that was easy to navigate and useful. What had previously been the domain of geeks using special programs to navigate Usenet newsgroups or FTP sites now had a friendly wrapper in the form of the graphical web browser. Soon Netscape rolled out even more enhancements, like Javascript, which moved websites from a static text-and-image experience to interactive experiences. There are obvious parallels to AI here: If you can make something easy for anyone to use that previously required impenetrable expertise (like, say, writing code or generating content), you are going to completely change our perceptions of what computers can do. And such a fundamental shift can’t be confined to the products of a single company. In addition to the web browser, many of the big changes in the history of human/computer interaction–the first GUI at Xerox Parc in the 70’s, touchscreen gestures in the first iPhone–had a single origin but were soon ubiquitous.</p><p>Fueled by venture capital and no small amount of hype, the AI ubiquity is coming, in the form of direct ChatGPT competitors, but also niche AI players. <a href="https://info.ada.cx/forrester/customer-service-chatbot/?dv=c&amp;mt=e&amp;creative=649167877893&amp;utm_campaign=bol_always-on_na_prospect_ent-mm_mofu_search-mql_kw_brand&amp;utm_medium=paid-search&amp;utm_source=google&amp;utm_term=ada.cx&amp;utm_content=&amp;hsa_tgt=kwd-1242609560568&amp;hsa_acc=5466190196&amp;hsa_cam=19726273653&amp;hsa_src=g&amp;hsa_grp=145016389566&amp;hsa_net=adwords&amp;hsa_ver=3&amp;hsa_ad=649167877893&amp;hsa_mt=e&amp;hsa_kw=ada.cx&amp;gad=1&amp;gclid=CjwKCAjwl6OiBhA2EiwAuUwWZShu5dRUuJKwHlSA9OWuBXTFxtNf61ZNoc2VHw1hs7QPN4ql49R_JBoCKtoQAvD_BwE">Ada</a> creates AI customer service agents. <a href="https://www.grammarly.com/blog/how-grammarly-uses-ai/">Grammarly</a> writes and edits text for you. <a href="https://insilico.com/">Insilico</a> can model proteins and design drugs. <a href="https://github.com/features/copilot">Github copilot</a> writes code. Bloomberg has developed a <a href="https://www.bloomberg.com/company/press/bloomberggpt-50-billion-parameter-llm-tuned-finance/">language model for navigating its financial data</a>. And on and on.</p><p>Some of these use OpenAI’s API, but others don’t. After all, how many AI products truly need the collective intelligence of the entire internet–ChatGPT’s selling point–to be useful? Take <a href="https://www.expediagroup.com/investors/news-and-events/financial-releases/news/news-details/2023/Chatgpt-Wrote-This-Press-Release--No-It-Didnt-But-It-Can-Now-Assist-With-Travel-Planning-In-The-Expedia-App/default.aspx">Expedia’s recent integration of ChatGPT</a> into its product, where the chatbot can help you plan a vacation, identify points of interest, etc. All that’s truly required is a baseline understanding of human language, a whole bunch of information about vacation destinations, and integration with Expedia’s booking system. There’s no need for the chatbot to, say, write an episode of Seinfeld in the form of a Shakespeare play, or any of the millions of other things ChatGPT can do which would be irrelevant to travel planning. As a proof-of-concept, and a PR move, it makes sense for Expedia to work with OpenAI. But does it make sense in the long run?</p><p>For more targeted AI use cases, it turns out the technology overhead problem isn’t as insurmountable when you aren’t trying to do everything. <a href="https://www.databricks.com/">Databricks</a> created an LLM called Dolly, built on open source datasets and trained on only 3 billion parameters (compared to GPT-3’s 175 billion). Because of its smaller scale, Dolly was fully trained in three hours on a single machine. According to Databricks, Dolly has all the necessary foundational capabilities for anyone to build their AI product on top of it.</p><p><strong>It takes a village (of robots)</strong></p><p>Predicting the future of AI is harder than normal prognostication in the tech world. AI creates simultaneous technological uncertainty (we don’t know how good AI is going to get, or how fast) and cultural uncertainty (we don’t know in what ways AI is going to change society). But imagining these possible futures helps inform choices we make today.</p><p>The two questions we started with–the tech overhead problem, and the Star Trek vs Star Wars use of AI–are intrinsically linked. If users push for a centralized AI experience, because indeed a single platform becomes more powerful than the sum of its parts, then the technological complexity becomes so overwhelming–not to mention the aggregator power of the platform owner–that OpenAI stands a good chance of growing immensely powerful, the Google of the next 10 years.</p><p>Personally, I fall more on the Netscape side of the argument. I think we’re witnessing the dawn of a new age of computing–we can call it the fifth age, after mainframes, PC’s, the internet, and mobile. The places in our society and economy that AI will have an impact are impossible to measure, but will certainly be too numerous for a single firm to tackle. Also, and this is maybe wishful thinking, I think AI is too important and transformative for the technology to be concentrated in a single firm. The potential for abuse, or lack of oversight, is too great. OpenAI has opened the door, but here’s hoping we all go through it together.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b018e51124c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Did this come out of nowhere? How the inner workings of ChatGPT portend an AI revolution]]></title>
            <link>https://samhawes.medium.com/did-this-come-out-of-nowhere-how-the-inner-workings-of-chatgpt-portend-an-ai-revolution-ab0287b3ed71?source=rss-c97e8c01416a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab0287b3ed71</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Sam Hawes]]></dc:creator>
            <pubDate>Mon, 17 Apr 2023 17:18:21 GMT</pubDate>
            <atom:updated>2023-04-17T17:18:21.273Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Did this come out of nowhere? How the inner workings of ChatGPT portend an AI revolution</strong></p><p>Pretty much everything I’ve read about ChatGPT falls into three categories:</p><ol><li>“This is transformative and intimidating and might bring about the end of the world.”</li><li>“Haha look at all these dumb things that ChatGPT got wrong.”</li><li>“Click here for FIFTY AMAZING THINGS you can ask ChatGPT to do RIGHT NOW”</li></ol><p>One thing I <em>haven’t</em> seen is a good answer to the question, “How on earth did this all happen so quickly?” Products referred to as “AI” have been around for a while. I’ve had Siri on my iPhone for years, though she probably feels neglected since I only ever talk to her when I’m forced to because I’m driving. Ditto Alexa via the Echo Dot I got for $20 five years ago, whose main job is to turn the string lights over my porch off and on, and also randomly butt into conversations when she erroneously hears her name spoken. I’m pretty sure if either of these assistants could exhibit the sort of conversational ability that ChatGPT has demonstrated, we would have noticed. It appears that almost no one, even informed people in the AI community, thought we had gotten to the technological point where AI at this level was possible. But here we are.</p><p>For the last five years I worked on the product side of Machine Learning and AI models at Facebook, which were governed by many of the same principles, and employed similar techniques, as ChatGPT. So I have some experience, and since I haven’t found a good primer on how we got to this place with AI, I decided to write one myself. What follows is a slightly technical, but hopefully not too wonky, explanation of what is special about ChatGPT, what OpenAI discovered that was new, and what the implications are.</p><p><strong>In a surprise to pretty much everyone, you can encapsulate the essence of human language using “only” 175 billion parameters.</strong></p><p>You’ve probably heard that ChatGPT was created from the internet. In a sense, ChatGPT is us, as it is trained against millions of blog posts, wikipedia entries, news articles, and everything else that we put online every day. ChatGPT could never have been built without harnessing this raw material. The internet was a precondition for its construction, as it required a digital record of human creation against which to build a matrix of language.</p><p>What ChatGPT is <em>not</em> is a simple copy of everything it has scanned, like a pickier version of Google that plucks out a sentence from this source or a paragraph from that one and stitches together a response. ChatGPT is a language model first and foremost, which means the engineers at OpenAI had to create a colossal map of relationships between words by ingesting billions of examples. Imagine a giant 3D space with names and ideas and snippets of language laid out as far as you can see in every direction, organized in such a way that terms that relate to each other are physically close to one another and have strong connective tendrils running between them. So “dog” might be close to “furry”, “cat”, “companion”, and so on. The relationship space is so sophisticated, trained on such a high volume of content, that it develops the ability to ascertain context. For instance, while “dog” may be associated with “lab”, so would “chemical” or “research”, and the ability to understand this context–i.e. determine in what way you’re using “lab”–is one of the surprising abilities of this type of neural network.</p><p>GPT-3 features 175 billion connections like this (OpenAI hasn’t told us how many connections GPT-4 has). Something surprising has emerged out of this cleverly constructed model, even though its design is fundamentally similar to neural networks that computer scientists have been building for years. It is able to approximate real conversation to such a degree that even the most jaded users think there’s something ineffable about it, that we’ve stumbled upon something new in a way we haven’t seen for decades in the technology world, as innovation is usually a much less exciting series of incremental improvements. As Stephen Wolfram writes in his <a href="https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/">long discussion of ChatGPT</a>: “The basic answer, I think, is that language is at a fundamental level somehow simpler than it seems. And this means that ChatGPT — even with its ultimately straightforward neural net structure — is successfully able to ‘capture the essence’ of human language and the thinking behind it. And moreover, in its training, ChatGPT has somehow ‘implicitly discovered’ whatever regularities in language (and thinking) make this possible.”</p><p>This may be an important scientific discovery! Wolfram adds, “it’s suggesting that we can expect there to be major new ‘laws of language’ — and effectively ‘laws of thought’ — out there to discover.” But the takeaway here isn’t just that language is maybe less complex or more logical than we’ve all assumed. It’s that <em>human-like reasoning and creativity are easier to emulate than we believed.</em> (Note that I said “emulate” — unless there emerges extraordinary evidence to the contrary, I remain solidly in the camp of believing that these are machines approximating how humans think, not thinking themselves.) As we’ve seen with image and audio generation (with realistic video sure to come), you can do a lot more than just construct useful text with this kind of model.</p><p><strong>Training a machine learning model against the amorphous goal of “be a useful chatbot and give good answers” is surprisingly effective</strong>.</p><p>Traditionally, what you train a machine learning model to do (its so-called “optimization function”) is pretty unambiguous. I worked on the models that govern the Facebook Feed–that is, what posts you see and in what order you see them. There were models to, for instance, predict the likelihood that a user might comment on a story, or watch more than the first few seconds of a video. “Did this user comment” is a binary problem–either they did or didn’t–so you can know definitively whether the model achieved its goal. Lots of models are like this: you’re trying to predict the weather, or the stock market, and the results may be uncertain ahead of time but are easy to measure once they happen.</p><p>What OpenAI has done is quite different: they’ve simply tried to optimize ChatGPT for “good” answers, regardless of topic or prompt type or anything else, with all the built-in ambiguity and subjectivity contained in the word “good”. Once you hit the basic requirements of a cogent response (e.g. the chatbot didn’t fundamentally misunderstand the question), what makes one response better than another? Is it providing more detail, or just enough to make the point? Does suggesting a course of action or offering an opinion improve it, or just insult people that a robot is lecturing them? Is it better to be really creative but get some facts wrong, or limit the answers to be as anodyne and fact-based as possible?</p><p>These questions are debatable, which is why there is an art to building AI like this, a fact that gets lost within all the engineering talk. Consider how OpenAI used actual humans to help train the model. They took ChatGPT through a series of training sessions, demanding more from the AI at every step. The first session was primarily human-led: given a number of prompts (similar to the sorts of things they expected users to ask ChatGPT), the human trainers (known as “labelers”) wrote their own answers–without any help from the AI–to serve as initial examples to start the training. The second phase required ChatGPT to answer a prompt multiple times, and the labelers would rate the responses from worst to best. Finally, they built a “reward model”, which was based on the labelers’ work and would score responses on its own, to train ChatGPT at far higher volumes and speed than could be achieved by even an army of human labelers (which they didn’t have). <em>They built a separate AI to train the main AI</em>. This allowed them to scale! The whole thing boggles the mind.</p><p><strong>The “transformer” architecture (“transformer” being the “T” in GPT) turns out to be, well, transformative.</strong></p><p>A common criticism of the current wave of large language models is that they are fundamentally “fancy autocomplete”, and are just looking at what is statistically the next word in the sequence. I think this criticism is too reductive. It reminds me of the old joke: “How do you make a statue of an elephant? Easy, just get the biggest granite block you can find and chip away everything that doesn’t look like an elephant.” I suppose that is technically how sculpting works, but it’s not a very useful description of what the artist does.</p><p>In older language models, characterizing the output as basically autocomplete was more or less a fair assessment, since the model would look at only the preceding word or two to predict the following one. But the transformer architecture, which was originally laid out in 2017 by researchers at Google, is far more sophisticated: the transformer takes into account ALL preceding words in the conversation, which includes everything the chatbot has written up to that point, and anything that you, the user, has written in the initial prompt or in follow-up questions. What the transformer does next is fairly technical and probably not worth going into in detail, but there are two aspects that are especially important. The first is the idea of “embeddings”, which is the process of converting words into numbers that can be sent through the neural network. These numbers contain far more than just coded representations of the words themselves, but key data around context and relative position. The second important idea is “attention”, which is a method for understanding that some words earlier in the conversation are more important than others. “Attention” was core to the theory proposed in the 2017 paper, which had the catchy title “Attention is All You Need.”</p><p>To select the next word in a response, the transformer model essentially executes three steps, and please bear in mind that these are carried out over and over again <strong>for every successive word that ChatGPT writes</strong>:</p><ol><li>Create a representation (an embedding) of how each word in this conversation relates to every other word</li><li>Send this embedding through the ENTIRE neural network. Even though most will be ignored, all 175B connections in the model are activated. You are perhaps getting a sense of the computing power we’re talking about here</li><li>Generate a set of candidate words to be the next one in the sentence, each with a probability associated with it–the higher the probability, the more likely the model thinks it is the “right” next word</li></ol><p>Step (3) is the source of the “fancy autocomplete” criticism, but even that step hides some subtlety, because ChatGPT doesn’t always choose the highest probability option. There is a setting called “temperature”, a sliding scale where a value of zero means it should always choose the word with the highest probability–the safest (and maybe blandest) option. But as temperature increases it allows the model more and more leeway to choose words that have lower probabilities, and what you get in return is some semblance (or perhaps illusion) of creativity. Look at these strange ideas it came up with! Crank it up too high, though, and the output becomes <em>too </em>weird to be useful. OpenAI decided that the best value to balance creativity against predictability was a temperature value of 0.8. Why 0.8 as opposed to some other number? There’s no scientific answer, it’s just that the human labelers who assessed the output liked the answers best at this setting. Bing Chat, which is built on ChatGPT, actually lets you adjust this by selecting “more creative” or “more precise” before you ask your question.</p><p><strong>Closing thoughts</strong></p><p>There’s a lot more we can talk about, like what applications beyond conversational chat this tech may enable, or my take on why this new era we appear to be entering feels so different from any other (tl;dr is that this is the first computing revolution to include both technological uncertainty [we don’t know what this tech is capable of] and social uncertainty [we don’t know how it is going to change how we live]). And if people seem to like this, I will write a follow-up. But I’ll leave you with a final thought. In writing about the challenges of creating a civil society, Kant famously wrote: “Out of the crooked timber of humanity no straight thing was ever made.” The AI we’ve built is made from all of us, crooked though we may be, and as such reflects all our imperfections. And our response to this new technology is certainly going to be imperfect, and a test of our society, as a swirl of accelerated scientific progress, business interests, utopian visions, abuse potential, and portents of doom create a vertiginous feeling that we’re on the cusp of something and have no idea how it’s going to play out. Understanding the technology is a good first step, but it only gets you so far.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab0287b3ed71" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>