The year is 1945. The US, freshly winning the war, now has significant capacity for science without as clear of a goal. They could continue constructive weapons of mass destruction, and they will, but the effort need not be as centralized or focused without soldiers actively in trenches. Not wanting this research capacity to diffuse into meaninglessness, the head the wartime scientific effort, Vannevar Bush, publishes his manifesto for American science in the Atlantic: As We May Think.

In this piece he aims to refocus scientists on research toward a ‘memex’ as he calls it. This device is in many ways similar to the modern desktop computer: it stores a record of a person’s intellectual ‘trails’ which can be shared and searched freely. The vision is that such a device could help scientists keep up with the fire-hydrant of research that was regularly published by the thousands of scientists that had been recently funded by the government.

Eighteen years later, Doug Englebart wrote a letter thanking Bush for the article’s contribution to his thoughts on augmenting human intellect. Here, Englebart reframes Bush’s vision as a framework for making humans more efficient in intellectual pursuits in general. To Englebart, this is mostly about symbol manipulation: being able to read, write, edit, and send files in a maximally convenient way. Englebart went on to invent the mouse, hyperlinks, and the 2D graphical interface based on these ideas.

This arc of history doesn’t stop with Englebart, however. In the 1990s, the internet enabled the efficient sharing discussed by Bush. A few years later, Google’s PageRank algorithm satisfied Bush’s idea of indexing ideas by association in order to effectively solve the problem of selection.

One curious grain of inspiration I discovered in my reading of Bush’s article had to do with trails. He imagined ‘trails’ of ideas, one leading to the next, and constantly branching down interesting nooks. Trails, in my view, are reasonably adjacent to one’s search history on the internet. But Bush goes a step forward, envisioning a “new profession of trail blazers those who find delight in the task of establishing useful trails through the enormous mass of the common record.”

So why don’t we simply share search histories with each other? Because the internet is messy and imperfect. No two idea journeys are the same. Even if they could be in theory, we rely on the browser’s back button because we constantly make mistakes; we can’t know if websites have the information we need before we open them.

Still, maybe Bush was onto something. How frustrating is it to pave new trails on the internet. Compare the experience of learning and researching on the internet versus scrolling Twitter or TikTok. How nice would it be if we had instant curation systems that pre-determined our journey on the internet based on our goals.

Even before Bush and Englebart, there was known to be value in better connecting the thoughts of individual with the knowledge bank of society. Ralph Waldo Emerson described how we are all fragments “as the hand is divided into fingers” of the greater intellect of our species. With new technology to better handle fundamental units of symbol manipulation — indexing, selection, writing, etc — what other tools might exist for augmenting human intellect. Artificial Intelligence will surely be important, as it accomplishes each of these tasks with accuracy and speed. But its specific insertion into our lives is yet to be determined by the trail blazers of history to be made.

How One Man Predicted the Internet Before Computers was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the center of Google’s main corporate campus stands an imposing T-Rex Fossil named Stan. The rumor is that Stan presides over the company as a constant reminder that the ‘dinosaur companies’ eventually go extinct. The Dinosaurs are fierce and powerful, but old. They are undone by their own stability, which breeds complacency amid perpetually changing times. Having amassed the power and sustainability of a dinosaur (and having lost its pioneering founders) Google now stands at the precipice: will it sink slowly to the beat of its own bureaucracy? Or will the fire of what was once called Silicon Valley spring innovation eternal?

For context, I interned at Google this summer, and I learned a lot about what it is and isn’t. For starters, Mountain View is not composed of mountains, and is instead a vast suburbia (with some charm).

What caught me most off-guard about the way the business is run is how much they truly respect their users. Each engineer, at least on my team, cared deeply about each and every user experience. And not only in the context of their perception or use of the product, but genuinely about the product improving the user’s state of being, of living. If Google does become a dinosaur, they will do so with dignity. They do not fear death; they have too much money for that. They only fear making something that is not useful, and ultimately deteriorating the brand and spirit of the company.

At this point, it is clear that an ads-based internet works. It allows almost half the world’s population access to high quality services at no cost. The problem for ads serving companies (Google, Meta, Amazon, Microsoft, etc.) is that they are really in the market for attention, and humans are fickle creatures. Worse, they’re running out of users to gain. On top of that, the startup cost for creating a product like Google or Facebook today is dramatically lower than it was 20 years ago. The result is ever increasing competition for a piece of a pie whose growth is slowing to a creep.

Fortunately for Google, they do innovate on many other fronts. They are among the strongest AI companies in the world and they have an up-and-coming cloud business that sits close to consumers. And then, they have their secret weapon: Google Chrome.

Browsers are extremely counterintuitive in terms of business-sense. They are never sold, run no ads, and are very complicated and expensive to produce. Not to mention that they require vigilant security and constant innovation to remain popular. The most simple way to explain the power of the browser is to think about how you get to the internet: you pay for Wi-fi, you buy a laptop, and then you download a browser. There are exactly three companies with control over the pixels on your screen, and the browser is the only one with the leverage of a pure-software product. All you have to do is convince people to install the browser, and you steal all of the default-setting advantages of everyone who came before you. And despite almost all computers shipping with either Safari or Microsoft Edge, 65% of people go out of their way to install Google Chrome.

Google knows how important Chrome is, and you’ve likely seen their ads everywhere for it. Because Apple and Microsoft can remove Google’s everything from their hardware products, but they will never take away people’s favorite browser, and that’s where they have all the power to turn everything into a google search. Of course, this position is just as defensible as advertising, with tightening competition, and a problem of high activation cost for acquiring ever-so-fickle users. To avoid descending into extinction, I believe Google has to leverage its browser marketshare with its AI and cloud strength.

There is a change that I have written extensively about regarding AI and its scaling laws. Essentially, AI and specifically large transformer models are becoming extremely good at repeating human-like patterns. This obviously has implications extending far beyond a conversation about Google or technology. But for them specifically, it is an opportunity to change the way that people interact with their computers.

No other company controls nearly as much frequently used on-device software, or has the AI capabilities of Google. Making chrome a deeply AI-rooted product could change the way we interact with computers, and secure chrome (and hence, Google) as a winner for many years to come.

There is another trend that places chrome in the perfect position to leverage google’s resources to become a deeply innovative and necessary product: the increasing complexity of browser activity.

Crypto people will fight me on this, but I think that the real ‘web3’ is about editing, not ownership. For context, web1 traditionally refers to a read-only internet (think wikipedia), and web2 traditionally refers to a read-write internet (think youtube). But what about editing and working on the internet, instead of just uploading and reading finished products. Google knows about this trend and largely started it with sheets, docs, and slides, which remove what once were individual software packages and integrate them into the internet, where they can be edited from anywhere, by anyone. But this trend has accelerated beyond just docs. Nowadays, UI is made on the web with Figma, code is written on Repl.it, and teachers do not just upload grades, but administer tests themselves on the internet. The pandemic work-from-home trend has merely accelerated this change; no place, besides maybe an office, is as collaborative as the internet.

For google to take advantage of these trends, it has to go meta (no pun intended). They must make Chrome a uniquely productive browser. They must leverage the world’s information that they have so tirelessly organized, and put it to work for users. There are many ways that this can be done. Autocomplete in Gmail and Docs is definitely a step in the right direction, but what’s a step in a marathon? I believe that chrome has to fully embrace the new class of AI models that can understand natural language and generate action based on this language.

Language is the perfect interface for chrome, because it is already based on language. The largest non-changing part of a chrome browser is already a text input field. But what if the search bar could be used for purposes besides navigation? What if it could interact with the page itself to do things that were smart and productive?

Large, transformer based models have already demonstrated the ability to generate text and images based on text input. Moreover, Google is the owner of some of the best models for these exact purposes. A website-agnostic, built in AI language-interface would give Chrome an edge above competition, but more than that, it would revolutionize the way we use the internet and do work in general. And maybe the technology is not quite ready to work on every website, but even launching known, predictable products on a few websites (docs, for example) would drastically reduce the amount of user re-education that would have to occur as AI matures across other domains and tools like this become more available and predictable.

In fact, I believe Google’s ability to minimally and slowly roll out such a feature is its absolute biggest advantage in brining this product to life. A simple icon and coloring distinction in the suggested results popup could indicate web actions, rather than search results. Embedding such information into the results tab isn’t necessarily a new change, either, with results currently being embedded on some searches. From the user standpoint, there is near zero buy-in for instant productivity improvements.

A potential issue on the horizon for this type of software is that — at least based on current constraints — it can likely not be as cheaply served as the rest of the internet. A single pass through a state of the art language model requires on the order of 100 billion floating point operations. This isn’t an impossible hurdle to deploying the software in general (Moore’s law has been working remarkably well, thankfully), but it may be prohibitive to deploying such a tool on in a free, ad-supported way.

I view this not as an obstacle for Google, but an opportunity to change their incentives. Google’s current customer is ultimately not users, advertisers pay the bills, not searchers. And while there is overlap in the needs of users and advertisers, prioritizing users would allow Google to lean more solidly into its foundational mission of being useful to everyday people. What began as a place to search for web pages has grown into a place to search for information. And advertisers often aren’t as good at providing information as they are at providing web pages. But justifying a paid/subscription service with across-the-board AI integration into everyday work would allow them to wean off advertiser incentives, even if just slightly.

Whenever I write anything, I constantly google. I google words, I verify things I remember, etc. I’d say that in only about 1 in 3 of those searches do I actually need to leave Google. I’m sure Google knows this, so why do they automatically send you to an entire web page when you press enter in the search bar when you only want a morsel of information? Because if you make 66% less queries, you are seeing 66% less ads. A user-centric model (paid, if need be) eliminates this problem, and generates more value for the user.

I imagine that someone who is less of an AI fanboy might be skeptical of the usefulness of a permanent browser-level presence. So here are 6 ‘action queries’ if you will that I believe are fully possible to implement today:

• “Write an outline for an essay on [insert prompt]” — Google Docs
• “Make a chart out of this information” — Google Sheets
• “Paste in the top column all of the imdb ratings for the top 100 movies, and put in the second row the release date of the movies, and then make a chart out of that information” — Google Sheets
• “Write an email to my coworkers telling them that I’m sick and can’t come in today” — Gmail
• “Design me some UI for a note taking app” — Figma
• “Book me the cheapest airline ticket to X possible” — Google Flights

In terms of implementing these actions, I imagine that initially, they would have to be done one at a time (possibly with contribution from app developers themselves). But another piece of Google’s unique competitive advantage in making such a product is access to web browser interactions. It could be conceived that this data (fully and carefully anonymized, of course) could eventually be used to train such a model to learn the patterns of web interaction, and conduct them from a text based prompt input.

There is also the work being done at Adept.AI to connect AI to APIs across the internet. Such a product integrated to not just the google homepage, but the browser itself, has the potential to radically change how work is done.

We live in a very exciting time in that the many now have access to what was only available to the few. One consequence of this is that the most resourceful of us are now sufficiently resourced to take on seemingly impossible challenges. This is what gives me confidence that, if Google doesn’t build out this infinitely more productive future, some other company just might. Regardless of who does it, I’m excited to watch it and be a part of it.

The Browser of the Future was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

A few days ago, OpenAI released what I find to be the most striking display of the creative power of AI: DALL-E 2. On the most basic level, DALLE-2 is a function that maps text to images with remarkable accuracy, producing high quality and vibrant output images. But how does the technology work, really? In this post, I will try to explain as deeply as possible while keeping as many people in the loop as possible.

As stated above, DALL-E 2 takes text as input, and produces images as its output. But this isn’t all done in one go; rather, DALL-E 2 represents the culmination of a variety of techniques that have been built on and improved bit by bit over the past few years. These techniques are stacked on each other like lego blocks in order to compose the full model. To understand DALL-E 2, we first have to understand what these pieces are and what they do. Only then can we dive into the inner workings and training processes for understanding the underlying components.

CLIP

Among the most important building blocks in the DALL-E 2 architecture is CLIP. CLIP stands for Contrastive Language-Image Pre-training, and it’s essential to DALL-E 2 because it functions as the main bridge between text and images. Broadly, CLIP represents the idea that language can be a vehicle for teaching computers how different images relate to one another. Formally, it prescribes a simple way to carry out this teaching.

The best way to understand CLIP is to first see the shortcomings of previous computer vision systems. Until CLIP, neural methods for computer vision involved aggregating large datasets of images, and then hand labeling them into a set of categories. Though today’s models are exceptional at this task, it is inherently limited by the need for pre-selected categories. For example, imagine taking a picture of the street and asking such a system to describe it; it could tell you how many cars and signs there were, but it couldn’t give you a feel for the scene overall. What’s more, anything without enough images to produce a category would simply not be classified by the model.

The insight that makes CLIP so powerful is the bright idea to train models to not just identify which category (of a pre-defined list of options) an image belongs to, but to identify the caption of each image from a list of random captions. This allows the model to use language to more precisely understand the difference between “a shiba inu” and “a shiba inu wearing a beret,” rather than having a human labeler dictate in advance whether or not these belong in the same category.

In accomplishing the ‘pre-training task’ described above, CLIP is able to create a vector space whose dimensions represent both features of images and features of language. This shared vector space functionally provides models with a image-text dictionary of sorts, allowing them to translate (at least semantically) between one and the other.

Still, this shared understanding does not represent all of DALL-E 2. Even with a way to translate English words into Spanish words, one would need to learn Spanish pronunciations and grammar before speaking. In the case of our model, CLIP enables us to take textual phrases and understand how they map onto images. But still, we need a way to generate those images in a way that’s true to our understanding of the text. Enter our second lego building block: the diffusion model.

Diffusion Models

Imagine you have a Rubik’s Cube that is perfectly solved, with each side having only blocks of a single color. Now, you randomly choose a side and twist it. Then another side, and another, and another, and so on until you consider the Rubik’s Cube ‘scrambled.’ How would you go about solving it? Say you forgot all the twists you performed, could there be a general way to solve the cube without that information? Well, you could just take it one step at a time: keep twisting whatever face gets you closer to having all the same color on the same side until its solved. More practically, you could train a neural network to go from a state of disorder to a state of less disorder.

This is the crux of how diffusion models are created. You take an image, randomly scramble its pixels until you have an image of pure noise, and train a model to reduce the noise by changing its pixels step by step until you’re back at the original image (or something that resembles it). This leaves with a model that can effectively generate information — an image — from randomness! And by showing the model new, random samples to start from, we can even get new images.

To infuse these creations with the semantic meaning of our input text, we ‘condition’ the diffusion model on the CLIP embeddings. This simply means that we pass the vectors from our joint CLIP space described above into to the diffusion model that calculates what pixels to change upon each ‘step’ of the generation process. This allows the model to base its changes on that information.

Of course, the technical details of the math and implementation are more complicated than I have presented here. But at it’s core, DALL-E 2 is the optimization and refinement of these two technologies.

It has been said that “art is the window to [wo]man’s soul.” For this reason, some people find generative images to be boring or uninteresting. To me though, generative art is almost more interesting: it is the window to the computer’s soul. In producing art, the model reveals its understandings, misunderstandings, and — most of all — its creativity.

DALL-E 2.0, Explained was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

On April 17th, OpenAI leader Sam Altman tweeted the following:

Sam Altman on Twitter: "we need a name for this new computer interface of telling computers what you want in natural language and them doing it. any ideas? / Twitter"

we need a name for this new computer interface of telling computers what you want in natural language and them doing it. any ideas?

He ultimately came up with “language interface,” at least for now. This is more than just a one-off thought by Altman, though. In recent weeks, we have seen many of the most respected people in the AI community take the leap of faith to start companies targeting the exact moment evoked by the term “language interface,” the moment where AI becomes useful by bridging the gap between cognition and computation.

The first giants to move on this trend were LinkedIn founder Reid Hoffman and DeepMind co-founder Mustafa Suleyman, who announced inflection.ai together earlier this year. Their website contains a colorful description of the company’s motivation:

We will soon have the ability to relay our thoughts and ideas to computers using the same natural, conversational language we use to communicate with people. Over time, these new language capabilities will revolutionize what it means to have a digital experience.

But wait a minute, aren’t they just making a search box with an enter button? Isn’t that just Google? It’s more than that. To understand why, we need to understand the deep historical relationship between the functionality and the usability of new technologies.

Early computers could run computer programs that they were given, but they were expensive and slow. Appropriately, the way we interacted with them were reflective of their shortcomings. Dale Fisk, an IBM employee in the 1970s, described how “each punched card described one instruction in the program and that each box could hold about two thousand cards.” According to him, this box was “pretty heavy” and “more than one was awkward for a guy (or gal).” However, he goes on to say that these physical limits of usability made sense in terms of functionality because “The punched card hopper on these brand-new computers could hold just about that amount at one time.”

Half a century later, the standard interface between ones desire to run a program and a computer is the keyboard, mouse, and flat screen. Consider how perfectly these devices serve the functionality of computers. At a high level, the mouse allows us to run whichever programs we please. But even more remarkable is how the size of the screen somewhat limits your ability to overwhelm the computer with programs, just as the box imposed a limit on early programmers. Though the mouse-screen duo is seemingly trivial in hindsight, this interface was non-obvious at the time, and according to Stanford Historian Henry Lowood, “It wasn’t something that was immediately successful as a technology.”

This is not a coincidence; throughout the history of computing, the tools we build to use a technology must match the functionality of that technology. And as the functionality of our computers change, our interfaces must change as well.

Changing of interfaces is not merely a formality of technological progress, either. Rather, they are the catalysts for punctuated change in technology’s interaction with society. To see this, think about any phone that existed before 2007, and how much time you spent using it. Compare this to how much time you spent on your iPhone last week; touchscreen changes everything. Arguably, the introduction of the touchscreen as a mobile interface has been the driving force of the societal change in the American 2010s. But I digress, the point is that when the capabilities of technology exceed that which we can accomplish with our current interface, when there is capability that is not usable, society is almost predestined to develop a new interface. Moreover, this process unlocks entirely new paradigms of user interaction with potentially sweeping effects.

Now that we’re caught up, what is Reid Hoffman really doing? It may help us to look at another one of his investments, a newly launched startup called Adept. Adept promises to make “useful general intelligence,” a play on the concept “artificial general intelligence.” The basic premise is an AI trained not just in language generation, but in accessing and using the APIs that already exist. As software engineers know, there’s an API for just about anything you can think of. In this way, Adept is essentially plugging the AI of today into the modern web, making it useful to people who want to get things done. The team at Adept has already recruited an impressive technical founding team, including the lead on Google’s Transformer paper, arguably the most influential AI research paper of the past 20 years.

So what do all these really smart people know? And why are they all itching to start and back AI interface companies? They know that AI is getting really, really good. And that multi-modal architectures like OpenAI’s DALL-E, that can go from text to images, are improving very rapidly. And lastly, they know that technological advances are cool, but interfaces change societies. If they get it right, what they’re building could be as ubiquitous and abiding as the mouse itself.

If you thought this was cool, follow me on Twitter! @DanielFein7

My grandpa recently discovered that I’ve been writing medium articles. After reading a few that I wrote about AI, he sent me a text asking for me to write an article explaining it to “seniors” like him. This is my best attempt at that.

What is Artificial Intelligence?

Artificial Intelligence (AI) is a phrase used by computer scientists to describe programs that can do things that require applying knowledge. But isn’t any computer program ‘applying’ the knowledge given to it in its own programming? For example, a program that counts to ten is just applying its knowledge of the ordering of the numbers. In practice, we reserve the term AI for programs that do things that require not just intelligence, but specifically human-like intelligence. This includes harder problems that are often too complicated for simple machines to solve.

What’s the Point?

I think an important thing to know about to fully understand the spirit of AI is something called the Turing Test (also known as the Imitation Game). Developed by Alan Turing in 1950, near the dawn of computer science, it tests a machine’s ability to exhibit human-like intelligence. The idea is that you put a human and a machine in separate rooms and digitally communicate with each one. If you are unable to reliably tell which room has the machine and which has the person, the machine can be said to have ‘passed’ the Turing Test. Much of Artificial Intelligence research since then has been aimed toward eventually developing machines that can ‘pass’ this test.

Still, there are a whole category of other tasks that do not involve language but that still require applying some knowledge at a human level. The entire field of AI is rooted in a deep fascination with the power of the human mind to do all sorts of things. We can identify all the things around us, imagine how they interact, and communicate our thoughts about all of these things in any number of ways. Tackling the whole problem of machines that can replicate humans along all of these different abilities is what computer scientists call Artificial General Intelligence (AGI). To eventually be able to build these AGI machines, we will first need to be able to solve all of these challenges on their own. That’s why most research today is specific to different ‘tasks.’

There are AI researchers studying a whole host of tasks that require intelligence. The biggest two categories, however, are Natural Language Processing (NLP) and Computer Vision. NLP focuses on building AI that can read, understand, and write human languages. This includes AI that can summarize articles, translate between languages, or automatically detect misinformation on facebook. Computer vision is focused on solving problems relating to digital images or videos. The problem that gets the most attention in this space is that of self-driving cars, which take pictures of the world to try to make decisions about how to drive.

How do we Make Artificially Intelligent Machines?

From the time when Turing proposed his test all the way until the 2010s, the way we made machines ‘intelligent’ was pretty similar to the way you might go about learning a new skill yourself: asking the experts to explain it. You may have heard of the famous 1997 chess matches of IBMs Deep Blue vs. world champion Garry Kasparov. IBM’s computer worked in part by rating each possible move based on rules provided by chess grandmasters. Similarly, Google Translate before 2016 worked by reading a bunch of translated text and coming up with patterns to feed the computer. Rule-based and statistical programs (like early Google Translate) were everywhere, and they were humanity’s best attempt at recreating intelligence. But what changed?

In about 2010, a way to create AI programs called Machine Learning suddenly started blowing all these programs out of the water. Machine learning is a group of computer programs that can use data to get smarter. These programs basically take in and study lots of information in order to learn patterns, and then take in new data and apply the patterns they learn in order to make predictions.

To make it a little more clear what machine learning really entails, imagine you own a lemonade stand. You sell a cup of lemonade for $3, and you sell three cups for $9. If someone gave this data to a machine learning algorithm, and then asked it how much you should sell two cups of lemonade for, it should say $6. This example isn’t too complex, but you can imagine how these problems can get pretty hard pretty fast. Just imagine making a computer predict how much to sell a house for (which Zillow actually did!). There are a lot more factors to consider now: location, square footage, age of the construction, etc. But good machine learning models, with enough data to study from, can learn the patterns and make the predictions.

Deep Learning

Specifically, the type of program that has powered most of the recent eruption in Machine Learning progress is known as Deep Learning. This type of program has gotten so good that it has basically taken over the fields of AI and Machine Learning completely (all three of these terms are almost used interchangeably because of this). The way deep learning works is fairly complex, and getting into the math behind it is enough to give anyone a major headache. But what makes Deep Learning so powerful is the ability to guess and check very well.

At a high level, Deep learning programs take in numbers, then add, subtract, and multiply them with each other, and with other numbers called parameters. They then do some more addition and multiplication to get a single number that represents some belief that the program has. For example, the Zillow house-pricing model might take in everything that is known about a house, and then add and multiply this information with parameters to receive a final number that represents how much the house is predicted to cost. And I know what you’re thinking… how could we get anything that means anything by just doing a bunch of random math? The secret comes in the parameters and how we choose them; that’s where the learning comes in.

We start with random numbers for our parameters and just start making fairly random guesses. We then do some math to find out how wrong we were, and how we could change our parameters to get closer to the right answer. Continuing with our house analogy, say our first guess tells us that a million dollar home is worth only $5. We would then go back and update some of the parameters we are multiplying to be bigger, to try to make our answer closer to what is correct. By repeating this process over and over again with a lot of examples, the parameters eventually get shockingly good at making accurate guesses.

Why is All of This Such a Big Deal?

I wrote all about why I believe this is such a big deal today in a different article. But basically, Deep Learning has gotten really, really good. Most of this is because we have access to faster computers and more data every year. We can now make algorithms that can do all sorts of things from writing books to driving cars. Even though it’s still not clear if we can eventually use these programs to pass the Turing Test or to achieve Artificial General Intelligence, its still really exciting to think about how many ways programs like these can help us live better lives and get more done.

There is also the idea that she who can replicate something must understand it. Scientists aren’t yet sure if the way that Deep Learning works is actually similar to the human brain, but it’s enticing to think that getting closer to reproducing human-like intelligence might also give us the answers to mysteries about how our own minds function. We may have to look outwards first to look inwards.

Clearly, I think this stuff is a big deal. That’s why I hope that I’ve helped my grandpa — and anyone else who decides to read this — understand AI and its many implications for the world.

Artificial Intelligence Explained for My Grandpa was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

Over the past year, four Stanford students have taken their own lives. Why? That’s the question everyone seems to be asking. Some like to blame lack of mental health resources, others speculate that it’s a consequence of pressure to “be perfect.” I think these things are definitely problems, but cannot themselves fully explain why there is a mental health crisis at Stanford, specifically.

Suicide never makes sense. But, if you have ever visited Stanford’s campus, you know that it seems to make even less sense here. The campus is beautifully maintained, the weather is warm, and it rained about 4 times total this school year. As you learn more about Stanford, it becomes harder and harder to understand how anyone could be sad here; there is free access to gyms and pools, students swim and party in fountains, they get dinners paid for by Venture Capitalists and Companies looking to hire them, and most of them have GPAs above the national average. In fact, if you spend some time on YouTube or TikTok, you can find videos of Stanford student’s who appear to lead terrifically fun and happy lives.

But, of course, not all Stanford student’s are so happy. In fact, many of them are just pretending. This is so common that the school has a name for it: Duck Syndrome. It’s called this because, as described by the New York Times, “a duck appears to glide calmly across the water, while beneath the surface it frantically, relentlessly paddles.” I believe that Duck Syndrome as a cause for unhappiness at Stanford is understated and under-explored.

Operating solely from definitions, the existence of Duck Syndrome itself guarantees that it is understated as a concept. What I mean by this is that if people are concealing their own sadnesses and insecurities, then they are also concealing how much they are affected by Duck Syndrome. It’s why every student knows about Duck Syndrome, but very few will point to it affecting them personally and instead choose to point at outward at more tangible, external factors as causing the problems at Stanford.

Only after living here do I feel like I can write about how big the gap between how Duck Syndrome is talked about and what it actually is. The biggest misunderstanding, in my book, is where Duck Syndrome starts. The popular perception is that it begins on campus, a byproduct of students taking difficult classes while also trying to enjoy their lives. But I think it starts before that; it starts in high schools, in parent groups, and in popular culture.

Stanford has the lowest acceptance rate of any school in the US. Think about what that means. It doesn’t mean that they accept the fewest, or that they have the highest qualifications. It simply means that there are significantly more students that want to go than who get to go. Given that most top colleges admit a similar amount of students at a similar rigor, this mostly just means that more students want to go to Stanford. What’s more, although the title for ‘best’ college is hotly debated, the title of ‘dream’ college has more or less settled on Stanford in recent years (and sometimes Harvard). People don’t just want to go to Stanford, they dream about it.

Because of this, I believe that Duck Syndrome isn’t just a response to other students, it’s a variation of survivor’s guilt. Every student here knows that there are thousands of others who would take their spot among the palm trees in a second. Maybe they would’ve appreciated it all more. Maybe they wouldn’t be working on a PSET in their room all day, and would instead be out on the field playing volleyball with the grad students.

I first noticed that this was the case when I hopped on a call with a Senior at Stanford when deciding whether I would enroll back in High School. I remember going on about how great everything seemed and being met with a quiet resistance. She wasn’t saying it was a sad place, but she wasn’t affirming my view of the school either. And now I finally understand: Stanford is not inherently sad, but it will never live up to the wild expectations people like to put on it. To make matters worse, the disagreement between these expectations and the reality — that Stanford student’s are busy and often overworked people — is harmful to the mental health of many.

This unique truth is often hidden not only by students suffering from Duck Syndrome, but also by the greater institution. After all, the image of a happy go-lucky school is partially what fuels Stanford’s prestige. This prestige in turn benefits alumni, who in turn endow the school very well. That’s why the grass is always green on campus, even in the worst of California droughts. At the beating heart of this scheme is the student: who is believed by the world to be, and ultimately is, incredibly fortunate. Only, they are too acutely aware of their luck to acknowledge that sometimes things don’t go their way.

The ultimate result of this is the worst feeling of all: isolation and loneliness. If everyone else is having fun — as you should be — how can you possibly ask to join? If you are the only one not feeling the force of the sunny winter days on your mood, if you are the only one not playing volleyball in the field, who is there for you to relate to? Of course, more people relate to you than you could know. They just don’t want you, let alone their parents, let alone the world, to find out.

“I have meticulously decided your pairings such that you and your partners skills complement each other,” said Dr. Jill Helms, my professor for a writing class I’m taking at Stanford. At first, I didn’t believe that such a thing was possible: even if she perfectly understood all of our skills, how could she have optimized every single pairing? But then I got to thinking about it, and came up with an extremely over-engineered, but interesting, solution.

If you could represent each person’s skillset, along with something they wish to get better at, as vectors in space, all we would need to is match people according to the similarity between these vectors. Believe it or not, this is actually not too hard using a few standard python packages. Let’s get started!

To clarify exactly what it is we’re doing here, we will write code that asks a group of students to each give their name, a sentence about their skillset, and a sentence about what they wish to learn. The code will then output the ideal matchings for the students such that students are matched with others who have the skills that they desire.

Organizing the Project

I chose to split the function into two files for readability and reusability (if I wanted to spin this off as a Flask API, for example). We’ll call the one that handles data retrieval main.py, and the one that does the work of making the pairings get_pairings.py . We’ll also need to install some packages using pip to get the code working. Specifically, we’ll be relying on three main packages:

• sentence-transformers offers state-of-the-art pre-trained transformer architectures for producing sentence embeddings. In layman's terms, this package gives us the function that will take sentences and turn them into vectors.
• sklearn is a broadly useful data analysis library. In this project, we’ll be using it to get the function that calculates the similarity between vectors.
• numpy is a very popular framework that helps python deal with long arrays of numbers. We’ll use it to make our pairings.

These packages can all be installed by running the line pip install sentence-transformers sklearn numpy in a terminal. Once we’ve done this, we’re ready to write some code!

Getting Data

Our first step is to ask each student for their name and sentences. For our purposes, it works well to store an array of names, an array of the ‘skillset’ sentences’ and an array of the ‘desired skillset’ sentences. This will be handled by the main.py file. To make your lives easier, I’ll paste in all the code here and explain after.

First, we import our code in from get_pairings.py, which I’ll explain more below. Then, we write a function called pair that uses python’s built in input function to get user input from the command line as strings. We ultimately loop through each student, building up the arrays described above (which we’ll call knows, wants, and names). We then pass these arrays into out imported run function in order to get back a list of lists of students’ names. Each inner list in this return value contains a pairing of students. We then take this list and print out each of the pairs. We now have the main structure of our program down, the only thing left to do is to actually figure out how to make the pairings!

Making Pairings

We need to write a function that takes in the three lists that we received above and turns them into optimized pairings. If we were doing this all from scratch, it would probably require hundreds upon hundreds of lines of code. Luckily, we’re able to build on other people’s code using the packages described above. I hope you’ll find that the result is relatively straightforward. The code for my get_pairings.py file is pasted here:

We have four functions to break down. Let’s start with the last (yet most important) one: the run function. It takes the the three arrays representing the students and returns a list of lists containing the best pairings. It does this by first loading in a model (specifically BERT, a state of the art transformer model created by researchers at Google). It then runs the list of sentences to analyze through the model to receive ‘embeddings,’ which are functionally vector representations of each of the sentences. It then computes a table containing the cosine similarity of each of these vectors to one another. This can be thought of as the angle between each of the vectors if you were to plot them out in space.

The other three functions each work on processing this table in order to turn these similarities into good pairings. First, the tables are ‘averaged’ by the avg_scores function. Say Billy’s skills match perfectly with Bob’s desired skills, but Billy’s desired skills are very different from Bob’s actual skills. This average function would replace the respective table entries for these similarities with the average of both. So, if the Billy-desires and Bob-skill matchup is 1, but the Bob-desires and Billy-skills similarity is 0, then each would become 0.5.

Next, the get_pairs function reads the table into pairs. It does this by finding the highest average similarity scores in the table, storing them into an array, and then setting all of the similarity scores for each person in that pairing equal to negative infinity, so that the members of that group aren’t put into another group. It does this until all the pairs that can be made are found. Finally, it takes the indices of these pairs and converts them into the names given by the names array, and returns this result to be handled by main.py, or whatever other client it is used by.

Result

In the end, the working function should be able to do this.

$ python3 main.py
Enter how many people you would like to pair off: 4
Enter the name of student 1: Daniel
Enter one or more sentences describing Daniel’s skillset: I like art.
Enter one or more sentences describing Daniel’s desired skillset: Something relating to water, boats, or swimming.
Enter the name of student 2: Gaby
Enter one or more sentences describing Gaby’s skillset: I know a lot about the ocean and am a diver.
Enter one or more sentences describing Gaby’s desired skillset: I would like to know about history.
Enter the name of student 3: Jade
Enter one or more sentences describing Jade’s skillset: Computer Science and math.
Enter one or more sentences describing Jade’s desired skillset: Economics is very cool to me.
Enter the name of student 4: Ali
Enter one or more sentences describing Ali’s skillset: I love studying the economy and monetary policy.
Enter one or more sentences describing Ali’s desired skillset: Quantitative fields like science and math.
Here are the pairings:
1: Jade and Ali
2: Daniel and Gaby

I love stuff like this because it shows just how easy it is to use state of the art AI for everyday projects and use cases. I hope that some people can learn from this and start bringing AI to more people everywhere.

Using AI to Make Better Teams was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

The world of AI seems to have split into two camps recently: the die-hards and the ditchers.

The Die-hards

The hopeful camp counts members like OpenAI’s chief scientist Ilya Sutskever and Tesla’s Sr. Director of AI Andre Karpathy. They believe that some of the abilities of the best language models today are indicative of even more powerful emergent properties. Both researchers have gone as far as to lightly suggest that these models may have some inklings of consciousness. For perhaps the most dystopian and compelling take on this, check out Karpathy’s short story from the perspective of a conscious language model (related: The AI Moment).

The Ditchers

But there is another, more grim view with some traction as well. The ditchers not only say that models aren’t conscious (they really went after Sutskever for his take), some also say that they never can be. There are a lot of reasons that people hold this view. These ideas are tricky, in my experience, because most of them make sense at first, but their conclusions are ultimately a matter of opinion.

Take the most famous such idea, the Chinese Room Experiment. It basically purports that if an english speaker went into a room with an English-Chinese dictionary, she could convince people writing to her from the outside that she spoke Chinese. The argument is that the English-speaker can simulate language understanding without actually having it. It follows that ML models can learn rules to simulate consciousness without actually being conscious. But at the end of the day, even if the person in the room doesn’t understand Chinese, the question of whether the room (containing both the person and the dictionary) can be said to understand Chinese is a matter of personal opinion.

To see this, imagine that the brains of everyone you know are inhabited by little aliens who only speak an alien language. They translate their thoughts to English before speaking to you. Would you doubt if everyone you knew understood English? Even after discovering the aliens? A matter of personal opinion, in my book.

Model’s Are Cheating

The first time I ever felt the urge to ditch Karpathy and friends to become a pessimist was when I read this thread by Alex Tamkin, a researcher at Stanford who mentored me for some time. In it, he explains briefly how it may be the case that GPT style models are simply memorizing training data very well, rather than learning generalizable patterns. At face value, this seems to blow the murmurs of consciousness out of the water. How could a model be consciousness if it’s really just remembering the right thing at the right time?

But then, of course, aren’t we all just remembering the right thing at the right time? Think about exactly how many of the phrases you say you have heard before. One paper cited by Tamkin implies that the frequency of arithmetic expressions in training data for a model is predictive of how accurately the model can work with the expressions. Another neural system this is true for is my little cousin Isabella, who was able to do consistently better on her multiplication tables after she practiced them and received feedback. Filmmaker Jim Jarmusch put it best:

“Nothing is original. Steal from anywhere that resonates with inspiration or fuels your imagination. Devour old films, new films, music, books, paintings, photographs, poems, dreams, random conversations, architecture, bridges, street signs, trees, clouds, bodies of water, light and shadows. Select only things to steal from that speak directly to your soul. If you do this, your work (and theft) will be authentic. Authenticity is invaluable; originality is non-existent.”

Just like the Chinese room experiment, whether or not you would ever call an advanced memory-resurfacer conscious seems to be a matter of personal opinion. Could this sifting and selecting of relevant text really be the origin of thought? In the case that it is, perhaps the ideas that fire through our heads are nothing more than the flipping of pages in the book of our own experiences and exposures.

Massive Language Models Are Cheating. Should We Care? was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Who made the first flag of the USA,” my little sister’s third grade substitute teacher asked the class. My sister confidently replied “Betsy Ross” (fairly fringe knowledge for a third grader if you ask me, but she was really on it). “Nope,” fired back the sub, “it was actually Susan B. Anthony.”

There are many ways to be a bad teacher. In my 13 years of public school and few years of college, I have had my fair share of clueless, neurotic, and — on more than one occasion — criminal teachers. But I have also been blessed with some utterly exceptional teachers. Despite teaching different things in different styles, they always come with that same magical feeling. That feeling that what you are being taught is not just important, but necessary for your development as a human. In fact, that is why they are teaching it to you with such care, not to help you attain some indeterminate status or career, but because they need to.

Some people think that the best teachers know the most, or that the best teachers prepare the most. Personally, I have had exceptional teachers on either side of both of these spectra. There are two qualities that I think are non-obvious and non-negotiable when it comes to being a great teacher: precision of language and accessibility.

1. Precision of Language

When teachers are specific and intentioned with every word they say, the student can sit and think about any subset of those words and continue to extract knowledge from them as their understanding increases. The best lectures are like good movies or books, you can play them back again and again and learn something new each time.

The main way teachers bungle this is by not understanding what they’re teaching. It’s not always their fault when this happens. Still, the unfortunate truth is that to teach something, you have to know at least most of the dynamics behind that thing. It’s easy for teachers to present pre-made slides on a topic they’re shaky on, but when it comes down to being asked questions, things sometimes start to fall apart. Even among the teachers who know what they’re talking about, some answer with more precision than others. The best teachers don’t just respond with accuracy. They respond at exactly the right level of difficultly as to not lose the class, and reveal only as much information as the question — and the lecture — requires.

Another, more sad, way teachers mess this up is simply by not having enough faith in their students and their curiosity. It’s one thing when teachers intentionally omit information to be more clear, but when students are missing context that was simply deemed “too challenging” or “too boring” by teachers, they lose a vital piece of the puzzle that will affect their ability to build on their knowledge and further explore if they are interested. If teachers are precise in their descriptions, students have a reason to be interested and overcome the difficulties that teachers seek to avoid by glossing over details.

2. Accessibility

The word accessibility is often applied to classes in the same way it’s applied to bathrooms: add a wooden ramp, call it accessible. To be truly accessible takes effort. I have learned most of what I can say about this from Stanford’s very accessible CS department. Accessible does not mean easy. To me, it means mainly two things: everything starts at first principles, and growth is valued at least as much as success. These things are beneficial not just for some subset of the students, but for everyone involved.

First Principles

Teaching from first principles is useful to students because it mimics more closely how students have to think in real life to solve real problems. Oftentimes classes focus on understanding how to use existing frameworks, without speaking about how those frameworks were produced in the first place. For example, when I learned about the constitution in 8th grade, I remember being taught about each article and the exact functionality provisioned to each part of government. This turned out to be much less useful than a later discussion of the constitution I had when I was older about the constitutional conventions themselves, where I learned how the founding fathers thought about government and how they reached compromises.

Perhaps the more obvious reason for teaching from first principles is that it removes the barrier of prior contextual knowledge that often disproportionally affects those who come from less privilege. This leveling of the playing field is an essential part of what makes great teachers great.

Prioritizing Growth

Focusing on growth is also an extremely underrated philosophy in teaching. It’s sort of the teacher’s extension of Carol Dweck’s growth mindset. By focusing on growth, the students who know the most are pushed to continue growing, despite their knowledge, and the students who know the least are pushed to grow, despite their lack of knowledge. Straightforward enough. What’s less straightforward is how this is actually implemented in practice.

I think the only way that teachers really do this well is by internalizing and deeply accepting that the prior success of a student does not inform their growth potential. It’s difficult to do because if one student knows more than another, the simplest explanation is that they have previously grown their knowledge faster than the other student. But it’s arguably more likely that the difference can be attributed to the previous growth opportunities and environments that each student was exposed to. This should be the default assumption for variations in students’ baseline success. When teachers build this assumption into their teaching style, it becomes difficult for students performing at the top to become complacent, and it also makes it difficult for students near the bottom to give up.

Why Everyone Should Care

Of course, more goes into being a great teacher than I can possibly describe. There’s something spiritual involved in giving people a reason to care about wrapping their head around something complicated. From what I have seen though, many otherwise good teachers could be made better by perfecting these two elements. I care about this because in some ways, to be a good teacher is to be a good thinker. I don’t necessarily even mean this in a Richard Feynman way. I mean that the only way anyone can ever really prove that they know anything to anyone is by teaching it to them. To teach is to communicate. And being precise and accessible in the way we go about communication is ridiculously underrated. For what it’s worth, I bet my sister’s substitute teacher wouldn’t be so confused about American History if someone had put in the time and care to teach them.

If you liked this article, please consider giving a clap and a follow. :) I’m also on twitter @DanielFein7.

Don’t know what to read next? I recommend my brief article on emergent properties.

Let’s be honest… Isaac Asimov has weird facial hair, but he’s a really interesting thinker. When I was younger, I read his short story The Last Question, and was immediately hooked. The story tells a very interesting tale about technology and its relationship with humanity, and the ending is powerful to say the least. The thing that caught me off guard most when I read it was the notion that everyone should be at least a little worried about the heat death of the universe.

For a quick refresher, this is the idea that with the entropy (randomness) of the universe perpetually (and irreversibly) increasing — as it does by the second law of thermodynamics — there will be a point when it reaches thermodynamic equilibrium, and no more energy can be converted into work. In other words, the ‘heat’ in the universe will ‘die,’ becoming spread so thin and uniformly that it cannot be used to do things. No living things, stars, etc can exist in this state, hence the worry of Asimov’s characters.

In school, I was taught about the second law of thermodynamics by the following analogy. I was told to imagine dropping an egg on the floor. They told me that the egg would never naturally return to its original state of being an egg, for the rest of time it will just be disparate yolk particles scattered around first the floor, and eventually around the world and then the universe. After thinking about this example for all of ten minutes, I was confused. Of course, an egg in a void would just break and be broken. But an egg in real life would surely be reabsorbed by plants, and then eaten by small animals, and then possibly even by another chicken who lays another egg. But entropy cannot be reversed, so what gives?

Well, the physicists answer to this is that the life itself creates even more disorder when generating the second egg, respiring food into air. But its still curious that life is capable of taking some particles and putting them back into order in a way that no other phenomena can. Wait a minute, though. Because there is another thing that can reverse entropy: us. Using our brains and our hands, we can pick up each of those particles and put them into order with our hands. We must be able to, or how would we build buildings, iPads, and everything else we make to entertain ourselves.

So we’re up to two things in the universe that can ‘reverse’ entropy: the mind and life, thought and evolution. The thing about reversing entropy is that its synonymous with creating information. Because what informs the ordering of these disparate particles but new information. Another interesting thing is that it one leads to the other. Evolve long enough and you get a mind. What happens if you think long enough?

People talk a lot about whether we are in a simulation. I tend to not really care, but thinking about it now, it’s almost as if the mind is a simulation of evolution, at least along this axis. In our attempts to produce a human-like AI, maybe we should be considering models of evolution as we consider models of cognition.