This is the third in a series of 3 Tutorials.

In this tutorial you’ll add basic AI functionality to your web prototype (from the previous tutorial) by making an OpenAI API request.

• 1 —Using AI’s Non-Determinism
• 2—APIs, Keys, and OpenAI’s API
• 3— Making an OpenAI API Request
• 4 — Checking errors
• 5— How far can you get…

1 — Using AI’s Non-Determinism

For the sake of this portion of the class we will focus specifically on AI’s non-deterministic properties and thinking through how to leverage this in a web interaction.

What is Non-Determinism and how is it useful?

Non-determinism means something behaves differently each time. Its unpredictable, maybe even intentionally. If a user asks the same question, they may get a different response.

There are likely many useful ways to leverage AI’s non-determinism. For simplicity we’ll focus on two of non-determinism’s main utilities: using it to produce more choice (lots of options), or using it to choose (choose a single option from many possibilities).

Choice

Generative interfaces and interactions are becoming more popular with the rise of AI. Producing more and producing variety can help us iterate more quickly on an idea, or explore possibilities we might not have otherwise considered. However, more is not always better. More options can mean decision fatigue. So be intentional about how many options you use, and how varied the results are that you produce.

Choosing

Adaptive interfaces and interactions are also becoming more popular with the rise of AI. Adaptive interactions can help us choose from the wide range of possibilities. An “adaptive interface” reduces options and information that someone sees by adapting the interface to the specific preferences of a person, or context about the time of day, place, or context to help them make a choice. In a sense it is pre-bakes decisions before they are ever exposed to a user. Note that adaptive is also not always useful. For example the early Microsoft word adapted menus automatically based on which actions a someone clicked most often, but this led to users getting frustrated with menu structures that were constantly changing and unlearnable.

2— APIs, Keys, and OpenAI’s API

What is an API?

An API, or an Application Programming Interface, is just that, an interface for someone using a web app. For example, Google Maps has an API a software developer might use when adding a maps page on their personal website. The API exposes the various capabilities and controllable parameters of the Google Maps service.

If a website or application has an API, they typically have a page on their website dedicated to describing their API and how to interact with it such as this: API Documentation. Within the documentation you will often find pages within that page that describe the specific syntax in order to create API call/API requests to their application.

In this tutorial, we’ll create an API request to OpenAI to build a non-deterministic capability in your website. This can be either “generative” produce choices, or “adaptive” make a choice.

Get an API Key from OpenAI

To add AI functionality to your website prototype you’ll need to get an API Key from OpenAI. This key lets us connect to OpenAI and use it in our program. It’s similar to logging into an email service to prove who you are — this key “authenticates” our API requests, letting OpenAI know that we’re allowed to use their tools.

2a — Sign Up/Log In to OpenAI: Visit OpenAI’s website and sign up or log in. Once logged in, click “API.”

2b — Create a new Project: Click the “Default project” and create a new project. You can name this whatever you want. This is helpful for tracking the API spending for your different projects.

2c — Create an API key: Click “Dashboard” (upper right hand corner). In the left panel navigate down to “API Keys.” A new window will pop up. Click “Create New Secret Key.” Name it whatever you want, click “create” and then copy the key.

2d — Store the Key: Keep your API key safe and ready to use in your code. Once you close the window you will not be able to access the key again, but you can always create a new one. Be careful about where it is placed (e.g. never paste your key into ChatGPT or upload your API key to gitHub).

2e — add your API key to your code: Back in Visual Studio code, at the top of your Javascript add the below code, and paste your new API key where it says YOUR CODE HERE. This assigns this text, called a “string”of your API key to a constant named apiKey, so we can easily access it later in our code.

//IMPORTANT!!! do NOT add your API Key to CodePen unless you have a private account and can set this to private!
//Any sketch you make will be saved automatically and public and your API Key will likely be scraped by bots and used by other people leading to charges$$$!
const apiKey = 'YOUR KEY HERE';

⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️

Very important note about simplicity!

Note that in my case my prompts are about a fortune cookie and that my Javascript is specific to that interaction. You might choose to adapt this to your needs using your code. However, I recommend first starting with something very simple like this fortune teller interaction where a button gets a simple 5–10 word text response from OpenAI. Keep your code VERY simple to start, so you can first get a sense of how the API request works, before applying this to your interaction.

⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️

3— Create an OpenAI API Request

In this step, we’ll use our API key to request a specific response — “a short fortune cookie phrase” — from OpenAI.

3a — Check the OpenAI documentation: You can find API documentation under “API reference”. We’ll look at the “Create chat completion” here: https://platform.openai.com/docs/api-reference/chat/create?lang=curl. Select and copy the entire curl command.

3b — Use ChatGPT to Translate: Instead of curl, we want to translate this into Javascript since Javascript is the standard for building a website. This is where we can put ChatGPT to work. Paste the command into ChatGPT and ask it to translate the curl command into JavaScript code.

Your prompt should look something like this:

Can you help me translate this below curl command into JavaScript that runs locally? 
Please put it into an async function called 'getFortune' with a try catch. 
I do not need a system level prompt. 
The message should be 'Give me a short fortune cookie phrase.' 
I already have an API key above this code segment in a constant called "apiKey".

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

You’ll notice we can add a lot of instructions in this prompt to make the code better formatted for our needs. Here’s some of the rationale for those additional instructions in the above prompt:

Can you help me translate this curl command...

Why translate? We’re translating this cURL command to JavaScript so we can use the REST API directly without additional installations.

There are typically 3 ways to talk to an API: Python, Javascript, or REST.

Using something like Javascript would require us to install both openAI’s SDK and node.js. However the REST API allows us to use cURL a command-line tool used to send HTTP requests to interact with APIs which is a more standard universal protocol for multiple types of computers thus wont require the same installation process.

Please put it into an async function...

Why put it in an Async Function? This allows to wait for OpenAI’s response to our request asynchronously so that our entire program doesn’t stop and wait while OpenAI is working on the response.

I do not need a system level prompt.

Why take out the system level prompt? Because we just need a single response and are not handling a back and forth conversation, we can stick to a user prompt, and skip the system level one. A system level prompt as opposed the the user level prompt is akin to higher level instructions you’d give for an entire chat thread within ChatGPT — for example, “you are a medical assistant, please respond accordingly.” Whereas a user level prompt is akin to a specific prompt within that system thread, for example “I have a fever what might be causing this?”

Ok, so your original Javascript code was likely something like this:

const apiKey = 'YOUR KEY HERE';

//WE'LL WANT TO ADD YOUR NEW ASYNC FUNCTION WITH THE OPENAI REQUEST IN HERE

const fortuneButton = document.querySelector('#generate');
fortuneButton.addEventListener('click', () => {
    const fortune = "Your default fortune: Success is in your future!";
    
    // Update the h1 text with the default fortune
    const fortuneDisplay = document.querySelector('#fortune-text');
    fortuneDisplay.textContent = fortune;
});

but your new Javascript code likely looks more like this:

async function getFortune() {
  try {
    const response = await fetch("https://api.openai.com/v1/chat/completions", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Authorization": `Bearer ${apiKey}`
      },
      body: JSON.stringify({
        model: "gpt-4",
        messages: [
          {
            role: "user",
            content: "Give me a short fortune cookie phrase."
          }
        ]
      })
    });

    if (!response.ok) {
      throw new Error(`Error: ${response.status} ${response.statusText}`);
    }

    const data = await response.json();
    const fortune = data.choices[0].message.content;
    console.log("Fortune Cookie Phrase:", fortune);
    return fortune;
  } catch (error) {
    console.error("An error occurred:", error);
  }
}

We can now ask ChatGPT to adapt this into our code so that it runs when we press our button. With a prompt like this:

Here's my Javascript: 
PASTE THE 1ST SCRIPT
 
Can you adapt this code so that it runs whenever the button is pressed: 
PASTE THE 2ND SCRIPT

Your full Javascript code should look like this:

const apiKey = 'YOUR KEY HERE';

async function getFortune() {
  try {
    const response = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${apiKey}`,
      },
      body: JSON.stringify({
        model: "gpt-4o",
        messages: [
          {
            role: "user",
            content: "Give me a short fortune cookie phrase.",
          },
        ],
      }),
    });

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();
    const fortune = data.choices[0].message.content;
    console.log(fortune);
    return fortune;
  } catch (error) {
    console.error('Error fetching the fortune:', error);
    return "Your default fortune: Success is in your future!";
  }
}

const fortuneButton = document.querySelector('#generate');
fortuneButton.addEventListener('click', async () => {
    // Call the getFortune function and await the result
    const fortune = await getFortune();
    
    // Update the h1 text with the fetched fortune
    const fortuneDisplay = document.querySelector('#fortune-text');
    fortuneDisplay.textContent = fortune;
});

Paste this back into your VSCode.
Remember you’ll need to replace the API Key at the top of this Javascript with your own: const apiKey = ‘YOUR KEY HERE’;

Give it a try!

4 — Checking errors

If you run into errors. Don’t worry! This is actually part of a usual software development process. Errors are the norm.

If your code isn’t working. Here are some helpful tips to work through the error step by step:

• Check for errors.
• Can you determine what part of the code, or what step the error refers to? If not you can paste the error into ChatGPT and ask what the error is from. This will work better if you do this in the same ChatGPT thread as you’ve been using to develop your initial code so that it has context on your work.
• As much as possible try to isolate the problem: Simplify your code by isolating the part that is causing the issue. Comment out parts of your code or temporarily remove sections.
• Use print statements: Ask ChatGPT to write you print() or console.log() statements in your code to track where the error emerges.
• Consider going back a step back (e.g. refer to your code before your last change). Was there anything in this step that produced the issue?
• When in doubt, simplify!

5 — How far can you get?

Now that you’ve braved an OpenAI request, see if you can adapt this to an interaction you’d like to make that leverages non-determinism.

Keep it simple and slowly build up the capabilities. Paste in only small segments of code or subtle changes. For example can you change the content of the prompt content: “Give me a short fortune cookie phrase.” in the Javascript? Perhaps instead of a simple button click you try sending a short phrase to OpenAI when you click the button.

This is the first in a series of 3 Tutorials.

In this tutorial we’re going to create a very simple web prototype in the browser.
You‘ll choose a website interaction you like and use some prompting with ChatGPT to mimic it.

• What is HTML, CSS, & Javascript?
• Code Pen for quick web prototyping
• Natural Language Programming with ChatGPT

What is HTML, CSS, & Javascript?

Most websites today include three primary languages: HTML, CSS and Javascript. HTML: Structure, CSS: Style, JavaScript: Interactivity

• HTML (Hypertext Markup Language) is the standard language used to create and structure the content on web pages, such as text, images, and links.
• CSS (Cascading Style Sheets) is the language used to style and layout web pages, controlling the appearance of elements like colors, fonts, and spacing.
• JavaScript is a programming language that adds interactivity to web pages, for dynamic content, animations, and communication with APIs.

Code Pen for quick web prototyping

Code Pen

1 — Sign up for a free version of Code Pen
Code Pen allows you to quickly test a website without worrying about setting up a server just yet. Be sure not to use/paste any API keys in your Code Pen code, because it’s public and thus scrape-able/exploitable.

2 — Rename and Save Often
In the upper left hand corner name your sketch whatever you like. I’m naming mine “Basic Web.” Save regularly so you don’t lose work.

HTML

3 — Create the basic structure of your page.

<h1 id="Hello">🥠 Hello World!</h1>
<button id="generate">Click Me</button

CSS

4 — Add styling to your page

The CSS will style the page and its elements. Play with making some adjustments here. Can you make the font bigger? Change the background color?

body {
  font-family: system-ui;
  background: #f06d06;
  color: white;
  text-align: center;
}

Javascript

5 — Add functionality to your page

The Javascript will allow you to build interactivity and the functionality of the page. Its the web behavior. Below we’ll make the button clickable, and have it change both text and the website background color:

document.getElementById("generate").addEventListener("click", function() {
  // Change the text
  document.getElementById("hello").textContent = "🥠 Greeen is the color!";
  
  // Change the background color
  document.body.style.backgroundColor = "#4CAF50"; // Change to green
});

Natural Language Programming with ChatGPT

Prompt ChatGPT for an interaction pattern.

5 — Screenshot an interaction
Find an interaction pattern you like in a website you like. Screenshot just a portion of it. Don’t worry about anything too complex just yet like AI interactions or API requests. For now just focus on one portion of the site.

6 — Paste the screenshot into ChatGPT
and ask for a simple website that mimics this interaction, describing the interaction and asking for it to separate the HTML, CSS, and Javascript so that it can easily be pasted into Code Pen. Some basic natural language prompt techniques that have worked for me:

• keep it simple — one at a time prompts that focus on one segment of what you are trying to do.
• aesthetics first — I find starting prompts with the aesthetics first, and then adding more functionality (Javascript) after tends to work better. So your prompt might emphasize “make me a website that looks like this”

Example Prompt:
Can you build a website to make this into an interaction where if I click the buttons it will generate a textbox that is connected like this? Generate it as separate (html,css, javascript) so that I can paste each segment into codepen?

7— Paste your results into the respective segments of Code Pen.

8 —Debugging the console

If you run into errors, you will see them as a red “!”. You can click these to reveal the error. If its not immediately clear what the error is, try pasting this into ChatGPT for help on explaining the error.

Up Next, Tutorial: Create A Website Locally with a Server

This is the second in a series of 3 Tutorials.

In this tutorial we’re going to create a very simple web prototype in locally.
You’ll use the you created in the previous tutorial, but run it locally now.

• Why Run Websites Locally?
• Getting Set Up with Visual Studio Code
• What’s a server?
• Assignment — How Far Can You Get?

Why Run Websites Locally?

Protecting API Keys
Running websites locally allows us to securely make API requests without exposing sensitive keys. In environments like Code Pen, API keys are visible in the frontend code (unless you have a paid account). When API keys are posted publicly they can often be scraped and exploited — meaning someone will grab your API key and run up your bill using it for their own projects. Locally, we can keep these keys hidden in secure files.

Direct Control Over Code
Local development gives you full control over the code (HTML, CSS, JavaScript), so you can observe exactly how each segment works and make changes in real-time. This hands-on approach helps you understand the impact of each modification.

Why We’re Still Copying and Pasting Code
Although AI tools like Claude or CoPilot could automate some code generation, tools like Claude will cost an addition $20/mo and compress the HTML, CSS, and Javascript into a single file giving you less understanding and control over the code changes. Since we are very interested in what works/what doesn’t and understanding the relationships between the prompts and the code generated, copying and pasting may actually help us retain some control over what we are creating and understand how various prompts behave.

Getting Set Up with Visual Studio Code

Next we’re going to move your files from the Code Pen website from the last tutorial to your computer and install some tools for running that code locally.

1 — Create a local folder
Create a folder somewhere locally on your computer called “LLMs for UX Prototyping.”

2 — Install Visual Studio Code
Install whichever version matches your computer. Follow all instructions to make sure its installed. Then open Visual Studio Code. Visual Studio Code is a code editor for software developers.

3 —Create 3 files for each of your Code Pen files
Inside of Visual Studio Code, go to the lefthand explorer panel and click open folder, and navigate to the “LLMs for UX Prototyping” folder.

Next in the explorer panel in VS Code right click “New File” to add a new file and name it “index.html”. Next you will paste your HTML code from CodePen in this file. Save and close.

Do the same for your Javascript and CSS files as well naming them “basic.js” and “styles.css” respectively.

4 — Add code to the HTML file to reference the CSS and Javascript
In order to have your code work, the HTML needs to reference the CSS and Javascript. Copy and paste your HTML code into ChatGPT and ask it to add a reference to “basic.js” and “styles.css”. A reasonable output should look somewhat like this (though your html will likely be more complex):

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Green Fortune</title>
  <link rel="stylesheet" href="styles.css">
</head>
<body>
  <h1 id="hello">🥠 Hello World!</h1>
  <button id="generate">Click Me</button>
  <script src="basic.js"></script>
</body>
</html>

5 — Install — Live Preview VS Code Extension
Inside of VS Code, we will go to the “extensions” button on the left panel, search for “Live Preview” and click “install” to install it. Live Preview is an extension that will allow us to preview your files on a local server.

• Use — Live Preview
  Next click the “explorer” button on the left panel to return to your files. Click your index.html file and then click the “live preview” button in the upper right which looks like this:

FAQ — What is a Server?

A server stores your website’s files and makes them available to people when they visit your website. It handles requests, like loading a page or accessing information, and delivering the website content to someone who visits the site. A local server is a server that runs on your own computer instead of being hosted online (e.g. not Amazon Web Services, or Google Cloud). It lets you develop and test websites on your computer before making them live to the internet.

Assignment — How far can you get?

Now that you have a basic local website working, see how far you can get in mimicking the interaction? Try to get as far as you can.

You may choose to jump into portions of the code and make changes directly. Don’t forget to save as you go.

As you go, notice where AI prompting falls short and where it excels. What are your 3 main takeaways on what worked and what didn’t?

In this tutorial we’re going to create a simple prototype that returns a new 🥠 fortune from OpenAI each time a button is clicked.

Getting started in Code Pen

This is what we’ll use for basic programs that we’ll prototype directly in the browser. Why in the browser? Code Pen will allow us to quickly and easily prototype without worrying about setting up a server just yet.

1 — Sign up for Code Pen

2 —Rename and Save Often

In the upper left hand corner name your sketch whatever you like. I’m naming mine “AI Fortune.” Save regularly so you don’t lose work.

📖 What is HTML, CSS, & Javascript?📖

Most websites today include three primary languages: HTML, CSS and Javascript. HTML: Structure, CSS: Style, JavaScript: Interactivity

• HTML (Hypertext Markup Language) is the standard language used to create and structure the content on web pages, such as text, images, and links.
• CSS (Cascading Style Sheets) is the language used to style and layout web pages, controlling the appearance of elements like colors, fonts, and spacing.
• JavaScript is a programming language that adds interactivity to web pages, for dynamic content, animations, and communication with APIs.

Setting up your HTML

3 —Create the structure for the “fortune text” and “button” on your page.

Fortune Text:
First we’ll change <h1>👋 Hello World!</h1> to include whatever you want the starting phrase to be. We’ll also add an id id=”fortune-text” to this phrase which will allow us to update the h1 text element when we get a response from OpenAI.

Button:
We’ll also add a button which we’ll use to get a fortune from OpenAI.

<h1 id="fortune-text">🥠 Your fortune awaits!</h1>
<button id="generate">Get a Fortune</button>

Setting up your CSS

4 — Keep the styling the same.

For now you can keep the CSS styling the same. You are welcome to change it if you like.

body {
  font-family: system-ui;
  background: #f06d06;
  color: white;
  text-align: center;
}

Setting up your Javascript

5 — Make the button functional.

Before we start worrying about any AI functionality or the OpenAI API, lets just make sure we can make our button somewhat functional. Let’s start by having the button update the fortune text to a new default fortune phrase when clicked. In the bellow code we are:

5a — Selecting the Button: We use document.querySelector(‘#generate’) to grab the button by its id.

5b — Adding a Click Event: We attach a click event listener to the button. When clicked, the function inside will run.

5c — Setting a Default Fortune: We define fortune with a default phrase: “Your default fortune: Success is in your future!”.

5d — Updating the <h1> Text: We select the <h1> element using document.querySelector(‘#fortune-text’) and set its textContent to the fortune value.

const fortuneButton = document.querySelector('#generate');
fortuneButton.addEventListener('click', () => {
    const fortune = "Your default fortune: Success is in your future!";
    
    // Update the h1 text with the default fortune
    const fortuneDisplay = document.querySelector('#fortune-text');
    fortuneDisplay.textContent = fortune;
});

6 — Get an API Key

Next we‘ll need to get an API Key from OpenAI. This key lets us connect to OpenAI and use it in our program. It’s similar to logging into an email service to prove who you are — this key “authenticates” our API requests, letting OpenAI know that we’re allowed to use their tools.

6a — Sign Up/Log In to OpenAI: Visit OpenAI’s website and sign up or log in. Once logged in, click “API.”

6b —Create a new Project: Click the “Default project” and create a new project. You can name this whatever you want. This is helpful for tracking the API spending for your different projects.

6c —Create an API key: Click “Dashboard” (upper right hand corner). In the left panel navigate down to “API Keys.” A new window will pop up. Click “Create New Secret Key.” Name it whatever you want, click “create” and then copy the key.

6d — Store the Key: Keep your API key safe and ready to use in your code. Once you close the window you will not be able to access the key again, but you can always create a new one. Be careful about where it is placed (e.g. never paste your key into ChatGPT or upload your API key to gitHub).

6e — add your API key to your code: Back in Code Pen, at the top of your Javascript add the below code, and paste your new API key where it says YOUR CODE HERE. This assigns the string of your API key to a constant named apiKey, so we can easily access it later in our code.
IMPORTANT!!! Do NOT add your API Key to CodePen unless you have a private account and can set this to private! Any sketch you make will be saved automatically and public and your API Key will likely be scraped by bots and used by other people leading to charges to your account!

//IMPORTANT!!! do NOT add your API Key to CodePen unless you have a private account and can set this to private!
//Any sketch you make will be saved automatically and public and your API Key will likely be scraped by bots and used by other people leading to charges$$$!
const apiKey = 'YOUR KEY HERE';

7 — Create an OpenAI API Request

In this step, we’ll use our API key to request a specific response — “a short fortune cookie phrase” — from OpenAI.

7a — Check the OpenAI documentation: You can find API documentation under “API reference”. We’ll look at the “Create chat completion” here: https://platform.openai.com/docs/api-reference/chat/create?lang=curl. Select and copy the entire curl command.

7b— Use ChatGPT to Translate: Instead of curl, we want to translate this into Javascript so that it will work from the browser. This is where we can put ChatGPT to work. Paste the command into ChatGPT and ask it to translate the curl command into JavaScript code suitable for a web application.

You could use a prompt like:

Can you help me translate this curl command into JavaScript that runs in a browser? 
Please put it into an async function called 'getFortune' with a try catch. 
I do not need a system level prompt. 
The message should be 'Give me a short fortune cookie phrase.' 
I already have an API key above this code segment in a constant called "apiKey".

You’ll notice we can add a lot of instructions in this prompt to make the code better formatted for our needs. Here’s some of the rationale for those additional instructions in the above prompt:

Can you help me translate this curl command...

Why translate? We’re translating this to use the REST API. Typically, you’d use the node.js client, when developing locally on your computer with a server. However, to keep things simple and avoid installing additional libraries, we’ll use the REST API, which allows us to send HTTP requests directly from the browser. That’s why we are translating the curl command to javascript.

Please put it into an async function...

Why put it in an Async Function? This allows to wait for OpenAI’s response to our request asynchronously so that our entire program doesn’t stop and wait while OpenAI is working on the response.

I do not need a system level prompt.

Why take out the system level prompt? Because we just need a single response and are not handling a back and forth conversation, we can stick to a user prompt, and skip the system level one. A system level prompt as opposed the the user level prompt is akin to higher level instructions you’d give for an entire chat thread within ChatGPT — for example, “you are a medical assistant, please respond accordingly.” Whereas a user level prompt is akin to a specific prompt within that system thread, for example “I have a fever what might be causing this?”

Ok, so now your original code is this:

const apiKey = 'YOUR KEY HERE';

//WE'LL WANT TO ADD YOUR NEW ASYNC FUNCTION WITH THE OPENAI REQUEST IN HERE

const fortuneButton = document.querySelector('#generate');
fortuneButton.addEventListener('click', () => {
    const fortune = "Your default fortune: Success is in your future!";
    
    // Update the h1 text with the default fortune
    const fortuneDisplay = document.querySelector('#fortune-text');
    fortuneDisplay.textContent = fortune;
});

And your new code for getting a request looks like this:

async function getFortune() {
  try {
    const response = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${apiKey}`,
      },
      body: JSON.stringify({
        model: "gpt-4o",
        messages: [
          {
            role: "user",
            content: "Give me a short fortune cookie phrase.",
          },
        ],
      }),
    });

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();
    const fortune = data.choices[0].message.content;
    console.log(fortune);
    return fortune;
  } catch (error) {
    console.error('Error fetching the fortune:', error);
    return null;
  }
}

We can now ask ChatGPT to adapt this into our code so that it runs when we press our button. With a prompt like this:

Here's my Javascript: 
PASTE THE 1ST SCRIPT
 
Can you adapt this code so that it runs whenever the button is pressed: 
PASTE THE 2ND SCRIPT

Your full Javascript code should look like this:

const apiKey = 'YOUR KEY HERE';

async function getFortune() {
  try {
    const response = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${apiKey}`,
      },
      body: JSON.stringify({
        model: "gpt-4o",
        messages: [
          {
            role: "user",
            content: "Give me a short fortune cookie phrase.",
          },
        ],
      }),
    });

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();
    const fortune = data.choices[0].message.content;
    console.log(fortune);
    return fortune;
  } catch (error) {
    console.error('Error fetching the fortune:', error);
    return "Your default fortune: Success is in your future!";
  }
}

const fortuneButton = document.querySelector('#generate');
fortuneButton.addEventListener('click', async () => {
    // Call the getFortune function and await the result
    const fortune = await getFortune();
    
    // Update the h1 text with the fetched fortune
    const fortuneDisplay = document.querySelector('#fortune-text');
    fortuneDisplay.textContent = fortune;
});

Give it a try! If you run into errors check your console log!

Congrats!

Now make your own AI Interface

Create your own AI interface using AI prompting. You might use ChatGPT or a similar tool to change this further.

1 — Change the HTML Structure and Javascript Functionality

Change the Javascript functionality. Some possible ideas to make it your own:

• A clock that displays a new AI-generated haiku every minute.
• Two buttons, “Pro” and “Con,” that generate opposing arguments on any given topic.
• Devil and Angel — A text field that allows a user to type. With an interface that responds with a for and an against argument.
• A dial you spin to receive AI-generated advice, akin to a custom magic 8 ball.
• A brainstorm tool that generates 10 varied ideas based on an initial phrase.
• Get some ideas from creatives like this or from existing products here.

Consider the changes you’ll need to make to the OpenAI response and whether a system prompt is needed.

2 — Change the CSS Styling

• adapt the CSS styling
• find an existing style you’d like to mimic from another website, upload the image to ChatGPT. Again consider ideas from resources across the web.

This Week at Spatial Pixel — Sep 9, 2024

1—People are starting to test the Procession beta.
2—We added generative-text capabilities.
3—We ran a creative AI workshop with local artists.
4—We hosted some guest speakers in our classes who spoke about spatial interaction and AI+UX Trends.

How might we enable spatial computing beyond glasses? — In the spirit of fostering an open environment for research that can enable this future, we’re sharing experiments as we go.

1—People are starting to test the Procession beta.

Who is testing Procession?

We’re thrilled at the inbound interest for Procession, a spatial computing environment “for your space, not for your face.”

It’s currently being tested as a private beta by three classes, at Columbia University, University of Michigan, and University of Pennsylvania. We’ve also provided access to a few more academic and artist labs.

How can I get access to the Procession beta?

We’re a small team committed full time to this dream!

Because it’s early, we’re first prioritizing developers and researchers who can expand the reach of a glasses-free spatial computing.

• Research Beta
  If you have an idea in mind for developing a library or extension to Procession, please get in touch by messaging us through our LinkedIn page. We’re providing access to a small number of researchers who able to contribute new library capabilities. It’s our intent for contributions to first be open-source and non-commercial.
• Non-Research Beta (not released yet)
  Our broader focused non-research beta has not yet been released. If you’d like to stay tuned for this, you can follow along via our LinkedIn page.

A research brief.

This week we were invited to share a research brief for Columbia University GSAPP’s Computational Design Master’s program to solicit student researchers. We’re excited to invite them to explore opportunities with us!

We’re sharing it below in case you’re also working towards a similar glasses-free future of spatial computing.

A Special Request — Hardware Research

We’re particularly interested in folks who are investigating camera-projector combinations and spatial awareness with sensors. If you’re working on this kind of hardware, please reach out!

2—We added generative-text features to Procession.

NEW: Generative Text — AI inside the AI

This week we added generative-text capabilities to Procession. This means that in addition to using to programming with natural language to generate running code—the basic UI flow in Procession—you can also prompt Procession to incorporate text completions from OpenAI (and other APIs in the future). This means you can build sketches that generate haikus, poems, or other meaningful prose. We’ll soon be vetting function-calling to investigate the uses of generative logic.

IN PROGRESS: RFIDs & Sensor Inputs

We’re currently working on getting more physical inputs from sensors into Procession. We’ve built a working prototype of RFID sensor inputs (commercial readers and an Arduino-based ones) into Procession and workshopped this on Friday with artists (see below). We’re also aiming to extend this to environments like IFTTT, by which we can connect to all kinds of sensors and actuators downstream. This can help spatial computing extend beyond an individual device and more into the physical hardware that can enable an intelligent space.

FUTURE: Open Programmable Spaces

We’re also working towards a more general beta. We’re inspired by environments like Open Interpreter, Processing, P5.js, and Dynamicland that are community led and open. We hope to work with other researchers to build out such an environment.

Our long term vision is to have a programming environment where artists, creatives, and spatial designers can build intelligent spaces and develop and share research in the space in an open way.

3—We ran a creative AI workshop with local artists.

AI with not for Artists

This week we ran a creative AI workshop with a group of artists at the Ministry of Awe in Philadelphia. Artists came up with ideas for material objects throughout the museum that could serve as input into Procession and then eventually combine into a collective, generated, realtime mural.

The Ministry of Awe is a new immersive museum being collectively created by artists in the heart of Philadelphia’s Old City, spearheaded by the renowned mural artist Meg Saligman. She is pulling together multimedia artists from all domains, from painting to projection mapping to object-making. We’re excited to share more of our work with them as it develops.

4—Academic Recap

William Martin teaches “Spatial AI” and “Colloquium I for Computational Design Practices” at Columbia University GSAPP. Violet Whitney teaches “Spatial UX” at Columbia University,“LLMs for UX Prototyping” at University of Pennsylvania,“Reading and Writing the World” at University of Michigan.

Josh Goldblum serial entrepreneur, Founder & CEO
Bluecadet, FutureSpaces, Artwrld
Guest Lecture: Spatial UX @ Columbia University

• Find Your People—As a serial entrepreneur, Josh talked about working on the cutting edge and “finding your people.” If you don’t have a scene make it. Josh struck me as an explorer. Someone who’d gladly venture into new scenes whether that’s the world of crypto or the world of artists.
• It’s okay to pivot—As someone working in emerging spaces, a big part of working in an exploratory way is being okay to change course as you learn. When asked about shifting Artwrld from NFTs to artist talks, Josh spoke about taking stock of what you have. As the market shifts, or the ethos of movement shifts, don’t be afraid to change!
• Link to Bluecadet—many interactive spatial experiences
• Link to FutureSpaces—talks/community on physical/digital installations

Kshitij Agrawal, Creator of
AI Verse AI + UX Trends
Guest Lecture: LLMs for UX Prototyping @ University of Pennsylvania

• Research Aloud—Kshitij has been researching and categorizing interaction patterns and trends emerging around AI. His research has garnered quite the following since so much of what he does is useful those of us also trying to figure out this AI thing too.
• Trend: Nonlinear & Agentic UX—Techniques that give a user control to configure a UX flow in a nonlinear fashion. Natural language programming is all about being able to structure the flow of information.
• Trend: Ambient — Embedded directly on top of the environment around you: the tables, the walls, the books. Most of these interaction patterns exist in the world of museum installations where Josh Goldblum (above) and Bluecadet reside. But these types of interactions are becoming much more meaningful to the world of everyday technology.
• Link to his presentation.
• Link to his website.
• Link to his Twitter/X.

Thanks for reading. Have something to share or interested in learning more?

We run Spatial Pixel: spatial computing “for your space, not for your face.”

Spatial Pixel is committed to creating and fostering an open environment for spatial design research that can enable a future for spatial computing that is inherently embodied, environmental, and collective. We hope this research can build counter-practices to the glasses-only future being created by Big Tech.

Spatial Pixel — Research, Research & More Research was originally published in Spatial Pixel on Medium, where people are continuing the conversation by highlighting and responding to this story.

User Interface Bureaucracy and How a Philosophy from 1900s Capitalism Could Save Us.

Chapter 1 —Flat screen. Flat thoughts.

Chapter 2— We need space to think.

Chapter 3 —Can we study knowledge work like manual work?

I have this syncing feeling that every little task I have inside the computer is like a mini bullshit job — a bullshit task…

Chapter 1 — Flat screen. Flat thoughts.

If you’re anything like me, you spend most of your day looking at your screen. If you’re a knowledge worker it’s likely worse.

Steve Jobs said the computer is like a bicycle for the mind.
This is my “mind bicycle” right now:

I think my mind bicycle has two flat tires going uphill both ways.

Computers — like a scattered pile of papers

What makes computer work feel so scattered? Well, computers today lack the spatial structure that we’re used to in the material world. So while working on a computer requires tons of context switching between apps, there’s very little spatially to help us organize the context switching between apps.

I sometimes find myself clicking around tabs having completely forgotten what I was even doing. Was it my calendar I needed? A link from an email? Was I looking up a reference on the web?

Even more concerning is that we’re spending more and more time on our computers, often missing the world and people around us. It’s impacting how we think: shortening children’s attention spans, making us less empathetic.

We’re burnt out and screen fatigued.
And yet, we’re all still here.

Chapter 2 —We need space to think.

If you were a designer when tools like Miro and Mural emerged, it felt like a big step forward. Their killer feature: just the simple ability to zoom out on a canvas. What it meant was that you could finally organize work spatially. You could collect and group material like we do in the real world.

For knowledge workers who need to organize, research, brainstorm, or structure plans — the spatial relationships are the thinking. So the space helps us think.

We think through our bodies, not just our brains.

Humans have embodied cognition. Our thoughts, reasoning, and behaviors are affected by our physical body. How information is organized and arranged, and our sensory interaction with the environment impacts our thinking. Students in math competitions use physical gestures to solve the problems.

We use proximity — we’ll place tools we’re using now nearby and tools we plan to use later far away. We naturally organize our desks spatially. We put sticky notes on the sides of our monitors to remind us in a way that isn’t distracting, but close enough to help us remember. A sticky note is basically useless if it’s in the wrong place.

We group spatially — We group like items together: tools and materials for sewing and laundry get stored together while tools and materials for cooking go another place.

We use our gestures — I might say, “let’s brainstorm this later,” while pointing at a sketch. Or “can you make this part blue while circling an area of a drawing.” I might point at something and ask “what is it?”

We use imitation — an animator might demonstrate how water in a scene should behave by acting out its motions. A choreographer shows dancers how to move. Toy cars might can help us understand the the events of a car accident.

We use material affordances — Building physical models or prototypes might help us understand how elements work together and behave. “This will slide off of that.”

Some of my best thinking comes when I’m away from the computer. When I’m out for a walk I can associate ideas more freely. When I make a physical prototype it quickly clarifies what works and what doesn’t. Sometimes just taking a shower and disassociating completely is where I resolve problems I’ve forgotten about.

If our thinking is tied to our bodies, our rooms, our materials, our neighborhoods, the sounds and the smells around us, then we must find ways to make our tools for thinking leverage the spatial and embodied capabilities of the natural world.

In “Externalism, Epistemic Artefacts and the Extended Mind,” Kim Sterelny shows how humans use the environment and external tools for cognitive processing. Most transformative to me was the idea that the homosapien isn’t actually much smarter than our previous distant cousins. There wasn’t some genetic breakthrough in human brain structure making homosapiens smarter. Rather a subtle change in our genome allowed us to build external tools and use them in combination with our minds to build cognitive processing capabilities into the world around us. More-so, it was the “proliferation and transmission” of these tools for thinking that led to the explosion of human’s cognitive capabilities.

“Yet in all probability human cognitive powers have expanded without any significant change in human genomes. Instead, their expansion is a result of the proliferation and transmission of epistemic artifacts; of tools for thinking. As he sees it, we have got smarter, even though our naked brains have not.”
 — “Externalism, Epistemic Artefacts and the Extended Mind,” Kim Sterelny

In short, our thinking capacity is tied more to external tools than it is to our internal minds.

Thus, if our tools are what defines our intellect,
but our “mind bicycles” are modern day thought interrupters,
then its time to rethink the computer.

Chapter 3 — Can we study knowledge work like manual work?

Motion Studies of Manual Work

In the early 1900s, Frank and Lilian Gilbreth pioneered scientific management by looking at how processes in factories — and even kitchens in the home — could be optimized for efficiency. The looked at each step of a worker’s task whether that was laying brick in construction, assembling a part of a physical product in a factory, or baking a cake in the home. Each task study was documented against a gridded backdrop with cameras and timers.

By tracking movement over time they were able to study manual labor and diagnose its parts and make recommendations to “optimize” each job. The aim was to save wasted time in the work.

Their work shaped the structure of factories and even office work today.

But while it may have saved time on a job, the drive for efficiency ignored people’s humanity, their well-being, their creativity, and the broader social aspects of work. All were simply missing.

Can Knowledge Work Be Motion-Studied?

Just like manual work, knowledge work takes place in sequence through time. But when the computer is introduced, the sequences are chopped into fragments: I’m going to find a file, attach it to my email, then send it.

Just like manual work, knowledge work takes place through motions. But on the computer our motions get constrained to tiny finger movements on a mouse stuck on a two dimensional plane.

We need to view our digital interaction with the same scrutiny as the Gilbreth’s but with a fresh take. It’s not about physical labor, but a mental one.

We should investigate our own knowledge work interaction patterns. We should appropriate work measurement tools like time motion studies, employee monitoring, mouse tracking, UI heat maps, image segmentation, body tracking, gesture recognition.

But instead of focusing on “productivity,” our goal should be to reinvent knowledge work altogether-working from our own values, whether they are creativity, self-actualization, connection, or something else. We should look to the anti-work movement to understand what’s going wrong.

Like David Graeber’s postulation that much of our jobs today are bullshit, I have this syncing feeling that every little tasks I have inside the computer is like a mini bullshit job — a bullshit task. Just UI bureaucracy playing out, but I’m just the cog in the machine, jumping through hoops to make it all work.

Time Motion Studies made “therbligs” to describe and categorize the sub-tasks that made up a task like “search” or “grasp”. We can invent our own therbligs to describe and categorize the sub-tasks of knowledge work.

We can dissect the choreography of clicks, searches, drags and drops as spatial and sequenced, just as the motion studies did. Within the scatter between the apps and pop-ups, perhaps there’s a way after all to create order from the chaos.

Today’s work, knowledge work, is in crisis! It has us fixed to a chair 8 hours a day, completely screen fatigued, with a constant stream of thought patterns interrupters. Perhaps most importantly, the bounds of what we can think of is being held back by our 2 dimensional tools.

It’s time to deliver on the promise of computers as “bicycles for the mind.” We need to expand our thinking to the spatial and material dimensions, but to do so, we must untangle this 2-dimensional mess that is the UI bureaucracy of our time.

Thanks for reading. Have something to share or interested in learning more?

Violet Whitney teaches “Spatial UX” at Columbia University, “LLMs for UX Prototyping” at University of Pennsylvania, and “Reading and Writing the World (Physical Computing)” at University of Michigan.

She and her partner William Martin run Spatial Pixel: spatial computing for your space not your face.

Flat screen. Flat thoughts. was originally published in Spatial Pixel on Medium, where people are continuing the conversation by highlighting and responding to this story.

Low-level Abstraction Code Intro — for course LLMS for UX Prototyping

Learning Objectives

• Intro to the P5.js environment
• build a simple sketch
• learn how to find, understand, and use P5 documentation
• Introduction to abstraction in coding

The Basics

What is p5.js and why use it?

P5.js is a JavaScript library for creative code. It’s a flexible software sketchbook and a language for learning how to code visually and use it for the creative arts. Its an especially great environment for physical computing because it will allow you to prototype projection mapping that responds to interactive inputs like (a web cam, markers, or a multitude of triggers from the web). Many artists choose to work with P5 because of its visual interface and its simplicity. P5 can be a great environment to prototype interfaces, tie together IOT devices, and work with interactive audio.

Setting Up P5

1. Open the p5.js web editor at https://editor.p5js.org/.
   No downloads needed. It’s all online!
2. Login and create an account. (top right corner)
   This lets you save and revisit your sketches.

P5 Architecture

• text editor — where you write your code (called a “sketch” in P5). Your code is written in JavaScript, just like on the web.
• compiler — translates code into machine-readable instructions
• display window — where visuals created by your sketch are displayed

Basics

You will have two basic parts within a Processing “sketch”, the “setup” and the “draw” loop. The setup loop runs once when launching your script, whereas the draw loop runs repeatedly. This allows you to save computing power by sticking parts of the code that only need to run at the start in the setup loop. The draw loop will continually update so it can change over time, or get realtime information such as the location of your mouse or update a camera feed.

function setup() {
  createCanvas(1280, 960); // Create a canvas
}

function draw() {
  background(0); // Set black background
  ellipse(mouseX, mouseY, 500, 510); // Draw ellipse at mouse position
}

Reference

The basis of learning to code is really about learning how to find and use information (usually online). Rarely does anyone know off hand the exact syntax and operations to write all of their code from scratch. Rather, creative technologists and even seasoned software engineers regularly use the reference pages for a programming language or a development environment to understand how a particular programming language functions, its operations, and so they can borrow from what’s already written. You can find P5’s reference here: P5 Reference.

Using Documentation

If we look around within P5’s reference, we’ll see all kinds of operations and parameters we can use in our code. So much possibility! Poke around and click into some of the functions you see. Lets look into background(). You'll see if we click into background we'll get information on how to use this function, what parameters it expects, and some examples. It looks like it takes a single number background(51); for a value between black and white, or 3 numbers (background(152,190,100); for RGB values.

//images below from left to right
background(0);
background(0,255,0);
background(255,255,0)

🧠CHALLENGE🧠
Can you change the background to yellow?

Here are some other fun functions to play with in your sketch.

fill()
noFill()
stroke()
brightness()
color()
hue()
saturation()

🧠CHALLENGE🧠
Can you change the fill and stroke of the ellipse?

Understanding the Draw Loop

Remember the draw loop runs continuously so you can build interactive functionality.

Lets take advantage of this to see other ways we can get our ellipse on the screen to move. You can think about your sketch as one long algebra formula where substitutions can be made to the code. Most locations where you see () accept parameters that are specific to the function. Remember the "R,G,B" number values we added as parameters to our background() function. To make the ellipse move we can edit its x and y parameters. Let's look up what parameters the ellipse accepts. Looks like its syntax is ellipse(x-coord, y-coord, width, height) where each of those parameters is a number (a float to be more specific).

🧠CHALLENGE🧠
With the help of the exercises above, please create your own sketch. Consider changing the:

• colors,
• shape (ellipse, square, etc)
• time (second, month, minute, millis, hour, day)
• ADVANCED — Consider using a new function not referenced in this tutorial from the P5 Reference. For example use your mouse (mousePressed) to control the above elements of your script. Or consider adding text.

Abstractions

Abstraction is a crucial concept in programming that allows us to describe logic at a high level and simplify a program. In the context of code, an abstraction can help us create a general solution that can handle many specific cases, rather than writing out every possible scenario individually.

In a program an abstraction is meant to represent what a piece of that program does. For example: an ecologist’s program might have a representation (an abstraction) of “trees” that “grow”. Both a the tree and grow can be built in code are abstractions that will have to encapsulate what each does. A tree could have parameters for its tree-ness: a species, height, deciduous, average number of branches, while grow might include how something grows: fast, slow, or in rhythmic spurts. Now that “tree” and “grow” are abstracted out — they could be used elsewhere. “Grow” could be used on other parts of the program — like flowers and grass.

Abstractions are used all over programs. A racing game might have an abstraction for a “car” that “drives.” Or perhaps you could build a sketch with different “paint brushes” that can paint different colored “strokes”.
But before we get too complex, lets just start simple with our existing sketch.

Abstracting our circle’s position

Ok so lets say we have a circle that moves on the screen, but we want to create 5 circles that each get bigger and are each in a different place.

Code for box on the left:

function setup() {
  createCanvas(400, 400);
}

function draw() {
  background(255);
  fill(255, 0, 0);
  ellipse(100, second() * 3, 20);

Code for box on the right:

function setup() {
  createCanvas(400, 400);
}

function draw() {
  background(255);
  
  fill(255, 0, 0);
  ellipse(100, second() * 3, 20);
  ellipse(150, second() * 3, 30);
  ellipse(200, second() * 3, 40);
  ellipse(250, second() * 3, 50);
  ellipse(300, second() * 3, 60);
}

But once you get to 100, 1000 or more this gets very tedious. So abstraction can help us generalize the rule.

🧠CHALLENGE🧠

How can we draw the circle 5 times in a slightly increasing x location?

First we need a way to draw the circle 5 times. We can do this by making a for loop. This for loop will run all of the code inside of the brackets {}.
Try adding some additional shapes inside the for loop.

You’ll notice there’s still just 1 ellipse. Not 5. Why?

for (let i = 0; i < 5; i++) {
    ellipse(100, second()*3, 20);
  }

Well, this for loop will run until its condition is met. The for loops condition is here: (let i = 0; i < 5; i++).

• This means a variable i will start at 0 when the for loop begins, and
• i++ means each next time the loop runs i will increase by 1 each time until this statement is met: i < 5.

Since every time the for loop is run, i gets greater by 1, e.g. i=1, 2, 3, 4, 5 then we can multiple this by our the x position of our ellipse.

So if a circle is structured like this ellipse(x, y, radius); then if we want to have the x change location for each we just need to increment the size each time the circle is drawn: ellipse(100+i, second()*3, 20);. But you’ll notice these are each only 1 pixel apart. So we can increase this by multiplying i: ellipse(100+i*50, second()*3, 20).

You could abstract this even further by creating a separate variable called ‘x’ inside of a for loop that increases by 1 each time:

function setup() {
  createCanvas(400, 400);
}

function draw() {
  background(255);
  fill(255, 0, 0);
  
  for (let i = 0; i < 5; i++) {
    let x = 100 + i * 50; 
    ellipse(x, second() * 3, 20);
  }
}

This is a basic demonstration of abstraction. We now have a variable that represents the x position and the functionality of how x changes abstracted into its own logic.

🧠CHALLENGE🧠

Change this sketch with your abstractions.

Ex:

• A color abstraction — Make the color change for each circle?
• A shape abstraction — Create different shapes each time?
• A movement abstraction — make each circle move differently based on your mouse position?

You decide!

Or make an entirely new sketch that encapsulates an abstraction of something in the real world:

• Paintbrush: Changes color and stroke based on mouse movement.
• Staircase: Steps appear as you click upward.
• Tree: A fractal structure that grows more branches over time.
• Bouncing Ball: Changes direction and speed with clicks.
• Solar System: Planets orbit each other.
• Ripple Effect: Expanding circles from mouse clicks.
• Music Notes: Appear and move when keys are pressed.
• Growing Vine: Extends and wraps as you drag the mouse.
• Lightbulb: Adjusts brightness when clicked.

Tips and Tricks

• Example Sketches
  If you are looking for more examples — you can find some great things to try under File > Examples. This brings up a lot of various examples which you can play around with and learn how different bits of code are functioning.
• Commenting Code
  You can add comments to your code which don’t run and are stricly to help the readability of your code like this //these two slashes create a comment in your code.
• Print
  print() allows you to see the values of a parameter in the console. For example, if you add this print(second()); in the draw loop, you will see a continual update in the console of your second parameter. Give it a try!
• Use Giphy Capture
  For screen capturing gifs, Giphy Capture is great for macs.
  For more ways of creating gifs with windows and mac go here.

What is “programmable space”?

We’re imagining a “programmable space.” In a programmable space you create programs not in a computer or a screen, but rather in a physical space (a room, a street, a city).

In 2017, I was introduced to Dynamicland, a project led by Bret Victor aimed at interacting with computers in a tangible and communal way. Their pitch: the environment itself is the computer. This flipped the typical dynamic of using computers. Instead of typing in isolation, groups of people could work together on programs that would then change the environment around them; projections and sounds would emerge in the room.

Dynamicland was an experimental computing environment where physical objects and spaces were embedded with sensors, cameras, and projectors to enable real-time, tangible interaction with digital information. Users could collaboratively manipulate and create interactive programs by moving and arranging physical items like pieces of paper, thus blending the digital and physical worlds. It’s an amazing vision!

Programmable space, a failed vision?

Unfortunately, Dynamicland failed to gain funding and closed.

I wonder whether it was just ahead of its time. Was the technology struggling to catch up? When it first opened, Dynamicland required writing the programs in code. That meant it was painstaking for most folks (non-engineers) to really use it.

To enable programmable space, we’ll need even easier ways to program.

To enable programmable space, then we’ll need even easier ways to program. That means no writing code and no required software engineering degree. It should be easy for non-engineers, for designers, for the humanities, for artists, for kids to program.

Dynamicland may have emerged at time when the technology for programmable space wasn’t ready, however it seems the technology has finally caught up.

I recently saw the Dynamicland website says there’s a reboot coming summer 2024! 👀

AI → Programmable Space

The multi-modal AI technology emerging today seems like just the thing to make programmable spaces a viable reality.

Multimodal AI can understand and combine different types of information, like speech, gesture, imagery, all at once. It’s more akin to how people interpret information from all kinds of context. You might point at something and say, “Hey what’s this?” Because you’re pointing at it, I know that “this” object is what you’re asking about. Multimodal AI is about that same kind of reasoning from multiple points of context.

This was an experiment we did at Spatial Pixel where you can point and ask about objects in the environment to have conversations with an AI that understands what you’re asking about.

Instead of having to write code, we can speak instructions in natural language. Instead of having to interact with an obscure gaming environment, you could draw a shape with a pencil and assign rules to it by speaking. A programmable space will need to leverage the way people naturally interact with their environments: pointing at things, moving them around, rather than writing chunks of code.

Here are more experiments we did towards programmable spaces. It’s a way of creating interactive programs and sketches that adapt to what you say, how you move, and the environment around you.

I really dislike being on a computer, but this was fun. You end up:

• interacting more with people instead of your computer.
• interacting more with your environment instead of your computer.
• interacting more with your body than your computer.

And because you’re making the programs in a more tangible way — with your voice, your body, and objects in the environment — perhaps it makes you a bit more present. I’d like the future of computing to be more inherently collective, tangible and bodily.

Real world AI will get better

Multimodal AI has been a major investment by Google, Meta, OpenAI, and others. Major companies like NVIDIA are investing in “physical AI”. Physical AI means using devices like sensors or robots that leverage AI to understand and react to an environment.

We’re glad to see investment in technologies that interact with the real world. These will likely only get better, making programmable spaces more viable. But as these technologies develop, it will be important that artists, designers, and people outside of Big Tech shape this technology so that it’s designed for human aims, not just capitalist ones.

Programmable Space Lab

With the rise of AI, we’d like to see programmable spaces and academic labs that further explore this kind of computing in the real world, and ensure it’s done in ways that everyday people can create programs with more intuitive means like natural language and drawing.

We like the spirit of Dynamicland: the communal creative expression. We like the idea of co-creation and we think the technology is ripe for it. We’ve been thinking about how programs could adapt to be more environmental allowing us to more naturally reference the world around us.

And what more would we do with programmable spaces? Well, nearly everything we do on computers today, but in a way that allows for more collective creation and more presence in the physical world. I’d be curious to see how a programmable space would change what we create and how we relate to each other. Would it make us more creative? More cooperative? Maybe most importantly, more present in this world and life we live?

Programmable Space was originally published in Spatial Pixel on Medium, where people are continuing the conversation by highlighting and responding to this story.

Forget code editors for Python, Javascript, and computer code. What about code editing in natural language?

TLDR: This is the beginning of a research and design exercise for creating a natural language programming environment. My partner, William and I’ve only just started to experiment with these ideas: here’s a bare bones one for P5, and one for physical space. Now we’re looking for topics related to code editors, visual programming, and programming languages that can spur ideas to help in this journey. If you have references you think might help us, please send them to us!

1. Rise in Natural Language Programming.
2. Prompting Isn’t Enough.
3. What to Borrow from Code Editors.
4. Emerging Experiments in Natural Language Programming
5. Provocations for Natural Language Programming.

1 — Rise in Natural Language Programming (NLP)

If you write code you’ve probably spent a decent amount of time in a Code Editor or an IDE (Integrated Development Environment). Code editors like Visual Studio Code help developers write, edit, and test code. But as we see the emergence of “natural language programming” where we can write executable programs with everyday language, will we still need a programming environment for natural language? I think so.

Natural Language Programming, related but not to be confused with Natural Language Processing, is a way of programming with natural-language sentences, like writing in English.

A very simple NLP: “Can you turn off the bedroom light off after 10 o’clock?”

What makes this a program and not just plain english? It becomes a program when it moves beyond a description and becomes something a computer can execute. Its a set of instructions meant to be computed, calculated, decided on or carried out.

Last summer my partner, William Martin and I taught a course in the new Computational Design program at Columbia University GSAPP. For the first time, we used natural language (ChatGPT) to write programs rather than starting with raw code. Typically in academic programs, we have to dedicate a decent amount of time to up-skilling students in programming. In the past learning the ins and outs of Python or Javascript syntax was a prerequisite to really designing anything that leveraged computation.

However, in this go-round, students were able to develop complete websites, totally bespoke to their ideas, relying much more heavily on Natural Language Programming. And for as much as ChatGPT “did their homework” developing their website code, so much of these designs were inherently theirs. Students still had to design the website, its functionality; spend days prompting to get useful responses; and struggle through actual code to develop meaningful programs. It was ultimately their ideas, their designs, and their visions.

In all fairness NLP is not that good yet. Students still needed a good technical understanding of Javascript syntax to string these websites together.

But that’s changing.

Recently Jensen Huang, CEO of NVIDIA, has predicted the death of coding. He argued, “It is our job to create computing technology such that nobody has to program and that the programming language is human. Everybody in the world is now a programmer. This is the miracle of AI.” [source][video]

Programming Isn’t Going Away

So while coding may be going away, programming is not. There are useful ways of thinking that someone develops when they learn to code. They learn to think systematically about how things are connected. They learn concepts like recursion, and learn how the language itself can impact the capabilities of a program. Those ways of thinking will still be important to develop. But a lot of the nuances of coding keeps us too deep in the technical weeds and prevent us from thinking more strategically and conceptually. Natural language programming allows us to dedicate our brain power at a higher level to the problems that matter.

Today, most people aren’t writing sophisticated programs with ChatGPT, but I do believe that will come. A wave of “programming” literacy is coming via natural language.

Today, only 2.54% of the global workforce are software engineers. [source] In 1820 only 12% of the world was able to read and write, but today 87% can read and write. [source] Like the steep incline of literacy following the Industrial Revolution, will most people be programming literate 200 years from now? Or perhaps even sooner? Today so much control of our world is only achievable by this tiny % of people who are software engineers. I welcome a future where programming is far more accessible and more people can write programs specific to their own interests and domains.

2 — Prompting Isn’t Enough

Great so most people in the future will be programming and doing so in natural language. So does that mean we’re all just going to ChatGPT our programs into existence? Not exactly. Language is very useful for thinking, but writing prompts is hardly programming.

As folks who program regularly will tell you, programming is not about coding. It’s a way of thinking. My partner recently pointed out that when software engineers discuss problems with one another, they rarely talk about code. They speak more abstractly about the structure of the system and its behavior. They consider how to structure the semantics of the program and how that will translate into the program’s capabilities.

Programming teaches you to think holistically about the system grappling with scale, recursion, network affordances, and emergent behaviors. It teaches problem solving, forcing someone to think critically, logically, and about operational flow.

Prompting in contrast to programming today isn’t always the easiest way to think programmatically.

There’s a danger that as we move towards prompting that we’ll rely too heavily on computers to think on our behalf. Prompting is more about a wait and see what the computer does that’s more reactive, vs programming which is more about designing, orchestrating, thinking and creating. Prompting also doesn’t allow you to build the holistic system. You’re only ever working on segments. And you must bring those segments to a different environment to build anything more complex.

But there’s a great opportunity if we leverage natural language’s ease and flexibility with the more systemic capabilities of programming. My hope and aim is to help contribute to a best of both worlds scenario where we harness the capabilities of both to help people think at a higher level.

Why we need a programming environment!

But why do we need a programming environment if AI’s going to do all of the work anyway? Even if AI is going to do all the program writing on our behalf, we’ll still need a way to communicate with it, clarify our intentions, and hopefully design programs together with AI.

We need an environment better than a chatbot to write natural language programs.

My hope is that programming literacy is not about having AI think for you, but that it empowers many more people to think creatively in terms of holistic systems and interconnected programs. I’m bullish that we’ll need an environment better than a chatbot, beyond just language to help us become sophisticated programmers.

3 — What to Borrow from Code Editors.

Luckily we’re not starting from scratch. There’s a great lineage of programming environments. Integrated development environments include a place to make, see, and check your program. Each of these capabilities is useful for an NLP.

1. Make. . . . . . . . . .[Code Editor]
   This is where you author and design a program. Where you actually write the code. It usually includes some form of syntax highlighting. This might also include tools like auto-complete to improve the ease of writing code. It also might include higher level maps that help you author the hierarchy and components of your program.
2. See . . . . . . . . . . .[Playground]
   A way to run the programs to see how they behave, so you can actively test and iterate on your program. Some environments include a “playground” where you can “play” a version of your program.
3. Check . . . . . . . . .[De-bugging]
   Tools for testing and understanding bugs and errors, including things like error logs.

4 — Emerging Experiments in Natural Language Programming

A Showcase of NLP-First Programming

In 2021 OpenAI published CodeX, an environment for natural language programming. Today the programming capabilities you access in ChatGPT are from this CodeX model. This environment demonstrates the ability to write a longer structured program by iteratively providing natural language instructions. I’m really encouraged by this environment. However, it also shows there’s definitely a ways to go to make something that can write sophisticated programs and that anyone can use.

NLP Environments Aren’t There Yet

Lacks the Natural Language Program — In the CodeX environment you never see more than a single prompt at a time which really limits your ability to control and design the program structure. What I’d consider your program code (all of the iterative prompts provided) isn’t viewable, scannable, editable. You’re always prompting one at a time one or two sentences. How can this become a designed and structured part of the interface?

No Referencing Between Contexts — Additionally this environment makes it difficult to understand how the contexts interact. How does the prompt impacts portions of the formal code on the right? Where is “animate” in the formal code? And how does my prompt or elements in the formal code impact the behavior in the canvas (where the game is running)? Its hard to understand really how they all relate to one another beyond a general “they seem to relate”. In an environment that supports natural language which is inherently more abstract, it will be more important to be able to understand the relationships across these contexts.

Other Emerging NLP Trends

Engineer-First NLP — Since CodeX, there’s been an explosion in AI coding Copilots: from Github Copilot, Amazon AI Code Generator, and Sourcegraph. These copilots allow you to type prompts in natural language, but they still return responses in formal code. For the time being, you’ll still need to be a software engineer or at least rather technical to use them.

Novice Programming-NLP — Meanwhile, gaming companies like Roblox an Unity have started to advertise their aims of creating natural language programming capabilities natively inside of their development environments for novice programmers.

Designer-first-NLP — Designer-first AI tools like Draw-A-UI could point to how future natural language environments could evolve in a way that leverages more native design skills like drawing. Here someone can draw shapes on a canvas and by adding labels to the elements the environment can intuit the program functionality.

5 —Provocations for Natural Language Programming.

If we want a programming environment for natural language, it’ll need to be honed to the affordances of natural language. So how is natural language different from a formal language?

Ambiguity

Sequence Ambiguity — Today its hard to understand the order and sequence of natural language programs. To develop a complex program through prompt engineering, you’d have pages worth of instructions making it difficult to follow operational flow. It gets really muddy. A development environment should help reveal the inherent structure and sequence of the program. Perhaps like some visual programming environments attempt to do? This could facilitate a programmer’s ability to navigate and design that higher level structure of the program.

Syntax Ambiguity — Natural language in contrast to structured language programming is far more flexible and ambiguous. It relies more heavily on the contextual semantics of the language and even the context of where and when the program takes place.

So what might the syntax highlighting be in natural language? Perhaps its actually useful to reveal the grammar through highlighting each word class? Verbs are similar to functions and methods, whereas nouns are more like variables and objects.

But syntax highlighting that follows the grammar of natural language doesn’t quite feel useful because so much of the meaning is determined by the entire sentence from the semantics in the context of the whole sentence. Perhaps more appropriate than syntax highlighting would be a semantic highlighting.

But within natural language’s ambiguity, perhaps even more important is than highlighting is that in natural language programming, AI interprets intent, whereas in formal language the computer follows explicit instructions. , it’s more important to be able to continually control, check and account for that hidden context. I can give an abstract command like “make me something pretty” and AI will have to interpret what “pretty” is. Even for something that seems more concrete like “draw a line” the AI is still interpreting what I mean by “a line” and what I mean by “draw”.

In a natural language environment, it becomes more important to reveal what the program is inferring. We should make it easier to specify our intent by making the design and control its inference more explicit.

Formal language: follows explicit instructions

Natural language: interprets ambiguous intent

Inspecting hidden assumptions— This means future programs will need to be able to navigate between abstract and specific more quickly. Language is especially abstract in contrast to programs like C# which are highly specific. We’ll need to create better ways to help NL programmers navigate levels of specificity and abstraction. Consider a simple instruction for a robot “bring the cup downstairs” and how many assumptions (mini-decisions) the program would have to make. This designer of this program likely wants the ability to control more finite parameters of the program buried in the assumptions of that program. This might warrant an interface that allows you to zoom into levels of specificity within the text.

This article has gotten so long that I’ve decided to create a separate one to to focus solely on mock-ups of what a natural language programming environment could be. That way I can explore more of these little ideas, and focus on the design itself.

Hopefully you have ideas too! I’m looking for more.

Hey, you! Thanks for reading.
If you’re a researcher, designer, or just interested in natural language programming, please reach out! I’m actively looking for projects, case studies, research, and ideas for what a natural language code editor should be. If you have anything to share it’s much appreciated!

We Need a Natural Language Programming Environment! was originally published in Spatial Pixel on Medium, where people are continuing the conversation by highlighting and responding to this story.

Exploration of how bodies can create more natural interfaces

Sections of this article:

1. Precedents — Hands as controllers
2. Why — Why do we need a hand keyboard?
3. Natural Language Programming — The body in natural language.

1 — Precedents — Hands as controllers

My Student Gave Me This Idea

Might I start by saying, this idea is not wholly my own, but rather a Frankenstein of ideas stolen from many creative people who’ve built up these ideas around me. Most importantly, I’ve had a student in my Physical Computing course in the Urban Technology program this semester who has been very keen on developing screen-less remotes by using key presses on the fingers, which planted this idea in my brain. He’s been focusing on this interaction as a sort of universal remote. You can see his project here. He also pointed out this technology called Gest which was another point of inspiration.

I‘ve been inspired by so many projects, some which have shown the potential for digital interactions through more natural ways of interacting with the hands, others which leverage computer vision and pose detection, such as in the image above. Projects such as Google ATAP’s Soli pictured below show the potential for more natural microgetures without physically interacting with any hardware.

Many tech companies have this same fascination and there are numerous patents showing XR interfaces on top of the hands such as the Apple patent drawing below.

If you need anymore convincing about why our hands are so special, just take a look at Bret Victor’s useful rant about hands in his article on the future of interaction design.

It got me to thinking about what else we can do with hands?

2 — Why do we need a hand keyboard?

FAQ

Q — Why don’t you just bring a phone or computer?
A —I’ve been fascinated by the idea of not needing to carry a phone or a computer around to do computing tasks. I don’t want to bring my clunky keyboard. And I don’t really like the idea of carrying around a bunch of devices: computers, phones, or otherwise. What if we could rely on environmental computers and cloud-based software for interactions?

Q — Why don’t you just use speech?
A — Speech is great, but most computer interactions and conversations are private. Either you’re looking something up, or texting someone. I’d prefer not to be that person at the cafe saying “Hey Siri, can you text my mom and ask her when we’re all getting together this year.”

Q — Why don’t you just use sign language?
A — That’s a great idea. The idea didn’t come across my mind until I wrote this article. Since I don’t sign, I’m not sure what the cons and benefits may be between these two systems. I’d love to hear from anyone more familiar here. I assume signing may not be as fast.

Q — Doesn’t this hand keyboard thing exist already?
A — That’s what I was wondering. Apparently it does, and Samsung patented it. Amazing how quickly things get marked by large companies. But that’s a post for another time. In the mean time, I’ll let you enjoy their hilariously passive aggressive patent drawing below.

At last you can harass your coworkers while away from your computer.

But in all seriousness, this is a great idea.

FUN FACT:

Did you know the QWERTY keyboard was invented in 1867 and gains its logic from the printing press where typesetters manually set individual characters?

“Ok, you convinced me. Now I want a hand keyboard.

But how would I make one?” — You

It’s technically achievable.

The above is just a rough demo of how to use the position of the hand to recognize letters pressed. It is by no means a working keyboard. If you were building out a more robust system you’d instead use the skeleton structure of the hand. This is what we’re currently working on [end of this post] and what we hope to include in future posts along the way. If you’d like to see how to make this above simpler demo, its quite easy with Teachable Machine. You can find my tutorial for it here.

Hands are great symbols.

The hands are great symbols for, well, tons of things.

When I was working on this lecture, I spent a bunch of time looking at these ancient drawings from witchcraft and alchemy.

The more I perused the depth of the internet, I came across more and more examples where the hand is used as a symbol or for signs. It’s used for sign language, Palmistry, Mudras, cultural gestures like thumbs up, as a mnemonic device. The list could go on. So because we can map so many different recognizable meanings onto the hands, they become a really relevant interface for real world interactions.

This project is not really about turning your hand into a keyboard. Rather it’s an exploration of how to leverage our hands directly to interact with computers.

It’s not just about typing, its about all the ways we use computers with our hands.

Why should mice and keyboards confine our creativity? We should be able to tap into the flexibility of both our hands and of language to create our own controls for computers.

3 — Natural Language Programming

One idea we’ve been working on at Spatial Pixel is how to speak programs into the world around you like this working demo.

In the future, we’ll be able to speak the changes we want in an interface and even speak changes we want to make to change how our hands behave as inputs, to swap our hands from a keyboard, to a paintbrush, or from a paintbrush to a puppet, or from a puppet to a music control.

This “Speech to Hand” project above is about turning your hand into an interface. But its still in concept mode because there’s a lot to build! Each of these demos on the right is a working program with hand tracking. However, we’re still actively working on the speech functionality to translate programmed actions to the hand.

This includes inferring:

1. body identifier— what part of the hand the speaker is referring to: finger joints, hand, position of the whole hand or what part of each component (top, bottom, left, right)
2. task identifier — what action the speaker is trying to map to the hand: pressing, tracking distance between fingers, position of the hand, shape or structure of hand which must be inferred from a higher level command such as “turn my hand into a paintbrush”. We’re still trying to sort out the best ways to break down and categorize motions with the hand, but we did come across this amazing paper from the Yale Grab Lab.

If you have any ideas on how we might categorize tasks in terms of motions, please send your ideas, papers, or projects our way! There’s a lot we have to learn on converting language into actions that can be mapped to the hands.

By the way, I typed this whole article on my hand keyboard while lying under the canopy of a beautiful tree.

No, that was a lie.

I only made a demo version with a few letters.

But wouldn’t that have been cool?

Thanks for reading!

Now go type!

Turning Your Hand Into a Keyboard was originally published in Spatial Pixel on Medium, where people are continuing the conversation by highlighting and responding to this story.