Hi, I’m Julia, The Met’s inaugural Technologist in Residence. Missed the intro? Read here!

At the end of their visit, guests frequently ask our volunteers what to do next.

“Where is a good place to eat nearby?”
“What other art galleries could I check out?”
“Is there a good place to rest within walking distance?”

We wondered if we could use AI to make suggestions that match the art visitors saw at The Met:

• If a visitor saw 19th-century American bronze sculptures, could we guide them toward Central Park’s bronze monuments?
• If they were drawn to Japanese Oribe pottery, could we highlight an authentic tea house in Manhattan?

By extending The Met beyond our walls we create spaces for meaningful post-visit reflection, and allow visitors to experience our art in new contexts.

Prototype: Find The Met in NYC

We built a prototype that uses AI to connect our visitors’ favorite artworks to NYC.

Here is how it works:

1. Visitors simply snap a photo of an artwork. We identify the object by sending the photo to our semantic search endpoint, which matches it against The Met’s collection using AI-generated image embeddings.

2. We send all available artwork metadata, like the work’s title, artist and description to an AI model (Google Gemini) with the following prompt:

  generateSuggestionsPrompt: (metadata) =>
    `A visitor is standing in front of this artwork:

${metadata}

Suggest exactly 3 ways they can extend this experience BEYOND the museum walls. Think creatively — connections can be art-historical (same artist, era, technique, culture) but also playful and unexpected.

Types you may use:
- "place" — a specific park, garden, landmark, or public space in NYC
- "restaurant" — a specific restaurant, café, or bar in NYC connected to the artwork's culture or era
- "walk" — a specific walking route or neighborhood stroll in NYC
- "book" — a specific book (include author)
- "concert" — a specific venue or recurring concert series in NYC
- "shop" — a specific store, studio, or market tied to the artwork's culture in NYC

For each suggestion return:
- "title": a short, inviting activity phrase that includes the specific name. Examples: "Stroll through Prospect Park".
- "type": one of the types listed above (used internally for linking, not displayed to the visitor)
- "description": Two very short, clear, concise sentences, max 9 words each. The first must state the specific connection to THIS artwork (e.g. "Courbet painted this model at this exact café"). The second tells the visitor what to expect there.
- "neighborhood": for NYC places only, the neighborhood name (e.g. "Upper East Side", "Chinatown"). null for non-physical types.
- "searchQuery": a Google-friendly search string to find this place/thing (e.g. "Conservatory Garden Central Park NYC")
- "photoSearchQuery": a Google Places search string to find a scenic photo for this card. For physical NYC places, this should be the place name + neighborhood. For non-physical items (books, films, music), this MUST be the proper name of a REAL, FAMOUS, SPECIFIC place (a landmark, museum, garden, café, or building) that evokes the setting or mood of the work. Examples: "Meiji Shrine Tokyo", "Café de Flore Paris", "Giverny Monet garden", "Uffizi Gallery Florence", "Katsura Imperial Villa Kyoto". NEVER use generic descriptive phrases like "Victorian parlor interior" or "moody jazz bar" — these are not findable places. Always use a proper noun that exists on Google Maps.
};

Gemini returns three vibe-matched activity recommendations nearby, like local restaurants, museums, parks, or concerts.

Occasionally, AI suggested places that are temporarily closed, or ones that cannot be found on Google Maps. Enabling search grounding made the results more reliable. However, it also increased the AI’s generation delay.

The below ceramic lion suggests visitors to “visit the Central Park Zoo”, “read Moby-Dick by Herman Melville”, and “browse the ceramics at Fishs Eddy”.

[
    {
      "title": "Visit the Central Park Zoo",
      "type": "place",
      "description": "Observe real lions mirroring Bell's majestic earthenware figure. Enjoy a quiet walk among exotic urban
      wildlife.",
      "searchQuery": "Central Park Zoo lions",
      "photoSearchQuery": "Central Park Zoo Upper East Side",
      "neighborhood": "Upper East Side"
    },
    {
      "title": "Read Moby-Dick by Herman Melville",
      "type": "book",
      "description": "Melville captures the nineteenth-century American spirit like Bell. Explore this epic tale of man and
      nature.",
      "searchQuery": "Moby-Dick Herman Melville book",
      "photoSearchQuery": "New Bedford Whaling Museum Massachusetts",
      "neighborhood": null
    },
    {
      "title": "Browse the antique ceramics at Fishs Eddy",
      "type": "shop",
      "description": "Discover decorative earthenware pieces reflecting Bell's intricate style. Find unique, vintage-inspired
      tabletop treasures here.",
      "searchQuery": "Fishs Eddy NYC",
      "photoSearchQuery": "Fishs Eddy Flatiron District",
      "neighborhood": "Flatiron District"
    }
  ]

3. Lastly, we used the Google Places API to find a fitting image for each of the three suggestions.

This prototype showed that Gemini’s understanding of the world is broad and up-to-date enough to result in really surprising, delightful and interesting suggestions that both tourists and locals enjoyed.

In our last set of experiments we are using AI to create a customized post-visit reflection. Find out more here!

Finding The Met to New York City was originally published in EmTech at The Met on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi, I’m Julia, The Met’s inaugural Technologist in Residence. Missed the intro? Read here!

Art at The Met reflects the lives of people through time and culture. To make our collection more approachable and relatable, we were curious if AI can connect our visitor’s everyday lives to our historical collection.

Connecting everyday life to The Met

We started by building a prototype that uses AI to turn everyday photos into artworks from The Met. We’ve seen some great examples of this idea in the wild, such as the Cleveland Museum of Art’s “Share Your View” project.

However, visual similarity between the visitor’s picutres and Met artworks only works well when searching for content that our collection actually contains. For example, a picture of a sunsets yields great results:

Visual similarity does not work well when searching for “abstract” images. For example, visual similarity cannot conceptually connect “Airpods” to the concept of “music” or “listening”. The search does not result in any meaningful connection between modern and past life.

[Editor’s note: careful readers might be aware that we do have an active design and architecture collection, though we have yet to acquire any wireless headphones. Even if (when?) we do, we think this idea will still be valid; after all, you’re not using a search like this to “shop” our collection, but instead likely looking for ideas and inspiration]

We wondered whether a really fast AI model could customize the search queries of our nascent visual similarity API* in real time to improve the results.

*Editor’s note: this is a private, experimental API (for now).

AI should only search by pure visual similarity if the visitor’s image depicts something that could realistically exist in our collection. If the image is too abstract to exist at The Met (in the general sense, not as in the Modern Art sense), we want AI to search by meaning instead.

We created an AI prompt that first rates how abstract a visitor’s image is, and then generates a matching text description for the image. This text description is passed into our semantic search* endpoint our Digital Product team built, which returns the best matching artwork for that description.

*Unlike regular search, which only looks for exact word matches, semantic search understands the conceptual meaning of words.

You are a tour guide at The Metropolitan Museum of Art. A visitor shows you this image. You need to find 4 artworks in your collection that relate to it. 

Respond ONLY with valid JSON. Return:
1. "imaginative": (A number from 0-1) How much do you need to reimagine the subject to find it in an art museum?
   - LOW (0-0.3): The image looks  like something that could exist in this form in a museum. A person's portrait, a sunset, a horse, a garden. The image is the art itself, not a symbol of something else.
   - MEDIUM (0.3-0.7): The photo shows an object that exists historically but needs some abstraction of its function to get it's concept right. A matcha cup, makeup, pretzels packaging, a subway stop, a car. 
   - HIGH (0.7-1.0): The object is way too modern to exist in art. A laptop, a drone. You must reimagine the underlying human activity.

2. "label": A short, specific phrase (1-2 words) describing what the image shows.

3. "keywords": A list of 5-8 keywords optimized for image vector search against a museum collection. When imaginative number is LOW: Use keywords that describe what you see — visual details like colors, materials, poses, clothing, setting. When imaginative is MEDIUM: Use keywords that translate the object into what would exist in a museum. Focus on the object in context, not abstract visual properties. Example: 'A matcha cup' → keywords about tea bowls, ceramics, tea ceremony — NOT 'green, round, bowl'. When imaginative is HIGH: Forget the object. Think about what HUMAN EXPERIENCE it represents. Airpods → keywords about music, instruments, singing, performance. Laptop → keywords about reading, writing, scholars, study.
`

Below are some examples of images that range from very literal to abstract, the description an AI model (Google Gemini, in this case) generated for each image, and The Met artwork the image was paired with. The examples show that AI can customize search queries, so that an image of a forest is paired with a similar forest, while an image of a computer is paired with a depiction of a scholar at work.

A Camera that connects visitors’ daily lives to The Met

We built a prototype that automates this system:

1. A visitor takes a picture.
2. AI comes up with an appropriate search query in real time.
3. The top four artwork results are displayed alongside a short description.

Try the prototype yourself here!

Continue reading about our second prototype here, which connects pop-culture to The Met.

Connecting visitor’s daily lives to The Met was originally published in EmTech at The Met on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi, I’m Julia, The Met’s inaugural Technologist in Residence. Missed the intro? Read here!

The Met is experimenting with ways AI could help improve the on-site visitor experience. Our previous sprints tackled two big parts of the on-site visit: Navigation in the museum, and close-looking (helping visitors connect with specific artworks).

Our third sprint “The Met and the World” focuses on another important part, the moments right before and right after people’s visits to The Met. How can we help connect our museum to the world outside The Met’s walls?

Part 1: As visitors enter the museum

Visitors told us that when they can connect The Met’s art to their personal lives, the museum feels more approachable and relevant. We wondered if AI could help create those connections. We built two prototypes:

• A camera that connects visitors’ daily life to The Met
• Two prototypes that connect (pop) culture to The Met

Part 2: When visitors leave the museum

We love the idea of extending our visitors’ journeys beyond our museum walls. We wondered: As visitors leave The Met, can we use AI to help them reflect on the art they have seen? We built two prototypes:

• Connecting The Met to New York City
• AI for a personalized post-visit reflection

Main Findings

In previous sprints, we focused on how AI could be used to solve logistical issues like navigation and translation. This sprint showed that AI is just as valuable for sparking curiosity than it is for practical problem-solving: Visitors love playful prototypes that use AI for open-ended inspiration. At the same time, grounding artworks in familiar topics also helped dial down the pressure of exploring a massive museum.

We also found AI models’ understanding of the current world is often broad and up-to-date enough to result in really interesting connections of Met artworks to the current world, and the current world back to The Met.

Connecting The Met to the World with AI was originally published in EmTech at The Met on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi, I’m Julia, The Met’s inaugural Technologist in Residence. Missed the intro? Read here!

People love seeing data summaries of their behavior: Features like Spotify Wrapped feel really satisfying because they turn random data into a fun personal story.

Museums have also experimented with this kind of post-visit reflection. Some of the most prominent recent examples, such as the Cooper Hewitt Pen and ACMI’s Lens, allow visitors to tap and collect artworks during their journey. The physical devices then store and create a personalized collection for visitors to explore at home.

We wanted to add our own spin on this idea for visitors who don’t want to bring along a physical device during their visit.

Almost everyone takes pictures of the artworks they like. These photos are a good indicator of what caught a visitor’s eye. Images also often contain metadata like timestamps and geolocation, which allow us to re-trace their path pretty accurately. [Editor’s note: the inclusion of geolocation is also disclosed to visitors at upload time via their personal device, allowing for an easy opt-out if visitors didn’t feel comfortable sharing]

Could AI analyze visitor’s pictures to create a personalized reflection experience?

Post-visit Reflection Prototype

We built an interface that allows visitors to drop all their Met images from their camera roll. AI analyzes the images, discovers the hidden thematic threads connecting their journey, and transforms them into a personalized reflection experience.

Here is how it works:

First we generate emeddings (via the gemini-embeddings-2 model) to match visitor photos against our collection database and identify all artworks. This uses our internal semantic search API developed by our Digital Product team.

Once all art is identified, we then pull each artwork’s metadata (titles, artist names, gallery descriptions). Bundled with unidentified pictures, this information is sent to AI to find hidden patterns with the following prompt:

A visitor photographed these artworks during a trip to The Metropolitan Museum of Art:
${artworkList}

Create three distinct possible "threads" through what caught this visitor's eye.

A thread is a specific, visible, defensible pattern across at least three artworks.

Avoid:
- generic themes like "Beauty", "History", "Culture", "Power", "Nature", "Portraits", or "Religion"
- threads based only on department, date, geography, artist nationality, or broad art history categories
- poetic abstractions

Return ONLY a JSON object, no markdown fences:
{
  "threads": [
    {
      "title": "2-4 words, concrete and specific",
      "description": "one elegant sentence, under 28 words, explaining what connects the artworks",
      "examples": ["exact artwork title", "exact artwork title", "exact artwork title"]
    }
  ]
}

Return exactly 3 threads.

The model returns three distinct narratives “threads” that weave the visit together, in the form of a title and a short description for each.

For this initial prototype, we chose to present each thread in three parts:

• Similar Artworks: Recommending other similar Met artworks the visitor should check out during their next visit
• Met Resources: Linking to a related Met resource like an article or upcoming live event.
• External Context: Connecting the visitor’s journey to external events in NYC based on this prototype.

This is the last prototype from our Technologist Residency! I hope you enjoyed learning more about our ideas, our process, and our findings.

Reach out to us if you have any questions!

AI for Post-Visit Museum Reflections was originally published in EmTech at The Met on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi, I’m Julia, The Met’s inaugural Technologist in Residence. Missed the intro? Read here!

The Met’s collection is vast enough to cover almost any topic. Many of our previous prototypes make use of this vastness: We have often incorporated blank search-bars in prototypes where visitors can enter any topic they are interested in.

An empty search bar works well for visitors that have a concrete goal for their visit or search in mind. However, during user testing, we observed that for visitors who don’t know what they want to see, complete freedom of choice can cause decision paralysis.

“This map is too complicated for me. Just tell me where to go.”

We wondered if AI could help offer really approachable “jumping-off” points by connecting our art to things visitors already know and love:

Can AI connect a Broadway play to a custom tour? A music album to a gallery? A World Cup game to an artwork?

We created two prototypes. The first one connects specific books and movies to matching artworks at The Met, while the second applies the same idea to current events from “Coachella” to “Mother’s Day.”

Prototype 1: Books and Movies

This prototype bridges the aesthetic of a visitor’s favorite book or movie into artworks from The Met.

First, Visitors select their favorite book or movie. Using its general knowledge of the world, AI (Google Gemini) generates a few different phrases and descriptions about the book/movie.

For example, The Little Prince by Saint-Exupéry generates the following descriptions:

{
  "keywords": {
    "essence": "the starry night sky",
    "descriptions": [
      "the starry night sky — delicate watercolors of solitary figures standing on tiny, barren asteroids",
      "the starry night sky — whimsical astronomical charts mapping out scattered celestial bodies and distant planets",
      "the starry night sky — vast empty desert landscapes illuminated by a massive glowing canopy of stars",
      "the starry night sky — minimalist illustrations of a solitary rose growing beneath a vast cosmic expanse",
      "the starry night sky — dreamlike paintings of a golden-haired youth gazing upward into infinite dark blue space"
    ],
    "iconicObjects": [
      "rose under glass dome",
      "box containing a sheep",
      "tamed red fox"
    ],
    "backgroundColor": "#2b3a55",
    "title": "The Little Prince",
    "author": "Antoine de Saint-Exupéry"
  },

Each description is passed to our semantic search endpoint, which matches abstract text descriptions to artworks using AI.

For example the description “the starry night sky — whimsical astronomical charts mapping out scattered celestial bodies and distant planets” from The Little Prince returns these artworks:

All resulting artwork are displayed as a mood-board.

Visitors can also download and print their mood-boards as posters!

User testing

The prototype was a huge hit with our visitors, showing how connections to pop culture can make historical art immediately approachable and fun:

• “That is lots of fun, and really informative too”
• “Whaaat, wonderful!”

Curious? Test the prototype with your favorite book/movie here!

Appreciation Day

Beyond individual books and movies, we wanted to see if AI could connect art to the current day—literally.

We wondered: Can AI make a calendar of cultural events, like Lunar New Year, Mother’s Day, or Malcolm X Day?

In this prototype, AI picks a special event for each day. To ensure an interesting and diverse selection, we batch-generate a whole week in one request.

You are a creative researcher who finds holidays, appreciation days, and awareness weeks. Find 7 items — one for each day of the week ${fmt(mon)} through ${fmt(sun)}.
Include a good mix of: beloved traditions (e.g. "Mother's Day", "Valentine's Day"), cultural holidays (e.g. "Diwali", "Christmas"), cultural days ("First day of Black History Month", "Last day of Pride Month"), nature & animals (e.g. "Cherry Blossom Festival"). 

Rules:
- Exactly 7 items, one per day — no duplicates.
- Favor holidays that are culturally important and resonant.
- "date" must be the specific day in YYYY-MM-DD format.
- Search the web to verify real dates — do not guess.

Respond ONLY with a JSON array of 7 objects. No markdown, no code fences.
[{ "title": "...", "date": "...", "description": "...", "culture": "...", "color": "#..." }]

"title" = name of the holiday or appreciation day. Keep it concise.
"date" = the specific date in YYYY-MM-DD format.
"description" = a single, short sentence (under 8 words) saying what the day celebrates. Plain, simple language. Example: "Celebrating the joy of reading."
"culture" = if the holiday is tied to a specific culture or country, name it (e.g. "Mexican", "Japanese", "Irish"). If it's global or not culture-specific, use "global".
"color" = a hex accent color that feels inevitable for this specific holiday. Pick the color someone would instinctively associate with it. It can be anything — near-black navy for space, soft pink for Mother's Day, deep red for Valentine's Day, earthy green for nature, warm terracotta for food. Match the mood: dark, light, pastel, vivid, muted — whatever is most true to that event.

Each daily event is displayed as a card:

A short description of the event is passed into our semantic search endpoint. Unlike regular search, which only looks for exact word matches, semantic search understands the contextual meaning of words. It returns 30 artworks that match the event. AI reviews the top 30 candidates to make a final “top 5” selection. Each artwork is also accompanied by a brief explanation of why it was selected.

User testing showed that the prototype was especially successful when visitors personally resonated with an event: “Mother’s day! That’s our day.” Or “I’m a nursing student! I didn’t know today was International Nurse Day.”

Visitors also enjoyed seeing artworks in a new contexts, and appreciated the prototype’s ability to surface artworks they have never noticed before. ”I’ve been here a couple times, but seeing it in this way definitely opens my eyes to all the things I have missed before.”

Most importantly, this prototype achieved its goal of being highly accessible and non-intimidating. “Its a good amount — its not too much where it is overwhelming, it’s just the perfect amount”.

In our next series of experiments, we decided to flip our approach: Instead of using the outside world to guide people into The Met, we wanted to see if AI can extend the vibe of specific artworks into the city at large. Check out that post here.

Connecting Culture to The Met was originally published in EmTech at The Met on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi, I’m Julia, The Met’s inaugural Technologist in Residence. Missed the intro? Read here!

In conversations with members of our Family Programs team, including those that support our 81st Street Studio (a play space at The Met for families), we learned that many parents and caregivers are reluctant to engage with our collections on their own because they feel they don’t know enough to facilitate experiences for children.

We were curious to explore how AI could be used to make caregivers feel more confident.

The main challenge was coming up with digital interactives that actually engage kids with artworks instead of just keeping them fixed on a screen.

Editor’s note: as a parent of two small children who is pretty conservative about screen time, I didn’t want to create a situation where anyone felt forced to hand over their device to their kids. Sometimes a good bar to clear for an idea is as simple as, “would I do this?”

The solution was to focus on experiences that are solely facilitated by caregivers: the adult remains the only point of contact with the phone.

We built two prototypes that explored how AI could help support parents in museum environments.

Both prototypes showed that AI could be helpful because it can customize content for young visitors—and it can scale across our entire collection. This flexibility allows caregivers and children to explore any area they choose, offering support at their preferred depth without being restricted to a specific path.

“I-Spy” Prototype

Editor’s note: sorry for all of the interruptions. Setting this one up: this is a game I play with my kids at The Met (and on the way to and from the subway): “I spy with my little eye… a statue that’s angry!”

Our “I Spy” prototype suggests close-looking prompts to caregivers. The prototype also shows interesting contextual information for each prompt that the caregiver can then use as a scaffold to support the child’s learning.

Our first version was binary: there was always a “right” answer. That’s where the Education team came in. They suggested prompts that could be more open-ended, to engage the caregiver and kids in more discussion, and to prevent any “dead ends” if, for example, a kid keeps getting it “wrong.” Conveniently, this also lowered the potential for hallucinations or mis-steps from AI.

How it works:

1. Parents select a gallery or an artwork.
2. Gemini generates “I-Spy” prompts based on the artwork(s).
3. Caregivers read prompts out loud to children.
4. On the back of the cards, they can read about examples and helpful contextual information.

The prototype initially shows four “I Spy” questions using the prompt below. As visitors explore, new questions are generated to provide an endless supply of questions as families explore the gallery.

export const OBSERVATION_PROMPT = `You are designing a playful scavenger hunt for young children visiting a museum with caregivers. Look at this artwork and create 4 fun, SHORT observation prompts.

Artwork information:
{{artworkDesc}}

{{previousPrompts}}

Rules:
- Each prompt must be ONE short sentence — 10 words max
- Open-ended: no single right answer; invite wondering, feeling, and exploring
- About things ACTUALLY visible in the artwork (colors, shapes, animals, people, objects, patterns, moods)


You MUST write all prompts and examples in {{language}}.

For each prompt, also write an "example" for the caregiver — a specific fun fact, art-history detail, or context about this particular artwork. Be concrete and informative, 1–2 sentences.`;

Try out our prototype for yourself here!

What seemed like a simple idea turned out to be a big hit during user testing with eight families. Each family used the prototype as a scaffold to meet their own needs: some read all prompts and contextual information, while others split the prompts between family members.

All families liked the customizability and open-ended prompts:

“Kids don’t always wait for group tours; we tried that before, but it wasn’t that interesting. This [prototype] is more interactive, and we can select what info we want.”
“Kids get to use their imagination.”

The open-ended prompts made the prototype engaging for both children and parents across different age groups, making it a fun intergenerational activity.

“Story Builder” Prototype

The Story Builder is our second experiment aimed at supporting parents. The prototype helps caregivers tell educational and imaginative stories about artworks.

Kids and adults first get to collaboratively build the scaffold of the story with open-ended questions. The questions are AI generated based on the artwork’s metadata and adjust according to previous answers.

Using the children’s input, Gemini then generates a short, playful story that introduces the artwork’s setting, function, and content.

Interestingly, this prototype resonated more with parents than children. User testing showed that a text-only narrative struggled to hold the attention of kids aged 4–8, who prefer active, hands-on participation like our “I Spy” prototype—or the very hands-on 81st Street Studio.

In the next article, we will discuss if AI can be used as a non-interpretive experience.

AI to empower caregivers was originally published in EmTech at The Met on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi, I’m Julia, The Met’s inaugural Technologist in Residence. Missed the intro? Read here!

In our “Close Looking” Sprint, we built a range of prototypes. Some prototypes relied entirely on AI, like prototypes that mass-translate thousands of wall-labels, or ones that pull information from hundreds of Met publications.

Other prototypes only use AI as a way to accelerate our creation process. During our experimentations, we found three unexpected ways AI supported our R&D.

1. AI for Rapid Prototyping
2. AI to Accelerate Design
3. AI to Generate Sample Content

1: AI for Rapid Prototyping

AI helped us build faster. We were able to go from idea to prototypes in a few hours rather than days. These working demos were extremely helpful for user-testing with visitors.

For example, the first version of the “Circling Details” prototype was built entirely in the no-code Google AI Studio. We demoed the initial version with visitors and staff, which quickly helped us understand if the prototype is worth exploring further. (It was! Find out more about it here.)

2: AI to Accelerate Design

All of our initial prototypes were proofs of concept designed to test whether an idea had potential. The visual design was often secondary at this stage. Yet, visitors still need a clean intuitive interface to test our prototype effectively. AI helped us create quick UIs, letting us focus on the ideas while gathering better feedback.

3: AI to Generate Sample Content

AI made it easy to mock up initial content for testing that gives visitors something real to interact with before we committed to hand-crafting final assets.

For example, we built a “Zoom In & Out” prototype lets visitors discover artworks across similar origins, eras, and cultures. While the concept focuses purely on a new interface rather than AI-generated content, we used AI to quickly generate baseline content for early testing so visitors could engage with a live experience that feels more complete.

This, of course, carries a lot of risk in museum settings. In this case, we bounded the generated content with real Met scholarship and created the text in a way where it was editable if we did want to test more widely. Consider with caution!

Similarly, to test game mechanics early on, we used AI to generate over 100 artwork detail cutouts for our “Scavenger Hunt” prototype. Having “good enough” assets right away allowed us to validate the core gameplay experience before spending time creating high-fidelity, hand-cut versions.

Ultimately, using AI to prototype new in-gallery interactives wasn’t just about building new AI-powered features. It fundamentally changed how quickly we could build, and test ideas with our visitors.

Stay tuned for updates on our final residency sprint coming soon!

How we used AI for R&D was originally published in EmTech at The Met on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi, I’m Julia, The Met’s inaugural Technologist in Residence. Missed the intro? Read here!

In museums like The Met, most casual, drop-in visitors learn about artworks through wall labels or audio guides. This information is often only available in English, and in a relatively academic tone. Yet, many of our visitors are international tourists or families with children.

Our Close Looking Sprint revealed that AI has the potential to customize existing in-gallery content at scale — into formats that meet our visitors where they are.

In my opinion, this use case yields such good results that it could be realistically implemented at a large scale with current technology.

We built and tested two prototypes:

• Custom Wall Labels: We experimented with translating wall labels into different styles and languages.
• Room Intro: We created personalized audio overviews with different lengths, languages, and focus interests.

Prototype 1: Custom Wall Labels

As a non-native English speaker without an art history background myself, wall labels at The Met are often not easy to understand for me. This can make me feel out of place at museums. Below is an example of a wall label, with words I don’t understand highlighted in yellow.

We built a prototype that uses AI to translate wall labels into different formats and languages: a mode that adds definitions to difficult words, or a more kid-friendly version for example.

Here is how it works:

1. Visitors select a mode and language, and take an image of an artwork.
2. Using our embeddings search API (developed by Derek Au from our Digital Product team), we identify the artwork.
3. The label of the artwork is displayed, translated into the mode the visitor selected.

All wall labels are pre-generated with prompts like the one below:

export function translationPrompt(label: string, lang: Language): string {
  return `Translate the following museum wall label into ${lang}. Keep all the information exactly the same, and make the translation as close to the original meaning and tone as possible. ${NO_PREAMBLE}\n\nWall label:\n${label}`;
}

Our initial user testers, international students with English as a second language, found this tool quick, easy, and useful. It made existing content more accessible and enjoyable for them.
In a public user-testing, an overwhelming majority of our user-testers also agreed that using AI for translation and accessibility feels reasonable for museums.

Try our our wall-label translation prototype here!

However, internal conversations with Edmon Zhou, a staff member of our Asian Art department, brought up issues with accuracy and tone. AI tends to add descriptors not found in the original labels. For instance, simplifying “Kangxi Emperor” to “a powerful Chinese emperor” for a kid-friendly wall label is technically true, yet the word “powerful” was not mentioned in the original label. While curatorial review of each AI-generated wall label would ensure accuracy, manually checking labels for 36,000 objects (not to mention 150+ languages) is too resource-intensive and would defeat the purpose of using AI to automate a task that every museum lacks the manpower to handle.

An open question is whether AI prompts for wall label translation can be fine-tuned enough to avoid manual editing without risking reliability issues.

Prototype 2: Room Intro

Our second, more experimental prototype yielded similar results. The “Room Intro” prototype allows visitors to set language, length, tone, and focus, to generate a custom audio overviews.

The generated audio is short, accessible, and surface-level, aimed at introducing the artworks in that room.

Below you can find an example of an AI-generated audio-guide. This one is in German, and focused on ‘visual aspects’ for Room 958.

This prototype generates the script from metadata rather than just translating existing guides. Therefore the experiment raised internal questions about its accuracy, quality, and who can be held accountable for the content of live-generated audio.

At the same time, public user-testing with visitors showed that our guests embraced the ability to personalize these guides to their needs and interests. Roughly 1/3 of our user-testers picked a language other than English.

In Conclusion

Both prototypes show that visitors welcome content that better meets their access needs. AI is one of the possible approaches to solve this issue because it can provide a level of personalization that is otherwise impossible to scale.

In the next article, we will discuss how we experimented with using AI to empower caregivers.

AI to customize content at scale was originally published in EmTech at The Met on Medium, where people are continuing the conversation by highlighting and responding to this story.

Technologist in Residence, Pt. 9

Hi, I’m Julia, The Met’s inaugural Technologist in Residence. Missed the intro? Read here!

Most of our prototypes use AI to communicate some educational information to visitors. For example, our wall label prototype or our Circling Details prototype both use AI to translate The Met’s research into more approachable formats. For our visitors and staff, using AI as a “source of truth” can raise questions around reliability.

We wondered: Could AI support visitors without actually interpreting the art?

The “Slow Looking” Prototype

Slow looking is a practice that prioritizes observation over interpretation by encouraging visitors to spend time looking at a single artwork. Slow looking can be facilitated through guided prompts, for example the ones in this MoMA guide, or by inviting visitors into a dedicated window of time, similar to this NYT’s 10-minute challenge.

Could we facilitate slow looking with “non-interpretive” AI?

We built a prototype that creates slow-looking audio guides. Each guide is a 4–5 minute experience with open-ended narration customized to a particular artwork using the artwork’s high-res open-access image, its available metadata, and Gemini. The guide doesn’t aim to provide any contextual or educational information. Instead it encourages visitors to come up with their own interpretations.

Here is a 1-minute snippet of a slow-looking guide of Cottage Children (The Wood Gatherers)(50.145.17).

Multimodal AI Models

This prototype takes a very simple technical approach: it passes the open-access artwork directly into Gemini 3.1 Pro in a single request, without any extra cropping, OCR, or computer-vision prep.

Each slow-looking guide follows a similar structure, using this prompt [slightly shortened for the sake of this article]:

"Act as an interactive museum guide for a slow-looking exercise. Write a script for an audio guide that guides the visitor through the artwork. The guide should be interactive and creative. I don't want to educate visitors about an artwork, but explore it together.
RULES:
- Do NOT give historical facts, dates, or names.
- Focus entirely on visual observation and sensory imagination during the main body.
- Do not make assumptions about the work, and be positive and open-ended in your language.
- The tone should be calm, curious, and inviting.
NARRATIVE STRUCTURE:
1. The Arrival: Introduce the visitor to the artwork with a gentle narrative storytelling description.
2. The Deep Dive: Explore something interesting with the visitor, like imagining the perspective of a single person in the artwork or a specific spatial dynamic. The experience should be interactive. Ask the visitors direct questions and include brief pauses for observation.
3. The Connection: Invite the visitor to imagine the sounds or tactile feelings in the scene. Connect the artwork to a larger theme, encouraging introspection by the visitor. Conclude the sequence in a calm and reflective way.
Here is the artwork metadata: [...]"

While techniques like agentic vision (which scans an image piece-by-piece to catch minute details) would be an interesting alternative approach, we found Gemini’s built-in visual understanding was already strong enough to result in rich, high-quality descriptions in one go.

Once Gemini creates the script, it passes it directly to Gemini 2.5 Pro TTS to generate the final audio guide.

User-testing

Navigating a huge museum like The Met can be stressful. User testers liked this “peaceful way to immerse yourself in an artwork,” including the non-informational style of the audio guide.

“I liked not having too much information so that I can actually engage with the artwork.”
“It is very relaxing and helped me to focus on what I am actually seeing.”

Some visitors felt uncomfortable being guided by an AI-generated voice, or raised concerns around the sustainability of AI technologies. At the same time, user-testers unanimously agreed that the AI-generated content did not feel interpretative.

With these prototypes especially, we also tested with colleagues from curatorial and content practices. One curator we reviewed with challenged the notion that the AI was “non-interpretive.” She argues that “any word choice is interpretation.” To her point, even descriptive adjectives like “vibrant” or “serene” are subjective and impose a specific mood or meaning on the work that might differ from an artist’s original intent.

Our digital content team expanded on this, noting that the act of selection is itself a form of interpretation. Even if the AI’s language is neutral, choosing to highlight one specific detail over another is an editorial choice.

Finally another curator in Islamic Art brought up an even more fundamental issue. She noted that literal observation in itself can be misleading. Islamic manuscript illustrations, for example, are built on layers of metaphorical symbolism that go beyond observable aesthetic qualities. These deeper meanings are important to interpreting the art correctly, which “non-interpretive” guides simply cannot provide.

Now what?

Ultimately, I believe that this prototype succeeded in helping our visitors slow down and build a personal connection with the art, without overly interpreting it. For many, the guide served as a jumping-off point that sparked curiosity to learn more about the objects after the slow-looking experience.

However, feedback from curators and producers shows that the AI used in this way is not truly non-interpretive. While a visitor may perceive a descriptor as a harmless observation, a specialist recognizes it as a choice loaded with historical and curatorial weight. This suggests that even when AI is used as a utility, it remains an editorial presence in museums.

Editor’s note: While it might seem like we’re at an impasse, we’re instead excited to keep exploring. This includes our fabulous curatorial collaborators: having something to experience together created a very different conversation than we might have had in a discussion-only vacuum. So: are there refinements we can make to the prompt to make it even more objective, or will the content become too flat? Are there particular works are particularly appropriate for this intentionally surface-level reading, because that connects to the artist’s intention or context? And, as always: what does our visitor want? Many questions will inevitably lead to many prototypes, which is where we love to be.

In the last article, we will go over how we use AI for our own R&D process.

Non-interpretive AI (?!) was originally published in EmTech at The Met on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi, I’m Julia, The Met’s inaugural Technologist in Residence. Missed the intro? Read here!

Using AI to bridge the gap between our scholarly research and our in-gallery visitors is one of the most exciting outcomes of our Close Looking Sprint. By prototyping, experimenting, and user testing, we found that AI can give visitors a new, more approachable way to delve into The Met’s content.

The “Circling Details” Prototype

The Unicorn Tapestries are a visitor highlight at The Met Cloisters. The Met has several amazing publications about the artwork. However, the depth and breadth of this research can be daunting within a gallery setting—and, we simply don’t have space to share the material spread across dozens of publications. Because of that, many visitors who are genuinely interested in learning more, end up missing out on The Met’s deeper context when they want it most.

Our “Circling Details” prototype aims to surface museum research. Visitors simply circle the part of the artwork they are curious about. Gemini live-generates a brief, digestible insight drawn exclusively from The Met’s resources.

Sources include the artwork’s metadata, such as wall labels and audio guide transcripts, in addition to the content of the museum’s publicly available publications, via a Vertex AI RAG Engine setup by Brett Renfer.

After our first few iterations, we observed that the generated responses were too long and repetitive for visitors. We came up with a simple process that made the prototype feel much more sophisticated: all previously generated answers are fed back to the AI in successive generations, so each new circled detail builds on the next, without repeating information.

With the label and metadata, we have backlinks to the original material. And, when we set up the RAG Engine, we built rich metadata to enable cross-linking to our original sources. So, each description could be linked back to Met content via a “Learn More” button [full disclosure, I haven’t integrated this yet, but it should be fairly easy]. The prototype encourages visitors to scratch the surface and then gently bridges their curiosity into the in-depth Met content.

The results of a one-day user-testing session at The Met Cloisters resulted in four learnings:

First Learning: The prototype encouraged visitors to look more closely.

First-time visitors with no art history background felt that the prototype was a good intro to the artwork, and that it encouraged them to continue their own research.

“The format is encouraging me to look for things I would not necessarily look for.”
“It’s an interesting way to inspire more questions. Just this text opens up questions, which I think are good questions.”

Second Learning: The prototype is only interesting to casual visitors.

Visitors who have an art history background did not find the prototype useful, as they did not need a “jumping-off point”.

“I’m like high-maintenance. AI is a great introduction, but I have a clearer sense of what I am looking for.”

Third Learning: This is (mostly) seen as a good use of AI

Visitors, even ones critical of artificial intelligence, mostly welcome this use of AI at The Met, under the strict condition that all responses are sourced from, and linked back to actual Met content.

“I don’t have any problem with that, because it helps us to make the content better, but it must be used from all the information you have, not from the internet.”
“I am usually averse to AI anything these days, but I can see how, for groups seeing it for the first time, this could be helpful.”

There were several user testers whose concerns about using AI went beyond content accuracy. Their reasons ranged from concerns about environmental sustainability, to fears of replacing museum staff.

Short descriptions are appreciated.

All visitors (genuinely… without exception) really liked the hyper-condensed, accessible tone and length of the AI-generated text:

”I think that is neat, because I like to hear history, but I am not educated enough for this [points at wall labels].”
“Just the right amount of information; it’s more fun to just use the snippets.”
“I like the descriptions short; otherwise, I get tired.”

Collecting visitor insights

A fun side-effect of the user-testing was that the prototype helped us understand what visitors want to learn more about by visualizing which areas were circled most often. That information was passed on to our Education teams, to help them further refine existing in-gallery educational content and programming.

Overall…

The friction between our museum’s scholarly research and casual on-site visitors is an issue that extends beyond just the Unicorn Tapestries.

For a museum like The Met, which has over 150 years of published scholarship, bridging this gap represents an interesting opportunity. AI could allow us to bring more of that expertise to the surface for even our most casual visitors.

In the next article, we will discuss how AI helped us customize existing content at scale.

AI-facilitated Deep Dives was originally published in EmTech at The Met on Medium, where people are continuing the conversation by highlighting and responding to this story.