Labels, captions, and video players
We recently launched a new exhibition with RISING, The Vinyl Factory: Reverb. It explores sound and social movements through vinyl culture.
My team at ACMI had the fun job of building new technology to support and extend the visitor’s experience.
Digital labels
During the ACMI renewal we built digital museum exhibition labels that pull their content from XOS, our museum exhibitions system (CMS), and also have the ability to synchronise playback bars with video players.
For Reverb we added a new design template to help highlight when the video starts again using our existing RabbitMQ message synchronising service.
We also wanted to extend the accessibility of our museum labels to other languages, so added Simplified Chinese, Hindi and Punjabi translations using gpt-5.
Video captions
To provide visitors with video captions/subtitles without compromising the visual aesthetic of the Reverb artworks, we built a synced captions viewer component for our website using Codex agents.
This uses the same RabbitMQ synchronising service as our existing video players and digital labels, and also keeps the latest video caption highlighted and on the middle of the screen in time with the video’s audio track.
We then translated these video captions to Simplified Chinese, Hindi and Punjabi using gpt-5.
Video players
The Reverb artists’ preferred video playback computers were Mac Minis, so to enable seamless integration with our existing digital labels and captions viewer we ported our Python media player code to SwiftUI using Codex agents.
This enabled us to use the Apple Test Flight and App Store deployment systems for macOS, AppleTV, iOS and iPadOS devices.
Apple Media player code: github.com/acmilabs/apple-media-player
You can find it in the Apple Mac App Store: apps.apple.com/us/app/acmi-media-player/id6754896867
Try it out with our test base URL: https://acmi-public-api.s3.ap-southeast-2.amazonaws.com/media-player/
Translations
The amount of labels and captions to translate meant we needed to build a pipeline for computers to translate the video’s audio speech-to-text, and also an interface for humans to validate and edit them.
Get Simon Loffler’s stories in your inbox
Join Medium for free to get updates from this writer.
We reused our existing off-line Whisper speech-to-text pipeline so we could perform all of the text generation on our in-house computers, and then used cloud-based gpt-5 integrated in XOS for the translations.
This pipeline allows us to generate multi-language museum label and video captions translations with one button press, and then edit them inline while watching the source video.
We’re also experimenting with a digital label translations review system that makes use of large language models to suggest where translation drift might have happened.
To do this our pipeline:
- Sends our gpt-5 translations to the Google Translate API to translate back to English
- Passes our original English text, translated text, and translated-back English text to a language model to highlight potential issues
- Populates a dashboard for ACMI staff to review
We hope that using two different models to translate to and from English will help discover problems that one might have been over-confident on due to its pre-and-post-training data and techniques.
If any of these projects are on interest to you or your organisation, please get in touch so we might be able to build something together.
Try it out for yourself
If you’d like to experience the result of all of this in person, please buy a ticket to see Reverb, and scan the QR codes on the museum labels to see the access captions and labels: access.acmi.net.au
Our AI use
It’s worth noting that this would have been very difficult to build in the timeframe we had without using AI.
We have access to Claude, GPT+Codex, and Grok at work — and used Codex-5.4/5 agents for the majority of code in this project.
We acknowledge the energy use associated with training these models, and I make best efforts to offset this using solar+batteries+electric vehicles. Where we can we use local models on-site.
We’ve found that selective use of AI tools has enabled us to automate repetitive tasks across teams at ACMI, giving them more time to be creative and experimental in their work.
This article contributed to Digital labels, synced captions and AI translations by my team lead, Head of Digital Products & Interaction Design at ACMI, Lucie Paterson.
