We are shocked and deeply saddened by the passing of our dear friend and colleague, Jessica Clark.

Jessica originally started working with us on what we called the Immersive Media Impact Working Group. Not the most imaginative title but an apt description of what we were trying to achieve. We held an Interactive Media Impact Convening in New York in October 2014 to start to build the vision for what later turned into Immerse. Earlier that year Jessica had joined a hardware hackathon that Ingrid had produced at the Tribeca Film Institute. Based on her ongoing social photography experiment, The City Speaks, Jessica created an Arduino-based prototype for simulating/stimulating responses to her photos of street art. It included automated responses triggered by buttons featuring related photos, and then asked participants to record how a particular image “spoke” to them in exchange for a sticker of that photo that they could keep. We mention this here because Jessica’s work contained so many layers. She was deeply invested in online culture but always loved tangible, analog artifacts. She loved art and nature and beauty. She was endlessly curious, always looking at the world with her sharp, bridge-building mind and creating new creative pathways through it.

We launched Immerse in 2016, initially as a partnership between Dot Connector Studio (Jessica), MIT Open DocLab (Sarah) and Tribeca Film Institute (Ingrid). Jessica did the lion’s share of the work despite having a full schedule with Dot Connector Studio. She was an incredible editor: insightful, dedicated and so hard working. She did what she always did: joined the dots for us all. Jennifer Brandel has a wonderful description of how Jessica explained the concept of the interstitium. She was always dancing between fields and disciplines. Creating meaning, translating ideas big and small. She loved nothing more than taking a knotty issue, a wicked problem, and creating frameworks and spreadsheets to unpack it, map it, write about it, and share it. She was so generous with her knowledge and with the outputs of her work. Her deep love for public media was wrapped up in this generous approach to the world. The editorial vision for Immerse reflects some of the breadth of Jessica’s approach: “…we take a broad look at what we mean when we say “nonfiction.” Where does “media” end and “reality” begin? Our goal is to explore many different variations — yes, journalism and documentary, but also speculative fiction based on real-world events, experiential media that sneaks up on you in familiar (and unfamiliar) places, and new concepts such as distributed storytelling.”

She always had the latest productivity tools — which she needed because she was doing the work of an army of people. We discovered Zoom through her, long before the pandemic when everyone was suddenly zooming. Our weekly calls for Immerse were anchored by Jessica sitting in her purple office, surrounded by books and art and all the things that helped her make sense of the world. These Zooms were a mixture of work, brainstorming, resource sharing, executive coaching and therapy, a touchpoint in a busy week. Jessica was brilliant, grouchy, funny, strategic, always driving things forward. We were all trying to find a way to work through constant instability, create open and dynamic conversations while the ground shifted beneath our feet, and build community. Through these weekly calls we became real friends and would relish in person meetups at various festivals and conferences. Over the years others would join our Immerse team — Kat Cizek, Abby Sun, Julie Fukunaga, Ngozi Nwadiogbu, Claudia Romano, Carrie McLaren — but Jessica was always there at the center, steering the ship, weathering the storms.

In addition to editing Immerse, Jessica also wrote many insightful articles. She wrote about everything from haptic storytelling to journalists and collective wisdom to immersive monsters reflecting her wide-ranging, discipline-hopping interests. She worked with Kat Cizek to share key takeaways from Collective Wisdom. She also worked closely with Kamal Sinclair to parse out the important findings of Making a New Reality and reach a wider audience. She loved this work, collaborating with others who were equally passionate and driven, getting into the weeds on big ideas and finding ways to reach people, to communicate clearly.

It is hard to think about a future without Jessica in it. She was as Brandel said “the Platonic ideal of an interstitionary” and in this moment of polycrisis we need people like Jessica more than ever. She worked so hard to imagine a better future and to connect people and ideas to bring about a better future. Immerse went on hiatus in early 2023. We all just got busy and never quite figured out how to continue again, if at all. We would still meet for occasional Zoom meetings with Jessica in her purple office and those meetings continued to feel so full of possibility because there was always forward momentum with Jessica, always a sense of another connection, another big idea, another brilliant tool to use. We are forever inspired by her and will miss her hugely. Thank you Jessica for everything.

You can contribute to Jessica’s memorial fund here.

Our journey with Jessica Clark was originally published in Immerse on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dear readers:

When we launched Immerse in 2016 in partnership with The Fledgling Fund, we had no idea what was in store for us. Remember the world pre-pandemic, pre-Trump, pre-Musk? Who knew those were innocent times…

But what we did know is that we were uniquely positioned to document and curate the evolution of documentary practice into new and unplumbed realms. We could see artists and documentarians pushing boundaries all around us, exploring the potential and dangers of new technologies. But the work was elusive, fragmented and inaccessible — both critical discourse and key funding were lacking.

So, as we explained in our first newsletter, Immerse aimed to convene “all of you who are making, supporting and thinking about the field of interactive non-fiction — news, documentary and whatever comes next. While VR is one focus, we’ll be tackling the full range of interactive and participatory technologies, as well as related questions about ethics, creativity, inclusion and the types of futures we’re all making. Our goal is to spark dialogue about this fascinating and confounding field.”

And so we did! Our editorial team and writers ranged far and wide, across disciplines, topics, continents, platforms, modalities and existential debates, surfacing new forms of investigation and expression. We created a bridge between academia and practitioners. We gave makers a platform to reflect on their practice, creators of color a space to elevate their perspectives, and the next generation openings for reflection and dialogue.

Just a glimpse at our archives suggests how far we have roamed: from the history of transmedia to the future of avatars, from the power of decentralized narratives to the promise of constraint, from harassment in social VR to the potential for experiential design to connect us to nature and ourselves.

In the process we captured a snapshot of early conversations about VR, AI, AR, and other technologies that at the time seemed obscure, but are now dominating mainstream headlines. Along the way, we collaborated with leading documentary festivals around the world to publish special issues, lead conversations, and even world-hop through artist-created versions of the Metaverse.

Beyond storytelling and technology, social justice has been key to our mission. As a women and non-binary-led editorial collective, we know we’re unusual in tech and media realms that are still too often male-dominated. With support from our funders at the Ford, MacArthur, Knight foundations and the National Endowment for the Arts, we’ve held space for critics of color — particularly those early on in their careers — to contribute to the dialogue. As champions for inclusion and liberation, our work together supported the creation of the Making a New Reality Toolkit, and the recently published Collective Wisdom: Co-Creating Media for Equity and Justice.

We absolutely love working and exploring together. But now, we’ve decided to take some time to figure out what comes next. Inspired by our colleagues at the Allied Media Conference, we are taking a chrysalis period.

What’s that? It’s informed by the metamorphosis of a caterpillar into a butterfly. Inside of the protective shell of the chrysalis, the caterpillar dissolves. It grows new legs, eyes and wings. Its organs take fresh forms, as old tissues are broken down to build a new body. It goes into to the goo — and breaks out anew.

That’s where we are now.

The team is taking time to archive and sunset Immerse as we know it, and to reimagine what the Immerse community and dialogue of the future might be. To keep you posted on our progress, we’ll continue the monthly newsletter.

However things evolve, we see that we’re part of a grand tradition — as David Lang recently wrote via Substack: “The catalogers know something important that most of the startup hype obscures: the purest joy in the technology world is not a big funding round or other high achievements. It’s the simple pleasure of a curious mind and the camaraderie that develops in an amateur technology scene on the verge of something new — a mixture of possibility, personal power, and shared passion.”

We’re not sure where we’ll go next as a team, or which parts will emerge intact. But we know that the role Immerse has played in the ever-changing field of interactive documentary is important, and that the topics we cover are more relevant than ever.

Goodbye for now!

Jessica, Julie, Ingrid and Sarah

Why the Immerse Team is Taking Some Time Inside the Goo was originally published in Immerse on Medium, where people are continuing the conversation by highlighting and responding to this story.

At this year’s TED2023 Conference in Vancouver (April 17–21, 2023) — themed around “Possibility” — attendees were abuzz around the role increasingly intelligent and large-scale AI models such as ChatGPT, Stable Diffusion and Midjourney might play now and in the and distant future.

More than just an artfully curated week of talks, the TED Conference has always been a showcase of innovation. Interwoven between the familiar presentations were displays of the bright, dazzling and sometimes too-good-to-be-true world unlocked with AI. From live performance capture tools that could transform actors to aliens with barely any equipment to worldbuilding tools that could generate architectural styles from across the expanse of the globe, the audience was inundated with striking images of possibility. As my second time attending on the ground as conference support staff through the Bezos Scholars Program, the adult daycare culture of TED, with dragonboat courses and circus workshops squeezed between opportunities for idea generation, was a strong departure from my quotidian life, to say the least.

In the ecological domain, Karen Bakker spoke of the potential of harnessing AI that analysto protect and restore endangered coral reefs. Part-and-parcel with the product placement historically featured at TED, Sal Khan showcased the potential of Khanmigo, an AI writing coach chatbot that can help students write essays, structure arguments and even speak to literary figures to better understand the works with which they engage.

Throughout, the conversation always returned to a tempering of optimism around the possibilities of AI. Audience members were consistently probed for their level of fear, their own impending dread of the rapid change at the hands of tech corporations’ “space race” to own and dominate the landscape. As with other technologies that had come before (the most recently featured hype cycle at TED was 2022’s emphasis on Mars and the Metaverse), this year’s TED was a showcase of the bite-sized: a balance of stories of possibilities and constraints, all distilled to less than 15 minutes.

Beyond catastrophizing: The art of designing with constraints

One striking session focused on AI creativity offered insight into larger explorations of what it might look like to create art and documentary in tandem with AI tools. On April 20th, 2023, TED gathered AI artists and technologists K Allado-McDowell, Refik Anadol, Bilawal Sidhu and Eileen Isagon Skyers to speak more on the subject.

Admittedly, a lot of what I know about AI art is prompted from a fear of my impending obsolescence — and circles of artists, writers and creatives who find themselves up against the increasingly threatening foe of automation. As a journalist and creative in the games space, it’s difficult not to lose hope when every headline or Tweet is a reminder that a machine may soon be able to do what I do better, faster and cheaper.

When prompted with the same question — of whether artists should feel threatened by the proliferation of such tools — panelists offered more optimistic thoughts, preferring to frame the technology as adding to the range of possible forms of expression as opposed to replacing any singular art form. While the concerns are understandable, especially considering the professionalization of and specialization needed to “master” any particular suite of creative tools in an era pre-AI, this future is here whether we like it or not and we must be equipped to adapt.

This particular moment of flux, as AI artists establish not only their own distinctive signatures but also the legitimacy of their artistic medium, is one that AI artist and writer K Allado-McDowell likened to the emergence of photography. Photography, in their eyes, did not discredit painting and the value of collecting painting, just as AI art will not displace traditional forms of art. And people are not necessarily creating art solely for the final product — novelty, creative inspiration and trial-and-error might emerge as particularly important in the years to come.

What they cited instead was the emergence of a human-AI feedback loop, one that goes beyond simply prompting “make this and you get it back.” As a co-author of several book projects, operas and games with GPT3, they are particularly attuned to this feedback loop and finding ways to set rules and constraints that push both them and generative AI to “come up with things that wouldn’t be possible alone.”

The key challenge is sifting through the noise — what some of the panelists called “AI garbage.” “I do think the abundance with generative image making tools like Midjourney, Stable Diffusion and DALL-E makes it hard … to actually find what the constraints are in the system,” proposed Allado-McDowell. “I think we will see really incredible AI-generated artworks that are standalone images made solely with those tools and they will have their own vocabulary. But I don’t think we’ve seen the Duchampian moment yet with these tools. We’re waiting for that person to come and break the system open.”

Bilawal Sidhu, former project manager for Google Street View and now technology influencer and content creator, also echoed sentiments about the “journey” of creation. Sidhu, who grew up making home movie videos using clunky video editing software, has hope for the next generation of content creators. He said, “I think it’s going to be more about the canvas on which you create and the kind of story that you tell at the end of the day versus the tools that you end up using.” Sidhu sees even more potential now for anyone with talent and vision to succeed.

AI can cut the barriers to entry, elevating an artist’s craft to the next level. Sidhu pushed writers to consider visual image, visual artists to consider animation and 3D, and game makers to consider fully immersive worlds. While the metaverse may seem to currently be in decline, it now offers many more possibilities to be populated by a diversity of creators using these tools — not just the large AAA gaming companies and the Pixars of the world, but the indies and small studios. AI could also be used to generate new game mechanics or to create personalized gaming experiences based on a player’s individual preferences and playing style.

The unsolved problems of copyright and data stewardship

The panel also touched upon the collective ethical reckoning with these tools we are currently witnessing. The use of stolen, unattributed data for algorithmic art generation currently is not strongly matched by processes for legal recourse and copyright. AI may, as we’ve seen in numerous sectors, continue to amplify existing human biases, rely on incomplete training data and perpetuate harm against groups at the margins, including people of color, queer and trans communities, disabled folks and inhabitants of the Global South. And the question remains: In a situation where AI and humans work together, who is the rightful owner and how should contributor(s) be cited?

When prompted with this question, panelists had notably less to offer — not to my surprise. In many ways, we are still in the theory stage, struggling to keep informed with the rapid progress currently largely driven by OpenAI. That said, efforts to contain AI are by no means new. For example, the European Union has proposed a set of ethical guidelines for trustworthy AI, which includes principles such as human oversight and accountability, transparency and non-discrimination.

Artist Refik Anadol, works on projects that meld the physical and digital, including Artificial Realities: an attempt to construct physical corals using 3D printed AI sculptures. He cited a fascination with archival processes. Well known for striking data visualizations that have graced the MoMA and Gaudi’s Casa Battlo (the latter of which was auctioned as an NFT for $5 million in 2021), Anadol has been working in these spaces for almost a decade harnessing what he calls “large, focused and publicly available datasets, visualizing … ‘humanity’s collective memories.’”

On the panel, spoke about standing on the shoulders of those who came before and the importance of working in partnership with ancestral knowledge. AI, in his eyes, is a tool for preservation and learning — and these principles of respect and care were foundational in the creation of an “open-source AI rainforest model that can reconstruct extinct flora and fauna based on [Amazonian] tribes’ deep collective knowledge.”

Anadol also spoke extensively about the importance of open source technologies and the power of artists to create and harness their own AI models, putting their narratives into their own hands. While we have yet to see the full accessibility of such computation and tools at our disposal, the option is on the table for artists to continue to explore.

A final confession

After a week of sitting through TED, it became a running joke among presenters to have ChatGPT, or another large language model generate your talk for you — a tantalizing lure that journalist me, in all my sleep-deprivation, was inspired to explore for the first time in writing this recap. So I tried it on OpenAI:

JF: Can you write a summary of key points from a panel transcript on generative AI creativity and the role in the arts from this year’s TED conference, moderated by Chris Anderson?

And then:

JF: Can you expand that to be 800 words?

While (disclaimer) none of the content made its way to this final article, I was struck by the potential for near instantaneous information generation. I had with me an interactive, encyclopedic research assistant who might lack inspiration with their prose, but could constantly go back to the drawing board to get you a satisfactory response. I was able to organize my thoughts more clearly, understand a flow from an unwieldy transcript and get an article together I probably would not have had the time to do.

And I’m perhaps late to the game, out of resistance or a critical stance around the potential for harm such tools can cause. On the other side of the globe, my grandmother recently used the tool to solve her own interpersonal problems after being slighted by a colleague. After copy pasting their read receipts, she asked ChatGPT why someone might respond that way.

My grandmother likened using the OpenAI chatbot to her memories of raising spirits in her childhood — a Pandora’s Box too dangerous to be tampered with for too long. And that was something I felt strongly while continuing to prompt the tool for more, a dopamine rush with every “generate.” It was freeing in some ways to be pulled into rabbit-hole tangents and asking the chatbot to summarize, rephrase, expand. I saw the value of the tool the same way older generations complained about research before the internet — and may one day venture back to using it to organize thoughts and follow threads the same way I’d trace citations on Google Scholar.

In closing,, I reflect on the question Chris Anderson asked all of the panelists — what advice they would give to a young artist looking to start their career today. In media curator Isagon-Skyers’ words, “I would say to follow your intuition, because that’s one thing AI doesn’t have.”

“Yet,” says Sidhu.

For more news, discourse, and resources on immersive and emerging forms of nonfiction media, sign up for our monthly newsletter.

Immerse is an initiative of the MIT Open DocLab and Dot Connector Studio, and receives funding from Just Films | Ford Foundation, the MacArthur Foundation, and the National Endowment for the Arts. The Gotham Film & Media Institute is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing emerging nonfiction projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.

TED 2023: Possibilities in Artificial Intelligence and Creativity was originally published in Immerse on Medium, where people are continuing the conversation by highlighting and responding to this story.

A behind-the-scenes look at Pedro Harres’ interactive and immersive animated film From The Main Square

It’s not often that a student thesis takes home a prize at an A-list film festival but such was the case with From The Main Square, an immersive experience by Brazilian director Pedro Harres. It won the Immersive Grand Jury Prize at Venice Film Festival in 2022 and more recently, the grand prize at New Images Festival in France.

The project has traveled far in the recent festival circuit. I first encountered From the Main Square at Venice Immersive and was taken by the imaginary world that unfolded around me and the profound truths that it revealed. There, I sat down with Harres, for a fascinating discussion about the project’s origins, his intentions and process and the teamwork involved. I encountered it again at South by Southwest in March, where the team offered some of the original drawings on paper at the end of the exhibit. The drawings were brimming with life and imagination, while also documenting a very specific time and place. The art director and illustrator Dani Eizirik drew many of them on site — in the woods and in the cities — lending the project another level of veracity.

Seeds of the concept

The film’s origins started ten years ago when Harres was living in Brazil and became deeply disturbed by the vitriolic comments unfolding online, which he saw as a bad harbinger. He set out to make an animated film that showed a “portrait of social disruption,” highlighting the risks of social polarization.

His original vision was of a single building with inhabitants fighting over its structure, each trying to modify it in their own way until the building collapsed. To bring this project to life, he wrote a script for Eizirik to illustrate, hoping to apply for a grant. The night before the application was due, Eizirik sent him an unexpected drawing — of a city, not a building.

While they didn’t get the grant, the idea of a polarized city stayed with Harres. Years later when he was a Masters student in animation direction at Babelsberg Film University in Potsdam, he saw an opportunity to apply for funding for a VR film. He was in a support structure that allowed for experimentation and thought this could be the moment to try something new. The unexpected drawing of the city that his art director had presented to him years ago would be the perfect start to explore an immersive project he decided. In this film, the city would rise and fall as the polarized inhabitants turned to beating each other up and destroying it.

He also kept the idea of having two groups — the Rounds and the Squares — that represent opposing worldviews and “two tendencies that you see in society.” The Rounds have a connection to nature and build their houses as artisans, while the Squares tend to build with concrete, iron and glass and prefer order over freedom. This manifests in the Squares’ preference for dictators and the authoritarian rule they use to control the space around them.

He added a subcategory to the Rounds: Indigenous peoples who were the first inhabitants of this world and are shown living in harmony with the land and witnessing all that unfolds. This group is the first one you encounter in the experience. Then, the Rounds appear and start building their homes out of natural elements. Next, the Squares enter the picture and start building industrial buildings. Soon after buildings go up, the interactions between the groups become harsher and violence erupts.

Police, whose main function is to suppress the Rounds, start to appear. We see people begin carrying signs and hanging flags of dictators on their homes. They spout slogans and tanks arrive. Protests erupt on both sides, trees are cut down and more beatings between the sides occur. In the chaos, both the Rounds and the Squares start setting buildings and each other ablaze. As a viewer in a headset, you spin around quickly to witness all the scenes unfolding simultaneously around you. You start embodying the chaos.

Embodying rapid city change through game mechanics

Harres wanted it to be an experience where viewers have the freedom to look around and choose which scenes to activate with their gaze. While gaze-based scene activation is a common mechanism in VR, Harres flipped it: content is swapped when you look away. It loads while you are not looking and when you look back, the animated scene has completely changed. It was a difficult game mechanic to implement but important to Harres because it captured the feeling of gentrification and rapid development that he experienced while living in Berlin.

In his words:

Berlin is undergoing a harsh process of gentrification. The city is modifying at a very fast pace. You go to a street and you don’t see a building there, and one week later you have a brand new building… [I wanted to capture] this feeling of spookiness to the viewer. There are trees and a mountain and then you look one minute later and ten buildings are there.

He went on to explain the difficulty of creating user interactions where the content was swapped once the user looked away.

This was also tricky to do because at the beginning, the content would be already activated by the borders of your field of view and then you would be missing a lot of stuff [by the time it is centered in your field of vision]… So we needed to put a collider on certain objects — that is, an invisible square on your field of view that whenever those things hit, it [should trigger an animation] in the middle of your field of view.

Genre-bending: Fiction or documentary?

When I asked Harres if he considered his work a documentary, he said it was a hybrid — both fiction and documentary.

Many of the situations depicted are based on real life situations including the burning of Galdino Jesus dos Santos, an Indigenous leader and activist who was in Brasilia for a conference and was sleeping on a bus stop bench when five white men approached him and burned him to death. (The murder charges were later dismissed.) Galdino became an icon of the indigenous resistance in Brazil.

Buildings are based on observation drawings. The city is a conglomerate of different cities referencing real world sites he had observed. He referenced buildings, streets and scenes in São Paulo, Berlin, Porto Alegre, Maquiné and some cities in Turkey. The languages spoken are a mix of actual ones. The Square language mixes Greek, Bulgarian and Swedish while the Round language mixes Igbo, Mongolian, Somali and Samoan. The dictator represented in the signs is an amalgam of dictators past and present from Indonesia, Hungary, Israel, Chile and Brazil. The themes in his work stretch far beyond his experiences in Brazil and Germany, speaking to global phenomena and political systems.

Harres collaborated with a big team. The art directors — Eizirik along with Paulo Lange — drew all the buildings on paper, and Eizirik designed the characters for all of the figures inhabiting the city. The animations were created by animators Sophia Shönborn and Samuel Patthey, the latter known for his expertise in straight ahead animation. This style starts with one pose and adds progressive poses without the smooth “in-betweening” we see in traditional animation. Without a clear guide between frames, this style allows for spontaneous, more “organic-feeling” occurrences that reinforce the documentary aspect of the content.

In terms of future directions, Harres envisions distributing the project on Steam, an online game publisher, where incentives to return to the game are built in through achievements.

“It’s designed in a way that when you play it, you’re going to see around 70% of the content. Therefore if you want to see everything, you have to play again. “

Harres explains, “I saw people ending it and having this mix of amazement by seeing something that they had never seen before… but really worried about the message that was being portrayed inside. Since the first version of the script was written ten years ago, before this wave of authoritarianism started worldwide, it was already there. But back then it was much more fictional, and then it got somehow more attuned with reality. And now reality is scarily matching it.”

With its mix of references and a mesmerizing animated world, From the Main Square immerses you in a story that is both universal and specific and artfully blends fiction and documentary. It leaves you with a sense of delight at the magical world you inhabited but also with a deep sense of unease because of its resonance with the violent polarization that reflects our reality today.

For more news, discourse, and resources on immersive and emerging forms of nonfiction media, sign up for our monthly newsletter.

Immerse is an initiative of the MIT Open DocLab and Dot Connector Studio, and receives funding from Just Films | Ford Foundation, the MacArthur Foundation, and the National Endowment for the Arts. The Gotham Film & Media Institute is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing emerging nonfiction projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.

Illustrating Reality in a 3D World was originally published in Immerse on Medium, where people are continuing the conversation by highlighting and responding to this story.

This winter issue centers disability innovation in documentary and emerging tech — presenting perspectives from artists, activists, scholars and technologists at the vanguard of storytelling and disability justice. It’s inspired by existing efforts from our collaborators at the Access and Disability Innovation Working Group at MIT Open Doc Lab and Co-Creation Studio.

Lightning talks from artists, technologists, curators and scholars hosted by the working group provided a jumping off point for reflection for the Immerse editorial collective. The publication typically engages emergent technology forms that historically have not accommodated disabled artists and audiences. One of our issue pieces in particular, an interview with Sacha Wares, the director of the IDFA award-winning Museum of Austerity (2021), is directly inspired by one of the talks in the working group’s symposium.

With this issue, we aim to share learnings that inspired us, lessons learned in the creation and exhibition of art that centers access, that recognizes disabled people in the wholeness of their experiences, across the many spectrums and intersections of identity. We also wish to share the harsh realities, the impacts of access initiatives that fall short, enacting change that is unsustainable and harmful.

I ask our readers, creatives across the spectrum of engaging with (or perhaps persisting through) conversations around disability and access in their day to day, to consider these pieces with an open mind. Allow yourself forgiveness, time to “be.” In keeping with the spirit of many of these pieces, sit with the feelings of resonance, acknowledgement and discomfort, and commit yourself to learning and unlearning in the ways you can.

Some of these are stories of creative workarounds and “finessing,” of breaking apart and manipulating systems in place to imagine new realities. These stories speak of frustration, of triumph and of not only wishing for a new status quo, but detailing the steps needed to get there. Others call out ableist bullsh*t like they see it, like Leora Fridman’s critique of Apple’s The Greatest (2022) the newest disability ad that she argues falls short in nuance and representation. And don’t miss Aubrie Lee’s gripping essay explaining how technology that could be enabling her as a wheelchair user and self-proclaimed “cyborg” instead sets her back.

Our writers introduce us to a range of cutting-edge accessible tech — some of which, like Dylan Fox’s speculative fiction piece on being blind in 2040, hasn’t yet been invented. From Laurel Lawson, dancer, choreographer and engineer, we get an introduction to Protactile and the potential of haptic tech to revolutionize disabled dance and performing arts spaces. In their Zoom campfire chat, Chris Hainsworth and Harshadha Balasumbrian of Blind Burners reflect on their collective’s challenges in making a virtual reality (VR) experience accessible for blind and visually impaired creators and audiences — under the time crunch of Burning Man 2021.

Beyond the baseline of access (which we continue to emphasize as a necessary precondition of showcasing work), this issue also features ways of exhibiting and cultivating creative experiences that are intimate, intentional and trauma-informed. In her interview with seeley quest, multisensory artist Salima Punjani presents a plethora of ways of structuring care into art practice, spanning everything from heartbeat biodata to bubble baths. Artist-scholar Dr. Frank Mondelli grapples with questions about what it means to center Deaf, hard of hearing and disabled perspectives in their subversive re-imagining of the hearing exam — surfacing questions about how to design not only for a Deaf and/or disabled audience, but for a hearing, able-bodied one as well.

Now, I am proud to present perspectives from folks who do the day-to-day work of capacity building, existing both in solidarity and in allyship in order to create a more just, accessible world. As a first-time editor (from start to finish), I feel grateful for many things: the organizing efforts, triumphs and trials of activists such as Stacey Park Milbern, Alice Wong, Mia Mingus and Leah Lakshmi Piepzna-Samarasinha that have uplifted disability justice to the visibility we see today. The grace, sheer talent and patience of all the contributors. The fact that we are able to put this issue into the world. The labor of love that Immerse is for our editorial collective — many claps to associate editor Omar El-Sabrout, joining for this issue.

Issue Featured Stories:

For our stories in simple text, please refer to our linked Archives page.

• Into the “Museum of Austerity”
• Blind in 2040: Specs on Deck
• How Innovation Sets Me Backwards
• How Can Art Reach Out and Touch Us?
• “Words in Noise”: Looking for a Deaf Voice Amidst the Noise
• VR Beyond Sight
• Embodiment as an Entry Point — At the Intersections of Art, Tech & Care
• Make Medicine Sick

For more news, discourse, and resources on immersive and emerging forms of nonfiction media, sign up for our monthly newsletter.

Immerse is an initiative of the MIT Open DocLab and Dot Connector Studio, and receives funding from Just Films | Ford Foundation, the MacArthur Foundation, and the National Endowment for the Arts. The Gotham Film & Media Institute is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing emerging nonfiction projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.

Welcome to our Latest Issue: Disability Justice, Technology & Access! was originally published in Immerse on Medium, where people are continuing the conversation by highlighting and responding to this story.

Directed by Sacha Wares and based on the reporting of collaborator John Pring of Disability News Service, Museum of Austerity (2021) uses volumetric capture in a mixed reality project for the HoloLens headset to tell stories of disabled people in the UK who died as a result of welfare support cuts in the decade following the 2008 financial crisis. Co-produced by English Touring Theatre, National Theatre UK and Trial and Error, with creative technology by All Seeing Eye and Dimension Studio, the project previewed at London Film Festival Expanded in 2021, and subsequently won the IDFA DocLab Immersive Non-Fiction Award 2021. A touring version is currently being developed for theatrical presentation in the UK in 2023.

The Accessibility and Disability Innovation working group at MIT Open Documentary Lab & Co-Creation Studio invited Museum of Austerity’s Director Sacha Wares to discuss their team’s approach to accessibility and care in the production and exhibition of the work. The following is based on her March 2022 presentation.

The sound design in Museum of Austerity responds to the embodied presence of a viewer in physical and mixed reality space. Approaching each holographic representation of a person who died as a result of the Austerity era cuts activates the sound of a relative talking about that person’s last moments. In closer proximity to the person’s hologram, one can hear that actor’s breathing. Turning away to leave triggers comments from politicians in the House of Commons about the law itself, changes to the benefit system or the person’s death. This juxtaposition collapses the distance between political decision making, the individuals impacted and the audience.

Due to the nature of the subject matter, a significant consideration in the production of the work was the mental wellbeing of both the production team, the participating families, and the audience. Says Wares:

We had a psychotherapist and a theater wellbeing practitioner on hand for the whole (production) process… There was mental health support for participating families and anybody within the creative team who wanted it and an agreement that at any point that if anybody working on the project wanted to step away from the material, they could, and people did take that up and they were… vocally grateful for that having been said from the start.

That part of the creative process, I felt we managed really sensitively and that was a lot to do with our main collaborator John Pring’s take on it as a journalist. He came to the project with strong feelings about how you talk about suicide in journalism, which is quite different from how you maybe would in theater… There are strict rules in journalism about what you should and shouldn’t say about someone taking their own life to avoid triggering those with personal experience of mental distress. John was scrupulous about that, and that really limited a lot of what we could show, what we could say, and that was to the project’s benefit because often when you’re limited you have to think in alternative ways and I think that’s one of the reasons why it’s quite restrained as a piece of work, and I think that contributes to its power.

The project’s creative team includes disabled and non-disabled artists and offers comprehensive, inclusive and clearly signposted access information for audiences.

The access offer includes through-experience audio description, in-headset captioning, a BSL (British Sign Language) onboarding video, wheelchair access and a range of accessible amplification options for audiences.

Leaning into the access considerations of the subject matter resulted in innovation in the exhibition of the work, including a bespoke content editing tool whereby audiences can select not to see certain material:

At the testing phase, it was clear that there was potential for some audience members to be overwhelmed by the subject matter particularly if there was an overlap with their own experience. We took the audience care responsibility in terms of content warnings very seriously and also offered the option of content removal. We realized that because each story is independent and they’re holograms, you can take content out, and it doesn’t affect somebody’s experience, because the stories are thematically connected, but not narratively so — you don’t need to have seen one to understand the next one. For the audience there is a list of content warnings and if there’s anything on the content warning that they don’t want to see, we can remove those stories.

Sacha and the majority of the creative team have a background in theater, where there are industry standards for accessibility that are not currently present in the production or exhibition of XR/immersive work. Wares highlights that due to a lack of established accessibility approaches in immersive space, the creative team incorporated some existing conventions from theater and film, aiming to balance the interplay between audience expectations, aesthetic considerations and technical constraints.

Consideration of accessibility during production, recruiting an access testing panel and drawing on the lived experience within the creative team contributed significant, innovative approaches.

A team member’s own light sensitivity led to the addition of brightness adjustment as an audience access offer. Similarly, the iterative process involved in addressing another team member’s access to work in progress, led to a successful sighted guide solution being on offer to audiences by opening night.

The importance of testing with people where embodied audience movement is part of the design of the work was another learning, relating to differences in the ways devices respond to eye movement and physical mobility; for example awareness of the responses of sound zone triggers to wheelchair movements, including reversing rather than head turning. Importantly, information about accessible versions could be used to train hosts at exhibiting venues.

Innovation through inclusive design contributed in other unexpected ways. Rest points were offered in the design of the work for people with mobility difficulties, with folding chairs for people to use. Wares notes that, “Lots of people [without] mobility challenges took advantage of those chairs, so again, it was another example of where the whole experience benefited.”

Accessibility needs clear signposting in advance of the point of booking so audiences understand not only how work is available accessibly, but also what to anticipate at venues. Sacha explains.

We made our own BSL and captioned marketing video that told people a little bit about what to expect, organized a survey of the venue and, with the audio description company VocalEyes, produced an audio guide about what audiences would encounter in the venue…

The creative side of the access planning was really pleasurable and … where we’ve encountered difficulties it resulted in positive gains for the work … I think if we don’t take care to make our work accessible in the widest way, then we are guilty of excluding audiences and artists from the journey and underusing the combined power of art and technology to shape inclusive spaces of the future.

Further information from Museum of Austerity sound designer Gareth Fry about making immersive experience accessible can be found here.

This piece is part of Immerse’s 2023 issue centering disability innovation in documentary and emerging tech — presenting perspectives from artists, activists, scholars and technologists at the vanguard of storytelling and disability justice. You can find other featured stories and more information about the issue here.

For more news, discourse, and resources on immersive and emerging forms of nonfiction media, sign up for our monthly newsletter.

Immerse is an initiative of the MIT Open DocLab and Dot Connector Studio, and receives funding from Just Films | Ford Foundation, the MacArthur Foundation, and the National Endowment for the Arts. The Gotham Film & Media Institute is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing emerging nonfiction projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.

Into the “Museum of Austerity” was originally published in Immerse on Medium, where people are continuing the conversation by highlighting and responding to this story.

Note: The opinions in this article are solely those of its author.

In September, I was waiting to cross the Embarcadero in San Francisco. A woman standing nearby asked me, “Is that the wheelchair that can climb stairs?” I shook my head. (I hadn’t gotten this one before, so I didn’t have one of my joking comebacks ready.) She informed me, “The same company that makes that chair also makes ones that can climb stairs.”

Well. That would be impressive, considering the company that makes my chair cannot even keep the correct time on the joystick display.

Stair-climbing wheelchairs, exoskeletons, personal assistant robots, AI, CRISPR, and brain–machine interfaces are all called innovations instead of novelties. I call them shiny objects, and most of them will hurt me more in their mainstream mystique than they will ever help me.

I’m a cyborg and a futurist. I love innovation in theory, but, in practice, it sets me backwards when it means I can’t use what worked for me before.

My cyborg smorgasbord (my smorgasborg)

I use three power wheelchairs and five phones.

Each chair serves me in one of three environments: home, work, and going out. None can do every place well, and only one is still covered by insurance. And no, none of their manufacturers also makes a wheelchair that can climb stairs. They all cause me pain or pressure in different ways — which means switching between them relieves some pain in exchange for other pain.

I have five more mobility devices just in case. Three of them are Pride Revo scooters, which I loved but have been discontinued in favor of newer models. I use two of my Revos as spare part donors for the one that has broken down the least.

Wheelchair repairs are expensive and have months-long wait times, if I can get them at all. When the motors of my dear 15-year-old Quantum 600 started leaking oil onto my floors, I spent more than $1500 out-of-pocket to replace them both. One of these broke soon after, and fortunately, I had a friend knowledgeable and caring enough to repair it. He noted that the break was probably caused by improper installation.

For my going-out chair, I paid more than $4000 for a Quantum Edge 3 that can elevate. The resistance of its joystick was too high for me, so I needed a spring replaced with a thinner one. The process of replacing it required scheduling two in-person appointments, one of which was for a so-called evaluation that was nothing more than the technical representative showing up, confirming what I was asking for, and leaving five minutes later.

Springs like this one can be found online for less than a dollar. But the official spring part cost $40, and the entire process took five and a half months. And this was with the help of an excellent customer service representative; the process would have been even more frustrating without that person’s attentiveness.

My work chair is a Permobil F5. Over time, its button pad has broken off its mount. One might guess why, instead of undertaking another repair odyssey, I chose to fix it myself for the price of two hairbands.

As the rest of the micromobility industry charges forward, power wheelchairs get further and further behind. My Silicon Valley city has several places that sell and repair electric bikes, but it has no such place for power wheelchairs. When will my chairs have lithium-ion batteries instead of each being stuck with two heavy lead-acid car batteries? What are ebikes if not in-line power wheelchairs? My wheelchairs aren’t just medical devices; they’re echairs.

Micro-regressions

Just as each of my echairs can only do some of what I need, each of my five phones is limited. And the thing that none of them does quite well enough is the thing I need most in a phone: typing.

In the year 2023, when companies launch ever more advanced phones several times a year, I would like to be able to input text. Yet with ostensibly so many options, I still do not have a hardware and software combination that truly works for me. My favorite keyboard was one developed by TouchPal. I lost it six years ago when my HTC One M8 stopped working after an Android update. End-of-lifed but never forgotten.

Since then, the accessibility experience of my phones has only regressed. My next keyboard no longer had arrow keys. (Don’t ask a person with muscular dystrophy to edit text by reaching higher on the screen or wrestling with an erratic cursor.) The one after that no longer had accurate swipe-typing or useful word prediction.

My current best keyboard has been removed from the Play Store, so I only have it on my oldest phone. This is the state of that phone:

People tell me my phone is at risk of exploding. In the absence of a better keyboard, an explosion is the only thing that will make me stop using it.

Disability, technology, and autonomy

As a cyborg, I am not only my human body. My echairs are my means of self-locomotion — my keyboard features, my means of expression. I use my machines the way others use muscles. I don’t need autonomous technology. I need technology to make me autonomous.

I was born with muscles that die. I want technology to autonomize me and, as I lose muscles, to adapt to me. But the way technology is advancing today, it isn’t just leaving me behind. It’s setting me back.

When the dominant tech and corporate powers don’t help us, krips still find a way, and krips help each other. Liz Henry is starting a Disability Technology Foundation to open source information on assistive tech hacks. NMD United, whose board I’m honored to have joined, creates virtual spaces that people with neuromuscular disabilities use to troubleshoot their echairs. Abdullah’s design studio aims to show how technology should serve disabled artists in achieving the highest forms of human creativity. David Ortiz advocated from within the Colorado state government for the United States’ first right-to-repair law for echairs. And allies such as Dmitrii Votintcev and my kind Chorda team have been realizing my idea for a vertical hardware keyboard that I’d find a joy to use, much better than the one I’m using to write this article.

I wish I could take access for granted. I wish companies would realize that disabled people are the ones who need technology the most, when the entire point of technology is to expand what people can do. I wish I didn’t have to worry about fighting to keep old things just as much as fighting for better things. I wish I didn’t have to fight at all. And I wish I could easily input text.

My machines are part of how I exist, and therefore who I am. Obsolescence, whether planned or permitted, renders parts of me obsolete. End-of-lifed.

<EOF>

This piece is part of Immerse’s 2023 issue centering disability innovation in documentary and emerging tech — presenting perspectives from artists, activists, scholars and technologists at the vanguard of storytelling and disability justice. You can find other featured stories and more information about the issue here.

For more news, discourse, and resources on immersive and emerging forms of nonfiction media, sign up for our monthly newsletter.

Immerse is an initiative of the MIT Open DocLab and Dot Connector Studio, and receives funding from Just Films | Ford Foundation, the MacArthur Foundation, and the National Endowment for the Arts. The Gotham Film & Media Institute is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing emerging nonfiction projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.

How Innovation Sets Me Backwards was originally published in Immerse on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dr. Frank Mondelli is a UC Davis postdoc and historian of the material culture of disability, sound media and pop culture in 20th century and contemporary Japan. I had a chance to sit down and talk with them about their project Words in Noise (2021), a five-track 15 minute musical EP you can listen to here (or below).

The EP is quite short, each track abstracting a different hearing exam from the “Pure Tone exam” to the titular “Words-in-Noise exam,” layering in striated jazz chords, found sounds from the hearing exams in question and its fair share of colorful characters, from doctors to the artist themselves. We experience each track as someone going through “routine” hearing exams, prompted for comprehension and speaking as the tracks oscillate between medical prompt, music and controlled chaos. From start to finish, Words in Noise subverts, probes and questions its listeners — after all, it is an exam.

For me, Words in Noise represents a sonic “counterattack” of sorts against the archival erasure of Deaf voices, a singular and oft ignored experience of how a hearing exam could be felt. The EP is as musical as it is informative, subtly embedding critiques about medical diagnosis and hearing centrism amidst (for lack of better words) sick beats.

It is a thoughtful glimpse of an academic stepping outside their comfort zone to tell a different kind of story. More than a tool to promote empathy or education among hearing audiences, the narrative speaks of Mondelli’s experience being hard of hearing and as a scholar of disability culture and Deaf studies.

In this interview, edited for clarity and length, we work through their creative process and the theoretical backbones of such a project.

Julie Fukunaga: Could you talk a little bit about making your EP Words in Noise and how it originated?

Frank Mondelli: The longer I’ve been in academia, the more I’ve come to value personal and creative expression. I came to this in the middle of my PhD program. After I passed [my oral exams], I thought, “Man, like the only thing I’ve been doing with my life is just studying” — and working really, really hard to understand other stuff, of course, in preparation to contribute [to academia]. But I thought, “Wow, I really need a creative outlet.” I turned towards music as a way of doing that.

And at first, I didn’t have any desire or ambition to make original music. I just wanted to learn how to play the piano. But then as I got to Japan in 2019 to do my dissertation research [on the history of technologies for hearing impairment in Japan], I was getting really inspired by my archival work, which fits in, of course, with the history of testing hearing and electronic standardizing what counts at Deafness versus what does not. Part of the reason I was doing that research in the first place is because I myself am hearing impaired. And I have a history with hearing exams myself — of course, at a very different time and place from my research.

And I wanted to creatively think about what it means to be tested from more than just a historical, archival theoretical point of view. And so for my EP, for each track, it is each track represents a different kind of hearing exam, telling a loose narrative that’s more or less based on my own life.

JF: Could you talk a little bit more about how archival work shows up in this particular project? What are some of the different tracks you’re using, the different voices you’re using?

FM: So in terms of archival work, as a very rough rule of thumb the further back in history you go, the less and less you’re going to bump into Deaf voices. I was spending a lot of time with voices of people presuming to talk for other people’s experiences. I feel like we as disabled people often run up against [that].

For example, the academic Allison Kafer in her book, Feminist Queer Crip (2013), for example, talks about notions of “disability futures,” [in the context of] futures that non-disabled people tell disabled people that they have. This could be quite a violent act. It could be quite invasive, it could be uncomfortable. And I feel like that’s tied in with a lot of testing culture, in terms of testing for hearing and such. So I think, again, rather than rip stories from my archival work and make songs about them… it was feeling, “Man, I just I want to hear a Deaf voice.”

You can sometimes get at the Deaf experience through reading in between the lines. And so for the EP most of the voices you hear are not my own, they’re doctors or other children, other kinds of sound effects, other kinds of disordered voices. I wanted to both represent the fact that [you can] “hear” a Deaf voice in the midst of all that noise, but I also wanted to flip that around by the end of the EP as well and center my own voice and center one Deaf experience. And I try to be very clear that this is one person’s experience and only one aspect of one person’s experience as well.

JF: At the end, you start asking questions of the audience. To define particular notions of disability studies, or ask if they understand these words, and [ultimately you] invite the audience to find that information and to “join us,” which I think are really compelling last words. I’m curious if you could talk about how you embedded your own voice throughout the process of creating the EP.

FM: Sure. I really struggled with what precisely I wanted to say at the very end. And it was funny because writing the other doctor characters was really kind of easy for me. But then, when you write in your own voice and you’re talking to the audience directly, it feels really important to get that right. I go back and forth on how effective I was at the end of the EP, but there was one thing I knew I wanted to do, I wanted to perform a test myself.

I don’t wanna just give another hearing test at the end. I want to queer the exam, or I want to subvert it in some way, and I want to really take part of the spirit of the exam, but use it in a way that is inclusive and inviting of a wide audience. So in some ways I’m very nice to the listener at the end. I think it’s, it would’ve been very easy to strike a more angry tone, strike a more confrontational tone. I think that has a time and place for sure. You often don’t get social stuff done without making people uncomfortable at best, and really uncomfortable further on.

JF: How did you make decisions regarding who to curate the experience for and where, if at all, to include sort of explanations or education for hearing, for able-bodied audiences, for people who may not share these experiences with you?

FM: I try to be really guiding to the listener. It’s a short EP, about 15 minutes. About one third of that is much more experimental music, and that comes right in the middle of it. It’s like the main showcase of the EP, which is the pure tone test part.

And so in this case, lots of people may have taken hearing exams, but those experiences might have been so unremarkable that they barely remember how they worked. So the explanations of each test at the beginning of each song is a way of making it very clear here’s how this exam works. If I had just been making the EP for myself, for example, I wouldn’t need those kinds of explanations. But they’re there to help the listener understand even when the music gets abstract.

JF: [Probing] more about how you included these moments of discomfort or antagonism for audiences who would engage with the work: One track that jumps out to me in particular is track four — the Words in Noise Test, where you’re asking [the listener], “Can you say isolation?” “Can you say seclusion?” “Insecurity?” and having these really uncomfortable feelings droning out into nothingness. I’m really curious about your artistic decisions there.

FM: I think in terms of artistic decisions, you know, it’s funny ’cause growing up when you take some of these exams, a lot of the words that they make you say are mundane, like “boat” or “rabbit” or “dog.” But I have distinct memories of being so unsure what those words are and straining and struggling to hear.

I’ll speak only from my own experience. But when you’re uncomfortable, when you’re feeling isolated and somebody is saying something to you and you have to strain and struggle to interpret it, it’s almost like your brain sometimes goes towards the words that describe the emotions you’re feeling. And then when you do interpret, it can be something that’s almost revealing about yourself and your own feelings, which is of course prompted and partially created by the environment you’re in.

Another aspect of it in that track, Dr. Marigold says something to the effect of, “It’s not whether we understand you. It’s whether you understand us.” And that I felt I also wanted in that moment both the representation of the test taker’s emotional experience, as well as a representation of what the whole institution is working to do.

At the same time, I use a lot of distortion, I use a lot of audio effects on my voice, on Dr. Marigold’s voice. I also bring in a sound of a crowd as well because that’s what the Words in Noise Test in particular is supposed to be. It’s like a cocktail party test. And so I wanted to accomplish both confrontational … but also surrounding that with music that at least I thought was pretty or interesting.

JF: Whether pulling from this project or any others, what does it mean to you to center Deaf identities and experiences in your own artistic work?

FM: I would say within [the last 10 years of] disability studies, it’s been becoming even more exciting recently because we’ve, we’ve started getting lots of concepts that really are about the world from a disabled person’s point of view. One recent academic term that’s come up more recently is the “Disabled Gaze” or “disability gaze,” which is a play on words like male gaze or non-disabled gaze. There’s “Deaf gain,” which I think is about five years old now as a term. So many people focus on hearing loss, but what do we gain from Deafness as well? [And] “aesthetic access.” Rather than focusing on stuff like ramps for buildings, aesthetic access is more about what happens when disabled people are given tools of creative self-expression and are less shut out of things.

And again, I’m speaking only to my experience even within the Deaf world. I’m not a native signer. I didn’t grow up in the capital “D” Deaf community as well. So my perspective is, you know, quite different. We all have individual perspectives, which is exciting. In my work, I wanted to center my own experiences, but gesture towards larger institutional relationships with social, political, cultural moments in society.

This piece is part of Immerse’s 2023 issue centering disability innovation in documentary and emerging tech — presenting perspectives from artists, activists, scholars and technologists at the vanguard of storytelling and disability justice. You can find other featured stories and more information about the issue here.

For more news, discourse, and resources on immersive and emerging forms of nonfiction media, sign up for our monthly newsletter.

Immerse is an initiative of the MIT Open DocLab and Dot Connector Studio, and receives funding from Just Films | Ford Foundation, the MacArthur Foundation, and the National Endowment for the Arts. The Gotham Film & Media Institute is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing emerging nonfiction projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.

“Words in Noise”: Looking for a Deaf Voice Amidst the Noise was originally published in Immerse on Medium, where people are continuing the conversation by highlighting and responding to this story.

An interview with multisensory artist Salima Punjani on fostering intimate connection in immersive art installations

Salima Punjani is a Montreal artist working in installations and facilitating participatory environments, connecting audiences with access through multiple senses and a diverse set of media. Her art centers welcoming and counters alienation. Initially working in journalism to highlight under-documented communities, her recent pieces are multi-sensory, including soft sculptural installations as sources of sound, vibration and scent, as well as works that incorporate biometric data like recorded heartbeats and brainwaves. One that has particularly foregrounded co-authoring with disabled people is Progression, created in 2018.

Progression features MRI brain scans of several people with multiple sclerosis (MS), including Salima. Each participant contributed an MRI image and a photo portrait they wanted Salima to layer together to offer a narrative around cognitive change that might challenge assumptions.

Besides the photographic composites, she laser-engraved wood panels to reproduce fine detail from each subject’s brain scan, providing tactile, non-visual access to the pattern images. She also recorded and sonified each participant’s brainwaves while they were interviewed about their portraits. Then, she created small stuffed pillows containing vibrotactile transducers to send each subject’s distinct wave patterns through in tactile pulses, so exhibit attendees could feel them while touching the pillows, as a non-hearing mode of access. Wanting to familiarize attendees with these technologies in a non-intimidating way, Salima’s been present to offer introductions for trying on Muse headsets available in the installations, to directly experience the EEG headsets’ brainwave-sensing transmission to a vibration pattern format.

The most recent public activities she’s shared have been tactile/listening events with groups of youth or adults, facilitated online or in person outdoors. Attendees are introduced to vibrotactile pillows, which vibrate according to sound samples that each participant contributes. In the events, people then take turns experiencing and commenting on the sonic and haptic patterns they perceive.

Since meeting Salima soon after I moved to Montreal in 2017, I’ve been struck by her care in fostering artful narratives that focus on connecting with both the subjects featured in her projects and the audience, providing multiple entry points to enable inclusion and agency for people with a range of embodiments. I’m excited to share this interview conversation, edited for clarity. Find more on her work at: https://cargocollective.com/salima

seeley quest: Can you talk about your focus of inviting reflections on community experiences, and on choosing to explore different technologies for entry modes?

Salima Punjani: Community experiences are important to me. I don’t feel there needs to be a dichotomy between community or socially engaged art and what’s considered professional art. Working in community with technology, like [hosting] sensory listening workshops, is quite meaningful for me … [such public activities are] some of my favorite things that I do. People are invited to bring a sound based on a theme: comfort or belonging or whatever feels relevant at the time, and we come together. So far I’ve held the workshops at a city park, and with [the artists’ centre] Ada X in person as well as online.

It creates an opportunity to deeply listen to each other, not just through words but through felt sensation and a way to have an immersive experience even if maybe we don’t have so much capacity. It helps people have a certain quality of presence with each other, and people give really meaningful feedback and interpretations to each other. And it’s simple — it’s just a sound and a vibration and the energy of us together. Moving forward in my work I’m heading towards a bit more simplicity; when it comes to technologies I’m trying to simplify things because constantly figuring out new technology [isn’t that accessible for me].

sq: You’ve been working on projects involving translation: renderings of relationships to neurological change by people with MS, transmission of sound to pulse patterns that can be felt by Deaf audiences. How do you approach the concept of translation?

SP: When you’re feeling brain waves, you’re feeling the frequencies that I’ve chosen for you to feel, translat[ing] biodata in a way that isn’t to find something wrong, but instead builds connectedness between people.

sq: How are subjects responding to the presentation of their content? Are audiences responding in any ways you wouldn’t anticipate?

SP: I’m shocked [subjects are] so chill about using their biodata. I made Progression in 2018 and have exhibit[ed] it five times after that, even though people’s stories have changed. We joke about that; they actually are saying I should do an updated version with stories that are more recent. People are happy to share their stories and to dimensionalize the experiences of people with MS past the assumptions that exist.

Another piece, The Cost of Entry as a Heartbeat (2020) uses heartbeat data and sounds recorded in Budapest thermal baths to create a collective rest environment. For most people it’s very relaxing, yet for others the opposite; I’ve had some big reactions to the piece. Both of the times [it’s been installed], there’ve been people that had panic attacks or strong stress responses, because the heartbeat [sound] was too fast, or they hadn’t really had experience with such an immersive environment.

I’m grateful I have training in care work so I know what to do when that happens. It’s the reason I studied social work, to slow down and really think about how to make sure there are structures of care in my projects. There are often ways people react that aren’t what I anticipated in creating a project, and at the same time I feel relatively confident that I have skills to support if something comes up. [I take] the evocative potential of art seriously, not lightly at all.

q: Do you have goals regarding immersive encounters?

SP: Yes. I’ve had the opportunity to experience spatial sound in magnificent contexts: at Monom in Berlin, the Spatial Sound Institute in Budapest, and Lobe studio in Vancouver. My interests lie in bringing experiences to communities: disabled folks, BIPOC folks, people at intersections that might not have access to go into these sound studios, as there are only three in the world that use 4D sound technology. I’m currently, through support of the Canada Council of the Arts, working on developing a mobile spatial sound studio to bring into schools or different communities, to democratize the technology a bit. It’s in a space where people have to go to it now, but I’d like to bring it to people. It’s valuable to offer these immersive experiences of rest to communities underserved. So that’s big for me, to democratize vibrational multi-sensory environments.

This summer a collaborator, Greer Pester, and I worked on developing large vibrotactile soft sculptures based on body parts and we finished the project with making a ’zine: a version of an experience which can be mailed to people that has prompts and different materials that explore the senses. I’ve experienced powerful work, such as Ray Young’s THIRST TRAP (2022) through London’s Unlimited Festival that I had the opportunity to go to this year. THIRST TRAP was an at-home experience where you listened to an audio file, and received this box that had different materials. It was meant to be experienced in the bathtub and was about climate change …put together in a way that felt like a ritual — immersive through smell, light, water temperature and the depth of the water being so specific — and a bath bomb. Incredibly well done, it had great entry and exit rituals, made you feel part of something collectively [and was available for] a good length of time so you could access it whenever you had the capacity to spend that time to experience it.

I’m excited to see how these kinds of care or exhibition packages can support people in experiencing immersive work. I’m working on developing a project related to grief and collective care that will find a way to do this. [I’m] moving towards some simplicity, also play and pleasure in my work.

sq: Would you expand on your next directions?

SP: I’m working on this mobile spatial sound studio, with a grant from the Arts Council for two more years to dig into my research areas. I’m [building] a performance piece/ritual that invites letting go of something, holding something for someone else, and having some sort of collective release related to grief and loss to create a space where holding something for someone else feels possible without it feeling overwhelming. My activism is based on mutual care and mutual aid in a loving way, dealing with things collectively present and hard to hold, and finding ways to bring that into an artistic space that can give a tiny bit of relief.

I’ll host a gathering space for disabled artists at Trinity Square Video in Toronto, July 2023. Aimee Louw’s poem about being held in an unspectacular space was really inspiring to me. I’d like to host an informal gathering for artists to come together and just be with each other, without some event or conference.

In 2024 I’m heading towards curation at Tangled Gallery. I’ve been invited to create a show so I’m researching artists, just meeting people, visiting studios, talking to curators I admire, bringing learnings I’ve had from the disability arts community over the last few years to my curatorial practice.

This piece is part of Immerse’s 2023 issue centering disability innovation in documentary and emerging tech — presenting perspectives from artists, activists, scholars and technologists at the vanguard of storytelling and disability justice. You can find other featured stories and more information about the issue here.

For more news, discourse, and resources on immersive and emerging forms of nonfiction media, sign up for our monthly newsletter.

Immerse is an initiative of the MIT Open DocLab and Dot Connector Studio, and receives funding from Just Films | Ford Foundation, the MacArthur Foundation, and the National Endowment for the Arts. The Gotham Film & Media Institute is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing emerging nonfiction projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.

Embodiment as an Entry Point — At the Intersections of Art, Tech & Care was originally published in Immerse on Medium, where people are continuing the conversation by highlighting and responding to this story.

I. “Disability is an art.” — Neil Marcus

The history of dance and disability is fraught.

I refer to disability here as an identity, as personal and political practice. This is divorced from the medical model, which uses disability as a means-tested umbrella term for impairment, and from the social model, which examines the systemic and neglects the personal.

There have always been disabled artists, but a climate that views disability as critical context rather than debilitation has been until recently almost nonexistent. In the United States, small pockets of artists have sprung up more or less concurrent with our disability civil rights movement, often loosely united by shared access needs. The trend of recognizing disability as culture and practice is relatively new and institutions are struggling with the related but different concepts of access, equity and justice.

Community is one source of our strength. Often difficult, the work of uniting across varied bodyminds calls for a practice of de-centering: the conscious awareness that our way of being and sensing is not the only way, and that there is no best way. Disability is an identity which encompasses us however differently we exist in the world. We know that there is no such thing as normal: there is no justification to prioritize the way one person moves or communicates above another. Justice requires a shared commitment to bridging our radically different ways of being.

The multiple and varied experiences of disability offer fertile ground for artistry. What can we create when we decenter our individual, inherently limited preferences? What might we choreograph when we value each new embodiment that enters the room? The experience of difference and multiplicity of em(mind)bodiment is largely unique to disabled people; thus we claim the artistic fruits of infinite variety.

II. Reach out and touch someone

Experiencing performance visually is embedded in English. We go “see a show.” We sit at a remove and watch performers onstage. We are comfortable in the clarity of our respective roles and social scripts. And yet, there is tension between the passivity of viewing and artists’ goals of communication and interaction. Great art should change everyone involved.

Dance communicates primarily through movement: both large, like thrilling lifts or dramatic falls, and small, like the emotional nuance of the turn of a head or crook of a finger. Dancers train for thousands of hours to accurately communicate through this language of the body — and then we further augment the performer’s body with various technologies which create immersive and spectacular effects. Examples include wearable technologies such as the ballet dancer’s pointe shoe or aerialist’s unitard and environmental technologies such as lighting and projection, music and sound effects. Do these seem, well, low-tech? Behind the scenes, they are controlled by carefully programmed software cues and hardware ranging from a single laptop to racks of servers backstage. The hour or two we enjoy is the tip of an infrastructural iceberg.

And still, these widely used technologies cater to only two channels of perception: vision and hearing. Critics of ocularcentrism remind us that vision is our perception of reflected light; it separates the perception of a thing from the thing itself, creating objectification. There is safety in this perceptual, physical and psychological distance. Dance is a kinaesthetic art, transmitted most directly through taking on movement. To decrease separation we should turn to another sense: touch.

Touch is the most universal of our senses and the most biologically necessary: we do not thrive without skin contact. Many forms of perception live in our skin, with complex and individually different sensitivities in how and where our bodies experience touch. Kinaesthesis and proprioception are the truest senses of dance: the perception of our body in relation to itself and the space around us. Touch is devastatingly intimate, a direct form of knowing; someone close enough to touch us might be a threat or a lover. How can art reach out and touch us?

III. Haptic technology

You are likely familiar with haptic feedback: it’s the tiny vibrational blip with which your phone rewards you for interaction. It’s fast, effective and goes almost unnoticed except for the quick dopamine hit. But the potential of haptic experience goes far beyond this tiny affordance. Haptic is not quite the same thing as touch: it has specifically to do with our perception of objects. We can manipulate those perceptions by changing how our skin and muscles react using vibration or electricity. We can change the shape or texture of a physical object, create objects that don’t physically exist or imagine the conversion of ideas into something you can feel.

Towards a practical approach, haptics could be imagined as the intersection of touch and sound: after all, sound is our perception of vibration in air. Just as we use technology to augment the performers’ body, we can augment our receptive capacity. If you were seated onstage, you might feel secondary effects like the vibrations of the soundtrack from the speakers, from dancers landing on the floor. How might we imagine an experience of dance that is crafted to be felt?

IV. Practical Experiments

This is where I began my own explorations. Having experienced various kinds of vibration-based therapeutics, I had direct knowledge of how precisely calibrated vibration and tiny spurts of electricity can trick mind and body into perceiving intangible objects or imaginary movements. I wanted to explore three ideas: direct interpretation of sound; translation of proprioception; and communicating complex and abstract ideas.

VibraFusionLab is based in the city of Hamilton just 45 minutes west of Toronto. Founded by hard of hearing artist David Bobier, they are among the field leaders in developing new devices to record and transmit touch-based content. I met David at a gathering of international access tech leaders, and, through the screen, his snowy beard and twinkling brown eyes gave him the look of an impish, endlessly inquisitive wizard. Vibrafusion’s projects include devices held or worn by each audience member through human-scale constructions which can play recorded content or interpret bodily movement from artist-worn devices on the fly.

With Vibrafusion’s guidance, I assembled a simple implementation for Kinetic Light’s Wired (2022), an aerial meditation on barbed wire created and performed by disabled artists. Audience members hold a strand of wire which carries the show’s soundtrack interpreted into vibration. The music was vibration-ready: half of it created by a hard of hearing composer, half by a disabled composer working in layered sound and rhythm. The result was a striking and artistically communicative experience.

Kinetic Light, an intersectional disability arts company, espouses a practice of many entry points into our work. Audiences could have this haptic experience in combination with watching the stage or listening to the multi-track audio description delivered through the Audimance app. First tested in Wired’s Chicago and New York premieres, this technology proved so popular and powerful that we quickly adopted it for our other shows. And yet this only begins to touch (lightly upon) the potential of this new medium.

V. Centering disabled joy

Making art that values and centers disabled knowledge and joy means centering people with direct experience. We can find leadership in the medium of touch in DeafBlind practice. Created by and in DeafBlind community, Protactile is a language of the skin: perceived through touch, pressure and proprioception. It is distinct from tactile ASL (American Sign Language), which attempts to use touch and various forms of haptic feedback to translate visually-centered ASL to a nonvisual form. Protactile grew out of personal and cultural imperatives for direct experience, connection and communication.

Meaning never develops in a vacuum. We interpret language and art based on our cultural context, and we modulate our understanding according to the medium of our experience. Protactile and other touch-centered systems often use multiple interpreters to communicate complex meaning and rich aesthetic experiences. Small clusters of people form in space, self-organizing cells which blur the distinctions between artist, interpreter and audience. This might suggest challenges for larger audiences or smaller venues, and this is exactly where disability-centered technology can serve as a way to extend and amplify our artistic intentions. What can we reimagine when art is contextualized through touch and consensual, collaborative co-creation?

When access is segregated from the practices of creating and experiencing art, it’s easy to fall into a worldview where access is finite and scarce. If we consider that all forms of experience are equally valid, it becomes clear that access is part of the creative process and that both artists and audiences are necessary collaborators in performance. Tactile and kinaesthetic techniques and technologies are an excitingly direct way for dance and movement artists to share work across bodies and media, whether as literal as a shared proprioceptive encounter, or as abstract as a concept vibrated through a novel artistic language which blends physical, created and extended realities.

This piece is part of Immerse’s 2023 issue centering disability innovation in documentary and emerging tech — presenting perspectives from artists, activists, scholars and technologists at the vanguard of storytelling and disability justice. You can find other featured stories and more information about the issue here.

For more news, discourse, and resources on immersive and emerging forms of nonfiction media, sign up for our monthly newsletter.

Immerse is an initiative of the MIT Open DocLab and Dot Connector Studio, and receives funding from Just Films | Ford Foundation, the MacArthur Foundation, and the National Endowment for the Arts. The Gotham Film & Media Institute is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing emerging nonfiction projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.

How Can Art Reach Out and Touch Us? was originally published in Immerse on Medium, where people are continuing the conversation by highlighting and responding to this story.