During 14 weeks, students of the Interactive Media Arts BFA at Tisch School of the Arts (NYU) have the opportunity to explore the theoretical concepts along with hands-on experiences regarding creative coding and machine learning. Tools like Teachable Machine, RunwayML, TensorFlow.js and our dear ml5.js are explored, tested and discussed while students critically think about data collection, ethics and bias. All the different topics explored during those meetings evoke different interests and possibilities of exploring creativity with the use of Machine Learning and the class culminates into a final project presentation where students get to showcase their art pieces.

Here are some of the final projects created by the students from the 2023 Machine Learning for the Arts class using ml5:

Elemental Mastery by Aditya Pandhare

Elemental Mastery allows you to empower yourself to seamlessly control the four elements of fire, water, air, and earth, by unleashing a symphony of creativity and exploration and a selection of some amazing fighting moves! This project uses ml5.js next-gen handpose, speech recognition and particle systems.

(Video | Documentation | p5 sketch)

Picky Art Critic by Andrew Feng

This project is a DoodleNet Classifier which outputs different responses based on confidence level, as well as ideally a less strict response.

(Documentation | p5 sketch)

Strange Loops by Annie Zhu

Strange Loops is an exploration into generative deep learning models and self-referential patterns, questioning the notions of identity and the uncanny. The creator trained a machine learning algorithm (StyleGAN2-ADA-Pytorch) on a database of selfies, then had it generate its own version of the pictures; from there,they took it to Google’s Trainable Machine and ml5 Image Classifier to see if it was able to distinguish the real face from the imposter face. The final result is a short lenght film that functions both as documentation of the outcome as well as a standalone work of art

(Video | p5sketch)

ml5 KeyPoints Tool by Connie Hu

This project is meant to be a tool to help beginners get started with using ml5 pose estimation models in their creative coding projects and also for experienced user to prototype easily.

(GitHub repo | Documentation | p5sketch)

Fit Kit by Zoey Zhang & Vicky Wang

Fit Kit is a personal training platform combining multiple p5js sketches using the ml5 pose classifier. It reminds us of the right exercise movement and monitors workouts by tracking the user’s motion.

(p5sketch | p5sketch | p5sketch | p5sketch)

HandGun AR by Joonha Yu

HandGun AR aims to integrate the physical and digital realms, enabling users to interact with a virtual environment using their fingers as simulated pistols.

(Documentation | Live Example)

Emocean by Sirui Wang

This project is an envisioned technology that listens to your unpleasant feelings/thoughts/concerns to help you unload the mental baggage, relieving you from the need to burden your friends and family with these heavy thoughts

(Documentation | Live Example)

Glitch Identity by So Yeon Kim

This project seeks to reveal and critique how machine learning algorithms may categorize human and real-world diversity into oversimplified and stereotyped groups. It engages with audiences by scanning their faces, yet assigning a deliberately randomized identity, such as ‘A dinosaur that likes to dance,’ to underscore the absurdity and inaccuracies intrinsic to these algorithms.

(Documentation | Live Example)

TOUCH by Yizhen Xiang

TOUCH system works as a catalyst for profound human connection, where following its instructions becomes a journey of getting closeness — bridging not only physical distances but fostering a mental togetherness. It uses webcam and the hand pose model to estimating the distance between two hands (from two people) to encourage people getting closer to each other by following the instructions on the screen, such as touching fingertips, clapping, interlocking.

(Live Example)

Breathing With MossPillow by Yuntian Zhao

MossPillow is a biodesign installation that can help people deep breathing and regulate their memory and emotions. This project uses raw plants and materials and incoporates ml5 by using PoseNet and FaceMesh to play nature sounds.

(Video | Documentation | Live Example)

As always, make sure to follow us on social media to stay connected and see everytime we share new projects our community has been creating with ml5 (or to share your own creations with us)!

Follow us on X and join our Discord channel.

This blog post was written by Micaelle Lages and Yining Shi.

All images used in this post belong to the respective creators for each project displayed. All inspirations and resources utilized are their responsibility and should be acknowledged in their own documentation.

Today, we are asking you for your help in improving ml5.js and more specifically, our community. It’s been two years since we last updated the Code of Conduct, a document that governs our community and ensures the safety of each and every user and contributor of ml5.js.

We are currently in the process of building the next iteration of the ml5.js core library — a version with state-of-the-art machine learning models, robust technical features, and a narrower scope. The new ml5.js prioritizes the needs and experiences of students, artists, and other machine learning beginners. We believe that everyone should be able to experiment and build with AI tools, regardless of their experience with computer science or software engineering. While this is a work in progress, we believe that our Code of Conduct should evolve alongside the ml5.js library as well as the world around us.

Therefore, we want to hear your voice! We invite you to share feedback on the current state of the ml5.js community and our Code of Conduct. What’s missing? What needs to go? What does it need more or less of? What do you think can be improved? We need your input in order to make sure that we are making changes that truly benefit our community.

There are two ways to provide feedback. The first is to fill out this brief google form. Alternatively, you can create an issue directly in the Code of Conduct Repository in GitHub. If you’re not sure how to create an issue in GitHub, here’s an article to help!

We encourage you to take a look at our previous Call for Comment which outlined our intention and approach to the Code of Conduct and Software License. Additionally, here’s our Code of Conduct and License Announcement from 2021.

You can always reach out on Twitter or on Discord. We look forward to hearing from you!

– ml5.js team

Jacqueline Li is a fifth year at Northeastern University studying Computer Science and Design. She recently completed her capstone project the Synesthesia Lens, which is a synesthetic audio visualizer using ml5.js and p5.js. Continue reading to learn more about her work!

Interview by Ozioma Chukwukeme

Can you tell us more about yourself?

JL: I’m currently a fifth year undergrad student at Northeastern University graduating in May. I grew up in Virginia and moved to New Jersey for middle school and high school. I always had a preference towards STEM — I liked math, but wasn’t very good at it, and I liked physics. But even so, in high school I didn’t really know what I wanted to do in college. The funny thing is that I hated computer science in high school and didn’t want to pursue it in college. My mom kept telling me ‘You should do it, it’s really easy’ but I thought it was confusing and difficult, so I thought it would make sense to come into college undeclared as an engineer, as it still scratched that STEM itch in my brain. After a semester, I learned that engineering just wasn’t for me. So, I decided to give computer science a second chance; I switched majors from engineering and eventually settled on a combined major in computer science and design.

The reason why I chose design as the second half of my major was design was that I was always an artistic kid, but never thought of that as something I could pursue professionally. After exploring a bit more, I realized the potential that it had for careers, such as UI/UX, product design, and frontend development. Something that really sparked my interest though was going to Zach Lieberman’s ARTECHOUSE exhibit in DC. I genuinely thought, and still think, ‘That is one of the coolest things I’ve ever seen.’ It was a direct intersection between CS and design that I didn’t know existed before, and it became a niche I wanted to explore.

Tell us about your capstone project; what was your inspiration and approach?

JL: So the prompt for the capstone project was very broad — the purpose of design capstones is really just to allow senior design students to work on whatever passion project they want to do using the skills they learned in undergrad so far. Going into it I knew I wanted to create an interactive installation with creative code because I never had the outlet or space to do that during my time at Northeastern. In the beginning of the semester, I was playing around with p5.js and doing a lot with audio and randomly generated bubbles. During one of the crits, my professor mentioned that I should look into synesthesia and specifically Alexander Scriabin, a composer that had synesthesia who saw specific colors for specific notes.

I wondered how I could visualize this as I don’t have synesthesia. Eventually, I figured that I wanted to showcase the fleeting nature of sound in addition to the colors that Scriabin saw when he heard music. To accomplish these tasks, I decided to randomly place lines on the x-axis with a color that is mapped to the note that is being picked up through the microphone. Immediately after the line is projected on the screen, it begins to fade away. I ended up titling this project the Synesthesia Lens to highlight that this allows you to see through Scriabin’s synesthetic lens.

If you’re interested in the nitty gritty, I used ml5.js to figure out the frequency of the audio data. Then, I converted each frequency to the note it falls under and visualized that via a line with a color of the specified note, as Scriabin saw. If you wanna see the exact mapping of notes to colors, check out the about section of my project website!

What were some of the challenges you faced while creating this project?

JL: I kind of went into this project blind because I didn’t have too much experience with creative code, so it was roadblock after roadblock. I found videos from The Coding Train on Youtube which helped me start when I didn’t know how things like the draw() function worked since I was so used to object oriented programming.

Once I got deeper into the project, one of the bigger challenges was figuring out how to get frequency data. I was initially using the p5.js FFT (fast Fourier transformation) library to get the frequencies but I was having trouble with it and thought there had to be an easier way. Eventually, I found The Coding Train’s Ukulele Tuner video and decided to use ml5.js.

What drew you to use ml5.js and p5.js over the others tools out there?

JL: I think it’s really the web editor for p5.js — its so fun and easy to get started. My initial plan was to work in p5 and see the extent of its capabilities, and if I needed more, then I would look into Processing. However, I felt that for this project p5 was more than enough. I didn’t even know ml5 existed until having the trouble with frequencies–and that was a godsend!

Did you have previous machine learning experience?

JL: Surprisingly, I don’t have any prior experience with machine learning. At my university, the mandatory classes we had to take were around the fundamentals. So my first computer science class there was in a language called Racket — it’s a functional programming language to get you thinking like a programmer and it was very difficult to learn. Some of my friends took machine learning as an elective, but I wasn’t interested in it at the time because I wanted to go into frontend and learn more about web development. So I didn’t have previous experience with machine learning but ml5.js made it super easy to understand, and I am interested in learning more about machine learning and AI.

Do you have any future plans for this project?

JL: My initial plan for this was to create several visualizations for the Synesthesia Lens, but because of the time constraint of the capstone I wasn’t able to. Over the summer, I plan on making more interactive and unique visualizations for this project. I also want to look into generative art and creative code and hope to do more with that in the future!

Be sure to follow Jacqueline’s socials to stay up to date on her work:

Live Project: https://www.synesthesia--lens.com

Jacqueline’s Website: https://www.jacquelineli.com

Instagram: @didotfanclub

TikTok: @influenczar

If you have a project you’d like to submit to be featured on our socials, submit it through this form.

Disclaimer: This post contains content that may not be suitable for minors or younger audiences. Reader discretion is advised.

Intro to ML for the Arts is a course in the ITP department at NYU Tisch where students can learn to create interactive web-based projects with the help of beginner friendly machine learning tools, taught by Yining Shi.

‘Quick, Madlibs!’ by Hannah Zhao and Carolina Herrera

Quick, Madlibs is a game that uses Google’s Quick, Draw! to fill in the blanks of various stories, Mad Libs-style

Tools Used: ml5js, p5.js

(Video | Documentation | Live Example)

‘NN Playground’ by Ziyuan Lin

Inspired by Teachable Machine. NN Playground is an in-browser graphical interface that provides both an intuitive and comprehensive approach to training machine learning models. It will provide us with a shortcut to our dream application or project.

Tools Used: ml5js, React

(Video | Live Example)

‘Performer — Canon remix’ by Jiayi Tan

Users use hand pose and facial features to interact with the elements, and perform the “Canon In D Major” that belongs to you.

Tools Used: ml5.js

(Video | Documentation | Live Example)

‘Choose your best Nike Shoe’ by Jasper Zhao

You describe features of a running shoe you want, and AI helps you choose which Nike running shoe is best for you.

Tools Used: ml5.js, Teachable Machine, RunwayML, p5.js

(Video | Documentation | Live Example)

‘AI Artist’ by Mingren Fu

AI Artist is a tool that will transform users’ imaginations into photorealistic images. In terms of how it works, each color on the canvas will correspond to a specific type of element (such as mountains, rivers, trees, etc.) in nature. Based on the shape and color of the doodles that users draw on the canvas, AI Artist will identify the associated nature elements and generate an output as a photorealistic image. Users can control and interact with my project by either clicking the buttons or making verbal commands such as “mountain” or “generate” through speech recognition.

Tools Used: RunwayML, p5.js

(Video | Documentation | Live Example)

‘MagentaMuse’ by Anzhelika Nastashchuk

MagentaMuse is a web-based tool that allows users to generate, manipulate, upload, and save MIDI files. This application was built using Magenta.js, a JavaScript library for creating music using machine learning algorithms. Two different models from the Magenta library were implemented: a MusicVAE and MusicRNN. This project was done specifically for music producers or composers who would use this application in case of writer’s block or for inspiration.

Tools Used: Magenta

(Video | Documentation | Live Example)

‘A drawing of a full body beautiful girl, young, hyper-realistic, very detailed, intricate, very sexy pose, unreal engine, damatic lighting, 8k, detailed, black and white.’ by Jiwon Yu

A mixed media project that explores interaction with a traditional media: the goal of the project is to protray the suffocating gazes that women recieve in the digital world. The audio of women breathing, will stop when a face is detected in front of the painting. Then, the audio resumes when the user gets out of the frame.

Tools Used: Posenet, RunwayML, p5.js

(Video | Documentation | Live Example)

We look forward to showcasing more incredible work from you all in the near future. Don’t hesitate to submit your project using the form above, we can’t wait to see what you create!

Today we are excited to announce two new parts of ml5.js: a new and improved Code of Conduct and our very own software license. These are part of our ongoing work to make machine learning for the web friendlier. This blog post explains how these parts work and how we got here.

For more than a year now, we have been working to understand a central question: how can we use the rules that govern the ml5.js community to encourage friendly, creative applications of machine learning?

In January 2021, after a year of internal discussion, we asked for community feedback on a draft version of a Code of Conduct and software license for ml5.js. We explained that our goal was to keep ml5.js grounded in the spirit of open source, while reducing the chance that ml5.js could be abused for purposes we feel run counter to the expectations of the ml5.js community.

Since then, we received feedback through a number of open channels (for example, Github issues and Discord) as well as through more directed, one-on-one conversations and feedback sessions (such as meetings with experts in creative technology, machine learning, community development, and open source). We have integrated that feedback into today’s versions of the Code of Conduct and software license.

The Code of Conduct and software license work together to affirm the importance of our neighborly approach to creating and exploring machine learning. This neighborly approach means a recognition of one another, extending care for the people around us and the places that we touch through the things we make — in this case through machine learning. Our Code of Conduct and software license are a recognition of the fact that machine learning is a powerful tool; it is as much about math, statistics, and data, as it is about the ways that power is employed in the world. It can be used for creative exploration that surprises and delights and improves the world. Unfortunately, machine learning can be employed (knowingly and unknowingly) to bring harm to real people, places, and things. We’d like to do our best to minimize these negative effects as much as possible.

The Code of Conduct describes how we expect everyone to behave while they are in shared ml5.js community spaces. Additionally, the software license requires projects that incorporate ml5.js to follow the rules of the Code of Conduct.

The Code of Conduct

The ml5.js Code of Conduct describes the expectations for how everyone behaves within the ml5.js community. That includes community spaces like workshops, GitHub repositories, and online discussion areas.

The Code of Conduct draws inspiration from and builds upon community statements and codes of conduct from the open source and creative arts community, such as the p5.js community statement and code of conduct. It also includes examples of welcome and unwelcome behavior while building projects with the ml5.js library and while in ml5.js community spaces.

It is important that members of the ml5.js community know that the Code of Conduct is an active part of the community. That is why it includes information about asking how your project matches up with the Code of Conduct’s expectations, how to report possible violations, and about how we will handle those allegations.

We know that ml5.js continues to evolve, and that the ways in which machine learning can be used for learning and creativity are hard to anticipate. That is why we have designed the Code of Conduct to be a living document.

We expect the Code of Conduct to evolve in at least two ways. First, we know that the text may change. This could involve adding, removing, or editing some of the expectations from the document.

Second, the Code of Conduct balances being broad in its scope with the need to provide clarity should specific situations occur that break or violate the Code of Conduct. We hope that these situations are uncommon, but we do believe that the Code of Conduct will help all ml5.js community members to understand the ml5.js rules and expectations. We will work to make these situations — e.g. the process of addressing breaches in the Code of Conduct and how we resolve them — public so that the entire community understands how the Code of Conduct works. We expect that these situations will help our collective understanding of the Code of Conduct evolve over time.

Code of Conduct Committee

The Code of Conduct Committee’s job is to help guide the Code of Conduct’s evolution and help apply its general rules to specific situations. The initial Code of Conduct Committee is made up of:

• Christina Dacanay
• Lydia Jessup
• Sam Krystal
• Joey Lee
• Ashley Jane Lewis
• Bomani Oseni McClendon
• Daniel Shiffman
• Michael Weinberg

The Code of Conduct Committee is responsible for enforcing the Code of Conduct. It is also responsible for making sure that the Code of Conduct Committee is sustainable and represents the ml5.js community.

For now, that means that we expect at least some members of the Code of Conduct Committee to rotate off every year. That gives Code of Conduct Committee members a chance to step away from their responsibilities and provides an opportunity to new members of the ml5.js community to take part. The Code of Conduct Committee also intends to explore ways to make it easier to participate. That includes defining sustainable workloads and seeking grants to recognize the labor required to be part of the Committee.

The License

The new ml5.js software license uses copyright law to require projects to follow the Code of Conduct. Using ml5.js requires you to promise that your project follows the Code of Conduct. Our license is a modified version of the Blue Oak Model License 1.0.0, which is a modern permissive license.

Although ml5.js is far from the first project to explore ethical licensing, there are a few elements of this new approach that we want to take a moment to explain.

Separate the License from the Code of Conduct

One of the hardest parts of designing an ethical open source license is finding a way to describe what is considered “acceptable” use under the license. Instead of trying to describe all of these “good” and “bad” uses in the license text, we are linking the license to our evolving Code of Conduct.

We think this helps in a few ways. First, the Code of Conduct is easier to read for most members of the ml5.js community, who are not formally trained as lawyers. We hope that makes it more approachable than immediately asking our community members to parse through formal legal language.

Second, it allows the rules around behavior to evolve without having to revise the license itself. It is easier to update the Code of Conduct than it is to try and revise licensing language.

Empower the Code of Conduct Committee to Review Alleged Violations

The Code of Conduct is broad, and there are many ways that people might use ml5.js. We do not want creators to worry that a misunderstanding about how the Code of Conduct applies to a specific project could result in a violation of the software license. The Code of Conduct Committee must investigate and agree that a project violates the Code of Conduct before the project formally violates the license. We hope that this will make it more likely that the Code of Conduct will be enforced fairly.

License Decay

Copyright protection can easily last for over 100 years. That means a poorly thought out software license can create problems for decades into the future. We have incorporated decay into our license in order to try and avoid these types of problems.

The ml5.js library is made up of thousands of individual commits (in Git terms) by individual contributors. For the first three years after each commit is made, that commit will be licensed under our new software license, which will require users to comply with the Code of Conduct. That means that versions of the ml5.js library that include commits created in the last three years are covered under the license. If you are using a version of the ml5.js library that has been updated in the last three years, you are also bound by the software license. After three years, the license decays into the very permissive Blue Oak Model License 1.0.0.

In practice, this means that an actively developed ml5.js will be licensed under the ml5.js software license because some of its components will always be less than three years old. Projects that use the current version of the library will be required to comply with the ml5.js Code of Conduct. However, the decay helps us address three concerns involved with trying out our newly developed, ml5.js-specific software license.

Concern #1: The license relies on an active Code of Conduct Committee. The first danger is what would happen if ml5.js was abandoned by the community. The ml5.js software license does not really work without an actively developed Code of Conduct and Code of Conduct Committee to oversee it. License decay means that the Code of Conduct will no longer be binding three years after the last commit if the Code of Conduct and Code of Conduct Committee are no longer active.

Concern #2: The Code of Conduct could evolve in unexpected ways. The second danger is what would happen if the Code of Conduct evolved in a way that parts of the community were no longer comfortable with. In this case, that part of the community can rewind to three years before the change, fork the project, and move in a different direction. This also means that project creators do not need to worry about a change to the Code of Conduct years after they create their project. After three years, the old version of ml5.js used by an archived project will no longer be legally required to comply with the Code of Conduct as it evolves.

Concern #3: We want to account for our own learning. The third danger is simply that this new approach to ethical licensing turns out to not work as intended. The license decay feature makes it much easier to change course if we need to do so.

Is ml5.js Still Open Source?

The ml5.js community is rooted in open source approaches to software development. ml5.js is built on other open source software — the machine learning functionality and pre-trained models of ml5.js are built entirely on top of TensorFlow.js, the JavaScript implementation of Google’s end-to-end open source ML platform Tensorflow licensed under Apache-2.0 License — and is developed in an open, collaborative manner.

We understand that there is an ongoing discussion in the open source software community around how ethical concerns might be integrated into open source software licenses. We understand that incorporating ethical concerns into open source licenses violates the terms of the Open Source Definition maintained by the Open Source Initiative. That may mean that some people will no longer view ml5.js as open source software. As we wrote in our call for comment, we respect the fact that different parts of the community view this question differently.

As a community, we remain committed to developing ml5.js in a way that is open, inclusive, and encourages others to use, build upon, and remix the library. We have also decided that the cost of losing some users who are uncomfortable with our approach to an ethically-grounded license is outweighed by the importance of building a community focused on ethical uses of machine-learning. There are many alternatives to ml5.js for those who are not interested in our approach.

Can Other People/Projects Use This Approach?

Yes please!

Our approach builds on the work of many others. We encourage everyone to take what they want from it. Use this approach as a starting point, copy it exactly, or whatever else makes sense for you.

The Code of Conduct is licensed under a Creative Commons public domain dedication. That means that anyone is free to copy it, build upon it, and remix it without any restriction or obligation to credit ml5.js. We chose to use CC0 on the Code of Conduct because we know it can be hard to give legally correct attribution when you are borrowing parts of existing work to create something new. We did not want the kind of formal attribution requirement of other Creative Commons licenses (like a Creative Commons BY-SA license) to get in the way of reuse by others.

The software license itself is licensed under the same permissive terms as the Blue Oak Model License 1.0.0.

Of course, even though we do not require it, we would love a shoutout if you do use it.

Thank You!

We want to end this post with a big thank you to everyone who took the time to review drafts, discuss choices, and suggest alternatives. This has been a long undertaking and would not have been possible without your help. While it would be impossible to list everyone who has helped, we do want to especially mention:

• Coraline Ada Ehmke
• Albert Fox Cahn
• Austin Gillett
• Nabil Hassein
• Jesse Kirkland
• Jilayne Lovejoy
• Lauren Lee McCarthy
• shawné michaelain holloway
• Kyle E. Mitchell
• Sarah Pearson
• Caroline Sinders
• Luis Villa
• Aaron Williamson

We are mentioning all of these people to thank them for helping us develop this approach, not because they necessarily endorse the final outcome. Any mistakes or poor choices that we made in crafting this approach is in spite of — not because of — their generous feedback.

In a similar spirit, we would like to acknowledge projects that we have drawn language and inspiration from. As with the people who helped us develop this approach, while we found these projects to be incredibly helpful none of them necessarily endorse the final outcome.

Projects that we have drawn language from:

• p5.js — Community Guidelines
• Cloud 9 — Community Guidelines
• Contributor Covenant — Code of Conduct
• Open Source Guides — Writing Codes of Conduct
• Blue Oak Council — Model License

Other projects that we have drawn inspiration from:

• The Creative Independent — Designing for Diversity
• UnifiedJS — Moderation Policy
• TC39 — Code of Conduct
• Rust — Code of Conduct
• United Nations — Universal Declaration of Human Rights
• Cities for Digital Rights — Declaration
• The Hippocratic License — An Ethical License for Open Source Projects
• The Anti-Capitalist Software License
• MIT License

This is the first version of our Code of Conduct and software license. It will probably not be the last. We encourage you to raise specific concerns by filing an issue in the repo, reaching out on twitter or Discord, or via email at feedback@ml5js.org. We are especially interested in hearing about terms in the Code of Conduct that could benefit from more explanations or context. You can add those in this issue.

Acknowledgements

This work is the result of many hours of collaboration, discussions, emails, coordination, and perseverance of a handful of ml5.js community members (listed below). Additionally, there have been many invaluable conversations and feedback from friends, family, colleagues, and open source contributors (e.g. those that were part of the early ml5 license discussions) who may not have directly contributed to these documents, but have nonetheless influenced these outcomes — we are grateful for your presence and contributions. The authors of this document include, but are not limited to (alphabetical order):

• Christina Dacanay
• Lydia Jessup
• Sam Krystal
• Joey Lee
• Ashley Jane Lewis
• Bomani Oseni McClendon
• Daniel Shiffman
• Michael Weinberg

A New Code of Conduct and License for ml5.js was originally published in ml5js on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today we are asking for your help to make ml5.js even better.

We are always looking for ways to help ml5.js live up to its goal of being a neighborly approach to creating and exploring artificial intelligence in the browser. We have been working on a new way to do that and are excited to share it with you for feedback! We are proposing a way to link our Code of Conduct with the software license to make ml5 even more friendly. There is information about how to send us feedback at the bottom of this post.

ml5.js helps artists, coders, and students do creative things with machine learning by making it approachable and friendly to learn. ml5.js can be a force for powerful creativity. This is rewarding when it is used for creative exploration that improves the world. It is concerning when it is used to create tools that makes the world a less friendly place.

ml5.js — the library, the website, the community — encourages you to build positive things and discourages you from building harmful ones. Almost a year ago, we began exploring ways that we might be able to use a license to help further encourage good things while discouraging bad things.

Our goal was to find a way to keep ml5.js grounded in the spirit of open source, while reducing the chance that it could be abused for purposes that we feel run counter to the expectations of the ml5.js community.

We know that using a license for this purpose is controversial in the open source world. Nonetheless, we believe that we may have a model that integrates the goals of open source with a concern about the ethical use of powerful software. We know that this model will not work for everyone. We hope that it will work for the ml5.js community we are building together.

The Approach: Combining a License with a Code of Conduct

Defining ‘good’ and ‘bad’ behavior is one of the biggest challenges in drafting ethically-grounded licenses. While it may be easy to categorize any given use of software at a moment in time, capturing the breadth of possible uses of software that is freely available on the internet — possibly for decades to come — is an almost insurmountable challenge. Even the most carefully crafted language is likely to miscategorize edge cases that exist at the moment of its drafting. It will inevitably also fail to anticipate uses in the decades that follow.

We have attempted to address this by shifting our behavioral expectations out of the license and into the Code of Conduct. Structurally, that allows us to move the discussion of behavior away from the license itself and into a more community-accessible document. This document has freedom to evolve over time. It has the additional advantage of being structured in a way that is more approachable for the majority of the ml5.js community.

Collaborating on, working with, and improving codes of conduct are familiar activities for large parts of the ml5.js community. We hope that using a Code of Conduct instead of a license to document expectations makes it easier for the community to understand and contribute to them.

Interpretive Committee

Even the most comprehensive Code of Conduct will fail to anticipate all possible uses. Additionally, various members of the community may apply the Code of Conduct differently to the same set of facts.

To address both of these challenges we are establishing a Code of Conduct review committee drawn from the ml5.js community. The combination of a Code of Conduct and interpretation committee will inject a human element into decisions about morality that can evolve along with the larger community. We think people should be involved in making decisions, especially those that have to do with ethics. The committee is empowered to apply the Code of Conduct to novel scenarios that have not yet been integrated into the text of the Code of Conduct itself. That allows the Code of Conduct to evolve in the face of new uses.

While this flexibility can be an asset, it can also undermine user confidence in the license itself. Users might be concerned that a single contributor with an idiosyncratic interpretation of the Code of Conduct could decide that a well-meaning project is outside the scope of the license. That is why the committee must agree that a user is in violation of the Code of Conduct before the user is considered to be in violation of the license. In these scenarios, the committee acts as a stabilizing force in the interpretation of the license.

Process

Anyone who uses ml5.js does so under the terms of a license. Our proposed license includes an obligation to only use ml5.js in ways that fall within the behavior outlined in our Code of Conduct.

If someone uses ml5.js in a way that does not meet the expectations in the Code of Conduct their project would no longer be covered by the license. Contributors to ml5.js can ask the Code of Conduct Committee to decide if a project goes against the Code of Conduct. The license requires that the Code of Conduct Committee agrees that the project violates the Code of Conduct before a project is no longer covered by the license.

This review process creates a single interpretation of the Code of Conduct, and prevents ml5.js users from being tripped up by fringe interpretations of the rules.

Rules for a Limited Time

We have built an additional safeguard into the license: license decay.

A challenge in defining ethical behavior in a license is the length of time the definition will need to apply. In almost all countries, copyright lasts for decades. That means that a license that is applied in 2020 needs to create ethical rules that are still applicable in 2050, 2100, and beyond.

We don’t think we can do that. That’s why the new parts of our license are only in force for three years. After that initial period, the license ‘decays’ and reverts to a much less restrictive MIT license. In addition to limiting the amount of time a static ethical interpretation may apply, this decay also makes it easier for us to fix problems or make changes going forward.

In practice, as long as ml5.js is being actively developed there will always be parts of the source code that are new enough to be covered by our new license and the Code of Conduct. However, if active development ever stops (or we decide to change our licensing approach), after three years the heightened obligations will fade away.

A Different Approach

The open source licensing community has long discouraged attempts to incorporate ethical concerns into open source licenses. Doing so violates the terms of the Open Source Definition maintained by OSI. More practically, many commentators have pointed out the challenges inherent in creating robust definitions of ‘good’ and ‘bad’ behavior that could reliably be applied to the wide range of possible contexts in which open source software might be used. The extended term of copyright protection only enhances this challenge.

Nonetheless, many committees steeped in open source ethos continue to seek ways to limit what they view as non-ethical uses of the technology they help to develop. Projects such as the Hippocratic License and the Anti-Capitalist Software License speak to a desire within the open source software community to limit what software creators view as non-ethical downstream uses of their work.

We recognize that our proposal to use a license (based on — but certainly not endorsed by — the Blue Oak Model License 1.0.0) to bind users of ml5.js to a code of conduct as interpreted by a committee may not be for everyone. Some users will be wary of a code of conduct, or of the uncertainty that it injects into using ml5.js. Others will be skeptical that the ml5.js community is willing to maintain the infrastructure required by the licensing approach. These kinds of user-limiting concerns drive some of the historical wariness of ethically-grounded open source licenses.

Nonetheless, the ml5.js community has decided that the cost of losing some of these users is worth the benefits of the license itself, as well as the benefits of the signal that the use of the license sends to potential users. ml5.js is a community that is dedicated to friendly machine learning for the web. Users who are not interested in that type of machine learning are invited to go elsewhere.

Contact Us

ml5.js is a community effort and we welcome your thoughts on this approach. We encourage you to raise specific concerns about the Code of Conduct by filing an issue in the repo. As always, you can always reach out on Twitter, on Discord, or via email at feedback@ml5js.org. We look forward to hearing from you!

ml5.js Code of Conduct was originally published in ml5js on Medium, where people are continuing the conversation by highlighting and responding to this story.

Every few months we want to highlight creative projects from people in our community—that means you! If you have a project you’d like to submit to the round-up, submit it through this form. This month we’re featuring work from Yining Shi’s Machine Learning for the Web class.

Machine Learning for the Web is a 14-week class at ITP/NYU with a focus on building creative and interactive projects in the browser with machine learning techniques. Tools like Teachable Machine, RunwayML, ml5.js and TensorFlow.js are really making the creating process easy and beginner friendly. I’m so happy and proud to see so many creative projects made by students for every week’s homework. I’d love to share some of the projects with you and discuss creative applications of Machine Learning with you. — Yining

Bubble World by Sihan Zhang

Bubble World includes three mini games in which you can interact with bubbles in fun ways through your webcam. You can move the bubbles with your hands, bounce the bubbles with your head, or use your voice to control the bubbles. (Video | Documentation | Live Example)

Voice Candy Dispenser by Philip Cadoux

This project uses Google’s Teachable Machine Audio Classifier to identify when a user says “candy.” Upon hearing the keyword, a servo motor moves 90 degrees, allowing m&ms to fall on to a ramp and to the user. (Video | Documentation | Live Example)

Mooncake Hunter by KJ Ha

Mooncake Hunter is an interactive game using ML5.js PoseNet and P5.js. In this game, PoseNet is used to track the user’s nose position. The user’s nose turns into a fork, which serves as a tool to hunt for mooncakes. (Video | Documentation | Live Example)

Nose Guitar by Michelle Sun

Move your head to make some noises! A virtual guitar that will be triggered by the physical movements of the user. (Documentation | Live Example)

Dancing Pumpkin Character by Julie Lizardo

In the holiday spirit, I decided to recreate my favorite halloween gif: the dancing Halloween man. This project allows users to become their own dancing pumpkin character by using ml5.js Posenet model to detect your body from your webcam and move the character on the canvas. (Video | Documentation | Live Example)

Release v0.6.0 and more!

The ml5.js team is so excited to announce some new changes to the project.

We have a new release! 🎉

This release introduces two new models ported from Tensorflow.js: Handpose and Facemesh. Handpose can be used to predict the location of 21 3D landmarks on the palm and fingers of a single hand. Facemesh can be used to predict 486 3D landmarks on the human face and can work on multiple faces at once. Facemesh provides more detail about the positions of key locations on the face than any of the models currently supported in ml5.js’s implementation of FaceApi.

In addition to new models, the 0.6.0 release also includes a variety of fixes and improvements by our community of contributors and maintainers:

• Updated tf.js version to 1.7 (up from 1.2) (#846)
• Documentation for the Object Detector class
• Switched fetch calls in core library code to Axios for improved browser support (#979)
• Added nearestFromSet function to Word2Vec (#1015)
• Fixes to the Neural Network class (#919, #1096)
• Increase the performance of the FaceAPI by supporting the Tiny Face model (#936)

We’re also going to be using this Medium page to have more direct connection to our community. In addition to our editorial profiles we’ll be including casual updates on the state of the library and featuring small creative projects in round-ups. Do you have a project you’d like featured on the website or the Medium post? Submit it here!

Lastly, the team has been hard at work investigating the current landscape of machine learning and defining a set of standards that we want to uphold. These standards reflect our commitment to keeping ml5.js a friendly, safe, and accessible community for all learners. Our investigation has culminated in a living Code of Conduct and license that we’ll be releasing for feedback later this month.

What would you like to see next from the ml5.js team? We would love to hear from you! Let us know in the comments or on Twitter!

Written by Itay Niv. Edited by Ashley Jane Lewis.

The Heart of Reading Together

In a society full of screens as potential social inhibitors, the work of ITP student Itay Niv has been a whimsical breathe of fresh air, blowing the cobwebs off of a timeless idea — tech that brings loved ones together. His thesis project, Let’s Read a Story, emerged in the third semester of his masters degree and the third trimester of his wife’s pregnancy as a concept that would open up new modes of storytelling with his future child. This colorful responsive app uses machine learning to adapt classic fables into a fun interactive reading activity.

It comes as no surprise that the experience of using this project is just as contemplative and considered as the creator himself. Let’s Read a Story is the beautiful outcome of what happens when you build from the heart, working towards something special for the ones that you love. This agile system will help parents and kids cuddle up, sprinkling creativity into the quiet book nooks, the bedtime stories, the read-on-go moments and everything in between.

As an undergrad, Itay studied graphic design while working as a motion vfx artist and exploring interactivity and physical computing. Before coming to ITP, he worked for several years as a UX and UI designer. In this masters degree he learned how to code robustly and endeavoured into machine learning for the first time shortly before building Let’s Read a Story. Itay’s work is impressive as a stand alone piece but also a motivator to those who have a big idea before they have the skills to execute it.

It has been a pleasure to help document and share this wonderful thesis project with our community. Follow Itay Niv directly to see where Let’s Read a Story goes next! You can find his website here and follow him on Twitter here.

— Ashley Jane Lewis, 1st year ITP student and ml5 Community Manager Research Assistant

Let’s Read A Story is a speculative exploration on how computers and technology can turn story time into a conversation between parents, children and a computer. The project takes the corpus of Aesop fables and investigates the possibility of exploring the connections between different characters and ideas from the original fables in a new and fun way using recently available machine learning language models.

You can find a working demo here (best performance on chrome desktop).

In the following post I will try to describe the thought process and some of the technical aspects I encountered while designing and building the project.

👶 Some Background

Let’s Read A Story was created as my thesis project at ITP (NYU’s Interactive Telecommunications program). At ITP, a program devoted to exploring creative uses of technology and the recently possible. The 4th semester in the program is usually dedicated to developing a thesis project. Just a few months before it was time for me to start my second year at ITP my partner and I discovered the exciting news that we were expecting our first child.

So when the moment came to think about what I wanted to work on for the thesis project, the arrival of my child was a great inspiration on my thought process. I understood quite fast that I wanted to build something with technology that would allow me to connect with him. As my partner and I started to read texts about child development and parenthood, I came across this quote by the poet Friedrich Schiller:

“Deeper meaning resides in the fairy tales told to me in my childhood than in the truth that is taught by life.”

Reading this quote reminded me of my childhood and the memories of children books my parents and brothers had read to me, before there were any screens in our lives. The simple pleasure of browsing through pages of a children’s book, the smells and textures, the colors and sounds that lived once when we were kids. No different than other children, as a child, books and movies were my entire world. From them I learned what friendship meant, the values of family, love and at a later stage it is from animated movies which from I learned english.

So I got to think of what has changed in the last 30 years since I was a child? How will my child explore the stories, sounds and textures of fairy tales and other children’s narratives?

In recent years there has been a rise in popularity of smart devices and speakers using advanced NLP technologies. Next year, on average, each American household will have a smart device like Apple Homepod, amazon Alexa or Google Home. With applications Intended for preschoolers available on various of these popular platforms, children are talking to technology and technology is starting to talk back.

Does that mean we can start talking to books? This question echoed in my mind for days and was the main drive in the research of this idea for my thesis project.

🏗️ The Narrative Structure

For over more than 27,000 years, since the first cave paintings were discovered, telling stories has been one of our most fundamental communication methods. Aristotle, who analyzed the dramatic structure in his poetics in 335 BCE determined that a whole is what has a beginning and middle and end. In the 19th century, it was Gustav Freytag that coined the narrative pyramid — the dramatic structure is divided into five parts; exposition, rising action, falling action, climax and dénouement → the story’s catharsis.

For children, a story is an interactive experience — as a story progresses and develops, children ask questions. This is a great learning activity, kids learn to associate images in the book with the story and this develops their visualization capacity and imagination. They learn how to read and think.

For me, a good children’s book is much more than just text. It’s an immersive experience that includes(but is not limited to) textures, illustrations and music that accompanies it. One of my favorite books growing up was peter and the wolf by Sergei Prokofiev — that book had an orchestral soundtrack, a symphony — each character in the book had its own theme music played by a different instrument. The illustrations were amazing too!

For my son, as we move away from printed matter into digital storytelling for children, my hopes are that we would seize the opportunity inherent in new technologies to somehow, not only preserve but enhance these unique experiences.

📚 Finding a Suitable Dataset

I knew that I wanted to work with existed text from children’s books to create this experience, and I wanted the project to facilitate a conversation and imaginative thinking, to incorporate various components like illustrations and sound, characters etc, and for it to be as accessible as possible to children and parents.

Since the main part of this project is the text generation, it was clear to me that the avenues which I wanted to research and explore were first and foremost language models, and I had to choose a dataset to analyze.

After experimenting with a few canonic corpuses, I chose to focus and analyze Aesop Fables. I was drawn to the Aesop Fables text because of its concise, yet rich story lines, the use of animal archetypes as metaphors, and the strong morals embedded in each story.

Each original Aesop Fable contains:

1. A short title, usually very descriptive of the story’s content and characters.
2. The story itself, usually no more than 30 sentences.
3. The moral of the story, usually contains a metaphor built on the inherent nature or trait of the animals in the story.

⚙️ Analyzing the Data

After research with various machine learning models, such as LSTM’s, I came across the idea of word vectors. Word vectors belong to a group of machine learning models, or neural networks that are trained to reconstruct linguistic contexts of words and produce word embeddings. In natural language processing tasks, an essential part of the research and what requires effortful work is to convert text data into vectors. In that manner, it’s easier for a computer to then process and analyze the text. Once the text is embedded into a vector, one can easily determine whether two points are similar or not by calculating their euclidean distance. Techniques like word2vec do just that by converting a word into a vector — thus the corresponding vector of the word ‘burger’ will be closer to the word ‘sandwich’ than ‘salad’ or ‘dessert’. A simple example for word2vec is available within the ml5 library.

While embedding a sentence or a paragraph, along with words the context of the whole sentence needs to be captured in that vector, this makes the embedding process a little more tricky. This is where the Universal Sentence Encoder comes into the picture. The publicly available pre-trained model enables users to encode corpuses of text (sentences as well as whole paragraphs!) into high dimensional vectors that can be used for text classification, semantic similarity, clustering, and other natural language tasks.

For Let’s Read A Story I used the publicly available tensorflow.js model of universal Sentence Encoder and implemented it to work on a node application. After compiling a JSON file that holds all the stories broken down to individual sentences, their titles, characters, and animals.

The node application analyzed all sentences derived from the fables (~1500 sentences) and created another file that holds all the embeddings. This yields an ‘embedding map’ containing sentence embeddings for each sentence in a high dimensional space, we can then visualize it in 2 dimensions like this:

In this interactive map we can see that similar semantic meaning sentences are clustered next to each other.

💡 First Results

The first time I generated a new narrative out of the map I wrote a simple node application that picks a random sentence from the analyzed Aesop fables corpus and gives back the 10 closest sentences in the vector space. I used ml5’s word2vec class that uses cosine similarity to measure distance between the different sentences and modified it a bit so the class could work with my data structure(you can find the modified class here):

This was the result I got:

This was exciting to me as this was a new piece of content that made sense and had some sort of narrative structure to it.

I continued experimenting with this technique. After talking to Allison Parrish about this technique, an idea popped up to try and generate content based on an original story’s progression. To do this I started exploring different ways in which I can visualize Aesop fables on the embeddings map. Using tensorflow.js projector tool, If we visualize the story “The Swallow and the Crow” on the map, this is how it looks like this:

If we can visualize a story and see its progression through vector space, what if we replace the the original sentences with the nearest semantic neighbor of each sentence in the story; this might give us a version of that story while keeping the story’s structure.

*** Hypothetically, if we use a larger dataset of sentences, or augment the current one, the output would appear less nonsensical.

🛠️ Building the Application

As I started the work on this project I knew I wanted it to live on various devices, surfaces and spaces, to be as accessible as possible to whoever wanted to explore. But for the first version, I thought it would be best if I just prototype for web browsers, that way it will be accessible to almost everybody. The prototype used Node.JS for the server side (vector analyzing, sentiment analysis and serving the content) and html, css and javascript, for the frontend functionality (everything else).

Based on the concept of retrieving similar stories I began building the application which the main focus was to try and simulate a conversation. As speech is the most natural user interface for kids and allows minimum friction between action and result, I decided to utilize the web speech api. Readers can input content using their voice.

Using Universal Sentence Encoder in the backend with tensorflow.js-node, the captured speech is turned into a vector.

Using ml5's Word2Vec modified Sentence2Vec class (see code snippet here) the script iterates through all the sentences embedded beforehand by the Universal Sentence Encoder, i.e the ‘embedding map’. This gives back the sentence which is most similar in its semantic meaning to the reader input. Once I obtain the most similar sentence from the dataset i serve it to the reader as the first line of the new story. Also, the story algorithm takes the similar sentence and finds out the origin story of that sentence. The story algorithm generates a similar story based on that original story, derived from the reader’s prompt, and that is the seed structure that each story begins with.

As the story unfolds itself, readers can intervene and change the direction of the story by inputting their idea on how the story should continue. The story algorithm knows to fetch relevant content based on similar plot lines form the dataset. Every time the reader inputs text the story the algorithm pivots the narrative arc and matches a new narrative arc based on the new readers input.

To enrich the stories, I chose to use a recurrent neural network trained on the quick, draw! dataset. The story algorithm knows how to identify different elements in the text and using the ml5 library and p5.js, draws them to the page as the sentence is being added.

There is a simple RegEx search on the resulting sentences. The story algorithm functionality determines which animal, character or object appears in the generated story and then reconstructs an illustration from the trained sketch RNN model using p5.js. If the sentence contains an animal that does not exist in the model, there is another functionality that ‘enriches’ the model’s keywords and matches similar animals, objects and character specified in the story, for example a dog class will generate foxes and wolves.

Using AFINN-based sentiment analysis library the story algorithm analyzes each sentence to determine whether it has a positive or negative sentiment. Positive sentiments get a major scale melody and negative sentiments get a minor scale accordingly. Each animal gets a different musical instrument as well, according on its characteristics.

After user testing the application on dozens of children, I noticed the growing desire for kids to engage with the screen and story medium. As a solution to this I turned the entire screen space into a drawing canvas — in this manner, a child can add drawings to the story and be even more engaged.

In the end of each generated story, the story algorithm embeds the new generated story using Universal Sentence Encoder. By preprocessing all the stories through the Universal Sentence Encoder and compiling a vector embedding for each fable, the story algorithm is able to fetch a moral from the dataset that best fits, based on the semantic similarity to original stories.

👨‍👩‍👦‍👦 User Testing

A key part of that process was user testing. I user the project with kids in different age groups, from 3yr olds to 11yr olds, and iterated on their feedback and behaviour I documented.

Through testing with children I learned that my early assumptions about content creation and interaction had to change with different stages of the development.

A strong example to this was the reaction I saw from kids before and after enabling drawing capabilities in the experience. I hope to iterate on this in the next versions. I believe drawing could be a great force for creation, connecting kids drawings as input to the experience would be an extremely interesting experiment.

🏁 Final Thoughts

During the creation process of Let’s Read a Story, I constantly found myself thinking about the ethics of this experiment. By generating content with an algorithm, am I contributing to replacing traditions and professions so precious as authors, illustrators and musicians? That ironic thought disturbed me, as this was the exact opposite of the objective I had in mind. My experiment was meant to create a contemporary engagement with storytelling, not to replace it — to find ways these different mediums can be remixed and explored in new, exciting ways.

As I tested and documented Let’s Read A Story I noticed something magical in the interaction between the child and the parent, something that transforms ordinary story time into a creative adventure. My son, Carmi, is 3 months now, and I can’t help but anticipate using it with him.

This is not the end of my experiment and there’s much more to learn and explore. But I did reach one firm conclusion that i’ll take with me through the next parts of the project; There is a human soul at the core of every story, that’s irreplaceable. And I must always remember to create tools that would help people tell their story.

Let’s Read A Story was completed with the help and guidance of Nancy Hechinger, Daniel Shiffman, Allison Parrish, Gene Kogan and friends from the ITP + ml5 communities, spring 2019.

Let’s Read A Story: talking to books using semantic similarity was originally published in ml5js on Medium, where people are continuing the conversation by highlighting and responding to this story.

In 2019 the conversational pairing of diversity and machine learning largely results in rediscoveries of biased societal patterns but the subject matter offers much more nuance to be explored. This redundant rhetoric needs more heart, more compassion and, before all else, more creative applications that nudge us to be self-reflective as a united community and empower underprivileged voices. As a creative technologist and woman of colour, I look to the work of black speculative designers who are leading these innovative conversations. At the helm of this challenging navigation are two artists who seem to be eloquently articulating all the questions that are on the collective tip of our tongue.

Ayodamola Okunseinde is an award winning Afrofuturistic Reclamation artist, designer, and technologist whose work questions and connects the past, present and future for underrepresented communities. His masterful way of problematizing relationships between physical and virtual environments has garnered a body of knowledge that not only humanizes technological spaces but also forces positive dialogue through the contemplation of speculative futures, prototyping elements of the present that can be leveraged for social change.

With a keen eye on etymology and the preservation of cultural language, Nikita Huggins is a researcher and creative technologist whose work centers social impact and digital self-exploration. She asks the hard questions — “if Google translate is for everybody than why can’t it understand my dialect?” Her work persists to hold tech accountable and queries the norms that we have established in our daily use.

As active educators of marginalized populations, both artists work, in part, to bridge the gap, bringing forward youth who will go on to mindfully diversify the tech industry. As a team, operating under the name C.T.R.L. Alliance, they have shared an ml5 focused Something in Residency position at ITP this semester. I sat down with Ayo and Nikita to discuss their intentions for the term and had the pleasure of discovering the breadth of considerations that go into their practice.

If you like this conversation as much as I do, then you’re in luck! Nikita and Ayo have generously agreed to one more Medium check-in at the end of the semester! Follow us here or on Twitter for updates.

— Ashley Jane Lewis, 1st year ITP student and ml5 Community Manager Research Assistant

ml5: It’s thrilling to speak to you about the creative and artistic ways that you’ll be investigating machine learning. The ITP student body is excited to learn about the method of looking through the lens of representation in order to rally new voices that haven’t usually be at the table. What is the goal of the residency as it stands right now?
Nikita: Ayo and I are working together to create Black Corpus, a project under C.T.R.L. Alliance, our collaboration that investigates culture, language, research and technology. We’ve created several installations, one of which was at playtest at the Library of Congress. We’re interested in the deeper meaning of words and we think that working with machine learning would be a good opportunity to analyze the written works of black authors to potentially reveal an additional layer of subtext.

We have three major goals of the SIR. First, we’re aiming to draw out elements from our findings that could then be represented or showcased to the public. Secondly, we’ve developed a prototyping workshop for ITP that walks students through the process of designing data sets while considering issues around gender and racial bias. The last ambition is to contribute to ml5 by creating new, unexplored models. Ayo and I are planning to work with Temple University to get our data sets. They’re developing a corpus of literary works including those by authors of colour. With this partnership we’ll be able to collect the data, train models and then put them on ml5 so that the greater public can have access to them.

ml5: Since you can train models on a variety of different kinds of data, the idea of using language specifically is very interesting to me. What inspired the decision to focus on texts by black authors?
Nikita: I guess because it was missing.

Ayo: Yeah. *laughs*

Nikita: It stemmed from a conversation where we sat down and asked, “how do we fit into this?” At the time I was working with the Trinidad Creole language for my thesis. Words just felt particularly meaningful. We did some research and found that you can get a whole corpus from a British author. It’s online because someone bothered to scrape it, but no one is doing it for us. The classics are great but sometimes you reach a point where you just need to see something that is more representative of yourself. Show me something I can relate to! We saw that others were using these common text generation examples to derive additional meaning from the literature which got us thinking, “can we discover new meaning from the authors that matter to us as well?” It’s about the idea that if you want to see something in the world you need to create it.

Ayo: I have to give credit to ml5 as well. Before ml5, I thought about exploring machine learning projects and I even tried taking classes on the subject but there wasn’t the same the ease and access. ml5, with its web enabled editor, can be taught without the steep learning curve. The examples on the website really sparked my interest. One of the first ones that I looked at was the Virginia Woolf LSTM, followed by the Ernest Hemingway LSTM. Around that time, Nikita and I were speaking about collaborating and, in our exploration, we realized, “wait, where’s the Langston Hughes LSTM? Where are the W.E.B Du Bois examples?”

ml5: Yeah, you two are really piloting that content.
Ayo: I was very inspired by those models and I want people to be inspired by models of different authors. Creating black examples also perpetuates their work, cycling their text back into a modern context. It would be great to see what could come from that.

ml5: I don’t see many demonstrations of machine learning that pertain to blackness right now, which, of course, is at the root of your creative pursuit. Are there other examples of data sets or collectives that are trying to build this kind of content?
Ayo: We’re not familiar with other groups that are creating artworks based on machine learning about blackness or language. The reason we wanted to look into machine learning is that it is literally this other kind of mind that has its own understanding of the world.

The way it manifests itself is a space that, as people of colour, we haven’t had a lot of access to. We want to explore its accessibility but also what it means for a machine to taste blackness, in a sense.

Is there something within the text that can be transferred and understood through the machine that may not be understood otherwise? Similar to the work of Stephanie Dinkins, we’ll be trying to coax a personality into a machine and then query its understanding of itself as a black entity.

ml5: That’s amazing. I’ll probably be ruminating on the idea of “tasting blackness” for a long time. There’s such personification in that phrasing and it really presumes an identity for the machine. If this is its first taste of blackness, what is the machine’s identity right now?
Ayo: I would argue that the machine’s current identity or rather the flavours within it now are very “tech bro” in classification. The identity is not diverse enough, not only in blackness but in gender, in approach, in understanding other epistemologies, you know? From my understanding and research of the works that I’ve seen come out of machine learning, it’s still struggling to take all of those elements into consideration, which is a direction it will have to venture in order to address the societal issues that the field is dealing with right now. Truthfully, current projects that address critical issues of racism and bias can be stark and sometimes sterile. We’ve seen few examples that have approached these topics in nuanced, speculative or poetic ways.

Nikita: You’re right, there are lots of articles about bias towards people of colour, gender, sexuality and the like. The approach is almost repetitious. Yes, it’s important to discuss the impact of machine learning on policing but what about everything else?

ml5: These are prominent story arcs. They’re very easy to tell but this kind of siloed approach is such a common flaw in the translation of stories about blackness. The undercurrent reality is far more multifaceted and certainly more expansive and expressive than just these narratives.
Ayo: Right. There can be and are many intersections of blackness and machine learning to be explored. For example, what does Octavia Butler’s work tell us about black bodies that we may not be able to gain through the medium as it stands? What does Octavia Butler’s work, now combined with Langston Hughes, or W.E.B Du Bois, tell us about the condition of blackness? Is it possible to read even deeper into these texts by using algorithms to sort and parse out the spaces that we may not be able to grasp just by the sheer fact that we’re saturated by the norms of our societal mores and ways of thinking? Is there, perhaps, a way to ask a machine to reinterpret and reconsider those structures?

Nikita: Based on the outcomes that we get, we’ll discover questions for ourselves, like whether we should we even try to recreate the algorithm to be more inclusive.

ml5: Or does blackness need its own system entirely?
Nikita: Right, is there an alternate space that needs to be built for inclusivity to be taken into consideration? Is it more than just a diversity issue? Is the algorithm itself inherently biased?

Ayo: Some of our other works have spoken to these kinds of questions. For example, in our piece Unbroken Meaning, we put Trinidad English Creole and Nigerian Pidgin into the model. It then tries to understand what the language is, which is obviously a flawed pursuit because the model was developed in English. Once it has finished processing, it speaks the language out to us. Our questions are around what is gained and what is lost in the process of taking a deeply rich language and passing it through a sterile sieve. Is it possible to create a system that not only understands Pidgin and Creole but also puts it into context?

These languages come out of an ontology that is derived from a particular African or West African epistemology, which is in turn based on a cosmology that provides an understanding of the world. The ways in which this informs behaviours, thought processes and conversations are very interesting to explore because it often requires us to analyze the double meaning of phrases and words. A lot of these languages rely heavily on mythological parables and metaphors. It’s storytelling and there’s context within the language which shifts the meaning.

Let’s not forget the intentionality in that; the idea that the patterns of these languages exist to obfuscate and create distance between an oppressor and the oppressed group.

I don’t know if a machine can understand that. In some ways it’s obvious that, of course, a machine could never understand that, but it’s worth investigating.

ml5: It sounds like this work is just as much about parsing a nuanced language as it is about spending the time to appreciate and rediscover that language.
Nikita: Exactly. In that process we’re unveiling parts of history and examining power structures — there’s so much to unpack. What we also hope to find through this work is an examination of those constructs for ourselves and our own self-identification. Hopefully this can help inform other black people as well.

ml5: By making these kinds of machine learning examples available, it may cause a ripple effect, inspiring young black creatives to continue making identity based content of their own.
Nikita: Definitely. A few weeks ago I taught an ml5 workshop to a class of high school students. They really enjoyed the conversation about algorithmic bias but when I told them that you could use lyrics or poems to train an LSTM model you could see the creativity flowing.

Ayo: I actually have a student at Parsons who has started working on an ml5 LSTM. She’s trained a model on Cardi B lyrics *laughs*. Of course that’s not a big body of text and what comes out of it is sometimes questionable, but at least that’s a start! With young people in particular, machine learning is a good tool to start discussions about bias, ownership and access. If is that’s all these projects do, then that’s a lot!

There’s one really important consideration that we haven’t discussed yet. Hmm how do I express it? It’s extremely important that we engage mindfully with this content because certain authors have more privilege than others. For instance, when you run a Hemingway model and it outputs garbage or gibberish that’s okay, right? That doesn’t change the way we idolize Hemingway. That doesn’t shift our impression of his content because his work, along with the likes of other common text generation authors like Shakespeare or Victoria Woolfe, are canonized. However, if you run the works of James Baldwin through a model and it outputs nonsense, that’s not okay. There’s a lack of representation already and there is a lot misrepresentation of blackness out there. One doesn’t want to add to that. One doesn’t want to leave space to perpetuate stereotypes.

Hemingway has a privilege of being able to be battered and misrepresented, without great consequence because we’ve been taught that Hemingway is quality stuff. Langston Hughes, Malcolm X, Octavia Butler however… we [people of colour] know that it’s quality stuff but a lot of people don’t even know their works. It’s precious and it can be bruised or damaged very easily. That’s the best way I can describe it.

ml5: It’s a common paradox for black creative technologists. Do you create content that diversifies monocultural tech spaces or preserve the sanctity of black culture to avoid appropriation or misuse?
Nikita: It’s complicated. In part, that’s why we’re hoping to bring other artists of color who are in the machine learning field to ITP. It would be meaningful to host panels or artist talks to hear their perspective and learn about their approach. That might also help us develop ideas for the art we’re trying to make in this residency.

ml5: I’m looking forward to is how all of this research for Black Corpus will manifest artistically! You’re working with so many cerebral concepts and playing with a lot of tangible tools. What will the visuals look like? Will they be screen based or physical installations?
Ayo: I think moving away from the screen based and leaning more towards physical objects. Something utilitarian, meaning that they won’t be purely aesthetic wall or table decorations. We’re modelling the traditions of West Africa art [like masks and instruments], which are aesthetically beautiful but also meant to be engaged with, worn, touched. I think it’s important for our artworks to live in physical spaces and be interacted with.

ml5: That sounds very exciting! Let’s end on something introspective so that we have a topic to lead the conversation when we interview you at the end of term. What is the question you’re posing the artwork today
Ayo: For me, it’s what is the work going to look like? In my head I have a vision of everything up to the manifestation of the work.

Nikita: I think that’s why we came up with the workshop. It will tangentially help us with the design process. My question is will the art be truly representative or is it going to be simply our interpretation of blackness?

Ayo: Is it really going to make an impact? It has to be something good. It has to be a portal bridging two worlds.

Nikita: Is the algorithm going to do it justice? Which will lead to a bigger discussion of why or why not?

ml5: The main thing is just to be asking (and attempting to answer) those questions. The more questions you pose publicly through the application of this residency, the more questions we will get. Ultimately this opens the doors for other people to pick through those queries and begin to tackle them via their own creative practice. Thank you so much for sharing so open-heartedly about your intentions for this SIR. ITP and ml5 are very lucky to have you this semester.

Follow Ayo and Twitter and Nikita online and stay tuned for the end of term posts in this series!

The Subtext of a Black Corpus was originally published in ml5js on Medium, where people are continuing the conversation by highlighting and responding to this story.