Every few months we want to highlight creative projects from people in our community — that means you! If you have a project you’d like to submit to the round-up, submit it through this form. This month we are, once again, featuring work from Yining Shi’s Introduction to Machine Learning for the Arts class, an IMA/NYU class focused on getting undergrad students initiated in the world of creative possibilities of AI and Machine Learning.
During 14 weeks, students of the Interactive Media Arts BFA at Tisch School of the Arts (NYU) have the opportunity to explore the theoretical concepts along with hands-on experiences regarding creative coding and machine learning. Tools like Teachable Machine, RunwayML, TensorFlow.js and our dear ml5.js are explored, tested and discussed while students critically think about data collection, ethics and bias. All the different topics explored during those meetings evoke different interests and possibilities of exploring creativity with the use of Machine Learning and the class culminates into a final project presentation where students get to showcase their art pieces.
Here are some of the final projects created by the students from the 2023 Machine Learning for the Arts class using ml5:
Elemental Mastery by Aditya Pandhare
Elemental Mastery allows you to empower yourself to seamlessly control the four elements of fire, water, air, and earth, by unleashing a symphony of creativity and exploration and a selection of some amazing fighting moves! This project uses ml5.js next-gen handpose, speech recognition and particle systems.
(Video | Documentation | p5 sketch)
Picky Art Critic by Andrew Feng
This project is a DoodleNet Classifier which outputs different responses based on confidence level, as well as ideally a less strict response.
Strange Loops by Annie Zhu
Strange Loops is an exploration into generative deep learning models and self-referential patterns, questioning the notions of identity and the uncanny. The creator trained a machine learning algorithm (StyleGAN2-ADA-Pytorch) on a database of selfies, then had it generate its own version of the pictures; from there,they took it to Google’s Trainable Machine and ml5 Image Classifier to see if it was able to distinguish the real face from the imposter face. The final result is a short lenght film that functions both as documentation of the outcome as well as a standalone work of art
ml5 KeyPoints Tool by Connie Hu
This project is meant to be a tool to help beginners get started with using ml5 pose estimation models in their creative coding projects and also for experienced user to prototype easily.
(GitHub repo | Documentation | p5sketch)
Fit Kit by Zoey Zhang & Vicky Wang
Fit Kit is a personal training platform combining multiple p5js sketches using the ml5 pose classifier. It reminds us of the right exercise movement and monitors workouts by tracking the user’s motion.
(p5sketch | p5sketch | p5sketch | p5sketch)
HandGun AR by Joonha Yu
HandGun AR aims to integrate the physical and digital realms, enabling users to interact with a virtual environment using their fingers as simulated pistols.
(Documentation | Live Example)
Emocean by Sirui Wang
This project is an envisioned technology that listens to your unpleasant feelings/thoughts/concerns to help you unload the mental baggage, relieving you from the need to burden your friends and family with these heavy thoughts
(Documentation | Live Example)
Glitch Identity by So Yeon Kim
This project seeks to reveal and critique how machine learning algorithms may categorize human and real-world diversity into oversimplified and stereotyped groups. It engages with audiences by scanning their faces, yet assigning a deliberately randomized identity, such as ‘A dinosaur that likes to dance,’ to underscore the absurdity and inaccuracies intrinsic to these algorithms.
(Documentation | Live Example)
TOUCH by Yizhen Xiang
TOUCH system works as a catalyst for profound human connection, where following its instructions becomes a journey of getting closeness — bridging not only physical distances but fostering a mental togetherness. It uses webcam and the hand pose model to estimating the distance between two hands (from two people) to encourage people getting closer to each other by following the instructions on the screen, such as touching fingertips, clapping, interlocking.
Breathing With MossPillow by Yuntian Zhao
MossPillow is a biodesign installation that can help people deep breathing and regulate their memory and emotions. This project uses raw plants and materials and incoporates ml5 by using PoseNet and FaceMesh to play nature sounds.
(Video | Documentation | Live Example)
As always, make sure to follow us on social media to stay connected and see everytime we share new projects our community has been creating with ml5 (or to share your own creations with us)!
Follow us on X and join our Discord channel.
This blog post was written by Micaelle Lages and Yining Shi.
All images used in this post belong to the respective creators for each project displayed. All inspirations and resources utilized are their responsibility and should be acknowledged in their own documentation.
