How Jacqueline Li is Visualizing Sound with Machine Learning
Jacqueline Li is a fifth year at Northeastern University studying Computer Science and Design. She recently completed her capstone project the Synesthesia Lens, which is a synesthetic audio visualizer using ml5.js and p5.js. Continue reading to learn more about her work!
Interview by Ozioma Chukwukeme
Can you tell us more about yourself?
JL: I’m currently a fifth year undergrad student at Northeastern University graduating in May. I grew up in Virginia and moved to New Jersey for middle school and high school. I always had a preference towards STEM — I liked math, but wasn’t very good at it, and I liked physics. But even so, in high school I didn’t really know what I wanted to do in college. The funny thing is that I hated computer science in high school and didn’t want to pursue it in college. My mom kept telling me ‘You should do it, it’s really easy’ but I thought it was confusing and difficult, so I thought it would make sense to come into college undeclared as an engineer, as it still scratched that STEM itch in my brain. After a semester, I learned that engineering just wasn’t for me. So, I decided to give computer science a second chance; I switched majors from engineering and eventually settled on a combined major in computer science and design.
The reason why I chose design as the second half of my major was design was that I was always an artistic kid, but never thought of that as something I could pursue professionally. After exploring a bit more, I realized the potential that it had for careers, such as UI/UX, product design, and frontend development. Something that really sparked my interest though was going to Zach Lieberman’s ARTECHOUSE exhibit in DC. I genuinely thought, and still think, ‘That is one of the coolest things I’ve ever seen.’ It was a direct intersection between CS and design that I didn’t know existed before, and it became a niche I wanted to explore.
Tell us about your capstone project; what was your inspiration and approach?
JL: So the prompt for the capstone project was very broad — the purpose of design capstones is really just to allow senior design students to work on whatever passion project they want to do using the skills they learned in undergrad so far. Going into it I knew I wanted to create an interactive installation with creative code because I never had the outlet or space to do that during my time at Northeastern. In the beginning of the semester, I was playing around with p5.js and doing a lot with audio and randomly generated bubbles. During one of the crits, my professor mentioned that I should look into synesthesia and specifically Alexander Scriabin, a composer that had synesthesia who saw specific colors for specific notes.
I wondered how I could visualize this as I don’t have synesthesia. Eventually, I figured that I wanted to showcase the fleeting nature of sound in addition to the colors that Scriabin saw when he heard music. To accomplish these tasks, I decided to randomly place lines on the x-axis with a color that is mapped to the note that is being picked up through the microphone. Immediately after the line is projected on the screen, it begins to fade away. I ended up titling this project the Synesthesia Lens to highlight that this allows you to see through Scriabin’s synesthetic lens.
If you’re interested in the nitty gritty, I used ml5.js to figure out the frequency of the audio data. Then, I converted each frequency to the note it falls under and visualized that via a line with a color of the specified note, as Scriabin saw. If you wanna see the exact mapping of notes to colors, check out the about section of my project website!
What were some of the challenges you faced while creating this project?
JL: I kind of went into this project blind because I didn’t have too much experience with creative code, so it was roadblock after roadblock. I found videos from The Coding Train on Youtube which helped me start when I didn’t know how things like the draw() function worked since I was so used to object oriented programming.
Once I got deeper into the project, one of the bigger challenges was figuring out how to get frequency data. I was initially using the p5.js FFT (fast Fourier transformation) library to get the frequencies but I was having trouble with it and thought there had to be an easier way. Eventually, I found The Coding Train’s Ukulele Tuner video and decided to use ml5.js.
What drew you to use ml5.js and p5.js over the others tools out there?
JL: I think it’s really the web editor for p5.js — its so fun and easy to get started. My initial plan was to work in p5 and see the extent of its capabilities, and if I needed more, then I would look into Processing. However, I felt that for this project p5 was more than enough. I didn’t even know ml5 existed until having the trouble with frequencies–and that was a godsend!
Did you have previous machine learning experience?
JL: Surprisingly, I don’t have any prior experience with machine learning. At my university, the mandatory classes we had to take were around the fundamentals. So my first computer science class there was in a language called Racket — it’s a functional programming language to get you thinking like a programmer and it was very difficult to learn. Some of my friends took machine learning as an elective, but I wasn’t interested in it at the time because I wanted to go into frontend and learn more about web development. So I didn’t have previous experience with machine learning but ml5.js made it super easy to understand, and I am interested in learning more about machine learning and AI.
Do you have any future plans for this project?
JL: My initial plan for this was to create several visualizations for the Synesthesia Lens, but because of the time constraint of the capstone I wasn’t able to. Over the summer, I plan on making more interactive and unique visualizations for this project. I also want to look into generative art and creative code and hope to do more with that in the future!
Be sure to follow Jacqueline’s socials to stay up to date on her work:
Live Project: https://www.synesthesia--lens.com
Jacqueline’s Website: https://www.jacquelineli.com
Instagram: @didotfanclub
TikTok: @influenczar
