Digital Music Box
An interactive project on an ESP32 TTGO T-Display with a joystick, button, and potentiometer for Professor Mark Santolucito’s Creative Embedded Systems (COMS 3930) Course in Spring 2022
Documentation: https://github.com/sedona-thomas/TTGO-Interactive-Device
Overview
Last semester, in Professor Santolucito’s Computational Sound class, we all programmed our own web-based synthesizers. I had always intended to come back to this code to improve upon it, and I decided to take this opportunity to not only reprogram it to have a more accessible user interface and additional features but also to add a peripheral device with sensors to control some of the features such as the pitch and various modes. Instructions for setting up the synthesizer and making the music box can be found at the github repository (linked above).
How It Works
The Digital Music Box webpage allows a user to play a digital synthesizer from their computer keyboard. Using the box with the sensors, users can turn the potentiometer to increase the keyboard octave, press the button to toggle a low-frequency oscillator, move the joystick horizontally to change the low-frequency oscillator’s frequency, and move the joystick vertically to change the number of partials when in additive synthesis mode. Other mode changes can be accessed by clicking the buttons on the screen.
Learning to Use Esp32 Pins
I began by trying to get one sensor at a time to work but realized that there were a lot of values to maintain, and since I was still learning to simply use the hardware, I decided to completely overhaul my Python and C++ programs. In the Python program, the serial messages with just the sensor value worked alright for a single sensor, but I needed a way to keep track of multiple sensors at the same time with only some of them changing state. I decided to use a tagging system like HTML but with my own tag names.
Initially, I used the built-in HTMLParser class to identify open and close tags, but this made it difficult to add more levels of abstraction. Then, I switched to my own parser which I initially manually coded to identify tags before switching to regex expressions. This allowed me to break down the different tags into their own methods. It became really easy to fix issues with storing information, but I noticed an issue with the joystick’s x-axis potentiometer and button being switched. I realized the way I had organized the ESP32 C++ program was prone to errors with keeping track of which pin is being used with each sensor, so I began reorganizing and abstracting out parts of my code.
I began by abstracting out all of the Serial.print() statements for each sensor into their own functions to address the value of one wire at a time. Then, I reordered all of the variables and functions to group each sensor, so it was easier to fix issues one sensor at a time. Eventually, I began searching for why the joystick’s x-axis potentiometer and button were switching and after checking the wires, I determined that it must have been the code. I finally realized that I had written down pin 27 as T9 and pin 32 as T7 when they were supposed to be the other way around. Once I realized that I was using “T7” and “T9” in multiple methods to access the joystick sensors, I searched the internet to figure out the data type of the touch pins, so I could update a single variable if I decided to change pin connections for any of the sensors.
My improvements to the python and C++ programs were tedious, but they really helped me to understand how all of the pins work and can be accessed. Once I understood how to connect devices to the touch pins, I needed to more efficiently connect multiple devices that could be read in the appropriate manner.
Designing the Class-Based Sensor Readers
Building on the framework I made when practicing using the sensors, I realized that I was still writing too much repetitive code, and it was making it difficult to keep track of each of the pieces. At this point, I decided to completely rework my system to make distinct classes for a button, potentiometer, and joystick, so I could read any number of sensors with ease and not have to manage multiple instances of the same information or algorithm. As I was designing these classes, I realized that they made the joystick much easier to manage since it could simply handle instances of my previously built button and potentiometer classes rather than reimplement these classes themselves.
Once I had this code working, I decided to smooth out the values for the buttons by looking at the past few values before deciding if the button was being pressed. This made the button slightly less responsive, but it didn’t impact its use since it was consistently jumping before smoothing the values. I then extended this to smooth out the values of the potentiometers to take a rolling average since the values jumped around even when the potentiometer was not touched.
Finally, once I had my code sending serial data to a python script that interpreted my tags, I added a mode to send serial data as a JSON format to prepare for linking a webpage to the output. Finally, I completely abstracted out interactions with the Esp32 screen and list of values to abstract out over 200 lines of code and make future projects significantly easier. I also began attempting to add wifi functionality but decided I didn’t know enough about it at this point.
Linking Esp32 to Webpage
Linking the Esp32 to a webpage simply involved switching the Arduino code to the JSON format by changing a single variable definition before uploading and debugging. I initially had an issue with the JSON parser not recognizing the serial data since it took multiple messages to receive the entire JSON string. I fixed this by simply looping through serial port reads until the combined string was in the proper format. I then began connecting and adding to a digital synthesizer I had already created. The initial functionality was already working, so I added the ability to handle changes in octaves and integrated the values read from the sensors into the already existing framework. The only real issue I came across was that I had to remove a 1/16 octave since it was too quiet to hear. However, this was where my project caused the biggest issue.
My intention had been to make the synth adjust parameters based on sensor input, but after hours of debugging, I finally found something that explained how async functions in Javascript can’t be relied on to update global variables. I am still extremely confused about what was happening within the async readLoop() function, and this was where I almost gave up. However, right before designing an entirely new webpage, I realized that the octave variable was able to be changed, but all of the others stayed the same. As a last attempt, I removed all of the if statements and replaced them with variable declarations using ternary operators and it worked. I still have no idea what the difference is, but all of the sensors are now able to change the synth modes.
Learned
Throughout this project, I learned more about physical design than any other project I have completed to date. I learned how to design a physical enclosure and upload it to a laser cutter. I learned how to make a lo-fi physical prototype to test my measurements before committing to the more costly materials. I learned how to safely use an electric band saw and orbital sander despite my previous discomfort with heavy-duty power tools. I also learned how to connect sensors to an Esp32 as well as how to send serial data and format the data in a useful way to be parsed by the main program. All in all, this project really tested my patience, but it gave me enough familiarity to be comfortable connecting sensors to a microcontroller as well as made me much more comfortable using all of the tools available in a maker space.
Intended Improvements
I had initially intended to improve my program by switching to a class-based system, but it ended up being less work to simply invest the time rewriting my code since I was using touch sensors rather than analog and digital read for the sensors. At some point, I hope to move the sensor classes into their own files and import their header files, but I struggled to figure out how to use analogRead() and digitalRead() from an external file. Other than these small improvements, my approach to adding sensors to the Esp32 works extremely well and is much easier to manage than my original attempt. I also intend to finish the wifi connectivity and ensure that it works, but I decided to wait until I fully understood what I was doing.
Special Thanks to Professor Mark Santolucito and the staff at the Barnard Design Center for all of their help!!!
