Program notes
How MAiSTRO came to be
An LSTM composer trained on Mozart, Beethoven, and Chopin, built by a team of eight at Western Cyber Society. The story so far, in the order we lived it. For where the project went next, read how it works.
Origins
Motivation
Outside of engineering, music has always been a huge part of my life, listening to it, picking apart how a track is built, appreciating the work that goes into composing something that actually moves people. I've always been fascinated by how music artists compose, layering melody, rhythm, and emotion into something that connects with listeners in a way that's hard to put into words. That fascination made me wonder how I could bring my engineering skills into that world instead of just enjoying it from the outside.
At the same time, AI was exploding everywhere: essays written in seconds, images that made you question what was even real anymore. It felt like AI was closing in on doing anything a human could do. But one space seemed strangely untouched: music composition. That gap felt like the perfect intersection of two things I wanted to explore, could AI genuinely create music the way a person could, and could I build something that gave engineers a real way to support musicians instead of sidelining them?
The plan
Our plan
We decided to start simple: build a machine learning model that could learn from classical piano music and generate its own compositions. This meant finding a suitable dataset in a format the model could understand, selecting the right model architecture, running inference to generate new sequences, and finally converting those outputs back into audible music.
The dataset
Data collection & preprocessing
For the dataset, we gathered a collection of piano pieces by Mozart, Beethoven, and Chopin since they are easily recognizable, which made checking for overfitting much more straightforward. Initially, we thought we would need to convert audio recordings of these compositions into a format the model could understand. However, because these pieces are so well-known, computer-readable versions already exist online. These files, known as MIDI files, essentially represent digital sheet music, making them perfect for feeding into our model.
The model
The model
When planning this project, Henry and I had a few contenders for models we could use to accomplish the task. Autoencoders can be used by learning compressed representations of musical patterns and then decoding them to create new compositions. Generative Adversarial Networks (GANs), consisting of two models in competition, one generating music and the other evaluating its realism, also presented a compelling option. Lastly, Recurrent Neural Networks (RNNs) work by learning patterns in sequences of musical notes and predicting the next note based on previously played ones. After doing research, we felt that RNNs most closely mimicked the way humans process and compose music, leading us to choose a specific variant known as the Long Short-Term Memory (LSTM) model for our implementation.
You would undoubtedly ask, “Why not just use a regular RNN?” While RNNs and LSTMs essentially aim to accomplish the same task, standard RNNs have a much smaller memory capacity compared to LSTMs. For example, if I asked an RNN, “The sky is…”, it would (hopefully) respond with “blue.” But if I asked it, “Explain why the sky is blue,” by the second sentence it would start rambling about how the sky is actually just a reflection of the ocean and that's why deserts don't exist at night. This is because of a common mathematical issue called the vanishing gradient problem, where the gradient of the loss function decays exponentially over time. This is fine if the model is outputting one-word answers, but worse if the answers are meant to be longer since the model will forget more information faster and faster as the length of the answer increases.
LSTMs help solve this problem by allowing a larger memory of past inputs. An additional enhancement to these models is the attention mechanism. While RNNs and LSTMs use previous inputs to influence future outputs, they lack the ability to label what parts of previous inputs are important to a specific output. This is important in music generation, where the model must understand structural elements like verses and choruses to repeat or develop them appropriately. The attention mechanism allows the model to focus on these significant sections, improving its ability to generate musically structured compositions. Here's an example that helped me understand what the model was doing:
Unless you've seen this joke before, you probably fell for it (like I did… more times than I'd like to admit).
As humans, we don't read word-by-word. Instead, we read the most important parts and assume where the sentence is going from there. This is what attention is doing within our model: guiding the LSTM in what it needs to remember or forget as time goes on.
Our final model had the following architecture:
Bidirectional LSTM layer
Reads the note sequence both forward (start to end) and backward (end to start), then combines both readings into a single, better-informed representation.
Dropout layer
Randomly ignores a portion of the network during training, used throughout the model to keep it from overfitting.
Attention layer
Weighs which parts of the sequence actually matter to the next note, the way a person skims a sentence for the load-bearing words instead of reading every one equally.
LSTM layer
A second pass that folds the attention-weighted context back into the sequence.
Dropout layer
A second guard against overfitting before the final decision is made.
Dense layer
Organizes every note and chord the model has learned into a single set of candidates.
Softmax activation
Turns that set of candidates into probabilities, so the model can pick the most likely next note or chord.
Results
Alright, let's hear how it sounds
I'm done with the theory. Here's the model at four points along the same training run.
Epoch 1
That definitely sounded like the first time someone tried playing the piano. Clearly, it hasn't learned much from the dataset yet.
Epoch 10
The model has picked up some more complex chords and started developing patterns, but still struggles to vary its rhythm.
Epoch 20
Noticeably more melodic coherence here, and we're finally seeing some variation in note lengths, although still limited.
Epoch 50
loss 0.1534Arguably the best-sounding output we generated, though at a loss this low, there's a good chance some of it is memorized rather than composed.
By the way, the model only outputs MIDI files during generation, and to turn one into something audible we manually applied a piano soundfont. Generated pieces now play straight in the browser, no soundfont required. With the remaining Google Colab credits we had, we trained the model for a final 50 epochs, that's the last clip above. This is something I would genuinely listen to if I was trying to zone in during a study session, at a cafe, or any other situation where background music would fit in.
Then we kept going
Everything above is where the project stood when the team finished it. Since then the decoder has been rebuilt, two more architectures joined the model, and the whole thing became something you can steer rather than re-roll. How it works covers that, with the measurements. Or skip the reading and compose something.