This was a ton of fun to work on — it was one of those projects where, when I was part-way through, progress slowed a bit because I was just enjoying testing it — sitting in front of the piano, playing around, watching things happen, and getting new ideas of what to add next. The project consists of two applications: the MIDI processor and the visualizer itself.

The MIDI processor application is written in JUCE, a fantastic framework for multi-platform audio applications. I’d worked with it previously, and thought it’d be a great way to achieve the processing I wanted. The application itself is fairly simple: it accepts MIDI input from two separate devices, with one being mapped to function as the actual piano input, and the other functioning as a control surface. Fortunately, I had all the hardware necessary already, given my involvement in music and the fact that most of my piano playing these days is done on my Roland digital piano, routed through my Mac to Logic Pro using Keyscape.

The communication between the two applications is achieved with OSC. I translate all of the midi events of interest into simple OSC messages, since the specific structure of the MIDI data itself is unnecessary for the visualization. The visualizer simply needs to know about the NoteOn / NoteOff events (which include information about note number and the velocity of the key presses), the values of the knobs on the MPK mini, presses of the pedals, etc. The messages passed with OSC can easily include all of this information, in a more readable format suited to the task at hand.

The visualization, meanwhile, is implemented using Unity. I modeled the pianos in Blender and am quite happy with how they look. Initially, I ran the visualizer on my Mac as well, using OSC to communicate locally between the applications. That was asking for quite a bit of total processing power, so I built the visualizer on my Windows machine (which has a more powerful graphics chip) and configured the applications to communicate wirelessly over the network. This communication is fast enough that, after separating the two applications to the separate machines, overall performance seemed to benefit substantially!

It’s been fun to iterate on this and to find more and more distinct visual “looks” for different pieces.