Bela+Puredata based assistive guitar project

spDuchamp

We're doing work at the Responsive Ecologies Lab at Toronto Metropolitan University on an assistive guitar project.

We're working with John Kelly who, with folks at Drake Music in the UK, created the first version of this guitar which John uses.

Here is John playing his original Kellycaster...
(

We're now developing a new version that is designed to be more robust for touring, and adaptable to various musicians to meet their needs.

The original version had the chord engine built using Max4Live and only operated in Ableton. Our new simplified chord engine is done entirely in Puredata, so we can run in Plugdata or on the Bela.

It is preloaded with all these chords (https://tombatossals.github.io/react-chords/guitar) and they can be called up by name.

Here is a little demo video using a 6 chord palette wired up.

We'll be releasing all this code and build instructions soon.

giuliomoro

Nice, I remember this project from way back when and met John a couple of times. I am wondering about the latency that can be heard in the video ... I can distinctly hear the acoustic sound from the strings coming through before the processed version. Is it still going through the computer? Or are you using a bluetooth speaker perhaps?

spDuchamp

This system isn't calibrated yet and the bonk threshold is a bit too high, so there can be action on the strings before the note sounds, but when this is in your hands and you're playing it, the latency seems undetectelable.

I haven't measured the latency yet, but having examined the original Kellycaster code, I can say for certain that this implementation has much more simple/faster communication between the Bela and Ableton.

We'll be using Ghost saddle bridge piezo pickups with the hexpander preamp wired to a Bela Gem Multi with all the hardware installed in a small off-the-shelf practice guitar. All powered by one USB cable attached to the computer.

giuliomoro

Nice. If you haven't finalised the hardware yet, you may consider moving to a Bela Gem Multi for more processing power and future maintainability.

spDuchamp bonk

Ah that may explain part of it. As it uses spectral flux, it uses long-ish FFTs which affect response time. Consider using the default values (-npts 256 -hop 128), though that on six strings may be too taxing for the original Bela? Whatever values you choose, make sure -hop has the same value as the block size you set on Bela (or a submultiple of it).

If latency is a concern, you should consider moving to time-based onset detection, which should work well enough for processing single-string signals and require less resources.

spDuchamp

Next time I'm in the lab, I'll give those values of bonk a whirl. Yeah, the new Gem multi is what we'll be using. I'll post an update when we have it all wired up in the new guitar.

I'm curious what onset detection you recommend, because I spent a whole lot of time trying to do fast onset detection that ended up failing horribly with real guitar playing, so I went back to using good ol' bonk, as was used in the original Kellycaster.

giuliomoro

The Jen Syntar GS-3000 (among others) was doing onset detection on signal from hexaphonic pickup in the analog domain in 1978, so we ought to be able to do something similar or better with all this power! :-)

I'd do a band-passed envelope detector with a dynamic threshold. The threshold is set based on the current value of the signal: you want to see a large enough spike from the current 'steady state-ish' level. That's sort of what spectral flux does: how much of the spectrum changes from one frame to the next.

The example PureData/envelope-trigger shows an example of sample-accurate threshold detection with manually-adjustable threshold. It doesn't have a proper envelope detector, but you can add one by adding a tunable lowpass filter after the [abs~]. The hv_ abstractions come from https://github.com/enzienaudio/heavylib. As it is, it may be a bit too sensitive to noise, so the filters would help to get rid of that.

Then the three parameters to tune are:

the bandpass's cutoffs (you may want to remove low-frequencies and concentrate on the higher-frequency burst, ot keep it full-range).
the threshold above current steady state-ish level
the envelope detector's low pass cutoff

Instead of a low pass after the [abs~], you could use a peak detector (see e.g.: https://github.com/giuliomoro/peak-/blob/main/peak~.c) which follows increases in the input signal faster and it only applies smoothing when the input signal is below the past level.

I know it's easier said than done, but I am sure you can achieve something lower-latency than spectral flux.