Ah! Thanks so much. That did it. Of course, I was running AT= on a project without any custom source files, so the compiler was never invoked at all :facepalm:

FYI, we have decided to use OSC for all the communication between the Touchkeys process and Pd. This simplifies things, and the actual MIDI on/off we require is just encapsulated in an OSC message automatically anyway.

As you suggested, I have edited the way the Mappings work to avoid unnecessary reallocation during a state change. My solution was to add a member MRPMapping* mrpMapping_; to the PianoKey class, which is pre-allocated when the class is instantiated. The mapping is simply reset() and engage()d when active, and disengage()d when idle. Seems to work well so far. That was certainly a good suggestion!

The next stage is seeing how it runs on the Bela, as is (likely extremely slow, if at all) and then seeing what we can strip out. My gut would be to look at the Scheduler classes first, and try to eliminate them. I think there is a chance that all actions (except for timeouts, obviously) could simply be performed immediately, without too much ill effect. We may not need timeouts at all, really, since we are not receiving from an OSC or MIDI device (other than sysex messages from the external audio routing hardware). If we do, then we could simply delegate only the timeout actions to the scheduler, and perform all other actions (mappings, mainly) immediately. It's a theory that is worth testing, at least.

Let me know if you have any more insights in to the matter, you've been more than helpful so far.

Edit: For reference/posterity, here's the final compile command:
g++ -std=c++14 -I/usr/local/include/libpd/ -I/home/juniper/Downloads/liblo-0.29 -I/home/juniper/Downloads/boost_1_69_0 -I/root/Bela/include -I/root/Bela/build/pru/ -I/usr/xenomai/include/cobalt -I/usr/xenomai/include -march=armv7-a -mfpu=vfp3 -D_GNU_SOURCE -D_REENTRANT -fasynchronous-unwind-tables -D__COBALT__ -D__COBALT_WRAP__ -DXENOMAI_SKIN_posix -DXENOMAI_MAJOR=3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -DNDEBUG -DBELA_USE_RTDM -I/root/Bela/resources/stretch/include -DNDEBUG -O3 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"Main.d" -MT"Main.o" -o "Main.o" "../Main.cpp"

One thing I would like to try, and I will let you know how this works out:

If communicating through USB (UART) we are not utilizing Bela's real-time capabilities for anything but the audio thread, which is doing the lifting for our Pd patch. Hence, I think it is worth trying to communicate to the Bela via ethernet from a dedicated board (likely a Raspberry Pi), which will be running the Touchkeys code and generating OSC messages.

My thinking is this, although that would be adding another layer of latency, it may be acceptable and more importantly, it would ensure that the audio processing blocks always have enough time to complete each cycle, maintaining their real-time promise, and other threads don't have to compete for the CPU time that is left over. In essence, this would constitute a mult-core machine (in abstract, at least).

The RPi 3 B+ has a 1.4GHz A8 quad-core CPU, with 1 GB of RAM, so the increased performance may actually beat out any latency introduced from the ethernet interface. I will post updates once we get some testing done on this.

Did you hit the CPU limit on Bela already?
Do you have a sense of how much CPU is needed by the audio engine and by the MRP software on Bela
Adding a computer to the game seems more hassle than is needed, if it's not strictly necessary.

I haven't been able to test the CPU load yet. That is more of a contingency plan than anything else. We are also trying to incorporate other features (wifi enabled parameter adjustment and presets, machine learning features), which might end up pushing us over the line. Ideally, yes we wouldn't go that route.

alt text

Something is not right here. I have everything working on my personal (Ubuntu Linux) machine to the point where I can run my adapted Touchkeys code on my machine and send OSC messages to Bela which is running out Pd patch, and everything works as expected and is quite responsive.

However, once I attempt to run Touchkeys on the Bela at the same time, it hangs while attempting to open the Touchkey device. gdb does not seem to play well with the Bela core, so I cannot see the exact line where it is hanging (blocking perhaps?) but through deduction it is this line: device_ = open(inputDevicePath, O_RDWR | O_NOCTTY | O_NDELAY); Is there any reason this would not work on Bela? That is should I use the same flags as you did here: https://github.com/giuliomoro/serial-piano-scanner/blob/master/SerialInterface.cpp. Namely _handle = open(portname, O_RDWR | O_NOCTTY | O_SYNC).

I am attempting to run all threads as vanilla pthreads. Is it possible my pthread calls are being wrapped (and turned into realtime threads) because of the flags being passed to the compiler? Or does that happen just during linking? Do you think there something else that I need to change about the LDFLAGS I posted above?

For now, I will attempt building the Touchkeys code alone, with the only the normal CFLAGS and LDFLAGS to rule those out as the culprits, and to allow me to use gdb if needed.

Thanks again.

    wa3573 Invoking: GCC C++ Linker
    g++ -o "bela_custom_main" ./Main.o -L/root/Bela/lib/ -pthread -lbelaextra -lbela -llo -Wl,--no-as-needed -L/usr/xenomai/lib -lcobalt -lmodechk -lpthread -lrt -lprussdrv -lstdc++ -Wl,--no-as-needed -L/usr/xenomai/lib -lcobalt -lmodechk -lpthread -lrt -lasound -lseasocks -lNE10 -lmathneon -lsndfile -lpd -lpthread

    Nothing here is automatically wrapping pthread calls.

    wa3573 That is should I use the same flags as you did here: https://github.com/giuliomoro/serial-piano-scanner/blob/master/SerialInterface.cpp. Namely _handle = open(portname, O_RDWR | O_NOCTTY | O_SYNC).

    That was used to open an actual UART port. I have not tried opening a USB serial instead.

    Can you simply cat /dev/tty... from the terminal for the port you want to use? Does it work?

        so in this case portname is actually "/dev/serial/by-id/usb-APM_TouchKeys_B6A358563433-if00". On both my machine and Bela, cat /dev/serial/by-id/usb-APM_TouchKeys_B6A358563433-if00 echoes terminal input until it receives a SIGINT, as expected. I'm not sure if /dev/tty... ports would work for out case because we will (likely) be using a USB hub, as we need to connect to two USB interfaces (the key sensors and the MRP routing hardware). Not sure how the tty... ports are assigned in that case.

        So, after compiling and running the Touchkeys app alone on Bela, the same issue persists. I can confirm open() is not functioning as expected.

        Thread 1 "serial-piano-sc" hit Breakpoint 1, 0xb6fbfbea in open () from /lib/arm-linux-gnueabihf/libpthread.so.0
        (gdb) step
        Single stepping until exit from function open,
        which has no line number information.
        Starting the TouchKeys on /dev/serial/by-id/usb-APM_TouchKeys_8D73428C5751-if00 ... failed: Failed to open
        [Thread 0xb2879450 (LWP 1048) exited]

        What is the return value of open() ? Send me the exact code you are using, and I can try with the piano scanner I have here.

        I'll get you the return value shortly. If you're interested, the full working project is available in the repo: https://github.com/wa3573/Drexel_MRP_Key_Scanner/tree/master/serial-piano-scanner

        It's very stable and working well for us on my machine. The relevant part for this discussion would be in TouchkeyDevice.cpp specifically:

        int device_; // File descriptor

        ...

        bool TouchkeyDevice::openDevice(const char * inputDevicePath)
        {
        // If the device is already open, close it
        if (isOpen())
        closeDevice();
        device_ = open(inputDevicePath, O_RDWR | O_NOCTTY | O_NDELAY);
        if (device_ < 0) {
        // fprintf(stderr, "%s", explain_open(inputDevicePath, O_RDWR | O_NOCTTY | O_NDELAY, 0));
        return false;
        }
        return true;
        }

        Edit:

        Ah, you know, I have been getting lucky in some regard. I just realized an error in my adapter class which replaces JUCE's CriticalSection and ScopedLock with functional adaptations using the pthread_ library. I made the error of having ScopedLock inherit from CriticalSection, so at it's constructor implicitly called CriticalSection and effectively made a new mutex, which is not the intended functionality. (ScopedLock sl = ScopedLock(&criticalSection_); is called to lock, and when that object goes out of scope, it's destructor is called, which exits the enclosed criticalSection). Anyway, that is remedied now, and the repo has been updated.

        Not sure if this was having an effect on my problem above, but it is possible. I will retest tomorrow.

        I just got a Raspberry Pi to experiment with, and interestingly, while the Pi is able to open the device file and obtain a file descriptor, it cannot successfully detect the presence of the Touchkeys device. More testing ahead, but this leads me to wonder if there is a fundamental difference in the way these "files" are being accessed by their ARM Linux implementations.

        Here's the solution (for the RPi at least, but both are embedded Debian distros so I suspect it would work on Bela as well)

        First: learn the Baud rate of the device:

        stty -F /dev/serial/by-id/usb-APM_TouchKeys_8D73428C5751-if00

        output:

        speed 9600;

        Then, git rid of that pesky O_NDELAY ~= O_NONBLOCK flag. This may work fine in most cases, but we're chewing up CPU cycles just to get an EAGAIN when there's no data. Add the O_SYNC flag which ensures any data written to the port is flushed before the write() call returns, seemingly more reliably than the existing flush calls written in.

        In place of blocking, set up a pure timed read using the termios attributes, by setting VMIN to 0 and VTIME to 5 for a 0.5 second timeout. Then setup the termios attributes for non-canonical mode with the baud rate attained above.

        So finally, using a couple helper functions (second answer here https://stackoverflow.com/questions/6947413/how-to-open-read-and-write-from-serial-port-in-c) we have the new openDevice() logic:

        ...

        if (isOpen())
	closeDevice();

device_ = open (inputDevicePath, O_RDWR | O_NOCTTY | O_SYNC);

if (device_ < 0) {
	return false;
}

set_interface_attribs(device_, B9600);
set_mincount(device_, 0);                // set to timed read

return true;

        Seems to work pretty consistently, so far.

        Ok, so it seems the only problem was with the speed of the port? I am actually surprised it was B9600 ... I am wondering if that's just a dummy number when dealing with USB serial?

        All the rest seems to be just optimizations (blocking upon read seems indeed a better option in general). Maybe O_SYNC doesn't help much in terms of CPU performance; I think the integrity of the data is not at risk either way. I guess this only helps when using multi-threaded programs?

        Yeah that seemed very slow to me as well. Very possible it's a dummy value. I'll experiment with different speeds. Not sure if that was the root of the problem, threading likely played a role. O_SYNC is possibly unnecessary, I will also try without that flag.

        Apart from it being purely speed related, my best guess would be that using a timed read had some impact.

        2 months later

        Here's an update:

        First of, I've got to thank you for all your help, this wouldn't have been possible without it.

        We've stuck with the Touchkeys software running on a RPi, and then transmitting OSC to Bela which is running a Pd patch to do the DSP, it works well, and the Bela doesn't seem to have the CPU resources to run both DSP and the Touchkeys software at the same time. Here's a video demo from last term:

        Pardon the wildly out-of-tune piano. We're working on enhancements and usability improvements now, and I am building an enclosure to solidify the system's cohesion.
        alt text

        alt text

        My next question for you:
        do you know what specific connector is used for the 3-pin audio connectors on the audio expander caplet? I am making a custom panel output with stereo 3.5mm jacks, and would like to purchase connectors, rather than cut and splice the connectors which we have already.

        I've also addressed a couple of bugs in the Touchkeys code which you all might be interested in.

        5 days later

        Thanks! Here's a link to our public Google drive folder with the Pd patch:

        https://drive.google.com/drive/folders/1iODqtzqR_P8H6xlysx1xJJUQo8ssPyp2?usp=sharing

        I'm not sure if we are running the sound generator, I am unfamiliar with it. The Touchkeys source code (running on the Pi) is available at the Github repo here:
        https://github.com/wa3573/Drexel_MRP_Key_Scanner/tree/master/serial-piano-scanner

        Currently, it is set up to log key position into a log file, so latency would not be as low as possible, but it does seem to be acceptable. This can be changed with a preprocessor directive in PianoKey.cpp, if desired. I can point out the bug fixes I mentioned earlier, if you're interested.