Streaming data to PC for real time processing in Python

Ccfogarty · Apr 23, 2020

Hello,

I'm trying to stream data that the Bela is recording from 4 mics to PC via OSC. However I'm a bit of a novice with communications/networks and having difficulty figuring out how to send buffer to PC.

At the moment, I'm saving data to a .wav file using writeRt command in render like so

gPipe.writeRt(context->analogIn, context->analogFrames * context->analogInChannels);
Bela_scheduleAuxiliaryTask(gFillBufferTask);

Then using a writeBuffer function I found in example:

void writeBuffer(void*) {
  unsigned int numItems = gAnalogFrames * gAnalogInChannels;
  float buf[numItems];
  int ret;
  while((ret = gPipe.readNonRt(buf, numItems) ) > 0)
  {
    sf_write_float(outfile, &buf[0], ret);
  }
}

I'd like to instead perhaps rewrite the writeBuffer function as follows

void writeBuffer(void*) {
  unsigned int numItems = gAnalogFrames * gAnalogInChannels;
  float buf[numItems];
  int ret;
  while((ret = gPipe.readNonRt(buf, numItems) ) > 0)
  {
    oscSender.newMessages("\fmcw_print", &buf[0], ret);
  }
}

Is this the correct way to think about this? Any thoughts and tips would be greatly appreciated

giuliomoro · Apr 23, 2020

cfogarty Is this the correct way to think about this?

overall the idea seems fine: write to a pipe from the audio thread, read the pipe from a non-rt thread and from there send to the network. I haven't use the OscSender class much, but it seems that you'd need a separate call to add() after newMessage(), e.g.:

 while((ret = gPipe.readNonRt(buf, numItems) ) > 0)
  {
    oscSender.newMessages("\fmcw_print").add(&buf[0], ret);
  }

I am also wondering if this would be sending too many small packets. With default settings gAnalogFrames = 8 and gAnalogInChannels = 8 you'd get 64 samples per packet, that is 256 bytes. A single message can take up to slightly less than 1500 bytes if I remember correctly, so If I were you I would try to make it at least 5 times as large as this, to save some overhead during the transmission. You'd have to do this buffering on the writeBuffer() side, using buffer with a size 5 times the current one, keeping a pointer of where you are writing the stuff that comes from the pipe, and once the buffer is full, you send it over.

Ccfogarty · Apr 24, 2020

Thanks giuliomoro !

I've managed to establish communication between Python code on PC and Bela, but the frames I'm receiving on PC end seem jumbled

Here is my function as it is now

//Auxiliary task that sends buffer to remoteIP via osc
void oscBuffer(void*) {
  unsigned int numItems = gAnalogFrames * gAnalogInChannels;
  float buf[numItems];
  int ret;
  while((ret = oscPipe.readNonRt(buf, numItems) ) > 0)
  {
    std::cout << std::setprecision(5) << ret << '\n';
    oscSender.newMessage("/fmcw_print").add(&buf[0],ret).send();
  }
}

And my Python code at the moment simply tries to print on the other end for a given packet sent, but all I get is this:

b'\x8a\xd3^\xbd,\xe1B<\x80E\xf1\xbbeb\xfd\xba6\xc8\xb3\xbd\xdb56=\xc7\xd6\x08\xbd\xa5\x1b\xa7\xbb\xc1\x8e\x9d\xbd\x83\xf35=\xeblJ\xbd\xa4\xc1\xf6\xbb\x16\x1f+\xbd\x85{\x19=\xfa}X\xbdT-\r\xbc\xe6&\xef\xbc\x07\xb3F=\xd3F\x17\xbdA \xa0\xbb\x17\x92#\xbcm\xdar=\x9e\x1b\x89\xbc\xca\x11\x00\xbb\x075\x1c<\xbd>{=\xd8\xde:;\x8b\x1a\xef9?\x9d\x03=V)\x94=h\x83\x0f<]\xb0\x0b;VBy=on\xa1=\xec7O<\x8fj?;\xcf\xdd\x8c=3{\x8c=\n\\\xaf<\x05\x1fO;\x13\xbf}=O\x03\x0c=\x12\x1c\xe6;\xb6\xba\xa89\x0b\xf9?=\xdd\x92\xe2\xbb\x9e _\xbc\xae\xa9,\xbb\xa7\x1f\xe2<\xd7\x10\xe6\xbc\x0bU\xa2\xbchVd\xbbT|7<_\xc8\xc6\xbc\xce\xb2\x18\xbd\x8f\xd9\xdf\xbb\x02\xc4.\xbc)ID\xbc\xcdfP\xbda|\xff\xbb\x15\xc9\xf3\xbc\x9f5\x99;\xb3D5\xbdG:\xb5\xbb'

Part of the problem is I'm not quite sure what the conditions of the while loop are doing in the above function. I tried adapting the writeBuffer function to perform this task but perhaps it requires some more tweaking.

Some of my IDE settings in case they're relevant:

gAnalogInChannels = 4;
gAnalogSampleRate = 44100; //Operating at 15-22kHz
gAnalogFrames = 64; //To keep under 1500 bytes as recommended in prev response

Lastly,

giuliomoro read the pipe from a non-rt thread

This has been bugging me, what is significance of rt vs non-rt? seeing them appended to end of read and write calls in bela examples. Not having success finding explanation online

giuliomoro · Apr 24, 2020

cfogarty
gAnalogInChannels = 4;
gAnalogSampleRate = 44100; //Operating at 15-22kHz
gAnalogFrames = 64; //To keep under 1500 bytes as recommended in prev response

Do you set these manually ? You should set them from within setup() based on the actual current setting, which you can change from the Project Settings tab in the IDE.

e.g.:

unsigned int gAnalogInChannels;
unsigned int gAnalogSampleRate;
unsigned int gAnalogFrames;

bool setup(BelaContext* context, void*)
{
  gAnalogInChannels = context->analogInChannels;
  gAnalogSampleRate = context->analogSampleRate;
  gAnalogFrames = context->analogFrames;
...
}

cfogarty And my Python code at the moment simply tries to print on the other end for a given packet sent, but all I get is this:

I know nothing about python, but it seems that you are printing a hexadecimal representation of the buffer. Assuming you have a OSC parsers on the python side, which will take care of stripping out the header of the message, you should then re-interpret the binary payload blob of the message as a 32-bit floating point number.

Ccfogarty · Apr 24, 2020

giuliomoro

giuliomoro Do you set these manually ? You should set them from within setup() based on the actual current setting, which you can change from the Project Settings tab in the IDE.

No I really just did it to convey to you the settings, it's set up pretty much as you say here.

giuliomoro I know nothing about python, but it seems that you are printing a hexadecimal representation of the buffer.

I will look into the Python side of things, just figured I'd check here in case something looked egregious on the Bela side of the system. I'll check back in here if after printing as 32-bit float if there's something still amiss. Many thanks!