On image

Bela image, v0.3.8f,  4 January 2022

More info at https://github.com/BelaPlatform/bela-image-builder/releases

Built with bela-image-builder master@f7d3b8d11dc3b7b89ee4be9b1ec94eefcf22524f
on Tue Jan  4 02:28:17 UTC 2022

Suddenly I get this error when I try to send an Osc Message over my network. The code was working fine before I updated...

Error while sending to pipe from OscSndTsk_192.168.0.1327132: (1) Operation not permitted (size: 44)

Same if I try to send on local host ? Any idea ?

I can send and receive messages -but it seems to happen when I for instance use the receive osc callback to send an osc message with another sender... or when I send two messages in the same function call....

You can only write to the pipe from a Xenomai cobalt thread. The Bela audio thread is one such thread, but your osc callback may not be. You should be able to turn a Linux thread into a Xenomai thread by calling:

#include <cobalt/sched.h> // or #include <sched.h> ? ?
...
// from within thread:
__wrap_sched_setscheduler_ex(0, SCHED_COBALT, NULL); // or SCHED_OTHER ???

however, you may not want to write from both the audio thread and a secondary thread to the same pipe: I am not sure what it happens but either you will get corrupted messages or you will have the audio thread wait for the other thread to finish its write operation.

Just a side note: OscSender's newMessage() and add() methods are not exactly real-time friendly, although if you are always sending messages of the same length to the same address it shouldn't cause any memory allocation after the first one (if at all).

ok, but did something change? I have used this code without having this issue for years. no problem on image v0.3.8d …

My Osc Sends are happening on a thread created by AuxTask (the non-realtime one).

Nope. Something is definitely broken.

Here is how I reproduced on a fresh v0.3.8f image with no of my own code or cross compiler in the mix 🙂

  1. Fix the (also new) naming issue on OscSender.cpp ... as mentioned in this post: https://forum.bela.io/d/2033-error-while-loading-shared-libraries-libseasocks-so-1-4-3/8
    ... e.g., just shorten name by changing OscSenderTask_ to OscSndTsk_
  2. Start the osc handskaker in node: node /root/Bela/resources/osc/osc.js
  3. Run the example: Communication/OSC

Output:

Waiting for handshake ....
handshake received!
received a message with int 0 and float 3.140000
Error while sending to pipe from OscSndTsk_127.0.0.17563: (1) Operation not permitted (size: 60)
received a message with int 1 and float 3.140000
Error while sending to pipe from OscSndTsk_127.0.0.17563: (1) Operation not permitted (size: 60)
received a message with int 2 and float 3.140000
Error while sending to pipe from OscSndTsk_127.0.0.17563: (1) Operation not permitted (size: 60)
received a message with int 3 and float 3.140000
Error while sending to pipe from OscSndTsk_127.0.0.17563: (1) Operation not permitted (size: 60)
received a message with int 4 and float 3.140000

This is a major showstopper for me 😭

    So did some more digging. The problem is introduced by the change in OscReceiver, from AuxTaskNonRT to std::thread in git commit

    cd056c40969a2c0c4a43385cc9747c6e3341842a : OscReceiver: various improvements.
    - no more reliance on AuxTaskNonRt, which means it can run without Xenomai at all.
    - not sleeping if a message was received, so that we do not limit throughput
    - using vector instead of char[]
    - reduced includes in header

    So that is why it is no longer a Xenomai cobalt thread ?
    I tried

    #include <cobalt/sched.h> // or #include <sched.h> ? ?
    ...
    void on_receive(oscpkt::Message* msg, void* arg)
    {
    	__wrap_sched_setscheduler_ex(0, SCHED_COBALT, NULL); // or SCHED_OTHER ???
    	if(msg->match("/osc-setup-reply"))
    		handshakeReceived = true;
    	else if(msg->match("/osc-test")){
    		int intArg;
    		float floatArg;
    		msg->match("/osc-test").popInt32(intArg).popFloat(floatArg).isOkNoMoreArgs();
    		printf("received a message with int %i and float %f\n", intArg, floatArg);
    		oscSender.newMessage("/osc-acknowledge").add(intArg).add(4.2f).add(std::string("OSC message received")).send();
    	}
    }

    but

    use of undeclared identifier '__wrap_sched_setscheduler_ex'; did you mean '__wrap_sched_setscheduler´´'? column: 2, line: 50

    also tried with

    __wrap_sched_setscheduler(0, SCHED_COBALT, NULL);

    but

    Waiting for handshake ....
    handshake received!
    received a message with int 1656 and float 3.140000
    Segmentation fault
    Makefile:606: recipe for target 'runide' failed
    make: *** [runide] Error 139
    Bela stopped

    Tried debugging in gdb ... did not tell me much ...

      HjalteBestedMoeller So that is why it is no longer a Xenomai cobalt thread ?

      yes, good catch!

      HjalteBestedMoeller Segmentation fault

      I see a bug in the implementation of Xenomai that doesn't check for a NULL pointer as third argument. Attempted fixes below.

      use of undeclared identifier 'wrap_sched_setscheduler_ex'; did you mean 'wrap_sched_setscheduler´´'? column: 2, line: 50

      ah it my be simply sched_setscheduler_ex(): try that?

      HjalteBestedMoeller also tried with
      __wrap_sched_setscheduler(0, SCHED_COBALT, NULL);

      on sched_setscheduler() you need to pass the thread id as the first argument and it turns out the third argument cannot be NULL:

      #define _GNU_SOURCE
      #include <unistd.h>
      #include <cobalt/sched.h> // or #include <sched.h> ? ?
      
      // in some function ...
          pid_t pid = gettid();
          struct sched_param param;
          param.sched_priority = 0;
          int ret = __wrap_sched_setscheduler(pid, SCHED_OTHER, &param);
      
      // or, with _ex (amended version):
          struct sched_param param;
          param.sched_priority = 0;
          int ret = sched_setscheduler_ex(0, SCHED_OTHER, &param);
      
          // both, check for error
          if(ret) fprintf(stderr, "Error during setscheduler: %d %s\n", ret, strerror(ret));

      HjalteBestedMoeller Fix the (also new) naming issue on OscSender.cpp ... as mentioned in this post

      This one is weird: we added the port number with your commit below, so I assume it was working for you at the time.

      commit 8cb25e9df2255275155287f3463edabf7c67936f
      Author: hjbm <hjalte.moeller@man-es.com>
      Date:   Mon Dec 28 19:35:27 2020 +0100
      
          Fix OscSender Task name
      
      diff --git a/libraries/OscSender/OscSender.cpp b/libraries/OscSender/OscSender.cpp
      index 0faf9c56..93ce26c1 100644
      --- a/libraries/OscSender/OscSender.cpp
      +++ b/libraries/OscSender/OscSender.cpp
      @@ -29,7 +29,7 @@ void OscSender::setup(int port, std::string ip_address){
              socket->setup(port, ip_address.c_str());
      
              send_task = std::unique_ptr<AuxTaskNonRT>(new AuxTaskNonRT());
      -       send_task->create(std::string("OscSenderTask_") + std::to_string(port), [this](void* buf, int size){send_task_func(buf, size); });
      +       send_task->create(std::string("OscSenderTask_") + ip_address + std::to_string(port), [this](void* buf, int size){send_task_func(buf, size); });
       }
      
       OscSender &OscSender::newMessage(std::string address){

      not sure why it would stop working. There may be some internal string length limit. Either you are using larger port numbers/ip addresses than before, thus increasing the streng lenght beyond the internal limit, or the limit itself has changed inside Xenomai.

        giuliomoro Yes! Totally weird with the naming of AuxTask... I have the same IP's and port names, so something must have changed in Xenomai.

        Why the change to std::thread ?

          HjalteBestedMoeller Why the change to std::thread ?

          Because AuxTaskNonRT is meant for a thread that sleeps until it receives data from a RT thread. That is not what's happening here: OscReceiver::waitForMessage() is simply polling a socket and calling a callback when new data is available, so a std::thread is semantically more appropriate. Also, there was no need for it to be a Xenomai thread and that added a dependency that made it harder to reuse elsewhere (though I can't remember whether I actually used it anywhere else). Furthermore, note that the issue you are encountering is not with OscReceiver, but with OscSender, or rather with AuxTaskNonRT.

          Here's some tested-working (verbose and future-proof) code to turn a thread into a Xenomai thread (from here):

          static bool turnIntoCobaltThread(bool recurred = false) {
              struct sched_param param;
              memset(&param, 0, sizeof(param));
              int policy;
              // Guaranteed to succeed as pthread_self() cannot fail and pthread_getschedparam()'s only error condition is when
              // the given thread does not exist.
              pthread_getschedparam(pthread_self(), &policy, &param);
              pid_t tid = getTid();
          
              if (int ret = __wrap_sched_setscheduler(tid, policy, &param)) {
                  fprintf(stderr, "Warning: unable to turn current thread into a Xenomai thread : (%d) %s\n", -ret,
                          strerror(-ret));
                  initializeXenomai();
                  if (!recurred)
                      return turnIntoCobaltThread(true);
                  else
                      return false;
              }
              xprintf("Turned thread %d into a Cobalt thread %s\n", tid, recurred ? "with recursion" : "");
              return true;
          }

          In your case you can simplify the error handling by skipping the initializeXenomai() (which is defined elsewhere in the document) part. Just remember not to run this code before setup() is called (i.e.: don't put it in the constructor of a global object), or it will take place before Xenomai is initialised.

          If this works then we could do a further step to integrate it into the AuxTaskNonRT::schedule(const void* ptr, size_t size) method, following the logic we use in Supercollider: if writing to the pipe fails with -EPERM it means that you should call turnIntoCobaltThread() and then write to the pipe again. This way turnIntoCobaltThread() will be called at most once per thread. Something like this (untested and not super elegant):

          diff --git a/core/AuxTaskNonRT.cpp b/core/AuxTaskNonRT.cpp
          index 6bc8c11c..c5b40c92 100644
          --- a/core/AuxTaskNonRT.cpp
          +++ b/core/AuxTaskNonRT.cpp
          @@ -84,8 +84,19 @@ int AuxTaskNonRT::schedule(const void* ptr, size_t size){
           #endif
                  if(ret < 0)
                  {
          -               rt_fprintf(stderr, "Error while sending to pipe from %s: (%d) %s (size: %d)\n", name.c_str(), errno, strerror(errno), size);
          -               return errno;
          +               if(errno == EPERM)
          +                       initializeXenomai();
          +#ifdef XENOMAI_SKIN_native
          +               ret = rt_pipe_write(&pipe, ptr, size, P_NORMAL);
          +#endif
          +#ifdef XENOMAI_SKIN_posix
          +               ret = __wrap_sendto(pipeSocket, ptr, size, 0, NULL, 0);
          +#endif
          +               if(ret < 0)
          +               {
          +                       rt_fprintf(stderr, "Error while sending to pipe from %s: (%d) %s (size: %d)\n", name.c_str(), errno, strerror(errno), size);
          +                       return errno;
          +               }
                  }
                  return 0;
           }

            giuliomoro Tnhx 🙂

            A bit confusing to me. I might hack me thru it, but any chance you could make a version of AuxTaskNonRT that implements this and share it with me -then I will test it ?

            this should build and run (untested)

            diff --git a/core/AuxTaskNonRT.cpp b/core/AuxTaskNonRT.cpp
            index 6bc8c11c..db62d79a 100644
            --- a/core/AuxTaskNonRT.cpp
            +++ b/core/AuxTaskNonRT.cpp
            @@ -5,6 +5,8 @@
             #include <fcntl.h>
             #include <stdlib.h>
             #include <errno.h>
            +#include <sched.h>
            +#include <sys/syscall.h>
            
             extern int volatile gRTAudioVerbose;
            
            @@ -75,6 +77,28 @@ void AuxTaskNonRT::__create(){
             	}
             }
            
            +// Standard Linux `gettid(2)` not available on Bela
            +static inline pid_t getTid() {
            +    pid_t tid = syscall(SYS_gettid);
            +    return tid;
            +}
            +
            +static bool turnIntoCobaltThread() {
            +    struct sched_param param;
            +    memset(&param, 0, sizeof(param));
            +    int policy;
            +    // Guaranteed to succeed as pthread_self() cannot fail and pthread_getschedparam()'s only error condition is when
            +    // the given thread does not exist.
            +    pthread_getschedparam(pthread_self(), &policy, &param);
            +    pid_t tid = getTid();
            +    if (int ret = __wrap_sched_setscheduler(tid, policy, &param)) {
            +        fprintf(stderr, "Warning: unable to turn current thread into a Xenomai thread : (%d) %s\n", -ret,
            +                strerror(-ret));
            +            return false;
            +    }
            +    return true;
            +}
            +
             int AuxTaskNonRT::schedule(const void* ptr, size_t size){
             #ifdef XENOMAI_SKIN_native
             	int ret = rt_pipe_write(&pipe, ptr, size, P_NORMAL);
            @@ -82,6 +106,16 @@ int AuxTaskNonRT::schedule(const void* ptr, size_t size){
             #ifdef XENOMAI_SKIN_posix
             	int ret = __wrap_sendto(pipeSocket, ptr, size, 0, NULL, 0);
             #endif
            +	if(ret < 0 && EPERM == errno)
            +	{
            +		turnIntoCobaltThread();
            +#ifdef XENOMAI_SKIN_native
            +		ret = rt_pipe_write(&pipe, ptr, size, P_NORMAL);
            +#endif
            +#ifdef XENOMAI_SKIN_posix
            +		ret = __wrap_sendto(pipeSocket, ptr, size, 0, NULL, 0);
            +#endif
            +	}
             	if(ret < 0)
             	{
             		rt_fprintf(stderr, "Error while sending to pipe from %s: (%d) %s (size: %d)\n", name.c_str(), errno, strerror(errno), size);

            Now its just my git skills ..

            can I automatically merge this into my branch or should I just manually add the stuff that has a + in front of it ?

            whichever you prefer. You could just put those lines in a file called mypatch and do git apply mypatch

              Weird error

              Building AuxTaskNonRT.cpp...
              In file included from core/AuxTaskNonRT.cpp:8:
              In file included from /usr/xenomai/include/cobalt/sys/cobalt.h:32:
              In file included from /usr/xenomai/include/cobalt/uapi/kernel/vdso.h:21:
              /usr/xenomai/include/cobalt/uapi/kernel/urw.h:60:22: error: expected ')'
               token = (*(volatile typeof(urw->sequence) *)&(urw->sequence));
                                   ^
              /usr/xenomai/include/cobalt/uapi/kernel/urw.h:60:12: note: to match this '('
               token = (*(volatile typeof(urw->sequence) *)&(urw->sequence));
                         ^
              /usr/xenomai/include/cobalt/uapi/kernel/urw.h:60:22: error: C++ requires a type specifier for all declarations
               token = (*(volatile typeof(urw->sequence) *)&(urw->sequence));
                          ~~~~~~~~ ^
              2 errors generated.
              Makefile:452: recipe for target 'build/core/AuxTaskNonRT.o' failed
              make: *** [build/core/AuxTaskNonRT.o] Error 1
              Connection to 192.168.6.2 closed.

              giuliomoro That seems super weird. The ACCESS_ONCE Macro is defined in the kernel, right ?

              And the code you posted must also include the same urv.h, right ?

              oh well there was one include too many (cobalt/sys/cobalt.h). Amended the patch above