audio thread mode switch - std::thread

thetechnobear

I've written a pure data external, which (among other things ) sends OSC messages over UDP.

I've noticed each time I send a message , I get a MSW.

the way Ive implemented this, is the audio thread writes the required messages to a lock-free queue.
and then there is a writer thread (using std::thread) that picks off the queue and sends it.

Im a bit surprised, surely std::thread is posix thread, and so still not the audio thread.... and the only thing on the audio thread is not blocking.

do I have to use the xenomai threads (or AuxTaskThread) to avoid this?

if this is the issue, and as this is cross platform code, whats the easiest way to get a xenomai thread, without too many includes/libs for the build ... bare in mind this is also inside a PD external.

also, unrelated, I saw some mention of a bela 'netreceive' external , is this around somewhere?
(I need to replace udpreceive and udpsend)

thanks for any pointers
Mark

giuliomoro

thetechnobear also, unrelated, I saw some mention of a bela 'netreceive' external , is this around somewhere?

latest libpd for Bela has Pd's [netreceive] and [netsend] working ok. I implemented queues for them to push and send messages from and to the audio thread.

thetechnobear is the audio thread writes the required messages to a lock-free queue.
and then there is a writer thread (using std::thread) that picks off the queue and sends it.

this seems fine: I can see no reason why the std::thread should cause a mode switch on the audio thread. I think the problem may be inside the lock-free queue. Did you have a look at the code in there? For most things I tend to use ringbuffer that comes with libpd: https://github.com/BelaPlatform/libpd/tree/bela/libpd_wrapper/util .

thetechnobear

giuliomoro
ah cool, so you are shipping some externals in the build? where are they located?

std::thread, thanks that useful, as i didnt want to swap out unless absolutely necessary.

lockfree... hmm, puzzlingly,
I was using port audio PaUtilRingBuffer, and protecting it with std c++ condition variables, which also shouln't have blocked. ... but when I saw the MSW.

so, I switched to moodycamels , and saw exactly the same thing
(moody camel : https://github.com/cameron314/readerwriterqueue)
so seemed unlikely both would have a hidden lock...

hmm, ok, I'll dig deeper...

thetechnobear

giuliomoro
ok, found the underlying cause... I was using a version of the queue which blocked the reader... (non-audio thread)
unfortunately, this uses a semaphore to signal between the reader and writer, and its the semaphore.signal() that causes the mode switch ... in the same way the c++ condition var do 🙁

so if I remove this, and just get the reader thread to sleep, and then dequeue the message, no mode switches.

however, this is not ideal...(but at least it works - yeh! )
id prefer to wake up the (reader) thread when new items are placed on the queue...

@giuliomoro do you know a way to do this in xenomai without causing a mode switch? perhaps the bela code has to do something similar somewhere?

giuliomoro

thetechnobear ah cool, so you are shipping some externals in the build? where are they located?

No externals, they are built into libpd.

thetechnobear std::thread, thanks that useful, as i didnt want to swap out unless absolutely necessary.

Your problem is the condition variable then, do you use it for the std::thread to be woken up? When writing (partly-(portable code, I normally just poll a flag in memory from the non-RT thread. If going full-Xenomai, I use its own message-queues or RT-pipes (see some examples in core/AuxTask*.cpp or in core/Midi.cpp).

Anyhow, if you really need a condition variable, that should be a Xenomai one. So the deal with this is: just use plain-C pthread_cond_ and compile and link normally for other platforms, but follow the following instructions on Bela.

By passing the flag -Wl,--wrap=pthread_cond_init to the compiler at linking time, the linker will replace all the calls to pthread_cond_init with calls to __wrap_pthread_create *. The Xenomai libs will provide an implementation for __wrap_pthread_cond_init() (and many other Posix functions!), so that an appropriately linked program (or library )will use those instead of the regular ones **.
In practice you have to add one of those entries for each function for which you want to use the Xenomai implementation, e.g.:

-Wl,--wrap=pthread_cond_create -Wl,--wrap=pthread_cond_destroy -Wl,--wrap=pthread_cond_timedwait -Wl,--wrap=pthread_cond_signal

***.
An alternative approach to using linking-time symbol wrapping, is to manually call the __wrap_ function from your code where appropriate. This approach is the one that allows you to use the Xenomai implementation only for some of the calls to a given function.
You can #ifdef this, and/or use some clever (and to me obscure) C-preprocessor syntax as appropriate. You'd have to add -I/usr/xenomai/include to the compiler flags, so that when you#include <posix.h> it loads /usr/xenomai/include/posix.h which contains the forward declarations for the __wrap_ functions ****.

Additionally to the above (whichever approach you chose), you have to include the Xenomai libraries, The appropriate flags are given by running:

$ /usr/xenomai/bin/xeno-config --skin=cobalt --no-auto-init --no-mode-check --ldflags

which returns:

-Wl,--no-as-needed     -L/usr/xenomai/lib -lcobalt -lmodechk -lpthread -lrt

Just add this and the string you crafted above to the linker stage and you should be sorted *****.

* it also replaces the calls to __real_pthread_create with calls to pthread_create, but this is not relevant for us just now.
** Xenomai provides an automated method to wrap all the calls to any Posix function for which it provides an implementation , however that has some problems when used with C++, and also I think it is better to only wrap those you want. A full list of all the wrappable functions is in /usr/xenomai/lib/cobalt.wrappers.
*** Alternatively, you can pass a file containing all the flags for the functions you want to wrap (modelled on /usr/xenomai/lib/cobalt.wrappers) by doing: -Wl,@/path/to/your/custom/file.wrappers.
**** or you can forward-declare them yourself, as we do in include/xenomai_wraps.h.
***** unless you see that your external is not loaded, ping me if that is the case.

giuliomoro

thetechnobear s, which also shouln't have blocked. ... but when I saw the MSW.

Point is Linux's threads, condtion variables, message queues, clocks, semaphores, clocks, all use the Linux kernel, so you have to use Xenomai's ones for real-time application. A full list of what Xenomai implements is here (Modules->Cobalt->POSIX interface).

Also, if you want to read more on how to port an application to Xenomai, see here, but really all you have to do is follow the instructions above.

Also this thread and the associated documentation patch are interesting with regards to what actually is the Cobalt API.

thetechnobear

giuliomoro

yeah, I saw all this, so the question is... do the std c++ libs map to posix, and if so (which I thought they did) , how to i get Xenomai to use this posix wrapper?

I see this in the doc you linked to...

 XENO_DESTDIR:=/home/gilles/cross-install
 XENO_CONFIG:=$(XENO_DESTDIR)/usr/xenomai/bin/xeno-config
 XENO_POSIX_CFLAGS:=$(shell DESTDIR=$(XENO_DESTDIR) $(XENO_CONFIG) --skin=posix --cflags)
 XENO_POSIX_LIBS:=$(shell DESTDIR=$(XENO_DESTDIR) $(XENO_CONFIG) --skin=posix --ldflags)

does this mean, we should be compiling with something similar? to make sure we use the posix skin, rather than the linux kernel posix interface?

(Ive just noticed std::steadyclock is also triggering MSW , again presumably because its hitting the linux posix , rather than this skinned clock?)

giuliomoro

thetechnobear does this mean, we should be compiling with something similar?

Did you see my long post above?
I think this is answered there already? Just pass the appropriate flags at linking time, but make sure you do not use --skin=posix, as that comes with automatically wrapping all the Posix functions, whereas you probably only want to wrap the ones you are using, by adding -Wl,--wrap=pthread_cond_init or similar

giuliomoro

thetechnobear (Ive just noticed std::steadyclock is also triggering MSW , again presumably because its hitting the linux posix , rather than this skinned clock?)

it definitely is. Point is your libstdc++ is already linked and it uses Linux's services by default. No way around it, because you would need to re-link it with the wrapper flags in order for that to call Xenomai's services *. You need to use the C API, which can be easily handled by Xenomai with appropriate linker flags as explained above.

*: (except perhaps using LD_PRELOAD with a wrapper library that contains calls like:

int
     pthread_create(pthread_t *restrict thread, const pthread_attr_t *restrict attr, void *(*start_routine)(void *), void *restrict arg)
{
  __wrap_pthread_create(thread, attr, start_routine, arg);
}

for each function implemented by Xenomai.

thetechnobear

giuliomoro ah, sorry no... didn't see your long post - some how I missed that
(looks, like we may have overlapped in time slightly)

k, all this code is fairly isolated, so switching it over to using the posix (wrapped) interface is not too bad, and definitely looks like will cure the issues.
I guess in the end, I'll end up having variants using posix/wrap and then std c++ stl, but thats not too bad.

bit late to start it tonight...

good news is that apart from these MSW all appears to be working ok 🙂

thanks for the help, much appreciated
Mark

thetechnobear

@giuliomoro

so ive got the basics working, and seems to be ok, for threads/cond, but if I try to wrap pthread_mutex_lock,
i get

terminate called after throwing an instance of 'std::system_error'
  what():  Operation not permitted
Aborted

searching on the internet, says this is usually caused by not linking pthread, but i can see this is linked, and also using nm, i can see that pthread_mutex_lock is wrapped....

i saw a note about issues with static mutex, but this is wrapped in a class that is dynamically created, so I'm not sure this is the case,

any thoughts?

giuliomoro

From what context are you invoking __wrap_pthread_mutex_lock()?

The error you are getting is a c++ exception, so I would be surprised if it is thrown as a consequence of a C call, although I am not familiar with the inner workings of libstdc++.

thetechnobear

so so it appears i always get this behaviour even in a simple example

with the following defined:

--wrap pthread_mutex_init
--wrap pthread_mutex_lock
--wrap pthread_mutex_unlock

using the following code:

    pthread_mutex_t m;
    LOG_0("cobalt ");
    LOG_0("c init " << pthread_mutex_init(&m,0));
    LOG_0("c lock1 " << pthread_mutex_lock(&m));

i get the following output:

cobalt 
c init 1
c lock1 1

note: all calls should return 0 for success, but 1 = EPERM
only really the first, pthread_mutex_init, is the real issue as its obviously not initialising the mutex successfully.

(also the second lock does not block, indicating that its not just the error codes, really the lock is not taken)

if I remove the pthread_mutex_* wraps , it all works as expected...

Im using the following compiler options

            message(STATUS "BELA optimized")
            # /usr/xenomai/bin/xeno-config --skin=cobalt --no-mode-check --cflags
            set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -I/usr/xenomai/include/cobalt -I/usr/xenomai/include -march=armv7-a -mfpu=vfp3 -D_GNU_SOURCE -D_REENTRANT -D__COBALT__")
            set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wl,@${PROJECT_SOURCE_DIR}/xenomai.wrappers")
            set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize --fast-math")
            #  /usr/xenomai/bin/xeno-config --skin=cobalt --no-auto-init --no-mode-check --ldflags
            set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--no-as-needed -L/usr/xenomai/lib -lcobalt -lmodechk -lpthread -lrt ")
            set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} -Wl,--no-as-needed -L/usr/xenomai/lib -lcobalt -lmodechk -lpthread -lrt ")
            set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,--no-as-needed -L/usr/xenomai/lib -lcobalt -lmodechk -lpthread -lrt ")
            #set(CMAKE_STATIC_LINKER_FLAGS "${CMAKE_STATIC_LINKER_FLAGS} -Wl,--no-as-needed -L/usr/xenomai/lib -lcobalt -lmodechk -lpthread -lrt ")

they are defined in cmake, but I can see when I compiled , they are applied correctly.
also, if I use nm on the built binary, I can clearly see the wrap_pthread functions are bound, rather than the underlying pthread functions.

not sure what to try next really ...

EDIT:
so If I remove --wrap pthread_mutex_init , and leave the wrapped lock/unlock this fails on the lock
( I guess this is expected, as the mutex is now a pthread mutex, and wrapped lock is for a cobalt mutex ?!)
i.e.

cobalt 
c init 0
c lock1 1

giuliomoro

You need to call xenomai_init() before you can use Xenomai services. This is normally not needed if running inside a Bela project because it is called in the background.

This program:

#include <iostream>
#include <pthread.h>
#include <xenomai/init.h>

int main ()
{
	int argc = 0;
	char *const *argv;
	xenomai_init(&argc, &argv);
	pthread_mutex_t m;
	std::cout << "cobalt " << " \n";
	std::cout << "c init " << pthread_mutex_init(&m,0) << " \n";
	std::cout << "c lock1 " << pthread_mutex_lock(&m) << " \n";
	std::cout << "c try lock "<< pthread_mutex_trylock(&m) << " \n";
	std::cout << "c lock2 "<< pthread_mutex_lock(&m) << " \n";
	std::cout << "c lock3 "<< pthread_mutex_lock(&m) << " \n";
	std::cout << "c unlock "<< pthread_mutex_unlock(&m) << " \n";
	std::cout << "c lock4 "<< pthread_mutex_lock(&m) << " \n";
	std::cout << "//cobalt" << " \n";
	return 0;
}

compiled like this:

g++ -c -pthread test.cpp `/usr/xenomai/bin/xeno-config --skin=cobalt --cflags` -o test.o  &&\
 g++ test.o -o test `/usr/xenomai/bin/xeno-config --skin=cobalt --ldflags --no-auto-init`  -Wl,--wrap=pthread_mutex_init -Wl,--wrap=pthread_mutex_lock -Wl,--wrap=pthread_mutex_trylock

gives:

root@bela:~# ./test
cobalt
c init 0
c lock1 0
c try lock 16

and then it hangs, correctly.

thetechnobear

thanks for that... that solved that issue...

now getting errors with semaphores (arghh, this is tiring !)

#include <iostream>
#include <pthread.h>
#include <semaphore.h>
#include <xenomai/init.h>

#include <thread>
#include <chrono>


sem_t sem;

void* func(void*) {
    while(true) {
        bool timeout = false;

        int t = 1000 * 1000;
        struct timespec ts = { t/1000000, 1000*(t%1000000) };
        clock_gettime(CLOCK_REALTIME, &ts);
        t += (ts.tv_nsec/1000);
        ts.tv_nsec = 0;
        ts.tv_sec += t/1000000;
        ts.tv_nsec += 1000*(t%1000000);
        errno=0;
        int rc=sem_timedwait(&sem,&ts);
        std::cout << "timedwait " << rc << " : " << errno << std::endl;
    }
    return 0;
}

int main ()
{
    int argc = 0;
    char *const *argv;
    xenomai_init(&argc, &argv);
    std::cout << "si " << sem_init(&sem,0,0) << std::endl;

    pthread_t thr;
    std::cout << "thr " << pthread_create(&thr,0,func,0) << std::endl;

    for(int i=0;i<10;i++) {
        sem_post(&sem);
        std::this_thread::sleep_for(std::chrono::seconds(2));
    }
    std::this_thread::sleep_for(std::chrono::seconds(10));

    return 0;
}

so what should happen , is the semaphore should timeout every other time, as its waiting 1 second, but the producer is adding every 2 seconds

so without the wrapper i correctly get

si 0
thr 0
timedwait 0 : 0
timedwait -1 : 110
timedwait 0 : 0
timedwait -1 : 110
timedwait 0 : 0
timedwait -1 : 110
timedwait 0 : 0

but if i wrap with

--wrap pthread_create
--wrap sem_init
--wrap sem_post
--wrap sem_timedwait

I get nothing useful 🙂

si 0
thr 0
timedwait 0 : 0
timedwait 0 : 11
timedwait 0 : 11
timedwait 0 : 11

the return code is wrong.. not -1, and error 11 (EAGAIN) is not even defined for sem_timedwait.
as far as i can see , sem_timedwait, seems to be behaving like set_wait, i.e. it just waits for a post, and does not timeout.

... im wondering if its something to do with clocks, but i cannot find any examples with xenomai for timedwaits

note: Im compiling the same as your example , just I use -Wl, @file ...
(using nm, i then check things are wrapped as expected)

Im starting to get the feeling i should ditch the posix wrappers, they seem really unreliable...
(they kind of worked in some instances with std c++ classes, and then in other cases not)
but that means a whole load of work to start abstracting stuff, just to get bela working 🙁

(and i thought c++ stl was the abstraction 😉 )

giuliomoro

if it returns 0 then the errno should be the last errno set by some other call? This explains why EAGAIN is not returning an expected value.

This said, I am not sure what the problem is. I can see by cat /proc/xenomai/registry/usage that a Xenomai registry entry is used when initing the semaphore.

There actually is an instance where sem_timedwait() is used in the Xenomai code base:

./testsuite/smokey/cpu-affinity/cpu-affinity.c:	if (!__Terrno(ret, __STD(sem_timedwait(&context->done, &ts)))) {

have a look in /opt/xenomai-3/testsuite/smokey/cpu-affinity/cpu-affinity.c and see if that helps at all.

giuliomoro

@thetechnobear got it: it was easier than expected. You need to wrap clock_gettime().
If you try this with the same options as above (that is without wrapping clock_gettime() at compile time)

#include <iostream>
#include <pthread.h>
#include <semaphore.h>
#include <xenomai/init.h>

#include <thread>
#include <chrono>


sem_t sem;

void* func(void*) {
    while(true) {
        bool timeout = false;

        int t = 1000 * 1000;
        struct timespec ts = { t/1000000, 1000*(t%1000000) };
	double time;

        clock_gettime(CLOCK_REALTIME, &ts);
	time = ts.tv_sec + ts.tv_nsec/1000000000.0;
	std::cout << "clock_gettime: "<< time << "\n";

        __wrap_clock_gettime(CLOCK_REALTIME, &ts);
	time = ts.tv_sec + ts.tv_nsec/1000000000.0;
	std::cout << "clock_gettime: "<< time << "\n";

        t += (ts.tv_nsec/1000);
        ts.tv_nsec = 0;
        ts.tv_sec += t/1000000;
        ts.tv_nsec += 1000*(t%1000000);
        errno=0;
        int rc=sem_timedwait(&sem,&ts);
        std::cout << "timedwait " << rc << " : " << errno << std::endl;
    }
    return 0;
}

int main ()
{
    int argc = 0;
    char *const *argv;
    xenomai_init(&argc, &argv);
    std::cout << "si " << sem_init(&sem,0,0) << std::endl;

    pthread_t thr;
    std::cout << "thr " << pthread_create(&thr,0,func,0) << std::endl;

    for(int i=0;i<10;i++) {
        sem_post(&sem);
        std::this_thread::sleep_for(std::chrono::seconds(2));
    }
    std::this_thread::sleep_for(std::chrono::seconds(10));

    return 0;
}

(note that I am calling the POSIX clock_gettime() and the Cobalt __wrap_clock_gettime() and comparing the results). You will get:

si 0
thr 0
clock_gettime: 1.52479e+09
__wrap_clock_gettime: 707.068
timedwait 0 : 0
clock_gettime: 1.52479e+09
__wrap_clock_gettime: 707.069
timedwait -1 : 110
clock_gettime: 1.52479e+09
__wrap_clock_gettime: 708.07
timedwait 0 : 11
clock_gettime: 1.52479e+09
__wrap_clock_gettime: 709.068
timedwait -1 : 110
clock_gettime: 1.52479e+09
__wrap_clock_gettime: 710.068
timedwait 0 : 11
clock_gettime: 1.52479e+09
__wrap_clock_gettime: 711.068

so there are two CLOCK_REALTIME, one for Xenomai and one for Linux. Earlier on, you were reading from the Linux one and using it to set an event for the Xenomai one.

So you can keep your code as above but add --wrap clock_gettime to your @ file.

thetechnobear

Thanks @giuliomoro thats fixed it...

after a few other small hurdles, I finally I have success on my project, no MSW (ok, one at startup but thats ok 😉 )
so very happy!

most of my fights to get this working has been to do with the way std c++ and the posix layer interact, or don't.
many places you can step around the issues (mostly using native_handle()) , but some you can't
this is not because the lib has pthreads compiled in (it doesn't it just has references, which correctly get wrapped) , but i think its because it does have some internal state, which just seems to get in the way of some objects.
(e.g. std::thread with std::mutex, can be made to work, but std::thread with semaphores, nope!)

add to that , creating test programs often creates other issues... and its a bit of a nightmare,
thought I'm sure once you have done it once, its not so bad.

anyway thanks Giulio , I really appreciated your support, and patience 🙂
... you'll be able to see it at superbooth.

giuliomoro

This got me confused a while ago, and to this day, I am still not quite sure I get the answer: http://www.xenomai.org/pipermail/xenomai/2017-March/037220.html

thetechnobear you'll be able to see it at superbooth.

am I getting a cake? 🙂

thetechnobear

giuliomoro This got me confused a while ago, and to this day, I am still not quite sure I get the answer: http://www.xenomai.org/pipermail/xenomai/2017-March/037220.html

yeah, thats in line with what Ive found... I had a bit of a look at the C++ STL code, I think the issue is whilst it does use the pthreads library (and these get wrapped) theres some additional state and logic, which messes sometimes
e.g. conditional vars dont work at all, mutex and threads kind of work, sometimes.
it was the 'sometimes' that really hit me, especially as i could see the calls were being wrapped.

as i said, in some places you can workaround using the native_handle() of STL, this was useful for me for threads, where I didn't want to change the C++ class interface, which would get really messy... but still ended up in quite a few #ifdef .
I guess once Ive got everything work as I want it on Bela, I'll take the pain, and refactor , so I can remove the conditional macros and replace with a 'platform layer' for bela. (will be useful for win32 as well )