Hi! Im trying to compile some JUCE code to run on bela and have come across many issues, managing to find a solution to each one by one. However, I've reached a point where I'm afraid I might break things apart.

I've managed to set a virtual cross compilation environment (a virtualBox xenomai image), to speed up builds for the embedded platform. In the pursuit of trying to get a successful build and linked binary, I ended up using the latest g++ arm-linux-gnueabihf (version 9) to have access to some of the latest features of c++17, and now the executable won't run on bela because of a GLIBC mismatch (requires version 2.28 but has 2.24 installed).

I know upgrading glibc, while keeping the same linux kernel might be risky so I wonder what you guys recommend in this case. Below are the possible solutions in order of preference and minimum risk that I'm thinking of:

  1. Upgrading firmware on bela with the latest march version and hoping that supports the latest g++ 9 version
    (current version: Bela image, v0.3.7a, 16 October 2019,
    Linux version 4.4.113-ti-xenomai-r149 (root@giulio-VirtualBox) (gcc version 5.5.0 (Linaro GCC 5.5-2017.10) ) #1 PREEMPT Wed Sep 26 15:40:33 BST 2018)

  2. Having some kind of virtual environment on bela with the required GLIBC 2.28 and somehow update runpath LD flags of my app to use that instead of the default ones

  3. Manually update the glibc version on bela and hope that it will actually be backwards compatible

I have a feeling that option 1 falls immediately, from what I've seen on the bela-image-builder repo. So please let me know which one would be the best option (or some other that I have not thought of)

p.s. I know I could go and edit the code and get rid of the latest features of c++17, but it's too much work and changes would be intrusive so I have excluded this from the start

    There is a cross compiling option with clang10 that doesn't seem to have this issue, see here. Ultimately the best solution would be to update the Bela image. I will try and build an image with the latest version of Debian today or tomorrow.

    It would be awesome if you could provide a Bela image with an up-to-date linux kernel in such a short time!

    I have already checked out the clang cross-compiler, but I have found it poorly documented and it didn't work out of the box so I did not want to invest more time in it. However I'm not really sure what issue are you saying is mitigated. If you are talking about the need of having a virtual linux machine, yes I agree. But fixing the glibc mismatch? Is clang10 linking statically against the latest c++17 libraries? Otherwise I don't see how using clang10 will help me in this case.

    Anyways I'm impatiently waiting for the new OS Image, as that should indeed be the best solution. Please look for a linux version that is coupled with gcc/g++ version 9.

    Thanks a lot for your responsiveness!

      @thetechnobear ?

      alexvoina Anyways I'm impatiently waiting for the new OS Image, as that should indeed be the best solution. Please look for a linux version that is coupled with gcc/g++ version 9.

      I just built a Buster image with the 4.14 kernel. Flashing it now to see if it boots.

        great! let me know how it goes and where can i get it from 😃

        the above apt install will depending upon what you have set your apt-repo too, and if it has C++17 cross compiled or not. if you need a later version that is available you'd have to compiled it for yourself.

        many thanks for your quick replies! amazing work

        Unfortunately I still have a glibc mismatch of version 2.29 (my bad, cause I wrote 2.28 above but I also had a 2.29 dependency). The current Debian buster image has it's latest glibc version 2.28-10, so in order to get rid of the issue, I still need to do one of the hacky things listed above.

        However I will try and do the cross compilation with g++ version 8 and hope that has the c++ library new enough to compile my code. I'll keep you updated on this

        If you could provide another image with a Linux kernel that has gcc version 9 it would be awesome.

        I really appreciate that you have acted so fast, not keeping me blocked on it.

          alexvoina If you could provide another image with a Linux kernel that has gcc version 9 it would be awesome.

          I am only set up to build Debian images. I guess this means I have to try and build bullseye/sid (which has 2.30-4)?

            I'm not really sure how glibc mismatch rolls out when the software requires 2.29 and the device has 2.30. If building bullseye does not require a big amount of time, you could try 😃

            I have managed to build with arm-linux-gnueabi-g++-8 but it somehow linked with the same version of GLIBC and my program still does not run on bela. The g++-cross compiler has different libraries/folders for each version, e. g. c++/8 or c++/9 but uses the same libc.so file. I'm not really sure if g++8 is supposed to use glibc version 2.29, or it is just there and has not been updated (more precisely downgraded to something older than 2.29) because I previously had g++9 installed.

            There were minor code modifications needed for the switch between C++ (g++-9) and C++(g++-8) so I should be fine if I can enforce an older glibc version.

            Still it would be nice in the future to have a bela image compatible with g++9, so one can simply install g++9-cross and compile.

              alexvoina If building bullseye does not require a big amount of time, you could t

              it's building now. Fact is, we are very unlikely to be maintaining an unstable image, when we barely have the resources to maintain the stable one. Once I get it to build, I will send you the binary and push the needed changes to the bela-image-builder repo.

                @alexvoina could you please describe in detail what features of C++17 aren't supported by libc-2.28 ? I don't actually know a whole lot about these things, but I figured that any C++ features would be in libstdc++?

                  Finally I have managed to run my code on (v0.4.0alpha-1-gc5cfd95-dirty, 26 April 2020), built with arm-linux-gnueabihf-g++-8, and glibc version 2.28-7

                  giuliomoro could you please describe in detail what features of C++17 aren't supported by libc-2.28 ?

                  it's not that libc-2.28 does not support features from C++17, it's just the whole g++ toolchain comes with specific libraries for both C++ and glibc.

                  In my first attempt I used the same gcc as, Bela image, v0.3.7a, 16 October 2019 which is gcc version 5.5.0.
                  This gcc version had an old c++ library that did not include features like std::optional, and std::variant, which were extensively used in my code.

                  Then I have upgraded to g++9, and this showed more or less the same behavior as compiling with XCode (Clang), which was very convenient for me. The problem here was that g++9 was using glibc-2.29 (or 2.30) and my executable would need an environment with the corresponding glibc version, hence the need of a newer bela image.

                  Finally I have installed g++8, which had a few missing features (because it had a different C++ version): new errors appeared for example when declaring static constexpr char arrays, but again, no big deal.

                  The only way I could run the exec was to downgrade my glibc to version 2.28, which I hope is in sync with g++ 8. Now I have a seg fault of course, but still I have some thing to work with 🙂)

                      Thanks for the explanation.

                      alexvoina that g++9 was using glibc-2.29 (or 2.30)

                      You mean ld was using it? Or is there's something in the headers that has actually changed between the versions? (again, sorry I am not experienced enough in these things).

                      So, if the actual problem is that you do not have an old enough version of glibc for ARM for your host, then why don't you grab the one from the board instead?

                      If - instead - you need a newer version of libc than the one on the board, then I'd suggest you go for

                      alexvoina Having some kind of virtual environment on bela with the required GLIBC 2.28 and somehow update runpath LD flags of my app to use that instead of the default ones

                      My understanding is that it should be as easy as copying the library from the building host to the board, and run the program with LD_PRELOAD=/path/to/your/libc ./executable ?

                      To avoid issues with linking, I tend to use distcc instead of full cross-build, and performing the linking on the board. However, with JUCE the linking stage itself is quite slow, as I reported here.

                      alexvoina , Bela image, v0.3.7a, 16 October 2019 which is gcc version 5.5.0.

                      it's actually gcc 6.3 on that image, but I don't think this makes any difference for you.

                              giuliomoro You mean ld was using it? Or is there's something in the headers that has actually changed between the versions? (again, sorry I am not experienced enough in these things).

                              I think the guys who maintain the g++ project choose the suite of packages. G++ usually has all the files contained in the directory where it is installed, and has a specific structure if having multiple g++ versions

                              e.g: root# arm-linux-gnueabihf-g++-8 -print-file-name=libc.so yields
                              /usr/lib/gcc-cross/arm-linux-gnueabihf/8/../../../../arm-linux-gnueabihf/lib/libc.so
                              which would be the exactly the same if you would have g++9 and run the same command

                              however if you run root# arm-linux-gnueabihf-g++-8 -print-file-name=libstdc++.so yielding
                              /usr/lib/gcc-cross/arm-linux-gnueabihf/8/libstdc++.so, and then for arm-linux-gnueabihf-g++-9
                              it will point you to a file in a different directory.

                              So my understanding is g++ uses whatever it has in its directories. And it is strongly advised not to mess up with standard libraries, therefore just use sudo apt-get install g++ version, and yeah copy additional libraries if needed to be linked with your project.

                              giuliomoro then why don't you grab the one from the board instead?

                              because glibc is used in most of the code that is written out there, and changing the version without taking care of dependencies could lead to total chaos.

                              giuliomoro My understanding is that it should be as easy as copying the library from the building host to the board, and run the program with LD_PRELOAD=/path/to/your/libc ./executable ?

                              Yes, I think this is feasible. But again your whole binary will use LD_PRELOAD=/path/to/your/libc, and there might be a call to some system function that the OS can't handle.

                              I can't think of a potential bug to describe that might arise from unsynced standard libraries, but I wanted to follow the best practices and avoid risky approaches at all cost.

                              Linking stage is not slow for me. Only the first build is slow, but that is to compile most of the code, and for minor modifications in the source afterwards it runs really quick. Always remember to pass -j5 (could be even more) for parallel jobs.

                                      ok thanks for the further explainer. Bullseye is still building (had a few issues, (including libseasocks's CMake files not being compatible with whatever CMake comes with it) but getting there)

                                      alexvoina Linking stage is not slow for me

                                      I meant running it on Bela when using distcc from within it.

                                        thanks a lot! at the moment I'm going with the working build, but I will test the latest image soon and tell you if it works out of the box with g++-9

                                        Thanks again for being so responsive!

                                        12 days later

                                        so was a kernel upgrade needed for c++17? or was this just the easiest way forward?

                                        Ive also come across a library that needed c++17, so id be interested in clang being updated on the bela board.

                                        importantly though... ive been cross-compiling on my mac, using -std=c++17 (with xcBela) and then copying over the binaries - and it works absolutely fine - so there doesn't appear to be a runtime dependancy for me.