PLEASE format your posts properly.
Enclose blocks of code or terminal logs with ``` , .e.g:
```
/tmp/distcc/distccd_502b6055.ii🙁.text+0x3e8): undefined reference to MHD_create_response_from_fd'
/tmp/distcc/distccd_502b6055.ii:(.text+0x414): undefined reference to
MHD_add_response_header'
/tmp/distcc/distccd_502b6055.ii🙁.text+0x42c): undefined reference to MHD_add_response_header'
```

so that it gets visualized as

/tmp/distcc/distccd_502b6055.ii:(.text+0x3e8): undefined reference to `MHD_create_response_from_fd'
/tmp/distcc/distccd_502b6055.ii:(.text+0x414): undefined reference to `MHD_add_response_header'
/tmp/distcc/distccd_502b6055.ii:(.text+0x42c): undefined reference to `MHD_add_response_header'

I have been editing your posts to be able to read them.

Digital-Larry This does not agree with the directions given. They hint that there should be some .so* files also:

libHTTPDFaust.so*

some files are missing there, I will fix that shortly.

Digital-Larry ok I see many references to MHD_stop_daemon for example, what is that I wonder, well I found this, next question is how do I integrate it with this project?

https://www.gnu.org/software/libmicrohttpd/manual/libmicrohttpd.html

The instructions from here actually say:

Install libmicrohttpd-dev on the board:
connect the board to the internet and run
apt-get install libmicrohttpd-dev

you may have skipped that?

Digital-Larry ext question, when I ran the faust2bela script while watching ~/distcc.log, nothing is appended to the log file (this is on the host of course). The file is actually compiled and running on the Bela though. What do I need to do to enable the distcc to run when I run faust2bela -tobela also from the host?

You need to enable it to build all Bela projects. It is described in the distcc-on-bela documentation:

Using distcc by default on Bela
Make the following edit to the Bela Makefile:

 endif

 ifeq ($(COMPILER), clang)
-  CLANG_PATH?=/usr/bin/clang
+ CLANG_PATH=distcc-clang
   CC=$(CLANG_PATH)
   CXX=$(CLANG_PATH)++
   DEFAULT_CPPFLAGS += -DNDEBUG # Maybe we should add back in -no-integrated-as?

have you done these steps yet? It doesn't look like it, because your linker line starts with /usr/bin/clang++ instead of distcc-clang+

    giuliomoro Thank you!!!! Sorry about the bad formatting, it wasn't obvious what I should be doing, but I suppose I should be aware this is a markdown environment. My building of the httplib locally was probably unnecessary as I had previously used the apt-get step. Using apt-get gives you 0.9.51 while building from source gives you a version that is a couple of years more recent, not that I think it matters.

    I did not edit the Bela makefile.

    I'll try this later today. I'm sick with the flu and have a pounding headache. 8( But I trust that if this succeeds, I will be singing in the sunshine. 8^) with pitch shifted delays

    giuliomoro THANK YOU!!!!! The updated library files made a bunch of difference. However I had to copy them to /usr/lib as /usr/local/lib did not work. OMG I can see the GUI! I'm even using remote access software so I'm viewing the Bela attached to my Linux box downstairs on USB while I'm upstairs using Windows. Tonight I will try out some audio tests!

      Digital-Larry However I had to copy them to /usr/lib as /usr/local/lib did not work.

      hmmm not sure why that would be the case. Possibly it just needed running ldconfig ... Or possibly, the linker would find the file libHTTPDFaust.a you previously had in /usr/local/lib and looked no further? Not sure.

        not sure really. However I have an explanation as to why it worked with the shared library (.so) and not with the static (.a) one:

        a static library is simply an archive of object files, while a shared library - among other things - has been linked with all the libraries it depends upon. So when using the static library, in order for that to work, you need to explicitly pass to the linker the library(ies) that your your static library depends upon (in this case: -lmicrohttpd), so that the external symbols in the objects files that make up the .a file can be resolved. When using the shared library, instead, these symbols have been already resolved when it was linked together in a .so file, so there is no need to explicitly state it again.

        Still having trouble at this step.

        Using distcc by default on Bela
        Make the following edit to the Bela Makefile:

        By "the Bela Makefile" I presume that you mean the file /root/Bela/Makefile. Sorry, it wasn't obvious to me.

        I found the lines and made the change.

        echo 'Building render.cpp...'
        Building render.cpp...
        distcc-clang++  -I/root/Bela/projects/delay01 -I./include -I./build/pru/ -I/usr/xenomai/include/cobalt -I/usr/xenomai/include -march=armv7-a -mfpu=vfp3 -D_GNU_SOURCE -D_REENTRANT -fasynchronous-unwind-tables -D__COBALT__ -D__COBALT_WRAP__ -DXENOMAI_SKIN_posix -DXENOMAI_MAJOR=3 -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -DNDEBUG -DBELA_USE_RTDM -I/root/Bela/resources/stretch/include -std=c++11 -Wno-varargs -DNDEBUG  -Wall -c -fmessage-length=0 -U_FORTIFY_SOURCE -MMD -MP -MF"/root/Bela/projects/delay01/build/render.d" -o "/root/Bela/projects/delay01/build/render.o" "/root/Bela/projects/delay01/render.cpp" 
        distcc[2855] ERROR: nonblocking connect to 192.168.7.1:3632 failed: Connection refused
        distcc[2855] (dcc_build_somewhere) Warning: failed to distribute and fallbacks are disabled
        Makefile:434: recipe for target '/root/Bela/projects/delay01/build/render.o' failed
        make: *** [/root/Bela/projects/delay01/build/render.o] Error 103
        Connection to 192.168.7.2 closed.
        gary@audio-workstation:~/faust/examples/bela$ 

          OK the distccd wasn't running. Now that I started that, I get:

          distccd[14751] (dcc_standalone_server) not detaching
          distccd[14751] (dcc_new_pgrp) already a process group leader
          distccd[14751] (dcc_log_daemon_started) preforking daemon started (3.3.2 x86_64-pc-linux-gnu, built May 26 2019 14:11:45)
          distccd[14751] (dcc_create_kids) up to 1 children
          distccd[14751] (dcc_create_kids) up to 2 children
          distccd[14751] (dcc_create_kids) up to 3 children
          distccd[14751] (dcc_create_kids) up to 4 children
          distccd[14751] (dcc_create_kids) up to 5 children
          distccd[14751] (dcc_create_kids) up to 6 children
          distccd[14752] (dcc_check_client) connection from 192.168.7.2:59746
          distccd[14752] (check_address_inet) match client 0x207a8c0, value 0x207a8c0, mask 0xffffffff
          distccd[14752] (dcc_r_token_int) got DIST00000001
          distccd[14752] (dcc_r_token_int) got ARGC00000013
          distccd[14752] (dcc_r_argv) reading 19 arguments from job submission
          distccd[14752] (dcc_r_token_int) got ARGV0000000f
          distccd[14752] (dcc_r_token_string) got 'clang++-3.9-arm'
          distccd[14752] (dcc_r_argv) argv[0] = "clang++-3.9-arm"
          distccd[14752] (dcc_r_token_int) got ARGV0000000e
          distccd[14752] (dcc_r_token_string) got '-march=armv7-a'
          distccd[14752] (dcc_r_argv) argv[1] = "-march=armv7-a"
          distccd[14752] (dcc_r_token_int) got ARGV0000000a
          distccd[14752] (dcc_r_token_string) got '-mfpu=vfp3'
          distccd[14752] (dcc_r_argv) argv[2] = "-mfpu=vfp3"
          distccd[14752] (dcc_r_token_int) got ARGV0000001c
          distccd[14752] (dcc_r_token_string) got '-fasynchronous-unwind-tables'
          distccd[14752] (dcc_r_argv) argv[3] = "-fasynchronous-unwind-tables"
          distccd[14752] (dcc_r_token_int) got ARGV00000003
          distccd[14752] (dcc_r_token_string) got '-O3'
          distccd[14752] (dcc_r_argv) argv[4] = "-O3"
          distccd[14752] (dcc_r_token_int) got ARGV0000000e
          distccd[14752] (dcc_r_token_string) got '-march=armv7-a'
          distccd[14752] (dcc_r_argv) argv[5] = "-march=armv7-a"
          distccd[14752] (dcc_r_token_int) got ARGV00000010
          distccd[14752] (dcc_r_token_string) got '-mtune=cortex-a8'
          distccd[14752] (dcc_r_argv) argv[6] = "-mtune=cortex-a8"
          distccd[14752] (dcc_r_token_int) got ARGV00000010
          distccd[14752] (dcc_r_token_string) got '-mfloat-abi=hard'
          distccd[14752] (dcc_r_argv) argv[7] = "-mfloat-abi=hard"
          distccd[14752] (dcc_r_token_int) got ARGV0000000a
          distccd[14752] (dcc_r_token_string) got '-mfpu=neon'
          distccd[14752] (dcc_r_argv) argv[8] = "-mfpu=neon"
          distccd[14752] (dcc_r_token_int) got ARGV00000010
          distccd[14752] (dcc_r_token_string) got '-ftree-vectorize'
          distccd[14752] (dcc_r_argv) argv[9] = "-ftree-vectorize"
          distccd[14752] (dcc_r_token_int) got ARGV0000000b
          distccd[14752] (dcc_r_token_string) got '-ffast-math'
          distccd[14752] (dcc_r_argv) argv[10] = "-ffast-math"
          distccd[14752] (dcc_r_token_int) got ARGV0000000a
          distccd[14752] (dcc_r_token_string) got '-std=c++11'
          distccd[14752] (dcc_r_argv) argv[11] = "-std=c++11"
          distccd[14752] (dcc_r_token_int) got ARGV0000000c
          distccd[14752] (dcc_r_token_string) got '-Wno-varargs'
          distccd[14752] (dcc_r_argv) argv[12] = "-Wno-varargs"
          distccd[14752] (dcc_r_token_int) got ARGV00000005
          distccd[14752] (dcc_r_token_string) got '-Wall'
          distccd[14752] (dcc_r_argv) argv[13] = "-Wall"
          distccd[14752] (dcc_r_token_int) got ARGV00000002
          distccd[14752] (dcc_r_token_string) got '-c'
          distccd[14752] (dcc_r_argv) argv[14] = "-c"
          distccd[14752] (dcc_r_token_int) got ARGV00000012
          distccd[14752] (dcc_r_token_string) got '-fmessage-length=0'
          distccd[14752] (dcc_r_argv) argv[15] = "-fmessage-length=0"
          distccd[14752] (dcc_r_token_int) got ARGV00000002
          distccd[14752] (dcc_r_token_string) got '-o'
          distccd[14752] (dcc_r_argv) argv[16] = "-o"
          distccd[14752] (dcc_r_token_int) got ARGV0000002a
          distccd[14752] (dcc_r_token_string) got '/root/Bela/projects/delay01/build/render.o'
          distccd[14752] (dcc_r_argv) argv[17] = "/root/Bela/projects/delay01/build/render.o"
          distccd[14752] (dcc_r_token_int) got ARGV00000026
          distccd[14752] (dcc_r_token_string) got '/root/Bela/projects/delay01/render.cpp'
          distccd[14752] (dcc_r_argv) argv[18] = "/root/Bela/projects/delay01/render.cpp"
          distccd[14752] (dcc_r_argv) got arguments: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /root/Bela/projects/delay01/build/render.o /root/Bela/projects/delay01/render.cpp
          distccd[14752] (dcc_scan_args) scanning arguments: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /root/Bela/projects/delay01/build/render.o /root/Bela/projects/delay01/render.cpp
          distccd[14752] (dcc_scan_args) found object/output file "/root/Bela/projects/delay01/build/render.o"
          distccd[14752] (dcc_scan_args) found input file "/root/Bela/projects/delay01/render.cpp"
          distccd[14752] compile from render.cpp to render.o
          distccd[14752] (dcc_run_job) output file /root/Bela/projects/delay01/build/render.o
          distccd[14752] (dcc_input_tmpnam) input file /root/Bela/projects/delay01/render.cpp
          distccd[14752] (dcc_r_token_int) got DOTI0018d9d3
          distccd[14752] (dcc_r_file) received 1628627 bytes to file /tmp/distccd_1db50d83.ii
          distccd[14752] (dcc_r_file_timed) 1628627 bytes received in 0.146217s, rate 10877kB/s
          distccd[14752] (dcc_set_input) changed input from "/root/Bela/projects/delay01/render.cpp" to "/tmp/distccd_1db50d83.ii"
          distccd[14752] (dcc_set_input) command after: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /root/Bela/projects/delay01/build/render.o /tmp/distccd_1db50d83.ii
          distccd[14752] (dcc_set_output) changed output from "/root/Bela/projects/delay01/build/render.o" to "/tmp/distccd_1dc50d83.o"
          distccd[14752] (dcc_set_output) command after: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /tmp/distccd_1dc50d83.o /tmp/distccd_1db50d83.ii
          distccd[14752] (dcc_check_compiler_masq) /usr/local/bin/clang++-3.9-arm is not a symlink
          distccd[14752] (dcc_spawn_child) forking to execute: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /tmp/distccd_1dc50d83.o /tmp/distccd_1db50d83.ii
          distccd[14752] (dcc_spawn_child) child started as pid14971
          distccd[14971] (dcc_new_pgrp) entered process group
          distccd[14971] (dcc_increment_safeguard) setting safeguard: _DISTCC_SAFEGUARD=1
          distccd[14752] (dcc_collect_child) cc child 14971 terminated with status 0x100
          distccd[14752] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
          distccd[14752] (dcc_x_token_int) send DONE00000001
          distccd[14752] (dcc_x_token_int) send STAT00000100
          distccd[14752] (dcc_x_file) send 248 byte file /tmp/distcc_161f0d83.stderr with token SERR and compression 69
          distccd[14752] (dcc_x_token_int) send SERR000000f8
          distccd[14752] (dcc_x_file) send 0 byte file /tmp/distcc_16d70d83.stdout with token SOUT and compression 69
          distccd[14752] (dcc_x_token_int) send SOUT00000000
          distccd[14752] (dcc_x_token_int) send DOTO00000000
          distccd[14752] clang++-3.9-arm /root/Bela/projects/delay01/render.cpp on localhost failed
          distccd[14752] job complete
          distccd[14752] (dcc_cleanup_tempfiles_inner) deleted 5 temporary files
          distccd[14752] (dcc_job_summary) client: 192.168.7.2:59746 COMPILE_ERROR exit:1 sig:0 core:0 ret:0 time:988ms clang++-3.9-arm /root/Bela/projects/delay01/render.cpp

          in the distcc log.

          On host console I get:

          echo 'Building render.cpp...'
          Building render.cpp...
          distcc-clang++  -I/root/Bela/projects/delay01 -I./include -I./build/pru/ -I/usr/xenomai/include/cobalt -I/usr/xenomai/include -march=armv7-a -mfpu=vfp3 -D_GNU_SOURCE -D_REENTRANT -fasynchronous-unwind-tables -D__COBALT__ -D__COBALT_WRAP__ -DXENOMAI_SKIN_posix -DXENOMAI_MAJOR=3 -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -DNDEBUG -DBELA_USE_RTDM -I/root/Bela/resources/stretch/include -std=c++11 -Wno-varargs -DNDEBUG  -Wall -c -fmessage-length=0 -U_FORTIFY_SOURCE -MMD -MP -MF"/root/Bela/projects/delay01/build/render.d" -o "/root/Bela/projects/delay01/build/render.o" "/root/Bela/projects/delay01/render.cpp" 
          distcc[3031] ERROR: compile /root/Bela/projects/delay01/render.cpp on 192.168.7.1 failed
          distcc[3031] (dcc_build_somewhere) Warning: remote compilation of '/root/Bela/projects/delay01/render.cpp' failed, retrying locally
          distcc[3031] (dcc_build_somewhere) Warning: failed to distribute and fallbacks are disabled
          /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
          Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          Makefile:434: recipe for target '/root/Bela/projects/delay01/build/render.o' failed
          make: *** [/root/Bela/projects/delay01/build/render.o] Error 1
          Connection to 192.168.7.2 closed.
          gary@audio-workstation:~/faust/examples/bela$ 

            Think you were talking about this before, but not sure I understood.

            ary@audio-workstation:~/faust/examples/bela$ cat /usr/local/bin/*arm
            #!/bin/bash
            /usr/bin/clang -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@
            #!/bin/bash
            /usr/bin/clang++ -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $0
            gary@audio-workstation:~/faust/examples/bela$ 

            These don't currently point to clang-3.9. They just point to clang which is actually version 6. I think you were saying that it has to be version 3.9?

            However, this step works:

            distcc-clang -c ~/Bela/resources/network/udp-client.c
              gary@audio-workstation:~$ ld -V
              GNU ld (GNU Binutils for Ubuntu) 2.30
                Supported emulations:
                 elf_x86_64
                 elf32_x86_64
                 elf_i386
                 elf_iamcu
                 i386linux
                 elf_l1om
                 elf_k1om
                 i386pep
                 i386pe
              gary@audio-workstation:~$ 

              Digital-Larry By "the Bela Makefile" I presume that you mean the file /root/Bela/Makefile.

              yes

              Digital-Larry On host console I get:

              I guess that's technically the "client" console (i.e.: what you see there is while you are logged in on Bela).

              So it seems that the compiler is failing, and my guess is that it is because it is being passed -march=armv7-a but no -target armv7l-unknown-linux-gnueabihf, and this - I guess - is because you are not calling the wrapper but you are calling the vanilla compiler directly (through the symlink you created).
              The point of the wrapper is exactly to automatically add -target armv7l-unknown-linux-gnueabihf to all calls.

              I think what is happening is what I mentioned in the other thread:

              giuliomoro But if you do put some symlinks there, they should be pointing NOT to the regular clang, but to the clang-3.9-arm and clang++-3.9-arm files you created in /usr/local/bin/, and they will eventually call clang with the appropriate option -target armv7l-unknown-linux-gnueabihf, to tell it to build code for ARM.

              Digital-Larry gary@audio-workstation:/usr/local/lib$ ls -l distcc/
              total 0
              lrwxrwxrwx 1 root root 14 May 28 17:45 clang-3.9-arm -> /usr/bin/clang
              lrwxrwxrwx 1 root root 16 May 28 17:45 clang++-3.9-arm -> /usr/bin/clang++

              Is this working? I find it surprising, but I guess it may depend on how you clang has been built. This way you are simply using the regular clang on your computer, but it should need a -target armv7l-unknown-linux-gnueabihf option in order to know it should build for arm (at least, it does with mine, which I installed with macports on Mac).

              basically, remove your current symlinks from /usr/local/lib/distcc/ and make some new ones that point NOT to the vanilla clang/clang++, but to the wrapper clang-3.9-arm/clang++-3.9-arm you created:

              sudo rm /usr/local/lib/distcc/*
              sudo ln -s /usr/local/bin/clang-3.9-arm/ /usr/local/lib/distcc/
              sudo ln -s /usr/local/bin/clang++-3.9-arm/ /usr/local/lib/distcc/

              Digital-Larry However, this step works:

              distcc-clang -c ~/Bela/resources/network/udp-client.c

              I see now how that can easily give false positives: it just checks that the server can compile, but does not check the nature of the resulted file. If you were to do file a.out, it would probably say it's an x86 file and not an ARM one.
              So the test should probably be amended to :

              distcc-clang -c ~/Bela/resources/network/udp-client.c -march=armv7-a

              This should then fail similarly to the how the Bela project is failing in your current configuration, and should be fixed with the updated symlinks.

                OK let's see where we are now.

                root@bela:~/Bela# grep distcc Makefile 
                  CLANG_PATH=distcc-clang
                root@bela:~/Bela#

                reference command line build on bela works (without specifying architecture):

                root@bela:~/Bela/resources/network# distcc-clang -v -c udp-client.c
                clang version 3.9.1-9 (tags/RELEASE_391/rc2)
                Target: armv7l-unknown-linux-gnueabihf
                Thread model: posix
                InstalledDir: /usr/bin
                Found candidate GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6
                Found candidate GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0
                Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/6
                Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/6.3.0
                Selected GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0
                Candidate multilib: .;@m32
                Selected multilib: .;@m32
                 "/usr/lib/llvm-3.9/bin/clang" -cc1 -triple armv7-unknown-linux-gnueabihf -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name udp-client.c -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu cortex-a8 -target-abi aapcs-linux -mfloat-abi hard -v -dwarf-column-info -debugger-tuning=gdb -resource-dir /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1 -internal-isystem /usr/local/include -internal-isystem /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/include -internal-externc-isystem /usr/include/arm-linux-gnueabihf -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir /root/Bela/resources/network -ferror-limit 19 -fmessage-length 80 -fallow-half-arguments-and-returns -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -o - -x c udp-client.c
                clang -cc1 version 3.9.1 based upon LLVM 3.9.1 default target armv7l-unknown-linux-gnueabihf
                ignoring nonexistent directory "/include"
                #include "..." search starts here:
                #include <...> search starts here:
                 /usr/local/include
                 /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/include
                 /usr/include/arm-linux-gnueabihf
                 /usr/include
                End of search list.
                clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)
                Target: armv7l-unknown-linux-gnueabihf
                Thread model: posix
                InstalledDir: /usr/bin
                 "/usr/lib/llvm-3.9/bin/clang" -cc1 -triple armv7-unknown-linux-gnueabihf -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name distccd_ba396596.i -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -target-cpu cortex-a8 -target-abi aapcs-linux -mfloat-abi hard -v -dwarf-column-info -debugger-tuning=gdb -coverage-file /tmp/distccd_ba666596.o -resource-dir /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1 -fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length 0 -fallow-half-arguments-and-returns -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -o /tmp/distccd_ba666596.o -x cpp-output /tmp/distccd_ba396596.i
                clang -cc1 version 3.9.1 based upon LLVM 3.9.1 default target x86_64-pc-linux-gnu
                #include "..." search starts here:
                End of search list.
                root@bela:~/Bela/resources/network# 

                reference build works with specifying architecture:

                root@bela:~/Bela/resources/network# distcc-clang -c -v ~/Bela/resources/network/udp-client.c -march=armv7-a
                clang version 3.9.1-9 (tags/RELEASE_391/rc2)
                Target: armv7l-unknown-linux-gnueabihf
                Thread model: posix
                InstalledDir: /usr/bin
                Found candidate GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6
                Found candidate GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0
                Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/6
                Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/6.3.0
                Selected GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0
                Candidate multilib: .;@m32
                Selected multilib: .;@m32
                 "/usr/lib/llvm-3.9/bin/clang" -cc1 -triple armv7-unknown-linux-gnueabihf -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name udp-client.c -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu cortex-a8 -target-abi aapcs-linux -mfloat-abi hard -v -dwarf-column-info -debugger-tuning=gdb -resource-dir /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1 -internal-isystem /usr/local/include -internal-isystem /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/include -internal-externc-isystem /usr/include/arm-linux-gnueabihf -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir /root/Bela/resources/network -ferror-limit 19 -fmessage-length 80 -fallow-half-arguments-and-returns -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -o - -x c /root/Bela/resources/network/udp-client.c
                clang -cc1 version 3.9.1 based upon LLVM 3.9.1 default target armv7l-unknown-linux-gnueabihf
                ignoring nonexistent directory "/include"
                #include "..." search starts here:
                #include <...> search starts here:
                 /usr/local/include
                 /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/include
                 /usr/include/arm-linux-gnueabihf
                 /usr/include
                End of search list.
                clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)
                Target: armv7l-unknown-linux-gnueabihf
                Thread model: posix
                InstalledDir: /usr/bin
                 "/usr/lib/llvm-3.9/bin/clang" -cc1 -triple armv7-unknown-linux-gnueabihf -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name distccd_ff4c663b.i -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -target-cpu cortex-a8 -target-abi aapcs-linux -mfloat-abi hard -v -dwarf-column-info -debugger-tuning=gdb -coverage-file /tmp/distccd_fcec663b.o -resource-dir /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1 -fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length 0 -fallow-half-arguments-and-returns -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -o /tmp/distccd_fcec663b.o -x cpp-output /tmp/distccd_ff4c663b.i
                clang -cc1 version 3.9.1 based upon LLVM 3.9.1 default target x86_64-pc-linux-gnu
                #include "..." search starts here:
                End of search list.
                root@bela:~/Bela/resources/network# 

                Here's the distcclog from the Ubuntu machine:

                distccd[14752] (dcc_check_client) connection from 192.168.7.2:59770
                distccd[14752] (check_address_inet) match client 0x207a8c0, value 0x207a8c0, mask 0xffffffff
                distccd[14752] (dcc_r_token_int) got DIST00000001
                distccd[14752] (dcc_r_token_int) got ARGC00000006
                distccd[14752] (dcc_r_argv) reading 6 arguments from job submission
                distccd[14752] (dcc_r_token_int) got ARGV0000000d
                distccd[14752] (dcc_r_token_string) got 'clang-3.9-arm'
                distccd[14752] (dcc_r_argv) argv[0] = "clang-3.9-arm"
                distccd[14752] (dcc_r_token_int) got ARGV00000002
                distccd[14752] (dcc_r_token_string) got '-v'
                distccd[14752] (dcc_r_argv) argv[1] = "-v"
                distccd[14752] (dcc_r_token_int) got ARGV00000002
                distccd[14752] (dcc_r_token_string) got '-c'
                distccd[14752] (dcc_r_argv) argv[2] = "-c"
                distccd[14752] (dcc_r_token_int) got ARGV0000000c
                distccd[14752] (dcc_r_token_string) got 'udp-client.c'
                distccd[14752] (dcc_r_argv) argv[3] = "udp-client.c"
                distccd[14752] (dcc_r_token_int) got ARGV00000002
                distccd[14752] (dcc_r_token_string) got '-o'
                distccd[14752] (dcc_r_argv) argv[4] = "-o"
                distccd[14752] (dcc_r_token_int) got ARGV0000000c
                distccd[14752] (dcc_r_token_string) got 'udp-client.o'
                distccd[14752] (dcc_r_argv) argv[5] = "udp-client.o"
                distccd[14752] (dcc_r_argv) got arguments: clang-3.9-arm -v -c udp-client.c -o udp-client.o
                distccd[14752] (dcc_scan_args) scanning arguments: clang-3.9-arm -v -c udp-client.c -o udp-client.o
                distccd[14752] (dcc_scan_args) found input file "udp-client.c"
                distccd[14752] (dcc_scan_args) found object/output file "udp-client.o"
                distccd[14752] compile from udp-client.c to udp-client.o
                distccd[14752] (dcc_run_job) output file udp-client.o
                distccd[14752] (dcc_input_tmpnam) input file udp-client.c
                distccd[14752] (dcc_r_token_int) got DOTI00016dac
                distccd[14752] (dcc_r_file) received 93612 bytes to file /tmp/distccd_ba396596.i
                distccd[14752] (dcc_r_file_timed) 93612 bytes received in 0.011009s, rate 8304kB/s
                distccd[14752] (dcc_set_input) changed input from "udp-client.c" to "/tmp/distccd_ba396596.i"
                distccd[14752] (dcc_set_input) command after: clang-3.9-arm -v -c /tmp/distccd_ba396596.i -o udp-client.o
                distccd[14752] (dcc_set_output) changed output from "udp-client.o" to "/tmp/distccd_ba666596.o"
                distccd[14752] (dcc_set_output) command after: clang-3.9-arm -v -c /tmp/distccd_ba396596.i -o /tmp/distccd_ba666596.o
                distccd[14752] (dcc_check_compiler_masq) /usr/local/bin/clang-3.9-arm is not a symlink
                distccd[14752] (dcc_spawn_child) forking to execute: clang-3.9-arm -v -c /tmp/distccd_ba396596.i -o /tmp/distccd_ba666596.o
                distccd[14752] (dcc_spawn_child) child started as pid32524
                distccd[32524] (dcc_new_pgrp) entered process group
                distccd[32524] (dcc_increment_safeguard) setting safeguard: _DISTCC_SAFEGUARD=1
                distccd[14752] (dcc_collect_child) cc child 32524 terminated with status 0
                distccd[14752] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
                distccd[14752] (dcc_x_token_int) send DONE00000001
                distccd[14752] (dcc_x_token_int) send STAT00000000
                distccd[14752] (dcc_x_file) send 1002 byte file /tmp/distcc_b8066596.stderr with token SERR and compression 69
                distccd[14752] (dcc_x_token_int) send SERR000003ea
                distccd[14752] (dcc_x_file) send 0 byte file /tmp/distcc_b8246596.stdout with token SOUT and compression 69
                distccd[14752] (dcc_x_token_int) send SOUT00000000
                distccd[14752] (dcc_x_file) send 2192 byte file /tmp/distccd_ba666596.o with token DOTO and compression 69
                distccd[14752] (dcc_x_token_int) send DOTO00000890
                distccd[14752] clang-3.9-arm udp-client.c on localhost completed ok
                distccd[14752] job complete
                distccd[14752] (dcc_cleanup_tempfiles_inner) deleted 5 temporary files
                distccd[14752] (dcc_job_summary) client: 192.168.7.2:59770 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:318ms clang-3.9-arm udp-client.c
                distccd[14753] (dcc_check_client) connection from 192.168.7.2:59772
                distccd[14753] (check_address_inet) match client 0x207a8c0, value 0x207a8c0, mask 0xffffffff
                distccd[14753] (dcc_r_token_int) got DIST00000001
                distccd[14753] (dcc_r_token_int) got ARGC00000006
                distccd[14753] (dcc_r_argv) reading 6 arguments from job submission
                distccd[14753] (dcc_r_token_int) got ARGV0000000d
                distccd[14753] (dcc_r_token_string) got 'clang-3.9-arm'
                distccd[14753] (dcc_r_argv) argv[0] = "clang-3.9-arm"
                distccd[14753] (dcc_r_token_int) got ARGV00000002
                distccd[14753] (dcc_r_token_string) got '-c'
                distccd[14753] (dcc_r_argv) argv[1] = "-c"
                distccd[14753] (dcc_r_token_int) got ARGV00000029
                distccd[14753] (dcc_r_token_string) got '/root/Bela/resources/network/udp-client.c'
                distccd[14753] (dcc_r_argv) argv[2] = "/root/Bela/resources/network/udp-client.c"
                distccd[14753] (dcc_r_token_int) got ARGV0000000e
                distccd[14753] (dcc_r_token_string) got '-march=armv7-a'
                distccd[14753] (dcc_r_argv) argv[3] = "-march=armv7-a"
                distccd[14753] (dcc_r_token_int) got ARGV00000002
                distccd[14753] (dcc_r_token_string) got '-o'
                distccd[14753] (dcc_r_argv) argv[4] = "-o"
                distccd[14753] (dcc_r_token_int) got ARGV0000000c
                distccd[14753] (dcc_r_token_string) got 'udp-client.o'
                distccd[14753] (dcc_r_argv) argv[5] = "udp-client.o"
                distccd[14753] (dcc_r_argv) got arguments: clang-3.9-arm -c /root/Bela/resources/network/udp-client.c -march=armv7-a -o udp-client.o
                distccd[14753] (dcc_scan_args) scanning arguments: clang-3.9-arm -c /root/Bela/resources/network/udp-client.c -march=armv7-a -o udp-client.o
                distccd[14753] (dcc_scan_args) found input file "/root/Bela/resources/network/udp-client.c"
                distccd[14753] (dcc_scan_args) found object/output file "udp-client.o"
                distccd[14753] compile from udp-client.c to udp-client.o
                distccd[14753] (dcc_run_job) output file udp-client.o
                distccd[14753] (dcc_input_tmpnam) input file /root/Bela/resources/network/udp-client.c
                distccd[14753] (dcc_r_token_int) got DOTI00016eeb
                distccd[14753] (dcc_r_file) received 93931 bytes to file /tmp/distccd_20816628.i
                distccd[14753] (dcc_r_file_timed) 93931 bytes received in 0.012232s, rate 7499kB/s
                distccd[14753] (dcc_set_input) changed input from "/root/Bela/resources/network/udp-client.c" to "/tmp/distccd_20816628.i"
                distccd[14753] (dcc_set_input) command after: clang-3.9-arm -c /tmp/distccd_20816628.i -march=armv7-a -o udp-client.o
                distccd[14753] (dcc_set_output) changed output from "udp-client.o" to "/tmp/distccd_20dc6628.o"
                distccd[14753] (dcc_set_output) command after: clang-3.9-arm -c /tmp/distccd_20816628.i -march=armv7-a -o /tmp/distccd_20dc6628.o
                distccd[14753] (dcc_check_compiler_masq) /usr/local/bin/clang-3.9-arm is not a symlink
                distccd[14753] (dcc_spawn_child) forking to execute: clang-3.9-arm -c /tmp/distccd_20816628.i -march=armv7-a -o /tmp/distccd_20dc6628.o
                distccd[14753] (dcc_spawn_child) child started as pid32547
                distccd[32547] (dcc_new_pgrp) entered process group
                distccd[32547] (dcc_increment_safeguard) setting safeguard: _DISTCC_SAFEGUARD=1
                distccd[14753] (dcc_collect_child) cc child 32547 terminated with status 0
                distccd[14753] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
                distccd[14753] (dcc_x_token_int) send DONE00000001
                distccd[14753] (dcc_x_token_int) send STAT00000000
                distccd[14753] (dcc_x_file) send 0 byte file /tmp/distcc_27c16628.stderr with token SERR and compression 69
                distccd[14753] (dcc_x_token_int) send SERR00000000
                distccd[14753] (dcc_x_file) send 0 byte file /tmp/distcc_27996628.stdout with token SOUT and compression 69
                distccd[14753] (dcc_x_token_int) send SOUT00000000
                distccd[14753] (dcc_x_file) send 2192 byte file /tmp/distccd_20dc6628.o with token DOTO and compression 69
                distccd[14753] (dcc_x_token_int) send DOTO00000890
                distccd[14753] clang-3.9-arm /root/Bela/resources/network/udp-client.c on localhost completed ok
                distccd[14753] job complete
                distccd[14753] (dcc_cleanup_tempfiles_inner) deleted 5 temporary files
                distccd[14753] (dcc_job_summary) client: 192.168.7.2:59772 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:286ms clang-3.9-arm /root/Bela/resources/network/udp-client.c
                distccd[14754] (dcc_check_client) connection from 192.168.7.2:59774
                distccd[14754] (check_address_inet) match client 0x207a8c0, value 0x207a8c0, mask 0xffffffff
                distccd[14754] (dcc_r_token_int) got DIST00000001
                distccd[14754] (dcc_r_token_int) got ARGC00000007
                distccd[14754] (dcc_r_argv) reading 7 arguments from job submission
                distccd[14754] (dcc_r_token_int) got ARGV0000000d
                distccd[14754] (dcc_r_token_string) got 'clang-3.9-arm'
                distccd[14754] (dcc_r_argv) argv[0] = "clang-3.9-arm"
                distccd[14754] (dcc_r_token_int) got ARGV00000002
                distccd[14754] (dcc_r_token_string) got '-c'
                distccd[14754] (dcc_r_argv) argv[1] = "-c"
                distccd[14754] (dcc_r_token_int) got ARGV00000002
                distccd[14754] (dcc_r_token_string) got '-v'
                distccd[14754] (dcc_r_argv) argv[2] = "-v"
                distccd[14754] (dcc_r_token_int) got ARGV00000029
                distccd[14754] (dcc_r_token_string) got '/root/Bela/resources/network/udp-client.c'
                distccd[14754] (dcc_r_argv) argv[3] = "/root/Bela/resources/network/udp-client.c"
                distccd[14754] (dcc_r_token_int) got ARGV0000000e
                distccd[14754] (dcc_r_token_string) got '-march=armv7-a'
                distccd[14754] (dcc_r_argv) argv[4] = "-march=armv7-a"
                distccd[14754] (dcc_r_token_int) got ARGV00000002
                distccd[14754] (dcc_r_token_string) got '-o'
                distccd[14754] (dcc_r_argv) argv[5] = "-o"
                distccd[14754] (dcc_r_token_int) got ARGV0000000c
                distccd[14754] (dcc_r_token_string) got 'udp-client.o'
                distccd[14754] (dcc_r_argv) argv[6] = "udp-client.o"
                distccd[14754] (dcc_r_argv) got arguments: clang-3.9-arm -c -v /root/Bela/resources/network/udp-client.c -march=armv7-a -o udp-client.o
                distccd[14754] (dcc_scan_args) scanning arguments: clang-3.9-arm -c -v /root/Bela/resources/network/udp-client.c -march=armv7-a -o udp-client.o
                distccd[14754] (dcc_scan_args) found input file "/root/Bela/resources/network/udp-client.c"
                distccd[14754] (dcc_scan_args) found object/output file "udp-client.o"
                distccd[14754] compile from udp-client.c to udp-client.o
                distccd[14754] (dcc_run_job) output file udp-client.o
                distccd[14754] (dcc_input_tmpnam) input file /root/Bela/resources/network/udp-client.c
                distccd[14754] (dcc_r_token_int) got DOTI00016eeb
                distccd[14754] (dcc_r_file) received 93931 bytes to file /tmp/distccd_ff4c663b.i
                distccd[14754] (dcc_r_file_timed) 93931 bytes received in 0.011458s, rate 8006kB/s
                distccd[14754] (dcc_set_input) changed input from "/root/Bela/resources/network/udp-client.c" to "/tmp/distccd_ff4c663b.i"
                distccd[14754] (dcc_set_input) command after: clang-3.9-arm -c -v /tmp/distccd_ff4c663b.i -march=armv7-a -o udp-client.o
                distccd[14754] (dcc_set_output) changed output from "udp-client.o" to "/tmp/distccd_fcec663b.o"
                distccd[14754] (dcc_set_output) command after: clang-3.9-arm -c -v /tmp/distccd_ff4c663b.i -march=armv7-a -o /tmp/distccd_fcec663b.o
                distccd[14754] (dcc_check_compiler_masq) /usr/local/bin/clang-3.9-arm is not a symlink
                distccd[14754] (dcc_spawn_child) forking to execute: clang-3.9-arm -c -v /tmp/distccd_ff4c663b.i -march=armv7-a -o /tmp/distccd_fcec663b.o
                distccd[14754] (dcc_spawn_child) child started as pid32551
                distccd[32551] (dcc_new_pgrp) entered process group
                distccd[32551] (dcc_increment_safeguard) setting safeguard: _DISTCC_SAFEGUARD=1
                distccd[14754] (dcc_collect_child) cc child 32551 terminated with status 0
                distccd[14754] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
                distccd[14754] (dcc_x_token_int) send DONE00000001
                distccd[14754] (dcc_x_token_int) send STAT00000000
                distccd[14754] (dcc_x_file) send 1002 byte file /tmp/distcc_f203663b.stderr with token SERR and compression 69
                distccd[14754] (dcc_x_token_int) send SERR000003ea
                distccd[14754] (dcc_x_file) send 0 byte file /tmp/distcc_f22c663b.stdout with token SOUT and compression 69
                distccd[14754] (dcc_x_token_int) send SOUT00000000
                distccd[14754] (dcc_x_file) send 2192 byte file /tmp/distccd_fcec663b.o with token DOTO and compression 69
                distccd[14754] (dcc_x_token_int) send DOTO00000890
                distccd[14754] clang-3.9-arm /root/Bela/resources/network/udp-client.c on localhost completed ok
                distccd[14754] job complete
                distccd[14754] (dcc_cleanup_tempfiles_inner) deleted 5 temporary files
                distccd[14754] (dcc_job_summary) client: 192.168.7.2:59774 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:320ms clang-3.9-arm /root/Bela/resources/network/udp-client.c

                Now trying "faust2bela" from the Ubuntu machine. /usr/Bela/Makefile still references distcc.

                gary@audio-workstation:~/faust/examples/bela$ faust2bela -tobela -gui FXChaine2.dsp 
                Send to bela
                /home/gary/faust/examples/bela/FXChaine2
                Start communication with bela
                /home/gary/faust/examples/bela/FXChaine2
                FXChaine2
                Checking the board is up and running at root@192.168.7.2...done
                Stop running process...
                Run bela now
                uploadBuildRun START
                Copying new source files to BeagleBone...Using rsync...
                Automatically detected PROJECT_TYPE: cpp 
                Running on __stretch__ with Xenomai __3__
                systemctl stop bela_startup || true
                PID=`grep bela-audio  /proc/xenomai/sched/stat | cut -d " " -f 5 | sed s/\s//g`; if [ -z $PID ]; then [ true = true ] || echo "No process to kill"; else [  true = true  ] || echo "Killing old Bela process $PID"; kill -2 $PID; sleep 0.2; kill -9 $PID 2> /dev/null; fi; screen -X -S Bela quit > /dev/null; exit 0;
                killall scsynth 2>/dev/null& killall sclang 2>/dev/null& true
                echo 'Building render.cpp...'
                Building render.cpp...
                distcc-clang++  -I/root/Bela/projects/FXChaine2 -I./include -I./build/pru/ -I/usr/xenomai/include/cobalt -I/usr/xenomai/include -march=armv7-a -mfpu=vfp3 -D_GNU_SOURCE -D_REENTRANT -fasynchronous-unwind-tables -D__COBALT__ -D__COBALT_WRAP__ -DXENOMAI_SKIN_posix -DXENOMAI_MAJOR=3 -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -DNDEBUG -DBELA_USE_RTDM -I/root/Bela/resources/stretch/include -std=c++11 -Wno-varargs -DNDEBUG  -Wall -c -fmessage-length=0 -U_FORTIFY_SOURCE -MMD -MP -MF"/root/Bela/projects/FXChaine2/build/render.d" -o "/root/Bela/projects/FXChaine2/build/render.o" "/root/Bela/projects/FXChaine2/render.cpp" 
                distcc[5035] ERROR: compile /root/Bela/projects/FXChaine2/render.cpp on 192.168.7.1 failed
                distcc[5035] (dcc_build_somewhere) Warning: remote compilation of '/root/Bela/projects/FXChaine2/render.cpp' failed, retrying locally
                distcc[5035] (dcc_build_somewhere) Warning: failed to distribute and fallbacks are disabled
                /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
                Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
                clang: error: linker command failed with exit code 1 (use -v to see invocation)
                Makefile:434: recipe for target '/root/Bela/projects/FXChaine2/build/render.o' failed
                make: *** [/root/Bela/projects/FXChaine2/build/render.o] Error 1
                Connection to 192.168.7.2 closed.
                gary@audio-workstation:~/faust/examples/bela$ 

                This still seems to be the failure:

                /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
                Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe

                This link takes us to some hints on that error message, which seem to culminate in "install the toolchain". The question is in the context of Android NDK though so I don't know if it applies here.

                https://stackoverflow.com/questions/52646860/ld-unrecognised-emulation-mode-armelf-linux-eabi-when-cross-compiling-with-a

                Here's the distcc log from this step:

                distccd[14755] (dcc_check_client) connection from 192.168.7.2:59776
                distccd[14755] (check_address_inet) match client 0x207a8c0, value 0x207a8c0, mask 0xffffffff
                distccd[14755] (dcc_r_token_int) got DIST00000001
                distccd[14755] (dcc_r_token_int) got ARGC00000013
                distccd[14755] (dcc_r_argv) reading 19 arguments from job submission
                distccd[14755] (dcc_r_token_int) got ARGV0000000f
                distccd[14755] (dcc_r_token_string) got 'clang++-3.9-arm'
                distccd[14755] (dcc_r_argv) argv[0] = "clang++-3.9-arm"
                distccd[14755] (dcc_r_token_int) got ARGV0000000e
                distccd[14755] (dcc_r_token_string) got '-march=armv7-a'
                distccd[14755] (dcc_r_argv) argv[1] = "-march=armv7-a"
                distccd[14755] (dcc_r_token_int) got ARGV0000000a
                distccd[14755] (dcc_r_token_string) got '-mfpu=vfp3'
                distccd[14755] (dcc_r_argv) argv[2] = "-mfpu=vfp3"
                distccd[14755] (dcc_r_token_int) got ARGV0000001c
                distccd[14755] (dcc_r_token_string) got '-fasynchronous-unwind-tables'
                distccd[14755] (dcc_r_argv) argv[3] = "-fasynchronous-unwind-tables"
                distccd[14755] (dcc_r_token_int) got ARGV00000003
                distccd[14755] (dcc_r_token_string) got '-O3'
                distccd[14755] (dcc_r_argv) argv[4] = "-O3"
                distccd[14755] (dcc_r_token_int) got ARGV0000000e
                distccd[14755] (dcc_r_token_string) got '-march=armv7-a'
                distccd[14755] (dcc_r_argv) argv[5] = "-march=armv7-a"
                distccd[14755] (dcc_r_token_int) got ARGV00000010
                distccd[14755] (dcc_r_token_string) got '-mtune=cortex-a8'
                distccd[14755] (dcc_r_argv) argv[6] = "-mtune=cortex-a8"
                distccd[14755] (dcc_r_token_int) got ARGV00000010
                distccd[14755] (dcc_r_token_string) got '-mfloat-abi=hard'
                distccd[14755] (dcc_r_argv) argv[7] = "-mfloat-abi=hard"
                distccd[14755] (dcc_r_token_int) got ARGV0000000a
                distccd[14755] (dcc_r_token_string) got '-mfpu=neon'
                distccd[14755] (dcc_r_argv) argv[8] = "-mfpu=neon"
                distccd[14755] (dcc_r_token_int) got ARGV00000010
                distccd[14755] (dcc_r_token_string) got '-ftree-vectorize'
                distccd[14755] (dcc_r_argv) argv[9] = "-ftree-vectorize"
                distccd[14755] (dcc_r_token_int) got ARGV0000000b
                distccd[14755] (dcc_r_token_string) got '-ffast-math'
                distccd[14755] (dcc_r_argv) argv[10] = "-ffast-math"
                distccd[14755] (dcc_r_token_int) got ARGV0000000a
                distccd[14755] (dcc_r_token_string) got '-std=c++11'
                distccd[14755] (dcc_r_argv) argv[11] = "-std=c++11"
                distccd[14755] (dcc_r_token_int) got ARGV0000000c
                distccd[14755] (dcc_r_token_string) got '-Wno-varargs'
                distccd[14755] (dcc_r_argv) argv[12] = "-Wno-varargs"
                distccd[14755] (dcc_r_token_int) got ARGV00000005
                distccd[14755] (dcc_r_token_string) got '-Wall'
                distccd[14755] (dcc_r_argv) argv[13] = "-Wall"
                distccd[14755] (dcc_r_token_int) got ARGV00000002
                distccd[14755] (dcc_r_token_string) got '-c'
                distccd[14755] (dcc_r_argv) argv[14] = "-c"
                distccd[14755] (dcc_r_token_int) got ARGV00000012
                distccd[14755] (dcc_r_token_string) got '-fmessage-length=0'
                distccd[14755] (dcc_r_argv) argv[15] = "-fmessage-length=0"
                distccd[14755] (dcc_r_token_int) got ARGV00000002
                distccd[14755] (dcc_r_token_string) got '-o'
                distccd[14755] (dcc_r_argv) argv[16] = "-o"
                distccd[14755] (dcc_r_token_int) got ARGV0000002c
                distccd[14755] (dcc_r_token_string) got '/root/Bela/projects/FXChaine2/build/render.o'
                distccd[14755] (dcc_r_argv) argv[17] = "/root/Bela/projects/FXChaine2/build/render.o"
                distccd[14755] (dcc_r_token_int) got ARGV00000028
                distccd[14755] (dcc_r_token_string) got '/root/Bela/projects/FXChaine2/render.cpp'
                distccd[14755] (dcc_r_argv) argv[18] = "/root/Bela/projects/FXChaine2/render.cpp"
                distccd[14755] (dcc_r_argv) got arguments: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /root/Bela/projects/FXChaine2/build/render.o /root/Bela/projects/FXChaine2/render.cpp
                distccd[14755] (dcc_scan_args) scanning arguments: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /root/Bela/projects/FXChaine2/build/render.o /root/Bela/projects/FXChaine2/render.cpp
                distccd[14755] (dcc_scan_args) found object/output file "/root/Bela/projects/FXChaine2/build/render.o"
                distccd[14755] (dcc_scan_args) found input file "/root/Bela/projects/FXChaine2/render.cpp"
                distccd[14755] compile from render.cpp to render.o
                distccd[14755] (dcc_run_job) output file /root/Bela/projects/FXChaine2/build/render.o
                distccd[14755] (dcc_input_tmpnam) input file /root/Bela/projects/FXChaine2/render.cpp
                distccd[14755] (dcc_r_token_int) got DOTI0019610a
                distccd[14755] (dcc_r_file) received 1663242 bytes to file /tmp/distccd_261c6794.ii
                distccd[14755] (dcc_r_file_timed) 1663242 bytes received in 0.150477s, rate 10794kB/s
                distccd[14755] (dcc_set_input) changed input from "/root/Bela/projects/FXChaine2/render.cpp" to "/tmp/distccd_261c6794.ii"
                distccd[14755] (dcc_set_input) command after: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /root/Bela/projects/FXChaine2/build/render.o /tmp/distccd_261c6794.ii
                distccd[14755] (dcc_set_output) changed output from "/root/Bela/projects/FXChaine2/build/render.o" to "/tmp/distccd_27a96794.o"
                distccd[14755] (dcc_set_output) command after: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /tmp/distccd_27a96794.o /tmp/distccd_261c6794.ii
                distccd[14755] (dcc_check_compiler_masq) /usr/local/bin/clang++-3.9-arm is not a symlink
                distccd[14755] (dcc_spawn_child) forking to execute: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /tmp/distccd_27a96794.o /tmp/distccd_261c6794.ii
                distccd[14755] (dcc_spawn_child) child started as pid32659
                distccd[32659] (dcc_new_pgrp) entered process group
                distccd[32659] (dcc_increment_safeguard) setting safeguard: _DISTCC_SAFEGUARD=1
                distccd[14755] (dcc_collect_child) cc child 32659 terminated with status 0x100
                distccd[14755] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
                distccd[14755] (dcc_x_token_int) send DONE00000001
                distccd[14755] (dcc_x_token_int) send STAT00000100
                distccd[14755] (dcc_x_file) send 248 byte file /tmp/distcc_58bf6794.stderr with token SERR and compression 69
                distccd[14755] (dcc_x_token_int) send SERR000000f8
                distccd[14755] (dcc_x_file) send 0 byte file /tmp/distcc_5b7e6794.stdout with token SOUT and compression 69
                distccd[14755] (dcc_x_token_int) send SOUT00000000
                distccd[14755] (dcc_x_token_int) send DOTO00000000
                distccd[14755] clang++-3.9-arm /root/Bela/projects/FXChaine2/render.cpp on localhost failed
                distccd[14755] job complete
                distccd[14755] (dcc_cleanup_tempfiles_inner) deleted 5 temporary files
                distccd[14755] (dcc_job_summary) client: 192.168.7.2:59776 COMPILE_ERROR exit:1 sig:0 core:0 ret:0 time:1109ms clang++-3.9-arm /root/Bela/projects/FXChaine2/render.cpp

                  This is very confusing to me. The error message you get seems to be about the linker, ld, and it seems to come from the host, because of the list of supported emulations. The weird thing about this is that distcc should never attempt to run the linker on the host (only the compiler), and also that the line that fails really has nothing to do with the linker.

                  So this is the line that is run on the host and fails:

                  Digital-Larry distccd[14755] (dcc_spawn_child) forking to execute: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /tmp/distccd_27a96794.o /tmp/distccd_261c6794.ii

                  As you see, it tries to compile the pre-processed file /tmp/distccd_261c6794.ii into the object file /tmp/distccd_27a96794.o. The pre-processed file is generated by distcc on the client (Bela), where it runs cpp (the C-preprocessor) on the file to bring in all the includes and apply all the defines and resolve the macros, generating a stand-alone C++ file that has no dependency on any external files.

                  Let's troubleshoot this with incremental steps. Create a simple C++ file with no includes on the host:

                  • Create a simple C++ file with no includes on the HOST: create a file called test.ii:
                    int func()
                    {
                        return 0;
                    }
                    and then run the same command as above (except for the file paths at the end) :
                    clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii
                    if this is successful, make sure the generated file test.o is indeed an ARM binary:
                    $ file test.o
                    test.o: ELF 32-bit LSB relocatable, ARM, version 1 (SYSV), not stripped
                  • if the above worked fine, then copy the test.ii file to the board and run ON THE BOARD:
                    distcc-clang++ -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii
                    and if this is successful, check that the generated file is indeed an ARM binary:
                    $ file test.o
                    test.o: ELF 32-bit LSB relocatable, ARM, version 1 (SYSV), not stripped
                  • if this is still good, then rename test.ii to test.cpp and run the same line again, this time with test.cpp at the end:
                    distcc-clang++ -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.cpp
                  • if this is still good, still on the board, do cd /root/Bela/ and there try to re-run the full line that was failing while building your project:
                    distcc-clang++  -I/root/Bela/projects/FXChaine2 -I./include -I./build/pru/ -I/usr/xenomai/include/cobalt -I/usr/xenomai/include -march=armv7-a -mfpu=vfp3 -D_GNU_SOURCE -D_REENTRANT -fasynchronous-unwind-tables -D__COBALT__ -D__COBALT_WRAP__ -DXENOMAI_SKIN_posix -DXENOMAI_MAJOR=3 -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -DNDEBUG -DBELA_USE_RTDM -I/root/Bela/resources/stretch/include -std=c++11 -Wno-varargs -DNDEBUG  -Wall -c -fmessage-length=0 -U_FORTIFY_SOURCE -MMD -MP -MF"/root/Bela/projects/FXChaine2/build/render.d" -o "/root/Bela/projects/FXChaine2/build/render.o" "/root/Bela/projects/FXChaine2/render.cpp" 
                    does this successfully generated the file /root/Bela/projects/FXChaine2/build/render.o? And is it an ARM binary?

                    giuliomoro first step fails.

                    gary@audio-workstation:~/test$ clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii
                    /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
                    Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
                    clang: error: linker command failed with exit code 1 (use -v to see invocation)
                    
                    gary@audio-workstation:~/test$ cat test.ii
                    int func()
                    {
                        return 0;
                    }
                    gary@audio-workstation:~/test$ 
                    
                    gary@audio-workstation:~/test$ which clang++-3.9-arm 
                    /usr/local/bin/clang++-3.9-arm
                    
                    gary@audio-workstation:~/test$ cat /usr/local/bin/clang++-3.9-arm 
                    #!/bin/bash
                    /usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $0
                    
                    gary@audio-workstation:~/test$ clang++-3.9-arm 
                    /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
                    Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
                    clang: error: linker command failed with exit code 1 (use -v to see invocation)
                    gary@audio-workstation:~/test$ 

                    Calling that compiler throws the error with no parameters given at all! [edit - it's calling the script which does add parameters] So maybe that is broken somehow. I used apt to install it.

                    gary@audio-workstation:~/test$ sudo apt list | grep ^clang-3.9
                    
                    WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
                    
                    clang-3.9/bionic,now 1:3.9.1-19ubuntu1 amd64 [installed]
                    clang-3.9-doc/bionic,bionic 1:3.9.1-19ubuntu1 all
                    clang-3.9-examples/bionic 1:3.9.1-19ubuntu1 amd64
                    gary@audio-workstation:~/test$ 
                    
                    gary@audio-workstation:~/test$ sudo dpkg -s clang-3.9
                    Package: clang-3.9
                    Status: install ok installed
                    Priority: optional
                    Section: devel
                    Installed-Size: 140661
                    Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
                    Architecture: amd64
                    Source: llvm-toolchain-3.9
                    Version: 1:3.9.1-19ubuntu1
                    Replaces: clang-3.1, clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1), clang-include-fixer-3.9, compiler-rt
                    Provides: c++-compiler, c-compiler, objc-compiler
                    Depends: libc6 (>= 2.14), libclang1-3.9 (= 1:3.9.1-19ubuntu1), libgcc1 (>= 1:3.4), libjsoncpp1 (>= 1.7.4), libllvm3.9 (>= 1:3.9.1-6~), libstdc++6 (>= 5.2), libstdc++-7-dev, libgcc-7-dev, libobjc-7-dev, libclang-common-3.9-dev (= 1:3.9.1-19ubuntu1), libc6-dev, binutils
                    Recommends: llvm-3.9-dev, python
                    Suggests: gnustep, gnustep-devel, clang-3.9-doc
                    Breaks: clang-3.1, clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1), clang-include-fixer-3.9, compiler-rt
                    Description: C, C++ and Objective-C compiler (LLVM based)
                     Clang project is a C, C++, Objective C and Objective C++ front-end
                     for the LLVM compiler. Its goal is to offer a replacement to the GNU Compiler
                     Collection (GCC).
                     .
                     Clang fully implements all published ISO C++ standards including C++11, as
                     well as the upcoming C++14 standard, and some parts of the fledgling C++1z
                     standard, and is considered a production-quality C++ compiler.
                    Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
                    Homepage: http://www.llvm.org/
                    gary@audio-workstation:~/test$ 

                    Some other random tests to see what exactly we are calling and the simplest command to generate the error.

                    gary@audio-workstation:~/test$ which clang++-3.9-arm 
                    /usr/local/bin/clang++-3.9-arm
                    gary@audio-workstation:~/test$ cat /usr/local/bin/clang++-3.9-arm 
                    #!/bin/bash
                    /usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $0
                    gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 
                    clang: error: no input files
                    gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf
                    clang: error: no input files
                    gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf ./test.ii
                    /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
                    Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
                    clang: error: linker command failed with exit code 1 (use -v to see invocation)
                    gary@audio-workstation:~/test$ /usr/bin/clang++-6.0 -target armv7l-unknown-linux-gnueabihf ./test.ii
                    /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
                    Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
                    clang: error: linker command failed with exit code 1 (use -v to see invocation)
                    gary@audio-workstation:~/test$ /usr/bin/clang++ -target armv7l-unknown-linux-gnueabihf ./test.ii
                    /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
                    Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
                    clang: error: linker command failed with exit code 1 (use -v to see invocation)
                    gary@audio-workstation:~/test$ 

                      right, what about ls -l /usr/bin/clang++-3.9 ? is it a symlink to something?
                      Further to this, you could try running

                      clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii

                      and remove one option at a time until it succeeds (or better: start with just clang++-3.9-arm -march=armv7-a ./test.o ./test.ii, is it successful? Then add more and more options until it fails .

                      I am wondering whether maybe your clang-3.9 does not support hard-float, in which case the failing option should be -mfloat-abi=hard. Can you dpkg -l | grep clang and perhaps then dpkg --info <names of the 3.9 packages from the previous result>?