This is very confusing to me. The error message you get seems to be about the linker, ld, and it seems to come from the host, because of the list of supported emulations. The weird thing about this is that distcc should never attempt to run the linker on the host (only the compiler), and also that the line that fails really has nothing to do with the linker.

So this is the line that is run on the host and fails:

Digital-Larry distccd[14755] (dcc_spawn_child) forking to execute: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /tmp/distccd_27a96794.o /tmp/distccd_261c6794.ii

As you see, it tries to compile the pre-processed file /tmp/distccd_261c6794.ii into the object file /tmp/distccd_27a96794.o. The pre-processed file is generated by distcc on the client (Bela), where it runs cpp (the C-preprocessor) on the file to bring in all the includes and apply all the defines and resolve the macros, generating a stand-alone C++ file that has no dependency on any external files.

Let's troubleshoot this with incremental steps. Create a simple C++ file with no includes on the host:

  • Create a simple C++ file with no includes on the HOST: create a file called test.ii: 
    int func()
{
    return 0;
}
    and then run the same command as above (except for the file paths at the end) : 
    clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii
    if this is successful, make sure the generated file test.o is indeed an ARM binary: 
    $ file test.o
test.o: ELF 32-bit LSB relocatable, ARM, version 1 (SYSV), not stripped
  • if the above worked fine, then copy the test.ii file to the board and run ON THE BOARD: 
    distcc-clang++ -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii
    and if this is successful, check that the generated file is indeed an ARM binary: 
    $ file test.o
test.o: ELF 32-bit LSB relocatable, ARM, version 1 (SYSV), not stripped
  • if this is still good, then rename test.ii to test.cpp and run the same line again, this time with test.cpp at the end: 
    distcc-clang++ -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.cpp
  • if this is still good, still on the board, do cd /root/Bela/ and there try to re-run the full line that was failing while building your project: 
    distcc-clang++  -I/root/Bela/projects/FXChaine2 -I./include -I./build/pru/ -I/usr/xenomai/include/cobalt -I/usr/xenomai/include -march=armv7-a -mfpu=vfp3 -D_GNU_SOURCE -D_REENTRANT -fasynchronous-unwind-tables -D__COBALT__ -D__COBALT_WRAP__ -DXENOMAI_SKIN_posix -DXENOMAI_MAJOR=3 -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -DNDEBUG -DBELA_USE_RTDM -I/root/Bela/resources/stretch/include -std=c++11 -Wno-varargs -DNDEBUG  -Wall -c -fmessage-length=0 -U_FORTIFY_SOURCE -MMD -MP -MF"/root/Bela/projects/FXChaine2/build/render.d" -o "/root/Bela/projects/FXChaine2/build/render.o" "/root/Bela/projects/FXChaine2/render.cpp"
    does this successfully generated the file /root/Bela/projects/FXChaine2/build/render.o? And is it an ARM binary?

      giuliomoro first step fails.

      gary@audio-workstation:~/test$ clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)

gary@audio-workstation:~/test$ cat test.ii
int func()
{
    return 0;
}
gary@audio-workstation:~/test$ 

gary@audio-workstation:~/test$ which clang++-3.9-arm 
/usr/local/bin/clang++-3.9-arm

gary@audio-workstation:~/test$ cat /usr/local/bin/clang++-3.9-arm 
#!/bin/bash
/usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $0

gary@audio-workstation:~/test$ clang++-3.9-arm 
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$

      Calling that compiler throws the error with no parameters given at all! [edit - it's calling the script which does add parameters] So maybe that is broken somehow. I used apt to install it.

      gary@audio-workstation:~/test$ sudo apt list | grep ^clang-3.9

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

clang-3.9/bionic,now 1:3.9.1-19ubuntu1 amd64 [installed]
clang-3.9-doc/bionic,bionic 1:3.9.1-19ubuntu1 all
clang-3.9-examples/bionic 1:3.9.1-19ubuntu1 amd64
gary@audio-workstation:~/test$ 

gary@audio-workstation:~/test$ sudo dpkg -s clang-3.9
Package: clang-3.9
Status: install ok installed
Priority: optional
Section: devel
Installed-Size: 140661
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Source: llvm-toolchain-3.9
Version: 1:3.9.1-19ubuntu1
Replaces: clang-3.1, clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1), clang-include-fixer-3.9, compiler-rt
Provides: c++-compiler, c-compiler, objc-compiler
Depends: libc6 (>= 2.14), libclang1-3.9 (= 1:3.9.1-19ubuntu1), libgcc1 (>= 1:3.4), libjsoncpp1 (>= 1.7.4), libllvm3.9 (>= 1:3.9.1-6~), libstdc++6 (>= 5.2), libstdc++-7-dev, libgcc-7-dev, libobjc-7-dev, libclang-common-3.9-dev (= 1:3.9.1-19ubuntu1), libc6-dev, binutils
Recommends: llvm-3.9-dev, python
Suggests: gnustep, gnustep-devel, clang-3.9-doc
Breaks: clang-3.1, clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1), clang-include-fixer-3.9, compiler-rt
Description: C, C++ and Objective-C compiler (LLVM based)
 Clang project is a C, C++, Objective C and Objective C++ front-end
 for the LLVM compiler. Its goal is to offer a replacement to the GNU Compiler
 Collection (GCC).
 .
 Clang fully implements all published ISO C++ standards including C++11, as
 well as the upcoming C++14 standard, and some parts of the fledgling C++1z
 standard, and is considered a production-quality C++ compiler.
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Homepage: http://www.llvm.org/
gary@audio-workstation:~/test$

      Some other random tests to see what exactly we are calling and the simplest command to generate the error.

      gary@audio-workstation:~/test$ which clang++-3.9-arm 
/usr/local/bin/clang++-3.9-arm
gary@audio-workstation:~/test$ cat /usr/local/bin/clang++-3.9-arm 
#!/bin/bash
/usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $0
gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 
clang: error: no input files
gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf
clang: error: no input files
gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$ /usr/bin/clang++-6.0 -target armv7l-unknown-linux-gnueabihf ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$ /usr/bin/clang++ -target armv7l-unknown-linux-gnueabihf ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$

          right, what about ls -l /usr/bin/clang++-3.9 ? is it a symlink to something?
          Further to this, you could try running

          clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii

          and remove one option at a time until it succeeds (or better: start with just clang++-3.9-arm -march=armv7-a ./test.o ./test.ii, is it successful? Then add more and more options until it fails .

          I am wondering whether maybe your clang-3.9 does not support hard-float, in which case the failing option should be -mfloat-abi=hard. Can you dpkg -l | grep clang and perhaps then dpkg --info <names of the 3.9 packages from the previous result>?

            I used the -### command with just clang++ (the 6.0 version installed by default, but the error is the same so far regardless of the compiler version used, 3.9 or 6.0):

            gary@audio-workstation:~/test$ /usr/bin/clang++ -### -target armv7l-unknown-linux-gnueabihf ./test.ii
clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
Target: armv7l-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
 "/usr/lib/llvm-6.0/bin/clang" "-cc1" "-triple" "armv7-unknown-linux-gnueabihf" "-emit-obj" "-mrelax-all" "-disable-free" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "test.ii" "-mrelocation-model" "static" "-mthread-model" "posix" "-mdisable-fp-elim" "-fmath-errno" "-masm-verbose" "-mconstructor-aliases" "-fuse-init-array" "-target-cpu" "generic" "-target-abi" "aapcs-linux" "-mfloat-abi" "hard" "-fallow-half-arguments-and-returns" "-dwarf-column-info" "-debugger-tuning=gdb" "-resource-dir" "/usr/lib/llvm-6.0/lib/clang/6.0.0" "-fdeprecated-macro" "-fdebug-compilation-dir" "/home/gary/test" "-ferror-limit" "19" "-fmessage-length" "80" "-fno-signed-char" "-fobjc-runtime=gcc" "-fcxx-exceptions" "-fexceptions" "-fdiagnostics-show-option" "-fcolor-diagnostics" "-o" "/tmp/test-dc97e4.o" "-x" "c++-cpp-output" "./test.ii"
 "/usr/bin/ld" "-z" "relro" "-X" "--hash-style=gnu" "--eh-frame-hdr" "-m" "armelf_linux_eabi" "-dynamic-linker" "/lib/ld-linux-armhf.so.3" "-o" "a.out" "crt1.o" "crti.o" "crtbegin.o" "-L/usr/lib/llvm-6.0/bin/../lib" "-L/lib/../lib" "-L/usr/lib/../lib" "-L/usr/lib/llvm-6.0/bin/../lib" "-L/lib" "-L/usr/lib" "/tmp/test-dc97e4.o" "-lstdc++" "-lm" "-lgcc_s" "-lgcc" "-lc" "-lgcc_s" "-lgcc" "crtend.o" "crtn.o"
gary@audio-workstation:~/test$

            giuliomoro yes there is a symlink there that appears to be from the installation (I didn't do it).

            gary@audio-workstation:~/test$ ls -l /usr/bin/clang++-3.9 
lrwxrwxrwx 1 root root 27 Dec 26  2017 /usr/bin/clang++-3.9 -> ../lib/llvm-3.9/bin/clang++
gary@audio-workstation:~/test$
              gary@audio-workstation:~/test$ clang++-3.9-arm -march=armv7-a ./test.o ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$

gary@audio-workstation:~/test$ dpkg -l | grep clang
ii  clang                                      1:6.0-41~exp5~ubuntu1                        amd64        C, C++ and Objective-C compiler (LLVM based)
ii  clang-3.9                                  1:3.9.1-19ubuntu1                            amd64        C, C++ and Objective-C compiler (LLVM based)
ii  clang-6.0                                  1:6.0-1ubuntu2                               amd64        C, C++ and Objective-C compiler
ii  libclang-common-3.9-dev                    1:3.9.1-19ubuntu1                            amd64        clang library - Common development package
ii  libclang-common-6.0-dev                    1:6.0-1ubuntu2                               amd64        clang library - Common development package
ii  libclang1-3.9:amd64                        1:3.9.1-19ubuntu1                            amd64        C interface to the clang library
ii  libclang1-6.0:amd64                        1:6.0-1ubuntu2                               amd64        C interface to the clang library
gary@audio-workstation:~/test$

              dpkg --info doesn't show anything but apt show does:

              gary@audio-workstation:~/test$ apt show clang-3.9
Package: clang-3.9
Version: 1:3.9.1-19ubuntu1
Priority: optional
Section: universe/devel
Source: llvm-toolchain-3.9
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 144 MB
Provides: c++-compiler, c-compiler, objc-compiler
Depends: libc6 (>= 2.14), libclang1-3.9 (= 1:3.9.1-19ubuntu1), libgcc1 (>= 1:3.4), libjsoncpp1 (>= 1.7.4), libllvm3.9 (>= 1:3.9.1-6~), libstdc++6 (>= 5.2), libstdc++-7-dev, libgcc-7-dev, libobjc-7-dev, libclang-common-3.9-dev (= 1:3.9.1-19ubuntu1), libc6-dev, binutils
Recommends: llvm-3.9-dev, python
Suggests: gnustep, gnustep-devel, clang-3.9-doc
Breaks: clang-3.1, clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1), clang-include-fixer-3.9, compiler-rt
Replaces: clang-3.1, clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1), clang-include-fixer-3.9, compiler-rt
Homepage: http://www.llvm.org/
Download-Size: 37.4 MB
APT-Manual-Installed: yes
APT-Sources: http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
Description: C, C++ and Objective-C compiler (LLVM based)
 Clang project is a C, C++, Objective C and Objective C++ front-end
 for the LLVM compiler. Its goal is to offer a replacement to the GNU Compiler
 Collection (GCC).
 .
 Clang fully implements all published ISO C++ standards including C++11, as
 well as the upcoming C++14 standard, and some parts of the fledgling C++1z
 standard, and is considered a production-quality C++ compiler.

gary@audio-workstation:~/test$ 

gary@audio-workstation:~/test$ apt show libclang-common-3.9-dev
Package: libclang-common-3.9-dev
Version: 1:3.9.1-19ubuntu1
Priority: optional
Section: universe/libdevel
Source: llvm-toolchain-3.9
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 39.1 MB
Depends: libc6 (>= 2.15), libgcc1 (>= 1:3.4), libstdc++6 (>= 5.2), libtinfo5 (>= 6), libllvm3.9 (= 1:3.9.1-19ubuntu1)
Homepage: http://www.llvm.org/
Download-Size: 2,575 kB
APT-Manual-Installed: no
APT-Sources: http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
Description: clang library - Common development package
 Clang project is a C, C++, Objective C and Objective C++ front-end
 for the LLVM compiler. Its goal is to offer a replacement to the GNU Compiler
 Collection (GCC).
 .
 Clang fully implements all published ISO C++ standards including C++11, as
 well as the upcoming C++14 standard, and some parts of the fledgling C++1z
 standard, and is considered a production-quality C++ compiler.
 .
 This package contains the clang generic headers and some libraries
 (profiling, etc).

gary@audio-workstation:~/test$ 

gary@audio-workstation:~/test$ apt show libclang1-3.9
Package: libclang1-3.9
Version: 1:3.9.1-19ubuntu1
Priority: optional
Section: universe/devel
Source: llvm-toolchain-3.9
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 22.9 MB
Depends: libc6 (>= 2.14), libgcc1 (>= 1:3.4), libllvm3.9 (>= 1:3.9.1-6~), libstdc++6 (>= 5.2)
Homepage: http://www.llvm.org/
Download-Size: 5,950 kB
APT-Manual-Installed: no
APT-Sources: http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
Description: C interface to the clang library
 Clang project is a C, C++, Objective C and Objective C++ front-end
 for the LLVM compiler. Its goal is to offer a replacement to the GNU Compiler
 Collection (GCC).
 .
 Clang fully implements all published ISO C++ standards including C++11, as
 well as the upcoming C++14 standard, and some parts of the fledgling C++1z
 standard, and is considered a production-quality C++ compiler.
 .
 This package contains the clang library.
 .
 The C Interface to Clang provides a relatively small API that exposes
 facilities for parsing source code into an abstract syntax tree (AST),
 loading already-parsed ASTs, traversing the AST, associating physical source
 locations with elements within the AST, and other facilities that support
 Clang-based development tools.

gary@audio-workstation:~/test$

              Digital-Larry gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf ./test.ii

              you nee -c otherwise it will attempt to use the linker.

              giuliomoro
              or better: start with just clang++-3.9-arm -march=armv7-a ./test.o ./test.ii, is it successful?

              I forgot -c, too, and also -o.

              so try:

              clang++-3.9-arm -march=armv7-a -c -o ./test.o ./test.ii

              if that succeeds, add one option at a time until it fails ..

                    giuliomoro

                    gary@audio-workstation:~/test$ clang++-3.9-arm -march=armv7-a -c -o ./test.o ./test.ii
                    /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
                    Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
                    clang: error: linker command failed with exit code 1 (use -v to see invocation)
                    gary@audio-workstation:~/test$

                      giuliomoro

                      gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 -###  -march=arm7-a -c -o ./test.o ./test.ii
clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
 "/usr/lib/llvm-3.9/bin/clang" "-cc1" "-triple" "x86_64-pc-linux-gnu" "-emit-obj" "-mrelax-all" "-disable-free" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "test.ii" "-mrelocation-model" "static" "-mthread-model" "posix" "-mdisable-fp-elim" "-fmath-errno" "-masm-verbose" "-mconstructor-aliases" "-munwind-tables" "-fuse-init-array" "-target-cpu" "arm7-a" "-dwarf-column-info" "-debugger-tuning=gdb" "-coverage-file" "/home/gary/test/./test.o" "-resource-dir" "/usr/lib/llvm-3.9/bin/../lib/clang/3.9.1" "-fdeprecated-macro" "-fdebug-compilation-dir" "/home/gary/test" "-ferror-limit" "19" "-fmessage-length" "80" "-fobjc-runtime=gcc" "-fcxx-exceptions" "-fexceptions" "-fdiagnostics-show-option" "-fcolor-diagnostics" "-o" "./test.o" "-x" "c++-cpp-output" "./test.ii"
gary@audio-workstation:~/test$

                      sorry, hang on, swimming in stuff here...

                      gary@audio-workstation:~/test$ clang++-3.9-arm -### -march=armv7-a -c -o ./test.o ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$

                        I added the "-###" right in the clang++-3.9-arm script itself (I also tried adding "-fno-autolink"):

                        gary@audio-workstation:~/test$ sh -x /usr/local/bin/clang++-3.9-arm -### -march=armv7-a -c -o ./test.o ./test.ii
+ /usr/bin/clang++-3.9 -### -target armv7l-unknown-linux-gnueabihf -fno-autolink --sysroot /home/gary/arm /usr/local/bin/clang++-3.9-arm
clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)
Target: armv7l-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
 "/usr/bin/ld" "--sysroot=/home/gary/arm" "-X" "--eh-frame-hdr" "-m" "armelf_linux_eabi" "-dynamic-linker" "/lib/ld-linux-armhf.so.3" "-o" "a.out" "crt1.o" "crti.o" "crtbegin.o" "/usr/local/bin/clang++-3.9-arm" "-lstdc++" "-lm" "-lgcc_s" "-lgcc" "-lc" "-lgcc_s" "-lgcc" "crtend.o" "crtn.o"
gary@audio-workstation:~/test$

                        OK after that last one I found it! It wasn't until I ran the script with "sh -x" that I saw the wrong command line substitution. The last thing on the command line should be the source file name, but it is the script file name!

                        /usr/bin/clang++-3.9 -### -target armv7l-unknown-linux-gnueabihf -fno-autolink --sysroot /home/gary/arm /usr/local/bin/clang++-3.9-arm

                        In this file, I had "$0" instead of "$@" - I changed it and it works!!!!

                        gary@audio-workstation:~/faust/examples/bela$ cat /usr/local/bin/clang++-3.9-arm 
#!/bin/bash
/usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@
gary@audio-workstation

                        Giuilo thanks so much for your patient help. Sorry it was so difficult but maybe it helps the next guy. At this point I think Faust on Bela is a very viable idea. Hopefully my next questions and comments will be more on the implementation side.

                        giuliomoro I think so. At least, I see it shown in some of the things I posted a day or two ago. Just very hard to see! One question I have which I will defer answering for now is whether clang 3.9 is really needed on the Ubuntu system or whether "clang" as installed (which by now is at 6.0) is OK. I understand that the distcc as being called by faust2bela may be making the specific reference to 3.9. Lots of different interacting parts here.

                          Linker problem

                          One issue of 3.9 vs 6.0 is that linking is always done on the board by calling clang++-3.9, and so if there is any incompatibility between the .o files generated by 6.0 and the way clang++-3.9 calls the ld linker, or the way the older ld linker on the board deals with object files, then you could have some issues. I guess this is unlikely to happen, but I am not sure what guarantees clang and ld give about interoperability of different versions.

                          Compiler problem

                          Another issue is that different versions of clang may decide to pack structs (or at least ARM intrinsics) in different ways, for instance:

                          giuliomoro What is surprising is that while alignof(float32x4_t) is 16 bytes on clang-3.9, it is actually 8 bytes on gcc-6.3 and clang-6.0.

                          and

                          giuliomoro However, I am wondering where that difference comes from, as the arm_neon.h files for the two versions of clang provide identical definition for all the neon vector types.

                          More in general, there would be a problem if you have object files that both include some .h file that is interpreted differently by the two versions of the compiler, and you compile the two objects with different versions of the compiler. Again, I am not sure what backward/forward compatibility guarantees clang provides. Clearly, if you ALWAYS use distcc to build all the files on Bela, there would be no problem. The problem only comes up (potentially) if you are intermixing .o files compiled with different versions of the compiler.
                          For instance - in you current situation - you built all the Bela core/ files with the local version of clang-3.9 the first time you built the first program. If you switch to using clang-6.0, I'd recommend you do a make -C ~/Bela clean on the board, so that those files are re-built next time you build a project.

                                giuliomoro ah well never mind, all that stuff gives me a headache! I'm OK leaving 3.9 installed. Thanks for the detailed explanation.