OK let's see where we are now.

root@bela:~/Bela# grep distcc Makefile 
  CLANG_PATH=distcc-clang
root@bela:~/Bela#

reference command line build on bela works (without specifying architecture):

root@bela:~/Bela/resources/network# distcc-clang -v -c udp-client.c
clang version 3.9.1-9 (tags/RELEASE_391/rc2)
Target: armv7l-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6
Found candidate GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0
Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/6
Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/6.3.0
Selected GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0
Candidate multilib: .;@m32
Selected multilib: .;@m32
 "/usr/lib/llvm-3.9/bin/clang" -cc1 -triple armv7-unknown-linux-gnueabihf -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name udp-client.c -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu cortex-a8 -target-abi aapcs-linux -mfloat-abi hard -v -dwarf-column-info -debugger-tuning=gdb -resource-dir /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1 -internal-isystem /usr/local/include -internal-isystem /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/include -internal-externc-isystem /usr/include/arm-linux-gnueabihf -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir /root/Bela/resources/network -ferror-limit 19 -fmessage-length 80 -fallow-half-arguments-and-returns -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -o - -x c udp-client.c
clang -cc1 version 3.9.1 based upon LLVM 3.9.1 default target armv7l-unknown-linux-gnueabihf
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/include
 /usr/include/arm-linux-gnueabihf
 /usr/include
End of search list.
clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)
Target: armv7l-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
 "/usr/lib/llvm-3.9/bin/clang" -cc1 -triple armv7-unknown-linux-gnueabihf -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name distccd_ba396596.i -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -target-cpu cortex-a8 -target-abi aapcs-linux -mfloat-abi hard -v -dwarf-column-info -debugger-tuning=gdb -coverage-file /tmp/distccd_ba666596.o -resource-dir /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1 -fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length 0 -fallow-half-arguments-and-returns -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -o /tmp/distccd_ba666596.o -x cpp-output /tmp/distccd_ba396596.i
clang -cc1 version 3.9.1 based upon LLVM 3.9.1 default target x86_64-pc-linux-gnu
#include "..." search starts here:
End of search list.
root@bela:~/Bela/resources/network#

reference build works with specifying architecture:

root@bela:~/Bela/resources/network# distcc-clang -c -v ~/Bela/resources/network/udp-client.c -march=armv7-a
clang version 3.9.1-9 (tags/RELEASE_391/rc2)
Target: armv7l-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6
Found candidate GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0
Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/6
Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/6.3.0
Selected GCC installation: /usr/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0
Candidate multilib: .;@m32
Selected multilib: .;@m32
 "/usr/lib/llvm-3.9/bin/clang" -cc1 -triple armv7-unknown-linux-gnueabihf -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name udp-client.c -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu cortex-a8 -target-abi aapcs-linux -mfloat-abi hard -v -dwarf-column-info -debugger-tuning=gdb -resource-dir /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1 -internal-isystem /usr/local/include -internal-isystem /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/include -internal-externc-isystem /usr/include/arm-linux-gnueabihf -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir /root/Bela/resources/network -ferror-limit 19 -fmessage-length 80 -fallow-half-arguments-and-returns -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -o - -x c /root/Bela/resources/network/udp-client.c
clang -cc1 version 3.9.1 based upon LLVM 3.9.1 default target armv7l-unknown-linux-gnueabihf
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/include
 /usr/include/arm-linux-gnueabihf
 /usr/include
End of search list.
clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)
Target: armv7l-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
 "/usr/lib/llvm-3.9/bin/clang" -cc1 -triple armv7-unknown-linux-gnueabihf -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name distccd_ff4c663b.i -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -target-cpu cortex-a8 -target-abi aapcs-linux -mfloat-abi hard -v -dwarf-column-info -debugger-tuning=gdb -coverage-file /tmp/distccd_fcec663b.o -resource-dir /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1 -fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length 0 -fallow-half-arguments-and-returns -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -o /tmp/distccd_fcec663b.o -x cpp-output /tmp/distccd_ff4c663b.i
clang -cc1 version 3.9.1 based upon LLVM 3.9.1 default target x86_64-pc-linux-gnu
#include "..." search starts here:
End of search list.
root@bela:~/Bela/resources/network#

Here's the distcclog from the Ubuntu machine:

distccd[14752] (dcc_check_client) connection from 192.168.7.2:59770
distccd[14752] (check_address_inet) match client 0x207a8c0, value 0x207a8c0, mask 0xffffffff
distccd[14752] (dcc_r_token_int) got DIST00000001
distccd[14752] (dcc_r_token_int) got ARGC00000006
distccd[14752] (dcc_r_argv) reading 6 arguments from job submission
distccd[14752] (dcc_r_token_int) got ARGV0000000d
distccd[14752] (dcc_r_token_string) got 'clang-3.9-arm'
distccd[14752] (dcc_r_argv) argv[0] = "clang-3.9-arm"
distccd[14752] (dcc_r_token_int) got ARGV00000002
distccd[14752] (dcc_r_token_string) got '-v'
distccd[14752] (dcc_r_argv) argv[1] = "-v"
distccd[14752] (dcc_r_token_int) got ARGV00000002
distccd[14752] (dcc_r_token_string) got '-c'
distccd[14752] (dcc_r_argv) argv[2] = "-c"
distccd[14752] (dcc_r_token_int) got ARGV0000000c
distccd[14752] (dcc_r_token_string) got 'udp-client.c'
distccd[14752] (dcc_r_argv) argv[3] = "udp-client.c"
distccd[14752] (dcc_r_token_int) got ARGV00000002
distccd[14752] (dcc_r_token_string) got '-o'
distccd[14752] (dcc_r_argv) argv[4] = "-o"
distccd[14752] (dcc_r_token_int) got ARGV0000000c
distccd[14752] (dcc_r_token_string) got 'udp-client.o'
distccd[14752] (dcc_r_argv) argv[5] = "udp-client.o"
distccd[14752] (dcc_r_argv) got arguments: clang-3.9-arm -v -c udp-client.c -o udp-client.o
distccd[14752] (dcc_scan_args) scanning arguments: clang-3.9-arm -v -c udp-client.c -o udp-client.o
distccd[14752] (dcc_scan_args) found input file "udp-client.c"
distccd[14752] (dcc_scan_args) found object/output file "udp-client.o"
distccd[14752] compile from udp-client.c to udp-client.o
distccd[14752] (dcc_run_job) output file udp-client.o
distccd[14752] (dcc_input_tmpnam) input file udp-client.c
distccd[14752] (dcc_r_token_int) got DOTI00016dac
distccd[14752] (dcc_r_file) received 93612 bytes to file /tmp/distccd_ba396596.i
distccd[14752] (dcc_r_file_timed) 93612 bytes received in 0.011009s, rate 8304kB/s
distccd[14752] (dcc_set_input) changed input from "udp-client.c" to "/tmp/distccd_ba396596.i"
distccd[14752] (dcc_set_input) command after: clang-3.9-arm -v -c /tmp/distccd_ba396596.i -o udp-client.o
distccd[14752] (dcc_set_output) changed output from "udp-client.o" to "/tmp/distccd_ba666596.o"
distccd[14752] (dcc_set_output) command after: clang-3.9-arm -v -c /tmp/distccd_ba396596.i -o /tmp/distccd_ba666596.o
distccd[14752] (dcc_check_compiler_masq) /usr/local/bin/clang-3.9-arm is not a symlink
distccd[14752] (dcc_spawn_child) forking to execute: clang-3.9-arm -v -c /tmp/distccd_ba396596.i -o /tmp/distccd_ba666596.o
distccd[14752] (dcc_spawn_child) child started as pid32524
distccd[32524] (dcc_new_pgrp) entered process group
distccd[32524] (dcc_increment_safeguard) setting safeguard: _DISTCC_SAFEGUARD=1
distccd[14752] (dcc_collect_child) cc child 32524 terminated with status 0
distccd[14752] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
distccd[14752] (dcc_x_token_int) send DONE00000001
distccd[14752] (dcc_x_token_int) send STAT00000000
distccd[14752] (dcc_x_file) send 1002 byte file /tmp/distcc_b8066596.stderr with token SERR and compression 69
distccd[14752] (dcc_x_token_int) send SERR000003ea
distccd[14752] (dcc_x_file) send 0 byte file /tmp/distcc_b8246596.stdout with token SOUT and compression 69
distccd[14752] (dcc_x_token_int) send SOUT00000000
distccd[14752] (dcc_x_file) send 2192 byte file /tmp/distccd_ba666596.o with token DOTO and compression 69
distccd[14752] (dcc_x_token_int) send DOTO00000890
distccd[14752] clang-3.9-arm udp-client.c on localhost completed ok
distccd[14752] job complete
distccd[14752] (dcc_cleanup_tempfiles_inner) deleted 5 temporary files
distccd[14752] (dcc_job_summary) client: 192.168.7.2:59770 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:318ms clang-3.9-arm udp-client.c
distccd[14753] (dcc_check_client) connection from 192.168.7.2:59772
distccd[14753] (check_address_inet) match client 0x207a8c0, value 0x207a8c0, mask 0xffffffff
distccd[14753] (dcc_r_token_int) got DIST00000001
distccd[14753] (dcc_r_token_int) got ARGC00000006
distccd[14753] (dcc_r_argv) reading 6 arguments from job submission
distccd[14753] (dcc_r_token_int) got ARGV0000000d
distccd[14753] (dcc_r_token_string) got 'clang-3.9-arm'
distccd[14753] (dcc_r_argv) argv[0] = "clang-3.9-arm"
distccd[14753] (dcc_r_token_int) got ARGV00000002
distccd[14753] (dcc_r_token_string) got '-c'
distccd[14753] (dcc_r_argv) argv[1] = "-c"
distccd[14753] (dcc_r_token_int) got ARGV00000029
distccd[14753] (dcc_r_token_string) got '/root/Bela/resources/network/udp-client.c'
distccd[14753] (dcc_r_argv) argv[2] = "/root/Bela/resources/network/udp-client.c"
distccd[14753] (dcc_r_token_int) got ARGV0000000e
distccd[14753] (dcc_r_token_string) got '-march=armv7-a'
distccd[14753] (dcc_r_argv) argv[3] = "-march=armv7-a"
distccd[14753] (dcc_r_token_int) got ARGV00000002
distccd[14753] (dcc_r_token_string) got '-o'
distccd[14753] (dcc_r_argv) argv[4] = "-o"
distccd[14753] (dcc_r_token_int) got ARGV0000000c
distccd[14753] (dcc_r_token_string) got 'udp-client.o'
distccd[14753] (dcc_r_argv) argv[5] = "udp-client.o"
distccd[14753] (dcc_r_argv) got arguments: clang-3.9-arm -c /root/Bela/resources/network/udp-client.c -march=armv7-a -o udp-client.o
distccd[14753] (dcc_scan_args) scanning arguments: clang-3.9-arm -c /root/Bela/resources/network/udp-client.c -march=armv7-a -o udp-client.o
distccd[14753] (dcc_scan_args) found input file "/root/Bela/resources/network/udp-client.c"
distccd[14753] (dcc_scan_args) found object/output file "udp-client.o"
distccd[14753] compile from udp-client.c to udp-client.o
distccd[14753] (dcc_run_job) output file udp-client.o
distccd[14753] (dcc_input_tmpnam) input file /root/Bela/resources/network/udp-client.c
distccd[14753] (dcc_r_token_int) got DOTI00016eeb
distccd[14753] (dcc_r_file) received 93931 bytes to file /tmp/distccd_20816628.i
distccd[14753] (dcc_r_file_timed) 93931 bytes received in 0.012232s, rate 7499kB/s
distccd[14753] (dcc_set_input) changed input from "/root/Bela/resources/network/udp-client.c" to "/tmp/distccd_20816628.i"
distccd[14753] (dcc_set_input) command after: clang-3.9-arm -c /tmp/distccd_20816628.i -march=armv7-a -o udp-client.o
distccd[14753] (dcc_set_output) changed output from "udp-client.o" to "/tmp/distccd_20dc6628.o"
distccd[14753] (dcc_set_output) command after: clang-3.9-arm -c /tmp/distccd_20816628.i -march=armv7-a -o /tmp/distccd_20dc6628.o
distccd[14753] (dcc_check_compiler_masq) /usr/local/bin/clang-3.9-arm is not a symlink
distccd[14753] (dcc_spawn_child) forking to execute: clang-3.9-arm -c /tmp/distccd_20816628.i -march=armv7-a -o /tmp/distccd_20dc6628.o
distccd[14753] (dcc_spawn_child) child started as pid32547
distccd[32547] (dcc_new_pgrp) entered process group
distccd[32547] (dcc_increment_safeguard) setting safeguard: _DISTCC_SAFEGUARD=1
distccd[14753] (dcc_collect_child) cc child 32547 terminated with status 0
distccd[14753] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
distccd[14753] (dcc_x_token_int) send DONE00000001
distccd[14753] (dcc_x_token_int) send STAT00000000
distccd[14753] (dcc_x_file) send 0 byte file /tmp/distcc_27c16628.stderr with token SERR and compression 69
distccd[14753] (dcc_x_token_int) send SERR00000000
distccd[14753] (dcc_x_file) send 0 byte file /tmp/distcc_27996628.stdout with token SOUT and compression 69
distccd[14753] (dcc_x_token_int) send SOUT00000000
distccd[14753] (dcc_x_file) send 2192 byte file /tmp/distccd_20dc6628.o with token DOTO and compression 69
distccd[14753] (dcc_x_token_int) send DOTO00000890
distccd[14753] clang-3.9-arm /root/Bela/resources/network/udp-client.c on localhost completed ok
distccd[14753] job complete
distccd[14753] (dcc_cleanup_tempfiles_inner) deleted 5 temporary files
distccd[14753] (dcc_job_summary) client: 192.168.7.2:59772 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:286ms clang-3.9-arm /root/Bela/resources/network/udp-client.c
distccd[14754] (dcc_check_client) connection from 192.168.7.2:59774
distccd[14754] (check_address_inet) match client 0x207a8c0, value 0x207a8c0, mask 0xffffffff
distccd[14754] (dcc_r_token_int) got DIST00000001
distccd[14754] (dcc_r_token_int) got ARGC00000007
distccd[14754] (dcc_r_argv) reading 7 arguments from job submission
distccd[14754] (dcc_r_token_int) got ARGV0000000d
distccd[14754] (dcc_r_token_string) got 'clang-3.9-arm'
distccd[14754] (dcc_r_argv) argv[0] = "clang-3.9-arm"
distccd[14754] (dcc_r_token_int) got ARGV00000002
distccd[14754] (dcc_r_token_string) got '-c'
distccd[14754] (dcc_r_argv) argv[1] = "-c"
distccd[14754] (dcc_r_token_int) got ARGV00000002
distccd[14754] (dcc_r_token_string) got '-v'
distccd[14754] (dcc_r_argv) argv[2] = "-v"
distccd[14754] (dcc_r_token_int) got ARGV00000029
distccd[14754] (dcc_r_token_string) got '/root/Bela/resources/network/udp-client.c'
distccd[14754] (dcc_r_argv) argv[3] = "/root/Bela/resources/network/udp-client.c"
distccd[14754] (dcc_r_token_int) got ARGV0000000e
distccd[14754] (dcc_r_token_string) got '-march=armv7-a'
distccd[14754] (dcc_r_argv) argv[4] = "-march=armv7-a"
distccd[14754] (dcc_r_token_int) got ARGV00000002
distccd[14754] (dcc_r_token_string) got '-o'
distccd[14754] (dcc_r_argv) argv[5] = "-o"
distccd[14754] (dcc_r_token_int) got ARGV0000000c
distccd[14754] (dcc_r_token_string) got 'udp-client.o'
distccd[14754] (dcc_r_argv) argv[6] = "udp-client.o"
distccd[14754] (dcc_r_argv) got arguments: clang-3.9-arm -c -v /root/Bela/resources/network/udp-client.c -march=armv7-a -o udp-client.o
distccd[14754] (dcc_scan_args) scanning arguments: clang-3.9-arm -c -v /root/Bela/resources/network/udp-client.c -march=armv7-a -o udp-client.o
distccd[14754] (dcc_scan_args) found input file "/root/Bela/resources/network/udp-client.c"
distccd[14754] (dcc_scan_args) found object/output file "udp-client.o"
distccd[14754] compile from udp-client.c to udp-client.o
distccd[14754] (dcc_run_job) output file udp-client.o
distccd[14754] (dcc_input_tmpnam) input file /root/Bela/resources/network/udp-client.c
distccd[14754] (dcc_r_token_int) got DOTI00016eeb
distccd[14754] (dcc_r_file) received 93931 bytes to file /tmp/distccd_ff4c663b.i
distccd[14754] (dcc_r_file_timed) 93931 bytes received in 0.011458s, rate 8006kB/s
distccd[14754] (dcc_set_input) changed input from "/root/Bela/resources/network/udp-client.c" to "/tmp/distccd_ff4c663b.i"
distccd[14754] (dcc_set_input) command after: clang-3.9-arm -c -v /tmp/distccd_ff4c663b.i -march=armv7-a -o udp-client.o
distccd[14754] (dcc_set_output) changed output from "udp-client.o" to "/tmp/distccd_fcec663b.o"
distccd[14754] (dcc_set_output) command after: clang-3.9-arm -c -v /tmp/distccd_ff4c663b.i -march=armv7-a -o /tmp/distccd_fcec663b.o
distccd[14754] (dcc_check_compiler_masq) /usr/local/bin/clang-3.9-arm is not a symlink
distccd[14754] (dcc_spawn_child) forking to execute: clang-3.9-arm -c -v /tmp/distccd_ff4c663b.i -march=armv7-a -o /tmp/distccd_fcec663b.o
distccd[14754] (dcc_spawn_child) child started as pid32551
distccd[32551] (dcc_new_pgrp) entered process group
distccd[32551] (dcc_increment_safeguard) setting safeguard: _DISTCC_SAFEGUARD=1
distccd[14754] (dcc_collect_child) cc child 32551 terminated with status 0
distccd[14754] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
distccd[14754] (dcc_x_token_int) send DONE00000001
distccd[14754] (dcc_x_token_int) send STAT00000000
distccd[14754] (dcc_x_file) send 1002 byte file /tmp/distcc_f203663b.stderr with token SERR and compression 69
distccd[14754] (dcc_x_token_int) send SERR000003ea
distccd[14754] (dcc_x_file) send 0 byte file /tmp/distcc_f22c663b.stdout with token SOUT and compression 69
distccd[14754] (dcc_x_token_int) send SOUT00000000
distccd[14754] (dcc_x_file) send 2192 byte file /tmp/distccd_fcec663b.o with token DOTO and compression 69
distccd[14754] (dcc_x_token_int) send DOTO00000890
distccd[14754] clang-3.9-arm /root/Bela/resources/network/udp-client.c on localhost completed ok
distccd[14754] job complete
distccd[14754] (dcc_cleanup_tempfiles_inner) deleted 5 temporary files
distccd[14754] (dcc_job_summary) client: 192.168.7.2:59774 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:320ms clang-3.9-arm /root/Bela/resources/network/udp-client.c

Now trying "faust2bela" from the Ubuntu machine. /usr/Bela/Makefile still references distcc.

gary@audio-workstation:~/faust/examples/bela$ faust2bela -tobela -gui FXChaine2.dsp 
Send to bela
/home/gary/faust/examples/bela/FXChaine2
Start communication with bela
/home/gary/faust/examples/bela/FXChaine2
FXChaine2
Checking the board is up and running at root@192.168.7.2...done
Stop running process...
Run bela now
uploadBuildRun START
Copying new source files to BeagleBone...Using rsync...
Automatically detected PROJECT_TYPE: cpp 
Running on __stretch__ with Xenomai __3__
systemctl stop bela_startup || true
PID=`grep bela-audio  /proc/xenomai/sched/stat | cut -d " " -f 5 | sed s/\s//g`; if [ -z $PID ]; then [ true = true ] || echo "No process to kill"; else [  true = true  ] || echo "Killing old Bela process $PID"; kill -2 $PID; sleep 0.2; kill -9 $PID 2> /dev/null; fi; screen -X -S Bela quit > /dev/null; exit 0;
killall scsynth 2>/dev/null& killall sclang 2>/dev/null& true
echo 'Building render.cpp...'
Building render.cpp...
distcc-clang++  -I/root/Bela/projects/FXChaine2 -I./include -I./build/pru/ -I/usr/xenomai/include/cobalt -I/usr/xenomai/include -march=armv7-a -mfpu=vfp3 -D_GNU_SOURCE -D_REENTRANT -fasynchronous-unwind-tables -D__COBALT__ -D__COBALT_WRAP__ -DXENOMAI_SKIN_posix -DXENOMAI_MAJOR=3 -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -DNDEBUG -DBELA_USE_RTDM -I/root/Bela/resources/stretch/include -std=c++11 -Wno-varargs -DNDEBUG  -Wall -c -fmessage-length=0 -U_FORTIFY_SOURCE -MMD -MP -MF"/root/Bela/projects/FXChaine2/build/render.d" -o "/root/Bela/projects/FXChaine2/build/render.o" "/root/Bela/projects/FXChaine2/render.cpp" 
distcc[5035] ERROR: compile /root/Bela/projects/FXChaine2/render.cpp on 192.168.7.1 failed
distcc[5035] (dcc_build_somewhere) Warning: remote compilation of '/root/Bela/projects/FXChaine2/render.cpp' failed, retrying locally
distcc[5035] (dcc_build_somewhere) Warning: failed to distribute and fallbacks are disabled
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:434: recipe for target '/root/Bela/projects/FXChaine2/build/render.o' failed
make: *** [/root/Bela/projects/FXChaine2/build/render.o] Error 1
Connection to 192.168.7.2 closed.
gary@audio-workstation:~/faust/examples/bela$

This still seems to be the failure:

/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe

This link takes us to some hints on that error message, which seem to culminate in "install the toolchain". The question is in the context of Android NDK though so I don't know if it applies here.

https://stackoverflow.com/questions/52646860/ld-unrecognised-emulation-mode-armelf-linux-eabi-when-cross-compiling-with-a

Here's the distcc log from this step:

distccd[14755] (dcc_check_client) connection from 192.168.7.2:59776
distccd[14755] (check_address_inet) match client 0x207a8c0, value 0x207a8c0, mask 0xffffffff
distccd[14755] (dcc_r_token_int) got DIST00000001
distccd[14755] (dcc_r_token_int) got ARGC00000013
distccd[14755] (dcc_r_argv) reading 19 arguments from job submission
distccd[14755] (dcc_r_token_int) got ARGV0000000f
distccd[14755] (dcc_r_token_string) got 'clang++-3.9-arm'
distccd[14755] (dcc_r_argv) argv[0] = "clang++-3.9-arm"
distccd[14755] (dcc_r_token_int) got ARGV0000000e
distccd[14755] (dcc_r_token_string) got '-march=armv7-a'
distccd[14755] (dcc_r_argv) argv[1] = "-march=armv7-a"
distccd[14755] (dcc_r_token_int) got ARGV0000000a
distccd[14755] (dcc_r_token_string) got '-mfpu=vfp3'
distccd[14755] (dcc_r_argv) argv[2] = "-mfpu=vfp3"
distccd[14755] (dcc_r_token_int) got ARGV0000001c
distccd[14755] (dcc_r_token_string) got '-fasynchronous-unwind-tables'
distccd[14755] (dcc_r_argv) argv[3] = "-fasynchronous-unwind-tables"
distccd[14755] (dcc_r_token_int) got ARGV00000003
distccd[14755] (dcc_r_token_string) got '-O3'
distccd[14755] (dcc_r_argv) argv[4] = "-O3"
distccd[14755] (dcc_r_token_int) got ARGV0000000e
distccd[14755] (dcc_r_token_string) got '-march=armv7-a'
distccd[14755] (dcc_r_argv) argv[5] = "-march=armv7-a"
distccd[14755] (dcc_r_token_int) got ARGV00000010
distccd[14755] (dcc_r_token_string) got '-mtune=cortex-a8'
distccd[14755] (dcc_r_argv) argv[6] = "-mtune=cortex-a8"
distccd[14755] (dcc_r_token_int) got ARGV00000010
distccd[14755] (dcc_r_token_string) got '-mfloat-abi=hard'
distccd[14755] (dcc_r_argv) argv[7] = "-mfloat-abi=hard"
distccd[14755] (dcc_r_token_int) got ARGV0000000a
distccd[14755] (dcc_r_token_string) got '-mfpu=neon'
distccd[14755] (dcc_r_argv) argv[8] = "-mfpu=neon"
distccd[14755] (dcc_r_token_int) got ARGV00000010
distccd[14755] (dcc_r_token_string) got '-ftree-vectorize'
distccd[14755] (dcc_r_argv) argv[9] = "-ftree-vectorize"
distccd[14755] (dcc_r_token_int) got ARGV0000000b
distccd[14755] (dcc_r_token_string) got '-ffast-math'
distccd[14755] (dcc_r_argv) argv[10] = "-ffast-math"
distccd[14755] (dcc_r_token_int) got ARGV0000000a
distccd[14755] (dcc_r_token_string) got '-std=c++11'
distccd[14755] (dcc_r_argv) argv[11] = "-std=c++11"
distccd[14755] (dcc_r_token_int) got ARGV0000000c
distccd[14755] (dcc_r_token_string) got '-Wno-varargs'
distccd[14755] (dcc_r_argv) argv[12] = "-Wno-varargs"
distccd[14755] (dcc_r_token_int) got ARGV00000005
distccd[14755] (dcc_r_token_string) got '-Wall'
distccd[14755] (dcc_r_argv) argv[13] = "-Wall"
distccd[14755] (dcc_r_token_int) got ARGV00000002
distccd[14755] (dcc_r_token_string) got '-c'
distccd[14755] (dcc_r_argv) argv[14] = "-c"
distccd[14755] (dcc_r_token_int) got ARGV00000012
distccd[14755] (dcc_r_token_string) got '-fmessage-length=0'
distccd[14755] (dcc_r_argv) argv[15] = "-fmessage-length=0"
distccd[14755] (dcc_r_token_int) got ARGV00000002
distccd[14755] (dcc_r_token_string) got '-o'
distccd[14755] (dcc_r_argv) argv[16] = "-o"
distccd[14755] (dcc_r_token_int) got ARGV0000002c
distccd[14755] (dcc_r_token_string) got '/root/Bela/projects/FXChaine2/build/render.o'
distccd[14755] (dcc_r_argv) argv[17] = "/root/Bela/projects/FXChaine2/build/render.o"
distccd[14755] (dcc_r_token_int) got ARGV00000028
distccd[14755] (dcc_r_token_string) got '/root/Bela/projects/FXChaine2/render.cpp'
distccd[14755] (dcc_r_argv) argv[18] = "/root/Bela/projects/FXChaine2/render.cpp"
distccd[14755] (dcc_r_argv) got arguments: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /root/Bela/projects/FXChaine2/build/render.o /root/Bela/projects/FXChaine2/render.cpp
distccd[14755] (dcc_scan_args) scanning arguments: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /root/Bela/projects/FXChaine2/build/render.o /root/Bela/projects/FXChaine2/render.cpp
distccd[14755] (dcc_scan_args) found object/output file "/root/Bela/projects/FXChaine2/build/render.o"
distccd[14755] (dcc_scan_args) found input file "/root/Bela/projects/FXChaine2/render.cpp"
distccd[14755] compile from render.cpp to render.o
distccd[14755] (dcc_run_job) output file /root/Bela/projects/FXChaine2/build/render.o
distccd[14755] (dcc_input_tmpnam) input file /root/Bela/projects/FXChaine2/render.cpp
distccd[14755] (dcc_r_token_int) got DOTI0019610a
distccd[14755] (dcc_r_file) received 1663242 bytes to file /tmp/distccd_261c6794.ii
distccd[14755] (dcc_r_file_timed) 1663242 bytes received in 0.150477s, rate 10794kB/s
distccd[14755] (dcc_set_input) changed input from "/root/Bela/projects/FXChaine2/render.cpp" to "/tmp/distccd_261c6794.ii"
distccd[14755] (dcc_set_input) command after: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /root/Bela/projects/FXChaine2/build/render.o /tmp/distccd_261c6794.ii
distccd[14755] (dcc_set_output) changed output from "/root/Bela/projects/FXChaine2/build/render.o" to "/tmp/distccd_27a96794.o"
distccd[14755] (dcc_set_output) command after: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /tmp/distccd_27a96794.o /tmp/distccd_261c6794.ii
distccd[14755] (dcc_check_compiler_masq) /usr/local/bin/clang++-3.9-arm is not a symlink
distccd[14755] (dcc_spawn_child) forking to execute: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /tmp/distccd_27a96794.o /tmp/distccd_261c6794.ii
distccd[14755] (dcc_spawn_child) child started as pid32659
distccd[32659] (dcc_new_pgrp) entered process group
distccd[32659] (dcc_increment_safeguard) setting safeguard: _DISTCC_SAFEGUARD=1
distccd[14755] (dcc_collect_child) cc child 32659 terminated with status 0x100
distccd[14755] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
distccd[14755] (dcc_x_token_int) send DONE00000001
distccd[14755] (dcc_x_token_int) send STAT00000100
distccd[14755] (dcc_x_file) send 248 byte file /tmp/distcc_58bf6794.stderr with token SERR and compression 69
distccd[14755] (dcc_x_token_int) send SERR000000f8
distccd[14755] (dcc_x_file) send 0 byte file /tmp/distcc_5b7e6794.stdout with token SOUT and compression 69
distccd[14755] (dcc_x_token_int) send SOUT00000000
distccd[14755] (dcc_x_token_int) send DOTO00000000
distccd[14755] clang++-3.9-arm /root/Bela/projects/FXChaine2/render.cpp on localhost failed
distccd[14755] job complete
distccd[14755] (dcc_cleanup_tempfiles_inner) deleted 5 temporary files
distccd[14755] (dcc_job_summary) client: 192.168.7.2:59776 COMPILE_ERROR exit:1 sig:0 core:0 ret:0 time:1109ms clang++-3.9-arm /root/Bela/projects/FXChaine2/render.cpp

    This is very confusing to me. The error message you get seems to be about the linker, ld, and it seems to come from the host, because of the list of supported emulations. The weird thing about this is that distcc should never attempt to run the linker on the host (only the compiler), and also that the line that fails really has nothing to do with the linker.

    So this is the line that is run on the host and fails:

    Digital-Larry distccd[14755] (dcc_spawn_child) forking to execute: clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o /tmp/distccd_27a96794.o /tmp/distccd_261c6794.ii

    As you see, it tries to compile the pre-processed file /tmp/distccd_261c6794.ii into the object file /tmp/distccd_27a96794.o. The pre-processed file is generated by distcc on the client (Bela), where it runs cpp (the C-preprocessor) on the file to bring in all the includes and apply all the defines and resolve the macros, generating a stand-alone C++ file that has no dependency on any external files.

    Let's troubleshoot this with incremental steps. Create a simple C++ file with no includes on the host:

    • Create a simple C++ file with no includes on the HOST: create a file called test.ii: 
      int func()
{
    return 0;
}
      and then run the same command as above (except for the file paths at the end) : 
      clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii
      if this is successful, make sure the generated file test.o is indeed an ARM binary: 
      $ file test.o
test.o: ELF 32-bit LSB relocatable, ARM, version 1 (SYSV), not stripped
    • if the above worked fine, then copy the test.ii file to the board and run ON THE BOARD: 
      distcc-clang++ -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii
      and if this is successful, check that the generated file is indeed an ARM binary: 
      $ file test.o
test.o: ELF 32-bit LSB relocatable, ARM, version 1 (SYSV), not stripped
    • if this is still good, then rename test.ii to test.cpp and run the same line again, this time with test.cpp at the end: 
      distcc-clang++ -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.cpp
    • if this is still good, still on the board, do cd /root/Bela/ and there try to re-run the full line that was failing while building your project: 
      distcc-clang++  -I/root/Bela/projects/FXChaine2 -I./include -I./build/pru/ -I/usr/xenomai/include/cobalt -I/usr/xenomai/include -march=armv7-a -mfpu=vfp3 -D_GNU_SOURCE -D_REENTRANT -fasynchronous-unwind-tables -D__COBALT__ -D__COBALT_WRAP__ -DXENOMAI_SKIN_posix -DXENOMAI_MAJOR=3 -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -DNDEBUG -DBELA_USE_RTDM -I/root/Bela/resources/stretch/include -std=c++11 -Wno-varargs -DNDEBUG  -Wall -c -fmessage-length=0 -U_FORTIFY_SOURCE -MMD -MP -MF"/root/Bela/projects/FXChaine2/build/render.d" -o "/root/Bela/projects/FXChaine2/build/render.o" "/root/Bela/projects/FXChaine2/render.cpp"
      does this successfully generated the file /root/Bela/projects/FXChaine2/build/render.o? And is it an ARM binary?

        giuliomoro first step fails.

        gary@audio-workstation:~/test$ clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)

gary@audio-workstation:~/test$ cat test.ii
int func()
{
    return 0;
}
gary@audio-workstation:~/test$ 

gary@audio-workstation:~/test$ which clang++-3.9-arm 
/usr/local/bin/clang++-3.9-arm

gary@audio-workstation:~/test$ cat /usr/local/bin/clang++-3.9-arm 
#!/bin/bash
/usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $0

gary@audio-workstation:~/test$ clang++-3.9-arm 
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$

        Calling that compiler throws the error with no parameters given at all! [edit - it's calling the script which does add parameters] So maybe that is broken somehow. I used apt to install it.

        gary@audio-workstation:~/test$ sudo apt list | grep ^clang-3.9

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

clang-3.9/bionic,now 1:3.9.1-19ubuntu1 amd64 [installed]
clang-3.9-doc/bionic,bionic 1:3.9.1-19ubuntu1 all
clang-3.9-examples/bionic 1:3.9.1-19ubuntu1 amd64
gary@audio-workstation:~/test$ 

gary@audio-workstation:~/test$ sudo dpkg -s clang-3.9
Package: clang-3.9
Status: install ok installed
Priority: optional
Section: devel
Installed-Size: 140661
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Source: llvm-toolchain-3.9
Version: 1:3.9.1-19ubuntu1
Replaces: clang-3.1, clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1), clang-include-fixer-3.9, compiler-rt
Provides: c++-compiler, c-compiler, objc-compiler
Depends: libc6 (>= 2.14), libclang1-3.9 (= 1:3.9.1-19ubuntu1), libgcc1 (>= 1:3.4), libjsoncpp1 (>= 1.7.4), libllvm3.9 (>= 1:3.9.1-6~), libstdc++6 (>= 5.2), libstdc++-7-dev, libgcc-7-dev, libobjc-7-dev, libclang-common-3.9-dev (= 1:3.9.1-19ubuntu1), libc6-dev, binutils
Recommends: llvm-3.9-dev, python
Suggests: gnustep, gnustep-devel, clang-3.9-doc
Breaks: clang-3.1, clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1), clang-include-fixer-3.9, compiler-rt
Description: C, C++ and Objective-C compiler (LLVM based)
 Clang project is a C, C++, Objective C and Objective C++ front-end
 for the LLVM compiler. Its goal is to offer a replacement to the GNU Compiler
 Collection (GCC).
 .
 Clang fully implements all published ISO C++ standards including C++11, as
 well as the upcoming C++14 standard, and some parts of the fledgling C++1z
 standard, and is considered a production-quality C++ compiler.
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Homepage: http://www.llvm.org/
gary@audio-workstation:~/test$

        Some other random tests to see what exactly we are calling and the simplest command to generate the error.

        gary@audio-workstation:~/test$ which clang++-3.9-arm 
/usr/local/bin/clang++-3.9-arm
gary@audio-workstation:~/test$ cat /usr/local/bin/clang++-3.9-arm 
#!/bin/bash
/usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $0
gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 
clang: error: no input files
gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf
clang: error: no input files
gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$ /usr/bin/clang++-6.0 -target armv7l-unknown-linux-gnueabihf ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$ /usr/bin/clang++ -target armv7l-unknown-linux-gnueabihf ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$

            right, what about ls -l /usr/bin/clang++-3.9 ? is it a symlink to something?
            Further to this, you could try running

            clang++-3.9-arm -march=armv7-a -mfpu=vfp3 -fasynchronous-unwind-tables -O3 -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -ftree-vectorize -ffast-math -std=c++11 -Wno-varargs -Wall -c -fmessage-length=0 -o ./test.o ./test.ii

            and remove one option at a time until it succeeds (or better: start with just clang++-3.9-arm -march=armv7-a ./test.o ./test.ii, is it successful? Then add more and more options until it fails .

            I am wondering whether maybe your clang-3.9 does not support hard-float, in which case the failing option should be -mfloat-abi=hard. Can you dpkg -l | grep clang and perhaps then dpkg --info <names of the 3.9 packages from the previous result>?

              I used the -### command with just clang++ (the 6.0 version installed by default, but the error is the same so far regardless of the compiler version used, 3.9 or 6.0):

              gary@audio-workstation:~/test$ /usr/bin/clang++ -### -target armv7l-unknown-linux-gnueabihf ./test.ii
clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
Target: armv7l-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
 "/usr/lib/llvm-6.0/bin/clang" "-cc1" "-triple" "armv7-unknown-linux-gnueabihf" "-emit-obj" "-mrelax-all" "-disable-free" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "test.ii" "-mrelocation-model" "static" "-mthread-model" "posix" "-mdisable-fp-elim" "-fmath-errno" "-masm-verbose" "-mconstructor-aliases" "-fuse-init-array" "-target-cpu" "generic" "-target-abi" "aapcs-linux" "-mfloat-abi" "hard" "-fallow-half-arguments-and-returns" "-dwarf-column-info" "-debugger-tuning=gdb" "-resource-dir" "/usr/lib/llvm-6.0/lib/clang/6.0.0" "-fdeprecated-macro" "-fdebug-compilation-dir" "/home/gary/test" "-ferror-limit" "19" "-fmessage-length" "80" "-fno-signed-char" "-fobjc-runtime=gcc" "-fcxx-exceptions" "-fexceptions" "-fdiagnostics-show-option" "-fcolor-diagnostics" "-o" "/tmp/test-dc97e4.o" "-x" "c++-cpp-output" "./test.ii"
 "/usr/bin/ld" "-z" "relro" "-X" "--hash-style=gnu" "--eh-frame-hdr" "-m" "armelf_linux_eabi" "-dynamic-linker" "/lib/ld-linux-armhf.so.3" "-o" "a.out" "crt1.o" "crti.o" "crtbegin.o" "-L/usr/lib/llvm-6.0/bin/../lib" "-L/lib/../lib" "-L/usr/lib/../lib" "-L/usr/lib/llvm-6.0/bin/../lib" "-L/lib" "-L/usr/lib" "/tmp/test-dc97e4.o" "-lstdc++" "-lm" "-lgcc_s" "-lgcc" "-lc" "-lgcc_s" "-lgcc" "crtend.o" "crtn.o"
gary@audio-workstation:~/test$

              giuliomoro yes there is a symlink there that appears to be from the installation (I didn't do it).

              gary@audio-workstation:~/test$ ls -l /usr/bin/clang++-3.9 
lrwxrwxrwx 1 root root 27 Dec 26  2017 /usr/bin/clang++-3.9 -> ../lib/llvm-3.9/bin/clang++
gary@audio-workstation:~/test$
                gary@audio-workstation:~/test$ clang++-3.9-arm -march=armv7-a ./test.o ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$

gary@audio-workstation:~/test$ dpkg -l | grep clang
ii  clang                                      1:6.0-41~exp5~ubuntu1                        amd64        C, C++ and Objective-C compiler (LLVM based)
ii  clang-3.9                                  1:3.9.1-19ubuntu1                            amd64        C, C++ and Objective-C compiler (LLVM based)
ii  clang-6.0                                  1:6.0-1ubuntu2                               amd64        C, C++ and Objective-C compiler
ii  libclang-common-3.9-dev                    1:3.9.1-19ubuntu1                            amd64        clang library - Common development package
ii  libclang-common-6.0-dev                    1:6.0-1ubuntu2                               amd64        clang library - Common development package
ii  libclang1-3.9:amd64                        1:3.9.1-19ubuntu1                            amd64        C interface to the clang library
ii  libclang1-6.0:amd64                        1:6.0-1ubuntu2                               amd64        C interface to the clang library
gary@audio-workstation:~/test$

                dpkg --info doesn't show anything but apt show does:

                gary@audio-workstation:~/test$ apt show clang-3.9
Package: clang-3.9
Version: 1:3.9.1-19ubuntu1
Priority: optional
Section: universe/devel
Source: llvm-toolchain-3.9
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 144 MB
Provides: c++-compiler, c-compiler, objc-compiler
Depends: libc6 (>= 2.14), libclang1-3.9 (= 1:3.9.1-19ubuntu1), libgcc1 (>= 1:3.4), libjsoncpp1 (>= 1.7.4), libllvm3.9 (>= 1:3.9.1-6~), libstdc++6 (>= 5.2), libstdc++-7-dev, libgcc-7-dev, libobjc-7-dev, libclang-common-3.9-dev (= 1:3.9.1-19ubuntu1), libc6-dev, binutils
Recommends: llvm-3.9-dev, python
Suggests: gnustep, gnustep-devel, clang-3.9-doc
Breaks: clang-3.1, clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1), clang-include-fixer-3.9, compiler-rt
Replaces: clang-3.1, clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1), clang-include-fixer-3.9, compiler-rt
Homepage: http://www.llvm.org/
Download-Size: 37.4 MB
APT-Manual-Installed: yes
APT-Sources: http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
Description: C, C++ and Objective-C compiler (LLVM based)
 Clang project is a C, C++, Objective C and Objective C++ front-end
 for the LLVM compiler. Its goal is to offer a replacement to the GNU Compiler
 Collection (GCC).
 .
 Clang fully implements all published ISO C++ standards including C++11, as
 well as the upcoming C++14 standard, and some parts of the fledgling C++1z
 standard, and is considered a production-quality C++ compiler.

gary@audio-workstation:~/test$ 

gary@audio-workstation:~/test$ apt show libclang-common-3.9-dev
Package: libclang-common-3.9-dev
Version: 1:3.9.1-19ubuntu1
Priority: optional
Section: universe/libdevel
Source: llvm-toolchain-3.9
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 39.1 MB
Depends: libc6 (>= 2.15), libgcc1 (>= 1:3.4), libstdc++6 (>= 5.2), libtinfo5 (>= 6), libllvm3.9 (= 1:3.9.1-19ubuntu1)
Homepage: http://www.llvm.org/
Download-Size: 2,575 kB
APT-Manual-Installed: no
APT-Sources: http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
Description: clang library - Common development package
 Clang project is a C, C++, Objective C and Objective C++ front-end
 for the LLVM compiler. Its goal is to offer a replacement to the GNU Compiler
 Collection (GCC).
 .
 Clang fully implements all published ISO C++ standards including C++11, as
 well as the upcoming C++14 standard, and some parts of the fledgling C++1z
 standard, and is considered a production-quality C++ compiler.
 .
 This package contains the clang generic headers and some libraries
 (profiling, etc).

gary@audio-workstation:~/test$ 

gary@audio-workstation:~/test$ apt show libclang1-3.9
Package: libclang1-3.9
Version: 1:3.9.1-19ubuntu1
Priority: optional
Section: universe/devel
Source: llvm-toolchain-3.9
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 22.9 MB
Depends: libc6 (>= 2.14), libgcc1 (>= 1:3.4), libllvm3.9 (>= 1:3.9.1-6~), libstdc++6 (>= 5.2)
Homepage: http://www.llvm.org/
Download-Size: 5,950 kB
APT-Manual-Installed: no
APT-Sources: http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
Description: C interface to the clang library
 Clang project is a C, C++, Objective C and Objective C++ front-end
 for the LLVM compiler. Its goal is to offer a replacement to the GNU Compiler
 Collection (GCC).
 .
 Clang fully implements all published ISO C++ standards including C++11, as
 well as the upcoming C++14 standard, and some parts of the fledgling C++1z
 standard, and is considered a production-quality C++ compiler.
 .
 This package contains the clang library.
 .
 The C Interface to Clang provides a relatively small API that exposes
 facilities for parsing source code into an abstract syntax tree (AST),
 loading already-parsed ASTs, traversing the AST, associating physical source
 locations with elements within the AST, and other facilities that support
 Clang-based development tools.

gary@audio-workstation:~/test$

                Digital-Larry gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf ./test.ii

                you nee -c otherwise it will attempt to use the linker.

                giuliomoro
                or better: start with just clang++-3.9-arm -march=armv7-a ./test.o ./test.ii, is it successful?

                I forgot -c, too, and also -o.

                so try:

                clang++-3.9-arm -march=armv7-a -c -o ./test.o ./test.ii

                if that succeeds, add one option at a time until it fails ..

                      giuliomoro

                      gary@audio-workstation:~/test$ clang++-3.9-arm -march=armv7-a -c -o ./test.o ./test.ii
                      /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
                      Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
                      clang: error: linker command failed with exit code 1 (use -v to see invocation)
                      gary@audio-workstation:~/test$

                        giuliomoro

                        gary@audio-workstation:~/test$ /usr/bin/clang++-3.9 -###  -march=arm7-a -c -o ./test.o ./test.ii
clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
 "/usr/lib/llvm-3.9/bin/clang" "-cc1" "-triple" "x86_64-pc-linux-gnu" "-emit-obj" "-mrelax-all" "-disable-free" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "test.ii" "-mrelocation-model" "static" "-mthread-model" "posix" "-mdisable-fp-elim" "-fmath-errno" "-masm-verbose" "-mconstructor-aliases" "-munwind-tables" "-fuse-init-array" "-target-cpu" "arm7-a" "-dwarf-column-info" "-debugger-tuning=gdb" "-coverage-file" "/home/gary/test/./test.o" "-resource-dir" "/usr/lib/llvm-3.9/bin/../lib/clang/3.9.1" "-fdeprecated-macro" "-fdebug-compilation-dir" "/home/gary/test" "-ferror-limit" "19" "-fmessage-length" "80" "-fobjc-runtime=gcc" "-fcxx-exceptions" "-fexceptions" "-fdiagnostics-show-option" "-fcolor-diagnostics" "-o" "./test.o" "-x" "c++-cpp-output" "./test.ii"
gary@audio-workstation:~/test$

                        sorry, hang on, swimming in stuff here...

                        gary@audio-workstation:~/test$ clang++-3.9-arm -### -march=armv7-a -c -o ./test.o ./test.ii
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gary@audio-workstation:~/test$

                          I added the "-###" right in the clang++-3.9-arm script itself (I also tried adding "-fno-autolink"):

                          gary@audio-workstation:~/test$ sh -x /usr/local/bin/clang++-3.9-arm -### -march=armv7-a -c -o ./test.o ./test.ii
+ /usr/bin/clang++-3.9 -### -target armv7l-unknown-linux-gnueabihf -fno-autolink --sysroot /home/gary/arm /usr/local/bin/clang++-3.9-arm
clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)
Target: armv7l-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
 "/usr/bin/ld" "--sysroot=/home/gary/arm" "-X" "--eh-frame-hdr" "-m" "armelf_linux_eabi" "-dynamic-linker" "/lib/ld-linux-armhf.so.3" "-o" "a.out" "crt1.o" "crti.o" "crtbegin.o" "/usr/local/bin/clang++-3.9-arm" "-lstdc++" "-lm" "-lgcc_s" "-lgcc" "-lc" "-lgcc_s" "-lgcc" "crtend.o" "crtn.o"
gary@audio-workstation:~/test$

                          OK after that last one I found it! It wasn't until I ran the script with "sh -x" that I saw the wrong command line substitution. The last thing on the command line should be the source file name, but it is the script file name!

                          /usr/bin/clang++-3.9 -### -target armv7l-unknown-linux-gnueabihf -fno-autolink --sysroot /home/gary/arm /usr/local/bin/clang++-3.9-arm

                          In this file, I had "$0" instead of "$@" - I changed it and it works!!!!

                          gary@audio-workstation:~/faust/examples/bela$ cat /usr/local/bin/clang++-3.9-arm 
#!/bin/bash
/usr/bin/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@
gary@audio-workstation

                          Giuilo thanks so much for your patient help. Sorry it was so difficult but maybe it helps the next guy. At this point I think Faust on Bela is a very viable idea. Hopefully my next questions and comments will be more on the implementation side.

                          giuliomoro I think so. At least, I see it shown in some of the things I posted a day or two ago. Just very hard to see! One question I have which I will defer answering for now is whether clang 3.9 is really needed on the Ubuntu system or whether "clang" as installed (which by now is at 6.0) is OK. I understand that the distcc as being called by faust2bela may be making the specific reference to 3.9. Lots of different interacting parts here.

                            Linker problem

                            One issue of 3.9 vs 6.0 is that linking is always done on the board by calling clang++-3.9, and so if there is any incompatibility between the .o files generated by 6.0 and the way clang++-3.9 calls the ld linker, or the way the older ld linker on the board deals with object files, then you could have some issues. I guess this is unlikely to happen, but I am not sure what guarantees clang and ld give about interoperability of different versions.

                            Compiler problem

                            Another issue is that different versions of clang may decide to pack structs (or at least ARM intrinsics) in different ways, for instance:

                            giuliomoro What is surprising is that while alignof(float32x4_t) is 16 bytes on clang-3.9, it is actually 8 bytes on gcc-6.3 and clang-6.0.

                            and

                            giuliomoro However, I am wondering where that difference comes from, as the arm_neon.h files for the two versions of clang provide identical definition for all the neon vector types.

                            More in general, there would be a problem if you have object files that both include some .h file that is interpreted differently by the two versions of the compiler, and you compile the two objects with different versions of the compiler. Again, I am not sure what backward/forward compatibility guarantees clang provides. Clearly, if you ALWAYS use distcc to build all the files on Bela, there would be no problem. The problem only comes up (potentially) if you are intermixing .o files compiled with different versions of the compiler.
                            For instance - in you current situation - you built all the Bela core/ files with the local version of clang-3.9 the first time you built the first program. If you switch to using clang-6.0, I'd recommend you do a make -C ~/Bela clean on the board, so that those files are re-built next time you build a project.

                                  giuliomoro ah well never mind, all that stuff gives me a headache! I'm OK leaving 3.9 installed. Thanks for the detailed explanation.