Debugging (again)

AndyCap

Hi Guys,

OK I have built a Linaro cross compiler for OS X that closely matches the Bela (4.4.59 kernel, 2.3 glibc, linaro-6.3-2017.02), I have also updated the Eclipse project so it builds, everything works fine. I did try 2.4 glibc but had lots of problems there with TLS, not sure why.

Remote debugging seems to work most of the time, even in the audio thread but occasionally we get a bit of a panic:

Message from syslogd@bela at May 20 08:51:06 ...
 kernel:[  347.388363] Internal error: : 1b [#5] PREEMPT ARM

Message from syslogd@bela at May 20 08:51:06 ...
 kernel:[  347.509629] Process bela-audio (pid: 797, stack limit = 0xdf7c4218)

Message from syslogd@bela at May 20 08:51:06 ...
 kernel:[  347.515924] Stack: (0xdf7c5fb0 to 0xdf7c6000)

Message from syslogd@bela at May 20 08:51:06 ...
 kernel:[  347.520307] 5fa0:                                     000c07e8 00000000 000c07e8 000519b0

Message from syslogd@bela at May 20 08:51:06 ...
 kernel:[  347.528525] 5fc0: 00000000 b60b8e18 befff7a4 000f0042 b60b8e1c 00000001 00000000 b60b8c04

Message from syslogd@bela at May 20 08:51:06 ...
 kernel:[  347.536742] 5fe0: 000d7b38 b60b8be8 00074b60 000519c8 600d0010 ffffffff 00000000 00000000

Message from syslogd@bela at May 20 08:51:06 ...
 kernel:[  347.544973] Code: f1080080 e3130020 1a000006 e2424004 (e4b40000)

I also tried local gdb with clang compiled code and you get the same issue.

It is intermittent, you can ctrl-c in gdb to get it back but then you cannot continue, a re-run works.

You can force the issue by continuing from the breakpoint quickly, if you wait a couple of seconds it is fine.

Any ideas?

AndyCap

giuliomoro Hope that that is gcc 6.3

AndyCap (4.4.59 kernel, 2.3 glibc, linaro-6.3-2017.02)

There is a downloadable version for Linux with the same settings, older versions I tried and different gcc and glibc versions all had various problems, thats why I had to build the toolchain for OS X in the end (not a job that was fun).

I tried clang but Eclipse support for clang is laughably bad, it basically doesn't work! It seems there was a couple of community attempts to integrate it, then an official CDT release and then nothing, seems it was dropped really. I gave up trying to get it to work.

For the linux instructions I think it is best to get the user to download and install the matching linaro toolchain rather than relying on the distro.

giuliomoro

I have had this before while debugging directly on Bela. I do not think this is a local vs remote thing. Not sure what the cause is, but it was not too frequent, so much that I taught a class about debugging Bela code using gdb and I managed to get through the two hours with only a handful of people experiencing this once (I would then have them reboot the machine and they could keep going after that).

I am not sure what the underlying issue is, probably something in Xenomai's interrupt pipeline. I have been meaning to post on the Xenomai list about it for a while but never got around to doing it. Did you find a reliable way of consistently triggering this? The only way I got it to reliably OOPS - iirc - is to set a conditional breakpoint in the real-time thread. Note that this would be bad practice anyhow, because conditional breakpoints are incompatible with real-time, but perhaps that is a good starting point for the Xenomai people to debug it. So if I need a conditional breakpoint, I have to add the condition in the C++ code and add an unconditional breakpoint.

AndyCap

Yeah I agree it isn't a remote vs local thing, I'm guessing it is something to do with the thread being passed to and fro between the kernels on the bp. Interestingly ctrl-c works consistently without any problems.

I can reliable trigger it by putting a breakpoint on render() and then quickly continuing after it hits the breakpoint, c <enter>, c<enter> etc till it goes bang.

AndyCap

Also I have a zip file here of the cross-compiler I have built for OS X. Would you like to host it on your site somewhere?

I also have an updated Eclipse project fully working on OS X, I need to test it on Linux as well but then do you want to merge it into your github?

I also have a script that creates a "mini" sysroot in /usr/opt/linaro that the project then uses for includes/libs. So the user would have to:

Install the cross-compiler
Run the script
Import the bela folder into eclipse

giuliomoro

AndyCap Also I have a zip file here of the cross-compiler I have built for OS X.

Hope that that is gcc 6.3 ? It'd be good to have the same version of the compiler on the board and the cross-compiling environment? Also perhaps you can get away with using clang for compilation (with -target armv7l-unknown-linux-gnueabihf), and only using the linker from arm-linux-gnueabihf?

giuliomoro

Sounds good yes, happy to do all the above. Please create a merge request for the Eclipse project against our dev branch.
Would you be happy to edit this page accordingly as well? https://github.com/BelaPlatform/Bela/wiki/Compiling-Bela-projects-in-Eclipse
That'd be ace. We left out the cross-compiling thing out in the past few years because of the trouble of supporting it across different operating systems. However, I have a draft blog post on how to use distcc with clang to speed up "native" builds by offloading them to the host: https://github.com/BelaPlatform/bela_newBlog/blob/master/docs/_drafts/2017-10-31-cross-compiling.markdown

AndyCap

Ok, will do.

Happy the edit the page as well with updated instructions.

When I test it on Linux I will get on with it.

Is it only OS X and linux that need supporting?

giuliomoro

AndyCap Is it only OS X and linux that need supporting?

Nothing is needed everything is welcomed! I think on Linux most distros have a pre-built cross-compiler that you can get with the package manager.

giuliomoro

In my experience, there is no clear winner between clang and gcc in respect to the CPU performance of the generated code. It really depends on the code you are compiling. Code generated by the Heavy Pd-to-C++ compiler used to perform much better with clang, for instance, but that is far from true across the board.

AndyCap Eclipse support for clang is laughably bad

What is lacking/needed? I cannot think of anything compiler-specific in Eclipse (but again, I have not used Eclipse much and not in the last three years!).

AndyCap

I will also do some benchmarks between the cross compiled code and locally compiled clang to check nothing it too wrong!

AndyCap

giuliomoro What is lacking/needed? I cannot think of anything compiler-specific in Eclipse (but again, I have not used Eclipse much and not in the last three years!).

I would say nearly everything, it doesn't integrate with the IDE correctly, I just could not get it to work. You can only create cross projects using clang by choosing "toolchains not compatible with the host" which is a sign of things to come, I only looked at OS X though but as OS X is clang centric I had no hope. This was with a simple hello world project! I may look at it again but I just wanted to get something working.

jarm Hi @AndyCap, I have a use case for this toolchain and would be happy to test all your instructions and documentations once you're ready for that! Looks great.

Nice, when it is all up and going I will contact you. Struggling on Linux with release builds at the moment...

jarm

Hi @AndyCap, I have a use case for this toolchain and would be happy to test all your instructions and documentations once you're ready for that! Looks great.

AndyCap

Good stuff, we just need someone to test it now:

jarm Hi @AndyCap, I have a use case for this toolchain and would be happy to test all your instructions and documentations once you're ready for that! Looks great.

Hi @jarm

It is already to go if you follow the instructions here: https://github.com/BelaPlatform/Bela/wiki/Compiling-Bela-projects-in-Eclipse

Any questions or problems please post here and I will fix them up.

Cheers

Andy

AndyCap

I have it tested now in both Linux and OS X.

I have my changes in the dev (and dev-modular) branch of my forked repository.

I also have some text to update the wiki with.

I have no idea what I am doing in GitHub so instead of arseing anything up I thought I would ask how to create a merge request against your dev branch?

giuliomoro

hmm, did you fork your repo on github? If yes, then go to github.com/[yourusername]/Bela/tree/dev and you should find a button that says "Pull request". From there on, you follow their unintuitive GUI and try to get it to work. If it says "There isn’t anything to compare." then try switching the base. At some point you may be lucky enough to have managed to force it into displaying the one (or more) commit(s) that you have made on your branch and ask you to create a Pull Request (which they may intuitively shorten as PR) against the dev branch of BelaPlatform/Bela/

Good luck!

[disclaimer: it actually is easier than I make it look above].

giuliomoro

As for the wiki, you can just freely edit it as long as you are logged in with you github account. If you need to add images/files (or the binaries for the compiler?), then you would have to clone the wiki repo (https://github.com/BelaPlatform/bela.wiki.git) on your computer, add the files and push it back.

AndyCap

giuliomoro As for the wiki, you can just freely edit it as long as you are logged in with you github account. If you need to add images/files (or the binaries for the compiler?), then you would have to clone the wiki repo (https://github.com/BelaPlatform/bela.wiki.git) on your computer, add the files and push it back.

I'm getting a 403 when trying to push from my clone:

fatal: unable to access 'https://github.com/BelaPlatform/bela.wiki.git/': The requested URL returned error: 403

AndyCap

Well I tried the pull request (surely it should be push!) and I'm guessing something went wrong!

My changes are there along with craploads of others, see what you think!

I will clone the wiki as I want to add a few images, I did try adding the cross-compiler to github but it was too big. I could probably cut the size down by removing the generated sysroot but I still don't think it will go down to the required size. I did email you a link to download it though.

giuliomoro

didn't receive the download link (yet). Added comments on the PR. I can edit with more step-by-step guidance if needed.

AndyCap

Might have got flagged as junk because of the link, here it is: https://www.dropbox.com/s/5kurk4n7dzuryi7/arm-bela-linux-gnueabihf.zip?dl=0

AndyCap

I did just try cutting the size down but everything stopped working, so possibly best at the moment to keep it like this.