Hi,

Ive kind of got myself into a confused situation 🙂 (again!!)

Ive got a number of Belas, salt and BelaMini , and Im finding it difficult to track which have been upgraded and what versions of the software they have one them ...

( I guess I should keep notes, but im not that organised)

the first thing that 'confuses' me, is the difference between the 'bela image' and the 'bela software', and the relationship between them.

the former is an image you install directly onto the sdcard (and then in my case i transfer to emmc) ,
the latter you can upgrade from GitHub then apply with the browser.
what Im usually confused about is when the former is needed? (as its more effort)

ok, so this leads to a couple of suggestions, which would make my life easier 🙂
(in order of what helps the most for me 😉 )

a) version numbers
can we have a version for image and a separate version for software, and report these in the web interface.
such that we can quickly see if one or both need upgrading.

b) upgrade from web interface
rather than download zip, update...
that downloads directly the latest version (assuming you have a internet connection on the board)

c) check/update button in web interface
I suppose this is a combination of the above
quick 'check' button which tells you, should you upgrade the image or the software.

I do understand that really you don't need to keep bela on the latest/greatest, but the issue I have is when I come to start something new on Bela - I don't know if it a good idea/advantageous to upgrade or not (or if thats means the image or software) .to do this I need to :
a) try to work out what is currently on that particular bela
b) come to the forum to work out what changes have happened, so might be useful
honestly, its usually easier for me to just upgrade.

the only caveat to that is upgrading the emmc/image is actually a bit time consuming, and that this upgrading whenever I come to pick up an old bela, is precisely how ive got into my confused states, with belas on different versions 😉 (until I 'bite the bullet' and go around updating all belas/salt/bela mini at once)

thoughts?

how do others go about this?

EDIT:
Bonus question 🙂

so I did the update via web/zip file method.
but now I notice on the board if I do a git status its telling me that there are loads of 'changed' files.
does the zip file update , not bin the whole ~/Bela directory, and so get the git status from the zip fie?

    Right, so:

    • image version (the one you get when you ssh onto the board, or by grep v0 /etc/motd: this is the image version number, which corresponds to the tag of the bela-image-builder used to build it. Differences/updates between tags can be inferred by inspecting the changelog, but the most important ones are summarised in the release page *.
    • Bela source code: each image comes with the newest Bela code available at the time of release. - Sometimes minor (e.g.: v0.3.6a, v0.3.6b) image releases are built from the same bela-image-builder source code as the base release, just with newer Bela source code. This normally happens only if there were some critical bug fixes.
    • updating the Bela source code: this gets the files from the uploaded zip archive and installs them in the /root/Bela folder running the scripts/update_board file provided by the archive itself (this is achieved by calling (calling make checkupdate and make update). Unfortunately the archive does not come with a copy of the git repo, but only with a snapshot of the master, and update_board just rolls over the new files on the board (usingrsync so that files that disappeared/moved are also taken care of) and also installs some extra goodies (e.g.: all the packages in Bela/resources/stretch/deb/ and it make installs all the tools in Bela/resources/tools/).
    • what VERSION are you on? As the downloaded zip archive comes without a full git repo, there is no way to update the local repo, therefore running git status after an update will always show your files as being out of date. Similarly, this makes it hard to display any meaningful version information in the IDE: we simply DON'T KNOW what version we are on. The "About Bela" box in the IDE used to contain misleading information, and we had some sort of discussion about what should have been in there. I strongly discourage keeping some sort of fictitious version number baked in the repo, because that would be prone to being left behind while developing and there is no way to update it automatically and reliably. Having some sort of minimal build server would be great: you would request a zip archive from that server, and this would pull together the latest Bela code and the latest extra packages (e.g.: libpd, supercollider, csound, that currently live in resources/stretch/deb which is really annoying because the increase the size of the repo), and adds a version tag. However, we wouldn't want to find ourselves in the situation of maintaining any sort of public-facing online server until we have a more compelling/important use case for it.
      I think a good step forward here would be to at least show the date of the last successful update. I also just came up with this idea: if instead of suggesting the github.com/BelaPlatform/Bela/master.zip link for download, we suggest https://api.github.com/repos/BelaPlatform/Bela/zipball/master, then this will return a zip archive whose name is something like BelaPlatform-Bela-d94f045. This means that we could infer the commit hash and display that in the IDE. Still sub-optimal but relatively easy to implement. We should also display the image version in the IDE, yes.
    • how do I do it? I git pull. This most of the time works fine, unless there has been a change in the resources/ folder, in which case I either apply it manually (e.g.: installing an updated package), or even run scripts/update_board. In fact, you can safely run it from within a git pull'ed repo to just perform the extra-core-code bits.

    thetechnobear a) version numbers
    can we have a version for image and a separate version for software, and report these in the web interface.
    such that we can quickly see if one or both need upgrading.

    with the constraints outlined above, yes I think we should do it. I actually took it out when it was baked in because it was confusing.

    thetechnobear b) upgrade from web interface
    rather than download zip, update...
    that downloads directly the latest version (assuming you have a internet connection on the board)

    We don't want to implement workflows that require the board to be connected to the internet. That is because:
    a) not so many people have their board on the internet
    b) dealing with people's internet connection sharing would probably bring much more headache to us than the benefits that would come from this solution, so until there is a reliable cross-platform workflow for internet connection sharing from the host computer, I don't think this will happen.

    thetechnobear c) check/update button in web interface
    I suppose this is a combination of the above
    quick 'check' button which tells you, should you upgrade the image or the software.

    If we get the IDE to be more web-oriented (e.g.: interrogating the Github API). this is definitely the way to go. Again, this is really limited by the fact that we cannot assume that the board is connected to the internet, so we need to do Cross-origin requests (CORS), which are in general not encouraged by some browsers, and simply not possible if the server does not allow it. For instance, once we did this for the Heavy server, and one of either Chrome or Firefox assumed that receiving a 302 (redirect) in response to a POST request (which is expected behaviour of many APIs), was treated as a failure for a CORS. In that instance, we had to ask Enzien to change their server's response. They kindly did that, but we cannot ask GitHub or anyone else to do it. So this would mean that we would need our own server middleware to handle with these cases, which we don't really have the resources to go into just now.

    * For instance, from the releases page :

    v0.3.6b:

    Most relevant changes:
    Support for CTAG
    Installs hvcc

    v0.3.5b

    Most relevant changes:
    Drivers are no longer required on Windows or Mac (starting from OS X 10.11). On Windows, with no drivers, the board will show up at 192.168.6.2; on Mac, with no drivers, it will show up at 192.168.7.2 (or bela.local/).
    board and host keeps MAC address between reboots, so that the host can uniquely identify each board and store settings

    v0.3.4

    added support for Bela Mini
    automatic detection of the board you run on (e.g.: BeagleBone Green, Black, Black Wireless, Pocket Beagle), so the appropriate device-tree is loaded for each.
    support for uboot overlays: add more overlays to uEnv.txt with uboot_overlay_addrX= where X is a number between 0 and 7

          Thanks for the detail reply.
          (your detailed responses are an fantastic part of the Bela 'experience')

          Internet - yeah, I understand your point - though perhaps quite a few of us have connections - but for sure its just a 'simplicity thing'

          more important i think is clear information about versions / updates availability - its no really issue doing it as a 2 step process - download/apply.

          image verses update ("bela software update")...

          ok, I think the thing here is what warrants a new image?
          3.6 installs hvcc, but could that not have been done as an update? (in the same way as the update can install Csound/SC?)

          if I look to Organelle/Norns, basically an image is never usually required, you can simply do an update.. this will do things like run scripts to install packages, update configuration etc.
          really the only requirement for a new image, is if there is really big change, e.g. a new distro, or its its just a daunting task to do as an update (e.g. perhaps installing new kernel etc)
          (I think we all recognise installing an image is best avoided, its a pain on salt and some bbb, and also means you have to redo any customisations - hostname changes, wifi setup, installing your dev tools/projects)

          however I will say, I think that this is already your intention too,
          as I see with the image release notes, I guess many of these changes were 'big' e.g. driver changes , so perhaps the install hvcc is the 'exception' rather than the rule.

          git repo vs update

          ok, I think the underlying thing here, is that an 'update' by whatever method, should preferably lead to the same state on the bela board.

          so i think the update release, should be a clone (depth 1) of the repo, so then on the boards its a valid repo, which you can do git pull on.

          so I think the update image could be basically:
          - a cloned repo (depth 1 ) , and when pulled a file is created with either a version tag, or commit tag

          • an update script (which copies things into the right place, installs packages etc)
            ( i think this is what make install is kind of doing?)

          • the repo should contain resources (e.g packages) it needs

            ( I think this is whats in resources/stretch/deb)

          the idea then is updating then is:
          - you can see version number of update and image on Web UI
          - its clear from wiki? releases notes? GitHub? what the latest version is of update and image, so we know if to update or not.
          then we can update by one of two means

          • we can either download release package and apply from Web
            OR
          • git pull, and then run update.sh

          either way should lead to the same thing/image...

          (so those of us with an internet connection on bela, can do the second and 'know' the board is updated in the same way as if we ran the release package update)

          I think much of what Ive detailed about is pretty much what your doing already,
          perhaps some of it is just my lack of understanding or some minor inconsistencies which make it confusing for those of use not following the 'back story'

          ... and I think pretty much the same/similar to what you proposed in your reply.

            thetechnobear 3.6 installs hvcc, but could that not have been done as an update? (in the same way as the update can install Csound/SC?)

            hvcc by itself is just a folder, so it could be installed similarly, however it requires a number of python dependencies and apt-get packages to be installed. All the other packages we install are .deb packages that we put together, so they would not come with apt-get. Sometimes it is a bit of a mess messing around with apt-get- installed packages, so I would not want to ship any of those (and also I would not know how to install python pip packages offline, but maybe it's just a matter of going through that. We are not currently doing much with hvcc, and it can be easily installed following the instructions: apt-get install python-pip python-setuptools, pip install wheel && pip install enum && pip install Jinja2, clone the hvcc repo.

            thetechnobear ok, I think the thing here is what warrants a new image?

            that is normally new board support (e.g.: 0.3.4, 0.3.6), or driver updates (0.3.5). Earlier revisions (up to and including 0.3.3 were mostly bugfixes).

            thetechnobear (I think we all recognise installing an image is best avoided, its a pain on salt and some bbb, and also means you have to redo any customisations - hostname changes, wifi setup, installing your dev tools/projects)

            Yes we do. That is why the core code will always keep working with all older version of the image. For instance, this block: https://github.com/BelaPlatform/Bela/blob/master/core/PRU.cpp#L42-L48

            #ifdef BELA_USE_RTDM
#if __has_include(<linux/rtdm_pruss_irq.h>)
#include <linux/rtdm_pruss_irq.h>
#else /* has_include */
#define RTDM_PRUSS_IRQ_VERSION 0
#endif /* has_include */
#endif /* BELA_USE_RTDM */

            In order to support CTAG, we had to greatly extend the rtdm_pruss_irq driver, now adding an include for running ioctls. Earlier versions of the driver did not have the include, so we detect them and we handle that case below. So you see, unless you want CTAG you don't have to update the image, and there is not even the need to install the driver (which we could do with the udpate_board script). (As a side note, v0.2 or earlier images will stlil run because they don't have BELA_USE_RTDM defined.)

            This way we keep each "Bela code" update as safe as possible: no change to sensitive system locations (e.g.: the boot partition, the network configuration, the rtdm driver), which could potentially lead to a bricked board if the update is interrupted (e.g.: by a power shutdown while writing the bootloader, or the kernel modules dependencies file). It an update goes wrong, even if the IDE is screwed, worst case you can still ssh on to the board and run update_board again. We occasionally apply some system-wide changes - besides installing packages -, e.g.: everything in the resources/stretch/rootfsfolder, but these will not compromise system stability if something goes wrong. In principle most of the changes made by the image builder (except adding apt-get packages), could be probably applied here, but - as mentioned - we stay clear of the non-critical ones.

            thetechnobear if I look to Organelle/Norns, basically an image is never usually required, you can simply do an update.. this will do things like run scripts to install packages, update configuration etc.

            We try the same; with the limitations above. That's why, e.g.: libpd/Sc/Csound packages come as part of the repo.

            thetechnobear so i think the update release, should be a clone (depth 1) of the repo, so then on the boards its a valid repo, which you can do git pull on.

            That'd be great, but then there is need for a server that packs the cloned repo up in a zip archive. Assuming you could get a cloned repo on the board, then the update process would be straightforward. Actually, if we could take that for granted, I'd love to have it so that there are a number of subrepos all tied together by the main one.

            As I see it now, I think these are the practical, low-effort actions we could take:

            Update:
            - modify update_board so it detects whether you are within a git repo or not. If you are in a git repo, run git push to the board, then run the rest of update_board (skipping file copying), to run the post-install stuff. This will always give you a clean repo after update
            - if the zip archive that gets uploaded has a hash (as mentioned above), grab it and store it. If running update_board from within a git repo, delete the stored hash.
            Display:
            - always display the date of last update (needs to be cached)
            - if there is a stored hash, display that. Otherwise, use git describe --tags --dirty to display the version number.
            - document the above in the wiki

                      cool, so basically, for existing hardware its unlikely an image update is required..
                      (unless an improvement is found or xenomai was being update?)
                      and sounds like hvcc was a bit of an exception (no worries, just trying to figure it out)

                      personaly I don't think apt-get or pip installs are anything to worry about (anymore than deb pkg) in an update script - you just have to be a bit careful of ensuring you know dependencies, so that you provide these files as well. (and install in order ) - this can be a bit fiddly, but not too problematic if you do on a clean board.

                      on the organelle ive done a few automated pip installs (and also using pacman, since its arch linux)
                      (even downgrading reinstalling where necessary 😉 )

                      e.g. https://github.com/TheTechnobear/OTC/blob/master/OTC_Install/deploy.sh

                      cool, id love a bit more info on the wiki about how to do the updates 'manually'
                      if I can simply go to any of my boards (assuming network connectivity) and do :
                      git pull ; update_board ;
                      and know that's full update (except drivers), im more than happy 🙂

                      thanks again

                        thetechnobear if I can simply go to any of my boards (assuming network connectivity) and do :
                        git pull ; update_board ;
                        and know that's full update (except drivers), im more than happy 🙂

                        Yes that's it.

                          thetechnobear cool, so basically, for existing hardware its unlikely an image update is required..

                          correct.

                          thetechnobear and sounds like hvcc was a bit of an exception (no worries, just trying to figure it out)

                          yeah. It was more of a "extra-bonus": I'd still recommend users to run it on the host as it's much faster.

                          thetechnobear personaly I don't think apt-get or pip installs are anything to worry about (anymore than deb pkg) in an update script - you just have to be a bit careful of ensuring you know dependencies, so that you provide these files as well. (and install in order ) - this can be a bit fiddly, but not too problematic if you do on a clean board.

                          I see from your script how to install pip packages offline. I like how apt-get takes care of the dependencies, down to the exact version number. I think it's a powerful system and I would like to leverage it where possible/needed, to avoid extra headache

                                  giuliomoro yeah. It was more of a "extra-bonus": I'd still recommend users to run it on the host as it's much faster.

                                  yeah, I think that's what I'm going to do...

                                  giuliomoro I like how apt-get takes care of the dependencies, down to the exact version number. I think it's a powerful system and I would like to leverage it where possible/needed, to avoid extra headache

                                  im sure you'll find that apt-get can still do this from files (similar to pacman),
                                  when you install from files/directories its just using this as a 'cache' , it still follows the dependency chain, and expect these to be present in the 'cache' (rather than going off to an online repo)

                                  if its similar to pacman,
                                  a) creating installer - you can tell it to download the required packages (and dependencies) to a directory use as the basis for your
                                  b) running installer - in your update script tell it to pickup packages from a directory - it will still use the dependency info , so if you got the wrong version or forgot a package (though you woudn't if you did step a 🙂 , then the installer would still complain when trying to install the derivative package.

                                  of course, if you want to go the next step,
                                  you could even make the software update that you release a deb/apt package,
                                  that could be an interesting option.. and is not that difficult, they basically are a bit of meta data, file collection and an installer script that gets run.
                                  the advantage is this 'handles' your versioning, can deal with dependencies, and even allows a user to rollback an install.
                                  I think once done once its fairly straightforward, but not sure if worth the initial effort?!

                                        thetechnobear a) creating installer - you can tell it to download the required packages (and dependencies) to a directory use as the basis for your

                                        I think I do something like this over here, where I ask apt-get for the URLs of the packages, then copy them across and installing with dpkg (I guess I could place them in apt-get's cache folder and install using apt-get, though it does not make much difference: dpkg can sort the order of the packages out for itself so that dependencies are satisfied when it is given a list. Again, this requires running on the host and having a given set of tools available there (which rules out, e.g.: default Mingw installations on Windows). Also, this is not quite 100% working because I'd need to run the equivalent of apt-get update as well, to get up-to-date URLs from the apt-get install --print-uris command: if a package has been updated/fixed upstream, its URL may have become invalid.

                                        thetechnobear you could even make the software update that you release a deb/apt package,

                                        That's an interesting one, too.