I think I fixed this now (sorry a bit late for your application). If you update your board to the latest
dev branch of the Bela repo, then it should work for you. The issue was that the roundtrip latency was not fixed to the nominal level (blocksize * 2 + 128), but it started at a smaller value, depending on the CPU load, and then increase towards the nominal value (in steps of 128 or multiples) every time you had a CPU spike. Now it will stay at the nominal value throughout and when an underrun occurs it will not change.