Where is your audio coming from? The main advice would be to avoid ground loops (e.g.: don't run audio from your laptop's output into the Bela input while Bela is connected to the laptop via USB). Second comes: use a large-ish signal if you can (e.g.: 1Vpp) . While you can rely on the on-board preamplifier for small gain adjustments, you will hear an unhealthy amount of noise if you run it above +20dB or so (these are the "PGA Gain" controls in the IDE). If you only need one mono input from a low-level source (e.g.: a dynamic mic), you could consider plugging the Hot signal of the balanced connection to L and the Cold to R and then take the difference between the two in software, effectively making a balanced input. This boosts your SNR and gets rid of the correlated noise.
spoitras find a better power supply (low ripple)
bad power supplies will do bad. OK power supplies (and good ones) will be fine, as long as you don't use the 5V directly in your audio or sensor circuits. For instance, don't use it to provide voltage references in your circuit (get a proper voltage reference and buffer it with an opamp in that case, or use the 3v3 line from Bela if it's good enough for your applicatoin.
spoitras add a choke and capacitors to smooth out the power supply
Unless it's a very bad one, or you are using it for voltage sources in your circuit, this shouldn't be needed.
spoitras use shielded cables to connect to Bela
It always helps, although it's not too bad without, unless you have a very high impedance source (e.g.: guitar pickup), but anyhow Bela's input is low-ish impedance (20k to 80k).
spoitras use switching jacks on project enclosure to ground inputs when unused
another thing that always helps. You can test for yourself: run the passthrough example without any input connected and with the input shorted to ground. You can probably hear the difference.
spoitras lower PGA gain and use a louder signal input
spoitras use a grounded project enclosure
It can help, though its effect may depend on the impedance of your input source.
spoitras I'm also wondering about the CTAG cape and wether I would be happier with its sound quality? Is that the low-hassle solution to better sound on Bela?
The CTAG 's codec has better specs and so it's a bit better (still with the limitations of running on top of a single-board computer). It also performs better in terms of the ground loop with the USB cable to the host (for reasons that are not entirely clear to me), but it has no built-in preamp or headphone output.