Decoupling capacitors are useful for generating audio but not for CVs where you want response down to DC.
One way to get a bipolar output from a unipolar DAC is to use an op-amp mixer circuit. This is an inverting amplifier configuration with one feedback resistor and two input resistors. One of the inputs goes to the DAC, the other goes to a low-noise, regulated negative voltage, say -5V. The op-amp sums the two signals together at the output. Try, say, a 20k feedback resistor, 20k to -5V, and 10k to the DAC. This will turn the 0-5V range of the DAC into +5V to -5V (inverted), i.e. 0V from the DAC = +5V out, 2.5V from the DAC = 0V, 5V from the DAC = -5V out.