Does the same thing happen on both left and right channels? If yes then the odds of a bad component are reduced. Also, try measuring the headphone output under load (e.g. a 100 ohm resistor), and/or measuring the signal at J9.
A small DC offset at the input, even after the input filter caps, is not necessarily impossible. I would expect it to be in the region of .05 or less at rest, and probably much smaller (.01 or less). Anything bigger than 0.1, or anything substantially different between left and right channels, might indicate a component problem. The measure_noisefloor example will show you the DC level of each pin calculated over a window.
Don't necessarily trust the FFT to tell you a DC offset though. For any short-time window, the lowest bin will encompass a range of very low frequencies, and because of windowing I would expect it to show some energy in that bin even if there isn't a "real" DC offset. The best way to measure the actual DC offset is to take a mean over several seconds, ideally with no signal at all.