For a battery, you may want to use lipo cell, these come in all sort of sizes.

To get estimated battery duration, get the battery's advertised mAh and divide it by 1000 times the value from the table above for the use case you are targeting (most likely audio + 1 core load).
Speaker amps, see what adafruit and sparkfun have, keeping in mind that having an option for differential input may be better than having single ended inputs. Power amps may require larger voltages than provided by single-cell lipos, so you either need a dedicated battery for those or a power booster. If possible, avoid sharing the power between Gem and the power amp, unless the power amp has a differential input.
For speakers, I don't have any specific recommendations as it really depend on size and power needed.