Higher sensitivity to spurious electric fields (noise) with larger pot values. A small capacitor to ground at the analog input and tidy wiring will pretty much eliminate that concern. Alternatively a digital equivalent of an RC filter if you don't want to spend money on capacitors (I'm using digital filtering).
My prototype has a mix of pot values from 10k to 1 Meg with no discernable difference in performance on each channel. The selection of values was a consequence of using what I have on hand -- no real reason, but it does demonstrate that just about any pot value will work.
Leakage current on the analog in op amp inputs would become a concern much greater than 1 Meg, but I think 1 Meg is ok.
