I cannot reproduce the issue you are having. Depending on the encoder you have you may need to change the line
encoders.push_back({0, Encoder::ANY});
to
encoders.push_back({0, Encoder::ACTIVE_HIGH});
You could also try increasing the debouncing
usleep(500);
to something like 1000 (these are microseconds, so that would be 1 millisecond). If these still don't make it good enough for you, the next thing would be to add debouncing capacitors to the board. I seem to remember it doesn't have any, which is really lame. Are the schematic and PCB layout available anywhere?
