I have eQEP2 working properly with my EC11 rotary encoder
In your overlay PB-eqep2-00A0.dts here you have the following line
count_mode = <1>; // count_mode is not userspace op_mode
// 0 -> Quadrature mode, normal 90 phase offset cha & chb.
// 1 -> Direction mode. cha input = clock, chb input = direction
// 2 -> UP count mode for frequency measurement QDIR=1, ignore direction input
// 3 -> DOWN count mode for frequency measurement QDIR=0, ignore direction input
I changed this to
count_mode = <0>;
and now the encoder behaves very well.
The strange thing is that when even when count_mode was set to 1 in the dts, the system would report that it was in mode 0 (absolute) , and sort of behaved like it was in absolute mode but with a lot of noise, or contact bouncing
/sys/devices/platform/ocp/48304000.epwmss/48304180.eqep# cat mode
I tried to set the absolute mode in my code, but still the same behavior until I changed the dts to count_mode = <0>;
I could set it to relative mode with this command with either 0 or 1 in the dts file, and the output behaved differently (count returned to zero after each movement)
I don't know enough to understand exactly what you mean by the comment you made on that line in the dts "count_mode is not userspace op_mode", but I suspect that it may be relevant to what I have been seeing.
Now to figure out eQEP0....