Are your rendering times good? If so don't mess with scaling options.
It sounds like you are not actually using the custom modes, this has happened to others as well. Does madVR's report of the refresh rate stay the same too? The decimal points should change as you tune a custom resolution. Getting the Nvidia drivers to actually switch to the custom mode can be tricky, try driver reinstalls with the start fresh option enabled...