[Asmodian]

Because as soon as you change any settings you end up with different average bitrates or file sizes.

With different size files how do you judge the results? If one file is larger it might look only a little bit better, so the setting is equivalent to lowering CRF a tiny bit (performance becomes the deciding factor). It might instead look a lot better and the setting improves quality even if a slightly higher CRF was used (a good setting, it could be worth lower performance). It could be worse too, increasing both the size and quality but making the encode less efficient, decreasing the CRF value slightly would have produced better quality at the same size. It is really hard to judge if the quality improved enough compared to the bitrate increase to be able to separate these. The same is true with a smaller file, is it simply slightly worse quality? The option might make the encode less efficient while also lowering the size, so is worse then simply raising CRF slightly.

The only situation you can judge with any accuracy would be if the larger file was worse quality. The only accurate way is to compare quality at the same average bitrate.

As an aside, be careful of judging still frames too. Good quality comparison is actually really hard. I can end up focusing on one aspect and tuning to improve that while missing some other aspect getting worse at the same time. I then come back in a few years and feel like an idiot, all that time figuring out the perfect settings and in total they were probably slightly worse than the preset would have been. Especially when I had less experiance but it is still really easy to do. I use more aggressive compression and two pass encoding when testing settings and then run high bitrate CRF encodes after I have them figured out.