From what I recall of quantization and inverse quantization:
Quantization: Quantized coeff = Floor(DCT coeff/QM coeff)
Inverse Quantization: Reconstructed coeff = (Quantized coeff + 0.5) * QM coeff
The problem arises with the rounding error involved in the floor and using the midpoint value of each coefficient to reconstruct.
Consider the DCT coefficients 75, 88, and 99 with a QM coefficient of 25. Each will result in a quantized coeffient of 3 and each will reconstruct to 87.5. As you can see, there is significant error.
However, I don't visualize well in the frequency domain and don't know how much nor exactly the type of effect it would have in the spacial domain (after iDCT).
I guess at least one of the lessons that should be taken away is that you should be really careful with putting really high coefficients in a QM, since large coeffs will result in large error (when the QM fails to zero that particular coefficient). If you do want to simply zero a certain number of the highest frequency values, you might want to have a look at the avisynth based DCTfilter (
http://forum.doom9.org/showthread.php?s=&threadid=38539 and
http://forum.doom9.org/showthread.php?s=&threadid=45695) then use a "normal" quant type.
I think quantization error is one of the reasons QMs look like they do (smaller values for low freq and increasingly larger for higher freq). You need to have the most accurate reconstruction possible of the most important coefficients (thus small quants/low error for low freq), whereas it is less important for the less important high freq coeffs (high quants for high freq with hopes of zeroing).
I hope this isn't too confusing
