Another take on quantization matrices - DCTCheck plugin for AviSynth

[Shalcker]

More then a month ago in a thread named "Understanding a Quantization Matrix" was interesting discussion which sadly rather quickly faded...

But people mentioned there that to create good custom matrix we should look at values entering quantization, and that previewing results for a given quantizer can be helpful too... I've been thinking about that for a while and it looked like rather simple thing to do, at least for keyframes.

Therefore I created AviSynth filter for that purpose.
It can:
- output various data about processed frame into specified text file
and
- process frame in a way close to actual MPEG4 processing (at least i hope so), with frequency data quantized according to specified quantizer and quantization matrix, allowing you to note all artifacts quantization may introduce to picture.

Here it is, available for download, sources included - DCTCheck v0.4.1 (http://web.etel.ru/~shalcker/DCTCheck_v041.zip)
Only accepts YV12 and mod 16 width and height frames.

Posting this to AviSynth board seems pointless since only place where you can use custom quantization matrices is XviD...

Usage is explained in detail within DCTCheck.txt in archive.

----
Few quick examples:
DCTCheck(clip, Quant=2, idct=2, Matrix="c:\custom_matrix.mtx")
should result in output frame close to actual keyframe with quantizer 2 and custom matrix from custom_matrix.mtx (unless custom matrix is specified quantization matrix defaults to all-8). You can use output of the filter to check difference with original frame by using Subtract or ConvertToYUY2() and Compare.

DCTCheck(clip, "FYCKTtMmXxNnEe", Quant=10, Output="c:\temp\dctcheck.txt")
should result in frame passed unchanged and some data about DCT values before and after quantization written into file dctcheck.txt

DCTCheck(clip, "FYCKTtIAa", Quant=2.5, idct=2, Matrix="c:\custom_matrix.mtx", Output="c:\temp\dct_blocks.txt")
should result in big dct_blocks.txt with tables for all DCT blocks in frame (beware - all blocks for each frame processed, several megabytes of text data per frame) as well as "original" pixel blocks (before quantization) and "output" pixel blocks (close to keyframe at quantizer 2 since idct=2).

---
DCTCheck is slow and written in pure C++ without a single bit of mmx code, and uses only fdct_int32 and idct_int32 from XviD sources.

Delta-frame modes are extremely dumb (no motion compensation) and is only there to provide you with possible examples of data for Inter/Deltaframe.

Frame Quantizer accepts float values, so you can easily scale your matrix as much as you want. Quantizer value is not limited to 31.

----
You should note that DCT process is not loseless by itself, and will introduce rounding errors because precision of DCT result is limited to integer values (and higher precision is required to make this process close to loseless). Different DCT methods may introduce a bit different errors as it was discussed in several threads lately.

In my tests with AviSynth performing DCT and then iDCT without any quantization (idct=1 for DCTCheck) results in 56-64dB PSNR.

There were a lot more tests but i think this post is long enough as it is.
----

Any suggestions are welcome.

[Nic]

Why aren't you using the NASM'd Assembly routines of XviD for the DCT? If you need assistance putting them into VC6 Workspace just post for help :)

-Nic

[Lefungus]

Cool, now i see two things you can add ;)

-A psnr test or any other video quality metric. There are some metrics that works in the DCT domain.

-A brute force algorithm that test all matrix possibilities, and give the one with best psnr results. DCTTune do something like that i think

[Shalcker]

Originally posted by Lefungus
Cool, now i see two things you can add ;)

-A psnr test or any other video quality metric. There are some metrics that works in the DCT domain.

If you can point me to sources for such metrics i can add them.

You can always use AviSynth Compare function to obtain PSNR, or use your VQMCalc filter to check another metric (as long as there is no image transformation between them) :)


-A brute force algorithm that test all matrix possibilities, and give the one with best psnr results. DCTTune do something like that i think
There is no need for brute force test to understand which matrix gives best PSNR result. Certainly that's all-1 at any quantizer. ;)

Figures may start to differ as quantizer and matrix values rise though.

And brute force approach to matrix optimization is... well... too long. If you wish to test all matrix possibilities you'll have to do something around factorial of 64 process/compare... that's around 10^150... we can cut that alot but even a few millions is a bit too much, don't you think so? :)

Maybe we can optimize each cell value independently based on average and/or peak error. That'll reduce us to a few thousands of comparisons in the worst case... got to try that out :)

[MfA]

Lefungus, 256^64 is a rather large number ... brute force aint going to work :) Designing optimal matrices should be done using statistics from real frames though, the statistics of real motion compensated frames are bound to be substantially different.

You can find some source code for a program which among other things calculates near optimal quantization tables, using a fast statistical approach using coefficient magnitude histograms, for JPEG here (ftp://ftp.cs.wisc.edu/pub/ratnakar/) (at a given lambda of course, that is how R-D optimization works).