Tool for codecs comparison (Release 1.4 + 3 metrics as plugins under LGPL)

[DmitriyV2]

MSU Video Quality Measurement Tool!

Last news:
Version 1.4 released! (http://www.compression.ru/video/quality_measure/vqmt_download_en.html)

Released several metric plugins in source code:

* MSU Brightness Independent PSNR. (http://www.compression.ru/video/quality_measure/metric_plugins/bi-psnr_en.htm) (with source code under LGPL)
Main idea: It's common situation during testing (we made big
amount of tests), when codec change brightness a little. This metric
is compensate such changes for ANY brightness modification (up
to negative frame).
Also speed optimizer's are welcome!

* MSU Brightness Flicking Metric. (http://www.compression.ru/video/quality_measure/metric_plugins/bfm_en.htm) (with source code under LGPL)
Main idea: Simple flicking calculation (can be enhanced).

* MSU Drop Frame Metric. (http://www.compression.ru/video/quality_measure/metric_plugins/dfm_en.htm) (with source code under LGPL)
Main idea: Some codecs on low bitrates drop more frames, some -
less. It's good idea to calculate not only PSNR/SSIM but also number
of dropped frames as codec characteristic for low bitrates. Also this metric is
useful for capture device/program evaluation.

Now you can create your own metrics, e.g.:
* Faster metrics (using MMX-SSE2)
* New special measures (blocking, interlacing, etc)
* New perfect formal metrics (again better than PSNR) :)
You are welcome!


Metric supports different input formats (AVI, AVS, YUV, BMP) and metrics (PSNR, Delta, VQM, SSIM, MSU Blurring/Blocking Measure). Main features: results are saved in *.CSV (can be opened everywhere), all metrics are visualized, comparative analysis (comparison of three files: original, and two other, for example processed by different codecs).

Program allow to answer on questions:
1) One codec has more bluring than another. On what frames? Average mark?
2) One codec has more blocking than another. On what frames? Average mark?
3) One codec has more lower quality than another. On what frames? Average mark in PSNR, VQM, SSIM?

Correspondent "bad frames" are saved automatically into BMP files.

Direct links: Metric home page (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html), link to program (http://www.compression.ru/video/quality_measure/src/msu_vmt.zip).
Plugin's pages: Brightness Independent PSNR (http://www.compression.ru/video/quality_measure/metric_plugins/bi-psnr_en.htm), Brightness Flicking Metric (http://www.compression.ru/video/quality_measure/metric_plugins/bfm_en.htm), Drop Frame Metric (http://www.compression.ru/video/quality_measure/metric_plugins/dfm_en.htm).


ChangeLog:

1.4
(*) Bugfixing in BMP processing (visualization saving, etc)

1.3
(+) Installation added
(*) Bug with YV12 yuv files fixed

1.2
(+) Now it is possible to compress visualization
(+) Plugin mechanism released
(!) Problem with some DV codecs is known now

1.0
(+) More YUV filetypes are supported, including YV12, YUY2, YUV
(+) Supports unicode
(+) Visualization dialog was extensively reworked
(*) Interface is more user-friendly

0.81
(+) New AVI Reading system (support large AVI Files, VP 70)
(*) Bugfixing in final dialog

0.8
(+) New dialog with visualisation of the comparison and for comparison of the selected frames added.

0.75
(+) Improved codecs support (x264)
(+) Now data from YUV-AVI files is extracted without convertion
(*) Bugfixing: #NAN in VQM calculation, calculation of RGB-metrics for YUV-files
(*) MSU Blocking Measure changed

Ver 0.74
(+) MSE, MSAD metrics added
(+) Saving of averange value of the metric added
(+) Improved codecs support (DivX3, WMV, mjpeg2000)
(*) Delta metric was changed
(*) Bugfixing with AviSynth

Ver 0.73
(+) All colorspaces from AVISynth are supported
(+) I420 (IYUV) support added
(+) XviD and B-frames support added
(*) Bugfixing (YUV-files, SSIM)

Ver 0.72
(+) AviSynth support added!

Any comment and ideas about future metric development are welcome!

[Sharktooth]

great tool, thanks a lot!
a nice addition would be avisynth support.

[Sirber]

+1 for avisynth

[DmitriyV2]

great tool, thanks a lot!
Thanks! :)

a nice addition would be avisynth support.
Ok, we think about avisynth.

[hpn]

Until avisynth support is added to this sweet tool, one can just use "MakeAvis.exe" that comes bundled with ffdshow (check in your folder X:\Program Files\ffdshow\) to "wrap" any avs scripts, creating small fake AVI files that the "MSU Video Measurement" can work with.

[Sharktooth]

i know but avisynth support is more elegant.

[hpn]

Definitely :)

[unskinnyboy]

Much useful tool, Thanks! :-)

[hpn]

Until avisynth support is added to this sweet tool, one can just use "MakeAvis.exe" that comes bundled with ffdshow (check in your folder X:\Program Files\ffdshow\) to "wrap" any avs scripts, creating small fake AVI files that the "MSU Video Measurement" can work with.

Forgot to mention that to see MakeAvis.exe one have to enable "AviSynth scripts serving" during ffdshow installation (it's disabled by default, I don't know why :confused: ):

http://img212.echo.cx/img212/5176/avisynthscriptsserving0kb.png

[DmitriyV2]

Much useful tool, Thanks! :-)Thanks!

"Twin" tool "MSU Human Perceptual Quality Measurement" (that will complete this formal measurement) will be released soon (hope till the end of June).

[dimzon]

WOW!!!
Awesome tool!

[TEB]

yea luve it.. but plz add avs support and 4:2:2 support

[Sharktooth]

TEB can you please stop asking for 4:2:2 support for every app/codec in every thread?

[TEB]

hehe why? i need it at work.. 4:2:2 storage of film...

[Sharktooth]

ok, but it's quite annoying having to read always the same thing in different threads.

[DmitriyV2]

Metric with AVS support will be released in a week approximately!
(Due to mass request :)).

[Sharktooth]

:eek:

[IgorC]

Metric with AVS support will be released in a week approximately!
(Due to mass request :)).

it would be GREAT!!

[acidsex]

Metric with AVS support will be released in a week approximately!
(Due to mass request :)).

Way sweet. Much thanks from me and many others I am sure. :)

[foxyshadis]

I'm curious if you know what could cause the error "Failed to open the AVI frame". I get it for most videos, even when I compare the same video to itself. Every other video application I have works fine. I have XP SP2.

Anyway, it looks very cool, hopefully by the time it comes out of beta I'll be able to use it.

[Edit: It turns out that it only happens for those videos decoded via xvid through vfw (which was all mpeg4 before). I'm trying to get ffdshow to take over xvid's so I can see how it works.]

[Edit2: Working fine using divx or ffdshow as the vfw handler. Seems to be an incompatibility in xvid, I've heard of it not working in some apps like Vegas.]

[CruNcher]

Hiya Dmitriy really a great new Tool from your Labs thx for it to the whole Team :) do you think the Samsung Deinterlacer will ever become publicly available or is there no chance that this will ever happen ? and will the new Motion Estimated Logo Removal thats announced for September also be faster ?

[DmitriyV2]

[Edit: It turns out that it only happens for those videos decoded via xvid through vfw (which was all mpeg4 before). I'm trying to get ffdshow to take over xvid's so I can see how it works.]

I't very strange, but looks like we have problems with xvid in this version. We reproduce situation (thanks for bugreport!) try to understand why???

[DmitriyV2]

Hiya Dmitriy really a great new Tool from your Labs thx for it to the whole Team :)
Thanks!

do you think the Samsung Deinterlacer will ever become publicly available or is there no chance that this will ever happen ? and will the new Motion Estimated Logo Removal thats announced for September also be faster ?
It's looks like Samsung Deinterlacer will ever become publicly available only in Samsung plasma panels and TFT TV sets... And we already have new better version of MSU Logo Removal but there is some restrictions in contract.
So we publish another filters for free, not this...

[Yama4050242]

what does error "failed to open The AVI frame" mean?

[DmitriyV2]

what does error "failed to open The AVI frame" mean?
Problem with XviD. We try to understand problem and fix it.

[DmitriyV2]

Metric 0.72 released!

We add AviSynth support to Metric Tool! Please test!

Direct links: Metric home page (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html), link to program (http://www.compression.ru/video/quality_measure/src/msu_vmt.zip).

[Yama4050242]

the link to the program still 0.71?

[Sharktooth]

It's 0.72... Avisynth seems to work correctly but i dont understand why it requires RGB24...
however there's a memory leak in SSIM test.

[DmitriyV2]

the link to the program still 0.71?
Yes, the same, but program was updated

[DmitriyV2]

It's 0.72... Avisynth seems to work correctly but i dont understand why it requires RGB24...
however there's a memory leak in SSIM test.
We working on this. Thanks for report!

[DmitriyV2]

We release version 0.73 of Metric! (Fixes with Xvid, added AviSynth colorspaces, added B-frames support) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Please test with Xvid and AviSynth colorspaces!

Enjoy! :)

[Sharktooth]

wow, thanks!

[DmitriyV2]

We release version 0.74! (added MSAD+MSE, support of DivX3, WMV, mjpeg2000, AviSynth fixes) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Enjoy! ;)

[trott]

I have the impression I cannot open avi files encoded with DV with your tool. (failed to open first video file)

[DmitriyV2]

I have the impression I cannot open avi files encoded with DV with your tool. (failed to open first video file)
New release will be 5 Dec (approx.), with many fixes in AVI loading part. Now please use AviSynth.

[ggab]

and how about the status of "MSU tool for human perceptual codecs comparison" ?
http://forum.doom9.org/showthread.php?t=99788

i would like to see a new version of this great program :)

[Sirber]

Adding to RealAnime LE, if you don't mind :)

[DmitriyV2]

and how about the status of "MSU tool for human perceptual codecs comparison" ?
http://forum.doom9.org/showthread.php?t=99788
i would like to see a new version of this great program :)
Thank you! Now we already test new version. Hope soon we release it!

[DmitriyV2]

We release version 0.75! (Improved codecs support (x264), data from YUV-AVI files is extracted without convertion, bugfixing(VQM, RGB metrics for YUV-files)) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Any comment and ideas about future metric development are welcome!

Enjoy! ;)

[Sirber]

can't connect to your website :(

The connection to the server was reset while the page was loading.

[DmitriyV2]

can't connect to your website :(
The connection to the server was reset while the page was loading.
10 Dec there was problems with provider! Try now please!

[quake74]

Cannot connect today (12/12).

[Eretria-chan]

Works for me (13:01 GMT 12/12/05)

[Sirber]

Works now, thanks!

[DmitriyV2]

Last news: 08 March - version 0.81 released! (New AVI Reading system (support large AVI Files, VP 70), Bugfixing in final dialog) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Enjoy! :)

[foxyshadis]

It is very useful. Some things I find odd: You cannot click on bottom graph to more easily click a specific frame, the last value of every metric is nonsense that leads to a silly end of the graph and changes the real "average" value. Maybe my codec is just chopping off the last frame, I'm not sure.

Still excellent!

[DmitriyV2]

It is very useful. Some things I find odd: You cannot click on bottom graph to more easily click a specific frame, the last value of every metric is nonsense that leads to a silly end of the graph and changes the real "average" value. Maybe my codec is just chopping off the last frame, I'm not sure.
Hm! Support of click - looks resonable! Thanks for idea!

About last frame - we test metric pretty long, but now AVI reading system was changed, so any bugreports on AVI reading may be very useful!

Still excellent!
Welcome! :)

[DmitriyV2]

Last news: 04 May - version 1.0 released! (More YUV modes are supported, interface and visualization SERIOUSLY reworked, bugfixing) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Enjoy! :)

[foxyshadis]

Awesome! I've been hoping for this release for a while. =D Maybe in the future you could have a way to load previous results, for comparing quickly multiple videos. The visualization is quite nice and usable now, and I wish I had something like that in Calc/Excel for charts!

Are you open to telling how blurring measure is calculated? I'm quite interested in measures of absolute quality, rather than comparisons to a source - partly that's because I work with a lot of videos that start badly and need too much filtering. Plus it would be nice to someday have a tool that can definitively say "this needs rainbow removal" or "this needs mild decombing", which an aplication could turn into filter recs, so newbies can do advanced processing without a guru's help. ...But I suppose that's way beyond the scope of this tool, sorry.

I wonder how one would go about turning these spatial measures into temporal measures to better match human perception, though. That would be quite a trick. (Perhaps only measurement averages need to be reweighted somehow.)

[siddharthagandhi]

It would be nice if there was an autocorrect feature that would correct any frame correspondence discrepancies.