Tool for codecs comparison (Release 1.4 + 3 metrics as plugins under LGPL)

[DmitriyV2]

MSU Video Quality Measurement Tool!

Last news:
Version 1.4 released! (http://www.compression.ru/video/quality_measure/vqmt_download_en.html)

Released several metric plugins in source code:

* MSU Brightness Independent PSNR. (http://www.compression.ru/video/quality_measure/metric_plugins/bi-psnr_en.htm) (with source code under LGPL)
Main idea: It's common situation during testing (we made big
amount of tests), when codec change brightness a little. This metric
is compensate such changes for ANY brightness modification (up
to negative frame).
Also speed optimizer's are welcome!

* MSU Brightness Flicking Metric. (http://www.compression.ru/video/quality_measure/metric_plugins/bfm_en.htm) (with source code under LGPL)
Main idea: Simple flicking calculation (can be enhanced).

* MSU Drop Frame Metric. (http://www.compression.ru/video/quality_measure/metric_plugins/dfm_en.htm) (with source code under LGPL)
Main idea: Some codecs on low bitrates drop more frames, some -
less. It's good idea to calculate not only PSNR/SSIM but also number
of dropped frames as codec characteristic for low bitrates. Also this metric is
useful for capture device/program evaluation.

Now you can create your own metrics, e.g.:
* Faster metrics (using MMX-SSE2)
* New special measures (blocking, interlacing, etc)
* New perfect formal metrics (again better than PSNR) :)
You are welcome!


Metric supports different input formats (AVI, AVS, YUV, BMP) and metrics (PSNR, Delta, VQM, SSIM, MSU Blurring/Blocking Measure). Main features: results are saved in *.CSV (can be opened everywhere), all metrics are visualized, comparative analysis (comparison of three files: original, and two other, for example processed by different codecs).

Program allow to answer on questions:
1) One codec has more bluring than another. On what frames? Average mark?
2) One codec has more blocking than another. On what frames? Average mark?
3) One codec has more lower quality than another. On what frames? Average mark in PSNR, VQM, SSIM?

Correspondent "bad frames" are saved automatically into BMP files.

Direct links: Metric home page (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html), link to program (http://www.compression.ru/video/quality_measure/src/msu_vmt.zip).
Plugin's pages: Brightness Independent PSNR (http://www.compression.ru/video/quality_measure/metric_plugins/bi-psnr_en.htm), Brightness Flicking Metric (http://www.compression.ru/video/quality_measure/metric_plugins/bfm_en.htm), Drop Frame Metric (http://www.compression.ru/video/quality_measure/metric_plugins/dfm_en.htm).


ChangeLog:

1.4
(*) Bugfixing in BMP processing (visualization saving, etc)

1.3
(+) Installation added
(*) Bug with YV12 yuv files fixed

1.2
(+) Now it is possible to compress visualization
(+) Plugin mechanism released
(!) Problem with some DV codecs is known now

1.0
(+) More YUV filetypes are supported, including YV12, YUY2, YUV
(+) Supports unicode
(+) Visualization dialog was extensively reworked
(*) Interface is more user-friendly

0.81
(+) New AVI Reading system (support large AVI Files, VP 70)
(*) Bugfixing in final dialog

0.8
(+) New dialog with visualisation of the comparison and for comparison of the selected frames added.

0.75
(+) Improved codecs support (x264)
(+) Now data from YUV-AVI files is extracted without convertion
(*) Bugfixing: #NAN in VQM calculation, calculation of RGB-metrics for YUV-files
(*) MSU Blocking Measure changed

Ver 0.74
(+) MSE, MSAD metrics added
(+) Saving of averange value of the metric added
(+) Improved codecs support (DivX3, WMV, mjpeg2000)
(*) Delta metric was changed
(*) Bugfixing with AviSynth

Ver 0.73
(+) All colorspaces from AVISynth are supported
(+) I420 (IYUV) support added
(+) XviD and B-frames support added
(*) Bugfixing (YUV-files, SSIM)

Ver 0.72
(+) AviSynth support added!

Any comment and ideas about future metric development are welcome!

[Sharktooth]

great tool, thanks a lot!
a nice addition would be avisynth support.

[Sirber]

+1 for avisynth

[DmitriyV2]

great tool, thanks a lot!
Thanks! :)

a nice addition would be avisynth support.
Ok, we think about avisynth.

[hpn]

Until avisynth support is added to this sweet tool, one can just use "MakeAvis.exe" that comes bundled with ffdshow (check in your folder X:\Program Files\ffdshow\) to "wrap" any avs scripts, creating small fake AVI files that the "MSU Video Measurement" can work with.

[Sharktooth]

i know but avisynth support is more elegant.

[hpn]

Definitely :)

[unskinnyboy]

Much useful tool, Thanks! :-)

[hpn]

Until avisynth support is added to this sweet tool, one can just use "MakeAvis.exe" that comes bundled with ffdshow (check in your folder X:\Program Files\ffdshow\) to "wrap" any avs scripts, creating small fake AVI files that the "MSU Video Measurement" can work with.

Forgot to mention that to see MakeAvis.exe one have to enable "AviSynth scripts serving" during ffdshow installation (it's disabled by default, I don't know why :confused: ):

http://img212.echo.cx/img212/5176/avisynthscriptsserving0kb.png

[DmitriyV2]

Much useful tool, Thanks! :-)Thanks!

"Twin" tool "MSU Human Perceptual Quality Measurement" (that will complete this formal measurement) will be released soon (hope till the end of June).

[dimzon]

WOW!!!
Awesome tool!

[TEB]

yea luve it.. but plz add avs support and 4:2:2 support

[Sharktooth]

TEB can you please stop asking for 4:2:2 support for every app/codec in every thread?

[TEB]

hehe why? i need it at work.. 4:2:2 storage of film...

[Sharktooth]

ok, but it's quite annoying having to read always the same thing in different threads.

[DmitriyV2]

Metric with AVS support will be released in a week approximately!
(Due to mass request :)).

[Sharktooth]

:eek:

[IgorC]

Metric with AVS support will be released in a week approximately!
(Due to mass request :)).

it would be GREAT!!

[acidsex]

Metric with AVS support will be released in a week approximately!
(Due to mass request :)).

Way sweet. Much thanks from me and many others I am sure. :)

[foxyshadis]

I'm curious if you know what could cause the error "Failed to open the AVI frame". I get it for most videos, even when I compare the same video to itself. Every other video application I have works fine. I have XP SP2.

Anyway, it looks very cool, hopefully by the time it comes out of beta I'll be able to use it.

[Edit: It turns out that it only happens for those videos decoded via xvid through vfw (which was all mpeg4 before). I'm trying to get ffdshow to take over xvid's so I can see how it works.]

[Edit2: Working fine using divx or ffdshow as the vfw handler. Seems to be an incompatibility in xvid, I've heard of it not working in some apps like Vegas.]

[CruNcher]

Hiya Dmitriy really a great new Tool from your Labs thx for it to the whole Team :) do you think the Samsung Deinterlacer will ever become publicly available or is there no chance that this will ever happen ? and will the new Motion Estimated Logo Removal thats announced for September also be faster ?

[DmitriyV2]

[Edit: It turns out that it only happens for those videos decoded via xvid through vfw (which was all mpeg4 before). I'm trying to get ffdshow to take over xvid's so I can see how it works.]

I't very strange, but looks like we have problems with xvid in this version. We reproduce situation (thanks for bugreport!) try to understand why???

[DmitriyV2]

Hiya Dmitriy really a great new Tool from your Labs thx for it to the whole Team :)
Thanks!

do you think the Samsung Deinterlacer will ever become publicly available or is there no chance that this will ever happen ? and will the new Motion Estimated Logo Removal thats announced for September also be faster ?
It's looks like Samsung Deinterlacer will ever become publicly available only in Samsung plasma panels and TFT TV sets... And we already have new better version of MSU Logo Removal but there is some restrictions in contract.
So we publish another filters for free, not this...

[Yama4050242]

what does error "failed to open The AVI frame" mean?

[DmitriyV2]

what does error "failed to open The AVI frame" mean?
Problem with XviD. We try to understand problem and fix it.

[DmitriyV2]

Metric 0.72 released!

We add AviSynth support to Metric Tool! Please test!

Direct links: Metric home page (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html), link to program (http://www.compression.ru/video/quality_measure/src/msu_vmt.zip).

[Yama4050242]

the link to the program still 0.71?

[Sharktooth]

It's 0.72... Avisynth seems to work correctly but i dont understand why it requires RGB24...
however there's a memory leak in SSIM test.

[DmitriyV2]

the link to the program still 0.71?
Yes, the same, but program was updated

[DmitriyV2]

It's 0.72... Avisynth seems to work correctly but i dont understand why it requires RGB24...
however there's a memory leak in SSIM test.
We working on this. Thanks for report!

[DmitriyV2]

We release version 0.73 of Metric! (Fixes with Xvid, added AviSynth colorspaces, added B-frames support) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Please test with Xvid and AviSynth colorspaces!

Enjoy! :)

[Sharktooth]

wow, thanks!

[DmitriyV2]

We release version 0.74! (added MSAD+MSE, support of DivX3, WMV, mjpeg2000, AviSynth fixes) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Enjoy! ;)

[trott]

I have the impression I cannot open avi files encoded with DV with your tool. (failed to open first video file)

[DmitriyV2]

I have the impression I cannot open avi files encoded with DV with your tool. (failed to open first video file)
New release will be 5 Dec (approx.), with many fixes in AVI loading part. Now please use AviSynth.

[ggab]

and how about the status of "MSU tool for human perceptual codecs comparison" ?
http://forum.doom9.org/showthread.php?t=99788

i would like to see a new version of this great program :)

[Sirber]

Adding to RealAnime LE, if you don't mind :)

[DmitriyV2]

and how about the status of "MSU tool for human perceptual codecs comparison" ?
http://forum.doom9.org/showthread.php?t=99788
i would like to see a new version of this great program :)
Thank you! Now we already test new version. Hope soon we release it!

[DmitriyV2]

We release version 0.75! (Improved codecs support (x264), data from YUV-AVI files is extracted without convertion, bugfixing(VQM, RGB metrics for YUV-files)) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Any comment and ideas about future metric development are welcome!

Enjoy! ;)

[Sirber]

can't connect to your website :(

The connection to the server was reset while the page was loading.

[DmitriyV2]

can't connect to your website :(
The connection to the server was reset while the page was loading.
10 Dec there was problems with provider! Try now please!

[quake74]

Cannot connect today (12/12).

[Eretria-chan]

Works for me (13:01 GMT 12/12/05)

[Sirber]

Works now, thanks!

[DmitriyV2]

Last news: 08 March - version 0.81 released! (New AVI Reading system (support large AVI Files, VP 70), Bugfixing in final dialog) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Enjoy! :)

[foxyshadis]

It is very useful. Some things I find odd: You cannot click on bottom graph to more easily click a specific frame, the last value of every metric is nonsense that leads to a silly end of the graph and changes the real "average" value. Maybe my codec is just chopping off the last frame, I'm not sure.

Still excellent!

[DmitriyV2]

It is very useful. Some things I find odd: You cannot click on bottom graph to more easily click a specific frame, the last value of every metric is nonsense that leads to a silly end of the graph and changes the real "average" value. Maybe my codec is just chopping off the last frame, I'm not sure.
Hm! Support of click - looks resonable! Thanks for idea!

About last frame - we test metric pretty long, but now AVI reading system was changed, so any bugreports on AVI reading may be very useful!

Still excellent!
Welcome! :)

[DmitriyV2]

Last news: 04 May - version 1.0 released! (More YUV modes are supported, interface and visualization SERIOUSLY reworked, bugfixing) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Enjoy! :)

[foxyshadis]

Awesome! I've been hoping for this release for a while. =D Maybe in the future you could have a way to load previous results, for comparing quickly multiple videos. The visualization is quite nice and usable now, and I wish I had something like that in Calc/Excel for charts!

Are you open to telling how blurring measure is calculated? I'm quite interested in measures of absolute quality, rather than comparisons to a source - partly that's because I work with a lot of videos that start badly and need too much filtering. Plus it would be nice to someday have a tool that can definitively say "this needs rainbow removal" or "this needs mild decombing", which an aplication could turn into filter recs, so newbies can do advanced processing without a guru's help. ...But I suppose that's way beyond the scope of this tool, sorry.

I wonder how one would go about turning these spatial measures into temporal measures to better match human perception, though. That would be quite a trick. (Perhaps only measurement averages need to be reweighted somehow.)

[siddharthagandhi]

It would be nice if there was an autocorrect feature that would correct any frame correspondence discrepancies.

[DmitriyV2]

Maybe in the future you could have a way to load previous results, for comparing quickly multiple videos. The visualization is quite nice and usable now, and I wish I had something like that in Calc/Excel for charts!
We think it's difficult to repeat Excel/MathCad/MathLab visualization functionality :). So we implement instant analysis tool and recomment to use Excel/MathCad/MathLab for saved csv files analysis.

Are you open to telling how blurring measure is calculated? I'm quite interested in measures of absolute quality, rather than comparisons to a source - partly that's because I work with a lot of videos that start badly and need too much filtering. Plus it would be nice to someday have a tool that can definitively say "this needs rainbow removal" or "this needs mild decombing", which an aplication could turn into filter recs, so newbies can do advanced processing without a guru's help. ...But I suppose that's way beyond the scope of this tool, sorry.

We plan to publish metric in paper soon.

Why not? :) We plan include plugin interface into new version. So ANY new metrics will be possible to add.


I wonder how one would go about turning these spatial measures into temporal measures to better match human perception, though. That would be quite a trick. (Perhaps only measurement averages need to be reweighted somehow.)
This is not so simple trick.

[DmitriyV2]

It would be nice if there was an autocorrect feature that would correct any frame correspondence discrepancies.
We plan to public such "correction" plug-in in open source to tune correction method with the help of other clever researchers.

[Livesms]

DmitriyV2

And what about batch support.

Some times it is usefull for comparing lots of files encoded by different codecs. And here is another option - "append only avarage result to a file file.csv" - this option will append to a file file.scv int next line "file.avi; result value". It will be nice for mass coparation.

For example http://forum.ixbt.com/topic.cgi?id=29:25574-2#26 here i should do a monkey job for clicking button and selectiung file each time to compare new variant.

[siddharthagandhi]

The direct input (non-avisynth) has a lot of bugs, many of my video files aren't accepted. That and the auto frame correction features are the two most important features I look foward to.

[WorBry]

Hi Dmitry,

In the on-line information that you provide regarding your denoise filters, you give an example of how LUV measure can be used to first visualize temporal noise, by comparison of adjacent frames in a source sequence, and then the effects of denoising, by comparing a processed frame with the original.

http://www.compression.ru/video/denoising/index_en.html

Could you please clarify which of the metrics (PSNR, MSAD, delta etc) you use for this. I'm assuming PSNR, but just wanted to make sure.

Actually, based on this assumption, I've done some preliminary tests with deinterlaced (50fps) DV test material and found that it has great potential for fine tweaking Temporal-Spatial denoiser filter settings, including the TS denoiser built into your lossless codec.

To do this, I first ran a PSNR L-LUV comparison (with visualization output) using the source clip as reference and the same clip shifted by one frame (trim(1,0)) as the test clip. I then did the same with the processed (denoised) clip, and visually compared the two RGB 'visualization' sequences to see how effectively the filter has eliminated temporal noise. In my tests, temporal noise mostly appears as blue dots. Anything in the 'yellow dot' domain tends to be associated with moving detail and I so I've tried to avoid removing it.

I'm wondering if this visual comparison of original vs processed visualizations could in turn be converted into a measured 'difference' parameter that provides an objective measure of denoise efficiency. What do you think?

I wondering also if it would be possible to provide an option for outputting the visualization frame sequences in a compressed format (maybe YUY2, YV12 or your MSU lossless format) as the uncompressed RGB sequences can get quite large?

Any possibility of LUV-LUV also?

Great tool. Thanks alot.

Edit: Here's an example, the first image showing a visualization frame from a PSNR L-LUV test of the source clip (original vs one frame shift) and the second, the same frame after denoising with hqdn3d(3). RemoveGrain(2). The third image is a straight PSNR L-LUV comparison of original source and processed frame, revealing the noise that has been removed. Pretty hard to make out I know, but the source image at full res was too large to upload. The clip (my son on a baby chair) was shot in fairly poor indoor light.

[WorBry]

Here are the same images inverted to negatives, which show the effects more clearly.

[siddharthagandhi]

You should upload them at imageshack or someplace else because it takes a while for moderators to approve attachments.

[WorBry]

Moderator,

Any chance of my attachments being approved?

Thanks.

[WorBry]

Patient: Doctor, why do people keep ignoring me?
Doctor: Next patient please !

:o

[DmitriyV2]

We are prepare now version with plugins support (and many other fixes), so anybody will be able to write his own metric - for example with GPU optimization or with measurement of some specific issues.

If you want to take part in metric pre-release testing - please let us know on e-mail video-measure(*)compression.ru (or http://www.compression.ru/video/quality_measure/images/video-measure_graphics_cs_msu_su.gif).

[DmitriyV2]

Last news: 20 June - version 1.2 released! (bugfixes + plugins interface(!)) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Now it's easy to add new metric to the tool.

For example:
* Faster metrics (using MMX-SSE2, HW acceleration, DirectShow
* New special blurring (blocking, interlacing, etc) measures
* New perfect formal metrics (again better than PSNR :))

Any ideas about future metric development are welcome!

Enjoy! :)

[DmitriyV2]

Last news:

04 October - released several metric plugins in source code:

* MSU Brightness Independent PSNR. (http://www.compression.ru/video/quality_measure/metric_plugins/bi-psnr_en.htm) (with source code under LGPL)
Main idea: It's common situation during testing (we made big
amount of tests), when codec change brightness a little. This metric
is compensate such changes for ANY brightness modification (up
to negative frame).
Also speed optimizer's are welcome!

* MSU Brightness Flicking Metric. (http://www.compression.ru/video/quality_measure/metric_plugins/bfm_en.htm) (with source code under LGPL)
Main idea: Simple flicking calculation (can be enhanced).

* MSU Drop Frame Metric. (http://www.compression.ru/video/quality_measure/metric_plugins/dfm_en.htm) (with source code under LGPL)
Main idea: Some codecs on low bitrates drop more frames, some -
less. It's good idea to calculate not only PSNR/SSIM but also number
of dropped frames as codec characteristic for low bitrates. Also this metric is
useful for capture device/program evaluation.

And as I wrote: Now you can create your own metric, e.g.:
* Faster metrics (using MMX-SSE2)
* New special measures (blocking, interlacing, etc)
* New perfect formal metrics (again better than PSNR) :)
You are welcome!

Enjoy! :)

[Sharktooth]

Nice stuff, thanx ;)

[Sergey A. Sablin]

Hi Dmitriy,

would be cool for non-russian speakers to place graphics on english pages with english tips ;)

[DmitriyV2]

Hi Dmitriy,
would be cool for non-russian speakers to place graphics on english pages with english tips ;)
Really? :)
Looks like good idea!

[DmitriyV2]

BTW: We have small anniversary 4 days ago - total number of metrics program downloads from official page become bigger than 25000! It's good value for such specific program and looks like next 25000 downloads will be faster.

Congratulations are welcome! :)

We have feedback from following big companies:
* Analog (analog.com)
* Samsung (samsung.com)
* Intel (intel.com)
* Hewlett-Packard (hp.com)
* Thomson (thomson.net)
* Harmonic (harmonicinc.com)
* Matrox (matrox.com)
* Zoran (zoran.com)
* Kodak (kodak.com)

And smaller companies:
* audiocodes.com
* envivio.fr
* ambric.com
* smartphones.no
* intec.ugent.be
* testpro.co.il
* britannia2000.co.uk
* bouyguestelecom.fr
* simonechiesa.com
* honeywell.com
* cdrinfo.com
* infoteck.ru
* astri.org
* cubiworld.com
* inescporto.pt

Of course this is good help for us.

Hope with metrics plugins we will help to develop new good metrics for video funs and pro's.

[DmitriyV2]

There was released 1.3 and 1.4 versions.

Main changes:
+ Now program has installation process (but can work without it also as well)
* Bugfixing in BMP processing (visualization saving, etc)
* Bug with YV12 yuv files fixed

Metric page:
http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html

[HeadBangeR77]

Cheers, I've been using this tool for some time already (mainly SSIM metrics), and I used to have problems with saving bad frames (original frames were sometimes saved and sometimes not, so I didn't have the comparison background). Good to hear the issue has been resolved.

Just a question - I really like the graphical way of showing the metrics values throughout the clip/film. Is there any chance to implement some other saving mechanism than the present one (video)? I think an apropriate container might be a problem here.

regards /HDBR77

[CruNcher]

DmitryV2 compared with the Video Quality Studios PSNR and SSIM measurement speed MSUs application is upto 3x slower
could you enhance something in those regards ?

[DmitriyV2]

2 HeadBangeR77 & CruNcher: I have answer for both questions.
We are implemented plugins mechanism now, so it's possible:
* to implement your own metric with alternative visualisation;
* to implement faster (MMX, SSE, 3DNow) metrics implementation;
We have wide front of work and sometime some plans (like metric speedup) moving forward too slow, thats why we implement plugins. So everybody can use our PSNR plugin (you can find it source codes in samples) and make it 3 times faster. And hope we also will make our metrics faster during this year.

[laminar]

Hello Dmitriy,

I recently bought your MSU Video Quality Measurement Tool.
It is really powerful !
I need your help to use the CLI version. We developped our own metrics (like PQA). We are able to use them through the GUI version, but I don't know how I can use them with the CLI version (msu_measure.exe), I even don't know whether it is possible !
Thanks for your support

[Sergey Grishin]

The reply is quite out-of-date, but may be it will be helpful for other people.
Of course it is possible!
You should implement virtual function GetName in your metric class which implements ICustomPlugin.
This function should return the name of your metric which is used in command line for metric identification.

For example, in case of BI-PSNR plugin we did the following:
In class
class CBIPSNRPlugin : public ICustomPlugin
there is function:
wstring CBIPSNRPlugin::GetName ()
{
return L"bi_psnr";
}

To use this metric from command line tool use this command (don't forget to put your vmp file into 'plugins' folder):
msu_metric.exe -f file1.avi -f file2.avi -metr bi_psnr -cc YYUV -sc 1 -cod out_dir

[Sergey Grishin]

The tool was updated recently. Version 2.6 released.
http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html

Here the changes since version 1.4:
2.6 (Windows Vista & Windows 7 support)

Fixed bug in Scene Change Detection plugin when working under Windows Vista or Windows 7
Fixed bug in saving visualization video when running on Windows Vista or Windows 7
Fixed dependency with vcomp.dll


2.5

Video with any resolution is now supported by all metrics. Video with resolution which is not appropriate for some metric is now expanded (via data duplication, separately for each metric) to make resolution acceptable
1.95 times speed up of command line tool multiple metrics calculation on average (PRO version only)
YUV files with size more than 2Gb are supported now
Output directory for *.CSV and visualization files is automatically set to folder of last specified reference file
Fixed bug in processing of *.YUV files with non-standard resolution
Fixed bug in loading the mask from *.YUV files
Fixed bug in masking of L (LUV colorspace) component
Fixed bug in processing of non-standard resolution *.AVS files
Fixed bug in calculation of SSIM (precise) for second reference file
Fixed bug in conversion from RGB32 to YUV color spaces for video with non-standard resolutions (affects calculation of metric for *.AVI files)

2.01 beta

1.5 times speed up of command line tool multiple metrics calculation on average (PRO version only)
Masking is added
Fixed bug in 4:2:2 raw files with more than 8 bits per component support

2.0 beta

*.MOV, *.VOB, *.WMV, *.MP4, *.MPG, *.MKV, *.FLV formats support simplified
HDTV support (PRO version only)
Raw files with more than 8 bits color depth per component are supported (PRO version only)
Alternative SSIM and PSNR are added for compatibility with other implementations
New version of *.CSV files with average metric values (PRO version only)
Minor acceleration
Preview buttons are added
Options save is improved
All MSU plugins are renamed (names are now more correct in GUI and simpler to call from PRO console)
MSU Noise Estimation plugin bug with incorrect (identical) values for some videos is fixed
MSU Noise Estimation and MSU BI-PSNR plugins provide correct information about their home pages now
MSU BI-PSNR plugin crash during visualizing a metric for video sequences with dimensions less than 255 is fixed


1.52

Error in saving CSV file for comparative analysis is fixed


1.51

Error at the opening YUV-files is fixed


1.5

Set of interface fixes

[Sergey Grishin]

The tool was updated recently. Version 2.7 released.
VQMT Home Page (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html)
Video Casts (http://www.compression.ru/video/quality_measure/vqmt_pro_video_en.html#basic)

Changes in v2.7:

MSSSIM (fast and precise) metric implemented
3SSIM metric implemented
Fixed bug in calculation of VQM metric under Windows 7
Fixed bug during program launch on some machines


Examples of VQMT metric visualizations:
http://compression.ru/video/quality_measure/images/metrics.gif

Description

Professional program for video quality assessment:

supports more than 30 formats (AVI, YUV, AVS, BMP, MOV, VOB, WMV, MP4, MPG, MKV, FLV and others via AviSynth);
20 objective metrics are implemented (PSNR, Delta, VQM, SSIM, 3SSIM, MS-SSIM, MSU Blurring/Blocking, MSE, MSAD and others);
any metirc can be implemented via open plug-in interface.


Examples of graphs builts based on data calculated using VQMT:
http://compression.ru/video/quality_measure/images/graphs.gif

Main features:

The tool provides functionality for calculating:

metric values for each frame,
average value for sequence,
metric values for specific color component.

Metric values are stored into CSV (Comma-Separated Values) files, which can be opened everywhere.
Visualization video can be generated additionally to CSV files for each metric. Visualization illustrates the amplitude of the given metric in each pixel of each video frame.
Professional color depth (10-, 14- and 16-bit) is supported for raw data (YUV).
ROI (Region of Interest) support is also implemented, it is possible to calculate any metric for specific part of the frame which is defined by provided mask.
VQMT provides GUI and command-line (CL) interfaces. It is possible using CL-tool to calculate several metrics for a given set of color components simultaneously (in parallel), thus achieveing significant speed advantage comparing to case of using GUI tool.
Detailed documentation is provided along with the tool.

[Keiyakusha]

Thanks. Any chances to see free version without resolution limitation?

[Sergey Grishin]

Thanks. Any chances to see free version without resolution limitation?

This limitation can be set off when there will be some other feature justifying companies for purchasing the license. Currently there is no such feature.

Any way FREE version already provides the majority of functionality of PRO:
http://compression.ru/video/quality_measure/video_measurement_tool_en.html#features

[yuri69]

I've already asked once but nobody replied.
url (http://forum.doom9.org/showpost.php?p=1480162&postcount=1)
tl;dr

Is there any way how to compare Theora or Dirac encoded video files with this 'MSU Video Quality Measurement Tool'?

[LoRd_MuldeR]

I've already asked once but nobody replied.
url (http://forum.doom9.org/showpost.php?p=1480162&postcount=1)
tl;dr

Is there any way how to compare Theora or Dirac encoded video files with this 'MSU Video Quality Measurement Tool'?

Besides the fact that these objective quality metrics can't tell you much about the overall visual quality, why not?

I don't know how the MSU tool decodes the input, maybe through VFW, through DirectShow or through it's own "internal" libavcodec-based decoder.

Anyway, you can always decompress the videos you want to compare to some lossless(!) format, such as HuffYUV, before feeding them into the MSU tool...

Oh, and if the MSU tool supports Avisynth input, you are of course free to use whatever source filter you like (FFM2 might be a good candidate).

[EDIT]

And of course you would not compare two "compressed" videos against each other directly, but compare each of them against the original video!

[yuri69]

Besides the fact that these objective quality metrics can't tell you much about the overall visual quality, why not?

I don't know how the MSU tool decodes the input, maybe through VFW, through DirectShow or through it's own "internal" libavcodec-based decoder.

Anyway, you can always decompress the videos you want to compare to some lossless(!) format, such as HuffYUV, before feeding them into the MSU tool...

Oh, and if the MSU tool supports Avisynth input, you are of course free to use whatever source filter you like (FFM2 might be a good candidate).

[EDIT]

And of course you would not compare two "compressed" videos against each other directly, but compare each of them against the original video!Thanks for your reaction.

I'm going to compare video files mostly to uncompressed files (.yuv).

I know that the PSNR really doesn't correlate with human perception, but SSIM metrics should be better. I'm going to perform some blinded experiments on a testing group too.

I will try to struggle with ffmpegsource2, AviSynth and Dirac now. Theora has worked fine :)

[LoRd_MuldeR]

Thanks for your reaction.

I'm going to compare video files mostly to uncompressed files (.yuv).

Nothing speaks against decompressing your Dirac or Theora clips to uncompressed .yuv files before feeding them into the MSU tool.

I know that the PSNR really doesn't correlate with human perception, but SSIM metrics should be better. I'm going to perform some blinded experiments on a testing group too.

Yes, SSIM should correspond to the human perception much better, still it's far from being perfect.

Just think of the various Psy optimizations in x264, which greatly improve the perceived quality, but hurt all the well-known metrics.

One basic problem is that all the commonly used metrics assume the "original" source is of perfect quality and that every divergence from the sources is inherently bad.

In reality the human eye might prefer a slightly more divergent version over a version that is less divergent (mathematically) but totally smoothed out...

[yuri69]

Nothing speaks against decompressing your Dirac or Theora clips to uncompressed .yuv files before feeding them into the MSU tool.Well, I'd rather to load them with ffmpegsource2 to AviSynth. However it looks that I will have to compile my own libschroedinger.dll, because FFindex fails when opening a Dirac coded file. Damn MinGW :(

Also yes, I'm aware of various psycho-visual enhancements. That's why I want to include the blind experiment.

[MSU G&M Lab Videogroup]

Hello everybody!

We have just released new VQMT version 3.0!

Whats new in the version 3.0?

1) Accelerated SSIM-based metrics calculation using NVIDIA hardware.
Now it works up to 13 times faster on GPU!
2) Added stSSIM metric, which uses temporal information and shows good
subjective quality correlation result
3) Added 64-bit version with up to 10% speedup
4) Added *.y4m internal support

For those who interested in metrics difference we made a subjective
quality comparison for some of them:
http://www.compression.ru/video/quality_measure/subj_metric_comparison_en.html

MSU VQMT 3.0 Free version can be downloaded here:
http://www.compression.ru/video/quality_measure/vqmt_download_en.html#free

As always, additional information you can read on the MSU VQMT Webpage:
http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html

Feel free to ask questions!