Tool for codecs comparison (Release 1.4 + 3 metrics as plugins under LGPL)

[DmitriyV2]

Maybe in the future you could have a way to load previous results, for comparing quickly multiple videos. The visualization is quite nice and usable now, and I wish I had something like that in Calc/Excel for charts!
We think it's difficult to repeat Excel/MathCad/MathLab visualization functionality :). So we implement instant analysis tool and recomment to use Excel/MathCad/MathLab for saved csv files analysis.

Are you open to telling how blurring measure is calculated? I'm quite interested in measures of absolute quality, rather than comparisons to a source - partly that's because I work with a lot of videos that start badly and need too much filtering. Plus it would be nice to someday have a tool that can definitively say "this needs rainbow removal" or "this needs mild decombing", which an aplication could turn into filter recs, so newbies can do advanced processing without a guru's help. ...But I suppose that's way beyond the scope of this tool, sorry.

We plan to publish metric in paper soon.

Why not? :) We plan include plugin interface into new version. So ANY new metrics will be possible to add.


I wonder how one would go about turning these spatial measures into temporal measures to better match human perception, though. That would be quite a trick. (Perhaps only measurement averages need to be reweighted somehow.)
This is not so simple trick.

[DmitriyV2]

It would be nice if there was an autocorrect feature that would correct any frame correspondence discrepancies.
We plan to public such "correction" plug-in in open source to tune correction method with the help of other clever researchers.

[Livesms]

DmitriyV2

And what about batch support.

Some times it is usefull for comparing lots of files encoded by different codecs. And here is another option - "append only avarage result to a file file.csv" - this option will append to a file file.scv int next line "file.avi; result value". It will be nice for mass coparation.

For example http://forum.ixbt.com/topic.cgi?id=29:25574-2#26 here i should do a monkey job for clicking button and selectiung file each time to compare new variant.

[siddharthagandhi]

The direct input (non-avisynth) has a lot of bugs, many of my video files aren't accepted. That and the auto frame correction features are the two most important features I look foward to.

[WorBry]

Hi Dmitry,

In the on-line information that you provide regarding your denoise filters, you give an example of how LUV measure can be used to first visualize temporal noise, by comparison of adjacent frames in a source sequence, and then the effects of denoising, by comparing a processed frame with the original.

http://www.compression.ru/video/denoising/index_en.html

Could you please clarify which of the metrics (PSNR, MSAD, delta etc) you use for this. I'm assuming PSNR, but just wanted to make sure.

Actually, based on this assumption, I've done some preliminary tests with deinterlaced (50fps) DV test material and found that it has great potential for fine tweaking Temporal-Spatial denoiser filter settings, including the TS denoiser built into your lossless codec.

To do this, I first ran a PSNR L-LUV comparison (with visualization output) using the source clip as reference and the same clip shifted by one frame (trim(1,0)) as the test clip. I then did the same with the processed (denoised) clip, and visually compared the two RGB 'visualization' sequences to see how effectively the filter has eliminated temporal noise. In my tests, temporal noise mostly appears as blue dots. Anything in the 'yellow dot' domain tends to be associated with moving detail and I so I've tried to avoid removing it.

I'm wondering if this visual comparison of original vs processed visualizations could in turn be converted into a measured 'difference' parameter that provides an objective measure of denoise efficiency. What do you think?

I wondering also if it would be possible to provide an option for outputting the visualization frame sequences in a compressed format (maybe YUY2, YV12 or your MSU lossless format) as the uncompressed RGB sequences can get quite large?

Any possibility of LUV-LUV also?

Great tool. Thanks alot.

Edit: Here's an example, the first image showing a visualization frame from a PSNR L-LUV test of the source clip (original vs one frame shift) and the second, the same frame after denoising with hqdn3d(3). RemoveGrain(2). The third image is a straight PSNR L-LUV comparison of original source and processed frame, revealing the noise that has been removed. Pretty hard to make out I know, but the source image at full res was too large to upload. The clip (my son on a baby chair) was shot in fairly poor indoor light.

[WorBry]

Here are the same images inverted to negatives, which show the effects more clearly.

[siddharthagandhi]

You should upload them at imageshack or someplace else because it takes a while for moderators to approve attachments.

[WorBry]

Moderator,

Any chance of my attachments being approved?

Thanks.

[WorBry]

Patient: Doctor, why do people keep ignoring me?
Doctor: Next patient please !

:o

[DmitriyV2]

We are prepare now version with plugins support (and many other fixes), so anybody will be able to write his own metric - for example with GPU optimization or with measurement of some specific issues.

If you want to take part in metric pre-release testing - please let us know on e-mail video-measure(*)compression.ru (or http://www.compression.ru/video/quality_measure/images/video-measure_graphics_cs_msu_su.gif).

[DmitriyV2]

Last news: 20 June - version 1.2 released! (bugfixes + plugins interface(!)) (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html).

Now it's easy to add new metric to the tool.

For example:
* Faster metrics (using MMX-SSE2, HW acceleration, DirectShow
* New special blurring (blocking, interlacing, etc) measures
* New perfect formal metrics (again better than PSNR :))

Any ideas about future metric development are welcome!

Enjoy! :)

[DmitriyV2]

Last news:

04 October - released several metric plugins in source code:

* MSU Brightness Independent PSNR. (http://www.compression.ru/video/quality_measure/metric_plugins/bi-psnr_en.htm) (with source code under LGPL)
Main idea: It's common situation during testing (we made big
amount of tests), when codec change brightness a little. This metric
is compensate such changes for ANY brightness modification (up
to negative frame).
Also speed optimizer's are welcome!

* MSU Brightness Flicking Metric. (http://www.compression.ru/video/quality_measure/metric_plugins/bfm_en.htm) (with source code under LGPL)
Main idea: Simple flicking calculation (can be enhanced).

* MSU Drop Frame Metric. (http://www.compression.ru/video/quality_measure/metric_plugins/dfm_en.htm) (with source code under LGPL)
Main idea: Some codecs on low bitrates drop more frames, some -
less. It's good idea to calculate not only PSNR/SSIM but also number
of dropped frames as codec characteristic for low bitrates. Also this metric is
useful for capture device/program evaluation.

And as I wrote: Now you can create your own metric, e.g.:
* Faster metrics (using MMX-SSE2)
* New special measures (blocking, interlacing, etc)
* New perfect formal metrics (again better than PSNR) :)
You are welcome!

Enjoy! :)

[Sharktooth]

Nice stuff, thanx ;)

[Sergey A. Sablin]

Hi Dmitriy,

would be cool for non-russian speakers to place graphics on english pages with english tips ;)

[DmitriyV2]

Hi Dmitriy,
would be cool for non-russian speakers to place graphics on english pages with english tips ;)
Really? :)
Looks like good idea!

[DmitriyV2]

BTW: We have small anniversary 4 days ago - total number of metrics program downloads from official page become bigger than 25000! It's good value for such specific program and looks like next 25000 downloads will be faster.

Congratulations are welcome! :)

We have feedback from following big companies:
* Analog (analog.com)
* Samsung (samsung.com)
* Intel (intel.com)
* Hewlett-Packard (hp.com)
* Thomson (thomson.net)
* Harmonic (harmonicinc.com)
* Matrox (matrox.com)
* Zoran (zoran.com)
* Kodak (kodak.com)

And smaller companies:
* audiocodes.com
* envivio.fr
* ambric.com
* smartphones.no
* intec.ugent.be
* testpro.co.il
* britannia2000.co.uk
* bouyguestelecom.fr
* simonechiesa.com
* honeywell.com
* cdrinfo.com
* infoteck.ru
* astri.org
* cubiworld.com
* inescporto.pt

Of course this is good help for us.

Hope with metrics plugins we will help to develop new good metrics for video funs and pro's.

[DmitriyV2]

There was released 1.3 and 1.4 versions.

Main changes:
+ Now program has installation process (but can work without it also as well)
* Bugfixing in BMP processing (visualization saving, etc)
* Bug with YV12 yuv files fixed

Metric page:
http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html

[HeadBangeR77]

Cheers, I've been using this tool for some time already (mainly SSIM metrics), and I used to have problems with saving bad frames (original frames were sometimes saved and sometimes not, so I didn't have the comparison background). Good to hear the issue has been resolved.

Just a question - I really like the graphical way of showing the metrics values throughout the clip/film. Is there any chance to implement some other saving mechanism than the present one (video)? I think an apropriate container might be a problem here.

regards /HDBR77

[CruNcher]

DmitryV2 compared with the Video Quality Studios PSNR and SSIM measurement speed MSUs application is upto 3x slower
could you enhance something in those regards ?

[DmitriyV2]

2 HeadBangeR77 & CruNcher: I have answer for both questions.
We are implemented plugins mechanism now, so it's possible:
* to implement your own metric with alternative visualisation;
* to implement faster (MMX, SSE, 3DNow) metrics implementation;
We have wide front of work and sometime some plans (like metric speedup) moving forward too slow, thats why we implement plugins. So everybody can use our PSNR plugin (you can find it source codes in samples) and make it 3 times faster. And hope we also will make our metrics faster during this year.

[laminar]

Hello Dmitriy,

I recently bought your MSU Video Quality Measurement Tool.
It is really powerful !
I need your help to use the CLI version. We developped our own metrics (like PQA). We are able to use them through the GUI version, but I don't know how I can use them with the CLI version (msu_measure.exe), I even don't know whether it is possible !
Thanks for your support

[Sergey Grishin]

The reply is quite out-of-date, but may be it will be helpful for other people.
Of course it is possible!
You should implement virtual function GetName in your metric class which implements ICustomPlugin.
This function should return the name of your metric which is used in command line for metric identification.

For example, in case of BI-PSNR plugin we did the following:
In class
class CBIPSNRPlugin : public ICustomPlugin
there is function:
wstring CBIPSNRPlugin::GetName ()
{
return L"bi_psnr";
}

To use this metric from command line tool use this command (don't forget to put your vmp file into 'plugins' folder):
msu_metric.exe -f file1.avi -f file2.avi -metr bi_psnr -cc YYUV -sc 1 -cod out_dir

[Sergey Grishin]

The tool was updated recently. Version 2.6 released.
http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html

Here the changes since version 1.4:
2.6 (Windows Vista & Windows 7 support)

Fixed bug in Scene Change Detection plugin when working under Windows Vista or Windows 7
Fixed bug in saving visualization video when running on Windows Vista or Windows 7
Fixed dependency with vcomp.dll


2.5

Video with any resolution is now supported by all metrics. Video with resolution which is not appropriate for some metric is now expanded (via data duplication, separately for each metric) to make resolution acceptable
1.95 times speed up of command line tool multiple metrics calculation on average (PRO version only)
YUV files with size more than 2Gb are supported now
Output directory for *.CSV and visualization files is automatically set to folder of last specified reference file
Fixed bug in processing of *.YUV files with non-standard resolution
Fixed bug in loading the mask from *.YUV files
Fixed bug in masking of L (LUV colorspace) component
Fixed bug in processing of non-standard resolution *.AVS files
Fixed bug in calculation of SSIM (precise) for second reference file
Fixed bug in conversion from RGB32 to YUV color spaces for video with non-standard resolutions (affects calculation of metric for *.AVI files)

2.01 beta

1.5 times speed up of command line tool multiple metrics calculation on average (PRO version only)
Masking is added
Fixed bug in 4:2:2 raw files with more than 8 bits per component support

2.0 beta

*.MOV, *.VOB, *.WMV, *.MP4, *.MPG, *.MKV, *.FLV formats support simplified
HDTV support (PRO version only)
Raw files with more than 8 bits color depth per component are supported (PRO version only)
Alternative SSIM and PSNR are added for compatibility with other implementations
New version of *.CSV files with average metric values (PRO version only)
Minor acceleration
Preview buttons are added
Options save is improved
All MSU plugins are renamed (names are now more correct in GUI and simpler to call from PRO console)
MSU Noise Estimation plugin bug with incorrect (identical) values for some videos is fixed
MSU Noise Estimation and MSU BI-PSNR plugins provide correct information about their home pages now
MSU BI-PSNR plugin crash during visualizing a metric for video sequences with dimensions less than 255 is fixed


1.52

Error in saving CSV file for comparative analysis is fixed


1.51

Error at the opening YUV-files is fixed


1.5

Set of interface fixes

[Sergey Grishin]

The tool was updated recently. Version 2.7 released.
VQMT Home Page (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html)
Video Casts (http://www.compression.ru/video/quality_measure/vqmt_pro_video_en.html#basic)

Changes in v2.7:

MSSSIM (fast and precise) metric implemented
3SSIM metric implemented
Fixed bug in calculation of VQM metric under Windows 7
Fixed bug during program launch on some machines


Examples of VQMT metric visualizations:
http://compression.ru/video/quality_measure/images/metrics.gif

Description

Professional program for video quality assessment:

supports more than 30 formats (AVI, YUV, AVS, BMP, MOV, VOB, WMV, MP4, MPG, MKV, FLV and others via AviSynth);
20 objective metrics are implemented (PSNR, Delta, VQM, SSIM, 3SSIM, MS-SSIM, MSU Blurring/Blocking, MSE, MSAD and others);
any metirc can be implemented via open plug-in interface.


Examples of graphs builts based on data calculated using VQMT:
http://compression.ru/video/quality_measure/images/graphs.gif

Main features:

The tool provides functionality for calculating:

metric values for each frame,
average value for sequence,
metric values for specific color component.

Metric values are stored into CSV (Comma-Separated Values) files, which can be opened everywhere.
Visualization video can be generated additionally to CSV files for each metric. Visualization illustrates the amplitude of the given metric in each pixel of each video frame.
Professional color depth (10-, 14- and 16-bit) is supported for raw data (YUV).
ROI (Region of Interest) support is also implemented, it is possible to calculate any metric for specific part of the frame which is defined by provided mask.
VQMT provides GUI and command-line (CL) interfaces. It is possible using CL-tool to calculate several metrics for a given set of color components simultaneously (in parallel), thus achieveing significant speed advantage comparing to case of using GUI tool.
Detailed documentation is provided along with the tool.

[Keiyakusha]

Thanks. Any chances to see free version without resolution limitation?

[Sergey Grishin]

Thanks. Any chances to see free version without resolution limitation?

This limitation can be set off when there will be some other feature justifying companies for purchasing the license. Currently there is no such feature.

Any way FREE version already provides the majority of functionality of PRO:
http://compression.ru/video/quality_measure/video_measurement_tool_en.html#features

[yuri69]

I've already asked once but nobody replied.
url (http://forum.doom9.org/showpost.php?p=1480162&postcount=1)
tl;dr

Is there any way how to compare Theora or Dirac encoded video files with this 'MSU Video Quality Measurement Tool'?

[LoRd_MuldeR]

I've already asked once but nobody replied.
url (http://forum.doom9.org/showpost.php?p=1480162&postcount=1)
tl;dr

Is there any way how to compare Theora or Dirac encoded video files with this 'MSU Video Quality Measurement Tool'?

Besides the fact that these objective quality metrics can't tell you much about the overall visual quality, why not?

I don't know how the MSU tool decodes the input, maybe through VFW, through DirectShow or through it's own "internal" libavcodec-based decoder.

Anyway, you can always decompress the videos you want to compare to some lossless(!) format, such as HuffYUV, before feeding them into the MSU tool...

Oh, and if the MSU tool supports Avisynth input, you are of course free to use whatever source filter you like (FFM2 might be a good candidate).

[EDIT]

And of course you would not compare two "compressed" videos against each other directly, but compare each of them against the original video!

[yuri69]

Besides the fact that these objective quality metrics can't tell you much about the overall visual quality, why not?

I don't know how the MSU tool decodes the input, maybe through VFW, through DirectShow or through it's own "internal" libavcodec-based decoder.

Anyway, you can always decompress the videos you want to compare to some lossless(!) format, such as HuffYUV, before feeding them into the MSU tool...

Oh, and if the MSU tool supports Avisynth input, you are of course free to use whatever source filter you like (FFM2 might be a good candidate).

[EDIT]

And of course you would not compare two "compressed" videos against each other directly, but compare each of them against the original video!Thanks for your reaction.

I'm going to compare video files mostly to uncompressed files (.yuv).

I know that the PSNR really doesn't correlate with human perception, but SSIM metrics should be better. I'm going to perform some blinded experiments on a testing group too.

I will try to struggle with ffmpegsource2, AviSynth and Dirac now. Theora has worked fine :)

[LoRd_MuldeR]

Thanks for your reaction.

I'm going to compare video files mostly to uncompressed files (.yuv).

Nothing speaks against decompressing your Dirac or Theora clips to uncompressed .yuv files before feeding them into the MSU tool.

I know that the PSNR really doesn't correlate with human perception, but SSIM metrics should be better. I'm going to perform some blinded experiments on a testing group too.

Yes, SSIM should correspond to the human perception much better, still it's far from being perfect.

Just think of the various Psy optimizations in x264, which greatly improve the perceived quality, but hurt all the well-known metrics.

One basic problem is that all the commonly used metrics assume the "original" source is of perfect quality and that every divergence from the sources is inherently bad.

In reality the human eye might prefer a slightly more divergent version over a version that is less divergent (mathematically) but totally smoothed out...

[yuri69]

Nothing speaks against decompressing your Dirac or Theora clips to uncompressed .yuv files before feeding them into the MSU tool.Well, I'd rather to load them with ffmpegsource2 to AviSynth. However it looks that I will have to compile my own libschroedinger.dll, because FFindex fails when opening a Dirac coded file. Damn MinGW :(

Also yes, I'm aware of various psycho-visual enhancements. That's why I want to include the blind experiment.

[MSU G&M Lab Videogroup]

Hello everybody!

We have just released new VQMT version 3.0!

Whats new in the version 3.0?

1) Accelerated SSIM-based metrics calculation using NVIDIA hardware.
Now it works up to 13 times faster on GPU!
2) Added stSSIM metric, which uses temporal information and shows good
subjective quality correlation result
3) Added 64-bit version with up to 10% speedup
4) Added *.y4m internal support

For those who interested in metrics difference we made a subjective
quality comparison for some of them:
http://www.compression.ru/video/quality_measure/subj_metric_comparison_en.html

MSU VQMT 3.0 Free version can be downloaded here:
http://www.compression.ru/video/quality_measure/vqmt_download_en.html#free

As always, additional information you can read on the MSU VQMT Webpage:
http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html

Feel free to ask questions!