Qualitix 0.2.4 : Psnr/Ssim/Vqm tester (Last rev 03/2006)

[Dams]

I've a little program in order to test compare an avisynth source with avi compressed file to get numbers from PNSR/SSIM algorithm.

http://membres.lycos.fr/dexec/screen1.jpg

Site : http://www.qualitix.cjb.net/
Mirror : http://www.qualitix.fr.tc/

Todo:
- Generate graphical reports (Charts)

Give me ur impressions!

[Dams]

Version 0.2.1 :
- added message at end of test.

[Dams]

Nobody post question ? :eek:

So I guess everything is OK :)

[esby]

two remarks:

* It's not for 'xvid/divx' files, but for avi files, and more probably for avisource() capable file if you are using avs filters for doing the hardwork... not a big deal, but it's better to be exact.

* you should state in this thread that it is a gui for automatizing steps with ssim / vqm lefungus filters. (not a big deal again, but that might help people reading this thread...)

I'll check it at home, anyway :)

esby

[Kaiousama]

Can you add graphical representation of the results, to better visualize differences between tested clips on graphs?

I mean one graph showing the differences in terms of average psnr/ssim/vqm values(an histogram graph).

And another more detailed grapg, showing the psnr/ssim/vqm values for each analyzed frame (a line graph would be good), something like what DRManalizer does for compression quantitizers.

[Dams]

Originally posted by esby
two remarks:

* It's not for 'xvid/divx' files, but for avi files, and more probably for avisource() capable file if you are using avs filters for doing the hardwork... not a big deal, but it's better to be exact.

* you should state in this thread that it is a gui for automatizing steps with ssim / vqm lefungus filters. (not a big deal again, but that might help people reading this thread...)

I'll check it at home, anyway :)

esby

Ok, I'll see it ;)

[Dams]

Originally posted by Kaiousama
Can you add graphical representation of the results, to better visualize differences between tested clips on graphs?

I mean one graph showing the differences in terms of average psnr/ssim/vqm values(an histogram graph).

And another more detailed grapg, showing the psnr/ssim/vqm values for each analyzed frame (a line graph would be good), something like what DRManalizer does for compression quantitizers.

It is in the todolist !
I admit that in the actual version, the view of the results is not the cooliest thing of this software...

I'm thinking about how I render this pictures, with HTML table or directly with picture...?
But I think I would render all this in one HTML Page, in order to load your default browser. It should be work on all computer.

[esby]

about graphical representation & XML
I'm not sure xml is really needed...

but if you want to do something like that:
do it as a separate application (with available command line options),
which would take csv files and transform to xml...
maybe it already exists...
if yes, you can just automatize the step.
if no, the tool could be used with more modularity anyway.

for the graphical stuff...

I think openoffice(excel) can provide the treatment
most users will actually need... of course it's not automatic...
but given that you can really need differents analysis, it still very powerful.

Of course, you can do a small application able to use csv file and able to create histograms or statistical data...

and then you'll just have to 'hardlink' the csv created with the right application, in your GUI, like an option...

Another option, would be to create an openoffice spreadsheet model, which would take csv and output data & histograms...
But for that, I cannot really help.

esby

PS: personnaly, i prefer csv over xml, for this kind of data usage,
you don't need particular or complex procedure for loading into memory the data...
Of course, if you use an application, that is able to process your data, and if it accepts xml (since it is a good exchange language), xml is the way to do it.
But if you stick to csv, for me it's the safest & easiest solution.

[Lefungus]

As i've coded vqmcalc, i know it's a completely borked and buggy filter. And as i don't "believe" in it anymore, i won't fix it either. So i don't think you should include it in your app as its results are absolutely untested and unproven.
People may criticize ssim, but at least this filter is a correct implementation of the ssim algorithm. VQMcalc isn't

[Dams]

@esby : I can also generate my own graph with some kind of opensource library which can do that job.. not only with office or openoffice ..

@Lefungus : ok, no problemo, I'll not forget to hide this option in the last release.

[Dams]

@Lefungus : I look at some other forums and web site, and I see, even if VQM is not finished, that many people use it.
So I decided to let it inside Qualitix, but I'll put information on the fact that results may be harsadous with, so to be attention to this and to use with caution.

[esby]

@dams : i was more suggesting workarounds than anything else.
Of course you can do it inside directly.

esby

[Dams]

@esby : what do u mean by workaround ?

[esby]

workaround =
easy solution that a normal user can use, without much problem,
thus making the application not really vital.
I talked of openoffice, since everybody can download & install it whatever the platform he uses (with the singular exception of people using <MacOsX)

esby

[Dams]

@esby : yeah :D
But I can use also TChart..

[Lefungus]

Originally posted by Dams
but I'll put information on the fact that results may be harsadous with, so to be attention to this and to use with caution.

It's not "they may be hasardous", it's "they are false".
Now, people are already using those metrics in wrong ways, i can't imagine what will happen if they also use broken metrics.

I wash my hands of all this stuff :(

[esby]

Just put a warning about it,
if they want to use it,
they won't have to complain if it's meaningless...

esby

[Dams]

Originally posted by esby
Just put a warning about it,
if they want to use it,
they won't have to complain if it's meaningless...

esby

U're all right :D

[Dams]

New version !

0.2.2(07-07-04)
- Added "Text Report Result" -> Report a summary of the results in a window in addition of text files generated
- Cosmetic changes

[Dams]

If u have any questions,remarks, bug reports or something else, tell me!

I'm planning the new version in some days, with charts reports..

[DDogg]

Dams, would you have any suggestions on how to use your tool for comparison with an encoded mpeg2 MPV? I attempted to use VFAPI to create a fake avi from an avs using directshowsource. It plays fine in VDub, so I thought it would function ok in your tool, but I get a "Fichier introuvable" which I think means file not found? (If so it would be great if you could point out what file was not found in English)

Later: ok, I see that both files must be YV12 from attempting to view the temp files you created. Vfapi will not work for this so I'll try other alternatives.

[Dams]

@DDog : would u please send me ur script which work well in vdub, in order to see what I can do for check this.

At which moment, u have the message, "Fichier introuvable" ??

Best regards.

[DDogg]

At which moment, u have the message, "Fichier introuvable" ?? When I press "Launch".

I have since used Link2 (in YV12 mode) to create the fake avi. The scripts for PSNR and SSIM are working ok when I manually run them, so now I am wondering if it is because I did not set a path to VQM? Given the comments made by Lefungus, I was not interested in it and I could not find the dll anyway.

[Dams]

Well, I try it (Link2) with Qualitix and it works well.
I used this script in input :

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec3.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\Undot.dll")
source = mpeg2source("G:\amp\projet.d2v").crop(8,12,704,552)
source = source.undot().LanczosResize(608,320)
source

and this script in output (fake avi) LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec3.dll")
source = mpeg2source("G:\amp\projet.d2v").crop(8,12,704,552)
source = sourceLanczosResize(608,320)
source

If u want to use SSIM or VQM, u have to set the right path in Configuration menu, but if u use only PSNR and SSIM (always with the right path) , it should well worked.

If u want I use ur own script, I can do.

[Dams]

http://membres.lycos.fr/dexec/screen2.jpg

New version :

0.2.3 (16-07-04)
&nbsp;&nbsp;- Correct bug no text showed with "Text Report Result" (oopsss!)

I noticed that report wasn't show !! LOL
If nobody tells me .. :D

[Dams]

New url! (see first post)

[SiXXGuNNZ]

thanks :)

[spyto]

Hi, I have a little problem :confused:
http://img71.exs.cx/img71/2138/problem1vp.jpg

[bgpop]

Site don't work for me... it's only a empty page... :sly:

[PlazzTT]

Me too. But I found this:

http://www.qualitix.fr.tc/

[PlazzTT]

Originally posted by spyto
Hi, I have a little problem :confused:
http://img71.exs.cx/img71/2138/problem1vp.jpg

I also have this error.

It means 'File not found'.

[Dams]

Hi all,

I activate again the url, now its works.

[zambelli]

I also have this error.

It means 'File not found'.

Does anyone know a workaround for this problem? It seems to happen when Qualitix invokes VirtualDubMod. I'm using VDubMod 1.5.10.2.

[Dams]

Hi all,

I've made little improvements, all must be fixed for file not found problem.. (new version 0.2.4)

Let's test !

[Niddhogg]

n.a. for download?

Would be interested in comparing it to subjective differences.

[Dams]

n.a. for download?

Would be interested in comparing it to subjective differences.

Lets see the first page, there is the link to my program ;)

[Niddhogg]

Damn, that was fast. :p
And :

I'm sorry, that file is not available to download at this time


Tried with both browsers (http://img124.imageshack.us/img124/1493/operavsie1sd.gif) to be sure?

[foxyshadis]

The download.php is jacked (probably the file isn't listed in some kind of database) but the file is there: http://membres.lycos.fr/qualitix/qualitix_0.2.4.zip (this is what download.php is pointed to)

[Dams]

If you have any questions/ remarks, don't hesitate ;)

[Audionut]

If you have any questions/ remarks, don't hesitate ;)


There's a slight problem with that.

The site says the file is unavailable for download.

[Dams]

sorry dude !!

Link removed

[Fizick]

Dams,
Your link is broken for me.

Lycos MultiMania hйbergement gratuit de sites perso

Vous essayez de tйlйcharger un fichier d'une page d'un membre Lycos MultiMania, en hйbergement gratuit :
http://membres.lycos.fr/dexec/qualitix_0.2.4.zip

Pour avoir accиs а ce fichier, vous devez visiter la page de ce membre :
http://membres.lycos.fr/dexec
(vous allez кtre redirigй dans 5s)

[Niddhogg]

Ditto for me, Lycos... :o
Can wait a bit though, need a few more codec samples to test.

[Dams]

File is now on line !

[BigDid]

Dams, Your link is broken for me.
...Lycos MultiMania, en hйbergement gratuit :
http://membres.lycos.fr/dexec/qualitix_0.2.4.zip
This one works for me: right-click, save as

...Pour avoir accиs а ce fichier, vous devez visiter la page de ce membre :
http://membres.lycos.fr/dexec ...
The url works but cannot DL the file with R or L-click
I have Firefox 1.51/french, hopes it's not a "cultural exception" :D

Did

Edit: that was more than 1hour ago

[Dams]

This one works for me: right-click, save as


The url works but cannot DL the file with R or L-click
I have Firefox 1.51/french, hopes it's not a "cultural exception" :D

Did

Edit: that was more than 1hour ago

Go the web site directly (one of the two link on the first post of the first page and), I test it, it works now..

[BigDid]

Go the web site directly (one of the two link on the first post of the first page and), I test it, it works now..
Yes, now works for me too, L-click, redirection and save-as box appears. Tks

Did

[Dams]

Yes, now works for me too, L-click, redirection and save-as box appears. Tks

Did

Now it is your turn : test it ;)

[quake74]

Hi Dams, I'm playing around ssim and tried your prog, but it seems to me you don't account for the delay in avisouce. If you do not use trim I think the source video and compressed one will be off by a few frames (I think 2) and so the SSIM calculation is useless. I think this comes from the bframes problem, and I just saw it using an avi created from avc2avi.

[Dams]

Hi Dams, I'm playing around ssim and tried your prog, but it seems to me you don't account for the delay in avisouce. If you do not use trim I think the source video and compressed one will be off by a few frames (I think 2) and so the SSIM calculation is useless. I think this comes from the bframes problem, and I just saw it using an avi created from avc2avi.

You mean that your AVI file is compressed with Bframe , and there is a shift between AVS source and AVI File ?

I'll try to test it with an AVI with Bframe and I tell you.

[foxyshadis]

Interesting tool, but sucks that it's avi only, not much use to me. Would it be hard to add a bunch of standard media extensions and have them use DirectShowSource, plus d2v for mpeg2source? As well as avisynth scripts directly for testing effects of certain filters.

Also directshowsource can help with avi with b-frames. Only issue is whether one wants post-processed video or not, could cause confusion.

I use MSU primarily presently, but it's hardly perfect and requires you to manually create the avisynth files, so yours is a lot more user-friendly that way.

Now the bad news: When I launch the comparison, I get "parameter lumimask is the wrong type"; perhaps it's just my patched version (http://forum.doom9.org/showthread.php?t=92532) but it seems it's supposed to be 0,1,or 2.

Commandline and/or batch testing would also be excellent for numerous different samples.