PSNRCreator

[DarkDudae]

PSNRCreator 1.0

Hi all, I have programmed a litle tool that can help to calculate PSNR, SSIM and VQM of any video. It uses VirtualDubMod and Avisynth.

Only thing you have to do is to choose the avs or avi source witch was used to encode a video, and that video itself. The program will launch automatically VirtualdubMod and it will show datailed results on screen. It has two methods to do the tests. Default method only uses lanczosresize, and it ignores all noise filters. (That is to keep the source totally clean). Second method, uses all filters and the resizers that your avs has. To use this method, you have to check the box: "Keep original avs filters"


Yo can download it here:

PSNRCreator (http://www.canalxvid.com/uploader/PSNRCreatorEng.zip)

In the file, there is a README.txt in spanish. I will translate it into english in next days. Anyways, for any doubt, bug, or wathever, you can find me up here.

Since this tool is a beta, use it under your own risk. (Don´t worry, it won´t format your HD, hehe)

Regards

[Koepi]

I'll test it after the 2nd pass of my tonights capture is finished - I'm looking forward to that :-)

(N.B.: how's avsoptimizer coming along?I've tried to use it yesterday, looked for a new version, it's still 0.9 [or is there a binary refresh without notice? Sorry, but my spanish is really non existant]- but unfortunately I started testing dev-api-4 already and it doesn't work with it. Any plans to adopt it soon as it's getting really closer to first dev-api-4 alphas?)

Uhm. Sorry for getting so much off-topic... anyways, thanks for the tool and the work spent on it, I'll check it out in a few hours and post my results here! :)

Best regards
Koepi

[jjseth]

Yes, avsoptimizer is in 0.9version (ARCalculator version)still but all are waiting for newbies.
For Devapi4 testing you can use a special version of ARCalculator0.81. Download here (http://www.canalxvid.com/uploader/ArCalc-DevApi4.zip) . There's an other version too for Syskin Devapi4 downloaded here (http://www.canalxvid.com/uploader/ARCalc_para_devapi4_10-08-2003_by_Syskin.zip). For comity of Mashabak in CanalXvid.

I excuse your spanish if you excuse my poor english;) :D

Ánimo DarkDudae, ahora en versión internacional:D
C'mon DarkDudae, now in internacional version:D

[DarkDudae]

Thanks jjseth :D

@Koepi

About ARCalculator, as jjseth says, there are some before versions with devapi4 support, but they are not translated into english. The ini language files was implemented for version 0.9. Anyways, a new version will be released in next days with the problem of DVD2AVI 1.77x incompatibility fixed, avi support, and that will work with any version of XviD, Dev-Api-4 included :). We are looking for info to expand AVSOptimizer functions, and add an optimized quantization matrix with optimal results for each analyzed video. But I am afraid it is very hard since there is not much info about it.

By the way, PSNRCreator still doesn´t support avs sources with Trims on it. :rolleyes:

Greetings

[Koepi]

Thanks for your replies :-)

Damnit. Make it trim-aware please please pretty-please, because my avs usually end with a line like this:

Trim(0,182919).LanczosResize(640,272)+Trim(182919,0).BilinearResize(640,272)

for making the credits less bitrate-intensive.... :-/

I'll try it anyways with that line commented out ;-)

Best regards
Koepi

[hellgauss]

The zip file doesn't work to me!!

I've written an application for calculating psnr. It allows to trim source and test (it's useful for bframe lag problem).

Since i'm not a programmer i've include the source code, so anyone can do this small prog better!

http://xoomer.virgilio.it/jbgenov/PSNR.zip

HellGauss

PS the readme is in italian, however the GUI it's very simple to use.

[DarkDudae]

@hellgauss

Try to download with Internet Explorer...some other people have reported me the same program, and they used Mozilla as Browser...anyways, the link works perfectly, at least for me. Nice program, good work :). Your program uses an Avisource to load the avs file? It works from different way to mine. By the way, it is very strange, but I only have got a PSNR of 20.45 with a video at quantizer 2 while with my program I get 46.78 . The avi had b-frames so I did a Trim(1,0) in the GUI.:confused: If you want, I can post here both avs scripts...

@Koepi
I have uploaded a new version that will be able to handle with that Trim perfectly.

NOTE: This tool works with almost avisynth script files.

Example of a Working Script file for PSNRCreator:

LoadPlugin("D:\ARCHIV~1\ARCALC~1\mpeg2dec3.dll")
LoadPlugin("D:\ARCHIV~1\ARCALC~1\undot.dll")
LoadPlugin("D:\ARCHIV~1\ARCALC~1\Fluxsmooth.dll")
Mpeg2Source("C:\PRUEBA~1\a.d2v")
Crop(6,2,-8,-2)
SeparateFields
SelectOdd
Undot()
Fluxsmooth(7,7)
BilinearResize(576,432)

Actually, only avs scripts witch have problems with this tool are the scripts with a few fields in a single line:

Example of a NON Working Script file for PSNRCreator:

LoadPlugin("D:\ARCHIV~1\ARCALC~1\mpeg2dec3.dll")
LoadPlugin("D:\ARCHIV~1\ARCALC~1\undot.dll")
LoadPlugin("D:\ARCHIV~1\ARCALC~1\Fluxsmooth.dll")
Mpeg2Source("C:\PRUEBA~1\a.d2v").Crop(6,2,-8,-2).SeparateFields.
SelectOdd.Undot().Fluxsmooth(7,7).BilinearResize(576,432)

Anyways, I will try to correct this in next versions.

Anyways, there are great tools, like Quality Studio of Everwicked ot the promising tool of hellgauss you can use in case of problems.
;)

Regards

[kalxas]

Hi everyone!

I was searching through this forum to find an answer about a problem I have with XviD. The problem is exactly the same as the one posted before: when I use the compare command in avisynth, in order to measure the PSNR of an XviD avi, I get results around 20-30. I just tried to add a trim command, as you suggested, but no change. Finaly, I used the PSNRCreator, and still the same. This must have something to do with B frames, as all tests that I performed with XviD avi's which didn't use B frames, gave me results of around 40.
Any ideas to work around this?
Thanks in advance :)

Kalxas

[Koepi]

Sure. Tweak your settings correctly (i.e. 1 or 2 bframes (or best: no bframes), quant ratio 150, quant offset 75, sensitivity 0).

Regards
Koepi

[kalxas]

Thanks for replying.

So, this is not a measurement or decoder problem? The video file does not seem to be so bad to give 25db PSNR... Anyway, I tried to encode with Bframes =2 and PSNR is still under 30. Everything ok without B frames.

And something else. Do you sugest not to use B frames? Have you concluded that they are buggy or something? Or they tend to result in worse encodes? How about Dev-api 4? The same there with B frames?
Thanks for your suggestions.

[DarkDudae]

Hi again,

I have uploaded a new version of PSNRCreator. This works correctly with almost all avs with Trims in default method. (Thanks Koepi for the help with that silly bug).

It has some minor changes that will help with all test proccess.

You can download it in the main post of this thread.

@kalxas
It is very strange what you say... because PSNRCreator trims the avi to avoid first frame decoder lag for videos with B-frames. Are you sure you didn´t cheched the box for "non b-frames videos"?
Anyways, try to encode a XviD with a constant quant of 2 and try again.

Greetings

[kalxas]

Hi DarkDudae,

I just downloaded your new version of PSNRCreator. I have a problem using it with an avs source. Here is the script I use:

#####Start#####
LoadPlugin("D:\Program Files\DivX Programs\Avisynth\plugins\MPEGDecoder.dll")
source=MPEGSource("D:\VCD\Matrix\VTS_1.D2V")
source=Crop(source,0,78,720,422)
return source
#####End#######

When I press the start button, an error message pops that says "Invalid trim" and after that "Unknown original". This can be worked around by editing the avs your program creates on screen, but when I edit "Comparacion.avs", the program does not use it, so I have to edit it always inside PSNRCreator. Could you, please, make it use the saved on disk avs?

As for the XviD issue, I tried encoding at Quantizer=2 with B frames =1 and still I get PSNR around 25.

Thanks for replying
Kalxas

[kalxas]

Hi everyone,

I just fixed the "problem" I was having with XviD avi source. It was an error on my avs script. By mistake I trimed both source and avi video streams and PSNR went crazy...:D
Sorry for any trouble I caused you trying to reproduce the problem.
Thanks for your suggestions, they helped me understand this thing.

I am working in a codec PSNR comparison right now, and when I am done, I will post here my results.

Thanks again,

Have a nice day,
Kalxas

[DarkDudae]

Try with

#####Start#####
LoadPlugin("D:\Program Files\DivX Programs\Avisynth\plugins\MPEGDecoder.dll")
MPEG2Source("D:\VCD\Matrix\VTS_1.D2V")
Crop(0,78,720,422)
#####End#######

you dont need to associate the source variable
:)

[kalxas]

I tried it and I still get the "unknown original" message.
The comparation.avs doesn't seem to copy the source line from my avs:
MPEGSource("D:\VCD\Matrix\VTS_1.D2V")

Shouldn't this be added to Comparation.avs as
original=MPEGSource("D:\VCD\Matrix\VTS_1.D2V")
???

when I add this line, always works.

Have a nice day,
Kalxas

[hellgauss]

The zip file is now working! Unfortunately at momement i have no video to compare... next rip i will use it.

My psnr calculator simply load the source and the test, convert them in yuy2 colorspace, trim them, and finally resize(lanczos) the test to source resolution. Then is done a compare in avisynth.

If the test has bframe, then it should be trimmed: trim(1,0).
I also noticed that if i convert video to rgb, the psnr is very low. I think it is due to bitperpixel value (24 in rgb, 16 in yuy2).

Is this the right way to calculate psnr using avisynth?

[DarkDudae]

@kalxas

That source is MPEG1? If as I can see, comes from a DVD(Matrix in this case), it is an MPEG2 source, so you can change that for MPEGSource instead MPEGSource. The program doesn´t recognizes it because of that. If it would be MPEG2 it would work perfectly.

Anyways, I have added mpegsources compatibility :D , you can download it when you want (thanks for help me with this bug)

So, this will work:

#####Start#####
LoadPlugin("D:\Program Files\DivX Programs\Avisynth\plugins\MPEGDecoder.dll")
MPEGSource("D:\VCD\Matrix\VTS_1.D2V")
Crop(0,78,720,422)
#####End#######

By the way, if you edit the script inside PSNRCreator, you have to push OK, and of course it saves the changes you do on it, and that will be the script it will be loaded in virtualdubmod.

@hellgauss
I don´t know why, I get lower PSNR values with your tool. Are you sure you can open an avs file as an Avisource? :(
Try my PSNRCreator and check the button: Edit Avs Script and you will see what I do (I read all data directly from the avs source) because when I use an avs as avisource, I don´t get true PSNR values (dunno why)

[kalxas]

Hi DarkDudae,

You are right, my source is from a DVD, so it is MPEG2, but MPEGSource.dll suports MPEG2 and it creates an D2V project to load into VirtualDubMod. I will try what you propose, it sounds more correct indeed :) . But since I started using this command combination during the past months, I got used to this (luck for you trying to find more features to add to your program. :D )
As for the "comparation.avs" issue, whouldn't it be nice to be able to load it in order to repeat the test several times without having to edit it every time?
Thanks for the new version. Keep up the good work!

[DarkDudae]

@kalxas

I don´t understand what you want to say exactly. :confused:

You don´t have to edit manually comparacion.avs every time. It does it automatically (allways the avs source be compatible with my tool).
You had a problem because PSNRCreator didn´t recognized that mpegsource line because I programmed it for Mpeg2source (mpeg2dec3.dll). Anyways, now it should work too with a mpegsource.

For example, if the avs file you are using is like this:

#####Start#####
LoadPlugin("D:\Program Files\DivX Programs\Avisynth\plugins\MPEGDecoder.dll")
MPEGSource("D:\VCD\Matrix\VTS_1.D2V")
Crop(0,78,720,422)
#####End#######

You won´t need to edit anything manually.

Greetings
:)

[kalxas]

@DarkDudae

You won´t need to edit anything manually.

Good point there :)
Everything works fine with the new version. Thanks.

[everwicked]

I have a couple of suggestions to make.

What I assume that your application is doing right now, is to parse the Avisynth script and then re-generate it with different settings/filters.

Why not:
- Allow the user to provide an AVS script without any resizing/filtering
- Provide the resulting AVI and define what resizing/filtering has been used there so it won't be repeated by PSNRCreator
- Use Video Quality Studio to do the processing.

It currently doesn't support command line arguments but if it would be useful to you, I could add it in no time. I think that would be easier than dynamically linking to my app in order to access the APIs.

[DarkDudae]

Originally posted by everwicked
IWhat I assume that your application is doing right now, is to parse the Avisynth script and then re-generate it with different settings/filters.


Well, my app has two methods, First one, reads resolution, crop, and what deinterlacing filters there are in original avs script you put on it. And then, with that resolution, crop or deinterlacing filters, creates a new avs script with only lanczos as resizer.

Second method uses the resizer, and all filters you have in your original avs. (to use this method, you only have to check "Keep original avs filters).

An example:

This is original avs script I used to encode a XviD:

LoadPlugin("D:\ARCHIV~1\ARCALC~1\mpeg2dec3.dll")
LoadPlugin("D:\ARCHIV~1\ARCALC~1\undot.dll")
LoadPlugin("D:\ARCHIV~1\ARCALC~1\Fluxsmooth.dll")
Mpeg2Source("C:\PRUEBA~1\a.d2v")
LumaFilter()
Crop(6,2,-8,-2)
SeparateFields
SelectOdd
Undot()
Fluxsmooth(7,7)
BilinearResize(576,432)

The program, from this script, creates automatically other witch will be loaded in VirtualDubMod:

#Script generated by PSNRCreator to obtain PSNR of a B-frames avi
loadplugin("d:\archiv~1\arcalc~1\mpeg2dec3.dll")
loadplugin("d:\archiv~1\arcalc~1\undot.dll")
loadplugin("d:\archiv~1\arcalc~1\fluxsmooth.dll")
original=mpeg2source("c:\prueba~1\a.d2v")
original=crop(original,6,2,-8,-2)
original=separatefields(original)
original=selectodd(original)
original=LanczosResize(original,576,432)
original=ConvertToYUY2(original)
filtered=avisource("C:\testcomp3\testcomp3.avi")
filtered=Trim(filtered,1,0)
filtered=ConvertToYUY2(filtered)
Compare(filtered,original,"","C:\testcomp2\testcomp3_PSNR.log")
#

As you can see, this script takes resolution, crop and deinterlacing filteres (in case there are), and then, it uses allways LanczosResize.

Anyways, you can use the second method, checking "keep original avs filters", and them, the script generated automatically by PSNRCreator would be:

#Script generated by PSNRCreator with source filters and b-frames
loadplugin("D:\ARCHIV~1\ARCALC~1\mpeg2dec3.dll")
loadplugin("D:\ARCHIV~1\ARCALC~1\undot.dll")
loadplugin("D:\ARCHIV~1\ARCALC~1\Fluxsmooth.dll")
original=mpeg2source("C:\PRUEBA~1\a.d2v").LumaFilter().Crop(6,2,-8,-2).SeparateFields.SelectOdd.Undot().Fluxsmooth(7,7).BilinearResize(576,432).Trim(0,0)
original=ConvertToYUY2(original)
filtered=avisource("C:\testcomp3\testcomp3.avi")
filtered=Trim(filtered,1,0)
filtered=ConvertToYUY2(filtered)
Compare(filtered,original,"","C:\testcomp2\testcomp3_PSNR.log")
#

As you can see, this time the tool uses all filters to do the compare.

Quality Studio is great for PSNR, since it hasn´t problems to calculate PSNR from any source or configuration, anyways, I think its speed is lower that Compare function of Avisynth, and SSIM is not implemented yet on it. However, if you implement commandline, of course I will link to it as an extra method to calculate :D (It would be great)

[everwicked]

Originally posted by DarkDudae
Quality Studio is great for PSNR, since it hasn´t problems to calculate PSNR from any source or configuration, anyways, I think its speed is lower that Compare function of Avisynth, and SSIM is not implemented yet on it. However, if you implement commandline, of course I will link to it as an extra method to calculate :D (It would be great)

It is slower, that is true. I wanted to work on some bottleneck issues I know about but I haven't had the time just yet.

SSIM is also on my TODO list but I wanted to release something I have been working on lately first.

I will make sure I enable the commandline in the next version.

[hellgauss]

@DarkDude
No... i'm not sure!
I'm not very expert in PSNR, i've written my app only to do some test and because i like programming.
It's very strange you get such low values. Did you check the temp avs file in temp dir? I usually get 38-42 for my 1cd rips (source is mpeg2source("***.d2v") + crop, test is the output avi), even if i add some filter like lumafilter or deen.

However why I 'can't open an avs file with avisource? Is there a specific reason?

I want to say 2 thigs about calculating psnr. I use avisynth 2.51, and the average value is incorrect. Since log a + log b =/= log (a+b), it is incorrect to calculate average psnr with aritmeticmean. Also compare function doesn't work in yv12.
A doom9 user (maybe superdump if i remember well...) said me that these 2 problem have been solved in cvs version of avisynth.


Bye
HG

[Wilbert]

said me that these 2 problem have been solved in cvs version of avisynth.
AFAIK Compare still doesn't work in YV12. I will ask Sh0dan to add Shalcker's code to AviSynth. See http://forum.doom9.org/showthread.php?s=&threadid=58187

[DarkDudae]

@hellgauss
I know you can load avisynth files with avisource, anyways, I couldn´t get normal PSNR values using that (dunno why) Here I post you all process.

Original avs script loaded in Quality Studio's everwicked, PSNR's hellgauss and my PSNRCreator:

#####testcomp3.avs - Original AVS Script used for XviD encoding avi##

SetWorkingDir("C:\ARCHIV~1\AVISYN~1\")
LoadPlugin("mpeg2dec3.dll")
mpeg2source("C:\testcomp3\testcomp3.d2v")
crop(0,78,718,420)
LanczosResize(640,256)

#########

Well, I load this avs script and an avi in 3 programs.

AvsScript generated by hellgauss'PSNR:
#
source=AviSource("C:\testcomp3\testcomp3.avs",false)
source=converttoyuy2(source)
test=AviSource("C:\testcomp3\testcomp3.avi",false)
test=converttoyuy2(test)
test=trim(test,1,0)
test=lanczosresize(test,source.width,source.height)
compare(test,source,"","C:\PSNR\hellgaussPSNR.log")
#
As you can see, I have trimmed the test because it is an XviD with b.frames.


AvsScript generated by PSNRCreator:

setworkingdir("c:\archiv~1\avisy~1\")
loadplugin("mpeg2dec3.dll")
original=mpeg2source("c:\testcomp3\testcomp3.d2v")
original=crop(original,0,78,718,420)
original=LanczosResize(original,640,256)
original=ConvertToYUY2(original)
filtered=avisource("C:\testcomp3\testcomp3.avi")
filtered=Trim(filtered,1,0)
filtered=ConvertToYUY2(filtered)
Compare(filtered,original,"","C:\PSNRCreator\testcomp3_PSNR.log")

##############

Avg PSNR:
---------
Hellgauss:12.807 :confused:
Quality Studio: 46.4801
PSNRCreator: 46.4824

Wilbert

Thanks for the info

[DarkDudae]

Hi all, I have fixed some issues, and I have added a lot of new features for the program. Shalcker's code for CompareYV12 has not been added yet, because I will wait next Avisynth version.

You can download it HERE (http://www.canalxvid.com/uploader/PSNRCreatorEng.zip)

PSNRCreator 0.9.9C What is new:

-Added possibility of choosing the % of video to test.

-Added possibility of select the range (trim) that will be tested, for example, from frame 124 to 1050.

-Added more compatibility with avs scripts. (Now it should work best in any cases)

-When you open an avs/avi, now you get info about if they are valid files and what sort of video, resolution, and number of frames it has.

-Added possibility of save the final avs script that PSNRCreator automatically creates for tests.

-Added possibility of save results.

-More stability and speed showing tests results on screen.

-Added color spaces detect, to avoid color conversions when they are not needed.

-Some other minor bugs fixed.

[CruNcher]

just to remind some guys here http://forum.doom9.org/showthread.php?s=&threadid=58102 :D

[DarkDudae]

PSNRCreator 1.0

Well, here it is final version of PSNRCreator.

You can download it HERE (http://canalxvid.urbenalia.com/uploader/PSNRCreatorEng.zip)

What is new?

-More avisynth scripts compatibility. Now, it is 100% compatible with all video scripts in "keep original avs filters" mode too.

-More info messages that will help user to know what is wrong.

-Now if PSNRCreator detects that video file is not a XviD, it will disable "box for NO-Bframes videos", since it is not neccesary for other codec and it can be confussing for some users.

-Fixed a little bug when cancel button was selected in Save dialog.

Thanks for all people who have helped me with this program :)

Regards