[TEST] Visual, PSNR, SSIM, VQM tests with RV9, XviD ,DivX, WMV9 ...

[SeeMoreDigital]

BTW: Could anyone tell me why doesn't Sorenson MPEG4 codec play on my computer? Hi deXtoRious,

In order to obtain a response. You will have to provide a little more info than the above statement!

Cheers

[deXtoRious]

Thanks for the answer :)
Expect to see it here after 1-2 days.

[deXtoRious]

I've got everything ready to encode, just one last question before I start: could anyone please suggest a decent method to measure the speed of the encode without sitting next to the computer with a chronometer?

[superdump]

Well if you're using vdub or vdubmod and you put all your jobs in the job list you can see the times it took to encode accurate to the nearest minute.

[Soulhunter]

And when You not use VDub...
You can watch the Date/Time of the resulting Log/Movie Files...

Bye

[deXtoRious]

All right, thanx. Gonna do the test today and post results either this evening or tommorow morning.

[deXtoRious]

I did some encoding and some PSNR tests and I've got another question. In the end of the log file there is the Average PSNR and the Overall PSNR, so which should I take into account?

It looks like this:

Minimum Average Maximum
Mean Absolute Deviation: 0.0095 5.3357 68.3030
Mean Deviation: -66.0349 +0.0084 +68.2864
PSNR: 7.0340 30.9158 68.3631
Overall PSNR: 22.9140

[Assault]

It's recommended to use the overall PSNR. The difference to the average PSNR is that it treats the whole video as one frame.
But are you sure that everything worked find when you encoded your clips because an overall psnr of 22.9 is very low.

Assault

[deXtoRious]

Well, that's what surprised me... The video looks quite well visually. It's a WMV9 encode at 550 kbps. If it helps, the SSIM I got was 0.965776.

EDIT: The PSNR script was this:
#PSNR
name = "trailer wmv9 550kbps.avi"
a = avisource("trailer.avi").assumefps(100).converttoyuy2()
b = avisource(name).assumefps(100).converttoyuy2().trim(1,0)
compare(b, a, "", name+".txt", false)

[superdump]

dextorious: If using xvid with b-frames and testing the PSNR, remember to remove the first frame from the xvid clip using either ".deleteframe(0)" or ".trim(1,0)". If you do not remove the null frame your results for xvid will be significantly (and incorrectly) marred.


EDIT!!!: Or conversely, remember to remove it when using other codecs. Retest the wmv9 clip without the ".trim(1,0)".

[deXtoRious]

Well, I just retested it without the trim and the Overall PSNR is now 39.3785! :)
Thank you for the help.
Are there any other things that I should about PSNR/SSIM/VQM tests?

[superdump]

I found RV9 to be a bit tricky. If you have a working test script for it then use it, if not get back to me and I'll dig mine out for you.

Aside from that, just the xvid with b-frames thing really. I can't think of anything else anyway.

[deXtoRious]

Well, it'd be really nice if you could give me yours...:o
And I'm truly sorry about another newbie-like question, but how do I open an rmvb through avisynth???

[Sirber]

DirectShowSource

but you won't have the sound.

[superdump]

To open an rmvb via avisynth you have to use:
DirectShowSource("filename.rmvb",fps=25)

And here is the test script for RV9:

name="test.rmvb"

source=avisource("source.avi").converttoyuy2().deleteframe(0).deleteframe(n-2)
test=directshowsource(name,fps=25).converttoyuy2().deleteframe(n-1)

compare(test,source,"",name+"psnr.txt",false)

Alter the fps accordingly. Also, if you have a clip which is 1500 frames long (counting from 1 to 1500) then let n=1500 and fill in the two 'deleteframe()'s.

I believe that is correct. I'm just about to run a quick test so I'll tell you if I find anything wrong with it.

EDIT: Sirber half beat me to it. ;)

[deXtoRious]

Thanks! :)
That should be all. I'll try to include Sorenson 3 MPEG4 Video codec as well, if I'll have problems with it again, I'll write it here.

[Sirber]

Does anyone succeded in installing Sorenson on non-english OS?

[SeeMoreDigital]

Originally posted by Sirber
Does anyone succeded in installing Sorenson on non-english OS? Sirber my friend. It's a real shame (no pun intended) that you've not had sucess installing Sorenson.

Can you confirm which version you are trying to install?

Cheers

[Sirber]

Confirming:

Sorenson Squeeze 3.0: 16-bit app errror when installing video codecs
Sorenson Squeeze 3.5: 16-bit app errror when installing video codecs
Sorenson Video Codecs 3.x: 16-bit app errror

Viva French Operating Systems!!! :D

[SeeMoreDigital]

Originally posted by Sirber
Confirming:

Sorenson Squeeze 3.0: 16-bit app errror when installing video codecs
Sorenson Squeeze 3.5: 16-bit app errror when installing video codecs
Sorenson Video Codecs 3.x: 16-bit app errror

Viva French Operating Systems!!! :D Hey, just a thought. If you think the French OS is the problem. Why don't you create another boot partition and load the English version of XP?

It's aways a good idea to have at least two boot partitions anyway!

Cheers

[Sirber]

I have 2 partitions, one for OS and one for Datas. I don,t like to have 2 OS and I'm french :)

[SeeMoreDigital]

It was just a thought!

All our work PC's (all 6 of them) we have 4no 8GB approx WinXP Pro 'boot' partitions, on an independant 40GB hard drive.
One boot partition for geneneral/office use.
One boot partition for video editing. And Mpeg1,2 & 4, WMV9 (.avi and wmv) and QuickTime encoding
One boot partition for video editing. And RealMedia encoding.
One boot partition to be used as a spare, should one of the other 'boots' crash or become unstable.

All four of the 'Hire Out' PC's have WinXP Pro installed on them 3no times - and are formatted at least 6 times a year.

At home my 'Media PC' has 3 boot partitions. XP Home x2 and Linux. And my main PC. And my main PC has XP Home x3.

I take the view, that with hard disc space being so cheap and M$ operating systems being so expensive. It's more economical to install the Windows O/S's more than once! And the whole exercise saves time (and money) in the long term!

Cheers

[Sirber]

hum... as a linux addict, I like to have a long and hard uptime :) Also, I don't want to waste my precious disk space on multiple Windows installation :D

[C0mPr355]

can you tell me what trailer you used ?

I would like to run some tests as well. :)

ty.

[deXtoRious]

2 C0mPr355

Whom did you ask?

2 all

Sorry for the delays, I'm currently experienced a lack of time because of the disturbing phenomena called school :(
I will try to finish the test tommorrow, but unfortunately I can't promise anything.

[deXtoRious]

I've finally got some free time, for two days :)
I've done almost all the encodes and tests, but my psnr script (which was kindly given to me by a helpful member of this forum) for rv9 doesn't work.

n=3635
name="trailer rv9 550kbps.rmvb"
source=avisource("trailer.avi").converttoyuy2().deleteframe(0).deleteframe(n-2)
test=directshowsource(name,fps=25).converttoyuy2().deleteframe(n-1)
compare(test,source,"",name+"psnr.txt",false)

DirectShowSource just shows that RV9 is an unsupported format. Could anyone help me on this?

[C0mPr355]

i meant to direct that question at sirber. Which LOTR trailer did he use.

[karl_lillevold]

Originally posted by deXtoRious
DirectShowSource just shows that RV9 is an unsupported format. Could anyone help me on this?
You need Gabest's RealMediaSplitter:
http://forum.doom9.org/showthread.php?s=&threadid=56940

You can also enable PSNR in RV9 encoder with the calcPSNR codecProperty [d9 search for calcPSNR], to double-check the Avisynth compare numbers.

[deXtoRious]

Thanks!
I thought I was using it, but it turned out I had removed it some time ago. Another stupid mistake from my part :(

[deXtoRious]

When I try to use this script:
n=3635
name="trailer rv9 550kbps.rmvb"
source=avisource("trailer.avi").converttoyuy2().deleteframe(0).deleteframe(n-2)
test=directshowsource(name,fps=25).converttoyuy2().deleteframe(n-1)
with ssim or vqm, they start yelling that they need yv12 instead of yuv2. So I tried to replace the convert with a converttoyv12. However, after that the compare functions just kept yelling as if nothing had happened...:(
So, how do I convert this to YV12?

[deXtoRious]

Well, I hate to beg for help, but I have to, otherwise my test will never be finished :(
PLEASE HELP! :rolleyes:

[superdump]

dextorious:

n=3635
name="trailer rv9 550kbps.rmvb"
source=avisource("trailer.avi").converttoyv12().deleteframe(0).deleteframe(n-2)
test=directshowsource(name,fps=25).converttoyv12().deleteframe(n-1)

That should work just fine. Although, iirc, RV9 uses YV12 anyway so you shouldn't need the conversion for that one and if your source avi is in VBLE or YV12 huffyuv or YV12 FFV1 or some other lossless YV12 format then that shouldn't need it either.

[deXtoRious]

It still doesn't work, because due to RV9's VFR, the clips are of different length :(

[superdump]

Well that's RV9's own damn fault isn't it? :D Maybe it would be reasonable when comparing a VFR and a CFR video stream to compare the CFR stream's frames to the currently decoded image in the VFR stream? I don't know if that would be possible though. Ask the avisynth gurus. :)

[deXtoRious]

The question is simple: how the hell do I get SSIM and VQM out of RV9???
The test in the first page of this thread clearly shows it's possible.

[superdump]

OK, firstly get the following and load the relevant one for each test using "LoadPlugin("c:\blah\blah\blah.dll")"

VQMCalc DLL (http://perso.wanadoo.fr/reservoir/dl/VqmCalc-0.21.rar)

SSIM DLL (http://perso.wanadoo.fr/reservoir/dl/SSIM-0.23.rar)

They should then be used by scripting your source file, or better, using VBLE and making an AVI as your source file. Then script the following to test VQM:
LoadPlugin("C:\blah\vqmcalc.dll")
source=AVIsource("C:\Video\Source.avs")
test=AVIsource("C:\Video\XviD.avi")
return VqmCalc(source,test,"results.log")

And this for SSIM:
LoadPlugin("C:\blah\SSIM.dll")
source=AVIsource("C:\Video\Source.avs")
test=AVIsource("C:\Video\XviD.avi")
return ssim(source,test,"log.csv","score.txt",lumimask=true,scaled=true)

And just change the function name to SSIM and add .Converttoyv12() and the delete frames as necessary.

Also, did you get the latest CVS build of avisynth? (google for 'CVS Avisynth', it should be the first result) As this provides an Overall PSNR value when running a compare. The average PSNR value that is stated can be biased as it just sums the PSNR of each frame and takes and average. The overall PSNR does something like sum all the MSEs (mean-squared-errors) of each frame, takes an average and then puts that into the PSNR formula giving an accurate output for the PSNR.

I hope this helps. :)

EDIT: Updates and changes to functions.

[superdump]

Oh, and the conditions of the VQMCalc plugin is that both dimensions of your video have to be an integer multiple of 8 (mod 8) and, as you know, they have to be in YV12 format for both SSIM and VQMCalc.

EDIT: Errors.

[lilhobo]

for a newbie into thread will means nothing, where can i go to get explanation for the diffrent tests ( encoding methods i guess, dll and such)

so which codec has the best results?

[CQ]

hey
1. after writing the script what do you do with it? run in VDub or what? and... please be nice to me :rolleyes:

2. how do you obtain those graphs? with excel? how? plz, someone, i really need to know. (to do a subj. test its easy but when it comes to obj i`m stuck, its not my fault that i`m almost a director (stil studying) not a video engineer :)


please superdump make some time and write a full (or almost) guide (with writing the avs, how to run the avs :( , how to make graphs from the cvs data....). a stupid, i mean very stupid codec tester guide :)

[superdump]

The method hasn't changed in the last year, apart from possibly using the CompareYV12 dll where appropriate. If you use either CompareYV12 or the internal Compare function you can use the Java XviD Stats Viewer to create some very nice antialiased graphs but I don't know how Sagittaire makes his SSIM graphs, he may have written a program for it. And yes, you do just play the resulting script in Vdub or mplayerc or whatever until it's played all the way through and then close it.

For XviD with b-frames in AVI you have to remove the first frame (unless using packed bitstream). Even then it's sometimes a real pain to get it to work. Thankfully everwicked is working on optimising his VQS objective program so that we can use that.