View Full Version : XviD 1.0 vs all other codec ... ;-)
Sagittaire
22nd May 2004, 03:23
http://jfl1974.free.fr/images/UNDERCON.GIF
Test (http://jfl1974.free.fr/HTM/32_Test_Codec_Video.htm) in french ... :devil:
... but graphs are universal language
- XviD 1.0, RV10, VP6, H264, WMV9
- 500-2000 Kbps for very low and very high compressibility encoding
- SSIM and PSNR test
- Samples (30 sec)
- Captures
http://jfl1974.free.fr/images/UNDERCON.GIF
virus
22nd May 2004, 03:54
:eek: hey, that's what I'd call a professional comparison!
What to say? I still believe that the MPEG quant. matrix for XviD would have performed better visually (maybe not for SSIM). Nevertheless, RV10 comes out on top in great style here. Time for a new Doom9 comparison?
thx for your work!
(BTW am I blind or you've totally ignored DivX? Not even mentioned in the "other codecs" section...)
cheers :)
virus
Neo Neko
22nd May 2004, 05:16
I think a new comparrison is still a way off. With h264 codecs starting to pop up and become usefull, the release of Xvid 1.0, and recent Real Media developments a new comparrison is on the horizon. I think one more codec development will be required before a new test is started.
As to this test it was a bit to heavy on the artificial metrics and way to light on the visual tests. 2 shots from one or two films is hardly a decent crossection. And from the meger image selection I actually thought the h264, and VP6 shots on the 500kbps "image" looked better than RM10. [YMMV] But it still goes to show that artificial metrics are no substitute for the human eye.
Teegedeck
22nd May 2004, 10:44
About SSIM see this post (http://forum.doom9.org/showthread.php?&postid=498172&highlight=SSIM#post498172) by Soulhunter. For SSIM h.263 quant looked better than an HQ custom-matrix at quant=2...
And we already knew that PSNR is pretty worthless in order to judge perceived quality. Better use your eyes, after the only (probably) valid testing-tool (JNDmetrix-IQ, c't used it) isn't downloadable for testing, anymore.
Sagittaire
22nd May 2004, 11:17
make a test only with captures is difficult ...
http://jfl1974.free.fr/images/XviD-WMV9-2000.jpg
Capture with Maxi difference peak PSNR for WMV9: advantage WMV9
Capture with Maxi difference peak PSNR for XviD: advantage XviD
make captures that supposes that Rate Control is identical for all codecs. the captures can also very different because of the I,P and Bframes ...
the best solution is to post small sample for visual test ... thus downlaod sample if you don't like metric tests.
Teegedeck
22nd May 2004, 12:13
Samples would be great. No offense, but metric tests may be useful for analyzing isolated aspect of codec behaviour but not for an overall judgement of quality. Except - maybe - JNDmetrics. But even if JNDmetrics imitates human visual perception for its analysis, it only comes close, and it is our perception what this is all about.
PSNR likes a smooth picture, most viewers' eyes don't - so testing PSNR is not useful IMHO.
Sagittaire
22nd May 2004, 13:20
SSIM is a very good quality metric ... Soulhunter test isn't a very good test ...
Originally posted by Sagittaire
Make test with animate it is not a good idea: visually you will have much difficulty to see the difference between matrix then how SSIM test could do it. 90 for SSIM is a very high value: it's impossible to make the difference visually between encoding with 90 and higher for SSIM ...
a small test with HPII trailer in XviD quant2 ...
andreas_78er matrix
35 586 Ko
SSIM=83.64
H263
28 516 Ko
SSIM=82.53
Make the test with a real movie ...
Doom9
22nd May 2004, 13:29
Guess I have the read the changelogs again.. have there even been visual improvements since the build I've tested last time (when XviD is concerned).
I'm still waiting for a bunch of H.264 encoders.. there are a few in the pipeline that might just appear in an upcoming comparison but right now I think it's too early to get at it again. I even considered quality metrix but then when I got a bunch of codec settings optimized for quality metrics, that was the death of such an idea.
SeeMoreDigital
22nd May 2004, 13:41
Personally I would rather trust my eyes than a bunch of graphs!
And as I've said many times before, why don't we make these codecs work a lot harder by generating some 'comp tests' using larger image pixel frame sizes!
For example, if you are encoding from an 2.35:1 NTSC source, lets have some 720x368 cropped 'anamorphicly framed' encodes or better still some 864x368 cropped 'true framed' encodes.
Similarly, if you are encoding from an 2.35:1 PAL source, lets have some 720x432 cropped 'anamorphicly framed' encodes or some 1024x432 cropped 'true framed' encodes.
Soulhunter has already shown (much better than I ever did) that 'astonishing' levels of detail are capable with XviD using such frame sizes. So lets see how well the other codecs compare.
Cheers
SeeMoreDigital
22nd May 2004, 13:47
Originally posted by Doom9
Guess I have the read the changelogs again.. have there even been visual improvements since the build I've tested last time (when XviD is concerned).
I'm still waiting for a bunch of H.264 encoders.. there are a few in the pipeline that might just appear in an upcoming comparison but right now I think it's too early to get at it again. I even considered quality metrix but then when I got a bunch of codec settings optimized for quality metrics, that was the death of such an idea. Hi Doom9,
Did you manage to take a look at the H.264 encode Tommy Carrot posted the other day http://forum.doom9.org/showthread.php?s=&postid=497454#post497454 The darn thing only ran at 253kbps!
Cheers
Lefungus
22nd May 2004, 13:48
Objective quality metrics are just tools, and such, can be misused. I believe they're the most useful when you try to improve one codec performance and not when comparing various codecs.
About JNDMetrix-IQ (Sarnoff), this (http://www.cns.nyu.edu/~zwang/files/papers/vssim.html) and that (http://www.cns.nyu.edu/~zwang/files/papers/ssim.html) have shown ssim (basic version: ie what the current filter do) to be better on the VQEG set of tests, meaning it has better correlation to human visual system.
Now it would be a real treat if someone is up to the task of adding latest improvements in ssim, that is multi-scale ssim, and motion compensation.
Doom9
22nd May 2004, 14:03
Did you manage to take a look at the H.264 encode Tommy Carrot posted the other daySo far I've yet to look at a single H.264 file. I have a couple of files around, codecs/tools to test and codecs I'm supposed to get for testing, but so far I haven't done anything. I'm seriously contemplating upping the ante though and use larger framesizes next time to make things a bit rougher on the codecs. If only codecs get better and the test remains the same, eventually there just won't be any differences that I can spot. As with PC hardware testing, where eventually newer generation games and benchmarking tools are used, I think 2004 will mark an overhaul of my testing methods. But all that is still a few months away.
amango
22nd May 2004, 14:42
http://jfl1974.free.fr/Video/Sample-XviD-2000.avi
http://jfl1974.free.fr/Video/Sample-WMV9-2000.avi
If you compare those samples, you'll see that WMV9 does really have problems with the background. XVID just produces a fine und still picture with much less visible wandering pixels.
SeeMoreDigital
22nd May 2004, 15:12
Thanks amango,
However, I thought Gladiator was a 2.35:1 movie. Why are your encodes 1.82:1 (1.85:1)?
Cheers
Sagittaire
22nd May 2004, 15:14
@ amango
Use PP=1 for best quality with WMV9
http://jfl1974.free.fr/Download/PP-WMV9.reg
For high resolution/high bitrate use PP=0
Use PP=4 for best quality with VP6
http://jfl1974.free.fr/Download/PP-VP6.reg
For high resolution/high bitrate use PP=0
@ SMD
Source HDTV 1080i 16/9 ...
# AviSynth 2.55 #
LoadPlugin("C:\PROGRA~1\Autorv9\SOFTS\AVSFILE\MPEG2DEC3.DLL")
LoadPlugin("C:\PROGRA~1\Autorv9\SOFTS\AVSFILE\DECOMB.DLL")
Video=Mpeg2Source("D:\Ripp\Gladiator\azerty.d2v", idct=2, ipp=true )
Video=Telecide(Video,post=true,chroma=true,gthresh=50,guide=1)
Video=Decimate(Video,cycle=5)
Video=trim(Video,15244,28001)
Video=Crop(Video,0,0,-8,-0)
Video=LanczosResize(Video,640,352)
Video=AssumeFPS(video,25)
return video
SeeMoreDigital
22nd May 2004, 16:05
Originally posted by Sagittaire
@ SMD
Source HDTV 1080i 16/9 ... Is this source still available for download? If so I might generate and post some of my own test files.
Cheers
virus
22nd May 2004, 16:35
about the "time for a new Doom9 comparison?" topic...
XviD: I don't have XviD beta3 clips anymore, so I cannot judge if the overall quality has been improved since then (my guess is: yes :))
VP6: this is worth a comparison in its current state, that's for sure. The last comparison featured version 6.0.8.3 IIRC. Three different versions came out since then, with the latest 6.2.0.10 being a nice jump ahead in flexibility/quality.
RV10: can't say too much, but reading some posts by Karl about the improvements and some impressions by the userbase, I'd say it's ready for comparison.
H.264: well, x264 (ffdshow) is still unusable. We simply cannot ask "Doom9, please encode your movies 10 times till you find the right constant quantizer to match the size". Will be ready when 2-pass is implemented, since quality-wise it's already worth a look.
As for Ahead's upcoming implementation, I don't know. I don't think we are going to see a _really_ good H.264 implementation before late 2004, though. There are so many degrees of freedom in H.264, so tuning these codecs is going to be a nightmare :)
IMHO we should not forget that DivX 5.2 is coming (last time I took a look at their forums at divx.com they said they are in track with their time schedule --> June). Waiting for it is mandatory IMHO.
just my 2 eurocents :)
virus
Sagittaire,
why did you use 3 b-frames, no qpel, h.263 and pp4 for xvid?
Sagittaire
22nd May 2004, 19:07
Max consecutive Bframe 3
in fact the coder places only one or two bframe to the maximum and rarely three. I put 3 bframe to let the coder choose the best number of bframe. That doesn't modify the PSNR and the SSIM ...
http://jfl1974.free.fr/images/FramesType.jpg
Qpel
PSNR, SSIM and for me quality degradation for low bitrate
H263
best matrix for SSIM and PSNR
PP4
best quality for very low bitrate ... without XviD blocks everywhere at 500 Kbps and 25% for compressibility ... download sample.
SeeMoreDigital
22nd May 2004, 19:31
Well, now that it's been brought to my attention that the source was HiDef (1080i 1.77:1). I really don't want to be negative but I personally can't see the point of generating an 640x352 encode in the first place.
Anyway, my reasons for thinking like this have already been posted here: - http://forum.doom9.org/showthread.php?s=&postid=499054#post499054
Cheers
Teegedeck
22nd May 2004, 22:00
Having had a look at the samples, I see nothing surprising. At 500 kbps without postprocessing almost all samples look irritating. Unacceptable, no differentiating necessary between those. Only RV10, whose pp I can't turn off, looks best. H.264 looks very promising (my own tests with ffdshow's indicated that, too) but the sample doesn't decode quite correctly over here, so it's hard to say and cannot be really judged. Note that I still have to install a decoder for VP6, so I don't have an opinion on this one. For that reason I skip the 1500 kbps samples.
At 1000 kbps RV10 and XviD both do fine; but XviD should have done better than that - see my comment later on. XviD shows blocking during highest motion while RV10 smoothes a lot in exactly these places and also generally. The smoothing is nicely done, edges are still sharp. Thing is, if you don't look at still-frames, you won't realize XviD's blocking but you will realize RV10's smoothing. While quality metrics will probably give better marks to the overall smoothed picture.
I really like that you've used GMC and AQ, but when I checked, I saw that B-frames have about the same quant as P-frames (offset=1 maybe), while their quantizer should be something like P-frame x 1.5 + 1. B-frames used in the way they're used here don't help much, I'd suggest using the standard 2 B-frames, ratio 1.5, offset 1. Using such a setting for the 500kbps-test would have been even more important.
At 2000 kbps there's nothing interesting. XviD would have looked much better with the SixOfNine or HVS-best-picture matrix and, of course, quarterpel. Still it looks a bit sharper to me than WM9. Maybe tomorrow I'll have the opportunity to watch it on the TFT, then I can perhaps tell better.
My bottom-line is: Sorry, but I cannot support your test-results i.e. the results of your quality-metrics. Except for the 500kbps-area, where we're not so much judging quality but pure ugliness. Please read below:
H263
best matrix for SSIM and PSNR
That's the point. I don't see any sense in optimizing a codec test set-up for artificial measuring; likewise I don't want a codec that is optimized for such artificial metrics. I guess we all agree that we don't want a codec that gives the best results in SSIM, but a codec that looks best! If SSIM doesn't like what my eyes perceive as good quality - then the software is wrong, not my eyes.
Also a test should really consist of comparing the encodings to the original. It is not sufficient that an encode looks nice in itself, it should also resemble the original's look as closely as possible.
Testing SSIM and PSNR is fun but hardly makes for a codec-comparison. It might have limited use for optimizing certain aspects of a single codec, but nothing more.
Sagittaire
22nd May 2004, 22:39
Use VSS decodeur for H264 ffvfw with good CPU (Athlon XP 1500+) ... H264 with Cabac, Inloop and all fct flacs is difficult to decode.
If HVS matrix give results spectacularly better than H263 why this Matrix isn't the defaut matrix for XviD ... ???
When the results is bad with XviD everyone says "it isn't the best setting ... with these setting the result is better" ... :D
New sample with XviD and this setting in prepation: XviD 1.0 Final Adjustments with VDM: 2 pass + Ultra high + Matrix HVS-best + vhq4 + chroma motion + trellis + bframe 1/1.50/0.75 + GMC + adaptative quant
Teegedeck
22nd May 2004, 22:43
Please excuse all that editing I did in the post above. :)
How nice that you do another test for us. Will you this time skip the PSNR and SSIM tests?
And yes, if it was for me I'd replace XviD's standard matrix with HVS-best-picture but this is not for me or you to decide. But XviD gives us the opportunity to adjust it to our needs, doesn't it?
P.S.: For 2000 kbps you should even use the SixOfNine matrix but do the 1st pass at quant=3. Isn't it a nicely complicated world we live in? Well, it's never been XviD's aim to make the world simpler. :D
P.P.S.: I realize in the 2000kbps-sample there is terrible blocking in the background of the scene. Forgive me, but your source obviously is really not HQ, be it hi-res or not.
Sagittaire
22nd May 2004, 23:00
H264 and RV10 don't use Post-Process ... it's "inloop filtering" ... it's a fonction to deblocking in loop during the encoding ... in fact the RV10 seems more to be a codec MPEG4 v10 (H264) than a codec MPEG4 v2 (XviD and DivX) ... visually too
Teegedeck
22nd May 2004, 23:01
I know. Sorry for the incorrect terminus. I've used inloop-filtering in my low-bitrate h.264 experiments, too.
Originally posted by Sagittaire
New sample with XviD and this setting in prepation: XviD 1.0 Final Adjustments with VDM: 2 pass + Ultra high + Matrix HVS-best + vhq4 + chroma motion + trellis + bframe 1/1.50/0.75 + GMC + adaptative quantwhy only 1 b-frame???
why not qpel???
2 b-frames and qpel is a must imo
Teegedeck
22nd May 2004, 23:04
Agreed. Didn't want to say anything...
But now we're two...
Sagittaire
22nd May 2004, 23:05
Re ... :D
When the results is bad with XviD everyone says "it isn't the best setting ... with these setting the result is better" ...
correction ...
New sample with XviD and this setting in prepation: XviD 1.0 Final Adjustments with VDM: 2 pass + Ultra high + Matrix HVS-best + vhq4 + chroma motion + trellis + bframe 2/1.50/0.75 + GMC + adaptative quant + QPel
When the results is bad with XviD everyone says "it isn't the best setting ... with these setting the result is better" ... well its you who always comes up with some strange settings, not us (but we already discussed this)
bframe 2/1.50/0.75 where does always the need come from to change the default b-frame settings?
use 2/1.50/1.00!
Teegedeck
22nd May 2004, 23:08
Might I shily ask again why there seems to be blocking in the source, already (as it is visible in both WM9 and XviD at quant=2, I suppose it is in the source)?
SeeMoreDigital
22nd May 2004, 23:08
Originally posted by Teegedeck
...I realize in the 2000kbps-sample there is terrible blocking in the background of the scene. Forgive me, but your source obviously is really not HQ, be it hi-res or not. I was thinking that also.
Can you confirm where it's from Sagittaire?
Cheers
Teegedeck
22nd May 2004, 23:11
Not that it would render a test invalid. But if it indicates that the source is highly quantized already, we of course can't expect a too high level of detail reproduction. Just keep that in mind.
Edit: Please encode those same scenes with all the caveats again, anyway, so we have something to compare.
maybe i am wrong on this, but i assume postprocessing on higher bitrates will do more harm than good propably
SeeMoreDigital
22nd May 2004, 23:15
In my experience you absolutely right!
But I usually view all my encodes via hardware anyway, on an big screen.
Cheers
and it would be nice if you could give us the same method as doom9 to be able to compare the clips visualy (being able to zoom in, switch to codecs as we want aso...) :)
Sagittaire
22nd May 2004, 23:26
not so quickly not so quickly ... lol
but not block for the source: 1080i MPEG2 HP@HL 12 Mbps ...
I will post the others sample for 2000 Kbps ... 4.5 Mo is not to strong ...
I quote himself ... lol
Originally posted by Sagittaire
make a test only with captures is difficult ...
http://jfl1974.free.fr/images/XviD-WMV9-2000.jpg
Capture with Maxi difference peak PSNR for WMV9: advantage WMV9
Capture with Maxi difference peak PSNR for XviD: advantage XviD
make captures that supposes that Rate Control is identical for all codecs. the captures can also very different because of the I,P and Bframes ...
the best solution is to post small sample for visual test ... thus downlaod sample if you don't like metric tests.
Teegedeck
22nd May 2004, 23:42
but not block for the source: 1080i MPEG2 HP@HL 12 Mbp
It is not? Then please post a screenshot of your original for this frame:
http://img52.photobucket.com/albums/v158/Teegedeck/XviD_I-frame_at_quant3.png
Look at the background. I've never managed to get a blocky picture with XviD at quant 3.whatever. And in case anyone should think AQ has something to do with it: the whole background is quantized at 3.
Also, why have you changed XviD's default and unchecked 'Closed GOV'? I realize that in your encode B-frames reference I-frames. They're not supposed to, please use the default here.
Originally posted by Teegedeck
Also, why have you changed XviD's default and unchecked 'Closed GOV'? I realize that in your encode B-frames reference I-frames. They're not supposed to, please use the default here.wtf? how can someone untick this?
sagittaire plz post all your exact settings you used for your xvid encodes and especially those which you changed from default!
Sagittaire
23rd May 2004, 00:22
no change defaut setting: closed GOP and packed bistream always actived for bframe it's strange ... :confused:
Soulhunter
23rd May 2004, 00:24
Originally posted by Sagittaire
Soulhunter test isn't a very good test ...
You mean my 2nd test, eh...
Nope, its a very good test !!!
Coz it shows that SSIM sucks for anime sources... ;)
But I will do a 3rd test, then again with a real movie source !!!
Bye
Originally posted by Sagittaire
no change defaut setting: closed GOP and packed bistream always actived for bframe it's strange ... :confused: well still, plz post all your codec settings! transparency in a comparison is very important
and on the quality measuring tools, may i quote karl lillevold:
Originally posted by karl_lillevold
I just wanted to mention that I believe PSNR and PSNR-derivatives (SSIM, JND) will become less and less important for video as it is getting closer to transparency, just as has been the case for audio for many years already. Psycho-visual enhancements already exist that make the video appear more like the original, even though PSNR is decreased. How can this happen, you may ask.. The simplest example could be the video being shifted one pixel compared to the original. PSNR and all measurements that do some kind of comparison with the original, pixel for pixel, will be completely ruined, but visually, seeing the one pixel shift is almost impossible.
For instance, HFE2 should not be used for PSNR comparisons. Even the original HFE, like I think Sagittaire is using for his test, is not recommended. Here's a quote from the HFE2 post (http://forum.doom9.org/showthread.php?s=&threadid=75278). "Mathematical analysis between the source and the encoded video has shown that even though PSNR is reduced from HFE2, the visual resemblance is closer to the original."
Then, with regards to RV10 Elysian, in this post (http://forum.doom9.org/showthread.php?s=&threadid=71022&perpage=20&pagenumber=10) I explained how PSNR can be increased 1/2 dB just by setting patternAdaptivity=2, and rcHighBitrateReduce to 60. Still, my experience is that I prefer the default settings for visual quality, maybe even with
<rcPFrameRefQuant type="uint">6</rcPFrameRefQuant>
<rcBFrameRefQuant type="uint">9</rcBFrameRefQuant>
instead of the default 6 and 10. Others might prefer to max out PSNR though.
Sagittaire
23rd May 2004, 00:59
HFE2 is a sharper in Post-Process who modifie image (visualy better) but it isn't the same image ... and it isn't good for PSNR/SSIM ...
Objectif for a codec is to reproduce most accurately possible in output the image in input ... :D
source
http://satsuki.yatoshi.free.fr/test/548s.jpg
real
http://satsuki.yatoshi.free.fr/test/548r.jpg
real HFE2
http://satsuki.yatoshi.free.fr/test/548rhfe.jpg
Teegedeck
23rd May 2004, 01:06
Originally posted by Sagittaire
no change defaut setting: closed GOP and packed bistream always actived for bframe it's strange ... :confused:
Yes; this is a little strange. My ffdshow version (athos' ffdshow-20040514) reports that frameorder for all my XviD files... Guess I should go over there and let them know. [Edit: Wait a tick! XviD stats report IBB frameorders, too. Is that allowed in closed GOV? Did I miss anything? Is it perhaps only an BBI-frameorder that isn't allowed? It is. I award myself the dunce-hat and shut my mouth. It is so nice to discover one probably misunderstood something... Sorry to have caused that commotion that didn't belong into this thread after all.]
BTW; Karl Lillevold seems to hope that HFE would counteract RV10's tendency to look too smooth, and thus make it look more like the original. But as we see, the effect is still often somewhat different from true to the original.
Manao
23rd May 2004, 02:06
For once, I'll take the defense of Sagittaire. If the source was blocky, since it has been scaled down to 640x352 ( from 1080 to whatever ), blocks would not be 8 x 8 but 9.5 x something.
So blocks in the background are created by XviD. ( edit : and WMV 9 )
Teegedeck
23rd May 2004, 02:29
That blocksize-oddity is a valid observation. Still, we absolutely need to see a screenshot of the source. I don't know what it would take to make that shot look blocky - as an I-frame in XviD at quant=3. And I don't know what 1080i version of Gladiator he has. Probably a DVB that has been badly upscaled?
If XviD produced that I'd say it's worth a bug-report. What do you think.
Good night then, it's getting very late.
Manao
23rd May 2004, 02:54
Strangely enough, I've always found that i-frame, regardless of the framerate / quantizer, where somehow blockier than p / b - frames. Motion compensation is a very good way of removing blocks, I guess, so I don't think it's a bug.
Teegedeck
23rd May 2004, 03:08
Upon closer examination, I see the borders of some more pronounced artefacts overlap in the XviD and WMV9 frame. XviD seems to have 'enhanced' the artefacts in the worst possible way, but I think they were there already.
Edit: OK, safe these pictures to disc, open both of them, then zoom them in to something like 2000x and switch between them while watching the areas I've marked (yes, I know these red lines show I can't hold a mouse steady).
In the upper field you should see some visible horizontal border, while in the lower field at least I see a visible vertical border.
http://img52.photobucket.com/albums/v158/Teegedeck/XviD_artefacts.png
http://img52.photobucket.com/albums/v158/Teegedeck/WMV9_artefacts.png
Sagittaire; could we now have the original, please?
s2001
23rd May 2004, 09:14
Dear All,
I have posted question about XVID vs. WMV comparision:
http://forum.doom9.org/showthread.php?s=&threadid=76688
Could you please reply me - I can't understand why XVID produce the same quality with larger size (vs. WMV9)
I will really appreciate your help!
Best wishes,
Sergey
SeeMoreDigital
23rd May 2004, 10:41
Sorry but I've 'UnSubscribed' from this thread.
Sagittaire
23rd May 2004, 11:33
I will remake the test with an accessible source: HD-WMV source with Punicher sample for exemple ...
Teegedeck
23rd May 2004, 11:59
No, post the screenshot first. We don't need new 'results' right now, we first need to clear up this thing.
Edit: An HD-WMV is not an accessible source. (Only on selected DVDs and/or require registering.)
Some words of adivce, as your tests seem to be the object of critique quite often:
1) Discuss your test-setup extensively before you start your test! Post your avs and all your codec settings and wait for comments for some days.
2) Use mainstream DVDs as source. (You don't need HD-sources for testing resolutions like 640x, you only need high-bitrate sources.) Use DVDs that many people posess, so they can reproduce your results. Ideally choose DVDs that have been used in tests by other people on the forum already, like Matrix 2.
3) Don't only use SSIM tests etc. These are fun, but they don't prove anything. Screenshots prove more than SSIM, full test-samples prove all.
4) Always provide screenshots of the original, too, when you compare screenshots.
If you take this advice, I'll look forward to more tests from you, if not I'm not interested.
DanielSun
23rd May 2004, 14:32
Originally posted by Teegedeck
My bottom-line is: Sorry, but I cannot support your test-results i.e. the results of your quality-metrics. Except for the 500kbps-area, where we're not so much judging quality but pure ugliness. Please read below:
That's the point. I don't see any sense in optimizing a codec test set-up for artificial measuring; likewise I don't want a codec that is optimized for such artificial metrics. I guess we all agree that we don't want a codec that gives the best results in SSIM, but a codec that looks best! If SSIM doesn't like what my eyes perceive as good quality - then the software is wrong, not my eyes.
Also a test should really consist of comparing the encodings to the original. It is not sufficient that an encode looks nice in itself, it should also resemble the original's look as closely as possible.
Testing SSIM and PSNR is fun but hardly makes for a codec-comparison. It might have limited use for optimizing certain aspects of a single codec, but nothing more.
I cannot agree more !
For me PSNR and visual quality are totally different things!
And your test setup should be optimized for our eyes not for the statistic numbers.
Regards
DanielSun
vinetu
24th May 2004, 02:01
@DanielSun - Make this and you'll understand:
Take a very HQ sample (nice colours,fine details and so on),
then encode that 5 times with codec ,which setings are "BEST"
for "our eyes" in that way - Source->encode A->encode B->encode C ...
Then look at the last( encode E ) ... if it still looks good,post
the setting you used here! :)
Best Regards!
sirluke
25th May 2004, 23:14
I might be wrong, but did anyone notice the difference in bitrate between the winner, RV10, and the other contenders? It'a about 25% at the same target bitrate:
XviD @ 500kbps: 25664K ->509kbps
RV10 @ 500kbps: 31905K ->633kbps (+24%)
XviD @ 2000kbps: 101036K ->2004kbps
RV10 @ 2000kbps: 127979K ->2538kbps (+27%)
With such higher bitrate I'm sure XviD could beat RV.
Bye.
p.s. how can someone put together a test and compare codecs not having the same constraints, e.g. bitrate (at least)!
Sagittaire
25th May 2004, 23:36
http://jfl1974.free.fr/images/UNDERCON.GIF
With LOTRII Two Towers PAL Extended Version Chapter 56 "the last walk of Ents" and chapter 57 "Nazkuls Attack" in 640*272 ... ;-)
http://jfl1974.free.fr/images/UNDERCON.GIF
Andrey
25th May 2004, 23:59
Teegedeck: frameorders, too. Is that allowed in closed GOV? Did I miss anything? Is it perhaps only an BBI
IBBP allowed
IBBI not allowed
As I understand it :)
But my question is why not to use "not closed GOP" ?
It seems that I have some miunderstanding here.
Can somebody explain it to me ?
Thanks in advance !
Teegedeck
26th May 2004, 07:34
@Sagittaire:
Originally posted by Teegedeck
1) Discuss your test-setup extensively before you start your test! Post your avs and all your codec settings and wait for comments for some days.
@Andrey: Yes, this is where I misunderstood it: IBBP is a closed GOP. In a previous version of XviD there was an option not to let B-frames reference I-frames and that seems to have caused a mix-up for me.
vBulletin® v3.8.4, Copyright ©2000-2009, Jelsoft Enterprises Ltd.