Why do I still get "best" results with Koepi's Nov. 14th build?

[TripleA]

I have a problem:

For me, all builds (Koepi's) after Nov. 14th, 2002 A.D. are extremely slow compared to it without a tangible increase in quality. Actually, compressibility goes down with them.

This without using B-frames (I don't think I'll be using them for anything but very low bitrate stuff). With QPel, lumi-masking & GMC.

An actual example:

A 2-pass encode of Stuart Little 2 with a target size of 660343KB.

AVS:

LoadPlugin("MPEG2Dec3.dll")
LoadPlugin("Convolution3D.dll")
mpeg2source("Stuart Little 2.d2v")
crop(2,14,716,548)
BicubicResize(640,336,0,0.75)
Convolution3d (preset="movieHQ")

Encode analysis of the 111402 encode:

XviD Analyzer v0.21 by MoonWalker
e-mail : s_ilias@gmx.net


Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 2196 Times, Percentage Used : 2.14%
Quant 3 Used : 46747 Times, Percentage Used : 45.50%
Quant 4 Used : 49298 Times, Percentage Used : 47.99%
Quant 5 Used : 4348 Times, Percentage Used : 4.23%
Quant 6 Used : 138 Times, Percentage Used : 0.13%
Quant 7 Used : 3 Times, Percentage Used : 0.00%

Average Quantizer Used for Movie : 3.547

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 9092 Times.

MPEG Quantization Type Used 62879 times, Percentage Used : 56.23%

H.263 Quantization Type Used 48943 times, Percentage Used : 43.77%

Quantizers prevented from rising too steeply 23 times


Intra-Frame (Key-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 6 Times, Percentage Used : 0.41%
Quant 3 Used : 1373 Times, Percentage Used : 92.83%
Quant 4 Used : 30 Times, Percentage Used : 2.03%
Quant 5 Used : 1 Times, Percentage Used : 0.07%

Average I-Frames Quantizer Used for Movie : 3.018

Credits
---------
Quant 20 Used : 69 Times, Percentage Used : 4.67%

Inter-Frame (P-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 2190 Times, Percentage Used : 1.98%
Quant 3 Used : 45374 Times, Percentage Used : 41.12%
Quant 4 Used : 49268 Times, Percentage Used : 44.65%
Quant 5 Used : 4347 Times, Percentage Used : 3.94%
Quant 6 Used : 138 Times, Percentage Used : 0.13%
Quant 7 Used : 3 Times, Percentage Used : 0.00%

Average P-Frames Quantizer Used for Movie : 3.555

Credits
---------
Quant 20 Used : 9023 Times, Percentage Used : 8.18%


B-Frames not Used


No Skipped Frames



Frame Analisis
----------------

Number Of Intra-Frames (Key-Frames) : 1479
Number Of Inter-Frames (P-Frames) : 110343
Number Of B-Frames : 0
Number Of Skipped Frames : 0
Total Number Of Frames : 111822

1.32% of the Movie is Intra-Frames (Key-Frames)
98.68% of the Movie is Inter-Frames (P-Frames)
0.00% of the Movie is B-Frames
0.00% of the Movie is Skipped Frames



Size Analysis
----------------

1-Pass Size : 1383504254 Bytes or 1351078 KBytes or 1319 MBytes
Scaled Size : 673507254 Bytes or 657721 KBytes or 642 MBytes
Actual Size : 672856962 Bytes or 657086 KBytes or 641 MBytes
Actual Size of Avi File without Sound : 675603456 Bytes or 659769 KBytes or 644 MBytes

Usefull Statistics
------------------
Compressibility : 48.63%
Relative Quality of XviD avi : 56.26%
Absolute Quality of XviD avi : 95.34%

Analysis of the 021703 encode:

XviD Analyzer v0.21 by MoonWalker
e-mail : s_ilias@gmx.net


Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 1761 Times, Percentage Used : 1.71%
Quant 3 Used : 49590 Times, Percentage Used : 48.27%
Quant 4 Used : 47502 Times, Percentage Used : 46.24%
Quant 5 Used : 3729 Times, Percentage Used : 3.63%
Quant 6 Used : 138 Times, Percentage Used : 0.13%
Quant 7 Used : 10 Times, Percentage Used : 0.01%

Average Quantizer Used for Movie : 3.522

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 9092 Times.

MPEG Quantization Type Used 60471 times, Percentage Used : 54.08%

H.263 Quantization Type Used 51351 times, Percentage Used : 45.92%

Quantizers prevented from rising too steeply 19 times


Intra-Frame (Key-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 6 Times, Percentage Used : 0.41%
Quant 3 Used : 1363 Times, Percentage Used : 92.09%
Quant 4 Used : 33 Times, Percentage Used : 2.23%
Quant 5 Used : 9 Times, Percentage Used : 0.61%

Average I-Frames Quantizer Used for Movie : 3.032

Credits
---------
Quant 20 Used : 69 Times, Percentage Used : 4.66%

Inter-Frame (P-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 1755 Times, Percentage Used : 1.59%
Quant 3 Used : 48227 Times, Percentage Used : 43.71%
Quant 4 Used : 47469 Times, Percentage Used : 43.02%
Quant 5 Used : 3720 Times, Percentage Used : 3.37%
Quant 6 Used : 138 Times, Percentage Used : 0.13%
Quant 7 Used : 10 Times, Percentage Used : 0.01%

Average P-Frames Quantizer Used for Movie : 3.529

Credits
---------
Quant 20 Used : 9023 Times, Percentage Used : 8.18%


B-Frames not Used


No Skipped Frames



Frame Analisis
----------------

Number Of Intra-Frames (Key-Frames) : 1480
Number Of Inter-Frames (P-Frames) : 110342
Number Of B-Frames : 0
Number Of Skipped Frames : 0
Total Number Of Frames : 111822

1.32% of the Movie is Intra-Frames (Key-Frames)
98.68% of the Movie is Inter-Frames (P-Frames)
0.00% of the Movie is B-Frames
0.00% of the Movie is Skipped Frames



Size Analysis
----------------

1-Pass Size : 1416495646 Bytes or 1383296 KBytes or 1350 MBytes
Scaled Size : 673507254 Bytes or 657721 KBytes or 642 MBytes
Actual Size : 673314342 Bytes or 657533 KBytes or 642 MBytes
Actual Size of Avi File without Sound : 676061184 Bytes or 660216 KBytes or 644 MBytes

Usefull Statistics
------------------
Compressibility : 47.53%
Relative Quality of XviD avi : 56.67%
Absolute Quality of XviD avi : 95.41%

Avg. quant. went a bit down. But more higher-quantized frames there are. And 1st-pass size went up...

Watching video sections I couldn't notice any difference. And objectively comparing single-frames didn't yield any better results (in case you're interested: I copy a frame from one encode to Paint Shop Pro. Paste the same frame from the other encode as a new layer. Convert it to negative and set its opacity to 50%. Theoretically, if the frames are exactly the same the resulting image is gray any deviations are the differences...)

Not much improvement. At least, not enough to warrant 4 times the encode time.



So why am I bitching about it here? Why don't I just use the build I like?

Well, I am, actually. But I just want to know what I'm doing wrong. 'Cause I must be doing quite a few things wrong, I think...

Any help is appreciated.

[Edit: I forgot to mention that I don't use GMC with 111402. Just lumi, there. The rest of the options are:

Motion search: 6
Quant. type: H.263 for 1st-pass and ModHQ for 2nd.
FourCC: XVID
Max. I-Frame: 132
Min. I-Frame: 5

I-Frame quant: [2,6]
P-Frame quant: [2,10]

Normal curve compression with only I-Frame boost set to 20.

Credits @ quant 20 with greyscale enabled.]

[sam_b]

Sometime the advanced features do not improve the image, but GMC should just make the file slightly smaller and not change the image (on first pass). Qpel is a bit funny, especially at the moment, but I tend to like it with H263. Not with MPEG or custom though.

Are you sure you looked at the Qpel encode? It should make a big difference to the quality. Also, what does 'normal curve compression' mean? Do you mean linear scaling?

Also you do know not to use Modulated with b-frames, should your tests with b-frames look a bit funny?

[ookzDVD]

For 79minutes movie like Stuart Little 2,
I use B-Frame + .AC3 into 1 CD with very good result :)

[BoNz1]

Originally posted by TripleA
For me, all builds (Koepi's) after Nov. 14th, 2002 A.D.

I am sorry, this is just way too funny, Nov. 14, 2002 A.D.? Lol, thats harilious, :D . You sure it wasn't the one from the 14th of 2002 B.C.? Anyhow, if you want to use a build from the 14th of November features like GMC and quarterpixel will be quite unstable and probably as you described slow even more than some of the latest builds. And as was mentioned before GMC will not provide a huge decrease in filesize either if you compare first pass sizes, and quarterpixel will not either really, but it will make the image much sharper. Using b-frames will show a nice decrease in filesize, but as was mentioned before don't use modulated quants.

[TripleA]

Yes, the A.D. bit might be a bit redundant :-).

It's just a thing I have where I try to make anything I say as accurate as I can. In this context, it's intended to show I'm not using any other calendar system...

Anyway, I thought I made things clear, but perhaps not enough:

0- I'm not using any fancy features with the 111402 build, only lumi and ModHQ on 2nd pass (lumi on both, of course).
1- I'm using QPel and GMC with the 021703 build, in addition to lumi. Also VHQ set to 4.
2- The slow-down is with 021703 build. It runs less than 1/4 the speed of 111402.
3- The 1st pass size with 021703 (with GMC) is larger than with 111403.
4- There is no discernable quality improvement with either live video or still frames.
5- By "normal curve compression" I meant I'm not using Alt.CC or external curve treatment. Just the plain-old "2 pass" tab on the CODEC's settings page.

Actually, looking at flat surfaces I now notice that 021703's output is grainier in "flat" areas of the picture (i.e. a wall). And before anyone jumps to any conclusions, the grain is *not* in the source (which's the same .AVS, remember), nor in the 111402 output. So I'd consider this a micro-example where 111403 produces better output than 021703 (I consider "better" in a video CODEC to mean "closer to source"). I'll gladly post samples if anyone wants them and can tell me how (where?) to post in a loss-less image format (that's also reasonably sized... TIFF should do the trick, I believe).

Please also note that:

0- I wasn't living under a rock for the past 3 months. I've been trying new builds often (more than once a week, I think) to see if I could get an increase in quality that would justify the drop in speed. To be honest, I don't remember *any* increase in quality, let alone one that would justify such a drastic (IMO) slow-down.
1- I'm not a week-end encoder, nor have I started encoding last week (or even 3 months ago). I make DVD-rips almost daily and my first serious encode was made using NanDUB v0.20 (I just looked that up, in case you're wondering).
2- This is *not* a "your work suxxx!!" post. I love XviD. I think it's the greatest thing since the Pentium. I very much appreciate the work done by the developers and I don't presume to know 1/10 what they know (not about video CODECs, anyway...). It's just that I've had this growing, nasty feeling I'm missing out on something for the past 3 months and I just can't take it any more.

Again, any help is most appreciated.

[Koepi]

I assume you know the search button.

search for GMC in combination with VHQ.

If you read the thread, come back and tell us where you're attempts are flawed ;)

Regards,
Koepi

[TripleA]

Hmmm... I had a feeling I should check what this VHQ thing was before using it. I really should have done that before posting my question shouldn't I? Just that I saw it first just when I was setting up this test encode and it was really late at night, and I was really tired, etc. etc.

Anyway, the slow-down/not-good-enough part is probably still valid. But I'll have to do another test encode to check for that.

I'll get back to you when I'm done with that.

[Teegedeck]

BTW, again I'd like to advertise the use of 'linear scaling'. Really. :)

[Koepi]

I have to emphasize Tee's objection :)

Regards
Koepi

[Defiler]

Originally posted by Teegedeck
BTW, again I'd like to advertise the use of 'linear scaling'. Really. :) I've been confused by this in recent builds. What is currently considered to be the best scaling setting? I got the impression that external scaling had fallen out of favor.. The search results are clouded by a lot of outdated threads on this topic.

[Koepi]

Well, together with the dynamic I/P/B-decision external scaling isn't of much use - but honestly, I didn't test that.

If you hit "load defaults" in your xvid build it'll get setup to use internal linear scaling, whcih should suffice. But I'd be interested to see external linear scaling results with bframes as well!

(btw., I have an "internal" statsreader-hack which makes it possible to resume aborted 1st pass [that happened once to often to me now ;) ] as it can merge 2 stats-file into a new third one... if there's interest for it I could clean it up and release it.)

Best regards
Koepi

[Defiler]

Originally posted by Koepi
If you hit "load defaults" in your xvid build it'll get setup to use internal linear scaling, whcih should suffice.Thanks. This is what I have been doing, and I'm glad to hear that I wasn't doing anything stupid. Heh.

I am considering making a "crosswalk" document for submission to the FAQ that would clear up which features are incompatible with each other. (For example: Can't use GMC with VHQ, can't use packing and B-frames at the same time, etc.)
Is there a list somewhere of which features are /not/ safe to enable in the first pass? I know that there was a time when lumi-masking was unsafe for use in the first pass, because it would double the intended effect..

[TripleA]

I'm back!

I did another bunch of test encodes. There was another episode of "too late, too tired" and I made a small mistake, hence the slight delay. But anyway, I did another 2 pass with VHQ off, lumi, chroma motion, QPel & GMC on. This, btw, is the same as my original flawed test, only with VHQ off. This resulted in an avg. quant. of 3.640 and a 1st-pass size of 1474348881B. I did another 2-pass with exactly the same options selected as with my 111402 encode. That is, with only lumi selected. The results were an average quantizer of 3.739 and a 1st-pass size of 1556135551B. I also did a few 1st-pass-only runs to check speed with other various options selected... And I think a summery table would be a fabulous idea right about now! ;)

Code 1st-pass Size (KB) Avg. Quant. Time (per-pass, hours:minutes)
111402 1351078 3.547 1:00
021703.00 1383296 3.522 4:19
021703.01 1439793 3.640 2:01
021703.02 1519663 3.739 1:42
021703.03 1454757 N/A 2:05
021703.04 1393938 N/A 4:32

About those codes: 021703.00 was my original flawed encode (scroll up to read details). .01 is with VHQ off, lumi, chroma motion, QPel & GMC on. .02 is with only lumi enabled (to contrast old & new lumi code). .03 is with lumi, chroma motion & QPel enabled. .04 is the same as .03 with VHQ=4.

I'll also attaching a nice sample of everything. It's a grid of 100x100 pixels rectangles. The sample is taken from frame #56670. The ones on top are all the same one (from the source) and the ones on the bottom are each from a different encode. Which do you think is "better"? (And I know motion video should not be compared with stills... but still! ;))

[Edit: Tried to fix language a bit...]

[DaveEL]

Originally posted by Koepi

(btw., I have an "internal" statsreader-hack which makes it possible to resume aborted 1st pass [that happened once to often to me now ;) ] as it can merge 2 stats-file into a new third one... if there's interest for it I could clean it up and release it.)

Best regards
Koepi

avs2avi http://forum.doom9.org/showthread.php?s=&threadid=36768 will allow this too (and allow resuming of the second pass also). Any chance i could take a look at the source to see how you handle B/skip/pad frames when appending files so i can finish off the avs2avi resuming support (if i ever get any time to finish avs2avi that is, my university seem to think im supposed to do work for them in my final year rather then write video encoders :) ).

DaveEL

[Defiler]

Originally posted by DaveEL
my university seem to think im supposed to do work for them in my final year rather then write video encoders.Man.. That's just weird. Time to change schools.

[TripleA]

Originally posted by DaveEL
my university seem to think im supposed to do work for them in my final year rather then write video encoders.


Just tell them it's Important Work to further the human race's development... ;)

[TripleA]

Eureka!

I have finally found my culprit! It's actually very clearly hinted at by my last table. But it took a leap of faith to another build to confirm things.

It's the "new" (3 months old...) lumi code. I'll need to do some further testing with a movie that's actually dark, but almost-half-speed with a lower compression to boot is not to my taste. So, Koepi, any chance you may set it so one can choose which lumi code to use? If it's not too much work (I mean, if the changes don't get CVSed, you'd have to make them every time you want to make a new build, right? And I can easily just go on using uManiac's insta-build... More on the bleeding-edge, anyway ;))

Thank you all for your patience.

[sam_b]

You REALLY want to use the old code? I'm talking about output quality, not compression here. The new code redistributes bits, rather than just taking bits away. Do not assume the visual quality will be the same with and without the new lumi, as was meant to be the case with the old lumi.

[TripleA]

As I said, I'll need to try this on a dark movie to see if new code is actually better (for me, I mean). But right now, with a movie like Stuart Little 2 (of which I now have 8 different encodes, btw) it seems to me that quality is actually hurt by this lumi code (the issue with flat color areas). And with such a slow-down to boot, I'm sticking with the old code. For now, at least. And if memory surves me right, the whole reason for lumi masking was to increase compressability...

I will post further findings here. I'm thinking it's probably time I re-encoded The Others or Panic Room... ;)

[Teegedeck]

Well, for high-bitrate encodes I agree that lumi-masking (ReferenceDivX-code) doesn't help anything. I'd not recommend it. And it doesn't save bits as it wasn't meant to do that.