XviD 29082002-3

[Koepi]

Hi,

I forgot to make a bigger announcement here ;)

XviD-29082002-3:
- Updated to Statsreader 1.1 - it's more precise now.
- Fixed macro block skip-threshold.
- Added a "new modulated HQ (experimental)" mode for modulated quantizers

The new modulated quant mode should be tested, please.
It's working in the opposite direction than the old modulated quantizer routine:
It uses quantizer type h263 for quantizers from 1 to 3, and above MPEG.
The idea is that h263 gives a "cleaner" image for those quantizers, while the noise which gets introduced through the MPEG quant. type makes blockyness less visible (compare a clip encoded with each quant type at quant 12-16 to make the effect easily visible).

Thanks for your help,

best regards,
Koepi

P.S.: if you already have XviD-29082002-2 then you just need to fetch the new statsreader from the statsreader thread to be up-2-date ;)

[ookzDVD]

@Koepi,

Thank you for your effort.

Btw, talking about your StatReader,
could you summarized in the simple word how to use the .stat result ?

From other thread I found that I should not use the MVhints
in the first pass ?

So... after I got the first .stat file, I input it to the StatReader,
scale to the certain size, and save the .stat to new.stat

...And then use the 2nd-pass-Ext and then input the new.stat ?

Is there any settings in the 2nd-pass-Ext should be properly configured ?

Thank you.

[HarryM]

Originally posted by Koepi
- Fixed macro block skip-threshold.
- Added a "new modulated HQ (experimental)" mode for modulated quantizers
[/B]


22 hours of my encode-time are unavailing :)

Must I repeat first passes too? Or only second passes again be sufficient?

[R3g]

Originally posted by Koepi
Hi,

I forgot to make a bigger announcement here ;)

XviD-29082002-3:
- Updated to Statsreader 1.1 - it's more precise now.
- Fixed macro block skip-threshold.
- Added a "new modulated HQ (experimental)" mode for modulated quantizers


Well, I couldn't find out what this macro block skip-treshold is. Could you please explain it, or at least tell me in which thread it is discussed ? The search for "skip-treshold" gave no results !

[lovelace]

This post now is like cutting my own fingers, but:

Hi Koepi,

this new codec now works fine with K6-2 Machines.

Fantastic for me, but maybe you forgot to optimize, like you did before?

But your statsreader 1.0, 1.1 doesn't work here. While starting to display the curve, it gives an exception fault. (Maybe also K6-2 problem; but wanted to inform you)

PS: Sorry for my insistence last time. Won't happen again. Became nicer ;)

[unplugged]

Originally posted by Koepi
- Fixed macro block skip-threshold.
It regards the same thing that you further fixed (starting from CVS) with the first 18-08-2002?
I don't remeber exactly... I think something about a db value.

[rui]

Ok, about the new modulated mode.
Since my small trailer test wasn’t giving me much help here (to small, so most of it was being done in mpeg quantizer), i decided to test this way: i made two tests with the same exact settings at fixed quantizer 3, one using h.263 and the other using mpeg.
Then i opened Vdub and compared the same frames in both tests.

My conclusions (subjective):

The test using h.263 has less “noise” around edges. This is the most significant point in his favour to me. The detail, since the quantizers used now with h.263 are or 2 or 3, is almost all there.
I say almost because in some situations (for example the wrinkle in faces) mpeg still retains a tiny more detail.
That’s it, as always critics are welcome, mainly because i don’t know if the testing method is the most correct.

[iago]

Originally posted by Koepi
P.S.: if you already have XviD-29082002-2 then you just need to fetch the new statsreader from the statsreader thread to be up-2-date ;) So, my test going on with XviD-29082002-2 and statsreader 1.1 (which means XviD-29082002-3) doesn't lack anything ;).

Testing Fight Club again with external curve compression, default altCC with mrq:100, h.263/h.263, lumi masking both passes, quantizers capped 2-4/2-8, credits 20quant, aiming for 746000kb.

Especially I'm curious about the credits treatment and oversize/undersize (predictability) issue.

Will take some four hours for second pass to finish. I'll report back the results then.

best regards,
iago

[Koepi]

As zulu reported on that thread, my matrix encode came out ~3mb oversized, too! but the overflow reported was just 532 bytes over targeted size. This is all strange.

The old stats files are useless, you have to redo them.

About MB skip:
syskIn reset the macro block skip threshold to 3 again (we had that before, decided it was crap, and set it to one) - and he added three more conditions to be checkede for MB skip. This results in a smaller file - BUT it removes details (MBs can be skipped in encoding if the change is below a certain threshold).

Hm, I even forgot to optimize the binaries? That's quite possible, it's all derived from the sources where I ried to fix SMP support, and I might have forgotten to change the settings for the usual "release" config.

The statsreader output stats-file has to be added as 2nd pass ext. stats file _additionally_ to the first pass stats file in XviD. To get a most pure linear scaled curve you have to either use "regular" CC with high/low set to 0, or altCC with automatic Min. rel. Quality at 100 _or_ automatic MRQ and strength set to 0.

*phew*

I hope I didn't forget about something now :)

Best regards,
Koepi

[Koepi]

LOL.

You're right, I forgot the optimizations, too.
and the MB skip threshold before.

...

Maybe I'm getting old ;)

Expect a new binary soon. Only change: re-optimized ;)
Sorry that it doesn't run on a k6 anymore then :-/

Regards,
Koepi

[Koepi]

XviD-30082002-1:
- Oops. Optimized for speed again *caugh*. I forget _too_ much lately.
- Tried to be more aggressive in overflow fix for credits/2nd pass external.


*whistle* I hope noone notices... ;)

Regards,
Koepi

[MoonWalker]

:)..We are humans, not robots (or automatons if you have played Syberia :) )

But. I am currenlty doing a old/new modulated test with the unoptimized version..I am on the 2-pass..I suppose it ok, or I have to stop it an do with the new version??

MoonWalker

[iago]

Originally posted by Koepi
XviD-30082002-1:
- Oops. Optimized for speed again *caugh*. I forget _too_ much lately.
- Tried to be more aggressive in overflow fix for credits/2nd pass external.@Koepi,

As I get it ;), the only difference of the new binary from 29082002-3 is that the new one is "optimized for speed" and a "new credits treatment" is employed, right?

So, I don't have to abort my test with 29082002-3 (that is 29082002-2 and statsreader 1.1 :)) when only two hours left to see the results! ;) (OK, as you and Zulu have reported, taking the risk of getting oversized of course ;), but no other deficiency in terms of quality?)

best wishes,
iago

(P.S.: LOL! Otherwise, I'll start a "sbc" encode with my first pass stats file, then do a fresh two pass with 30082002-1 in order to see and compare the results ;). Everything changes too fast these days, even I, a serial tester ;), cannot catch up with this speed! :))

[Koepi]

You can continue your tests with the old version.

Maybe if we all get a 3MB oversize there's something which can be done about it...

(damn, you all caught me... ;) )

Thanks so much for testing and supporting! I can't express my appreciation of it often enough!

(redoing matrix mpeg/mpeg test now... the h263 version looks SO yummie! somehow the threshold is really bad, now the backgrounds are as stable as they can be regarding the noise in that movie... the details are absolutely fine... and even if a few frames get up to quant 11, that's in scenes with small framesizes, not _too_ noticable [still they're annoying me, i don't like those numbers ;) ] ).

Best regards and thanks a million,
Koepi

[rui]

Koepi, just asking out of curiosity: ;)

Would it be possible to incorporate your statsreader into xvid itself?

I mean, when he/she choosed external 2nd pass, xvid would default to your statsreader, using the credits parameters that we already had configured in 1st pass, only asking the user to input, as always, the desired filesize and the other parameters that we can input when using the external 2nd pass.
Then it would scale the curve, not using the internal code, but the statsreader code, and create automatically the, for example, video2.stats in the folder we would indicate when configuring the 2nd pass external encode.

The advantage of this would be to automate the encoding process in the same way that it is automated when choosing internal encoding(using jobs in Vdub, for example).

[The Edge]

You can continue your tests with the old version.
@Koepi
Can I still use the stat file made by an older xvid release for use with a newer release? U probably answered this before I'm sure. Sorry.
I'm doing some testing of my own at the moment.
I would hate to have to a new stat file for every update as you update the code alot. Thanks for your hard work by the way:)

Edge

Edit
Or if anyone else can answer this.
Cheers

[Koepi]

Unfortunately, the compressability between builds from 18.08.-29082002-1 changed :-/ stats files generated with those builds will work, but give you a "wrong" bitrate distribution. Sorry to have to give bad news :/

Best regards,
Koepi

[iago]

@Koepi and all,

My Fight Club test with 29082002-3 (29082002-2 and statsreader 1.1) also came out (~4.5 mb) oversized :(.

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 16039 Times, Percentage Used : 8.18%
Quant 3 Used : 169464 Times, Percentage Used : 86.46%
Quant 4 Used : 10448 Times, Percentage Used : 5.33%
Quant 5 Used : 53 Times, Percentage Used : 0.03%
Quant 6 Used : 5 Times, Percentage Used : 0.00%

Average Quantizer Used for Movie : 2.972

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 4168 Times.
Quant 21 Used : 20 Times.

Also, (just a personal observation but) I have doubts that the quality is really better with h.263/h.263 than MPEG/MPEG with quantizers <4. I must test MPEG/MPEG too, in order to better see it.

As a result, it's time to test the 30082002-1 build to see if it solves the oversize/undersize issue, with the new more aggressive +/-2 credits treatment.

kind regards ;)
iago

P.S.: Can we use the 29082002-3 first pass stats file for second pass with 30082002-1 build?

[Koepi]

I thought I implied that - but yes, the 29082002-2 and 29082002-3 stats files are valid with 30082002-1 ;)

Thanks for testing!

Best regards,
Koepi

[iago]

OK ;). Starting a fresh two pass with the new 30082002-1 build using MPEG/MPEG quantization. The exact same parameters (with external curve compression) as the previous test encode will be used except the choice of quantization.

Then will come a comparison (maybe with screenshots, maybe just my visual observations ;)) between the h263/h263 and MPEG/MPEG encodes.

Of course, also hoping to see that the oversize/undersize issue is solved with the new more aggressive credits treatment :).

ciao,
iago

[Emp3r0r]

i apologize for my lack of tests but it seems to me it would be benificial to quality if you kept MPEG quants for quants 1 to 3 and h.263 for more highly compressed frames since h.263 compresses better. Does this not make sense?

[Koepi]

That's definatly a matter of taste, empe.

I'd give it a shot since the argumentation is reasonable. It has the "experimental" attached since it _is_ experimental ;)

Regards,
Koepi

[MoonWalker]

Some results of the new ModulatedHQ.

I have done The Matrix, 2-pass internal, motiob 6,cc used with high/low : 0/0(linear scaling), I-Frames : 2-5,all the rest at default.

H.263/ModulatedHQ


Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 726 Times, Percentage Used : 0.39%
Quant 3 Used : 76500 Times, Percentage Used : 41.09%
Quant 4 Used : 97996 Times, Percentage Used : 52.63%
Quant 5 Used : 10536 Times, Percentage Used : 5.66%
Quant 6 Used : 198 Times, Percentage Used : 0.11%
Quant 7 Used : 65 Times, Percentage Used : 0.03%
Quant 8 Used : 56 Times, Percentage Used : 0.03%
Quant 9 Used : 42 Times, Percentage Used : 0.02%
Quant 10 Used : 58 Times, Percentage Used : 0.03%
Quant 11 Used : 10 Times, Percentage Used : 0.01%

Average Quantizer Used for Movie : 3.646

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 9967 Times.

MPEG Quantization Type Used 118928 timed, Percentage Used : 60.63%

H.263 Quantization Type Used 77226 timed, Percentage Used : 39.37%

Quantizers prevented from rising too steeply 118 times

h.263/Old Modulated


Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 75 Times, Percentage Used : 0.04%
Quant 3 Used : 83028 Times, Percentage Used : 44.59%
Quant 4 Used : 99523 Times, Percentage Used : 53.45%
Quant 5 Used : 3298 Times, Percentage Used : 1.77%
Quant 6 Used : 106 Times, Percentage Used : 0.06%
Quant 7 Used : 62 Times, Percentage Used : 0.03%
Quant 8 Used : 37 Times, Percentage Used : 0.02%
Quant 9 Used : 31 Times, Percentage Used : 0.02%
Quant 10 Used : 24 Times, Percentage Used : 0.01%
Quant 11 Used : 4 Times, Percentage Used : 0.00%

Average Quantizer Used for Movie : 3.576

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 9967 Times.

MPEG Quantization Type Used 93070 timed, Percentage Used : 47.45%

H.263 Quantization Type Used 103085 timed, Percentage Used : 52.55%

Quantizers prevented from rising too steeply 6 times

In my opinion, h.263/Old Modulated is better for the partivular movie cause it smoothes out the blocks generated by the high quants.

Another think is that with H.263/ModulatedHQ I had 118 "prevetions" whereas with h.263/Old Modulated I had only 6.

@Koepi
I don't know the specs of the MPEG4 very well, but isn't supposed the an MPEG4 should end with an I-Frame? I say this because both tests end with P-Frame.

MoonWalker

P.S Still got consecutive I-Frames :(

[Koepi]

Hm, I don't think it's necessary to end on an Iframe - but b-frame is forbidden for sure! ;)

Hm, consecutive IFrames is unavoidable in Matrix, unless you play with "min. keyframe distance" (which is a bad thing i think).
I use iframe quant. capping and a iframe boost of 0, cc kf threshold 10, bitrate reduction 30-35%, this helps it a lot.

I hope this helps you too!

Best regards,
Koepi

[colasonic]

To end on an I-frame may be reasonable, since it's frenquently seached.

[iago]

Originally posted by iago
Also, (just a personal observation but) I have doubts that the quality is really better with h.263/h.263 than MPEG/MPEG with quantizers <4. I must test MPEG/MPEG too, in order to better see it. Vow, now I see what a big fault it is to say that! Later on, I realized that I'd watched the h.263/h.263 encode without any post-processing (Nic's XviD decoder filter disabled) and because of that I'd noticed some more blocks than usual!.. Then I watched it with full post-processing and I must admit that it "is" beautiful :). Sorry for the mistaken comment before.

However, it is a "matter of taste" at the same time, as Koepi said somewhere above; so I will finish my MPEG/MPEG encode, take some screenshots from both encodes and post them for comparison, so that everybody can decide for himself/herself which one is better ;).

Started the second pass of the MPEG/MPEG encode and looking forward to seeing the results, especially in terms of oversize/undersize issue.

kind regards,
iago

[HarryM]

I don't see why you present always the percent occurrence of quantizer distribution...

This hav'nt (surely) relations to subjective (human eye) quality!

[Koepi]

Thanks for your destructive critics harry!

I'm proud that you lack respect for every basic communication rule we have here.


Next time, give _constructive_ criticx, make it BETTER than what you think is wrong.

*GNARF*

[iago]

@Koepi and all,

Yes, my MPEG/MPEG encode with 30082002-1 build also came out ~4.5mb oversized, just like the h.263/h.263 one before with 29082002-3 build :(

regards,
iago

P.S.: I'm preparing the screenshots for comparison between h.263/h.263 and MPEG/MPEG encodes.

[HarryM]

Originally posted by Koepi
Thanks for your destructive critics harry!

I'm proud that you lack respect for every basic communication rule we have here.


Next time, give _constructive_ criticx, make it BETTER than what you think is wrong.

*GNARF*

I'm sorry to all.
But from simple list of 'quantizers used' you can't identify quality of rip.
Or do you?

Important is 'Why, when, how,...' THIS quantizer used at just THIS frame (scene)...

Mean no harm, but I thoroughly tested newest compilations (versions) XviD. I think (sorry), that using 'modulated' is "way to hell"...

a) mpeg quant is'nt more accurately on details, only better describes colors (e.g., problematic body-colour is better with mpeg quants)
b) mpeg quant much more errorized at edges (i don't know why) than h263 quant
c) at perfect noiseless rips is mpeg quant frame-size smaller(!) (about 5%) than h263 quant. For lower quant (6, 7, 8,...) maybe contrariwise...
d) with mpeg quant is movie in one way livelier, more natural (but not more detailed) than with h263 quant...

These my experiences.
I know safely only this (at present).
For noised movies DO NOT USE 'modulated'!!!

[Nic]

@Harry:
Modulated exists because people wanted it, so Koepi wrote it. & many have liked it & used it with great results. Please respect that. If you've had bad results, then thanks for tips & info of your experiences, but please try & express things in a "constructive" way (as you almost did in your second post)

Also quantizer is a loose way of determining quality, e.g a quant of 2 is always going to look better than a quant of 6. determing picture quality (especially when concerning hvs) is a complex thing. PSNR is another measure, but that would be slow & OTT for this. I hope this explains why the quantizer is used as a basic measure....

Important is 'Why, when, how,...' THIS quantizer used at just THIS frame (scene)...
That's very true & good testers like rui & others do try & give as much info as possible....

Thanks,
-Nic

[Koepi]

Thanks Nic for giving such a nice explanation! :)

Best regards,
Koepi

[HarryM]

And what 'simplest modulated' method?

Only simply altering between 'h263' and 'mpeg' quant? Frame by frame?

Sorry for my stupid idea, but this is a good logical compromise method between 'new modulated' and 'old modulated'...

[Koepi]

if you toggle that frame-by-frame you'll loose much bitrate for the movie (140bytes per frame) which then can't be compensated for by the "nice" sideeffects this quant type switching can have.

Regards,
Koepi

[HarryM]

Originally posted by Koepi
if you toggle that frame-by-frame you'll loose much bitrate for the movie (140bytes per frame) which then can't be compensated for by the "nice" sideeffects this quant type switching can have.

Regards,
Koepi


140 bytes per frame lost?
I don't know this.

Ordinaly (compressibility=50%) I have 40-50% toggles of sum frames (mainly toogles between quant3 and quant4). This is loss of cca 1 MB for every 10 minutes of movie!

[-h]

I just checked the VOL header length, it's more like 35 bytes. I think I said 140 bytes some time ago, I'm not sure where I pulled that figure from though.

-h

[HarryM]

Originally posted by -h
I just checked the VOL header length, it's more like 35 bytes. I think I said 140 bytes some time ago, I'm not sure where I pulled that figure from though.

-h

Thank for exact information!

[Koepi]

Thanks for clearing that up -h, I think you calculated that value from the 2 quant matrices which are stored at a quant matrix change (64x2=128 - and then some overhead).

dunno how it'll be handled now since the decoder can't know the quant matrix, but maybe the matrices are stored somewhere in the stream and then reused by a single reference number.

Regards,
Koepi

P.S.: i always came to ~3.5MB overhead at my calculations for a "usual" movies

[iago]

@Koepi

(Well, after solving the problem on screenshots in the other thread ;)) I want to mention some interesting point regarding the 30082002-1 build and filesize issue.

I made a (h.263/h.263 - lumi both passes) rip with 30082002-1 binary using internal curve compression and the default AltCC parameters, with quantizers capped 2-4/2-8, payback proportionally, and 20quant. credits. The final movie came out ~1mb oversized, spitting out the below report:

---------------------------------------------------------------------
DebugView analyzer for XviD codec v0.11 by MoonWalker
e-mail : s_ilias@gmx.net

Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 833 Times, Percentage Used : 0.41%
Quant 3 Used : 35205 Times, Percentage Used : 17.16%
Quant 4 Used : 119310 Times, Percentage Used : 58.14%
Quant 5 Used : 46050 Times, Percentage Used : 22.44%
Quant 6 Used : 3747 Times, Percentage Used : 1.83%
Quant 7 Used : 53 Times, Percentage Used : 0.03%
Quant 8 Used : 13 Times, Percentage Used : 0.01%

Average Quantizer Used for Movie : 4.082

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 7333 Times.

H.263 Quantization Type Used 212544 timed, Percentage Used : 100.00%

Quantizers prevented from rising too steeply 16 times


Intra-Frame (Key-Frame) Quantizers
------------------------------------

Movie
-------
Quant 4 Used : 2277 Times, Percentage Used : 98.44%

Credits
---------
Quant 20 Used : 36 Times, Percentage Used : 1.56%

Number Of Consecutive I-Frames : 23

Inter-Frame (P-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 833 Times, Percentage Used : 0.40%
Quant 3 Used : 35205 Times, Percentage Used : 16.75%
Quant 4 Used : 117033 Times, Percentage Used : 55.67%
Quant 5 Used : 46050 Times, Percentage Used : 21.90%
Quant 6 Used : 3747 Times, Percentage Used : 1.78%
Quant 7 Used : 53 Times, Percentage Used : 0.03%
Quant 8 Used : 13 Times, Percentage Used : 0.01%

Credits
---------
Quant 20 Used : 7297 Times, Percentage Used : 3.47%


Frame Analisis
----------------

Number Of Intra-Frames (Key-Frames) : 2313
Number Of Inter-Frames (P-Frames) : 210231
Total Number Of Frames : 212543

1.09% of the Movie is Intra-Frames (Key-Frames)
98.91% of the Movie is Inter-Frames (P-Frames)


Size Analysis
----------------

1-Pass Size : 1750943965 Bytes or 1709906 KBytes
Scaled Size : 762898835 Bytes or 745018 KBytes
Actual Size : 763703679 Bytes or 745804 KBytes

Usefull Statistics
------------------
Compressibility : 43.62%
Relative Quality of XviD avi : 48.99%
Absolute Quality of XviD avi : 93.75%
---------------------------------------------------------------------

I don't know if it can be considered an unpredictability somehow introduced into internal curve compression too, but I just wanted to inform you about the issue.

(Maybe due to capping quantizers 2-4/2-8 ?)

kindest regards,
iago


EDIT: BTW, credits size is ~7.5mb. Currently I'm doing another rip with internal curve compression with the exact same parameters above to see the results. If this one comes out OK, then it means I messed up something in the previous encode, which is also quite possible ;)

EDIT2: The above issue and results belong to the Blues Brothers rip. Now I'm doing another rip, Matrix, with internal curve compression to see the result.

[iago]

Well, this one came out the accurate size as usual, with only a deviation of ~100kb. I guess there occured a problem somehow in the previous encode.

So... it's time to test the new statsreader! :)