View Full Version : (XviD) Statsreader

Koepi
27th August 2002, 02:04
Ahoy,

I needed a little distraction (right word? dunno...), so I coded yet another stats file viewer.
And as I always wanted to go back to the roots again, I built in an amazing feature:

_linear curve compression for external xvid compression mode_!

Just enter your target filesize in MB, hit the "save as" button, give that baby a name and press "ok".
After that, you'll get the stats file for "2nd pass - external" mode in XviD.

Sure, you can use GKnot.

But this is
a) my first trial of doing that and
b) just 11kb big (zipped 8kb). No huge 7MB downloads just for a curve scaler... ;)

I hope you like it and send me plenty bug reports!

(Until now it's not at all tested. maybe it totally messes up. But the curve is nicely scaled at least ;) )

Best regards,
koepi
neo_sapien
27th August 2002, 04:19
Cool :)

Will this one increase the efficiency of bit distribution? Give us an even better quality picture? ;)

EDIT~w00t, apparently I'm a senior member now.
Koepi
27th August 2002, 04:35
Depending on your preferences this can give you better quality, right.

At least the quality should be more constant...

The nicest thing is, that quantizer ranges still get applied, and low bitrate situations get some compensation, too.... xvid's 2pass stuff rocks :)

I'm doing the marix (again) and let's see, I think it'll improve it.

Regards,
Koepi

P.S.: since it's so small I'll include it into my next binaries - if there will be any...
iago
27th August 2002, 08:07
@Koepi

Great news :). Gonna test it right away with Fight Club (as usual ;)).
BTW, are you using default altCC in your Matrix test with the external stats file?

thanks,
iago
ookzDVD
27th August 2002, 08:28
@Koepi,

Thank you for the tool,
even _xvid_ is stopped atm, but the tool is still on going. ;)
iago
27th August 2002, 08:46
@Koepi

I get this error message in VirtualDub when initializing the second pass, both in fast recompress and full processing mode. (Using avisynth 2.05, btw.)

regards,
iago
Koepi
27th August 2002, 09:18
Strange error, dunno where it comes from.

I manually start both passes and didn't check the job control on that...
how are you going to compress the stats file, output it to another filename (!important! XviD doesn't forgive if the first pass and external compressed stats file are identical ;) ) and in the same time start the second pass? ;)

Try entering that manually and start the jobs by hand, see if that helps it please!

And yes, I use all the rest my default settings. I did the credits at constant quant. 8 in the first pass, in the second pass it's not selectable - and that is a good sign :) I still have ~4hrs to wait from now to see the results, but I'm already impatient. I testet it on the first 2000 frames of matrix and those were the best results I got that far....

Thanks for finding the tool useful :)

Best regards,
Koepi
iago
27th August 2002, 09:28
@Koepi

That's the exact procedure I follow: I open the first pass stats file in statsreader, enter my desired filesize (634mb in this case), then I hit "save as" and save the file under another file name. Also, the tool doesn't automatically give the .stats extension so I type it as well: eg. "fightclb-ext.stats". (I also get the "scaled down to ...." confirmation.)

Then I go to XviD 2pass-2ndPass Ext. for the second pass and select the scaled file as my stats file, etc. The usual procedure I follow everytime ;).

Anything wrong with that procedure?

thanks,
iago
Koepi
27th August 2002, 09:35
uhm. nope, that's absolutely correct :-(

I have no idea where this comes from.
Did you check the file, if it's the same size as the first pass stats file?

But if you do it that way, I don't know why vdub complains about job control. Please delete your *.jobs file in your virtualdub directory and see if this helps!

Regards,
Koepi

EDIT: I just saw that you wrote "..and I choose that file as my stats file" - you have to just _add_ it as the file at the bottom which defaults to "\videogk.stats". The other field still has to contain your "original" first pass stats file!
iago
27th August 2002, 09:42
@Koepi,

I don't know how but it's OK now :). I think I messed up something before. Anyway, never mind ;), since it's working with no problem now! I'm looking forward to see the results too :).

best regards,
iago

EDIT: yes you're absolutely right, that's where I went wrong! ;) Stupid me...
Koepi
27th August 2002, 13:26
[1096] Quantizer distribution for 2nd pass:
[1096] Q:2:449
[1096] Q:3:76435
[1096] Q:4:96210
[1096] Q:5:12124
[1096] Q:6:949
[1096] Q:7:16
[1096] Q:8:3
[1096] Q:9:1
[1096] Q:10:1

Looks good to me... the "weapons rack" scene looks ok for my taste, but I have to check the whole movie. I'm amazed as there's still some noise of the source to see :)

EDIT: ouch. the credits stuff didn't work out, file is 9 MB oversized.... hrmpf.

Regards,
Koepi
rui
27th August 2002, 14:20
Well, after some experiences, i came to the conclusion that koepi's settings and using his new tool for external 2nd pass, gives us lower absolute quantizers. But using milan's simulation mode and choosing alt cc. parameters from there, but still using koepi's external 2ns pass tool, i can achieve higher number of frames with lower quntizers.

I didn't thought that the alt cc parameters still had influence in the encode, but they do.

I will explain:

##### koepi's settings:------------- milan simulation alt cc.
quant2 -- 92 ------------------------- 79
quant3 -- 36 ------------------------- 71
quant4 -- 112 ------------------------ 319
quant5 -- 365 ------------------------ 540
quant6 -- 595 ------------------------ 742
quant7 -- 873 ------------------------ 577
quant8 -- 795 ------------------------ 450
quant9 -- 383 ------------------------ 248
quant10 -- 212 ----------------------- 231
quant11 -- 81 ------------------------ 125
quant12 -- 25 ------------------------ 68
quant13 -- 1 ------------------------- 40
quant14 -- 0 ------------------------- 33
quant15 -- 0 ------------------------- 14
quant16 -- 0 ------------------------- 10
quant17 -- 0 ------------------------- 8
quant18 -- 0 ------------------------- 8
quant19 -- 0 ------------------------- 3
quant20 -- 0 ------------------------- 2

Now, all of you will say that using milan's simulation, i got much higher quantizers. True, but look at the number of 4, 5 and 6 quantizers i got. Using both koepi's 2nd pass external tool and milan simulation, i got much higher values for those quantizers. The quantizers from 13 to 20 are there, but in low numbers. Probably one doesn't notice them.
So, i believe is a personal choice. If you want a safe encode, one that doesn't use high quantizers, then use koepi's settings.
If you want to be brave :D, and take some chances, use koepi's tool with simulation mode to get the alt cc parameters. You probably can get access to much higher numbers of frames using lower quantizers, in spite of having also some frames that will use higher quantizers.

As always, critics are welcome.
Koepi
27th August 2002, 14:38
I assure you, the high quantizers will be visible.
I thought so too first, but I came to the conclusion that even on frames with "few" information the high quantization is visible.
This is especially true because that information is used to show the next picture...

But thanks for the test! :)

regards,
Koepi
rui
27th August 2002, 14:56
Ok, i have to correct some data that i posted in my earlier post. In the test using koepi's tool and milan simulator, i had enabled i-frame boost. Now i remade the test without i-frame boost, and the results are:

##### koepi's settings:------------- milan simulation alt cc.
quant2 -- 92 ------------------------- 95
quant3 -- 36 ------------------------- 57
quant4 -- 112 ------------------------ 327
quant5 -- 365 ------------------------ 528
quant6 -- 595 ------------------------ 650
quant7 -- 873 ------------------------ 583
quant8 -- 795 ------------------------ 539
quant9 -- 383 ------------------------ 271
quant10 -- 212 ----------------------- 231
quant11 -- 81 ------------------------ 122
quant12 -- 25 ------------------------ 72
quant13 -- 1 ------------------------- 33
quant14 -- 0 ------------------------- 33
quant15 -- 0 ------------------------- 18
quant16 -- 0 ------------------------- 6
quant17 -- 0 ------------------------- 2
quant18 -- 0 ------------------------- 3
quant19 -- 0 ------------------------- 0
quant20 -- 0 ------------------------- 0

It uses a little lower quantizers, but still there are some high quantizers. According to the expert (none other than our Koepi :D), these can mean trouble.
iago
27th August 2002, 15:06
Originally posted by Koepi
I'm amazed as there's still some noise of the source to see :) That's really interesting and nice. I hope I see the same noise (rather than blocks :)) in my test encode too...

best wishes,
iago
Koepi
27th August 2002, 15:15
Well, blocks are there, too. But I didn't find them annoying.

For example, before the guy at the computer hits the "enter" key (it's not the enter key, it's just any key... that key I'm always searching when asked to press it ;) ) and then the scene switches to the "loading program" where the weapon racks scroll in, his hand is full of blocks, but the keyboard is amazing stable and detailed.

I'm testing with h263 quant type on second pass ATM - and I hope I have some better news in that respect.

I think that on such "critical" movies the modulated quantizer type is a big no-no as every switch of quant type takes up 140bytes which could be better spent on the movie itself ;)

Regards,
Koepi
rui
27th August 2002, 15:53
About quantization modes: lately i am only using MPEG, even on dificult movies. I prefer the extra detail, and don't mind the little increase in blocks.
For smooth videos, i would go to divx5 ;)
iago
27th August 2002, 16:34
Originally posted by Koepi
ouch. the credits stuff didn't work out, file is 9 MB oversized.... hrmpf. Yes, my test finished with 7 mb oversize too ;), but I'm quite pleased with the quality though I used MPEG/Modulated quantization, which I believe will be even better after taking your advice and not switching between quantization type (MPEG/MPEG as rui does for example :)).

Predictability is indispensable to XviD, XviD is predictability! :) Koepi, please solve that credits issue, mate! Cause your tool really works great in terms of achieving better quality ;).

best regards,
iago
manono
27th August 2002, 17:52
Hi-

I also came out a bit oversized (aimed for 1301MB and got 1304MB-nothing important) on a movie with no end credits. But perhaps it's also due to having used Modulated Quant for the 2nd pass. I didn't know about switching Quant type adding size either. Assuming that the Modified Stats File is created as a straight percentage of the frame sizes of the original first pass Stats File, with no Motion or Luma Correction applied (unlike theWEF's DivX 3.11 version), then this is just what I've been waiting for.

Thanks a lot Koepi.
Koepi
27th August 2002, 18:07
ALL PEOPLE, FULL STOP _NOW_!

:)

Just talked to foxer again and we came up with some issues. Seems like we're getting closer to our target ;)

Ok, this is what is most important:
DON'T use MV hints.

Second:
DON'T use alt CC. OR set it like this: altCC of mrq100 or str0.


Third:
IF you don't use altCC, SET "old" CC low/high to 0 (!!!)

I prefer "payback proportionally". Set the payback delay to something like 10-12 times your framrate (thus 10-12 seconds payback delay).

I'm redoing the matrix _again_ now with second pass h263 quant.
All my problems with stacked up overflow are gone (shame on you, MV hints! why do you have to produce an error which has to be compensated? ;) ) Well, ok, at least that's the reason for _slightly_ worse quality, the mv hints stuff. This is a known issue, but with our games we play here it becomes severe - the effect isn't that bad when doing just straight-forward encodings.

The scene based keyframe overflow treatment is active anyways, so you get the most linear curve possible this way for now.

Ok, if you're not "stuffed up" already, please change your test settings ;)

Thanks again for all your help,

best regards,
Koepi

EDITED: to reflect that altCC can achieve that, too. the important point is NOT to use MV hints.
Koepi
27th August 2002, 18:13
Ah, one addition:

In this scenario there _shouldn't_ be any difference between internal and external compression.

Would be nice if you could check for that, too!

Thanks,
Koepi
Bulletproof
27th August 2002, 18:21
Lately I've been finding that the CBR mode in XviD works quite well.
rui
27th August 2002, 22:06
Humm.. By using default cc with high/low at 0, with no hme, i got this:

Quant 2 Used : 2 Times, Percentage Used : 0.06%
Quant 4 Used : 36 Times, Percentage Used : 1.01%
Quant 5 Used : 296 Times, Percentage Used : 8.29%
Quant 6 Used : 633 Times, Percentage Used : 17.73%
Quant 7 Used : 976 Times, Percentage Used : 27.34%
Quant 8 Used : 881 Times, Percentage Used : 24.68%
Quant 9 Used : 339 Times, Percentage Used : 9.50%
Quant 10 Used : 221 Times, Percentage Used : 6.19%
Quant 11 Used : 43 Times, Percentage Used : 1.20%
Quant 12 Used : 99 Times, Percentage Used : 2.77%
Quant 13 Used : 6 Times, Percentage Used : 0.17%
Quant 14 Used : 10 Times, Percentage Used : 0.28%
Quant 15 Used : 6 Times, Percentage Used : 0.17%
Quant 16 Used : 9 Times, Percentage Used : 0.25%
Quant 17 Used : 2 Times, Percentage Used : 0.06%
Quant 18 Used : 7 Times, Percentage Used : 0.20%
Quant 19 Used : 2 Times, Percentage Used : 0.06%
Quant 20 Used : 2 Times, Percentage Used : 0.06%

Average Quantizer Used for Movie : 7.540

I think that it's inferior to what i managed to achieve using alt cc and your defaults, that i posted earlier. Am i missing something?
Koepi
27th August 2002, 22:08
No, but we're testing, right?

:)

Regards,
Koepi
rui
27th August 2002, 22:14
Ok, boss, i will keep on testing and experimenting :), and will keep an eye (rather both:D) to any new developments and advices you can come up with.
Koepi
27th August 2002, 22:18
Just an observation I made:
You seem to test with a short sample (few thousand frames), right?

This gives you a slight idea how things work, but for propper curve treatment testing you should use a whole movie - one thing that didn't change compared to the nandub times ;)

Thanks for all your efforts!

Best regards,
Koepi
Koepi
27th August 2002, 22:38
Hm.

[900] Quantizer distribution for 2nd pass:
[900] Q:2:1322
[900] Q:3:95437
[900] Q:4:84934
[900] Q:5:3960
[900] Q:6:391
[900] Q:7:37
[900] Q:8:16
[900] Q:9:18
[900] Q:10:59
[900] Q:11:14

I'm getting a little annoyed with the time.
Well, I'll code a more precise "scaler" routine and add support for credits, maybe this helps (credits just as constant quantiser unfortunately).

Regards,
koepi
iago
27th August 2002, 23:05
@Koepi

Well, the first point, I've never used Hinted ME so far, so the source of my problems is certainly not due to MVH :).

Second, now I'll take your new advices and start a fresh test encode (with Fight Club again ;)), without using external curve compression, setting mrq to 100 with default altCC parameters (medium/500/90), payback proportionally, using a payback delay and max i-frame interval of 10-12 times my framerate (which I already do in my encodes), quantizers MPEG/MPEG and capped as 2-6/2-16, with no lumi masking at all.

Next, I will compare the results (with screenshots maybe) with a previous encode of the same movie encoded with the old routine (strength 30 with default altCC, lumi in both passes, payback with bias) and with MPEG/Modulated quantizers capped the same as above.

I really wonder how the final encode will be in comparison to the previous one. I'll report back the results then.

best regards,
iago
Koepi
28th August 2002, 00:08
I compiled a new version of the stats reader which should be a little more precise in downscaling.
Also, if you used fixed quantizer credits, it can be setup to not scale those.
Find it attached...
Koepi
28th August 2002, 00:24
Ok, while I started the first pass for my mpeg/mpeg quant-type test, I'm watching my h263/h263 encoding. It looks better without any doubt due to switched off mv hints (main difference: the backgrounds are more stable).

Ok, off for checking more... :)

The mpeg/mpeg test is a little unfair, becaus ethere it can reach the desired filesize (hopefully) due to unscaled credits ;)

Regards,
Koepi
iago
28th August 2002, 00:37
@Koepi

So, with the latest version, I'm modifying my test setup ;). Now it is:

"...a fresh test encode (with Fight Club again ), using external curve compression with the new statsreader, setting mrq to 100 with default altCC parameters (medium/500/90), payback proportionally, using a payback delay and max i-frame interval of 10-12 times my framerate (which I already do in my encodes), quantizers MPEG/MPEG and capped as 2-6/2-16, with no lumi masking at all.

OK? :)

regards,
iago
Koepi
28th August 2002, 00:44
Ok! I'm looking forward to the results! :)

Regards,
Koepi

P.S.: I#m still a little unsure if the quality boost i can see here is due to switched off MVHints or due to using fixed quantizer type (instead of modulated...) - or the mix of both. I think I'll leave my hands off both in future :)
rui
28th August 2002, 08:48
Originally posted by Koepi
Just an observation I made:
You seem to test with a short sample (few thousand frames), right?
Right. It's my favorite "The Replacements" trailer, the most demanding piece of video i can find.
Originally posted by Koepi
This gives you a slight idea how things work, but for propper curve treatment testing you should use a whole movie - one thing that didn't change compared to the nandub times ;)

Thanks for all your efforts!

Best regards,
Koepi
I am sure you are right, but most of the times i am at work, and here i have a p2-350 to work with. The only way of conducting some tests is using trailers or chapters from movies :(
Koepi
28th August 2002, 08:52
Thanks rui :)

That's ok with me, I just wanted to mention that you don't necessarily get "the whole picture" with a trailer. Any tests are welcome, don't misunderstand that! :9

Best regards,
Koepi
Emp3r0r
28th August 2002, 09:42
I'm testing with Planet of the Apes and using iago's setup. I prefer to use h.263 with bicubic as it seems to produce a better (less noise) picture than mpeg or modulated. I'll post my results tommarrow.

@koepie: how do you generate the Quantizer distribution for 2nd pass Information
Koepi
28th August 2002, 10:21
Foxer added some statistical info to XviD's 2pass code, it spits that info out like all other info, capturable via Debugview.

Regards,
koepi
rui
28th August 2002, 11:06
Well, regarding the internal vs external results, i made another small test, using default cc and o% high/low, with payback bias. The results are very diferent from those i got yesterday, using payback proportional:

Internal:

Quant 4 Used : 82 Times, Percentage Used : 2.30%
Quant 5 Used : 390 Times, Percentage Used : 10.92%
Quant 6 Used : 755 Times, Percentage Used : 21.15%
Quant 7 Used : 1065 Times, Percentage Used : 29.83%
Quant 8 Used : 824 Times, Percentage Used : 23.08%
Quant 9 Used : 300 Times, Percentage Used : 8.40%
Quant 10 Used : 115 Times, Percentage Used : 3.22%
Quant 11 Used : 23 Times, Percentage Used : 0.64%
Quant 12 Used : 13 Times, Percentage Used : 0.36%
Quant 13 Used : 3 Times, Percentage Used : 0.08%

Average Quantizer Used for Movie : 7.046

External:

Quant 4 Used : 104 Times, Percentage Used : 2.91%
Quant 5 Used : 480 Times, Percentage Used : 13.45%
Quant 6 Used : 892 Times, Percentage Used : 24.99%
Quant 7 Used : 1068 Times, Percentage Used : 29.92%
Quant 8 Used : 717 Times, Percentage Used : 20.08%
Quant 9 Used : 225 Times, Percentage Used : 6.30%
Quant 10 Used : 52 Times, Percentage Used : 1.46%
Quant 11 Used : 16 Times, Percentage Used : 0.45%
Quant 12 Used : 13 Times, Percentage Used : 0.36%
Quant 13 Used : 3 Times, Percentage Used : 0.08%

Average Quantizer Used for Movie : 6.806

So, there is a small diference between internal and external, using Koepi's latest statsreader. It seems that external is a little bit better. Take these results with a grain of salt, because they were achieved using only a trailer, not a full movie.
iago
28th August 2002, 11:37
@Koepi and all,

The results I had with Fight Club with the above mentioned test setup using external curve compression and entering credits range in the "statsreader" as well confused me a bit, especially regarding the credits treatment: although the encode came out exactly the desired filesize (hit the target accurately), that is 634mb/649000kb in this case, 16mb of the precious total file size is spent on credits with such a distribution:

Quantizers Used For Credits :
--------------------------------
Quant 1 Used : 106 Times.
Quant 2 Used : 3803 Times.
Quant 3 Used : 24 Times.
Quant 4 Used : 24 Times.
Quant 5 Used : 231 Times.

Well, the quality is still OK, maybe better than the previous routine encode, but 16mb is too much a waste for credits, most of which can be spent on an even better encode.

the exact debugview analyzer results:

DebugView analyzer for XviD codec v0.11 by MoonWalker
e-mail : s_ilias@gmx.net

Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 1927 Times, Percentage Used : 0.98%
Quant 3 Used : 94801 Times, Percentage Used : 48.36%
Quant 4 Used : 93031 Times, Percentage Used : 47.46%
Quant 5 Used : 6015 Times, Percentage Used : 3.07%
Quant 6 Used : 241 Times, Percentage Used : 0.12%
Quant 7 Used : 1 Times, Percentage Used : 0.00%

Average Quantizer Used for Movie : 3.530

Quantizers Used For Credits :
--------------------------------
Quant 1 Used : 106 Times.
Quant 2 Used : 3803 Times.
Quant 3 Used : 24 Times.
Quant 4 Used : 24 Times.
Quant 5 Used : 231 Times.

MPEG Quantization Type Used 200204 timed, Percentage Used : 100.00%

Quantizers prevented from rising too steeply 14 times


Intra-Frame (Key-Frame) Quantizers
------------------------------------

Movie
-------
Quant 4 Used : 2171 Times, Percentage Used : 91.10%
Quant 5 Used : 33 Times, Percentage Used : 1.38%
Quant 6 Used : 179 Times, Percentage Used : 7.51%

Credits
---------

Number Of Consecutive I-Frames : 151

Inter-Frame (P-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 1927 Times, Percentage Used : 1.00%
Quant 3 Used : 94801 Times, Percentage Used : 48.96%
Quant 4 Used : 90860 Times, Percentage Used : 46.92%
Quant 5 Used : 5982 Times, Percentage Used : 3.09%
Quant 6 Used : 62 Times, Percentage Used : 0.03%
Quant 7 Used : 1 Times, Percentage Used : 0.00%

Credits
---------

Frame Analisis
----------------

Number Of Intra-Frames (Key-Frames) : 2383
Number Of Inter-Frames (P-Frames) : 193633
Total Number Of Frames : 196015

1.22% of the Movie is Intra-Frames (Key-Frames)
98.78% of the Movie is Inter-Frames (P-Frames)

Size Analysis
----------------

1-Pass Size : 1525443749 Bytes or 1489691 KBytes
Scaled Size : 665047442 Bytes or 649460 KBytes
Actual Size : 659450253 Bytes or 643994 KBytes

Usefull Statistics
------------------
Compressibility : 43.23%
Relative Quality of XviD avi : 56.66%
Absolute Quality of XviD avi : 95.41%


Any comments?

best regards,
iago

EDIT: BTW, 1stPass/2ndPass ratio given by statsreader was 2.338:1, and that's my avisynth script, in which I apply TemporalSmoother(10) to the credits as well:

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec.dll")
mpeg2source("C:\FIGHT\FIGHT.d2v")
crop(4,60,716,354)
BicubicResize(640,272,0,0.5)
Trim(0,196015)+Trim(196016,0).TemporalSmoother(10)
Koepi
28th August 2002, 12:15
@rui:

surprising results. But hu, then the stats reader is useful for something, isn't it? ;)

@iago:

Dunno what to do about the credist, in ~2hours I'm in the credits range here, too (first pass: fixed quant 20), dunno if or how it will work.

Hopefully the visual quality is ok?

Thanks for your testing efforts, both of you! :)

Best regards,
Koepi
iago
28th August 2002, 12:26
@Koepi,

I forgot to say that I'd used fixed quantizer 31 for credits in both passes and as I said before I entered the credits range in the statsreader too before saving the external stats file.

Yes, the quality is absolutely OK, and (hopefully :)) will be even better imho when the credits issue is solved and the precious mb's spent for credits will be used in the movie itself ;).

Thanks are for "you" man, for this great tool and for all your valuable efforts :).

best wishes,
iago
rui
28th August 2002, 13:44
Originally posted by iago
[BThanks are for "you" man, for this great tool and for all your valuable efforts :).
[/B]

I can only subscrive iago's words.
Koepi
28th August 2002, 13:59
Ok, my new matrix encoding is through.

The credits don't get scaled by the codec, it stuck with fixed quant 5 instead of 20, thus oversizing the output file when using special credits treatment by a huge amount(26MB in my case).

I'm taking a small timeout for getting some sleep, after that will hack a small fix into xvid and release a new binary with "improved" 2nd pass external support for fixed quantizers (it'll be a hack like this: if credits are checked [only end-credits though], it'll take the quantizer which is given by the credit-frames).

Sorry for this. Still dunno if the scaling is correct after that as I didn't see the filesize when credits started, but the formulars in statsreader should be correct.

Ok, CU soon, and thanks for your support!

Best regards,
Koepi
Koepi
28th August 2002, 15:57
I've a "bonbon" for you:

XviD-28082002-1:
- Activated EPZS(^2) motion estimation algorithm.
- Enabled fixed quantizer mode for external compression/credits
- included statsreader in the package, for easier access.

It's experimental, I don't know yet if it works, it#s just a one-line-hack which tries to force a quantizer there. Please check on that (rui? your trailer should be fast to test ;) ).

Thanks & best regards,
Koepi
rui
28th August 2002, 17:08
Well, i made the test, but the credits still are encoded at various quantizers :( despite the fact that i configured them to be encoded at 20 quantizer.
But i used the 1st pass made with the prior build, and used the stats file created by the statsreader that you launched today. The one that comes with this build isn't diferent, is it? Could this afected the test in some way?
I will make a test with 1st and 2nd pass, but it will take some time. (my p2-350 is sweating all over :D)

DebugView analyzer for XviD codec v0.11 by MoonWalker
e-mail : s_ilias@gmx.net


Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 3 Used : 3 Times, Percentage Used : 0.09%
Quant 4 Used : 177 Times, Percentage Used : 5.12%
Quant 5 Used : 706 Times, Percentage Used : 20.43%
Quant 6 Used : 1095 Times, Percentage Used : 31.69%
Quant 7 Used : 820 Times, Percentage Used : 23.73%
Quant 8 Used : 490 Times, Percentage Used : 14.18%
Quant 9 Used : 140 Times, Percentage Used : 4.05%
Quant 10 Used : 22 Times, Percentage Used : 0.64%
Quant 11 Used : 2 Times, Percentage Used : 0.06%

Average Quantizer Used for Movie : 6.362

Quantizers Used For Credits :
--------------------------------
Quant 2 Used : 1 Times.
Quant 4 Used : 4 Times.
Quant 5 Used : 109 Times.
Quant 20 Used : 1 Times.
Koepi
28th August 2002, 17:15
Nope, it's the same build.
Damn, i just tested it too.

It's seriously messed up, I've to dig deeper into it :-(

Thanks for the test!

Expect a new version soon. (version number won't change, but I'll announce it here).

Regards,
Koepi
rui
28th August 2002, 17:22
Koepi, i don't think that it's so bad. With the exact same settings, this new build got the average quantizer down from 6,648 to 6,362, and it didn't used higher quantizers (the max was quant11, the same as the older build).

So to me this is an improvement. :)


EDIT: for what is worth, i found out that, when using the external stats file, using default cc with high%/low% at 0 is the exact same that using alt cc with the mrq at 100% level.
Koepi
28th August 2002, 18:46
New binary is up.

Didn't check whether it breaks anything else, but i coded "safely" so it shouldn't harm anything else. It finally hits the target size for me (requested to downscale to 8MB, it reported downscaled to 8.29MB - which it hit on the spot :) ).

Remember, it only works for constant quantizer credits.

Best regards,
Koepi

...who needs a pause now.
Koepi
28th August 2002, 18:48
Ah, nice rui - that's the way it's supposed to be! Nice you mention that.

Still, I don't understand why internal compression gives different result s as external...

Best regards,
Koepi
iago
28th August 2002, 18:51
@Koepi

Great ;). Starting the new test immediately :).

ciao,
iago
iago
28th August 2002, 23:27
Hello everybody,

Well, the 1stPass of my test encode with the new 28082002-1 (updated) binary (6.UltraHigh, MPEG, lumi masking, quant31 credits) is about to finish.

I'll do the second pass with external curve compression, 6.UltraHigh, MPEG, lumi masking, mrq 100 with default altCC parameters (medium/500/90), payback proportionally, capping quantizers 2-4/2-8 (don't tell Koepi ;)) or 2-6/2-16, and quant31 credits.

Hopefully, if the credits issue is solved and 2ndPass also comes out with quant31 credits :), I'm expecting a real quality boost in the final encode.

I think it'll take me some time to post the results.

now, just waiting :)
iago
unplugged
28th August 2002, 23:56
:confused: there is something that maybe not clear in XviD CC or alt-CC, to me and maybe others.
Does internal CC algo estimates all frames (size) just prior starting 2nd pass encode or calculates them one by one during the encoding process?
Excuse me, perhaps too stupid the question, but it's to know a bit more...
Of course, the first method that I have indicated is much less "redundant" whereas the second suffers the distorsion or difference between the linear (original) curve and the compressed curve.

With XviD when you tweak CC or alt-CC the frames sizes are scaled having the average frame size as "center of gravity" (everything close or equal to average size in 1st pass remains close to average also in 2nd pass, because of the center of gravity; much like ABR).

Koepi, do you call your last scaling method linear because it mainly divides everything with 1st-pass/2nd-pass ratio? (center of gravity for scaling is 0; much like VBR)
Maybe this was another stupid question, but again it's to know what's intended as old and new method, what's intended to be better, what's intended to be linear...

You indicate the linear method more precise with frame size distribution, If I have well understood it's because just before starting the external 2nd-pass process the *sum* of target frames' size is perfectly aligned with final requested size.

When we use a non-linear 2nd pass scaling method (like alt-CC) one strong point that we should care in the pre-encoding phase is to implement a numerical formula to draw up the *entire* frame size list to plan and start to manage the total distribution just before the encoding process:
- of course having the sum of pre-calculated frames sizes as target equivalent to requested file size
- having the underuse by excessive high curve compression (typical underuse of bits due to user param, too conservative) equally compensated in *all frames* (a sort of ABR... over impossible VBR)

I don't know at all how and when XviD decide 2nd-pass frames sizes (if on-fly frame by frame or precalculating), does alt-CC suffers by continuous up and down overflow due to lack of a total and prior precalculating phase?

I'm not a good programmer, I'm not trying to criticize or say anything. It's only to know how raw or perfect it is. ;) This is very interesting, infact I think noboby (except devs) really knows how XviD draws up CC. :)

Sorry, my few thoughts suffer very much by english exposure and my expressions often results too cumulative... :(
Koepi
29th August 2002, 01:36
hehe iago :)

actually, i just hacked up second pass treatment, there was no need to redo the first pass, sorry for not being clear about that :(

I'm still a little unsatisfied as i more or less disabled curve treatment for second pass while using 2nd pass external compression together with activated credits (thus resulting in only fixed quant credits work), and so any stacked-up overflow until the credits start get's carried over to the final filesize. my matrix rip is ~2mb oversized...

at least it's better than before ;)

Regards,
Koepi
iago
29th August 2002, 07:57
Originally posted by Koepi
hehe iago :) actually, i just hacked up second pass treatment, there was no need to redo the first pass, sorry for not being clear about that :( Yes I knew/guessed that but my previous first pass stats file was "without" lumi, so this time I did a first pass "with" lumi ;).
Originally posted by Koepi
I'm still a little unsatisfied as i more or less disabled curve treatment for second pass while using 2nd pass external compression together with activated credits (thus resulting in only fixed quant credits work), and so any stacked-up overflow until the credits start get's carried over to the final filesize. my matrix rip is ~2mb oversized... at least it's better than before ;)Same here. My Fight Club rip is ~5mb oversized but absolutely better than before, as I expected. (Now I'm producing another Ogg audio file with a lower filesize, as the only solution ;)).

best regards,
iago
Didée
29th August 2002, 08:48
Perhaps it´s too early in the morning for me, and my brain is still sleeping ...

I don´t see the advantage of the linear downscaling done by koepi´s statsreader:

CC/AltCC concept:
- make big frames smaller, to the benefit of small frames

Linear downscaling concept:
- make *everything* so small that it fits

I have to admit: So far, I´ve done no test with this new method (having to free up ~200 GB HD-space completely for urgent troubleshooting - no time for experiments).

But, to speak with unplugged: -> First of all the concept <-

From theory, I see that small frames are put at a disadvantage with linear downscaling - especially in addition with the encouraged "disabling" of any CC along with that.

Could someone give a short explanation *why* linear downscaling should give better results than AltCC ??
I would like to (really) understand the concept.
unplugged
29th August 2002, 09:15
Originally posted by Didée
CC/AltCC concept:
- make big frames smaller, to the benefit of small frames

Linear downscaling concept:
- make *everything* so small that it fits
Yes
Originally posted by Didée
But, to speak with unplugged: -> First of all the concept <-

From theory, I see that small frames are put at a disadvantage with linear downscaling - especially in addition with the encouraged "disabling" of any CC along with that.

Could someone give a short explanation *why* linear downscaling should give better results than AltCC ??
I would like to (really) understand the concept.
First, nobody has said that linear is better, mainly it has a positive point regarding the distribution, that is (more) homogenous.
Despite small frame penalization I think linear method may make better the final job only because the XviD CC is difficult to configure and may not have a good frame-distribution code for entire video (and it's difficult to setup), but who knows how XviD CC draws up 2nd-pass frame size list...
unplugged
29th August 2002, 09:32
First, nobody has said that linear is better, mainly it has a positive point regarding the distribution, that is (more) homogenous.
What I'm trying to mean is that linear could be sometimes better not because the concept is better than alt-CC but because the "linear" algo is very easy to implement in a homogenous way, much precise to approach the end of file without too overflow adjustments (up 1235, down 4345, up 7686, up 2000... fluctuations due to non-linear CC not well predicted).

It's what I know at this moment, maybe wrong or maybe not exact, but until nobody responds... :-|
iago
29th August 2002, 10:08
And here are the exact results:
(Btw, quantizers were capped 2-6/2-16 in the test.)

1stPass/2ndPass: 1.963:1
(from statsreader, aiming for 634mb (~649000kb), which came out ~654000kb in the end)
---------------------------------------------------------------------
DebugView analyzer for XviD codec v0.11 by MoonWalker
e-mail : s_ilias@gmx.net

Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 7124 Times, Percentage Used : 3.63%
Quant 3 Used : 139758 Times, Percentage Used : 71.30%
Quant 4 Used : 48440 Times, Percentage Used : 24.71%
Quant 5 Used : 644 Times, Percentage Used : 0.33%
Quant 6 Used : 49 Times, Percentage Used : 0.02%
Quant 7 Used : 1 Times, Percentage Used : 0.00%

Average Quantizer Used for Movie : 3.218

Quantizers Used For Credits :
--------------------------------
Quant 31 Used : 4188 Times.

MPEG Quantization Type Used 200204 timed, Percentage Used : 100.00%

Quantizers prevented from rising too steeply 14 times


Intra-Frame (Key-Frame) Quantizers
------------------------------------

Movie
-------
Quant 3 Used : 2217 Times, Percentage Used : 90.27%
Quant 4 Used : 46 Times, Percentage Used : 1.87%
Quant 5 Used : 177 Times, Percentage Used : 7.21%

Credits
---------
Quant 31 Used : 16 Times, Percentage Used : 0.65%

Number Of Consecutive I-Frames : 149

Inter-Frame (P-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 7124 Times, Percentage Used : 3.60%
Quant 3 Used : 137541 Times, Percentage Used : 69.55%
Quant 4 Used : 48394 Times, Percentage Used : 24.47%
Quant 5 Used : 467 Times, Percentage Used : 0.24%
Quant 6 Used : 49 Times, Percentage Used : 0.02%
Quant 7 Used : 1 Times, Percentage Used : 0.00%

Credits
---------
Quant 31 Used : 4172 Times, Percentage Used : 2.11%


Frame Analisis
----------------

Number Of Intra-Frames (Key-Frames) : 2456
Number Of Inter-Frames (P-Frames) : 197748
Total Number Of Frames : 200203

1.23% of the Movie is Intra-Frames (Key-Frames)
98.77% of the Movie is Inter-Frames (P-Frames)


Size Analysis
----------------

1-Pass Size : 1305113986 Bytes or 1274525 KBytes
Scaled Size : 665049007 Bytes or 649461 KBytes
Actual Size : 665047594 Bytes or 649460 KBytes

Usefull Statistics
------------------
Compressibility : 50.96%
Relative Quality of XviD avi : 62.15%
Absolute Quality of XviD avi : 96.35%
---------------------------------------------------------------------

@unplugged
Sorry to interfere with your posts.

best wishes,
iago
rui
29th August 2002, 10:54
About the diference between using internal and external compression:

Internal
DebugView analyzer for XviD codec v0.11 by MoonWalker
e-mail : s_ilias@gmx.net


Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 4 Used : 126 Times, Percentage Used : 3.65%
Quant 5 Used : 568 Times, Percentage Used : 16.44%
Quant 6 Used : 1012 Times, Percentage Used : 29.29%
Quant 7 Used : 911 Times, Percentage Used : 26.37%
Quant 8 Used : 605 Times, Percentage Used : 17.51%
Quant 9 Used : 191 Times, Percentage Used : 5.53%
Quant 10 Used : 40 Times, Percentage Used : 1.16%
Quant 11 Used : 2 Times, Percentage Used : 0.06%

Average Quantizer Used for Movie : 6.592

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 115 Times.

MPEG Quantization Type Used 3570 timed, Percentage Used : 100.00%

Quantizers prevented from rising too steeply 26 times

External
DebugView analyzer for XviD codec v0.11 by MoonWalker
e-mail : s_ilias@gmx.net


Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 4 Used : 158 Times, Percentage Used : 4.57%
Quant 5 Used : 718 Times, Percentage Used : 20.78%
Quant 6 Used : 1071 Times, Percentage Used : 31.00%
Quant 7 Used : 863 Times, Percentage Used : 24.98%
Quant 8 Used : 505 Times, Percentage Used : 14.62%
Quant 9 Used : 126 Times, Percentage Used : 3.65%
Quant 10 Used : 12 Times, Percentage Used : 0.35%
Quant 11 Used : 2 Times, Percentage Used : 0.06%

Average Quantizer Used for Movie : 6.369

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 115 Times.

MPEG Quantization Type Used 3570 timed, Percentage Used : 100.00%

Quantizers prevented from rising too steeply 30 times

As you can see, there is a diference between them, being external a little better.
iago
29th August 2002, 11:08
@rui

Thanks for that really helpful test, which supports my (and I guess Koepi's) visual observations with some real technical data.

However, the "oversize" issue, which points to the predictability being broken, (as also explained by Koepi in his above post,) is still a problem to be solved imho. (Entering a lower filesize in the statsreader should not be the only way to go ;))

best wishes,
iago
MoonWalker
29th August 2002, 14:46
Hi,

I have done some test my self too and more or less I had the same results.I am doing some more now. StatsReader - CC 0/0 - AltCC Default. The only thing that I didn't like is the consecutive I-Frames.. :


Start Frame | End Frame | Number of Consecutive I-Frames
-------------+-------------+-------------------------------
195927 | 195935 | 8 Credits
196026 | 196034 | 8 Credits
196037 | 196155 | 118 Credits


I have posted only the ones in the credits portion.
My parameters were the defaults using AltCC and Koepi's latest build..I had a total of 608 consecutive I-frames.

MoonWalker
Koepi
29th August 2002, 14:49
If you're using altCC you have to set either strength to 0 or disable automatic min. relative quality and set it to 100. Else you don't get linear scale...

Thanks for testing!

Regards,
Koepi
Koepi
29th August 2002, 14:55
Hm, my hack doesn't work well:

My matrix-h263-quant-type encoding came out 8MB undersized.

Maybe we should leave credits alone...

But the quality is very nice.

Regards,
Koepi
iago
29th August 2002, 15:26
Originally posted by Koepi
My matrix-h263-quant-type encoding came out 8MB undersized. Maybe we should leave credits alone... So the problem is not only getting oversized files, which could simply be handled (though not being the proper way of course) by entering a bit lower filesize in statsreader. The case is total lack of predictability then.

Koepi, I have a couple of questions on mind now, the most important of which is:

When we leave the credits alone, will we still be able to get better quality with external compression and linear scaling (due to oversized credits this time), although the final file size will be accurate)?

best regards,
iago

EDIT: And one more question, which is still unclear for me:
Do you advice strength:0 (or mrq:100) only for external compression/linear scaling, or also for internal compression with default altCC parameters?

Sorry to bother you with these questions, but I'm ready to start a new test encode ;).
Koepi
29th August 2002, 15:33
Hi iago,

most certainly: no.

I think I have to code that properly into XviD ;)

I can live with this improper size since the quality is nice, and maybe later I'll try again to fix this stuff.

Best regards,
Koepi
Koepi
29th August 2002, 15:35
Hm, use those values if you want to test linear scaled curves, else the defaults should be fine. I still don't understand why the internal curve sclaing routines give another result than the external linear scaled curve though.

Regards,
Koepi
iago
29th August 2002, 15:52
Originally posted by Koepi
I can live with this improper size since the quality is nice, and maybe later I'll try again to fix this stuff. LOL! But "I" can't live with "this" improper size Koepi, (5mb oversized), 'cause I've reached my limits and can't create a lower size ogg audio (q.001), although the quality is very nice! :) :)

Well, OK, I'm joking of course ;).

So, as a result, we are waiting for the next harvest of your valuable efforts (which will solve this "damn" credits issue :)), when you have the time of course.

with my best regards
and many thanks for your prompt replies,
iago

PS: XCD, which could be a solution in this case, doesn't work for me due to my CD writer (Asus 32x12x40) I guess :(. Whatever program I use for burning, I always get an error message with my writer!)
Koepi
29th August 2002, 15:58
Since I don't know the name of your drive i can just try to help:

ftp://ftp.asuscom.de/pub/ASUSCOM/

maybe you find a newer firmware there, I don't know where the problem derives from.

Maybe you should use "force-aspi" drivers and fireburner for burning the created image :)

Best regards,
Koepi
iago
29th August 2002, 16:02
@Koepi,

Thanks a lot, I will try :). Also, sorry for switching off-topic in this thread.

best regards,
iago
rui
29th August 2002, 16:36
Well, continuing off topic :D, you can try in this site.

http://www.firmware.fr.st/

This is where i could find a firmware to my dvd reader to unlock it.
Koepi
29th August 2002, 16:48
Hehe, I didn't find any "asus" burner news there so i directed iago directly to the asus ftp ;)

Enough of off-topic, we ride on:

XviD-29082002-1:
- Activated EPZS(^2) motion estimation algorithm.
- Fixed quantizer mode for external compression/credits

I tried again to fix the external compression/fixed quantizer credits treatment. You have to use a fixed quant smaller than 31 to let it work, I try to vary the quantizer +/- 1 in credits to get rid of overflow or underflow. Hopefully this works, my test run will take 5 hours again...

Regards,
Koepi
iago
29th August 2002, 17:38
@Koepi

I'm so glad to hear this almost-immediate good news, and waiting for your test results impatiently. And one more question to bother you ;):

If this time the credits treatment works the way as you expect, will it be possible to use the first pass "quant31" stats file in the second pass? (I guess no, which means I'll do the first pass again... :))

Anyway, no matter! If this issue is solved, I'll be so glad that I can start a thousand fresh two passes happily ;).

thanks,
iago
Koepi
29th August 2002, 17:55
Unfortunately you can't reuse that stats file as the statsreader just leves those credits values untouched.

Some more tests should be made, with short clips and some artificial credits, maybe comparing "before"(old build) and "new"(...build).

I just did a test on the start of matrix, 1000 frames, credits from 500-1000... and ended up with a overflow which was ~2kb, instead of stacking up to some MB or just not compensating for ~1mb variation.

Regards,
Koepi
Koepi
29th August 2002, 19:40
Ok, now full stop again!

I'll put up a new binary soon, somehow a macroblock skip threshold was set too high, resulting in a less detailed picture.

...and again, we have to redo our first pass stats files :-(
Koepi
29th August 2002, 20:06
XviD-29082002-2:
- Activated EPZS(^2) motion estimation algorithm.
- Fixed macro block skip-threshold. Sorry I didn't notice before
that this was changed in CVS - my "private" builds had them
disabled so i blindly trusted they were fixed in CVS, too.
- As bonbon I added a "new modulated HQ (experimental)" mode for
modulated quantizers, Isibaar suggested that: h263 quant for <4,
mpeg quant for higher quantizers. Should eleminate some noise
from the "huge" frames, and mpeg quant seems to perform better
with high quantizers vs. blockyness than h263 quant.

Man, I'm sorry pals! (hehe, same post as on another thread ;) )

Ok, let's get to work again... first pass h263 quant started.

Regards,
Koepi
iago
29th August 2002, 22:10
Originally posted by Koepi

XviD-29082002-2:
- Activated EPZS(^2) motion estimation algorithm.
- Fixed macro block skip-threshold. Sorry I didn't notice before
that this was changed in CVS - my "private" builds had them
disabled so i blindly trusted they were fixed in CVS, too.
- As bonbon I added a "new modulated HQ (experimental)" mode for
modulated quantizers, Isibaar suggested that: h263 quant for <4,
mpeg quant for higher quantizers. Should eleminate some noise
from the "huge" frames, and mpeg quant seems to perform better
with high quantizers vs. blockyness than h263 quant.
So... it's time to send 18082002 and 28082002 builds to the trash together with the previous first pass stats files, install the new 29082002-2 build with the fixed macro block skip-threshold and the new bonbon, and then start new fresh two pass encodes :).

@Koepi,

Are we trying external compression stats again with fixed quantizers <31 this time? ;)

(Off-topic: Btw, my XCD image burning problem is solved now, and I can happily aim for higher filesizes from now on ;))

best wishes,
iago
Koepi
29th August 2002, 22:14
Cool iago,

what solved your XCD problem?

And yes, the fixed quant for ext. comp. still should be lower than 31 - I 'd suggest the default with 20 is fine... :)

Regards,
Koepi
iago
29th August 2002, 22:26
Off-Topic: sorry everyone...
Originally posted by Koepi
...what solved your XCD problem? Well, actually "fireburner" did ;). I realized that it had got nothing to do with firmware, and after many unsuccessful tries with nero, cdrwin and cdrdao, "fireburner" solved the problem :).

I'm a much happier man now :)
iago
Koepi
29th August 2002, 23:43
Ok, prime time again...

Stats reader gets even more accurate, i introduce version numbering now ;)

Version 1.1 uses KB instead of MB as downscale-size. In my tests the accuracy hits in the bounds of 100kbytes... :) (for the stats file only, the final movie has to be encoded then ;))

Thanks iago for telling me that it's the right attempt :)

Find it attached to this post.

Regards,
Koepi
zulu
30th August 2002, 04:08
Hi everybody!

Here is my test contribution for the 29082002 build / statsreader 1.1.
The movie i tested is "The Timemachine", runtime ~92 mins.


I made the following settings:

Neutral Bicubic Resizing (0.5) to 576x240
aimed file size: 636MB (~945 kbps)

external second pass (of course..)
min/max I quants: 2/6
min/max P quants: 2/16
"old" CC: low and high at 0, payback delay 250, payback proportionaly
alternative CC disabled
no hinted ME
credits encoded with fixed quant 20
all othere setting were set to default


Results:

The final avi was ~3MB oversized.
Can't say much in terms of visual quality - my gfx card is broken, i temporarily use an old and crappy virge card. Direct show doesn't seem to like it :(
I'll watch it on my brothers pc, tomorrow.

Here are the results of the debug analysis:


DebugView analyzer for XviD codec v0.11 by MoonWalker
e-mail : s_ilias@gmx.net


Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 71 Times, Percentage Used : 0.05%
Quant 3 Used : 56726 Times, Percentage Used : 43.53%
Quant 4 Used : 70863 Times, Percentage Used : 54.38%
Quant 5 Used : 2612 Times, Percentage Used : 2.00%
Quant 6 Used : 43 Times, Percentage Used : 0.03%

Average Quantizer Used for Movie : 3.584

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 7510 Times.
Quant 21 Used : 3 Times.

H.263 Quantization Type Used 137828 timed, Percentage Used : 100.00%

Quantizers prevented from rising too steeply 8 times


Intra-Frame (Key-Frame) Quantizers
------------------------------------

Movie
-------
Quant 3 Used : 1146 Times, Percentage Used : 79.47%
Quant 4 Used : 48 Times, Percentage Used : 3.33%
Quant 5 Used : 217 Times, Percentage Used : 15.05%

Credits
---------
Quant 20 Used : 31 Times, Percentage Used : 2.15%

Number Of Consecutive I-Frames : 173

Inter-Frame (P-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 71 Times, Percentage Used : 0.05%
Quant 3 Used : 55580 Times, Percentage Used : 40.75%
Quant 4 Used : 70815 Times, Percentage Used : 51.92%
Quant 5 Used : 2395 Times, Percentage Used : 1.76%
Quant 6 Used : 43 Times, Percentage Used : 0.03%

Credits
---------
Quant 20 Used : 7479 Times, Percentage Used : 5.48%
Quant 21 Used : 3 Times, Percentage Used : 0.00%



Frame Analisis
----------------

Number Of Intra-Frames (Key-Frames) : 1442
Number Of Inter-Frames (P-Frames) : 136386
Total Number Of Frames : 137827

1.05% of the Movie is Intra-Frames (Key-Frames)
98.95% of the Movie is Inter-Frames (P-Frames)



Size Analysis
----------------

1-Pass Size : 1306169592 Bytes or 1275556 KBytes
Scaled Size : 666732921 Bytes or 651106 KBytes
Actual Size : 666733963 Bytes or 651107 KBytes

Usefull Statistics
------------------
Compressibility : 51.04%
Relative Quality of XviD avi : 55.80%
Absolute Quality of XviD avi : 95.25%
Koepi
30th August 2002, 04:25
Thanks for testing, zulu!

Hm. 3MB oversized. Still not good. Damn. I think I've to code harder ;)

Thanks again,

best regards,
Koepi
zulu
30th August 2002, 04:35
Hm. 3MB oversized. Still not good. Damn. I think I've to code harder



Harder? It's 5.30 am in göttingen! go to bed and get some sleep, man! ;)
Koepi
31st August 2002, 10:33
Ok, my matrix h263/h263 rip has 3 MB oversize, too.

Now my mpeg/new modulated rip is finsihed, and it's size is 2kb different from the h263 rip.

So the file size deviation seems to be systematic, but i don't know where the error derives from as the overflow is absolutely in the range where it belongs in the codec.... Let's see if I can find out.

Best regards,
Koepi
iago
31st August 2002, 11:15
@Koepi,

Same here with Fight Club man ;):

h.263/h.263 (29082002-3 build): ~4.5 mb oversize
MPEG/MPEG (30082002-1 build): ~4.5 mb oversize

best regards and thanks again for your ongoing efforts on solving this problem with linear scaling,
iago
sherpya
31st August 2002, 18:43
I've made some test without credits, now I want to try with credits...
Just 2 questions:
1) Credits into xivd codec must be disabled?
2) Into statreader, I must check credits then two options are: first start frame of credits and second the fixed quantizer, right?

I want to use h263 quant type for 1st and Mod HQ for 2nd, it's completly wrong or I can try these settings... Koepi I hope you have made this quant type to use it :D
Koepi
31st August 2002, 21:55
I#ve added that quant type because Isibaar had a good idea with it - it has to proove if it works out as thought...

Well. for the credits stuff:

in xvid, do the first pass with a fixed quantizer (e.g. 20) credits. Remember those values of the start and endframe for the credits.
After that's done, go to statsreader, open the stats, and check "use credits", in the first field enter the start frame, in the second the end frame.
This will have following result: statsreader doesn't scale those frames within the credits range - it's assumed the are correctly scaled already (hence the fixed quantizer stuff).

Sadly, you'll get a slightly oversized file. maybe you should take precautions like subtracting around 4 MB from your desired file size.

I've to track down where this overflow comes from, it shouldn't be there. I'll look into it when I find the time.

Regards,
Koepi
sherpya
1st September 2002, 19:26
Here is the results:
I've used h263/modhq
video size 1194.61->1198.1
3.5mb oversized

Options used:
Motion search precision: 6
Quant Type: h263/modhq
Credits fixed to 20
2nd pass opts:
Min I-frame: 2
Max I-frame: 6
Min P-frame: 2
Max P-frame: 16
Two Pass:
I frame boost %: 0
High bitrate scene %: 0
Low bitrate scente %: 0
playback proportionally

Video lenght 1h:41.38

The encoding was fine, some artifact but not so visible
Very small amount of blocks during high motion scenes but not visibile with postprocessing

I will try to mux into an ogm with 2 audio tracks and 2 subtitles... humm still have some problem embedding srt into ogms...

Edit: no problem with subtitles ... Now I've an ogm with 2 lang, 2 subtitles and chapiters...,
DVD Decripters does automaticatilly chapter list creation but I need to change chapter names by hand, anyone knows a software to take chapter names directly from dvd?
Koepi
1st September 2002, 19:48
Ok, since this is a somewhat constant error I thought about it again:

my scaling formular before was:

(1st pass size - credits size) / (desired size - credits size)

This is changed to

(1st pass size) / (desired size - credits size).

The tests I did scaled the curve to ~3.5MB less (!) than the desired size - but that's the amount it got oversized before!

Find attached statsreader 1.2, and please test if it solves the issue!

Thanks and best regards,
Koepi
Koepi
1st September 2002, 19:51
Originally posted by sherpya
DVD Decripters does automaticatilly chapter list creation but I need to change chapter names by hand, anyone knows a software to take chapter names directly from dvd? [/B]

The names aren't stored anywhere on the CD. They are given by a DVD menu that you needed to rip somehow. Since this is different for every DVD i don't think you'll have much luck doing that.

Regards,
Koepi
Bluedan
1st September 2002, 22:00
Look @ blockbuster (http://www.blockbuster.com)
Usually they have a chapter list (in english) for mostly any DVD on the market. But be aware that information given there refers to Region 1 products in general. So sometimes chapter info (for exemple: total amount of chapters!) isn't appropiate. But have a try ...with copy'n paste. I'm mostly satisfied.
sherpya
1st September 2002, 22:19
I will test version 1.2 for oversize problem later, main problem was splitting ogm with 2 audio 2 sub and chapters... but this is not the right forum to discuss this... sadly cutting ogm files is not easy
I've made a tricky process mux(without subs)->cut->demux(graphedit)->mux 2 splits (with subs and chapter list rearranged)
Really a nitemare...:(
OggCut really works only if ffdshow is deactivated and without subs.
I've deactivated also cladfx.ax (a filter from powerdvd???).
Btw remaining in topic video encoded with h263/modhq has some problem when viewing with ffdshow (even with use xivd and idct xvid). It shows artifatcs
iago
1st September 2002, 23:09
Now going for Matrix 2ndPass using external curve compression with statsreader 1.2, hoping to see that the oversize problem is finally overcome! :).

regards,
iago
Koepi
1st September 2002, 23:20
Hehe, thanks for testing :)

I'm looking forward to see results (and I'll test it myself, too, of course!)

Best regards,
Koepi
sherpya
2nd September 2002, 02:27
with 1.2 600k oversize, I think it would be ok
Koepi
2nd September 2002, 02:32
Sherpya:

YAY! I did it ;)

It's not logical for me why this solved it as it is wrong mathematics, frankly speaking, but 600kb is well in range for my taste.

Thanks!!! :)

Regards,
Koepi

...who's a little happier now...
ookzDVD
2nd September 2002, 04:16
@Koepi,

I'm using the StatReader v1.1,
try to scaling the "We Were Soldier", from 2760Mb to 965Mb(988160Kb)
I got the negative result :(

I'll try to use your latest one v1.2,
I hope it will resolve the problem.
Koepi
2nd September 2002, 04:19
negative result? That's odd :-/ hopefully 1.2 works for you.

Btw., you should change something in your settings, 2.7GB to not even 1GB is going to be bad quality, can't work out nicely.

My latest encode came out 1.7MB oversized again.

But I fell over a strange bahaviour of the encoding - the frames from the first pass file seem to be different as from the second pass file. I'll switch to mpeg2dec.dll and do a proper 2pass again to see if that's true.

Regards,
koepi
Koepi
2nd September 2002, 04:35
Ok, new stats-reader (1.3) with even more precise scaling - and hopefully fixed "huge size" problems like OokzDVD ran into.

(using double instead of single precision now...)

Best regards,
Koepi
ookzDVD
2nd September 2002, 05:22
@Koepi,

Thank you for the _fast_ response ;)
I hope it will resolve the problem.
iago
2nd September 2002, 08:13
@Koepi

My Matrix rip with statsreader 1.2 came out only ~400kb oversized, which is quite acceptable imho ;). (mpeg2dec.dll)

regards,
iago

EDIT: And now, time to give the newborn 1.3 a try! ;)
Koepi
2nd September 2002, 10:12
Full stop, iago ;)

I found my error:

I have to subtract the AVI overhead from the second pass desired size to get proper results, so my attempt above is flawed!!!

Here we go, StatsReader 1.4.... sorry for the massive amounts of releases!

Best regards,
Koepi
iago
2nd September 2002, 10:46
hehe, OK Koepi ;)

Here we go with statsreader 1.4 then! (dead-born 1.3 went to trash).

This time: Matrix MPEG/MPEG with external curve compression ;).

best regards,
iago
Koepi
2nd September 2002, 20:26
Matrix, MPEG/MPEG, target size: 599000 - reached: 599012 !!!

YES, finally! :)

The quantizer distribution is a little disturbing though:

[888] Quantizer distribution for 2nd pass:
[888] Q:2:1695
[888] Q:3:91746
[888] Q:4:75400
[888] Q:5:15899
[888] Q:6:1259
[888] Q:7:84
[888] Q:8:19
[888] Q:9:16
[888] Q:10:18
[888] Q:11:13
[888] Q:12:34
[888] Q:13:5

But i have to check the visuals first before drawing conclusions. It's bad to have even few of such highly quantized frames, but maybe they are hardly noticable - I hope so :)

Best regards,
Koepi

EDIT: starting first pass for H.263/H.263 encode... :)
MoonWalker
2nd September 2002, 21:36
@Koepi

I'll do a h.263/h.263 with the matrix to compare our results.I am thinking to do it with cc 0/0(linear scaling) to see the diferences with you statsreader..what do you think?

BTW What reslution are you using? Simpleresize??

MoonWalker
iago
2nd September 2002, 23:27
@Koepi

Matrix, MPEG/MPEG, target size: 599000 - reached: 599012 !!! That's really great! Yes, finally! So, thanks again Koepi, for your non-stop effort to solve this inaccuracy problem with linear scaling, which is finally overcome ;)...

Matrix, MPEG/MPEG, target size: 757000 - waiting (that means it's gonna be reached then! :))

kindest regards,
iago
Koepi
3rd September 2002, 01:13
@moonwalker:

avs:
SetMemoryMax(40)
LoadPlugin("D:\Video_TS\sbc-ripping\avisynth\mpeg2dec.dll")
mpeg2source("D:\Video_TS\thematrix.d2v")
Crop(0,78,720,420)
BicubicResize(640,272,0,0.5)

Credits: 20/20 quant from 186188-196155

(R2 PAL DVD source)

Let's compare if we finally have the same results! Thanks for helping out here :)

@iago:

Please report if the results are as expected :))

I'm glad that the problem is solved for me at least, let's hope that this wasn't just a random effect!

Best regards,
Koepi
Bulletproof
3rd September 2002, 07:42
I just tried it and my file came out 14 megs oversized. I set the ALT CC off and put everything to 0 and put proportinal distribution. And I didn't use credits either.
colasonic
3rd September 2002, 08:40
the statsreader 1.4 with XviD-30082002-1 is great! they hit my target precisely!
I used The Royal Tenenbaums and set the desired file size to 644121KB. they finally gave me a file as 644122KB!
Thanks to Koepi, iago and all testers for your efforts!

H.263/H.263
CC Hi/Lo=0/0, pay back proportionally 250, lumi masking in both passes, AltCC disabled
Credits 20 Quantizer
iago
3rd September 2002, 09:50
@Koepi and everybody

Matrix, MPEG/MPEG, target size: 757000 - final size: 757012 !!! ;)

(external cc, lumi both passes, quantizers: 2-4/2-8, hi/lo: 0/0, altCC disabled, payback proportionally, credits 20quant)

Only one word: Fantastic! :)

thanks again for this (finally achieved :)) great work,
iago
MoonWalker
3rd September 2002, 11:46
Originally posted by iago

(external cc, lumi both passes, quantizers: 2-4/2-8, hi/lo: 0/0, altCC disabled, payback proportionally, credits 20quant)


iago, when you use external cc it doesn't matter witch cc you are using in XviD(this is all internal :) )..That's why it is called external..

MoonWalker
Koepi
3rd September 2002, 12:21
Moonwalke,

now you're getting very inprecise.

The overflow treatment is internal while the curve compression is external.
Your post is flawed, iago's is right.

It doesn't matter if you use altCC or regularCC, you just have to diable it by setting it to the values i mentioned about a hundred times now, so that only the overflow treatment is in effect.

I hope you didn't do this in other ways now, else your results won't help comparing internalCC (!) vs external CC.

Regards,
Koepi
iago
3rd September 2002, 12:25
Originally posted by MoonWalker
iago, when you use external cc it doesn't matter witch cc you are using in XviD(this is all internal :) )..That's why it is called external..@MoonWalker
Yes, that's already clear I guess; so I couldn't fully understand what you're getting at :)...

Also, I remember it was discussed that when using external cc, we'd use either hi-lo: 0-0 with "normal internal cc" disabling altCC, or mrq: 100 (or strength: 0) with the other parameters default in "alternative internal cc" to get a proper linear scaling with correct overflow treatment. I just provided info about what path I followed considering these previous discussions.

kindest regards,
iago

EDIT: Oh, simultaneous reply from Koepi :).
MoonWalker
3rd September 2002, 15:41
Originally posted by Koepi
Moonwalke,

now you're getting very inprecise.

The overflow treatment is internal while the curve compression is external.
Your post is flawed, iago's is right.

It doesn't matter if you use altCC or regularCC, you just have to diable it by setting it to the values i mentioned about a hundred times now, so that only the overflow treatment is in effect.

I hope you didn't do this in other ways now, else your results won't help comparing internalCC (!) vs external CC.

Regards,
Koepi

:rolleyes: :rolleyes: Hmm...Sorry..I thought it was like that.I mean I knew that the overflow was done internaly, but I supposed it didn't matter which values we use for the cc/altcc cause all the scaling was done externaly(ie the distance were not in use)...But I was wrong :rolleyes: ...Thanks Koepi for clearing this out :)

MoonWalker
primitive
3rd September 2002, 17:00
Is hardcoding the AVI overhead in the statsreader a good idea?

1) What if you are using an alternate container format for your final encode? If your final output is ogg, and you use your statsreader, your result is going to be undersized.

2) What if you use gknot and use gknot's internal calculation for overhead (multiple audio streams, files)? If overhead is higher, your file is going to be oversized.

Just a thought,

-p
Koepi
3rd September 2002, 17:15
@primitive:
OGG has some overhead, too, and usually you can say muxing an avi with an ogg vorbis sound file gives you nearly no size increase due to overhead, the formular can be simplified like avi_size+ogg_vorbis_size=OGM_size. That's the way I calculate for my encodings, and I think everyone does when using OGM.

So if we hit that avi_size we were calculating, there's nothing wrong with it.

@Moonwalker:
external compression is somewhat messed in xvid, therefore my credits-hack in my latest build :-/ I was a little inprecise too btw., if you set those values to 0/0 etc (as iago posted), you simply have no further influence of curve compression, if you set it to other values the curve will get strangely influenced by it, resulting in a non-linear scaled curve at least.
I hope this gives a better picture of what's happening.
To compare internal vs external linear scale I suggest you disable altCC and use just the regularCC with high&low set to 0. The aimed target size should be the same and the credits range&quantizer restrictions, too (as well as the "usual" quantizer restriction, greyscale modes,....)

@iago:
hehe, tends to happen somewhat often ;)

Regards,
Koepi
Bulletproof
3rd September 2002, 20:43
I tried again and this is what the statsreader said:

"down to 1171.79mb (1199908.51)"

I set the target size exactly to 1.2 gigs, the resulting file came out to be:

09/03/2002 07:25a 1,228,814,336 blah.avi

29 megs (rounded) oversized.

Anyone know what I might be doing wrong?
Koepi
3rd September 2002, 21:05
bullet,

I tested with different things end everything's ok.
You'd get the same result with internal compression, if you have a size problem with statsreader (when using credits, you still rely on my latest build, but you sure know that as you read this thread).

btw.,
1199908.51 is very close to
1200000

So, it's definatly your settings which are mistaken(which you didn't provide - and they definatly don't belong here). See, noone else has a problem like you have.
If you restrict the quantizers too much,.... and so on, and so on...
you know, the usual stuff to check when not getting the right target file size.

Thanks,
regards,
Koepi
Bulletproof
3rd September 2002, 21:53
Could it possibly be the keyframes? It is quite a long movie, about 2 and a half hours long.
Koepi
3rd September 2002, 21:55
Did you use iframe boost? it was flawed in external compression mode.
Should be fixed with

XviD-03092002-1:
- Updated to Statsreader 1.4.
- Foxers 2nd pass external/credits fix

Still, this shouldn't result in oversized movies, it must be your quantizer restriction then i think.

Regards,
Koepi
rui
3rd September 2002, 22:05
Well, this is a little bit off-topic :o, but i was concerned about the quantizer to use in 1st pass when using the new ModHQ mode.
In the old modulated mode one should use h.263 quantizer in 1st pass.
But if i remember correctly, someone once said that if one knew that it would get in the 2nd pass most percentage of mpeg quantizers, then he/she should use in the 1st pass not h.263 but mpeg.

Now, in the new mode, since we have a majority, for 1 cd encodes, of mpeg quantizers, should we still use h.263 quantizer for 1st pass, or turn to mpeg?

In my short "The Replacements" trailer test, i got very similar results by using h.263 or mpeg in 1st pass, but when using h.263, in spite that in the 2nd pass the mpeg was used in much higher percentage, i got a little lower average quantizer and lower, by one point, absolute higher quantizer used.

What you guys think about this?
Koepi
3rd September 2002, 22:16
You're right rui, for the "new mod HQ" mode it would be wise to take mpeg quant for first pass when the second pass quant will be >3, h263 else.

Regards,
Koepi
rui
3rd September 2002, 22:23
So mpeg it is.
Thanks for the clearup, Koepi :)

P.S. Man, one of this days you will break the 2000 post barrier. :D
I believe that you will be the first person on this forum to achieve that:cool: (at least i didn't crossed with anyone else with that much posts in my lurkings).
iago
3rd September 2002, 22:39
Originally posted by Koepi
XviD-03092002-1:
- Updated to Statsreader 1.4.
- Foxers 2nd pass external/credits fixOh, I was just getting used to 30082002-1 with statsreader 1.4 ;).

Well, just kidding of course ;). That's another good news. As soon as my current long long encode is finished, the new build will be enthroned on my machine! ;)

kind regards,
iago
Bluedan
3rd September 2002, 22:41
My personal test encode resulted in 5.6MB oversized.
I left I-frame boost at default 20.
Aimed at 1249920 KB (that's what I entered in Statsreader1.4 last night), got 1255526 KB in the end as resulting file.
Not a desaster as the quality is pretty, and there is still room for cut points for 2CD splitting. ;)

settings
movie: together!
motion search prec 6
quant type h.263 both passes
min I-frame quant 2
max I-frame quant 6
min P-frame quant 2
max I-frame quant 14 or 16 (sorry, it was dark...)
lumi enabled
alt CC disabled
CC high/low set to 0 with prop.payback delay 250
left I-frame boost at 20
Hinted ME off of course
credits fixed quant 20

XVid build 30-8-2002 with SR_v1.4

Though the numbers don't match the expectations here (Koepi please tell me that the fault is on me, not the code) I'm happy with the quality I reached. Don't have time for further comparison testing, nor did I ever touched that debug analyser, shame on me maybe.
BTW, I had to run it in parallel to get the debug output, right?
Next time.
Keep up, I'm your fan !
Koepi
3rd September 2002, 22:49
bluedan,

I think your settings are all ok except for the iframe-boost...

so sorry that you have to redo the second pass, try iframeboost 0 :-/ Or use the new build, there it is fixed.

Thanks for testing,

Regards,
Koepi
iago
3rd September 2002, 22:50
Originally posted by Bluedan
I left I-frame boost at default 20.
XVid build 30-8-2002 with SR_v1.4 AFAIK, I-frame boost default is "0" not "20" in 30082002-1 build, and setting it to 20 may be the reason of your oversize.

best regards,
iago

EDIT: Koepi, come on man, I can't believe it! ;) Another simultaneous post !!! LOL! just strange... :)
Koepi
3rd September 2002, 22:55
@iago:

that's really impressive, right :) Maybe we're too often here at the same time ;)

Serefe,
Koepi
Bluedan
3rd September 2002, 23:03
Thanks for sonic speed posting...
I'm to lazy now, I think the overburn function of NERO can fix an eventual lack of size accuracy :p
Anyway, I'm not only a lazy but also anxious transcoder, so I always assume 695MB as 1CD size to be flexible with setting the cutting knife for 2CD split point (->chapter start).
Thanks both.
Decided to run dgbv_nt (new version 4.2 out today on http://www.sysinternals.com ) next time.
cult
3rd September 2002, 23:40
ok,i have a stupifd question.
Here it is:xvid has option for credits at start of the movie.So some of us may use it.Could it be it the cause for failure sometimes?I havent tested the new 03092002-1 so I dont know how it works,but when I tried the statsreader 1.4 I realized that I had 1st pass with some option checked for credits at the start of the movie,so I realized that was gonna be useless.Maybe adding this option too?
Koepi
3rd September 2002, 23:50
Hm, but this might take some time, right now I'm watching the season 9 opener of x-files to verify that it's as abd as it was talked about ;)

After that I'll happily add starting credits, too.

Regards,
Koepi
jpl
4th September 2002, 00:56
Is it possible that we could get a version that runs with older processors for those of us not lucky enough to have newer CPU's?

Please??

I'd like to test this out and see how well it works but it wont run on my K6-III. I'm asuming that it's a result of using the Intel Compilers in some kind of mode that breaks the compatibility.

Thanks
Koepi
4th September 2002, 01:33
Ok,

StatsReader 1.5 with starts- and endcredits separated, plausibility-warning (trarget size smaller than the credits... I think someone ran into this trouble already), compiled for old processor users convenience with MS compilers.

Regards,
Koepi
ookzDVD
4th September 2002, 05:49
@Koepi,

The v1.3 solves the "negative issue" ;) thank you.

_but_ sometimes I'm having error while closing the StatReader after scaling. The scaling is OK, the file is generated.
Koepi
4th September 2002, 05:55
Yeah, i get it too sometimes, I didn't get a plausable behaviour. Might be the memory allocated for the statsfile not freed properly, freed too often ;) or a file handle doesn't get destroyed.

But it's nothing vital :)

Thanks for the report!

Best regards,
Koepi
rui
4th September 2002, 08:55
The freeze on exit stuff happens to me also, but only in home (winXP).
Here at work (Win95B), i never got that, and tried all the builds. And they say that WinXp it the most evolved OS from Microsoft. Go figure:rolleyes:
rui
4th September 2002, 11:40
Koepi, i tried to use the modHQ mode with your latest build, but it is using quant mpeg all the time, when it should use quant h.263 for 2 and 3 quantizers, right?
Gannjunior
4th September 2002, 12:05
@Rui
Which is your film?
But the codec uses MPEG even for quantizer < 3 ?

ciao
cult
4th September 2002, 12:09
what can I say Koepi?thank u alot!U r the man!
rui
4th September 2002, 12:27
Gannjunior, the latest builds from Koepi have a feature called modulatedHQ (experimental), that uses, for quantizers 2 and 3 quantization H.263 (clears some of the noise in the edges), and mpeg for quantizers 4 and higher (retain more detail or more true collors according to some post i have seen).

But in this latest build, when using modhq, i saw that it's using mpeg for all quantizers.
jpl
4th September 2002, 12:44
Koepi,

Thanks recompiling for compatibility with those of us stuck with old CPU's. I can't wait to try it tonight when I get home.
Koepi
4th September 2002, 14:19
rui,

i was sure i implemented it correctly, strange... I'll take a look at it later tonight, when I'm bored again, now it's too much to catch up with first ;)

Regards,
Koepi
Gannjunior
4th September 2002, 14:43
@Rui
Yes,it's clear,I've followed the discussion.I'm not espressed well.What I want to say is: Which is the AVG quantizer showed in the log file? Maybe your film is done only by quantizers > 3,and in according to MOD. HQ,they're all mpeg.

ciao
Koepi
4th September 2002, 14:59
Nonono, rui is right.

I just directly took Foxers fixed 2pass.c and forgot to readd the quantizer switching for "new mod hq (exp)".

Hm, since i have a new statsreader anyways, I think i can build yet another binary... maybe even write a short note on statsreader use.

Thanks for the report,

best regards,
Koepi
rui
4th September 2002, 15:00
Sorry, i didn't got you first :)

No, i watched the encode in debugview, and in quantizers 2,3,4 and 5 it used always mpeg, when it should have used h.263 for 2 and 3.
Then i made another test with the old modulated, and this one used mpeg for 2 and 3 and h.263 for the rest, as it should be.

EDIT. Sorry Koepi, i was to slow in posting :)
Koepi
4th September 2002, 15:19
XviD-04092002-1:
- Updated to Statsreader 1.5.
- Fixed "new mod.HQ(ext.)" mode.

And yes, I know that the main GUI still shows 1.4 as version number ;)
Have fun with testing!

Regards,
Koepi
Gannjunior
4th September 2002, 15:30
@Rui
ah,ok,thanks...however Koepi is faster than a thunderbolt: He's already done a new upgrade ;)
Gannjunior
4th September 2002, 16:35
I'd like to do a question for my friend who is just registred,so he isn't still able to post:

"Hi Koepi,I'm trying to use stats reader 1.5, with a first pass stats file of 16Mb:

version 2.0 frames 243773
size (mb) 3375 max frame size 72747

if I set : target size (kb) 600000
I get : scaled down 1383.78 MB (1416986.33 kb)- compression ratio -1.241:1!

if I set : target size (kb) 700000
I get : scaled down 926.43 MB (948668.56 kb)- compression ratio -1.063:1!

if I set : target size (kb) 800000
I get : scaled down 469.09 MB (480350.87 kb)- compression ratio -0.929:1!

it seems to me there is some overflow problem with big stats files. insn't it ? (with smaller files it works)
any suggestion ?

thanks "
Koepi
4th September 2002, 16:49
Hm, well, first pass file sizes >2GB could've been an issue, tried to change to 64bit integers, dunno if this already helps - but it should in theory.

Find statsreader 1.6 attached and report if the problem is solved, please.

Best regards,
Koepi
iago
4th September 2002, 18:04
@Koepi

I had the same negative compression ratio problem as Gannjunior with statsreader 1.4 (with a >2gb first pass size), and statsreader 1.6 seems to solve that problem. It scaled the statsfile down correctly to 3.093:1, aiming for 730000kb 1 XCD ;).

best regards,
iago

(P.S.: What a compression ratio, isn't it?! ;) The movie is 3hr-8min "Spartacus")
Koepi
4th September 2002, 18:07
Iago,

thanks for reporting success :)

well, compression ratio of 3:1 is unlikely to give a "wow!" effect... but for just trying it, why not ;)

Regards,
Koepi
iago
4th September 2002, 18:14
O/T

@Koepi

Actually, I did two first passes with that movie, one with normal routine, second to test convolution3d(1,4,5,3,4,2.8,0) in terms of compressibility on a "normal" but rather long movie ;). And convolution3d(1,4,5,3,4,2.8,0) -before resizing- gave a compression ratio of 2.875:1 ;).

Anyway, back to topic, under the circumstances, it's much more reasonable to give it a go with the normal routine (without filtering)! :)

ciao,
iago
iago
4th September 2002, 18:44
@Koepi,

BTW, can we use a 30082002-1 build 1stPass stats file (scaled down with statsreader 1.6) for 2ndPass with externalCC using the new 04092002-1 build? (sorry if it sounds pointless... but I guess there "are" some changes in the builds later than 30082002-1, so I feel a bit unsure and have to ask that question ;))

thanks for your patience :)
iago
Koepi
4th September 2002, 19:11
Nope, no changes in the core or first pass stuff were made since then :) You can safely use the old stats files...

Best regards,
Koepi
iago
4th September 2002, 19:45
@Koepi

Thanks for the immediate reply. So I "safely" started my second pass with 04092002-1 ;).

sevgiler (kindly greetings) ;)
iago
Gannjunior
4th September 2002, 19:58
@Koepi
Thanks 4 your answer.Statsreader 1.6 has solved my firend's problem: it scaled the statsfile correctly.
P.S. If you want,you can download the stats file zipped(1.9Mb)from here:
video.stats (http://www.geocities.com/maurizioferreira/first.zip )

ciao :D
iago
5th September 2002, 00:50
Yep, just for trying and fun ;):
(mission impossible :D)

Movie: Spartacus (3hr-08min) / PAL R2
576*240 / SimpleResize
target video size: 730000kb (1XCD with ogg audio)
Koepi's 04092002-1 build - statsreader 1.6

First Pass:
6.UltraHigh / h.263 / lumi masking / credits 20 quant.

Second Pass:
6.UltraHigh / h.263 / lumi masking / quantizers capped:2-6/2-16 / external curve compression with statsreader 1.6 / hi-lo: 0-0 with altCC disabled / payback proportionally / payback delay: 300 / credits 20 quant.

I'll post the results when finished; I'm looking forward to see how it will look! :D

kindest regards,
iago
iago
5th September 2002, 16:00
Hello everybody,

Here are the results of the above mentioned encode (using 04092002-1 build and external cc with statsreader 1.6):

(Aimed video size: 730000kb Final size: 730006kb !!! ;) Great...)

1stPass:
file size: 2191mb compressibility ratio: 3.104:1

2ndPass:

XviD Analyzer v0.12 by MoonWalker & MarcFD
e-mails : s_ilias@gmx.net & marc.fd@libertysurf.fr

Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 3 Used : 7086 Times, Percentage Used : 2.54%
Quant 4 Used : 114744 Times, Percentage Used : 41.19%
Quant 5 Used : 111110 Times, Percentage Used : 39.89%
Quant 6 Used : 29966 Times, Percentage Used : 10.76%
Quant 7 Used : 3428 Times, Percentage Used : 1.23%
Quant 8 Used : 924 Times, Percentage Used : 0.33%
Quant 9 Used : 450 Times, Percentage Used : 0.16%
Quant 10 Used : 288 Times, Percentage Used : 0.10%
Quant 11 Used : 194 Times, Percentage Used : 0.07%
Quant 12 Used : 315 Times, Percentage Used : 0.11%
Quant 13 Used : 131 Times, Percentage Used : 0.05%
Quant 14 Used : 148 Times, Percentage Used : 0.05%
Quant 15 Used : 4100 Times, Percentage Used : 1.47%
Quant 16 Used : 5659 Times, Percentage Used : 2.03%

Average Quantizer Used for Movie : 5.082

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 4313 Times.

H.263 Quantization Type Used 282856 timed, Percentage Used : 100.00%

Quantizers prevented from rising too steeply 199 times


Intra-Frame (Key-Frame) Quantizers
------------------------------------

Movie
-------
Quant 6 Used : 1661 Times, Percentage Used : 98.05%

Credits
---------
Quant 20 Used : 33 Times, Percentage Used : 1.95%

Inter-Frame (P-Frame) Quantizers
------------------------------------

Movie
-------
Quant 3 Used : 7086 Times, Percentage Used : 2.52%
Quant 4 Used : 114744 Times, Percentage Used : 40.81%
Quant 5 Used : 111110 Times, Percentage Used : 39.52%
Quant 6 Used : 28305 Times, Percentage Used : 10.07%
Quant 7 Used : 3428 Times, Percentage Used : 1.22%
Quant 8 Used : 924 Times, Percentage Used : 0.33%
Quant 9 Used : 450 Times, Percentage Used : 0.16%
Quant 10 Used : 288 Times, Percentage Used : 0.10%
Quant 11 Used : 194 Times, Percentage Used : 0.07%
Quant 12 Used : 315 Times, Percentage Used : 0.11%
Quant 13 Used : 131 Times, Percentage Used : 0.05%
Quant 14 Used : 148 Times, Percentage Used : 0.05%
Quant 15 Used : 4100 Times, Percentage Used : 1.46%
Quant 16 Used : 5659 Times, Percentage Used : 2.01%

Credits
---------
Quant 20 Used : 4280 Times, Percentage Used : 1.52%



Frame Analisis
----------------

Number Of Intra-Frames (Key-Frames) : 1694
Number Of Inter-Frames (P-Frames) : 281162
Total Number Of Frames : 282855

0.60% of the Movie is Intra-Frames (Key-Frames)
99.40% of the Movie is Inter-Frames (P-Frames)

Size Analysis
----------------

1-Pass Size : 2297538270 Bytes or 2243689 KBytes or 2191 MBytes
Scaled Size : 740575358 Bytes or 723218 KBytes or 706 MBytes
Actual Size : 740574642 Bytes or 723217 KBytes or 706 MBytes

Usefull Statistics
------------------
Compressibility : -37.08%
Relative Quality of XviD avi : 39.35%
Absolute Quality of XviD avi : 90.75%


And, interestingly, the movie is "not" unwatchable; on the contrary it "is" a decent encode, perfectly watchable, in spite of the average quantizer ~5, with a pretty and smooth playback using Nic's decoder filter at strength 4 (full deblock) :). (15 and 16 quants in the report are only used at the beginning (first 4 or 5 minutes) of the film where long durations of only black screen is present with some still scenes occuring from time to time.

EDIT: Well, of course it is far from being "perfect and flawless" ;), but it is still "perfectly watchable" without disturbing you much; and even more "watchable" if you add some noise using ffdshow ;). In short, it is absolutely "not" a "messed up and unwatchable" encode, which really surprised me.

best regards from iago ;)
(who is much happier now with linear scaling)
Koepi
5th September 2002, 16:04
Hehe, interesting that this doesn't mess up the movie :)

Btw., internal scaling with regular CC at 0/0 or altCC.... you know the settings... - should result in the _same_ result. Now it's interesting if there still is a difference.

I need another movie to encode, can't see the matrix for some hours ;)

Regards,
koepi
iago
5th September 2002, 16:38
Originally posted by Koepi
Btw., internal scaling with regular CC at 0/0 or altCC.... you know the settings... - should result in the _same_ result. Now it's interesting if there still is a difference.@Koepi,

I encoded that movie only once, with regular cc at 0/0 with the above settings. I didn't use "altCC with strength 0 or mrq 100 and the other parameters default", which as you pointed out should give the same result as regular cc at 0/0 ;). So, there is nothing confusing or uncertain I guess? ;)

(And the encode mentioned above was a fresh two pass with 04092002-1, and statsreader 1.6 for scaling down the curve.)

kindest regards,
iago
Koepi
5th September 2002, 16:48
oh, the difference i meant was:

there should be no difference if you use internal or external scaling.

I'll encode something in some time to compare this.

Best regards,
Koepi
iago
5th September 2002, 17:08
Originally posted by Koepi
oh, the difference i meant was:

there should be no difference if you use internal or external scaling.

I'll encode something in some time to compare this.@Koepi

Oh, I see now ;). Sorry for the misunderstanding ;).

ciao,
iago
Emp3r0r
5th September 2002, 21:45
Originally posted by iago
quantizers capped:2-6/2-16I have yet to understand the purpose behind capping quants. Why not let xvid do it's magic?
Koepi
5th September 2002, 22:21
Originally posted by Emp3r0r
I have yet to understand the purpose behind capping quants. Why not let xvid do it's magic?

Well, sometimes magic fails and therefore some safety-belts ;)

Regards,
Koepi
iago
5th September 2002, 22:24
@Emp3r0r

Maybe a sort of caution or just a habit, I don't know ;). In fact, I usually prefer 2-4/2-8 for my rips, but this one was a real hard case (3hr-8min). Also, as I remember, somewhere Koepi had suggested 2-6/2-16 for 1CD rips and 2-5/2-16 for 2CD, if I'm not mistaken.

regards,
iago

EDIT: Koepi, I cannot believe it anymore !!! ;). Is this really a coincidence?! These simultaneous posts! ;)
MaTTeR
5th September 2002, 22:47
Originally posted by Koepi


Well, sometimes magic fails and therefore some safety-belts ;)

Regards,
Koepi
Hehehe...I totally agree. I remember Koepi stating that sometimes he could see the higher quant frames, well so can I and I suppose were not the only one's. I always cap the quants just as Iago mentioned(2-4/2-8). Admittingly the majority of my rips are targeted for either 970MB(XCD/99min) or 1410MB on 2CD's with full 5.1 AC3. For single CD rips I might fall back to something a little more conservative such as 2-6/2-12. Of course capping the quants also requires me to encode the movie several times for best results while I tweak but that's not really an issue with dual XP 1600's at less than 55mins per pass with TemporalSoften around 640 resolution. Perhaps I wouldn't be so eager to tweak the quants if I was still running dual PIII 800s:D
MoonWalker
5th September 2002, 23:00
@Matter

Out of topic a little. For 2-cd rips you use MPEG quant? Cause I use too but I get many MPEG arifacts.Can you post your avs ?? If you don't mind you can PM me, so we don't post on the topic..

Thanks,
MoonWalker
ookzDVD
6th September 2002, 08:56
@Koepi,

I've been trying v1.5 and the good news is the "Exit error" issue
is gone. Thank you.
Maurizio
6th September 2002, 09:05
hello everybody.
It's a lot of time I follow this forum and this is my first post.
I'm the friend of Gannjunior, who had the problem with negative results.
just a little question:

Originally posted by Koepi
oh, the difference i meant was:
there should be no difference if you use internal or external scaling.

Now I don't understand anything.
do you mean :
1) if I use external scaling, I get exactly the same results either I use
regular cc with 0/0 or alt cc whith known values
2) if I use regular cc with 0/0 or alt cc whith known values,
I get the same results either using internal or external scaling.
If this is the case, what's the point of using external scaling at all (since this requires one of the above cc settings) ?

Regards
Maurizio
Koepi
6th September 2002, 09:14
internal and external scaling should give the same results when using regCC with 0/0.

I started to add the scaling ability to the StatsReader as it was fun to code, and as a test.

Now we have the doubt that it _is_ the same as it should be. I'm conducting a test today (if the quantizer distribution differs between internal and external scaling, there's something wrong with the code :( )

You're right, if there's no difference, no need to use the statsreader for scaling. But if there is....

Well, let's ope everything is fine. My first small test showed a difference, but the statsfile might have been broken as it was from a crashed first pass session.

Regards,
Koepi
Maurizio
6th September 2002, 11:45
So if I'm not wrong, you are just experimenting with linear scaling
before passing to the more difficult task of unlinear one ?
btw, broadly speaking , should unlinear produce better results (whatever this might means) than not linear ?
regards
Maurizio
TripleA
6th September 2002, 12:57
I did some encoding tests of "Soul Survivors". Using Koepi's Sept. 4th build. All with a target size of 728984KB. All reasonably close (under 1MB off). Here are the analyses (XDA v0.11. v0.12 doesn't work for me...):

First pass h.263, 2nd pass modulated HQ, internal CC:

Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 63169 Times, Percentage Used : 54.85%
Quant 3 Used : 50667 Times, Percentage Used : 44.00%
Quant 4 Used : 1298 Times, Percentage Used : 1.13%
Quant 5 Used : 30 Times, Percentage Used : 0.03%

Average Quantizer Used for Movie : 2.463

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 7978 Times.

MPEG Quantization Type Used 9306 timed, Percentage Used : 7.56%

H.263 Quantization Type Used 113836 timed, Percentage Used : 92.44%

Quantizers prevented from rising too steeply 100 times


Intra-Frame (Key-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 1447 Times, Percentage Used : 94.51%
Quant 3 Used : 24 Times, Percentage Used : 1.57%

Credits
---------
Quant 20 Used : 60 Times, Percentage Used : 3.92%

Number Of Consecutive I-Frames : 0

Inter-Frame (P-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 61722 Times, Percentage Used : 50.75%
Quant 3 Used : 50643 Times, Percentage Used : 41.64%
Quant 4 Used : 1298 Times, Percentage Used : 1.07%
Quant 5 Used : 30 Times, Percentage Used : 0.02%

Credits
---------
Quant 20 Used : 7918 Times, Percentage Used : 6.51%



Frame Analisis
----------------

Number Of Intra-Frames (Key-Frames) : 1531
Number Of Inter-Frames (P-Frames) : 121611
Total Number Of Frames : 123141

1.24% of the Movie is Intra-Frames (Key-Frames)
98.76% of the Movie is Inter-Frames (P-Frames)



Size Analysis
----------------

1-Pass Size : 932481456 Bytes or 910626 KBytes
Scaled Size : 743497266 Bytes or 726071 KBytes
Actual Size : 743741918 Bytes or 726310 KBytes

Usefull Statistics
------------------
Compressibility : 79.76%
Relative Quality of XviD avi : 81.19%
Absolute Quality of XviD avi : 98.61%

First pass h.263, 2nd pass modulated HQ, external CC (StatsReader v1.5):

Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 46383 Times, Percentage Used : 40.28%
Quant 3 Used : 68661 Times, Percentage Used : 59.62%
Quant 4 Used : 111 Times, Percentage Used : 0.10%
Quant 5 Used : 9 Times, Percentage Used : 0.01%

Average Quantizer Used for Movie : 2.598

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 7978 Times.

MPEG Quantization Type Used 8098 timed, Percentage Used : 6.58%

H.263 Quantization Type Used 115044 timed, Percentage Used : 93.42%

Quantizers prevented from rising too steeply 0 times


Intra-Frame (Key-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 1453 Times, Percentage Used : 94.53%
Quant 3 Used : 24 Times, Percentage Used : 1.56%

Credits
---------
Quant 20 Used : 60 Times, Percentage Used : 3.90%

Number Of Consecutive I-Frames : 0

Inter-Frame (P-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 44930 Times, Percentage Used : 36.95%
Quant 3 Used : 68637 Times, Percentage Used : 56.44%
Quant 4 Used : 111 Times, Percentage Used : 0.09%
Quant 5 Used : 9 Times, Percentage Used : 0.01%

Credits
---------
Quant 20 Used : 7918 Times, Percentage Used : 6.51%



Frame Analisis
----------------

Number Of Intra-Frames (Key-Frames) : 1537
Number Of Inter-Frames (P-Frames) : 121605
Total Number Of Frames : 123141

1.25% of the Movie is Intra-Frames (Key-Frames)
98.75% of the Movie is Inter-Frames (P-Frames)



Size Analysis
----------------

1-Pass Size : 932481456 Bytes or 910626 KBytes
Scaled Size : 743466503 Bytes or 726041 KBytes
Actual Size : 743715324 Bytes or 726284 KBytes

Usefull Statistics
------------------
Compressibility : 79.76%
Relative Quality of XviD avi : 76.97%
Absolute Quality of XviD avi : 98.20%

First pass MPEG, 2nd pass modulated HQ, external CC (StatsReader v1.5):

Quantizers Analisis
---------------------

Quantizers Used For Movie :
------------------------------
Quant 2 Used : 61594 Times, Percentage Used : 53.48%
Quant 3 Used : 52631 Times, Percentage Used : 45.70%
Quant 4 Used : 937 Times, Percentage Used : 0.81%
Quant 5 Used : 2 Times, Percentage Used : 0.00%

Average Quantizer Used for Movie : 2.473

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 7978 Times.

MPEG Quantization Type Used 8917 timed, Percentage Used : 7.24%

H.263 Quantization Type Used 114225 timed, Percentage Used : 92.76%

Quantizers prevented from rising too steeply 11 times


Intra-Frame (Key-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 1397 Times, Percentage Used : 91.13%
Quant 3 Used : 56 Times, Percentage Used : 3.65%
Quant 4 Used : 20 Times, Percentage Used : 1.30%

Credits
---------
Quant 20 Used : 60 Times, Percentage Used : 3.91%

Number Of Consecutive I-Frames : 0

Inter-Frame (P-Frame) Quantizers
------------------------------------

Movie
-------
Quant 2 Used : 60197 Times, Percentage Used : 49.50%
Quant 3 Used : 52575 Times, Percentage Used : 43.23%
Quant 4 Used : 917 Times, Percentage Used : 0.75%
Quant 5 Used : 2 Times, Percentage Used : 0.00%

Credits
---------
Quant 20 Used : 7918 Times, Percentage Used : 6.51%



Frame Analisis
----------------

Number Of Intra-Frames (Key-Frames) : 1533
Number Of Inter-Frames (P-Frames) : 121609
Total Number Of Frames : 123141

1.24% of the Movie is Intra-Frames (Key-Frames)
98.76% of the Movie is Inter-Frames (P-Frames)



Size Analysis
----------------

1-Pass Size : 1063509869 Bytes or 1038583 KBytes
Scaled Size : 743466744 Bytes or 726041 KBytes
Actual Size : 743398910 Bytes or 725975 KBytes

Usefull Statistics
------------------
Compressibility : 69.90%
Relative Quality of XviD avi : 80.86%
Absolute Quality of XviD avi : 98.58%
iago
6th September 2002, 23:00
Originally posted by Koepi
internal and external scaling should give the same results when using regCC with 0/0.
Now we have the doubt that it _is_ the same as it should be. I'm conducting a test today (if the quantizer distribution differs between internal and external scaling, there's something wrong with the code :( )@Koepi,

Any (good) news about your test ;)?

best regards,
iago
Koepi
6th September 2002, 23:46
Just finished the test:
Comparison of linear external vs internal curve compression:

Target size 599.000kb.

mpeg2dec.dll, mpeg2dec-PP6, mpeg2dec-PP6,
h263/h263 h263/h263 h263/h263
ext.CC ext.CC int.CC lin.

Q:2:1324 Q:2:1200 Q:2:1210
Q:3:95354 Q:3:103581 Q:3:103755
Q:4:84936 Q:4:78063 Q:4:77890
Q:5:4006 Q:5:2991 Q:5:2991
Q:6:415 Q:6:195 Q:6:188
Q:7:69 Q:7:70 Q:7:88
Q:8:25 Q:8:38 Q:8:35
Q:9:17 Q:9:36 Q:9:22
Q:10:34 Q:10:14 Q:10:9
Q:11:8

599.012kb 599.024kb 599.114kb


So it isn't behaving the same. There is a small difference which could be causing this: Foxer is redistributing the rounding error, which the statsreader doesn't (thus you never hit you target rate to the byte with statsreader curve scaling [I mean the report at the end of the scaling]). But in the end it seems to hit the desired rate more accurately this way, strange.
I'll ask foxer when he's online again if he has some other ideas. Good thing is that the results aren't dramatically different.

Now I'm going to the visual check.

Regards,
Koepi
Dali Lama
7th September 2002, 05:30
Hello!

I have tried to perform some of the tests suggested by you guys. I am having difficulty with the StatsReader 1.5. When I open my first pass stats and enter a reasonable file size I get a a msg saying scaled down to -919.06 MB?? Also, the compression ratio is negative... -2.673:1???

I have used an earlier beta before, but a negative result did not occur.

Thanks to all and good day,

Dali
Koepi
7th September 2002, 05:43
Dali,

fetch yourself statsreader v1.6, some posts before this one.

It has the issue with first pass size >2GB solved, which resulted in the behaviour you observed.

Thanks,

best regards,
Koepi

EDIT: the post is on page 8, the link to the attachment is http://forum.doom9.org/attachment.php?s=&postid=175804
Dali Lama
7th September 2002, 05:48
Thank You Koepi,

Issue resolved...testing resumed :)

Dali
Sharro
10th September 2002, 11:39
Cmon Koepi...my encode came out 6KB SHORT.....tsssccc tsccc....

Just kidding... and taking the opportunity to say Thx!! :-)))

Take Care,


Sharro
iago
10th September 2002, 14:25
@Sharro

LOL! Sense of humour is something I really appreciate ;). Yes, XviD is absolutely outstanding, indisputable winner, in predictability. (Imho, this point should also be included in all sorts of codec comparisons or discussions (just like Doom9 does), besides its rivaling or surpassing quality.

regards,
iago
Koepi
11th September 2002, 17:12
StatsReader 1.7 now tries to not scale "minimum sized" frames (e.g. 96bytes or 101 bytes frames, depending on the resolution).

For those frames a compensation is built-in now, this should avoid problems where longer low bitrate scenes could stack up overflow and thus result in some highly quantized frames which are simply spoken ugly.

I hope you like it, please test it! (I added info about those conditions to the output, too... I think i could make a "full report" version which does a quantizer distribution simulation, too ;) ).

Find the new version attached,

thanks,

and best regards,
Koepi
iago
11th September 2002, 18:01
Wowww, just great man! ;)
Koepi's never stopping, never giving up! ;)

thanks for the new toy,
ciao,
iago
Koepi
12th September 2002, 08:44
"What's the worst that could happen?", 640x352...

targeting: 631.432kb, reached: 631.456kb.
First pass size: 1.210mb

[1016] Quantizer distribution for 2nd pass:
[1016] Q:2:22
[1016] Q:3:46194
[1016] Q:4:87069
[1016] Q:5:1683
[1016] Q:6:4
[1016] Q:7:3
[1016] Q:8:3
[1016] Q:9:2
[1016] Q:10:2

I think it's amazing that "only 14 bad frames" are left .)

I didn't test that movie with other settings, but I might do on the "timemachine" which is the next movie I'm trying to encode ;)

Just wanted to share my result...

Best regards,
Koepi

P.S.: maybe i must finetune the formular to get the min. frame size in respect to macroblocks... it's somewhat weird, it's still no "linear" function if i subtract the avi frame overhead, and I don't see what I'm missing :-/
zulu
12th September 2002, 13:15
I didn't test that movie with other settings, but I might do on the "timemachine" which is the next movie I'm trying to encode


great! :)

i'm wondering how this movie will turn out for you.
i tested it both with internal and extnernal CC but can't get it to look good.

There were too many macroblocks for my taste.
Please have in particular a look at the scene at frame pos. ~ 114700.

Looking forward for your results :)
rui
12th September 2002, 14:07
One question:

When using stastreader, one shouldn't use the alt.cc or normal cc values. That's why it's adviced to input, for example in default cc, high/low to 0, so one gets a linear curve.

BUT, i did some tests days ago using the statsreader AND alt cc, and got diferent results than using only statsreader, or only alt cc.
In this cases the codec is using what type of curve to distribute the bits? Is the alt cc getting priority over statsreader stats file, or the other way around? Or a mix of both (since i am getting diferent results when using both, like i said)

P.S. I am having problems in my comp at work, so i can't test here anymore :(. I can't reproduce those tests i mentioned above.
Koepi
12th September 2002, 15:14
@zulu:

my first pass size is 1.365mb, see the scaling results attached. It may be that you ran into trouble with minsize frames before that scene...

@rui:
it takes the downscaled values from the stats2-file. It then applied curve correction with the overflow and the desired rate given by that statsfile and thus results into something different.

Regards,
Koepi
iago
12th September 2002, 16:21
Originally posted by Koepi
StatsReader 1.7 now tries to not scale "minimum sized" frames (e.g. 96bytes or 101 bytes frames, depending on the resolution).

For those frames a compensation is built-in now, this should avoid problems where longer low bitrate scenes could stack up overflow and thus result in some highly quantized frames which are simply spoken ugly.That's simply great. One step closer to the solution and to excellence ;).

Originally posted by Koepi
...I think i could make a "full report" version which does a quantizer distribution simulation, too ;)In my opinion, that would really be very useful, another great option for the outstanding statsreader ;).


Thanks again for your never-ending efforts and great works,
best regards,

iago
Koepi
13th September 2002, 07:44
I'm testing with "from hell" now, a very compressable movie. Good that I found the lanczos3.dll resize filter on the avisynth forum, this made it possible that the movie has to be compressed in the second pass at all (640x272 neutral bicubic came out at just 601000kb, now I've to downscale from 631000kb to 621000kb ;) ).

This movie has long "scenes" with min framesize, and the statsreader curve works like expected, no stacked up overflow, no degradation due to that!

Hm. On the further development, thinking of this movie the simulated second pass will be very inaccurate as it would show that all frames get quant. 2 which isn't true. But it gives an idea at last...

Ok, I'll think about writing a report-file, this should be very useful for further checking out what could be done.

Best regards,
Koepi
toliman
13th September 2002, 11:09
i also have 'from hell', at a slightly lower res, sitting on the HDD waiting to be encoded. I replaced the older codec i had with the 04/09 build, with the new mod.HQ experimental settings, which missed the indicated size by about 40mb :scared: . Looked crisp though, which was a nice surprise, statsreader told me i was getting very close to the first pass, which is another surprise. a really good compressible movie that fits on 1cd. using the statsreader didnt gain any more accuracy.

(i might need different first-pass stats for each kind of quantizer, which i didnt think was necessary, the thought just came to mind then. is this a correct assumption?)

i've tried to follow the (wandering) discussion in this post, but there is not much info on mod.HQ, which i find more intrguing than modulated/h.263/mpeg as something purely experimental or as a recommended setting (what info i did read on mod.HQ, got lost about 40 posts ago)... anyway, i'll see if i can do first and second pass in mod.HQ instead of first pass with h.263. (i had the impression mod.HQ was trying the 'nandub approach' to low-motion-hi-motion quantisers that made SBC awesome over regular low-motion encoding.) i'm willing to try anything from b-frames to psy/lumi enhancements to gain a small amount of visual quality, ive got the HDD space and the DVD's to play with to try it out.

The other problem with 'from hell' is the credits are blood-red, what does setting "encode credits in greyscale" do, modifying YUV intensity/values, i.e. what effect does non-white/black sequences make to greyscale credit encoding ?

also, while on the subject of encoding credits in first/second pass, is it better to use something a bit easier on the eyes than q.20, like q.12, and duplicate the same settings in 2nd pass? does this effect statsreader's accuracy or is this no problem.

(you can also just point me towards a FAQ on these questions, if there is one)
Koepi
13th September 2002, 13:16
Unfortunately all those infos you're searching for a spread ovr diferent threads so I like to urge you to use the search function, view the results in a new tab/window and learn about those things.

I don't understand why mod. HQ would destroy size predictability - it were other settings you were using. Use "load defaults" and you're mostly fine :)

if you use mod quant or mod. HQ, you will gain nothing from using it in the first pass. But read about it in the thread where we tested that. (Hint: first pass is done at constant quantizer 2....).

Uff. You didn't read the XOE or the help file. If you use fixed. quant credits, you have to use the same settings for both passes, else your final filesize won't hit the targeted size. (I assume that's the error you did above and got 40mb oversized...). So, read the XOE even if it's outdated a little.

Ok, I lost motivation in going on and explain things that are explained all around this place once more, condensed in one post, so I'll simply stop the reply now ;) I hope you can understand that it's very frustrating typing all the stuff again and again (and please read the forum rules - No. 1 is VERY important :) ).

Regards,
Koepi
toliman
13th September 2002, 14:40
Originally posted by Koepi
I don't understand why mod. HQ would destroy size predictability - it were other settings you were using. Use "load defaults" and you're mostly fine :)


im redoing a lot of encoded files recently, so i'm reworking old assumptions and asking questions, the current encodes dont seem to be working so well with the defaults i had. there is a 40gb backload of dvd's & divx re-encodes to go through, so im a little uhh ... curious on the whole process with mod.HQ and the new external CC.

e.g. a 3cd version of the Thin Red Line (avg 1100kbps) turned out with massive segments between keyframes, choosing modulated quant in the 2nd pass, 120 frames taking up 7-9mb or more... was hard to cut around the massive 200 frame, 12mb sections between I-frames to fit neatly. Then, i take the opposite extreme and encode a 80 min movie, and it turns out to be 20mb undersized modulated quant. 2nd pass.

i'm taking the weekend to go through and find any method to actually hit the target size i enter in, it seems like an odd goal for 9-10 hours of work to aim for +/- 500kb, instead of +/- 10-20mb from the intended size.
Koepi
13th September 2002, 15:09
Sorry pal,

your settings are definatly messed.

_do_ as I wrote and USE the search (I'll attach a picture how to do that later). And, with all respect *caugh*, this is OFF TOPIC to the MAX.

Btw. - I hit in the bounds of +/- 24kb max. variation from my target size. So rethink your settings properly after reading all available documentation.

Did you read forum rule 1?

Koepi
Easy123
13th September 2002, 21:57
@Koepi

I also did a Test with "From Hell" and the movie looked damn good until it came to the end where "Jack the Ripper" puts the Heart into the Fire... The whole scene blocked up with ugly Macroblocks... I used MPEG Quantizers in first pass, and MOD HQ in Second Pass Ext (Quant 2-6 & 2-16) (stats File done with Statsreader). Do you know an Answer why it suddenly blocked up??? never had this prob before. All movies came out Perfect....


Cu
Easy123
manono
13th September 2002, 23:12
Hi-

Perhaps you started the lower quality End Credits too early?

And, of course, fire is hard to compress. Did you run DeBugView while encoding so you could check what quants were being assigned to the fire scene?
Easy123
14th September 2002, 07:48
No haven´t run debugview but guess I´ll do that and take a look....


Easy123
Koepi
14th September 2002, 09:53
The movie looks perfectly here no matter what I do.

Easy, credits range is from 166376-176028 ;) Please rechceck that, manono is probably right.

Best regards,
Koepi
Easy123
14th September 2002, 18:20
Well, now it works... Guess I must have typed a wrong number into statsreader, without seeing it... (am i dumb, doh :D )
Koepi
15th September 2002, 05:53
Results for Zoolander, lanczos3 resize to 640x272, first pass size 1233mb, target size 642999kb, reached 643020kb.

[1016] Quantizer distribution for 2nd pass:
[1016] Q:2:125
[1016] Q:3:46806
[1016] Q:4:70631
[1016] Q:5:2089
[1016] Q:6:9

No quantizers >6, statsreader now seems to work better than internal CC! :)

Just wanted to share that again :)

regards,
Koepi
iago
15th September 2002, 10:30
FULL EDIT: Sorry. Totally cancelled! Using Statsreader 1.7 and ColorYUY2 together (for a visual test, regarding the treatment of blockiness in black areas as well) is completely contradictory. Statsreader 1.7 itself is trying to do this, not scaling the minimum sized frames! Seems I got a bit confused recently :(.
MoonWalker
15th September 2002, 12:06
@Koepi

Hi,
Just an idea. Why don't you put the scaling code of the stats reader into XviD's scaling code? For example, at normal CC if we use high/low : -1/-1 could triger your scaling code..

Just a thought,

Excellent work

MoonWalker
Koepi
15th September 2002, 12:45
@MoonWalker:

That's exactly what I plan to do, but there is still the problem with the propper min. frame size formula... if that's solved, I'm going to hack that into 2pass.c (i thought I mentioned that some times now ;) ).

Else it would be "useless" work (well, not really, you can always use StatsReader ;) ).
We're investigating on this, but it seems that only -h and Eduard seem to care about that issue... let's see what happens, I'm definatly way out of "simple maths" training and don't find the magic formula.

Best regards,
Koepi
manono
15th September 2002, 15:06
Hi-

TheWEF's formula for Minimum Bit Rate in his Modified Stats File for DivX3.11 was 25% of the Average Bit Rate, with a bottom at 270. So, once the Bit Rate got above 1080, the minimum started to rise. It always seemed reasonable to me. Plus, he gave us the option to adjust it ourselves (useful for dark movies done for 1 CD).
Koepi
15th September 2002, 15:27
Thanks for the hint, but that's not what I've been looking for. XviD is _not_ nandub/divx3.

I now have the formula for pframes min. size, for iframes the error gets distributed much "finer" so a special treatment here isn't necessary IMHO (and very complicated - not possible without access to the bitstream itself... the _final_ bitstream with a possibility to reencode the frame when necessary...).

I'll implement it into the StatsReader soon and will call it v1.8.

Thanks for the help,

and best regards,
Koepi
Gazza
18th September 2002, 04:39
I've just caught up on the work on Koepi's statsreader and have decided to jump in with v1.7 and do some testing myself. After unloading the zip I tried to load my first .stats (first pass) file. Unfortunately statsreader dropped out with the following message.

Statsreader.exe has created errors and will be closed by Windows. A log file will be created.

This only happens when I try to load the stats file.

I have tried with previous versions of statreader (v1.2 onwards) and the same thing happens. I thought that my stats file could be corrupted by i can load it into perfectXvid by Unforgiven OK and get decent results back.

Please note my test was to rip a single vob from a dvd and process this through dvd2avi, vdub, etc.

Any ideas anyone?

Also, I use gknot to create the avs file. Is this necessary when using the statsreader?

Gazza
Koepi
18th September 2002, 08:36
A dll in your system is replaced by a damaged one.

fetch yourself a free, actual, updated virii-scanner from www.freeav.com and try to clean your system.
A fresh install could help as well.

Regards,
Koepi
Koepi
18th September 2002, 10:25
Ok, here it is: StatsReader V1.8!

Thanks to Michael Niedermayer (I think he's the mpeg4 guru for ffmpeg?) I now have a formula to correctly determine the min frame size for IFrames, too, and my first tests seem very promising.

Now I have to hack that into XviD's 2pass.c. And the overflow treatment should learn those values as well, with that we get _perfect_ scaled curves :)

Well, don't wait any longer, fetch yourself a fresh 9kb huge copy and test it! :)

Thanks and best regards,
Koepi
iago
18th September 2002, 11:17
@Koepi

Right away boss! ;)

many thanks!..
iago
SolarBurn
18th September 2002, 11:46
I'm having the statsreader crashing after the stats file is loaded problem aswell, I've checked my system for virii and nothing came up, I think it stopped working after I did a windows update, any idea what dll is broken?
Gazza
18th September 2002, 12:41
Koepi, I use a virus scanner that recieves new updates almost on a daily basis so a virus might not be the problem. I agree it could be a *.dll issue but not sure which one. Any guidance? Are there any specific *.dll's that statsreader calls up?

Thanks in advance

Gazza
Koepi
18th September 2002, 12:48
The StatsReader is a VS98/MFC app and thus should rely on MFC42.dll.

my version here is:

6.0.8665.0 (6.0.400)

972kb, 04.10.2000, 14:00h

I hope this helps.

If anyone has an idea why this doesn't work for everyone, please let me know and I'll try to make a "bugfix" for that.

Best regards,
Koepi
sherpya
18th September 2002, 12:54
I have no problem: winxp sp1
Koepi
18th September 2002, 13:03
I compiled a statically linked version, could you try if this solves your problem please? This doesn't replace the 9kb download, this is just experimental for users who have problems with the usual statsreader. So everyone please use the download in the post above, only users having problems should download this version(102kb):

EDIT:
deleted attachment since it doesn't solve the problem, no need to further abuse Swede's bandwidth.
SolarBurn
18th September 2002, 13:32
The staticly linked one breaks for my most recent stats file aswell.
I tried that and the 9kb version on an old stats file and it worked fine both times.
The old stats file was made with XviD-03092002-1 and the newer one with XviD-04092002-1. This is my first encode with the newer codec and I'm fairly sure the encode went smoothly but I'm going to redo it again just now to see what happens
SolarBurn
18th September 2002, 13:44
couple of short tests and the statsreader is working fine with my new stats files, I'm doing the whole stats file over again to see if there is any difference between the files, I think I am using the same settings as I used the first time
SolarBurn
18th September 2002, 15:55
I created a new first pass stats file for the same movie and it came out at exactly the same size using the same settings and it still crashes the statsreader, you want a copy of it?
Koepi
18th September 2002, 16:38
Yes please, compress it with zip, upload it anywhere and send me the link as PM please :)

Thanks,
Koepi
NiTroGen
18th September 2002, 23:36
Well, while using StatsReader I got this really wonderful idea:

Since StatsReader is used right away after the 1st pass, why don't you add a tiny little button than will import the parametres used in 1st pass (filename & credits range) directly from registry?

Yes, I know, I am too lazy to select the file manually, then check the necessary boxes and type the credits range, ( :p ) but I still think it is a wonderful idea.
rui
19th September 2002, 08:44
Not that i want to brag myself or anything, but i already posted something like this a long time ago, but no one seemed interested at the time: http://forum.doom9.org/showthread.php?s=&threadid=32538&highlight=statsreader

Koepi, just asking out of curiosity:

Would it be possible to incorporate your statsreader into xvid itself?

I mean, when he/she choosed external 2nd pass, xvid would default to your statsreader, using the credits parameters that we already had configured in 1st pass, only asking the user to input, as always, the desired filesize and the other parameters that we can input when using the external 2nd pass.
Then it would scale the curve, not using the internal code, but the statsreader code, and create automatically the, for example, video2.stats in the folder we would indicate when configuring the 2nd pass external encode.

The advantage of this would be to automate the encoding process in the same way that it is automated when choosing internal encoding(using jobs in Vdub, for example).
Koepi
19th September 2002, 12:09
@rui,

go to page 10 of this thread where I wrote about that - again.

@NitroGen:

I'll do so when I'm feeling so bored that I want to fiddle with the registry stuff again ;)

@solarburn:
I can reproduce the crash, but didn't find the cause yet, I'm still investigating. Thanks for the file!

Regards,
Koepi
NiTroGen
19th September 2002, 13:29
@rui:

Sorry, man. I've never seen your post, but I'm glad there are more brilliant minds in this forum, sharing the same great ideas. :D

@Koepi:

I'm looking forward to you get bored. :p
(Btw, I do use StatsReader in every encoding I make and I always get terrific results. Keep up the good work.:cool: )
rui
19th September 2002, 14:01
NiTroGen, please don't get me wrong. :)

Between the two of us, you are the briliant mind. :D
After all you wroted a software to try and help xvid encoding (it wasn't the most concensual software, but you made something :p ).

I am just a guy who likes movies, likes to try new things, and likes to experiment a little and try to improve what is established. I know nothing about the theory behind all this. All i know is by reading here and experimenting later. ;)
NiTroGen
19th September 2002, 14:21
Originally posted by rui
I am just a guy who likes movies, likes to try new things, and likes to experiment a little and try to improve what is established. I know nothing about the theory behind all this. All i know is by reading here and experimenting later. ;)

Well, you've just described me. I'm not a brilliant mind. And my software is not brilliant too (Right, Koepi? ;) ) But as you said, experimenting and improving is all I do. Anyway, Rui, enough with this off-topic talking (and the compliments). Let's continue our conversation about StatsReader...
ookzDVD
20th September 2002, 09:26
@Koepi,


No quantizers >6, statsreader now seems to work better than internal CC!


Any plans to add your scaling algorithm to the 2-Pass-Int ?
Koepi
20th September 2002, 10:16
@OokzDVD: as I wrote to rui, look at page 10 of this thread - I stated my point already ;)

@NitroGen: I didn't say your programs are BS. I just got annoyed as it seemed that you ignored the info I gave you and repeatedly tried the impossible thing: curve aggression can't be determined by stats files, it has to be set by the user accordingly to her/his preferences... :) As "apology" I felt bored, just for you ;)

@All:

Find StatsReader 1.9 attached. I still couldn't find the error that causes a crash with some stats files, but we now have an "Auto Load" button which reads XviD's registry settings for loading the statsfile and setting the credits ranges. (I feel it's wrong to setup the codec for the 2nd pass right away so i left that alone).

Enjoy, and please tell me if it works for you! :)

Best regards,
Koepi
Gannjunior
20th September 2002, 10:59
Hi Koepi,
The external pass by statsreader is really precise.I've only a problem: how could I choose the exact quantizer to assign to start and end credits?In the last film I've obtained all quantizers(in credits) of 2 and 3.

TIA

ciao
Koepi
20th September 2002, 11:03
@Gannjunior:

You have to setup the XviD 1st pass with credits and a fixed quantizer to make that work. When choosing "2nd pass external" make sure that the value in those boxes for the quantizers is still the same as used during first pass - after that just enter the credits ranges (or hit auto load ;) ) in the StatsReader and it won't touch those frames.

I hope this helps,

Regards,
Koepi
Gannjunior
20th September 2002, 13:22
@Koepi

I've setted I-frames=20 and P-frames=31 in the 1st pass.However it doesn't follow them.the film is LOTR:

720x304 Desired size: 1.592.000
real size by statsreader 1.592.034 (perfect!)

I've observed debugview during the end of 2nd pass and I've seen that,to respect the size,it has been used lower quantizers in the credits,and not in the film.
The problem isn't a possible saturation of the film,'cause at that resolution,LOTR satures near 2,5-3Gb.
:confused:

thx,ciao
Koepi
20th September 2002, 13:35
Which XviD build are you using?
Gannjunior
20th September 2002, 13:42
04/09. I think is your last build.And i've used statsreader 1.7(i've seen that now you've done the 1.9 :) )
Koepi
20th September 2002, 14:44
Dunno what went wrong, please read to the massive amount of posts here to see how to setup the codec so it produces what you want to see - if you didn't make any errors in the xvid setup, and you entered everything in statsreader accordingly, this thing shouldn't happen. You know you have to activate credits even if you can't choose anything for the second pass?

Regards,
Koepi
Defiler
20th September 2002, 15:27
Wait.. I've been leaving the credits setting alone in the first pass, and specifying the range of frames in the "Ecredits" section of StatsReader. Is that incorrect?
I have read every post in this thread, but some of it seems contradictory to me. Please excuse my confusion.
manono
20th September 2002, 15:42
Hi Koepi-

I can confirm a problem with End Credits with Ver. 1.7 (haven't tried the most recent version yet). For the first couple of movies I thought I had been careless either in the first pass settings or with the StatsReader End Credit settings. So I doublechecked everything with my most recent movie. In the XviD first pass, end credits were set for 10, but Moonwalker's nice little program gave me this:

Quantizers Used For Credits :
--------------------------------
Quant 1 Used : 1 Times.
Quant 2 Used : 22 Times.
Quant 3 Used : 3 Times.
Quant 4 Used : 22 Times.
Quant 5 Used : 5733 Times.
Koepi
20th September 2002, 16:29
Hm, how to put this all into 1 post?

I'll try:
- 1st pass xvid setup: usual first pass, if you want to use credits, select "fixed quant credits" and set the quantizers like you prefer them.
- after that, fire up statsreader (use the most recent version please.) Hit auto load. enter the size you want that movie to get and hit "save as.." and enter a name. Remember that name.
- back in xvid, select 2nd pass external. setup as usual, but disable altCC and use regular CC with high/low 0.
- select the 2nd pass stats file you just generated in the bottom field called "2nd pass stats". You still must have the first pass stats file! Leave it untouched in the upper field.

Don't touch your credits settings, they're the same as in the first pass.

With most recent builds from CVS the stats reader + credits might work, but i didn't verify that yet.
With this procedure and my 4/9/02 build everything works fine for me.

Regards,
Koepi
Gannjunior
20th September 2002, 19:59
@Koepi

I'm following the discussion about statsreader from the begin.I've used the same settings you have told now.However I'll try to use your last statsreader 1.9 anf I'll cross my fingers;)
I'll post my results.

Regards,
Gannjunior
Defiler
20th September 2002, 20:26
Thank you for that summary.
Just one final clarification..
The "Credits" tab should look like this for both the first AND second pass, correct?
http://hellninjacommando.com/misc/xvid-credits.png
iago
20th September 2002, 22:29
@Defiler

Yes, exactly like that for the first pass (if you want to use a fixed quantizer of 20 for the credits of course ;)). But for the second pass external, quantizer boxes are deactivated.

regards,
iago
Defiler
20th September 2002, 23:32
Thank you. I've been doing it wrong. :)
iago
21st September 2002, 07:06
@Koepi

My first pass stats file crashes with statsreader 1.9 and with 1.8 too (I haven't tried with the previous releases). First pass size of the file is 2165 mb (as analyzed by xda.exe). Can it be a problem of filesize >2gb ?

best regards,
iago
Koepi
21st September 2002, 11:05
Nope, it's no filesize problem, those variables are all "big enough" to carry that information - and it would throw a propper exception if those variables cause trouble.

Maybe you should try the new XviD build, with the new internal scaling modifications (to name min[i|p]framesize limitation and modified overflow treatment) it should give an apropiate scaled curve when using linear sclaing (disabling altCC and setting regular curve compression to 0/0 - sorry, I've to repeat that so noone misunderstands what's going on ;) you can of course use altCC with the settings described some times in this thread, too).

I hope this helps,

regards,
Koepi
NiTroGen
22nd September 2002, 02:33
Originally posted by Koepi
Thanks to Michael Niedermayer (I think he's the mpeg4 guru for ffmpeg?) I now have a formula to correctly determine the min frame size for IFrames, too, and my first tests seem very promising.Could you share the formulas about IFrames & PFrames?

And a question that I have for a long time: How is AVI overhead calculated? :confused:

(Yes, I'm trying to create a linear downscale program, too. I hope that you don't find it competitive and share some of your knowledge with us. ;) )
Gazza
23rd September 2002, 05:18
Originally posted by Koepi

Find StatsReader 1.9 attached. I still couldn't find the error that causes a crash with some stats files, but we now have an "Auto Load" button which reads XviD's registry settings for loading the statsfile and setting the credits ranges. (I feel it's wrong to setup the codec for the 2nd pass right away so i left that alone).


Hi Koepi,
Tried the latest version of statreader but still crashes when loading the .stats file. It does this also when pressing the Auto Load button. However when pressing the load button I did notice a small message in the statsreader form that showed that the statsreader obtained the correct number of frames but then also said 0 bytes? So it seems to be analysing the .stats file in part but then stops retrieval - its not something silly like a divide by zero or a counter not set right? - just a thought :(

I ran the above tests with the latest xvid (XviD-22092002-1.exe).
ookzDVD
23rd September 2002, 05:43
@Gazza,

I think you could use the 2-pass-int now, after Koepi applied
his scaling algo into the 2-pass-int.
Gazza
23rd September 2002, 07:10
Thanks ookzDVD - I will give it a go.
iago
23rd September 2002, 12:50
@Gazza,

You're right, I have the same problem here, Statsreader 1.9 still crashes, but only with "some" stats files, not with all of them in my experience. And with such problematic ones that the statsreader refuses ;), I follow the way that ookzDVD mentioned (2ndPass internal, in the "two pass" tab: setting high-low: 0-0, and disabling "alternative curve system", to linear-scale the curve).

regards,
iago
Koepi
24th September 2002, 10:25
Originally posted by NiTroGen
Could you share the formulas about IFrames & PFrames?

And a question that I have for a long time: How is AVI overhead calculated? :confused:

(Yes, I'm trying to create a linear downscale program, too. I hope that you don't find it competitive and share some of your knowledge with us. ;) )

Hm, want to render my proggi useless?

Well, here we go:

pframe minsize: (headersize + macroblocks + 8) / 8
iframe minsize: (headersize + 22 * macroblocks + 8) / 8

headersize is 80 bits in the pframes and 232 in iframes - this changes with cust. matrix for 1 frame but shouldn't be of bigger disturbance.

Avi frame overhead is 24bytes per frame.

(I still feel somewhat exploited now...)

Koepi
MoonWalker
24th September 2002, 11:00
I have found about the overhead this formula :

overhead=((23.086+((intra+inter)*0.19140625))/8)+5

I tested it and it works very well..But I think 24bytes per frames is much simpler :p

MoonWalker
Koepi
24th September 2002, 11:04
LOL! :)

Regards,
Koepi
NiTroGen
24th September 2002, 13:38
Hm, want to render my proggi useless?
Well, after your latest build, all linear curve downscaling proggies are useless. ;)

headersize is 80 bits in the pframes and 232 in iframes - this changes with cust. matrix for 1 frame but shouldn't be of bigger disturbance
What do you mean, by saying 80 and 232 bits? Headersize isn't the value of the dk_v parametre? Sorry, I didn't understand this.

Anyway, those formulas are needed for any kind of curve downscaling, linear or not, so it's good to be known to all people that want to experiment with downscaling. Maybe someone will find the ultimate formula. :rolleyes:

Thanks for the info, Koepi. I've already found the overhead formula in xvid sources.

Btw, since audio interleaving adds more overhead to the avi, I think it's a good idea to add an option to statsreader (if you continue its development) to calculate the total overhead (video + audio) by setting the audio interleaving options. So, when we'll have the final muxed avi, its size would be exactly the desired one. That's something that will never be implemented in the xvid codec, so our programs will still be usefull.

@MoonWalker:

Slow the cabbages! (MoonWalker should understand this expression). Check my formula about overhead, that I found using Gordian Knot and the method of the least squares:

Overhead = (34.418 * (MovieLength in minutes)) + 2.8673

:p
Koepi
24th September 2002, 13:43
That mux'ing overhead is not really an issue, use Nic's calculator for that - since I only (and I mean _only_) do OGMs that is a value which isn't "important" and error-prone...

And yes, of course you find those formulas in the xvid sources as foxer added them after my proposal.

Regards,
Koepi
sungey
30th October 2002, 20:48
hello ... i have a question

i have been using statsreader on nandub stats file .. it works fine and file size is very predictable ... but im wondering if
statsreader scales nandub stats the same way GKnot does ...
any idea ?

statsreader 0wns ^_^
Koepi
30th October 2002, 21:57
No, statsreader is different from GKnot.

GKnot can do far more "sophisticated" curve scaling for nandub as nandub uses "motion" and "luma" which don't get used by xvid (it's simply not necessary as XviD has no bugs/weaknesses there and is straight forward with those things, DivX3 has some features turned on by bitrate [and also switched off again if the bitrate range is left again...]). You'll get a plain linear scaled bitrate curve, and the gauge/bitrate buffer of nandub should cope well with it, too. But DivX3 has some bugs/features which really make it possible to gain more quality by getting (ab-)used :)

But thanks for the report that you successfully use it with DivX3/Nandub! Never thought someone would do that :)

Best regards
Koepi
bond@doom9
10th November 2002, 18:13
I once read somewhere that to achieve the same results as editing the 1st pass stats-file via statsreader i have to uncheck "use alt. curve" and then to set high/low curve compression to 0.

Now I made three test-files (using Koepis 041002 binary):
1) one with the procedure described above
2) one with an external statsreader19 edited stats-file
3) and one with the default settings

The results were that test-file 1 and 3 had exact the same filesize!
test-file 2 was a few bytes smaller!

So what does this mean? Does Koepis recent build automatically uses the statsreader features (so external editing of the 1st pass stats-file isnt needed anymore)? But why is then test-file 2 a little bit smaller?

:confused:

If this issue has been discussed before i am sorry to post it again but I didnt find any answers by using the search function
Koepi
10th November 2002, 18:47
Statsreader scales the curve a little bit different than XviD internally.
Therefore the file size difference (usually statsreader scaled curves reach the targeted file size within a smaller range, more on the spot).

But since filesize is not the important thing of that but "constant quality", you should take a look at the quantizer distribution spit out by XviD/VfW at the end of the 2nd pass.

Regards
Koepi
bond
13th July 2004, 17:58
an old thread, but still the main one for discussing koepis statsreader tool i assume :)

so here is what i found today:

i used statsreader 2.1 (able to open xvid 1.0 style .pass files) to analyse a .pass file i created with xvid 1.0.1

after opening the .pass i noticed that the displayed framenumber showed some frames more than in the input source (i tested virtualdub and virtualdubmod (both latest versions available today), with both the classic dvd2avi/mpegdecoder and new dgindex/dgmpegdec combination)

the .pass file produced with virtualdub showed 1 frame more than the input
the .pass file produced with virtualdub showed 2 frames more than the input

i checked the .pass files manually too: no d/n (delay frames) marked

i used 2 b-frames (no packed bitstream, other b-vop settings on default), gmc, qpel, vhq4, chroma, trellis, aq (well the whole stuff :D )

can anyone reproduce this? can anyone plz try it and check manually (for example in word) whether the number of frames is the correct one in the .pass itself
i assume its an issue in handling b-frames in vd(m), but might also be in statsreader
Dark-Cracker
13th July 2004, 18:38
hum a bit out of topic but does someone know how the old xvid scaled the curse using hight % and low % distance given by perfectxvid (who analysed the .stats file) .
it was different about linear scaling but does someone have an explain or a source code ?

++