(XviD) Statsreader

[iago]

hehe, OK Koepi ;)

Here we go with statsreader 1.4 then! (dead-born 1.3 went to trash).

This time: Matrix MPEG/MPEG with external curve compression ;).

best regards,
iago

[Koepi]

Matrix, MPEG/MPEG, target size: 599000 - reached: 599012 !!!

YES, finally! :)

The quantizer distribution is a little disturbing though:

[888] Quantizer distribution for 2nd pass:
[888] Q:2:1695
[888] Q:3:91746
[888] Q:4:75400
[888] Q:5:15899
[888] Q:6:1259
[888] Q:7:84
[888] Q:8:19
[888] Q:9:16
[888] Q:10:18
[888] Q:11:13
[888] Q:12:34
[888] Q:13:5

But i have to check the visuals first before drawing conclusions. It's bad to have even few of such highly quantized frames, but maybe they are hardly noticable - I hope so :)

Best regards,
Koepi

EDIT: starting first pass for H.263/H.263 encode... :)

[MoonWalker]

@Koepi

I'll do a h.263/h.263 with the matrix to compare our results.I am thinking to do it with cc 0/0(linear scaling) to see the diferences with you statsreader..what do you think?

BTW What reslution are you using? Simpleresize??

MoonWalker

[iago]

@Koepi

Matrix, MPEG/MPEG, target size: 599000 - reached: 599012 !!! That's really great! Yes, finally! So, thanks again Koepi, for your non-stop effort to solve this inaccuracy problem with linear scaling, which is finally overcome ;)...

Matrix, MPEG/MPEG, target size: 757000 - waiting (that means it's gonna be reached then! :))

kindest regards,
iago

[Koepi]

@moonwalker:

avs:
SetMemoryMax(40)
LoadPlugin("D:\Video_TS\sbc-ripping\avisynth\mpeg2dec.dll")
mpeg2source("D:\Video_TS\thematrix.d2v")
Crop(0,78,720,420)
BicubicResize(640,272,0,0.5)

Credits: 20/20 quant from 186188-196155

(R2 PAL DVD source)

Let's compare if we finally have the same results! Thanks for helping out here :)

@iago:

Please report if the results are as expected :))

I'm glad that the problem is solved for me at least, let's hope that this wasn't just a random effect!

Best regards,
Koepi

[Bulletproof]

I just tried it and my file came out 14 megs oversized. I set the ALT CC off and put everything to 0 and put proportinal distribution. And I didn't use credits either.

[colasonic]

the statsreader 1.4 with XviD-30082002-1 is great! they hit my target precisely!
I used The Royal Tenenbaums and set the desired file size to 644121KB. they finally gave me a file as 644122KB!
Thanks to Koepi, iago and all testers for your efforts!

H.263/H.263
CC Hi/Lo=0/0, pay back proportionally 250, lumi masking in both passes, AltCC disabled
Credits 20 Quantizer

[iago]

@Koepi and everybody

Matrix, MPEG/MPEG, target size: 757000 - final size: 757012 !!! ;)

(external cc, lumi both passes, quantizers: 2-4/2-8, hi/lo: 0/0, altCC disabled, payback proportionally, credits 20quant)

Only one word: Fantastic! :)

thanks again for this (finally achieved :)) great work,
iago

[MoonWalker]

Originally posted by iago

(external cc, lumi both passes, quantizers: 2-4/2-8, hi/lo: 0/0, altCC disabled, payback proportionally, credits 20quant)


iago, when you use external cc it doesn't matter witch cc you are using in XviD(this is all internal :) )..That's why it is called external..

MoonWalker

[Koepi]

Moonwalke,

now you're getting very inprecise.

The overflow treatment is internal while the curve compression is external.
Your post is flawed, iago's is right.

It doesn't matter if you use altCC or regularCC, you just have to diable it by setting it to the values i mentioned about a hundred times now, so that only the overflow treatment is in effect.

I hope you didn't do this in other ways now, else your results won't help comparing internalCC (!) vs external CC.

Regards,
Koepi

[iago]

Originally posted by MoonWalker
iago, when you use external cc it doesn't matter witch cc you are using in XviD(this is all internal :) )..That's why it is called external..@MoonWalker
Yes, that's already clear I guess; so I couldn't fully understand what you're getting at :)...

Also, I remember it was discussed that when using external cc, we'd use either hi-lo: 0-0 with "normal internal cc" disabling altCC, or mrq: 100 (or strength: 0) with the other parameters default in "alternative internal cc" to get a proper linear scaling with correct overflow treatment. I just provided info about what path I followed considering these previous discussions.

kindest regards,
iago

EDIT: Oh, simultaneous reply from Koepi :).

[MoonWalker]

Originally posted by Koepi
Moonwalke,

now you're getting very inprecise.

The overflow treatment is internal while the curve compression is external.
Your post is flawed, iago's is right.

It doesn't matter if you use altCC or regularCC, you just have to diable it by setting it to the values i mentioned about a hundred times now, so that only the overflow treatment is in effect.

I hope you didn't do this in other ways now, else your results won't help comparing internalCC (!) vs external CC.

Regards,
Koepi

:rolleyes: :rolleyes: Hmm...Sorry..I thought it was like that.I mean I knew that the overflow was done internaly, but I supposed it didn't matter which values we use for the cc/altcc cause all the scaling was done externaly(ie the distance were not in use)...But I was wrong :rolleyes: ...Thanks Koepi for clearing this out :)

MoonWalker

[primitive]

Is hardcoding the AVI overhead in the statsreader a good idea?

1) What if you are using an alternate container format for your final encode? If your final output is ogg, and you use your statsreader, your result is going to be undersized.

2) What if you use gknot and use gknot's internal calculation for overhead (multiple audio streams, files)? If overhead is higher, your file is going to be oversized.

Just a thought,

-p

[Koepi]

@primitive:
OGG has some overhead, too, and usually you can say muxing an avi with an ogg vorbis sound file gives you nearly no size increase due to overhead, the formular can be simplified like avi_size+ogg_vorbis_size=OGM_size. That's the way I calculate for my encodings, and I think everyone does when using OGM.

So if we hit that avi_size we were calculating, there's nothing wrong with it.

@Moonwalker:
external compression is somewhat messed in xvid, therefore my credits-hack in my latest build :-/ I was a little inprecise too btw., if you set those values to 0/0 etc (as iago posted), you simply have no further influence of curve compression, if you set it to other values the curve will get strangely influenced by it, resulting in a non-linear scaled curve at least.
I hope this gives a better picture of what's happening.
To compare internal vs external linear scale I suggest you disable altCC and use just the regularCC with high&low set to 0. The aimed target size should be the same and the credits range&quantizer restrictions, too (as well as the "usual" quantizer restriction, greyscale modes,....)

@iago:
hehe, tends to happen somewhat often ;)

Regards,
Koepi

[Bulletproof]

I tried again and this is what the statsreader said:

"down to 1171.79mb (1199908.51)"

I set the target size exactly to 1.2 gigs, the resulting file came out to be:

09/03/2002 07:25a 1,228,814,336 blah.avi

29 megs (rounded) oversized.

Anyone know what I might be doing wrong?

[Koepi]

bullet,

I tested with different things end everything's ok.
You'd get the same result with internal compression, if you have a size problem with statsreader (when using credits, you still rely on my latest build, but you sure know that as you read this thread).

btw.,
1199908.51 is very close to
1200000

So, it's definatly your settings which are mistaken(which you didn't provide - and they definatly don't belong here). See, noone else has a problem like you have.
If you restrict the quantizers too much,.... and so on, and so on...
you know, the usual stuff to check when not getting the right target file size.

Thanks,
regards,
Koepi

[Bulletproof]

Could it possibly be the keyframes? It is quite a long movie, about 2 and a half hours long.

[Koepi]

Did you use iframe boost? it was flawed in external compression mode.
Should be fixed with

XviD-03092002-1:
- Updated to Statsreader 1.4.
- Foxers 2nd pass external/credits fix

Still, this shouldn't result in oversized movies, it must be your quantizer restriction then i think.

Regards,
Koepi

[rui]

Well, this is a little bit off-topic :o, but i was concerned about the quantizer to use in 1st pass when using the new ModHQ mode.
In the old modulated mode one should use h.263 quantizer in 1st pass.
But if i remember correctly, someone once said that if one knew that it would get in the 2nd pass most percentage of mpeg quantizers, then he/she should use in the 1st pass not h.263 but mpeg.

Now, in the new mode, since we have a majority, for 1 cd encodes, of mpeg quantizers, should we still use h.263 quantizer for 1st pass, or turn to mpeg?

In my short "The Replacements" trailer test, i got very similar results by using h.263 or mpeg in 1st pass, but when using h.263, in spite that in the 2nd pass the mpeg was used in much higher percentage, i got a little lower average quantizer and lower, by one point, absolute higher quantizer used.

What you guys think about this?

[Koepi]

You're right rui, for the "new mod HQ" mode it would be wise to take mpeg quant for first pass when the second pass quant will be >3, h263 else.

Regards,
Koepi

[rui]

So mpeg it is.
Thanks for the clearup, Koepi :)

P.S. Man, one of this days you will break the 2000 post barrier. :D
I believe that you will be the first person on this forum to achieve that:cool: (at least i didn't crossed with anyone else with that much posts in my lurkings).

[iago]

Originally posted by Koepi
XviD-03092002-1:
- Updated to Statsreader 1.4.
- Foxers 2nd pass external/credits fixOh, I was just getting used to 30082002-1 with statsreader 1.4 ;).

Well, just kidding of course ;). That's another good news. As soon as my current long long encode is finished, the new build will be enthroned on my machine! ;)

kind regards,
iago

[Bluedan]

My personal test encode resulted in 5.6MB oversized.
I left I-frame boost at default 20.
Aimed at 1249920 KB (that's what I entered in Statsreader1.4 last night), got 1255526 KB in the end as resulting file.
Not a desaster as the quality is pretty, and there is still room for cut points for 2CD splitting. ;)

settings
movie: together!
motion search prec 6
quant type h.263 both passes
min I-frame quant 2
max I-frame quant 6
min P-frame quant 2
max I-frame quant 14 or 16 (sorry, it was dark...)
lumi enabled
alt CC disabled
CC high/low set to 0 with prop.payback delay 250
left I-frame boost at 20
Hinted ME off of course
credits fixed quant 20

XVid build 30-8-2002 with SR_v1.4

Though the numbers don't match the expectations here (Koepi please tell me that the fault is on me, not the code) I'm happy with the quality I reached. Don't have time for further comparison testing, nor did I ever touched that debug analyser, shame on me maybe.
BTW, I had to run it in parallel to get the debug output, right?
Next time.
Keep up, I'm your fan !

[Koepi]

bluedan,

I think your settings are all ok except for the iframe-boost...

so sorry that you have to redo the second pass, try iframeboost 0 :-/ Or use the new build, there it is fixed.

Thanks for testing,

Regards,
Koepi

[iago]

Originally posted by Bluedan
I left I-frame boost at default 20.
XVid build 30-8-2002 with SR_v1.4 AFAIK, I-frame boost default is "0" not "20" in 30082002-1 build, and setting it to 20 may be the reason of your oversize.

best regards,
iago

EDIT: Koepi, come on man, I can't believe it! ;) Another simultaneous post !!! LOL! just strange... :)

[Koepi]

@iago:

that's really impressive, right :) Maybe we're too often here at the same time ;)

Serefe,
Koepi

[Bluedan]

Thanks for sonic speed posting...
I'm to lazy now, I think the overburn function of NERO can fix an eventual lack of size accuracy :p
Anyway, I'm not only a lazy but also anxious transcoder, so I always assume 695MB as 1CD size to be flexible with setting the cutting knife for 2CD split point (->chapter start).
Thanks both.
Decided to run dgbv_nt (new version 4.2 out today on http://www.sysinternals.com ) next time.

[cult]

ok,i have a stupifd question.
Here it is:xvid has option for credits at start of the movie.So some of us may use it.Could it be it the cause for failure sometimes?I havent tested the new 03092002-1 so I dont know how it works,but when I tried the statsreader 1.4 I realized that I had 1st pass with some option checked for credits at the start of the movie,so I realized that was gonna be useless.Maybe adding this option too?

[Koepi]

Hm, but this might take some time, right now I'm watching the season 9 opener of x-files to verify that it's as abd as it was talked about ;)

After that I'll happily add starting credits, too.

Regards,
Koepi

[jpl]

Is it possible that we could get a version that runs with older processors for those of us not lucky enough to have newer CPU's?

Please??

I'd like to test this out and see how well it works but it wont run on my K6-III. I'm asuming that it's a result of using the Intel Compilers in some kind of mode that breaks the compatibility.

Thanks

[Koepi]

Ok,

StatsReader 1.5 with starts- and endcredits separated, plausibility-warning (trarget size smaller than the credits... I think someone ran into this trouble already), compiled for old processor users convenience with MS compilers.

Regards,
Koepi

[ookzDVD]

@Koepi,

The v1.3 solves the "negative issue" ;) thank you.

_but_ sometimes I'm having error while closing the StatReader after scaling. The scaling is OK, the file is generated.

[Koepi]

Yeah, i get it too sometimes, I didn't get a plausable behaviour. Might be the memory allocated for the statsfile not freed properly, freed too often ;) or a file handle doesn't get destroyed.

But it's nothing vital :)

Thanks for the report!

Best regards,
Koepi

[rui]

The freeze on exit stuff happens to me also, but only in home (winXP).
Here at work (Win95B), i never got that, and tried all the builds. And they say that WinXp it the most evolved OS from Microsoft. Go figure:rolleyes:

[rui]

Koepi, i tried to use the modHQ mode with your latest build, but it is using quant mpeg all the time, when it should use quant h.263 for 2 and 3 quantizers, right?

[Gannjunior]

@Rui
Which is your film?
But the codec uses MPEG even for quantizer < 3 ?

ciao

[cult]

what can I say Koepi?thank u alot!U r the man!

[rui]

Gannjunior, the latest builds from Koepi have a feature called modulatedHQ (experimental), that uses, for quantizers 2 and 3 quantization H.263 (clears some of the noise in the edges), and mpeg for quantizers 4 and higher (retain more detail or more true collors according to some post i have seen).

But in this latest build, when using modhq, i saw that it's using mpeg for all quantizers.

[jpl]

Koepi,

Thanks recompiling for compatibility with those of us stuck with old CPU's. I can't wait to try it tonight when I get home.

[Koepi]

rui,

i was sure i implemented it correctly, strange... I'll take a look at it later tonight, when I'm bored again, now it's too much to catch up with first ;)

Regards,
Koepi

[Gannjunior]

@Rui
Yes,it's clear,I've followed the discussion.I'm not espressed well.What I want to say is: Which is the AVG quantizer showed in the log file? Maybe your film is done only by quantizers > 3,and in according to MOD. HQ,they're all mpeg.

ciao

[Koepi]

Nonono, rui is right.

I just directly took Foxers fixed 2pass.c and forgot to readd the quantizer switching for "new mod hq (exp)".

Hm, since i have a new statsreader anyways, I think i can build yet another binary... maybe even write a short note on statsreader use.

Thanks for the report,

best regards,
Koepi

[rui]

Sorry, i didn't got you first :)

No, i watched the encode in debugview, and in quantizers 2,3,4 and 5 it used always mpeg, when it should have used h.263 for 2 and 3.
Then i made another test with the old modulated, and this one used mpeg for 2 and 3 and h.263 for the rest, as it should be.

EDIT. Sorry Koepi, i was to slow in posting :)

[Koepi]

XviD-04092002-1:
- Updated to Statsreader 1.5.
- Fixed "new mod.HQ(ext.)" mode.

And yes, I know that the main GUI still shows 1.4 as version number ;)
Have fun with testing!

Regards,
Koepi

[Gannjunior]

@Rui
ah,ok,thanks...however Koepi is faster than a thunderbolt: He's already done a new upgrade ;)

[Gannjunior]

I'd like to do a question for my friend who is just registred,so he isn't still able to post:

"Hi Koepi,I'm trying to use stats reader 1.5, with a first pass stats file of 16Mb:

version 2.0 frames 243773
size (mb) 3375 max frame size 72747

if I set : target size (kb) 600000
I get : scaled down 1383.78 MB (1416986.33 kb)- compression ratio -1.241:1!

if I set : target size (kb) 700000
I get : scaled down 926.43 MB (948668.56 kb)- compression ratio -1.063:1!

if I set : target size (kb) 800000
I get : scaled down 469.09 MB (480350.87 kb)- compression ratio -0.929:1!

it seems to me there is some overflow problem with big stats files. insn't it ? (with smaller files it works)
any suggestion ?

thanks "

[Koepi]

Hm, well, first pass file sizes >2GB could've been an issue, tried to change to 64bit integers, dunno if this already helps - but it should in theory.

Find statsreader 1.6 attached and report if the problem is solved, please.

Best regards,
Koepi

[iago]

@Koepi

I had the same negative compression ratio problem as Gannjunior with statsreader 1.4 (with a >2gb first pass size), and statsreader 1.6 seems to solve that problem. It scaled the statsfile down correctly to 3.093:1, aiming for 730000kb 1 XCD ;).

best regards,
iago

(P.S.: What a compression ratio, isn't it?! ;) The movie is 3hr-8min "Spartacus")

[Koepi]

Iago,

thanks for reporting success :)

well, compression ratio of 3:1 is unlikely to give a "wow!" effect... but for just trying it, why not ;)

Regards,
Koepi

[iago]

O/T

@Koepi

Actually, I did two first passes with that movie, one with normal routine, second to test convolution3d(1,4,5,3,4,2.8,0) in terms of compressibility on a "normal" but rather long movie ;). And convolution3d(1,4,5,3,4,2.8,0) -before resizing- gave a compression ratio of 2.875:1 ;).

Anyway, back to topic, under the circumstances, it's much more reasonable to give it a go with the normal routine (without filtering)! :)

ciao,
iago