(XviD) Statsreader

[Gazza]

I've just caught up on the work on Koepi's statsreader and have decided to jump in with v1.7 and do some testing myself. After unloading the zip I tried to load my first .stats (first pass) file. Unfortunately statsreader dropped out with the following message.

Statsreader.exe has created errors and will be closed by Windows. A log file will be created.

This only happens when I try to load the stats file.

I have tried with previous versions of statreader (v1.2 onwards) and the same thing happens. I thought that my stats file could be corrupted by i can load it into perfectXvid by Unforgiven OK and get decent results back.

Please note my test was to rip a single vob from a dvd and process this through dvd2avi, vdub, etc.

Any ideas anyone?

Also, I use gknot to create the avs file. Is this necessary when using the statsreader?

Gazza

[Koepi]

A dll in your system is replaced by a damaged one.

fetch yourself a free, actual, updated virii-scanner from www.freeav.com and try to clean your system.
A fresh install could help as well.

Regards,
Koepi

[Koepi]

Ok, here it is: StatsReader V1.8!

Thanks to Michael Niedermayer (I think he's the mpeg4 guru for ffmpeg?) I now have a formula to correctly determine the min frame size for IFrames, too, and my first tests seem very promising.

Now I have to hack that into XviD's 2pass.c. And the overflow treatment should learn those values as well, with that we get _perfect_ scaled curves :)

Well, don't wait any longer, fetch yourself a fresh 9kb huge copy and test it! :)

Thanks and best regards,
Koepi

[iago]

@Koepi

Right away boss! ;)

many thanks!..
iago

[SolarBurn]

I'm having the statsreader crashing after the stats file is loaded problem aswell, I've checked my system for virii and nothing came up, I think it stopped working after I did a windows update, any idea what dll is broken?

[Gazza]

Koepi, I use a virus scanner that recieves new updates almost on a daily basis so a virus might not be the problem. I agree it could be a *.dll issue but not sure which one. Any guidance? Are there any specific *.dll's that statsreader calls up?

Thanks in advance

Gazza

[Koepi]

The StatsReader is a VS98/MFC app and thus should rely on MFC42.dll.

my version here is:

6.0.8665.0 (6.0.400)

972kb, 04.10.2000, 14:00h

I hope this helps.

If anyone has an idea why this doesn't work for everyone, please let me know and I'll try to make a "bugfix" for that.

Best regards,
Koepi

[sherpya]

I have no problem: winxp sp1

[Koepi]

I compiled a statically linked version, could you try if this solves your problem please? This doesn't replace the 9kb download, this is just experimental for users who have problems with the usual statsreader. So everyone please use the download in the post above, only users having problems should download this version(102kb):

EDIT:
deleted attachment since it doesn't solve the problem, no need to further abuse Swede's bandwidth.

[SolarBurn]

The staticly linked one breaks for my most recent stats file aswell.
I tried that and the 9kb version on an old stats file and it worked fine both times.
The old stats file was made with XviD-03092002-1 and the newer one with XviD-04092002-1. This is my first encode with the newer codec and I'm fairly sure the encode went smoothly but I'm going to redo it again just now to see what happens

[SolarBurn]

couple of short tests and the statsreader is working fine with my new stats files, I'm doing the whole stats file over again to see if there is any difference between the files, I think I am using the same settings as I used the first time

[SolarBurn]

I created a new first pass stats file for the same movie and it came out at exactly the same size using the same settings and it still crashes the statsreader, you want a copy of it?

[Koepi]

Yes please, compress it with zip, upload it anywhere and send me the link as PM please :)

Thanks,
Koepi

[NiTroGen]

Well, while using StatsReader I got this really wonderful idea:

Since StatsReader is used right away after the 1st pass, why don't you add a tiny little button than will import the parametres used in 1st pass (filename & credits range) directly from registry?

Yes, I know, I am too lazy to select the file manually, then check the necessary boxes and type the credits range, ( :p ) but I still think it is a wonderful idea.

[rui]

Not that i want to brag myself or anything, but i already posted something like this a long time ago, but no one seemed interested at the time: http://forum.doom9.org/showthread.php?s=&threadid=32538&highlight=statsreader

Koepi, just asking out of curiosity:

Would it be possible to incorporate your statsreader into xvid itself?

I mean, when he/she choosed external 2nd pass, xvid would default to your statsreader, using the credits parameters that we already had configured in 1st pass, only asking the user to input, as always, the desired filesize and the other parameters that we can input when using the external 2nd pass.
Then it would scale the curve, not using the internal code, but the statsreader code, and create automatically the, for example, video2.stats in the folder we would indicate when configuring the 2nd pass external encode.

The advantage of this would be to automate the encoding process in the same way that it is automated when choosing internal encoding(using jobs in Vdub, for example).

[Koepi]

@rui,

go to page 10 of this thread where I wrote about that - again.

@NitroGen:

I'll do so when I'm feeling so bored that I want to fiddle with the registry stuff again ;)

@solarburn:
I can reproduce the crash, but didn't find the cause yet, I'm still investigating. Thanks for the file!

Regards,
Koepi

[NiTroGen]

@rui:

Sorry, man. I've never seen your post, but I'm glad there are more brilliant minds in this forum, sharing the same great ideas. :D

@Koepi:

I'm looking forward to you get bored. :p
(Btw, I do use StatsReader in every encoding I make and I always get terrific results. Keep up the good work.:cool: )

[rui]

NiTroGen, please don't get me wrong. :)

Between the two of us, you are the briliant mind. :D
After all you wroted a software to try and help xvid encoding (it wasn't the most concensual software, but you made something :p ).

I am just a guy who likes movies, likes to try new things, and likes to experiment a little and try to improve what is established. I know nothing about the theory behind all this. All i know is by reading here and experimenting later. ;)

[NiTroGen]

Originally posted by rui
I am just a guy who likes movies, likes to try new things, and likes to experiment a little and try to improve what is established. I know nothing about the theory behind all this. All i know is by reading here and experimenting later. ;)

Well, you've just described me. I'm not a brilliant mind. And my software is not brilliant too (Right, Koepi? ;) ) But as you said, experimenting and improving is all I do. Anyway, Rui, enough with this off-topic talking (and the compliments). Let's continue our conversation about StatsReader...

[ookzDVD]

@Koepi,


No quantizers >6, statsreader now seems to work better than internal CC!


Any plans to add your scaling algorithm to the 2-Pass-Int ?

[Koepi]

@OokzDVD: as I wrote to rui, look at page 10 of this thread - I stated my point already ;)

@NitroGen: I didn't say your programs are BS. I just got annoyed as it seemed that you ignored the info I gave you and repeatedly tried the impossible thing: curve aggression can't be determined by stats files, it has to be set by the user accordingly to her/his preferences... :) As "apology" I felt bored, just for you ;)

@All:

Find StatsReader 1.9 attached. I still couldn't find the error that causes a crash with some stats files, but we now have an "Auto Load" button which reads XviD's registry settings for loading the statsfile and setting the credits ranges. (I feel it's wrong to setup the codec for the 2nd pass right away so i left that alone).

Enjoy, and please tell me if it works for you! :)

Best regards,
Koepi

[Gannjunior]

Hi Koepi,
The external pass by statsreader is really precise.I've only a problem: how could I choose the exact quantizer to assign to start and end credits?In the last film I've obtained all quantizers(in credits) of 2 and 3.

TIA

ciao

[Koepi]

@Gannjunior:

You have to setup the XviD 1st pass with credits and a fixed quantizer to make that work. When choosing "2nd pass external" make sure that the value in those boxes for the quantizers is still the same as used during first pass - after that just enter the credits ranges (or hit auto load ;) ) in the StatsReader and it won't touch those frames.

I hope this helps,

Regards,
Koepi

[Gannjunior]

@Koepi

I've setted I-frames=20 and P-frames=31 in the 1st pass.However it doesn't follow them.the film is LOTR:

720x304 Desired size: 1.592.000
real size by statsreader 1.592.034 (perfect!)

I've observed debugview during the end of 2nd pass and I've seen that,to respect the size,it has been used lower quantizers in the credits,and not in the film.
The problem isn't a possible saturation of the film,'cause at that resolution,LOTR satures near 2,5-3Gb.
:confused:

thx,ciao

[Koepi]

Which XviD build are you using?

[Gannjunior]

04/09. I think is your last build.And i've used statsreader 1.7(i've seen that now you've done the 1.9 :) )

[Koepi]

Dunno what went wrong, please read to the massive amount of posts here to see how to setup the codec so it produces what you want to see - if you didn't make any errors in the xvid setup, and you entered everything in statsreader accordingly, this thing shouldn't happen. You know you have to activate credits even if you can't choose anything for the second pass?

Regards,
Koepi

[Defiler]

Wait.. I've been leaving the credits setting alone in the first pass, and specifying the range of frames in the "Ecredits" section of StatsReader. Is that incorrect?
I have read every post in this thread, but some of it seems contradictory to me. Please excuse my confusion.

[manono]

Hi Koepi-

I can confirm a problem with End Credits with Ver. 1.7 (haven't tried the most recent version yet). For the first couple of movies I thought I had been careless either in the first pass settings or with the StatsReader End Credit settings. So I doublechecked everything with my most recent movie. In the XviD first pass, end credits were set for 10, but Moonwalker's nice little program gave me this:

Quantizers Used For Credits :
--------------------------------
Quant 1 Used : 1 Times.
Quant 2 Used : 22 Times.
Quant 3 Used : 3 Times.
Quant 4 Used : 22 Times.
Quant 5 Used : 5733 Times.

[Koepi]

Hm, how to put this all into 1 post?

I'll try:
- 1st pass xvid setup: usual first pass, if you want to use credits, select "fixed quant credits" and set the quantizers like you prefer them.
- after that, fire up statsreader (use the most recent version please.) Hit auto load. enter the size you want that movie to get and hit "save as.." and enter a name. Remember that name.
- back in xvid, select 2nd pass external. setup as usual, but disable altCC and use regular CC with high/low 0.
- select the 2nd pass stats file you just generated in the bottom field called "2nd pass stats". You still must have the first pass stats file! Leave it untouched in the upper field.

Don't touch your credits settings, they're the same as in the first pass.

With most recent builds from CVS the stats reader + credits might work, but i didn't verify that yet.
With this procedure and my 4/9/02 build everything works fine for me.

Regards,
Koepi

[Gannjunior]

@Koepi

I'm following the discussion about statsreader from the begin.I've used the same settings you have told now.However I'll try to use your last statsreader 1.9 anf I'll cross my fingers;)
I'll post my results.

Regards,
Gannjunior

[Defiler]

Thank you for that summary.
Just one final clarification..
The "Credits" tab should look like this for both the first AND second pass, correct?
http://hellninjacommando.com/misc/xvid-credits.png

[iago]

@Defiler

Yes, exactly like that for the first pass (if you want to use a fixed quantizer of 20 for the credits of course ;)). But for the second pass external, quantizer boxes are deactivated.

regards,
iago

[Defiler]

Thank you. I've been doing it wrong. :)

[iago]

@Koepi

My first pass stats file crashes with statsreader 1.9 and with 1.8 too (I haven't tried with the previous releases). First pass size of the file is 2165 mb (as analyzed by xda.exe). Can it be a problem of filesize >2gb ?

best regards,
iago

[Koepi]

Nope, it's no filesize problem, those variables are all "big enough" to carry that information - and it would throw a propper exception if those variables cause trouble.

Maybe you should try the new XviD build, with the new internal scaling modifications (to name min[i|p]framesize limitation and modified overflow treatment) it should give an apropiate scaled curve when using linear sclaing (disabling altCC and setting regular curve compression to 0/0 - sorry, I've to repeat that so noone misunderstands what's going on ;) you can of course use altCC with the settings described some times in this thread, too).

I hope this helps,

regards,
Koepi

[NiTroGen]

Originally posted by Koepi
Thanks to Michael Niedermayer (I think he's the mpeg4 guru for ffmpeg?) I now have a formula to correctly determine the min frame size for IFrames, too, and my first tests seem very promising.Could you share the formulas about IFrames & PFrames?

And a question that I have for a long time: How is AVI overhead calculated? :confused:

(Yes, I'm trying to create a linear downscale program, too. I hope that you don't find it competitive and share some of your knowledge with us. ;) )

[Gazza]

Originally posted by Koepi

Find StatsReader 1.9 attached. I still couldn't find the error that causes a crash with some stats files, but we now have an "Auto Load" button which reads XviD's registry settings for loading the statsfile and setting the credits ranges. (I feel it's wrong to setup the codec for the 2nd pass right away so i left that alone).


Hi Koepi,
Tried the latest version of statreader but still crashes when loading the .stats file. It does this also when pressing the Auto Load button. However when pressing the load button I did notice a small message in the statsreader form that showed that the statsreader obtained the correct number of frames but then also said 0 bytes? So it seems to be analysing the .stats file in part but then stops retrieval - its not something silly like a divide by zero or a counter not set right? - just a thought :(

I ran the above tests with the latest xvid (XviD-22092002-1.exe).

[ookzDVD]

@Gazza,

I think you could use the 2-pass-int now, after Koepi applied
his scaling algo into the 2-pass-int.

[Gazza]

Thanks ookzDVD - I will give it a go.

[iago]

@Gazza,

You're right, I have the same problem here, Statsreader 1.9 still crashes, but only with "some" stats files, not with all of them in my experience. And with such problematic ones that the statsreader refuses ;), I follow the way that ookzDVD mentioned (2ndPass internal, in the "two pass" tab: setting high-low: 0-0, and disabling "alternative curve system", to linear-scale the curve).

regards,
iago

[Koepi]

Originally posted by NiTroGen
Could you share the formulas about IFrames & PFrames?

And a question that I have for a long time: How is AVI overhead calculated? :confused:

(Yes, I'm trying to create a linear downscale program, too. I hope that you don't find it competitive and share some of your knowledge with us. ;) )

Hm, want to render my proggi useless?

Well, here we go:

pframe minsize: (headersize + macroblocks + 8) / 8
iframe minsize: (headersize + 22 * macroblocks + 8) / 8

headersize is 80 bits in the pframes and 232 in iframes - this changes with cust. matrix for 1 frame but shouldn't be of bigger disturbance.

Avi frame overhead is 24bytes per frame.

(I still feel somewhat exploited now...)

Koepi

[MoonWalker]

I have found about the overhead this formula :

overhead=((23.086+((intra+inter)*0.19140625))/8)+5

I tested it and it works very well..But I think 24bytes per frames is much simpler :p

MoonWalker

[Koepi]

LOL! :)

Regards,
Koepi

[NiTroGen]

Hm, want to render my proggi useless?
Well, after your latest build, all linear curve downscaling proggies are useless. ;)

headersize is 80 bits in the pframes and 232 in iframes - this changes with cust. matrix for 1 frame but shouldn't be of bigger disturbance
What do you mean, by saying 80 and 232 bits? Headersize isn't the value of the dk_v parametre? Sorry, I didn't understand this.

Anyway, those formulas are needed for any kind of curve downscaling, linear or not, so it's good to be known to all people that want to experiment with downscaling. Maybe someone will find the ultimate formula. :rolleyes:

Thanks for the info, Koepi. I've already found the overhead formula in xvid sources.

Btw, since audio interleaving adds more overhead to the avi, I think it's a good idea to add an option to statsreader (if you continue its development) to calculate the total overhead (video + audio) by setting the audio interleaving options. So, when we'll have the final muxed avi, its size would be exactly the desired one. That's something that will never be implemented in the xvid codec, so our programs will still be usefull.

@MoonWalker:

Slow the cabbages! (MoonWalker should understand this expression). Check my formula about overhead, that I found using Gordian Knot and the method of the least squares:

Overhead = (34.418 * (MovieLength in minutes)) + 2.8673

:p

[Koepi]

That mux'ing overhead is not really an issue, use Nic's calculator for that - since I only (and I mean _only_) do OGMs that is a value which isn't "important" and error-prone...

And yes, of course you find those formulas in the xvid sources as foxer added them after my proposal.

Regards,
Koepi

[sungey]

hello ... i have a question

i have been using statsreader on nandub stats file .. it works fine and file size is very predictable ... but im wondering if
statsreader scales nandub stats the same way GKnot does ...
any idea ?

statsreader 0wns ^_^

[Koepi]

No, statsreader is different from GKnot.

GKnot can do far more "sophisticated" curve scaling for nandub as nandub uses "motion" and "luma" which don't get used by xvid (it's simply not necessary as XviD has no bugs/weaknesses there and is straight forward with those things, DivX3 has some features turned on by bitrate [and also switched off again if the bitrate range is left again...]). You'll get a plain linear scaled bitrate curve, and the gauge/bitrate buffer of nandub should cope well with it, too. But DivX3 has some bugs/features which really make it possible to gain more quality by getting (ab-)used :)

But thanks for the report that you successfully use it with DivX3/Nandub! Never thought someone would do that :)

Best regards
Koepi

[bond@doom9]

I once read somewhere that to achieve the same results as editing the 1st pass stats-file via statsreader i have to uncheck "use alt. curve" and then to set high/low curve compression to 0.

Now I made three test-files (using Koepis 041002 binary):
1) one with the procedure described above
2) one with an external statsreader19 edited stats-file
3) and one with the default settings

The results were that test-file 1 and 3 had exact the same filesize!
test-file 2 was a few bytes smaller!

So what does this mean? Does Koepis recent build automatically uses the statsreader features (so external editing of the 1st pass stats-file isnt needed anymore)? But why is then test-file 2 a little bit smaller?

:confused:

If this issue has been discussed before i am sorry to post it again but I didnt find any answers by using the search function

[Koepi]

Statsreader scales the curve a little bit different than XviD internally.
Therefore the file size difference (usually statsreader scaled curves reach the targeted file size within a smaller range, more on the spot).

But since filesize is not the important thing of that but "constant quality", you should take a look at the quantizer distribution spit out by XviD/VfW at the end of the 2nd pass.

Regards
Koepi