Manual compression test, very inacurate!

[kalehrl]

I'm trying to do a simple manual compression test using x264 and a MeGUI generated script to try to predict the file size of the entire encode. The problem is, it is very inaccurate. This is the script:
LoadPlugin("D:\Programs\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("D:\Dreambox\movie\projectx\Carl Sagan's Cosmos\DVDR_1\VIDEO_TS\VTS_01_1.d2v", cpu=4)
LoadPlugin("D:\Programs\MeGUI\tools\avisynth_plugin\TIVTC.dll")
tfm(order=1).tdecimate(hybrid=1)
crop(8, 0, -8, 0)
Spline36Resize(640,480) # Spline36 (Neutral)
fft3dgpu(sigma=1.8)
Nothing fancy and for compression test I added in the script 'selectrangeevery(300,15)' for 5% of the video.
The encode is 22,5MB which I then multiply by 20 to get the predicted final encode size - 450MB. However, when I encode the entire video, I get 363MB. This is way off for me! I know this type of compression test isn't accurate, but the difference is huge.
On another file I have predicted file size of 368MB but, in fact, the encode ended up being 286MB!
Am I wrong in my calculations or is there anything else I can do to get more accurate predictions?
Thank you

[detmek]

1. Try to use longer range, something like (600,30)
2. In your case multiply predicted files size with 0.8
368/450=0,817
286/368=0,777

[Groucho2004]

1. Try to use longer range, something like (600,30)
Agreed.
I did some experiments with this a while ago and it seems that the results become more accurate if the segments to be analysed are at least 2-3 seconds long. So, for 24p I would use something like (1440, 72).

[hello_hello]

I did a minor amount of experimenting a while ago and found a similar thing. File size was something like 15% less than the compression test indicated.

I think Groucho2004 is probably correct but I never got around to experimenting more myself. I put it down to gop size. The default maximum keyint is (I think) 250, which means gops can be up to around 10 seconds long. With (300,15) in a script it would mean encoding groups of 15 frames, and I'd imagine each group would be detected as a scene change with the possibility of scene changes within, so the compression test ends up consisting of much smaller gops on average than the final encode.

That's my theory anyway, but I did have another one regarding the difference between the compression test result and the final file size being a fairly consistent percentage when (300,15) is used, as per detmek's suggestion to multiply predicted files size with 0.8
If it always works out pretty close to that, then you could continue using (300,15). I guess you'd need to run a lot more compression tests to find out if it's consistent.

Edit: I assume Groucho2004 meant SelectRangeevery(1440,72)

[kalehrl]

Thank you for your thoughts on this guys.
I ran the same 2 compression tests but this time with SelectRangeEvery(1440,72) as Groucho2004 suggested and have much more accurate results.
The first clip ended up being 18,1MBx20=362MB which is almost exactly the size of the entire encoded video - 363MB.
The second clip ended up being 14,8MBx20=296MB which is pretty close to the size of the entire encoded video - 286MB.
I guess I'll stick to Groucho2004's formula 3xframerate.

[Groucho2004]

I assume Groucho2004 meant SelectRangeevery(1440,72)
Yes, sorry about that.