Manual comp.test

[bullfrog]

Open the avs file in VDub, select 1-pass quality based and set the quantizer to 2 (set the other options as you want).

Selecting Fast Recompress in menu Video?

And I have another questions:

1. The final value is an approximation of the comp.test value of Gordian Knot? I Can use this value for the rule de 60%-80% recommended in doom9's guide? Which is the rule?

2. I tested your manual method to make compressibility test with DivX 5.02 and I have obtained values smaller than those of the test of compressibility of Gknot also with 5.02...

Thanks.

[N_F]

Selecting Fast Recompress in menu Video?
Correct.
1. The final value is an approximation of the comp.test value of Gordian Knot? I Can use this value for the rule de 60%-80% recommended in doom9's guide? Which is the rule?
Correct. The values usually recommended are 60-75%
2. I tested your manual method to make compressibility test with DivX 5.02 and I have obtained values smaller than those of the test of compressibility of Gknot also with 5.02...
The methods are not exactly equal, but they should give quite similar results. And as Jonny says in his post the method is not precise for B-frames. How big difference have you noticed?

[jonny]

Selecting Fast Recompress in menu Video?

Yes, not selecting this will only increase the encoding time.

1. The final value is an approximation of the comp.test value of Gordian Knot?

Yes

I Can use this value for the rule de 60%-80% recommended in doom9's guide? Which is the rule?

You must find your rule with tests :)

2. I tested your manual method to make compressibility test with DivX 5.02 and I have obtained values smaller than those of the test of compressibility of Gknot also with 5.02...

If you enable b-frames there are some problems, take a look at my post here:
http://forum.doom9.org/showthread.php?s=&threadid=29357


Edit
@N_F:
We had replyed at the same time :D

[bullfrog]

I have tested with the Jonny's test.comp 46 seconds of an animated cartoon film and these are the results:

Codec DivX 5.02 basic

Resize filter: Bilinear - Gknot test: 56.0 - Jonny Test: 46.0
Resize filter: Neutral Bicubic - Gknot test: 46.7 - Jonny Test: 38.8
Resize filter: Sharp Bicubic - Gknot test: 44.3 - Jonny Test: 36.9


I have later tested 13 minutes of a normal film and the results approach more. These are the results:

Resize filter: Bilinear - Gknot test: 44.7 - Jonny Test: 41.7
Resize filter: Neutral Bicubic - Gknot test: 37.6 - Jonny Test: 35.2

The results come near more...

Why take place these differences?

Thanks.

[jonny]

The lower is the length of the clip, the higher is the error.
(this is valid for the manual method and for GKnot too)

You should run a full encode at quality 100% to see what is the real comptest value.

Anyway the main difference is that GKnot discard 2 frames every 14 frames, this method discard all the keyframes (so in the general case: 1 frames every 14 frames).
Without b-frames the 2 methods are very similar (but with small clips, as you can see, the difference is more evident because the number of frames used is too low)

[bullfrog]

Thanks, Jonny

[bullfrog]

Somebody can explain as it is the process that Gknot makes in his test.comp?

Thanks.

[jonny]

I'll try to give you the "short answer", in the hope this satisfy you.

GKnot encode 14 frames every 280 frames (for a 5% comp.test).
The encode is done setting bitrate=10000 and using the 1st pass of a 2 pass encode.

After this it reads the log file of the first pass and it calculate the predicted size (this size is comparable to an encode at QB 100%)
(making this it discard some frames every 14 frames... because those frames are statistical bads)

After this it calculates the new Bits/(Pixel*Frame)

So the comp.test value will be:
100 * Bits/(Pixel*Frame) of the movie / Bits/(Pixel*Frame) of the 1st pass

DivX5Enc make "exactly" the same process (with the latest versions i'm using a different log file for the predicted size calculations)

DivX5Enc comp.test use this at the end:
100 * targetsize / predictedsize


Note:

100 * Bits/(Pixel*Frame) of the movie / Bits/(Pixel*Frame) of the 1st pass

is the same as doing:

100 * targetsize / predictedsize

If someone don't trust this, i can show this with some math.

If you want to me to expand some points or you don't understand something... i'm here :)

Cheers
jonny

[bullfrog]

Originally posted by jonny

After this it calculates the new Bits/(Pixel*Frame)

So the comp.test value will be:
100 * Bits/(Pixel*Frame) of the movie / Bits/(Pixel*Frame) of the 1st pass


1)
How it obtains this value?
How is obtained this value in the movie?
How is obtained this value in the 1st pass?

Originally posted by jonny
DivX5Enc make "exactly" the same process (with the latest versions i'm using a different log file for the predicted size calculations)


2) Why uses a different log file?

Originally posted by jonny
DivX5Enc comp.test use this at the end:
100 * targetsize / predictedsize

Note:

100 * Bits/(Pixel*Frame) of the movie / Bits/(Pixel*Frame) of the 1st pass

is the same as doing:

100 * targetsize / predictedsize


3) Which are the differences with the manual method that your you have described (http://forum.doom9.org/showthread.php?s=&threadid=44414) in this forum and these methods?

Thanks.
I love DivX 5 Enc.

[jonny]

How it obtains this value?
How is obtained this value in the movie?

I'll make you an example:

Resolution: 704 x 384
Fps: 25.000
Bitrate: 734 kBit/s = 734,000 Bit/s (in DivX 4 & 5)

So:

Bits/(Pixel*Frame) = 734,000 / (704 * 384 * 25.000) = 0.109



How is obtained this value in the 1st pass?

Using the predicted size, GKnot calculates the average bitrate of the 1st pass, than uses the same formula with this new bitrate.


Why uses a different log file?

I've added a new option to VDubMod.
This option, if enabled, generate a log file containing info about the frames encoded.
I prefer to use this because in this way i'm indipendent from the log file generate by the codec (this means that if the next version of divx change the log format, i don't care about this)


Which are the differences with the manual method that your you have described in this forum and these methods?

The differences are mainly related to the frames discarded every 14 frames.
The manual method discards all the keyframes.
Those methods discard the first frame (an artificial keyframe) and the last frame (sometimes an oversized b-frame) every 14 frames.
(those frames are discarded because are statistically bad)

No big differences without b-frames since discarding all the keyframes is similar to discard the first frame (an artificial keyframe) every 14 frames.
(note that in the manual method i take care of only the delta-frames sizes)

But there is no way to discard b-frames with the manual method, that's way the method have problems in this case.

DivX 5 Enc & GKnot methods are generally better than the manual method (but i think it's useful to know about the manual method anyway)


Cheers
jonny


PS: a lot of questions from you :), do you want to write a new GKnot :)?

[bullfrog]

I am write a guide (http://peque.org/maurus) of Gknot 0.27 ( codecs 5.02).

If it does not seem bad, I think write a chapter describing your DivX 5 Enc 1.11...:rolleyes:

Thanks for all.

[jonny]

I am write a guide of Gknot 0.27 ( codecs 5.02).

I see, nice.


If it does not seem bad, I think write a chapter describing your DivX 5 Enc 1.11...

Nice too :)

[bullfrog]

Originally posted by jonny
I'll make you an example:

Resolution: 704 x 384
Fps: 25.000
Bitrate: 734 kBit/s = 734,000 Bit/s (in DivX 4 & 5)

So:

Bits/(Pixel*Frame) = 734,000 / (704 * 384 * 25.000) = 0.109

Using the predicted size, GKnot calculates the average bitrate of the 1st pass, than uses the same formula with this new bitrate.



How calculates here the predicted size?
You can put an example to me?

Thanks.

[jonny]

It's simple.
Reading the log you have the size for each frame encoded.
Summing up the frame sizes (discarding some frames) and dividing this sum by the number of frames used, you have the average frame size.
Multiply the average frame size by the total number of frames and bingo!

[bullfrog]

I see your utility comptest v1.02...

It does not work with the codecs 5.03?

If you update it for 5.03, they can use those that they prefer to use VirtualdubMod for manual encoding...

Thanks

[jonny]

I have no interest on updating my CompTest utility.
I'd made this tool because the first days DivX 5.0 was out, GKnot 0.23 was no more able to make compressibility test.

For fun, if you want to see the full story:

My manual method:
http://forum.doom9.org/showthread.php?s=&threadid=19927

CompTest (automatically make what i describe in the previous thread):
http://forum.doom9.org/showthread.php?s=&threadid=22272

Those 2 posts are really old :D

After this i've started programming DivX5Enc.
Now things are evolved too much and updating CompTest will be only lost time (i've more interesting things to work on :))


EDIT: anyway you can use DivX5Enc to make a comp.test and then encode manually with VDub (using the same parameters), i see no problem in doing this.

[bullfrog]

Originally posted by jonny
The differences are mainly related to the frames discarded every 14 frames.
The manual method discards all the keyframes.
Those methods discard the first frame (an artificial keyframe) and the last frame (sometimes an oversized b-frame) every 14 frames.
(those frames are discarded because are statistically bad)


No big differences without b-frames since discarding all the keyframes is similar to discard the first frame (an artificial keyframe) every 14 frames.

(note that in the manual method i take care of only the delta-frames sizes)

But there is no way to discard b-frames with the manual method, that's way the method have problems in this case.

DivX 5 Enc & GKnot methods are generally better than the manual method (but i think it's useful to know about the manual method anyway)


Why no big differences since discarding all the keyframes (manual compt.test method) or discard the first frame (an artificial keyframe) every 14 frames? I don't understand this well...

Thanks

[jonny]

I'll make you an example.

I assume 1% and snipsize=14.
With this setting the SelectRangeEvery will be something like this:

SelectRangeEvery(1400, 14)

This means that 14 frames every 1400 will be encoded.
The frames encoded will look something like this:

ADDDDDDDDDDDDDADDDDDDDDDDDDDADDDDDDDDDDDDD

A=artificial keyframe
D=delta frame

In some particular cases you'll have something like this:

ADDDDKDDDDDDDDADDDDDDDDDDDDDADDDDDDDDDDDDD

K=normal keyframe

The manual method discard all the A and K frames.
The other method discard only A frames.

There is only a little amount of cases where you have a K frame inside the snip, this is why the 2 methods are really similar.

[bullfrog]

Thank you very much by the explanation.:)

[jonny]

Ops, i forgot another thing :rolleyes:

There is one other particular case.
Sometime the codec doesn't detect a scene change between 2 snips.

ADDDDDDDDDDDDDODDDDDDDDDDDDDADDDDDDDDDDDDD

O=oversized delta frame instead of the artificial keyframe

Those O frames don't get discarded by the manual method.
(anyway the O case is a small amount of frames and cause only a little deviation from the correct value)

[maurus]

I have created a spreadsheet in Excel for the manual comp.test by Jonny.

Here (http://peque.homeftp.org/~maurus/comptest.xls)

This in Spanish, but I believe that it is possible to be understood well... :)

I hope that it serves as aid.

Thx.

[jonny]

Nice work maurus :)
I'm going to update the sticky.

[maurus]

Originally posted by jonny
I'll make you an example.

I assume 1% and snipsize=14.
With this setting the SelectRangeEvery will be something like this:

SelectRangeEvery(1400, 14)

This means that 14 frames every 1400 will be encoded.
The frames encoded will look something like this:

ADDDDDDDDDDDDDADDDDDDDDDDDDDADDDDDDDDDDDDD

A=artificial keyframe
D=delta frame

In some particular cases you'll have something like this:

ADDDDKDDDDDDDDADDDDDDDDDDDDDADDDDDDDDDDDDD

K=normal keyframe

The manual method discard all the A and K frames.
The other method discard only A frames.

There is only a little amount of cases where you have a K frame inside the snip, this is why the 2 methods are really similar.

1) What is exactly a "artificial keyframe"?

2) Which are the differences between a normal keyframe and a artificial keyframe?

Thx.

[jonny]

SelectRangeEvery(1400, 14)

Using this istruction, we take the first 14 frames, than we jump to the next 14 frames skipping 1400-14 frames.
This jumps cause the codec to thing that there is a scene change, so a keyframe is inserted (artificial, because in your normal encode you'll never have this keyframe)
Keyframes take more space compared to normal delta frames, so, to make a good statistic you must discard those frames (if not, you'll obtain a totally crazy result).

I hope it's clear :)

[maurus]

Originally posted by jonny
Using this istruction, we take the first 14 frames, than we jump to the next 14 frames skipping 1400-14 frames.
This jumps cause the codec to thing that there is a scene change, so a keyframe is inserted (artificial, because in your normal encode you'll never have this keyframe)


It's clear, but who inserted artificial keyframe?

Thx.

[jonny]

The codec insert those keyframes

[maurus]

Originally posted by jonny
I'll make you an example:

Resolution: 704 x 384
Fps: 25.000
Bitrate: 734 kBit/s = 734,000 Bit/s (in DivX 4 & 5)

So:

Bits/(Pixel*Frame) = 734,000 / (704 * 384 * 25.000) = 0.109



I think Fps = 25, so:

Bits/(Pixel*Frame) = 734,000 / (704 * 384 * 25) = 0.109

[jonny]

:)

I use: . for separing integer with her fractional part and: , for separing thousands

So:
25.000 = 25
734,000 = 734000

I think this is the correct english notation. (@english people: is it correct?)
In italy for example we use the exact opposite.

[UGAthecat]

Originally posted by jonny
:)

I use: . for separing integer with her fractional part and: , for separing thousands

So:
25.000 = 25
734,000 = 734000

I think this is the correct english notation. (@english people: is it correct?)
In italy for example we use the exact opposite.
Yes, that is correct for english speakers. I think it's actually based on the way '.' and ',' are used in english grammar. We basically use them the same for 'reading' numbers as we do for reading a sentence.
Anyway, I have a question for Jonny that is on topic.
You mentioned that you 'fix' the artificial inflation of the comptest by leaving the keyframes out of the equation. Wouldn't another way of getting around the artifitial inflation be to make a comptest using a different set of variables for selectrangeevery. As an example:
Selectrangeevery(3000,150)
should still give a test using about 5% of the movie, but each segment of the movie in the test would be long enough to not introduce a lot of artificial keyframes.
I guess the question really is, what are your thoughts on using this method for manual comptest?

[jonny]

Hi

The main problem in what you say is this:

Consider snip size=14.
For a 5% we encode 14 frames every 280.
Suppose the source is 25 fps, this can be expressed as 14 frames every 11.2 seconds. 11.2 is really a small interval, with 5% the sample used have a really good distribution in all the movie.

Now snip size=150, so 150 frames every 3000 (for a 5%).
Suppose again the source at 25 fps, all can be expressed as 150 frames every 120 seconds or 6 seconds encoded every 2 minutes.
In 2 minutes all could happen, so the sample will be really less accurate and less distribuited in the movie.

I can't give you numbers about this, since i've never done this test.
But the rule is: the smaller is the snip size, the higher is the accurancy of the method (testing done in the past by Steady and Dividee have demostrated that the best snip size is 13, recently changed to 14 to better handle b-frames).

Best regards
jonny

[UGAthecat]

Thanks for the reply. I guess the method I mentioned would only be good for movies that dont really have any variance in the amount of action throughout the movie. If it were used on a movie with action scenes, like you said, the test jumping past 2 minutes could miss an entire high action scene.

[jonny]

Yes, what you say is 100% correct.

[Sephiros]

First of all, thanks Jonny for all for all this information it's really interesting :)

I understand why we have to discard the I frame every 14 frames on a snip size of 14 (because of the "scene changes") but I'm not sure why should we discard B frames too, because if activated for the final encoding, we will find them in the resulting movie anyway, am I right? So if we discard them on the comptest, isn't it bad for the % value as the predicted size will be higher than it would with B-frames? (as they are much more compressed, if I understood it well from the stiky about GOP)

So is it all B-frames taht we must remove or
Just some B frames, i.e B-frames right after the I frame?


Thank you,

Patrick

[jonny]

Hi :)

You must discard the last frame every 14 too (sometime this is an oversized b-frame, usually this happen when the codec doesn't detect scene change).
You can verify this manually parsing the DivX 5.0.2 log file.

The DivX 5.0.2 log have a double 0 frame, so, the final working method is to discard the first and second frames every 14 (there is a "shift" to handle the double 0 frame).

I can't help you with the 5.0.3 log file, since now i'm using a totally different log file for calculations (the log file is generated via VDubMod-the latest cvs release)

Cheers
jonny

(i'm happy to see someone read and apply what i write ;))

[Sephiros]

I must say that to write my program ( more info here: http://forum.doom9.org/showthread.php?s=&threadid=48617 ) I read almost half of your posts :).

Ok now I understand how to get the right values for a comptest out of the codec log, but is there a way to extrapolate and "guess" the average size of a D-frame to a given resolution with only one comptest?

I almost filled up almost a whole sheet of excel with data and graphics to try to find a rule (that looks like a cubic function...) but didn't find anything satisfactory. I'm far from being a math genius so if anyone knows anything about that, I'll be gald to know more :)


Another strange thing i found while trying to find a rule:
With B-frames and GMC, Divx5.03, the log of the first pass is EXACTLY the same in 720*304 than in 700*296 (NTSC 16:9). Strange, heh?


Pat

[jonny]

Ok now I understand how to get the right values for a comptest out of the codec log, but is there a way to extrapolate and "guess" the average size of a D-frame to a given resolution with only one comptest?

Never done testing in this direction (you should try to contact TheWEF for this).
I don't like this because conpressibility test have an error, and using another approximation to figure out the value at a given resolution will sum up too much error imo (i can be wrong here, but this is what i feel).

Another strange thing i found while trying to find a rule:
With B-frames and GMC, Divx5.03, the log of the first pass is EXACTLY the same in 720*304 than in 700*296 (NTSC 16:9). Strange, heh?

This is really strange!
Anyway i give you a suggestion, forgive the 5.0.3 log file and go to use the latest VDubMod (using /log /nowrite command line options).
In this way "vdenc.log" will be generated in the current directory, containing the size of each frame encoded (you gain divx log file indipendence in this way)
The only thing you must know about this method is that, with b-frames, you'll see P and B frames packed in 1 frame and after an empty frame (but this doesn't change the method, discarding the first and the last frame works perfectly ... without "shifting" this time)

Keep up your good work and let me know if you find a good solution for your first question :)

[Sephiros]

Hi Jonny!

thanks for the encouragements! :)

For the extrapolation, I won't work on it at the moment, too much bug fix to do :) as it is for the moment, my equation has an error of +- 30% on a variation of 10% in the resolution which kinda....sucks bad lol.

Beside, I'm not sure how to use the vdenc.log here is a segment I got:
...
1 34854
0 10071
0 8
0 11047
0 8
1 39362
0 27985
0 8
0 27435
0 8
0 26059
0 8
...

Why is there 2 "1" in 5 frames here? is it a oversized B-frames or what ( assuming the 1st "1" should be correct") ?
The size of the frames are displayed in bits?
What frame should be discraded? Each "1" and "0 8" I guess, and the first and last of the log?


Thank you for your patience! :)


Pat


----Edit March 17:
I forgot to mention: Snipsize is 14,80 in the given log

[jonny]

0 = Delta frame
1 = Keyframe
(there is no indication of b-frames, since it's not possible atm to detect this with vfw)

In your example you have 2 keyframes in 1 snip (sometime happens :))

10071, 11047, 27985 ... actually are 2 frames packed in 1 (one P frame and one B frame).
frames with size = 8 are empty frames.

To make all work you should discard frames at offset 0, 13, 14, 27 and so on (so first and last frame every 14 frames in the log)

[Sephiros]

Many Thanks Jonny!

I released a patch for my program that includes Comp test and other few things...

If you have some time to spare, tell me what you think of it!


Patrick

[jonny]

If you have some time to spare, tell me what you think of it!

Surely the comp.test will be really accurate :)
(i haven't a large amount of free time in this days :(, but i've downloaded yor app and if i find the time i'll make some experiments with it)
atm i'm working on a new app, the first beta stage had passed, soon i'll start a new beta stage, do you want to partecipate?