Compressibility Check with Gordian Knot and DivX5

[jonny]

I have found a way to make a compressibility check with Gordian Knot and DivX5.


1 - Save .avs with compressibility check enabled and % you want

2 - Open the .avs in VirtualDub

3 - Select Video > Fast recompress

4 - Press Ctrl+C, select DivX Pro 5.0 Codec > Configure

5 - In Quick Config CLI copy:

-b21 780 -key 300 -log "C:\nnnxnnn.log" -dr 2,2,2000,10,20 -sc 50 -pq 5

6 - Edit "C:\nnnxnnn.log" with the resolution you have used in the .avs file (ex. "C:\704x384.log")

7 - In MPEG4 Tools check the options you want

8 - Press OK two times

9 - Press F7 and save the .avi

10 - Open the log file with a text editor (ex. "C:\704x384.log")

11 - Change the line 1 in:

##version 2

12 - Delete the lines 3 and 4:

ex.
bidirect 0
bidir_mult 2.000000

13 - The log file is now something like this:

##version 2
quality 5
Frame 0: intra 1, quant 2, texture 21262, motion 0, total 21627, complexity 53154
Frame 1: intra 0, quant 2, texture 0, motion 0, total 1031, complexity 0
Frame 2: intra 0, quant 2, texture 0, motion 0, total 1031, complexity 0
Frame 3: intra 0, quant 2, texture 0, motion 0, total 1031, complexity 0
Frame 4: intra 0, quant 2, texture 0, motion 0, total 1031, complexity 0
Frame 5: intra 0, quant 2, texture 0, motion 0, total 1031, complexity 0
Frame 6: intra 0, quant 2, texture 0, motion 0, total 1031, complexity 0
Frame 7: intra 0, quant 2, texture 0, motion 0, total 1031, complexity 0
Frame 8: intra 0, quant 2, texture 0, motion 0, total 1031, complexity 0
Frame 9: intra 0, quant 2, texture 0, motion 0, total 1031, complexity 0
...
...
...

14 - Save the log file

15 - Open Gordian Knot, press Load (in Compressibility Test) and select the log file



This method seems to work...

[rui]

O.k, it worked.

When I tried to load the .log file I had created, Gnot asked me something about that the numbers of pixels per frame wasn’t stored in a divx4 log file, and if I was sure that I wanted to load that log file.
I answered yes, and there I was, a divx5 comp check.
But I still have some doubts about how to interpret the percentages that gnot shows us.
The values that were good for divx4 maybe are a little high for divx5. When, in divx4 the low barrier was 35-40%, in divx5 maybe one can survive with lower values.

[jonny]

If you don't activate the new feature of divx5 in the comp. check, sure, you can go lower than 40%-45%
If you activate GMC and biframes the compressibility increase... but i dunno if the same range of divx4 is ok...

numbers of pixels per frame wasn’t stored in a divx4 log file...

6 - Edit "C:\nnnxnnn.log" with the resolution you have used in the .avs file (ex. "C:\704x384.log")

have you make this?

[rui]

No, i just named it divx.log.
Does it makes any diference??

By it worked, i just choosed to continue after that error, and gnot loaded the log file just fine.

[jonny]

For the tests i made it seems GKnot use the final part of the file name to calculate total amount of pixel, so you can rename it for example "dasdas_dasdas_640x480.log"
To check if the file is loaded correctly go to the encoder tab
Are you using 0.23b?

[rui]

Yes, i am using gnot 0.23 beta.

I will try renaming the log file, but like i said above, without renaming it it worked. Maybe if i rename it i will not have that error?

But either way, what % do you think are significative in divx5?

[Beave]

I tried an action movie that had a CompCheck of 55% with only biframe turned on. I must say it was to bad for my taste!
I am doing the encode again with 65%. Let's see how this will look like.

[jonny]

Yes, with renaming you have 0 errors

I'm going to make a movie at 53,3% with GMC and biframes...

@Beave: can you tell me at 55% what avg. quantizer have you reached in "analyse.log"?

[Beave]

I think the analyse.log is overwritten by a later encode. Sorry

[jonny]

@53,3%, with gmc & bframes, the quality is good for me... avg quantizer in analyse.log is 4,27...

With DivX 4.12 i usually stay between 55% - 65% ... and avg quantizer in analyse.log is around 4... I think the same % of comp. test work with DivX 5

[rui]

Ok, thanks for the info.

If i have some time, i will do some encoding tonight, use the comp check divx5 method.
But, with divx4, i usually was satisfied with lower values than that. I i would get about 40-50%, it was enough for me.

[jonny]

Ok :)
Try and tell here what you think ;)

[rui]

Humm.. This comp check %'s maybe aren't totally right. I have done a comp check using this method with the movie The Green Mile.
The movie has plus 3 hours, and i target for 2 cd's. But even using a 640x352 resolution, the comp check only lows to 80%.

That means that if i would like it, i could make the encoding in full resolution that the comp check would even then only go to about 60%. I know that this movie is slow motion, but this seems much too extreme. Maybe divx5 best values are ~70% and up?

[ookzDVD]

I just wait the next GKnot.

[jonny]

I'm trying a movie with ~70% and bframes only to see how it looks...
(for me seems divx4 and divx5 comp. check are really near... we should compare a clip in divx4-divx5 with the same comp.check % to see if it really works...)

[jonny]

New movie test: Ghosts of mars ~1h 34m (aspect ratio: ~2.35)

Output resolution: 704x416 (i play it correcting the aspect ratio)
Bitrate: 1031 (i target for ~800MB)

Compressibility test with bframes only and sharp bicubic: 69,3%

Results:
Avg quantizer in analyse.log: ~3,6
THE QUALITY IS GREAT

bframes really increase compressibility... look at the avg quantizer in analyse.log!!!

[EvilFoo]

I'm confused. At what point is the 'nnnxxnnn.log' created? I'm just told to use this config line and then go edit nnnxxnnn.log, but at what point in time is that actually made (I don't have it).

Also, you guys talk about using compression checks for values like 65 and 85%, I'm guessing this is the value you get AFTER you use that log file. What value is used (1-15%) before saving the .avs file?

Thanks.

[jonny]

>I'm confused. At what point is the 'nnnxxnnn.log' created? I'm just told to use this config line and then go edit nnnxxnnn.log, but at what point in time is that actually made (I don't have it).

It is created with VirtualDub
(9 - Press F7 and save the .avi)
It is the log file of a first pass with min-max quantizer set to 2 (this is the way compressibility check work)
Before run this first pass you must change the "nnnxnnn.log" with something like 704x384.log (you must use the resolution you have selected for the compressibility test)


>Also, you guys talk about using compression checks for values like 65 and 85%, I'm guessing this is the value you get AFTER you use that log file. What value is used (1-15%) before saving the .avs file?

In my opinion values between 55%-65% are good (this cover both DivX4 and DivX5). Activating the new features like BFrames and GMC (those features increase compressibility A LOT) it's easy to reach this values for compressibility test (of course this is not a general rule and depends by the length & type of movie).
Usually i run tests with 1%... when the compressibility value is ok for me, i remake the test with 5% (usually the values of comp. test with 1% and 5% are really close)


Hope this help :D

[EvilFoo]

You are telling me step 9 is save the avi which does the first pass. If I'm reading your new message correctly the log file is created at this point. If that's true, why does step 6 tell me to edit the nnnxnnn.log into the resolution I am using? Comparing the tutorial to what you just said makes no sense. Since I'm only on step 6, how is the logfile created? I'm not on step 9 yet, unless of course I had already made the 0kb log file beforehand? [frustration kicking in]

[b0b0b0b]

I have a movie whose compressibility is 0.960 with b-frames and no filtering (other than bicubic resize).

[EvilFoo]

Another n3wb question: How does this check affect the overall video? It's JUST a check though right?

[rui]

It's only to help you choose a correct resolution, one that will look good with the number of cd's which you want to use.

[EvilFoo]

So instead of making a block of a ~7 hr encode, I have to cut in the middle of it (means waiting around 3 hrs) for the first pass to finish so I can look at a percentage? :\

[jonny]

>You are telling me step 9 is save the avi which does the first pass. If I'm reading your new message correctly the log file is created at this point. If that's true, why does step 6 tell me to edit the nnnxnnn.log into the resolution I am using? Comparing the tutorial to what you just said makes no sense. Since I'm only on step 6, how is the logfile created? I'm not on step 9 yet, unless of course I had already made the 0kb log file beforehand? [frustration kicking in]

@step6 you edit the DivX5 options... OK??? In this step you choose the file name for the log file...

@step9 you save the avi... at this point the log file is created...

>So instead of making a block of a ~7 hr encode, I have to cut in the middle of it (means waiting around 3 hrs) for the first pass to finish so I can look at a percentage? :\

Have you ever used comp. test for DivX4???
I think you don't understand what is the comp check...

In the comp. check GKnot analise the log file of a first pass with quantizer min-max = 2
if you choose 1% of comp. check, 1% of the film will be processed
Read the guide for DivX4, i think this help...

For DivX5 comp. check does not work automatically... so you must make different step to have this...

:S

[jonny]

I've made another test... summary of the 3 tests:


Movie 1:
Compressibility test: 53.3%
Total frames: 182161 Non-zero I & P decisions: 93139 Avg quantizer: 4.286401

Movie 2:
Compressibility test: 69.3%
Total frames: 141171 Non-zero I & P decisions: 72061 Avg quantizer: 3.597155

Movie 3:
Compressibility test: 82.4%
Total frames: 144896 Non-zero I & P decisions: 75665 Avg quantizer: 2.990858

[theReal]

How about making a comp.-check with Divx 4.12 and just aiming for lower numbers for the final divx 5 encode? Has anybody tried this?

I have done a few tests with a DV avi-file recently, and I could just safely lower the bitrate by about 25% in divx5 with b-frames compared to Divx 4.12.
Has anybody tried this for different DVDs? Are b-frame encodes always about 25% smaller than Divx4.12 encodes? Then quality checks with Gknot would be rather easy (given one has installed both codecs with a little registry tweaking).

[jonny]

I've written an util that automate the comp. check for DivX 5... but is in Perl+TK... so you must have Perl to run it... if someone is interested i can post the Perl source...

[EvilFoo]

jonny: I've read the divx4 and 5 guides regarding to compression, but neither actually talk about the compressibility tests. The only thing discussed is the bits/(pixel*frame) and which would be the best value.

I took a 1 minute clip from DVD2AVI and demuxed the audio to get ac3. Using GKnot I converted ac3->wav->mp3 at 128 kb/s w/ 44.8Hz. The resolution being 544x304 and 0.195 as the bits/(pixel*frame) value I save the .avs file with sharp bicubic, smart deinterlacing, and 5% check. Successfully following your steps, I load the log file into Gknot and I get 60.1% of 0.325. 165376 pixels = 646 blocks. So what? How does this help me with the second pass?

theReal: No haven't tried this yet, sounds interesting though.

[EvilFoo]

jonny: Is there any way for you to change that Perl+TK into an executable of some sort?

[jonny]

>How does this help me with the second pass?

The 60,1% tells you how the movie is compressible with the resolution you select... IT'S ONLY A TEST
If for example you obtain 20%... you must decrease the resolution or go for another cd because 20% is too low...
I usually stay between 55%-65% and the final quality is ok for me

You must do a full 1° pass and 2° pass to encode the movie

>Is there any way for you to change that Perl+TK into an executable of some sort?

There is an util called perl2exe... but it's not free :(

[yosemite]

well, Gknot tells me (after these steps):

Average Quant: 4.076 Compressibility Check failed (use the "now" button)

Whats wrong?

[Beave]

You have to set the min and max quant to 2 when doing the Comp Check.

[yosemite]

isnt that done with this:

5 - In Quick Config CLI copy:

-b21 780 -key 300 -log "C:\nnnxnnn.log" -dr 2,2,2000,10,20 -sc 50 -pq 5


?

[theReal]

When you use this config, you do set the min and max quant to 2 by the part that says: "(...) -dr 2,2 (...)"

btw. the bitrate setting (here: "780") doesn't matter for the comp.check.
I tried setting it to 1 and it still produced the same values.


I also did some comparison of the divx5 comp. check and the divx3.11 nandub comp.check.
The movie was "Hannibal", using 5% of all frames at a resolution of 608x320, soft bicubic resizing, aiming for a one-cd rip at a bitrate of 627.

Divx 3.11: 51%
Divx 5 without b-frames: 45%
Divx 5 with b-frames: 57%

(I didn't write down exact numbers, but I memorized them accurately enough to show the differences, I think)

Then I encoded the end credits seperately, that increased the divx 5 with b-frames value to 59%. The movie looks very very good, I'm really proud of it ;)

[yosemite]

[QUOTE]Originally posted by theReal
[B]When you use this config, you do set the min and max quant to 2 by the part that says: "(...) -dr 2,2 (...)"

ok, so I have set the quantizers.

But why "Average Quant: 4.076 Compressibility Check failed (use the "now" button)" ?

[jonny]

Can you tell me exactly what you have done? (in detailed steps)

[theReal]

I'd say if you follow the guide step by step it should work. It worked for me on the first try, no hidden traps, nothing.
But of course, computers... you never know what they do...

[yosemite]

Originally posted by jonny
Can you tell me exactly what you have done? (in detailed steps)

well, I did exactly that, what u have written above, exactly that

[jggimi]

Originally posted by EvilFoo
I took a 1 minute clip from DVD2AVI...Evil: The compressability check in GKnot defaults to 5% of the entire content, by sampling frames throughout. It adds the following to the AviSynth script:SelectRangeEvery(260,13).

If you select a 5% compressability check and then save the .avs file, you should see this in the script.

[jonny]

@yosemite

It seems you don't load the correct log file (or the file you create is not with mix-max quant=2) but i can't tell more...

[jonny]

New tests:

Total frames: 182161 Non-zero I & P decisions: 93139 Avg quantizer: 4.286401
Compressibility test: 53.3%

Total frames: 190146 Non-zero I & P decisions: 97103 Avg quantizer: 3.934464
Compressibility test: 61.6%

Total frames: 141171 Non-zero I & P decisions: 72061 Avg quantizer: 3.597155
Compressibility test: 69.3%



In the next tests i got undersized files:

Total frames: 139010 Non-zero I & P decisions: 71199 Avg quantizer: 2.918067
Compressibility test: 81.5%

Total frames: 144896 Non-zero I & P decisions: 75665 Avg quantizer: 2.990858
Compressibility test: 82.4%

Total frames: 132617 Non-zero I & P decisions: 67883 Avg quantizer: 2.912970
Compressibility test: 89.2%

NOTE: seems DivX 5 don't go lower than 2.9 (avg quantizer)!!!



All the tests are made with full movies

[b0b0b0b]

Did you have bframes enabled for your compressibility tests?

They probably don't fix quantizers at 2, but at 4, which explains the avg quant of 3.

[rui]

So, would you agree that the max value we should accept for divx5 is about 70-75%? Above that we could have undersized files?

I am finishing a encode from the movie The Haunting. I had a comp check with this movie of around 88/89%, and probably i will get a undersized file.
The other day i made a encode of this very same movie, using a resolution that gave me a comp check of 69%, and i got a perfect file size.

[jonny]

@rui:
Yes i think the max is 75%-80%, after you'll get undersized

@b0b0b0b:
I use b-frames... but i think this is not for b-frames... i don't have 2-4-2-4-2-4 in analyse.log

[theReal]

I think there is a multiplier of 2 for b-frames, so you won't get 2-4-2-4... in the analyse.log, but 2-2-2-2-2..., only the b-frames' quantizers are automatically multiplied with 2.

However I'm not sure, can someone confirm this?

[jonny]

Another test:

Compressibility test: 87.4%
Avg quantizer: 3.038250

This time WITHOUT BIFRAMES

File still undersized (by 20MB)

[rui]

I just did an encode of the movie The Replacements, which gave me, using your comp check utility, a 29% value. It looked bad :(
So, i have still to make more tests to find out what percentage could be considered the bottom line for divx5.
I used b-frames and GMC. The file size was spot 700MB. This only confirms my theory that divx5, as long as we choose our resolution wisely, can reach our desired file size.

I can't see why doom9, in his codecs test, got a oversized file for Matrix.
I already made some encodings of The Replacements, with diferent resolutions (from 512xXXX to 640xXXX), which i consider to be much more dificult to encode for 1 cd than Matrix, and i never got an oversized file. Using the 512 resolution i got the mencioned 29%. When using 640, i got below 20%'s, and the movie looked really bad, but the filesize continued to be 700MB.

[jonny]

You haven't got the point

I'm saying that the limit for avg quantizer (in analyse.log) is around 3.
So if you have an undersized file, the frames are not all encoded at quantizer 2...
I dunno why this happen... someone have same experience or want to comment this?

[theReal]

Jonny, was that an encode at 100% quality, or was it at a certain bitrate setting?

If the av. quantizer was ~3 without b-frames at 100% quality mode, this would be even more weird. But I agree it is still weird for a 2-pass encode if the file is undersized and not all frames are encoded at a quantizer of 2 (without b-frames).

I must be very lucky - I had not one divx 5 encode up to now that was more than 1MB off...

[jonny]

All my tests are made in 2-pass mode.

As you can see, when i go after 80% of compressibility test:

1 - I get undersized file
2 - Avg quantizer in analyse.log is around 3

No matter if i use or not b-frames



Edit:

2-pass, b-frames only
Compressibility test: 81.5%
Avg quantizer in analyse.log: 2.918067
final file undersized


2-pass, b-frames only
Compressibility test: 82.4%
Avg quantizer in analyse.log: 2.990858
final file undersized


2-pass, b-frames only
Compressibility test: 89.2%
Avg quantizer in analyse.log: 2.912970
final file undersized


2-pass, no b-frames
Compressibility test: 87.4%
Avg quantizer: 3.038250
final file undersized