Getting the desired compressibility with AutoGK

[jsquare]

I know that AutoGK's compressibility test is not 100% accurate, but must of the time it's within 5% or less of the 1st pass result.

These are the step that I'm using to achieved the desired compressibility:

1. For movies less than 1h:45m I select 1CD(700MB), longer movies 2CDs(1400MB), and the desired Audio/Width/Subs settings.

2. Once the comp.test finishes I abort the process and write down test results.

3. Some movies require more space and other a lot less, so to get your target compressibility I use the following formula:

((DC/CT) x VideoSize) + OverHead + AudioSize = CustomSize

DC = Desired Compressibility
CT = Compressibility Test

4. Add a new job to AutoGK with the same settings but this time with custom size based on the previous result.

Example: the movie "Enemy at the Gates" is highly compressable so I went for 1CD encode with MP3 at 96Kb VBR and 640 width, the comp.test came out too high(80.43), to get the right size at 60 quality I tried my formula.

((60/80.43) x 603.09) + 7.19 + 89.72 = 546.81 or approx. 547MB achieved the 60.xx on the comp.test.

I just hope that len0x will consider this option to be included in future versions of AutoGK, but for now I have to do it the hard way.

[manono]

Why not just run a 1-pass target quality percentage 60% for a 640 width? Save yourself some trouble (and time).

[len0x]

yeah, manono's right - why would one want to have a given compressibility without a target size and still use two passes?

[jsquare]

Originally posted by manono
Why not just run a 1-pass target quality percentage 60% for a 640 width? Save yourself some trouble (and time).
I tried that method before with bad results, but for testing purpose I tried again with this material at Q60, the file came out 100Mb smaller, the picture looks really bad in low light scene and the mosquito effect is all over the place.

You guys are the experts/creators of AGK and should know better about bit distribution and higher quality of 2-Pass encoding vs 1-Pass VBR or Q based encoding.

My idea is really simple and can offer a balance between desired estimated quality vs size.

[manono]

Hi-

You ran a 2-pass using your method of getting 60% compressibility and a 1-pass quality 60% on the same movie (Enemy At The Gates?) and got a 100 MB file size difference? That's interesting.

I would guess that there would be some different settings used for the 2 (maybe sharper matrix and resizer for the 1-pass and a softer matrix and resizer for the 2-pass). And I understand what you're saying about the differences. If the film doesn't have a high enough bitrate, then dark areas will have the dark blocks moving all around, and sharp edges will have mosquito noise all around. That would be offset by the sharper picture that the 1-pass should have, if the 2 were about the same file size. I might have to test the differences myself sometime. However:

You guys are the experts/creators of AGK and should know better about bit distribution and higher quality of 2-Pass encoding vs CBR or Q based encoding.

Are you saying that you believe Quality based encoding is CBR? If so, then that's very much incorrect. What it does is to give a more-or-less even or constant quality to the whole movie. Or put another way, the average quant for the whole movie is pretty much the same for all parts of the movie. So, for complex scenes, the bitrate will jump in order to keep that average quant, and for static scenes the bitrate will diminish. This is also true for 2-pass encoding, but not nearly to the same degree. That is, where the 1-pass keeps the ave quant the same for all parts of the movie, for the 2-pass, the ave quant will rise slightly for complex scenes (giving them slightly poorer quality than the rest of the movie) and lower slightly for more static scenes (giving them better quality than the rest of the movie). All things being equal (filesize, resolution and other settings), I'll take a 1-pass any time.

But that still doesn't explain the great difference in the filesizes. Perhaps, just perhaps, the algo being used to calculate percentage .vs ave. quant for Quality based encoding worked for the older versions of XviD, but no longer works for these newer versions (post 1.0). But I don't really know. len0x may have to comment. But thanks for running the tests.

[jsquare]

I really meant 1-Pass VBR(not CBR), just fixed my comment.

After using D2SRoBa+DVD2SVCD for a long time I know the real benefits of Q based encoding, speed and quality are the main reason for this approach, but it seems like the same method is not working for XviD(Haven't tried Divx) and I think is due to the way encoders/codecs(CCE/MPEG2 vs XviD/MPEG4) manage the Q factor.

[len0x]

Originally posted by jsquare
but it seems like the same method is not working for XviD

Anything to support this statement? (like logs from both AutoGK's runs - Q-based vs. 2 passes)

[jsquare]

Since I deleted the logs/files for "Enemy at the Gates" I tried another test material, this time something smaller, chapters 2,29-32 of The Matrix(exactly 15:01 minutes)

720 width, audio 96Kb VBR, size 1/4 CD(175MB)

Here is a copy and paste of the log window, but if needed I'll include the AGK log files:

[11/4/2004 2:22:37 PM] AutoGK 1.73b
[11/4/2004 2:22:37 PM] OS: WinXP (5.1.2600).2
[11/4/2004 2:22:37 PM] Job started.
[11/4/2004 2:22:37 PM] Input dir: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS
[11/4/2004 2:22:37 PM] Output file: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS\THE_MATRIX(2PASS).avi
[11/4/2004 2:22:37 PM] Audio: English
[11/4/2004 2:22:37 PM] Subtitles: none
[11/4/2004 2:22:37 PM] Codec: XviD
[11/4/2004 2:22:37 PM] Target size: 175Mb
[11/4/2004 2:22:37 PM] Custom resolution settings: fixed width of 720 pixels
[11/4/2004 2:22:37 PM] Custom audio settings: VBR MP3 with average bitrate: 96Kbps
[11/4/2004 2:22:37 PM] Started encoding.
[11/4/2004 2:22:37 PM] Demuxing and indexing.
[11/4/2004 2:22:53 PM] Processing file: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS\VTS_02_1.VOB
[11/4/2004 2:22:53 PM] Source aspect ratio: 16:9
[11/4/2004 2:22:53 PM] Source resolution: 720x480
[11/4/2004 2:22:53 PM] Found NTSC source.
[11/4/2004 2:22:53 PM] Source seems to be pure FILM.
[11/4/2004 2:22:53 PM] Found 21617 frames
[11/4/2004 2:22:53 PM] Encoding audio.
[11/4/2004 2:25:07 PM] Audio size: 10,511,444 bytes (10.02 Mb)
[11/4/2004 2:25:07 PM] Overhead: 864,640 bytes (0.82 Mb)
[11/4/2004 2:25:07 PM] Video size: 172,124,716 bytes (164.15 Mb)
[11/4/2004 2:25:07 PM] Running compressibility test.
[11/4/2004 2:27:16 PM] Duration was: 2 minutes 8 seconds
[11/4/2004 2:27:16 PM] Speed was: 8.43 fps.
[11/4/2004 2:27:16 PM] Compressibility percentage is: 73.07
[11/4/2004 2:27:16 PM] Chosen resolution is: 720x304 ( AR: 2.37 )
[11/4/2004 2:27:16 PM] Predicted comptest value is: 73.07
[11/4/2004 2:27:16 PM] Running first pass.
*************************************
EXCEPTION: Job aborted
*************************************
[11/4/2004 2:27:23 PM] Job finished.
[11/4/2004 2:29:44 PM] AutoGK 1.73b
[11/4/2004 2:29:44 PM] OS: WinXP (5.1.2600).2
[11/4/2004 2:29:44 PM] Job started.
[11/4/2004 2:29:44 PM] Input dir: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS
[11/4/2004 2:29:44 PM] Output file: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS\THE_MATRIX(2PASS).avi
[11/4/2004 2:29:44 PM] Audio: English
[11/4/2004 2:29:44 PM] Subtitles: none
[11/4/2004 2:29:44 PM] Codec: XviD
[11/4/2004 2:29:44 PM] Target size: 146Mb
[11/4/2004 2:29:44 PM] Custom resolution settings: fixed width of 720 pixels
[11/4/2004 2:29:44 PM] Custom audio settings: VBR MP3 with average bitrate: 96Kbps
[11/4/2004 2:29:44 PM] Started encoding.
[11/4/2004 2:29:44 PM] Demuxing and indexing.
[11/4/2004 2:30:04 PM] Processing file: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS\VTS_02_1.VOB
[11/4/2004 2:30:04 PM] Source aspect ratio: 16:9
[11/4/2004 2:30:04 PM] Source resolution: 720x480
[11/4/2004 2:30:04 PM] Found NTSC source.
[11/4/2004 2:30:04 PM] Source seems to be pure FILM.
[11/4/2004 2:30:04 PM] Found 21617 frames
[11/4/2004 2:30:04 PM] Encoding audio.
[11/4/2004 2:32:17 PM] Audio size: 10,511,444 bytes (10.02 Mb)
[11/4/2004 2:32:17 PM] Overhead: 864,640 bytes (0.82 Mb)
[11/4/2004 2:32:17 PM] Video size: 141,716,012 bytes (135.15 Mb)
[11/4/2004 2:32:17 PM] Running compressibility test.
[11/4/2004 2:34:25 PM] Duration was: 2 minutes 8 seconds
[11/4/2004 2:34:25 PM] Speed was: 8.44 fps.
[11/4/2004 2:34:25 PM] Compressibility percentage is: 60.16
[11/4/2004 2:34:25 PM] Chosen resolution is: 720x304 ( AR: 2.37 )
[11/4/2004 2:34:25 PM] Predicted comptest value is: 60.16
[11/4/2004 2:34:25 PM] Running first pass.
[11/4/2004 2:46:16 PM] Duration was: 11 minutes 50 seconds
[11/4/2004 2:46:16 PM] Speed was: 30.41 fps.
[11/4/2004 2:46:16 PM] Expected quality of first pass size: 64.75
[11/4/2004 2:46:16 PM] Running second pass.
[11/4/2004 3:04:19 PM] Duration was: 18 minutes 2 seconds
[11/4/2004 3:04:19 PM] Speed was: 19.97 fps.
[11/4/2004 3:04:19 PM] Job finished.
[11/4/2004 3:04:19 PM] AutoGK 1.73b
[11/4/2004 3:04:19 PM] OS: WinXP (5.1.2600).2
[11/4/2004 3:04:19 PM] Job started.
[11/4/2004 3:04:19 PM] Input dir: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS
[11/4/2004 3:04:19 PM] Output file: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS\THE_MATRIX(Q60).avi
[11/4/2004 3:04:19 PM] Audio: English
[11/4/2004 3:04:19 PM] Subtitles: none
[11/4/2004 3:04:19 PM] Codec: XviD
[11/4/2004 3:04:19 PM] Target quality: 60%
[11/4/2004 3:04:19 PM] Custom resolution settings: fixed width of 720 pixels
[11/4/2004 3:04:19 PM] Custom audio settings: VBR MP3 with average bitrate: 96Kbps
[11/4/2004 3:04:19 PM] Started encoding.
[11/4/2004 3:04:19 PM] Demuxing and indexing.
[11/4/2004 3:04:39 PM] Processing file: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS\VTS_02_1.VOB
[11/4/2004 3:04:39 PM] Source aspect ratio: 16:9
[11/4/2004 3:04:39 PM] Source resolution: 720x480
[11/4/2004 3:04:39 PM] Found NTSC source.
[11/4/2004 3:04:39 PM] Source seems to be pure FILM.
[11/4/2004 3:04:39 PM] Found 21617 frames
[11/4/2004 3:04:39 PM] Encoding audio.
[11/4/2004 3:06:53 PM] Running single pass encoding.
[11/4/2004 3:27:10 PM] Duration was: 20 minutes 16 seconds
[11/4/2004 3:27:10 PM] Speed was: 17.77 fps.
[11/4/2004 3:27:10 PM] Job finished.
[11/4/2004 3:30:29 PM] AutoGK 1.73b
[11/4/2004 3:30:29 PM] OS: WinXP (5.1.2600).2
[11/4/2004 3:30:29 PM] Job started.
[11/4/2004 3:30:29 PM] Input dir: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS
[11/4/2004 3:30:29 PM] Output file: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS\THE_MATRIX(Q65).avi
[11/4/2004 3:30:29 PM] Audio: English
[11/4/2004 3:30:29 PM] Subtitles: none
[11/4/2004 3:30:29 PM] Codec: XviD
[11/4/2004 3:30:29 PM] Target quality: 65%
[11/4/2004 3:30:29 PM] Custom resolution settings: fixed width of 720 pixels
[11/4/2004 3:30:29 PM] Custom audio settings: VBR MP3 with average bitrate: 96Kbps
[11/4/2004 3:30:29 PM] Started encoding.
[11/4/2004 3:30:29 PM] Demuxing and indexing.
[11/4/2004 3:30:43 PM] Processing file: V:\00-RIPPED\THE_MATRIX_16X9LB_N_AMERICA\VIDEO_TS\VTS_02_1.VOB
[11/4/2004 3:30:43 PM] Source aspect ratio: 16:9
[11/4/2004 3:30:43 PM] Source resolution: 720x480
[11/4/2004 3:30:43 PM] Found NTSC source.
[11/4/2004 3:30:43 PM] Source seems to be pure FILM.
[11/4/2004 3:30:43 PM] Found 21617 frames
[11/4/2004 3:30:43 PM] Encoding audio.
[11/4/2004 3:32:57 PM] Running single pass encoding.
[11/4/2004 3:52:27 PM] Duration was: 19 minutes 30 seconds
[11/4/2004 3:52:27 PM] Speed was: 18.47 fps.
[11/4/2004 3:52:27 PM] Job finished.

As you can see I did 3 tests:

The first comp.test was 73.07 so I aborted the process and started a new one with a calculated size of 146MB, the final file size was 149,264KB, note that comp.test dropped to 60.16 but went up to Q=64.75 after 1st pass.

The second test is based on Q=60 and the file size was 133,636KB.

The third test is based on Q=65(approx 64.75) and the file size was 143,212KB.

[len0x]

So what does this prove? Nothing, because target size < 700Mb will always use soft matrix while quality based encoding will always use sharp matrix. 60 percent is pretty low for a clean material, so to your taste soft matrix may look better at low compressibility (its design for that). If you compare them say at 75% then you should see the benefit of sharper matrix. Size will always be different due to matrix difference.

[Darksoul71]

@jsquare:

After using D2SRoBa+DVD2SVCD for a long time I know the real benefits of Q based encoding, speed and quality are the main reason for this approach, but it seems like the same method is not working for XviD(Haven't tried Divx) and I think is due to the way encoders/codecs(CCE/MPEG2 vs XviD/MPEG4) manage the Q factor.

Now that is indeed an interesting point of view :o

Frankly: The way D2SRoBa works, will work with all codecs. Otherwise the codec would be worse designed. Why ?

Let me explain:
Quality is the most important thing for any AV codec out there. We all want crisp looking movies without any compression artefacts. As any movie scene requires a different amount of "bitrate" spent but we normally limit the amount of "Bits" available for the movie by defining a target size.

If you use a 2/n-pass mode, the codec will first analyzes the video determining which scenes of the movies need more "bits" and which scenes need less "bits". After this the codec will distribute the "bits" in the movie based on some algorithm with the primary aim to keep quality constant for the complete movie. You have often possibilites to influence this algorithm (e.g. flat versus complex slider in CCE). XVid offers you very detailed options by influencing the bitrate control.

Every one pass encoding mode with VBR (variable bitrate) which offer either constant quality or constant quantizers works the other way round. The encoder doesn´t care how many "bits" he has to spend for certain movie scenes and will allocate as much bits/s needed to keep quality constant.

You see ? 2/n-pass will use pretty much the same approach in order to judge what "good" quality is.

If you get a compressibility value of 70% for a certain resolution / filter combination / codec setting for a movie, you´ll be very likely able to hit your targeted filesize when using a Q-based one-pass mode using a quality of 70%.

Gordian Knot (the "uncle" of AutoGK) used the CQ-method as some sort of "Quality-o-meter": The movie was encoded at maximum constant quality to give you an estimation what quality to expect with your given configuration. The calculation behind this is quite simple:

1) Do a CQ encode at maximum quality (-> 100%) with a (more or less) representative 2-5% sample of the movie
2) Calculate the "real" filesize of the movie you would get at 100% CQ-mode (Size of AVI * sample size)
3) Calculate the ratio between targeted filesize and 100% CQ filesize

The resulting ration in % will be your "quality-o-meter". As the 100% file was encoded in CQ-mode it this ratio gave you a good guess what quality to expect from your encoding.

From my understanding AutoGK still uses the same approach to judge what parameters to use for the final encoding (e.g. soft versus sharp resize, no versus strong denoiser, etc).

Finally a CQ 1-pass mode will always offer equal or better quality for every movie compared to a 2/n-pass mode as the encoder "doesn´t have to care" about where "bits" have to be spent in order to keep quality. The encoder just encodes with constant quality which is what we want :)

Of couse this will not help when you´ve choosen inadequate settings which will lead to mediocre quality anyhow, e.g. if AutoGK reports 30% quality and you encode a 30% CQ your movie will look shitty and CQ will not help you anyhow.

Regarding your filesize difference: 100 MB for a 1CD rip is pretty much. I also would suspect that you´ve change something in your configuration.

-D$