Would 10% compressiblity check be better than 5% ?

[GreatKent]

5% compressibility check is already able to create file very close to the desired file size, I just wonder if 10% check would be any better?
Thanks!

[Iznogoud]

It would be more accurate conserning the compressibility/finding right resolution, but it has nothing to do with the filesize AFAIK.

[JohnMK]

After repeated tests I've found the difference in results between 5% and 10% check is negligible. I would recommend using a 5% check, or if you're in a rush, 3% is fine too for a ballpark answer.

[GreatKent]

oh ic, thanks a lot!

[OvERaCiD23]

3-5% is good. one important note, when comparing comptests of different filters, be sure to use the same % for the comptest. if not, results can vary 5-10% and may lead to a bad decision for filters.

[bkam]

I was recently pondering this, and I found a post by jonny to his program DivX 5 Enc which is useful for running multiple compressibility tests... I'm sort of a newbie but I wanted to see the effects of Psychovisuals, B-Frames, convolution3d, etc. on compressibility. With this program you can just write the avs scripts (I get mine from gordianknot then modify them) and queue them up then wait, and it spits out a list of compressibility check #s.

jonny's site (http://jonny.leffe.dnsalias.com/)

Also interesting (and maybe more pertinent to this post) is the tests that Jonny has done. In almost all cases, 2% compressibility checks or 4% come closer than 5%, so now I always run 2%s... Check that out too.

jonny's tests (http://jonny.leffe.dnsalias.com/whywork.htm)

Thanks jonny!

[Iznogoud]

After reading jonny's test I wonder... how can 2% be more accurate than 10% if 100% is perfect? In scale of 1-100 when 100 is best 2 is better then 10 :confused:?
Answer: By chance.
Try with multiple movies and results will be very different. (but anyways wheter you use 2% or 10% and get 39% or 41% means nothing/very little. :) )

[bkam]

You're right, it's only from one movie. But I guess it shows that you're likely to have a 2-3% error from the correct value whether you do 2% or 10%, and one is five times faster (usually I want the compressibility tests to go quickly so I can compare them, or else I want to start encoding the actual movie as soon as possible so I can leave my computer alone for a couple of hours and still be getting use out of it :) )

[Iznogoud]

Indeed. Iz no good to hurry, but sure it's better to run even 1% test then not do it at all.

[jonny]

Remember that it's a statistical method.

In my tests the bigger error was at 5%, this means that the sample extracted at 5% was the worst sample for this particular movie.

There is no way to predict the best & the worst sample (you can only do a 100% comp. test to be sure - no statistics at 100%, simply a full first-pass run)

But you can compare the advantage of one option/filter against another if you fix a %.

I use 1% for adjusting the settings, than i do my last comp.test at 5%-10%.

Anyway the max error in all my tests was 3.14%, i think is far from being bad.


jonny

[JohnMK]

So many steps. ;) I think I'll just use one 4% comp test and be happy with that.

[jonny]

I'm telling you what i do, not that you must do the same i do :)

[JohnMK]

By the way Jonny, your programs are very appreciated. I use comp test on an almost daily basis. Suppose you might do an XviD version soon?

[jonny]

Thank you

An "XVid Enc" will be probably a killer app :D, anyway i want to first complete the DivX 5 video part.
Specifications for DivX 5 Enc v1.09 are defined, i'm working on it.
After this, the next version will be probably 2.00 (i want to implement an original & new idea for this version)

After this i have 2 choice: audio for DivX or "XVid Enc"... we will see...

Regards
jonny

[scheps]

i did the test at 5% then my % was 88 with the bar all the way to the right!! (i.e. couldnt ring the % down anymore) so what can i do with that? or the test has some significant limitations?

[bkam]

Assuming you're using DivX 5, if you get that type of compressibility test it is pretty likely that you will end up with an undersized file. Read the sticky (http://forum.doom9.org/showthread.php?s=&threadid=24584) on this topic in the DivX 5 forum for ways to make the movie less compressible (i.e. higher quality).

[scheps]

oh i am using divx 3 not five so would what you said still apply?

[bkam]

I'm sorry but I don't know.

[jonny]

Teorically yes... but i've started encoding with DivX 4, so i'm not 100% sure, any DivX 3 freak around :) ?
(you can try to post this on the DivX3/4 forum)

[scheps]

thanx m8 :D

[jonny]

100% in compression test represents the highest possible quality with b-frames, if you enable b-frames.
It's a codec (divx5) fault if at >75%-80% the result is undersized (sometimes it's better to make a qbased encode instead of a 2-pass enc).

[TheWEF]

jonny knows, because his advice made sure that gknots compcheck works with divx5-b-frames. had to alter the way of reading the log-file for that. thanx again.

wef.

[scheps]

so are you guys saying divx 5 with gordian knot is a better combo then with divx 3? and aim for something > 75-80%? to ensure . a file that is not undersized as well as maximising quality?

i've never tried 5 been using 3 all along till now :p

[TheWEF]

Forum Rule 12)
Do not ask "what's best" because this question cannot be answered objectively. Each and everyone has their own view about what's best in a certain area. The best is what works best for you!

encode a movie two times with the same settings, just different codec, compare and make your own decision. and read Doom9s MPEG-4 Codec shoot-out 2002 (http://www.doom9.org/codec3-1.htm).

[hakko504]

Originally posted by TheWEF
jonny knows, because his advice made sure that gknots compcheck works with divx5-b-frames. had to alter the way of reading the log-file for that. thanx again.

wef. I see.
Is this because you use the bitrate for the first pass? I always assumed that since you got undersized frames in the 75-100% area this was because the comp checks didn't account for b-frames. This is the expected behaviour if you run a compcheck without enabling b-frames and then encodes with b-frames enabled.

[jonny]

@hakko504:

>Is this because you use the bitrate for the first pass?

Can you explain this better?


>I always assumed that since you got undersized frames in the 75-100% area this was because the comp checks didn't account for b-frames.

Unfortunately it's a codec fault. In 2-pass mode at higher comptest values it's very difficult to fill the target size (lowering the max quant to 3-4 can help, but i've not tested this intensively).
Some my friends often switch to 1pass quality based, it seems the only way to force the codec to break this limit.

Anyway compress. test always try to get the maximum size for the encode @ quality 100%, taking into account all the parameters you set (b-frames, qpel, gmc, pe...).
Even keyframe interval have little influence on compressibility.

You may look at the tests i've done here:
http://jonny.leffe.dnsalias.com/whywork.htm




@wef:

Thanks to you for inventing THE method :) !!!
(the "/n" method i'm testing seems to work very well, take a note that i want something like this in the GKnot successor :) )

[hakko504]

@jonny

In both XviD (I think) and DivX3.11 you set the first pass bitrate to something impossibly high and just tell encoder to use DRF 2/2. In DivX 5.02 to get correct result you should set bitrate to the target bitrate for the second pass. I've always wondered why.

[jonny]

I think we should ask this to some divx or xvid developer.

[N_F]

Originally posted by jonny
Some my friends often switch to 1pass quality based, it seems the only way to force the codec to break this limit.


Every now and then I find my encodings to be undersized by 5-10 MBs. The solution usually is to find by how many percent it's undersized and then increase the bitrate by this percent.

I guess this is a way around this codec fault you mention (couldn't some kind of 3-pass encoding do a better work distributing the quantizisers (spelling?) than the current 2-pass?).

[jonny]

5-10MB is an acceptable error (when you use comp.test values >80% you usually get 50-100MB less if i remember correctly...)

Simply add 5MB in your target size in GKnot & all will go ok (example: 705MB instead of 700MB).

DivX 4 have big problems... long time ago i've made a 3-pass utility for DivX 4 (it uses the method you describe), just for fun you can find it here:

http://www.angelfire.com/wizard/divx4pass3/
(it have problems running on Win2000/XP, i've abandoned it... 1-2 days after releasing it, DivX 5 was came out :()

Anyway DivX 5 don't need this kind of thing.

[N_F]

Originally posted by jonny
5-10MB is an acceptable error (when you use comp.test values >80% you usually get 50-100MB less if i remember correctly...)


I can't remember that I ever was satisfied if the file came out at 690 MB instead of 701. I rather do another second pass and adjust the bitrate to reach a size of 701. Of course the difference in quality is too small to be noticed, but I like things perfect and I have the time.

I've gotten the impression that some people are experiencing a few megabytes off the desired size at almost every encoding.

I've only experienced on a few rare occasions.

[jonny]

I usually get 3-4MB less than the target size, but since i often use 2 cd, this help to get a better cut.

Why don't you reencode the audio to fill the left space? It whould be a lot faster.

[N_F]

Well, reaching a very specific size with ogg is much more of a challenge than doing it with DivX (though I guess you could up it one step to fill space, but it'd still be hard to reach a very specific size.

Anyway, if the filesize is only off a few MB I usually reencode the credits with quant ~12 instead of 20

[N_F]

When you encode to 2 CDs, do you first create a ~1400 size file and then split it or do you split it in DVD2AVI? From what you said I guess you use the first method, have you ever considered splitting already in DVD2AVI and pretty much handle it as two different movies?

It seems pretty much impossible otherwise to reach a specific size and make the cut at a scene change.

[manono]

Hi-

have you ever considered splitting already in DVD2AVI

Personally, I don't think that's such a good idea. You don't know in advance the bit rate the 2 halves will need to achieve equal quality. That is, if you find a good place to split in DVD2AVI, somewhere near the halfway point, then in those cases where there's a whole lot more action in the second half of the movie, the first half may have way better quality than the second half. Admittedly it's a difficult problem, but I usually make my 2 CD rips at 1390-1395 MB and then look for a decent place to split.

But I'm with you N_F in being dissatisfied with a 1 CD rip winding up at 690-695 MB. In those cases I'll always redo the second pass to make it come out to 700-701 MB.

[jonny]

Yes i 100% agree with manono (... 99%, usually i play with audio to get perfect size :))

[bkam]

Originally posted by jonny
@wef:

Thanks to you for inventing THE method :) !!!
(the "/n" method i'm testing seems to work very well, take a note that i want something like this in the GKnot successor :) ) I'm sorry but I'm curious. I assume by the method you're speaking about compressibility checks (maybe I'm wrong). But what is the /n you speak of? I would search for it but it's under 3 chars :). Thank you.

[jonny]

It's a nice surprise for the next version :) (yes it have something to do with the comp.test)
Please, just wait a bit ;)

[bkam]

Ahh, now I understand why you were so cryptic :cool:. I am very excited now :D.

[JohnMK]

Whoah. :) I'm really looking forward to your new release jonny. I use all your programs in combination with GordianKnot (as a script-writer it's excellent).

[jonny]

Beta testing will start next week, so soon you'll have the next ver.
(i have a lot of nice ideas for the future... but my time is so limited :(, i cry when i think old times, when i went to university ... so much free time! :))

[N_F]

Originally posted by manono
You don't know in advance the bit rate the 2 halves will need to achieve equal quality. That is, if you find a good place to split in DVD2AVI, somewhere near the halfway point, then in those cases where there's a whole lot more action in the second half of the movie, the first half may have way better quality than the second half.

That's a very good point. But I wonder how common it really is for a movie to have very different comp. values in the first and second half. Just for fun, the next few movies I'll encode I'll split up in the middle and encode with quant 2 to get a feeling how common it is.

[bkam]

Originally posted by N_F
Just for fun, the next few movies I'll encode I'll split up in the middle and encode with quant 2 to get a feeling how common it is. Well, undoubtedly it will vary a lot from movie to movie, but that is a pretty sweet idea. :) I'll have to think of a movie that changes a whole lot in the middle to try it.

[manono]

Hi N_F-

You wouldn't even have to do that. You could do it the way you suggested earlier (split in DVD2AVI and encode each half to 1 CD) and just compare the average quants between the 2 halves as given at the top of the analyze.log file that's created. My guess is that there will often be a 10% or more difference between the two. Just depends on how great of a difference you can live with. Your way does allow you to completely fill the CDs, and mine doesn't. And your way does allow you to find a good scene change at which to cut, and sometimes with my way, I have to cut at a keyframe, and not at a real scene change.

Doing it your way, you could run compress tests on the 2 halves, and if the percentages aren't close enough, you could create new .d2vs with a slightly different split point.

[bkam]

Although the discussion in this part is mostly about encoding at approximately the same quality, if I had a long enough movie (3 hours min) whose ending was very climactic, I might try something like encoding the first two hours to one CD and the last hour to another CD. Could be a fun experiment, and what I find is that after a few minutes of watching a movie, I forget the quality unless it is really terrible.

This way, you learn to put up with the quality, and then 2/3 through you all of the sudden get a big quality increase :D. This isn't a very practical idea but I may try it thanks to the posts here ;).

[N_F]

Originally posted by manono
You wouldn't even have to do that. You could do it the way you suggested earlier (split in DVD2AVI and encode each half to 1 CD) and just compare the average quants between the 2 halves as given at the top of the analyze.log file that's created.
Yes, of course... I should have thought of that. It'll save me some time.

I'll report results in a week or so when I've done it with a few movies.

[N_F]

Well, yesterday before I got you suggestion Manono I was running a test according to the way I described earlier. I'm sorry to say you seem to be right.

Rules of Engagement:

1st half: ~1150 MB
2nd half: ~700 MB

This is a huge difference so it seems I'll follow your method from here on.


On a slightly different matter: Rules of Engagement seems to be one of those movies that are a bit below acceptable for 1 CD and serious overkill for 2 CDs.

[manono]

Ouch-big difference. But I was expecting the second part to be larger. I don't know of many movies where the first half has more action (haven't done Rules Of Engagement). However, there's another factor I thought of but didn't mention. It's really hard to find the halfway point using DVD2AVI as there's no indication of the frame numbers the way there is for the .d2v in GKnot.

If you want to pursue your theory some more, you might try and split by chapters during the SmartRipper-DVDDecrypter phase. Another way would be to find your split point in GKnot and use the Trim function in the .avs. Then you run into audio problems, though. When done you might have to rejoin the video, add the audio, and split again. Either that or split the audio also.

...a bit below acceptable for 1 CD and serious overkill for 2 CDs.

I've run into that problem recently also. I decided it's how much I like the movie that determines whether I go with 1 or 2 CDs. If it's really good I'll go for 2 CDs with higher resolutions, no filters, high quality audio (the AC3 if it makes good use of the surrounds). If I consider it only a so-so movie, I'll go for 1 CD, lower the resolution, put on the filters to help with the compressibility and use lower quality audio.

[N_F]

I actually did this test by opening a single avs file - generated by Gordian Knot - in VirtualDub and marked at the exact half point, so I've already done what you suggest.

Since I only use theses files for tests I don't care about the audio. The sizes you see are only video.


I haven't seen Rules of Engagement yet, but it seems to be a war/court movie, where a big part of the first half contains the war picture ad the second part the court part, which would explain why the first part is bigger.

I'll try to run a few more tests to see if this movie is unique or not.

[jonny]

@N_F:

One suggestion for the tests.

You can use DivX5Enc to quick made a lot of tests in this way:


(no dvd2avi cut is assumed)
Create 2 avisynth files, one for each part using the trim function:

First avs:

... your script here
trim(0, 99999)

Second avs:

... your script here
trim(100000, 0)

(this example suppose that the half point is at frame 100000)


Than open the first script with DivX5Enc, set the parameters you want (bitrate you set have no importance for the test, for the rest: better to keep the defaults settings, disabling b-frames, to speed up the tests)
Run a compressibility test at 1%
DivX5Enc will return the predicted size at quality 100% (refer to the docs here (last image): http://jonny.leffe.dnsalias.com/comptest.htm)
Than simply load the second script, don't change any setting, and run another compressibility test at 1%.

Now you have 2 nice predicted size for the 2 parts (in 10 minutes i think :)).
You should expect an error of about 3%-4% in the prediction, but considering the time you spend to make the test, i think it's accetable :)
(you can of course use higher % making the compressibility test (example 10%), this will give you lower errors)

[N_F]

Results from the tests I've run (I decided to use quant 2 sizes even though I got other suggestions from Manono and Jonny), movies was divided in the middle, end credits cut out:

Rules of Engagement

1st - 1 165 966 62%
2nd - 704 542 38%


The Birds

1st - 1 540 850 52%
2nd - 1 429 486 48%


The Expert

1st - 627 420 50%
2nd - 638 662 50%


X-Men

1st - 614 450 56%
2nd - 491 694 44%


Night of the Living Dead

1st - 576 236 46%
2nd - 625 100 54%


A Night at the Roxbury

1st - 615 250 50%
2nd - 621 142 50%


As can be expected it varies greatly from movie to movie. Though 4 of the 6 movies I did this with showed quite even numbers, while Rules of Engagement showed a huge difference.


Anyway, I've gone over to "your" way of doing 2 CD rips. It's much more comfortable since among other things you don't have to divide the subtitles and deal with asynchrous streams.

[manono]

Hi-

Good work N_F. Remind me not to encode The Birds. You've got almost a 3 GB first pass there. Must be real hard to compress to 2 CDs.

[jonny]

Nice work!
It's curious to see that for some movies the difference is very little

[N_F]

Originally posted by manono
Remind me not to encode The Birds. You've got almost a 3 GB first pass there. Must be real hard to compress to 2 CDs.

Yeah, though I used C3D for my encoding since it's a bit noisy. Since it won't affect the test values I removed it for the test to increase the speed. I was quite happy with the results, though I guess it could have been better. I refuse to go over 2 CDs when it's not even breaking the 2h barrier.


Night of the Living Dead is the only B&W movie I've encoded so far that actually compresses very well. Though it should be noted it was the 30th anniversary edition, it's quite possible they cleaned it up.