Compressibility

[apoptosis66]

I am working on backing up my entire dvd collection, going for transparency.

For you transparency junkies here are my options:

--crf 18 --ref 5 --bframes 3 --b-pyramid --weightb --b-rdo --bime --filter -2:-2 --analyse all --8x8dct --no-fast-pskip --subme 7 --me umh --trellis 1 --mixed-refs --progress --threads auto --no-psnr --thread-input --no-ssim --no-dct-decimate --vbv-maxrate 25000 --aq-strength 1.2 --aq-sensitivity 10

So I did the first couple of moves, and was seeing file sizes ~1/3 the original. I was very excited and started planning out my storage needs.

Then I encoded 12 Monkeys, the compressed file was ~2/3 the original. Throwing off all my estimates.

This probably doesn't surprise you guys but it caught me off guard.

So my queston is... What determines a movie's compressibility?

Here are some numbers:

MOVIE COMPRESSED/ORIGINAL(IN BYTES) = PERCENT
8 Mile 2023051264/6581579776 = 31%
Sin City 2588487680/7164538880 = 36%
12 Monkeys 3253305344/4909305856 = 66%

Thank you for any help,
Apoptosis

[R3Z]

Noise
Grain
Detail
Interlacing
High Motion Scenes
No. of scene changes
Color Pallete

With CRF18 you will likely go above the original movie size with some movies. Its just how it works, maintaining a minimum level of quality.

[PuzZLeR]

With CRF18 you will likely go above the original movie size with some movies.

Then why even bother? This intrigues me about some in this forum.

But yes. More movement, flashing colors, etc. = higher bitrate demands = higher file sizes.

Even though compressibility is not something that you can quantify easily without some algorithmic solution, a good way to understand this would be that an action thriller should end up a higher file size than a movie showing some stand-up comedian telling the jokes while just standing the whole film.

I personally find CRF 18 a bit low, and would prefer something like CRF 21. Nicer file sizes. But then again, who am I to tell you that?

If you're backing up DvDs, not files, then a really good, and fast, app is HandBrake. They seem to have the right x264 settings for great compressibility - great quality at lower bitrates, and fully QT compatible. I don't know how they do it, but they're great.

If you like CRF 18, their equivalent is 65%. CRF 21 would be more like 59% if you want to try that.

[apoptosis66]

R3Z

Thank you, I was thinking it was Grain because 12 Monkeys seems particularly grainy. I just hadn't seen any posts talking about it.

As for why CRF18, I tried handbrake and even at 90% I was seeing blocking. Blocking drives me nuts. So I started lurking in the transparency posts. Honestly I couldn't get rid of blocking until I started messing with the AQ settings.

[PuzZLeR]

I tried handbrake and even at 90% I was seeing blocking.

My friend, your eyes must be much better than mine.:confused:

[nm]

Display type and settings make a lot of difference in the visibility of blocking artifacts. Some LCDs are very demanding.

[ToS_Maverick]

My friend, your eyes must be much better than mine.:confused:

i think he means blocking caused by flat areas, and i think he read my post about AQ ;)

[R3Z]

Then why even bother? This intrigues me about some in this forum.

Depends how important the source is to you. Compression isnt really my first priority, picture quality is. I see no use archiving a peice of shit for the sake of a bit of hdd or dvd space :)


I personally find CRF 18 a bit low, and would prefer something like CRF 21. Nicer file sizes. But then again, who am I to tell you that?


Sometimes crf18 just isnt enough, it all depends on the source and how important it is to you. I usually stick with CRF19 and that seems to get me usually a 25% or more reduction in filesize whilst enabling all the great details and gains i have from processing.

[R3Z]

R3Z

Thank you, I was thinking it was Grain because 12 Monkeys seems particularly grainy. I just hadn't seen any posts talking about it.

As for why CRF18, I tried handbrake and even at 90% I was seeing blocking. Blocking drives me nuts. So I started lurking in the transparency posts. Honestly I couldn't get rid of blocking until I started messing with the AQ settings.

Try something like this;

My avs for a grainy movie with some blocking;


Load_Stdcall_plugin("c:\program files\avisynth 2.5\plugins\yadif.dll")
DGDecode_mpeg2source("D:\Nirvana Come As You Are [576p].d2v",cpu=4)
AssumeTFF().Yadif(mode=0,order=1)
BlockBuster(method="noise")

My X264 settings;

x264.exe --crf 19 --deadzone-inter 10 --deadzone-intra 5 --ref 2 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-1 --analyse p8x8,b8x8,i4x4,p4x4 --threads auto --thread-input --progress --no-dct-decimate --output "your output.mp4" "your.avs" --cqmfile "prestige.cqm"

I have attached the prestige.cqm which does a great job of keeping grain and keeping the file size in check. Worth a shot.

[burfadel]

Trellis 2 usually leads to a slightly smaller file size. Its different to Xvid trellis, straight from Akupenguin (author of x264) Trellis 2 is the most ideal setting even in CRF.

Your aq settings are pretty high, that can affect the compressibility a large amount. The setting you have set for aq strength of 1.2, however looking at the help options manuall from the command prompt it says the maximum aq strength is 1.1. Maybe try lowering this value as its probably throwing your settings out. Its meant to be adaptive, using more bandwidth for that area, so if you set it extrememly high and the sensitivity quite low(the low applies to more block), it may be counter productive. Its still in testing remember! Maybe try something like a strength of 0.8 and a threshold of 13 say? it will effect the file size, sometimes considerably. I know from experience!

The final size can be bigger than the initial size due to characteristics of the original encode. If the original encode was done in the format you are trying it would probably be much smaller. Display artifacts and the like are carried through to the new encode.

You may want to run a light noise filter over it. Don't know the best one, but it will often result in much smaller files without quality loss.

Try more b frames? they're adaptive!

Try --direct auto as --direct spatial is default if it isn't specified. Some say spatial is better, some say temporal is better. People comparing them usually specifically set it to either spatial or temporal! Setting it to auto the encoder will automatically choose which one it thinks is the best for the giver frames, so you end up with a mix of spatial and temporal frames. This is my preferred setting, I think it looks better than either 'spatial' or 'temporal' alone...

If there's lots of motion, try setting a high motion vector search range. The default is 16, try 24 or 28?. You have very high settings for the other options, why not for the vector search range?!

[Morte66]

IMO the inherent denoising effect of x264 encoding usually reveals blocking/banding that was dithered by noise on the DVD far more than x264 introduces blocks of its own. Since you already have pretty good anti-blocking settings for x264 (though you need them at -2-2), it might be worth trying deblocking/debanding on the DVDs. This can also do a lot for compressibility.

I've had the most luck with Deblock_QED for processing normal DVDs at sane speeds.

My automated backup script (generated by Robot4Rip) is something like...
#source
DGDecode_mpeg2source("D:\MeGUI\r408 Vegas Vegas 2\r408 Vegas Vegas 2.d2v",info=3)

#colour correct
ColorMatrix( d2v="D:\MeGUI\r408 Vegas Vegas 2\r408 Vegas Vegas 2.d2v", threads=0 )

#whatever you need to deinterlace, if anything

#deblock (multithreaded)
import("..\deblock_qed_mt2.avsi")
MT( "DeBlock_QED_mt2( quant1=18, quant2=24, aOff1=2, bOff1=2, aOff2=4, bOff2=4 )", threads=2, overlap=8, splitvertical=true )

#denoise (MV analysis block size of 16 seems just as good as 8 and way faster, at least for 720x576)
source = last
bv2 = source.MVAnalyse(isb = true, delta = 2, pel = 2, blksize=16, overlap=8, sharp=2, idx = 2, truemotion=true)
bv1 = source.MVAnalyse(isb = true, delta = 1, pel = 2, blksize=16, overlap=8, sharp=2, idx = 2, truemotion=true)
fv1 = source.MVAnalyse(isb = false, delta = 1, pel = 2, blksize=16, overlap=8, sharp=2, idx = 2, truemotion=true)
fv2 = source.MVAnalyse(isb = false, delta = 2, pel = 2, blksize=16, overlap=8, sharp=2, idx = 2, truemotion=true)
source.MVDegrain2( bv1, fv1, bv2, fv2, thSAD=600, idx=2 )

#deband, padding to get the outer edges done
addborders( 16, 16, 16, 16 )
gradfun2db()
crop( 16, 16, -16, -16)


Then I encode with:
--crf 18.0 --ref 3 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --trellis 1 --analyse all --8x8dct --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim

p.s. I've never personally had much joy with blockbuster and x264. It puts noise in, and x264 takes it out again. It does work nicely for me with Xvid, though. But give it a spin, lots of people like it.

p.p.s. The last time I heard, --direct-auto is recommended for two pass and --direct-spatial for crf.

{edit} p.p.p.s. When trying things out, I've often found it useful to run the avisynth decode/process script from the DVD into lossless huffyuv, then encode the huffyuv with x264 in a second job. That way you can tell what's down to x264 and what's down to the original/processing. MeGUI will do this automatically if you tick "Add a pre-rendering job".

[Terranigma]

p.p.s. The last time I heard, --direct-auto is recommended for two pass and --direct-spatial for crf.

So would you say it's sane if I were to use direct spatial for the first pass (using crf of course), then using direct auto for the second pass (ABR)?

[Morte66]

So would you say it's sane if I were to use direct spatial for the first pass (using crf of course), then using direct auto for the second pass (ABR)?

Don't know, I'm afraid -- you couldn't do 2-pass crf back when I last read about it. Maybe somebody else can help.

[Terranigma]

Don't know, I'm afraid -- you couldn't do 2-pass crf back when I last read about it. Maybe somebody else can help.

That's true, but you can do crf for the first pass by using the --crf xx.x commandline in 2pass - 1st pass. CRF, in itself, is not allowed to be used for a second pass. I'll just give it a go myself. :)

[ToS_Maverick]

i think he means these kind of blocks:
http://forum.doom9.org/showthread.php?p=996829#post996829
which you can almost eliminate with AQ

[SealTooGreat]

As for why CRF18, I tried handbrake and even at 90% I was seeing blocking. Blocking drives me nuts.
Try to turn off encoder's inloop or use command --nf insted of --filter -2:-2 and than tweak encoder if necessary.
And you may tray M4G MRM V2 (http://forum.doom9.org/showthread.php?p=887125) - this cqm is blocks killer.
cmd example: --cqmfile "M4G MRM V2.cfg"

[foxyshadis]

So would you say it's sane if I were to use direct spatial for the first pass (using crf of course), then using direct auto for the second pass (ABR)?

That's equivalent to using spatial for both passes, iirc, or maybe the equivalent of auto with one pass, I'd have to check the source. If you don't use auto the direct stats don't get filled in.

Since auto will normally choose spatial most of the time anyway, it's no great loss to use it.

[DarkZell666]

Try to turn off encoder's inloop or use command --nf insted of --filter -2:-2 and than tweak encoder if necessary.
Turn off deblocking ? When he wants to get rid of blocks ? Forgive me if I'm puzzled ... not that I don't believe you or anything, it just doesn't look right :)

[SealTooGreat]

Turn off deblocking ? When he wants to get rid of blocks ? Forgive me if I'm puzzled ... not that I don't believe you or anything, it just doesn't look right :)
XviD doesn't have inloop but you still can encode without having blocks.
And here we are talking about --crf 18, there's no need to use inloop for such low quantizer.
And it's very heartbreaking how x264 encode looks(with inloop encoded) while playing with decoder's inloop disabled. This option (encoder's inloop enabled)should be used only for low bitrate.
Let's wait what apoptosis66 has to say after testing without encoder's inloop. :)

[PuzZLeR]

Depends how important the source is to you. Compression isnt really my first priority, picture quality is. I see no use archiving a peice of shit for the sake of a bit of hdd or dvd space :)

It still doesn't explain why you are encoding using a lossy format like H.264 AND increasing the file size.

I agree in using higher bitrates for the higher quality. Too heck with HDD space if encoding from a source that is important to you. I too would be more inclined do the same thing.

But NEVER/RARELY if the file size will be larger.

You will ALWAYS lose quality with a lossy compression standard like H.264 going from MPEG-2 to H.264, no matter what quality level you choose.

If you like the smaller file size, and accept a certain quality loss, I understand the compromise. If you like the higher quality, I understand the higher file size. But the thing that intrigues me is converting to lower quality AND a higher file size. Sorry my friend, I'm confused here.:confused:

I can think of only 4 exceptions to encode to a lossy format AND accept a bigger file size:

1) Maybe one format is what you're going for regardless of the size of the files. Having a bunch of mixed clips like MPEG-1, MPEG-2, DivX, Xvid, H.264 can be annoying for some, so I understand. It's also best when playing on one system that may not have wide compatibility.

2) Maybe you're converting from a lower supported format to H.264. Same logic as the last reason, and would justify a higher file size and/or a minimal quality loss. However MPEG-2/DvD is very supported.

3) Maybe it's an audio issue. But I don't see it in this case either.

4) Maybe H.264 actually gives it a better "look" when resizing. This is the case for me with MPEG-1 material. I convert from 320x240 MPEG-1 to 720x480 H.264 anamorphic and the deblock filters rid many blocky artifacts of the older format and I increase the screen size to make it "look" better on a larger screen, even though in theory it's still lossy. Of course, the file size is almost 1.5-2x larger, but in this case it's worth it for me to get a slicker, larger and more modern look. I don't see this applying to your case with DvD and MPEG-2 to H.264. Sorry.

But, to each his/her own, right? :)

[chainring]

i think he means these kind of blocks:
http://forum.doom9.org/showthread.php?p=996829#post996829
which you can almost eliminate with AQ

I have to give props to you for the crf20 w/AQ setting in this post:
http://forum.doom9.org/showpost.php?p=996829&postcount=231

It gives, what I consider, good grain retention, good resistance to macro blocking, speed, and a very good size.

[apoptosis66]

Sorry it has taken me so long to respond, work all day...

R3Z: I don't think its your eyes I do think its my display. I have a dell 2407WFP which I purchase specificly to test the quailty of my encodings. I was definately seeing bad blocking on settings other people were saying had none. In particular the openning scene of 8 mile in the bathroom with lots of tiny details, and just about any black area in Sin City. I would not see these blocks on my rear projection HDTV.

ToS_Maverick: I have read just about everything you have posted on this forum, and your work is without a doubt the basis for my own command line.

Yes my CLI is probably overkill but I am more concerned with quality than, time to encode, or file size. I am not interested in finding the boundary where they are all optimum. Don't get me wrong, its a valid cause, but I just don't want to say years later, "Damn I wish this was better quality".

[foxyshadis]

Puzzler, an extension of your #4 is to apply processing that cannot be done in realtime - extensive cleaning, enlargement, sharpening, etc. I wouldn't ever go over the original size unless I'm restoring a badly damaged file, though.

[PuzZLeR]

Hey Foxy,

Can I get you to give me a bit more detail about what you're talking about (processing, etc.)? I'm really interested actually if anybody's got better tips going from blocky, small sized MPEG-1 to smoother, larger screened H.264, if it's actually feasible.

Or I can start a new thread if this is considered hijacking the thread.:)

At any rate, I consider alot of my MPEG-1 material "damaged" because alot of it, by nature of its older limitations, is blocky. H.264 gives it the "illusion" of improvement because I find the deblocking in the standard finds some sort of "love" between pixels, even with enlarging. It's amazing when comparing video standards that are about 15 years technologically apart! Wow!

Yes, it costs me a bigger file size going from 1150 kbps to about 2000 kbps, but it looks much nicer on a bigger screen and is one of the very few exceptions where I believe going from one lossy format to another lossy format, with a bigger file size, is worth it.

[R3Z]

It still doesn't explain why you are encoding using a lossy format like H.264 AND increasing the file size.

I agree in using higher bitrates for the higher quality. Too heck with HDD space if encoding from a source that is important to you. I too would be more inclined do the same thing.

But NEVER/RARELY if the file size will be larger.

You will ALWAYS lose quality with a lossy compression standard like H.264 going from MPEG-2 to H.264, no matter what quality level you choose.

If you like the smaller file size, and accept a certain quality loss, I understand the compromise. If you like the higher quality, I understand the higher file size. But the thing that intrigues me is converting to lower quality AND a higher file size. Sorry my friend, I'm confused here.:confused:

I can think of only 4 exceptions to encode to a lossy format AND accept a bigger file size:

1) Maybe one format is what you're going for regardless of the size of the files. Having a bunch of mixed clips like MPEG-1, MPEG-2, DivX, Xvid, H.264 can be annoying for some, so I understand. It's also best when playing on one system that may not have wide compatibility.

2) Maybe you're converting from a lower supported format to H.264. Same logic as the last reason, and would justify a higher file size and/or a minimal quality loss. However MPEG-2/DvD is very supported.

3) Maybe it's an audio issue. But I don't see it in this case either.

4) Maybe H.264 actually gives it a better "look" when resizing. This is the case for me with MPEG-1 material. I convert from 320x240 MPEG-1 to 720x480 H.264 anamorphic and the deblock filters rid many blocky artifacts of the older format and I increase the screen size to make it "look" better on a larger screen, even though in theory it's still lossy. Of course, the file size is almost 1.5-2x larger, but in this case it's worth it for me to get a slicker, larger and more modern look. I don't see this applying to your case with DvD and MPEG-2 to H.264. Sorry.

But, to each his/her own, right? :)

I speak about filesizes increasing because the processing i use makes them look better and increases the amount of detail. Think about extra sharpness, added grain, deblocking, de-interlacing. It all takes bitrate.

[PuzZLeR]

I speak about filesizes increasing because the processing i use makes them look better and increases the amount of detail. Think about extra sharpness, added grain, deblocking, de-interlacing. It all takes bitrate.

Well, if it actually "improves" the video, and I believe you believe it at least gives a better look, then fine.

It read last time like you were trying to fit four people in a car and to solve the problem you found a fifth person to squeeze in too. :p