Problem with achieving quality encode with reasonable file size

[Octo-puss]

I've got this older DVD with live concert which is giving me hell.
I just can't find a way to encode it so the image quality is as close to the source without the resulting file being ridiculously high.
Could anyone help me with some settings?

The source is interlaced (I hate to deal with that stuff), for which I use QTGMC (which, compared to yadif is like night and day difference, so I will be reencoding everything I have).
I am using MeGui and x264, where I don't touch the advanced settings (I don't understand one bit of those), and just select quality-based encoding, tuning film, and very slow preset. I experimented with various CRF values from 23 all the way to 15.
I picked two random frames where I could easily tell detail changes, and reencoded them several times with different CRF values.

Example:
source (http://www.mediafire.com/view/jzdq9gmn3mqqr7b/Mother+Earth+QTGMC.png#)-CRF21 (http://www.mediafire.com/view/ikhybq93ob26lim/Mother+Earth+CRF21+QTGMC.png#)-CRF15 (http://www.mediafire.com/view/hyx85a911pa3tgc/Mother+Earth+CRF15+QTGMC.png#)

The detail simply is not there, and going to CRF15 produces 1,6GB file, which is ridiculous (size of the source is very slightly over 4GB).
Any tips appreciated.

[Sharc]

Assessing the quality of an encode based on single frames is doubtful, but anyway:
- You re-compress an already compressed (DVD mpeg2) source
- CRF15 gives you a reduction from 4GB to 1.6GB as you write, which is a significant reduction in filesize of 60% still preserving details well. It is far from a ridiculous file size.
- CRF21 shows some somoothing which is to be expected. What is your filesize at CRF21?

You may want to add a slight sharpening like sharpen(0.2) in the CRF21 case and see if you like the result better, or you could try changing the AQ-strength, but I wouldn't expect wonders.

[Octo-puss]

I know, I just wanted somewhere to start from.
How would I change the AQ strength setting? I don't really know what it does, and x264 settings guide is not really helpful (if you don't know anything about it).

[Sharc]

AQ = Adaptive Quantizer. I am not really experienced with MeGUI tool but I think you can set it in the x264 configuration settings, Tab Rate Control. Try
Mode: Variance AQ
Strength: 0.5 ..... 2.0 (experiment what you like best)

The CLI at the bottom will show --aq-strength x.x

AQ controls how bits are allocated: It takes bits from flat areas and throws these bits to more complex areas of the picture. So it is a trade off: Steal here .... spend there.

[Octo-puss]

That's what I read as well. But I don't know what number shall I put there.

[Sharc]

So then?
Make two tests: one with --aq-strength 0.5, and one with aq-strength 2.0 and see what the effect is and what YOU like better. I have neither your original nor your eyes ...

If you want to archive your source it is probably best to go for a 1:1 copy ... :D

[nm]

The source is interlaced (I hate to deal with that stuff), for which I use QTGMC

How about encoding as interlaced? Might save some bits, although you then obviously need to deinterlace during playback with something worse than QTGMC.

Having encoded DVB recordings with QTGMC processing, I'd say that 1.6 GB is quite normal for an hour of 576p50 or 480p60 video if you want high quality.

[Octo-puss]

I encoded it with CRF19 now and it came out at very surprising 817MB. That's weird. But it leaves some room for experimenting.
So anyway, what are some other x264 parameters that can affect detail in a positive way (after looking at some examples of sharpen function, I decided against using it)?

[Sharc]

What exactly is surprising and weird?
You had 1.6GB@CRF15 and 817MB@CRF19 which is well possible. What you get at the end depends strongly on the character of the source.
Higher CRF reduces the file size and softens the picture because there are less bits available for details.
How to improve details? Lower the CRF = increase bitrate => increase filesize. Details are bitrate hungry.

[Octo-puss]

Is it possible to get different results each time I encode something with exactly the same settings?
I tried with CRF18, and it looks almost identical to the source, BUT when I encoded just that particular frame, the quality was noticeably lower.

[Groucho2004]

Is it possible to get different results each time I encode something with exactly the same settings?
No (assuming you're not changing anything besides the encoder settings and you're not using "--non-deterministic"). Even with "--non-deterministic", you won't see a difference.

BUT when I encoded just that particular frame, the quality was noticeably lower.
Judging the quality of a video encoder by encoding one frame is probably the least intelligent thing you could do.

[Octo-puss]

Judging the quality of a video encoder by encoding one frame is probably the least intelligent thing you could do.
I covered that before.
I simply compared that particular frame in 1) an encode of just that frame and 2) full encode, and the frame looked very different, so that's why I asked.

[Sharc]

I covered that before.
I simply compared that particular frame in 1) an encode of just that frame and 2) full encode, and the frame looked very different, so that's why I asked.
The encoding result of P or B frames (which constitute the vast majority of the frames) depends on a sequence of pictures (taken from past and future) which change from frame to frame in a real movie. Means when you encode only a single frame this "sequence" is static i.e. there is no change between subsequent frames. Therefore you cannot draw any conclusions on quality or on encoder settings for the movie based on the encoding of a single static picture.

[Octo-puss]

Oh I see! Thanks for explanation.

[fumoffu]

I find that on not perfect sources those settings work well:
--preset slow --ref 6 --deblock -1:-1 --subme 9 --trellis 0 --psy-rd 0.3:0 --aq-strength 1.3
optionally also --bframes 5 for better compression
the idea is that disabling trellis and lowering psy to 0.3 (or 0.2) reduces bitrate wasted on artifacts, higher aq 1.3 (or even 1.4) prevents loss of detail on gradients and blurriness from low psy, and might not be necessary on all sources.
recommended crf 18 - 20

ykhm, works great on porn ;)

[Ghitulescu]

I simply compared that particular frame in 1) an encode of just that frame and 2) full encode, and the frame looked very different, so that's why I asked.
Because lossy compression are unpredictable, in particular in algorithms where the previous (and sometimes the next) frame plays a significant role.

[benwaggoner]

Did you turn on "MPEG-2 deblocking?" in MeGUI. That can help quite a lot with over-compressed MPEG-2 sources like this.

If the content is at all noisy, the x264 "noise reduction" parameter can be helpful. Start with 500 to see if it makes a difference.

[Octo-puss]

It really looks like encoding much more than one frame produces better results. I didn't realize that before, even though it makes perfect sense (but I don't know much about video encoding to begin with, so...). I seem to be getting fairly satisfactory results now even without advanced x264 settings. Sometimes even with lower CRF value.

I will experiment with the deblocking feature when I move onto the next DVD in line. Never really payed any attention to it before, possibly because I thought it could compromise picture quality in a way.

[therube]

I find that on not perfect sources those settings work well:
--preset slow --ref 6 --deblock -1:-1 --subme 9 --trellis 0 --psy-rd 0.3:0 --aq-strength 1.3
optionally also --bframes 5 for better compression
the idea is that disabling trellis and lowering psy to 0.3 (or 0.2) reduces bitrate wasted on artifacts, higher aq 1.3 (or even 1.4) prevents loss of detail on gradients and blurriness from low psy, and might not be necessary on all sources.
That's an interesting selection of settings.
I've messed around with it just a bit, & depending on what you're looking to accomplish, can do a decent job. I'm seeing only slight quality decrease with a decent file size drop. Haven't seen any benefit to --beframes. Interesting.

[movmasty]

Did you turn on "MPEG-2 deblocking?" in MeGUI. That can help quite a lot with over-compressed MPEG-2 sources like this.

If the content is at all noisy, the x264 "noise reduction" parameter can be helpful. Start with 500 to see if it makes a difference.Both options are blurry.....
in particular the x264 "noise reduction" gives artifacts over 200-250

I encoded it with CRF19 now and it came out at very surprising 817MB. That's weird. But it leaves some room for experimenting.
So anyway, what are some other x264 parameters that can affect detail in a positive way (after looking at some examples of sharpen function, I decided against using it)?
In my experience a CRF of 21 is very low and should give perfect results.
What about Key frames? H264 likes Key frames
Max kf not more than fps x 5
Min kf not more than 5
The lover the better

And since the CRF settings are averages and not deterministic
to boost kf give an I/P ratio at least of 1.8
KF are always a few and the size impact wont be noticeable,
and more kf reduce the size of P B ones.

In this way the most of any frame will be composed of light compressed kf blocks.

Set also the max/min quantizer, a side effect of CRF is that if you lover the quantizer, higher quantizers will be used too,
Try min 17/18 max 27/28

Reduce the curve compression, to do you have to put an higer value,
i use 0,81.

A characteristic of CFR is that uses different quantizers with same settings for different movies, or different content inside a movie.

[Asmodian]

Personally I wouldn't mess with min/max quantizer settings or I/P ratio, just dropping CRF a small amount usually adds more quality for less file size.

[osgZach]

Just make sure when reducing Keyframes you don't go overboard and destroy reasonable seek-ability of your clip

Although this has piqued my interest. Care to post a sample of the source clip, (1GB?) I'm bored and need something to play with.

[movmasty]

Personally I wouldn't mess with min/max quantizer settings or I/P ratio, just dropping CRF a small amount usually adds more quality for less file size.
CRF works in the old way, compresses more high bitrate frames, not high motion
Thus has the tendency to use high quant in detailed bright scenes that need low quant
and gives reasonabily quality there only if you set a too low general quant
High quality doesnt need quant over 27
setting the max quant will assure that too low quant wont be used elsewhere

And setting an high I/P ratio will preserve key quality
a ratio of 1.8 that that will give a difference of 5 in CQP, gives barely a 2 in CRF
(And, if in CQP 1.7 and 1.8 give same result, in CRF do not.)

[ty2010]

--detelecine --decomb --strict-anamorphic -e x264 -q 21 -a 1 -E faac -B 128 -6 auto -R Auto -D 0 --gain=0 --subtitle 1 -x ref=4:weightp=1:subq=3:rc-lookahead=80:trellis=0:8x8dct=0:scenecut=60:deblock=1,1:aq-strength=0.8:aq-mode=1 --verbose=1 --previews 30

Or --crop 0:0:0:0 and drop the previews if you don't want auto cropping.

This is for main profile, drop the 8x8 and trellis if you're ok with high. If you're looking to retain grain, it's going to be large, period(upscale 720 with lower quality is an option). On that note, I've had better luck with xvid with grain retention without monster file sizes.

You have to translate the settings but they also work well with x264/x264vfw.

[Asmodian]

That command line looks to be setup for speed over quality. I still think most users should stick with the defaults unless they have a good reason not to (and can articulate it), all these random command lines tend to get people confused and using less than optimal settings.

[Blue_MiSfit]

CRF works in the old way, compresses more high bitrate frames, not high motion
Thus has the tendency to use high quant in detailed bright scenes that need low quant
and gives reasonabily quality there only if you set a too low general quant
High quality doesnt need quant over 27
setting the max quant will assure that too low quant wont be used elsewhere

And setting an high I/P ratio will preserve key quality
a ratio of 1.8 that that will give a difference of 5 in CQP, gives barely a 2 in CRF
(And, if in CQP 1.7 and 1.8 give same result, in CRF do not.)

Maybe for some sources. But I'd be surprised if you see this behavior for the majority of sources.

[ty2010]

That command line looks to be setup for speed over quality. I still think most users should stick with the defaults unless they have a good reason not to (and can articulate it), all these random command lines tend to get people confused and using less than optimal settings.

No, quite purposeful. ref=4, no 8x8 and no trellis are for main profile and maximum compatibility, not everyone watches on PCs.

detelecine, decomb are hardly fast at all, there is a slow setting for decomb but that is overkill if normal is doing the job. Restoring progressive frames most times yields the highest quality and smallest file size.

strict anamorphic outputs the exact frame size and PAR of the original, no distortions from scaling.

subq=3 gives the greatest compressibility while still taming grain, lookahead 80 is not fast, nor extreme, scenecut 60 will actually boost quality a bit.
deblock=1,1:aq-strength=0.8:aq-mode=1 These work together to preserve edges and detail while still suppressing noise/grain. This will also clean up minor blocking/artifacts from the original mpeg2.

The original inquiry was encoding size reduction while maintaining quality, not just highest quality.

[benwaggoner]

...no trellis are for main profile and maximum compatibility...
Trellis shouldn't have any decoder-side impact other than slightly reducing bitrate and thus slightly increasing speed of decode.

No?

[nm]

subq=3 gives the greatest compressibility while still taming grain

You'll always get better compression with higher subq (=subme in x264 CLI). If you encode with subq=3 and subq=8 at the same bitrate, the subq=8 encode will have more grain because of improved quality. Or the other way around, if you encode at the same visual quality (which is difficult in practice), subq=8 encode will have smaller file size. In this case the amount of grain left would likely be similar in both videos.

The same principle can be applied to many other options too. Just take a look at what x264 speed presets (as listed by x264 --longhelp) set: all the options listed under --preset affect quality at a given bitrate, or vice versa, bitrate at a given quality. Slower options give higher quality. Speed presets are wisely chosen and you really shouldn't overlook them in general encoding tasks. In specific cases such as animation or grain retention, take a look at --tune settings.

And by the way, if you want to "tame" grain efficiently, denoise before encoding or play with --nr



The original inquiry was encoding size reduction while maintaining quality, not just highest quality.

Yep, Asmodian was talking about encoding size reduction. When talking about "quality" in the context of video encoding, people here usually mean quality per file size. Therefore "improving quality" (at given bitrate) equals "reducing encoding size" (at given quality).


If you're looking to retain grain, it's going to be large, period(upscale 720 with lower quality is an option). On that note, I've had better luck with xvid with grain retention without monster file sizes.

Could this have something to do with too high deblocking strength? I always use -1:-1 or less for live action, no matter if it's low or high bitrate. (Although my "low" could be "medium" for somebody else). Also try other options set by --tune film or --tune grain.

If you want to keep some kind of grainy (or at least detailed) look at lower bitrates, try mdegrain. It will remove most of the temporal noise/grain and therefore improves compression a lot, which in turn helps retaining textures and any leftover grain. I usually get better results this way than by letting the bitrate-starved encoder mash all the noise and details into a pile of blur and artifacts.



deblock=1,1:aq-strength=0.8:aq-mode=1 These work together to preserve edges and detail while still suppressing noise/grain. This will also clean up minor blocking/artifacts from the original mpeg2.

If the source has blocking artifacts and you want to keep details from the more detailed frames, deblock with something like Deblock QED.

I know you probably want to use HandBrake, but it just doesn't have all the really nice filters that people have written for AviSynth. When encoding high quality sources at higher bitrates, you can get away with x264 alone, but at lower bitrates filtering really helps a lot.

[ty2010]

nm-

subq3, depends on quality level, get higher than q20 and it starts to drop the grain as noise rather than preserved as a detail, with the higher subq's it's kept.

nr works for some sources and softens a bit, others it generates artifacts and breaks up edges, definitely not a relaible way to go

deblocking, aqmode 1 is less likely to generate macroblocks but doesn't help sharpness in high contrast areas, mode 2 sharpens those contrast areas and is usually more compressible but will macroblock readily in flat areas at lower bit rates but does a lot better at higher bit rates. 1,1 works well if the aq strength is lowered.

Want to use handbrake? yea, one click on a batch file is the height of efficiency. If there's a less than optimal output, I'll run with HB filters, if still no go on to virtualdub, if it doesn't have the tools then avisynth joins the game. If I can code something at 600fps, that's what I'm going to do, there's absolutely no sense in knocking something to 40 or 20fps without a very good reason.

benwaggoner

some content I actually get higher encoding rate with trellis on. The problem is it adds a little unpredictability at lower bitrates, random little smears and artifacts, up the bitrate and it goes away, and you're right back where you started. I make as much as I can to set and forget so I have more time for other issues.
Allegedly this is fixed after looking it up, I suppose I can give it yet another test run.

[ty2010]

OK, looks like trellis is 90% fixed, still causes some smears and/or artifacts on transitions and low contrast areas. Transitions I don't mind, I'll see if the low contrast gets on my nerves too much.

*Not going to use trellis, besides files being slightly lower quality and slower to code overall, they're larger when it's used with my other settings. Maybe I'm already optimized?