Help Requested to Maintain Fine Detail in an x264 Encode

[anonmous]

Hi,
Like most who come into these forums, I am looking to maximize the quality to size ratio for some video footage I have using the x264 codec. I started off with a 347 Mb/s source file which came from Fraps. I plan on using this footage and others in some compositing projects that I hope to get to sooner than later, so I want to keep the quality of the footage as high as possible, but, as is usual, with the least space footprint.
Through various experiments, I’ve found that using the ‘Very slow’ preset within TMPGEnc Video Mastering Works 5.3.1.85, that very fine detail is maintained all the way down to a one-pass VBR (Average bitrate) with the bitrate set to 30,000 kb/s and a maximum bitrate of 34,000 kb/s (I will abbreviate this type of average/maximum bitrate from here on out as 30-34 Mb/s). Quite frankly, to my mind, this is nothing short of amazing: a 10-fold decrease in size with no apparent loss in quality. If I go any lower than this, certain details which are 2x2 pixels, 1x3 pixels, etc., tend to disappear.
I have set up a project within Adobe Premiere CS6 in which I compare 25 different frames from two different encodes. The first sequence is what I refer to as the ‘reference’ sequence, and the second is what I call the ‘current’ sequence. I initially started with an encode at 60-64 Mb/s. The 60-64 Mb/s encode when compared with the original 347 Mb/s file, showed no difference of any consequence in any of the 25 different frames being compared. I then kept reducing the bitrate until I found that dropping below 30-34 Mb/s showed a drop in detail in more than half of the 25 frames being compared. After that, I dropped the bitrate down to 20 Mb/s and started experimenting with the so-called ‘advanced’ settings. I also set up the comparative sequences so that I could apply the Difference blending mode to the reference and current sequences, so I would be able to see precisely where the two encodes differed.
As I changed each advanced setting, the resulting encode would become the ‘current’ sequence and throughout all of the advanced settings experiments, the ‘reference’ sequence remained the 30-34 Mb/s encode. For each advanced setting change I would tabulate which of the frames in the ‘current’ encode had maintained their quality when compared to the ‘reference’ encode, and which had lost quality when compared to the ‘reference’ encode. Based on that, I would give the advanced setting being experimented with a thumbs up or thumbs down. I have tried at least 40 different encodes with a variety of settings, and no matter how high I crank the settings on the lower bitrate encodes, the detail in very fine detail largely disappears.
What follows are some of the settings I’ve experimented with and found to (in general) maintain more detail in the encode:
Setting modified value | original value
VBR (average bitrate) pass count 2 | 1
Max number of reference frames 4 | 3
P Picture relative quality 2x | 0 (Automatic)
Motion Search Type Hadamard Exhaustive | Uneven Multihex
Use fast P-Skip search Unchecked | Checked
Apply DCT decimation Unchecked | Checked
B Frame pyramid mode Strict | Disabled
Sub-pixel motion estimation mode 11 | 9

I’ve also tried combining some as well as all of the above settings, and none of them have been able to get a 20 Mb/s encode to be as detailed as the 30-34 Mb/s encode. So, ultimately what I’m looking for is this:
1. What is a recommended recipe of x264 settings which would have a good chance of maintaining detail which is visible at 30-34 Mb/s at a significantly reduced bitrate (i.e. settings which achieve the objective at 29-33 Mb/s would not be significant in my mind, but settings which achieve the objective in a 24-28 Mb/s encode would be a significant and relevant reduction in bitrate)? I welcome as many suggestions as possible here because it's easy for me to run a couple of encodes, drop them into sub-sequences in Premiere and have 25 different frame-by-frame comparison sequences pick up the changes automatically.
2. Also, is there a resource where I could get some definitive guidelines as to how to pick a maximum bitrate to correspond with an average bitrate (e.g. if I’m doing a VBR average bitrate encode, and my average bitrate is set to 10 Mb/s, what should my maximum bitrate be set to? What about for an encode with an average bitrate set to 20 Mb/s? etc.).

For the source file, MediaInfo yields the following:
Format : Fraps
Codec ID : FPS1
Duration : 7mn 28s
Bit rate : 347 Mbps
Width : 1920 pixels
Height : 1200 pixels
Display aspect ratio : 1.600
Frame rate : 30.000 fps
Color space : YUV
Bit depth : 8 bits
Bits/(Pixel*Frame) : 5.019
Stream size : 18.1 GiB (100%)


For the encoded file 30-34 Mb/s file which shows no relevant loss of quality, MediaInfo yields the following:
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.1
Format settings, CABAC : Yes
Format settings, ReFrames : 3 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 1mn 31s
Bit rate mode : Variable
Bit rate : 29.8 Mbps
Nominal bit rate : 0 bps
Maximum bit rate : 34.0 Mbps
Width : 1728 pixels
Height : 1080 pixels
Display aspect ratio : 1.600
Frame rate mode : Constant
Frame rate : 30.000 fps
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.533
Stream size : 326 MiB (100%)
Writing library : x264 core 124
Encoding settings : cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x133 / me=umh / subme=9 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=2 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-4 / threads=12 / sliced_threads=0 / slices=1 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=0 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=2pass / mbtree=1 / bitrate=30000000 / ratetol=1.0 / qcomp=0.60 / qpmin=5 / qpmax=69 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / vbv_maxrate=34000000 / vbv_bufsize=34000000 / nal_hrd=vbr / ip_ratio=1.40 / aq=1:1.00
Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.709-5, BT.1361, IEC 61966-2-4 709, SMPTE RP177

I’ve tried many different settings. I’ve run well over 50 different encodes at lower bitrates, but once I drop below 30-34 Mb/s, I lose the details spoken of. Now, although these encodes look absolutely fantastic when viewed as video, they don't maintain the type of frame-by-frame detail which I'm looking for. It very well could be that as in the realm of engines "there is no replacement for displacement", it is true in the realm of video quality "there is no replacement for bitrate." If so, no problem, I just want to feel reasonably confident that I've taken proper advantage of what x264 has to offer.

Thanks very much!

Matt

P.S. as the saying goes, a picture is worth a thousand words, so I've attached a few screen captures. In each set, the first is from the 60-64 Mb/s encode and the second is from the 30-34 Mb/s encode (I'm not sure if the pictures will appear in-line after this, but here's hoping):

[Asmodian]

The only settings I mess with when going for quality >> size encodes are no-fast-pskip and no-dct-decimate.

It looks like you already tried that so from my experience you are good.

Why "B Frame pyramid mode: Strict" instead of normal? Are you targeting bluray compatibility? Not that the change would allow the same quality at 80% of the bitrate.

I don't think you want to use "P Picture relative quality", did it help? Making the P-frames larger at the expense of B-frames doesn't seem good if you want all frames to be at max quality.

Of course increasing the number of reference frames would help, so might increasing the number of B-frames.

[anonmous]

Hi Asmodian, well the list of modified and original values was compiled simply through trial and error, I make no claims as to the appropriateness of any setting, only that for the encodes which I examined, all of those listed showed a greater retention of detail in more than half of the 25 sample frames examined side-by-side. For some of those settings, a number of the 25 frames did, in fact, look worse, but if over half of the 25 looked better, I gave the setting a tentative 'thumbs up.' Thanks for your thoughts.

Matt

[nibus]

If quality is your main concern you might try using CRF mode (http://mewiki.project357.com/wiki/X264_Settings#crf) instead of 2-pass. Start with 16 and move up from there, stopping once you notice loss of detail.

You will also want to increase psy (http://mewiki.project357.com/wiki/X264_Settings#psy-rd). You can do this easily by using the built-in tunings (http://mewiki.project357.com/wiki/X264_Settings#tune), or you can manually set it. Because this is game footage I'm not sure which tuning to recommend, but film is a common one.

AQ (http://mewiki.project357.com/wiki/X264_Settings#aq-strength) is the other setting you will want to experiment with. You are using AQ mode 1, with a default strength of 1. On HD footage I've found that lowering this value to .6-.8 can help increase detail retention. AQ mode 2 (http://mewiki.project357.com/wiki/X264_Settings#aq-mode) is thought to be superior by some but you will want to test with it and compare the two. Depending on your footage, lowering deblock (http://mewiki.project357.com/wiki/X264_Settings#deblock) may also help detail retention. Try using -3:-3, -2:-2, or -1:-1 and see what you think.

I would start with these options and then go from there:


--crf 16 --preset slower --tune film --deblock -3:-3 --aq-strength 0.75


You can also add these at the expense of time and somewhat higher bitrate:


--no-fast-pskip --no-dct-decimate


These are pretty high settings, but if you still are not satisfied you could try --preset veryslow, increase --qcomp (http://mewiki.project357.com/wiki/X264_Settings#qcomp), and also try disabling mbtree (http://mewiki.project357.com/wiki/X264_Settings#no-mbtree). Those last two options will increase the bitrate dramatically. Raising qcomp will allocate more bitrate to motion.

[raffriff42]

You are right, there is no substitute for bits. I for one am against using any sort of lossy compression on an 'intermediate' file (one which will be edited later).

you could (in fact you will) get an encoding artifact but not notice it until it is too late. Murphy's law, you know.
less information redundancy means greater susceptibility to bit rot. (OTOH, you can make more backup copies since they are smaller)
for me, highly compressed video seems very sluggish in the editor (try it with a dozen layers onscreen at once...)
any time you dub it's another opportunity to mess up the color (Rec601/Rec.709) and luma range (TV/PC), among other things

So I say, buy more hard drives and keep the original Fraps footage. If you must compress, don't take chances: encode with the best possible quality. To me, that means '--crf 6.0' or lower.

[anonmous]

Hi nibus, I've been running encodes all weekend based upon your suggestions. Thanks by the way for your efforts! Frustratingly, what I've found is that invariably, once I drop below 30 MB/s, that no matter which combination of settings I use, some frames will appear better, and others will appear worse, almost exactly 50/50. I have yet to experiment with the deblock settings and the psy settings.

Matt

[anonmous]

Hi raffriff42, you very well may be right. It does seem that once a certain bitrate is hit, that there's no combination of settings which can recover the fine details I'm looking to keep, or that the secret combination of settings to do so is lost in the sands of time.

Thanks,

Matt

[mogobime]

Sorry, I'm a little late.
I don't know if it will help you, but you can try my tool to automatically raise various parameters in crf-mode (for example psychovisual rtate distortion and desicion/trellis).

You will get quality as PSNR/SSIM and bitrate and it also can keep the output video files if the setting "L" (save generated videos...) is set to "on":
http://forum.doom9.org/showthread.php?t=166295

P.S.:
Try to modify adaptive quantization: Set aq-mode to "2" and aq-strength to "0.5" in the X-QuaSaT tool (or change the parameter "aq=1:1.00" to "aq=2:0.50").
Maybe this can help.

Cu
mogobime

[Sagittaire]

Don't use VBV setting else you have buffer saturation and potentially low quality in difficult part ...

[benwaggoner]

Don't use VBV setting else you have buffer saturation and potentially low quality in difficult part ...
Have you actually seen cases where there was a visible quality reduction in using the Level max VBV versus no VBV?

If a file needs more VBV, I think it would be preferable to just use a higher Level that allows the desired VBV in order to maintain compliance with the H.264 spec. A higher Level can also provide more reference frames and other features that can help compression efficiency.

Lots of reference frames can be particularly helpful for synthetic content like this.

[Sagittaire]

Have you actually seen cases where there was a visible quality reduction in using the Level max VBV versus no VBV?

If a file needs more VBV, I think it would be preferable to just use a higher Level that allows the desired VBV in order to maintain compliance with the H.264 spec. A higher Level can also provide more reference frames and other features that can help compression efficiency.

Lots of reference frames can be particularly helpful for synthetic content like this.

Yes and it's really easy to show that if your average bitrate is really near of max bitrate. In this case you have that:
bitrate=30000000 / ... / vbv_maxrate=34000000

... and certainely empty buffer for 90% of encoding. And empty buffer mean CBR encoding ... :scared:

Without the output file I can already say that you have really higher quant (than average quant) in high complexity part of encoding. For BluRay encoding the VBV model is really important (perhaps more important than all other parameters) if you use average bitrate near than max bitrate. In extreme case average encoding at 20 Mbps with max at 40 Mbps will be better in complex part than average encoding at 40 Mbps with max bitrate at 40 Mbps.

[benwaggoner]

Yes and it's really easy to show that if your average bitrate is really near of max bitrate. In this case you have that:
bitrate=30000000 / ... / vbv_maxrate=34000000

... and certainely empty buffer for 90% of encoding. And empty buffer mean CBR encoding ... :scared:
Yeah, but have you actually seen a visual quality degradation in that case?

A buffer that's empty because the bitrate is really high should look better than a buffer that's full because bitrate is really low.

Any file is always going to have a VBV, even if it can only be known after the fact. In practice, I suspect many encodes don't ever get that close to entirely filling their buffer.

Without the output file I can already say that you have really higher quant (than average quant) in high complexity part of encoding. For BluRay encoding the VBV model is really important (perhaps more important than all other parameters) if you use average bitrate near than max bitrate. In extreme case average encoding at 20 Mbps with max at 40 Mbps will be better in complex part than average encoding at 40 Mbps with max bitrate at 40 Mbps.
But in that case, it would be a rate control problem, no? VBV~max bitrate, so assuming a perfect rate control, a 20/40 VBR encode should never be better than a 40 CBR encode.

[sneaker_ger]

What's with those vbv values? vbv_maxrate=34,000,000 and vbv_bufsize=34,000,000 should be totally out of spec. Or does your program handle it differently than x264cli?

[Sagittaire]

Yeah, but have you actually seen a visual quality degradation in that case?

Yes ... simply because encoder say "I want pframe at 500 Ko for maintain constant visual quality" but VBV say "it's not possible, buffer is empty, only 208 Ko for your Pframe here"

A buffer that's empty because the bitrate is really high should look better than a buffer that's full because bitrate is really low.

No ... simply because max bitrate is not strictly max local bitrate. If your buffer is empty, frame can't have size higher than 208 Ko. With buffer the local bitrate can be localy higher than max bitrate (on small duration).


But in that case, it would be a rate control problem, no? VBV~max bitrate, so assuming a perfect rate control, a 20/40 VBR encode should never be better than a 40 CBR encode

VBV control is really complex. For exemple good codec must use always buffer for IFrame (2x-3x higher size in average than average local bitrate). For encoding at 40 Mbps with max rate at 40 Mbps average Iframe size will be at 600 Ko (average local bitrate at .... 120 Mbps). If encoder doesn't make that you have catastrophic quality for Iframe ... and quality flicking at each Iframe (the famous "Quality Poping").

[mogobime]

did you try to modify adaptive quantization as I suggested before?
For me this looks like a problem caused by adaptive quantization strength that is set too high.
If this doesn't help, try X-QuaSaT: http://forum.doom9.org/showthread.php?t=166295
(the problems with US version Windows should be solved now)
-Make a short high quality mpeg-2 clip of 120 seconds (with for example mediacoder) including the problematic scene and put it to VIDEOFILES folder
-Set SAVETEMPFILES to ON.
-Set ENCODINGMODE to qp and use a very low quantizer
-Configure the ranges of the loop-parameters you want X-QuaSaT to raise automatically

Example:

BFLOOP: 1 1 3
PSY-RD: STEP
PSYRDLOOP: 0 20 140
PSYTRLOOP: 0 20 140
SUBLOOP: 9 1 10
ENCODINGMODE: qp
QUANTIZATION: 2
REFFRAMES: 6
AQ-MODE: 2 or 0
AQ-STRENGTH: 0.5 or 0
TRELLIS: 2
SAVETEMPFILES: ON
If you want to use MS Excel / OpenOffice (will be clearer): OUTPUTFORMAT: csv

Now make sure you have enough free diskspace, hit [ENTER] and let X-QuaSaT run.
Go out and have some beer, this will take time:
You will get roundabout 300 clips, and the used setting can be identified with the help of STATISTIC.LOG/CSV

Cu

mogobime

[benwaggoner]

Yes ... simply because encoder say "I want pframe at 500 Ko for maintain constant visual quality" but VBV say "it's not possible, buffer is empty, only 208 Ko for your Pframe here"
Is this a single-pass encode without any lookahead rate control? A decent rate control mechanism would know that a hard P-frame is coming up and would save bits from previous frames in order to make room in the buffer for it.

I'd still be very interested in seeing a real-world example where the quality actually goes down when using VBV with -rc-lookahead on.

[Sagittaire]

Is this a single-pass encode without any lookahead rate control? A decent rate control mechanism would know that a hard P-frame is coming up and would save bits from previous frames in order to make room in the buffer for it.

Certainely but Buffer is by definition limited. If your complex part is very long (typicaly more 1 sec for BD for example), then when buffer is empty your Pframe will be always limited at 208 Ko. Good codec with good VBV will only reserve buffer for Iframe.


I'd still be very interested in seeing a real-world example where the quality actually goes down when using VBV with -rc-lookahead on.

I make study with x264 (influance of bitrate variability with Bitrate/maxbitrate/qcomp). I will post clear example when I have free time ... ;-)