x264: guess crf for given bitrate

[WasF]

OK, "fast first pass" helped ! But, still, no general info about how 1st/2nd pass options relate/should be balanced.

"Fast first pass" is --pass 1, precisely defined in x264's help as: force no-8x8dct, partitions none, me dia, ref 1, subme 1:user,else:2, trellis 0, fast-pskip
Unless you set slow-firstpass/perest placebo.

In my case, that leaves (besides level/profile) : rc-lookahead 60, b-adapt 2, merange 24, no-dct-decimate, direct auto
So, these are the potential offenders. That narrows it down, but I'm still not able to tell if disabling those in first pass would affect final result much.
I guess I'm only left with trying it out and comparing the stats files numbers with and without those options. Could be inconclusive, though, if difference is very small..

[Sharc]

This reminds me of another proposal (made by Dark Shikari?) how to speed up a 2-pass encode. I am not sure though if it has ever been realized as it requires some deeper knowledge:
Run the first pass on a reduced resolution of the video, e.g. 1280x720 instead of the original 1920x1080. Then modify the stats file somehow (?) before doing pass 2 with the original resolution.

I try to find the thread / post and report back when I find it.....

[sneaker_ger]

Has been realized already. Too lazy to search for the thread.

[nm]

In my case, that leaves (besides level/profile) : rc-lookahead 60, b-adapt 2, merange 24, no-dct-decimate, direct auto
So, these are the potential offenders. That narrows it down, but I'm still not able to tell if disabling those in first pass would affect final result much.

Don't change other settings between passes than what's listed there in x264's help under --slow-firstpass. If you'd like to speed up more, use faster settings in both passes. Placebo-level settings such as --me esa are pretty useless for their cost in CPU time.

[WasF]

Maybe. Without doing a detailed comparison :
--pass 1 + Full options:
[info]: frame I:895 Avg QP:35.52 size: 35034
[info]: frame P:20819 Avg QP:38.94 size: 6666
[info]: frame B:54135 Avg QP:43.47 size: 859
[info]: consecutive B-frames: 3.0% 4.9% 10.2% 68.0% 13.9%

Just --pass 1:
[info]: frame I:909 Avg QP:35.22 size: 35797
[info]: frame P:21879 Avg QP:38.53 size: 6170
[info]: frame B:53061 Avg QP:42.95 size: 927
[info]: consecutive B-frames: 3.3% 5.8% 13.7% 77.2%

Selection of frame types is affected (even though the QP numbers don't change much). I see stats there jumping from 68 to 77%. I'd say the full options do have a bit of an effect (and don't think doing a more detailed analysis would produce poorer decisions). So, yes, final quality would be affected.
How much ? I don't know. It seems like a small effect (the full options produced 1.54% less I frames, which would indicate a slightly wiser bit allocation to come in second pass - and more computations). But the raw numbers are hard to interpret for an algorithm as complex as H.264.

Use your eyes, I guess.

[Groucho2004]

So, yes, final quality would be affected.
How much ? I don't know. It seems like a small effect (the full options produced 1.54% less I frames, which would indicate a slightly wiser bit allocation to come in second pass - and more computations).
You compared two extreme sets of settings, you could just pick something in between for pass 1. For example:
--slow-firstpass --ref 4 --me hex --subme 3 --trellis 0

[Dark Shikari]

the full options produced 1.54% less I frames, which would indicate a slightly wiser bit allocation to come in second pass - and more computationsThis conclusion doesn't follow at all from the premise. Slight differences in frame type choices have literally nothing to do with ratecontrol or the "amount of computations".

There is also no particular reason -- beyond pure chance -- that slower settings would produce more or fewer I-frames.

This thread doesn't make much sense. If you're using such needlessly slow encoding settings to begin with, you're better off sacrificing 0.2% compression and doing 2-pass, and using slightly faster settings.

[WasF]

You compared two extreme sets of settings

I didn't choose "extreme" settings on purpose, those are my actual settings. And "extreme" is actually good here, to make differences (if any) pop up.

you could just pick something in between for pass 1.
Until a suitable crf estimation method is found (will it ever ?), that looks like the only reliable option.

Slight differences in frame type choices have literally nothing to do with ratecontrol or the "amount of computations".
I don't know about ratecontrol, but I thought for a fixed set of frames, picking less I frames means more P & B frames (which need ME), which means more ME calculations (the bulk of the algorithm)..

This thread doesn't make much sense. If you're using such needlessly slow encoding settings to begin with, you're better off sacrificing 0.2% compression and doing 2-pass, and using slightly faster settings.
I would sacrifice 0.2% size in a heartbeat. My concern is quality (which doesn't mean I would not care about +20% size for its sake, though). What "reasonable" settings would you use for "very good" encoded result at 1500 kb/s from a 720p source ? (assuming an average source: some action, some dark scenes, mostly slow).

[detmek]

OK, "fast first pass" helped ! But, still, no general info about how 1st/2nd pass options relate/should be balanced.

"Fast first pass" is --pass 1, precisely defined in x264's help as: force no-8x8dct, partitions none, me dia, ref 1, subme 1:user,else:2, trellis 0, fast-pskip
Unless you set slow-firstpass/perest placebo.

In my case, that leaves (besides level/profile) : rc-lookahead 60, b-adapt 2, merange 24, no-dct-decimate, direct auto
So, these are the potential offenders. That narrows it down, but I'm still not able to tell if disabling those in first pass would affect final result much.
I guess I'm only left with trying it out and comparing the stats files numbers with and without those options. Could be inconclusive, though, if difference is very small..

How can comparing stats files give you information about video quality?:confused:
When it comes to lossy video encoding (x264 is lossy encoder except for qp 0) quality is measured/compared by watching video, not by reading statistics.
You said that chainging first pass options changed number of I frames by 1,54%. What that means? Is the quality 1,54% better or 1,54% worse? Or is it 20% better or 20% worse? Maybe it is the same? Is this differece even visible when you watch the video?

[WasF]

@detmek: have you read just that one post ? You're missing my point completely.

[detmek]

Ok, change of plan. Here is accumulated data so far:

Options: x264-10bit_32bit.exe --rc-lookahead 60 --bframes 4 --b-adapt 2 --subme 10 --trellis 2 --partitions all --8x8dct
--me esa --merange 24 --threads 4 --no-dct-decimate --direct auto --no-interlaced --level 4.2 --ref 8 --profile high422
-------------------------
Source: TV, nature (60% bright, 20% night, 75% normal/slow, 25% action), 1280x720p, 29.97fps, 75835 frames
AVS: SelectRangeEvery(600,60) # 10%. Assuming 25fps: pick 2.4s at the begining of every 24s slice.
Or SelectRangeEvery(1200,60) # 5%. Assuming 25fps: pick 2.4s at the begining of every 48s slice.
Or SelectRangeEvery(600,180) # 30%. Assuming 25fps: pick 7.2s at the begining of avery 24s slice.
-------------------------
NeedBitrate: 1500
Komisar's formula: stop when GotBitrate in [NeedBitrate-100, NeedBitrate+100], or when reach 4 runs
Run 1 2 3 CRF Seed: 23.00
GotBitrate 2727.18 1818.28 1564.87 Bitrate error = GotBitrate - NeedBitrate
GotCRF 23 25.70 26.75 Reverse rel. Bitrate err = - Bitrate_error / GotBitrate
NextCRF 25.70 26.75 27.00 NextCRF = GotCRF - [ 6 * Reverse_relative_Bitrate_error ]
Duration 1h18 1h16 1h12
-------------------------
Results after full encode:
My method 30%(26.15) Komisar 10%(27.00)
Bitrate 1605 eta 9h30 .. :(
Error 105 ?
Cost 3h24 3h46



Done already : --rc-lookahead 60 --bframes 4 --b-adapt 2 --subme 10 --trellis 2 --partitions all --8x8dct --me esa --merange 24 --threads 4 --no-dct-decimate --direct auto --no-interlaced --level 4.2 --ref 8 --profile high10 (--fps if needed)

That's my only "preset": I don't compromise on quality. For me an encode is maximum quality, or isn't (use CUDA !).
With this, even a first pass is slow.
When you have average CRF of 26 quality is seriously compromised and there is no option or parametar that can do much to save it.
Maybe. Without doing a detailed comparison :
--pass 1 + Full options:
[info]: frame I:895 Avg QP:35.52 size: 35034
[info]: frame P:20819 Avg QP:38.94 size: 6666
[info]: frame B:54135 Avg QP:43.47 size: 859
[info]: consecutive B-frames: 3.0% 4.9% 10.2% 68.0% 13.9%

Just --pass 1:
[info]: frame I:909 Avg QP:35.22 size: 35797
[info]: frame P:21879 Avg QP:38.53 size: 6170
[info]: frame B:53061 Avg QP:42.95 size: 927
[info]: consecutive B-frames: 3.3% 5.8% 13.7% 77.2%


QP 38,94?! I don't want to know how that looks like.
Main problem is:"
...So, yes, final quality would be affected.
How much ? I don't know.
This is probably the answer:

This thread doesn't make much sense. If you're using such needlessly slow encoding settings to begin with, you're better off sacrificing 0.2% compression and doing 2-pass, and using slightly faster settings.
OK, maybe little more, say, 1-2%. It depends what you change.
Maybe you could try with:

--preset veryslow --b-frames 4 --ref 8

or even

--preset slower --b-frames 4 --subme 10

And if you want to quantify difference betweet this and your settings, add:

--tune ssim --ssim

to your and this command line. Then compare SSIM values.
It is not perfect measure but it helps. Also do a visual comparison, too.

[Dark Shikari]

Until a suitable crf estimation method is found (will it ever ?), that looks like the only reliable option.There is no such thing as a "suitable crf estimation method": this is what 2-pass is for.I don't know about ratecontrol, but I thought for a fixed set of frames, picking less I frames means more P & B frames (which need ME), which means more ME calculations (the bulk of the algorithm)..This is not true, please stop guessing wildly about how x264 works. Your statement is so wrong it's fractally wrong -- not merely incorrect, but even the premise doesn't make any sense.

I-frames are not faster to encode at typical settings (they are usually slower). The number of I-frames is a miniscule, irrelevant variable in terms of the total encoding time. Practically everything else matters more.
I would sacrifice 0.2% size in a heartbeat. My concern is quality (which doesn't mean I would not care about +20% size for its sake, though). What "reasonable" settings would you use for "very good" encoded result at 1500 kb/s from a 720p source ? (assuming an average source: some action, some dark scenes, mostly slow).The same as for any video ever: the slowest preset you have the patience to use. I don't understand why this is so incredibly difficult!

I really don't know why I bother posting here when people simply ignore the answers from someone who actually wrote the code they're using.

[WasF]

@detmek: OK, you don't have to quote all previous posts to prove you read them :p

I'm re-encoding 720p material with the goal of using the smallest possible bitrate without affecting quality. 26 is actually too low. Often times, I use values closer to 27, and I can't see a difference on my 22" screen. I would have to get within 4 inches from the screen to see the slight loss of detail (and this is a serial encoder speaking, highly sensitive to codec limits). Maybe 4 Mb/s bitrates are useful on a 55" screen, but for my needs, I noticed 1.5 Mb/s is generally enough, maybe 1.8/1.9 Mb/s when the source has lots of dark scenes (rarely more). Doesn't hurt the disk space usage.
I guess if I used a fast preset, I wouldn't be as satisfied with 1.5 Mb/s, and the "low" bitrate would affect quality. So I'm paying for "low" bitrate with "stronger" settings (very long encoding time). That's how I don't compromise on quality (and I'm paying the price).

The price is hefty, and that's why I'm here looking for cheat-sheets.

You're still missing the point of my last post. The QP you emphasized comes from a test I ran to check if going from full options in pass 1 (slow) to just --pass 1 (sweet) would have an effect on the final quality, in general, and how much?
To that end, I was comparing the output of the two (and used a 480p source to run the test faster). The small difference I noticed lead me the exact same conclusion you reached (use balanced settings), confirmed by Dark Shikari's assertive contribution.

[WasF]

@Dark Shikari: thanks for the clarifications. Will do as you said.

[detmek]

True. Visual quality is subjective. One advice if you are keeping your settings - don't use --me esa. It is very very slow, especialy with --merange 24 and you gain nothing. Also, you can improve compressibility by doing light denoiseing. MVTools is very good for that as it can remove grain but keep details.

[sirt]

Dark Shikari, I personally don't think anybody ignores what you say or what a developper advises. I grant some people may have inflexible beliefs about such and such parameter or options Or they are simply confused (including myself) because they don't understand how it really works.

But look, you can't say "stick to the presets and be happy" albeit it is probably the safest thing to do unless you want to experiment or understand. But alow me to raise a critical issue : what do you think people will do with http://mewiki.project357.com/wiki/X264_Settings ? Probably nothing constructive as reading the similar AVIdemux wiki related page will not illumine them either if they don't understand the principles. I think you need a deep understanding of video compression to adjust parameters but you can't blame somebody for thinking wrong or melting with some options that should not be adjusted when it is not really clearly explained how they work.

I've seen an old tutorial by you where you *recommand* some settings for *very high quality* encodes (http://community.eveonline.com/ingameboard.asp?a=topic&threadID=503062) but you never explain why this parameters are great or not. It's like you suggest using them and that's all. Of course you could retort me or anybody else that wants to understand should just read about h264/h265 norms and related books. It is surely a good idea but it is perhaps conceited. In my opinion, you should write a complete guide to explain exactly what does either parameter and where it is implemented in the source code. It's not too much to ask to help novices to understand what they do ! Saying use this and that and don't worry about won't help to get anything done.

[Dark Shikari]

I've seen an old tutorial by you where you *recommand* some settings for *very high quality* encodes but you never explain why this parameters are great or not.This was years ago before I was a developer, and before I made the presets. I was also rather clueless back then; please don't use posts of mine from 5 years ago as guides to anything at all, except maybe "what not to do".
In my opinion, you should write a complete guide to explain exactly what does either parameter and where it is implemented in the source code. It's not too much to ask to help novices to understand what they do ! Saying use this and that and don't worry about won't help to get anything done. I'm always happy to explain things for people who want to develop x264, but the obscure parameters aren't intended for ordinary users. All of this this leads to people recommending extremely complicated and convoluted methods of using x264, posting terrible and inaccurate information from others, and scaring people away from using x264 because it's "too hard".

Have you seen the number of awful guides around the internet recommending insanely complicated commandlines? Practically every day I have to deal with users who have been confused by these, from ordinary home users to major corporations using x264. Trying to convince everyone to mess with these obscure options is a great way to get people to stop using x264.

Also, a large part of x264's internals can't be explained without literally dozens of pages of background information about how codecs work. It's not that easy to just "write a guide".

[WasF]

Visual quality is subjective. One advice if you are keeping your settings - don't use --me esa. It is very very slow, especialy with --merange 24 and you gain nothing. Also, you can improve compressibility by doing light denoiseing. MVTools is very good for that as it can remove grain but keep details.

Indeed. With end-of-GOP images taken from esa/umh encodes of the same source, and even after 4x zoom, I couldn't decide which one I liked more. Maybe it can make a difference at higher bitrates, but at 1.5 Mb/s, there was no benefit to spending several hours more on --me esa.

Regarding the pre-denoising, I wouldn't think of using it on a clean source, even if improves compressibility (how ?), if it hurts visual quality. Does-it ?
I guess it's less risky with high-bitrate sources (8 or 10 Mb/s and more for 720p, I'm working with 4-6 Mb/s ones).

[nm]

Indeed. With end-of-GOP images taken from esa/umh encodes of the same source, and even after 4x zoom, I couldn't decide which one I liked more. Maybe it can make a difference at higher bitrates
It makes even less difference at higher bitrates.

Regarding the pre-denoising, I wouldn't think of using it on a clean source, even if improves compressibility (how ?)
Even videos that appear "clean" usually have some amount of temporal noise.

if it hurts visual quality. Does-it ?

Noise can hide artifacts such as slight banding, but if you are encoding at a low bitrate, chances are that denoising and adding noise back on playback gives better quality than trying to keep the noise. 10-bit encoding is also a good idea to avoid banding, if you only target PCs for playback.

I guess it's less risky with high-bitrate sources (8 or 10 Mb/s and more for 720p, I'm working with 4-6 Mb/s ones).

No, it's the other way around. The lower the bitrate, the more denoising makes sense.

[detmek]

As nm said.
But, before you start messing arount with MVTools you can try x264 internal denoise filter. Just add --nr xxx to command line where xxx is in 100-500 range to see results. MVTools is far better then x264 denoiser but is is good for experiment.

[WasF]

@nm: thanks for your kind answer. I do use 10-bit. I had no concept of what "temporal noise" is, but I found and example (http://www.youtube.com/watch?v=WHV6SP0lKuU) on Youtube. I would describe it as "a frenzy of semi-transparent tiny bugs crawling all over the screen" !
My source didn't look anything like that, so I wasn't exactly sure that noise reduction would make things any better (if not the opposite !).
But detmek's suggestion was easy to try out..

And the results are quite surprising :

http://desmond.imageshack.us/Himg254/scaled.php?server=254&filename=67492139.jpg&res=landing

No revolution there, but noise reduction does have a slight beneficial effect !

My understanding is that MVTools does a better job than x264's internal filter, so I can't wait to test it out ! Any references ?

[detmek]

Yes, it does. It preserves fine details better and does not blur image much.
Use MVTools that comes in dither package (http://forum.doom9.org/showthread.php?p=1386559#post1386559) and try with light denoising. Something like:
tr = 1 # Temporal radius
mt = true # Internal multithreading
lsb = false # 16-bit
thSAD = 200 # denoising strength
blksize = 16 # block size
overlap = 4 # block overlap
super = MSuper (mt=true)
multi_vec = MAnalyse (super, mt=true, multi=true, blksize=blksize, overlap=overlap, delta=tr)
MDegrainN (super, multi_vec, tr, mt=true, lsb=lsb, thSAD=thSAD, thSAD2=150)