x264: -crf18 results in high bitrates for odd sources?

[Loomes]

Hey there,

I encoded many of my movies from dvds and blu-rays to put them on my NAS at home, using the -crf18 option of x264 to preserve the quality that suits best for me. Everything went just fine but there is one thing that makes me wonder and costs a lot of space:

Some of the movies are rather odd or grainy original sources, e.g. "Psycho" from Hitchcock or "Morning of the Earth" from Albert Falzon which is a musical documentary from the 70s, consisting mostly of rather odd 8mm material. I presumed that doing -crf18 on low quality videos would result in very low bitrates but its quite the contrary.

While modern and excellent sources like e.g. "Oblivion" (2013) (Blu-Ray, encoded to 1280x720) results in lean 2067 kbit/s with x264 -crf18, "Psycho" (1960) (Blu-Ray, encoded to 1280x720) comes out with 6524 kbit/s and "Morning of the Earth" (1971) (DVD, 720x480) with 3089 kbit/s. The quality is excellent though.

So my questions are: Are the high bitrates from rather bad material really necessary or is x264 fooled somehow? Is there a way to avoid this without switching to constant bitrates? Why is it that x264 behaves like that? Thanks in advance!

[Sharc]

That's quite normal, any good encoder would do the same.
The noise can drive the bitrate up by factors due to the random nature of the noise.
If you want smaller files I recommend to denoise (or degrain) the source. It's offen better than increasing the CRF. Try for example hqdn3d(3) and see what you get.

[mandarinka]

As said, it is not as much that "CRF X results" in high bitrate, but that those sources are high complexity, hard to compress. Bitrates ending up "high" under CRF mode for given factor on the scale are just a logical consequence of that.

[Sparktank]

"bad material" is just the limitations at the then time.

Pyscho is really old. From 1960.
There's no way they could have filmed with digital or even newer cameras.
http://www.imdb.com/title/tt0054215/technical
35mm, Mitchell BNC
https://www.youtube.com/watch?v=RjrW-rM0JLs

Oblivion is 2013. Technology has obviously improved over the years for newer cameras. With the option of using digial capturing.
One of the cameras used for Oblivion was Red Epic.
http://www.imdb.com/title/tt1483013/technical

There's going to be considerably less noise and grain coming from newer movies using newer cameras.
Of course, it's all up to the director. Some insist on using film for a movie rather than digital.

On static backgrounds, older movies using older technology the grain/noise will be visible and variable.
For newer movies, the static background will likely only suffer banding since there's no grain/noise to smooth out the grading.

Can't really blame the times for being behind. If they didn't make Oscar winners with bad cameras, there would have been no interest to further invest in making movies and we wouldn't have better technology today and who knows what Oblivion might look like. Or if it would have been made at all.

[Loomes]

If you want smaller files I recommend to denoise (or degrain) the source. It's offen better than increasing the CRF. Try for example hqdn3d(3) and see what you get.
Thanks, but that would mean deviating from my standard process for certain movies and I also don't want to mess with the picture since grain/noise could also be a wanted effect in today's movies.

"bad material" is just the limitations at the then time.
On static backgrounds, older movies using older technology the grain/noise will be visible and variable.
For newer movies, the static background will likely only suffer banding since there's no grain/noise to smooth out the grading. Can't really blame the times for being behind.
Either this is rather offtopic or I don't get your point. Of course I dont "blame" Hitchcock for decreasing my harddrive space ;) I also have to say that nowadays quite a lot filmmakers choose grain/noise on purpose, e.g. Michael Mann or Martin Scorsese. So there is no point complaining about the source especially grain/noise. The expression "bad" was only meant to state the difference to noise-free material :)

[raffriff42]

Experiment with --vbv-maxrate at some medium-high figure. Unfortunately you will have to set a different max bitrate for standard def, high def etc.

The idea being that a less-than-perfect reproduction of a grainy image is an acceptable compromise.

Also try --tune grain or --tune film

[Sharc]

Thanks, but that would mean deviating from my standard process for certain movies and I also don't want to mess with the picture since grain/noise could also be a wanted effect in today's movies.

Sure, but preserving noise and grain drives the bitrate. There is no magic around it.

If size really matters you should do 2-pass encodes.

[Sparktank]

The expression "bad" was only meant to state the difference to noise-free material :)

Then you misspoke the first time.

So you can't really call me out on offtopic. You picked sources that are clearly noise vs clean and then ask why.
And then just turn around to say "I know" like a 5yo.
Sounds like you're just baiting here.

Clearly you cannot drive a point to begin with for others to work with.

[Loomes]

Then you misspoke the first time.
So you can't really call me out on offtopic. You picked sources that are clearly noise vs clean and then ask why.
And then just turn around to say "I know" like a 5yo.
Sounds like you're just baiting here.
Clearly you cannot drive a point to begin with for others to work with.
You still don't seem to get the whole point of my question. I asked if the behavior of x264 regarding grain/noise/odd/bad/8mm (call it what you want) material is normal and if, why that is and what I can do about it. Your response to these questions if mine was an explanation why older material is more "noisy" than modern material. I did not ask for that. And yes, -- I know -- why older material is of lower quality but -- I don't know -- why x264 behaves like that. This is not about the state or origin of the sources but about processing them. Seems everyone else got it so my English can't be that bad ;) So don't be pissed like a "5yo" (I just learned another expression here, thank you) and make such fuss about nothing.
By the way: I like chinese food -- and just went offtopic by saying that, so we're equal now. Happy? :)

[Groucho2004]

I asked if the behavior of x264 regarding grain/noise/odd/bad/8mm (call it what you want) material is normal and if, why that is and what I can do about it.Yes, it's normal and you got your answer(s) already in posts #2 and #3.
If you want more info on rate control, have a look here (http://akuvian.org/src/x264/ratecontrol.txt).

[benwaggoner]

Tossing in some --nr can help with random noise, while preserving the texture for the most part. the grain doesn't tend to change as much frame-to-frame, which can be more visually appealing, even if less accurate. And can save a lot of bits.

[Sharc]

Tossing in some --nr can help with random noise, while preserving the texture for the most part. the grain doesn't tend to change as much frame-to-frame, which can be more visually appealing, even if less accurate. And can save a lot of bits.
Out of interest: How does --nr compare with "commonly used" avisynth noise or grain reduction filters? Does it has an advantage, other than being included in the x264 encoder? (I have actually never tried it....)

Edit:
I found this:
http://forum.doom9.org/archive/index.php/t-154628.html
Still valid?

[asarian]

So my questions are: Are the high bitrates from rather bad material really necessary or is x264 fooled somehow? Is there a way to avoid this without switching to constant bitrates? Why is it that x264 behaves like that? Thanks in advance!


Yeah, not getting into the whole fight, but it seems pretty straightforward to me, that when the source material is pretty 'bad' (aka, lotsa grain/noise, etc), that it takes more bitwidth to keep all detail. Kinda like a no-brainer.

In fact, I often use the bitrate (via the output size) as a temporary indicator (during the job) of whether or not I denoised too much (aka, over-denoised: too much loss of detail, leading to sizes that are too small for what you'd expect).

[Loomes]

Yeah, not getting into the whole fight, but it seems pretty straightforward to me, that when the source material is pretty 'bad' (aka, lotsa grain/noise, etc), that it takes more bitwidth to keep all detail. Kinda like a no-brainer.
Besides the fact that it's very impolite to title any question as a "no-brainer" it's not "pretty straightforward" at all, because it's not a matter of logical reasoning but pretty much a matter of knowledge about how encoders work.

When x264 -crf18 gives a 6000kbit result on a 720p movie and it looks identical (to the human eye) with a 1500kbit result from x264 -bitrate 1500, there is no obvious reason at all. There is a big difference between logical reasoning and just not knowing some details about something. Moreover, I would not title "grain" as "detail". Grain is a kind of noise (wanted or not) and gives a certain effect but no detail to the movie at all.

[Stereodude]

Besides the fact that it's very impolite to title any question as a "no-brainer" it's not "pretty straightforward" at all, because it's not a matter of logical reasoning but pretty much a matter of knowledge about how encoders work.

When x264 -crf18 gives a 6000kbit result on a 720p movie and it looks identical (to the human eye) with a 1500kbit result from x264 -bitrate 1500, there is no obvious reason at all. There is a big difference between logical reasoning and just not knowing some details about something. Moreover, I would not title "grain" as "detail". Grain is a kind of noise (wanted or not) and gives a certain effect but no detail to the movie at all.
Insulting and attacking the people you're trying to get to help you tends to poison the well. Just because you don't understand how encoders work doesn't mean the answers are illogical.

There is absolutely no way a x264 -crf18 gives a 6000kbit result on a 720p movie and it looks identical to the human eye with a 1500kbit result from x264 -bitrate 1500 unless you're viewing a very small screen or viewing from quite a distance. Otherwise your chosen CRF is just much too high and you're wasting bits.

What's the problem with a 6Mbit/sec bitrate? Why do you need a 2Mbit/sec bitrate? If you want low bitrates on all your movies use 2-pass VBR or constant bitrate instead of CRF.

[Groucho2004]

Moreover, I would not title "grain" as "detail". Grain is a kind of noise (wanted or not) and gives a certain effect but no detail to the movie at all.
The encoder doesn't care what you call it - detail, a flaw in the source or an artistic effect. It qualifies as very fine detail which requires a lot of bitrate if you want to preserve it.

You can of course use filters to de-noise/de-grain before encoding and/or tune the encoder algorithms to discard some of the noise in order to reduce the bitrate.

[Loomes]

Sorry, Stereodude. I did not insult anyone, I was insulted, obviously. You should read the thread again.

It is very funny but strange that my question is called a "no-brainer" and it seems absolutely normal that some users misread the thread and even get backup for bashing on me for nothing. I can only smile and shake my head on such childish behaviour.

Thanks to anyone who answered politely and gave me some info about this.

[foxyshadis]

You weren't insulted. A no-brainer simply means an easily solved problem, it doesn't imply that you have no brains or that it should be easy for you, just that it's easy for the person saying it. It's a common phrase in English and I've never seen it used as an insult.

[kuchikirukia]

It is very funny but strange that my question is called a "no-brainer" and it seems absolutely normal that some users misread the thread and even get backup for bashing on me for nothing.

For nothing? You thought x264 knew the difference between detail you're interested in and detail you're not. x264 is an encoder, so if grain is a detail of the source you're feeding it, x264 tries to capture it with everything else.

[Loomes]

For nothing? You thought x264 knew the difference between detail you're interested in and detail you're not. x264 is an encoder, so if grain is a detail of the source you're feeding it, x264 tries to capture it with everything else.
Correct, for nothing. Because there is no need for bashing on anyone. By the way: Your post is just repeating what someone else already said, so it's completely useless (besides increasing your post counter +1, right?).

@Mod: Maybe this thread should be deleted or closed.