"Bounded" CRF Encoding with x264

[SomeJoe]

Hi Gents,

I have a question regarding a method I'm trying to use to get a specific rate control profile on my x264 encodes.

I like the idea of CRF encoding -- one pass so that it's marginally faster, and a known quality. The problem is that there are many instances when a CRF encode will generate a file with a final average bitrate that exceeds what I want.

As an example, I can do a CRF 19 encode of 1920x1080 material and sometimes get a final average bitrate between 10.0 and 12.0 Mbps. This is right in the range where I want. But sometimes, a CRF 19 encode of different 1920x1080 material can have a bitrate approaching 14.0 Mbps or sometimes higher.

I really don't see the value in allowing the average bitrate to go much above say 12.5 Mbps. I would rather have the quality begin to slightly go down at that point.

I've been experimenting with what I call a "bounded" CRF encode, which uses the VBV buffer settings to cap the maximum bitrate. The idea is to use the keyint, rc-lookahead, vbv-maxrate, and vbv-bufsize settings to force x264 to lower the quality if the bitrate begins to get outrageous.

As an example, I use the following settings to allow the bitrate to exceed 12.5 Mbps for short 2 second bursts:

--crf 19 --keyint 48 --rc-lookahead 48 --vbv-bufsize 25000 --vbv-maxrate 12500

This indeed does work for rate control, and prevents the overall average bitrate from exceeding 12.5 Mbps.

However, there are sometimes scenes in some material that exceed 2 seconds that need a higher average bitrate. As an example, the movie Coming to America has a scene about 15 minutes into the film of dancers that are wearing elaborate costumes that include multicolored, rapidly-moving feathers. This entire scene is about 90 seconds long, and clearly needs an average bitrate well in excess of 12.5 Mbps to maintain quality. With my bounded bitrate method, the quality naturally suffers because the VBV buffer quickly runs out of available bits.

I can only increase the VBV buffer size so much before certain hardware devices will have problems. Exceeding around 40000 will start to cause issues with a Blu-Ray player or a Popcorn Hour. Plus, if I do that I may also need to increase rc-lookahead which will increase encoding time.

Is the only way around this issue to increase the VBV max rate or go back to a 2-pass encode? If I increase the VBV max rate, then certain footage may end up with high average bitrates for the entire file. Going back to a 2-pass encode means no advantage of the known quality or marginally faster encode of CRF.

[Dark Shikari]

There's a far better way to do this.

Just run the first pass in CRF mode, and run the second pass using the bitrate from the first pass. This means, of course, that you can change the bitrate if it's too high.

Example A:

First pass resulting bitrate: 6732kbps
Max bitrate we'll accept: 10000kbps
Second pass bitrate: 6732kbps

Example B:

First pass resulting bitrate: 15731kbps
Max bitrate we'll accept: 10000kbps
Second pass bitrate: 10000kbps

Another similar option is to run the first pass with --slow-firstpass, and only run the second pass if the first pass bitrate is too high.

[SomeJoe]

That indeed is probably a good way to do it.

However, is there a good way to make the first CRF pass run faster than the second pass? For example, if I normally use --preset slow for the second pass, can I use --preset fast for the first CRF pass and still have the bitrate be valid to compare to my maximum?

[Dark Shikari]

That indeed is probably a good way to do it.

However, is there a good way to make the first CRF pass run faster than the second pass? For example, if I normally use --preset slow for the second pass, can I use --preset fast for the first CRF pass and still have the bitrate be valid to compare to my maximum?x264 already does this automatically when you use --pass 1.

[SomeJoe]

Ahh, so I run pass 1 with

--crf 19 --pass 1

which will run quickly and give me a bitrate, and also generate a stats file.

Now I run pass 2 with

-- pass 2 --bitrate xxxx --preset <whatever>

where xxxx is the appropriate bitrate (either the one that the CRF pass gave me if its low enough, or my maximum), and use the stats file that was generated.


Yes, this sounds excellent. It may not save me any encoding time over a standard 2-pass bitrate encode, but it will allow me to reduce the file size if my maximum bitrate is not required.

[detmek]

Another similar option is to run the first pass with --slow-firstpass, and only run the second pass if the first pass bitrate is too high. - This is your best option.

If you want to run 2-pass encode, just use --bitrate xxxx --pass 1 --preset xxx and then --bitrate xxxx --pass 2 --preset xxx. This way you will get fast first pass (no need to use --preset fast or veryfast as x264 automaticly lowers settings if you don't put --slow-firstpass in command line for first pass)

[SomeJoe]

- This is your best option.

If you want to run 2-pass encode, just use --bitrate xxxx --pass 1 --preset xxx and then --bitrate xxxx --pass 2 --preset xxx. This way you will get fast first pass (no need to use --preset fast or veryfast as x264 automaticly lowers settings if you don't put --slow-firstpass in command line for first pass)

The problem with the slow first pass is that the encoding time would go up significantly in the cases where a 2nd pass is required, which is many of them.

Using a straight bitrate 2-pass mode wouldn't give me the opportunity to reduce the bitrate/filesize if the content doesn't need my maximum bitrate.

[detmek]

I understand. There is another way. Not as accurate as your but faster.
Avisynth has function SelectRangeEvery(clip, int "every", int "length", int "offset", bool "audio").
For 25fps video, SelectRangeEvery(500, 25) will pick up 1 sec sample on every 500 frames (20 sec) ie. 5% of your video.
You encode this sample, check the final bitrate and asume that will be final bitrate od the video and change CRF to get desired bitrate.

[SomeJoe]

I did some tests and using "--pass 1 --crf 19" for the first pass results in a bitrate report that isn't very accurate.

I've instead tested using "--crf 19 --slow-firstpass --preset fast", and the bitrate here is closer to the actual bitrate that "--crf 19 --preset slow" would give. This pass is slightly slower than the standard 1st pass settings that --pass 1 gives you, but I like the more accurate bitrate.

So, it looks like I will try the following with some more extensive testing:

Pass 1:

--pass 1 --slow-firstpass --preset fast --crf 19

Pass 2:

--pass 2 --preset slow --bitrate xxxx

[Manao]

You also have to ask yourself why you consider more than xxx mbps too much for a movie. You seem to be interested in an average bitrate over plenty of movies. So you should have issues with single movies having a high bitrate. The chances are, other movies will have a bitrate low enough to compensate. Imho, in such cases, you ought to target constant quality (and thus CRF) over everything else.

[buzzqw]

There's a far better way to do this.

Just run the first pass in CRF mode, and run the second pass using the bitrate from the first pass. This means, of course, that you can change the bitrate if it's too high.

Example A:

First pass resulting bitrate: 6732kbps
Max bitrate we'll accept: 10000kbps
Second pass bitrate: 6732kbps

Example B:

First pass resulting bitrate: 15731kbps
Max bitrate we'll accept: 10000kbps
Second pass bitrate: 10000kbps

HDC offers this kind of encoding :)

(just set "Allow CRF + 1 pass encoding" to 1 in hdconverttox.ini)

BHH

[Sharc]

Originally Posted by Dark Shikari
There's a far better way to do this.

Just run the first pass in CRF mode, and run the second pass using the bitrate from the first pass. This means, of course, that you can change the bitrate if it's too high.

Example A:

First pass resulting bitrate: 6732kbps
Max bitrate we'll accept: 10000kbps
Second pass bitrate: 6732kbps

Example B:

First pass resulting bitrate: 15731kbps
Max bitrate we'll accept: 10000kbps
Second pass bitrate: 10000kbps

HDC offers this kind of encoding :)

(just set "Allow CRF + 1 pass encoding" to 1 in hdconverttox.ini)

@buzzqw
Yes it takes the bitrate of the 1st pass (crf) for pass2, but
where can I set the acceptable max. bitrate (10000kbps in DS's examples)? Could perhaps the value under 'Manually set bitrate' be used for this purpose?

[buzzqw]

where can I set the acceptable max. bitrate (10000kbps in DS's examples)? Could perhaps the value under 'Manually set bitrate' be used for this purpose?

in profile

you should put the --vbv-maxrate in hdconverttox_profile.txt row you want to use
(just duplicate and edit the line..)

BHH

[SomeJoe]

You also have to ask yourself why you consider more than xxx mbps too much for a movie. You seem to be interested in an average bitrate over plenty of movies. So you should have issues with single movies having a high bitrate. The chances are, other movies will have a bitrate low enough to compensate. Imho, in such cases, you ought to target constant quality (and thus CRF) over everything else.

Well, this is kind of why I was thinking of these methods. I would like to target a constant quality and use CRF mode exclusively, but there's times when that comes up with an outrageous bitrate.

Most 720x480 material I've seen will end up between 1.8-2.2 Mbps at CRF 19. But occasionally I see one that comes up over 3.0 Mbps. That can result in a fairly large file for what is usually substandard SD material anyway (e.g. has a lot of film damage or is excessively grainy).

Sometimes on grainy film you can bring down the average bitrate by using a denoiser like FFT3DGPU, but any denoiser can leave its own artifacts.

With HD material, this is generally not as much of a problem because HD material tends to be a lot cleaner and the bitrates tend to be more consistent when using CRF mode.

I'm not sure if this method or any variations of it is precisely what I'm looking for (or even necessary), but experimenting with it is quite interesting.

[Sharc]

in profile

you should put the --vbv-maxrate in hdconverttox_profile.txt row you want to use
(just duplicate and edit the line..)

Hmmm...., yes, but I was thinking about specifying a bitrate which at the end prevents that a certain file size is exceeded, e.g. in order to fit onto a DVD5. So the 2-pass (HDC's crf + 1pass) would actually produce a file size which would never exceed a maximum specified size (or bitrate, respectively).
(But never mind, I would normally just use the standard 2-pass mode for filesize constrained encodes).

[buzzqw]

so.. x264 will need a --dvd5 switch ;)

BHH

[Manao]

So your rational behind lowering the bitrate in such cases is that the source material is of a poor quality. That's understandable. But then, why don't you raise the CRF preventively, instead of waiting to see if the bitrate gets too high. That way, at least, you keep treating all your videos of equal quality in the same way.

[Sharc]

so.. x264 will need a --dvd5 switch
Not necessarily ...
HDC could just check in its (CRF+1-pass) mode:
if bitrate_from_pass1 > maxbitrate then bitrate_pass2=maxbitrate else bitrate_pass2=bitrate_from_pass1).

'maxbitrate' could be the value which I enter in the HDC Video Tab under 'Manually Set Bitrate'.
Just an idea following DS's examples A and B. Maybe not worth the pain of implementing in HDC though......

[buzzqw]

Just an idea following DS's examples A and B. Maybe not worth the pain of implementing in HDC though......

... not worth the try for me

BHH

[Sharc]

So your rational behind lowering the bitrate in such cases is that the source material is of a poor quality. That's understandable. But then, why don't you raise the CRF preventively, instead of waiting to see if the bitrate gets too high. That way, at least, you keep treating all your videos of equal quality in the same way.
Well, yes. Another rationale could be to just capping the bitrate(filesize), given the fact that crf does not truly stand for a certain quality, although it's often seen as such.
So one could be typically happy with a crf 20, but wants to make sure that even for a high motion/complex or long source the file size (or bitrate) must not exceed a maximum. If it would consume less space (at crf 20) it's always fine e.g. on a memory stick, but not more than a maximum (e.g. not exceeding the capacity of the medium).

Added:
Maybe an even more interesting strategy is DS's second proposal:
- Run a 1st pass in crf mode with --slow-firstpass.
- If the bitrate (filesize) is below the target, you're done; otherwise run a 2nd pass with the final (target) bitrate.

[SomeJoe]

So your rational behind lowering the bitrate in such cases is that the source material is of a poor quality. That's understandable. But then, why don't you raise the CRF preventively, instead of waiting to see if the bitrate gets too high. That way, at least, you keep treating all your videos of equal quality in the same way.

That is true, if one has enough experience to be able to choose a good CRF value beforehand.

I, unfortunately, do not. I have not encoded enough to be able to look at some source material and say "OK, this one should be fine at CRF 19, but this other one I should go with CRF 21". Actually, if I had that ability, I wouldn't need CRF at all, I could just choose the proper bitrate to begin with.

In those cases I would frequently guess wrong, and end up encoding twice anyway. Which is what most of these methods we're discussing do. :)

[Manao]

I, unfortunately, do not. I have not encoded enough to be able to look at some source material and say "OK, this one should be fine at CRF 19, but this other one I should go with CRF 21". Actually, if I had that ability, I wouldn't need CRF at all, I could just choose the proper bitrate to begin with.No, you're thinking backward here. Firstly, CRF - bitrate relationship is in no (direct) way linked to quality - bitrate relationship. Secondly, even if it was, knowing the relationship quality - bitrate wouldn't help you, since your problem is that you don't actually know which quality you want (when the source already has a bad quality). If you did, you'd be able to choose the proper CRF, without even caring for the bitrate.

So what I advise you to do is to take some short samples of those low quality videos, and encode them at different CRF until you get the quality that you want. Then, whenever you deem the source to have a low quality, use that CRF instead of CRF 19.

[SomeJoe]

So what I advise you to do is to take some short samples of those low quality videos, and encode them at different CRF until you get the quality that you want. Then, whenever you deem the source to have a low quality, use that CRF instead of CRF 19.

I will try that (along with several other good ideas I've received here) and experiment with them and see what I feel works best for me.

I appreciate all your input, and of the others who've offered their ideas in this thread.

:thanks:

[jordisound]

I assume CRF 19 gives me very good quality. I've no limitations for final bitrate or size
Is better a 2 pass encoding than single pass CRF encoding?

for exemple, I run a single pass CRF 19. the final bitrate is 3500
I run 1st and 2nd pass with target 3500.
the 2 pass method gives me better quality or is not worth it? (extra time encoding)

[nm]

for exemple, I run a single pass CRF 19. the final bitrate is 3500
I run 1st and 2nd pass with target 3500.
the 2 pass method gives me better quality or is not worth it? (extra time encoding)

The 2-pass encode gives the same quality, so it's definitely not worth it.