x265 buffer underflows

[kolak]

Getting buffer underflow errors when encoding CBR with x265 using strict-cbr. Is it a known issue?
Another issue could be fact that I'm forcing I frames with ffmpeg's -force_key_frames "expr:eq(mod(n,50),0)" This may be messing with rate control.
Is there any other way of getting fixed 2 seconds IDR frame distance (and still letting encoder to use I frames inside those 2 seconds)? Using latest ffmpeg.

[poisondeathray]

Getting buffer underflow errors when encoding CBR with x265 using strict-cbr. Is it a known issue?
Another issue could be fact that I'm forcing I frames with ffmpeg's -force_key_frames "expr:eq(mod(n,50),0)" This may be messing with rate control.
Is there any other way of getting fixed 2 seconds IDR frame distance (and still letting encoder to use I frames inside those 2 seconds)? Using latest ffmpeg.

Did you try -g 50 ?

Or -x265-params keyint=50

[kolak]

g or keying itself means max GOP size. It doesn't guarantee you IDR frame at every n*50fps at all (only that no GOP will be bigger than 50).
I need IDR frame at ever n*50 + I frames (to cover scene changes etc.) in between if encoder decides so.

[Sagittaire]

g or keying itself means max GOP size. It doesn't guarantee you IDR frame at every n*50fps at all (only fact that no GOP will be bigger than 50).
I need IDR frame at ever n*50 + I frames (to cover scene changes etc.) in between if encoders decides so.

use multipass perhaps. x265 has problem for size RC estimation.

[kolak]

Looks like 2 pass may be the only option.
Problem is that it cost so much extra time. I don't necessarily have this luxury to waste 2x more time.

[Blue_MiSfit]

Is there any other way of getting fixed 2 seconds IDR frame distance (and still letting encoder to use I frames inside those 2 seconds)


Yep, just set keyint == min-keyint and it will work fine inserting I frames (not IDR) on scene changes inside your GOP.

[Sagittaire]

Looks like 2 pass may be the only option.
Problem is that it cost so much extra time. I don't necessarily have this luxury to waste 2x more time.

well try that:

x265.exe --input C:\ToS_1920x800_xdither.y4m --output ToS-2000-psy.265 --input-res 1920x800 --fps 24000/1000 --bitrate 2000 --strict-cbr --vbv-maxrate 2000 --vbv-bufsize 12000 --pass 1 --slow-firstpass --stats ToS-2000-psy.log --preset veryslow --rd-refine --min-keyint 1 --keyint 120 --ipratio 1.00 --pbratio 1.00 --analysis-save ToS_1000_cbr_analysis.dat --analysis-save-reuse-level 10

x265.exe --input C:\ToS_1920x800_xdither.y4m --output ToS-2000-psy.265 --input-res 1920x800 --fps 24000/1000 --bitrate 2000 --strict-cbr --vbv-maxrate 2000 --vbv-bufsize 12000 --pass 3 --slow-firstpass --stats ToS-2000-psy.log --preset veryslow --rd-refine --min-keyint 1 --keyint 120 --ipratio 1.00 --pbratio 1.00 --analysis-load ToS_1000_cbr_analysis.dat --analysis-load-reuse-level 10

x265.exe --input C:\ToS_1920x800_xdither.y4m --output ToS-2000-psy.265 --input-res 1920x800 --fps 24000/1000 --bitrate 2000 --strict-cbr --vbv-maxrate 2000 --vbv-bufsize 12000 --pass 3 --slow-firstpass --stats ToS-2000-psy.log --preset veryslow --rd-refine --min-keyint 1 --keyint 120 --ipratio 1.00 --pbratio 1.00 --analysis-load ToS_1000_cbr_analysis.dat --analysis-load-reuse-level 10

..... etc etc




Reuse encoding use complete research from first pass and refine Rate Control at really high speed (like fastest preset speed but with veryslow preset quality). And reuse mode will refine and refine and refine again Rate Control in Npass mode certainely with better quality than classic 2 pass RC mode, and with better constrained Rate Control at each pass.

IMO with this particular mode encoding, --strict-cbr will be even useless if you use many pass and you can expect better quality too.

[Sagittaire]

and you can certainely expect more constant overall quality and less underflow with something like that:

--bitrate 1900 --vbv-maxrate 2000 --vbv-bufsize 12000

if your target bitrate is 2000, choose little lower bitrate like 1900 to have better buffer gestion and more constant quality in GOP between I, P and Bframe.

[kolak]

How is this then CBR encoding?
My spec says CBR at 2% tolerance which is bit crazy :)

[Sagittaire]

How is this then CBR encoding?
My spec says CBR at 2% tolerance which is bit crazy :)

certainely usefull for max bitrate limit, but why for min bitrate limit?

[kolak]

Because spec is set by someone who has not much clue about encoding, I assume :)

[kolak]

Yep, just set keyint == min-keyint and it will work fine inserting I frames (not IDR) on scene changes inside your GOP.

This is what I thought but had strange results. Need to check it again as it was possibly due to reading some bad first pass details left after some testing:)

[Sagittaire]

Because spec is set by someone who has not much clue about encoding, I assume :)

Well if you have multiple consecutive pure black frame, encoder will certainely never use high bitrate for that ... ;-)

[kolak]

Yes, this is very possible as content changes a lot and has many dips to blacks and overall is very dark.
Is there a solution for this?

[kolak]

This is what I thought but had strange results. Need to check it again as it was possibly due to reading some bad first pass details left after some testing:)


Strange getting GOPs at 61, 72 ... for keyint=minkeyint=50
(update: it was due to gop-lookahead not been set to 0)

I'm actually not getting underflow errors anymore (at least not so far), so looks like ffmpeg's force_keyframe option messes a bit with rate control in x265.

[benwaggoner]

Strange getting GOPs at 61, 72 ... for keyint=minkeyint=50
(update: it was due to gop-lookahead not been set to 0)
What was your scenario for "gop-lookahead?" I've never figured out what the feature was actually for, versus just changing min-keyint directly.

[kolak]

I did not understand it either, now I think I do.
Let's say you have keyint set to 50 frames. If you set gop-lookahead to eg. 25 frames, then x265 will look those 25 further ahead to see if it may be better to set an IDR frame somewhere there. It may decide that best IDR place will be eg. 10 frames after those 50, so you will end up with 60 frames GOP.
If you do the same, but set gop-lookahead=0 at those 10 frames pass 50 frames x265 will most likely set I frame (not an IDR). This will properly respect 50 frames maximum GOP.
With gop-lookahead your longest GOP is keyint+gop-lookahead.
I may be wrong :)
No idea how it's different than setting keyint=75 :)

--gop-lookahead <integer>
Number of frames for GOP boundary decision lookahead. If a scenecut frame is found within this from the gop boundary set by –keyint, the GOP will be extented until such a point, otherwise the GOP will be terminated as set by –keyint. Default 0.

Range of values: Between 0 and (–rc-lookahead - mini-GOP length)

It is recommended to have –gop-lookahaed less than –min-keyint as scenecuts beyond –min-keyint are already being coded as keyframes.

[Blue_MiSfit]

Ah, so if you do want constant IDR cadence (like for ABR streaming) then you would presumably want to make sure that's disabled! (it is by default)

Seems like a great feature for a satellite / broadcast feed where the main constraint on keyint is latency and channel switch time. Having slight wiggle room is probably acceptable in those cases.

[benwaggoner]

Ah, so if you do want constant IDR cadence (like for ABR streaming) then you would presumably want to make sure that's disabled! (it is by default)

Seems like a great feature for a satellite / broadcast feed where the main constraint on keyint is latency and channel switch time. Having slight wiggle room is probably acceptable in those cases.
So, --keint 50 --gop-lookahead 25 would still have a max GOP duration of 50 frames, but the average would tend to be closer to 50 than 75?

Seems worthy of some testing.

[Blue_MiSfit]

--keint 50 --gop-lookahead 25


My interpretation is that the nominal GOP length would be 50, but that it might be as high as 75 in some cases, if a scenecut is found within that extra 25 frames.

Put another way, starting at frame 0, assuming scene change detection is not triggered from frames 0-49, frame 50 will be an IDR... UNLESS frames 51-74 are flagged by scene change detection, in which case the earliest of those will be the IDR.

At least.. I think I did the math right there :D I always have to think twice about duration vs frame number etc!

[MGarret]

Yes, that's exactly what it is. Some broadcast hw encoders had that feature for years. They have "default" interval that is further "extended" in case of a detected scenecut.

[benwaggoner]

Yes, that's exactly what it is. Some broadcast hw encoders had that feature for years. They have "default" interval that is further "extended" in case of a detected scenecut.
So, same worst case, but shorter average GOP duration. Makes sense for channel-switching optimization. Less so for most other scenarios.

[kolak]

So, --keint 50 --gop-lookahead 25 would still have a max GOP duration of 50 frames, but the average would tend to be closer to 50 than 75?

Seems worthy of some testing.

Max would be 75.

[kolak]

My interpretation is that the nominal GOP length would be 50, but that it might be as high as 75 in some cases, if a scenecut is found within that extra 25 frames.

Put another way, starting at frame 0, assuming scene change detection is not triggered from frames 0-49, frame 50 will be an IDR... UNLESS frames 51-74 are flagged by scene change detection, in which case the earliest of those will be the IDR.

At least.. I think I did the math right there :D I always have to think twice about duration vs frame number etc!

This is how it seems to work.