Video stuttering when skipping ahead (converted with ffmpeg)

[jriker1]

I have been encoding 4K HDR10 video with similar ffmpeg settings as below, just tweaked in theory for 1080p for some older footage I need to convert. I play most of my content thru the NVidia Shield and never have issues. Until now. I have a H.264 AVC video that I am converting to H.265 and it plays fine on VLC on Windows, but when I play this one and a couple other similar videos that are 1080p on the Shield, they seem to play OK, until I jump ahead. The audio keeps working, but the video will either not move for a while or will but the video will stutter. And keep doing that. If I jump back 10 seconds or so it will start playing normally again. Never saw this before. Below is the command I'm running essentially:

ffmpeg.exe -i <input_video>.mkv -c:v libx265 -x265-params level=51:high-tier=1:repeat-headers=1:colorprim=bt709:transfer=bt709:colormatrix=bt709:crf=13:chromaloc=0:no-sao=1:info=0:range=limited:vbv-maxrate=17500:vbv-bufsize=17500 -vf crop=1920:816:0:0 -preset slower -pix_fmt yuv420p10le -sn -an <output_video>.hevc

So when that had this issue tried extracting all the frames as images and then ran the below but same deal:

ffmpeg.exe -framerate 24000/1001 -f image2 -i img%09d.bmp -c:v libx265 -x265-params level=51:high-tier=1:repeat-headers=1:colorprim=bt709:transfer=bt709:colormatrix=bt709:crf=13:chromaloc=0:no-sao=1:info=0:range=limited:vbv-maxrate=17500:vbv-bufsize=17500 -vf crop=1920:816:0:0 -preset slower -pix_fmt yuv420p10le -sn -an <output_video>.hevc

Any thoughts on what's going on?

Thanks.

JR

[poisondeathray]

What are you using to multiplex streams into the container ?

Shorter max keyframe interval might help with seeking

[jriker1]

What are you using to multiplex streams into the container ?

Shorter max keyframe interval might help with seeking

Thanks for the reply. Will try a shorter max keyframe interval. With multiplexing using mkvtoolnix latest version.

[jriker1]

By the way, my source was 8 bit so changed the pix_fmt but that didn't help but still makes sense to do. What would the thought be if...

1. The converted video put in an mkv container plays super fast and no hickups by itself. I can't even fast forward thru the footage with it immediately playing. No audio of course.
2. If I add in the DTS audio file untouched, and the ass subtitles which are text and always make them, it starts up again.

Should I assume something is wrong with the DTS file?

[benwaggoner]

If using x265, --no-open-gop can improve random access with some players.

--repeat-headers and maybe --aud could potentially help as well.

[jriker1]

OK I have --repeat-headers but will try your other suggestions. See you in a couple days when it's done. :)

Note the video is fine as encoded when I have no audio in the MKV. When I add the audio then it starts playing odd various times when I jump around. Audio is DTS XLL on these issue items if that helps at all. Not sure if the NVidia Shield has issue with this audio file though it plays fine. Even while the video is stuttering the audio keeps chugging along.

[benwaggoner]

OK I have --repeat-headers but will try your other suggestions. See you in a couple days when it's done. :)

Note the video is fine as encoded when I have no audio in the MKV. When I add the audio then it starts playing odd various times when I jump around. Audio is DTS XLL on these issue items if that helps at all. Not sure if the NVidia Shield has issue with this audio file though it plays fine. Even while the video is stuttering the audio keeps chugging along.
If you use AAC audio, do you have the same problems?

[jriker1]

If you use AAC audio, do you have the same problems?

I switched the DTS Master to AAC. Obviously not the same bitrate but no the stuttering couldn't be reproduced. Still had the lag in the video starting at times when jumping ahead a bunch but no stuttering playback. Thinking this must be the player. On a PC with VLC it's fine. One a NVidia Shield Pro 2019 on VLC it's fine. On Emby this happens. All are direct playing, no transcoding.

[benwaggoner]

I switched the DTS Master to AAC. Obviously not the same bitrate but no the stuttering couldn't be reproduced. Still had the lag in the video starting at times when jumping ahead a bunch but no stuttering playback. Thinking this must be the player. On a PC with VLC it's fine. One a NVidia Shield Pro 2019 on VLC it's fine. On Emby this happens. All are direct playing, no transcoding.
Could well be a player-specific issue with DTS, or a muxing issue.

Lag in random access is most common with very long GOP sizes. The default --keyint 250 can require decoding a lot of frames to be able to display an arbitrary one near the end of the GOP, as all frames that frame references need to be decodes, and all the frames those frames reference.

If that's it, lowering --keyint should proportionally reduce worst-case random access time.

[jriker1]

Got derailed with my other testing so running now. Added in keyint=24:min-keyint=24.

Do we think this would help the playback issue when jumping around though pretty convinced it's due to the audio. Also there was a mention about how it was multiplexed. Would mkvtoolnix be an issue multiplexing the content?

[poisondeathray]

You can try a transport stream instead of mkv to see if it makes a difference

[benwaggoner]

Got derailed with my other testing so running now. Added in keyint=24:min-keyint=24.

Do we think this would help the playback issue when jumping around though pretty convinced it's due to the audio. Also there was a mention about how it was multiplexed. Would mkvtoolnix be an issue multiplexing the content?
Raising min-keyint could hurt average seek time a little, and definitely can impact quality, as keyframes take a lot more bits for a given quality. I suggest leaving min-keyint alone, so it can adaptively put the keyframes in their optimal place (like hard edits).

[excellentswordfight]

Raising min-keyint could hurt average seek time a little, and definitely can impact quality, as keyframes take a lot more bits for a given quality. I suggest leaving min-keyint alone, so it can adaptively put the keyframes in their optimal place (like hard edits).
min-keyint is set to 23 automatically for 23,976 content by default so he is not really changing anything. And I-slices can still be placed more frequent, they will just not be IDR-frames.

[jriker1]

You can try a transport stream instead of mkv to see if it makes a difference

Encoding with the change however transport stream was a total bust. With or without the audio the video was hanging so long between jumps thru the footage I gave up. Audio would play but 30 seconds later video was still sitting (when there was audio included). Going back to the MKV version with no audio everything worked fine or seemed to. What's funny is my 4k HDR content all loads fast, no lag, no video taking forever to start and the file size is huge compared to this. Below is an example of what I do with my 4K HDR content:

ffmpeg.exe -i "<source>.mkv" -c:v libx265 -x265-params level=51:hdr10=1:hdr-opt=1:high-tier=1:repeat-headers=1:colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:master-display=G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,0):crf=13:chromaloc=2:no-sao=1:info=0:range=limited:vbv-maxrate=17500:vbv-bufsize=17500 -vf crop=3840:1600:0:280 -preset slower -pix_fmt yuv420p10le -sn -an "<output>.hevc"

[benwaggoner]

OH! I bet High-tier is the problem. Lots of consumer decoders don't handle it well, and may go into a weird mode just from that flag. Your bitrates caps are well below the Main Tier thresholds, so just use Main. With a CRF of 13 and a 17.5 Mbps max bitrate, you're probably getting something pretty close to CBR.

Level 5.1, which you are using, can go up to 40 Mbps, so there's no reason to use High Tier unless you want to go above that. For 17.5 Mbps, you might as well use Level 5.0 (max of 25 Mbps), which might boost compatibility or performance on some older devices.

[excellentswordfight]

OH! I bet High-tier is the problem. Lots of consumer decoders don't handle it well, and may go into a weird mode just from that flag. Your bitrates caps are well below the Main Tier thresholds, so just use Main. With a CRF of 13 and a 17.5 Mbps max bitrate, you're probably getting something pretty close to CBR.

Level 5.1, which you are using, can go up to 40 Mbps, so there's no reason to use High Tier unless you want to go above that. For 17.5 Mbps, you might as well use Level 5.0 (max of 25 Mbps), which might boost compatibility or performance on some older devices.
I noticed high tier as well, but it feels a bit odd that it would create issues if 2160p works with lvl5.1 flagged as high tier, and especially since rather low vbv limits are set.

But yes having that low crf value and vbv doesnt seems like that great, but that should be more in terms of encoding quality then decoding compat, right?

Was thinking that it was related to ref-frames, as you are using a low resolution for that level, with a low preset I thought that x265 would behave like x264 and use ref 16. But it doesnt looks like x265 go above 5 with that command line...

But it might be something to try out. What happens if you remove the vbv settings and set high-tier=0, and use level 4.1 (that would automatically set vbv to 20Mbps)?

[benwaggoner]

I noticed high tier as well, but it feels a bit odd that it would create issues if 2160p works with lvl5.1 flagged as high tier, and especially since rather low vbv limits are set.
I'd hope it wouldn't, but some devices have pretty prescriptive decoders to reduce risk of playback errors and security threats. There's certainly no reason to use High Tier 5.1 with Main Tier 5.0 VBV limits.

But yes having that low crf value and vbv doesnt seems like that great, but that should be more in terms of encoding quality then decoding compat, right?
Correct.

Was thinking that it was related to ref-frames, as you are using a low resolution for that level, with a low preset I thought that x265 would behave like x264 and use ref 16. But it doesnt looks like x265 go above 5 with that command line...
Default --ref is based on --preset, with a maximum of 5 for --preset veryslow. HEVC itself has an 8 ref max IIRC.

It's been a big problem that x264 historically didn't enforce Profile and Level constraints, resulting in out of spec files that had big playback issues with hardware decoders. x265 did the right thing here, where you have to use an override to violate spec compliance.

But it might be something to try out. What happens if you remove the vbv settings and set high-tier=0, and use level 4.1 (that would automatically set vbv to 20Mbps)?
Level 4.1 can't do 2160p, so it would just error out. Main Tier Level 5.0 is the lowest that can support 2160p, and then only up to 32 fps IIRC. If you set --level 5.0, vbv-bufsize and vbv-maxrate should be automatically set to 25000 unless otherwise specified.

Unlike classic x264, where setting --level would just set that flag in the bitstream but not change any defaults or validate that settings were legal.

[excellentswordfight]

Level 4.1 can't do 2160p, so it would just error out. Main Tier Level 5.0 is the lowest that can support 2160p, and then only up to 32 fps IIRC. If you set --level 5.0, vbv-bufsize and vbv-maxrate should be automatically set to 25000 unless otherwise specified..
TS has issues when trying to encode 1080p, 2160p works fine apperently. Or did i miss something?

[jriker1]

Not done encoding yet, however I think ffmpeg is ignoring the high tier setting. At least this is part of the output I copied from the screen:

Press [q] to stop, [?] for help
x265 [info]: HEVC encoder version 3.5+21-8a15ef537
x265 [info]: build info [Windows][GCC 11.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-5.1 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices

[benwaggoner]

TS has issues when trying to encode 1080p, 2160p works fine apperently. Or did i miss something?
TS? Transport stream?

[excellentswordfight]

Not done encoding yet, however I think ffmpeg is ignoring the high tier setting. At least this is part of the output I copied from the screen:

Press [q] to stop, [?] for help
x265 [info]: HEVC encoder version 3.5+21-8a15ef537
x265 [info]: build info [Windows][GCC 11.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-5.1 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices
If you still manually set vbv limits that are bellow Main tier for that level It might do that i guess

TS? Transport stream?
Topic/thread starter 😀

[jriker1]

OK so conversion done. So changing the keyint=24 made it track real fast audio and video, but when I'm jumping quickly thru the footage still had the issue where when I took my hand off the jump forward button the playback was jumpy. Even stopping the video and playing still did it. Going back 10 seconds or so played thru that area fine then. Ugh. Oh and the footage even though the call was:

ffmpeg.exe -i <input>.mkv -c:v libx265 -x265-params level=51:high-tier=1:repeat-headers=1:colorprim=bt709:transfer=bt709:colormatrix=bt709:crf=13:chromaloc=0:no-sao=1:info=0:range=limited:vbv-maxrate=17500:vbv-bufsize=17500:keyint=24 -vf crop=1920:816:0:2 -preset slower -pix_fmt yuv420p -sn -an <output>.hevc

The footage itself was 1920*816 (2.35:1), at 23.976 (24000/1001) FPS, HEVC (Main@L5.1@Main)

With the DTS Master audio - stutters
With the DTS Core extracted - Fine now with the keyint
Converted to AC3 audio - Fine

Guess I either leave the audio I guess and don't fast forward, switch to DTS Core or get the Emby folks to look into issues with their player and DTS XLL audio. Think the core sounds quieter than the master audio but may be me.

[benwaggoner]

Sounds likely a player-specific issue. Are you using --no-open-gop?

[jriker1]

Sounds likely a player-specific issue. Are you using --no-open-gop?

No, but I can do another conversion and try it. Only concern I'm having is do we think based on info so far, that the video is an issue?

[excellentswordfight]

OK so conversion done. So changing the keyint=24 made it track real fast audio and video, but when I'm jumping quickly thru the footage still had the issue where when I took my hand off the jump forward button the playback was jumpy. Even stopping the video and playing still did it. Going back 10 seconds or so played thru that area fine then. Ugh. Oh and the footage even though the call was:

ffmpeg.exe -i <input>.mkv -c:v libx265 -x265-params level=51:high-tier=1:repeat-headers=1:colorprim=bt709:transfer=bt709:colormatrix=bt709:crf=13:chromaloc=0:no-sao=1:info=0:range=limited:vbv-maxrate=17500:vbv-bufsize=17500:keyint=24 -vf crop=1920:816:0:2 -preset slower -pix_fmt yuv420p -sn -an <output>.hevc

The footage itself was 1920*816 (2.35:1), at 23.976 (24000/1001) FPS, HEVC (Main@L5.1@Main)

With the DTS Master audio - stutters
With the DTS Core extracted - Fine now with the keyint
Converted to AC3 audio - Fine

Guess I either leave the audio I guess and don't fast forward, switch to DTS Core or get the Emby folks to look into issues with their player and DTS XLL audio. Think the core sounds quieter than the master audio but may be me.
Just courious, why are you using souch low crf with very aggresive vbv limits for that crf range? And using a level that way above that resolution and vbv restrictions?

From the information you har given us it seems to be audio relatated, but i’ve seen x265 to have some vbv issues in the past. What happens if you use:


ffmpeg.exe -i <input>.mkv -c:v libx265 -x265-params level=41:vbv-maxrate=20000:vbv-bufsize=20000:repeat-headers=1:colorprim=bt709:transfer=bt709:colormatrix=bt709:crf=18:chromaloc=0:no-sao=1:info=0:range=limited:keyint=96 -vf crop=1920:816:0:2 -preset slower -pix_fmt yuv420p -sn -an <output>.hevc

If the issue persist that at least should rule out bitrate/vbv/level/tier issues.

[jriker1]

Just courious, why are you using souch low crf with very aggresive vbv limits for that crf range? And using a level that way above that resolution and vbv restrictions?

Is that limited for 1080p? When I used to create my videos thru Premiere Pro and a difference here, was using NeatVideo to get rid of some of the noise, I was using 8,000 for a CBR and 10,000 for shows I liked a lot. I was trying to reproduce CBR with ffmpeg but it's my understanding those settings set thte upper limit to how high a bitrate the encoder can use. This would be a lot higher than what I used to do. I think. And picture seems very clean.

[Asmodian]

Don't use VBV to try to get CBR, it isn't meant for that and it isn't good at it.

Do you really need to cap the max rate? Pick a higher crf without VBV (or the normal VBV for the level) for better quality at the same size. If you want to always get the same size use 2-pass and target a bitrate. If you want a flatter bitrate distribution for some reason use qComp (https://x265.readthedocs.io/en/2.5/cli.html).

VBV is designed to avoid buffering issues when streaming over a network or off of slow storage. It is not designed to cutoff the bitrate of all the high motion scenes. The rate control works better if you let it fluctuate as it desires, with VBV only stepping in rarely to keep extreme bitrate spikes within the limits of the hardware.

[benwaggoner]

Is that limited for 1080p? When I used to create my videos thru Premiere Pro and a difference here, was using NeatVideo to get rid of some of the noise, I was using 8,000 for a CBR and 10,000 for shows I liked a lot. I was trying to reproduce CBR with ffmpeg but it's my understanding those settings set thte upper limit to how high a bitrate the encoder can use. This would be a lot higher than what I used to do. I think. And picture seems very clean.
Unless you're trying to do real-time streaming over a constrained bitrate, CBR really doesn't make sense. Compared to VBR, at the same file size, CBR will look worse in difficult sections while wasting bits in easier sections.

For stuff that'll be played of a drive, VBR (either 2-pass VBR or using CRF) will be flat-out better most of the time.

[benwaggoner]

Don't use VBV to try to get CBR, it isn't meant for that and it isn't good at it.

Do you really need to cap the max rate? Pick a higher crf without VBV (or the normal VBV for the level) for better quality at the same size. If you want to always get the same size use 2-pass and target a bitrate. If you want a flatter bitrate distribution for some reason use qComp (https://x265.readthedocs.io/en/2.5/cli.html).

VBV is designed to avoid buffering issues when streaming over a network or off of slow storage. It is not designed to cutoff the bitrate of all the high motion scenes. The rate control works better if you let it fluctuate as it desires, with VBV only stepping in rarely to keep extreme bitrate spikes within the limits of the hardware.
VBV is required to meet profile @ level decoder requirements, so it is required for compatibility with lots of hardware decoders.

x265 generally picks a sane default level based on the input, and then constrains VBV appropriately for that, so setting VBV isn't needed in a lot of cases. But we want VBV to be used by the encoder in almost all cases.

Also, if you know you have a better decoder available, setting that and VBV can improve quality. For example, encoding 2160p24 would default to Level 5 and thus a 25 Mbps max VBV. But if targeting a Level 5.1 decoder, max VBV can be raised to 40 Mbps, which can help quality with high complexity or grain.

[excellentswordfight]

VBV is required to meet profile @ level decoder requirements, so it is required for compatibility with lots of hardware decoders.

x265 generally picks a sane default level based on the input, and then constrains VBV appropriately for that, so setting VBV isn't needed in a lot of cases. But we want VBV to be used by the encoder in almost all cases.

Also, if you know you have a better decoder available, setting that and VBV can improve quality. For example, encoding 2160p24 would default to Level 5 and thus a 25 Mbps max VBV. But if targeting a Level 5.1 decoder, max VBV can be raised to 40 Mbps, which can help quality with high complexity or grain.
I mentioned this to you before, x265 does not behave like that by default. if level and vbv is unspecified x265 does not use vbv, its only when level is manually set it will use vbv and when level is set it will prioritize high tier if tier is unspecified.

It's also in the docs

--level-idc
...
Beware, specifying a decoder level will force the encoder to enable VBV

--vbv-bufsize
...
Default 0 (vbv disabled)

[benwaggoner]

I mentioned this to you before, x265 does not behave like that by default. if level and vbv is unspecified x265 does not use vbv, its only when level is manually set it will use vbv and when level is set it will prioritize high tier if tier is unspecified.
Right. I keep forgetting because I never, ever do an encode without setting --level-idc. I like to know where I can trust it to decode reliably to reduce risk of encoder and decoder issues getting conflated.