x264 possibly not safe for 4K UHD encoding?

[LigH]

Last night I captured a long series of night sky photos, hoping to catch some shooting stars. This worked out pretty well, so I tried to turn this photo series into a time lapse video, using ImageSource and several nested Layers with different blend modes in AviSynth+.

When I watched the encoded material, I noticed heavy glitches for brief intervals in some areas close to the bottom. To ensure that the reason was not already in AviSynth+, I split my complex script into parts and saved intermediate results with Ut Video, reading these AVIs in following scripts. I encoded with x264 to MP4 both in VirtualDub2 and with the CLI encoder. The results differed in details, but in general, they all contained glitches now and then.

Encoding the same video with x265 appeared to be safe, I could not spot any glitches.

So I guess that the motion detection and estimation in x264 may have been fooled by the material. I noticed that it was usually provoked in one scene by a passing-by shooting star or whatever appeared in this frame. The glitches are always in the bottom part of the video.

I am not sure how to provide test material if anyone would be interested, it takes up several GB. I guess if bit accuracy is not necessary, a low CRF x265 or AV1 encode might be suitable as compromise source...

[Sharc]

Maybe you want to try and compare with an older release of x264. Perhaps you found a bug?

[LigH]

I will, as time permits...

[Blue_MiSfit]

Interesting, are you certain this is not a decoding issue, like a hardware decoder going beyond its limits?

[FranceBB]

Does it happen with low --ref values as well? I remember several years ago some people were complaining about motion estimation not working correctly on a video recorded at very high altitude containing several different clouds which was encoded with --ref 16.
That was the only time in which I ever heard of something similar, but it could be.
By the way, for general purpose files, I have encoded 4K 4:2:2 planar 10bit files in AVC Intra Class 300 (i.e 500 Mbit/s) with x264 and I never had such a problem, so it's probably not resolution-specific but rather content-specific.

As a very last question: does it happen with both x264 8bit and 10bit? And which version were you using?

[LigH]

Interim findings:


happens with a range of x264 builds
happens for 8 bpc encodes with slower preset (the only explicit parameters are --crf 15 --preset slower)
does not happen for 10 bpc encodes with slower preset
does not happen for x265 with fast preset


Thank you for your testing suggestions, I will add such parameters too.

Next scheduled test series: x264 8 bpc with all the presets.

[LigH]

Interesting, are you certain this is not a decoding issue, like a hardware decoder going beyond its limits?

:eek: Interesting idea. I opened the encoding result in VirtualDub2 and could not notice any decoding errors. I'll see how to configure LAV Filters in MPC-HC to check this again.

PS: You nailed it. It breaks in DXVA2 and in CUVID mode but looks fine without HW (avcodec). GPU is an Nvidia 1660 Super.

[benwaggoner]

And also a good point about x264 by default not enforcing Profile @ Level requirements historically. It was a huge problem back in the day, because you'd get content flagged as High Level 4.1 which should only have 4 reference frames but it would have 16 because --preset placebo was used. Which is actually level 5.1 or something. Which is why Xbox 360 media apps used to be limited to 720p, because it could decode 720p with 16 reference frames.

x265 does contrain parameters to the specified profile and level by default, so it never became an issue in HEVC.

[Blue_MiSfit]

Nice! Glad you found the issue. A good example of hardware being a PITA :)

You should be able to encode 4k H.264 while maintaining hardware decode compatibility if you use VBV and cap reference frames etc.

https://en.wikipedia.org/wiki/Advanced_Video_Coding#Levels

So, I'd look at using Level 5.1, and setting vbv-maxrate to no more than 240 Mbps. I'd set vbv-bufsize to no more than 240 Mbits, and set ref to no more than 5.

[LigH]

Thank you for the detailed recommendations.

I also discussed in the LAV Filters thread (https://forum.doom9.org/showthread.php?p=1921153#post1921153) whether a pessimistic fallback to software decoding might be advisable when the video's P@L flags exceed HW decoder constraints.