Starange glitch in x265 encoded video

[hellgauss]

I encoded a video file and I obtain a strange glitch using x265. This is the 'broken' part (7 seconds, the glitch is in the last second), extracted with virtualdub2 in direct stream copy.

https://limewire.com/d/ijLko#vUofJ5b2FA

The problem does not occour on every player, it seems to be related to hardware decoding on windows pc and integrated intel graphics, but I'm not 100% sure. On TV via usb the file is OK.

I tryed to reencode the video but the same file is generated, as well as I checked for errors using virtualdub2 or ffmpeg ( -v error ) and the stream seems to be ok

Do someone else experienced similar issue? Is the file hevc-compliant?

Thanks you for any feedback.

[Selur]

Does play fine on MacOS (Player: mpv)
looking the clip with mediainfo:

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L4@Main
Codec ID : V_MPEGH/ISO/HEVC
Duration : 8 s 716 ms
Bit rate : 5 319 kb/s
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0 (Type 0)
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.241
Stream size : 5.53 MiB (98%)
Writing library : x265 4.1:[Windows][GCC 10.2.0][64 bit] 10bit
Encoding settings : cpuid=1111039 / frame-threads=1 / numa-pools=none / no-wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=1280x720 / interlace=0 / total-frames=0 / level-idc=40 / high-tier=1 / uhd-bd=0 / ref=6 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-eob / no-eos / no-hrd / info / hash=0 / temporal-layers=0 / open-gop / min-keyint=23 / keyint=250 / gop-lookahead=0 / bframes=16 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=99 / lookahead-slices=0 / scenecut=42 / no-hist-scenecut / radl=0 / no-splice / no-intra-refresh / ctu=64 / min-cu-size=8 / rect / amp / max-tu-size=32 / tu-inter-depth=4 / tu-intra-depth=4 / limit-tu=0 / rdoq-level=2 / dynamic-rd=0.00 / no-ssim-rd / signhide / tskip / nr-intra=10 / nr-inter=25 / no-constrained-intra / strong-intra-smoothing / max-merge=5 / limit-refs=0 / no-limit-modes / me=3 / subme=4 / merange=58 / temporal-mvp / no-frame-dup / no-hme / weightp / weightb / no-analyze-src-pics / deblock=1:1 / sao / no-sao-non-deblock / rd=6 / selective-sao=2 / no-early-skip / rskip / rskip-edge-threshold=0.020000 / no-fast-intra / no-tskip-fast / no-cu-lossless / b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=0.70 / psy-rdoq=0.80 / no-rd-refine / no-lossless / cbqpoffs=1 / crqpoffs=1 / rc=crf / crf=18.6 / qcomp=0.59 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=8000 / vbv-bufsize=12000 / vbv-init=0.9 / min-vbv-fullness=50.0 / max-vbv-fullness=80.0 / crf-max=0.0 / crf-min=0.0 / ipratio=1.40 / pbratio=1.30 / aq-mode=3 / aq-strength=0.70 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=5 / range=0 / colorprim=1 / transfer=1 / colormatrix=1 / chromaloc=1 / chromaloc-top=0 / chromaloc-bottom=0 / display-window=0 / cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr10 / no-hdr10-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=0 / analysis-save-reuse-level=0 / analysis-load-reuse-level=0 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field / qp-adaptation-range=1.00 / scenecut-aware-qp=0conformance-window-offsets / right=0 / bottom=0 / decoder-max-rate=0 / no-vbv-live-multi-pass / no-mcstf / no-sbrc / no-frame-rc

one of these:
... / ref=6 / ... /bframes=16 / b-adapt=2 / b-pyramid / ...
is likely the troublemaker for some hardware decoders.
I would try:
1. disabling b-pyramid
2. lowering the b-frame count to maybe 6
3. lower ref count to maybe 4

Cu Selur

[rwill]

I am not so sure.

I can reproduce it with DXVA (Windows Video Hardware Stuff API) on an NVidia card.

It kinda looks like this using the reference software:


Layer 0 POC 255 TId: 0 ( p-SLICE, QP 24 ) [DT 0.154] [L0 245 240 235 230 225 223 ] [L1 ] [:,(unk)]
Layer 0 POC 250 TId: 0 ( b-SLICE, QP 25 ) [DT 0.147] [L0 245 240 235 230 223 ] [L1 255 ] [:,(unk)]
Layer 0 POC 246 TId: 0 ( b-SLICE, QP 26 ) [DT 0.175] [L0 245 240 235 230 ] [L1 250 255 ] [:,(unk)]
Layer 0 POC 247 TId: 0 ( b-SLICE, QP 26 ) [DT 0.164] [L0 245 240 235 230 ] [L1 250 255 ] [:,(unk)]
Layer 0 POC 248 TId: 0 ( b-SLICE, QP 26 ) [DT 0.177] [L0 245 240 235 230 ] [L1 250 255 ] [:,(unk)]
Layer 0 POC 249 TId: 0 ( b-SLICE, QP 26 ) [DT 0.166] [L0 245 240 235 230 ] [L1 250 255 ] [:,(unk)]
Layer 0 POC 251 TId: 0 ( b-SLICE, QP 26 ) [DT 0.158] [L0 250 245 240 235 230 ] [L1 255 ] [:,(unk)]
Layer 0 POC 252 TId: 0 ( b-SLICE, QP 26 ) [DT 0.173] [L0 250 245 240 235 230 ] [L1 255 ] [:,(unk)]
Layer 0 POC 253 TId: 0 ( b-SLICE, QP 26 ) [DT 0.173] [L0 250 245 240 235 230 ] [L1 255 ] [:,(unk)]
Layer 0 POC 254 TId: 0 ( b-SLICE, QP 26 ) [DT 0.160] [L0 250 245 240 235 230 ] [L1 255 ] [:,(unk)]
Layer 0 POC 259 TId: 0 ( p-SLICE, QP 24 ) [DT 0.183] [L0 255 250 245 240 235 230 ] [L1 ] [:,(unk)] ** here the glitch starts


DXVA duplicates the picture with POC 255 (picture 181 of your sample) into POC 259 (picture 185 of your sample), at least thats the output. This breaks subsequent decoding order motion compensation. I am not so sure whats going on because I cannot look into the DXVA workings. If you can reproduce it with Intel it might be a Microsoft thing. But then again maybe x265 is doing some weired buffer stuff, I cannot really tell, I am no decoder guy.

Software decoders output the correct decoded picture and work correctly, at least the ones I tested.

[hellgauss]

Thanks for your replies!

@Selur
I do not think that is the problem. I had always used that preset with very minor tuning for 720p encodes and never had problems. Also, for best compatibility, I used level 4.0. If options are not compatible with level, x265 reports a warning. I'm 99.99% sure it is an encoder or a decoder bug.

@rwill
I did not fully understand the log, but from your answer it seems that it is probably a decoder bug.

[nevcairiel]

I tried different hardware and different APIs to access the hardware, and all tested configurations show the same issue, which rules out the problem being a particular video driver, or the software feeding the data to the driver, unless they happen to have the exact same bug.
So the most likely issue is that the stream somehow violates the conformance in some way that hardware is much more susceptible to then a software decoder.

[Ritsuka]

Could it be this one https://bitbucket.org/multicoreware/x265_git/commits/1e993ee88063103904867ced35f7fcfc8e0748cf ?
Does it happens with a x265 from the main git branch?

[hellgauss]

I used Animmouse ffmpeg stable autobuild of 7 march 25 , 64 bit.
https://github.com/AnimMouse/ffmpeg-stable-autobuild/releases/tag/2025-03-07-17-13-6ae81d8-7.1.1

[Z2697]

It's not the problem with weighted prediction, just checked.
Can you "hint" the way to get the input video, for testing?

(does this not violate forum rule? :p)
This is probably very source-specific so there's no better way I think.

[hellgauss]

The source is copyrighted material, I think it is reasonably ok to share here 7 second of encoded video for technical discussion (e.g. I see a lot of screenshots on the board), but I prefer not to share the source. And if you change the source a little bit the bug goes away, you should catch the same GOP and key frame, so it is useless to share a source sample.

The origianal is Uma Musume Pretty Derby, Season 3 Ep 10, @8Mbits perhaps you can somewhat find it somewhere. The glitch is at 20:45. Please note that a few seconds before there is a nearly identical scene which is encoded/decoded well.

Command line used:
ffmpeg -threads 1 -y -hide_banner -nostdin -stats_period 15 -bitexact -i "u.mkv" -map 0 -profile:v main10 -vf "removegrain=1:1:1,format=yuv420p10le,scale=1280:720:flags=bicublin:param0=0.125:param1=0.4375" -sws_flags accurate_rnd -c:v libx265 -preset veryslow -crf 18.6 -x265-params "psy-rd=0.7:psy-rdoq=0.8:aq-mode=3:aq-strength=0.7:deblock=1,1:nr-intra=10:nr-inter=25:tu-intra-depth=4:tu-inter-depth=4:merange=58:ref=6:rc-lookahead=99:bframes=16:rd=6:rskip=2:rskip-edge-threshold=2:cbqpoffs=1:crqpoffs=1:scenecut=42:scenecut-bias=5.2:qcomp=0.59:tskip=1:tskip-fast=0:limit-sao=1:selective-sao=2:vbv-maxrate=8000:vbv-bufsize=12000:frame-threads=1:pools=none:no-wpp=1:level-idc=40:high-tier=1:colorprim=1:colormatrix=1:transfer=1:range=limited" -bitexact "out.mp4"

I performed another encode with the official BtbN build and it is ok. However frames size are slightly different in the whole encode. My opinion is that it is a bug that manifest only in very specific circumstances combo. Next days I will try to change some parameters and to reproduce the glitch on a short sample.

[jpsdr]

Just for information, played the test file your provided on a HW player : Zappity Neo, didn't notice any glitch.

[benwaggoner]

Try bframes=8. There aren't any presets that use more than that in any encoder I am aware of, so it'll be a much less tested scenario. And I doubt >8 is providing much extra compression efficiency for you. Although grain-free anime is the scenario where it would be most likely to have any positive impact. That may be exposing a rare bug, as most real-world content wouldn't actually use >8 consecutive b-frames even if the bframe value is >8.

[hellgauss]

Actually slower + anime leads to brames=10. That clip contains maximum 9 bframes. And in slow and clean anime, or anime with somewhat "repeated" frames, they are often used.

The specific problem for me is no longer for that video, since I performed a test encode with another build and it is ok. But I'm interested to know if it is an encoder bug (i.e. the stream is not compliant with hevc or with level) or decoder bug. If it is an encoder bug, I assume that it can be present even in the official build, which simply avoided the specific circumstances of animmouse build.

It is interesting to notice that two hardware decoder (my tv and jpsdr's Zappity) are ok. I still bet for decoder bug, but I'll do some tests when I have time.

[benwaggoner]

Actually slower + anime leads to bframes=10. That clip contains maximum 9 bframes. And in slow and clean anime, or anime with somewhat "repeated" frames, they are often used.
Yeah, cel animation is the primo use case for very high bframe counts. But that doesn't mean they've been tested thoroughly. If you have to debug this again sometime, you might try setting it to 8 and seeing if it changes anything. It likely won't, but was the only thing that stood out for me a little.

The specific problem for me is no longer for that video, since I performed a test encode with another build and it is ok. But I'm interested to know if it is an encoder bug (i.e. the stream is not compliant with hevc or with level) or decoder bug. If it is an encoder bug, I assume that it can be present even in the official build, which simply avoided the specific circumstances of animmouse build.

It is interesting to notice that two hardware decoder (my tv and jpsdr's Zappity) are ok. I still bet for decoder bug, but I'll do some tests when I have time.
rwill above identified a POC issue that seems a likely culprit, and may violate the spec in some small way.

Real-world decoders generally get lots of fuzz testing with a variety of non-compliant streams, so they are often able to pay things that aren't perfectly spec legal. Making a decoder crash in an interesting way has been a big source of security holes over the decades, so they really work to make sure all kinds of wrong things either still work or result in a safe exit and error message/black frame.

[rwill]

rwill above identified a POC issue that seems a likely culprit, and may violate the spec in some small way.

Real-world decoders generally get lots of fuzz testing with a variety of non-compliant streams, so they are often able to pay things that aren't perfectly spec legal. Making a decoder crash in an interesting way has been a big source of security holes over the decades, so they really work to make sure all kinds of wrong things either still work or result in a safe exit and error message/black frame.

Its not really a POC issue and I am still somewhat puzzled. It looks like the decoder copies the first reference frame from L0 into the glitch picture framebuffer and does nothing else. The first glitch picture itself looks normal so there are no decoding artifacts or so, its just the wrong picture. Subsequent pictures are broken because the glitch picture is the wrong picture and so motion compensation fails. I checked the decoded picture buffer management and picture indices look ok to me.

Might be that something triggers some sort of error concealment. But I have not found out what that might be. The picture triggering the issue is a single P slice, so really simple, I do not know yet what could go wrong there that might trip off hardware decoders but not software decoders.

[benwaggoner]

Might be that something triggers some sort of error concealment. But I have not found out what that might be. The picture triggering the issue is a single P slice, so really simple, I do not know yet what could go wrong there that might trip off hardware decoders but not software decoders.
Yeah, when some decoders work and others don't it is often because of error concealment.

Have you decoded just that p-slice, and the correct one, and compared them visually? Might be a clue in there.

[rwill]

Have you decoded just that p-slice, and the correct one, and compared them visually? Might be a clue in there.

Yes, they are bit identical.

[Z2697]

Have we tested this on AMD and Intel decoders?

Zen4 igpu decoder seems fine...
An old i5 with HD 630 shows different looking artifact. It copied a shifted version of frame 176 into a partially decoded frame 185. (176 is next to 181 in decoding order)
That's interesting, maybe the block where it stops decoding correctly is the "offending" block.
And an interesting error message (when using FFmpeg -c:v hevc_qsv for decoding)
[hevc_qsv @ 000001dbf267fec0] A decode call did not consume any data: expect more data at input (-10)

[jpsdr]

No one has some kind of H265 compliant data stream analyzer tool ?

[Z2697]

I'd say HM is good enough...
Having a lot more assertions in the code.

(JM? not so good.
https://web.archive.org/web/20100228010450/http://x264dev.multimedia.cx/?p=212
Edit/Note: It seems that earlier we were testing with a stream that was actually invalid; the reference decoder didn’t complain, but after putting it through a stream validator, it found an obvious issue (which we fixed in x264 r1342). As such, the list is a lot shorter now.
Also weighted prediction? LOL
That's a slightly different bug, I think, HEVC changed how chroma offset is coded)

[GeoffreyA]

Have we tested this on AMD and Intel decoders?

Zen4 igpu decoder seems fine...
An old i5 with HD 630 shows different looking artifact. It copied a shifted version of frame 176 into a partially decoded frame 185. (176 is next to 181 in decoding order)
That's interesting, maybe the block where it stops decoding correctly is the "offending" block.
And an interesting error message (when using FFmpeg -c:v hevc_qsv for decoding)
[hevc_qsv @ 000001dbf267fec0] A decode call did not consume any data: expect more data at input (-10)

Shows up on Arc B580 hardware decoding as well. Software decoding is all right.

[SquallMX]

I had a similar issue (https://forum.doom9.org/showthread.php?t=185706) (hardware decoding on nVidia was buggy, software decoding was OK), the culprit was "--rskip 2". Using "--rskip 1" fixed the issue but increased encoding time a lot.

[Z2697]

I don't think so... rskip 2 is not that different from rskip 1, it just uses different metric to make skip decision.

[rwill]

Have we tested this on AMD and Intel decoders?

Zen4 igpu decoder seems fine...
An old i5 with HD 630 shows different looking artifact. It copied a shifted version of frame 176 into a partially decoded frame 185. (176 is next to 181 in decoding order)
That's interesting, maybe the block where it stops decoding correctly is the "offending" block.
And an interesting error message (when using FFmpeg -c:v hevc_qsv for decoding)
[hevc_qsv @ 000001dbf267fec0] A decode call did not consume any data: expect more data at input (-10)

Can you upload the .yuv output from the i5 quicksync somewhere? I don't have any Intel systems available currently.

[Z2697]

Can you upload the .yuv output from the i5 quicksync somewhere? I don't have any Intel systems available currently.

I have it losslessly compressed in H264
https://workupload.com/file/VgTXNyAtrbF

[rwill]

Thanks Z2697, this got me further. I hope I got the numbers right here.

The block in question, POC 259, CTU 73 is a single 64x64 INTER_2Nx2N coded CU.

Due to certain mechanics (motion vector predictor POC scaling) the motion vector predictor (MVP) ends up being:
X= -32768, Y= -19437
with the alternative (not used) being
X= -32768, Y= -19506

So x265 codecs the motion vector difference as
MVD X= 32768, Y= 18385

so the final MV in the decoder is
MVD X= 0, Y= -1052

The element of a MVP can take the range of -32768 to 32767, so X is at the maximum here.

It appears the NVidia and Intel implementation gets confused by the large MVD X, which would take 17 bit to represent if signed, I don't know how they have implemented it in hardware though.

The picture where the glitch starts is the only picture in the sequence with such a large MVD.

I further inspected SquallMX his stream and there his glitch picture is also the only picture in the stream with a MVD greater or equal 2^15.

I suggest the x265 ppl to limit their motion vectors to sane ranges so hellgaus can try again. Letting MVs get out of hand is just bad style, even if it saves some bits.

[hellgauss]

Thanks for nice reverse engineering, although I did not understand everything! Sometimes bugs are very subtle to find.

In the end, is the stream hevc-compliant and it was decoder issue? For motion range, I replaced merange=57 with merange=58, I read in reference that "If the search range were any larger than this, another CTU row of latency would be required for reference frames." (but I do not know what does it means). For "hex" -me, the limit is 57, but i read that it can be raised to 58 for "star".

[rwill]

Well, to me it looks like a decoder issue. There is also currently nothing you can do with x265 configurations to avoid it.

I quickly scanned the standard for some motion vector delta limit but was not able to find something specific. There is an equation that puts the final motion vector in the range -2^15 to (2^15)-1 but there does not seem to be a limit on the coded delta. The delta just gets wrapped each 2^16 so internaly only the lower 16 bits need to be kept of the coded MVD I guess. Maybe this is where things go wrong, I don't know.

[Z2697]

Well, to me it looks like a decoder issue. There is also currently nothing you can do with x265 configurations to avoid it.

Disable temporal-mvp? :devil:

[Z2697]

So x265 codecs the motion vector difference as
MVD X= 32768, Y= 18385


The MVD X I got from HM decoder is -32768...
Which means the MVP is 32768? Int16 Overflow?
Either way, one of them is 32768. And is greater than maximum signed 16bit integer.

Wait... the sign is probably saved separately in the bitstream...


Void TDecSbac::parseMvd( TComDataCU* pcCU, UInt uiAbsPartIdx, UInt uiPartIdx, UInt uiDepth, RefPicList eRefList )
{
UInt uiSymbol;
UInt uiHorAbs;
UInt uiVerAbs;
UInt uiHorSign = 0;
UInt uiVerSign = 0;
...


I'd say this is completely decoders' fault.

[Z2697]

If the MVP is causing problem, maybe they stored the value in a single signed int16 internally, the problem is more complicated than MVD, right?

I mean, if the encoder should do workaround for the decoders' fault.
What are you gonna do? Going all the way back to the reference block and redo its motion search?

[GeoffreyA]

Well, if it's a decoder fault, I'd be happy to submit the issue on the Intel side. Hopefully, something can be done in the driver. Perhaps somebody else can tackle Nvidia.

[rwill]

I did some more MVD experiments:

https://drive.google.com/drive/folders/1jlpHekxYG_ZL7-OLTl3rYQ_0GtfcRqIG?usp=sharing

The problem gets triggered even with the small MVPs here, so MVP value seems unrelated.

MVD in short, for my GTX 1650 (don't judge plz): -0x8000 works, 0x7fff works, -0x8001 does not work, 0x8000 does not work.

The quick fix I think that can be applied to an encoder is to cast the MVD to int16 and back to int32 before writing.

[Z2697]

MVD in short, for my GTX 1650 (don't judge plz): -0x8000 works, 0x7fff works, -0x8001 does not work, 0x8000 does not work.

But is the MVD in the sample 32768 or -32768?
Perhaps MVP and MVD both can trigger the bug?

How are you able to make them arbitrary numbers?

[rwill]

But is the MVD in the sample 32768 or -32768?
Perhaps MVP and MVD both can trigger the bug?

How are you able to make them arbitrary numbers?

In hellgauss his sample? 32768 (positive).

For my experiments I modified an encoder to output such MVDs.

[Z2697]

In hellgauss his sample? 32768 (positive).

For my experiments I modified an encoder to output such MVDs.

I can figure that out... I'm asking how to modify the encoder, I know it's your company's proprietary encoder, but is there a general idea that can be reproduced in x265?

[rwill]

Rule of three: motion vector = 0x8000 + MVP.

Somewhere after the inter search. Best done with 2Nx2N inter at depth = 0.

[Z2697]

In hellgauss his sample? 32768 (positive).

For my experiments I modified an encoder to output such MVDs.

This thing here in HM really tells me that the MVD is -32768...
Am I misunderstanding something?
https://vcgit.hhi.fraunhofer.de/jvet/HM/-/blob/master/source/Lib/TLibDecoder/TDecEntropy.cpp?ref_type=heads#L279

[rwill]

This thing here in HM really tells me that the MVD is -32768...
Am I misunderstanding something?
https://vcgit.hhi.fraunhofer.de/jvet/HM/-/blob/master/source/Lib/TLibDecoder/TDecEntropy.cpp?ref_type=heads#L279

Thats because it returns the value of a motion vector which class is defined here:

https://vcgit.hhi.fraunhofer.de/jvet/HM/-/blob/master/source/Lib/TLibCommon/TComMv.h?ref_type=heads#L51

where the X and Y are defined as a 16 bit Short.

One has to check where the Mvd is set at

https://vcgit.hhi.fraunhofer.de/jvet/HM/-/blob/master/source/Lib/TLibDecoder/TDecSbac.cpp?ref_type=heads#L844

[Z2697]

I understand it now... so HM and FFmpeg also store the internal MVD as int16, and -32768 is the overflowed result of the actual 32768.
But how do they get the decoding result right? I'm confused.

And now it's x265's fault to produce such value?

[rwill]

A motion vector element is always in the range -2^15 to (2^15)-1.

Its in the standard at:

8.5.3.2 Derivation process for motion vector components and reference indices
8.5.3.2.1 General
Equation 8-94 to 8-97


Which more or less defines that motion vector elements are stored in int16.


uLX[ 0 ] = ( mvpLX[ 0 ] + mvdLX[ 0 ] + 2^16 ) % 2^16
mvLX[ 0 ] = ( uLX[ 0 ] >= 2^15 ) ? ( uLX[ 0 ] − 2^16 ) : uLX[ 0 ]


So for the final motion vector element X mvLX[ 0 ] it does not matter if a 0x8000 MVD overflowed or not because:

( ( mvpLX[ 0 ] + 0x8000 + 2^16 ) % 2^16 ) == ( ( mvpLX[ 0 ] + -0x8000 + 2^16 ) % 2^16 )


So the issue most likely can be avoided by
- Clipping motion vectors to sane ranges
- Casting the MVD to "int16" before writing

But as I wrote above, I have not found any actual limits regarding the MVDs in the bitstream in the standard.

[GeoffreyA]

With the view that this is a decoder issue, I posted the bug on Intel's side:

https://github.com/IGCIT/Intel-GPU-Community-Issue-Tracker-IGCIT/issues/1153

[Z2697]

I think this is both decoder and encoder issue :p

[GeoffreyA]

I think this is both decoder and encoder issue :p

Oh, well, let's see what happens :)

[Z2697]

There is some sort of limit in x265... is this not enough?
Or is it something with the specific binary? I can't reproduce the Uma Musume bug in my test build of x265 stable branch, and can reproduce using the FFmpeg from AnimMouse.
https://bitbucket.org/multicoreware/x265_git/src/cd4f0d6e9d30766f5e2be89dab2e6809f3b76507/source/encoder/search.cpp#lines-4523:4527

[GeoffreyA]

What changes are in AnimMouse's version?

[hellgauss]

As far as I can tell, it takes some time to receipt every changes/improvements of the official ffmpeg build. I use it because it is compiled with libfdk for aac encode. The building process seems to be automated.

Perhaps the bug pointed by Ritsuka at the beginning of the thread has still not be fixed in Animmouse build?

PS sorry for not being more precise, but I am not so skilled with git, builds, scrutinize complex code etc... I'm just able to do basic things in c and other languages.

[GeoffreyA]

Yes, it could be something fixed, in x265, in the past few months. I was under the impression that AnimMouse's build was a fork.

[Z2697]

Yes, the weighted prediction bug is not fixed in current AnimMouse build because it uses stable branch of x265, which basically is the "lastest release version".
There haven't been a release after that bug fix.
But, that bug is not what caused your problem.

There's no modification being done in that build, other than a CMake patch.
https://github.com/rdp/ffmpeg-windows-build-helpers/blob/083f55b7cbad2fb4339dca9ef26fe4f647abeeb4/patches/x265_x86_noasm_fix.patch

But some compiler bug is not impossible...
(For example if you disable too many compiler cpu flag in a X86 build, the ratecontrol is completely fried)

[hellgauss]

I noticed that the bug occurs at POC=256. If I use trace_header BSF in ffmpeg, POC is reset at zero after 255 (it seems to be only one byte).

Perhaps some weird things happens in the decoder which overwrites some carry flag?

[benwaggoner]

I noticed that the bug occurs at POC=256. If I use trace_header BSF in ffmpeg, POC is reset at zero after 255 (it seems to be only one byte).

Perhaps some weird things happens in the decoder which overwrites some carry flag?
If it is supposed to be 0-255 not 1-256, a 256 could be an issue.

If so, that's one of the classic programming bugs.