Why do I have strange artifacts in my encode

[ShortKatz]

I've encoded two movies of mine using HandBrake and the x265 encoder into HEVC. Both movie files suffer from pixelated artifacts.
Here are some screenshots:
https://imgur.com/LMiDEIT
https://imgur.com/ffnEmqB
Sadly this is not reproducible if I only encode this scene. I've cut out the effect sequence of the original file and encoded only this piece, which turned out to be perfectly fine. Recently I've changed my x265 encoding options, so maybe I now have a bad combination or have chosen a delicate option?

I used x265 3.5 RC1 and preset "slower" with the following options:
--early-skip --no-rect --no-amp --no-sao --rd-refine --opt-cu-delta-qp --no-strong-intra-smoothing --psy-rd=3 --psy-rdoq=0 --hme --hme-search=hex,umh,star --rskip=2

All I've changed to my previous encodes was, I switch from preset "medium" to "slower" and I've added --no-rect --no-amp --rd-refine --opt-cu-delta-qp --rskip=2.

I've never before have seen something like this with x265, so I'm a bit puzzled what this could be. I also have no idea why I cannot reproduce this with the affected sequence only.


Edit: For a sample file see https://forum.doom9.org/showpost.php?p=1936949&postcount=17

[poisondeathray]

Rule out a decoding/playback issue first . Try something else like mpv, ffplay

Run some hardware stability tests. Memory integrity, psu etc... If you're overclocked , revert to defaults

If you can't reproduce with just that scene, try encoding a segment that includes a few scenes before and after

[Boulder]

What is your script? I've seen those with MDegrain but pinterf's latest version of MVTools has fixed them.

[ShortKatz]

Rule out a decoding/playback issue first . Try something else like mpv, ffplay

Run some hardware stability tests. Memory integrity, psu etc... If you're overclocked , revert to defaults

If you can't reproduce with just that scene, try encoding a segment that includes a few scenes before and after

I've tried with Apple Quicktime, VLC and ffplay. It looks all the same. hardware stability test told me, all is fine. Nope, I did not overclock. I will try your suggestion with the scenes before, thanks.

What is your script? I've seen those with MDegrain but pinterf's latest version of MVTools has fixed them.
I did not use a script. Its just HandBrake with an updated x265 and some "Additional options". I used the same build for the same movies before, but with a different preset and different x265 options. And all was fine at that time.

[charliebaby]

try with staxrip and update your FFMPEG and take the x265 stable version and see if it's good

[quietvoid]

That's a rskip 2 problem AFAIK, it was mentioned before in the x265 thread.
https://forum.doom9.org/showthread.php?p=1920461#post1920461

[ShortKatz]

If you can't reproduce with just that scene, try encoding a segment that includes a few scenes before and after
That has worked. I've cut a sample 30sec before and after of the affected part and now I can reproduce it. Encoded it three times and every time I see this issue.


That's a rskip 2 problem AFAIK, it was mentioned before in the x265 thread.
https://forum.doom9.org/showthread.php?p=1920461#post1920461

I've set rskip to 1 or 0. The pattern did change a bit, but the issue is still there.

[Boulder]

Can you try encoding that segment into a lossless AVI with FFV1 for example, and see if you have the same artifacts?

Also, which Avisynth version you have? I had strangely similar issues a long time ago, https://forum.doom9.org/showthread.php?p=1698102#post1698102. If you can edit the script HandBrake uses, you could do a test by adding for example RequestLinear(clim=100) right after the source is loaded. It requires TIVTC to be installed.

[poisondeathray]

Also, which Avisynth version you have? I had strangely similar issues a long time ago, https://forum.doom9.org/showthread.php?p=1698102#post1698102. If you can edit the script HandBrake uses, you could do a test by adding for example RequestLinear(clim=100) right after the source is loaded. It requires TIVTC to be installed.

He used handbrake for those tests, so avisynth is irrelevant


That has worked. I've cut a sample 30sec before and after of the affected part and now I can reproduce it. Encoded it three times and every time I see this issue.


The next step since you have a cut source that you're able to consistently reproduce the issue with - is to try other decoding pathways, and binaries for encoding

If you can't figure the issue, post the sample and information on the bug tracker for the devs.

[Boulder]

He used handbrake for those tests, so avisynth is irrelevant

Ah, for some reason I've always understood that Avisynth can be used inside HB if you needed any advanced filtering techiques.

[poisondeathray]

Ah, for some reason I've always understood that Avisynth can be used inside HB if you needed any advanced filtering techiques.

You can use avisynth as input, but not directly . e.g. avfs as a "virtual" file by mounting an .avs . Direct avs filters cannot be used inside HB

[ShortKatz]

Can you try encoding that segment into a lossless AVI with FFV1 for example, and see if you have the same artifacts?
I will try that.


The next step since you have a cut source that you're able to consistently reproduce the issue with - is to try other decoding pathways, and binaries for encoding


Yes, I did many encodes already. I found out so far, that I get the same artifacts if I add rskip=2 and hme the slower preset. I do not see the artifacts if I remove hme from the options set of my initial post, but removing rskip did not help. Which does not make sense to me, because I used hme many times in the past without issues.

Ah, for some reason I've always understood that Avisynth can be used inside HB if you needed any advanced filtering techiques.

I don't use any kind of filter.

[poisondeathray]

Yes, I did many encodes already. I found out so far, that I get the same artifacts if I add rskip=2 and hme the slower preset. I do not see the artifacts if I remove hme from the options set of my initial post, but removing rskip did not help. Which does not make sense to me, because I used hme many times in the past without issues.


Good job, keep on trying to narrow it down

Also try to rule out other versions (older, newer, non HB) , the might be a regression in the HB x265 version you're using

If you post the sample, other people might be willing to help with the testing

[ShortKatz]

Also try to rule out other versions (older, newer, non HB) , the might be a regression in the HB x265 version you're using

Yes, I've tried different versions of x265 so far, 3.5 RC1 and 3.4 both gave me the same issue. Next, I will try FFmpeg instead of HandBrake.




If you post the sample, other people might be willing to help with the testing

Good idea, but the sample file is 340MB in size and I don't know where to upload. And I don't know if I violated any copyrights by doing so.

[Boulder]

Good idea, but the sample file is 340MB in size and I don't know where to upload. And I don't know if I violated any copyrights by doing so.

If you have a Google account, you could upload it to Google Drive and share it from there. I wouldn't worry about the copyrights issue, I'd say that for these testing purposes it's fine.

I would be very interested in testing with my normal settings. I have used HME but only with 1440p encodes and with umh,umh,star as the search method.

[ShortKatz]

If you have a Google account, you could upload it to Google Drive and share it from there. I wouldn't worry about the copyrights issue, I'd say that for these testing purposes it's fine.

OK, will try to upload it somewhere. It seems I need some help on this issue. I have tested so much now and still I have no clue what the issue is.


What I found out by now is, it is not a HandBrake issue. I was also able to get this issue if I use FFmpeg with:

ffmpeg -i input.mkv -vf crop=3824:1600:8:280 -c:v libx265 -tag:v hvc1 -crf 20 -preset slower -profile:v main10 -x265-params "rect=0:amp=0:early-skip=1:rskip=2:rd-refine=1:strong-intra-smoothing=0:sao=0:psy-rd=3:psy-rdoq=0:opt-cu-delta-qp=1:hme=1:hme-search=hex,umh,star:hdr10=1:hdr10-opt=1" -c:a eac3 -b:a 768k output.mp4

I also found out, I can workaround this issue by setting b-adapt to 0 or 1. Which also means I only see the issue for the presets medium to placebo.

[ShortKatz]

OK, I found a place to upload the sample file. The link is available for 7 days.
https://we.tl/t-XfaW773Hd8

To See the issue do the following:
- Compile FFmpeg with x265 v3.4 or 3.5 RC1.
- Use the sample from the link and convert it with the following command:
ffmpeg -i input.mkv -vf crop=3824:1600:8:280 -c:v libx265 -tag:v hvc1 -crf 20 -preset slower -profile:v main10 -x265-params "rect=0:amp=0:early-skip=1:rskip=2:rd-refine=1:strong-intra-smoothing=0:sao=0:psy-rd=3:psy-rdoq=0:opt-cu-delta-qp=1:hme=1:hme-search=hex,umh,star:hdr10=1:hdr10-opt=1" -c:a eac3 -b:a 768k output.mp4
- Inspect the result using a video player that can skip forward frame by frame. The issue is normally at second 37 or 38. It can be just one frame or it can be more than one. So you need to be careful to not jump over the issue.

The issue can look like in the initial post, but it also can look like this
https://imgur.com/qhcEIhM
or this
https://imgur.com/gAoR40w

You can also use the latest HandBrake snapshot with the same settings (HB 1.3.3 will not work, because it uses an old x265 which has no hme).

[Boulder]

Just tested your ffmpeg command line and also my settings with Avisynth input and x265, and I cannot reproduce the issue.

Are you a Mac user (based on the zip file contents)? Perhaps that is a factor to consider.

[poisondeathray]

No artifacts either with that ffmpeg commandline . Windows.

[ShortKatz]

Thanks for all your testing. If you can not reproduce, then I am even more confused why I see this artifacts.



Are you a Mac user (based on the zip file contents)? Perhaps that is a factor to consider.

This is true, I am a Mac user. Well spotted!
Do you think the x265 encoding depends on the OS? I only see architecture specific code in the x265 source.
Just in case, I will write some Mac users I know and ask them to reproduce.

[poisondeathray]

This is true, I am a Mac user. Well spotted!
Do you think the x265 encoding depends on the OS? I only see architecture specific code in the x265 source.
Just in case, I will write some Mac users I know and ask them to reproduce.

It should not matter, but post your HW specs , OS, and yes try to get some other Mac users to run some tests

That distribution and variety of artifacts in those screenshots strongly looks like HW issue to me, such as bad memory stick - I would revisit those HW tests also; and check your temps and cooling