x265 HEVC Encoder

[stax76]

If somebody is looking for vpy code:

https://github.com/rigaya/NVEnc/blob/master/NVEncCore/rgy_input_vpy.cpp

https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/vapoursynth.c

[stax76]

Maybe I build a chunk encoding console app as it seems to be relatively simple and nobody else seems to be interested in building it. Are there any special x265 settings needed? x264 has a --stitchable option. How does ripbot264 do it? vspipe and avs2pipemod allow defining start and end frame. Are two chunks enough?

[vpupkind]

Maybe I build a chunk encoding console app as it seems to be relatively simple and nobody else seems to be interested in building it. Are there any special x265 settings needed? x264 has a --stitchable option. How does ripbot264 do it? vspipe and avs2pipemod allow defining start and end frame. Are two chunks enough?

You may want to use their "chunked encoding" mode which will improve performance at the chunk boundaries

[LigH]

x265 3.3+27-g4780a8d99 (https://www.mediafire.com/file/94e52feibz00y7b/x265_3.3+27-g4780a8d99.7z/file) (multilib, GCC 10.1)

[stax76]

You may want to use their "chunked encoding" mode which will improve performance at the chunk boundaries

Thanks, so --chunk-start and --chunk-end, would be helpful if somebody can share experience with it. How many extra frames are needed? I'll use vspipe and avs2pipemod for trimming, modifying the avs/vpy file has too many implications.

[benwaggoner]

Thanks, so --chunk-start and --chunk-end, would be helpful if somebody can share experience with it. How many extra frames are needed? I'll use vspipe and avs2pipemod for trimming, modifying the avs/vpy file has too many implications.
I'd want to use at least as many frames as the greater of keyint and fps*vbv-maxrate/vbv-bufsize. That'll let rate control and IDR placement converge pretty close on what they would be in a full pass encode. If using larger gaps and being kinda paranoid, I'd probably use double the above.

The compute overhead is proportional to chunk duration, so is much less of an issue with 10 minute chunks than 30 second chunks.

[crystalfunky]

I noticed something about the crf value. One step to the next number is around 4000-5000 kbits bitrate.
So for example I have a clip with 27 Mbit/s and I want to lower the bitrate, I need to lower the crf one step. If I want to get the bitrate around 17 Mbit/s, I need to do 2 steps.
Can somebody confirm that?

[Boulder]

I noticed something about the crf value. One step to the next number is around 4000-5000 kbits bitrate.
So for example I have a clip with 27 Mbit/s and I want to lower the bitrate, I need to lower the crf one step. If I want to get the bitrate around 17 Mbit/s, I need to do 2 steps.
Can somebody confirm that?

No, that's not how it works. I'm afraid there is no mathematical way to calculate the "correct" value for CRF. It depends on the source and also the encoder parameters.

[RanmaCanada]

I noticed something about the crf value. One step to the next number is around 4000-5000 kbits bitrate.
So for example I have a clip with 27 Mbit/s and I want to lower the bitrate, I need to lower the crf one step. If I want to get the bitrate around 17 Mbit/s, I need to do 2 steps.
Can somebody confirm that?
If you want to lower the bitrate, then lower the bitrate with a 2 pass. CRF has nothing to do with bitrate.

[Blue_MiSfit]

Well, that's not accurate. If you're generally ok with the quality + bitrate from a given crf with a given preset, and one particular piece of content ends up too big, you can always just increase CRF.

Capped CRF is reasonable too.

[LoRd_MuldeR]

x265 3.3+27-g4780a8d99 (https://www.mediafire.com/file/94e52feibz00y7b/x265_3.3+27-g4780a8d99.7z/file) (multilib, GCC 10.1)

:thanks:

[filler56789]

x265.exe 3.3+31-d5695e2

http://msystem.waw.pl/x265/

[stax76]

Sorry to post here again, I've added this chunk encoding feature to staxrip, available in the last beta. It turned out everything needed was already in place so it could be finished in four hours.

Documented here:

https://staxrip.readthedocs.io/usage.html#parallel-processing

RipBot264 and Media Coder support it too.

[RanmaCanada]

Sorry to post here again, I've added this chunk encoding feature to staxrip, available in the last beta. It turned out everything needed was already in place so it could be finished in four hours.

Documented here:

https://staxrip.readthedocs.io/usage.html#parallel-processing

RipBot264 and Media Coder support it too.

This is fantastic. I haven't used this feature since mediacoder.

thank you.

[filler56789]

x265.exe 3.3+32-516a579

Store histogram of Y, U, V and Y's edge in analysis file.

Enables the library to compute Y Histogram and store
Histogram of Y, U, V Channel and Y's Edge in analysis
file at all reuse levels when --hist-scenecut is enabled.

http://msystem.waw.pl/x265/

[Barough]

x265 v3.3+31-g431a22e82 (http://www.mediafire.com/file/emtac09geqs2tzm/x265-3.3%252B31-g431a22e82_Win_GCC101.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 10.1.0)
https://bitbucket.org/multicoreware/x265_git/commits/branch/master

[benwaggoner]

No, that's not how it works. I'm afraid there is no mathematical way to calculate the "correct" value for CRF. It depends on the source and also the encoder parameters.
As a heuristic, file size doubles with about 5-6 lowerCRF, and halves with about the same in the other direction. Content dependent, of course.

[Boulder]

Perhaps a silly question, but something I've wondered ever since MakeMKV was able to add Dolby Vision to Matroska files. If at some point mkvmerge can handle the DV layer in a similar way, would it be possible to re-encode the actual video to a lower bitrate and combine the two (provided that mkvmerge supports such action)? Rescaling would definitely be out of question, but some light denoising should not do much, since it's the same with HDR.
The reason I'm asking this is that I have a small 2.5" HDD next to my TV, which I've been using to playback DV on the TV player since Kodi doesn't support DV. The HDD is only 500GB in size so re-encoding could effectively double the amount of movies I could store there.

[quietvoid]

The specs do mention the 1080p enhancement layer always comes with a 2160p base layer.
It is also something I have been wondering, you could easily test it by using yusesope's tool and merging the metadata into a reencoded HEVC file.

[benwaggoner]

The specs do mention the 1080p enhancement layer always comes with a 2160p base layer.
It is also something I have been wondering, you could easily test it by using yusesope's tool and merging the metadata into a reencoded HEVC file.
Yes, the enhancement layer doesn't need to be full rez as it's really spatial metadata. The enhancement layer is unwatchable by itself.

That said, almost all new DoVi content for several years uses a non-backwards compatible (NBC) base layer without an enhancement layer. This provides more efficient encoding and lower complexity decode/tone mapping. x265 can encode the NBC base layer just fine, although I'd use --preset slower at a minimum, and the DoVi-specific encoding tweaks.

[Blue_MiSfit]

3.4 has been tagged for release!

https://bitbucket.org/multicoreware/x265/branch/Release_3.4

Release notes - v3.4
-----------------------------

New features
------------------
• Edge-aware quadtree partitioning to terminate CU depth recursion based on edge information. --rskip 2 enables the feature and --rskip-edge-threshold <0..100> denotes the minimum expected edge-density percentage within the CU, below which the recursion is skipped. Experimental feature.
• Application-level feature --abr-ladder for automating ABR ladder generation. Shows ~65% savings in the over-all turn-around time required for the generation of a typical HLS ladder in Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz over a sequential ABR-ladder generation approach that leverages save-load architecture.
Enhancements to existing features
----------------------------------------------
• Improved 2-pass rate-control algorithm. The savings in the bitrate is ~1.72% with visual improvement in quality in the initial 1-2 secs.
Encoder enhancements
--------------------------------
• Faster ARM64 encodes enabled by ASM contributions from Huawei. The speed-up over no-asm version for 1080p encodes @ medium preset is ~15% in a 16 core H/W.
• Strict VBV conformance in zone encoding.
Bug fixes
------------
• Multi-pass encode failures with --frame-dup.
• Corrupted bitstreams with --hist-scenecut when input depth and internal bit-depth differ.
• Incorrect analysis propagation in multi-level save-load architecture.
• Failure in detecting NUMA packages installed in non-standard directories.

[Patman]

I've updated my x265 builds. x265-3.4+1 is built from the stable branch (stable release) and x265-3.4+6 is built from the master branch. Both builds are 64-bit multilib windows binaries.

[alexantr]

Who try --rskip 2? Did you catch rare glitches like that

[Sagittaire]

3.4 has been tagged for release!

https://bitbucket.org/multicoreware/x265/branch/Release_3.4

Release notes - v3.4
-----------------------------

New features
------------------
• Edge-aware quadtree partitioning to terminate CU depth recursion based on edge information. --rskip 2 enables the feature and --rskip-edge-threshold <0..100> denotes the minimum expected edge-density percentage within the CU, below which the recursion is skipped. Experimental feature.
• Application-level feature --abr-ladder for automating ABR ladder generation. Shows ~65% savings in the over-all turn-around time required for the generation of a typical HLS ladder in Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz over a sequential ABR-ladder generation approach that leverages save-load architecture.
Enhancements to existing features
----------------------------------------------
• Improved 2-pass rate-control algorithm. The savings in the bitrate is ~1.72% with visual improvement in quality in the initial 1-2 secs.
Encoder enhancements
--------------------------------
• Faster ARM64 encodes enabled by ASM contributions from Huawei. The speed-up over no-asm version for 1080p encodes @ medium preset is ~15% in a 16 core H/W.
• Strict VBV conformance in zone encoding.
Bug fixes
------------
• Multi-pass encode failures with --frame-dup.
• Corrupted bitstreams with --hist-scenecut when input depth and internal bit-depth differ.
• Incorrect analysis propagation in multi-level save-load architecture.
• Failure in detecting NUMA packages installed in non-standard directories.

well now Npass encoding seem work like crf mode: It is not a progess. For me the quantizer ratio between pframe and bframe is too high by default.

[burnix]

Patman. Can you build an ffmpeg x64 with it ????

Thanks

[Patman]

Patman. Can you build an ffmpeg x64 with it ????

Thanks

Here (http://www.mediafire.com/file/6920cu884gwplxk/ffmpeg-N-98036-gd538ca5175-gb6d7c4c1d4-gcc10.1.0.7z/file) :D

[burnix]

You're the best.
I'm going to try now, and after i'm going to sell on internet all crappy computer and server to make a real decent computer and work on ffmpeg compil.

Thanks man.

[K.i.N.G]

I have been stuggling for months with ugly distortions (smearing that kinda looks like motion blur with banding) in fast moving scenes and finally found out that x265's CUTree is to blame.
Turning it off solves it, but the bitrate blows up when trying to keep the same overal quality in other scenes.

Is there any way to make CUTree 'less agressive' or set it to 50% for example or some threshold parameter?

[Blue_MiSfit]

try a lower ctu value maybe? What resolution + bitrate are you targeting?

[K.i.N.G]

try a lower ctu value maybe? What resolution + bitrate are you targeting?

4K Bitrate depends on source, but the current footage is decent quality with little grain (which i'd like to keep).
For this footage, lets say arround 15K since its for streaming.

I restarted the encode with --ctu 32 (instead of --ctu 64)

[K.i.N.G]

Lowering the CTU size to 32 (--ctu 32) doesnt really seem to help

[Boulder]

By your description, it looks like the infamous onion artifacts. Have you tried --rd 6 with --rd-refine?

[poisondeathray]

Does increasing qcomp reduce the cutree effect? Similar to how increasing qcomp reduced mbtree effect in x264 ?

[K.i.N.G]

By your description, it looks like the infamous onion artifacts. Have you tried --rd 6 with --rd-refine?

I have not.
I will make sure to try those out next after this encode, which will run for a few days.

Does increasing qcomp reduce the cutree effect? Similar to how increasing qcomp reduced mbtree effect in x264 ?

I can confirm it does.
Normally I raise the qcomp to something like 0.70-0.75 and raise the CRF by ±0.5 which 'fixes' it to more acceptable levels.

Though... I always wondered if there was another (more efficient) way to control CUTree's strength/agressiveness more directly... (if I understand correctly qcomp controls a lot more than just things that are influenced by cutree). Hence my question here.

[Boulder]

I have not.
I will make sure to try those out next after this encode, which will run for a few days.And then it will run even slower :devil:

Though... I always wondered if there was another (more efficient) way to control CUTree's strength/agressiveness more directly... (if I understand correctly qcomp controls a lot more than just things that are influenced by cutree). Hence my question here.
I wouldn't be surprised if aq-mode also played a part here as in my opinion they are related, and the default mode 2 is something I will not use based on my tests. I'm still using mode 1 with strength 0.8-1.0 depending on the source. Higher for cleaner sources.

[benwaggoner]

I wouldn't be surprised if aq-mode also played a part here as in my opinion they are related, and the default mode 2 is something I will not use based on my tests. I'm still using mode 1 with strength 0.8-1.0 depending on the source. Higher for cleaner sources.
I note from the x265 dev list that an --auto-aq mode is in development. Maybe we'll be able to stop sweating aq-mode someday.

> This patch does the following:
> 1. Automatically decides the AQ Mode for each frame, using its scene
> statistics, such as luma intensity and edge density.
> 2. Add option "--auto-aq" to enable auto detection of AQ Mode per frame.

[fauxreaper]

CUTree granularity depends on CU depth recursion. --rskip 0 or 2 reduces onion effect. More early exits from CU recursion = bigger CUs = higher chances of wrong CU propagation and artifacts in motion.

x265 RDO loves to skip blocks, and CUTree artifacts are a consequence of this. Decreasing max-merge also helps, reducing artifacts and skipped blocks. Sometimes even setting bframes=0 decrease artifacts in motion.

[Boulder]

Decreasing max-merge also helps, reducing artifacts and skipped blocks.

I've never understood why max-merge gets increased in the slower and supposedly higher quality profiles. I always have it at '2'.

[Boulder]

CUTree granularity depends on CU depth recursion. --rskip 0 or 2 reduces onion effect. More early exits from CU recursion = bigger CUs = higher chances of wrong CU propagation and artifacts in motion.

x265 RDO loves to skip blocks

I did a little test with a UHD source, downscaled to 1440p. Interesting results, and easy to see that skips and merges are happening more with the default rskip setting compared to mode 2. I didn't do any visual comparions to find any onion artifacts yet.

rskip 1 rskip 2
Intra 64x64 DC 0,00 % 0,00 %
Intra 64x64 Planar 0,00 % 0,00 %
Intra 64x64 Ang 0,00 % 0,00 %
Intra 32x32 DC 2,03 % 1,24 %
Intra 32x32 Planar 1,21 % 0,79 %
Intra 32x32 Ang 9,08 % 3,61 %
Intra 16x16 DC 0,92 % 0,36 %
Intra 16x16 Planar 0,43 % 0,15 %
Intra 16x16 Ang 8,00 % 2,46 %
Intra 8x8 DC 0,25 % 0,14 %
Intra 8x8 Planar 0,06 % 0,04 %
Intra 8x8 Ang 3,75 % 1,61 %
4x4 0,45 % 0,44 %
Inter 64x64 6,50 % 49,71 %
Inter 64x64 (Rect) 0,29 % 15,61 %
Inter 32x32 19,90 % 6,50 %
Inter 32x32 (Rect) 1,05 % 0,91 %
Inter 16x16 16,78 % 4,27 %
Inter 16x16 (Rect) 0,91 % 0,05 %
Inter 8x8 6,65 % 2,21 %
Inter 8x8 (Rect) 0,83 % 0,30 %
Skip 64x64 0,00 % 0,00 %
Skip 32x32 0,81 % 0,24 %
Skip 16x16 3,73 % 1,04 %
Skip 8x8 4,27 % 1,91 %
Merge 64x64 0,27 % 1,63 %
Merge 32x32 2,49 % 0,82 %
Merge 16x16 4,86 % 1,67 %
Merge 8x8 4,47 % 2,27 %

[Greenhorn]

the comparatively enormous number of 64x64 blocks with rskip 2 to me, though-- from a total of 6.79% (normal + rect) to 65.33% (normal + rect)!

Is there any way to get CU-type in a more readable format than setting csv-log-level=1 and viewing the generated file in an Excel/CSV viewer? Would greatly help with some comparison's I've been doing for myself.

[Boulder]

the comparatively enormous number of 64x64 blocks with rskip 2 to me, though-- from a total of 6.79% (normal + rect) to 65.33% (normal + rect)!
That puzzled me as well. Disabling early exits seems to favour bigger CUs in this case. Maybe it's something related to this test clip being HDR, so the image is quite flat compared to the graded result (edit: correctly displayed HDR) on the TV. Plenty of action, but plenty of plain white looking areas when I play the result back on my SDR display.

I checked a couple of frames where I found some onion artifacting -- happened to be next to edges -- and rskip 2 did fix them. So it's definitely helpful.

Is there any way to get CU-type in a more readable format than setting csv-log-level=1 and viewing the generated file in an Excel/CSV viewer? Would greatly help with some comparison's I've been doing for myself.
I'm afraid not. That is something I'd like to see as well, averages like I put there and also information regarding refs. All those you can see with x264 so I don't know why the statistics were not considered for x265. If I was a coder, I would definitely submit a patch for it.

[K.i.N.G]

Thanks for all the suggestions guys!
I will try them out as soon as I can.

I've never understood why max-merge gets increased in the slower and supposedly higher quality profiles. I always have it at '2'.

Funny you should mention this, as i always wondered why I seem to get less of these 'artifacts' when encoding at lower presets. Will have a look at max-merge.

I checked a couple of frames where I found some onion artifacting -- happened to be next to edges -- and rskip 2 did fix them. So it's definitely helpful.

Will try this too.

[vpupkind]

the comparatively enormous number of 64x64 blocks with rskip 2 to me, though-- from a total of 6.79% (normal + rect) to 65.33% (normal + rect)!


Do you see the difference in file size or average quantizer as well?

[benwaggoner]

Do you see the difference in file size or average quantizer as well?
Or visually?

[Greenhorn]

I just meant to say it surprised me, but forgot a word. For most of the stuff I've tried it one (just my personal collection, nothing terribly interesting) rskip 2 produces slightly larger but very similar looking encodes, so realizing that it was making such different files on a basic level was a jolt.

IDK how it behaves when you have studio-grade clean/detailed sources, but for me rskip mode 2 needs an annoying amount of tuning, because with the default edge threshold some clips are a little better than with mode 2, while others just fall apart entirely.

[benwaggoner]

I just meant to say it surprised me, but forgot a word. For most of the stuff I've tried it one (just my personal collection, nothing terribly interesting) rskip 2 produces slightly larger but very similar looking encodes, so realizing that it was making such different files on a basic level was a jolt.

IDK how it behaves when you have studio-grade clean/detailed sources, but for me rskip mode 2 needs an annoying amount of tuning, because with the default edge threshold some clips are a little better than with mode 2, while others just fall apart entirely.
You can try with the source from my encoding challenge: https://forum.doom9.org/showthread.php?t=175776.

It'd be good to try with some files that used to have artifacts with rskip and see if and how they are improved with --rskip 2.

[charliebaby]

Patman Thank you the Build x265 :-)

[stax76]

Do following x265 crf values still make sense?

--crf 10 Super High
--crf 12 Very High
--crf 14 Higher
--crf 16 High
--crf 18 Medium
--crf 20 Low
--crf 22 Lower
--crf 24 Very Low
--crf 26 Super Low

I think the values were established many years ago when 4K was not popular.

There is a request to increase the value by 4:

--crf 14 Super High
--crf 16 Very High
--crf 18 Higher
--crf 20 High
--crf 22 Medium
--crf 24 Low
--crf 26 Lower
--crf 28 Very Low
--crf 30 Super Low

It would be identical to the x264 values then which also start at 14 and end at 30.

Change or not?

[K.i.N.G]

Sorry for the late reply but our remote desktop software crashed and didn't have access to the servers...

After a quick test adding these parameters to the encode improved the quality in parts with motion "--rskip 2 --max-merge 2 --rd-refine" with about the same bitrate.
The artifacts are still there but far less (and much less perceptible when playing the video).
I'll see if i can experiment some more with the settings later, maybe it can be improved even more.

Thanks to everyone for the usefull input and suggestions!

[K.i.N.G]

Do following x265 crf values still make sense?

--crf 10 Super High
--crf 12 Very High
--crf 14 Higher
--crf 16 High
--crf 18 Medium
--crf 20 Low
--crf 22 Lower
--crf 24 Very Low
--crf 26 Super Low

I think the values were established many years ago when 4K was not popular.

There is a request to increase the value by 4:

--crf 14 Super High
--crf 16 Very High
--crf 18 Higher
--crf 20 High
--crf 22 Medium
--crf 24 Low
--crf 26 Lower
--crf 28 Very Low
--crf 30 Super Low

It would be identical to the x264 values then which also start at 14 and end at 30.

Change or not?

I wouldn't change them. Its just going to add more confusion than it will fix when using presets with different versions.
And then there's the fact it also depends on the type of source. HDR needing a lower CRF than SDR etc.

I dont think 'normalizing' the CRF with x264 is that important if it is even possible... since result will always be different depending on the footage imho.