x265 HEVC Encoder

[Boulder]

Not myself yet. I'm currently at the HPA tech retreat. I should have some physical and CPU time to run some tests next week.

I just did a very quick test myself, encoded my Hot Fuzz testclip with --hist-scenecut and it looks like the default settings are way off. 1220 frames of which 251 were I-frames :confused:.. The normal encode would have 12 I-frames (max keyint 480), which looks about the correct amount of true scene changes there.

[LigH]

How do I retrieve version 3.3+3 via git? I only get version 3.3+1 via git pull origin master (in MSYS2) and the TortoiseGit Revision Graph (in Windows) tells me this is the HEAD.

[Greenhorn]

How do I retrieve version 3.3+3 via git? I only get version 3.3+1 via git pull origin master (in MSYS2) and the TortoiseGit Revision Graph (in Windows) tells me this is the HEAD.

All 3 branches appear to be at version 3.3+1 with different commit numbers; the two patches after 3.3+1 consisted solely of merging the Master branch with the Stable and Release_3.3 branches.

(IDK anything about any VCS, so I have no idea if this is as expected or not-- but my point being, the latest files at the moment will be tagged 3.3+1.)

[benwaggoner]

I just did a very quick test myself, encoded my Hot Fuzz testclip with --hist-scenecut and it looks like the default settings are way off. 1220 frames of which 251 were I-frames :confused:.. The normal encode would have 12 I-frames (max keyint 480), which looks about the correct amount of true scene changes there.Have you looked at where the cuts are relative to the IDR frames? Is it sticking IDRs in the middle of shots?

Sent from my SM-T837V using Tapatalk

[LigH]

Let's assume those are the tiny differences between Git and Mercurial, regarding the counting of version numbers and patch increments. Hard to compare "the same build" then ... :rolleyes:
_

PS:

In previous versions I was used to reading only CMake output on the console while x265 is built. Now, suddenly, make clutters the output by reporting verbosely that it is e.g. entering and leaving directories. May that be a new default behaviour of make, of the shell, or of the media-autobuild suite having set an environment variable in a way that it changes the output also in interactive shell mode?


x265 3.3+1-g396395b2b (https://www.mediafire.com/file/gtrc6wzf1gofi9o/x265_3.3+1-g396395b2b.7z/file)

[filler56789]

LigH's latest build of x265.exe says:

version = 3.3+1-g396395b2b

whereas the latest build of x265.exe by http://msystem.waw.pl/x265/ says:

version = 3.3+1-g554c887ac

But according to the page https://bitbucket.org/multicoreware/x265/commits/ ,
those commit numbers don't exist.

:confused:

:mad:

:rolleyes:

[LigH]

You are looking in the Mercurial repository, while I got the sources via Git instead (the hash has a g prefix). Try: https://bitbucket.org/multicoreware/x265_git/commits/ (note the x265_git)

Apparently, merges don't increase the patch count for Git; they did in Mercurial, though.

[filler56789]

Yeah, git is a total mess.

Mercurial apparently is less messy.

x264, for example, doesn't add the prefix "g" to the commit number.

x264 0.159.2991 1771b55

[filler56789]

Actually git is not the culprit to blame, but MulticoreWare. It's MulticoreWare's weird strategy to prefix the hash with "g" in its cmake file (https://bitbucket.org/multicoreware/x265/src/30eb4de83092bddcb4497a47bc9d2211dfb69cfc/source/cmake/version.cmake#lines-92).

:goodpost: :thanks:

[Boulder]

Is it sticking IDRs in the middle of shots?

Yes, that's what it does. It looks like motion and camera movement confuses it a lot. It also doesn't respect the min-keyint setting, which I have at 5 but places I-frames right next to each other.

[benwaggoner]

Yes, that's what it does. It looks like motion and camera movement confuses it a lot. It also doesn't respect the min-keyint setting, which I have at 5 but places I-frames right next to each other.


Sounds like time for a bug report to MCW. Min-keyint NEEDS to be respected. Putting in a non-IDR I-frame is okay more often.

Have you tried raising the threshold value?

[Boulder]

Sounds like time for a bug report to MCW. Min-keyint NEEDS to be respected. Putting in a non-IDR I-frame is okay more often.

Have you tried raising the threshold value?

Actually I don't know which type of I-frames they are as the source filter (ffms2 in Vapoursynth) only reports I, P or B. x265 itself also reports only I-frames as well so no help from there.

I could try and file a report tomorrow and also play with the threshold value. I only made one test with the default values to see what happens.

[vpupkind]

Actually I don't know which type of I-frames they are as the source filter (ffms2 in Vapoursynth) only reports I, P or B. x265 itself also reports only I-frames as well so no help from there.

I could try and file a report tomorrow and also play with the threshold value. I only made one test with the default values to see what happens.
Have a look at the x265 logs

[LigH]

Well, ffmpeg uses the "g" prefix too.

Any idea why make became so verbose?

[filler56789]

Well, ffmpeg uses the "g" prefix too.

Blame cehoyos :D

Any idea why make became so verbose?

Just gave a try to the latest git-download from MCW's x265 and saw the problem doesn't happen to me :confused:

Maybe you'd better ask directly to the maintainers of MSYS2, or perhaps to the MABS people...

the [##%] Building etc message format is "induced" (so to speak) by CMake, ¿maybe they have changed something in their latest release? :confused:
I'm using version 3.15.2, FWIW.

My MSYS2 environment is "old" as well, its release date is
*checks notes*
2016/Oct/25

And I stopped executing the command "pacman -Syuu" ages ago for a good reason.

[LigH]

The cmake messages were all I was used to see. Now I see additional make messages; I guess there are two most possible reasons: a) the current make version is so verbose by default now; b) something in the MSYS2 environment changed, and I am not sure if MABS may have caused that.

Now it looks like this to me:

https://frupic.frubar.net/shots/39086.png

[MeteorRain]

Just so you know that Git and Mercurial are not similar at all. They have HUGE differences and I'd rather think they are completely, totally, different kinds of DVCS. Git has a completely different design, different flexibility, different concepts, etc. than HG. If you apply your knowledge from one tool onto the other you'll likely get lost.

One of the biggest difference is anything you do to the HG repository has side effects. For example, creating a branch and making some commits in HG will permanently leave records, such as affecting commit revision numbers, and leaving a permanent branch that, while hidden, cannot be deleted. In Git, branches and commits can be deleted or changed as you wish. Syncing others' work no longer creates diversity between repositories. Conflicts can be resolved in a nicer way.

The Git command line, however, was pretty hard to use. It's inconsistent in many ways. I always recommend people to stick to a great GUI and only use command lines for occasional batch process. Depending on your role it could be very difficult to work on Git CLI. For example, my mod has 4 branches, all of them being rebased frequently, so `git pull` won't work on any of them, and you have to stick to the low level way of `git fetch` and then rebase what you have onto the branches, or use `git checkout` or `git reset` (dangerous) to update current branch reference. If you are a downstream modder (who takes my patches and integrates into your own mod) you'll likely be transplanting commits across projects and branches. Things get complicated very quick.

I'm kinda surprised to read that many replies talking about Git. Hopefully you'd get used to its concept soon.

[LigH]

I noticed that MABS has a quite elaborate helper routine handling Git. So I shall preferably trust its update results. Thank you for your remark, MeteorRain.

And thanks for this workaround parameter, HolyWu.

[vpupkind]

Yes, that's what it does. It looks like motion and camera movement confuses it a lot. It also doesn't respect the min-keyint setting, which I have at 5 but places I-frames right next to each other.


Have you tried higher threshold values? The algorithm does not look at motion, only on colors and texture. They differ a lot when you have a strong scene cut, but low thresholds can result in it false positives.

[Barough]

x265 v3.3+2-gbe2d82093 (http://www.mediafire.com/file/xb0y5n6ckxu2b0j/x265-3.3%252B2-gbe2d82093_Win_GCC920.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 9.2.0)
https://bitbucket.org/multicoreware/x265_git/commits/branch/master

[Boulder]

Have you tried higher threshold values? The algorithm does not look at motion, only on colors and texture. They differ a lot when you have a strong scene cut, but low thresholds can result in it false positives.

With the default threshold, lots of false positives. Min-keyint is respected as all were i-frames. The only I-frame was right at the beginning of the file, the same in the encode with the standard scenecut detection. Due to open GOP?

Some tests; Hot Fuzz is an easy clip with clear scene cuts. The correct amount of keyframes in the testclip is 13.

Threshold 0.5, it misses all scene changes
Threshold 0.1, it misses a lot of them
Threshold 0.05, it still misses some quite obvious ones
Threshold 0.025, it begins to place too many (29)
Threshold 0.03, still too many (19)
Threshold 0.035, almost the correct amount (14)

The default encoding missed two very obvious scene cuts which the histogram one was able to find. The histogram one had one false detection in a scene of higher motion. The false one was in the middle of the scene and slightly less blurry, and I suspect that red text on a white shirt skewed the detection routine there even more.

[Boulder]

I uploaded the source clip here: https://drive.google.com/open?id=19cIzFGAocSHYlBNJdZJ08OaQbfBIHUjm

To encode, I cropped off the black borders and downscaled with Spline36 to 1280x544. The histogram sc detection picks up frame 805 incorrectly, others seem good to me.
For some reason, I am unable to tweak the --scenecut and --scenecut-bias parameters to get the standard sc detection to pick up frames 380 and 582 which are perfect cuts between scenes.

[foxyshadis]

I uploaded the source clip here: https://drive.google.com/open?id=19cIzFGAocSHYlBNJdZJ08OaQbfBIHUjm

To encode, I cropped off the black borders and downscaled with Spline36 to 1280x544. The histogram sc detection picks up frame 805 incorrectly, others seem good to me.
For some reason, I am unable to tweak the --scenecut and --scenecut-bias parameters to get the standard sc detection to pick up frames 380 and 582 which are perfect cuts between scenes.

At this point, copy-pasting everything you've discovered into the x265 bug tracker sounds like the way to go.

[Boulder]

At this point, copy-pasting everything you've discovered into the x265 bug tracker sounds like the way to go.

Agreed, I'll do that right away.

[Stereodude]

Are there no more Wolfberry x265 builds (since mid 2019) or another x265 build that can open a .avs directly without the need for piping?

[Forteen88]

I'd appreciate it if devs here, that releases x265-builds, comments on what compiling-setting they use!
My favorite is compiler-setting is "Optimization: Compile for speed", although maybe it doesn't matter that much since x265 is mostly coded in assembler.

[Stereodude]

My build (https://forum.doom9.org/showthread.php?p=1901563#post1901563) is compiled with lavf support.
Thanks for pointing that out. I had missed that detail. I hate having to pipe into x265.

[fauxreaper]

Is it viable to make a --frame-dup mode 2 without generating VFR video? I don't know why x265 spends so many bits on duplicated frames.

Example:

1) Download wallpaper on https://wallpaperscraft.com/download/planet_clouds_light_star_94996/1920x1080
2) Create avs with
ImageSource("planet_clouds_light_star_94996_1920x1080.jpg",start=0,end=199,fps=23.976)
3) Encode avs with x265 x64 10bit, crf 22, preset slow and --csv-log-level 2 --csv
4) Open csv and examine used bits

https://i.imgur.com/vSwL4KQ.png

Why frames with 100% skip 64x64 need to use more than one thousand bits? Why the bits variation on duplicated frames, shouldn't almost all of them have similar size in bits?

x264 has much less variation on bitrate with same video. x64, 10bit, crf22.

I've filed an issue on x265's bugtracker, https://bitbucket.org/multicoreware/x265/issues/537/high-bit-cost-and-bit-cost-variation-on

[jlpsvk]

change from 3.3+1 to 3.3.+2 that haven't been mention anywhere??? command give "unknown parameter" with 3.3+2

--no-rskip --> --rskip 0

[benwaggoner]

Is it viable to make a --frame-dup mode 2 without generating VFR video? I don't know why x265 spends so many bits on duplicated frames.

What is mode 2? The syntax just has --frame-dup as a Boolean.

Frame-dup should be fully compatible with constant frame rate, since it is putting in skip frames, not changing frame durations (which was how WMV9 did it back in the day).

[Greenhorn]

change from 3.3+1 to 3.3.+2 that haven't been mention anywhere??? command give "unknown parameter" with 3.3+2

--no-rskip --> --rskip 0

There's information at (https://x265.readthedocs.io/en/default/cli.html) (just search for "rskip"), or if you run x265 --fullhelp from the CLI.

Basically, --rskip 0 is (old) --no-rskip, (old) --rskip 1 is --rskip, --rskip 2 and --rskip 3 use a different method of analysis to decide when to perform the skip. --rskip-edge-threshold controls how much recursion the encoder skips, if you've set --rskip 2 or 3.

From my limited testing it seems to still need

[jlpsvk]

There's information at (https://x265.readthedocs.io/en/default/cli.html) (just search for "rskip"), or if you run x265 --fullhelp from the CLI.

Basically, --rskip 0 is (old) --no-rskip, (old) --rskip 1 is --rskip, --rskip 2 and --rskip 3 use a different method of analysis to decide when to perform the skip. --rskip-edge-threshold controls how much recursion the encoder skips, if you've set --rskip 2 or 3.

From my limited testing it seems to still need

In the docs, there it is, but nowhere in changelog... :) That's what i am talkong about.. I am using --rskip 0 (--no-rskip till now)...

[stax76]

RD Level Rskip Mode Skip Recursion Heuristic

0 - 4 1 Neighbour costs.
5 - 6 1 Comparison with inter2Nx2N.
0 - 6 2 CU edge denstiy.
0 - 6 3 CU edge denstiy with forceful skip for lower levels of CTU.


Is row 1 column 2 correct?

What is the default? --rskip 0 is the same as no --rskip ?

https://x265.readthedocs.io/en/default/cli.html#cmdoption-rskip

I hope I added it correctly.


New OptionParam With {
.Switch = "--rskip",
.Text = "Recursion Skip",
.Expand = True,
.Options = {
"Disabled",
"0 - RD Level 0-4 Neighbour costs",
"1 - RD Level 5-6 Comparison with inter2Nx2N",
"2 - RD Level 0-6 CU edge denstiy",
"3 - RD Level 0-6 CU edge denstiy with forceful skip for lower levels of CTU"},
.Values = {"", "0", "1", "2", "3"}}

[jlpsvk]

@Stax76

--rskip 0 should be DISABLED (should do the same as --no-rskip) according to docs

[Boulder]

So --rskip 1 is now the same as --rskip previously?
Once again, would be very nice to know which use cases would benefit from the other settings. To me it looks like the changes commited lately have something to do with cartoons and anime, which I never touch myself.

[nevcairiel]

RD Level Rskip Mode Skip Recursion Heuristic

0 - 4 1 Neighbour costs.
5 - 6 1 Comparison with inter2Nx2N.
0 - 6 2 CU edge denstiy.
0 - 6 3 CU edge denstiy with forceful skip for lower levels of CTU.


Is row 1 column 2 correct?

It is correct. "--rskip 1" behavior depends on the RD Level, it differs between RD Level 0-4 and 5-6.

So, your option mapping would not be correct. 0 is disabled. 1 is both those modes, depending on RD.
Default is actually 1, and not disabled, unless --tune grain is used, according to the docs.

[RainyDog]

So --rskip 1 is now the same as --rskip previously?

Appears so, yes. I've not changed my command line since the change and it just defaults to --rskip 1 now.

I'll have a play around with --rskip 2 and 3 shortly but looks like there's also a --rskip-edge-threshold setting to take into account with those too.

I didn't find the speed difference between the old --rskip and --no-skip to be worth the trade off though, so unless the speed drop-off with --rskip 2 and 3 is substantially less then I don't expect them to be of much use.

[stax76]

@nev

I think I got it now, thank you.


New OptionParam With {
.Switch = "--rskip",
.Text = "Recursion Skip",
.Expand = True,
.Options = {
"Undefined",
"0 - Disabled",
"1 - RD Level 0-4 Neighbour costs, RD Level 5-6 Comparison with inter2Nx2N",
"2 - RD Level 0-6 CU edge denstiy",
"3 - RD Level 0-6 CU edge denstiy with forceful skip for lower levels of CTU"},
.Values = {"", "0", "1", "2", "3"}},

[foxyshadis]

In the docs, there it is, but nowhere in changelog... :) That's what i am talkong about.. I am using --rskip 0 (--no-rskip till now)...

If you want to use the bleeding edge builds, you have to be prepared for things to break occasionally, and UIs to catch up later. Otherwise, stick to stable builds or the build provided by the UI.

[MeteorRain]

In the docs, there it is, but nowhere in changelog... :) That's what i am talkong about.. I am using --rskip 0 (--no-rskip till now)...

I just checked the commit log, and found that rskip changes is on future 3.4. You must have used a 3.3+X where X includes the 3.4 changes.

It could be confusing to write 3.3+X in version number indicating a future 3.4 dev version.

[Lucius Snow]

Hi all,

I'm trying to optimize the encoding speed with my Ryzen 3990X (64 cores - SMT disabled) on Windows 10 64 bit.

When encoding UHD @ 23,76 fps file, i only reach 55-58% of CPU charge.

I tried to add "--numa-pools=64" "settings but there's no change.

I currently use 3.2+38-fdbd4e4 build with VS 2019 / AVX2.

Do you have any idea how to speed up encoding?

Thank you.

[Stereodude]

Hi all,

I'm trying to optimize the encoding speed with my Ryzen 3990X (64 cores - SMT disabled) on Windows 10 64 bit.

When encoding UHD @ 23,76 fps file, i only reach 55-58% of CPU charge.

I tried to add "--numa-pools=64" "settings but there's no change.

I currently use 3.2+38-fdbd4e4 build with VS 2019 / AVX2.

Do you have any idea how to speed up encoding?

Thank you.
1) That's not the right switch. You'd want: --pools 64, but it x265 should generate 64 threads automatically if there are 64 logical processors showing to the OS.

2) It depends where the bottleneck is. Your source could be unable to feed frames to the encoder fast enough.

3) Split the encode into pieces on a scene change and run them simultaneously.

[Atak_Snajpera]

Hi all,

I'm trying to optimize the encoding speed with my Ryzen 3990X (64 cores - SMT disabled) on Windows 10 64 bit.

When encoding UHD @ 23,76 fps file, i only reach 55-58% of CPU charge.

I tried to add "--numa-pools=64" "settings but there's no change.

I currently use 3.2+38-fdbd4e4 build with VS 2019 / AVX2.

Do you have any idea how to speed up encoding?

Thank you.

Disabling smt is ultra stupid idea because you lose a lot of performance! Up to 40%. The only good idea is to split video in chunks and encode then simultanously.

[Stereodude]

Disabling smt is ultra stupid idea because you lose a lot of performance! Up to 40%. The only good idea is to split video in chunks and encode then simultanously.
Windows 10 generally doesn't support more than 64 logical cores per NUMA node, so disabling SMT is the best way to get the most performance out of a 3990 under Windows 10 unless he uses Windows 10 Enterprise which can use all 128. Using 64 real cores is better than 64 logical cores across 32 real cores.

[Lucius Snow]

Exactly StereoDude.

I use Windows 10 LTSC so I have better performances with SMT disabled.

[Lucius Snow]

Interesting details:

I have a little better performance on Linux (Centos 8.1) but still far from 100% of CPU charge.

However, I made this X265 benchmark test: http://www.xin.at/x265/index-en.php

I reached 00:40:09:644 on Linux and 01:14:24.446 on Windows.

[Atak_Snajpera]

Windows 10 generally doesn't support more than 64 logical cores per NUMA node, so disabling SMT is the best way to get the most performance out of a 3990 under Windows 10 unless he uses Windows 10 Enterprise which can use all 128. Using 64 real cores is better than 64 logical cores across 32 real cores.

In video encoding 64 logical cpus per CPU group is not a problem. You just have to run additional x265 instances. Problem solved! Disabling smt is a bizarre workaround to low CPU usage by video encoder.

[MeteorRain]

Up to 40%.

Do you have benchmarks to support that? I thought we'd lose at most 20% performance by not using SMT but that's just my wild guess. Would love to see actual numbers showing the difference.

[Stereodude]

What is an exception code of 0xc0000005 in x265?
Faulting application name: x265.exe, version: 3.3.0.1, time stamp: 0x5e4bfd9a
Faulting module name: x265.exe, version: 3.3.0.1, time stamp: 0x5e4bfd9a
Exception code: 0xc0000005
Fault offset: 0x00000000004fb4d2
Faulting process id: 0x23a4
Faulting application start time: 0x01d5f9aa83090eef
Faulting application path: C:\HDTV Tools\x265\x265.exe
Faulting module path: C:\HDTV Tools\x265\x265.exe
Report Id: 921cdb5a-b329-4dac-b876-d60ff8a06c73
I made a few changes to my x265 command line and have gotten the same exception code twice in 10 hours (out of 8 simultaneous encodes on two different encode segments) mid encode after never seeing it ever before of many days of simultaneous encoding on the same system with the same version of x265. Frankly, I've never seen this error before from any version of x265 on any system before with any combination of command line switches.

I changed from this:
START "Enc #6" /NORMAL /NODE 0 /AFFINITY 00000F00 "x265.exe" --pools 4 -F 1 --crf 16.0 -p veryslow --no-sao --aq-mode 1 --aq-strength 1.15 --vbv-maxrate 25000 --vbv-bufsize 25000 --level 5.0 --keyint 120 --open-gop -D 10 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --qpfile 6.chp -o "out_6.265" "in_6.avs"
to this:
START "Enc #6" /NORMAL /NODE 0 /AFFINITY 00000F00 "x265.exe" --pools 4 -F 1 --crf 16.0 -p veryslow --aq-strength 1.15 --vbv-maxrate 25000 --vbv-bufsize 25000 --level 5.0 --keyint 120 --open-gop -D 10 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --qpfile 6.chp -o "out_6.265" "in_6.avs"

The video has the following properties:
AVSMeter 2.9.8 (x64), 2012-2020, (c) Groucho2004
AviSynth+ 3.4 (r2925, master, x86_64) (3.4.0.0)

Number of frames: 30565
Length (hh:mm:ss.ms): 00:21:14.815
Frame width: 1920
Frame height: 1080
Framerate: 23.976 (24000/1001)
Colorspace: YUV420P10

I restarted the first of the two segment that crashed and it has made it past the prior crash point.

I'm using HolyWu's build (https://forum.doom9.org/showthread.php?p=1901563#post1901563) if that matters.

[Stereodude]

In video encoding 64 logical cpus per CPU group is not a problem. You just have to run additional x265 instances. Problem solved! Disabling smt is a bizarre workaround to low CPU usage by video encoder.
*sigh*
Disabling SMT is how you optimize the performance of an application that will only run on a single NUMA node with a processor like a 3990. The FPS of his single encode with SMT disabled from his single x265 instance is significantly higher than it would be if SMT was enabled. I'd estimate 70-80%.

Is there an alternative workaround for maximizing x265 performance with running multiple simultaneous encodes, yes.