x265 HEVC Encoder

[Midzuki]

https://cosgan.de/images/smilie/konfus/g080.gif

307 patches with AVX-512 (and other improved assembly) code uploaded to the developer mailing list. That will take a little while to review.

They are up and running 0_o

https://bitbucket.org/multicoreware/x265/commits/all

[LigH]

Damn. I waited for the "Re: 0/307 — approved" mail.

Time to build.
_

P.S.: Compiling x265 with AVX-512 support works only for x86-64 architecture targets. A "bailout" for x86 (Win32) architecture targets seems to be missing, so it throws "invalid opcode" errors for the 8-bit depth core where assembler is still enabled.
_

x265 2.7+332-593e63cda903 (Win64) (http://www.mediafire.com/file/vm9jyrb6k40vc2h/x265_2.7%2B332-593e63cda903_Win64.7z)

Support for AVX-512 assembly optimized kernels; remember: enable it manually by adding --asm avx512 to the CLI — and don't fry your CPU...

Only x86-64 (Win64) version available, skipping it in x86 (Win32) mode for NASM is necessary not to break compilation completely.

[hajj_3]

anyone with an avx512 capable processor fancy doing benchmarks comparing it to the previous build?

[LigH]

I already have that feeling that one day, x265 will be used rather as a benchmark for the efficiency of the AVX implementations in a specific CPU, rather than as a benchmark for efficient video encoding ... :o

[Midzuki]

More AVX-512 code = bigger filesize :scared:

https://forum.videohelp.com/attachments/45153-1523497745/x265-with-tons-of-AVX512.png

[FranceBB]

Yep, but in 2018 20.7 MB is still very small and the increase is negligible. Unfortunately I can't test the latest AVX512 instruction set 'cause I have a Intel Xeon E5-2660 v4 that supports AVX2 only, sadly.
I look forward for benchmarks.

[foxyshadis]

I thought having Kaby Lake meant I had them, but nope, servers only. I have one customer who has a brand spanking new Skylake-X server that I can remote into, I should be able to get benchmarks tomorrow.

[Asmodian]

AVX-512 is faster! :D;)

I did some benchmarks using LigH's build x265 2.7+332-593e63cda903 (Win64) (https://forum.doom9.org/showthread.php?p=1839225#post1839225) above. I used the same build for the AVX2 tests, simply without the "--asm avx512" command.
i9-7900X @ 4.5 GHz all cores, 3.0 GHz mesh/cache, DDR4 4000-17-18-18-41-1T. No AVX2 or AVX-512 multiplier offsets. Max 92 degC package CPU temperature during both veryslow encodes. The faster modes did not saturate all 20 threads.

The source is 1920x1080 8-bit gradient MagicYUV 4:2:0 on a NvME SSD encoding to another NvME SSD. I used the first 1000 frames from Firefly episode 9 which I had already denoised (SMDegrain) and had on my drive.

avs2pipemod.exe -y4mp=1:1 "fireflyshort.avs" | x265_AVX512.exe --input - --y4m -o "D:\temp\fireflyshort.mkv" --asm avx512 --preset veryslow --crf 18.5 --output-depth 10
x265 [info]: HEVC encoder version 2.7+332-593e63cda903
x265 [info]: build info [Windows][GCC 7.3.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
x265 [info]: Main 10 profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 20 threads

veryslow:
AVX512: encoded 1000 frames in 303.44s (3.30 fps), 4037.41 kb/s, Avg QP:20.64
AVX2: encoded 1000 frames in 335.83s (2.98 fps), 4037.41 kb/s, Avg QP:20.64
medium:
AVX512: encoded 1000 frames in 28.41s (35.20 fps), 3183.67 kb/s, Avg QP:20.46
AVX2: encoded 1000 frames in 30.71s (32.57 fps), 3183.67 kb/s, Avg QP:20.46
veryfast:
AVX512: encoded 1000 frames in 15.47s (64.64 fps), 2769.26 kb/s, Avg QP:20.89
AVX2: encoded 1000 frames in 16.89s (59.20 fps), 2769.26 kb/s, Avg QP:20.89
ultrafast:
AVX512: encoded 1000 frames in 6.86s (145.77 fps), 1398.46 kb/s, Avg QP:25.00
AVX2: encoded 1000 frames in 7.22s (138.41 fps), 1398.46 kb/s, Avg QP:25.00

Thanks to everyone who works on x265 and thanks for the regular builds LigH. :)

[excellentswordfight]

Slower here.

Using LGHs build with a dell 2u rack server with a Xeon Gold 6126 (12c/24t). CPU utilization dropped with about 10% (both for 1080p and 2160p) and clockspeed dropped from 2.9Ghz to 2.4Ghz. I'm guessing that the gains for AVX512 didnt outweight the dropp in clockspeed and utilization.

Tears of steal source (10bit UHD-Bluray compat x265 source for 2160p test, 8bit bluray compat x264 soruce for 1080p)

2160p with avx512: 80-90% CPU usage, 2.28 fps
--asm avx512 --preset slow --profile main10 --level-idc 51 --crf 22


2160p: 100% CPU usage, 2.36 fps
--preset slow --profile main10 --level-idc 51 --crf 22


1080p with avx512: 45-55% CPU usage, 6.54 fps
--asm avx512 --preset slow --profile main10 --level-idc 41 --crf 18


1080p: 55-65% CPU usage, 7.14 fps
--preset slow --profile main10 --level-idc 41 --crf 18

[WhatZit]

I'm guessing that the gains for AVX512 didn't outweight the drop in clockspeed and utilization.

Yep, a Catch-22 also discovered by Cloudfare after some cryptography assessments: https://blog.cloudflare.com/on-the-dangers-of-intels-frequency-scaling/

[nevcairiel]

Asmodian runs without a AVX512 offset, which would instantly crash his system if a strong AVX512 workload would run, so clearly its faster with some "light" AVX512 usage. Usually you need at least a -10 offset or such to get it working stable under strong AVX512 load (or boost voltages substantially for more heat). Non-OCed Xeon CPUs probably downlock quite substantially.

[Barough]

x265 v2.7+337-54ff74d2b635 (http://www.mediafire.com/file/1wohb1plbu02qvy/) (GCC 7.3.0, 32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[burfadel]

Probably best to utilise AVX-512 where it gives the best gains without triggering thermal throttle. The good thing at least with 307 separate patches this can be whittled down. If a function is frequently used and gives only a small gain, it may actually encode faster if on mitred fire to the throttling the patch causes. Even if throttling isn't triggered on a particular rig, temperature difference should be taken into account to cover typical situations.

[LigH]

x265 2.7+337-54ff74d2b635 (http://www.mediafire.com/file/m1m441ep3eeyy5g/x265_2.7%2B337-54ff74d2b635.7z)

Merge with default; prep for v3.0
Support for HLG-graded content and pic_struct
Fix conditions for single-sei NAL
Fix 32 bit build error (means: AVX-512 support is only included in x86-64 architecture target)

(VMAF support to report per frame and aggregate VMAF score — unfortunately not yet? available for Windows builds)

New CLI parameters:

--atc-sei <integer> Emit the alternative transfer characteristics SEI message where the integer is the preferred transfer characteristics. Default disabled
--pic-struct <integer> Set the picture structure and emits it in the picture timing SEI message. Values in the range 0..12. See D.3.3 of the HEVC spec. for a detailed explanation.

[Asmodian]

Asmodian runs without a AVX512 offset, which would instantly crash his system if a strong AVX512 workload would run, so clearly its faster with some "light" AVX512 usage. Usually you need at least a -10 offset or such to get it working stable under strong AVX512 load (or boost voltages substantially for more heat). Non-OCed Xeon CPUs probably downlock quite substantially.

I had downclocked from my normal max clocks when running without an AVX offset.

I also ran some tests at my normal OC settings with -2, -4 multiplier offsets. 4.8 GHz max core, 4.6 GHz AVX2, 4.4 GHz AVX-512.

AVX512: encoded 1000 frames in 310.46s (3.22 fps), 4037.41 kb/s, Avg QP:20.64
AVX2: encoded 1000 frames in 335.85s (2.98 fps), 4037.41 kb/s, Avg QP:20.64

It would probably still melt with a heavy AVX-512 load but it also wasn't completely maxed. AVX-512 ran cooler than AVX2 at these settings. I am not sure why my AVX2 run only had the same speed as the previous 4.5 GHz encode, maybe a latency penalty due to the core changing states.

This is a binned, delidded, and water cooled CPU... other systems may have different results. :)

Edit: If I run Prime95 (p95v294b8) with AVX-512 at 4.5 GHz I do get thermal throttling.

[nevcairiel]

Edit: If I run Prime95 (p95v294b8) with AVX-512 at 4.5 GHz I do get thermal throttling.

Try with LinX/Linpack and see your system die. Prime95 does not fully use AVX512 yet (only trial factoring, not full FFTs)

[Stephen R. Savage]

Try with LinX/Linpack and see your system die. Prime95 does not fully use AVX512 yet (only trial factoring, not full FFTs)

It's actually not so bad at higher frequencies, because each 100 MHz increment saves a lot more power, compared to 2.5 GHz server SKUs. i9-7900X can reach 4.1-4.2 GHz AVX-512 frequency with an aftermarket cooling solution.

[jlpsvk]

x265 has static levels of refinement(--refine inter <level>/refine intra <level>) which can be used with --analysis-reuse-level 10.
Efficiency in terms of quality increases as the levels of refinement increases. This quality increase results from additional computation thereby increasing the overall encoding time.
For a better quality-speed trade-off, dynamic refinement was introduced where the encoder dynamically switches between different inter refine levels.
This basically exploits the fact that not all CUs are required to be encoded with same level for better performance/quality.
Considering the complexity of video content and the analysis information from first pass, the encoder can intelligently decide the optimal level of refinement for each CU.
Intra frames are usually encoded with best quality as they are used as references by the consecutive frames. Hence error introduced in intra frames due to reusing analysis data can propagate to frames that use these intra frames as reference.
To minimize the chances of error propagation, refine-intra 4 (level with best quality) restricts reusing analysis data for intra frames and forces the encoder to perform full intra analysis in the second pass.
This is why x265 documentation suggests to use dynamic refinement along with refine-intra 4 and this setting is expected to give improved quality than other refine intra levels for some videos.

any suggested quality wise settings recommendation for 4K HDR encoding? with CRF ie 17? :)

[nevcairiel]

It's actually not so bad at higher frequencies, because each 100 MHz increment saves a lot more power, compared to 2.5 GHz server SKUs. i9-7900X can reach 4.1-4.2 GHz AVX-512 frequency with an aftermarket cooling solution.

You can reach that if you boost the power you give the CPU, but unfortunately that also boosts the power outside of AVX512 mode, making your CPU overall less efficient. The integrated voltage controller has no option to increase the core voltage only in AVX512 mode, unfortunately.

But this is probably going a bit off-topic for X265. :)
I would've thought the X265 people already learned the down-clocking lesson with AVX2 though, where they experienced the same effect - fancy instructions that made the overall encode slower, especially on server systems, due to clock changes.

[mandarinka]

https://forums.anandtech.com/threads/intel-skylake-kaby-lake.2428363/page-662#post-39149633

RZN vs. CFL vs. SKL-X in X265 2.5+31:

RZN: /w AVX2 = 100.00%, /wo AVX2 = 105.21%
CFL: /w AVX2 = 130.61%, /wo AVX2 = 101.13%
SKL-X: w/ AVX2 = 135.47%, /wo AVX2 = 105.21%

Ryzen's performance without AVX2 is impressive, but it is sad to see that there is still a penalty (like on Excavator) when running 256-bit code.

I wish somebody would adjust the CPU detection code to disable AVX2 on Zen. Easy performance gain just from that simple change: Zen gets 5.2% faster by disabling AVX2.

[jlpsvk]

is it just me? using cpu capabilities with the new x265 not listing AVX-512. :( i7-7820X

[Asmodian]

Don't forget the "--asm avx512", it isn't enabled by default. This seems good if it is slower on most systems due to the multiplier offsets for AVX-512.

[jlpsvk]

@Asmodian

aaaaah... forgot it.. :D

[nevcairiel]

To be fair, the AVX2 speedup is still larger than the frequency penalty, so it made sense.

It is now, because they reigned in AVX2 use in some irrelevant functions with minimal speedups to reduce the effect of downclocks. They even had a presentation about that "adventure" and their findings on some conference once

[RieGo]

no AVX512:
encoded 1780 frames in 169.38s (10.51 fps), 2541.97 kb/s, Avg QP:20.40
AVX512:
encoded 1780 frames in 161.94s (10.99 fps), 2541.97 kb/s, Avg QP:20.40

makes a ~5% speed increase
considering avx512 encode was almost 10°C cooler, so maybe i can get away with +100MHz. i like it :D

[Selur]

okay, so nice, but not worth buying a new cpu because of it.

[Xizer]

Is anyone else having problems with getting it to work on Skylake Xeons?

x265 crashes on my Xeon Platinum 8176 server when I start it with the --asm avx512 flag.

Error: fwrite() call failed when writing frame: 3, plane: 2, errno: 32
Output 80 frames in 13.46 fps (5.90 fps)

It'll work fine on the Xeon Platinum machine as soon as I remove the avx512 flag.

And it works on my i9 7940X with the avx512 flag.

[WhatZit]

okay, so nice, but not worth buying a new cpu because of it.

Not until the i7-9700K (https://en.wikichip.org/wiki/intel/core_i7/i7-9700k) shows up (November 2018?).

[Ma]

x265 crashes on my Xeon Platinum 8176 server when I start it with the --asm avx512 flag.

One bug that can lead to crash is fixed in version 2.7+338 so please do not use older versions. Which version do you use?

[RieGo]

i'm seeing very inconsistent results

CRF-20 preset-"medium"
--------------------
Run1:
encoded 1128 frames in 27.42s (41.14 fps), 3355.47 kb/s, Avg QP:20.21
Run2 (AVX512):
encoded 1128 frames in 30.23s (37.32 fps), 3355.47 kb/s, Avg QP:20.21
Run3:
encoded 1128 frames in 27.77s (40.62 fps), 3355.47 kb/s, Avg QP:20.21
Run4 (AVX512):
encoded 1128 frames in 27.04s (41.71 fps), 3355.47 kb/s, Avg QP:20.21

but temps are still very low on avx512. i didn't expect this. probably x265 just uses very little avx512 and that's why we don't see much improvement - if any - running on lowered frequency.
maybe we can run at default speed without getting too much heat/power, if we only use it on x265... i'll give it a try

btw: i agree that avx512 is not worth getting a new cpu :D

[LigH]

x265 2.7+340-aa9102400f24 (https://www.mediafire.com/file/k0qeet33wv0053d/x265_2.7%2B340-aa9102400f24.7z)

remove unused asmname from x265_param; added a newline in the help
(fixed VMAF warning not applicable under Windows)

[Xizer]

x265 2.7+340-aa9102400f24 (https://www.mediafire.com/file/k0qeet33wv0053d/x265_2.7%2B340-aa9102400f24.7z)

remove unused asmname from x265_param (may fix some crashes on Xeons); added a newline in the help
(fixed VMAF warning not applicable under Windows)

2.7+340 is the version I'm using when it crashes.

[LigH]

Sorry to hear ... so the fix is not yet committed, only proposed?

[Ma]

2.7+340 is the version I'm using when it crashes.

Thanks for more info. It looks like more serious bug (x265 works for a while). The error message is probably from decoding app that writes graphics data via pipe to x265.

There are many possible reasons for the crash, one of them is the OS (and msvcrt.dll file in your OS). Could you test VS 2015 and VS 2017 x265 binaries? They are not based on msvcrt.dll.

[LigH]

I don't have such compilers installed; someone else may have to build them.

[foxyshadis]

x265-2.7+336-07defe235cde.7z VS 17 x64 (https://www.dropbox.com/s/hfyp95irk7j145i/x265-2.7%2B336-07defe235cde.7z?dl=0), crt is statically linked so no install needed. Debug pdbs included.

[Ma]

Bisection is one of the most common method of finding bugs.
In source file common/x86/asm-primitives.cpp it is function pointers assignment with avx512 code -- from line # 4696 to 5385.
We can turn off (comment) first half of this functions (x265-1.exe) and then second half (x265-2.exe).
You can download avx512-bisect1.7z (http://www.msystem.waw.pl/x265/avx512-bisect1.7z) VS 2015 binaries with diff files -- if x265.exe (from clean sources) hangs, please try x265-1.exe and then x265-2.exe -- it should be that one of them hangs and second not.
Please report back the results (which from the 3 binaries hangs and which not).

[Xizer]

Bisection is one of the most common method of finding bugs.
In source file common/x86/asm-primitives.cpp it is function pointers assignment with avx512 code -- from line # 4696 to 5385.
We can turn off (comment) first half of this functions (x265-1.exe) and then second half (x265-2.exe).
You can download avx512-bisect1.7z (http://www.msystem.waw.pl/x265/avx512-bisect1.7z) VS 2015 binaries with diff files -- if x265.exe (from clean sources) hangs, please try x265-1.exe and then x265-2.exe -- it should be that one of them hangs and second not.
Please report back the results (which from the 3 binaries hangs and which not).

All three binaries crash with the avx512 flag I'm afraid.

Is there additional information that can be provided for debug?

[Ma]

All three binaries crash with the avx512 flag I'm afraid.

Is there additional information that can be provided for debug?

Thanks for the info.

Could you post results of the command
x265 -V && x265 --asm avx2 -V && x265 --asm avx512 -V
For example in my system it looks likeF:\x265p\ma\avx512>x265 -V && x265 --asm avx2 -V && x265 --asm avx512 -V
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512


This bug could be not related directly to avx512 code -- could you check if it hangs if you use '--asm avx2' instead of '--asm avx512' (it is important to use --asm avx2 option).
------------------------------------------------------
Your OS is Win 8.1 that do not support avx512. This bug in x265 is not technical but conceptual -- avx512 is not auto recognized by default so option '--asm avx512' should not turn on avx512 without any check.
You can test file avx512-patch.7z (http://www.msystem.waw.pl/x265/avx512-patch.7z) with x265 that check what is supported by CPU & OS up to avx512 if option '--asm avx512' is used.
On Win 8.1 it should work with '--asm avx512' exactly like without this option, on Win 10 it should turn on avx512 if you have CPU with avx512 and you use option '--asm avx512'.

[Bhavnahari]

any suggested quality wise settings recommendation for 4K HDR encoding? with CRF ie 17? :)

If the data that is being reused comes from encoding a downscaled video (scale-factor=2), it does not make a lot of sense to use intra/inter refinement levels 0/1 as levels 0 and 1 reuse most of the information from the previous pass with no or minimal re-evaluation of analysis information. This can have a huge impact on the quality of the encode.

We have observed that --refine-inter=3 and --refine-intra=4 gives the best quality, even better than standalone x265 encodes in some cases, with a performance gain of up to 1.8X.

For 4K HDR content, you will have to modify the display settings based on the monitor. Please refer the docs for more information - http://x265.readthedocs.io/en/default/cli.html#vui-video-usability-information-options

[jlpsvk]

If the data that is being reused comes from encoding a downscaled video (scale-factor=2),
it does not make a lot of sense to use intra/inter refinement levels 0/1 as levels 0 and 1 reuse most of the information from
the previous pass with no or minimal re-evaluation of analysis information. This can have a huge impact on the quality of the encode.

We have observed that --refine-inter=3 and --refine-intra=4 gives the best quality, even better than standalone x265 encodes in some cases,
with a performance gain of up to 1.8X.

My settings then...

--crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock
--no-sao --no-strong-intra-smoothing --high-tier --refine-inter 3 --refine-intra 4

Of course, display settings are entered too... :)

Warning:
x265 [warning]: Intra refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling intra refine.
x265 [warning]: Inter refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling inter refine.

Can it be used with CRF? Or? Some parameters missing?

[Loomes]

So far no success for me running the avx512 option on my 7820x on Windows 10 64it. I tried
ffmpeg -i source.mkv -f yuv4mpegpipe - | x265.exe --preset slow --asm avx512 --crf 21 --y4m - -o dest.h265
which crashes immediately. Using --asm avx2 works fine.

I ran
x265.exe -V && x265.exe --asm avx2 -V && x265.exe --asm avx512 -V
and it showed
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512


You can test file avx512-patch.7z (http://www.msystem.waw.pl/x265/avx512-patch.7z) with x265 that check what is supported by CPU & OS up to avx512 if option '--asm avx512' is used.

I tried from that package x265.exe, x265-1.exe and x265-2.exe. All of them crashed immediately. I am using the latest nightly version of ffmpeg.
Am I missing something?

[Asmodian]

I tried from that package x265.exe, x265-1.exe and x265-2.exe. All of them crashed immediately. I am using the latest nightly version of ffmpeg.
Am I missing something?

I had crashes using the builds foxyshadis posted but I was successful using LigH's build from this post (https://forum.doom9.org/showthread.php?p=1839464#post1839464). Win10 and an i9-7900X. Have you tried them?

[Loomes]

I had crashes using the builds foxyshadis posted but I was successful using LigH's build from this post (https://forum.doom9.org/showthread.php?p=1839464#post1839464). Win10 and an i9-7900X. Have you tried them?
I did now but it also results in an instant crash. Are you using a ffmpeg pipe like me or what is your command line?

[Ma]

I tried from that package x265.exe, x265-1.exe and x265-2.exe. All of them crashed immediately. I am using the latest nightly version of ffmpeg.
Am I missing something?

x265-1.exe == x265-2.exe == x265.exe from package avx512-bisect1.7z (I commented out code for 8-bit encoding and compiled 10-bit version, sorry for that). So this package is to forget.

I want to precise: x265.exe from package avx512-patch.7z (http://www.msystem.waw.pl/x265/avx512-patch.7z) gives in your OS output:
x265 --asm avx512 -V
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
?

[Loomes]

I want to precise: x265.exe from package avx512-patch.7z (http://www.msystem.waw.pl/x265/avx512-patch.7z) gives in your OS output:
x265 --asm avx512 -V
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
?
Confirmed. I tried again on Windows 10 64bit with an 7820x using the command line shown above.

[Asmodian]

I did now but it also results in an instant crash. Are you using a ffmpeg pipe like me or what is your command line?

No, actually I was using avs2pipemod.exe (ver 1.1.1 Aug 15, 2016) piping to x265.exe with y4m:
C:\Tools\avs2pipemod.exe -y4mp=1:1 %1 | C:\Tools\x265.exe --input - --y4m -o "D:\Temp\%~n1.mkv" --asm avx512 --preset veryslow --crf 18.5 --output-depth 10

edit:
"x265 --asm avx512 -V" gives:
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][GCC 7.3.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512

[Ma]

Confirmed. I tried again on Windows 10 64bit with an 7820x using the command line shown above.

Thanks for confirmation -- it looks like your CPU & OS supporting avx512.

I've prepared some binaries to narrow this bug -- avx512-bisect2.7z (http://www.msystem.waw.pl/x265/avx512-bisect2.7z).
x265-0.exe -- 10-bit encoder with commented out all avx512 function pointers (it works on my i7 8700 with --asm avx512 option). It is for check if the bug is before avx512 code is running but the execution path is exactly like in avx512 case.

x265-01.exe -- assigned only first 1/4 function pointers with avx512 code.
x265-02.exe -- assigned only second 1/4 function pointers with avx512 code.
x265-03.exe -- assigned only third 1/4 function pointers with avx512 code.
x265-04.exe -- assigned only last 1/4 function pointers with avx512 code.

Could you test these 5 exe's and report back which works and which not?

[Loomes]

Could you test these 5 exe's and report back which works and which not?
I'm home in about 6 hours and will do. By the way: Is it correct that --asm avx512 will not work on Windows 7/8 64bit, no matter what?

[nevcairiel]

Is it correct that --asm avx512 will not work on Windows 7/8 64bit, no matter what?

That would appear to be the case. If you force avx512 usage through --asm avx512 on an other OS it'll just crash.

Thats really the problem with using the --asm option, it overrides the CPU/OS feature detection, a different option to enable it when present would be better.

[Loomes]

Could you test these 5 exe's and report back which works and which not?
I am afraid, none of the x265-xx.exe worked in my Windows 10 and crashed instantly. But I found something else: I tried to use directly the libx265 of ffmpeg by
ffmpeg-2018-04-18.exe -i source.mkv -c:v libx265 -x265-params crf=21 -preset slow -pix_fmt yuv420p10le -bsf hevc_mp4toannexb -f rawvideo target.h265
and it also crashed instantly. Then I changed this new ffmpeg-2018-04-18.exe for an older one ffmpeg-2018-03-24.exe and it worked. Then I ran again your x265-xx.exe files with the older ffmpeg-2018-03-24.exe but they all crashed again with
ffmpeg-2018-03-24.exe -i source.mkv -f yuv4mpegpipe - | x265-0.exe --preset slow --crf 21 --y4m - -o target.h265
Even more strange that ffmpeg-2018-04-18.exe works flawlessly on Windows 7 and so the x265-0.exe of your files does (it even claims to make use of AVX512 when it starts). All the other x265-0x.exe crashed immediately on Windows 7.

This is really weird. Let me know if I can provide you any more info of my system or do some more tests.

[Ma]

Thanks for the details. It is quite a riddle.
On Win7 everything is OK, x265-0 should work (all avx512 code is not active), x265-0x should crash.
On Win10 it is interesting -- in your 2 examples it is no '--asm avx512' option. It crashes without avx512 too?
Is something special with your Win10, for example it works in VirtualBox?

[Loomes]

On Win10 it is interesting -- in your 2 examples it is no '--asm avx512' option. It crashes without avx512 too?
Yes, I had --asm avx512 included in the first run and all crashed. Then I had the idea that it's about ffmpeg and let it run without the asm option but well, did not work either.

Is something special with your Win10, for example it works in VirtualBox?
No, not at all. It is Windows 10 Professional 64bit, nothing special about it as far as I know. The system drive has 14GB free space, the drive where x265, ffmpeg etc. is running with 80G free space. As I said, it works fine with the older ffmpeg version or older x265.exe.
Is it possible that there is something special about the 7820x CPU?

[Ma]

Is it possible that there is something special about the 7820x CPU?

It is possible that there are hardware problems -- easiest way to check this out is Prime95 https://www.mersenne.org/download/
If you run this app with button 'Just Stress Testing' it should find memory/CPU problems (in file 'results.txt' should be only [date] and XXX passed! entries).

There was some changes in ffmpeg, so you can test ffmpeg release 3.4.2 -- it should be different approach to thread synchronization.

There was some reports about x265 hangs with pipe input from ffmpeg but it is hard to reproduce -- maybe there is bug in x265 that is not fixed yet.

[Bhavnahari]

My settings then...

--crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock
--no-sao --no-strong-intra-smoothing --high-tier --refine-inter 3 --refine-intra 4

Of course, display settings are entered too... :)

Warning:
x265 [warning]: Intra refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling intra refine.
x265 [warning]: Inter refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling inter refine.

Can it be used with CRF? Or? Some parameters missing?


refine-inter and refine-intra features may be used with any rate control techniques including ABR, CRF and CQP. However, as indicated by the warnings, these features are enabled only for scaled save/load encodes. The "save" run will encode the content of resolution WxH (1920x1080, say) and generate an analysis.dat file. This analysis information is scaled up by the factor and used by the "load" run which works on the content of resolution 2Wx2H (3840x2160, say). Using the analysis information from a low-res encode to encode high-res content will impact the quality of the encode hence we try to minimize the repercussions by incorporating various intra and inter refinement strategies.

Your CLI should be:
HD encode : --crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-save <filename> --analysis-reuse-level 10 --scale-factor 2

UHD encode: -crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-load <filename> --analysis-reuse-level 10 --scale-factor 2 --refine-inter 3 --refine-intra 4

[jlpsvk]

refine-inter and refine-intra features may be used with any rate control techniques including ABR, CRF and CQP. However, as indicated by the warnings, these features are enabled only for scaled save/load encodes. The "save" run will encode the content of resolution WxH (1920x1080, say) and generate an analysis.dat file. This analysis information is scaled up by the factor and used by the "load" run which works on the content of resolution 2Wx2H (3840x2160, say). Using the analysis information from a low-res encode to encode high-res content will impact the quality of the encode hence we try to minimize the repercussions by incorporating various intra and inter refinement strategies.

Your CLI should be:
HD encode : --crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-save <filename> --analysis-reuse-level 10 --scale-factor 2

UHD encode: -crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-load <filename> --analysis-reuse-level 10 --scale-factor 2 --refine-inter 3 --refine-intra 4

Thanks!!! So it won't downscale and upscale video? Only stats? Right? :) So it's better for retain quality/details or not? :)

[Boulder]

When watching some x265 encoded content on my TV, I can see very, very thin greyish vertical lines in scenes that are almost or completely black. Is this due to slice boundaries?

[foxyshadis]

When watching some x265 encoded content on my TV, I can see very, very thin greyish vertical lines in scenes that are almost or completely black. Is this due to slice boundaries?

That'd more likely be tile boundaries. Definitely sounds like a decoder bug.

[Boulder]

(I did some more testing and I cannot see the strips if lookahead-slices was disabled (vertical resolution less than 720).) EDIT: tested with existing encodes - couldn't replicate this with the latest x265 using the same source.

This is how it looks on the TV and on my display, a snippet from the upper left corner (seriously exaggerated by getting the brightness real high):

https://imgur.com/a/gO45OYB

I recall seeing this also with x264 encoded videos. Different decoders have been used, on the computer it's the NVIDIA GPU and on the TV it's whatever is on Apple TV 4K - VideoToolBox I think.

EDIT: this older clip shows the thing at the start, turn up the brightness to see it: https://drive.google.com/open?id=1uibj4uJdaZAMf_Vvu8b7A0ob9nqtJKa2

[jlpsvk]

just an addition to x265 AVX-512 problems... :)

ASRock X299 OC Formula + i9-7940X + Win10 x64 ... no crashing... AVX512 working flawlessly. :)

Binary from here (compiled in VS2017):
http://msystem.waw.pl/x265/x265-2.7+340-aa91024_vs2017.7z

[Bhavnahari]

Thanks!!! So it won't downscale and upscale video? Only stats? Right? :) So it's better for retain quality/details or not? :)

The input video used to generate the analysis file(stats) in "save" encode must be the downscaled version of the video used to encode "load", x265 scales only the analysis data, not the input video.

The refinement techniques are to improve the speed of the encode with similar quality when compared to encoding the high-res video without any prior analysis information. Among the different refinement levels, --refine-inter 3 --refine-intra 4 gives the best quality retention.

[Loomes]

ASRock X299 OC Formula + i9-7940X + Win10 x64 ... no crashing... AVX512 working flawlessly. :)
Good for you ;P
What is your command line? Are you using ffmpeg?

[jlpsvk]

Good for you ;P
What is your command line? Are you using ffmpeg?

No... x265 ... 5 instances at once (RipBot264 Distr. encoding) to penetrate all cores at 100%. :)

My current command line:
--profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info
--no-deblock --no-sao --no-strong-intra-smoothing --high-tier --asm avx512

[LigH]

No... x265 ... 5 instances at once (RipBot264 Distr. encoding) to penetrate all cores at 100%. :)

This requires a powerful fan ... :cool:

https://technabob.com/blog/wp-content/uploads/2009/04/computer-fan-1.jpg

[jlpsvk]

This requires a powerful fan ... :cool:

Nope... Noctua NH-D15 keeping it at 63 degrees. :)

[Loomes]

Nope... Noctua NH-D15 keeping it at 63 degrees. :)
That's great but what is your frequency and your AVX offset? Because that's what makes the heat.

[jlpsvk]

That's great but what is your frequency and your AVX offset? Because that's what makes the heat.

All 14 cores running at 3.7GHz (instead of 3.1 stock).

[RieGo]

All 14 cores running at 3.7GHz (instead of 3.1 stock).

so no AVX512 offset?

i got really weird results while using avx512 and x265:
A) temps don't really go much higher while using avx512, at least on my system
B) AVX512 offset almost doesn't make a difference. i had same performance on 12x 100MHz difference in offset

[Asmodian]

Are you sure you are using AVX-512 and not bottlenecked by something else? I could notice even 2x offset (200 MHz), not hugely significant but a reproducible decrease in performance. Temperatures weren't much, if any, higher for me, running AVX2 or AVX512.

If you use something like hwinfo do you notice any cores running at the offset multiplier?

[Ma]

Are you using ffmpeg?

In case that the bug is in ffmpeg instead of x265, you can test ffmpeg binaries compiled by VS2015 -- ffmpeg-20180424.7z (http://msystem.waw.pl/x265/ffmpeg-20180424.7z) (without any libs, only for decoding video). I've added also ffmpeg compiled by GCC 7 and GCC 8 with only '-O2' optimize option instead of default '-O3 -fno-tree-vectorize'.

In my Win10/i7 8700 all 3 ffmpeg binaries works (and current Zeranoe builds also works).

[Loomes]

In case that the bug is in ffmpeg instead of x265, you can test ffmpeg binaries compiled by VS2015 -- ffmpeg-20180424.7z (http://msystem.waw.pl/x265/ffmpeg-20180424.7z) (without any libs, only for decoding video). I've added also ffmpeg compiled by GCC 7 and GCC 8 with only '-O2' optimize option instead of default '-O3 -fno-tree-vectorize'
Thanks a lot, I will do some more tests at the weekend. I have a feeling that it's not x265 or ffmpeg but that there's something wrong with my system. At least my Win10 install is rather fresh, almost vanilla and from an ISO directly downloaded from the Microsoft site. Well, I'll find out!

[sdml]

Am I correct that all *-refine-* features useful only for multi-pass scenarios?

[microchip8]

Am I correct that all *-refine-* features useful only for multi-pass scenarios?

rd-refine works in crf too

[Boulder]

I'm a bit confused as Bhavnahari stated that "refine-inter and refine-intra features may be used with any rate control techniques including ABR, CRF and CQP". Why would you do a 2-step encode in CRF mode to utilize the refining? Shouldn't a normal CRF encode already bring all the quality you can get from your settings with the least amount of time spent on encoding or analysing?

[Asmodian]

I believe the point is to do much of the analysis on a lower resolution video, then reuse that analysis with the higher resolution while still doing some refinement to fine tune the analysis for the full resolution. This gives a speed boost while also not lowering the quality too much, like simply using the analysis done on the low resolution video directly would.

I am not sure I would do this, but then I tend to run far up the quality/speed curve, e.g. preset veryslow with a few extra options that slow it down even more. It would be interesting to see the speed and quality impacts from using this method compared to slower or faster presets and/or other options.

[WhatZit]

I believe the point is to do much of the analysis on a lower resolution video, then reuse that analysis with the higher resolution while still doing some refinement to fine tune the analysis for the full resolution.

Basically correct. The idea is to "Analyse once, encode many".

Many of the recent CLI functions added to x265 are there to suit the demands of professional broadcast/OTT content providers, who may wish to create multiple resolution and bitrate copies of any one source.

The practical consumer application of this particular "reuse" functionality would be very niche.

[jlpsvk]

so no AVX512 offset?

i got really weird results while using avx512 and x265:
A) temps don't really go much higher while using avx512, at least on my system
B) AVX512 offset almost doesn't make a difference. i had same performance on 12x 100MHz difference in offset

probably Intel is OC-ing itself as it has good thermal conditions. :) temps with AVX-512 are bit lower in fact on my system. :)

[jlpsvk]

x265 crashing on "2nd pass"... (--profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star --max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-load c:\stats.dat --analysis-reuse-level 10 --scale-factor 2 --refine-inter 3 --refine-intra 4 --asm avx512)

x265 [warning]: specified frame type (5) at 2091 is not compatible with keyframe interval

any suggestion? :(

[Selur]

wild guess: still got free space on c: ? (iirc analysis files etc can get really large)

[LigH]

And: Write access rights in the root of C:? Better use a secondary drive for video processing.

[jlpsvk]

wild guess: still got free space on c: ? (iirc analysis files etc can get really large)

yeah... 600GB free on C: SSD. :) It's happening in the second phase, READ, when the stats are reading... :) First, 1080p encode, with writing stats is OK. :(

Stats file 27GB big. :)

[LigH]

Well, it's not completely impossible that encoding in AVX-512 mode might create a different output. But probably hard to confirm...

[jlpsvk]

Well, it's not completely impossible that encoding in AVX-512 mode might create a different output. But probably hard to confirm...

first phase was created using AVX-512 too... tried READ phase without it, result is the same. Probably will try to use first without also. :(

[divxmaster]

As a follow up to my post a few months ago re spectre/meltdown patching and its influence on x265 encoding fps,
(where I found on my skylake quad it made virtually no difference),
I have now test my haswell system (4770k) with the microsoft microcode updates release two days ago (KB4091663 for 1703 I am running).

The latest KB patches the microcode for spectre variant two (branch target injection), at boot time only. GRC's Inspectre shows my system is now fully patched.
This latest KB DOES affect throughput fps. I have seen a drop of about 2.5%. Not a big deal luckily. I will just adjust my conservative OC
from 4.1ghz to 4.2ghz. (which is +2.5%).

Cheers,
Divxmaster

[jlpsvk]

first phase was created using AVX-512 too... tried READ phase without it, result is the same. Probably will try to use first without also. :(

So... stats write encode (I am not writing here mastering display, options), spline36 downsized from 3840x1600 to 1920x800:
--crf 17 --keyint 240 --min-keyint 24 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star --max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-save c:\stats.dat --analysis-reuse-level 10 --scale-factor 2 --no-cutree

Stats read encode at full 3840x1600:
--crf 17 --keyint 240 --min-keyint 24 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star --max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-load c:\stats.dat --analysis-reuse-level 10 --scale-factor 2 --refine-inter 3 --refine-intra 4 --no-cutree

With or withou AVX-512, result is the same. :(

[LigH]

x265 2.7+346-69aafa6d70ad (https://www.mediafire.com/file/lh0hdw3w5y3tx8f/x265_2.7%2B346-69aafa6d70ad.7z) (stable merge)

AVX-512 cleanups and a few nits (getting a RC soon?)

[pradeeprama]

As a follow up to my post a few months ago re spectre/meltdown patching and its influence on x265 encoding fps,
(where I found on my skylake quad it made virtually no difference),
I have now test my haswell system (4770k) with the microsoft microcode updates release two days ago (KB4091663 for 1703 I am running).

The latest KB patches the microcode for spectre variant two (branch target injection), at boot time only. GRC's Inspectre shows my system is now fully patched.
This latest KB DOES affect throughput fps. I have seen a drop of about 2.5%. Not a big deal luckily. I will just adjust my conservative OC
from 4.1ghz to 4.2ghz. (which is +2.5%).

Cheers,
Divxmaster

Thanks for that. I typically classify any performance difference under 5% (in either direction) as noise when measured on a real system... so I suppose it is fair to say that Specrte and Meltdown don't affect us. This is not surprising as we barely make syscalls that require escalated privileges, which is where these bugs kick-in and safe guard!

[pradeeprama]

So... stats write encode (I am not writing here mastering display, options), spline36 downsized from 3840x1600 to 1920x800:
--crf 17 --keyint 240 --min-keyint 24 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star --max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-save c:\stats.dat --analysis-reuse-level 10 --scale-factor 2 --no-cutree

Stats read encode at full 3840x1600:
--crf 17 --keyint 240 --min-keyint 24 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star --max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-load c:\stats.dat --analysis-reuse-level 10 --scale-factor 2 --refine-inter 3 --refine-intra 4 --no-cutree

With or withou AVX-512, result is the same. :(

What CPU are you using?

[jlpsvk]

What CPU are you using?

Core i9-7940X

[Barough]

x265 v2.7+348-0968a46d6ba4 (http://www.mediafire.com/file/bw8wawdyapz7uit/) (GCC 7.3.0, 32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/all

[LigH]

x265 2.7+348-0968a46d6ba4 (https://www.mediafire.com/file/x3pkf15m8h5zl1f/x265_2.7%2B348-0968a46d6ba4.7z)

fixes a build error on Mac and a calculation bug in scalefactor 0

[Selur]

Does anyone see where my mistake is?

I'm trying to convert a 8bit pc range clip to 10bit x265 using:
ffmpeg -y -threads 8 -i "C:\Users\Selur\Desktop\Color Range Test.MOV" -map 0:0 -an -sn -vsync 0 -strict -1 -pix_fmt yuv420p10le -f yuv4mpegpipe - | x265 --input - --output-depth 10 --y4m --profile main10 --no-open-gop --crf 18.00 --range full --colormatrix bt470bg --output "H:\Temp\x265.265"
Problem is the output histogram is compressed, see: https://s14.postimg.cc/pq865qnfj/compare.png
Uploaded the source to my google drive as Color Range Test.MOV (https://drive.google.com/drive/folders/0B_WxUS1XGCPASUZibG5XZkRfeTg?usp=sharing).

for the comparision I used:
# Imports
import vapoursynth as vs
core = vs.get_core()
# Loading Plugins
core.std.LoadPlugin(path="G:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")


# Loading H:\Output\Color Range Test.MOV using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="H:/Output/Color Range Test.MOV")
# making sure input color matrix is set as 470bg
clip = core.resize.Point(clip, matrix_in_s="470bg")
# making sure frame rate is set to 30000/1001
clip = core.std.AssumeFPS(clip, fpsnum=30000, fpsden=1001)
# Making sure input color range is set to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
clip = core.text.Text(clip=clip, text="Original")
clip = core.hist.Levels(clip=clip)

clip2 = core.lsmas.LWLibavSource(source="H:/Temp/x265.265", format="YUV420P10", cache=0)
# making sure input color matrix is set as 470bg
clip2 = core.resize.Point(clip2, matrix_in_s="470bg")
# making sure frame rate is set to 30000/1001
clip2 = core.std.AssumeFPS(clip2, fpsnum=30000, fpsden=1001)
# Making sure input color range is set to PC (full) range.
clip2 = core.std.SetFrameProp(clip=clip2, prop="_ColorRange", intval=0)
clip2 = core.text.Text(clip=clip2, text="x265")
# adjusting output color from: YUV420P8 to YUV420P10
clip2 = core.resize.Bicubic(clip=clip2, format=vs.YUV420P8)
clip2 = core.hist.Levels(clip=clip2)


clip = core.std.StackHorizontal([clip, clip2])
# Output
clip.set_output()

My guess is that I need to change the ffmpeg call, but I don't see where.

Cu Selur

Ps.: Adding '-color_range 2' to the ffmpeg call doesn't help either.

[sneaker_ger]

As you say this has nothing to do with x265.

You could try to "trick" ffmpeg by pretending the input was limited range.
-vf zscale=rangein=limited:range=limited -pix_fmt yuv420p10le

Maybe someone with more ffmpeg knowledge knows a "proper" solution ...

[Selur]

As you say this has nothing to do with x265.
That is what I guessed, not sure whether ffmpeg does something wrong or x265 has some problem with y4m output or ffmpeg.

You could try to "trick" ffmpeg by pretending the input was limited range.
-vf zscale=rangein=limited:range=limited -pix_fmt yuv420p10le
That still changes the luma histogram. (https://s31.postimg.cc/tnpvy7cq3/compare2.png)
using:
-vf zscale=rangein=full:range=full
is better, but there are still some changes. (https://s31.postimg.cc/9hmdzaehn/compare3.png)

Maybe someone with more ffmpeg knowledge knows a "proper" solution ...
I hope so. :)

Cu Selur

[poisondeathray]

maybe there is a bug with full range ?

waveform/histograms only match if you use lossless (even compared to low crf like --crf 8, where you 'd only expect tiny differences) , regardless if you use vapoursynth input or ffmpeg scaling with rawvideo pipe

crf8 - waveform is compressed

ffmpeg -i "Color Range Test.MOV" -vf scale=iw:ih:in_range=pc:out_range=pc,format=yuv420p10le -f rawvideo - | x265_10bit.exe --fps 30000/1001 --input-res 1280x720 --input-depth 10 --output-depth 10 --input-csp i420 --crf 8.00 --range full --colormatrix smpte170m -o rawpipe_x265crf8.265 -


--lossless works with ffmpeg rawvideo pipe

ffmpeg -i "Color Range Test.MOV" -vf scale=iw:ih:in_range=pc:out_range=pc,format=yuv420p10le -f rawvideo - | x265_10bit.exe --fps 30000/1001 --input-res 1280x720 --input-depth 10 --output-depth 10 --input-csp i420 --lossless --range full --colormatrix smpte170m -o rawpipe_x265lossless.265 -



vspipe y4m --crf 8 waveform is compressed

vspipe --y4m 1.vpy - | x265_10bit.exe --y4m --crf 8.00 --colormatrix smpte170m --range full -o vs_x265.265 -



vspipe y4m --lossless waveform is ok

vspipe --y4m 1.vpy - | x265_10bit.exe --y4m --lossless --colormatrix smpte170m --range full -o vs_x265_lossless.265 -


vpy input

clip = core.lsmas.LibavSMASHSource(r'F:\Color Range Test.MOV')
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10)
clip.set_output()

[Selur]

waveform/histograms only match if you use lossless (even compared to low crf like --crf 8, where you 'd only expect tiny differences) , regardless if you use vapoursynth input or ffmpeg scaling with rawvideo pipe
I understand some small changes like when using '-vf zscale=rangein=full:range=full' and my lines from above might be a side effect of the video compression, but the compression of the histogram really looks like either a bug in x265 or all the other tools.

Cu Selur

[Ma]

I'm trying to convert a 8bit pc range clip to 10bit x265 using:
ffmpeg [...] -f yuv4mpegpipe - | x265 [...]

yuv4mpegpipe is for limited range only. Please use raw video for full range.

[poisondeathray]

I understand some small changes like when using '-vf zscale=rangein=full:range=full' and my lines from above might be a side effect of the video compression, but the compression of the histogram really looks like either a bug in x265 or all the other tools.



I agree . That's why I used crf 8, you'd expect only tiny tiny difference, not that large. It looks like CbCr are affected too

It affects x264 too . vs rawpipe too instead of y4m for fun

vspipe 1.vpy - | x264_10bit --demuxer raw --input-csp i420 --input-depth 10 --input-res 1280x720 --fps 30000/1001 --crf 8.00 --colormatrix smpte170m --input-range pc --range pc -o vs_raw_x264.264 -


But if it was a toolchain problem, you'd expect lossless to exhibit the same problem

Something is up...

[jlpsvk]

ok. x265 devs. what is the command line to output uhd bd compliant stream?

[Jamaika]

yuv4mpegpipe is for limited range only. Please use raw video for full range.
Never heard about it. A few threads further users advise against the use of rawvideo.

I test conveters:
ffmpeg_080518.exe -i "Color Range Test.MOV" -f yuv4mpegpipe -vf scale=1280:720:in_color_matrix=470bg:in_range=full:out_color_matrix=470bg:out_range=full,format=yuv420p10,lutyuv=val:val:val -strict -1 - |
x264-10bit.exe --demuxer y4m --muxer mp4 --input-csp i420 --input-depth 10 --input-range pc --output-csp i420 --threads 4 --preset veryslow --tune grain --crf 28 --fps 29.970 --keyint 60 --nal-hrd none
--colormatrix bt470bg --colorprim bt709 --transfer bt709 --range pc
--output "x264_420p10le_crf28_1.mp4" -
x264-10bit_2851.exe --muxer mp4 --input-csp i420 --input-depth 8 --input-range pc --output-csp i420 --threads 4 --preset veryslow --tune grain --crf 28 --fps 29.970 --keyint 60 --nal-hrd none
--colormatrix bt470bg --colorprim bt709 --transfer bt709 --range pc
--output "x264_420p10le_crf28_2.mp4" "Color Range Test.MOV"
bpgenc_0.9.8.exe -v -b 10 -f 420 -c ycbcr_bt709 -e x265 -m 9 -q 28 -o x265_420p10le_crf28.bpg rgba.png

Result of the test:
1)bad range,
2)bad range, no latest version 10bit 2901 (2018),
3)good.

Conclusion:
For me, the problem is in BT470bg.

[Ma]

After some tests it is a bug in ffmpeg (in my opinion).
Sorry for confusions.

[Jamaika]

If you use -pix_fmt yuv420p or yuv420p10 in ffmpeg it means limited range. Please use yuvj420 or yuv[B]
I know that I use x264, but...
[yuv4mpegpipe @ 000001d226440300] ERROR: yuv4mpeg can only handle yuv444p, yuv422p, yuv420p, yuv411p and gray8 pixel formats. And using 'strict -1' also yuv444p9, yuv422p9, yuv420p9, yuv444p10, yuv422p10, yuv420p10, yuv444p12, yuv422p12, yuv420p12, yuv444p14, yuv422p14, yuv420p14, yuv444p16, yuv422p16, yuv420p16, gray9, gray10, gray12 and gray16 pixel formats. Use -pix_fmt to select one.
Could not write header for output file #0 (incorrect codec parameters ?): I/O error
Error initializing output stream 0:0 --
or
[Parsed_format_1 @ 000001587211bec0] Invalid pixel format 'yuvj420p10'
[AVFilterGraph @ 00000158714fd980] Error initializing filter 'format' with args 'yuvj420p10'
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0

[Selur]

'yuvj420p10'
try:
-vf zscale=rangein=full:range=full -strict -1 -pix_fmt yuv420p10le
this seems to work for me when using ffmpeg als decoder, but that wouldn't help when using vspipe which poisondeathray tested,..

[poisondeathray]

'yuvj420p10'
try:
-vf zscale=rangein=full:range=full -strict -1 -pix_fmt yuv420p10le
this seems to work for me when using ffmpeg als decoder, but that wouldn't help when using vspipe which poisondeathray tested,..

But you still get problems unless you use lossless, right ?

You can look at histogram classic and there is a big line artifact in the waveform; you wouldn't expect that at low crf values (maybe at high lossy quantizers)

So it's not a pipe or toolchain problem

[Selur]

But you still get problems unless you use lossless, right ?
Not really. (used crf 18 and 8bit)
https://s9.postimg.cc/xjjpx9fl7/compare_x265.png (https://postimg.cc/image/xjjpx9fl7/)
Uploaded my compare script and the files I use to my GoogleDrive (https://drive.google.com/drive/folders/1M4E3CSWSi2cTV66csIPvCKmlYXgYUVkh?usp=sharing)
Argh,.. I used crf 18 and 8bit.
-> will try again with 10bit
You are right there are still 'line artifacts' in the histogram when using 10bit.
https://s9.postimg.cc/mmie8i0jf/compare_x265_10bit_-_0.png (https://postimg.cc/image/mmie8i0jf/)

Cu Selur
Ps.: As a side note: either I'm making something wrong or x264 is totally broken when handling with pc scale video. (see: https://forum.doom9.org/showthread.php?p=1841998)

[poisondeathray]

In case it's a vpy or algorithm issue (you're using zimg/z.lib in vpy to convert back to 8bit to check at 8bit waveform) , I double checked with the vpy only also with nearest neighbor /Point instead of Bicubic - you don't get those artifacts (when going back/forth in the vpy only) ; you only get those artifacts with x264/x265 lossy encoding (and CRF 8 should be near lossless) . Lossless encoding fixes it, so it's definitely not a pipe or "toolchain" issue .

[Selur]

Lossless encoding fixes it, so it's definitely not a pipe or "toolchain" issue .
Not necessarily, lossless should be YUV4:4:4 and not YUV4:2:0.
Atm. my best guess is that these line artifacts are due to the compression, but it might be a but.

[poisondeathray]

Not necessarily, lossless should be YUV4:4:4 and not YUV4:2:0.

No; "lossless" is a relative term with respect to whatever your input is

If you have a YUV4:2:0 source, upscaling to YUV4:4:4 is not lossless (unless you use nearest neighbor to up and downscale back to YUV4:2:0)

[poisondeathray]

Atm. my best guess is that these line artifacts are due to the compression, but it might be a but.


That's what it looks like (because lossless encoding fixes it), but it's very bizarre pattern (look at the classic histogram) . Many errors congregate around a low midtone value, creating a line artifact .

[Selur]

okay, looking at the classic histogram view, I see what you mean, and I guess those might be due to compression artifacts.

[poisondeathray]

You would expect random quantization artifacts to be more random, not coalesce around 1 value.

And you would expect it to be "less lossy" or higher quality, certainly at CRF 8 .

It might have just been a "perfect storm" conditions . But do you have another clip/scene/section from that camera to test ?

[Selur]

But do you have another clip/scene/section from that camera to test ?
Sadly, no. Got it from a Hybrid user who stumbled over the problem. :)
(wrote an email to the user whether he could share some additional footage from that source)

[foxyshadis]

Never heard about it. A few threads further users advise against the use of rawvideo.

I test conveters:
ffmpeg_080518.exe -i "Color Range Test.MOV" -f yuv4mpegpipe -vf scale=1280:720:in_color_matrix=470bg:in_range=full:out_color_matrix=470bg:out_range=full,format=yuv420p10,lutyuv=val:val:val -strict -1 - |
x264-10bit.exe --demuxer y4m --muxer mp4 --input-csp i420 --input-depth 10 --input-range pc --output-csp i420 --threads 4 --preset veryslow --tune grain --crf 28 --fps 29.970 --keyint 60 --nal-hrd none
--colormatrix bt470bg --colorprim bt709 --transfer bt709 --range pc
--output "x264_420p10le_crf28_1.mp4" -
x264-10bit_2851.exe --muxer mp4 --input-csp i420 --input-depth 8 --input-range pc --output-csp i420 --threads 4 --preset veryslow --tune grain --crf 28 --fps 29.970 --keyint 60 --nal-hrd none
--colormatrix bt470bg --colorprim bt709 --transfer bt709 --range pc
--output "x264_420p10le_crf28_2.mp4" "Color Range Test.MOV"
bpgenc_0.9.8.exe -v -b 10 -f 420 -c ycbcr_bt709 -e x265 -m 9 -q 28 -o x265_420p10le_crf28.bpg rgba.png

Result of the test:
1)bad range,
2)bad range, no latest version 10bit 2901 (2018),
3)good.

Conclusion:
For me, the problem is in BT470bg.

Rematrixing YUV in swscale goes through an RGB intermediate; it's not supposed to rematrix when you pass the same value in and out, but that's my best guess as to why the histogram gains drastic peaks.

[Ma]

I want to precise -- first command line gives o1.raw with full range inside, second gives o2.raw with limited range inside:
ffmpeg -i "../Color Range Test.MOV" -vf scale=out_range=pc,format=yuv420p -f rawvideo o1.raw
ffmpeg -i "../Color Range Test.MOV" -color_range pc -pix_fmt yuv420p -f rawvideo o2.raw

Is it ffmpeg bug or it should work that way?

[poisondeathray]

I want to precise -- first command line gives o1.raw with full range inside, second gives o2.raw with limited range inside:
ffmpeg -i "../Color Range Test.MOV" -vf scale=out_range=pc,format=yuv420p -f rawvideo o1.raw
ffmpeg -i "../Color Range Test.MOV" -color_range pc -pix_fmt yuv420p -f rawvideo o2.raw

Is it ffmpeg bug or it should work that way?

It's expected

The source is yuvj420p . The "j" means full range, and it's read correctly by ffmpeg

The order matters in ffmpeg. When you use -pix_fmt yuv420p, from a yuvj420p source - that tells ffmpeg to scale full range to limited range

If you were passing 8bit in/out , there is no need for any of it -f rawvideo is enough. It's just decoding to raw video. (But the receiving application might need you to enter parameters if piping raw)

In the earlier example, it was scaling to 10bit. That's why the -vf scale was used with format. (ffmpeg does not have a yuvj420p10le or be equivalent pixel format; the "j" is only for 8bit)

[poisondeathray]

I'm going to call this issue that Selur brought up, the 8bit=>10bit , 10bit encoding a bug. Both x264 , x265 are affected. You can see similar problem on other full range DSLR clips.

This is a test video from an old 7D review
http://www.imaging-resource.com/PRODS/E7D/E7DVIDEO.HTM

Direct Link

https://www.imaging-resource.com/PRODS/E7D/FULLRES/YE7VID_1080_24_P_MVI_1908.MOV


Lossless encoding doesn't exhibit the problem, nor does the scaling to 10 back to 8 to compare in vapoursynth only - so it's not a "tool chain" issue. But as low as crf 8 still clearly shows the coalescing values.

The likelihood that you'd get something like that is extremely low for several different videos (I tested 3 so far). You'd expect some quantization errors to round up, some down, but roughly be close to the original....especially at lower quantizers or crf values... Not redistribute to aggregate around 1 value, causing a line in the waveform. In this example, it was a higher midtone

The loss logic seems the same in x264 and x265, they both exhibit the same behaviour

lossless
https://s31.postimg.cc/av74s7963/lossless.png

x265_crf8
https://s31.postimg.cc/58aru7jcr/x265_crf8.png

[foxyshadis]

I'm going to call this issue that Selur brought up, the 8bit=>10bit , 10bit encoding a bug. Both x264 , x265 are affected. You can see similar problem on other full range DSLR clips.

Another possibility: It could be an artifact of the decoder, or even of the spec. I'd check against Mainconcept, or one of the GPU encoders & decoders. x264/x265 and ffmpeg/libav are so ubiquitous that it's hard to test anything else.

[poisondeathray]

Another possibility: It could be an artifact of the decoder, or even of the spec. I'd check against Mainconcept, or one of the GPU encoders & decoders. x264/x265 and ffmpeg/libav are so ubiquitous that it's hard to test anything else.


The testing routine used by selur converted back to 8bit to compare. A native 10bit waveform does not demonstrate the problem. But it's still a bizarre aggregation, and why this occurs at CRF 8 but not lossless.

I'll swap out and double check the decoders and encoders and do some more tests when I have more time

[Barough]

x265 v2.7+351-d3b714bd5471 (http://www.mediafire.com/file/pp6c8shkc4p2he8/) (GCC 7.3.0, 32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/all

[Ashok Kumar Mishra]

Version 2.8
===========

Release date - 21/05/2018

New features
-------------
1. :option:`--asm avx512` used to enable AVX-512 in x265. Default disabled.
For 4K main10 high-quality encoding, we are seeing good gains; for other resolutions and presets, we don't recommend using this setting for now.

2. :option:`--dynamic-refine` dynamically switches between different inter refine levels. Default disabled.
It is recommended to use :option:`--refine-intra 4' with dynamic refinement for a better trade-off between encode efficiency and performance than using static refinement.

3. :option:`--single-sei`
Encode SEI messages in a single NAL unit instead of multiple NAL units. Default disabled.

4. :option:`--max-ausize-factor` controls the maximum AU size defined in HEVC specification.
It represents the percentage of maximum AU size used. Default is 1.

5. VMAF (Video Multi-Method Assessment Fusion)
Added VMAF support for objective quality measurement of a video sequence.
Enable cmake option ENABLE_LIBVMAF to report per frame and aggregate VMAF score. The frame level VMAF score does not include temporal scores.
This is supported only on linux for now.

Encoder enhancements
--------------------
1. Introduced refine-intra level 4 to improve quality.
2. Support for HLG-graded content and pic_struct in SEI message.

Bug Fixes
---------
1. Fix 32 bit build error (using CMAKE GUI) in Linux.
2. Fix 32 bit build error for asm primitives.
3. Fix build error on mac OS.
4. Fix VBV Lookahead in analysis load to achieve target bitrate.

[LigH]

x265 2.8+1-478bfe2b7673 (https://www.mediafire.com/file/9ni1v095jkvvhwu/x265_2.8+1-478bfe2b7673.7z) (stable release, GCC 7.3.0, Win32+Win64)

last few fixes in analysis mode

[user1085]

x265 2.8+1-478bfe2b7673 (https://www.mediafire.com/file/9ni1v095jkvvhwu/x265_2.8+1-478bfe2b7673.7z) (stable release, GCC 7.3.0, Win32+Win64)

last few fixes in analysis mode

Do you also have a Linux build?

[Sagittaire]

Version 2.8
5. VMAF (Video Multi-Method Assessment Fusion)
Added VMAF support for objective quality measurement of a video sequence.
Enable cmake option ENABLE_LIBVMAF to report per frame and aggregate VMAF score. The frame level VMAF score does not include temporal scores.
This is supported only on linux for now.


VMAF is objective metric from Netflix:
https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652

Really good idea to impliment this metric. VMAF seem have really good correlation with HVS. Why not include PSNR-HVS-M (seem have really good correlation too)?

Possible to introduce tune profil for VMAF or PSNR-HVS-M like for PSNR and SSIM?

[jd17]

Possible to introduce tune profil for VMAF or PSNR-HVS-M like for PSNR and SSIM?

I would like that too! :)

[Asmodian]

Any tune profile for VMAF should probably be the default, given that it correlates very well with the human visual system. ;)

[user1085]

Anyone know how to build x265 2.8 with AVX512 enabled on Ubuntu?

I built x265 from 2.8 release tarball but x265 -V doesn't show me it's using AVX512 capability on a Intel Xeon

x265 [info]: HEVC encoder version 2.8
x265 [info]: build info [Linux][GCC 7.3.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

Here's the CPU info

uso@xvmub:~/x265_2.8/build/linux$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 2
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
Stepping: 4
CPU MHz: 2693.677
BogoMIPS: 5387.35
Virtualization: VT-x
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 33792K
NUMA node0 CPU(s): 0,1
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti tpr_shadow vnmi ept vpid fsgsbase bmi1 hle avx2 smep bmi2 erms rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves

[LigH]

AVX-512 is only enabled if you explicitly add --cpu avx512 to the command line; without it is not even checked for in CPU capabilities, apparently... so try:

x265 --cpu avx512 -V

Is it reported then?
__

Do you also have a Linux build?

No, I was a) quite confident that Linux users know how to install and use GNU compilers (it's good enough documented on Multicoreware's Bitbucket), and b) not sure how portable one build under one specific distro is. Windows builds are quite portable, and knowledge about compiling from sources is probably not so common among Windows users.

[user1085]

AVX-512 is only enabled if you explicitly add --cpu avx512 to the command line; without it is not even checked for in CPU capabilities, apparently... so try:

x265 --cpu avx512 -V

Is it reported then?
__



No, I was a) quite confident that Linux users know how to install and use GNU compilers (it's good enough documented on Multicoreware's Bitbucket), and b) not sure how portable one build under one specific distro is. Windows builds are quite portable, and knowledge about compiling from sources is probably not so common among Windows users.

No worries I figured out how to compile, didn't see the instructions earlier :)

Re:avx512, this worked for me
./x265 --asm avx512 -V

[LigH]

Oops, should have read the manual, "asm" instead of "cpu".

Thank you for confirming anyway.

[benwaggoner]

VMAF is objective metric from Netflix:
https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652

Really good idea to impliment this metric. VMAF seem have really good correlation with HVS. Why not include PSNR-HVS-M (seem have really good correlation too)?

Possible to introduce tune profil for VMAF or PSNR-HVS-M like for PSNR and SSIM?
VMAF is our least-bad metrics, but it definitely has limitations and blind spots. It doesn't catch banding well, particularly in lower luma. It is only trained on x264 8-bit SDR, and without a lot of different psychovisual tuning, so it doesn't do a great job of discriminating between some kinds of psychovisual tradeoffs. And it's never seen the newer kind of artifacts in HEVC, so it doesn't know about customer preference between them. It doesn't know about <300 Kbps or >1080p. It was only tested against a particular display at 3x screen heights. The temporal comparison base metric is pretty weak.

It's the best thing we've got, but it definitely isn't perfect.

Also, the question of how to go from scores for individual frame to a per-clip score remains unanswered. A file that varies from 30-80 VMAF is going to be perceived as a lot worse than one that varies from 50-60, but the mean of the frames won't show any difference.

That's not a VMAF specific problem; it applies to all metrics where we are trying to figure out quality over time from individual frame metrics.

[jd17]

Wait a minute.... The Amazon video specialist does NOT think Netflix's metric is the holy grail?
I'm shocked! Shocked! :eek:
Kidding of course... I just think someone had to say it. ;)

I do think Netflix abuses VMAF findings though.
Bitrates between 2000kbit/s and 3000kbit/s for 1080p x264 video are just not good. No matter what VMAF tells you...

[LigH]

The main limit is probably a bandwidth cap; restricted VBR. Some scenes will have a demand that can't be satisfied in a streaming usage. But quality tests will often be done for a local playback usage, I guess...

[jd17]

The main limit is probably a bandwidth cap; restricted VBR. Some scenes will have a demand that can't be satisfied in a streaming usage.

Is this a comment on my complaint about the 2000-3000kbit/s?
If that is the case - I do not think bandwidth cap is a factor.
I was referring to the "highest" quality Netflix streams at 1080p for some content. You cannot get more, no matter how high your bandwidth is.
This not only used to be much higher in the past, it still is for some content with more grain.
Additionally, Amazon streams significantly higher 1080p AVC bitrates across the board, so bandwidth can hardly be a limitation...

I think it's just Netflix saving bandwidth, because most people either watch this stuff on smaller screens with a viewing distance too large, or they just do not care / do not see the flaws...
Either way - Netflix abuses it.

[nevcairiel]

The bitrate you target is really not related to the metric you use. You can use a metric to optimize encoding settings for any target bitrate to get the best out of the bits available to you.

[jd17]

You might be misunderstanding what I tried to say...
I know how metrics work.

I merely tried to point out that I think Netflix is abusing their internal VMAF findings to justify streaming at much lower bitrates.
I would argue they lowered their bitrate target (at least partly) as a result of their use of VMAF.

No matter how clean your video is, 1080p x264 will not be good at 2000-3000kbit/s, especially in the backgrounds.

Maybe you understand where I am coming from when you look at this blog:
https://medium.com/netflix-techblog/per-title-encode-optimization-7e99442b62a2

In theory, the arguments in that blog are well reasoned.
Use low bitrates for cartoons and clean video, use high bitrates for lots of movement, grain, and so on...

However, especially within the last year or so (look at recent releases of Netflix series for instance), they have moved to the aforementioned low bitrates as highest 1080p target bitrate. On live action video!

While 2000-3000kbit/s is indeed completely fine for Cartoon, it is not for live action video.

Is that VMAF's fault? Of course not.
But I think Netflix saw that the VMAF scores are not too bad for 1080p video at those bitrates, so they decided to lower their targets accordingly.


Sorry for the long off-topic discussion...
Originally, I just wanted to make a little joke and complain about Netflix's quality in the process...
I did not mean for it to drift so far away from x265! :)

[foxyshadis]

Sorry guys, this was getting way too offtrack. AVX512 vs Threadripper has been moved to the PC Hardware forum. (https://forum.doom9.org/showthread.php?t=175476)

[IgorC]

Also, the question of how to go from scores for individual frame to a per-clip score remains unanswered. A file that varies from 30-80 VMAF is going to be perceived as a lot worse than one that varies from 50-60, but the mean of the frames won't show any difference.


It was solved in HVMAF

https://medium.com/netflix-techblog/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f

About Harmonic mean https://en.wikipedia.org/wiki/Harmonic_mean
Since the harmonic mean of a list of numbers tends strongly toward the least elements of the list, it tends (compared to the arithmetic mean) to mitigate the impact of large outliers and aggravate the impact of small ones.

P.S. It's not news at all. Geometric and harmonic means were in use since forever.

[K.i.N.G]

Hi, I have a possible request/idea.

Wouldn't it be possible to integrate a bitrate multiplier by mask (a greyscale image or video) as source? Preferably with an option to set strength/influence (so pure black doesn't always give 0% :D )
I think this would be quite useful. Not sure how difficult it would be to add.

It's just an idea, I don't want to come over as someone who 'demands' something from open source devs.

[LigH]

x265 2.8+10-a7bd0622ece5 (https://www.mediafire.com/file/3mnslai3miffxl9/x265_2.8%2B10-a7bd0622ece5.7z) (merge with stable)

several bug fixes (build errors, dyn.ref., scaled enc.)

[benwaggoner]

It was solved in HVMAF

https://medium.com/netflix-techblog/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f

About Harmonic mean https://en.wikipedia.org/wiki/Harmonic_mean
Looks like a promising improvement, but this stuff never gets SOLVED.

Averaged over a two hour movie, a harmonic mean can still not catch periods of terrible quality.

[Midzuki]

5. VMAF (Video Multi-Method Assessment Fusion)
Added VMAF support for objective quality measurement of a video sequence.

Enable cmake option ENABLE_LIBVMAF to report per frame and aggregate VMAF score. The frame level VMAF score does not include temporal scores.

This is supported only on linux for now.

Because... ???

[LigH]

I already wondered some time before you ... maybe due to the way how libvmaf is available for and can be installed in the OS; maybe due to the way how the library can be linked.

But since I read that placing VMAF into the encoder may not be the best solution for comparable results with other encoders, I will possibly not include it in my MinGW builds, even if it would ever work at all.

[Barough]

x265 v2.8+13-617590cb8b98 (http://www.mediafire.com/file/hudywji1f680o8o/) (GCC 7.3.0, 32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[LigH]

x265 2.8+19-bcdc610cf5f0 (https://www.mediafire.com/file/7pfbi9ugf0m40ut/x265_2.8%2B19-bcdc610cf5f0.7z)

Support scale factor with analysis-reuse-level 1-6; fix build issue when using icc; add support for chunked encoding*; use the data structure of analysis-save/load for multi-pass-opt-analysis/multi-pass-opt-distortion; add vbv-end tolerance check

--chunk-start <integer> First frame of the chunk. Default 0 (disabled)
--chunk-end <integer> Last frame of the chunk. Default 0 (disabled)

* this seems to allow the output of parts of a GOP, with bitrate control beyond this GOP where required, possibly to support "smart rendering"

[Selur]

@developers: Any insight into chunked encoding and what it is meant to be for would be nice.
@LigH: sounds to me more like something meant for distributed encoding rather than 'smart rendering'.

[LigH]

--chunk-start (https://x265.readthedocs.io/en/default/cli.html?highlight=chunk#cmdoption-chunk-start) <integer>

First frame of the chunk. Frames preceeding this in display order will be encoded, however, they will be discarded in the bitstream. This feature can be enabled only in closed GOP structures. Default 0 (disabled).

--chunk-end (https://x265.readthedocs.io/en/default/cli.html?highlight=chunk#cmdoption-chunk-end) <integer>

Last frame of the chunk. Frames following this in display order will be used in taking lookahead decisions, but, they will not be encoded. This feature can be enabled only in closed GOP structures. Default 0 (disabled).

Distributed encoding can possibly be limited to GOP ranges and VBV level control. Okay, frame exact splitting may be a bonus.

[benwaggoner]

@developers: Any insight into chunked encoding and what it is meant to be for would be nice.
@LigH: sounds to me more like something meant for distributed encoding rather than 'smart rendering'.
That said, the underlying technology can be used in all sorts of way.

Selective reencoding is another obvious application.

[benwaggoner]

Distributed encoding can possibly be limited to GOP ranges and VBV level control. Okay, frame exact splitting may be a bonus.
About the bare minimum useful chunk duration is at least one GOP and practically one (vbv-maxrate+vbv_bufsize)/vbv-maxrate. The second requirement can be loosened if the system will selectively reencode chunks when a sequence of chunks would violate VBV. Short chunks without the ability to reencode means that VERY conservative VBV must be used.

Consider two chunks that are relatively easy to encode except for a big complexity spike at the split point between chunks. Using normal rate control, each chunk would get most of their bits at the split point, since neither chunk knew the other was going to stress the VBV, and so the combination causes a VBV violation.

Cutting vbv-maxrate and vbv-bufsize to half of the "real" maximum values you want can fix the problem, but has obvious quality impact for many use cases.

[Stereodude]

Is there any way to x265 to let Windows know that it's busy encoding to prevent a Windows update from rebooting the PC during an encode? I thought I read before that there's a way for an application to let the OS know it's "busy" so that it won't reboot while the application is crunching away.

Currently Windows 10 will happily reboot mid x264 or x265 encode even if the CPU is pegged at 100%.

[LigH]

This may possibly be solved easier in user interfaces; but I will forward this to the mailing list for you to lead the developers' attention here.

P.S.: I wonder if you could use the ShutdownBlockReasonCreate (https://msdn.microsoft.com/en-us/library/windows/desktop/aa376877%28v=vs.85%29.aspx) function to postpone it.

[Barough]

x265 v2.8+21-a8a5ccf5aaf7 (http://www.mediafire.com/file/lqhocffrmbgvyve/x265-2.8+21-a8a5ccf5aaf7_Win_GCC730.7z) (GCC 7.3.0, 32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[Stereodude]

This may possibly be solved easier in user interfaces; but I will forward this to the mailing list for you to lead the developers' attention here.

P.S.: I wonder if you could use the ShutdownBlockReasonCreate (https://msdn.microsoft.com/en-us/library/windows/desktop/aa376877%28v=vs.85%29.aspx) function to postpone it.
Well, I call it directly from the command line with no GUI.

What's the x265 equivalent to --stitchable with x264 or is nothing needed to piece separate encodes together (that used the same command line)?

[sneaker_ger]

AFAIK there is no equivalent as x265 doesn't do the kind of "header optimizations" x264 did. So you should be able to stitch x265 encodes done with the same settings without problems.

[Selur]

I would have assumed that --repeat-headers (https://x265.readthedocs.io/en/latest/cli.html#cmdoption-repeat-headers) should be used for this scenario. :)

[sneaker_ger]

Repeating headers can mitigate problems. It's the same with x264 if you encode to raw .264 ES, i.e. the SPS/PPS will be at every keyframe. But SPS/PPS can still be different for every encode without --stitchable even with identical settings. But not with x265. Well, at least that was what I concluded when I tested it a long time ago...

[Barough]

x265 v2.8+23-656b5b442f0b (http://www.mediafire.com/file/2r5pq9i9lqa3xqa/) (GCC 7.3.0, 32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[Selur]

CEA 608/708 Support Parse the SEI messages from text file and insert it into the userSEI with cli option. .
Okay, so now we can add closed caption subtitle into the bitstream. Anybody got a script or small tool to convert srt to CEA 608/708 ?

[Ma]

Currently Windows 10 will happily reboot mid x264 or x265 encode even if the CPU is pegged at 100%.

My solution is to switch off Windows Update (before I want encoding):
https://www.easeus.com/todo-backup-resource/how-to-stop-windows-10-from-automatically-update.html (Solution 1. Disable Windows Update Service)

[LigH]

New build postponed, a patch renaming the new option is being proposed...

[LigH]

x265 2.8+24-289b8a3730ae (https://www.mediafire.com/file/940dk2d107fgnfe/x265_2.8%2B24-289b8a3730ae.7z)

--nalu-file <filename> Text file containing SEI messages in the following format : <POC><space><PREFIX><space><NAL UNIT TYPE>/<SEI TYPE><space><SEI Payload>

was --usersei-file a "moment" ago...

[MeteorRain]

My solution is to switch off Windows Update (before I want encoding):
https://www.easeus.com/todo-backup-resource/how-to-stop-windows-10-from-automatically-update.html (Solution 1. Disable Windows Update Service)

AFAIK Windows will try to change it back by itself.
My solution is to add both WUAU and delivery optimization service into Windows Firewall rules, and block all the outgoing traffic.

[RieGo]

adding more offtopic:
best way to prevent windows 10 updates is by disabling automatic installation in group policies (only on win 10 pro i think).
this is an easy and permanent solution.

[LigH]

But some people may not accept the risk of disabling updates and leaving Windows unpatched against new threats...

[Bhavnahari]

Greetings open source enthusiasts!

I am writing on behalf of the x265 developers. We notice that the open source community across the world has been using and experimenting with the x265 encoder and some of these works have been extremely intriguing. We would like to bring such ideas/experiments to the limelight and as an effort, we are reaching out to you to send in non-copyrighted articles that can be posted on the official x265 blog. We urge everyone who has published research papers based on x265 to share short write-ups on your analysis and your recommendations to improve x265. We believe that such contributions have the potential to become the gateway to new and better dimensions of video technology.

Your contributions will be of immense value to the open source fraternity. Looking forward to seeing the fascinating ideas that you all have. Please write to us at <pradeep@multicorewareinc.com> <bhavna@multicorewareinc.com> <vignesh@multicorewareinc.com>

[LigH]

x265 2.8+40-0106f9f2f867 (https://www.mediafire.com/file/st4y5zp5zpsq2rh/x265_2.8%2B40-0106f9f2f867.7z)

several new AVX(2) assembler routines

[registoni]

sorry if offtop but I am currious what settings amazon is using for encoding TV series in HEVC using x265 encoder.
In the MI there is rc=crf / crf=21.5. Are they using some encoding preset (medium, slow, slower, etc) or some fine tuned own setting? (probably stupid question)

[sneaker_ger]

Do they even use x265? x265 is one of many different H.265/HEVC encoders available on the market.

[LigH]

If you can find CRF in the MI, then it's quite probable that they used x265. Other encoders may not know this mode, and possibly not even store encoder options in MI.

But x265 only stores the internal API level (encoder core relevant) options. They are not equal to CLI options in every case. Selur has a tool which can calculate back which preset and tuning you could use to minimize the options. But you can't know whether a preset and tuning was indeed used. Or if a CLI encoder was used at all (maybe they have their own application using the encoder library via API).

[registoni]

Do they even use x265? x265 is one of many different H.265/HEVC encoders available on the market.

some info from MI:
General
Format : Matroska
Format version : Version 4 / Version 2
File size : 3.62 GiB
Duration : 59 min 30 s
Overall bit rate : 8 696 kb/s
Encoded date : UTC 2018-07-02 19:29:19
Writing application : mkvmerge v24.0.0 ('Beyond The Pale') 64-bit
Writing library : libebml v1.3.6 + libmatroska v1.4.9

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main@L4@Main
Codec ID : V_MPEGH/ISO/HEVC
Duration : 59 min 30 s
Bit rate : 8 054 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Bits/(Pixel*Frame) : 0.162
Stream size : 3.35 GiB (93%)
Writing library : x265 0.0:[Linux][GCC 4.8.2][64 bit] 8bit+10bit+12bit
Encoding settings : cpuid=1173503 / frame-threads=1 / wpp / no-pmode / no-pme / psnr / ssim / log-level=2 / csvfn=/apollo/env/YoshiEncodingWorkflowActivitiesLinuxEncoding/var/tmp/62271779/5e5e8977-f266-439e-9262-f5c3dcf1ac15/771fd41e-a6d5-4188-90ed-6d02fb27bab5_video_1080p_9000kbps.csv / csv-log-level=2 / input-csp=1 / input-res=1920x1080 / interlace=0 / total-frames=0 / level-idc=40 / high-tier=1 / uhd-bd=0 / ref=5 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / no-open-gop / min-keyint=61 / keyint=120 / bframes=5 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=120 / lookahead-slices=0 / scenecut=0 / no-intra-refresh / ctu=64 / min-cu-size=8 / rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=2 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=200 / no-constrained-intra / strong-intra-smoothing / max-merge=3 / limit-refs=3 / limit-modes / me=3 / subme=3 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao / no-sao-non-deblock / rd=4 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=1.00 / no-rd-refine / analysis-reuse-mode=0 / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=21.5 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=9000 / vbv-bufsize=12000 / vbv-init=0.9 / crf-max=0.0 / crf-min=0.0 / ipratio=1.40 / pbratio=1.30 / aq-mode=3 / aq-strength=2.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=2 / range=0 / colorprim=1 / transfer=1 / colormatrix=1 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=255 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / opt-qp-pps / opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / no-limit-sao / ctu-info=0
Default : Yes
Forced : No
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

[LigH]

Yes, quite certainly x265. With some additional information, possibly controlled by RipBot264 or similar tools for network distributed encoding.

[sneaker_ger]

Well, it certainly isn't one of the presets. (e.g. aq-mode 3 is in none of the presets)

https://x265.readthedocs.io/en/default/presets.html

[benwaggoner]

I mistakenly used x264 syntax in a x265 encode, and used --nr 350. But the x265 docs don't even document plain --nr as a command!

Anyone know what it does? --nr-inter 350 --nr intra 350? Just --nr-inter 350? That would match x264's behavior; it doesn't have that kind of intra-block filtering. If it's undefined, I'd recommend that --nr just alias--nr-inter.

[RainyDog]

Is --no-slow-firstpass safe to use for 2-pass encodes? I seem to recall reading a post some time ago which said that a fast/turbo 1st pass can be too unreliable for a slow 2nd pass with much higher settings to use as a basis.

Thanks.

[LigH]

x265 CLI option: slow-firstpass (https://x265.readthedocs.io/en/default/cli.html?highlight=slow-first#cmdoption-slow-firstpass)

Enable first pass encode with the exact settings specified. The quality in subsequent multi-pass encodes is better (compared to first pass) when the settings match across each pass. Default enabled.

The "turbo" options are indeed quite different from the original options in slow presets. But I don't remember any test showing verbosely how obvious the loss of quality is... the following pass(es) will still use the intended efforts, just the bitrate calculation may be based on a not really optimal reference distribution.

[benwaggoner]

x265 CLI option: slow-firstpass (https://x265.readthedocs.io/en/default/cli.html?highlight=slow-first#cmdoption-slow-firstpass)

The "turbo" options are indeed quite different from the original options in slow presets. But I don't remember any test showing verbosely how obvious the loss of quality is... the following pass(es) will still use the intended efforts, just the bitrate calculation may be based on a not really optimal reference distribution.
I did some noodling around using the slow-firstpass but reducing subme and a few other low-level things for the first pass. Testing wasn't that complete, but it looked like for a net total encoding time, I could get slightly better results by allowing the second pass to be slower by making the first pass faster. It went to more a 25/75 than 50/50 ratio.

The big problem I see with --no-slow-firstpass is ref=1, which can really impact things with some kinds of content. I would think it is critical to use the same number of reference and b-frames and same badapt mode between passes.


So, using --no-slow-firstpass and then setting --ref to the same as the second pass is likely a good starting point to experiment from

[benwaggoner]

sorry if offtop but I am currious what settings amazon is using for encoding TV series in HEVC using x265 encoder.
In the MI there is rc=crf / crf=21.5. Are they using some encoding preset (medium, slow, slower, etc) or some fine tuned own setting? (probably stupid question)
That is not a question I will be answering in any form :). Everything I write here is my personal opinion and experience only.

[foxyshadis]

That is not a question I will be answering in any form :). Everything I write here is my personal opinion and experience only.

I can just see the headlines in six months. "Famed compressionist Ben Waggoner caught in dirty format lies!" "World scoffs as he claims they were true at the time"

As for --nr... I thought that would be a quick find. No, you actually found a bug in getopt, the venerable GNU freaken getopt. It lets you abbreviate options, normally, and has a sanity check in case it matches a second option... but that sanity check only triggers if the shortname (eg, -V) or whether it needs an argument differs. Just in case the developer brain-farted an inserted the same option twice, it wouldn't blow up. But they never bothered to compare the full names!

So yes, entirely by accident, --nr means --nr-intra. I wonder if my getopt.c patch will break anyone's workflow.

[LigH]

I have just noticed a proposed patch in the mailing list that addresses this issue. Was that yours? ... They prefer patch sources in the mail body, not as attachment only.

[RainyDog]

I did some noodling around using the slow-firstpass but reducing subme and a few other low-level things for the first pass. Testing wasn't that complete, but it looked like for a net total encoding time, I could get slightly better results by allowing the second pass to be slower by making the first pass faster. It went to more a 25/75 than 50/50 ratio.

The big problem I see with --no-slow-firstpass is ref=1, which can really impact things with some kinds of content. I would think it is critical to use the same number of reference and b-frames and same badapt mode between passes.


So, using --no-slow-firstpass and then setting --ref to the same as the second pass is likely a good starting point to experiment from

Thanks benwaggoner.

Yeah, ref=1 and rd=2 were the two settings in --no-slow-firstpass that I saw as cause for concern too. Especially since I sometimes use rd=5 for the 2nd pass too.

How can I specify --ref 4 and --rd 3 with --no-slow-firstpass?

[LigH]

Just in the same command line.

x265 --preset ... --no-slow-firstpass --ref 4 --rd 3 ...

First the "meta options" which change several parameters at once, then atomic options which override these changes.

[RainyDog]

Just in the same command line.

x265 --preset ... --no-slow-firstpass --ref 4 --rd 3 ...

First the "meta options" which change several parameters at once, then atomic options which override these changes.

Thanks LigH.

Yeah I do that anyway as I always make quite a few tweaks to the standard presets. But what if I want to specify --rd 3 only for the fast 1st pass but use --rd 5 for the slow 2nd pass?

My thoughts are that --rd 3 will provide a much more suitable analysis than --rd 2 for the 1st pass and use psy-rdoq.

[LigH]

Just do it; compare --rd 2 / 3 / 5 in 1st pass (and always 5 in last) among each other, tell us if you see any difference. I doubt. You will be sure afterwards.

[RainyDog]

Just do it; compare --rd 2 / 3 / 5 in 1st pass (and always 5 in last) among each other, tell us if you see any difference. I doubt. You will be sure afterwards.

Well, I would... If I knew how to specify the --rd level and ref's in the 1st pass. That's what I was hoping you could tell me :)

Would it be :-

x265 --preset slow --no-slow-firstpass --pass 1 --ref 4 --rd 3(or 5) --pass 2... followed by the remainder of my manually set options.

[LigH]

No. You have to complete the 1st pass before you run the 2nd.

So first, run
x265 --preset slow --pass 1 --no-slow-firstpass --ref 4 --rd 3 ...
until it finished, to produce mainly a statistics file (if it writes also a video file, then this is surely not yet optimally encoded). Then run
x265 --preset slow --pass 2 --ref 4 --rd 5 ...
and now you have your resulting video file.

[IgorC]

An interesting article about compression/performance ratio of x265's presets

http://x265.org/performance-energy-consumption-analysis-x265/

[benwaggoner]

No. You have to complete the 1st pass before you run the 2nd.

So first, run
x265 --preset slow --pass 1 --no-slow-firstpass --ref 4 --rd 3 ...
until it finished, to produce mainly a statistics file (if it writes also a video file, then this is surely not yet optimally encoded). Then run
x265 --preset slow --pass 2 --ref 4 --rd 5 ...
and now you have your resulting video file.
--dynamic-rd could also be an interesting feature to use in tuning quality/speed. In theory it would allow for faster encoding for easy parts of the video, only switching to higher subme etcetera for the harder bits.

[jd17]

An interesting article about compression/performance ratio of x265's presets

http://x265.org/performance-energy-consumption-analysis-x265/

Thank you for posting this.

I feel quite confirmed in my personal findings, i.e. --preset slow being the sweetspot for great quality at a still reasonable encoding speed, since the jump to slower is already immense, while the benefit to quality is practically non-existent in my eyes. :)

[Boulder]

There's also at least the value of --max-merge being raised in the slower presets which (at least in my opinion) affect by degrading quality. It will smooth things more --> lower bitrate.

[benwaggoner]

There's also at least the value of --max-merge being raised in the slower presets which (at least in my opinion) affect by degrading quality. It will smooth things more --> lower bitrate.
The key thing about a first pass is that it has to maintain roughly proportional rate control and similar frame/slice type decisions. If it is just 15% less efficient, no biggie. But if it’s efficiency gap varies a lot throughout the clip, than it just isn’t an accurate initial effort to refine.

So a no-slow-firstpass should focus on features that don’t have highly variable impact based on content. Things looking different is probably a good proxy for that, so if you see max-merge changing the character of the encore significantly, you’d probably want to have it st the same value in both passes.

[imhh11]

Hi, thanks to all the developers.

UHD-BD are encoded at level 5.1 but when I select that level, I get this warning.
Should I ignore the warning or just use 5.0 which isn't uhdbd compliant ?

https://extraimage.net/images/2018/07/13/deb6472663fb6bfbb4810a81a638bd3a.png

https://extraimage.net/images/2018/07/13/a578fe21813d0c7a5cabe7edd6c81208.png

Also, is there anything in my setting that I should modify in order to retain grain better?

Thank you and sorry for my English

[Boulder]

Better grain retention = add --no-sao and --no-strong-intra-smoothing.

[imhh11]

Better grain retention = add --no-sao and --no-strong-intra-smoothing.

thanks, I'll try that.


EDIT. Wow its working well . still not perfect but way better than before.

http://screenshotcomparison.com/comparison/116849

http://screenshotcomparison.com/comparison/116846

http://screenshotcomparison.com/comparison/116847



and never mind about my first question, added --vbv-bufsize 160000 --vbv-maxrate 160000 and no more warning.

[Capella Systems]

--dynamic-rd could also be an interesting feature to use in tuning quality/speed. In theory it would allow for faster encoding for easy parts of the video, only switching to higher subme etcetera for the harder bits.

We have implemented this in our transcoding software (Cambria FTC) on top of x265 and it works quite well - a lot of videos are mostly easy to encode and the 80/20 rule is in our favor here.

[RainyDog]

--dynamic-rd could also be an interesting feature to use in tuning quality/speed. In theory it would allow for faster encoding for easy parts of the video, only switching to higher subme etcetera for the harder bits.

But doesn't --dynamic-rd only affect the RD level and only become applicable when VBV rate control is enabled?

Increases the RD level at points where quality drops due to VBV rate control enforcement. The number of CUs for which the RD is reconfigured is determined based on the strength. Strength 1 gives the best FPS, strength 4 gives the best SSIM. Strength 0 switches this feature off. Default: 0.

Effective for RD levels 4 and below.

[Boulder]

and never mind about my first question, added --vbv-bufsize 160000 --vbv-maxrate 160000 and no more warning.

If you are going to watch the result on a PC or a modern HTPC/media player, I think you can safely skip specifying the decoder level and leave the VBV buffer things out. They can only lower the quality of the result.

For better quality including grain retention, I also recommend testing raising qcomp. Try values between 0.6 and 0.8. The bitrate will rise as well though.

[imhh11]

If you are going to watch the result on a PC or a modern HTPC/media player, I think you can safely skip specifying the decoder level and leave the VBV buffer things out. They can only lower the quality of the result.

For better quality including grain retention, I also recommend testing raising qcomp. Try values between 0.6 and 0.8. The bitrate will rise as well though.

yes, I'm watching it on an HTPC /gtx1070.

Thank you so much for your recommendations, I'm already blown away by the difference. :D

[uneedme]

Hello Dudes,

I have tried to encode a video...

From 5 mins and on, the quality is getting worse and worse steadily(it is like auto crf downgrade). and then I split the source video into pieces and encoded them again. Every outputs are good then.

with the same params, I dont know why......

I suspect "--psy-rd 3.2" is the cause or it is a bug?......Or some wrong commands-combinations cause the chaos...

Cheers.


--crf 18.6 -m 2 --me 1 --rd 5 --rc-lookahead 120 --merange 114 --bframes 6 --ref 6 --min-keyint 23 --keyint 240 --max-merge 5 --tu-intra-depth 3 --tu-inter-depth 2 --aq-motion --qg-size 16 --no-sao --no-strong-intra-smoothing --psy-rd 3.2 --scenecut 80 --deblock -1:3 --aq-mode 2 --qcomp 0.8 --aq-strength 0.8 --colorprim bt709 --colormatrix bt709 --transfer bt709 --input-res 1862x768 --fps 24000/1001 --input-depth 10 -D 10 -o "1" --rc-grain --input -

[microchip8]

remove --aq-motion and try again

[LigH]

x265 2.8+47-e2759ae31c36 (https://www.mediafire.com/file/9ra7f7rvi4oszwb/x265_2.8%2B47-e2759ae31c36.7z)

several fixes related to HDR10+ LLC JSON and dependencies between effects of options

[alex1399]

I've got some 60fps video materials with wrongly encoded frame-rate which has lots of duplicated frames.
Encoding those video tends to exceed the bits per second budget.
1. remove duplicated frame
2. increase qcomp
3. set tu-inter-depth=4:tu-intra-depth=4
The 1. and 2. don't perform well, those "still" frames seems to compromised by the poorly unstable sampling rate and ghosting artifacts.
The 3. is interesting.
Instead, how to set x265 parameters to act just like a corrected frame-rate video but farther Intra-coded (I) frames and high level hierarchy Bi-directional predicted (B) frames?

[FranceBB]

If it's 60fps with many dups, you may wanna decimate it using a frameserver first.
I would use something like this in Avisynth:

tdecimate(mode=2, rate=30)

change "rate" according to the original frame-rate of your source.

Then, you said that "it tends to exceed the bits per second budget".
I assume you mean that it requires too much bitrate than the one you are trying to keep as maximum.
If that's the case, I would increase the reframe parameter and a few other things, but beware that it's gonna be more resource intensive to both encode and decode.
Try with --ref 16 --me esa --subme 11

[Jamaika]

How to get around this?
api.cpp:491:1: error: jump to label 'fail' [-fpermissive]
fail:
^~~~
In file included from api.cpp:24:
common.h:225:18: note: from here
goto fail; \

[Ma]

Easiest solution:
before building x265 please execute
export CXXFLAGS="-fpermissive"

[alex1399]

After some suggestions and a fast review of x265 documents, quick conclusion comes out for the x265 encoder parameters.
My boss will reject those that are not heuristic methods so trivial trimming frame is not option.

keyint = 10*frame-rate. Long distance key-frames are best for Still frames, based on frame-rate without affecting decoder side too much. Default is 250.
scenecut = 30. Scene-cut detection is less aggressive to ignore some ghosting / blending scene. Default is 40.
rc-lookahead = 0.8*frame-rate. Add some more lookahead for frame-type decision. Default is 20.

Rest of the options i.e., bframes, ref and many motion prediction would be based on the presets by default. Feel free to give your comments about those custom options and or de-noising (mostly from high-bit-depth to 8-bit-depth dithering) or de-ghosting stuffs?

[kento]

I just wish to thank everyone for the hard work and the suggestions made through the years !

[uneedme]

remove --aq-motion and try again

ooh cheers,

I will have another try...

(just cant logon the forum dont know why)



From current processing, it is sure aq-motion cause the Chaos......But Why. Is this experimental param still Buggy?

[foxyshadis]

After some suggestions and a fast review of x265 documents, quick conclusion comes out for the x265 encoder parameters.
My boss will reject those that are not heuristic methods so trivial trimming frame is not option.

keyint = 10*frame-rate. Long distance key-frames are best for Still frames, based on frame-rate without affecting decoder side too much. Default is 250.
scenecut = 30. Scene-cut detection is less aggressive to ignore some ghosting / blending scene. Default is 40.
rc-lookahead = 0.8*frame-rate. Add some more lookahead for frame-type decision. Default is 20.

Rest of the options i.e., bframes, ref and many motion prediction would be based on the presets by default. Feel free to give your comments about those custom options and or de-noising (mostly from high-bit-depth to 8-bit-depth dithering) or de-ghosting stuffs?

If you have a bit-budget, in any way, shape, or form, then you need to turn on VBV. You're only exceeding it because you didn't set up VBV correctly. If you're consistently exceeding it throughout the whole video, you can either raise the profile (longer encoding, bit of a crapshoot on how much it lowers bitrate) or raise the crf (guaranteed bitrate and quality reduction).

B-frames aren't set by the profile, only the tune, so setting them manually to a higher value is a solid option, but after 3 b-frames the value falls off enormously. (Except in certain classes of animation.)

[alex1399]

Increasing bframes with those duplicated frames doesn't help at all, maybe. Might require more experiments under the PSNR tunes to figure out. The hierarchy stuffs in x265 is quite abstract, so left those for presets.

VBV seems to be great if videos would be publicly released on air in advance. On the contrary, increasing qcomp and qpstep provide something better in motion view.

[katzenjoghurt]

https://media.giphy.com/media/4Td3vJgrjPKO0mOGMe/giphy.gif (https://imgbb.com/)

Sources:
https://www2.pic-upload.de/img/35671437/encoding_original.png
https://www2.pic-upload.de/img/35671439/encoding_x265.png

Slider Comparison Tool Link:
https://cdn.knightlab.com/libs/juxtapose/latest/embed/index.html?uid=1cd634b2-8dcc-11e8-b263-0edaf8f81e27

Hey community!

Hm... I feel something's funky with my settings but I don't know what.
SOMETIMES the encodings come out with a bit washed out colors.
It especially seems to happen in darker scenes...
See the gif above.

For most movies it seems to not happen at all.

What is it and how can I detect / prevent it?

I used Staxrip with these encoding settings:

short:

--crf 22 --tune grain --profile main10 --output-depth 10 --rskip --qcomp 0.8 --no-open-gop --sar 12:11 --no-deblock --no-strong-intra-smoothing


detailed settings:

Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L4@Main
Codec ID : V_MPEGH/ISO/HEVC
Duration : 2 h 13 min
Bit rate : 6 416 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.129
Stream size : 6.00 GiB (77%)
Writing library : x265 2.8+40-0106f9f2f867:[Windows][GCC 7.3.0][64 bit] 10bit
Encoding settings : cpuid=1111039 / frame-threads=3 / numa-pools=8 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=1920x1080 / interlace=0 / total-frames=192702 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=3 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / no-open-gop / min-keyint=23 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=20 / lookahead-slices=6 / scenecut=40 / radl=0 / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-merge=2 / limit-refs=3 / no-limit-modes / me=1 / subme=2 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / no-deblock / no-sao / no-sao-non-deblock / rd=3 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=4.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=22.0 / qcomp=0.80 / qpstep=1 / stats-write=0 / stats-read=0 / ipratio=1.10 / pbratio=1.00 / aq-mode=0 / aq-strength=0.00 / no-cutree / zone-count=0 / no-strict-cbr / qg-size=64 / rc-grain / qpmax=69 / qpmin=0 / const-vbv / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-mv-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei
Default : Yes
Forced : No


Someone mentioned that washed out colors may be caused by a too low bitrate... but hum... I don't see much of a difference when rising it.

[Atak_Snajpera]

gif in 256 colors is a bad way to demonstrate your issue.

[katzenjoghurt]

gif in 256 colors is a bad way to demonstrate your issue.

It visualizes the point: The warm face color turns grayish.

Added the sources to the original posting.

[Selur]

looking at it in Firefox at 200% on a HP 5k z27q with a Geforce GTX 1070t, I see the color change around the eye, but sure what is causing this.

[WhatZit]

...
colorprim=2 / transfer=2 / colormatrix=2
...

There's a lot we don't know about your source, such as its bitdepth, frame size, encoder family, or colour tagging.

A conversion from any of those source states could introduce colour-shifting depending entirely on what specific process you're using, especially rounding errors when resizing or denoising, or even a bad source decoder. Heaps of variables.

However, given that your MediaInfo dump of the x265 encode shows "unknown" for all of the colour characteristics, I'd start my investigation there.

[jd17]

Hm... I feel something's funky with my settings but I don't know what.
SOMETIMES the encodings come out with a bit washed out colors.
It especially seems to happen in darker scenes...
See the gif above.

Without knowing all the details already requested by others, I can at least offer a suspicion:

Assumption:
Your source is 8bit and you encode to 10bit.
If that is the case, what you see is most likely not an issue with your encode, but rather the decoder.
I stumbled over this a while back - I could only see the issue on my PC, but not when I watched the videos on my TV (clean 10bit decoding chain).

I think this was already discussed somewhere here on doom9 as well. :)

[katzenjoghurt]

Hi jd17! :)

OMG! You are right!!
It's an 8bit source.

Tried a bit around and compared it against a main 8bit encoding and the problem was gone. (Using StaxRip's comparison tool)

I then opened the 8bit and the main10 10bit file in VLC and... again no washed out colors.

So the issue seems to be related to StaxRip's screen comparison tool.


Thx a lot, Sherlock! :thanks:

[LigH]

I remember people putting two media players side by side, wondering about color differences ... but they set up the media player to use the Hardware Overlay renderer. Which can have different color controls, compared to the desktop. And can only be used by one player. The other player must then select an alternative desktop-contained renderer.

One renderer in the overlay, the other in the desktop: in this case, color differences are no surprise.

[alex1399]

If you have a bit-budget, in any way, shape, or form, then you need to turn on VBV. You're only exceeding it because you didn't set up VBV correctly. If you're consistently exceeding it throughout the whole video, you can either raise the profile (longer encoding, bit of a crapshoot on how much it lowers bitrate) or raise the crf (guaranteed bitrate and quality reduction).

OK, I found the culprit of the encoding process. The frame numbers of encoded video reported by ffprobe is one more than the original video material. In 2-pass encoding, the -f null - in the first pass encoding seems to bypass vsync detection and provide a invalid analysis for the coming second pass encoding which has a duplicated frame at the start.

If the -f null - in the first pass encoding is replaced by any dummy output, both output of the first pass encoding and the second pass encoding will have the same duplicated frame and the analysis is valid.

Is it possible to log only these error(warning) into txt file during the batch process, so I could read it when I'm back to screen?

[LigH]

VSync (Vertical Retrace Synchronisation) is related to a video monitor, a synchronism between monitor and video frame rate ... you probably mean audio/video synchronism. Whatever it means in detail, this will be an issue related to ffmpeg; x265 (as a separate encoder, or as the core library) processes video only and can only rely on the sequence of input frames. If the ffmpeg core serves a duplicate frame, blame ffmpeg.

[alex1399]

I'm sorry that confusing, the audio/video sync stuffs that ffmpeg names not the monitor sync stuffs. Does x265 itself have any mechanic to log out error, for example, somebody purposely use a different video at the second pass encoding to do something wrong?

[LigH]

If x265 uses a statistics file in a 2nd pass which is based on analyzing a different movie in the 1st pass, then it is possible that just the bitrate distribution is not optimal; if you have additional VBV restrictions, it might happen that they fail; if they have a different number of frames, the mismatch may be discovered in the end; but in general, just the quality may vary unexpectedly. There are no security measures to match statistics file and video source (which would be nearly impossible anyway if it comes out of a pipe). Your only workarounds I can imagine right now:

a) let the 1st pass create an output file too;
b) ignore audio during video encoding to avoid this issue if it is not in sync in the source file, always process audio separately, and multiplex it to the final video only

In any case, x265 is not to be blamed.

[alex1399]

The second pass encoding just use the slice-type order from the first pass encoding where the first pass intently encodes video A and the second pass intently encodes video B respectively. Both video A and B have the same frame-rate, resolution and amount of frames. the slice-type order is checked by ffprobe -show_frames.

Before I closed the paused batch window during the weekend, it shows something like "[warning]: specified frame type (*) at **** is not compatible with keyframe interval" and "[error]: slice=I but 2pass stats say B", never mind. Now a dummy output is assigned for the first past encoding output.

During debug mode, to record the PSNR statics, options tune PSNR for x265, psnr stats_file for ffmpeg, and move stats_file SomeGenericNameNum are mixed in the batch process. If the result is promising, then the debug stuffs will be canceled and use the default tuning for the rest of the videos.

A bonus demonstration of the mad Adam from RWBY V5E2 for the encoding misplacement
https://i.imgur.com/b6wmHFz.jpg

[alex1399]

No VBV restrictions right now. Just trying to simulate how could those videos badly up-sampled and up-scaled and low-bit-depth to high-bit-depth / high-bit-depth to low-bit-depth conversion back and forth. Now I feel disgusting about those superfluous 10-bit-depth video which was encoded from 8-bit-depth raw source.

A problematic USB stick is fine to reproduce binary corruption / missing in poor network to observe blocking artifacts. --gop-lookahead value larger than --min-keyint value is great on letting x265 produces more noise. Hard-coded artifacts from --dithering and --deblock that make things look not natural : a little subjective. And so on.

[FranceBB]

Now I feel disgusting about those superfluous 10-bit-depth video which was encoded from 8-bit-depth raw source.

Actually, it makes sense to encode in 10bit starting from an 8bit source. I know that if there's banding already in the 8bit source, there's gonna be banding in the 10bit encode as well, but if I had to deband an 8bit source, I wouldn't encode it back to 8bit unless I really need to. Besides, Main10 is the "de-facto" standard and is widely compatible with players (unlike H.264 High10 that wasn't very supported).

[MeteorRain]

Now I feel disgusting about those superfluous 10-bit-depth video which was encoded from 8-bit-depth raw source.

Have you ever taken the internal bit-depth / precision loss into consideration? IIRC 8-bit HEVC uses 8-bit internal processing which will effectively reduce the bit-depth to lower than 8-bit. (Correct me if I'm outdated)

Take math as an example. The area of a circle of 4.5, computing in 1-digit after point internal precision is 4.5 * 4.5 -> 20.2, then 20.2 * pi -> 62.6. However computing in 2-digit internal precision gives you 63.58 which is much closer to the real number 63.61725015. Totally different result.

[Asmodian]

That was true in AVC but HEVC uses a high bitdepth for most of the math internally, for all output bitdepths, but it is still a good idea to always encode to 10 bit HEVC.

[MeteorRain]

That was true in AVC but HEVC uses a high bitdepth for most of the math internally, for all output bitdepths

My memory says that, but I did a quick search and couldn't find anything saying x265 is using high bitdepth for internal precision. So not very certain about that part.

[Boulder]

I also recall reading somewhere that internally everything happens in 16-bit precision.

[Barough]

x265 v2.8+56-613074c6714f (http://www.mediafire.com/file/fd532su2k80zbxv/) (GCC 7.3.0, 32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[foxyshadis]

My memory says that, but I did a quick search and couldn't find anything saying x265 is using high bitdepth for internal precision. So not very certain about that part.

The spec absolutely hates being clear about things mere mortals like to know, but there are a few points: All dct transform processing is from -2^15 to 2^15 (7.4.3.2.2), meaning signed 16-bit. That's about it, though; for purposes of processing they're immediately scaled to bitdepth, and reference prediction & filtering are performed in the bit-depth of the encode. (8.5.3-4 & 8.7.2 has much of the gory details.)

[alex1399]

Thanks for everybody's input. To provide an additional reason why lossless x265 is superior, Video A and B are presented for a trivial scheme : interweave.
First, interweave both video; Second, select and mux the even frame; third, compare it under PSNR measure.

Video A: https://www5.zippyshare.com/v/uVQP4Ccc/file.html
Video B: https://www87.zippyshare.com/v/pkkl1Eph/file.html

x265 crf 0 interweave scheme
ffmpeg -i in.y4m -i out.y4m -filter_complex framepack=frameseq -c:v libx265 -crf 0 output.mp4
ffmpeg -i output.mp4 -vf select=mod(n\,2) -r 24000/1001 -c:v libx265 -crf 0 output1.mp4
ffmpeg -i output1.mp4 -i out.y4m -lavfi psnr=stats_file=psnr.csv -f null -

x265 lossless interweave scheme
ffmpeg -i in.y4m -i out.y4m -filter_complex framepack=frameseq -c:v libx265 lossless=1 output.mp4
ffmpeg -i output.mp4 -vf select=mod(n\,2) -r 24000/1001 -c:v libx265 -x265-params lossless=1 output1.mp4
ffmpeg -i output1.mp4 -i out.y4m -lavfi psnr=stats_file=psnr.csv -f null -

raw lossless interweave scheme
ffmpeg -i in.y4m -i out.y4m -filter_complex framepack=frameseq -f yuv4mpegpipe output.y4m
ffmpeg -i output.y4m -vf select=mod(n\,2) -r 24000/1001 -f yuv4mpegpipe output1.y4m
ffmpeg -i output1.y4m -i out.y4m -lavfi psnr=stats_file=psnr.csv -f null -

A mean-square-error of 0.00 but PSNR of non-inf is quite surprised me in the crf=0 x265 scheme.
Maybe there exists some bit-exact-accuracy options I didn't activate for x265, but crf=0 is definitely not lossless.

Out of topic
x264 crf 0 interweave scheme
ffmpeg -i in.y4m -i out.y4m -filter_complex framepack=frameseq -c:v libx264 -crf 0 output.mp4
ffmpeg -i output.mp4 -vf select=mod(n\,2) -r 24000/1001 -c:v libx264 -crf 0 output1.mp4
ffmpeg -i output1.mp4 -i out.y4m -lavfi psnr=stats_file=psnr.csv -f null -

[Boulder]

--crf 0 is not meant to be lossless.

[sneaker_ger]

It is for 8 bit x264. I suspect the error is not in libx264, maybe ffmpeg muxing/demuxing. Either way this has nothing to do with x265. Please move to a new thread or better yet report on ffmpeg bug tracker.

[alex1399]

If x265 is fine, I'm fine. Just messing around to seek some ordinary mistakes that common people will make during encoding process. Those rare conditions are not bugs, maybe.

[K.i.N.G]

I really like x265 but I seem to be unable to get rid of linear smearing/stretching artifacts when there are fast moving objects in a scene.
Is there a specific parameter targeted at improving this, without increasing the bit rate in other areas (those are fine)?

My settings are:
--crf 17 --preset veryslow --profile main10 --level-idc 5 --output-depth 10 --psy-rdoq 4 --aq-mode 3 --qg-size 64 --qcomp 0.7 --subme 5 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50)" --colorprim bt2020 --colormatrix bt2020nc --transfer smpte2084 --max-cll "457,179" --hdr --hdr-opt --deblock -1:-1 --no-sao --no-strong-intra-smoothing

example:
https://i.imgur.com/Nm3l7L2.png

[Boulder]

--qg-size could be something to look at. Lowering it will cause the bitrate to rise though.

[K.i.N.G]

--qg-size could be something to look at. Lowering it will cause the bitrate to rise though.

Thank you, I've lowered it to 16 and will report back when its finished.

I can live with 'slight' bitrate increases (which is perfectly understandable).

[LigH]

A value of 64 was only recommended for UHD material and has caused known over-blurring issues in the history of x265; I am not sure if they were completely fixed. 32 used to be a default; 16 may even be a bit pessimistic.

[K.i.N.G]

A value of 64 was only recommended for UHD material and has caused known over-blurring issues in the history of x265; I am not sure if they were completely fixed. 32 used to be a default; 16 may even be a bit pessimistic.

Its an UHD source
I tried a value of 16 to see if its better (would've tried 32 afterwards to see if its good enough).

Anyway, setting it to 16 made the artifacts smaller and thus a bit less obvious, but they still annoy me...

Right now I'm doing an encode with strong intra smoothing to see if that maybe helps...

Going to try an higher AQ strength after that but I'm affraid that's going to boost the bit rate a bit too much since it also affects areas that are already good enough (if im not mistaken?)...

[Boulder]

A value of 64 was only recommended for UHD material and has caused known over-blurring issues in the history of x265; I am not sure if they were completely fixed. 32 used to be a default; 16 may even be a bit pessimistic.

I started using 16 after I switched to --ctu 32.

[foxyshadis]

I really like x265 but I seem to be unable to get rid of linear smearing/stretching artifacts when there are fast moving objects in a scene.
Is there a specific parameter targeted at improving this, without increasing the bit rate in other areas (those are fine)?

My settings are:
--crf 17 --preset veryslow --profile main10 --level-idc 5 --output-depth 10 --psy-rdoq 4 --aq-mode 3 --qg-size 64 --qcomp 0.7 --subme 5 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50)" --colorprim bt2020 --colormatrix bt2020nc --transfer smpte2084 --max-cll "457,179" --hdr --hdr-opt --deblock -1:-1 --no-sao --no-strong-intra-smoothing

example:
https://i.imgur.com/Nm3l7L2.png

The only way to get x265 to retain detail/grain in absolutely all circumstances is --tune grain combined with enough bitrate to not starve it. That will massively change but the bitrate and distribution of your entire movie, not a first choice if you're happy with the rest.

I noticed you raised qcomp to 0.7 from the default 0.6, effectively telling x265, "I want fast-motion and difficult-to-compress scenes compressed harder than usual." Was there a specific reason for raising it?

Lastly, are you sure you're not fixating on three frames that in motion will be utterly unnoticeable everyone who actually watches? It's a very common ailment when you go down the rabbit hole of settings. You have to balance frame analysis, especially in "fast moving objects" scenes, with real-life watching of a whole whole scene.

[alex1399]

I've got some 60fps video materials with wrongly encoded frame-rate which has lots of duplicated frames.

OK, seems like I underestimated the benefit from cranking up --bframes. Provides 5% bit-rate reduction with the same quality.
The options are
--preset slower --tune psnr --profile main --ctu 32 --keyint 600 --scenecut 30 --rc-lookahead 48 --bframes 16 --qg-size 16 --pass * --qpstep 10 --range limited

Note that --tune psnr is removed in advance.

[K.i.N.G]

The only way to get x265 to retain detail/grain in absolutely all circumstances is --tune grain combined with enough bitrate to not starve it. That will massively change but the bitrate and distribution of your entire movie, not a first choice if you're happy with the rest.

I noticed you raised qcomp to 0.7 from the default 0.6, effectively telling x265, "I want fast-motion and difficult-to-compress scenes compressed harder than usual." Was there a specific reason for raising it?

Lastly, are you sure you're not fixating on three frames that in motion will be utterly unnoticeable everyone who actually watches? It's a very common ailment when you go down the rabbit hole of settings. You have to balance frame analysis, especially in "fast moving objects" scenes, with real-life watching of a whole whole scene.

Well I mentioned it is specifically a fast moving scenes/objects I am having trouble with...
Maybe I should try actually raising it even more to 0.8

It's not really noticable on my monitor, but when viewed on a 65" oled I got a bit annoyed by it...

Even with 1080p encodes with average bitrates around 7000 I still notice it sometimes.
I can't speak for anyone else, but to me it looks very ugly and in some cases it is really noticible.


Also, not related to this (i think), when encoding HDR content with x265 it seems as if the adaptive algorithms don't take the tonemapping/different contrast handling of HDR footage into account?
What i mean is that when you play HDR content on a non-HDR display, it looks really flat and it seems to me this is how the footage gets analyzed by x265's CRF/VBR which results in an out of balance preservation of detail in flat areas.
When the encoded footage is played back on a HDR screen, I get the impression there is more loss in detail in low contrast areas than usual...
I hope you guys understand what I mean? It's like x265 doesnt take the contrast adjustments of HDR content into account.
Already started a separate thread about this, but maybe I should've posted this here aswell...

[Boulder]

I noticed you raised qcomp to 0.7 from the default 0.6, effectively telling x265, "I want fast-motion and difficult-to-compress scenes compressed harder than usual."

This is a bit strange. When you raise qcomp, the bitrate gets substantially higher with CRF mode. Shouldn't it be the opposite?

[K.i.N.G]

Oh damn, i miss-read... so qcomp compresses harder?
But that also makes no sense to me since it does indeed result in higher bit rates (at same CRF) and improves quality concerning my fast motion artifacts issue...

[LigH]

From Xvid I remember a "quantizer curve compression", there it changed the quantizer distribution character. I believe it is still true to some extent for x265. From the official documentation:

Command line options: --qcomp <float> (https://x265.readthedocs.io/en/default/cli.html?highlight=qcomp#cmdoption-qcomp)
qComp sets the quantizer curve compression factor. It weights the frame quantizer based on the complexity of residual (measured by lookahead). Default value is 0.6. Increasing it to 1 will effectively generate CQP

CQP doesn't care about the complexity = degree of details in a quantized unit; frame parts with higher complexity will probably survive a coarser quantization better than parts with less details, banding will become more obvious where it is not cut by an edge or covered by a pattern.

[Wolfberry]

qcomp trades off the number of bits allocated to "expensive" high-motion versus "cheap" low-motion frames. At one extreme, qcomp=0 aims for true constant bitrate (CBR). Typically this would make high-motion scenes look completely awful, while low-motion scenes would probably look absolutely perfect, but would also use many times more bitrate than they would need in order to look merely excellent. At the other extreme, qcomp=1 achieves nearly constant quantization parameter (CQP). Constant QP does not look bad, but most people think it is more reasonable to shave some bitrate off of the extremely expensive scenes (where the loss of quality is not as noticeable) and reallocate it to the scenes that are easier to encode at excellent quality.

A higher qcomp will allow distributing more bits to high-motion scenes, thus making them look better.

[LigH]

x265 2.8+57-eea92165b035 (https://www.mediafire.com/file/h0au8vvwnq2tj5w/x265_2.8%2B57-eea92165b035.7z)

Enhance VBV lookahead of RADL pictures (and a few more small building and API fixes)

[Magik Mark]

I'm getting this error since v2.8 + 49 in Staxrip:

Error Video encoding using x265 2.8+56 (1.7.0.6)

Video encoding using x265 2.8+56 failed with exit code: -1073741819 (0xC0000005)

The exit code might be a system error code: The instruction at 0xp referenced memory at 0xp. The memory could not be s.

StaxRip.Proc.Start() in D:\Projekte\VS\VB\StaxRip\General\Proc.vb:line 338
at StaxRip.x265Enc.Encode(String passName, String commandLine, ProcessPriorityClass priority) in D:\Projekte\VS\VB\StaxRip\Encoding\x265Enc.vb:line 65
at StaxRip.x265Enc.Encode() in D:\Projekte\VS\VB\StaxRip\Encoding\x265Enc.vb:line 42
at StaxRip.GlobalClass.ProcessVideo() in D:\Projekte\VS\VB\StaxRip\General\GlobalClass.vb:line 225
at System.Threading.Tasks.Parallel.<>c__DisplayClass4_0.<Invoke>b__0()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at StaxRip.GlobalClass.ProcessJob(String jobPath) in D:\Projekte\VS\VB\StaxRip\General\GlobalClass.vb:line 137

Not sure if this is the result in changes in the switches

[LigH]

Most interesting here: There are placeholders for pointers and strings in a format string which are not interpreted as if preceded by a %. I would suspect issues with string functions.

Where did you get these x265 builds from, which compiler did they use? (e.g. run "x265 -V" in a console, that should reveal used compilers, or additional format string failures)

There was a patch e2759ae (https://bitbucket.org/multicoreware/x265/commits/e2759ae31c3638518d4a6358a884f569efae1298):
dhdr: Replace the header "string" with its C++ equivalent <cstring> to fix build failures.
It looks a bit compiler dependent to me. But I am no expert regarding C++ compilers...

[Magik Mark]

Thanks LigH for the thoughts. I have been getting my build from here:

http://msystem.waw.pl/x265/

VS 2017 AVX 2

[LigH]

Please compare with my latest 2.8+57 (https://forum.doom9.org/showthread.php?p=1847763#post1847763) and previous 2.8+47 (https://forum.doom9.org/showthread.php?p=1846673#post1846673) (GCC 7.3.0, generic) and Barough's 2.8+56 (https://forum.doom9.org/showthread.php?p=1847447#post1847447).

And please try to execute the encodes directly in a console window, to avoid mixing up error messages from x265 with error messages from StaxRip.

[Motenai Yoda]

A higher qcomp will allow distributing more bits to high-motion scenes, thus making them look better.

A lower qcomp will limit ditributing bits to high-motion scenes, allowing to assign them to low-motion scenes whose you notice more a quality loss

[Magik Mark]

Please compare with my latest 2.8+57 (https://forum.doom9.org/showthread.php?p=1847763#post1847763) and previous 2.8+47 (https://forum.doom9.org/showthread.php?p=1846673#post1846673) (GCC 7.3.0, generic) and Barough's 2.8+56 (https://forum.doom9.org/showthread.php?p=1847447#post1847447).

And please try to execute the encodes directly in a console window, to avoid mixing up error messages from x265 with error messages from StaxRip.

Hi Ligh!

Thanks for helping out. First of all I'm not familiar with with "console window". I exclusively encode using staxrip. I have noticed that build 47 below exhibits no problem at all. This made me think that the problem lies with new or modified switches. Unfortunately, the author of staxrip called-in quit, so no help from him at all

[LigH]

Yes, stax76 is not so active anymore. To use x265 in a more frequently updated GUI, which also allows more customization, I'd suggest MeGUI.

It will also be important to know your CPU. Does it support AVX2 at all? And it is interesting to know when x265 crashes. Immediately after starting an encoding job, or after calculating for a while? The log in MeGUI is more verbose here. It would report all of its output, including the CPU ID part.

[Ma]

I'm getting this error since v2.8 + 49 in Staxrip:

I've tried to reproduce the problem (but no hangs):
ffmpeg -i ../original.mkv -v warning -f yuv4mpegpipe - | x265-49 --y4m - --bitrate 1500 --pass 1 --ssim-rd --aq-mode 3 --pools 28 NUL
y4m [info]: 1920x1080 fps 24000/1001 i420p8 sar 1:1 unknown frame count
raw [info]: output file: NUL
x265 [info]: HEVC encoder version 2.8+49-5d34bbf671f7
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main 10 profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 28 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 4 / wpp(17 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 3 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 3 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : ABR-1500 kbps / 0.60
x265 [info]: tools: rd=3 ssim-rd rskip signhide tmvp strong-intra-smoothing
x265 [info]: tools: lslices=6 deblock sao stats-write
x265 [info]: frame I: 11, Avg QP:24.83 kb/s: 18638.87
x265 [info]: frame P: 349, Avg QP:26.64 kb/s: 4358.80
x265 [info]: frame B: 1173, Avg QP:32.98 kb/s: 431.42
x265 [info]: Weighted P-Frames: Y:20.3% UV:9.7%
x265 [info]: consecutive B-frames: 3.9% 1.1% 12.5% 30.3% 52.2%

encoded 1533 frames in 64.16s (23.89 fps), 1456.17 kb/s, Avg QP:31.48

If I understood you correctly: 2.8+47 works, 28+49 hangs at beginning, this might be problem with x265 initialization. There are no new switches from 2.8+47 to 2.8+49.

Could you confirm that 2.8+47 works, 2.8+49 hangs, and what with version 2.8+48 -- please download only 10-bit VS2015 AVX2 builds (2.8+47, 2.8+48 and 2.8+49) and report back which works and which not: www.msystem.waw.pl/x265/test.7z

[katzenjoghurt]

Just replaced x265 in Staxrip with LigH's 2.8+57 version.
No error yet.

(Current encoding seems to be a bit sluggish but need to observe further - it could very well just be movie related)


I'm using StaxRip 1.7.0.6 from https://github.com/stax76/staxrip/blob/master/changelog.md.

[Magik Mark]

Ma & LigH,

Found the error in 2pass encoding:

Multipass analysis refinement along with multipass rate control
Multipass refinement of qp based on distortion data

If these two are deactivated everything is ok

Maybe the syntax has changed?

[Ma]

Thanks for the info!

It looks like a bug in x265.

You could/should find exact commit that hangs, for example 2.8+47 works/2.8+48 hangs or 2.8+48 works/2.8+49 hangs. And find in StaxRip log x265 command line and copy it in this thread.

[Magik Mark]

Build above 47 all hangs

[iAvoe]

@x265_Project it seems that you would be the only one who can answer my question, since it's becoming complex and I believe the answer can be very long... please have a look(In HEVC, only QP=4 is truly lossless quantization... what about 0~3?): forum.doom9.org/showthread.php?t=175638

[Dclose]

I really like x265 but I seem to be unable to get rid of linear smearing/stretching artifacts when there are fast moving objects in a scene.
Is there a specific parameter targeted at improving this, without increasing the bit rate in other areas (those are fine)?

My settings are:
--crf 17 --preset veryslow --profile main10 --level-idc 5 --output-depth 10 --psy-rdoq 4 --aq-mode 3 --qg-size 64 --qcomp 0.7 --subme 5 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50)" --colorprim bt2020 --colormatrix bt2020nc --transfer smpte2084 --max-cll "457,179" --hdr --hdr-opt --deblock -1:-1 --no-sao --no-strong-intra-smoothing

example:
https://i.imgur.com/Nm3l7L2.png
1) imo, if you care about things that move, (and picture quality in general), you have to use sub-motion pixel subme 7. 5 is good, and is as low as I ever set that even on files I'm trying to finish fast, but 5 is easily visually inferior to 7 imo. 7 of course takes longer to encode though.

2) You have qg-size 64. I almost never encode 4k lately, but for 1080, my coding, quant, and tree unit settings are a low of 8 and high of 32, and Max Intra/Inter are maxed. 64 didn't look as sharp, and didn't have any noticeable advantages, even during a fairly recent test I did of them.

3) AQ mode. "auto" (mode 2?) was too inconsistent in quality for me. A main thing is faces tend to lack quality. And faces tend to be the main place on the screen to look at. If mode 3 is the "experimental/dark area" mode, the file sizes were too inconsistent for me. That mode tended to throw a lot of bitrate at the file and too often made the sizes huge. I use normal mode now, for consistency of video quality and filesize. I haven't retested the others in a year or so, so maybe they have improved.

4) With later releases of x265, I stopped messing with the q-comp type of settings. I did a big test on them again a couple months ago and found the default settings are very good.

5) -1/-1 is a lot of deblocking. I use -5/-4 even on encodes most people would probably consider very low bitrate. I'm usually around crf 21-24 though, not 17, so maybe deblocking has less effect at 17 anyway. At crf 17, I would think some obvious setting is wrong somewhere for it to not look great.

[Asmodian]

Deblocking strength scales with the amount of compression, at lower CRF values deblocking is automatically weaker.

[brumsky]

I'm having issues with Ripbot264 and Staxrip when trying to encode a 4k video. Do they support UHD? They work fine for 1080p content....

If not, what should I use?

thanks,
Brumsky

[user1085]

Ripbot works for me for 4k out of the boxI'm having issues with Ripbot264 and Staxrip when trying to encode a 4k video. Do they support UHD? They work fine for 1080p content....

If not, what should I use?

thanks,
Brumsky

[LigH]

Always the same mistake: The resolution alone is not the relevant attribute of a video. There is still a wide variety of possible container and content formats which could be used to store video with such a resolution. Use MediaInfo to tell us relevant technical attributes.

And if you have issues, tell us about the nature of these issues as verbose as necessary. A minimum requirement is quoting an error message letter by letter, if there is any, possibly even providing a log file. "I'm having issues" is not a sufficient description.

If the conversion crashes for downloaded moviez, you are left at your own peril.

In any case, it would be off-topic in a thread related to the x265 encoder, the reason for issues is usually rather the decoding than the encoding. You may have created a separate thread instead because it happens for more than one converter application.

[LigH]

MSYS2 recently updated MinGW64 with GCC 8.2.0; due to some internal compiler errors, MinGW32 will stay with GCC 7.3.0, though, until these issues are solved.

x265 2.8+58-d17bc7714ed2 (Win32-GCC730 & Win64-GCC820) (https://www.mediafire.com/file/k2qjvytar5odqjd/x265_2.8%2B58-d17bc7714ed2%28Win32-GCC730_Win64-GCC820%29.7z)

[brumsky]

@user1085

Thank I wanted to make sure it was supported out of box.

@LigH

LigH, I certainly agree that I did not provide enough information for proper troubleshooting. I know my question would be borderline, at best, for this thread. I just wanted to confirm that those application supported UHD out of the box. Now that I know it is I can proper troubleshoot the issue. I didn't want to waste a ton of time if it wasn't supported to begin with.

It is during decode, mostly ffms2.dll, while it is being indexed. I say mostly as I have had another error not related to ffms2.dll.


Thanks for the quick response and sorry for the "I'm having issues" post. :)

[LigH]

@brumsky:

Indexing already requires scanning the whole source video. If that already fails, there is a chance that your source has a "hole" ...

[FranceBB]

I asked it years ago, but I'm gonna ask it again:

Any chance to see assembly optimisations for Main10 on x86 anytime soon in the future?

I know that x64 is what pretty much anyone use nowadays, but it would be useful to have manual assembly optimisation in x86 as well, not just for 8bit, but also for Main10, 'cause it would speed things up a lot.

Test performed with x265 2.8+58-d17bc77 x86 using the following system:

CPU: Intel i7 6700HQ 4c/8th 3.20GHz
RAM: 16 GB (8x2) DDR4
OS: Windows XP Professional x86 with PAE (unlocked HAL) + Microsoft Extended Support
OS: Windows 7 Professional x64

Clip encoded: 4K UHD 10bit 4:2:0 23.976fps source.
Common settings: --preset medium --level 5.0 --tune fastdecode --ref 2 --rc-lookahead 3 -b 2 --profile main10 --bitrate 25000 --deblock -4:-4 --no-open-gop --min-keyint 1 --keyint 24 --repeat-headers --rd 3

1) x265 Main10 plain C++ (GCC 8.2 Optimisation disabled) Win XP x86 = 0.15fps
2) x265 Main10 plain C++ (GCC 8.2 Optimisation SSE4.2) Win XP x86 = 0.44fps
3) x265 Main10 SSE4.2 asm (GCC 8.2 Optimisation SSE4.2) Win 7 x64 = 1.88fps
4) x265 Main10 AVX2 asm (GCC 8.2 Optimisation AVX2) Win 7 x64 = 2.60fps

As you can see from the results, GCC manages to speed up the code by optimising plain C++ code to SSE4.2 automatically, but it's nearly not as fast as the manual assembly optimisation written by x265 developers, which is more than 4 time faster, but unfortunately it's available for x64 only. I'm well aware that implementing manual SSE4.2 assembly optimisation in x86 wouldn't give the same speed boost as it does in x64 due to the different architectures, but it would definitely improve performances over plain C++ (which is all we have for Main10 in x86 right now).
I would post benchmarks of x265 compiled with Visual Studio 2017 as well, but unfortunately I didn't manage to compile the multilib. (8/10/12bit) versions for Win32 with Visual Studio 2017. I did manage to compile the 8bit version, though, but that's not really useful.

So... do you think assembly optimisations on x86 will be introduced for Main10 too anytime soon?

Thank you in advance.

[NikosD]

I'm well aware that implementing manual SSE4.2 assembly optimisation in x86 wouldn't give the same speed boost as it does in x64 due to the different architectures...

So... do you think assembly optimisations on x86 will be introduced for Main10 too anytime soon?

Nice post, but I think you have already given yourself the answer.

x64 doubles the number of registers and is a lot easier, not only faster, for a developer to implement assembly optimizations.

I don't think that in 2018 it's some kind of priority to optimize for x86.

The percentage of x86-only OSes and CPUs are close to 0.

Of course, nothing stops you from asking.

[sneaker_ger]

So... do you think assembly optimisations on x86 will be introduced for Main10 too anytime soon?
You mean "re-introduce". Because in the past those existed but the developers deliberately removed them. Not because it wasn't faster but because they wanted to spend their dev time on other things.

So get an old version, new PC/OS or find someone who still develops it. I believe Ma had some branch for it but I don't know how old/recent it is.

[LigH]

And again the same answer: Very doubtful. The developers already decided to abandon this part of x265, because of reasons:


twice the efforts to make assembly routines with fewer and smaller CPU registers in 32 bit CPU mode
half the available RAM because 10 bit precision per color channel need 16 bit RAM instead of 8 bit for storage, and the limitation to 2 GB (or 4 GB for LAA processes) does not even allow encoding of FullHD (not to mention UHD)

_

Damn, something delayed my reply remarkably. I thought I posted it right after the question...

[Asmodian]

and the limitation to 2 GB (or 4 GB for LAA processes) does not even allow encoding of FullHD (not to mention UHD)


Interesting point, I had not thought of the memory footprint.

And LAA only applies to 32 bit on 64 bit systems, so the extra work optimizing 32 bit x265 for 10 bit does seem like a poor use of talent.

[Barough]

x265 v2.8+59-b44d5f0e42f8 (http://www.mediafire.com/file/pg6ady7j79vd7r9/) (32-bit GCC 7.3.0 / 64-bit GCC 8.2.0 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[Ma]

@Magik Mark

I've looked at commits 2.8+48 (b0d31e2) and 2.8+49 (5d34bbf). In version +49 there is potentially dangerous change from one (atomic) 32-bit operation to two 16-bit operations. I've reverted these changes -- you can test if patched version 2.8+58 hangs or not (patch file inside)
www.msystem.waw.pl/x265/x265-2.8+58-patched_vs2017-AVX2.7z

[Magik Mark]

Same problem ma

[Ma]

Same problem ma

Thanks for info!

Did you check ver. 2.8+48 (form test.7z in post #6260 (https://forum.doom9.org/showthread.php?p=1847951#post1847951))?

[Atak_Snajpera]

Ryzen Threadripper 2990wx uses 4 NUMA nodes and I would like to check if running 4 instances with manually adjusted --numa-pools could improve performance.
Can somebody verify if those are correct switches?


Instance 1 = --numa-pools "+,-,-,-"
Instance 2 = --numa-pools "-,+,-,-"
Instance 3 = --numa-pools "-,-,+,-"
Instance 4 = --numa-pools "-,-,-,+"


Without any adjustments 5 instances give this
https://p.xfastest.com/~sinchen/GIGABYTE-X399-AORUS-XTREME/GIGABYTE-X399-AORUS-XTREME-66.jpg
2990wx@3.4GHz(all core turbo) is only 20% faster than 1950@3.4GHz

[Sagittaire]

Ryzen Threadripper 2990wx uses 4 NUMA nodes and I would like to check if running 4 instances with manually adjusted --numa-pools could improve performance.
Can somebody verify if those are correct switches?


Instance 1 = --numa-pools "+,-,-,-"
Instance 2 = --numa-pools "-,+,-,-"
Instance 3 = --numa-pools "-,-,+,-"
Instance 4 = --numa-pools "-,-,-,+"


Without any adjustments 5 instances give this
https://p.xfastest.com/~sinchen/GIGABYTE-X399-AORUS-XTREME/GIGABYTE-X399-AORUS-XTREME-66.jpg
2990wx@3.4GHz(all core turbo) is only 20% faster than 1950@3.4GHz

well 5 instance just became too low for 1080p source ...:eek:

32C/64T for 5 instance for 1080p is more than 6C/12T for each 1080p instance. Unfortunaly, x265 have threading problem at 8 thread (and more) for 1080p source.

If you want really saturate 64 thread CPU, you must use at least 8 instance for 1080p source or at least 2 instance for 2160p source. And perhaps that 8x 1080p instance will saturate RAM with particular CCX connexion (even with quad DDR4 channel).

[Atak_Snajpera]

No it is not too low. Dual socket (2 NUMA) Intel Xeon E5-4660 v3 (56 threads total) still scales much better than single socket (4 NUMA) 2990WX.
It would probably scale even better if I set numa pools manually.

According to x265 documentation ( https://x265.readthedocs.io/en/default/threading.html )
If you are running multiple encoders on a system with multiple NUMA nodes, it is recommended to isolate each of them to a single node in order to avoid the NUMA overhead of remote memory access.
Can somebody verify than I'm setting numa pools correctly in my previous post?

[zub35]

x264 has a good optimization option
--tune film [--deblock -1:-1 --psy-rd <unset>:0.15]

why not have the same for x265 ?
--tune film [--no-sao --no-strong-intra-smoothing --psy-rd 4]

[RieGo]

x264 has a good optimization option
--tune film [--deblock -1:-1 --psy-rd <unset>:0.15]

why not have the same for x265 ?
--tune film [--no-sao --no-strong-intra-smoothing --psy-rd 4]

afaik a film preset is on the todo list... probably may take a while till they/we figure out all sane parameters.

now... why does everyone think it's a good idea to switch off Sample Adaptive Offset in-loop filter? i read about it and it sounds like a nice feature to improve efficiency - no matter what kind of video content is encoded.
i understand that there was supposely a little problem in the early stages of x265 with sao integration. but is this still a thing or is everybody just blindly turning off sao?

[microchip8]

afaik a film preset is on the todo list... probably may take a while till they/we figure out all sane parameters.

now... why does everyone think it's a good idea to switch off Sample Adaptive Offset in-loop filter? i read about it and it sounds like a nice feature to improve efficiency - no matter what kind of video content is encoded.
i understand that there was supposely a little problem in the early stages of x265 with sao integration. but is this still a thing or is everybody just blindly turning off sao?

SAO still blurs too much so many people disable it if they want to retain as much details as possible. However, at very low bitrates where other artifacts are more visible/present, the blur of SAO produces "better looking" images than an encode without it

[RieGo]

SAO still blurs too much so many people disable it if they want to retain as much details as possible. However, at very low bitrates where other artifacts are more visible/present, the blur of SAO produces "better looking" images than an encode without it

thanks.
i did some visual comparisons lately but wasn't able to detect any kind of differences at high bitrate - i didn't look at still images, only at video scenes.
at very low bitrate (300kbit/s) there was a lot of quality differences with different parameters, but I didn't look at no-sao...

so probably i'm just a bad quality judge. :D

[benwaggoner]

thanks.
i did some visual comparisons lately but wasn't able to detect any kind of differences at high bitrate - i didn't look at still images, only at video scenes.
at very low bitrate (300kbit/s) there was a lot of quality differences with different parameters, but I didn't look at no-sao...

so probably i'm just a bad quality judge. :D
SAO should do less as QP goes down, so what you see is how it should work.

[NikosD]

Can somebody verify if those are correct switches?

Instance 1 = --numa-pools "+,-,-,-"
Instance 2 = --numa-pools "-,+,-,-"
Instance 3 = --numa-pools "-,-,+,-"
Instance 4 = --numa-pools "-,-,-,+"



Can somebody verify than I'm setting numa pools correctly in my previous post?

Please, don't expect answers regarding AMD optimizations in this thread.

They are all Intel fans or worse fanboys.

Even the developers.

[LigH]

In general, generalizations are wrong. If I could afford a new PC, I would buy a Ryzen. But I could still not buy the insight in its NUMA structure.

[FranceBB]

If you want really saturate 64 thread CPU, you must use at least 2 instance for 2160p source.

Not just 64 thread CPU, at work I have two Intel Xeon E5-2660V4 14c/28th for a total of 28c/56th and I can't still saturate both CPUs with a 2160p 10bit HDR10 content encoded with preset --medium and bluray compatible specs.



They are all Intel fans or worse fanboys.

Some consumers are moving to AMD, but the majority of businesses are using Intel Xeon CPUs (my company included), so that's what they ask for optimizations.
They are simply following the market needs, nothing more.

[Barough]

x265 v2.8+66-88ee12651e30 (http://www.mediafire.com/file/087abao10la3nsd/) (32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[benwaggoner]

Not just 64 thread CPU, at work I have two Intel Xeon E5-2660V4 14c/28th for a total of 28c/56th and I can't still saturate both CPUs with a 2160p 10bit HDR10 content encoded with preset --medium and bluray compatible specs.
That's not surprising. Something like --preset slower would probably be better, but there's only so much threading that can be usefully done in a single instance, and Blu-ray restrictions reduce even that (b-frames can encode in parallel, but BD only allows 2 consecutive). Increasing -F will help, but high values can cause rate control issues.

[jlpsvk]

Not just 64 thread CPU, at work I have two Intel Xeon E5-2660V4 14c/28th for a total of 28c/56th and I can't still saturate both CPUs with a 2160p 10bit HDR10 content encoded with preset --medium and bluray compatible specs.

could you post your uhd bd compatible command line?

[excellentswordfight]

afaik a film preset is on the todo list... probably may take a while till they/we figure out all sane parameters.

now... why does everyone think it's a good idea to switch off Sample Adaptive Offset in-loop filter? i read about it and it sounds like a nice feature to improve efficiency - no matter what kind of video content is encoded.
i understand that there was supposely a little problem in the early stages of x265 with sao integration. but is this still a thing or is everybody just blindly turning off sao?
Using --no-sao for a tune film is imo valid. In my experience no-sao does improve fine detail alot with almost no negative effects for general "film" content with lower crf values. Preset slow together with no-sao is imo enough for detail retention now days. Not sure what setting does it, but I find preset Medium to be way softer then preset slow (imo there should only be a bitrate difference between them when doing a CRF encode, but it doesnt work like that I guess).

I have found sao to be usefull for both animation and low bitrate content though (as expected).

Not just 64 thread CPU, at work I have two Intel Xeon E5-2660V4 14c/28th for a total of 28c/56th and I can't still saturate both CPUs with a 2160p 10bit HDR10 content encoded with preset --medium and bluray compatible specs.
To add to this, I see around 70-80% utilization on dual Xeon E5-2680 v3 (48t) systems for 2160p content using preset slow. Imo that is a very reasonable ammount of multithread performance. For 1080p I wouldnt bother with anything more then 8-12C. Start using chunk-encoding if better multithread utilization is needed.

But I still think Atak question is valid, does 2990wx need any NUMA tweaking to perform correctly?

[RieGo]

Using --no-sao for a tune film is imo valid. In my experience no-sao does improve fine detail alot with almost no negative effects for general "film" content with lower crf values. Preset slow together with no-sao is imo enough for detail retention now days. Not sure what setting does it, but I find preset Medium to be way softer then preset slow (imo there should only be a bitrate difference between them when doing a CRF encode, but it doesnt work like that I guess).

I have found sao to be usefull for both animation and low bitrate content though (as expected).

thanks for your opinion :)
i will do some more visual tests with high and low bitrates and only film content. maybe i can finally understand all your motivation to turn off sao.

update: wow.
so I did a quick test @6000/1000/100 kbit/s with and without sao.
so you are 100% right. no-sao looks just much sharper and retains more details. even I can see it...
not just true on high bitrate but also on medium/low bitrate. but I can understand why it would make kinda sense to have a smooth low bitrate encoding.
so basically i'm sorry for not believing, need to change my presets now. lol

[benwaggoner]

update: wow.
so I did a quick test @6000/1000/100 kbit/s with and without sao.
so you are 100% right. no-sao looks just much sharper and retains more details. even I can see it...
not just true on high bitrate but also on medium/low bitrate. but I can understand why it would make kinda sense to have a smooth low bitrate encoding.
so basically i'm sorry for not believing, need to change my presets now. lol
Can you share the bitrates and/or command lines you were using?

[FranceBB]

That's not surprising. Something like --preset slower would probably be better, but there's only so much threading that can be usefully done in a single instance, and Blu-ray restrictions reduce even that (b-frames can encode in parallel, but BD only allows 2 consecutive). Increasing -F will help, but high values can cause rate control issues.

Yes... With slower I might get something more but still, it's still acceptable.

I see around 70-80% utilization on dual Xeon E5-2680 v3 (48t) systems for 2160p content using preset slow. Imo that is a very reasonable ammount of multithread performance.

Yes, it kinda is.

could you post your uhd bd compatible command line?

Sure.

UHD HDR10 BD50:
x265.exe --y4m - --dither --preset medium --level 5.1 --tune fastdecode --no-high-tier --ref 4 --profile main10 --bitrate 75000 --deblock -1:-1 --hdr-opt --hrd --min-luma 64 --max-luma 940 --chromaloc 2 --range limited --videoformat component --colorprim bt2020 --transfer smpte2084 --colormatrix bt2020nc --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,0.0050)" --max-cll 1000,400 --overscan show --no-open-gop --min-keyint 1 --keyint 24 --repeat-headers --rd 3 --vbv-maxrate 75000 --vbv-bufsize 75000 --asm=avx2 --wpp -o "H:\raw_video.hevc"

I know that some of you might be screaming "Ah!!" to that "brutal" clipping, but the uncompressed 16bit stream that x265 is gonna encode has already been brought in Tv Range with a proper LUT that tries to avoid to clip too much, so that's just for "safety reasons", especially 'cause otherwise QC refuses it, even if there's just a single scene in the video that is out of range. No, they don't stare at the video-scope all the time; such a process is automated by a machine that checks the file 1:1 and reports details about luma, chroma, whether there are freeze-frames, blocking of whatever type and so on. Sometimes it fails and it is spot-checked by a human, but still, they refuse the content if it's out of range.


Still, using --preset slow might help a bit, but I would have to specify parameters myself, especially 'cause I'm not using --uhd-bd and I would end up by limiting myself anyway.
Increasing --ref from 4 to 6 might also help.
As to the 75Mbit/s, the specs require the bitrate to stay below 82Mbit/s, but 75Mbit/s plus a bit of oscillation up and down and audio tracks is gonna be fine.

[RieGo]

Can you share the bitrates and/or command lines you were using?

nothing fancy, just a simple "--pass x --bitrate 6000 --preset slow --pmode [--no-sao]"

[LigH]

Was --pmode useful in your case? It does not cause a speedup in general, it depends on the circumstances, I read...

[RieGo]

Was --pmode useful in your case? It does not cause a speedup in general, it depends on the circumstances, I read...

i *think* it improves my cpu saturation. but only on 1080p or lower using 24 threads. wasn't able to get a constant saturation without it.
with 4k content everything is fine even without pmode.
i didn't really make any extensive speed tests though.

[benwaggoner]

i *think* it improves my cpu saturation. but only on 1080p or lower using 24 threads. wasn't able to get a constant saturation without it.
with 4k content everything is fine even without pmode.
i didn't really make any extensive speed tests though.
Pmode can easily increase CPU utilization AND reduce encoding speed if you don’t have a whole lot of unused cores when running without it. I’ve seen it speed up encoding 400x224 on a 32 logical core system, but never 1080p or above. But I’ve not tried on anything with >36 logical cores.

Pmode can also theoretically increase quality a bit, since a lot of its parallel work is stuff that would normally have gotten skipped due to early exit. Occasionally it’ll find something better than what was found before the early exit. I’ve never seen it really make a material difference compared to veryslow or placebo.

[RieGo]

Pmode can easily increase CPU utilization AND reduce encoding speed if you don’t have a whole lot of unused cores when running without it. I’ve seen it speed up encoding 400x224 on a 32 logical core system, but never 1080p or above. But I’ve not tried on anything with >36 logical cores.

Pmode can also theoretically increase quality a bit, since a lot of its parallel work is stuff that would normally have gotten skipped due to early exit. Occasionally it’ll find something better than what was found before the early exit. I’ve never seen it really make a material difference compared to veryslow or placebo.

yes my feeling was that it might be slower with pmode, but as i said i never actually did speed tests, just looked at cpu usage lol. my bad...
maybe it's a good idea for me to just remove it.
but going to slower is not an option (for me) - slow -> slower almost increases encoding time 100%

[benwaggoner]

yes my feeling was that it might be slower with pmode, but as i said i never actually did speed tests, just looked at cpu usage lol. my bad...
maybe it's a good idea for me to just remove it.
but going to slower is not an option (for me) - slow -> slower almost increases encoding time 100%

Yeah, it is quite likely that pmode is slowing you down a bunch and turning it off could get you that 100% speed back.

[vidschlub]

I'm having a discussion with someone online and they are citing an early 2016 discussion about 264 vs 265.
Does anyone know if there's a much more recent comparison of 264 to 265?

My assumption is, by now, with the correct settings used in the encoder, 265 should basically provide a superior image at the same bitrate, almost always (until the returns diminish at very high bitrates)
Surely, that is now the case?

[alex1399]

h.264 is still the grain king if you don't care the blocking it have.

[Blue_MiSfit]

h.264 is still the grain king if you don't care the blocking it have.

Unless we're talking 4K, particularly for HDR.

If you can't afford archival level bitrates, HEVC is dramatically better in almost every case, especially at high resolution.

[Forteen88]

I'm having a discussion with someone online and they are citing an early 2016 discussion about 264 vs 265.
Does anyone know if there's a much more recent comparison of 264 to 265?Video Codecs Comparison 2017,
http://www.compression.ru/video/codec_comparison/hevc_2017/ (http://www.compression.ru/video/codec_comparison/hevc_2017/)

Although they probably didn't compare 10-bit x265 vs 8bit x264 (x264 10-bit isn't supported by hardware-decoders!).
EDIT: OK, Nvidia GeForce 950/960 or better PC GPU supports full H265/HEVC 10-bit hardware-decoding, it doesn't support H264 10-bit hardware-decoding.

[NikosD]

Although they probably didn't compare 10-bit x265 vs 8bit x264 (x264 10-bit isn't supported by hardware-decoders!).

I think that mobile SOCs include HW decoding of H.264 10bit.

Probably Smart TVs, too.

[microchip8]

I think that mobile SOCs include HW decoding of H.264 10bit.

Probably Smart TVs, too.

I have 2 recent Smart TVs (Samsung and Panasonic) and 3 blu-ray players (2 from Samsung and 1 from LG). The Samsung BD players are UHD models

None of the devices above support 10-bit H.264 decoding

[excellentswordfight]

I'm having a discussion with someone online and they are citing an early 2016 discussion about 264 vs 265.
Does anyone know if there's a much more recent comparison of 264 to 265?

My assumption is, by now, with the correct settings used in the encoder, 265 should basically provide a superior image at the same bitrate, almost always (until the returns diminish at very high bitrates)
Surely, that is now the case?
I would say yes, if speed is not considered. But when tuning x265 to be as fast as x264 it falls behind imo.

Most of the test I've done has been in the "rip" catagory, I found that x265 --slow --no-sao --crf 18 has very similar fidelity to x264 --slower --tune film --crf 18 for 1080p bluray re-encoding with a 20-30% bitrate reduction. Most test I've done has been on tears of steel, which is a pretty good source for "general" film content imo, but it could ofc be sources were these numbers dont apply at all (but it has been the case on a few other random blurays I've tested on as well).

[Forteen88]

I hope that they finish the Video Codecs Comparison 2018 on that website soon. They've released an Express Report 2018, but it doesn't include "Ultra Ripping: Comparison on extremely slow presets" yet,
http://www.compression.ru/video/codec_comparison/hevc_2018/

[Przemek_Sperling]

I have 2 recent Smart TVs (Samsung and Panasonic) and 3 blu-ray players (2 from Samsung and 1 from LG). The Samsung BD players are UHD models

None of the devices above support 10-bit H.264 decoding

Weird, I own a cheap settop box (Opticum Sloth Combo Plus) and it decodes H.264 10-bit as well as H.265 12-bit. Maybe because of its chipset (Sunplus 1507). Most such devices have Ali chipsets and maybe they cannot decode such material.

[microchip8]

Weird, I own a cheap settop box (Opticum Sloth Combo Plus) and it decodes H.264 10-bit as well as H.265 12-bit. Maybe because of its chipset (Sunplus 1507). Most such devices have Ali chipsets and maybe they cannot decode such material.

I don't know the chipsets of my devices, but you are most likely correct. That said, there are quite a few devices (TVs, BD players & co) that don't support 10 bit H.264. My Samsung TV, however, supports decoding of 10 bits HEVC but not 10 bits H.264. I haven't tested 12 bits HEVC on it yet

[singhkays]

Using --no-sao for a tune film is imo valid. In my experience no-sao does improve fine detail alot with almost no negative effects for general "film" content with lower crf values. Preset slow together with no-sao is imo enough for detail retention now days. Not sure what setting does it, but I find preset Medium to be way softer then preset slow (imo there should only be a bitrate difference between them when doing a CRF encode, but it doesnt work like that I guess).

I have found sao to be usefull for both animation and low bitrate content though (as expected).


To add to this, I see around 70-80% utilization on dual Xeon E5-2680 v3 (48t) systems for 2160p content using preset slow. Imo that is a very reasonable ammount of multithread performance. For 1080p I wouldnt bother with anything more then 8-12C. Start using chunk-encoding if better multithread utilization is needed.

But I still think Atak question is valid, does 2990wx need any NUMA tweaking to perform correctly?

Not just 64 thread CPU, at work I have two Intel Xeon E5-2660V4 14c/28th for a total of 28c/56th and I can't still saturate both CPUs with a 2160p 10bit HDR10 content encoded with preset --medium and bluray compatible specs.


Some consumers are moving to AMD, but the majority of businesses are using Intel Xeon CPUs (my company included), so that's what they ask for optimizations.
They are simply following the market needs, nothing more.

I recently did some investigations around x265 scaling with 128 cores. You might be interested in the results https://www.singhkays.com/blog/x265-128-core-scaling-4k-hevc-hdr-azure-vm/

https://www.singhkays.com/img/128-core-x265-scaling/fps-vs-cores.png

No it is not too low. Dual socket (2 NUMA) Intel Xeon E5-4660 v3 (56 threads total) still scales much better than single socket (4 NUMA) 2990WX.
It would probably scale even better if I set numa pools manually.

According to x265 documentation ( https://x265.readthedocs.io/en/default/threading.html )

Can somebody verify than I'm setting numa pools correctly in my previous post?

See my investigation above. The CPU details are in the blog post. Not sure if I can help you verify something.

[Atak_Snajpera]

4k veryslow is the best case scenario for core utilization. 1080p with default medium preset would require at least 8 concurrent encodes to saturate all those 128 cores.

Ps. I'm not surprised that 2160p scales up to 32 cores. If We divide 2160 by default CU of 64 then we get value of 33.75.

[K.i.N.G]

I have 2 recent Smart TVs (Samsung and Panasonic) and 3 blu-ray players (2 from Samsung and 1 from LG). The Samsung BD players are UHD models

None of the devices above support 10-bit H.264 decoding

my nvidia shields, sony led tv (4k non-hdr) and lg oled tv (4k hdr) all decode it just fine, even my phone (samsung s5) plays them.

[microchip8]

my nvidia shields, sony led tv (4k non-hdr) and lg oled tv (4k hdr) all decode it just fine, even my phone (samsung s5) plays them.

With the exception of the two Samsung BD players, all my other devices are Full HD only. I must have bad luck because none can decode 10 bits H.264, including the Samsung BD players that say "file unsupported" when trying to feed them 10 bit H.264

That said, it's not important for me since I moved over to 10 bits HEVC which is decodable on the Samsung BD players *and* my Full HD Samsung TV. My Panasonic TV (also FHD) doesn't support it so I use one of the BD players to decode and feed it. I also prefer using the BD players to stream to my TVs as I can use Bitstream passthrough for the audio, which I can't when using the TVs directly (they internally convert it to AC3 - I use Toslink to feed audio from TVs to Yamaha receiver. Neither Toslink nor ARC supports lossless audio passthrough)

[benwaggoner]

4k veryslow is the best case scenario for core utilization. 1080p with default medium preset would require at least 8 concurrent encodes to saturate all those 128 cores.

Ps. I'm not surprised that 2160p scales up to 32 cores. If We divide 2160 by default CU of 64 then we get value of 33.75.
There is a 3 CTU lag in frame parallelism in x265, however. Multithreading performance tuning in x265 is a pretty complex matter. I find this invaluable:

https://x265.readthedocs.io/en/default/threading.html

(x265 has the best documentation of any codec, ever!)

[benwaggoner]

I recently did some investigations around x265 scaling with 128 cores. You might be interested in the results https://www.singhkays.com/blog/x265-128-core-scaling-4k-hevc-hdr-azure-vm/

https://www.singhkays.com/img/128-core-x265-scaling/fps-vs-cores.png



See my investigation above. The CPU details are in the blog post. Not sure if I can help you verify something.
Interesting data.

I would expect that, running multiple instances on multiple sockets like that, your 4x performance would be better if you used --pools to lock each instance to one socket to improve cache coherency and reduce NUMA utilization. Ala --pools "+,-,-,-" to lock to just the first socket of four.

[singhkays]

Interesting data.

I would expect that, running multiple instances on multiple sockets like that, your 4x performance would be better if you used --pools to lock each instance to one socket to improve cache coherency and reduce NUMA utilization. Ala --pools "+,-,-,-" to lock to just the first socket of four.

Thanks! I'm doing a follow up based on the comment below on the blog, so I'll include the above optimization as well. Are there other optimizations you'd like to see?

I'm confused about the using HEVC as the input file codec -- this will force decode delay, and unless it is full-intra, it will be multi-frame decode which will churn memory and processing just to get a decoded frame into the encode pipe.

A commercial application of this would be to encode a feature-length mezzanine, say an uncompressed MXF with a bit-depth and raster-size matching the output. Such a file would test raw encoding capability in high-cpu environments.

[benwaggoner]

Thanks! I'm doing a follow up based on the comment below on the blog, so I'll include the above optimization as well. Are there other optimizations you'd like to see?
That's the only one that popped out. Changes --pools will reduce the number of logical cores available and thus will also reduce the default --frame-threads and anything else that is based on core count, but that should happen automatically.

If you really want to stress single-instance encoding across all those sockets, try --pmode, or maybe even --pme if that doesn't saturate things. Those both increase CPU utilization more than they increase speed, but I bet you'd get more net speed out of a single instance with four cores with --pmode.

If you are looking to add more work for the encoder to do, both --cu-lossless and --tskip will help.

[RainyDog]

Unfortunately my PC's just crashed during the 2nd pass of a 2-pass encode.

If I run a 1-pass ABR encode using/reading the stats file that was generated during the 1st pass of the encode that crashed, am I right in assuming the result will be the same as if the 2nd pass had completed?

Thanks.

[LigH]

Why not running the 2nd pass of the 2-pass encode again?

[RainyDog]

Why not running the 2nd pass of the 2-pass encode again?

Thanks LigH. Well, that's kind of what I'm asking... Will a 1-pass ABR encode using the existing stats file produce the same result as if it were a 2nd pass?

Basically I'm using staxrip as the GUI and the encode options are 2-pass, Bitrate (1-pass ABR) and Quality (CRF). If I select 2-pass again then it will just start again and run another 1st pass creating a new stats file.

But if I select Bitrate ABR mode and add --pass 2 and --stats as custom encode options, when I set it going it takes a few seconds to read the stats file and then states --stats read once it starts. So its definitely using the stats file but my concern is is it using the 'proper' 2nd pass algorithm and not the poorer 1-pass ABR algorithm which I believe is internally different to the other modes...

Thanks.

[LigH]

I don't know which GUI you are using (x265 is a CLI encoder). If you can't select "2-pass, 1st pass" and "2-pass, 2nd pass" separately, but only "2-pass" as a bundle, then it may not be very helpful for your case.

The workaround you describe may work, but I didn't think much about it. After you tell us which GUI you are using, we might know a better solution, with or without this GUI.

By the way, the 1st pass may have been so quick, it could have been ran in the meantime...

[RainyDog]

I don't know which GUI you are using (x265 is a CLI encoder). If you can't select "2-pass, 1st pass" and "2-pass, 2nd pass" separately, but only "2-pass" as a bundle, then it may not be very helpful for your case.

The workaround you describe may work, but I didn't think much about it. After you tell us which GUI you are using, we might know a better solution, with or without this GUI.

By the way, the 1st pass may have been so quick, it could have been ran in the meantime...

I'm using Staxrip.

Well, the reason I didn't necessarily want to re-run the 1st pass is that I use custom 1st pass settings which are almost the same as the 2nd pass settings with a few changes to speed it up a bit. So it this instance the 1st pass took aalmost 6 hours... so if I can re-use it's stats file then it would save me 6 hours at least anyway!

Would also like to know for future reference really, in case it happens again.

Thanks.

[nevcairiel]

So just run the 2nd pass again using the stats file? Thats part of what makes the separate passes so nice, you can just run the 2nd pass again. Ordinarily one might use it to tune settings or whatnot, but if it failed you can also just start over.

[RainyDog]

So just run the 2nd pass again using the stats file? Thats part of what makes the separate passes so nice, you can just run the 2nd pass again. Ordinarily one might use it to tune settings or whatnot, but if it failed you can also just start over.

Hi nevcairiel, yes that's what I think I'm doing.

But I was just unsure whether I'd be unwittingly running an encode using ABR rate control mode as that's the option I'm having to select in Staxrip even though it's using a stats file already created from a successful 1st pass.

I believe ABR rate control mode is not recommended under any circumstances unless you must hit a certain bitrate/size and simply dont have the time for a 1st pass as it uses a different interal algorithm to CRF or 2pass encodes...

[nevcairiel]

You should probably ask the authors of that GUI how to run only a 2nd pass again. With the x265 CLI interface its pretty straight forward.

[RainyDog]

You should probably ask the authors of that GUI how to run only a 2nd pass again. With the x265 CLI interface its pretty straight forward.

Do you just add --pass 2 and --stats to the commnd line using the CLI interface?

[nevcairiel]

Basically yes, together with all other encoding options.

[RainyDog]

Basically yes, together with all other encoding options.

Thanks. Yeah, that's what I've done in the Staxrip custom command line options section.

I set it going before I headed off to work this morning and it definitely read/loaded the stats file before starting the encode and stated "stats-read" on the GUI output display anyway. Which it always does on a 2nd pass and states "stats-write" on a 1st pass.

[alex1399]

Hey guys, you might overlook the three pass encode for a fine result. The 1-pass ABR encode have two meaning, the --pass 1 activated first pass ABR encode and the initial ABR encode without --pass 1 activated. Once the --pass 1 is not included in the first pass ABR encode, add the --pass 1 and the --pass 2 for the following second and third pass encode. Else you could add --pass 3 and --pass 2 for the following three pass encode.

[benwaggoner]

Hey guys, you might overlook the three pass encode for a fine result. The 1-pass ABR encode have two meaning, the --pass 1 activated first pass ABR encode and the initial ABR encode without --pass 1 activated. Once the --pass 1 is not included in the first pass ABR encode, add the --pass 1 and the --pass 2 for the following second and third pass encode. Else you could add --pass 3 and --pass 2 for the following three pass encode.
Has anyone seen meaningful improvements from a third pass in x265? Since the 1st pass is a slow firstpass by default, I'd guess the gains would be less than in x264.

[WhatZit]

Has anyone seen meaningful improvements from a third pass in x265?

I toyed with this for several months, and was hard-pressed to identify any tangible benefit, even using naughty flick-screen frame grabs as comparison.

There was only ever the most microscopic detail improvements in low-contrast/high motion over a 2-pass, but these were easily homogenised out by the video processors in my TV's.

[benwaggoner]

I toyed with this for several months, and was hard-pressed to identify any tangible benefit, even using naughty flick-screen frame grabs as comparison.

There was only ever the most microscopic detail improvements in low-contrast/high motion over a 2-pass, but these were easily homogenised out by the video processors in my TV's.
I’ve only seen tangible improvements in 2-pass at low —bitrate —vbv-maxrate and -vbv-maxsize , OR when —vbv-maxrate is significantly higher than —bitrate

[RainyDog]

On the subject of multi-pass encodes, has anyone done much testing of the multi-pass-opt options --multi-pass-opt-analysis and --multi-pass-opt-distortion?

--multi-pass-opt-analysis, --no-multi-pass-opt-analysis

Enable/Disable multipass analysis refinement along with multipass ratecontrol. Based on the information stored in pass 1, in subsequent passes analysis data is refined and also redundant steps are skipped. In pass 1 analysis information like motion vector, depth, reference and prediction modes of the final best CTU partition is stored for each CTU. Multipass analysis refinement cannot be enabled when ‘analysis-save/analysis-load’ option is enabled and both will be disabled when enabled together. This feature requires ‘pmode/pme’ to be disabled and hence pmode/pme will be disabled when enabled at the same time.

--multi-pass-opt-distortion, --no-multi-pass-opt-distortion

Enable/Disable multipass refinement of qp based on distortion data along with multipass ratecontrol. In pass 1 distortion of best CTU partition is stored. CTUs with high distortion get lower(negative)qp offsets and vice-versa for low distortion CTUs in pass 2. This helps to improve the subjective quality. Multipass refinement of qp cannot be enabled when ‘analysis-save/analysis-load’ option is enabled and both will be disabled when enabled together. ‘multi-pass-opt-distortion’ requires ‘pmode/pme’ to be disabled and hence pmode/pme will be disabled when enabled along with it.

Seems that opt-analysis is more for speeding up subsequent passes by storing more data during the 1st pass for re-use. So... I think this might be more safely suited to multi-pass encodes where all passes use the exact (or almost) same settings?

And opt-distortion refines and re-balances QP distribution during subsequent passes based on info gathered in the 1st pass. Which sounds like a no brainer to use as long as it performs as it should... And perhaps even provides a genuine quality advantage to multi-pass encodes over CRF.

[alex1399]

The three pass encode is a refine of two pass encode where the original second pass encode is put at the end and an additional pass of slice-type / bit distribution encode inserted after the first pass. It is often to see that x265 places b-frame on some arbitrary location (not sub-optimal at all) even when the Quantization Parameter is not fluctuating everywhere. For the first pass encode, it get worse when x265 has to guess the target Rate Factor and the slice-type decision would be interfered by those Quantization Parameter fluctuation. The second pass in the two pass encode simply read the slice-type distribution from the first pass without any enhancement. The b-frame distribution is much consistent in the three pass encode. The three pass encode would place those b-frame like a specific case of two pass encode that utilize Constant Rate Factor encode for the first pass and Average Bit Rate encode for the second pass. That's why I prefer three pass encode.

For the --multi-pass-opt-analysis and the --multi-pass-opt-distortion, x265 prefers the Psycho-visual Rate Distortion measure to bias the Quantization Parameter in advance. I'm not a fan of those stuff. The compression efficiency based on non Psycho-visual metrics, for example the PSNR per bit, should still be a fundamental parts of encoder development. I'm aware that some encoder could trick on PSNR, but it doesn't sway my decision making.

[benwaggoner]

On the subject of multi-pass encodes, has anyone done much testing of the multi-pass-opt options --multi-pass-opt-analysis and --multi-pass-opt-distortion?





Seems that opt-analysis is more for speeding up subsequent passes by storing more data during the 1st pass for re-use. So... I think this might be more safely suited to multi-pass encodes where all passes use the exact (or almost) same settings?

And opt-distortion refines and re-balances QP distribution during subsequent passes based on info gathered in the 1st pass. Which sounds like a no brainer to use as long as it performs as it should... And perhaps even provides a genuine quality advantage to multi-pass encodes over CRF.
Yes, that's the one that should improve quality, and I imagine it should always be used in any second pass, particularly when doing CBR or having a constrained VBV. In the cases where I found 2-pass helpful, I was using --opt-distortion.

[RainyDog]

Ma & LigH,

Found the error in 2pass encoding:

Multipass analysis refinement along with multipass rate control
Multipass refinement of qp based on distortion data

If these two are deactivated everything is ok

Maybe the syntax has changed?

Found yesterday that this bug is still present in x265 2.8+66.

1st pass runs ok and creates the usual stats file plus a huge (11.5gb!) analysis file. Then crashes when it tries to run the 2nd pass if --multi-pass-opt-distortion has been set.

Are the developers aware of this? Thanks.

[RainyDog]

Here's my Staxrip log for the error when trying to start the 2nd pass when --multi-pass-opt-distortion is set.

"C:\Portable Programs\StaxRip\Apps\avs2pipemod\avs2pipemod64.exe" -y4mp "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976).avs" | "C:\Portable Programs\StaxRip\Apps\x265\x265.exe" --pass 1 --preset slow --profile main10 --output-depth 10 --rd 2 --psy-rd 2.0 --psy-rdoq 1.0 --subme 2 --aq-mode 3 --aq-strength 0.8 --qcomp 0.7 --bframes 6 --ipratio 1.2 --pbratio 1.1 --ctu 32 --no-cutree --merange 32 --qg-size 8 --no-rect --rc-lookahead 40 --lookahead-slices 0 --no-open-gop --early-skip --fast-intra --me dia --cbqpoffs -2 --crqpoffs -2 --deblock -2:-2 --no-sao --no-strong-intra-smoothing --no-constrained-intra --rdpenalty 1 --limit-tu 3 --tu-intra-depth 3 --tu-inter-depth 3 --multi-pass-opt-distortion --analysis-reuse-file "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976)_out.analysis" --bitrate 9000 --no-rect --no-cutree --weightb --no-open-gop --no-sao --no-strong-intra-smoothing --no-constrained-intra --frames 128857 --y4m --stats "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976).stats" --output NUL -

avs2pipemod[info]: writing 128857 frames of 24000/1001 fps, 1920x1040,
sar 0:0, YUV-420-planar-8bit progressive video.
y4m [info]: 1920x1040 fps 24000/1001 i420p8 unknown frame count
raw [info]: output file: NUL
x265 [info]: HEVC encoder version 2.8+66-88ee12651e30
x265 [info]: build info [Windows][MSVC 1915][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main 10 profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 3 / wpp(33 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: ME / range / subpel / merge : dia / 32 / 2 / 3
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 40 / 5.00
x265 [info]: Cb/Cr QP Offset : -2 / -2
x265 [info]: Intra 32x32 TU penalty type : 1
x265 [info]: Lookahead / bframes / badapt : 40 / 6 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 4 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 3 / 0.8 / 8 / 0
x265 [info]: Rate Control / qCompress : ABR-9000 kbps / 0.70
x265 [info]: tools: limit-modes rd=2 psy-rd=2.00 rdoq=2 psy-rdoq=1.00
x265 [info]: tools: early-skip rskip limit-tu=3 signhide tmvp fast-intra
x265 [info]: tools: deblock(tC=-2:B=-2) stats-write
avs2pipemod[info]: finished, wrote 128857 frames [100%].
avs2pipemod[info]: total elapsed time is 20045.940 sec.
x265 [info]: frame I: 1114, Avg QP:25.00 kb/s: 16601.32
x265 [info]: frame P: 23173, Avg QP:25.86 kb/s: 10044.17
x265 [info]: frame B: 104570, Avg QP:26.85 kb/s: 8597.98
x265 [info]: Weighted P-Frames: Y:8.1% UV:7.0%
x265 [info]: Weighted B-Frames: Y:8.3% UV:6.6%
x265 [info]: consecutive B-frames: 7.7% 1.6% 4.1% 6.5% 13.2% 53.3% 13.7%
encoded 128857 frames in 20047.53s (6.43 fps), 8927.25 kb/s, Avg QP:26.65

Start: 21:13:51
End: 02:47:59
Duration: 05:34:07

---------- Error Video encoding second pass using x265 2.6+31 ----------

Video encoding second pass using x265 2.6+31 failed with exit code: -1073741819 (0xC0000005)

The exit code might be a system error code: The instruction at 0xp referenced memory at 0xp. The memory could not be s.


------------- Video encoding second pass using x265 2.6+31 -------------

"C:\Portable Programs\StaxRip\Apps\avs2pipemod\avs2pipemod64.exe" -y4mp "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976).avs" | "C:\Portable Programs\StaxRip\Apps\x265\x265.exe" --pass 2 --preset slow --profile main10 --output-depth 10 --rd 4 --psy-rd 2.0 --psy-rdoq 1.0 --subme 4 --aq-mode 3 --aq-strength 0.8 --qcomp 0.7 --bframes 6 --ipratio 1.2 --pbratio 1.1 --ctu 32 --weightb --no-cutree --merange 32 --qg-size 8 --no-rect --rc-lookahead 40 --lookahead-slices 0 --no-open-gop --cbqpoffs -2 --crqpoffs -2 --deblock -2:-2 --no-sao --no-strong-intra-smoothing --no-constrained-intra --rdpenalty 1 --limit-tu 3 --tu-intra-depth 3 --tu-inter-depth 3 --multi-pass-opt-distortion --analysis-reuse-file "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976)_out.analysis" --bitrate 9000 --no-rect --no-cutree --no-open-gop --no-sao --no-strong-intra-smoothing --no-constrained-intra --frames 128857 --y4m --stats "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976).stats" --output "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976)_out.hevc" -

avs2pipemod[info]: writing 128857 frames of 24000/1001 fps, 1920x1040,
sar 0:0, YUV-420-planar-8bit progressive video.
y4m [info]: 1920x1040 fps 24000/1001 i420p8 unknown frame count
raw [info]: output file: C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976)_out.hevc
x265 [info]: HEVC encoder version 2.8+66-88ee12651e30
x265 [info]: build info [Windows][MSVC 1915][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main 10 profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 3 / wpp(33 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: ME / range / subpel / merge : star / 32 / 4 / 3
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 40 / 5.00
x265 [info]: Cb/Cr QP Offset : -2 / -2
x265 [info]: Intra 32x32 TU penalty type : 1
x265 [info]: Lookahead / bframes / badapt : 40 / 6 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 4 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 3 / 0.8 / 8 / 0
x265 [info]: Rate Control / qCompress : ABR-9000 kbps / 0.70
x265 [info]: tools: limit-modes rd=4 psy-rd=2.00 rdoq=2 psy-rdoq=1.00 rskip
x265 [info]: tools: limit-tu=3 signhide tmvp deblock(tC=-2:B=-2) stats-read
avs2pipemod[info]: finished, wrote 55 frames [0%].
avs2pipemod[info]: total elapsed time is 7.488 sec.
avs2pipemod[error]: only wrote 55 of 128857 frames.

Ignore the parts about using x265 2.6+31, this is just the version of x265 that was bundled with the last official release of Staxrip. I'm using 2.8+66.

[Ma]

Thanks, now I'm able to reproduce the problem.
My first observation is:
8bit+10bit+12bit versions of x265 hangs/not working in second pass
10bit or 10bit+8bit versions of x265 works OK

Under investigation...

[Ma]

The bug is in commit 2.8+48 https://bitbucket.org/multicoreware/x265/commits/b0d31e2bc0dfd79abfc87a7bf366b1fbeb421b2c

Now it is only one function to allocate analysis_data in multilib x265 8bit+10bit+12bit -- it is from 8bit part and sse_t type is uint32_t in this function. For 10bit and 12bit sse_t type is uint64_t, so this function allocate only half required memory for distortionData if you use multilib x265 for encoding 10bit or 12bit.

Patch proposition:
diff -r 88ee12651e30 source/encoder/api.cpp
--- a/source/encoder/api.cpp Thu Aug 16 18:27:01 2018 +0530
+++ b/source/encoder/api.cpp Sat Sep 08 06:45:23 2018 +0200
@@ -411,13 +411,18 @@
bool isVbv = param->rc.vbvMaxBitrate > 0 && param->rc.vbvBufferSize > 0;
int numDir = 2; //irrespective of P or B slices set direction as 2
uint32_t numPlanes = param->internalCsp == X265_CSP_I400 ? 1 : 3;
+#if X265_DEPTH < 10
+ int numCUs_sse_t = param->internalBitDepth > 8 ? analysis->numCUsInFrame * 2 : analysis->numCUsInFrame;
+#else
+ int numCUs_sse_t = param->internalBitDepth > 8 ? analysis->numCUsInFrame : (analysis->numCUsInFrame + 1) / 2;
+#endif

//Allocate memory for distortionData pointer
CHECKED_MALLOC_ZERO(distortionData, x265_analysis_distortion_data, 1);
- CHECKED_MALLOC_ZERO(distortionData->distortion, sse_t, analysis->numPartitions * analysis->numCUsInFrame);
+ CHECKED_MALLOC_ZERO(distortionData->distortion, sse_t, analysis->numPartitions * numCUs_sse_t);
if (param->rc.bStatRead)
{
- CHECKED_MALLOC_ZERO(distortionData->ctuDistortion, sse_t, analysis->numCUsInFrame);
+ CHECKED_MALLOC_ZERO(distortionData->ctuDistortion, sse_t, numCUs_sse_t);
CHECKED_MALLOC_ZERO(distortionData->scaledDistortion, double, analysis->numCUsInFrame);
CHECKED_MALLOC_ZERO(distortionData->offset, double, analysis->numCUsInFrame);
CHECKED_MALLOC_ZERO(distortionData->threshold, double, analysis->numCUsInFrame);

Test x265 Windows binary (multilib exe + patch) www.msystem.waw.pl/x265/x265-2.8+66-patched.7z

[K.i.N.G]

I'm encoding an 4:2:2 1920x1080p 10bit video with HDR and i'm getting this error/warning:

x265 [error]: Recommended Settings for HDR: colour primaries should be BT.2020,
transfer characteristics should be SMPTE ST.2084,
matrix coeffs should be BT.2020,
the input video should be 10 bit 4:2:0
Disabling offset tuning for HDR videos

But the encoding just continues... is it safe to ignore this or should i convert the video to 4:2:0 (i'd rather not but if i really have to...)?

[SeeMoreDigital]

I'm encoding an 4:2:2 1920x1080p 10bit video with HDR and i'm getting this error/warning....

But the encoding just continues... is it safe to ignore this or should i convert the video to 4:2:0 (i'd rather not but if i really have to...)?4:2:0 is still the chroma sub-sampling standard for all commercially released video. No domestic hardware players supports 4:2:2.

[K.i.N.G]

4:2:0 is still the chroma sub-sampling standard for all commercially released video. No domestic hardware players supports 4:2:2.

I'm pretty sure my nvidia shield tv plays it just fine. And I'd be surprised if even the internal player of my tv couldn't handle 4:2:2 to be honest, but I cant test it right now (not at home for the whole weekend).

My concern is that I'm not sure what "disabling offset tuning for HDR videos" exactly means... Will the resulting encode still be HDR?

[SeeMoreDigital]

...And I'd be surprised if even the internal player of my tv couldn't handle 4:2:2 to be honest...None of my smart TV's can. And my OPPO can't recognise such video streams placed within any supporting container!

[RieGo]

i agree. you won't be happy with a 4:2:2 video file. i encoded one file in 422 by accident. none of my devices could play it. plex also couldn't read it iirc. only pc could handle the file.
IMHO best thing to do would be convert your file to 4:2:0 before encoding. (or maybe it's just a problem with your source filter) you won't see a big difference and x265 won't have any problems with hdr metadata

[mini-moose]

Odd question maybe...
Do switches like --max-cll or --master-display have any effect on the actual video encode or are they more like flags for your HDR device?

[sneaker_ger]

flags

[katzenjoghurt]

Hi guys,

I'm currently trying to convert Star Wars Episode III to x265... and hell... it's a beast.
This movie gives me super ugly artifacts in dark scenes.

Original: https://i.imgur.com/PA9cWuR.png
Encoding: https://i.imgur.com/HpVbyf4.png (with meagre 2,300kb/s in that scene, 4,500kb/s on avg for the whole movie)

My settings were: --crf 23 --tune grain --profile main12 --output-depth 12 --rskip --qcomp 0.8 --no-open-gop --no-deblock --no-strong-intra-smoothing

I already tried aq-mode 3 but to retain an okayish quality I had to set the aq-strength to 1.5 and ended up with 12 mbit/s for this scene.
I then tried aq-strength 3 and ended up with with perfect quality... and wopping 43 mbit /s.

Why are dark scenes so much more costly than brighter scenes?
This does not feel right...
What can I do?

[sneaker_ger]

I already tried aq-mode 3 but to retain an okayish quality I had to set the aq-strength to 1.5 and ended up with 12 mbit/s for this scene.
Why did you not lower crf?

[Selur]

Also:
Boosting the brightness and looking at the source sometimes shows lots of compression artifacts, so some smoothing/denoising of the dark areas might help to understand where the bit rate goes.
Dark areas often contain lots of noise and artifacts which the encoder interprets as details that should be preserved.

[benwaggoner]

Why did you not lower crf?
Yeah, I recommend doing 2-pass VBR encoding when trying to figure out quality/bitrate tradeoffs, and then figuring out the optimal CRF from there.

[Forteen88]

@katzenjoghurt. Maybe you should set x265: --zones
and increase/decrease bitrate where you want.

[LigH]

Intentionally, aq-mode 3 (https://x265.readthedocs.io/en/default/cli.html#cmdoption-aq-mode) is supposed to help with dark scenes, even without zones; but it needs a matching aq-strength too, and sufficient bitrate, and possibly other circumstances — and still can't work wonders...

[Forteen88]

My settings were: --crf 23 --tune grain --profile main12 --output-depth 12 --rskip --qcomp 0.8 --no-open-gop --no-deblock --no-strong-intra-smoothingAre you aware of that GPU-decoding (certain newer GPUs) doesn't support 12-bits x265-video, but maximum 10-bits x265-video?

[LigH]

And if you have source material with only 8 bit precision per color component, an internal precision of 12 bit per quantization coefficient would be a waste anyway.

[brumsky]

Hi guys,

I'm currently trying to convert Star Wars Episode III to x265... and hell... it's a beast.
This movie gives me super ugly artifacts in dark scenes.

Original: https://i.imgur.com/PA9cWuR.png
Encoding: https://i.imgur.com/HpVbyf4.png (with meagre 2,300kb/s in that scene, 4,500kb/s on avg for the whole movie)

My settings were: --crf 23 --tune grain --profile main12 --output-depth 12 --rskip --qcomp 0.8 --no-open-gop --no-deblock --no-strong-intra-smoothing

I already tried aq-mode 3 but to retain an okayish quality I had to set the aq-strength to 1.5 and ended up with 12 mbit/s for this scene.
I then tried aq-strength 3 and ended up with with perfect quality... and wopping 43 mbit /s.

Why are dark scenes so much more costly than brighter scenes?
This does not feel right...
What can I do?

To me it looks like your PSY values are set very high. That could be an effect of tune grain though. Have you tried it without tune grain? Maybe take tune grain out and try playing with the PSY values. I've found aq-mode 3 to work well but it needs a bit high bit rate to help it. When I use aq-mode 3 I try to set the strength setting between .85-1, then up the bit rate as needed to reach my target quality.

[katzenjoghurt]

Hey brumsky et al.!

Yes... psy_rd was set to 4.
I kept grain mode but reduced psy_rd to 1 and set crf to 21 now and the artifacts are gone (while the file size got a bit smaller) ... the result is a bit smeared though.
Also it kills most of the grain.
https://i.imgur.com/eqQJXy7.png

Lowering Crf or Psy further won't do any good though... quite the contrary. At least on that example scene.

[katzenjoghurt]

Why did you not lower crf?

I didn't know that this is good practice.
Tried it out but didn't find any sweet spot yet though. :-/

Are you aware of that GPU-decoding (certain newer GPUs) doesn't support 12-bits x265-video, but maximum 10-bits x265-video?

Just something I tried out... 12-bit seemed to have helped in this thread (https://forum.doom9.org/showthread.php?t=173870). But no. Didn't work out for me.

[excellentswordfight]

I didn't know that this is good practice.
Tried it out but didn't find any sweet spot yet though. :-/



Just something I tried out... 12-bit seemed to have helped in this thread (https://forum.doom9.org/showthread.php?t=173870). But no. Didn't work out for me.
Tune grain usually gives pretty bad results, I found it to be usefull in very few instances.

I would start over with main10, keep --no-strong-intra-smoothing and --no-sao, try a slower preset and a lower CRF value. If that doesnt give you good results start playing with AQ and qcomp

[Forteen88]

My settings were: --crf 23 --tune grain --profile main12 --output-depth 12 --rskip --qcomp 0.8 --no-open-gop --no-deblock --no-strong-intra-smoothingTry setting --psy-rdoq 0.0 (default for --tune grain is --psy-rdoq 10.0) or set it to around 2.0.
I never liked that corresponding PSY-setting in x264 either, it was too strong.

[katzenjoghurt]

Try setting --psy-rdoq 0.0 (default for --tune grain is --psy-rdoq 10.0) or set it to around 2.0.
I never liked that corresponding PSY-setting in x264 either, it was too strong.

huh! I set --psy-rdoq 0 and got the exact same output (same pic and bytesize).
I then checked with mediainfo and all my test videos so far were already encoded with --psy-rdoq 0.

[katzenjoghurt]

Tune grain usually gives pretty bad results, I found it to be usefull in very few instances.

I would start over with main10, keep --no-strong-intra-smoothing and --no-sao, try a slower preset and a lower CRF value. If that doesnt give you good results start playing with AQ and qcomp

I will try tomorrow or during the weekend. As with any proposal that seems to be a bit more time consuming.

I'm quite happy with Tune Grain... I use it for every movie.
E.g. every Star Wars movie came out well with it so far... only Star Wars III seems like it was specifically made to make x265 look bad - pick your poison:
a) no grain b) 15k+ bitrate c) wobbling faces with artefacts

[Forteen88]

huh! I set --psy-rdoq 0 and got the exact same output (same pic and bytesize).
I then checked with mediainfo and all my test videos so far were already encoded with --psy-rdoq 0.Oh, weird, I got that info about --tune grain from,
https://x265.readthedocs.io/en/default/presets.html
That webpage says that --tune grain sets --psy-rdoq 10.0

Did you set --psy-rdoq 0 AFTER setting --tune grain? I think it overwrites the --psy-rdoq that --tune grain sets then.

[jd17]

He was most likely using a non-slow preset.
--psy-rdoq 10.0 is only set starting at --preset slow and slower.

This is why --tune grain actually looks quite good on medium or fast preset.

@katzenjoghurt:
Simply try --preset slow --profile main10 --output-depth 10 --no-sao at a CRF between 17 and 20.
Maybe that helps with bitrate and quality. :)

[Forteen88]

My settings were: ... --no-deblockThat'll create blocks on big screens. I would not remove deblock, most people use minimum --deblock -3:-3
I've never seen an encode (including x264-encodes) with lower --deblock than --deblock -3:-3

[katzenjoghurt]

Hi guys,

just wanted to thank you for your support.
Unfortunately I'm too dead tired from work to do some test encodings today.
But thanks! Much appreciated.
Will try out some of your proposals during the weekend and let you know what worked out.

[jlpsvk]

That'll create blocks on big screens. I would not remove deblock, most people use minimum --deblock -3:-3
I've never seen an encode (including x264-encodes) with lower --deblock than --deblock -3:-3

why? i am using --no-deblock with CRF ano NO BLOCKING... depends on bitrate and with CRF, no blocking should be presented. :)

[Forteen88]

why? i am using --no-deblock with CRF ano NO BLOCKING... depends on bitrate and with CRF, no blocking should be presented. :)Yeah, that's true to a point (until the screen is VERY big) :) What is the inch of the big screen you're using for playing such encodes?
If you're setting --no-deblock on UHD-encodes, it's obviously also less of a difference compared to encodes at lower resolutions.

[Boulder]

In theory, is it better to use smaller (C)TU sizes to retain detail? I've been thinking about the rdpenalty parameter - earlier I've used --ctu 32 --max-tu-size 16 --tu-inter-depth 4 --tu-intra-depth 4 --limit-tu 3, but I've wondered if --max-tu-size 32 --rdpenalty 1 would allow some more flexibility for the encoder without sacrificing detail.

[benwaggoner]

Yeah, that's true to a point (until the screen is VERY big) :) What is the inch of the big screen you're using for playing such encodes?
If you're setting --no-deblock on UHD-encodes, it's obviously also less of a difference compared to encodes at lower resolutions.
Deblocking also improves quality by making prediction more efficient. Even if you can't see a sharp block boundary on a given screen, it being there makes that area a poor reference for a predicted block. Using deblocking can reduce residual substantially, and thus lower QPs overall.

[Midzuki]

x265.exe 2.8+68-fa57fa584898

https://forum.videohelp.com/threads/357754-[HEVC]-x265-EXE-mingw-builds?p=2529259#post2529259

[katzenjoghurt]

@katzenjoghurt:
Simply try --preset slow --profile main10 --output-depth 10 --no-sao at a CRF between 17 and 20.
Maybe that helps with bitrate and quality. :)

The quality is nice at crf 17 slow.
I tried a bigger test encode with crf 18 medium but looks like I'll get close to a 10mbit bitrate then.
But lowering the bitrate further will kill the grain on the other hand.
*sigh*

At least I can confirm that the artifacts indeed seem to be caused by tune grain. As soon as I enable it artifacts start showing up at bitrates <10mbit.


This movie is really weird. Looking at it my gut feeling is that it should be encodable with 5,000kbit/s.
But it demands far more... more than some 60's movie with grain everywhere.

[Barough]

x265 v2.8+70-33a782b23f2c (http://www.mediafire.com/file/ikq55der79o1fb9/) (32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[RainyDog]

Is someone able to explain what --hdr-opt does please?

I've started to try a couple of hdr encodes and have always used --cbqpoffs -2 and --crqpoffs -2 in my standard x265 settings. But wondered if setting --hdr-opt together with those might cause issues as they all seem to be chroma/luma QP optimizations...

Thanks.

[tuanden0]

Is someone able to explain what --hdr-opt does please?

I've started to try a couple of hdr encodes and have always used --cbqpoffs -2 and --crqpoffs -2 in my standard x265 settings. But wondered if setting --hdr-opt together with those might cause issues as they all seem to be chroma/luma QP optimizations...

Thanks.

https://x265.readthedocs.io/en/default/cli.html#cmdoption-hdr-opt :D

[benwaggoner]

Is someone able to explain what --hdr-opt does please?

I've started to try a couple of hdr encodes and have always used --cbqpoffs -2 and --crqpoffs -2 in my standard x265 settings. But wondered if setting --hdr-opt together with those might cause issues as they all seem to be chroma/luma QP optimizations...

Thanks.
Indeed, --hdr-opt does adjust QP in a way that probably will allow you to turn off or at least lower the c?cpoffs options. I've been surprised how much a little chroma offset can increase bitrate with HDR content, so using --hdr-opt instead improves overall quality/efficiency.

[RainyDog]

https://x265.readthedocs.io/en/default/cli.html#cmdoption-hdr-opt :D

Well, yes... That's why I asked what it actually does :D As that doesn't really explain.

--hdr-opt, --no-hdr-opt

Add luma and chroma offsets for HDR/WCG content. Input video should be 10 bit 4:2:0. Applicable for HDR content. It is recommended that AQ-mode be enabled along with this feature. Default disabled.

By 'add' luma and chroma offsets, is it also tweaking cbqpoffs and crqpoffs and, if so, how?

[Magik Mark]

May I ask which among the switches will enable faster encoding in 2 pass (default medium) with minimal or no degradation in picture quality? So far I have identified the ff:

1. --no slow first pass
2. --multi-pass-opt-analysis
3. --multi-pass-opt-distortion

I might be missing couple more things?

[alex1399]

There're two elephants in the room. Your frame-crushing machine or your expectation about the speed of x265 at medium preset or both of them.

[benwaggoner]

Well, yes... That's why I asked what it actually does :D As that doesn't really explain.







By 'add' luma and chroma offsets, is it also tweaking cbqpoffs and crqpoffs and, if so, how?


Yes, the -cp?offs ARE the chroma offsets.

I don’t recall the exact math; it was based on an IEEE paper a couple years back. But it is adaptive based on luma, and so is going to be better than just fixed offsets.

[benwaggoner]

May I ask which among the switches will enable faster encoding in 2 pass (default medium) with minimal or no degradation in picture quality? So far I have identified the ff:

1. --no slow first pass
2. --multi-pass-opt-analysis
3. --multi-pass-opt-distortion

I might be missing couple more things?


Those are what I know about.

When using —no-slow-firstpass you probably want to make sure to set the number of ref and b-frames to match the second pass. The first pass is less useful as a reference without those, which will hurt quality a bit, and might slow encoding.

That said, the multi-pass options and no-slow-firstpass might be incompatible in practice, since x265 can’t refine stuff that the first pass didn’t do. Or at least, some parameter fine-tuning might be required to make sure the 1st pass is a good reference.

[NikosD]

I have 2 recent Smart TVs (Samsung and Panasonic) and 3 blu-ray players (2 from Samsung and 1 from LG). The Samsung BD players are UHD models

None of the devices above support 10-bit H.264 decodingSorry for the OT.

I just got my budget Chinese tablet with MediaTek MT6797 (Xelio X20) inside - a 2016 SoC

It supports in hardware MPEG2, VC-1/ WMV3, VP8, VP9 (8bit/10bit), H.264 (8bit/10bit), H.265 (8bit/10bit)

Also, my Sony is an ATV2 with MediaTek MT5891 SoC inside and supports H.264 10bit too.

End of OT.

[Majorlag]

May I ask which among the switches will enable faster encoding in 2 pass (default medium) with minimal or no degradation in picture quality? So far I have identified the ff:

1. --no slow first pass
2. --multi-pass-opt-analysis
3. --multi-pass-opt-distortion

I might be missing couple more things?

--no slow first pass, I think misses the point of 2 pass encoding. 2pass means to spend that much more time on encoding to get that much more quality or to hit that target size. Otherwise you would just stick to 1pass encoding for speed.

I would keep options 2 and 3, and add --no-strong-intra-smoothing --no-sao --multi-pass-opt-rps for some speedups as well.

[RainyDog]

Yes, the -cp?offs ARE the chroma offsets.

I don’t recall the exact math; it was based on an IEEE paper a couple years back. But it is adaptive based on luma, and so is going to be better than just fixed offsets.

Ok, thanks again benwaggoner.

I'll leave --c*qpoffs at default for HDR ecndoes then and set --hdr-opt instead.

[Barough]

x265 v2.8+74-fd517ae68f93 (http://www.mediafire.com/file/23ssbmzdtdwxzx2/) (32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[benwaggoner]

Ok, thanks again benwaggoner.

I'll leave --c*qpoffs at default for HDR ecndoes then and set --hdr-opt instead.
And please report back your results!

[Boulder]

Apart from the required bitrate to encode a frame, is the only real difference between --rd 4 and --rd 6 just the amount of effort required by the encoder? So I could switch to using --rd 4 if I'm ready to accept that it will probably produce a bigger file but take much less time? The presets don't seem to change the setting at all from the default 3, but 4 is the one where the other settings related to RDO kick in.

[Barough]

x265 v2.9+1-169e76b6bbcc (https://www.mediafire.com/file/5yy1p714hc41iyv/) (32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/all

[LigH]

What? v2.9? Without any announcement yet? Is the mailing list broken?

[Barough]

What? v2.9? Without any announcement yet? Is the mailing list broken?

v2.9 have been out since yesterday.

[LigH]

x265 2.9+1-169e76b6bbcc (http://www.mediafire.com/file/mv79mzv7kdgx485/x265_2.9%2B1-169e76b6bbcc.7z) (MSYS2; MinGW32: GCC 7.3.0 / MinGW64: GCC 8.2.0)

surprise

[benwaggoner]

Apart from the required bitrate to encode a frame, is the only real difference between --rd 4 and --rd 6 just the amount of effort required by the encoder? So I could switch to using --rd 4 if I'm ready to accept that it will probably produce a bigger file but take much less time? The presets don't seem to change the setting at all from the default 3, but 4 is the one where the other settings related to RDO kick in.
That seems a reasonable premise to me, for CRF encoding where VBV limits aren’t a quality constraint.

I don’t know if it’d be THAT much less time, but it’ll help.

[katzenjoghurt]

Thanks for providing builds for v2.9, guys! :)

I just always wonder - how to tell which cpu capabilities a build is using (AVX? AVX2? None?).
As far as I can tell x265.exe -V is misleadingly saying "using CPU capabilities X Y Z" but in fact it is actually just listing MY CPU's capabilities.
Is that right?

[Forteen88]

Thanks for providing builds for v2.9, guys! :)

I just always wonder - how to tell which CPU capabilities a build is using (AVX? AVX2? None?).
As far as I can tell x265.exe -V is misleadingly saying "using CPU capabilities X Y Z" but in fact it is actually just listing MY CPU's capabilities.
Is that right?This page got links to different builds of x265, some explicitly supports AVX2,
All binaries do the same, so it is only about encoding speed. My recommendations are: for AVX2-CPU the fastest should be VS 2017 AVX2 version, for AVX-CPU – VS 2017 AVX version, for SSE4-CPU – VS 2017 none or GCC none version, for SSSE3-CPU – GCC SSSE3 version, for CPU without even SSSE3 – GCC none version. You can determine fastest version by comparing encoding time on the same short sample. http://www.msystem.waw.pl/x265/

[nevcairiel]

I just always wonder - how to tell which cpu capabilities a build is using (AVX? AVX2? None?).
As far as I can tell x265.exe -V is misleadingly saying "using CPU capabilities X Y Z" but in fact it is actually just listing MY CPU's capabilities.
Is that right?

x265 uses all those instruction sets by default using runtime selection. If you CPU support it, and x265 has code for it, it'll get used.

Some people try to get a tiny bit of extra performance by allowing the C/C++ compiler to use those instructions sets to optimize the code, however one should know that compilers are limited in what they can do, and the majority of the "hot" (ie. important) code is manually optimized and unaffected by the compilers choices.

In my experience, the differences between a generic build with just the ASM enabled that the x265 developers wrote, and a build that allows the compiler to additionally use AVX/AVX2 to optimize the remaining code is relatively small, probably low single digits percentages.

[hajj_3]

are we ever going to get a changelog for v2.9 then?

[birdie]

Version 2.9

Release date - 05/10/2018
New features

Support for chunked encoding
Option:`--chunk-start and --chunk-end` Frames preceding first frame of chunk in display order will be encoded, however, they will be discarded in the bitstream. Frames following last frame of the chunk in display order will be used in taking lookahead decisions, but, they will not be encoded. This feature can be enabled only in closed GOP structures. Default disabled.

Support for HDR10+ version 1 SEI messages.

Encoder enhancements

Create API function for allocating and freeing x265_analysis_data.
CEA 608/708 support: Read SEI messages from text file and encode it using userSEI message.

Bug fixes

Disable noise reduction when vbv is enabled.
Support minLuma and maxLuma values changed by the commandline.

[katzenjoghurt]

@Forteen88:
Thanks, man. That's where I'm usually grabbing my builds from. But no 2.9 yet.
But there are cool guys here too, providing builds. :)

@nevcairiel:
Moin nevcairiel,
thanks for the insights! Huh.
The x265.exe --V output made me think that there are some AVX2 optimizations in the project's code itself. That's why I'm asking.
But you're saying it's just about compiler optimizations. Hm.

[Forteen88]

@Forteen88:
Thanks, man. That's where I'm usually grabbing my builds from. But no 2.9 yet.Np. There are 2.9 builds there under "x265 binaries for Win64/32 — stable branch"!

[LigH]

@katzenjoghurt:

For the core functions of the x265 encoder (especially those which are most often called in tight loops), some of the following sentences may be true, depending on performance gain, development progress in different bit depths, etc.:

There is basic C/C++ code. It depends on the compiler options which instruction set is used. If your CPU supports it, x265 can use this code. If not, it will crash due to unsupported instructions.
There is hand-optimized assembler code with MMX/SSE2 optimization. If your CPU supports it, x265 can use this code. If not, x265 should use simpler code.
There is hand-optimized assembler code with SSSE3/SSE4 optimization. If your CPU supports it, x265 can use this code. If not, x265 should use simpler code.
There is hand-optimized assembler code with AVX optimization. If your CPU supports it, x265 can use this code. If not, x265 should use simpler code.
There is hand-optimized assembler code with AVX2 optimization. If your CPU supports it and you enable it explicitly, x265 can use this code.

Well, for any x86-64 CPU today, SSE2 should be the minimum sensible supported instruction set. But already there are small differences. I remember the Athlon 64 (AMD K8) family being a threshold of providing an SSE2 implementation which is considered relatively "fast", but despite supporting SSE3 in specs, x264 and x265 will refuse to use it.

Usually all these code variants are present in a binary of (lib)x265, except they are excluded during compilation (e.g. you may disable all assembler code paths; but why would you want that?).

[katzenjoghurt]

Np. There are 2.9 builds there under "x265 binaries for Win64/32 — stable branch"!

OMG! You are right.
I ignored the right table as I always pick my versions from the left side.
Thanks again!


Usually all these code variants are present in a binary of (lib)x265, except they are excluded during compilation (e.g. you may disable all assembler code paths; but why would you want that?).

Thanks, LigH!
I understand this as: Unless a build mentions something else all optimizations are enabled. (?)
I just was looking for some way to verify.
Some context: What led me to the question was a current version of StaxRip which contained an x265.exe and a "x265 AVX2" zip file - and I couldn't tell if the zip was just some oversight or if the "normal" x265.exe is a version without AVX2 optimiziations.
I will just replace it and be done with it... though... I was wondering how to make sure an unknown x265.exe is indeed the "right" version for my machine.

[LigH]

Just a hint:

x265.exe --no-asm --version
x265 : HEVC encoder version 2.9+1-169e76b6bbcc
x265 [info]: build info [Windows][GCC 8.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: none!

All assembler optimizations forbidden = only basic C/C++ code used.

But if all assembler optimizations are enabled (and all of them are usually linked in the encoder), it only means they are available in case your CPU supports them (which is detected at runtime). It doesn't mean all of them are used on every hardware. If you don't limit them with the --asm [i]mask parameter, x265 detects what your CPU supports while starting, and selects the code paths with the optimal speed supported by your specific CPU.

x265.exe --version
x265 [info]: HEVC encoder version 2.9+1-169e76b6bbcc
x265 [info]: build info [Windows][GCC 8.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT

This x265.exe contains code paths for time-critical assembler routines for MMX+SSE2, SSSE3+SSE4, AVX, and even AVX2. But it runs on an AMD Phenom-II, so it is limited to MMX+SSE2 by the CPU auto-detection.

The very same x265.exe can use AVX or even AVX2 code if you copy it onto a PC with a CPU that supports AVX or even AVX2 and run it there.

{EDIT}If your CPU even supports AVX512, and you insist in using AVX512 instructions, then you need to enable it with an additional parameter --asm avx512 in your command line because it is a bit risky and does not always provide better performance, especially not when your CPU gets temperature throttled. And it will crash if your CPU does not support AVX512.{/EDIT}

So what are these special executables provided on a few sites? In addition to multiple code paths for time-critical assembler routines, also non-critical C/C++ routines get optimized for a modern instruction set, which limits their compatibility; these builds will not even start on older CPU's.

If I would use an x265.exe which was built with C/C++ compiler optimizations for AVX (that is probably what you read for special binaries), it would crash right at the start if run on an AMD Athlon/Phenom which doesn't support AVX, because it would use AVX instructions already for the initialization, already before the encoding even starts. But this is not a time-critical part. There is no serious need to speed up code which runs only once or a few times. – (Jedi mind powers) "This is not the build you are looking for."

Rather generic builds, like mine or Barough's or Midzuki's, are fine for a large range of PC's; builds for x86-64 are probably optimized at least for SSE2 in the code generated by the C/C++ compiler, which is the minimal widely supported instruction set of AMD64 compatible CPU's. And the selection of highly optimized assembler routines for the really time-critical parts is done in the encoder at runtime.

[benwaggoner]

If your CPU even supports AVX2, and you insist in using AVX2 instructions, then you need to enable it with an additional parameter --asm avx2 in your command line because it is a bit risky and does not always provide better performance, especially not when your CPU gets temperature throttled. And it will crash if your CPU does not support AVX2.
I thought that AVX2 would be used automatically, but AVX512 would only be activated via --asm (which appears to be undocumented in x265.readthedocs.io (https://x265.readthedocs.io/en/default/cli.html#performance-options))
Do I have that wrong?

AVX-512 is only useful with slower UHD resolutions, so it makes sense for it to require an opt in.

Rather generic builds, like mine or Barough's or Midzuki's, are fine for a large range of PC's; builds for x86-64 are probably optimized at least for SSE2 in the code generated by the C/C++ compiler, which is the minimal widely supported instruction set of AMD64 compatible CPU's. And the selection of highly optimized assembler routines for the really time-critical parts is done in the encoder at runtime.
Do we have any ballpark sense for how much platform-specific compilation can help encoding performance? I've heard some speculation about ~5% but that was a while ago before the current-gen AMD and Intel processors were out.

[LigH]

Oops, my mistake ... yes, AVX2 is automatic, only AVX512 is manual.

[Atak_Snajpera]

AVX-512 is only useful with slower UHD resolutions, so it makes sense for it to require an opt in.
I would like to see how useful is AVX-512 on 28 core xeon ;) I'm expecting negative speed-up ;)

[benwaggoner]

I would like to see how useful is AVX-512 on 28 core xeon ;) I'm expecting negative speed-up ;)
If you aren't doing Main10 UHD with a slower+ preset, you are almost certainly correct.

That said, an updated microarchitecture could potentially make AVX-512 be more generally useful. AVX2 became a lot more useful with (IIRC) Skylake's microarchitectural change which reduced thermal throttling doing AVX2, really improving throughput.

[qyot27]

That said, an updated microarchitecture could potentially make AVX-512 be more generally useful. AVX2 became a lot more useful with (IIRC) Skylake's microarchitectural change which reduced thermal throttling doing AVX2, really improving throughput.
So, expect a reasonably-mature AVX-512 ca. Sapphire Rapids?

[benwaggoner]

So, expect a reasonably-mature AVX-512 ca. Sapphire Rapids?
Plausibly. But x265 maybe the most CPU stressful real software in the world, hitting the cores, caches, and SIMD super hard at once. Hopefully Intel is benchmarking x265 during development!

It is hard to predict the optimal performance tuning of a given CPU without actually having it, as theoretical improvements don’t always work as expected.

I’m curious if anyone has benchmarked performance improvements from arch-specific and profile-driven builds.

X265 is also pretty stressful for compilers too.

[alex1399]

The x265 reported an error that the file "F:\x265" is not found(maybe it shows file could not open if I recall correctly) when I use --analysis-reuse-file F:\x265 during the second pass encoding.

Great, now I couldn't reproduce this error again.
It has been five days that zeranoe ffmpeg does not release a new version with x265 2.9.

[Atak_Snajpera]

Hopefully Intel is benchmarking x265 during development!
Nah... They will ask Principled Technology to do "proper" benchmarks ;) They are very good at disabling cores before testing...

[LigH]

Fresh build (https://www.mediafire.com/file/tz7w5f8qgenb28e/ffmpeg_N-92161-gf6d48b618a.7z/file) by media-autobuild_suite, GPL v3, Zeranoe-like selection.

ffmpeg version N-92161-gf6d48b618a Copyright (c) 2000-2018 the FFmpeg developers
built with gcc {7.3.0|8.2.0} (Rev3, Built by MSYS2 project)
configuration: --disable-autodetect --enable-amf --enable-bzlib --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-iconv --enable-lzma --enable-nvenc --enable-zlib --enable-sdl2 --disable-debug --enable-ffnvcodec --enable-nvdec --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-fontconfig --enable-libass --enable-libbluray --enable-libfreetype --enable-libmfx --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libwavpack --enable-libwebp --enable-libxml2 --enable-libzimg --enable-libshine --enable-gpl --enable-avisynth --enable-libxvid --enable-libaom --enable-version3 --enable-mbedtls --extra-cflags=-DLIBTWOLAME_STATIC --extra-libs=-lstdc++ --extra-cflags=-DLIBXML_STATIC --extra-libs=-liconv
libavutil 56. 19.101 / 56. 19.101
libavcodec 58. 33.100 / 58. 33.100
libavformat 58. 18.104 / 58. 18.104
libavdevice 58. 4.105 / 58. 4.105
libavfilter 7. 33.101 / 7. 33.101
libswscale 5. 2.100 / 5. 2.100
libswresample 3. 2.100 / 3. 2.100
libpostproc 55. 2.100 / 55. 2.100

[Forteen88]

x265 2.9+2 released now!
http://www.msystem.waw.pl/x265/

[LigH]

^ ffmpeg contains v2.9+2.

[DotJun]

Is there a downside to enabling avx512 on an intel X chip?


Sent from my iPhone using Tapatalk

[LigH]

As far as I remember from previous discussions...

Most of all: Temperature throttling. AVX512 can be a heavy burden.

Furthermore, switching the CPU into and out of AVX modes can be quite time consuming, which has to be considered in the optimization efforts, and it can make it less efficient for lower resolutions.

Please try to read back, and I believe a thread about AVX512 and AMD Ryzen got even separated from this generic x265 encoder thread.

[StvG]

Tested binaries download from here (http://msystem.waw.pl/x265/).
input.mkv - hevc (Main 10), yuv420p10le(tv), 3840x1606
AVX2 clock speed = AVX512 clock speed

ffmpeg -i input.mkv -f yuv4mpegpipe -strict -1 - | .\resources\x265-10b.exe --y4m - --ctu 32 -o .\OUTPUT.mkv

x265 [info]: HEVC encoder version 2.8+74-fd517ae68f93
x265 [info]: build info [Windows][GCC 8.2.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

encoded 498 frames in 51.34s (9.70 fps), 5044.61 kb/s, Avg QP:31.42

x265 [info]: HEVC encoder version 2.8+74-fd517ae68f93
x265 [info]: build info [Windows][MSVC 1915][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

encoded 498 frames in 51.80s (9.61 fps), 5044.61 kb/s, Avg QP:31.42
ffmpeg -i input.mkv -f yuv4mpegpipe -strict -1 - | .\resources\x265-10b.exe --y4m - --ctu 32 -o .\OUTPUT.mkv

x265 [info]: HEVC encoder version 2.9+2-7e978ed93d60
x265 [info]: build info [Windows][GCC 8.2.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

encoded 498 frames in 52.82s (9.43 fps), 5044.61 kb/s, Avg QP:31.42

x265 [info]: HEVC encoder version 2.9+2-7e978ed93d60
x265 [info]: build info [Windows][MSVC 1915][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

encoded 498 frames in 51.55s (9.66 fps), 5044.61 kb/s, Avg QP:31.42
ffmpeg -i input.mkv -f yuv4mpegpipe -strict -1 - | .\resources\x265-10b.exe --y4m - --ctu 32 -o .\OUTPUT.mkv

VS 2017 Generic compilation ("none")

encoded 498 frames in 51.49s (9.67 fps), 5044.61 kb/s, Avg QP:31.42

VS 2017 AVX2 compilation ("AVX2")

encoded 498 frames in 52.27s (9.53 fps), 5044.61 kb/s, Avg QP:31.42
ffmpeg -i input.mkv -f yuv4mpegpipe -strict -1 - | .\resources\x265-10b.exe --y4m - --ctu 32 (--asm avx512) -o .\OUTPUT.mkv

x265 [info]: HEVC encoder version 2.9+2-7e978ed93d60
x265 [info]: build info [Windows][MSVC 1915][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

encoded 498 frames in 52.05s (9.57 fps), 5044.61 kb/s, Avg QP:31.42

x265 [info]: HEVC encoder version 2.9+2-7e978ed93d60
x265 [info]: build info [Windows][MSVC 1915][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512

encoded 498 frames in 50.79s (9.80 fps), 5044.61 kb/s, Avg QP:31.42

[DotJun]

I tried a short test clip with avx512 enabled and disabled on a 4K source using the slower preset. FPS went up to 1.37 from 0.84 when I enabled 512.

Encoded clip looks good, no obvious errors that is. File size is roughly the same, but clip length and crf might have something to do with the tiny difference between the two.

64bit x265 on an intel 7820x. Temps are roughly equal to when 512 is disabled. Load is mostly at 100% on all cores with the occasional dip down to 87% every minute or so.

[LigH]

So it appears to be efficient on your specific CPU model.

[Atak_Snajpera]

I tried a short test clip with avx512 enabled and disabled on a 4K source using the slower preset. FPS went up to 1.37 from 0.84 when I enabled 512.

Encoded clip looks good, no obvious errors that is. File size is roughly the same, but clip length and crf might have something to do with the tiny difference between the two.

64bit x265 on an intel 7820x. Temps are roughly equal to when 512 is disabled. Load is mostly at 100% on all cores with the occasional dip down to 87% every minute or so.

You should encode whole movie (130k frames) instead of ultra short clip with few hundred of frames.
The longer you encode the more heat your cpu will produce and hence more aggressive AVX negative offset will be activated.

[Boulder]

Has anybody made any recent tests with different CTU and TU sizes? I made a quick test yesterday on a 720p encode, and max CTU (and TU) 16 turned out to produce the smallest file but also looked best compared to the original frame. At CTU max 64, the frame was clearly more blurry in places where there were more small details such as hair etc.

[RieGo]

I tried a short test clip with avx512 enabled and disabled on a 4K source using the slower preset. FPS went up to 1.37 from 0.84 when I enabled 512.

Encoded clip looks good, no obvious errors that is. File size is roughly the same, but clip length and crf might have something to do with the tiny difference between the two.

afaik avx512 enabled and disabled should produce exact same output. at least on my tests it did. can you share your command line?

[Clare]

Pushing Encoding Quality and Speed with x265 (https://youtu.be/7YT2KJwt4KI)
Massively Parallel Encoding (https://youtu.be/MHvpEVCvajk)

from Mile-High Video Workshop videos http://mile-high.video/files/mhv2018/

[benwaggoner]

x265 2.9+2 released now!
http://www.msystem.waw.pl/x265/
Anyone know what the new patch actually does:

rc: Fix rowStat computation in const-vbv (https://bitbucket.org/multicoreware/x265/commits/7e978ed93d6086973f87f607645339642ebb6ed0)

It looks like it might fix a serious issue in a given RC mode, but it isn't actually self-documenting.

[Jamaika]

Something seems to me that these aren't the only x265 bugs.
Github is still at 2.8. There will probably be some fixes to implement version 2.9.

[Boulder]

Has anybody made any recent tests with different CTU and TU sizes? I made a quick test yesterday on a 720p encode, and max CTU (and TU) 16 turned out to produce the smallest file but also looked best compared to the original frame. At CTU max 64, the frame was clearly more blurry in places where there were more small details such as hair etc.

Tests with 4K and 1080p encodes also showed the same behaviour. It's quite strange as it's said that the big thing in HEVC is that it can use larger CTUs than AVC to increase efficiency and that the bigger the CTU is, the less bitrate should be required. It doesn't seem to be like this at least in CRF mode. Too bad it's not allowed to use 16x16 CTUs with 1080p or 4K encodes if they are to be compliant.

[LigH]

Github is still at 2.8. There will probably be some fixes to implement version 2.9.

Bitbucket recently provided tag v2.9; but the "tip" is currently in the "stable" branch, not in "default".

Still, there seems to be a lack of communication recently. I reported compiler warnings of GCC 8.x already 2 times, and nobody replied until today.

[SmilingWolf]

Still, there seems to be a lack of communication recently. I reported compiler warnings of GCC 8.x already 2 times, and nobody replied until today.

Don't get me wrong, I have nothing but the conspiracy theories in my head, BUT, seeing as MulticoreWare sells en/decoding products, I wouldn't be surpised to see some announcement around the turn of the year about their work on some of the new codecs, be it AV1 or VVC.
But that's probably just me wishfully hoping for a sane AV1 encoder with proper performance, multithreading, profiles and documentation...

[benwaggoner]

Tests with 4K and 1080p encodes also showed the same behaviour. It's quite strange as it's said that the big thing in HEVC is that it can use larger CTUs than AVC to increase efficiency and that the bigger the CTU is, the less bitrate should be required. It doesn't seem to be like this at least in CRF mode. Too bad it's not allowed to use 16x16 CTUs with 1080p or 4K encodes if they are to be compliant.
It would be helpful to see the command line used, or at least bitrate/CRF.

Larger CTUs are definitely helpful at lower bitrates. Comparing at high perceptual quality can bring in other subtler differences between modes.

I've even been able to see a slight improvement at sub-SD resolutions using CTU 64 at very low bitrates.

[Boulder]

Source was 4K, downsampled to 1080p. These are the basic parameters that I've set:
--input-depth 16
--dither
--profile main10
--min-keyint 5
--keyint 480
--merange 44
--splitrd-skip
--preset veryslow
--rc-lookahead 60
--deblock -2:-2
--no-strong-intra-smoothing
--no-sao
--qcomp 0.8
--aq-mode 3
--aq-strength 0.8
--ctu 64
--max-tu-size 32
--rdpenalty 1
--qg-size 16
--tu-inter-depth 4
--tu-intra-depth 4
--limit-tu 4
--limit-refs 3
--max-merge 2
--rd-refine
--ref 6
--bframes 10
--crf 19
I've then also tested ctu 32 with max-tu-size 32 or 16.

In the 1080p encode, the bitrate is as follows:

CTU 64 / TU 32 - 15078 kbps
CTU 32 / TU 32 - 14608 kbps
CTU 32 / TU 16 - 14472 kbps

Of these, CTU 32 / TU 32 resembles the original the most. It's interesting that setting TU 16 also causes distortion in the same areas as CTU 64 / TU 32. I checked areas like eyes, hair etc. which have easily some sort of distortion because there are many fine lines and things that can be compared quite easily.

I've just started testing what CRF value is visually enough for 1080p so the final bitrate will probably be lower than what I got from my tests. I'd estimate CRF 20-21 would be the final value.

[benwaggoner]

Source was 4K, downsampled to 1080p. These are the basic parameters that I've set...

I've then also tested ctu 32 with max-tu-size 32 or 16.

In the 1080p encode, the bitrate is as follows:

CTU 64 / TU 32 - 15078 kbps
CTU 32 / TU 32 - 14608 kbps
CTU 32 / TU 16 - 14472 kbps

Of these, CTU 32 / TU 32 resembles the original the most. It's interesting that setting TU 16 also causes distortion in the same areas as CTU 64 / TU 32. I checked areas like eyes, hair etc. which have easily some sort of distortion because there are many fine lines and things that can be compared quite easily.

I've just started testing what CRF value is visually enough for 1080p so the final bitrate will probably be lower than what I got from my tests. I'd estimate CRF 20-21 would be the final value.
Target CRF will also depend on other parameters, if you are still adjusting those.

Overall, that is a very idiosyncratic set of options. Nothing looks wrong (and I'd love to hear who you picked some of those!, but it's definitely outside of any combination of settings that MCW would be doing psychovisual optimization for.


I'm curious about why you chose these particular settings:

--merange 44
--splitrd-skip
--max-merge 2
--deblock -2:-2
--rdpenalty 1
--qg-size 16
--bframes 10

Is this for encoding anime or some other kind of synthetic or mixed synthetic/natural image encoding?

[Boulder]

My sources are just regular movies or TV series. I'll tune CRF as the last item once I've got all the rest in place.

--merange 44, basically lowering it from the default 57 which I understand is meant for 4K. For 720p, I've used 38 all the time. I think this is remains of littlepox's set of "tune film" parameters.
--splitrd-skip, in my old notes, I didn't find it cause any ill effects. Do you have any specific information why it's a "bad idea"?
--max-merge 2, values 3-4 tested and it caused blur. One of the things in x265 that is different from x264 - the slower presets don't mean similar quality at lower final bitrate
--deblock -2:-2, no need for intense deblocking according to my tests
--rdpenalty 1, trying to favour smaller blocks.
--qg-size 16, tested values from 64 to 8, 16 looked best (in terms of distortion in small details again) when compared frame-by-frame. Tested with a 720p encode, so I'll need to check that also with 1080p later.
--bframes 10, some video utilizes a lot of B-frames for some reason. Not a big slowdown so I've kept it at that all the time.

[benwaggoner]

My sources are just regular movies or TV series. I'll tune CRF as the last item once I've got all the rest in place.

--merange 44, basically lowering it from the default 57 which I understand is meant for 4K. For 720p, I've used 38 all the time. I think this is remains of littlepox's set of "tune film" parameters.
It's not THAT frame size dependent, but 44 is probably fine.
--splitrd-skip, in my old notes, I didn't find it cause any ill effects. Do you have any specific information why it's a "bad idea"?
No reason it would be a bad idea; I just haven't seen it used before. Generally parameters that have a reliable quality/speed tradeoff are in a preset. But it isn't listed as experimental...

Anyone from MultiCoreWare care to weigh in? What's the tradeoff? Is this something that should get added to the faster presets, or default to On but off in --preset placebo?
--max-merge 2, values 3-4 tested and it caused blur. One of the things in x265 that is different from x264 - the slower presets don't mean similar quality at lower final bitrate
Yeah, with the many more tools available in HEVC, the differences between presets are greater. Odd that it causes blur; this should be a speed/quality tradeoff. You should file an issue here with repro details: https://bitbucket.org/multicoreware/x265/issues?status=new&status=open
--deblock -2:-2, no need for intense deblocking according to my tests
Well, to improve compression efficiency. Did you see any issues with using 0:0?
--rdpenalty 1, trying to favour smaller blocks.
Have you seen any experimental validation of it helping in x265 2.4+? I found places where it was helpful in older versions, but not recently.
--qg-size 16, tested values from 64 to 8, 16 looked best (in terms of distortion in small details again) when compared frame-by-frame. Tested with a 720p encode, so I'll need to check that also with 1080p later.
I would expect it would look better, but at some cost to efficiency due to the signaling overhead. --opt-cu-delta-qp might help that some. What sorts of bitrates are you targeting/getting?
--bframes 10, some video utilizes a lot of B-frames for some reason. Not a big slowdown so I've kept it at that all the time.
Why 10 specifically? I'd use either 8 (most tested, as it's in the slower+ presets) or 16 (maximum allowed).

[Boulder]

I think the general problem is that the presets and tunings are really old. I'm quite sure things are very much different now compared to what they were when most of the parameters and settings were come up with. Also, we are still missing the most common tuning which is --tune film. I would very much like to see what the creators of the encoder think of proper settings as they should know the internal workings best.

I recall someone else also complaining about max-merge for smoothing things if the value is too big. I'll need to retest and file a bug report if it's still reproducable. It's been some time since I tested it.

Will also retest deblock 0:0. When I tested it earlier, I did find it smooth things slightly so I went one notch down from the standard -1:-1 of x264's --tune film.

--rdpenalty 1 is something I have not tested recently.

--qg-size 16 does cause the bitrate to jump compared to 32 or 64, but it's worth it in my opinion.
I had tested it with a 720p encode with quite a big filesize difference:
qg-size 64 : 3147,41 kbps
qg-size 32 : 3648,01 kbps
qg-size 16 : 3954,23 kbps
It is something I need to separately retest for 1080p. The bitrates fluctuate a lot, I'd say between 2.5 - 6 Mbps for 720p encodes. I have no specific target so I just use CRF 19.

10 B-frames because for some quite noisy sources, I noticed that 8 consecutive B-frames were being used 5-10% of the time. Setting ten usually meant that the longest sequence was used 1-2% of the time.

[jlpsvk]

Is there a downside to enabling avx512 on an intel X chip?


Sent from my iPhone using Tapatalk

Heat. :D

[benwaggoner]

Heat. :D
Would it be hotter? Or would it just be slower due to thermal throttling.

I wouldn't be surprised if Intel's next big microarchitecture revision makes AVX512 useful in cases where it isn't today. We saw the same thing with Skylake and AVX2.

[Wolfberry]

No reason it would be a bad idea, I just haven't seen it used before. Generally parameters that have a reliable quality/speed tradeoff are in a preset. But it isn't listed as experimental...
Anyone from multicoreware care to weigh in? What's the tradeoff? Is this something that should get added to the faster presets, or default to on but off in --preset placebo?


In fact, this skip is not a fast skip algorithm.
As the sum of split cost is larger than none split CU's best cost (both rdcost of sub-cu and none split CU are without split flag cost), which means splitting into 4 parts at this depth of cu is a worse case compared with none split CU. So that, the remain N * 1/4 parts of CU analysis is useless.

If I understood the patch comment in the mailing list correctly, it should speed up intra split cost calculation a little while possibly preserving identical output.

splitrd-skip sounds like a small speedup with no trade off, but still disabled by default (according to the doc)

[Atak_Snajpera]

Would it be hotter? Or would it just be slower due to thermal throttling.

I wouldn't be surprised if Intel's next big microarchitecture revision makes AVX512 useful in cases where it isn't today. We saw the same thing with Skylake and AVX2.

This will require 10nm process. 14nm++++++++++++ has reached its thermal limits. Upcoming 8C/16T CPUs already are very hot at 5GHz.

[RieGo]

Would it be hotter? Or would it just be slower due to thermal throttling.

in my personal experience avx512 isn't generating much more heat than avx2 - on x265
so as long as you are only using it on x265 and use a safe thermal throttling setting u should be fine imho :)

in case i am wrong and anyone else has different results please correct me :)

[Boulder]

I recall someone else also complaining about max-merge for smoothing things if the value is too big. I'll need to retest and file a bug report if it's still reproducable. It's been some time since I tested it.

Will also retest deblock 0:0. When I tested it earlier, I did find it smooth things slightly so I went one notch down from the standard -1:-1 of x264's --tune film.
I've done some testing on a 1080p encode, basically I've set CRF 21 and then tested one parameter at a time by comparing still frames. I know it's not the optimal way but it is very hard to notice the differences in motion. What I've tried to compare are areas which show distortion quite well, such as eyes and their surroundings, hair etc. I've also tried picking frames from fast motion and almost still scenes.

I'll post my results as soon as I finish the 720p tests. From what I can tell is that 1080p requires slightly different parameters, but of course it could be that things have changed so much under the hood that my set of parameters have been obsolete all the time :)

What I already found strange is that deblock doesn't really affect bitrate. For example deblock 6:6 ended up around the same size as deblock 0:0.

[Forteen88]

What I already found strange is that deblock doesn't really affect bitrate. For example deblock 6:6 ended up around the same size as deblock 0:0.But did the avg QP, or visual quality remain the same?

[Boulder]

But did the avg QP, or visual quality remain the same?

The average QP remained pretty much the same, differences were around 0.01-0.03 units between 1:1 - -3:-3. Based on my tests, the higher values did soften the image more at least in places I checked, so maybe the bits were allocated elsewhere in the image.

[benwaggoner]

in my personal experience avx512 isn't generating much more heat than avx2 - on x265
so as long as you are only using it on x265 and use a safe thermal throttling setting u should be fine imho :)
That’s what I’d expect; TDP is TDP, and thermal throttling should kick in regardless of where the heat is coming from.

Picojoules per pixels is the more relevant metric here. Heat produced is equal to power draw. Better coolers can get the heat away from the CPU power better of course, but watts to the CPU is going to be the same as watts of heat to dissipate.

[benwaggoner]

I'll post my results as soon as I finish the 720p tests. From what I can tell is that 1080p requires slightly different parameters, but of course it could be that things have changed so much under the hood that my set of parameters have been obsolete all the time :)
Pretty much any settings from before x264 2.4 are invalid now due to the new lambda tables. Lots of other things have improved, but that was probably the biggest change.

What I already found strange is that deblock doesn't really affect bitrate. For example deblock 6:6 ended up around the same size as deblock 0:0.
Lots of codecs’ internal decisions are done ignoring in-loop deblocking to improve speed. So that’s not terribly surprising. And if this is CRF, it is a delta from QP based on spatial complexity, so it’s not surprising that it would change.

It IS surprising that you wouldn’t see bitrate OR QP change, even with a significant loss of detail. Not that using 6:6 is something anyone is likely to do in practice. But reduced detail should result in lower bitrate and/or QP. With lowered detail (including grain/gain noise), prediction should be more efficient...

Maybe log your repro for this as an issue for MCW?

[benwaggoner]

The average QP remained pretty much the same, differences were around 0.01-0.03 units between 1:1 - -3:-3. Based on my tests, the higher values did soften the image more at least in places I checked, so maybe the bits were allocated elsewhere in the image.
Hmmm. You should look at it in motion then; maybe the improvements are there.

Playing back at 1/4 speed is generally okay to see temporal artifacts better while still keeping temporal coherence. You can still detect discontinuities between IDR/i/P/B/b that way.

Testing at a higher CRF, like maybe 28, can also make differences at lot more obvious. Working at a quality where most things look pretty good can make differences much harder to detect.

[nevcairiel]

Would it be hotter? Or would it just be slower due to thermal throttling.

I wouldn't be surprised if Intel's next big microarchitecture revision makes AVX512 useful in cases where it isn't today. We saw the same thing with Skylake and AVX2.

The real problem with AVX512 is the strong downclock on stock CPUs, and the high density of the AVX512 instructions, causing a high power draw in a small die area.

With overclockable CPUs, you can control the AVX512 offset and make it useful for x265, but on Xeons or the like, you probably don't get that level of control, and the downclock during AVX512 might offset the advantages it offers.

Additionally, x265 is a pretty "light" AVX512 load. If you change the offset for a lower downclock and then run a strong AVX512 load (like pure math, ie. FFTs, for a prolonged time), your system may become unstable, due to the extreme density of all the heat and power draw. So its really hard to balance.

[DotJun]

You should encode whole movie (130k frames) instead of ultra short clip with few hundred of frames.
The longer you encode the more heat your cpu will produce and hence more aggressive AVX negative offset will be activated.
I encoded a full length movie and temps did not go over 80c which I think is ok for my chip? I should have stated that this computer is in a climate controlled area set to 70F.

My chip is OC'd to 4.5ghz with a -4 offset so that it drops to 4.1 when using avx. I guess I should have tried to compare encoding speed with avx on and off instead of just avx-512 enabled and disabled. I have no throttling issues and the load is a pretty consistent 100% with the occasional 1 second dip down to 87% every 30 seconds or so.

What is the command to disable avx entirely?

afaik avx512 enabled and disabled should produce exact same output. at least on my tests it did. can you share your command line?

This is what I use for 4k source since there is no Film preset:
--preset slower --crf 17 --profile main10 --me 3 --subme 5 --psy-rd 1.5 --psy-rdoq 5.0 --rdoq-level 1 --qcomp 0.8 --deblock -1:-1 --no-sao --repeat-headers --hdr-opt --range limited --colorprim 9 --transfer 16 --colormatrix 9 --master-display "G(13250, 34500)B(7500, 3000)R(34000, 16000)WP(15635, 16450)L(10000000, 1)"

[Atak_Snajpera]

I encoded a full length movie and temps did not go over 80c which I think is ok for my chip? I should have stated that this computer is in a climate controlled area set to 70F.
It is funny how you in USA use two different scales for temperature ;)

What is the command to disable avx entirely?
Bad idea! AVX2 is very useful in x265.

[Ma]

What is the command to disable avx entirely?

--asm sse4

Main asm levels are:
--asm no
--asm sse2
--asm ssse3
--asm sse4
--asm avx2
--asm avx512

--asm avx512 works a bit different -- it only enable possibility to use AVX-512 in auto-detection of CPU capabilities. For hard use of AVX-512 code (without checking) please use
--asm avx,avx512

[excellentswordfight]

Not sure if there is anyone that works on x265 is active in this thread anymore, but I've seen some discussion regarding --CTU and --merange both in this thread in the past and recently in a few threads.

From what I’ve read here and from what I’ve experienced from my testing is that using a CU size of 64 is overkill for 1080p and bellow, I see both a speed increase and a multithread increase when lowering it together with merange with no apparent loss in compression. I also saw some posts way back in this thread that suggested that the default CTU size should be based on resolution, wasn’t this implemented for any specific reason or is it just that no one has committed a patch for it?

I also have a question for merange, it says in the docs that the value of 57 is based on CTU-size and search method, but the value is set to 57 for all presets even though CTU-size and search method vary. How come?

[Boulder]

I finished my short tests today, and these are the settings that I found to suit my requirements.

1080p:

--deblock -3:-3 --no-strong-intra-smoothing --merange 44 --no-sao --qcomp 0.75 --aq-mode 3 --aq-strength 0.8 --ctu 32
--max-tu-size 32 --qg-size 8 --tu-inter-depth 4 --tu-intra-depth 4 --limit-tu 4 --limit-refs 3 --max-merge 3 --rd-refine
--ref 6 --bframes 10 --crf 20.5

720p:
--deblock -3:-3 --no-strong-intra-smoothing --merange 38 --no-sao --qcomp 0.8 --aq-mode 3 --aq-strength 0.8 --ctu 16
--max-tu-size 16 --qg-size 8 --tu-inter-depth 3 --tu-intra-depth 3 --limit-tu 4 --limit-refs 3 --max-merge 3 --rd-refine
--ref 6 --bframes 10 --crf 19.5

Very few changes between the two, but the CTU/TU size was an obvious change based on the comparison I made. Upon playback, the video looks very good in both resolutions on my 65" LCD even when watched from 1 meter or so. As a test source, I used the first episode of Black Sails as it's generally considered a very high quality Blu-ray release.

I did try finding out differences between deblock 1:1 and -3:-3 in motion but couldn't tell. If someone had ABX'd me, I probably wouldn't have been able to say which one is which. Mind you, the scene I used contained quite a lot of motion and some fast cuts so maybe they were just not visible there. Anyway, I chose a low value as I think it will retain detail and sharpness better. I watch everything from about 3-3.5 meters anyway, so any small blocking won't be that visible.

Edit: both encodes using the preset "veryslow".

[Dclose]

Did you see any issues with using 0:0?

I did a 4k down to 1080 encode the other day, with an accidental 0:0 deblock setting. It looks good in a "slick," "photoshopped" way, but it definitely loses a lot of detail, at least at CRF 23.

That's too much deblocking for me. Though I do put db at 0:0 when increasing CRF and dropping resolution a lot.

[DotJun]

--asm sse4



Main asm levels are:

--asm no

--asm sse2

--asm ssse3

--asm sse4

--asm avx2

--asm avx512



--asm avx512 works a bit different -- it only enable possibility to use AVX-512 in auto-detection of CPU capabilities. For hard use of AVX-512 code (without checking) please use

--asm avx,avx512



Thanks, I’ll put up three test runs of default, sse4 and avx512 to see what changes there are in FPS and efficiency.

Will medium or fast preset be ok to use or it has to be slower or higher?
Will it be ok to use “asm sse4” or do I have to specify sse4.2 like my log file shows?

[Boulder]

--qg-size 16, tested values from 64 to 8, 16 looked best (in terms of distortion in small details again) when compared frame-by-frame. Tested with a 720p encode, so I'll need to check that also with 1080p later.
I would expect it would look better, but at some cost to efficiency due to the signaling overhead. --opt-cu-delta-qp might help that some.
I was wondering about this parameter. Does it actually shift bits inside the frame so that the distribution is closer to the average - so that some areas with fine detail could suffer at the cost of flat areas looking better? Or the other way around as flat areas could be compressed using a lower QP to make them look equally good?

[benwaggoner]

I was wondering about this parameter. Does it actually shift bits inside the frame so that the distribution is closer to the average - so that some areas with fine detail could suffer at the cost of flat areas looking better? Or the other way around as flat areas could be compressed using a lower QP to make them look equally good?



AFAIK it works by reducing signaling overhead without changing actual pixels. There are several bitstream options that do things like that. This is the one likely to have the most impact, as WP signaling can take place MANY times per frame. The others are per frame or per GOP.


Sent from my iPhone using Tapatalk

[DotJun]

I did a few more test runs with the same parameters I stated before using 50k frames test clip. Here are the results.

Normal Preset:
sse4 2.19fps
avx2 2.89fps
512 3.18fps

They all had the exact same kbps of 23463.

Slower Preset:
sse4 0.71fps
avx2 0.89fps
512 0.95fps

All of them ended up with the exact same kbps of 22868.

My previous test from the other day seems to be a failure since I didn't use correct the correct switches for --asm.

[Boulder]

AFAIK it works by reducing signaling overhead without changing actual pixels.

There is a quite clear difference in the frame, it was quite obvious in this flat area (zoomed in though). The filesize difference was about 5% (the optimized version was smaller).

Without the optimization:
http://thumbs2.imagebam.com/6f/13/25/6e88661006880184.jpg (http://www.imagebam.com/image/6e88661006880184)

Optimization enabled:
http://thumbs2.imagebam.com/9f/c3/8d/85be161006880194.jpg (http://www.imagebam.com/image/85be161006880194)

In higher detailed area, the optimized encode looked better. I think I need to find a scene with some still background like sky, and see which one looks better in motion. The banding in flat areas can be quite eye-catching once you notice it.

[benwaggoner]

There is a quite clear difference in the frame, it was quite obvious in this flat area (zoomed in though). The filesize difference was about 5% (the optimized version was smaller).

In higher detailed area, the optimized encode looked better. I think I need to find a scene with some still background like sky, and see which one looks better in motion. The banding in flat areas can be quite eye-catching once you notice it.
Wow, awesome info! Thanks! I will iterate on further.

Was there any measurable perf difference?

[Boulder]

Wow, awesome info! Thanks! I will iterate on further.

Was there any measurable perf difference?

I just ran a longer encode to measure, and the difference is quite high. 4.95 fps for the normal encode and 5.22 fps for the one with optimization enabled. The normal encode is about 8% bigger.

I also checked how the optimization works in normal playback, and my eyes didn't like the result. The flat areas suffered a bit too much, there was a short scene with a nice, slightly noisy but flat coloured background which was lit by some flickering candlelight. The normal encode was slightly better looking there, there was not as much swimming blocks effect as there was with the optimized version.

[benwaggoner]

I just ran a longer encode to measure, and the difference is quite high. 4.95 fps for the normal encode and 5.22 fps for the one with optimization enabled. The normal encode is about 8% bigger.
An 8% file size reduction for a 5.5% speed increase would be an incredible optimization. An optimization can that give 1% reduction for a 5% speed increase is a big deal.

I also checked how the optimization works in normal playback, and my eyes didn't like the result. The flat areas suffered a bit too much, there was a short scene with a nice, slightly noisy but flat coloured background which was lit by some flickering candlelight. The normal encode was slightly better looking there, there was not as much swimming blocks effect as there was with the optimized version.
...but only if that 8% reduction doesn't impact quality, alas.

It would be interesting to see what the difference was in a stream analyzer. Or just looking at the log-level 2 csv files.

[Boulder]

I ran the encodes again to produce the logs, if you want to have a look: https://drive.google.com/open?id=1EjBUkFQjPreh4NsiZJnsPbY8qyl6tFMA . The flat part scene I mentioned appears several times, for example frames 0-117 contain that one.

[Boulder]

This is probably a silly question, but here goes anyway: if I use --hdr-opt, do I need to feed the encoder with 10-bit data or is 16-bit data as good if the source is a standard UHD with HDR? I always process things in 16-bit domain and let the encoder dither down to 10 bits.

[iwod]

Is there a release note for v2.9?

[LigH]

I never saw any...

[LoRd_MuldeR]

Is there a release note for v2.9?

https://bitbucket.org/multicoreware/x265/src/f9681d731f2e56c2ca185cec10daece5939bee07/doc/reST/releasenotes.rst?at=stable&fileviewer=file-view-default

[benwaggoner]

This is probably a silly question, but here goes anyway: if I use --hdr-opt, do I need to feed the encoder with 10-bit data or is 16-bit data as good if the source is a standard UHD with HDR? I always process things in 16-bit domain and let the encoder dither down to 10 bits.
The actual x265 encoder instance is going to start encoding with 10-bit 4:2:0 pixels one way or another. It gets converted somewhere upstream, perhaps even in the x265 exe. But that’s a filter that runs before the actual codec itself.

If you’re changing bit depth in x265, remember to always use —dither.

[Wolfberry]

It is kind of weird that the v2.9 release notes is only available in the stable (https://x265.readthedocs.io/en/stable/releasenotes.html) version, but not default (https://x265.readthedocs.io/en/default/releasenotes.html) or even latest (https://x265.readthedocs.io/en/latest/releasenotes.html).

[alex1399]

It seems that even zeranoe ffmpeg stays at x265 2.8 for a long time.

By the way, I'm not a fan of hard-coded dithering. Dithering should be handled as part of creative editing before the encoding process or left to the rendering after the decoding process.

[FranceBB]

This is probably a silly question, but here goes anyway: if I use --hdr-opt, do I need to feed the encoder with 10-bit data or is 16-bit data as good if the source is a standard UHD with HDR? I always process things in 16-bit domain and let the encoder dither down to 10 bits.

I would let x265.exe do the dithering, 'cause other dithering options like the Floyd Steinberg error diffusion may have a nicer look, but they could increase the bitrate required by x265. The built in dithering filter in x265 is supposed to dither everything down to the target bit depth without introducing banding. Blocks and macro blocks dithered by x265 are more likely to be recognised during the motocompensation by x265 than the ones dithered using a third party dithering method, therefore compression should be better.

In a nutshell, let x265 do the dithering and always pipe to it the highest bit depth you have, unless you like a specific dithering method and you have enough bitrate.

[benwaggoner]

I would let x265.exe do the dithering, 'cause other dithering options like the Floyd Steinberg error diffusion may have a nicer look, but they could increase the bitrate required by x265. The built in dithering filter in x265 is supposed to dither everything down to the target bit depth without introducing banding. Blocks and macro blocks dithered by x265 are more likely to be recognised during the motocompensation by x265 than the ones dithered using a third party dithering method, therefore compression should be better.

In a nutshell, let x265 do the dithering and always pipe to it the highest bit depth you have, unless you like a specific dithering method and you have enough bitrate.
Note there are two dithering modes in x265. The basic one if you don’t specify anything, and the more advanced one if you use —dither.

[benwaggoner]

Say, does anyone have any data showing potential benefits of using some of the "beyond placebo" settings?

For example

--subme 6 or 7 instead of 5
--me sea instead of star
--bframes 16 instead of 8
--ref 6+ instead of 5
--tskip
--cu-lossless


I've fond some value in very low bitrates or unusual content from all but the higher subme and me, which I've not tried.
--me sea in particular cuts speed enormously.

Tskip and cu-lossless can help with anime, small text, screen captures, and other stuff with sharp detailed edges.

[RainyDog]

Say, does anyone have any data showing potential benefits of using some of the "beyond placebo" settings?

For example

--subme 6 or 7 instead of 5
--me sea instead of star
--bframes 16 instead of 8
--ref 6+ instead of 5
--tskip
--cu-lossless


I've fond some value in very low bitrates or unusual content from all but the higher subme and me, which I've not tried.
--me sea in particular cuts speed enormously.

Tskip and cu-lossless can help with anime, small text, screen captures, and other stuff with sharp detailed edges.

Is --subme tied to RDO in the way that it is in x264?

In x264, --subme 6 upwards have increased levels of RDO. But seems that RDO is its own separate thing in x265 with RD levels 1-6 and rd-refine being a separate switch too.

[benwaggoner]

Is --subme tied to RDO in the way that it is in x264?

In x264, --subme 6 upwards have increased levels of RDO. But seems that RDO is its own separate thing in x265 with RD levels 1-6 and rd-refine being a separate switch too.
Yeah, subme does have some impact on rate control, but --rd is where the action is at.

From x265.readthedocs.io

Amount of subpel refinement to perform. The higher the number the more subpel iterations and steps are performed. Default 2

At –subme values larger than 2, chroma residual cost is included in all subpel refinement steps and chroma residual is included in all motion estimation decisions (selecting the best reference picture in each list, and chosing between merge, uni-directional motion and bi-directional motion). The ‘slow’ preset is the first preset to enable the use of chroma residual.

[TomV]

Say, does anyone have any data showing potential benefits of using some of the "beyond placebo" settings?

For example

--subme 6 or 7 instead of 5
--me sea instead of star
--bframes 16 instead of 8
--ref 6+ instead of 5
--tskip
--cu-lossless


I've fond some value in very low bitrates or unusual content from all but the higher subme and me, which I've not tried.
--me sea in particular cuts speed enormously.

Tskip and cu-lossless can help with anime, small text, screen captures, and other stuff with sharp detailed edges.
When I was tuning x265's presets, I tried all of these options that go beyond placebo, to see what should be included in placebo. You and anyone else are welcome to try them again, but I found it was easy to massively increase encode times, but impossible to get any meaningful improvement in efficiency.

[LigH]

What idiom would you give a configuration "beyond placebo"? Possibly "insane", like LAME MP3...

[mandarinka]

What idiom would you give a configuration "beyond placebo"? Possibly "insane", like LAME MP3...
ultra placebo. Or very placebo since that would match with very slow.

[benwaggoner]

When I was tuning x265's presets, I tried all of these options that go beyond placebo, to see what should be included in placebo. You and anyone else are welcome to try them again, but I found it was easy to massively increase encode times, but impossible to get any meaningful improvement in efficiency.
Indeed. “—me full —cu-lossless” will tank speed unbelievably without even trivial quality improvements for typical content and use cases.

I have seen visible and measurable value from ref 6, bframes 16, and tskip at <150 Kbps. Of course, using a lot of MIPS/pixel isn’t nearly so painful at low frame sizes and bitrates.

[RainyDog]

Yeah, subme does have some impact on rate control, but --rd is where the action is at.

From x265.readthedocs.io

Well yeah, that's why I asked. As --subme in the x265 docs doesn't specifically mention RDO.

But in x264 it's :-

subme

Default: 6

Set the subpixel estimation complexity. Higher numbers are better. Levels 1-5 simply control the subpixel refinement strength. Level 6 enables RDO for mode decision, and level 8 enables RDO for motion vectors and intra prediction modes. RDO levels are significantly slower than the previous levels.

QPel SAD 1 iteration
QPel SATD 2 iterations
HPel on MB then QPel
Always QPel
Multi QPel + bime
RD on I/P frames
RD on all frames
RD refinement on I/P frames
RD refinement on all frames
QP-RD (requires --trellis=2, --aq-mode > 0)


So it seems to me that --subme in x265 is as per --subme 1-5 in x264.

And the RD stuff of x264 --subme 6-11 is branched off into it's own thing in x265 with selectable RDO levels, rd-refine as a separate option etc.

[LigH]

x265 2.9+4-471726d3a046 (https://www.mediafire.com/file/0w5wivvwhcdb1y7/x265_2.9%2B4-471726d3a046.7z)

fixes: rowStat computation in const-vbv; memory reset size in dynamic-refine; linking issue on non x86 platform

[Dclose]

Say, does anyone have any data showing potential benefits of using some of the "beyond placebo" settings?

For example
[LIST=1]
--subme 6 or 7 instead of 5
I don't know what data you're looking for besides "yes, it looks better." I wrote the following some months back in this thread:

1) imo, if you care about things that move, (and picture quality in general), you have to use sub-motion pixel subme 7. 5 is good, and is as low as I ever set that even on files I'm trying to finish fast, but 5 is easily visually inferior to 7 imo. 7 of course takes longer to encode though.

I'm usually around CRF 22-23 (with nearly all quality settings turned on). It probably has lesser effect at CRF 18.

[RainyDog]

For 2-pass encodes, I normally use a custom faster 1st pass command line which is the same as my slow 2nd pass just with RDO level and subme turned down to level 2, --me dia, --early-skip and --fast-intra.

But I've been testing using identical command lines for both passes and using --multi-pass-opt-analysis instead which speeds up the 2nd pass considerably to the point where a complete 2-pass encode is almost the same speed as my usual approach.

Which should technically yield the higher quality final result? Is there any potential harm to using --multi-pass-opt-analysis?

[Majorlag]

Which should technically yield the higher quality final result? Is there any potential harm to using --multi-pass-opt-analysis?
Don't forget to also include --multi-pass-opt-rps --multi-pass-opt-distortion to your command line as well.
I understand that if your NOT turning down --RDO, --me and other settings then it should produce better results since it will spend more time on those settings in first pass. The --mulit-pass options are great in reusing the values obtained in the first pass to increase the speed of the second pass.

[atrin]

Hi,

I have some sample tabels for lamda2 and I generated a table of lambda based on it. This is the address of my sample https://mailman.videolan.org/pipermail/x265-devel/2017-March/010936.html
The second table is not related with its formula (lambda2 = 0.038 * pow(0.234, QP))

is there any document or information that explains lambda and lambda2 tables and relations?

Many thanks

[Jamaika]

Problems with metric VMAF.

After many hours, I managed to adjust the items VMAF. The new addition even recalculates something.

Read input model (libsvm) at ./vmaf_rb_v0.6.2/vmaf_rb_v0.6.2.pkl.model ...
…
Initialize storage arrays...
Extract atom features...
frame: 0, adm: 0.986, adm_num: 792.386, adm_den: 803.249, adm_num_scale0: 102.293, adm_den_scale0: 105.547, adm_num_scale1: 148.311, adm_den_scale1: 151.745, adm_num_scale2: 230.474, adm_den_scale2: 232.816, adm_num_scale3: 311.307, adm_den_scale3: 313.141, motion: 0.000, motion2: 0.000, vif_num_scale0: 3201540.000, vif_den_scale0: 4265149.500, vif_num_scale1: 915769.438, vif_den_scale1: 973015.313, vif_num_scale2: 237178.438, vif_den_scale2: 244942.781, vif_num_scale3: 62793.887, vif_den_scale3: 64002.453, vif: 0.796,
Generate final features (including derived atom features)...
Normalize features, SVM regression, denormalize score, clip...
frame: 0, adm2: 0.986477, adm_scale0: 0.969174, adm_scale1: 0.977376, adm_scale2: 0.989942, adm_scale3: 0.994142, motion: 0.000000, vif_scale0: 0.750628, vif_scale1: 0.941167, vif_scale2: 0.968301, vif_scale3: 0.981117, vif: 0.796321, motion2: 0.000000,
Exec FPS: 2.742952
VMAF score (mean) = 100.000000
x265 [info]: frame I: 1, Avg QP:23.64 kb/s: 5696.20
x265 [info]: frame P: 2, Avg QP:29.09 kb/s: 874.10
x265 [info]: frame B: 7, Avg QP:35.33 kb/s: 204.03
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: Weighted B-Frames: Y:0.0% UV:0.0%
x265 [info]: consecutive B-frames: 33.3% 0.0% 0.0% 33.3% 33.3% 0.0% 0.0% 0.0% 0.0%

However, how to use it? So many ads on the forum.

static const x265_vmaf_commondata vcd_yuv420p[] = { { (char *)"yuv420p", (char *)"./vmaf_rb_v0.6.2/vmaf_rb_v0.6.2.pkl", (char *)"vmaf_yuv%04d.json", (char *)"json", 0, 1, 1, 0, 0, 0, 0, (char *)"mean", 0, 3, 1 } };

The first two items are obvious. They concern the color of subsampling and the version VMAF. Due to the fact that I chose version 0.6.2, the last item 'enable_conf_interval' must be included.
Then, the recording items metric VMAF in the files json or xml. These are the next two positions from the left. Here are the problems. First of all, I don't know why the program doesn't save all parameters in one file. Secondly, I can't force a program to save json/xml files one after another to the number of processed frames. (vmaf_yuv%04d)
{
"version":"1.3.7",
"params":{
"model":"",
"scaledWidth":1920,
"scaledHeight":1080,
"subsample":3
},
"metrics":[
"adm2",
"bagging",
"ci95_high",
"ci95_low",
"motion2",
"stddev",
"vif_scale0",
"vif_scale1",
"vif_scale2",
"vif_scale3",
"vmaf"
],
"frames":[
{
"frameNum":0,
"metrics":{
"adm2":0.98648,
"bagging":99.62585,
"ci95_high":100.0,
"ci95_low":98.12069,
"motion2":0.0,
"stddev":0.7394500000000001,
"vif_scale0":0.75063,
"vif_scale1":0.94117,
"vif_scale2":0.9683,
"vif_scale3":0.98112,
"vmaf":100.0
}
}
]
}
Next, what is the items 'disable_clip' and 'enable_transform' for?
The next four items phone_model, psnr, ssim, ms_ssim should be turned off.
Choice of data processing method
Choosing the number of cores. In my case, zero.
Problem with the color n_subsample parameter. For BPG, once there are three for YUV, once there should be one for the alpha color. The instruction is five.

Ok, I created x265 files with VMAF and without:
- The X265 VMAF codec doesn't work with FFmpeg.
av_interleaved_write_frame(): Broken pipe
No more output streams to write to, finishing.
Error writing trailer of pipe:: Broken pipe
ffmpeg.exe -loglevel verbose -i Untitled.mp4 -an -f yuv4mpegpipe -vf scale=1920:1080:in_color_matrix=bt709:in_range=limited:out_color_matrix=bt709:out_range=limited,format=yuv420p -strict -1 - |
x265_081012bit_hdr_vmaf.exe --y4m --input-csp i420 --input-depth 8 --output-depth 8 --preset veryslow --crf 28 --fps 25.000 --keyint 50 --info --no-open-gop
--colormatrix bt709 --colorprim bt709 --transfer bt709 --limit-ref 0 --range limited --recon 111.yuv --output 111.h265 -
- I don't know what is the 'recon' function for VMAF for?
In the description:
-r/--recon <filename> Reconstructed raw image YUV or Y4M output file name

- Strange, the x265 vmaf itself works, but it isn't known whether the codec should have an output file or not?
Assuming he has. This file does not differ in content from the recon file. In addition, these files don't differ from x265 files without VMAF. I don't have a concept for what it is and what is the recon file for?

[LigH]

The "recon" feature writes a YUV or Y4M raw video file that contains the reconstructed video which has been decoded right after encoding it, so you can compare the compression results with the original source (assuming it was a YUV or Y4M file too) without calling an additional decoder. It is available independently of VMAF functions linked into x265 – which may still be possible only under Linux, I believe; are you sure your Windows build contains any VMAF comparison code? The build script source\CMakeLists.txt contains the check clearly in a "if(UNIX)" block.

[Jamaika]

... which may still be possible only under Linux, I believe; are you sure your Windows build contains any VMAF comparison code? The build script source\CMakeLists.txt contains the check clearly in a "if(UNIX)" block.
I created a version for Windows 2.9+8. I almost doesn't change anything. Codec hasn't only 'threads' for VMAF as I wrote earlier.
https://www.sendspace.com/file/r90y0d
Probably it can also be created in MSVC.

[Barough]

x265 v2.9+8-27d8424c799d (http://www.mediafire.com/file/bs49cg9rcjik169/) (32 & 64-bit 8/10/12bit Multilib Windows Binaries)

https://bitbucket.org/multicoreware/x265/commits/branch/stable

[Ma]

(64-bit GCC 8.2.0 8+10+12bit multilib / ICC 19.0 8/10/12 cli+shared)

ICC binaries not working in my Win10 (missing dll's).

[Natty]

hi, i would like to know how qcomp works in simple language, and it's impact on bitrate when its lowered or increased from its default value :thanks:

[LigH]

--qcomp <float> (https://x265.readthedocs.io/en/default/cli.html?highlight=qcomp#cmdoption-qcomp)
qComp sets the quantizer curve compression factor. It weights the frame quantizer based on the complexity of residual (measured by lookahead). It’s value must be between 0.5 and 1.0. Default value is 0.6. Increasing it to 1.0 will effectively generate CQP.

The default value 0.6 is a balance between a constant quantizer (regardless of the video content) and the complexity of the video content (degree of details and amount of motion) providing chances to spare bitrate by increasing the quantizer slightly in scenes where it may be sufficient to preserve enough quality with little noticeable loss.

IIRC, if you could decrease it to 0.0, the encoder would try its best to keep a constant bitrate (CBR), which would cause a very varying amount of quality loss (I might be wrong here, for x265, though). Increasing it to 1.0 instead would cause a constant quantization which would not take advantage of the possible ways to spare bitrate in scenes where convenient quality preservation could already be achieved with less bitrate, at a coarser quantization than the target.

You may increase this value a little (e.g. towards 0.8) when you notice that there is too much loss of precision in areas with very little detail, e.g. darkness and smooth ramps, especially in cases when your target bitrate is rather low. On the other hand, there may be other (psycho-visual) options to let the encoder not spare too much bitrate.

[Ma]

@Ma Test version available. If any of these works, some benchmarks will be appreciated.

x265_MT.exe works, thanks! I will make some test with '--no-asm' option to compare only C++ compilers.

[Ma]

Test platform: Win10 64-bit home, i7 8700 + be quiet pure rock, 16 GB RAM DDR4 @ 3866
Command line (only 8-bit encoding):
x265 --no-asm --crf 20 ../Bosphorus_1920x1080_120fps_420_8bit_YUV.y4m w.hevc

Results in fps (encoding speed, mean value from 2 runs):
8.98 fps -- ICC AVX2
8.41 fps -- GCC 9.0 AVX2 ucrt
7.36 fps -- GCC 8.2 AVX2
7.34 fps -- GCC 8.2 AVX2 ucrt
7.12 fps -- GCC 7.3 AVX2 ucrt
7.11 fps -- GCC 6.5 AVX2 ucrt
6.30 fps -- GCC 5.5 AVX2 ucrt
5.93 fps -- GCC 8.2 generic Barough build
5.57 fps -- GCC 4.9.4 AVX2 ucrt
5.06 fps -- VS 2017 AVX2
4.89 fps -- VS 2015 AVX2
4.72 fps -- GCC 4.8.5 AVX2 ucrt

ucrt means Universal CRT (it is replacement for msvcrt.dll)
Results with asm was 29 up to 30 fps for all contenders (full results in screen.txt).

ICC 19 is clear winner, GCC 9 in second place. VS 2017/2015 without asm are really slow (but with asm are good/the best).

[poisondeathray]

Thanks Ma for those tests. Wow, that's a large % variation in speed

[FranceBB]

Test platform: Win10 64-bit home, i7 8700 + be quiet pure rock, 16 GB RAM DDR4 @ 3866
Command line (only 8-bit encoding):
x265 --no-asm --crf 20 ../Bosphorus_1920x1080_120fps_420_8bit_YUV.y4m w.hevc

Results in fps (encoding speed, mean value from 2 runs):
8.98 fps -- ICC AVX2
8.41 fps -- GCC 9.0 AVX2 ucrt
7.36 fps -- GCC 8.2 AVX2
7.34 fps -- GCC 8.2 AVX2 ucrt
7.12 fps -- GCC 7.3 AVX2 ucrt
7.11 fps -- GCC 6.5 AVX2 ucrt
6.30 fps -- GCC 5.5 AVX2 ucrt
5.93 fps -- GCC 8.2 generic Barough build
5.57 fps -- GCC 4.9.4 AVX2 ucrt
5.06 fps -- VS 2017 AVX2
4.89 fps -- VS 2015 AVX2
4.72 fps -- GCC 4.8.5 AVX2 ucrt
.

Very interesting.
I knew that Intel Parallel Studio (and its compiler) was good, but what surprises me is that GCC has become better and better.
Visual Studio used to be good for AVX2, while GCC used to be better for SSE2/SSSE3/SSE4.1, but perhaps things have changed and GCC now totally outperforms Visual Studio.

[nevcairiel]

Do keep in mind that these tests are absolutely disjunct from reality. Noone is going to run something like x265 without ASM, so for any real-world use these numbers are meaningless.

[FranceBB]

Do keep in mind that these tests are absolutely disjunct from reality. Noone is going to run something like x265 without ASM, so for any real-world use these numbers are meaningless.

Well, of course.
Still, in an ideal world, compilers would be able to produce optimized assembly code as fast as manually-written intrinsics, so there's no need to manually write them.

Unfortunately, that's still an utopia.

Anyway, for 10/12bit x265 on x86 32bit systems (for which there aren't manually written intrinsics available and builds rely on compiler optimization only), compilers "speed tests" are kinda useful. ^_^

[nevcairiel]

Still, in an ideal world, compilers would be able to produce optimized assembly code as fast as manually-written intrinsics, so there's no need to manually write them.

Unfortunately, that's still an utopia.


And it will always remain nothing but a dream. A compiler does not have enough information about the restrictions and requirements of the algorithm to perform the same sort of optimization a developer can do when manually writing ASM - especially with advanced SIMD.

PS:
Anyone that runs on a 32-bit system deserves what they get. Upgrade already, stop wasting developers and your own time. A simple change from 32-bit to 64-bit on the same hardware will yield a massive speedup already. And if you're encoding with x265 on hardware thats not even 64-bit compatible, then you should *really* upgrade.

[benwaggoner]

And it will always remain nothing but a dream. A compiler does not have enough information about the restrictions and requirements of the algorithm to perform the same sort of optimization a developer can do when manually writing ASM - especially with advanced SIMD.
Yeah, QFT++. Autovectorization has been a dream for decades, and it’s never gotten anywhere near what hand assembly can do. Same with autoparallelization. A good compiler and the right code can maybe get 2-3x faster. Intel’s whole Itanium debacle was premised on, and failed because of, very over optimistic assumptions about compilers being able to do this stuff.

Apple switching to Intel was such a boon to the industry because it eliminated the need for media apps to have to implement on both SSEx and AltiVec. Because you couldn’t even port between them; often the whole algorithm had to be refactored to get decent performance.

As it is, x265 is probably some of the most advanced and complex SIMD code on the planet, with a lot of complex threading to boot. It’s likely about the worse case for a speed gap between compiler-generated SIMD versus hand-coded SIMD.

Anyone that runs on a 32-bit system deserves what they get. Upgrade already, stop wasting developers and your own time. A simple change from 32-bit to 64-bit on the same hardware will yield a massive speedup already. And if you're encoding with x265 on hardware thats not even 64-bit compatible, then you should *really* upgrade.
Are people actually still doing this? I can’t imagine how slow x265 must be on pre x64 hardware. I don’t think I’ve had a machine NOT running 64-bit since Windows 7 launched, and everything I was running when Win 7 launched was already 64-bit capable.

The joules per pixel on a pre x264 system has to be a couple of orders of magnitude worse than the latest Intel and AMD processors deliver. And upgrade would pay for itself quickly in lowered elecctricity & cooling costs alone!

[qyot27]

Are people actually still doing this? I can’t imagine how slow x265 must be on pre x64 hardware. I don’t think I’ve had a machine NOT running 64-bit since Windows 7 launched, and everything I was running when Win 7 launched was already 64-bit capable.
It's been a long, long time since I ever tried, although I do keep a full set of 32-bit build instructions for all the pieces in my FFmpeg/mpv build guide. For posterity, mostly.

My guess would be that you could get decent encode times with x265 on a PIII only by encoding at most 480p under --profile ultrafast at 8bit (since the >8bit ASM has to be explicitly turned off to build for 32-bit). And possibly not even then, as I'm pretty sure we'd still be looking at maybe 5fps. At that point you're dealing with strictly academic 'because I can' types of things, and you'd absolutely get better framerates (at a preciser preset) by just using x264 instead.

The joules per pixel on a pre x264 system has to be a couple of orders of magnitude worse than the latest Intel and AMD processors deliver. And upgrade would pay for itself quickly in lowered elecctricity & cooling costs alone!
Exactly. My Coppermine system was only a main system until 2015, and it only held out that long because of not having an income up until then (it is still alive, though, but now serves as a file archive).

Inexpensive mini-PCs have really filled the gap here, and get vastly better performance than an ancient system like that would get. Even with the power draw restraints and lack of AVX 1 or 2, Bay Trail-T (and now Apollo Lake, since I inadvertently fried the other one) could run circles around the Coppermine while being dead silent because the power consumption is so low they don't even need a cooling fan. Plus access to better SIMD - up to SSE4.2, plus the AES stuff - and multithreading. Running 64-bit OSes can be a bit of a task - the Bay Trail-T era would normally only ship with 32-bit versions of Windows on tiny eMMC storage (and since they come with 32-bit UEFI, you can't use 64-bit Windows, although 64-bit Linux distros loaded from flash drives or external USB hard drives are an option), but by now 64-bit Windows installs seem common, along with allowing for putting secondary SSDs into the system.

I've done 4K->4K and 4K->1080p transcodes on the Apollo Lake at about 5fps (ultrafast, 10bit, preserving HDR, crf 18), and at least Apollo Lake has a 10bit HEVC decoder in the GPU. Had I known the exact way to get that enabled in mpv at the time, I wouldn't have even bothered trying to transcode. But it let me get working figures, and I suppose 1080p 10bit HEVC would be less of a burden on the GPU than 4K would, so there's that.

[aegisofrime]

x265 v2.9+8-27d8424c799d (https://drive.google.com/open?id=1xZQABtoaSFgGu11YstmHKYLzO3elemlC)
(64-bit GCC 8.2.0 8+10+12bit multilib / ICC 19.0 8/10/12 cli+shared)

Apologies, I'm only seeing a v2.9+9 build on that, and that's built with GCC 8.2. I can't find an ICC build in that Google Drive, or am I blind? :(

[K.i.N.G]

I have to glue several shots, which all have different resolution, together.
Ideally I'd like to keep the original resolution.
Is it possible to have variable resolution? (If so, is it UHD compliant/will it play nicely on modern UHD devices?)

[Selur]

I have to glue several shots, which all have different resolution, together.
...
Is it possible to have variable resolution?
iirc mkv added support for this a while back, but that has nothing to do with x265
(best ask Mosu to be sure)

If so, is it UHD compliant/will it play nicely on modern UHD devices?
No.

[K.i.N.G]

Ah, too bad... Thanks for the quick & clear reply!

[mandarinka]

iirc mkv added support for this a while back
I tried searching the changelog for resolution, variable and so on, but could not find anything. Do you recall which version was it roughly, or where can I find information about that? Support for variable resolution, or better said, variable display aspect ratio as well is something I tried to solve for years. The problem was that matroska stored such data in a single header entry valid for the whole file; back when designing it, nobody imagined files where there would be need to switch these parameters.

AFAIK the best possible solution was to use separate files and link them using the next/previous segment linking fucntionality.

[Selur]

@mandarinka: Like I wrote 'best ask Mosu to be sure' :)
I thought there was a way to combine file with multiple resolutions and the problem was to find a player supporting it, but Mosu most definitely can shine some light on it whether I just remember this wrong of if there is a way to create such files.

[benwaggoner]

@mandarinka: Like I wrote 'best ask Mosu to be sure' :)
I thought there was a way to combine file with multiple resolutions and the problem was to find a player supporting it, but Mosu most definitely can shine some light on it whether I just remember this wrong of if there is a way to create such files.
The only software player I know of that can handle arbitrary resolution changes while maintaining constant displayed image size and aspect ratio is Windows with a MFT from Expression Encoder installed.

That at least works with variable-sized VC-1; I've never tried it with H.264. If a player is attending to SAR, doing a full-screen playback should Just Work. But it seems players rarely account for the SAR changing each GOP even if they can handle different resolutions.

This does work in most players on living room devices, though, since it's common for SAR to change (e.g. 720x480 to 1280x720 both at 16:9) with stream switching.

[benwaggoner]

After a bit of a draught, we have five new checkins for x265! Three for Dolby Vision Profile 5 support, one to make muxing of chunked encoding easier, and one to allow cutree to be used in analysis reuse, finally.

https://bitbucket.org/multicoreware/x265/commits/all

[LigH]

x265 2.9+14-3023bd8b05c0 (https://www.mediafire.com/file/gqaq9939bjwgdhw/x265_2.9%2B14-3023bd8b05c0.7z) (MSYS2, MinGW32 + GCC 7.4.0 / MinGW64 + GCC 8.2.1)

support for Dolby Vision profile 5 and RPU multiplexing; Cutree offset for analysis reuse

[imhh11]

Thanks for the update.

is DV profile 5 single layer ?
if yes, then does it mean we can now convert our 2 layer DV uhd disc to a single layer file playable via usb into our TV ?

[utack]

Netflix dropped a new article.
H264/H265/VP9 comparison using HVMAF with reference encoders and production encoders.
x265 is doing quote ok but can't always beat HM, EVE-VP9 does similarly well
Relative Bitrate savings for low and high quality using three test sets:
https://i.imgur.com/Cf7fQ9V.png


https://medium.com/netflix-techblog/performance-comparison-of-video-coding-standards-an-adaptive-streaming-perspective-d45d0183ca95

[Barough]

x265 v2.9+15-81373aab81df (http://www.mediafire.com/file/mfww6xxjiaodna9/) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit : GCC 7.4.0 / 64bit : GCC 8.2.1)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[LigH]

^ just a fixed compiler warning

[kuchikirukia]

I tried searching the changelog for resolution, variable and so on, but could not find anything. Do you recall which version was it roughly, or where can I find information about that? Support for variable resolution, or better said, variable display aspect ratio as well is something I tried to solve for years. The problem was that matroska stored such data in a single header entry valid for the whole file; back when designing it, nobody imagined files where there would be need to switch these parameters.

AFAIK the best possible solution was to use separate files and link them using the next/previous segment linking fucntionality.

Yup, linked mkvs and ordered chapters (the latter has more widespread support) can handle frame rate changes as well as resolution.

mpv used to use the bitstream DAR instead of the container until I pointed out that I could troll mpv users by encoding a bitstream with rapidly shifting DARs that would work perfectly fine on every other player.
There's really little real-world need to change DAR midstream given the video resolution has to be constant, and container DAR is better than bitstream since that allows it to be trivially fixed if wrong.

[jlpsvk]

Thanks for the update.

is DV profile 5 single layer ?
if yes, then does it mean we can now convert our 2 layer DV uhd disc to a single layer file playable via usb into our TV ?

i would like to know that too. :) and if yes, how. :)

[seandarcy]

Do I use -tune grain, or just decrease crf for the same bitrate ?

I've ripped some old film-based blurays which have film grain that I'd like to retain. I understand that -tune grain will retain the grain, at a 50% increase in file size, or equivalently bitrate. So, as I understand it, if a given crf has 2000kbs, adding -tune grain will result in 3000kbs. But what if I decreased the crf until I got 3000kbs without -tune grain ? Will I retain some/all of the original grain ?

Assume I have a specific bitrate budget, say 3000kbs.

Am I better off with -tune grain crf X (say 29), or no grain and crf X - 2 or 3 (say crf 26 ) ? Again I've picked the crf's so that the output has the same bitrate.

x265, 2.9.
My command line :
ffmpeg -i in.mkv -c:a copy -c:v libx265 -preset medium [-tune grain] crf X -pix_fmt yuv420p10le -x265-params colorprim=bt709:colormatrix=bt709:transfer=bt709 out.mkv

I'm sure that some place in this thread this question has been asked and answered, but none that I've found discuss grain vs crf for a fixed bitrate.

[asarian]

So, does x265 need to be compiled manually, or does someone distribute the binaries too?

[LigH]

You did not read this thread before asking that?!

[asarian]

You did not read this thread before asking that?!

I read the x265 documentation listed in the start post, but they don't seem to offer binaries (and if binaries exist here, they're, subsequently, usually listed in the start-post too). But I'll keep reading. :)

[LigH]

Try bottom-up; just one or two pages ago, you will find about 3-4 people offering latest builds. Including mine.

[asarian]

Try bottom-up; just one or two pages ago, you will find about 3-4 people offering latest builds. Including mine.

Thx. :) I took yours.

[MeteorRain]

So do I, although I rarely post in this thread.

[seandarcy]

I've been ripping a bunch of 480p DVDs with x264. A lot of them are early technicolor.

I've been wondering whether 10 bit x265 would better preserve the color. (I don't have anything that would play 10 bit x264). But if I do that, what's the color range, still TV (limited) or full ? And if TV, what is the limited color range for 10 bit ? 16-235 goes to xxx - yyy ?

And the colorprim stays smpte170m, correct ?

Anybody tried this ?

sean

[SeeMoreDigital]

Given the low cost of HDD's these days, why not back-up the original (8-bit) MPEG-2 stream from the DVD?!

No encoding required... Easy.


Cheers

[LigH]

The 8 or 10 bit of the AVC or HEVC encoder "engine" is not related to the bit depth of the image, but to the resolution of encoder internal quantization parameters after a mathematical transformation. If you never learned about the Fourier transformation of discrete samples to a frequency spectrum, you may not understand much what that means... and despite a limited range, the Discrete Cosine Transform as well as the HEVC Integer Transform are still closely related to Fourier sequences (which are originally infinite).

MPEG-2 video may technically have 8..11 bit internal DCT precision, but in DVD Video media most commonly 9 bit, rarely 10 bit in high quality productions. Recoding will lose quality already by the requantization of the material. I can only agree to SeeModeDigital: The best way to preserve the quality in your DVD material is not to convert it anymore. Just remultiplex the main movie PGC to an MKV (makeMKV) for easy handling on a PC or smart TV.

[seandarcy]

OK, I'll remux to mkv. But what about the soft telecine ? This is a 24fps film soft telecined to 29.97. Can I undo the soft telecine without reencoding ?

sean

[videoh]

You can undo the soft telecine with DGPulldown.

[seandarcy]

Unless I'm misreading this, DGPulldown telecines a stream. It doesn't undo the telecine.

[videoh]

From the user manual:

"Note that if the custom rate conversion is selected, and if the source rate is specified as equal to the destination rate, then all pulldown is removed and the stream is flagged as having a rate equal to the specified destination rate."

[FranceBB]

I've been wondering whether 10 bit x265 would better preserve the color. (I don't have anything that would play 10 bit x264).

x265 uses high bit depth internally and 10bit files are compatible with the majority of players; in other words, even if you feed it with an 8bit stream and you choose Main10, it will use high bit depth internally and encode it to 10bit.


But if I do that, what's the color range, still TV (limited) or full ?


You can encode it in both Full Range or Limited Range, depending on your target.
Please note that the standard is Limited Tv Range.
Besides, if the content was originally in Tv Range, you shouldn't change it.


And if TV, what is the limited color range for 10 bit ? 16-235 goes to xxx - yyy ?


The old 8bit Tv Range was 0.0-0.7V, in other words, 16-235.
For 10bit, Tv Range is 64-940.
x265 has a flag to specify "limited range" which is --range limited (just like x264), and it also has two additional parameters --min-luma 64 --max-luma 940 to apply clipping if you have to be 100% sure that values don't exceed Tv Range (useful for TV Stations).


Can I undo the soft telecine without reencoding ?


If it's really soft telecine (just a flag specified while muxing the original MPEG-2 file), yes, you can, otherwise if it's really telecined with repeated fields, you gotta do reverse telecine and re-encode.

MPEG-2 video may technically have 8..11 bit internal DCT precision, but in DVD Video media most commonly 9 bit, rarely 10 bit in high quality productions.

I used to make official DVDs time ago and you are correct, but I've seen some atrocities done in the past, including a playback 1:1 from a Sony VTR playing a BetaCAM "master", or playback using an Omneon SD playback port connected via SDI, or "recently" an Omneon FULL HD playback port routed via SDI to an hardware downscaler which performed FastBilinear and outputted the result to the Master DVD copy, when all this could have been done better and faster via software... but hey, stone-age procedures and stubborn co-workers...
Luckily, these days are over and nowadays everything it's handled via software.

[BLKMGK]

Some advice on chunked encoding for a newb please!

I've been working out how to use the chunk commands to encode pieces of a file with an eye towards using some form of cluster to speed single file encoding. One thing that I'm observing is slow encode speed but it may be because of the VM I'm using despite 12 cores being made available. When chunk is used and the specified frames are deep into the file the encoder is able to skip to those frames yes, it need not start at the beginning and search for them? I'm not currently specifying key frames and am allowing chunk to find those on its own, I also realize it will encode some frames before and after the keyframes it chooses to start on but I'd like to be sure that in the end I've saved time! I'm encoding a test now with a csv file specified for statistics and it's got nothing in it after 5 mins, I was hoping to see some information as it went along in hopes of determining progress :(

In some tests I've done things like specify color space, key intervals, and other settings, am I better off allowing the encoder to do most of that itself? My intuition says yes lol.

I have attempted to use a cropping filter in ffmpeg feeding X265, would I be better off doing this in X265? I've had trouble figuring that out so far so an example would rock - especially if it uses the same syntax as ffmpeg. Would cropping in ffmpeg prevent a video from seeking to the chunk frame I desire or be slower than using X265? I'm just trying to ensure I'm doing this as efficiently as possible, honestly must ffmpeg be used at all if I'm not adding filters? I've not been able to use X265 standalone to accomplish what I want yet.

I've also noticed that when I encode a chunk like this that VLC won't open the file with a MKV extension but if I use .265 (which I've associated with VLC) it opens, it's as if assumptions are made from the extension that prove invalid. When I've concatenated a number of chunks together and muxed in audio the result has been fine though.

Here are two examples of commandline I'm working with right now:

ffmpeg -hide_banner -i INPUT.mkv -f yuv4mpegpipe - | x265 - --no-open-gop --chunk-start 166675 --chunk-end 169500 --colorprim bt709 --transfer bt709 --colormatrix bt709 --crf=20 --fps 24000/1001 --min-keyint 24 --keyint 240 --sar 1:1 --preset slow --ctu 16 --y4m --pools + -o chunky-nocrop.265

ffmpeg -hide_banner -i INPUT.mkv -filter:v "crop=1920:800:0:140" -f yuv4mpegpipe - | x265 - --no-open-gop --chunk-start 166675 --chunk-end 169500 --crf=20 --fps 24000/1001 --preset slow --ctu 16 --y4m -o chunky1.mkv

Any advice would be welcomed, I'm doing this in Linux FWIW. :thanks:

[FranceBB]

Encoding in a Virtual Machine is gonna be slower than encoding files on your physical hardware; it doesn't matter if you allow many cores and you have. Since you are doing this on Linux in a VM, unless you need a frameserver (like Avisynth) to do any sort of post-processing, drop the Windows Virtual Machine and do it on Linux via ffmpeg and x265 (they can both be compiled using GCC, CMake and NASM).
As to the player, MPV works perfectly well on Linux, you just need to install the additional codecs and you are good to go; no need to use VLC.

For the records, I use Windows at work, but I have Fedora at home. ;)

[BLKMGK]

Encoding in a Virtual Machine is gonna be slower than encoding files on your physical hardware; it doesn't matter if you allow many cores and you have. Since you are doing this on Linux in a VM, unless you need a frameserver (like Avisynth) to do any sort of post-processing, drop the Windows Virtual Machine and do it on Linux via ffmpeg and x265 (they can both be compiled using GCC, CMake and NASM).
As to the player, MPV works perfectly well on Linux, you just need to install the additional codecs and you are good to go; no need to use VLC.

For the records, I use Windows at work, but I have Fedora at home. ;)

Right now it's a Linux VM on ESX for testing, I'm able to run the same command in Windows however as it's crossplatform (my intent). What seems odd to me is the destination file remains empty for an extended length of time but in the end has data. I've shortened up the gap between the chunks to 2875 frames to speed things and am dropping encoding quality to try and speed further but it still takes awhile.

My concern is that I'm somehow forcing it to read through and do processing on the entire file before it reaches it's target range and does real work - that would make breaking jobs up pointless!

End goal: break a large HD encoding task into 60 or more "jobs" and throw those jobs to encoding containers on various machines around the house that have idle cycles. Combine the resulting sections of video, mux in audio, have a complete video. Kubernetes or something like that to manage the containers. Have to have a proper working "job" before I can start the clustering part though hence my testing.

I've done this already, not using chunk, by indexing a video, making jobs that started and ended on keyframes, and then combining etc. but "chunk" seems made for this and allows me to skip the indexing, mathing of start/end points, and sounds like it uses the encoder better.

Hoping that makes more sense!

P.S. Yes, I intend to share whatever I come up with. Every other project I've found that does anything similar hasn't been updated in ages other than RipBot which works great on Windows but I've got more Linux resources and desire cross-platform as a result. :cool:

[asarian]

Encoding in a Virtual Machine is gonna be slower than encoding files on your physical hardware; it doesn't matter if you allow many cores and you have. Since you are doing this on Linux in a VM, unless you need a frameserver (like Avisynth) to do any sort of post-processing, drop the Windows Virtual Machine and do it on Linux via ffmpeg and x265 (they can both be compiled using GCC, CMake and NASM).
As to the player, MPV works perfectly well on Linux, you just need to install the additional codecs and you are good to go; no need to use VLC.

For the records, I use Windows at work, but I have Fedora at home. ;)


I have done many (x264) encodings in a VM (ESXi). Surprisingly, it actually isn't all that much slower, as, while the CPU's are virtualized, of course, they're not like software-emulated or something, and, for the most part, are accessed almost directly. The speed drop was about 2-3% vs. the real deal. Mostly VM I/O was actually a bottleneck, for me. :)

The beauty of VM encoding, is that you can suspend the entire VM, mid-encoding! And it will continue just fine where you left of, when you start it up again. I stopped doing it, though, as I love using GPU-assisted encoding, and figured, if I have to buy a second video-card, just to make it passthru to ESXi, then I'm better off just using my main card directly, outside Vmware.

[BLKMGK]

Agree, I don't see ESX slowing things too much. I am presently shying from GPU encoding mostly because I'm under the impression it's not as high a quality and only one Windows machine I've got could do it. Learning abnout clustering is also an incentive I'll admit.

That said - I have two jobs running right now in CMD Windows in WIN10. One seems to be saying it's run through 94K frame, the other 115K - they've been running over an hour each. The chunk start and end were only 2800 frames apart! One is moving at 23FPS, the other 56FPS, neither datafile has any damn data in it yet.

Am I doing something wrong? Surely chunk isn't meant to read from SOF and should jump to where it's been told to begin encoding right? I really don't want to use --seek and --frames to do this :(

Edit: Rereading the little I can find on chunk. Surely they don't mean ALL frames preceeding do they?! I had read this as a function to help multiple machines work on a single file but it's surely not that if ALL preceeding frames are encoded and discarded as you get into a file :(

option:`--chunk-start and --chunk-end` Frames preceding first frame of chunk in display order will be encoded, however, they will be discarded in the bitstream. Frames following last frame of the chunk in display order will be used in taking lookahead decisions, but, they will not be encoded. This feature can be enabled only in closed GOP structures. Default disabled.

[BLKMGK]

Just answered part of my own question "chunk" reads and encodes from start of file all the way to it's given starting point to create it's "chunk". It then reads to the given end point and completes. I've seen this mentioned as something that could be used for clustering jobs but not with this behavior it isn't! Is this intended? Why does it not advance to it's given start frame, look backwards to a previous keyframe, and start encoding there? It could discard the frames prior to it's given start but have given the encoder frames to prime it. If this is intended behavior I don't understand the usefulness.

Edit: Just tried an end point for the chunk that wasn't right against EOF. When it hits its endpoint it keeps encoding but discards the work.

[asarian]

Agree, I don't see ESX slowing things too much. I am presently shying from GPU encoding mostly because I'm under the impression it's not as high a quality and only one Windows machine I've got could do it.

Coding HEVC directly with (hardware-accelerated) NVENC, for instance, is an order of magnitude faster (and sometimes several orders of magnitude even) than what we can achieve with x265 itself, but you have no control over the process, and the quality is definitely lower than that of the latter.

I was more talking about OpenCL in x264; and especially used for the KNLMeansCL denoiser (QTGMC), where it speeds things up significantly. Even though I recently heard even OpenCL yields a theoretical lower quality.

That said - I have two jobs running right now in CMD Windows in WIN10. One seems to be saying it's run through 94K frame, the other 115K - they've been running over an hour each. The chunk start and end were only 2800 frames apart! One is moving at 23FPS, the other 56FPS, neither datafile has any damn data in it yet.

As for the latter, typically you have to refresh your explorer window (a few times) to see how much is actually output already, or it will stay at 0 bytes. :)

As for chunks, I never used them before, so I can't really say something useful about that.

[Ma]

If you want to encode without VBV buffer, you could use --seek and --frames options, for example:
for %i in (0 1000 2000) do ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek %i --frames 1000 c%i.hevc

which expands to:
ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 0 --frames 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 1000 --frames 1000 c1000.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 2000 --frames 1000 c2000.hevc

Finally you can put these 3 parts together:
mkvmerge -o c-whole.mkv c0.hevc + c1000.hevc + c2000.hevc
-----------------
example for Windows

[BLKMGK]

Coding HEVC directly with (hardware-accelerated) NVENC, for instance, is an order of magnitude faster (and sometimes several orders of magnitude even) than what we can achieve with x265 itself, but you have no control over the process, and the quality is definitely lower than that of the latter.

I was more talking about OpenCL in x264; and especially used for the KNLMeansCL denoiser (QTGMC), where it speeds things up significantly. Even though I recently heard even OpenCL yields a theoretical lower quality.



As for the latter, typically you have to refresh your explorer window (a few times) to see how much is actually output already, or it will stay at 0 bytes. :)

As for chunks, I never used them before, so I can't really say something useful about that.

Ah, makes sense to use them for programmatic speedups and for filters I agree. I may try using my GPU a few times just to see if the results are acceptable to me but so far haven't been able to get it working the few times I've tried :(

This definitely wasn't a refresh issue in explorer, Chunk really doesn't appear to save any data until it hits its start frame and continues encoding after it hits its end frame. Ugh!

If you want to encode without VBV buffer, you could use --seek and --frames options, for example:
for %i in (0 1000 2000) do ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek %i --frames 1000 c%i.hevc

which expands to:
ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 0 --frames 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 1000 --frames 1000 c1000.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 2000 --frames 1000 c2000.hevc

Finally you can put these 3 parts together:
mkvmerge -o c-whole.mkv c0.hevc + c1000.hevc + c2000.hevc
-----------------
example for Windows

I've done this except I started and ended on keyframes with the understanding that not doing so could have poor joins. The issue with that is it requires first pulling an index and then skipping through it to build the jobs. The first test attempted I built the latter part by hand (PITA) and after encoding I couldn't detect the seams at all. Seemed a good path forward!

I then noticed chunk in the release notes. Chunk, from its description and some discussion I'd seen here, seemed tailor made for clutering but my tests sure don't seem to prove it out. I'll try your example and see if I can detect the seams, perhaps I've been too paranoid?

Thank you for the example of how to build the jobs!

Edit: Tested this method ignoring keyframes and thought I detected a hitch in the video. Sure enough VLC shows two dropped frames where the hitch occurs and I see it elsewhere too. I'll keep testing and see if the keyframes are the issue. I'll pull an index and create some jobs. I need an automated way to build jobs from the index but not at every break lol

[Ma]

I tried to add --chunk-start/--chunk-end to the example and it is like this:
Encoding in parts of 1000 frames (+100 before, +100 after):

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 0 --frames 1100 --chunk-start 1 --chunk-end 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 900 --frames 1200 --chunk-start 101 --chunk-end 1100 c1.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 1900 --frames 1200 --chunk-start 101 --chunk-end 1100 c2.hevc

mkvmerge -o chunks.mkv c0.hevc + c1.hevc + c2.hevc

[BLKMGK]

I tried to add --chunk-start/--chunk-end to the example and it is like this:
Encoding in parts of 1000 frames (+100 before, +100 after):

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 0 --frames 1100 --chunk-start 1 --chunk-end 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 900 --frames 1200 --chunk-start 101 --chunk-end 1100 c1.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 1900 --frames 1200 --chunk-start 101 --chunk-end 1100 c2.hevc

mkvmerge -o chunks.mkv c0.hevc + c1.hevc + c2.hevc

That looks like a very good idea to get the encoder to take into account frames before and after the specified range in order for the encoder to be more efficient! It's not breaking on keyframes, am I wrong about their importance?

I tried your example on my sample file, I'm not spotting glitches or dropped frames in the statistics and I managed to extend it out some.

ffmpeg -i INPUT.MKV -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 3100 --frames 1200 --chunk-start 101 --chunk-end 1100 --csv stats.csv c3.hevc

If I'm wrong about the keyframes that will make life much easier. I'll try doing an entire film once I've incorporated this into my bash script, mux in sound, and check the synch.

Thank you! I really appreciate you providing examples too as it makes understanding what you're doing much simpler. :thanks:

[nghiabeo20]

What can SAO do? Does it improve quality/file size?
I'm targeting good quality with reasonable bitrate (5-6000 kbps for 1080p10, BD remux source) with CRF 21. Thanks!

[excellentswordfight]

What can SAO do? Does it improve quality/file size?
I'm targeting good quality with reasonable bitrate (5-6000 kbps for 1080p10, BD remux source) with CRF 21. Thanks!
Yes, SAO increases efficiency/compression. But it usually does it with the drawback of an slight reduction in fine detail. So its pretty great for low bitrate encodings and animation, but not so much if you wanna keep most of the detail.

Its best if you test it for yourself, and see what settings gives you the best results, but if you target that bitrate range I whould say that you might wanna turn it off.

And btw, I think crf21 will end up at a much lower bitrate for most content if you dont us tune grain or something similar. I get arround those numbers with crf18.

[nghiabeo20]

Yes, SAO increases efficiency/compression. But it usually does it with the drawback of an slight reduction in fine detail. So its pretty great for low bitrate encodings and animation, but not so much if you wanna keep most of the detail.

Its best if you test it for yourself, and see what settings gives you the best results, but if you target that bitrate range I whould say that you might wanna turn it off.

And btw, I think crf21 will end up at a much lower bitrate for most content if you dont us tune grain or something similar. I get arround those numbers with crf18.

Maybe I'm doing something wrong. All I edit from the veryslow preset is these commands

bframes=6:subme=3:rd=5:qcomp=0.75:no-sao:deblock=-2:-2

however CRF 22 10bit without any tuning gives me a whopping 8500kbps for a 30s test clip from my The Hundred-Foot Journey bluray!

[Boulder]

The bitrate depends a huge deal on the content. Grainy or noisy stuff requires much more than some clean CGI content.

[excellentswordfight]

Maybe I'm doing something wrong. All I edit from the veryslow preset is these commands

bframes=6:subme=3:rd=5:qcomp=0.75:no-sao:deblock=-2:-2

however CRF 22 10bit without any tuning gives me a whopping 8500kbps for a 30s test clip from my The Hundred-Foot Journey bluray!
As boulder said, thats very source dependant. And a 30s sample might not be representative either. But with that said i think your qcomp setting plays a role here (maybe deblock as well).

[singhkays]

Can anybody give any advice if there is any benefit by taking the following approaches? and is either one of the splitting methods preferred?


Split a video file on keyframes
Split a video file based on the shots (i.e. different camera angles)


other than the obvious ability to parallelize the encoding task

[Selur]

other than the obvious ability to parallelize the encoding task
Aside from further parallelizing the encoding more there is no reason to split a source unless you want to.
When having the split a source for parallelizing the encoding I would prefer to split 'a video file based on the shots' since the source simply might have key frames placed in a fixed interval not caring about coding efficiency. That said splitting on key frames is easier.
But splitting on key frames in respect to scenes and using the segmented encoding feature of x265 is probably the best easy to implement way. Ideally you wound split at the points where the encoder would want to set key frame for coding efficiency.

[singhkays]

But splitting on key frames in respect to scenes and using the segmented encoding feature of x265 is probably the best easy to implement way. Ideally you wound split at the points where the encoder would want to set key frame for coding efficiency.

Thank for the advice. I have specific question on the below

splitting on key frames in respect to scenes

How would this work? Should I split on keyframes within the scenes?

segmented encoding feature of x265

I haven't heard about this. How does this feature work?

Ideally you wound split at the points where the encoder would want to set key frame for coding efficiency

How do I figure out where the encoder wants to set key frames? I thought that was a user controlled option i.e. keyint

[filler56789]

To whom this may interest...
latest revision(s) cannot be compiled under MinGW-w64 && MSYS(2).
Some recent changes prevent the libraries from being linked to the command-line .EXE.

[Ma]

Some recent changes prevent the libraries from being linked to the command-line .EXE.

Multilib builds are broken (and builds with ENABLE_HDR10_PLUS too).
Simplest solution:
diff -r 537bba0b7fdc source/encoder/encoder.cpp
--- a/source/encoder/encoder.cpp Thu Dec 27 11:56:52 2018 +0530
+++ b/source/encoder/encoder.cpp Thu Dec 27 14:25:51 2018 +0100
@@ -69,7 +69,7 @@
int doviProfileId;
}DolbyVisionProfileSpec;

-DolbyVisionProfileSpec dovi[] =
+static DolbyVisionProfileSpec dovi[] =
{
{ 1, 1, 1, 1, 1, 5, 1, 2, 2, 2, 50 },
{ 1, 1, 1, 1, 1, 5, 0, 16, 9, 9, 81 },
diff -r 537bba0b7fdc source/x265.cpp
--- a/source/x265.cpp Thu Dec 27 11:56:52 2018 +0530
+++ b/source/x265.cpp Thu Dec 27 14:25:51 2018 +0100
@@ -690,7 +690,7 @@
args[argCount++] = token;
token = strtok(NULL, " ");
}
- args[argCount] = '\0';
+ args[argCount] = NULL;
CLIOptions cliopt;
if (cliopt.parseZoneParam(argCount, args,param, i))
{

[filler56789]

^ Thanks for answering.

Multilib builds are broken (and builds with ENABLE_HDR10_PLUS too).

OUCH! :-/ So I will have to wait.

Simplest solution:
diff -r 537bba0b7fdc source/encoder/encoder.cpp
--- a/source/encoder/encoder.cpp Thu Dec 27 11:56:52 2018 +0530
+++ b/source/encoder/encoder.cpp Thu Dec 27 14:25:51 2018 +0100
@@ -69,7 +69,7 @@
int doviProfileId;
}DolbyVisionProfileSpec;

-DolbyVisionProfileSpec dovi[] =
+static DolbyVisionProfileSpec dovi[] =
{
{ 1, 1, 1, 1, 1, 5, 1, 2, 2, 2, 50 },
{ 1, 1, 1, 1, 1, 5, 0, 16, 9, 9, 81 },
diff -r 537bba0b7fdc source/x265.cpp
--- a/source/x265.cpp Thu Dec 27 11:56:52 2018 +0530
+++ b/source/x265.cpp Thu Dec 27 14:25:51 2018 +0100
@@ -690,7 +690,7 @@
args[argCount++] = token;
token = strtok(NULL, " ");
}
- args[argCount] = '\0';
+ args[argCount] = NULL;
CLIOptions cliopt;
if (cliopt.parseZoneParam(argCount, args,param, i))
{

Since I didn't know the multilib builds are impossible for the time being,
the old error message

(something about "invalid conversion from char to char*")

just went replaced with a new one :-/

[Yanak]

Hi,
Introduce --tune animation option
--psy-rd 0.4
--aq-strength 0.4
--deblock 1:1
--bframes increased by 2

Sorry if it's a silly question but Bframes increased by 2, means +2 from the default parameter which is 4 Bframes according to the doc (https://x265.readthedocs.io/en/latest/cli.html#cmdoption-bframes), so it will be --bframes 6, or from 0 so it will be --bframes 2 ?

Thank you

[Wishbringer]

Or from bframes you give per commandline?
What has a higher priority?
I expect: +2 from preset, except explicit given by command, then the manual value

[LigH]

You will usually first use a preset, then a tuning (so the "animation" tuning will add 2 to the number of B frames of your preset); but if you also add an explicit bframes parameter later in the command line, it will supersede previously calculated values.

[RainyDog]

Looks like we have a new AQ mode in x265 2.9+28 :cool:

It scales the quantization step size according to the spatial activity of one coding unit relative to frame average spatial activity. This AQ method utilizes the minimum variance of sub-unit in each coding unit to represent the coding units spatial complexity.

Default 1.0.

**Range of values:** 0.0 to 3.0

+.. option:: --hevc-aq
+
+ Enable adaptive quantization
+ It scales the quantization step size according to the spatial activity of one
+ coding unit relative to frame average spatial activity. This AQ method utilizes
+ the minimum variance of sub-unit in each coding unit to represent the coding
+ unit’s spatial complexity.
+
+.. option:: --qp-adaptation-range
+
+ Delta-QP range by QP adaptation based on a psycho-visual model.
+
+ Default 1.0.
+ **Range of values:** 1.0 to 6.0

[Boulder]

Seems interesting. It would be nice to hear of the cases which this new method targets.

[Boulder]

x265 version 2.9+34-17120cf win64 [ICC 1900] (https://drive.google.com/open?id=1pqnbKmfE0pnvx-rlo5wYCB5OPMvuo_92)

libiomp5md.dll seems to be a dependency as I get a missing file warning. Where can I get that one (and others which may also be needed)?

[Forteen88]

libiomp5md.dll seems to be a dependency as I get a missing file warning. Where can I get that one (and others which may also be needed)?"libiomp5md.dll is from the Intel C compiler".
More info here, https://stackoverflow.com/questions/7270639/why-does-intel-compiler-produce-output-that-requires-libiomp5mt-dll-even-though

[Ma]

I found this statement about Intel C++:
***
If you want to link against the static OpenMP RTL, you must add /Qopenmp-link:static, which is a new switch for 11.0. So to produce a purely static executable, compile/link with /MT /Qopenmp-link:static
***

[Wolfberry]

@Ma
Thanks for the information.
Without /Qopenmp, /MT should be fine.
With /Qopenmp, /Qopenmp-link:static needs to be used to produce a purely static executable.

[Ma]

I hope that you pass /MT by
-DSTATIC_LINK_CRT=ON
CMake option.

[shinchiro]

In case, people want to play around with new hevc-aq: :)

x265-2.9+34-17120cfb5c99-multilibs-gcc-8.2.1.7z (https://transfer.sh/o933m/x265-2.9+34-17120cfb5c99-multilibs-gcc-8.2.1.7z)

[Yanak]

You will usually first use a preset, then a tuning (so the "animation" tuning will add 2 to the number of B frames of your preset); but if you also add an explicit bframes parameter later in the command line, it will supersede previously calculated values.

Or from bframes you give per commandline?
What has a higher priority?
I expect: +2 from preset, except explicit given by command, then the manual value

It makes sense, was a bit confused about it.

Thank you guys.

[mini-moose]

In the recent default branch builds, aq-mode default is now 2 instead of 1. Reading some on it, it seems (at least on paper) like a better mode to use: "applies separate aq-strength for each frame, based on the complexity of current frame. I tested on a short clip (10k frames) and it seems the avg bitrate with mode 2 is lower and avg qp is higher.
I'm not an expert so would appreciate comments on this.

The is also some feature called hevc-aq which I'm not sure what it does: Mode for HEVC Adaptive Quantization. Default disabled

Also see something called "no-splice" in mediainfo encoder settings, which I can't find any reference to in --full help.

[Boulder]

I did a quick 2-pass test and a CRF test encode and hevc-aq smoothed the flat background quite heavily in both cases. The bitrate rose quite a lot in CRF mode:

AQ-mode 1 : 4425 kbps
AQ-mode 2 : 4706 kbps
AQ-mode 3 : 5999 kbps
HEVC-AQ : 5094 kbps

I'm really not convinced that aq-mode 2 is a better option than mode 1. To me it seems to also smooth the flat areas too much and that becomes quite annoying once you start noticing the gradients that result from it. See the wall between the women.

Original frame before encoding:
https://i.postimg.cc/9w9HttFx/original.png (https://postimg.cc/9w9HttFx)

AQ-mode 1:
https://i.postimg.cc/wySSvycD/aqmode1.png (https://postimg.cc/wySSvycD)

AQ-mode 2:
https://i.postimg.cc/7J8WDvwV/aqmode2.png (https://postimg.cc/7J8WDvwV)

HEVC-AQ:
https://i.postimg.cc/xNvF1sdS/hevcaq.png (https://postimg.cc/xNvF1sdS)

[LigH]

Single frame screenshots often don't tell the impression of a movie clip accurately.

[Boulder]

Fortunately the testclips are quite small so I could put them in my Google Drive:

https://drive.google.com/open?id=1NrgUKJG0SJ8Y8Kat1j1fjYfflCArMx6C (aq-mode 1)
https://drive.google.com/open?id=1__fF2lLWGhqH1yB4DpfvuE55NBeMo_ui (aq-mode 2)
https://drive.google.com/open?id=1iWzq4i__QZYxdXXyNaNt7364LDlQQI2p (hevc-aq)

These are all 2-pass encodes with the same avg bitrate. To me hevc-aq smooths things too much, part of the problem is that the bitrate is not enough to keep up. Aq-mode 2 doesn't look as good as mode 1. Near the end of the clip, the difference between 1 and 2 is clearer as mode 2 has produced more swimming blocks in the flat background near the top of the image.

[tuanden0]

Fortunately the testclips are quite small so I could put them in my Google Drive:

https://drive.google.com/open?id=1NrgUKJG0SJ8Y8Kat1j1fjYfflCArMx6C (aq-mode 1)
https://drive.google.com/open?id=1__fF2lLWGhqH1yB4DpfvuE55NBeMo_ui (aq-mode 2)
https://drive.google.com/open?id=1iWzq4i__QZYxdXXyNaNt7364LDlQQI2p (hevc-aq)

These are all 2-pass encodes with the same avg bitrate. To me hevc-aq smooths things too much, part of the problem is that the bitrate is not enough to keep up. Aq-mode 2 doesn't look as good as mode 1. Near the end of the clip, the difference between 1 and 2 is clearer as mode 2 has produced more swimming blocks in the flat background near the top of the image.

Agree with you, --hevc-aq just smooth (blur) too much, some details were blurred and destroyed.

I tested with setting --preset slow --tune animation --crf 18 --hevc-aq

https://i.imgur.com/B0xclJl.png

[Selur]

Does '--hevc-aq' exclude the usage of '--aq-mode', I mean is it one or the other, or can they both be applied at the same time? (can't test myself atm.)
Also is '--qp-adaptation-range' a sub-option of '--hevc-aq' or '--aq-mode, or of both?

[Boulder]

Hevc-aq replaces the other aq-modes so now we have 4 possible aq-modes to use. Somehow it looks to me that qp-adaptation-range is available to all modes. It really should be tested to see what it actually does. The problem with these new parameters and functionalities has lately been the fact that no one explains why they were implemented (which scenarios will probably benefit and which may suffer). Also the test cases would be interesting to see and compare.

[Wolfberry]

qp-adaptation-range doesn't seem to do anything at the moment (There's no variance in final results nor I can find what it does in the code.)

[Selur]

Thanks for the help!

Hevc-aq replaces the other aq-modes so now we have 4 possible aq-modes to use.
may be we get lucky and they adjust the code so that there simply is a new mode for 'aq-mode' instead of a new flag,..

qp-adaptation-range doesn't seem to do anything at the moment (There's no variance in final results nor I can find what it does in the code.)
okay, so ignoring that parameter for now ;)

It really should be tested to see what it actually does. The problem with these new parameters and functionalities has lately been the fact that no one explains why they were implemented (which scenarios will probably benefit and which may suffer).
100% agree

[Yanak]

Edit : nevermind, i think i finally figured it, looks like i needed a little break from the screen to ease a bit my mind and see a bit more clear in all this, having the nose pointed at all this stuff since hours now :p

Hi all,

I'm trying to compile x265 using VS2017 (just want to compare speed vs some releases, especially interested in the Intel compiler linked in my vs2017 as my old CPU supports max AVX only, a little gain will be nice, no matter if it's a tiny one),

I followed all the steps, installed all stuff needed and i can compile successfully the 8 or 10 or 12 Bits exe's but I'm scratching my head since a couple of hours about how to compile a multilib 8+10+12Bits version.

I launched the multilib.bat, it creates the 8,10,12bits folders and it starts to populate some of them with a bunch of stuff and then i get prompted by an error message "10bit build failed". I'm pretty sure I'm missing something obvious but don't know what.

- Do i need first to generate the Solution using make-solutions.bat and repeat the process 3 times, using cmake GUI and selecting the different depth options to in the end get one for each 8,10,12Bits depths and generate each .exe and only after that i can run the multilib.bat ?

After spending a few hours on this i'm a bit out of ideas, if anyone have any tips about the steps to follow to create the multilib 8+10+12Bits exe please ?

Thank you very much.

[Ma]

I decided to test aq-mode 0/1/2/3 vs. hevc-aq at preset veryslow encoding lighthouse_lossless.mp4 to 10-bit hevc at bitrates 750/1500/3000/6000.

Command line:
for %b in (750 1500 3000 6000) do (
for %a in (0 1 2 3) do (
ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --bitrate %b --y4m - --qg-size 64 --preset veryslow --psnr --ssim --aq-mode %a a%a-%b-1.hevc --pass 1
ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --bitrate %b --y4m - --qg-size 64 --preset veryslow --psnr --ssim --aq-mode %a a%a-%b-2.hevc --pass 2
)
ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --bitrate %b --y4m - --qg-size 64 --preset veryslow --psnr --ssim --hevc-aq ah-%b-1.hevc --pass 1
ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --bitrate %b --y4m - --qg-size 64 --preset veryslow --psnr --ssim --hevc-aq ah-%b-2.hevc --pass 2
)

Results (sorted by PSNR):
bitrate 750:
aq-mode 1 - PSNR 43.244, SSIM 15.697 dB
aq-mode 3 - PSNR 43.063, SSIM 15.549 dB
aq-mode 2 - PSNR 42.919, SSIM 15.451 dB
--hevc-aq -- PSNR 42.514, SSIM 14.240 dB
aq-mode 0 - PSNR 41.108, SSIM 13.925 dB

bitrate 1500:
aq-mode 1 - PSNR 45.388, SSIM 17.483 dB
aq-mode 3 - PSNR 45.283, SSIM 17.366 dB
aq-mode 2 - PSNR 45.172, SSIM 17.288 dB
--hevc-aq -- PSNR 44.372, SSIM 15.980 dB
aq-mode 0 - PSNR 43.491, SSIM 16.001 dB

bitrate 3000:
aq-mode 3 - PSNR 47.310, SSIM 18.908 dB
aq-mode 2 - PSNR 47.276, SSIM 18.872 dB
aq-mode 1 - PSNR 47.240, SSIM 18.869 dB
--hevc-aq -- PSNR 46.424, SSIM 17.772 dB
aq-mode 0 - PSNR 46.084, SSIM 18.084 dB

bitrate 6000:
aq-mode 2 - PSNR 48.800, SSIM 19.839 dB
aq-mode 3 - PSNR 48.661, SSIM 19.710 dB
aq-mode 0 - PSNR 48.384, SSIM 19.659 dB
aq-mode 1 - PSNR 48.268, SSIM 19.436 dB
--hevc-aq -- PSNR 48.093, SSIM 19.248 dB

hevc-aq is not ready yet -- especially first pass is totally broken.
It's funny like aq-mode 1 wins at small bitrates and looses at high (even with aq-mode 0).

Full results in attached screen.txt

[Selur]

It's funny like aq-mode 1 wins at small bitrates and looses at high (even with aq-mode 0).
which makes it strange that aq-mode 2 is the new default,... (https://bitbucket.org/multicoreware/x265/commits/b14834a9d1c1864ea7e94d9cfed4e33f37e767c6)

[Forteen88]

@Ma. Isn't SSIM a better image-quality metric?
And isn't VMAF the best image-quality metric?! I saw that someone here shared a VMAF-version of x265.

[Ma]

For me the clip lighthouse_lossless.mp4 is interesting because of the sky -- it is challenging to encode this clip at low bitrates to be not annoying if you look at the sky. There are not sharp details -- it is about the gradients. PSNR is not perfect but for the sky it is OK.

For me at low bitrates aq-mode 1 is the least annoying, for medium bitrates aq-mode 3 is OK too. The default aq-mode 2 is not the best at low and medium bitrates with this clip (for me).

[Ma]

It looks like there is (at least one) bug in '--hevc-aq' option.
Workaround is quite simple, please use
--hevc-aq --no-cutree
instead of plain '--hevc-aq' option.

[imhh11]

Thanks for the update.

is DV profile 5 single layer ?
if yes, then does it mean we can now convert our 2 layer DV uhd disc to a single layer file playable via usb into our TV ?

i would like to know that too. :) and if yes, how. :)



even better, we can now mux DV layers (without re-encoding) into a mp4 file playable in my DV TV.

https://extraimage.net/images/2018/12/31/dc8f90ba1323839e80650471dc566510.png

only work with AC3 / E-AC-3 / AC-4 / AAC audio though.

https://github.com/DolbyLaboratories/dlb_mp4base

mp4muxer --dv-profile 7 --input-file (baselayername.hevc1) --input-file (dvlayername.hevc2) --input-file (audio.ac3) --media-lang eng --output-file (nameofmovie).mp4

[SeeMoreDigital]

...only work with AC3 audio though.

Actually, the new .MP4 muxer supports the following audio formats: Dolby Digital (AC-3), Dolby Digital Plus (E-AC-3) and Dolby AC-4 audio formats. Along with all AAC flavours.

Cheers

[DJATOM]

It looks like there is (at least one) bug in '--hevc-aq' option.
Workaround is quite simple, please use
--hevc-aq --no-cutree
instead of plain '--hevc-aq' option.

We (Beatrice-Raws) also discovered some annoying bug with --aq-motion. It's easy to trigger that on the static background and a single moving object. Samples reproducing that issue here (https://dropmefiles.com/Dpar2).

[Aruna Matheswaran]

Hello Everyone,
Our latest blog on Dolby Vision integration into x265 is out! Do check out the blog @ http://x265.org/x265-delivers-dolby-vision-streams/

[agressiv]

Looks like 3.0 will be released soon?

[LigH]

x265-3.0_RC+2-5251075203e1-win64-static+shared [ICC 1900] (https://drive.google.com/open?id=1xZQABtoaSFgGu11YstmHKYLzO3elemlC)

No real difference compared to previous build.

Doesn't build in MSYS2/MinGW — I believe because the Win Resource Compiler doesn't accept non-number parts of the version string, and "0_RC" is not a number.

[agressiv]

Hello Everyone,
Our latest blog on Dolby Vision integration into x265 is out! Do check out the blog @ http://x265.org/x265-delivers-dolby-vision-streams/

Anyone have a real-world example (with sample clips?) of this function?

I've extracted a secondary HEVC stream from a Dolby Vision UHD blu-ray with tsMuxer and tried to use that as the --dolby-vision-rpu parameter, but I'm guessing it's looking for something different. Encoder aborts about 5 seconds in.

x265 [error]: Invalid Dolby Vision RPU size in POC 8218:19:24

I'd love to get this working and not have to deal with multiple streams for DV.

[chinobino]

Hello Everyone,
Our latest blog on Dolby Vision integration into x265 is out! Do check out the blog @ http://x265.org/x265-delivers-dolby-vision-streams/

Exciting stuff!

Looks like v3 will have a lot of new features.

[FranceBB]

Hello Everyone,
Our latest blog on Dolby Vision integration into x265 is out! Do check out the blog @ http://x265.org/x265-delivers-dolby-vision-streams/

Very interesting, indeed.
Unfortunately I don't have any Dolby Vision file to extract the Dolby Vision RPU and fancy an encode, though.

[SeeMoreDigital]

Anyone have a real-world example (with sample clips?) of this function?Is this 'video only' sample (supplied by algorithm_colon) what you're looking for: https://forum.doom9.org/showthread.php?p=1861429#post1861429

I've been able to add audio Dolby Digital and AAC audio to it using a standard MP4Box build...

[Ma]

We (Beatrice-Raws) also discovered some annoying bug with --aq-motion. It's easy to trigger that on the static background and a single moving object. Samples reproducing that issue here (https://dropmefiles.com/Dpar2).

Yes, I see the problem. There are 3 functions that make quality of the one butterfly on static background worse:
--aq-mode
--cutree
--aq-motion
All of them penalize CU with butterfly and the result is extremely low quality in one place. It should be some limit in x265 that if all 3 functions penalize one place, the applied quality penalty is not bigger than some MAX_PENALTY.

As a workaround we could use only 2 of these 3 functions and/or lower aq-strength.

[asarian]

Hello Everyone,
Our latest blog on Dolby Vision integration into x265 is out! Do check out the blog @ http://x265.org/x265-delivers-dolby-vision-streams/

Wonderful! :goodpost:

If I understood the process right, I'm most interested in 'Profile 8.2 single layer with SDR compatibility' myself, as any extra layer that can be used for a better HDR->SDR conversion, will be very welcome.

I guess this also means you guys have 'cracked' the DV layer, and it's now known how to handle the meta-data therein, right?

Bit off-topic, perhaps, but I wouldn't mind seeing DV SDR layer info getting added to x264 too, one day (after all, x264 has always been predominantly about 8-bit).

[benwaggoner]

I decided to test aq-mode 0/1/2/3 vs. hevc-aq at preset veryslow encoding lighthouse_lossless.mp4 to 10-bit hevc at bitrates 750/1500/3000/6000.

…

hevc-aq is not ready yet -- especially first pass is totally broken.
It's funny like aq-mode 1 wins at small bitrates and looses at high (even with aq-mode 0).
Did you do a visual comparison? It's pretty typical that psychovisual improvements in AQ will reduce PSNR, and even SSIM in some cases.

[benwaggoner]

Yes, I see the problem. There are 3 functions that make quality of the one butterfly on static background worse:
--aq-mode
--cutree
--aq-motion
All of them penalize CU with butterfly and the result is extremely low quality in one place. It should be some limit in x265 that if all 3 functions penalize one place, the applied quality penalty is not bigger than some MAX_PENALTY.

As a workaround we could use only 2 of these 3 functions and/or lower aq-strength.
Sounds like there is some sort of race condition happening. Better to fix that than just have a parameter to limit the impact of the features, which could reduce quality in other kinds of content.

That said, aq-motion is still experimental. I wouldn't trust it as a "Default On" feature without a lot of testing.

[benwaggoner]

Hevc-aq replaces the other aq-modes so now we have 4 possible aq-modes to use. Somehow it looks to me that qp-adaptation-range is available to all modes. It really should be tested to see what it actually does. The problem with these new parameters and functionalities has lately been the fact that no one explains why they were implemented (which scenarios will probably benefit and which may suffer). Also the test cases would be interesting to see and compare.
Really? it seems it should be --aq-mode 4, then, if it is exclusive to the others.

[benwaggoner]

Really? it seems it should be --aq-mode 4, then, if it is exclusive to the others.
Well, just did a test encode and it appears to be true

x265 [warning]: hevc-aq enabled, disabling other aq-modes

This is confusing syntax. We really should have aq-mode 4 and 5. aq-mode 4 would be HEVC-AQ, and 5 would be HEVC-AQ with low luma bias (like aq-mode 3 versus 2). The need for low-luma bias is content dependent, and we shouldn't be locked into one or another.

Also I am getting VBV violations multiple times a second. From the first two seconds of my test clip:

x265 [warning]: poc:0, VBV underflow (-2596808 bits)
x265 [warning]: poc:8, VBV underflow (-13025705 bits):06:32
x265 [warning]: poc:4, VBV underflow (-13047849 bits):38:49
x265 [warning]: poc:16, VBV underflow (-12824903 bits)11:58
x265 [warning]: poc:12, VBV underflow (-12960465 bits):23:11
x265 [warning]: poc:24, VBV underflow (-12939991 bits)27:49
x265 [warning]: poc:20, VBV underflow (-13441281 bits):34:43
x265 [warning]: poc:32, VBV underflow (-12651471 bits)23:47
x265 [warning]: poc:28, VBV underflow (-12985385 bits):28:15
x265 [warning]: poc:40, VBV underflow (-12559455 bits)20:36
x265 [warning]: poc:36, VBV underflow (-13122593 bits)24:13


Was the Experimental Feature tag left off for HEVC-AQ? I love the idea of improved AQ, but it would be good to have a clear indication in the help text about how close to production ready this feature is.

[benwaggoner]

Wonderful! :goodpost:

If I understood the process right, I'm most interested in 'Profile 8.2 single layer with SDR compatibility' myself, as any extra layer that can be used for a better HDR->SDR conversion, will be very welcome.
It wouldn't be any HDR->SDR conversion. The base layer would just be the SDR. I don't know of any real-world use of Profile 8.2, but I imagine it would be mainly used to take SDR sources and add metadata for how it should be enhanced for HDR. It is probably possible to derive profile 8.2 from an HDR DoVi master. But I don't know if full HDR dynamic range could be reconstructed without an actual secondary media layer, or a metadata layer with LOTS of spatial tagging.

Bit off-topic, perhaps, but I wouldn't mind seeing DV SDR layer info getting added to x264 too, one day (after all, x264 has always been predominantly about 8-bit).
H.264 Dolby Vision was certainly used a bunch in early days. But I don't know why anyone would use it today; all consumer products I can think of that can display Dolby Vision have HEVC decoders.

But encoding isn't the hard part. The hard part is the shaping, inverse tone mapping, and metadata generation. Profile 8.1 is only easy because uncompressed HDR-10 is pretty much the raw format of a DoVi master (plus metadata). x26? doesn't even attempt to implement the complex tech required to do a good job converting between color volumes.

[asarian]

It wouldn't be any HDR->SDR conversion. The base layer would just be the SDR. I don't know of any real-world use of Profile 8.2, but I imagine it would be mainly used to take SDR sources and add metadata for how it should be enhanced for HDR. It is probably possible to derive profile 8.2 from an HDR DoVi master. But I don't know if full HDR dynamic range could be reconstructed without an actual secondary media layer, or a metadata layer with LOTS of spatial tagging.


H.264 Dolby Vision was certainly used a bunch in early days. But I don't know why anyone would use it today; all consumer products I can think of that can display Dolby Vision have HEVC decoders.

But encoding isn't the hard part. The hard part is the shaping, inverse tone mapping, and metadata generation. Profile 8.1 is only easy because uncompressed HDR-10 is pretty much the raw format of a DoVi master (plus metadata). x26? doesn't even attempt to implement the complex tech required to do a good job converting between color volumes.


Thanks for the great explanations! :)

[jlpsvk]

Wonderful! :goodpost:

If I understood the process right, I'm most interested in 'Profile 8.2 single layer with SDR compatibility' myself, as any extra layer that can be used for a better HDR->SDR conversion, will be very welcome.

I guess this also means you guys have 'cracked' the DV layer, and it's now known how to handle the meta-data therein, right?

Bit off-topic, perhaps, but I wouldn't mind seeing DV SDR layer info getting added to x264 too, one day (after all, x264 has always been predominantly about 8-bit).

why the hell you would downgrade HDR10 base to SDR? Main interest should be Profile 8.1. But still.... what is the DV RPU METADATA file? Is that the second HEVC stream on UHD BD disc?

[algorithm_colon]

Is this 'video only' sample (supplied by algorithm_colon) what you're looking for: https://forum.doom9.org/showthread.php?p=1861429#post1861429

I've been able to add audio Dolby Digital and AAC audio to it using a standard MP4Box build...

I tried using the same input files (base layer and metadata layer) to x265 as I used with the dolby tool and I get the same "Invalid Dolby Vision RPU" error

[Barough]

x265 v3.0_RC+4-8aebc58efe5c (https://www.mediafire.com/file/l6e0rrj4q1c2r3c/) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit : GCC 7.4.0 / 64bit : GCC 8.2.1)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[benwaggoner]

why the hell you would downgrade HDR10 base to SDR? Main interest should be Profile 8.1. But still.... what is the DV RPU METADATA file? Is that the second HEVC stream on UHD BD disc?


The RPU is just the metadata. Profiles 5 and 8.* don’t use a second encoded video layer. That’s just a legacy mode these days, as the dual-layer approach had a lot more bitrate and decode overhead.

Dual layer made a lot of sense when 8-bit H.264 encoders were standard, but with 10-bit HEVC it because a lot less necessary. Plus all the non-backwards compatible shaping used in Profile 5.


Sent from my iPad using Tapatalk

[benwaggoner]

Well, just did a test encode and it appears to be true



x265 [warning]: hevc-aq enabled, disabling other aq-modes



This is confusing syntax. We really should have aq-mode 4 and 5. aq-mode 4 would be HEVC-AQ, and 5 would be HEVC-AQ with low luma bias (like aq-mode 3 versus 2). The need for low-luma bias is content dependent, and we shouldn't be locked into one or another.



Also I am getting VBV violations multiple times a second. From the first two seconds of my test clip:



x265 [warning]: poc:0, VBV underflow (-2596808 bits)

x265 [warning]: poc:8, VBV underflow (-13025705 bits):06:32

x265 [warning]: poc:4, VBV underflow (-13047849 bits):38:49

x265 [warning]: poc:16, VBV underflow (-12824903 bits)11:58

x265 [warning]: poc:12, VBV underflow (-12960465 bits):23:11

x265 [warning]: poc:24, VBV underflow (-12939991 bits)27:49

x265 [warning]: poc:20, VBV underflow (-13441281 bits):34:43

x265 [warning]: poc:32, VBV underflow (-12651471 bits)23:47

x265 [warning]: poc:28, VBV underflow (-12985385 bits):28:15

x265 [warning]: poc:40, VBV underflow (-12559455 bits)20:36

x265 [warning]: poc:36, VBV underflow (-13122593 bits)24:13





Was the Experimental Feature tag left off for HEVC-AQ? I love the idea of improved AQ, but it would be good to have a clear indication in the help text about how close to production ready this feature is.

In talking with MCW, they confirmed that

A) The new HEVC-aq is in fact experimental, ala AQ-motion
B) It does override the specified AQ-mode, and there is no way to get low-luma bias ala AQ-mode 3 yet.
C) Rate control should be working. I used a high-grain clip with some atypical properties in my initial test (because AQ was messing up on it before), so if anything was going to give it pause, it was that one.


Sent from my iPad using Tapatalk

[MonoS]

i'm trying to encode a 4k hevc file using an DV Enhancement Layer, i have a few question in this regard:
Can i crop the input frame?
Can i see from a MediaInfo log that the DV Layer got muxed inside the stream?
Can i mux the resulting h265 file into a mkv container?
Do i need to set DV Profile 8.1 to mux the additional layer?

Regards.

[benwaggoner]

i'm trying to encode a 4k hevc file using an DV Enhancement Layer, i have a few question in this regard:
Can i crop the input frame?
Can i see from a MediaInfo log that the DV Layer got muxed inside the stream?
Can i mux the resulting h265 file into a mkv container?
Do i need to set DV Profile 8.1 to mux the additional layer?.
I am not sure what you are trying to do, but I am doubtful it would work :). Can you clearly define what your sources and desired outputs are? If you don’t already have an RPU file previously generated, you can’t do Profile 8.1. and there aren’t low-cost tools to generate one. ColorFront Transkoder is the most accesssble option I’m aware of.

Also, Profile 8.1 support isn’t common yet; the majority of existing Dolby Vision TVs in consumer hands today don’t support it. Profile 5 is what is universally supported.

The actual compression part is a relatively small and relatively straightforward part of making Dolby Vision content. Getting a properly shaped non-backwards compatible Y’CtCp source file and it’s metadata is the new and complex part.

Dolby Vision is a ways away from being something a consumer can make. It is totally feasible technically, but the tools just aren’t broadly available at this point.

[Forteen88]

I created a version for Windows 2.9+8. I almost doesn't change anything. Codec hasn't only 'threads' for VMAF as I wrote earlier.
https://www.sendspace.com/file/r90y0d
Probably it can also be created in MSVC.Could you please make a x265-VMAF version for the latest x265 (v3.0 RC4)?

[K.i.N.G]

x265 v3.0_RC+4-8aebc58efe5c (https://www.mediafire.com/file/l6e0rrj4q1c2r3c/) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit : GCC 7.4.0 / 64bit : GCC 8.2.1)

https://bitbucket.org/multicoreware/x265/commits/branch/default


Is quite a bit slower than current version i'm using (2.9.9)

Test Settings:
1080p source
--crf 18 --preset veryslow --profile main10 --level-idc 4 --output-depth 10 --ctu 32 --psy-rdoq 2.5 --tskip --aq-mode 3 --qcomp 0.7 --vbv-bufsize 20000 --vbv-maxrate 20000 --ipratio 1.35 --pbratio 1.25 --subme 7 --merange 64 --colormatrix bt709 --deblock -1:-1 --no-sao

x265 2.9.9 (64bit GCC 8.2.0) avg. speed: 0.66fps
x265 3.0 RC4 (64bit GCC 8.2.1) avg. speed: 0.40fps

[MonoS]

I am not sure what you are trying to do, but I am doubtful it would work :). Can you clearly define what your sources and desired outputs are?

sure i can, my apologies for not being clear from the start :)
I have a standard 4k bluray and i've extracted the tracks using the latest eac3to, so i have a source.mkv and dv_layer.h265 and i want to encode video and mux the additional Dolby Vision layer using the --dolby-vision-rpu option, but i don't know if the process is successful because i can't see any additional information when i generate a MediaInfo report.

I would like to crop the video stream as I'm encoding it then mux the resulting video into a mkv file.

TL;DR: how do i check that the muxing of the layer went fine? can i crop the picture? can i mux the resulting video stream to MKV?

[Ma]

x265 2.9.9 (64bit GCC 8.2.0) avg. speed: 0.66fps
x265 3.0 RC4 (64bit GCC 8.2.1) avg. speed: 0.40fps

New --preset slower == old --preset veryslow and new --preset veryslow is really new and really veryslow. See https://bitbucket.org/multicoreware/x265/commits/537bba0b7fdcbfc56ab4eb5315909c7a5cdb5a23?at=default

To exact compare please use in new version command line '--preset slower' instead of old '--preset veryslow'

[agressiv]

sure i can, my apologies for not being clear from the start :)
I have a standard 4k bluray and i've extracted the tracks using the latest eac3to, so i have a source.mkv and dv_layer.h265 and i want to encode video and mux the additional Dolby Vision layer using the --dolby-vision-rpu option, but i don't know if the process is successful because i can't see any additional information when i generate a MediaInfo report.

I would like to crop the video stream as I'm encoding it then mux the resulting video into a mkv file.

TL;DR: how do i check that the muxing of the layer went fine? can i crop the picture? can i mux the resulting video stream to MKV?

MKV doesn't support Dolby Vision (at least, not yet) - it won't work. Your best bet is to use the Dolby MP4 muxer and use ac3 audio.

[benwaggoner]

MKV doesn't support Dolby Vision (at least, not yet) - it won't work. Your best bet is to use the Dolby MP4 muxer and use ac3 audio.
Yeah, .mkv isn’t a professional format. Pretty much all professional development around containers these days is around mp4, and to a lesser and declining degree, MPEG transport streams.

[Barough]

x265 v3.0_RC+10-672ce0547e97 (https://www.mediafire.com/file/no3410zfz9ar6bo/) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit : GCC 7.4.0 / 64bit : GCC 8.2.1)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[MonoS]

MKV doesn't support Dolby Vision (at least, not yet) - it won't work. Your best bet is to use the Dolby MP4 muxer and use ac3 audio.

So what's the purpose of the --dolby-vision-rpu option?
If i understood correctly it should mux the DV metadate directly inside the h265 stream (and studying the relevant commit it seems to include such information inside an 0x3E NAL unit) so why i shouldn't be able to mux the resulting stream inside an mkv?

[agressiv]

So what's the purpose of the --dolby-vision-rpu option?
If i understood correctly it should mux the DV metadate directly inside the h265 stream (and studying the relevant commit it seems to include such information inside an 0x3E NAL unit) so why i shouldn't be able to mux the resulting stream inside an mkv?

DV metadata for x265, but not necessarily the elementary h265 stream from a blu-ray.

We have yet to see a clip which works with --dolby-vision-rpu. And once we get it, we can only use .ts or .mp4, not .mkv.

If you have one, we're all ears, but nothing from a UHD blu-ray works with that parameter with x265.

[LigH]

:o DV is still "Digital Video" for me, not "Dolby Vision", so I keep wondering about metadata...

[excellentswordfight]

@Ma

I asked some questions regarding mereange a while ago but didnt get a reply, so I thought that I give it another shot.

I've seen discussions here and in other threads that CTU 64 is overkill for resolutions bellow 4k, and I've seen from my own testing that lowering CTU from 64 to 32 on systems with plenty of threads gives an speed improvment of up to 50% for 1080p video, and lowering merange gives another 10%. And this is with a very minor compression hit. This behavior is also stated in this document https://media.readthedocs.org/pdf/x265/default/x265.pdf when it comes to threading performance.

What I find a bit odd is that this is stated in the document: "Given these considerations, you can understand why the faster presets lower the max CTU size to 32x32 (making
twice as many CTU rows available for WPP and for finer grained frame parallelism) and reduce --merange" and this: "The default is derived from the default CTU size (64) minus the luma interpolation half-length (4) minus maximum subpel distance (2) minus one extra pixel just in case the hex search method is used."

But I cant see that any preset changes the merange value of 57, even the two fastes ones that do lower the CTU value to 32. How come? And since lowering CTU (and merange), can have such massive influence on speed, wouldn't be a good idea to have these values set based on resolution?

And giving the explanation of the default merange value, would the same calculation stand when lowering CTU to say 32? I.e. would that give an "best practice" value of 26 if me star is used?

[MonoS]

We have yet to see a clip which works with --dolby-vision-rpu. And once we get it, we can only use .ts or .mp4, not .mkv.

I thought that the Dolby Video Enhancement Layer available in commercially available BDs where the RPU information, if i'm understanding correctly this is not the case, but x265 wasn't giving me any error (like you can see here https://bitbucket.org/multicoreware/x265/commits/e50f803e26fb3926dc695e0aeea39681fe1eacbd#Lsource/x265.cppT587 ) so i expected to be doing things right.

:o DV is still "Digital Video" for me, not "Dolby Vision", so I keep wondering about metadata...

It won't happen again ;)

[benwaggoner]

I thought that the Dolby Video Enhancement Layer available in commercially available BDs where the RPU information, if i'm understanding correctly this is not the case, but x265 wasn't giving me any error (like you can see here https://bitbucket.org/multicoreware/x265/commits/e50f803e26fb3926dc695e0aeea39681fe1eacbd#Lsource/x265.cppT587 ) so i expected to be doing things right.
I think the older discs were mainly Profile 5, where the enhancement layer includes both quarter-scale video and metadata. x265 doesn't support that. Newer discs would be Profile 5, where the enhancement layer is just RPU metadata. But that requires a properly "shaped" base layer specific to the source and metadata, which is Y'CtCp and dynamically adjusts to the currently-used subset of the PQ curve for greater precision.

It won't happen again ;)[/QUOTE]
I like DoVi as the short way to type it. I am an old fellow, and I hear DV I start thinking "Ah, 25 Mbps with 720x480 4:1:1 color in its NTSC variant." It's amazing what we can do with 25 Mbps a couple decades later!

[MonoS]

I think the older discs were mainly Profile 5, where the enhancement layer includes both quarter-scale video and metadata. x265 doesn't support that. Newer discs would be Profile 5, where the enhancement layer is just RPU metadata. But that requires a properly "shaped" base layer specific to the source and metadata, which is Y'CtCp and dynamically adjusts to the currently-used subset of the PQ curve for greater precision.


I think that what i'm talking about it's BDs with DoVi 7.6 (as per this document https://www.dolby.com/us/en/technologies/dolby-vision/dolby-vision-profiles-levels.pdf ) so standard 4k video content and additional enhancement layer

[Ma]

I've made a test with --merange 12/26/40/57/74/92 for --ctu 64 --qg-size 64 vs. --ctu 32 --qg-size 32

Command line:
for %m in (12 26 40 57 74 92) do (
x265 -p7 --bitrate 500 -f3333 --psnr --ssim --qg-size 64 --merange %m ../big_buck_bunny_1080p24.y4m w64-%m.hevc
x265 -p7 --bitrate 500 -f3333 --psnr --ssim --ctu 32 --merange %m ../big_buck_bunny_1080p24.y4m w32-%m.hevc
)

Results (in ctu block: speed (fps), PSNR, SSIM (dB)):

ctu
merange | 64 | 32
12 | 6.93 39.980 12.996 | 6.44 39.565 12.742
26 | 6.69 40.303 13.599 | 6.21 39.903 13.345
40 | 6.46 40.393 13.746 | 6.07 39.999 13.499
57 | 6.13 40.404 13.757 | 5.90 40.019 13.512
74 | 5.85 40.409 13.757 | 5.67 40.024 13.515
92 | 5.58 40.413 13.760 | 5.47 40.026 13.514

When --merange grows, the quality increases regardless of the size of --ctu (it may depend on the source movie).

For CPU with only 12 logical cores (6 physical) --ctu 64 is just better in preset slower for 1080p encoding.

[Motenai Yoda]

@Ma can you make results about --ctu 64 --qg-size 32 and --ctu 32 --qg-size 16 too?
maybe just for --merange 40 only

[jlpsvk]

why the hell was default aq-mode changed to 2 from 1???? my all encodes are bad now. :(

[nevcairiel]

If you rely on a specific setting that strongly, you should not rely on it being the default, and just specify it. Or as an alternative, not blindly update your encode pipeline without verifying that it still produces expected results.
For the record, the default was changed from 1 to 2 (https://bitbucket.org/multicoreware/x265/commits/b14834a9d1c1864ea7e94d9cfed4e33f37e767c6), not the other way around.

[Boulder]

If you rely on a specific setting that strongly, you should not rely on it being the default, and just specify it. Or as an alternative, not blindly update your encode pipeline without verifying that it still produces expected results.
For the record, the default was changed from 1 to 2 (https://bitbucket.org/multicoreware/x265/commits/b14834a9d1c1864ea7e94d9cfed4e33f37e767c6), not the other way around.

That's the reason why I have a long string of options in the command line even if most of them are the defaults of a specific preset.

[nevcairiel]

Personally I like the second option. Not blindly update, but validate new versions first.

[Selur]

Still waiting for them to update the documentation https://x265.readthedocs.io/en/latest/presets.html,..
Opened an issue entry for it (https://bitbucket.org/multicoreware/x265/issues/460/update-documentation) at the end of last year.
I get it that they change the defaults(/presets/tune) from time to time, but not updating the documentation do represent such fundamental changes is really annoying,...

[Boulder]

And a funny thing is that --no-rskip is supposed to be a default in --preset veryslow (and also --slower) but it's not according to the code. I reported the issue a long time ago but no one fixed it :) Doesn't matter to me though, I use --rskip all the time anyway.

[Selur]

@Boulder: yeah, looking at https://bitbucket.org/multicoreware/x265/src/default/source/common/param.cpp?at=default rskip is only disabled in placebo,
but the documentation:
- https://x265.readthedocs.io/en/latest/presets.html reports that is is disabled starting with slower.
- https://x265.readthedocs.io/en/default/presets.html reports that is is disabled starting with very slow.
-> wrong documentation is even worse than missing documentation :(
(same for https://bitbucket.org/multicoreware/x265/src/default/source/common/param.cpp?at=latest)

[jlpsvk]

ok... can you tell me why aq mode 2 is better? i noticed rapid lower bitrate with aq-mode 2... any impact on picture quality and fine detail retention? i am encoding with crf18

[Boulder]

ok... can you tell me why aq mode 2 is better? i noticed rapid lower bitrate with aq-mode 2... any impact on picture quality and fine detail retention? i am encoding with crf18

I didn't find mode 2 better than 1 in my test cases (reported earlier here in this thread) so I'm still using mode 1 with the default strength.

I think you need to find a new sweet spot for CRF since the bitrate demand is indeed quite different compared to mode 1, and I also think it depends heavily on the content.

[Ma]

@Ma can you make results about --ctu 64 --qg-size 32 and --ctu 32 --qg-size 16 too?
maybe just for --merange 40 only

Results in attachment -- it is a bit faster encoding but quality is worse (at the same bitrate).

[excellentswordfight]

I've made a test with --merange 12/26/40/57/74/92 for --ctu 64 --qg-size 64 vs. --ctu 32 --qg-size 32

Command line:
for %m in (12 26 40 57 74 92) do (
x265 -p7 --bitrate 500 -f3333 --psnr --ssim --qg-size 64 --merange %m ../big_buck_bunny_1080p24.y4m w64-%m.hevc
x265 -p7 --bitrate 500 -f3333 --psnr --ssim --ctu 32 --merange %m ../big_buck_bunny_1080p24.y4m w32-%m.hevc
)

Results (in ctu block: speed (fps), PSNR, SSIM (dB)):

ctu
merange | 64 | 32
12 | 6.93 39.980 12.996 | 6.44 39.565 12.742
26 | 6.69 40.303 13.599 | 6.21 39.903 13.345
40 | 6.46 40.393 13.746 | 6.07 39.999 13.499
57 | 6.13 40.404 13.757 | 5.90 40.019 13.512
74 | 5.85 40.409 13.757 | 5.67 40.024 13.515
92 | 5.58 40.413 13.760 | 5.47 40.026 13.514

When --merange grows, the quality increases regardless of the size of --ctu (it may depend on the source movie).

For CPU with only 12 logical cores (6 physical) --ctu 64 is just better in preset slower for 1080p encoding.
I see, thanks for sharing.

I think you are correct, since the biggest benefit of reducing it only appears when going above arround 8C/16T (I was testing with 12/24 and went from 7fps to 11fps cause of the less thread utilization at CTU 64, and I would say that it was definitely was worth the trade off), it does makes since to keep it as an manual setting.

I would say though that it might be worth consideration to lower the merange for those fastets presets, since it doesnt seem to benefit quality that much.

[LigH]

I like DoVi as the short way to type it.

That might sound stupid to a German (Doofi ~ retarded kid) :sly:

a couple decades later!

:eek: I am old!
_

Some on-topic:

Support for Dolby Vision is announced, and immediately people believe that applications for decoding, possibly even encoding, for consumer PCs are available too ... which is doubtful. I wonder how "homeopathic" its additional features will be in a generic furnished living room, where average people can hardly tell the difference between DD AC3 and dts on a DVD Video, not to mention HD Audio formats.

To me, it seems that supporting such data is mainly for professional content producers. Might be a field where hobbyists can't help the developers much.

[sneaker_ger]

Support for Dolby Vision is announced, and immediately people believe that applications for decoding, possibly even encoding, for consumer PCs are available too ... which is doubtful. I wonder how "homeopathic" its additional features will be in a generic furnished living room, where average people can hardly tell the difference between DD AC3 and dts on a DVD Video, not to mention HD Audio formats.
Oh, I'm sure the difference will be noticable to consumers. Simply because authors will make it so, not because it's superior. Just like in the past authors used different mixes for AC3 and DTS tracks on the same DVD. It's all about marketing. :devil:

[jd17]

Oh, I'm sure the difference will be noticable to consumers. Simply because authors will make it so, not because it's superior.

That.

[Lucius Snow]

Hello all,

There's still no GPU support to speed up the encoding?

Thanks.

[LigH]

No. I believe because GPU features would not speed x265 up. At least not without risking a loss of quality or losing the independence from hardware and software platforms (portability).

GPU features are not magical general speed-ups for every case of use, sometimes they just don't match the requirements.

[jlpsvk]

Ok. Did a test. 3840x2160 res., same x265 settings. One with aq-mode 1 crf18 and second aq-mode 2 (which is now new default) with crf16. aq-mode 2 with crf16 still gives lower bitrate. which one "should" be better in terms of detail retention?

whole CLI (HDR parameters are added automatically by RipBot264):
-preset slow --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --aq-mode 1 --amp --no-rskip --qg-size 8 --vbv-bufsize 160000 --vbv-maxrate 160000 --bframes 8
--rc-lookahead 48 --gop-lookahead 30 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --allow-non-conformance --no-strong-intra-smoothing --high-tier --asm avx512 --crf 18

-preset slow --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --aq-mode 2 --amp --no-rskip --qg-size 8 --vbv-bufsize 160000 --vbv-maxrate 160000 --bframes 8
--rc-lookahead 48 --gop-lookahead 30 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --allow-non-conformance --no-strong-intra-smoothing --high-tier --asm avx512 --crf 16

[asarian]

Oh, I'm sure the difference will be noticable to consumers. Simply because authors will make it so, not because it's superior. Just like in the past authors used different mixes for AC3 and DTS tracks on the same DVD. It's all about marketing. :devil:

Having an HD Amp myself, trust me, you can hear de difference! :) Try listening to Blade Runner, the Final Cut, with full HD audio, or the same movie, just with DD5.1, and I guarantee you that everything below HD audio will sound crap from there on in. :)

[jd17]

Having an HD Amp myself, trust me, you can hear de difference! :) Try listening to Blade Runner, the Final Cut, with full HD audio, or the same movie, just with DD5.1, and I guarantee you that everything below HD audio will sound crap from there on in. :)

That is not a very good argument against what sneaker said.

How do you know both mixes are based on the same mastering?
How do you know the DD-mix is not willingly different/worse than the HD mix?
Are you comparing identical speaker layouts (both just 5.1)?
Are you sure there is no dynamic compression in the DD-mix?
Are the volume levels identical?
Is the DD-mix limited in bitrate (384/448) or does it use the full 640kbit/s potential?

If you want a comparison of codec audio quality only, try encoding your own DD-mix from the HD-audio stream, considering all the factors above.
Even then, you cannot be sure in case of AC-3, because people claim that the original, commercial encoder is superior to those available freely... ;)

[asarian]

That is not a very good argument against what sneaker said.

How do you know both mixes are based on the same mastering?
How do you know the DD-mix is not willingly different/worse than the HD mix?
Are you comparing identical speaker layouts (both just 5.1)?
Are you sure there is no dynamic compression in the DD-mix?
Are the volume levels identical?
Is the DD-mix limited in bitrate (384/448) or does it use the full 640kbit/s potential?

If you want a comparison of codec audio quality only, try encoding your own DD-mix from the HD-audio stream, considering all the factors above.
Even then, you cannot be sure in case of AC-3, because people claim that the original, commercial encoder is superior to those available freely... ;)

Blade Runner is just one example. Let's not derail this thread too much, but I can assure you, that once you go HD audio, you'll never want to go back, ever. And why should that surprise anyone?! A typical DTS-MA/TrueHD track is often larger in size than an entire DVD! With bitrates around 10x higher as regular DTS, the superiority of HD audio is a no-brainer, far as I'm concerned.

[excellentswordfight]

And why should that surprise anyone?!
Cause 48Khz 16bit audio is very much sufficient for human hearing, and if you have enough experience of audio compression you know that there is several codecs that can achive very close to transperent audio with say 64kbps per channel (and most formats are above that!). I've encoded several surround tracks form lossless to multiple formats, you be surprised how little it affects hearable fidelity.

With that said, I'm sure that the HD-track sounds better for you, but most of that doesnt come from the higher specs (sample rate, bit depth and losslessness)

[jd17]

With bitrates around 10x higher as regular DTS, the superiority of HD audio is a no-brainer, far as I'm concerned.

Nobody questions that lossless - in theory, or better measurably - is always better.
However, this is a forum that pretty much evolves around compression efficiency, i.e. saving bits where we don't see or hear it.

This is why I (and plenty others here too) take these lossless audio streams and compress them to 200-600kbit/s AAC or opus.

And while these codecs are obviously far superior to the ancient AC-3, you'd be surprised how good 640kbit/s Dolby Digital really is.
Accordingly, if the same master (and volume) is used, you would most likely fail in telling TrueHD apart from Dolby Digital, if you don't know which is playing. ;)

[asarian]

Nobody questions that lossless - in theory, or better measurably - is always better.
However, this is a forum that pretty much evolves around compression efficiency, i.e. saving bits where we don't see or hear it.

This is why I (and plenty others here too) take these lossless audio streams and compress them to 200-600kbit/s AAC or opus.

And while these codecs are obviously far superior to the ancient AC-3, you'd be surprised how good 640kbit/s Dolby Digital really is.
Accordingly, if the same master (and volume) is used, you would most likely fail in telling TrueHD apart from Dolby Digital, if you don't know which is playing. ;)

'CD Quality' is 44.1 KHz/16 bit audio, generally considered 'very good' for music. Movies are a different thing, though. AC3 (aka DD 5.1) yields reasonable results at (almost always) 640kbps. But the difference between DTS-MA/TrueHD vs. AC3 is, well, unimaginably high. Believe me, I was in the same camp as you, one day, thinking you wouldn't be able to tell the difference... until I got an actual HD Amp. :) The fullness of the track when you switch to HD audio is staggering -- to the point where I can barely bear to listen to AC3 any more. In fact, for every Blu-ray I order (unless it's some sort of vintage deal), I first check to see whether it comes with HD Audio. If not, I simply don't buy it.

I'm not familiar with compressing audio 'to 200-600kbit/s AAC or opus' myself, so I'll take your word on those codecs.

[nevcairiel]

But the difference between DTS-MA/TrueHD vs. AC3 is, well, unimaginably high.

Except on a technical level it really is not.

At least DTS has a "core" you can use for a direct comparison, since it has to be from the same master. If you were to decode both the core and the full HD stream to PCM and send that PCM to your "HD Amp", I'm positive that in most cases you wouldn't even be able to tell the difference. What these devices do is cheat you by playing with volume and EQ settings that play towards how people perceive audio. Just a slight bit more volume for HD, and most people already perceive it as "better", and there is more such tricks.

Or take a HD track and re-encode it as AC3, just to ensure its the same master, and then decode both back to PCM and send that to the Amp so the Amp does not know what the original format was.

There is a lot of trickery to try to sell you on "HD" stuff, because they had to sell you something, and in fact the audio quality difference have been minimal for years. If one really goes deep into it on a technical level, you'll eventually find that out. If you just blindly trust the Amp, then sure, HD probably sounds better to you, but not because its HD audio, but because the Amp cheats you.

[Atak_Snajpera]

Except on a technical level it really is not.

At least DTS has a "core" you can use for a direct comparison, since it has to be from the same master. If you were to decode both the core and the full HD stream to PCM and send that PCM to your "HD Amp", I'm positive that in most cases you wouldn't even be able to tell the difference. What these devices do is cheat you by playing with volume and EQ settings that play towards how people perceive audio. Just a slight bit more volume for HD, and most people already perceive it as "better", and there is more such tricks.

Or take a HD track and re-encode it as AC3, just to ensure its the same master, and then decode both back to PCM and send that to the Amp so the Amp does not know what the original format was.

There is a lot of trickery to try to sell you on "HD" stuff, because they had to sell you something, and in fact the audio quality difference have been minimal for years. If one really goes deep into it on a technical level, you'll eventually find that out. If you just blindly trust the Amp, then sure, HD probably sounds better to you, but not because its HD audio, but because the Amp cheats you.

That whole TrueHD/Lossless audio topic reminds of 192KHz 24bit bullshit promoted by some well known companies. (Super Audio-CD , DVD-Audio and others). Facts are simple. 48Khz 16bit is more than enough for humans. Adults have hearing range up to ~18KHz ,so it still below 24Khz. The same story with bit depth.
TrueHD can sounds better than AC3 640kbps only if was encoded from better source or your amp is doing some tricks boosting artificially volume for some frequencies. (bass/trebles). If 128kbps OPUS/AAC is transparent for 2.0 then I see no reason why 320kbps wouldn't be enough for 5.1.

[jd17]

Believe me, I was in the same camp as you, one day, thinking you wouldn't be able to tell the difference... until I got an actual HD Amp. :) The fullness of the track when you switch to HD audio is staggering -- to the point where I can barely bear to listen to AC3 any more.

I get the feeling that you do not really understand where we are coming from...
We don't doubt you hear a difference!
But that difference is not based on codec superiority, but mastering, volume, dynamic compression and so forth.

However, we are turning in circles here, it's all been said already. You either do some legwork yourself and compare apples with apples, or you keep believing. :)

BTW, lossless-capable AVRs have been around for at least 10 years... Do you really think you are the only person here who owns one?


Sorry for the off-topic loop, let's get back to x265. :)

[benwaggoner]

No. I believe because GPU features would not speed x265 up. At least not without risking a loss of quality or losing the independence from hardware and software platforms (portability).

GPU features are not magical general speed-ups for every case of use, sometimes they just don't match the requirements.
I could see a 20-25% speedup for high quality encoding using GPU and fixed-function encoder features with the new, deeper Intel encoder/decoder APIs for preanalysis and such. Not because it would natively produce high quality output, but because it could help determine some optimal encoding parameters.

[benwaggoner]

Nobody questions that lossless - in theory, or better measurably - is always better.
I'll exactly argue against that. Information is a difference that makes a difference. A difference people can't discriminate in a proper double-blind test isn't a difference that matters.

Lossless or near lossless for sources makes sense, to the degree the extra information can eventually result in a detectable difference in derived content. Lossless, if used at all, is only used for archiving deep in studios.

[jd17]

I agree with you. That's why I inserted "in theory". ;)

[Selur]

Question regarding:
--qp-adaptation-range
is this a suboption of '-aq-mode 1+', '--hevc-aq', both, or is it independent of both?

Cu Selur

[benwaggoner]

Question regarding:
--qp-adaptation-range
is this a suboption of '-aq-mode 1+', '--hevc-aq', both, or is it independent of both?

I believe it only applies to --hevc-aq.

[kosta1000]

hi guys. Where can I find changelog for x265 3.0 ???

on official site, changelog exists only for version 2.9

another question : When Ryzen 2 get out around this June, would this processor have much better performance in x265 encoding, while AMD put 256-bit AVX2 now, along with many other notable improvments.

[Selur]

hi guys. Where can I find changelog for x265 3.0 ???
Probably hasn't been written yet,... iirc there hasn't been a 3.0 final/stable flag last tag was '3.0_RC'.

[benwaggoner]

Probably hasn't been written yet,... iirc there hasn't been a 3.0 final/stable flag last tag was '3.0_RC'.
The best bet for the moment is to read the documentation that got added to —fullhelp in the checking.

https://bitbucket.org/multicoreware/x265/commits/branch/stable

[agressiv]

Looks like 3.0 is finally released! Congrats to the team! Here are the release notes:

New features
-------------
1. option:: '--dolby-vision-profile <integer|float>' generates bitstreams confirming to the specified Dolby Vision profile. Currently profile 5, profile 8.1 and profile 8.2 enabled, Default 0 (disabled)

2. option:: '--dolby-vision-rpu' File containing Dolby Vision RPU metadata. If given, x265's Dolby Vision metadata parser will fill the RPU field of input pictures with the metadata
read from the file. The library will interleave access units with RPUs in the bitstream. Default NULL (disabled).

3. option:: '--zonefile <filename>' specifies a text file which contains the boundaries of the zones where each of zones are configurable.

4. option:: '--qp-adaptation-range' Delta-QP range by QP adaptation based on a psycho-visual model. Default 1.0.

5. option:: '--refine-ctu-distortion <0/1>' store/normalize ctu distortion in analysis-save/load. Default 0.

6. Experimental feature option:: '--hevc-aq' enables adaptive quantization
It scales the quantization step size according to the spatial activity of one coding unit relative to frame average spatial activity. This AQ method utilizes
the minimum variance of sub-unit in each coding unit to represent the coding unit’s spatial complexity.

Encoder enhancements
--------------------
1. Preset: change param defaults for veryslow and slower preset. Replace slower preset with defaults used in veryslow preset and change param defaults in veryslow preset as per experimental results.
2. AQ: change default AQ mode to auto-variance
3. Cutree offset reuse: restricted to analysis reuse-level 10 for analysis-save -> analysis-load
4. Tune: introduce --tune animation option which improves encode quality for animated content
5. Reuse CU depth for B frame and allow I, P frame to follow x265 depth decision

Bug fixes
---------
1. RC: fix rowStat computation in const-vbv
2. Dynamic-refine: fix memory reset size.
3. Fix Issue #442: linking issue on non x86 platform
4. Encoder: Do not include CLL SEI message if empty
5. Fix issue #441 build error in VMAF lib

[Selur]

4. option:: '--qp-adaptation-range' Delta-QP range by QP adaptation based on a psycho-visual model. Default 1.0.
6. Experimental feature option:: '--hevc-aq' enables adaptive quantization

Sadly still unkown how they are related to the normal aq-modes,..

[benwaggoner]

Sadly still unkown how they are related to the normal aq-modes,..
--hevc-aq overrides whatever --aq-mode is set to. I suspect that --aq-strength may get overridden itself, with --aq-adaption-range being the equivalent. Or both parameters could be used together, ala the interaction of CRF with maxrate/bufsize.

[brumsky]

@excellentswordfight

I've come to the same conclusion as well. merange should be based off of the CTU. If you leave CTU at 64 then merange should also be reduced. It'll be searching outside of it's original block. Not to say that is bad necessarily though. I've seen certain "high quality" encodes that use meranges larger than the CTU. I'm not 100% sold that it provides tangible differences thoughs.

I personally drop merange to 26 when using CTU 32. Since I never plan on using hex search I should probably change merange to 58 when using CTU64...

Best way to find out is to try it for yourself.

[brumsky]

another question : When Ryzen 2 get out around this June, would this processor have much better performance in x265 encoding, while AMD put 256-bit AVX2 now, along with many other notable improvments.

I hope it does. I also hope AMD allows them to be fused like they current do with two 128-bit AVX FPUs that can run one 256-bit AVX2. If they do this then they will have the lead in AVX512!

Intel's top of the line 18 core CPU only has 2 AVX512 units on it. Imagine a Zen 2 CPU with up to 8 AVX512 fused units!!! I know there is an extra cycle or two when doing a fused operation but still up to 8 AVX512 operations will be nice!

[Selur]

--hevc-aq overrides whatever --aq-mode is set to.
okay
I suspect that --aq-strength may get overridden itself, with --aq-adaption-range being the equivalent. Or both parameters could be used together, ala the interaction of CRF with maxrate/bufsize.
And here it get's confusing to me. :)
Questions are:
a. is '--aq-strength' an option for both '--aq-mode' and '--hevc-aq' or just '--aq-mode'?
b. is '--aq-adaption-range' an option for both '--aq-mode' and '--hevc-aq' or just '--hevc-aq'?

Cu Selur

[nevcairiel]

I know there is an extra cycle or two when doing a fused operation but still up to 8 AVX512 operations will be nice!

Other then compatibility, that really doesn't offer anything. AVX/AVX2 instructions making use of those 256-bit units would be about the same speed.

256-bit AVX on current Ryzen isn't that much faster then 128-bit SSE due to that.

In any case, there have been zero hints about AVX512 support.

[benwaggoner]

Other then compatibility, that really doesn't offer anything. AVX/AVX2 instructions making use of those 256-bit units would be about the same speed.

256-bit AVX on current Ryzen isn't that much faster then 128-bit SSE due to that.

In any case, there have been zero hints about AVX512 support.
And so far, AVX512 in x265 has only been demonstrated to be helpful on Intel systems doing UHD at slower+ presets; it'll make things slower for other scenarios on existing processors. That's why it is off by default even on a system with AVX512 support. AVX & AVX2 are always used if there is hardware support because they help significantly most of the time, and I don't know of any cases where they hurt.

Generally the value of AVX? instructions have improved over time, as microarchitecture improvements help with thermal throttling and other bottlenecks.

[benwaggoner]

okay

And here it get's confusing to me. :)
Questions are:
a. is '--aq-strength' an option for both '--aq-mode' and '--hevc-aq' or just '--aq-mode'?
b. is '--aq-adaption-range' an option for both '--aq-mode' and '--hevc-aq' or just '--hevc-aq'?

It is definitely not b. So either a, or:

c. --aq-strength is an parameter for --aq-mode, and --aq-adaption-range is a parameter for --hevc-aq, and neither is used when the other aq type is used.

[Boulder]

And so far, AVX512 in x265 has only been demonstrated to be helpful on Intel systems doing UHD at slower+ presets; it'll make things slower for other scenarios on existing processors. That's why it is off by default even on a system with AVX512 support. AVX & AVX2 are always used if there is hardware support because they help significantly most of the time, and I don't know of any cases where they hurt.

Generally the value of AVX? instructions have improved over time, as microarchitecture improvements help with thermal throttling and other bottlenecks.

At least with a first generation Ryzen, you'll want to disable AVX2 in the x265 command line. It's slightly faster that way.

[Stereodude]

At least with a first generation Ryzen, you'll want to disable AVX2 in the x265 command line. It's slightly faster that way.
The Zen 2 architecture doubles the width of the datapath and the execution units to 256-bits which should double the performance of AVX2 on it vs. Zen.

The whole FPU got supersized in Zen 2 (vs. 1).

2x wider datapath (256-bit, up from 128-bit)
2x wider EUs (256-bit FMAs, up from 128-bit FMAs)
2x wider LSU (2x256-bit L/S, up from 128-bit)

from: https://en.wikichip.org/wiki/amd/microarchitectures/zen_2

[NikosD]

Another important advantage of Zen 2 architecture and its implementation of Ryzen 3000 series, will be the CPU clock during heavy execution of AVX2 instructions, like running x265 app.

If AMD has been interpreted correctly, we will see no performance penalty due to lower clocks during x265 AVX2 code execution.

Intel sees lower clocks leveraging AVX2 instructions of x265 with all CPU architectures, so far.

I think Ryzen 3000 (and Threadripper, EPYC) based on Zen 2 architecture has all the benefits to be a lot faster than any Intel CPU ever released with the same number of cores.

And due to the fact that all AMD CPUs have more cores than Intel nowadays, then x265 could be a killer app for AMD, like Cinebench.

[chinobino]

x265-v3.0+1-ed72af837053 (https://drive.google.com/open?id=1Hj4yBUrk_w8yVkkmWFib1gWcGDWeQS2H) [ICC 1900][64 bit]

Redistributable Libraries for Intel® C++ (https://software.intel.com/sites/default/files/managed/72/e7/ww_icl_redist_msi_2019.1.144.zip)

x265-v3.0+1-ed72af837053-multilib (https://drive.google.com/open?id=1rGceIuQCOnGsjpnAMEBfCMNnITmIBLJt) [GCC 8.2.1][64 bit]

Perfect, thankyou.

[hajj_3]

HEVC licensing info article: http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/A-Video-Codec-Licensing-Update-129386.aspx

[WhatZit]

HEVC licensing info article: http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/A-Video-Codec-Licensing-Update-129386.aspx

"Tragedy Of The Commons” a.k.a. "Stake The Velos Vampires"

[Barough]

x265 v3.0_RC+13-ae085e5cd8a2 (http://www.mediafire.com/file/pl12i5umh5sck83/) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit : GCC 7.4.0 / 64bit : GCC 8.2.1)

https://bitbucket.org/multicoreware/x265/commits/branch/default

NOTE :
Checked with Pradeep (@MulticoreWare) about why the Default Branch haven't been pushed to v3.0 'Stable' and this is the reply/info i got

"
Our plan is to continue to use 3.0_RC on the default branch and have completed tags only on the stable branch. So we don't intend to merge back.
"

[NikosD]

This is only true on server CPU. There is no separate frequency for AVX on client CPU (7700K, 8700K, 9900K, etc.). The extent to which you may see "lower clocks" on non-server is if you are encoding faster and running into the 65/95 W power limit, and in that case AMD will be no different. But this is exactly the case for AMD, that they will not hit that power limit because of better efficiency/ architecture of Zen 2 implementing AVX2 instructions than any Intel architecture so far.

Probably 7nm could help too, keeping the same clocks for AVX2 like all the other instruction sets.

[StvG]

And so far, AVX512 in x265 has only been demonstrated to be helpful on Intel systems doing UHD at slower+ presets...

A simple test with 4K video downscaled to 1080p with avs+ and passed to x265 with avs2yuv, preset slower + ctu 32, AVX2@4500, AVX512@4500:
AVX2 - 7.08 fps
AVX512 - 7.87 fps

Another 1080p encoding with the same preset slower + ctu 32:
AVX2 - 7.68 fps
AVX512 - 8.08 fps

Also:
AVX2 ~ 290W
AVX512 ~ 250W

I'm using adaptive offset for vcore. So my vcore is 1.24v for @4800 (non-avx) and when encoding with AVX2 my core speed is @4500 but vcore remains the same 1.24v. When encoding with AVX512 my core speed is @4500 and vcore is 1.13v.

[nevcairiel]

AVX-512 only reduces performance on server chips because of artificial frequency limits that Intel imposes. After the downclock, x265 actually draws less power and runs cooler, showing how it was unnecessary. Hopefully the next generation of server improves the power management algorithm so downclocking will not be needed.

On Skylake-X/WS and Cannon Lake, AVX-512 only ever increases performance. It will presumably also be the case on Ice Lake.

Actually the offset is needed to maintain stability. I know, because I have a 7900X, and tried to get the best out of it.
You wouldn't notice this problem with x264 or x265, because its AVX512 usage is pretty "light", but if you run some heavy AVX512 tasks on all cores, and don't configure an appropriate offset, the chip just crashes. The energy density of the AVX512 units is just too high for running at full turbo clocks, nevermind OCed.

If x265 is the only AVX512 you ever run, and you want to risk it, sure, you can disable the offset and hope that it never happens. But I prefer to know that my system is stable no matter what software does. :)
But be careful, and do know that you can not judge the requirement from one pretty lightweight workload.

The only way the offset is getting lower is when the cores get more efficient, which they really only do on a process shrink. So hopefully that'll significantly reduce the AVX512 offset, even if I don't expect it to go away quite just yet.

This is easily testable by anyone with such a chip. For example, recent versions of the Intel LINPACK floating-point benchmark will put enough AVX-512 stress on the CPU to cause this.

[excellentswordfight]

A simple test with 4K video downscaled to 1080p with avs+ and passed to x265 with avs2yuv, preset slower + ctu 32, AVX2@4500, AVX512@4500:
AVX2 - 7.08 fps
AVX512 - 7.87 fps

Another 1080p encoding with the same preset slower + ctu 32:
AVX2 - 7.68 fps
AVX512 - 8.08 fps

Also:
AVX2 ~ 290W
AVX512 ~ 250W

I'm using adaptive offset for vcore. So my vcore is 1.24v for @4800 (non-avx) and when encoding with AVX2 my core speed is @4500 but vcore remains the same 1.24v. When encoding with AVX512 my core speed is @4500 and vcore is 1.13v.
I think he is refering to this https://software.intel.com/en-us/articles/accelerating-x265-with-intel-advanced-vector-extensions-512-intel-avx-512

And his statment is true for xeons. I did some tests on a Xeon Gold 6126 and even got lower performance for 2160p preset slow, clockspeeds down almost 20%, while gains of running avx512 gave maybe 10%. OCd X299 platforms are a niche (altough maybe not here).

[NikosD]

That statement makes no sense. If AVX2 does not "hit that power limit," that means that the frequency for non-AVX was too low. As long as it can reach a higher throughput with AVX, that means that either AVX will reduce frequency or non-AVX will be underutilized. But it does make sense in the context of real-world code optimization/execution.

Do you really hear from me for the first time that most non-SIMD code can't utilize a modern CPU in the way that SIMD code can ?

The only way to reach TDP limits of a modern CPU is from optimized SIMD code.

There are other limits to reach before power limits for non-AVX code.

[NikosD]

Non-AVX code can and does reach power limits. SKX 28-core can easily reach 165 W in p95 non-AVX. RyZen 2000 hits around 150 W in p95 non-AVX with power limits disabled (which implies throttling with 95 W power limit enforced). Anyway, SIMD code existed before AVX2, like MMX, SSE, SSE2, SSE3, SSSE3, SEE4.x etc.

For all those SIMD instruction sets, we never had lower clocks not even for AVX1.

AVX2 is even denser than all of those sets, but that doesn't mean that there is no architecture with no performance hit.

Zen 2 could be the first one.

[katzenjoghurt]

Oh my.
Am I the only one having such a hard time encoding scenes with red light / red backgrounds?

E.g. if a character turns on a red light, his face would turn suddenly totally blocky.
Blue light is fine, green light also seems to be a bit bad, but red light is the devil.
Looks like x265 (also x264 I think) detects the scene as super-dark and reduces
the bitrate like crazy.

I doubt that it's just a display thing as I can see the problem on my Dell display,
my Benq display and my Samsung TV.

By now I fix it by scanning every source for red scenes before encoding and setting
zones like crazy via --zones startframe,endframe,b=1.5/startframe,endfr....
Super-tedious.

I was shocked again today after I checked my encoding of Disney's Aladdin...
Red sand with black dots -> blurred to unshaded flat areas.
Red stone wall backgrounds -> bluuuurr.

Looks like I need to double or triple the bitrate manually in these scenes just to keep
the subjectively visible detail level compared to the non-reddish scenes.

AQ3 won't help all too much either. The overall bitrate would get just too high if
I want to retain the details that way. *sigh*

[benwaggoner]

AQ3 won't help all too much either. The overall bitrate would get just too high if
I want to retain the details that way. *sigh*
Raise CFR with aq-mode 3. The question is what looks best at a given bitrate.

I wish x265 gave us a way to change the SAO parameters, so we could adjust how the smoothing works in different luma ranges.

[benwaggoner]

That is why I said that hopefully the next version will have better power management to enable higher AVX and AVX-512 frequency in low intensity workloads.
And we saw thermal throttling for AVX2 get a lot better between Haswell and Skylake SP, so there is precedent for exactly that.

I suspect a truly optimized x265 would actually have different ASM depending on processor generation due to this kind of stuff.

[katzenjoghurt]

Raise CFR with aq-mode 3. The question is what looks best at a given bitrate.

I wish x265 gave us a way to change the SAO parameters, so we could adjust how the smoothing works in different luma ranges.

AQ3 isn't the solution for me...
I tried again yesterday... the file size doubled and still I didn't really
reach the quality I could achieve by defining zones manually.

SAO might add to it... but the issue is also there with SAO disabled.


In fact two of my past problems eventually seemed to come just from
the lighting / red area issue.

I wondered why Star Wars III gave me so much more trouble encoding than other Star Wars movies -> Answer: As Anakin turns evil here, the movie got a more reddish color coding.
https://forum.doom9.org/showpost.php?p=1851237&postcount=6353

Here the problem was in fact the character's dark red hair:
https://forum.doom9.org/showpost.php?p=1848566&postcount=11


With Aladdin I have the same issue now... it's dark red all over the place and super-hard to encode due to that.
It's even a problem in bright scenes.
Everything is crisp... and than there is this character wearing a dark red Fez. And the Fez is blocky and blurry.

[fauxreaper]

Oh my.
Am I the only one having such a hard time encoding scenes with red light / red backgrounds?

Use --cbqpoffs and --crqpoffs with negative values.

[katzenjoghurt]

Woah! These parameters sound interesting! I need to get home and try them out!
Thank you, fauxreaper!

[benwaggoner]

AQ3 isn't the solution for me...
I tried again yesterday... the file size doubled and still I didn't really
reach the quality I could achieve by defining zones manually.
You need to compare at the same bitrate. So try 2-pass VBR or a higher CRF. SDR content can often get away with a higher CRF with aq-mode 3.

Trying the chroma offsets is also a great idea, at least for those scenes. So glad to have —zones!

[katzenjoghurt]

I'm giving up... crqpoffs and its brother also didn't work wonders. The bitrate went up... but even at the maximum of -12 it still didn't reach a detail retention level I liked.

And aq-mode 3 didn't work out for me as well... I went as far as setting aq-strength from 1 to 5, ultimately with a crf of 33 (~6000kbps) but... no.


If someone is out for a challenge... grab Aladdin and try to bring it down to a 5000kbps encoding without totally smeared red backgrounds.
You'll start to hate the ladies at the window at 0:07:19 - 00:07:24. :)

Personally I'll go on with zones... It'll take hours but it will work in the end.

[Stereodude]

I'm giving up... crqpoffs and its brother also didn't work wonders. The bitrate went up... but even at the maximum of -12 it still didn't reach a detail retention level I liked.

And aq-mode 3 didn't work out for me as well... I went as far as setting aq-strength from 1 to 5, ultimately with a crf of 33 (~6000kbps) but... no.


If someone is out for a challenge... grab Aladdin and try to bring it down to a 5000kbps encoding without totally smeared red backgrounds.
You'll start to hate the ladies at the window at 0:07:19 - 00:07:24. :)

Personally I'll go on with zones... It'll take hours but it will work in the end.
What speed preset are you using? Ultrafast?

[katzenjoghurt]

Medium

[benwaggoner]

Medium

Ah. I suggest slower as the fastest preset that really starts exercising HEVC’s more unique features. I might throw in --tskip to see if that might help this weird case. It can sometimes help with sharp and weird edges.

Also, if you have a target file size and are comparing different encoder settings, you really should be using 2-pass VBR. It’s really hard to compare features when both output AND file size is changing.

Lastly, Are we sure that negative values reduce chroma QP? I had it in my head it was the other way around. But I rarely use that feature.

[katzenjoghurt]

http://forum.doom9.net/showpost.php?p=1720028&postcount=2269 :)

[benwaggoner]

http://forum.doom9.net/showpost.php?p=1720028&postcount=2269 :)
Thanks! Yes, negative numbers reduce chroma QP and thus enhance chroma quality.

[jlpsvk]

Ok... give up on Dolby Vision encoding now. Question is. How to encode my first HDR10+ bluray with x265 to preserve HDR10+? Can HDR10+ be played from MKV? Or must be m2ts?

[Selur]

@jlpsvk: What do you use for playback on a pc? (For reencoding HDR-10+ data: 1. extract meta data using hdr10plus_parser 2. reencode with x265 while specifying the extracted data 3. multiplex to the container of your choice)

[jlpsvk]

@jlpsvk: What do you use for playback on a pc? (For reencoding HDR-10+ data: 1. extract meta data using hdr10plus_parser 2. reencode with x265 while specifying the extracted data 3. multiplex to the container of your choice)

Thanks for that. I am playing on OPPO, not PC.

[SeeMoreDigital]

Thanks for that. I am playing on OPPO, not PC.Do you have an HDR10+ compatible TV too?

[jlpsvk]

Do you have an HDR10+ compatible TV too?

yes i have. :) Samsung UE65NU7442

Should I use --dhdr10-opt too??? can i crop?

[Selur]

Should I use --dhdr10-opt too???
should be optional and do no harm, so you can probably use it without running into issues.

can i crop?
I guess the HDR-10+ data would need to be adjusted, since at least the 'AverageRGB' should change when applying most filters.
But since you got the equipment, try and report back.

[Barough]

x265 v3.0_RC+14-46b84ff665fd (http://www.mediafire.com/file/4j25u9ds0mvzklp/) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit : GCC 7.4.0 / 64bit : GCC 8.2.1)

https://bitbucket.org/multicoreware/x265/commits/branch/default

NOTE :
Checked with Pradeep (@MulticoreWare) about why the Default Branch haven't been pushed to v3.0 'Stable' and this is the reply/info i got

"
Our plan is to continue to use 3.0_RC on the default branch and have completed tags only on the stable branch. So we don't intend to merge back.
"

[sonnati]

Oh my.
Am I the only one having such a hard time encoding scenes with red light / red backgrounds?

E.g. if a character turns on a red light, his face would turn suddenly totally blocky.
Blue light is fine, green light also seems to be a bit bad, but red light is the devil.
Looks like x265 (also x264 I think) detects the scene as super-dark and reduces
the bitrate like crazy.

I doubt that it's just a display thing as I can see the problem on my Dell display,
my Benq display and my Samsung TV.

By now I fix it by scanning every source for red scenes before encoding and setting
zones like crazy via --zones startframe,endframe,b=1.5/startframe,endfr....
Super-tedious.

I was shocked again today after I checked my encoding of Disney's Aladdin...
Red sand with black dots -> blurred to unshaded flat areas.
Red stone wall backgrounds -> bluuuurr.

Looks like I need to double or triple the bitrate manually in these scenes just to keep
the subjectively visible detail level compared to the non-reddish scenes.

AQ3 won't help all too much either. The overall bitrate would get just too high if
I want to retain the details that way. *sigh*

HVS is more sensible to yellow-red chrominance (>>550nm)
than violet-blue (<<500nm). Red has always been a mess in encoding, especially when encoding in yuv420 where chroma is subsampled and red pictures appear in their glorious blockiness.

There's still a lot of work to be done to optimize encoders to accomodate for perception: we should improve quantization both in dark picture and red-dominant picture, but this depends also on the viewing conditions (i.e. not so important in small mobile screens)

Not to mention higher order of complexity optimizations like saliency based optimizations or similar.

I have developed many times models that change params using zones-like params to improve the performance of a given encoder. This is one of those cases.

[LigH]

x265 3.0 RC+14-46b84ff665fd (https://www.mediafire.com/file/xnu5ejz4kbmi4ww/x265_3.0_RC%2B14-46b84ff665fd.7z) (MSYS2, MinGW32 + GCC 7.4.0 / MinGW64 + GCC 8.2.1)

:rolleyes:

[birdie]

Version 3.0 (https://bitbucket.org/multicoreware/x265/downloads/x265_3.0.tar.gz)
===========

Release date - 23/01/2019

New features
-------------
1. option:: '--dolby-vision-profile <integer|float>' generates bitstreams confirming to the specified Dolby Vision profile. Currently profile 5, profile 8.1 and profile 8.2 enabled, Default 0 (disabled)

2. option:: '--dolby-vision-rpu' File containing Dolby Vision RPU metadata. If given, x265's Dolby Vision metadata parser will fill the RPU field of input pictures with the metadata
read from the file. The library will interleave access units with RPUs in the bitstream. Default NULL (disabled).

3. option:: '--zonefile <filename>' specifies a text file which contains the boundaries of the zones where each of zones are configurable.

4. option:: '--qp-adaptation-range' Delta-QP range by QP adaptation based on a psycho-visual model. Default 1.0.

5. option:: '--refine-ctu-distortion <0/1>' store/normalize ctu distortion in analysis-save/load. Default 0.

6. Experimental feature option:: '--hevc-aq' enables adaptive quantization
It scales the quantization step size according to the spatial activity of one coding unit relative to frame average spatial activity. This AQ method utilizes
the minimum variance of sub-unit in each coding unit to represent the coding unit’s spatial complexity.

Encoder enhancements
--------------------
1. Preset: change param defaults for veryslow and slower preset. Replace slower preset with defaults used in veryslow preset and change param defaults in veryslow preset as per experimental results.
2. AQ: change default AQ mode to auto-variance
3. Cutree offset reuse: restricted to analysis reuse-level 10 for analysis-save -> analysis-load
4. Tune: introduce --tune animation option which improves encode quality for animated content
5. Reuse CU depth for B frame and allow I, P frame to follow x265 depth decision

Bug fixes
---------
1. RC: fix rowStat computation in const-vbv
2. Dynamic-refine: fix memory reset size.
3. Fix Issue #442: linking issue on non x86 platform
4. Encoder: Do not include CLL SEI message if empty
5. Fix issue #441 build error in VMAF lib

[LigH]

Hooray, I should have waited one more day... :o

[Selur]

@LigH: Why? Last code change was from 2019-01-25,... 3.0 release was out there since 2019-01-23 -> https://bitbucket.org/multicoreware/x265/commits/72188bd2f03447e71e789a5fd2f10364bb232c2c

[LigH]

So if there is a v3.0 release since 2 weeks, why do I still receive an RC version ... did I not update to the "tip"? Usually I do...

Ah, there was no back merge. Stable and default branch are still distinct.

[jlpsvk]

So if there is a v3.0 release since 2 weeks, why do I still receive an RC version ... did I not update to the "tip"? Usually I do...

Ah, there was no back merge. Stable and default branch are still distinct.

Barought wrote that few days ago:

Our plan is to continue to use 3.0_RC on the default branch and have completed tags only on the stable branch. So we don't intend to merge back.

So RC14 is newer than 3.0+1

[pradeeprama]

Barought wrote that few days ago:

Our plan is to continue to use 3.0_RC on the default branch and have completed tags only on the stable branch. So we don't intend to merge back.

So RC14 is newer than 3.0+1

We are starting to use 3.0_Au (Gold) to signal that the default branch has moved to gold state. We are just trying to avoid merging back from stable to default as it is good practice to avoid merges. Hope this isn't too confusing!

[benwaggoner]

HVS is more sensible to yellow-red chrominance (>>550nm)
than violet-blue (<<500nm). Red has always been a mess in encoding, especially when encoding in yuv420 where chroma is subsampled and red pictures appear in their glorious blockiness.

There's still a lot of work to be done to optimize encoders to accomodate for perception: we should improve quantization both in dark picture and red-dominant picture, but this depends also on the viewing conditions (i.e. not so important in small mobile screens)

Not to mention higher order of complexity optimizations like saliency based optimizations or similar.

I have developed many times models that change params using zones-like params to improve the performance of a given encoder. This is one of those cases.
Yeah, very flat gradients in red and blue have always been hard (since luma is mainly green, green is pretty much non subsampled).

Using --crqpoffs -* would help red without spending more bits on blue. It makes a certain logical sense that, when decreasing chroma QP, it would make sense to reduce 2 red for every blue given the relatively higher sensitivity to red. However, red also accounts for a greater portion of luma than blue which likely would offset that some.

Encoding in 10-bit instead of 8-bit should help as well. Sometimes these problems can be as much in the 10-bit Y'CbCr -> 8-bit RGB conversion on a device than in the encoding itself. Display controllers that just truncate the 8-bit instead of dithering can cause all kinds of issues even in SDR. Dithering should be done even when doing a 16-235 Y'CbCr to 0-255 RGB conversion.


To humorously paraphrase Gen. Robert H. Barrow, USMC

"Hobbyists talk about average bitrates and per-title metrics, but experts study color volume algorithms and rate control."

[benwaggoner]

should be optional and do no harm, so you can probably use it without running into issues.

All it does is remove duplicates of identical metadata; the majority of frames in a given title will be identical to the prior frame in display order. It'll still ensure that every IDR has its metadata.

This can save some bits (only material at very low bitrates), and potentially some work by the tone mapper. Since display<>decode order, you'll get cases where the tone mapper could find out in advance that several frames in a row have identical color characteristics.

[katzenjoghurt]

HVS is more sensible to yellow-red chrominance (>>550nm)
than violet-blue (<<500nm). Red has always been a mess in encoding, especially when encoding in yuv420 where chroma is subsampled and red pictures appear in their glorious blockiness.

There's still a lot of work to be done to optimize encoders to accomodate for perception: we should improve quantization both in dark picture and red-dominant picture, but this depends also on the viewing conditions (i.e. not so important in small mobile screens)

Not to mention higher order of complexity optimizations like saliency based optimizations or similar.

I have developed many times models that change params using zones-like params to improve the performance of a given encoder. This is one of those cases.

Thanks for the confirmation, sonnati. :)
I'll continue sticking to zones then or live with rising the overall bitrate beyond general need if the pain becomes too much.

[benwaggoner]

Thanks for the confirmation, sonnati. :)
I'll continue sticking to zones then or live with rising the overall bitrate beyond general need if the pain becomes too much.
And when you find examples that need zones, grab that snippet and open an Issue with MCW. An encoder should be able to detect when this would be an issue and then compensate for it.

[redbtn]

I've made some tests for 1080p with --tune-psnr, --tune-ssim, and no-tune for different variants --ctu --max-tu-size --qg-size

Does this mean that for 1080p encoding default settings (--ctu 64 --max-tu-size 32 --qg-size 32) is the best option?

Results:

no-tune
===
--ctu 64 --max-tu-size 32 --qg-size 32 Global PSNR: 55.009, SSIM Mean Y: 0.9965665 (24.643 dB) higher value
--ctu 32 --max-tu-size 32 --qg-size 16 Global PSNR: 54.982, SSIM Mean Y: 0.9965609 (24.636 dB)
--ctu 32 --max-tu-size 32 --qg-size 8 Global PSNR: 54.866, SSIM Mean Y: 0.9964265 (24.469 dB)
--ctu 32 --max-tu-size 16 --qg-size 16 Global PSNR: 54.908, SSIM Mean Y: 0.9964738 (24.527 dB)
--ctu 32 --max-tu-size 16 --qg-size 8 Global PSNR: 54.791, SSIM Mean Y: 0.9963344 (24.359 dB)
===

tune-psnr
===
--ctu 64 --max-tu-size 32 --qg-size 32 Global PSNR: 54.936 higher value
--ctu 32 --max-tu-size 32 --qg-size 16 Global PSNR: 54.910
--ctu 32 --max-tu-size 32 --qg-size 8 Global PSNR: 54.806
--ctu 32 --max-tu-size 16 --qg-size 16 Global PSNR: 54.869
--ctu 32 --max-tu-size 16 --qg-size 8 Global PSNR: 54.765
===

tune-ssim
===
--ctu 64 --max-tu-size 32 --qg-size 32 SSIM Mean Y: 0.9964914 (24.549 dB) higher value
--ctu 32 --max-tu-size 32 --qg-size 16 SSIM Mean Y: 0.9964839 (24.539 dB)
--ctu 32 --max-tu-size 32 --qg-size 8 SSIM Mean Y: 0.9963653 (24.395 dB)
--ctu 32 --max-tu-size 16 --qg-size 16 SSIM Mean Y: 0.9964291 (24.472 dB)
--ctu 32 --max-tu-size 16 --qg-size 8 SSIM Mean Y: 0.9963079 (24.327 dB)

[Selur]

Does this mean that for 1080p encoding default settings (--ctu 64 --max-tu-size 32 --qg-size 32) is the best option?
to state the obvious: only if your quality perception aligns with PSM and SSIM Mean,..

[benwaggoner]

to state the obvious: only if your quality perception aligns with PSM and SSIM Mean,..
The modern codec design process biases heavily for fixed QP maximizing mean PSNR, so adaptive quant and psychovisual optimizations are generally only helpful when targeting subjective quality. Which is all viewers care about. I’ve seen lots of beautiful PSNR plots turn into lousy looking video.

[redbtn]

to state the obvious: only if your quality perception aligns with PSM and SSIM Mean,..

The modern codec design process biases heavily for fixed QP maximizing mean PSNR, so adaptive quant and psychovisual optimizations are generally only helpful when targeting subjective quality. Which is all viewers care about. I’ve seen lots of beautiful PSNR plots turn into lousy looking video.

I do not have enough knowledge yet to make difficult conclusions. I'm just trying to find the optimal settings for 4K HDR -> 1080p HDR.
Which option can you advise?

And one more question. Can I somehow optimize the settings for quality improvement without significantly reducing speed? Or will these settings be enough? My traget bitrate 20-25mbs

--preset placebo --crf 12 --profile main10 --output-depth 10 --level-idc 51 --sar 1:1 --colorprim 9 --colormatrix 9 --transfer 16 --range limited --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1)" --hdr --hdr-opt --chromaloc 2 --repeat-headers --hrd --max-cll "0,0" --min-luma 0 --max-luma 1023 --subme 7 --me star --merange 48 --limit-tu 4 --max-merge 4 --limit-modes --limit-refs 3 --rd 6 --rd-refine --no-tskip --rskip --no-cutree --no-sao --no-open-gop --no-b-pyramid --no-strong-intra-smoothing --vbv-bufsize 160000 --vbv-maxrate 160000 --cbqpoffs -2 --crqpoffs -2 --min-keyint 5 --keyint 240 --ipratio 1.30 --pbratio 1.20 --deblock -3:-3 --qcomp 0.65 --aq-mode 2 --aq-strength 0.8 --psy-rd 1.5 --psy-rdoq 5

[excellentswordfight]

I do not have enough knowledge yet to make difficult conclusions. I'm just trying to find the optimal settings for 4K HDR -> 1080p HDR.
Which option can you advise?

And one more question. Can I somehow optimize the settings for quality improvement without significantly reducing speed? Or will these settings be enough? My traget bitrate 20-25mbs

--preset placebo --crf 12 --profile main10 --output-depth 10 --level-idc 51 --sar 1:1 --colorprim 9 --colormatrix 9 --transfer 16 --range limited --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1)" --hdr --hdr-opt --chromaloc 2 --repeat-headers --hrd --max-cll "0,0" --min-luma 0 --max-luma 1023 --subme 7 --me star --merange 48 --limit-tu 4 --max-merge 4 --limit-modes --limit-refs 3 --rd 6 --rd-refine --no-tskip --rskip --no-cutree --no-sao --no-open-gop --no-b-pyramid --no-strong-intra-smoothing --vbv-bufsize 160000 --vbv-maxrate 160000 --cbqpoffs -2 --crqpoffs -2 --min-keyint 5 --keyint 240 --ipratio 1.30 --pbratio 1.20 --deblock -3:-3 --qcomp 0.65 --aq-mode 2 --aq-strength 0.8 --psy-rd 1.5 --psy-rdoq 5
Imo, the bitrate you are targeting is so high that most of your settings are way overkill. You will spend alot of cpu cycles for virtually no quality gain, you will be well beyond the dimishing return point.

Have you actually tried those settings? It wouldn't surprise me that placebeo crf12 could render a file bigger then the original 4k one, and you will be lucky if you do it above 1fps. I'm sure you have your reasons for this scenario, but if quality was that important to me, I would just go with the original.

[redbtn]

Imo, the bitrate you are targeting is so high that most of your settings are way overkill. You will spend alot of cpu cycles for virtually no quality gain, you will be well beyond the dimishing return point.

Have you actually tried those settings? It wouldn't surprise me that placebeo crf12 could render a file bigger then the original 4k one, and you will be lucky if you do it above 1fps. I'm sure you have your reasons for this scenario, but if quality was that important to me, I would just go with the original.
I tried this settings only without (--rd 6 --rd-refine --opt-cu-delta-qp), with (--rd 3 --no-rskip). For example for file bitrate 57 Mbs size 45 GB output bitrate 26 Mb/s size 21 GB with 1.4fps encode speed. I think with new settings it's be around 0.9-1fps on my 6 core i5 8400.
I view at 4K UHD TV, but it is connected to a computer as duplicated with a monitor with a resolution of 1080p because windows 10 cannot normally scale many applications in 4K (it's sadly). Therefore, I decided that there is no point in reducing the image via MadVR in real time. Although I have a GTX 1080 which does it without problems. And also save about 50 percent of the HDD space.

[Boulder]

You can safely use --rskip, it will speed up things a lot and you won't notice any difference. I'd also use something like --subme 3.

[redbtn]

Boulder thank you!

This is probably a silly question, but here goes anyway: if I use --hdr-opt, do I need to feed the encoder with 10-bit data or is 16-bit data as good if the source is a standard UHD with HDR? I always process things in 16-bit domain and let the encoder dither down to 10 bits.

I would let x265.exe do the dithering, 'cause other dithering options like the Floyd Steinberg error diffusion may have a nicer look, but they could increase the bitrate required by x265. The built in dithering filter in x265 is supposed to dither everything down to the target bit depth without introducing banding. Blocks and macro blocks dithered by x265 are more likely to be recognised during the motocompensation by x265 than the ones dithered using a third party dithering method, therefore compression should be better.

In a nutshell, let x265 do the dithering and always pipe to it the highest bit depth you have, unless you like a specific dithering method and you have enough bitrate.

Im encode 10bit HDR, so if I want to do the same, pipe 16bit to x265 with the flag --dither, I need LWLibavSource format="YUV420P10" or format="YUV420P16" ?
And i should remove this? clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")

Thank you!

My VS scrypt:

clip = core.lsmas.LWLibavSource(source="video.mkv", format="YUV420P10", cache=1)
clip = core.resize.Point(clip, matrix_in_s="2020ncl",range_s="limited")
clip = core.std.AssumeFPS(clip, fpsnum=24000, fpsden=1001)
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
clip = core.std.CropRel(clip=clip, left=0, right=0, top=276, bottom=276)
clip = core.fmtc.resample(clip=clip, kernel="spline64", w=1920, h=804, interlaced=False, interlacedd=False)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
clip.set_output()

[hevc_enocder]

--crf 18 --profile main10 --level-idc 5.1 --output-depth 10 --rd 4 --ctu 32 --amp --aq-mode 2 --vbv-bufsize 160000 --vbv-maxrate 160000 --ipratio 1.3
--pbratio 1.2 --no-cutree --subme 7 --me star --merange 24 --max-merge 3 --bframes 12 --rc-lookahead 60 --lookahead-slices 4 --ref 6 --min-keyint 24 --keyint 240 --deblock -3:-3
--no-sao --no-strong-intra-smoothing --high-tier

Hi guys, I would like to know, how to improve my setting, Iam quite satified, but there is problem with red color and if there is sometig to change for encoding normal new movies.

Someone told me, that there is no reason to use subme 7, I am use to from x264 use subme 10 so I thought that it could be usefull.

Thank you for your advice and some explonation.

[Selur]

there is problem with red color
You probably did use the search in this thread and already tried using negative chroma offsets (--cbqpoffs and --crqpoffs), right?

[RainyDog]

--crf 18 --profile main10 --level-idc 5.1 --output-depth 10 --rd 4 --ctu 32 --amp --aq-mode 2 --vbv-bufsize 160000 --vbv-maxrate 160000 --ipratio 1.3
--pbratio 1.2 --no-cutree --subme 7 --me star --merange 24 --max-merge 3 --bframes 12 --rc-lookahead 60 --lookahead-slices 4 --ref 6 --min-keyint 24 --keyint 240 --deblock -3:-3
--no-sao --no-strong-intra-smoothing --high-tier

Hi guys, I would like to know, how to improve my setting, Iam quite satified, but there is problem with red color and if there is sometig to change for encoding normal new movies.

Someone told me, that there is no reason to use subme 7, I am use to from x264 use subme 10 so I thought that it could be usefull.

Thank you for your advice and some explonation.

In x264, RDO is tied to subme. So higher levels of subme also incorporate higher levels of RDO.

Subme and RD(O) level are separate settings in x265 and I personally don't think it's worth using anything higher than subme 5.

Whenever I've tested, it's never actually been worth it to use anything higher than subme 3 really which is the lowest subme to include chroma residual cost.

I'd definitely choose --rd 5 (or 6 since they're the same) plus --rd-refine over subme 7 everytime. Though that will half your encoding time so I rarely ever bother with that either :p

[excellentswordfight]

I tried this settings only without (--rd 6 --rd-refine --opt-cu-delta-qp), with (--rd 3 --no-rskip). For example for file bitrate 57 Mbs size 45 GB output bitrate 26 Mb/s size 21 GB with 1.4fps encode speed. I think with new settings it's be around 0.9-1fps on my 6 core i5 8400.
I view at 4K UHD TV, but it is connected to a computer as duplicated with a monitor with a resolution of 1080p because windows 10 cannot normally scale many applications in 4K (it's sadly). Therefore, I decided that there is no point in reducing the image via MadVR in real time. Although I have a GTX 1080 which does it without problems. And also save about 50 percent of the HDD space.
As I said, I'm sure that you have your reasons, but I dont find it that appealing to spend 48h per title to get a filesize that probably would look undisguisable at half the bitrate.

I just did a quick test on tears of steal with those settings and compared it an encode at native res with slow preset (+ --deblock -1:-1 --no-sao --no-strong-intra-smoothing)... the 1080p one ended up at 40Mbps (downscaled with spline), and the 2160 one at 25, the one at at native res was sharper and was about 3x faster to encode...

Ofc, if you find the time/compression/quality to be a good tradeoff for you go ahead with that. But when you even have a 4k TV I would jut go with a 4k workflow cause you can get away with 4k re-encodes with that bitrate target. Not gonna tell you what to do, but I would at least play with different preset levels and crf values and look at the actually video to see if you can find a better "sweetspot". Cause it sounds awful to me o spend 2days for an encode, just to get a bloated file that still has a loss of detail/sharpness (cause of downscaling).

[LigH]

x265 3.0 Au+4-dcbec33bfb0f (https://www.mediafire.com/file/3js85elpouv0cxl/x265_3.0_Au%2B4-dcbec33bfb0f.7z) (MSYS2, MinGW32 + GCC 7.4.0 / MinGW64 + GCC 8.2.1)

Gold!

[poller]

is there a way to compile the x64 version without avx512?

it only increases the size of the library and i will not use avx512 anyway. for now i'm sticking to v2.7
thanks :)

[MeteorRain]

is there a way to compile the x64 version without avx512?

Not sure how the size of the library matters, but anyway.

Open \source\common\x86\asm-primitives.cpp, find X265_CPU_AVX512 and note all the avx512 asms.

Remove them from *.asm files, remove the X265_CPU_AVX512 code block, and then try to compile them.

[benwaggoner]

I just did a quick test on tears of steal with those settings and compared it an encode at native res with slow preset (+ --deblock -1:-1 --no-sao --no-strong-intra-smoothing)... the 1080p one ended up at 40Mbps (downscaled with spline), and the 2160 one at 25, the one at at native res was sharper and was about 3x faster to encode...

Ofc, if you find the time/compression/quality to be a good tradeoff for you go ahead with that. But when you even have a 4k TV I would jut go with a 4k workflow cause you can get away with 4k re-encodes with that bitrate target. Not gonna tell you what to do, but I would at least play with different preset levels and crf values and look at the actually video to see if you can find a better "sweetspot". Cause it sounds awful to me o spend 2days for an encode, just to get a bloated file that still has a loss of detail/sharpness (cause of downscaling).
Yeah, an important thing about 4K encoding is that the pixels are getting near the minimize visible size in typical viewing environments, especially with non-HDR content. So small artifacts that could be visible in 1080p content become MUCH less so at 4K. I've heard a lot of "but 4K will take 4x the bitrate of 1080p), but the frequency distribution gets softer and latitude for non-perceptible distortion gets greater. Doubling bitrate from 1080p to 2160p is typically sufficient to get value out of the extra pixels without any regressions.

In fact a satisfactory H.264 8-bit 1080p bitrate will generally be a satisfactory HEVC 10-bit 2160p bitrate. The "4K will kill the internets!" panic was badly overblown.

[poller]

Not sure how the size of the library matters, but anyway.

Open \source\common\x86\asm-primitives.cpp, find X265_CPU_AVX512 and note all the avx512 asms.

Remove them from *.asm files, remove the X265_CPU_AVX512 code block, and then try to compile them.

that's how i started, but i thought there might be a simpler way.

but it does work, size is only slighter bigger than v2.7 now.

thanks.

[benwaggoner]

but it does work, size is only slighter bigger than v2.7 now.
I am curious as to why that is worth this effort.

[poller]

I am curious as to why that is worth this effort.
well, it didn't take really long, maybe 30 minutes.
i need ffmpeg for recording from an emulator, and i don't want to see the ffmpeg.dll being 5 times bigger than the actual emu. :stupid:
so i'm trying to keep it small.


and now... more questions. :p

first, v3.0 is slower and produces bigger output than previous versions. i did read the changelog, but fail to see what could be the reason (yes, noob here). :confused:
/edit: ok, it seems to be aq-mode=2


second, x86 builds by LigH are about 10% (!) faster than mine. is there some magical compiler flag? i tried -O2 and -Ofast... no luck. -Ofast being the slowest of them actually.
x64 builds are at the same speed.

:thanks:

[LigH]

No magical flags. Just out of media-autobuild suite (more or less, negligible differences in the handling). Do we use the same compiler?

[Boulder]

v3.0 changed some presets, so maybe the change in speed comparing to v2.7 comes from there. I don't know if the x265 docs are helpful in checking out what changed (I doubt :p)

[benwaggoner]

v3.0 changed some presets, so maybe the change in speed comparing to v2.7 comes from there. I don't know if the x265 docs are helpful in checking out what changed (I doubt :p)
Specifically, slower got a lot slower, and veryslow is even more very slow. Slower was a nice sweet spot quality/perf step; I think it would have made more sense to add a superslow or something.

Try using slow and seeing if that helps.

[poller]

No magical flags. Just out of media-autobuild suite (more or less, negligible differences in the handling). Do we use the same compiler?
strange. i now tested with GCC 7.3 from MSYS2 (it seems this is the one you are using), the executable is a lot bigger and even slightly slower than my other builds (GCC 4.9.3).
i am using the default values of x265.exe for testing.
this is beyond me. :confused:


v3.0 changed some presets, so maybe the change in speed comparing to v2.7 comes from there. I don't know if the x265 docs are helpful in checking out what changed (I doubt :p)
as mentioned, if i set --aq-mode=1 speed of v3.0 is pretty much the same as before, also the size of output.
no idea about the quality, there might be some other changes as you say.

[benwaggoner]

strange. i now tested with GCC 7.3 from MSYS2 (it seems this is the one you are using), the executable is a lot bigger and even slightly slower than my other builds (GCC 4.9.3).
i am using the default values of x265.exe for testing.
this is beyond me. :confused:
By "default values" you would be implicitly using --preset medium. No idea if that is what you want to be using for your scenario.

as mentioned, if i set --aq-mode=1 speed of v3.0 is pretty much the same as before, also the size of output.
no idea about the quality, there might be some other changes as you say.
--aq-mode 2 is the default in 3.0, but was 1 in previous versions. I wouldn't think it would change performance THAT much.

[poller]

it seems i started some confusion here.

i tested another video, about the same results.

--aq-mode=1
own builds: 40.5 sec
LigH builds: 35.5 sec
so my builds are even more than 10% slower, no matter what preset i use. :( so annoying.
GCC 9.01, GCC 8.2... always the same bad speed.

--aq-mode=2
own builds: 47.5 sec
LigH builds: 40.5 sec
so yes, for MY short low res test videos mode 2 is slower.
that might be different for other input files.

[MeteorRain]

second, x86 builds by LigH are about 10% (!) faster than mine. is there some magical compiler flag? i tried -O2 and -Ofast... no luck. -Ofast being the slowest of them actually.
x64 builds are at the same speed.

:thanks:

Could be some profiling magic. I'm unsure if LigH uses any profiling when compiling the binary though.

I used it once, made it slower. And I haven't used it since, but things might have changed.

[Selur]

Nope, already answered that he simply uses media-autobuild suite (see:https://forum.doom9.org/showthread.php?p=1866145#post1866145), so no profiling.

[Boulder]

Some build log could be useful, maybe there is something missing. I'd expect that if assembler was not used, the difference would be much bigger though.

[poller]

i tried hard with GCC again.

seconds. lower is better. :)

137.0 no assembly
47.0 (default)
45.5 (PGO build) -mtune=ivybridge (default here is -O3 which makes 1st pass PGO .exe crash, thus no better speed i guess)
44.5 (PGO build) -mtune=ivybridge -O2
43.9 (PGO build) -mtune=ivybridge -funroll-loops -finline-functions -ftree-loop-vectorize -O2
39.5 LigH

so i get little improvement with all that fiddling, but still far away from LigH's GCC builds. :(


giving up here, i have no ideas left.

[LigH]

OK, I forgot little details I edited a long time ago, while testing some compiling issues with a faulty compiler version. A leftover string is:

export CXXFLAGS="-march=pentium4 -mtune=generic"

for the 32-bit compilation (which is still quite generic, just a sensible minimum). That might bring a little advantage. For the 64-bit compilation, the CXXFLAGS is empty.

Furthermore, for the 32-bit compilation, assembly is disabled for 10 and 12 bit precision cores, but enabled for the 8 bit core.

[WhatZit]

Supply --svt in the command line to use the SVT-HEVC encoder.

Never expected that one! :confused:

From http://x265.org/x265-svt-hevc-house/:

With changeset a41325fc854f, the x265 library can invoke the SVT-HEVC library for encoding through the —svt option. We have mapped presets and command-line options supported by the x265 application into the equivalent options of SVT-HEVC, and have added a few specific options that are available only when the SVT-HEVC library is invoked. This page in our documentation describes the steps to build, and invoke the SVT-HEVC library in more detail.

Our reason for this integration was to enable our users to evaluate additional relative trade-offs between performance and compression efficiency while working behind the familiar API of the x265 library. In the long term, we plan to leverage this integration to further improve x265’s ability to handle real-time and low turn-around scenarios in pure software; this is the space that SVT-HEVC was focused on. In parallel, we will continue to innovate on our flagship presets that are used in offline encoding where x265 dominates. You can expect to see these changes in the coming releases of x265, increasing the reach of open-source for video compression!

Am I being cynical to suggest that Multicoreware couldn't achieve such speed optimisations on their own, so they formed this "synergy"?

[nevcairiel]

Personally I think its stupid to incorporate another encoder into the x265 "frontend". If one wanted to use different encoders, one would use say ffmpeg, or just use them directly. x265 should be x265, and nothing else. But oh well. Probably some business driving over common sense. :)

[shinchiro]

Lol, good to know I'm not the only one who thinks x265's decision to include another encoder inside itself is stupid. Well, if there's money involved here, I'm not even surprised

[benwaggoner]

Lol, good to know I'm not the only one who thinks x265's decision to include another encoder inside itself is stupid. Well, if there's money involved here, I'm not even surprised
From the link, it sounds like the big plan is to start incorporating use of certain SVT-HEVC features/tools within x265. Having a highly accelerated coarse motion search mode could help. Kind of like the OpenGL/CUDA experiments with x264 a while ago.

x265 has a TON of features where it can take input from a first pass and then refine it. Some of those don't require the stream be made with x265, and a few work with H.264 sources IIRC.

[Forteen88]

I just wanted to say that I did a little x265 speed-test, one compile vs another,
x265-3.0_Au+7-cb3e172_vs2017-AVX2 (msystem) vs x265-v3.0_Au+7-cb3e172a5f51-SVT-win64 [ICC 1900][MSVC 1916 Multilib][SVT][64 bit].

I encoded a 44 second long cartoon animation, 00096.m2ts, with this setting:
x265.exe --crf 18 --preset veryslow --output-depth 10 --rdoq-level 0 --psy-rdoq 0 --aq-mode 1 --aq-strength 0.4 --qcomp 0.65 --bframes 16 --rc-lookahead 48 --ref 6 --min-keyint 24 --keyint 240 --frame-threads 1 --colormatrix bt709 --deblock -2:-2 --no-sao --psy-rd 0.4 --tskip --tskip-fast --tu-inter 4 --tu-intra 4 --frames 1066


x265-3.0_Au+7-cb3e172_vs2017-AVX2 (msystem) Duration: 00:53:41
x265-v3.0_Au+7-cb3e172a5f51-SVT-win64 [ICC 1900][MSVC 1916 Multilib][SVT][64 bit] Duration: 00:53:32

Not a big difference in speed, considering I have a Intel Core i5-5200U CPU (I thought that the ICC 1900-compile would be much faster).

EDIT: By "much faster", I meant much faster than this encode was, I meant like 10% faster than the non-ICC compile.

[Selur]

I thought that the ICC 1900-compile would be much faster
to be frank I would have been surprised using a different compiler to have much of an impact,...

[poller]

OK, I forgot little details I edited a long time ago, while testing some compiling issues with a faulty compiler version. A leftover string is:

export CXXFLAGS="-march=pentium4 -mtune=generic"

for the 32-bit compilation (which is still quite generic, just a sensible minimum). That might bring a little advantage. For the 64-bit compilation, the CXXFLAGS is empty.

Furthermore, for the 32-bit compilation, assembly is disabled for 10 and 12 bit precision cores, but enabled for the 8 bit core.
well, here not even -march=corei7 did help much.
assembly needs to be disabled for x86 high bit, it does not compile when enabled.


Not a big difference in speed, considering I have a Intel Core i5-5200U CPU (I thought that the ICC 1900-compile would be much faster).

the same here, actually, all x64 builds (from the net) i tested are pretty much on the same level, my own builds included and also the ICC compile.

but i see differences in the x86 builds. but honestly, not many people will use those anyway.

[LigH]

One more build to compare, with two variants:

x265 3.0_Au+7-cb3e172a5f51 MABS (https://www.mediafire.com/file/qekmo78ifrr9e9k/x265_3.0_Au+7-cb3e172a5f51_MABS.7z/file) compiled with media-autobuild_suite only (EXE only, no DLL)

x265 3.0_Au+7-cb3e172a5f51 (https://www.mediafire.com/file/bnnonqapsk0gp7m/x265_3.0_Au+7-cb3e172a5f51.7z/file) compiled with custom build scripts to obtain libx265.dll too, running in interactive MinGW32 / MinGW64 shells

[poller]

nice, some small test:

x265_3.0_RC+14-46b84ff665fd
20.5 seconds
cpuid=1049583 / frame-threads=3 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=352x288 / interlace=0 / total-frames=2101 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=3 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=25 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=0 / b-pyramid / bframe-bias=0 / rc-lookahead=15 / lookahead-slices=0 / scenecut=40 / radl=0 / no-splice / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=2 / limit-refs=3 / no-limit-modes / me=1 / subme=2 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao / no-sao-non-deblock / rd=2 / no-early-skip / rskip / fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=21.0 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / ipratio=1.40 / pbratio=1.30 / aq-mode=2 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=255 / sar-width / : / sar-height=128:117 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=255 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / qp-adaptation-range=1.00

x265_3.0_Au+7-cb3e172a5f51
20.5 seconds
cpuid=1049583 / frame-threads=3 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=352x288 / interlace=0 / total-frames=2101 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=3 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=25 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=0 / b-pyramid / bframe-bias=0 / rc-lookahead=15 / lookahead-slices=0 / scenecut=40 / radl=0 / no-splice / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=2 / limit-refs=3 / no-limit-modes / me=1 / subme=2 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao / no-sao-non-deblock / rd=2 / no-early-skip / rskip / fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=21.0 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / ipratio=1.40 / pbratio=1.30 / aq-mode=2 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=255 / sar-width / : / sar-height=128:117 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=255 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / qp-adaptation-range=1.00

x265_3.0_Au+7-cb3e172a5f51_MABS
23.3 seconds
cpuid=1049583 / frame-threads=3 / numa-pools=8 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=352x288 / interlace=0 / total-frames=2101 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=3 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=25 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=0 / b-pyramid / bframe-bias=0 / rc-lookahead=15 / lookahead-slices=0 / scenecut=40 / radl=0 / no-splice / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=2 / limit-refs=3 / no-limit-modes / me=1 / subme=2 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao / no-sao-non-deblock / rd=2 / no-early-skip / rskip / fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=21.0 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / ipratio=1.40 / pbratio=1.30 / aq-mode=2 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=255 / sar-width / : / sar-height=128:117 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=255 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / qp-adaptation-range=1.00

my own build
22.6 seconds
cpuid=1049583 / frame-threads=3 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=352x288 / interlace=0 / total-frames=2101 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=3 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=25 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=0 / b-pyramid / bframe-bias=0 / rc-lookahead=15 / lookahead-slices=0 / scenecut=40 / radl=0 / no-splice / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=2 / limit-refs=3 / no-limit-modes / me=1 / subme=2 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao / no-sao-non-deblock / rd=2 / no-early-skip / rskip / fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=21.0 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / ipratio=1.40 / pbratio=1.30 / aq-mode=2 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=255 / sar-width / : / sar-height=128:117 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=255 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / qp-adaptation-range=1.00


the MABS build has some additional setting (numa-pools=8) but that did not affect the performance.

this was tested on a i7-3770k

[LigH]

What you may not find here are default GNU C/C++ compiler options.

Please note that MABS scripts may set up some specific CFLAGS and CXXFLAGS (e.g. O2 or O3?). The interactive MinGW consoles should not ... so GCC / G++ defaults may apply. Except for the 32-bit build where I explicitly set CXXFLAGS with pretty generic options suitable for 32-bit code on any AMD64 capable CPU, minimally (see above).

I have no clue what I may do "right".

[benwaggoner]

to be frank I would have been surprised using a different compiler to have much of an impact,...
It seems we've seen compilers make about a 10% difference from slowest to fastest. Which is kinda surprising to me given all the hand-tuned assembly that doesn't get compiled.

[LigH]

With this amount, the only reason I could imagine is memory alignment...

[FranceBB]

Since everyone was concerned about x64 platforms and nobody used x86, I tested it on a real x86 platform running Windows Server 2003 x86 with PAE and 16 GB of RAM.
The CPU is an old, dusty Intel Xeon 4c/8th running at 2.60GHz with instruction sets up to SSE4.2:

4/N.A) - x265 3.0_Au+7 - MABS compiled by LigH with media-autobuild_suite only (EXE only, no DLL)

It didn't even start. It refused to start due to missing kernel calls: GetNumaNodeProcessorMaskEx, InitializeConditionVariable, SetThreadGroupAffinity, SleepConditionVariableCS, WakeAllConditionVariable
No luck on Windows Server 2003, so it won't run on XP and its derivatives either.

3) - x265 3.0_Au+7 - compiled by LigH with custom build scripts to obtain libx265.dll too, running in interactive MinGW32 / MinGW64 shells

3.7fps/3.9fps

2) - x265 3.0_Au+7 - compiled with GCC9 (Preview) target SSE4.2

4.2fps/4.3fps

1) - x265 3.0_Au+7 - compiled with GCC8 target SSE4.2

4.7fps/4.8fps

Very basic low-complex Command line:
x265.exe --y4m - --dither --preset medium --level 5.0 --tune fastdecode --no-high-tier --ref 2 --rc-lookahead 3 -b 2 --profile main10 --bitrate 25000 --deblock -4:-4 --min-luma 64 --max-luma 940 --chromaloc 2 --range limited --videoformat component --colorprim bt709 --transfer bt709 --colormatrix bt709 --overscan show --no-open-gop --min-keyint 1 --keyint 24 --repeat-headers --rd 3 --vbv-maxrate 25000 --vbv-bufsize 25000 --asm=sse4.2 --wpp -o "\\VBOXSVR\Share_Windows_Linux\raw_video.hevc"

Lossless 16bit SD (UHD SDR downscaled) footage.


Anyway, I don't think the comparison is fair, 'cause LigH targeted pentium4, which means only SSE2 are supported.
In other words, I'm comparing SSE4.2 vs SSE2 and it's pretty clear that SSE4.2 have an advantage over SSE2.
As to GCC9, it seems that they changed something in the way -mtune behaves or maybe they changed something else; anyway, it produces an SSE4.2 build slower than the GCC8 SSE4.2 one.
It would be interesting to find out how ICC targeting SSE4.2 behaves on old Intel x86 systems (if Intel Parallel Studio can produce a Windows Server 2003 compatible binary).

[Boulder]

Has anyone else noticed how the bitrate shown during the encoding phase and the final bitrate differ from each other quite a lot sometimes? Yesterday I happened to be watching an 70000-frame encode finish and at the last frames, the average bitrate was ~6800 kbps. When the encode finished, the final bitrate was suddenly over 7100 kbps.

[Wolfberry]

@FranceBB I made a x86 test binary targeting SSE4.2 (only 8 bit).

You can test if it works or not, I also compiled ffmpeg with libvmaf, and you can test that as well.

[FranceBB]

@FranceBB I made a x86 test binary (https://drive.google.com/open?id=1YAdAX8xhX5wPYllY4_tOZX0GEzpelCsb) targeting SSE4.2 (only 8 bit).

You can test if it works or not, I also compiled ffmpeg (https://drive.google.com/open?id=1eq85KZ3YLxbbrUbOpYpFelrDBlqS1oVm) with libvmaf, and you can test that as well.

It's weird.
Dependency Walker didn't find any issue with the installer, but when I try to run it it says that's not a valid x86 application.
Are you sure that you targeted SSE4.2? Out of curiosity, can you try with 4.1 and one without any assembly optimisation?
The CPU is fully capable of handling SSE4.2, so I don't understand.
The OS is Windows Server 2003 x86 with PAE Enabled and 16GB of RAM.

[qyot27]

It's not weird when you consider that Windows Server 2003 is subject to the same problem that Windows XP faces: the error is almost definitely coming from x265 expecting modern Windows APIs (read: NUMA). The 'not a valid x86 application' error doesn't occur when the assembly is wrong (in that case it would crash and throw a SIGILL), it occurs when you try to run 64-bit programs on 32-bit OSes (which in this case it's not; the executable and dll are standard 32-bit PE32) or other similar OS incompatibilities.

https://bitbucket.org/multicoreware/x265/src/cb3e172a5f51c6a4bf8adb7953fe53277f5a1979/source/CMakeLists.txt?at=default&fileviewer=file-view-default#CMakeLists.txt-443

[katzenjoghurt]

Hm.
Could it be that zones are broken in AU+7?
StaxRip crashes away for me with zones defined. AU+3 still worked.
(I'm using the VS 2019 Preview 3 AVX2 AU+7 build from http://msystem.waw.pl/x265/)

[Ma]

Could it be that zones are broken in AU+7?
StaxRip crashes away for me with zones defined. AU+3 still worked.

Yes, it could. Second suspect is commit Integrate SVT-HEVC encoder to x265 (https://bitbucket.org/multicoreware/x265/commits/878541319ea1375be0e981f6ea5fefdb4d509fbd) (first suspect was innocent)

Proposed fix is (it is only technical change, I totally don't understand x265_copy_params function):
diff -r cb3e172a5f51 source/common/param.cpp
--- a/source/common/param.cpp Tue Feb 19 20:20:35 2019 +0530
+++ b/source/common/param.cpp Tue Feb 26 16:16:16 2019 +0100
@@ -2240,16 +2240,7 @@
dst->rc.zoneCount = src->rc.zoneCount;
dst->rc.zonefileCount = src->rc.zonefileCount;

- if (src->rc.zones)
- {
- dst->rc.zones->startFrame = src->rc.zones->startFrame;
- dst->rc.zones->endFrame = src->rc.zones->endFrame;
- dst->rc.zones->bForceQp = src->rc.zones->bForceQp;
- dst->rc.zones->qp = src->rc.zones->qp;
- dst->rc.zones->bitrateFactor = src->rc.zones->bitrateFactor;
- }
- else
- dst->rc.zones = NULL;
+ dst->rc.zones = src->rc.zones;

if (src->rc.lambdaFileName) dst->rc.lambdaFileName = strdup(src->rc.lambdaFileName);
else dst->rc.lambdaFileName = NULL;

Could you test if it helps? (compiled gcc 8.3 binary with this patch: test-Au7.7z (http://msystem.waw.pl/x265/test-Au7.7z))

[katzenjoghurt]

Thx Ma... I'll try your version out as soon as I'm back from work (in 6 hours).

AU+7 crashed for me for two videos when I tried it out this morning just by setting
--zones 1,100,b=1.3

No crash any more after I replaced the exe with an older AU+3 version.

[Ma]

Thanks for the info. The prime suspect is innocent (I am able to reproduce the problem) -- please ignore the patch and binary to test. We should investigate the problem further...

[Ma]

In x265 version 3.0_Au+5 there is new function x265_copy_params which I don't know what problem solves -- for sure it create new problems (zones for example).

My question is: if we remove x265_copy_params function and go back to old code (memcpy) is there something wrong? I've attached patch that revert changes with x265_copy_params function (maybe instead of patching this function it is better to remove it).

[mandarinka]

Has anyone else noticed how the bitrate shown during the encoding phase and the final bitrate differ from each other quite a lot sometimes? Yesterday I happened to be watching an 70000-frame encode finish and at the last frames, the average bitrate was ~6800 kbps. When the encode finished, the final bitrate was suddenly over 7100 kbps.

Happens to me too, even with a bigger deltas possibly (not sure, I use higher rates generally).

[katzenjoghurt]

Hi Ma,

I fear I can't be of help finding an answer to your question but thanks a lot for looking into it! :) *thumbsup*

[FranceBB]

The only difference between Au+7 and Au+3 are SVT-HEVC (not enabled in the build you use) and the zone fix in linux (which probably breaks staxrip) (need to investigate further)

@FranceBB New test build Not Set (https://drive.google.com/open?id=1oBQ7GCPYojFrhn_gXI29_0cTaDC4LzTe) SSE4.2 (https://drive.google.com/open?id=1sz4flIccYFBqSA5ANnONJzq9LMPzdQcF)

This one works, however I can't encode anything higher than 8bit (if I try to use Main10 it reverts back to the default bit-depth).
I remember that I had this kind of issue when I tried to compile x265 x86 with Visual Studio (I never managed to get 10bit and 12bit builds for x86 out of Visual Studio).

I tried now with a longer test, but encoding at 8bit this time, letting x265 calculate the average speed for me and feeding it with a 16bit uncompressed clip:

- GCC 8.2 SSE4.2

6.19fps

- Wolfberry not SSE4.2

6.14fps

- GCC 9 Preview SSE4.2

6.11fps

- LigH

6.22fps


However, encoding at 8bit and comparing different compilers shows different results, 'cause for 8bit x265 there are manually written intrinsics by x265 developers.
The interesting part is with 10bit for which there aren't manually written intrinsics, so it's up to the compiler to optimize the plain C++ code as much as possible, that's why I was testing them with Main10 in my previous test.
For the 8bit, apparently, GCC9 is still slower than GCC8, your build without targeting SSE4.2 is somewhere in-between and the one made by LigH is faster than mine for whatever reason.
But again, for 8bit there are manually written intrinsics, so I can't really test the effectiveness of compilers.

[Ma]

@katzenjoghurt
In new binaries from http://msystem.waw.pl/x265/ (ver. 3.0_Au+8) zones are working.

Now I must do some work for living but at the weekend I should look into this bug closer and prepare proper patch that works with SVT_HEVC too.

[LigH]

x265 3.0 Au+8-31ab7e09a3b5 (https://www.mediafire.com/file/rt5dypooppdpkze/x265_3.0_Au+8-31ab7e09a3b5.7z/file) (MSYS2, MinGW32 + GCC 7.4.0 / MinGW64 + GCC 8.3.0)

New 64-bit GCC 8.3.0, and zone issues should be fixed

[katzenjoghurt]

@katzenjoghurt
In new binaries from http://msystem.waw.pl/x265/ (ver. 3.0_Au+8) zones are working.

Now I must do some work for living but at the weekend I should look into this bug closer and prepare proper patch that works with SVT_HEVC too.

Yes! Wonderful! :) ... works like a charm for me again.
Immediately tried it out this morning but had to get to work as well (really quick) and was running out of time to post about it.

Thank you, Ma. :)

[Barough]

x265 v3.0_Au+9-2abd2a1909a4 (http://www.mediafire.com/file/69a6j1zgkt2k21g/x265-3.0_Au%252B9-2abd2a1909a4_Win_GCC.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit-GCC v7.4.0 / 64bit-GCC v8.3.0)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[filler56789]

↑ @Barough: many :thanks: for the new build :)

[Barough]

x265 v3.0_Au+14-c7e5878bdd31 (https://www.mediafire.com/file/g37kine05h521q3/x265-3.0_Au+14-c7e5878bdd31_Win_GCC.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit-GCC v7.4.0 / 64bit-GCC v8.3.0)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[LigH]

x265 3.0 Au+14-c7e5878bdd31 (https://www.mediafire.com/file/j8c5ddgmpb3tmub/x265_3.0_Au+14-c7e5878bdd31.7z/file) (MSYS2, MinGW32 + GCC 7.4.0 / MinGW64 + GCC 8.3.0)

3 flavours:

"Win32XP" (XP compatibility enabled, should disable NUMA support)
"Win32" (XP compatibility disabled, NUMA support should be enabled in Win7+)
"Win64"

AVX2 for normFactor and ssimDistortion

[filler56789]

x265.exe 3.0_Au+16-0018ca1ca42f

(GCC 7.4.0, 64-bits, multilib)

http://www.mediafire.com/file/gtr4ehsp52vho20/x265-3.0_Au%2B16-0018ca1ca42f.7z

[Forteen88]

@Wolfberry. You're not releasing a ICC19-compile of x265 anymore?

[filler56789]

x265-3.0_Au+21-bac0e1acb874-win64-multilib (https://drive.google.com/open?id=1nfhUE3fHqKefSwIhDf3LN67jANU9zroF)

x265 [info]: build info [Windows][GCC 8.3.1][64 bit] 8bit+10bit+12bit
x265 [info]: (libavcodec 58.51.100)
x265 [info]: (libavformat 58.27.102)
x265 [info]: (libavutil 56.26.100)
x265 [info]: (lsmash 2.16.1)

Compiled with LAVF & LSMASH support, patches from msg7086.

Your build doesn't work on 64-bit Windows 7, it simply crashes :confused:

[lvqcl]

Your build doesn't work on 64-bit Windows 7, it simply crashes :confused:

I suspect that the problem is not an OS. Probably your CPU doesn't support AVX (or AVX2?), and this build requires it for some reason.

[filler56789]

Yeah, that build requires AVX.

New build (https://drive.google.com/open?id=1nKb6zeAT7pqC3LhEky66NglWTMNPy-XI)

x265 [info]: HEVC encoder version 3.0_Au+21-bac0e1acb874
x265 [info]: build info [Windows][GCC 8.3.1][64 bit] 8bit+10bit+12bit
x265 [info]: (libavcodec 58.52.100)
x265 [info]: (libavformat 58.27.103)
x265 [info]: (libavutil 56.26.100)
x265 [info]: (lsmash 2.16.1)

I'm using march=core2 for this build (I use --with-arch=core2 when configuring gcc), so it should work for most people, but no guarantee as I don't have a spare machine to test.

Thanks for the compatible build :thanks:

It is somewhat bloated because of the additional stuff, but I compressed it with upx and now it has a healthy weight.

[Boulder]

How does one get rid of the "onion gradient" artifacts of x265? In the sample images, they are apparent in the character's forehead and also in the purplish stripe near his hair (zoom a bit to see both better). The bitrate of the encodes is ~6900 kbps.

x264
https://i.ibb.co/RprZzRn/x264.png (https://ibb.co/3y6GBgX)

x265, aq-mode 1
https://i.ibb.co/zfyTdx0/x265-aq1.png (https://ibb.co/9YFkPgC)

x265, aq-mode 2
https://i.ibb.co/19DYDXf/x265-aq2.png (https://ibb.co/55NXNjW)

Some base settings of the x265 encode:
--deblock -2:-1 --no-strong-intra-smoothing --cbqpoffs -3 --crqpoffs -3 --subme 3 --merange 24 --no-sao --no-rect --qcomp 0.7 --rd 6 --rd-refine
--aq-mode 1/aq-mode 2 --aq-strength 0.9 --ctu 16 --max-tu-size 8 --qg-size 16 --tu-inter-depth 2 --tu-intra-depth 2 --limit-tu 1 --limit-refs 3
--max-merge 2 --ref 5 --bframes 10

The x264 settings are pretty much --preset veryslow --tune film --no-fast-pskip --no-dct-decimate --deblock -2:-1 --qcomp 0.7.

[mini-moose]

How does one get rid of the "onion gradient" artifacts of x265? In the sample images, they are apparent in the character's forehead and also in the purplish stripe near his hair (zoom a bit to see both better).

Maybe post a source cap too.

[benwaggoner]

How does one get rid of the "onion gradient" artifacts of x265? In the sample images, they are apparent in the character's forehead and also in the purplish stripe near his hair (zoom a bit to see both better).

Some base settings of the x265 encode:
--deblock -2:-1 --no-strong-intra-smoothing --cbqpoffs -3 --crqpoffs -3 --subme 3 --merange 24 --no-sao --no-rect --qcomp 0.7 --rd 6 --rd-refine
--aq-mode 1/aq-mode 2 --aq-strength 0.9 --ctu 16 --max-tu-size 8 --qg-size 16 --tu-inter-depth 2 --tu-intra-depth 2 --limit-tu 1 --limit-refs 3
--max-merge 2 --ref 5 --bframes 10

The x264 settings are pretty much --preset veryslow --tune film --no-fast-pskip --no-dct-decimate --deblock -2:-1 --qcomp 0.7.
You are doing some weird settings in there which may have combinatorial effects. Can you try just --preset slower and see how that turns out?

A 8x8 max tu is pretty unusual, and those are also big chroma offsets which can suck bits away from luma. What are the goals of these settings, and how tdid you come to them?

[Boulder]

I got you the original clip here. I basically did a very slight MDegrain and then downscaled to 1280x720 using a very sharp Bicubic and fed to the encoders.

https://drive.google.com/open?id=1Qm_JWO-bksBTRgPZkN9NZavzXFTSkGIr

My x265 settings are based on frame-by-frame comparisons that I made several times. A small CTU and TU size look better (least distortion compared to the original image) with 720p encodes. For 1080p, I would use one notch higher values (also based on my tests). The chroma offsets don't change the bitrate much in CRF mode so the difference in 2-pass shouldn't also be big. Besides, x264 already does a similar thing by default.

EDIT: does anyone else have problems with topic notifications? I don't get any emails at all from any thread I've subscribed to.

[excellentswordfight]

I got you the original clip here. I basically did a very slight MDegrain and then downscaled to 1280x720 using a very sharp Bicubic and fed to the encoders.

https://drive.google.com/open?id=1Qm_JWO-bksBTRgPZkN9NZavzXFTSkGIr

My x265 settings are based on frame-by-frame comparisons that I made several times. A small CTU and TU size look better (least distortion compared to the original image) with 720p encodes. For 1080p, I would use one notch higher values (also based on my tests). The chroma offsets don't change the bitrate much in CRF mode so the difference in 2-pass shouldn't also be big. Besides, x264 already does a similar thing by default.

EDIT: does anyone else have problems with topic notifications? I don't get any emails at all from any thread I've subscribed to.
Dont seem to be an "general" issue with x265. Did a re-encode of your sample and I didnt get an gradiant issue, and overall I think x265 did a better job. I think benwaggoner is right, its your settings thats causing it.

x264 2pass @ 5Mbps, veryslow tune film
https://i.ibb.co/F08sSQf/x264.png (https://ibb.co/yYghwzM)

x265 2pass @ 5Mbps, slow, --no-sao --no-strong-intra-smoothing --deblock -1:-1
https://i.ibb.co/rGn5b5H/x265.png (https://ibb.co/fYmr8rH)

[Boulder]

Dont seem to be an "general" issue with x265. Did a re-encode of your sample and I didnt get an gradiant issue, and overall I think x265 did a better job. I think benwaggoner is right, its your settings thats causing it.

Thanks, that's exactly what I was looking for since I was experimenting a lot with aq-strength and the psy options to try to figure out what is wrong but was clearly looking in the wrong place.

I did some more testing now that I found those troublesome frames (it seems they evaded me when I pinpointed my go-to settings a long time ago), and using CTU 32 makes the onionize effect go away. I also left max-tu-size and qg-size at their defaults since things looked good.

[jethro]

Thanks, that's exactly what I was looking for since I was experimenting a lot with aq-strength and the psy options to try to figure out what is wrong but was clearly looking in the wrong place.

I did some more testing now that I found those troublesome frames (it seems they evaded me when I pinpointed my go-to settings a long time ago), and using CTU 32 makes the onionize effect go away. I also left max-tu-size and qg-size at their defaults since things looked good.

Encoder Settings:
x265 CRF 28, veryslow, --no-sao --no-strong-intra-smoothing --deblock -1:-1


https://i.ibb.co/7C52R4Z/Av-ringing-1.png (https://ibb.co/fF54nvy)

https://i.ibb.co/cyzLmnh/Av-ringing-2.png] (https://ibb.co/9pRtM5H)

https://i.ibb.co/wp58wYv/Av-ringing-3.png] (https://ibb.co/mhfdXSx)

More bitrate removes the artifacting.

[Boulder]

In those cases, I had thrown quite a lot of bitrate there. Using CRF18 produced a smaller file than 6900 kbps, which was x264@CRF18. CRF28 is something I'd never use :) The onion artifacts were visible only in certain frames with very high motion, that's probably why I missed them in my test rounds earlier. I do recall someone complaining that tune grain sometimes did those as well.

[Blue_MiSfit]

I'd caution against doing too much settings tuning using still frame comparisons, especially when evaluating max CU size (--ctu).

I've spent a lot of time doing this. Using --ctu 32 can indeed produce a sharper / more detailed / less blurred image when looking at single frames - relative to keeping the default (--ctu 64) in some cases. However, the trade-off is typically more blocking, which you're more likely to notice in motion. In other words, an inferior user experience unless you're using very high bitrates.

You're kind of performing a brute force non-adaptive psy tuning. x265 will use smaller CUs when it sees fit, you're just forcing it to never consider 64x64, even when coding giant uniform areas.

In my opinion, you're typically best off trusting the extensive tuning and testing that MulticoreWare has done, and just using a preset. They set --ctu to 64 starting with --preset veryfast, and keep it that way all the way through placebo. In fact, in looking at the documentation for this parameter: https://x265.readthedocs.io/en/default/cli.html#cmdoption-ctu it appears the only reason they set this to 32 for superfast and lower is that this increases parallelism and can therefore be faster.

Do what's right for you, of course :)

[tuanden0]

I got crash when using zone in x265_12bit.
So I tried to use VSEdit to test it and here is result.
Here is x265 file: http://msystem.waw.pl/x265/x265-3.0_Au+21-bac0e1a_gcc83-AVX2.7z
Here is everything else: https://i.imgur.com/FFlFylW.png

[Boulder]

I'd caution against doing too much settings tuning using still frame comparisons, especially when evaluating max CU size (--ctu).

I've spent a lot of time doing this. Using --ctu 32 can indeed produce a sharper / more detailed / less blurred image when looking at single frames - relative to keeping the default (--ctu 64) in some cases. However, the trade-off is typically more blocking, which you're more likely to notice in motion. In other words, an inferior user experience unless you're using very high bitrates.

You're kind of performing a brute force non-adaptive psy tuning. x265 will use smaller CUs when it sees fit, you're just forcing it to never consider 64x64, even when coding giant uniform areas.

In my opinion, you're typically best off trusting the extensive tuning and testing that MulticoreWare has done, and just using a preset. They set --ctu to 64 starting with --preset veryfast, and keep it that way all the way through placebo. In fact, in looking at the documentation for this parameter: https://x265.readthedocs.io/en/default/cli.html#cmdoption-ctu it appears the only reason they set this to 32 for superfast and lower is that this increases parallelism and can therefore be faster.

Do what's right for you, of course :)I've always considered 64x64 to be quite a lot for 720p video, and also the difference in performance is substantial. I can see that the average QP is lower with 32x32 than 16x16 though so it could be that there are situations where the encoder could squeeze some more out of the video. Of course, I could always run two simultaneous encodes to keep the CPU as busy as possible.

As you apparently have been testing things quite a lot, do you have any opinion regarding AQ-mode? I'm still quite unsure if the new default of mode 2 is the best general option for film.

[excellentswordfight]

I've always considered 64x64 to be quite a lot for 720p video, and also the difference in performance is substantial. I can see that the average QP is lower with 32x32 than 16x16 though so it could be that there are situations where the encoder could squeeze some more out of the video. Of course, I could always run two simultaneous encodes to keep the CPU as busy as possible.

As you apparently have been testing things quite a lot, do you have any opinion regarding AQ-mode? I'm still quite unsure if the new default of mode 2 is the best general option for film.
I wrote to Ma a fem months back regarding ctu 64 being overkill for reslutions under 2160p, and asking why this as the default value. And he responded with numbers that showed a loss in compression with using lower ctu values, even for lower res material. The difference is probably pretty negligible though, but i dont see any reason to lower it IF youre not trying to saturate more threads (were it can do wonders). I only lower it to 32 for 1080p if i’m at 16 threads or more.

[Barough]

x265 v3.0_Au+22-feec4bdf9866 (https://www.mediafire.com/file/5y74h3vjekjvho4/x265-3.0_Au+22-feec4bdf9866_Win_GCC.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit-GCC v7.4.0 / 64bit-GCC v8.3.0)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[Blue_MiSfit]

The difference isn't huge, but it does help.

IMO aq-mode 2 is definitely better in most cases. I set this always unless doing extremely low bitrate when I'll use mode 3.

[~ VEGETA ~]

I wanted to encode a 10-bit H.264 MKV video (lossless, 30 gb) to HEVC 10-bit... I piped it via ffmpeg but I put this "-pix_fmt yuv420p" instead of this "yuv420p10le"... I suspect this is wrong, but it is the only way it worked. Can you kindly guide me here?

I encoded .vpy file to lossless 10-bit H.264 since my device is slow with the filters I used... so now how to encode the 10-bit .mkv using x265? << I assumed using -pix_fmt yuv420p is wrong.



Plus, next time when I want to encode like this (lossless 10-bit H.264 then 10-bit HEVC), should I encode to .yuv/.h264 in x264 (--muxer yuv\raw)? So I can input it directly to x265 without piping??



Here is my command line:

"C:\.....\ffmpeg.exe" -i "C:\....\lossless_H264_video.mkv" -pix_fmt yuv420p -f yuv4mpegpipe - | C:\....\x265_64bit_10bit.exe --y4m --preset slower --crf 17 --ref 6 --rd 6 --psy-rd 1 --ctu 64 --aq-strength 0.8 --output-depth 10 --input-res 1920x1080 - --output "C:\....\ep07_1080p_video_HEVC.hevc"

[sneaker_ger]

Post your complete command line and ffmpeg log.

(Yes, if you want 10 bit output setting -pix_fmt yuv420p is wrong. Make sure your ffmpeg build is compiled with 10 bit libx265.)

[~ VEGETA ~]

Post your complete command line and ffmpeg log.

(Yes, if you want 10 bit output setting -pix_fmt yuv420p is wrong. Make sure your ffmpeg build is compiled with 10 bit libx265.)

I posted my command line as you see now.

Where can I find such ffmpeg build for windows?

I thought -pix_fmt is to get the input as 10-bit...

[sneaker_ger]

I thought you wanted to use libx265 from within ffmpeg. Ignore my last post.

Use:
ffmpeg -i "INPUT" -f yuv4mpegpipe -strict -1 -
Then make sure the log says it's "yuv420p10le".

[excellentswordfight]

I wanted to encode a 10-bit H.264 MKV video (lossless, 30 gb) to HEVC 10-bit... I piped it via ffmpeg but I put this "-pix_fmt yuv420p" instead of this "yuv420p10le"... I suspect this is wrong, but it is the only way it worked. Can you kindly guide me here?

I encoded .vpy file to lossless 10-bit H.264 since my device is slow with the filters I used... so now how to encode the 10-bit .mkv using x265? << I assumed using -pix_fmt yuv420p is wrong.



Plus, next time when I want to encode like this (lossless 10-bit H.264 then 10-bit HEVC), should I encode to .yuv/.h264 in x264 (--muxer yuv\raw)? So I can input it directly to x265 without piping??



Here is my command line:

"C:\.....\ffmpeg.exe" -i "C:\....\lossless_H264_video.mkv" -pix_fmt yuv420p -f yuv4mpegpipe - | C:\....\x265_64bit_10bit.exe --y4m --preset slower --crf 17 --ref 6 --rd 6 --psy-rd 1 --ctu 64 --aq-strength 0.8 --output-depth 10 --input-res 1920x1080 - --output "C:\....\ep07_1080p_video_HEVC.hevc"

I never had issues with the following:
"ffmpeg.exe" -i "input.mkv" -pix_fmt yuv420p10le -f yuv4mpegpipe -strict -1 - | "x265.exe" --y4m
So maybe you need to ad -strict -1

[~ VEGETA ~]

I thought you wanted to use libx265 from within ffmpeg. Ignore my last post.

Use:
ffmpeg -i "INPUT" -f yuv4mpegpipe -strict -1 -
Then make sure the log says it's "yuv420p10le".

I see:

video: wrapped_avframe, yuv420p10le in cmd so I guess this is ok.

what about my other question about better way to encode next time?

[sneaker_ger]

I encoded .vpy file to lossless 10-bit H.264 since my device is slow with the filters I used

Plus, next time when I want to encode like this (lossless 10-bit H.264 then 10-bit HEVC), should I encode to .yuv/.h264 in x264 (--muxer yuv\raw)? So I can input it directly to x265 without piping??
I don't understand the need for a temporary file at all in this case. Usually this is to speed up encoding when we do slow filtering + multiple passes of video encoding and time of
filtering + 1st pass + filtering + 2nd pass + .. > filtering + encode to lossless + 1st pass from lossless + 2nd pass from lossless ..

But you are encoding CRF, e.g. now instead of filtering + x265 crf you do filtering + x264 lossless + x265 crf. That's slower?

[~ VEGETA ~]

I don't understand the need for a temporary file at all in this case. Usually this is to speed up encoding when we do slow filtering + multiple passes of video encoding and time of
filtering + 1st pass + filtering + 2nd pass + .. > filtering + encode to lossless + 1st pass from lossless + 2nd pass from lossless ..

But you are encoding CRF, e.g. now instead of filtering + x265 crf you do filtering + x264 lossless + x265 crf. That's slower?

I have very low fps doing my filters, so I put everything to lossless x264 --preset ultrafast... then I can encode directly using x264 with slow settings in both 1080p and 720p. While If I want to encode in 1080 and 720 from vpy in both times I guess it will be slower since now I've gotta encode the filtering twice.

while if I wanna encode 720p only (from horriblesubs tv shows), then yes I go vpy directly without lossless.


what do you do for encoding BDs to 1080 and 720?

[sneaker_ger]

So you encode to multiple resolutions? Ok, then it may make sense to create an intermediate to only do (slow) filtering once.

I think you may save a little bit of time/CPU by saving to uncompressed y4m/yuv instead of encoding x264 lossless as intermediate. But with slow filtering + slow x265 it will probably not make too much of a difference either way. And the size of the file increases so the wear on the HDD/SSD increases. P.S.: ffmpeg allows multiple outputs but I don't know how it affects speed and it may result in HDD fragmentation.


what do you do for encoding BDs to 1080 and 720?
I don't.

[~ VEGETA ~]

So you encode to multiple resolutions? Ok, then it may make sense to create an intermediate to only do (slow) filtering once.

I think you may save a little bit of time/CPU by saving to uncompressed y4m/yuv instead of encoding x264 lossless as intermediate. But with slow filtering + slow x265 it will probably not make too much of a difference either way. And the size of the file increases so the wear on the HDD/SSD increases. P.S.: ffmpeg allows multiple outputs but I don't know how it affects speed and it may result in HDD fragmentation.




save to uncompressed video?? like using virtualdub2 to do that? not bad but I never tried it. I just hope it won't be more size than lossless h.264, at least not too much. plus, it has to be 10-bit.

I encode on a dedicated server so no problem about HDD xD, or so I think.

[sneaker_ger]

You don't need VirtualDub. vspipe can create y4m files. But yes, unsurprisingly, uncompressed files are bigger than compressed ones...

But again: if you are doing slow filtering and slow x265 encoding the difference between uncompressed or lossless x264 --preset ultrafast intermediate is probably not worth worrying about. If you want to know exactly you have to benchmark it.

[~ VEGETA ~]

You don't need VirtualDub. vspipe can create y4m files. But yes, unsurprisingly, uncompressed files are bigger than compressed ones...

But again: if you are doing slow filtering and slow x265 encoding the difference between uncompressed or lossless x264 --preset ultrafast intermediate is probably not worth worrying about. If you want to know exactly you have to benchmark it.

I thought of doing this:

ffmpeg -i inputfile.vpy -c:v rawvideo outputfile.yuv

so you suggested vspipe, so maybe this:

vspipe script.vpy output.raw

or add --y4m

I will try next time to see how much time will it take. Lossless x264 ultrafast 1080p is about 8 hours.

[FranceBB]

I will try next time to see how much time will it take. Lossless x264 ultrafast 1080p is about 8 hours.

If you really need a lossless mezzanine that doesn't require much computational cost, I think you're better off with UTVideo.
Alternatively, you can use other lossless codecs like FFV1, HuffYUV or Lagarith, although my suggestion is UTVideo or FFV1.

[~ VEGETA ~]

If you really need a lossless mezzanine that doesn't require much computational cost, I think you're better off with UTVideo.
Alternatively, you can use other lossless codecs like FFV1, HuffYUV or Lagarith, although my suggestion is UTVideo or FFV1.

I tried make it .yuv raw from virtualdub2 using ffv1 lossless, it took the same time as x264-10bit lossless if not a bit more.

Also, now x265 inputs it directly but it encodes it WAY much slower for IDK reason. Also file size is so freaking huge, maybe I did something bad despite using the exact same encoding commands:



x265_64bit_10bit.exe --preset slower --crf 17 --ref 6 --rd 6 --psy-rd 1 --ctu 32 --aq-strength 0.8 --output-depth 10 --input-res 1920x1080 --fps 24000/1001 --input "C:\videp.yuv" --output "C:\output.hevc"

pause

[FranceBB]

Also file size is so freaking huge

That's not surprising, considering that lossless codecs have limited algorithms to use in order to achieve a compression; as to the uncompressed .yuv file, well, uncompressed is uncompressed.

it took the same time as x264-10bit lossless if not a bit more.
Oh... I didn't expect that- Apparently x264 lossless is more optimized or perhaps it's because you used VirtualDub rather than ffmpeg to encode it.

now x265 inputs it directly but it encodes it WAY much slower
I didn't expect that. Encoding an uncompressed yuv source shouldn't take more than encoding an uncompressed A/V stream like the one Avisynth pipes to x265 nor encoding a lossless x264 source.
I mean, x265 is still gonna use its internal decoder to take the source in input, right?
Encoding-wise, I wonder whether its own internal decoder is more optimized for taking one thing in input rather than the other, or perhaps is just the way things have been allocated... but that cannot be since each source is lossless and there are little differences on filesize to justify a big speed different, besides compressed lossless, although with less space, still have to be "decoded", while uncompressed lossless files are ready to go but way bigger, so perhaps the hard drive is not fast enough to justify an uncompressed source as it takes longer to buffer it than decoding a losslessly compressed one... I don't know.
Perhaps someone can enlighten me as well about this.

[Wolfberry]

x265-3.0_Au+24-4217e691387c-win64-multilib (https://drive.google.com/open?id=1gCld9Iy8BubZdvR0o-V5sI07OEwbSuWT)

x265 [info]: HEVC encoder version 3.0_Au+24-4217e691387c
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 8bit+10bit+12bit
x265 [info]: (libavcodec 58.52.101)
x265 [info]: (libavformat 58.27.103)
x265 [info]: (libavutil 56.26.101)
x265 [info]: (lsmash 2.16.1)

[Leeloo Minaï]

now x265 inputs it directly but it encodes it WAY much slower

FFV1 offers one of the most efficient compression rate, size-wise, but it is also slow as hell.
And if it is slower than others lossless codecs in compression, it is also slower in decompression !

I am not surprised that x265 encoded slower too, you should try UTVideo instead.

[stax76]

The help output has one dash too much I think:

---hrd-concat Set HRD concatenation flag for the first keyframe in the buffering period SEI. Default disabled

[MeteorRain]

I tried make it .yuv raw from virtualdub2 using ffv1 lossless

Did you say you make .yuv raw using ffv1 lossless? So would that be an AVI file instead?

[iAvoe]

I wish the next x265 update or build can get rid of piping files in. Something like a straight file input as x264 builds (I'm not sure why it's not working through so many years tho)

[benwaggoner]

I wish the next x265 update or build can get rid of piping files in. Something like a straight file input as x264 builds (I'm not sure why it's not working through so many years tho)
No one is making a build with ffmpeg integrated is why.

[qyot27]

*cough* (https://forum.doom9.org/showpost.php?p=1873612&postcount=6811). We're still on the same forum page.

LAVF input is only supported through patches. MCW more or less put the kibosh on the idea of it being in the upstream source a few years ago, IIRC. Something about not wanting to deal with the maintenance burden of input modules other than raw and y4m, as I recall.

I mean, I use the LAVF support patches for my personal builds of x265 CLI, but more often than not, I just use libx265 through FFmpeg, which is probably more what MCW expects people to do if they want all the exotic input file support.

[StvG]

Last time I tested avs input with this patch (https://github.com/qyot27/x265-Yuuki-Asuna/commit/28b1da7c97abb5ba5f53f751a9d7338eb8e68ee6) was quite slower than avs2yuv+x265.

[Blue_MiSfit]

+1 for just using ffmpeg :)

The only downside AFAICT is when you want to use qpfiles, this still requires piping. I'm sure there are other things too, but this is a big one

[stax76]

How do I use the patch?

x265.exe --crf 18 --output C:\out.hevc C:\test.avs

[avs2 @ 0000000002763500] Format avs2 detected only with low score of 1, misdetection possible!
[avs2 @ 0000000002763500] Could not find codec parameters for stream 0 (Video: avs2, none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[NULL @ 0000000002747480] No codec provided to avcodec_open2()
[NULL @ 0000000002747480] No codec provided to avcodec_open2()
lavf [error]: could not find decoder for video stream
x265 [error]: unable to open input file <C:\Users\frank\Daten\Samples\test_temp\test.avs>

[qyot27]

+1 for just using ffmpeg :)

The only downside AFAICT is when you want to use qpfiles, this still requires piping. I'm sure there are other things too, but this is a big one
Do you mean zonefile? qpfile isn't marked with a CLI only disclaimer in the docs (https://x265.readthedocs.io/en/default/cli.html), so I'd assume it's usable through -x265-params. Most of the stuff marked as CLI only wouldn't be relevant to libx265 use anyway.

How do I use the patch?
Build the patched x265 source against an FFmpeg that has AviSynth support enabled first.

[qyot27]

Last time I tested avs input with this patch (https://github.com/qyot27/x265-Yuuki-Asuna/commit/28b1da7c97abb5ba5f53f751a9d7338eb8e68ee6) was quite slower than avs2yuv+x265.
Don't use 32-bit builds of x265 to encode high bit depth. They have no asm.

Proof:
J:\>x265 --preset ultrafast --crf 18 -o test-direct.mkv testavi.avs
lavf [info]:
Format : avisynth
Codec : rawvideo ( raw video )
PixFmt : yuv422p
Framerate : 24000/1001
Timebase : 1001/24000
Duration : 0:00:10
lavf [info]: 1920x1080 fps 24000/1001 i422p8 frames 0 - 239 of 240
x265 [info]: Using preset ultrafast & tune none
mkv [info]: output file: test-direct.mkv
x265 [info]: HEVC encoder version 2.9+2-7e978ed93d608697
x265 [info]: build info [Windows][GCC 8.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2
x265 [info]: Main 4:2:2 10 profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 2 / wpp(34 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 16
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : dia / 57 / 0 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 0 / 5.00
x265 [info]: Lookahead / bframes / badapt : 5 / 3 / 0
x265 [info]: b-pyramid / weightp / weightb : 1 / 0 / 0
x265 [info]: References / ref-limit cu / depth : 1 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 0.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-18.0 / 0.60
x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip tmvp fast-intra
x265 [info]: tools: strong-intra-smoothing lslices=6 deblock
x265 [info]: frame I: 1, Avg QP:22.97 kb/s: 119.50
x265 [info]: frame P: 60, Avg QP:22.41 kb/s: 5591.55
x265 [info]: frame B: 179, Avg QP:25.33 kb/s: 1784.29
x265 [info]: consecutive B-frames: 1.6% 0.0% 1.6% 96.7%

encoded 240 frames in 16.02s (14.99 fps), 2729.17 kb/s, Avg QP:24.59

J:\>avs2yuv testavi.avs -o - | x265 --y4m --preset ultrafast --crf 18 -o test-pipe.mkv -
testavi.avs: 1920x1080, 24000/1001 fps, 240 frames
converting input clip to YV12
y4m [info]: 1920x1080 fps 24000/1001 i420p8 unknown frame count
x265 [info]: Using preset ultrafast & tune none
mkv [info]: output file: test-pipe.mkv
x265 [info]: HEVC encoder version 2.9+2-7e978ed93d608697
x265 [info]: build info [Windows][GCC 8.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 2 / wpp(34 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 16
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : dia / 57 / 0 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 0 / 5.00
x265 [info]: Lookahead / bframes / badapt : 5 / 3 / 0
x265 [info]: b-pyramid / weightp / weightb : 1 / 0 / 0
x265 [info]: References / ref-limit cu / depth : 1 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 0.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-18.0 / 0.60
x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip tmvp fast-intra
x265 [info]: tools: strong-intra-smoothing lslices=6 deblock
x265 [info]: frame I: 1, Avg QP:22.97 kb/s: 116.81
x265 [info]: frame P: 60, Avg QP:22.44 kb/s: 4963.89
x265 [info]: frame B: 179, Avg QP:25.33 kb/s: 1607.71
x265 [info]: consecutive B-frames: 1.6% 0.0% 1.6% 96.7%

encoded 240 frames in 14.40s (16.67 fps), 2440.54 kb/s, Avg QP:24.60

J:\>

Summary:
Direct input of testavi.avs to 64-bit x265 with LAVF input: 14.99 fps
Piping from 32-bit avs2yuv bm2 to 64-bit x265: 16.67 fps

That's not 'quite slower', it's within the margin of error (especially considering the overhead of having the libav* libraries loaded in the x265 process). Re-running the test with the script converting to 4:2:0 before handing it to x265 or avs2yuv bm3 flipped the values around: direct use was faster than piping:

J:\>x265 --preset ultrafast --crf 18 -o testdirect.mkv testavi.avs
lavf [info]:
Format : avisynth
Codec : rawvideo ( raw video )
PixFmt : yuv420p
Framerate : 24000/1001
Timebase : 1001/24000
Duration : 0:00:10
lavf [info]: 1920x1080 fps 24000/1001 i420p8 frames 0 - 239 of 240
x265 [info]: Using preset ultrafast & tune none
mkv [info]: output file: testdirect.mkv
x265 [info]: HEVC encoder version 2.9+2-7e978ed93d608697
x265 [info]: build info [Windows][GCC 8.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 2 / wpp(34 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 16
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : dia / 57 / 0 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 0 / 5.00
x265 [info]: Lookahead / bframes / badapt : 5 / 3 / 0
x265 [info]: b-pyramid / weightp / weightb : 1 / 0 / 0
x265 [info]: References / ref-limit cu / depth : 1 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 0.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-18.0 / 0.60
x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip tmvp fast-intra
x265 [info]: tools: strong-intra-smoothing lslices=6 deblock
x265 [info]: frame I: 1, Avg QP:22.97 kb/s: 116.81
x265 [info]: frame P: 60, Avg QP:22.44 kb/s: 4963.89
x265 [info]: frame B: 179, Avg QP:25.33 kb/s: 1607.71
x265 [info]: consecutive B-frames: 1.6% 0.0% 1.6% 96.7%

encoded 240 frames in 14.11s (17.01 fps), 2440.54 kb/s, Avg QP:24.60

J:\>avs2yuv testavi.avs -o - | x265 --y4m --preset ultrafast --crf 18 -o testdirect.mkv -
testavi.avs: 1920x1080, 24000/1001 fps, 240 frames
y4m [info]: 1920x1080 fps 24000/1001 i420p8 unknown frame count
x265 [info]: Using preset ultrafast & tune none
mkv [info]: output file: testdirect.mkv
x265 [info]: HEVC encoder version 2.9+2-7e978ed93d608697
x265 [info]: build info [Windows][GCC 8.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 2 / wpp(34 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 16
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : dia / 57 / 0 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 0 / 5.00
x265 [info]: Lookahead / bframes / badapt : 5 / 3 / 0
x265 [info]: b-pyramid / weightp / weightb : 1 / 0 / 0
x265 [info]: References / ref-limit cu / depth : 1 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 0.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-18.0 / 0.60
x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip tmvp fast-intra
x265 [info]: tools: strong-intra-smoothing lslices=6 deblock
x265 [info]: frame I: 1, Avg QP:22.97 kb/s: 116.81
x265 [info]: frame P: 60, Avg QP:22.44 kb/s: 4963.89
x265 [info]: frame B: 179, Avg QP:25.33 kb/s: 1607.71
x265 [info]: consecutive B-frames: 1.6% 0.0% 1.6% 96.7%

encoded 240 frames in 14.35s (16.72 fps), 2440.54 kb/s, Avg QP:24.60

J:\>

Direct: 17.01 fps
Piped: 16.72 fps

The DJATOM fork of avs2yuv is, amusingly, only relevant for 64-bit tests. The 32-bit build would be comparing apples to oranges.

[StvG]

Never used 32-bit x265.
Try the latest version (https://github.com/MasterNobody/avs2yuv/releases). Also I don't pipe 32-bit avs to 64-bit x265. Using avs2yuv_x64 for 64-bit avs+ to 64-bit x265.

Edit:
Avs scriptffvideosource("4K.sample.mkv")
z_convertformat(width/2, height/2, resample_filter="spline36")
pipe - avs2yuv_x64 + x265avs2yuv_x64 -depth 10 1.avs -o - | x265.exe --y4m - --ctu 32 --preset ultrafast -o nul --pools 10 --frame-threads 2 > pipe.txt 2>&1

y4m [info]: 1920x1080 fps 24000/1001 i420p10 unknown frame count
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+25-39b35ea86283
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main 10 profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 10 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 2 / wpp(34 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 16
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : dia / 57 / 0 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 0 / 5.00
x265 [info]: Lookahead / bframes / badapt : 5 / 3 / 0
x265 [info]: b-pyramid / weightp / weightb : 1 / 0 / 0
x265 [info]: References / ref-limit cu / depth : 1 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 0.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-28.0 / 0.60
x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip tmvp fast-intra
x265 [info]: tools: strong-intra-smoothing lslices=6 deblock

x265 [info]: frame I: 18, Avg QP:30.77 kb/s: 3755.34
x265 [info]: frame P: 1084, Avg QP:32.66 kb/s: 714.73
x265 [info]: frame B: 3214, Avg QP:35.33 kb/s: 158.74
x265 [info]: consecutive B-frames: 1.5% 1.2% 1.4% 95.9%

encoded 4316 frames in 45.50s (94.85 fps), 313.38 kb/s, Avg QP:34.64
no pipex265.exe 1.avs --ctu 32 --preset ultrafast -o nul --pools 10 --frame-threads 2 > no_pipe.txt 2>&1

lavf [info]:
Format : avisynth
Codec : rawvideo ( raw video )
PixFmt : yuv420p10le
Framerate : 24000/1001
Timebase : 1001/24000
Duration : 0:03:00
lavf [info]: 1920x1080 fps 24000/1001 i420p10 frames 0 - 4315 of 4316
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+25-39b35ea86283
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main 10 profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 10 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 2 / wpp(34 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 16
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : dia / 57 / 0 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 0 / 5.00
x265 [info]: Lookahead / bframes / badapt : 5 / 3 / 0
x265 [info]: b-pyramid / weightp / weightb : 1 / 0 / 0
x265 [info]: References / ref-limit cu / depth : 1 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 0.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-28.0 / 0.60
x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip tmvp fast-intra
x265 [info]: tools: strong-intra-smoothing lslices=6 deblock

x265 [info]: frame I: 18, Avg QP:30.77 kb/s: 3755.34
x265 [info]: frame P: 1084, Avg QP:32.66 kb/s: 714.73
x265 [info]: frame B: 3214, Avg QP:35.33 kb/s: 158.74
x265 [info]: consecutive B-frames: 1.5% 1.2% 1.4% 95.9%

encoded 4316 frames in 52.69s (81.91 fps), 313.38 kb/s, Avg QP:34.64

It's my bad that I wrote "quite slower", I should write just "slower", but I was surprised that the difference is >10%, I expected to be marginal within statistical error (~1%).

[qyot27]

Never used 32-bit x265.
Try the latest version (https://github.com/MasterNobody/avs2yuv/releases). Also I don't pipe 32-bit avs to 64-bit x265. Using avs2yuv_x64 for 64-bit avs+ to 64-bit x265.

Edit:
Avs scriptffvideosource("4K.sample.mkv")
z_convertformat(width/2, height/2, resample_filter="spline36")
pipe - avs2yuv_x64 + x265avs2yuv_x64 -depth 10 1.avs -o - | x265.exe --y4m - --ctu 32 --preset ultrafast -o nul --pools 10 --frame-threads 2 > pipe.txt 2>&1

y4m [info]: 1920x1080 fps 24000/1001 i420p10 unknown frame count
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+25-39b35ea86283
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

...snip...

encoded 4316 frames in 45.50s (94.85 fps), 313.38 kb/s, Avg QP:34.64
no pipex265.exe 1.avs --ctu 32 --preset ultrafast -o nul --pools 10 --frame-threads 2 > no_pipe.txt 2>&1

lavf [info]:
Format : avisynth
Codec : rawvideo ( raw video )
PixFmt : yuv420p10le
Framerate : 24000/1001
Timebase : 1001/24000
Duration : 0:03:00
lavf [info]: 1920x1080 fps 24000/1001 i420p10 frames 0 - 4315 of 4316
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+25-39b35ea86283
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

...snip...

encoded 4316 frames in 52.69s (81.91 fps), 313.38 kb/s, Avg QP:34.64

It's my bad that I wrote "quite slower", I should write just "slower", but I was surprised that the difference is >10%, I expected to be marginal within statistical error (~1%).
Using the same options and bm5 binary of avs2yuv, the x265 binary I built a week ago (rather than one from last October), a longer chunk of frames as a representative sample, and having moved all the relevant input files and avs2yuv binaries to my SSD rather than running it from a USB 3.0 flash drive:

Script:
FFVideoSource("test.mp4").ConvertBits(10)

Piped:
E:\>avs2yuv64 -depth 10 test.avs -o - | x265.exe --y4m - --ctu 32 --preset ultrafast --output-depth 10 -o nul --pools 10 --frame-threads 2 --frames 2400
test.avs: 1920x1080, YUV420P10, 10-bits, progressive, 24000/1001 fps, 31122 frames
y4m [info]: 1920x1080 fps 24000/1001 i420p10 unknown frame count
x265 [info]: Using preset ultrafast & tune none
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+22-feec4bdf98663ac4
x265 [info]: build info [Windows][GCC 9.1.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2

...snip...

encoded 2400 frames in 164.00s (14.63 fps), 1002.53 kb/s, Avg QP:34.62
error: wrote only 6096584 of 6220800 bytes

Direct:
E:\>x265.exe test.avs --ctu 32 --preset ultrafast --output-depth 10 -o nul --pools 10 --frame-threads 2 --frames 2400
lavf [info]:
Format : avisynth
Codec : rawvideo ( raw video )
PixFmt : yuv420p10le
Framerate : 24000/1001
Timebase : 1001/24000
Duration : 0:21:38
lavf [info]: 1920x1080 fps 24000/1001 i420p10 frames 0 - 2399 of 31122
x265 [info]: Using preset ultrafast & tune none
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+22-feec4bdf98663ac4
x265 [info]: build info [Windows][GCC 9.1.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2

...snip...

encoded 2400 frames in 160.87s (14.92 fps), 1002.53 kb/s, Avg QP:34.62

Piped: 14.63 fps
Direct: 14.92 fps

I even enabled MT (Prefetch(4)).

Piped:
E:\>avs2yuv64 -depth 10 test.avs -o - | x265.exe --y4m - --ctu 32 --preset ultrafast --output-depth 10 -o nul --pools 10
--frame-threads 2 --frames 2400
test.avs: 1920x1080, YUV420P10, 10-bits, progressive, 24000/1001 fps, 31122 frames
y4m [info]: 1920x1080 fps 24000/1001 i420p10 unknown frame count
x265 [info]: Using preset ultrafast & tune none
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+22-feec4bdf98663ac4
x265 [info]: build info [Windows][GCC 9.1.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2

...snip...

encoded 2400 frames in 172.52s (13.91 fps), 1002.53 kb/s, Avg QP:34.62
error: wrote only 6096584 of 6220800 bytes

Direct:
E:\>x265.exe test.avs --ctu 32 --preset ultrafast --output-depth 10 -o nul --pools 10 --frame-threads 2 --frames 2400
lavf [info]:
Format : avisynth
Codec : rawvideo ( raw video )
PixFmt : yuv420p10le
Framerate : 24000/1001
Timebase : 1001/24000
Duration : 0:21:38
lavf [info]: 1920x1080 fps 24000/1001 i420p10 frames 0 - 2399 of 31122
x265 [info]: Using preset ultrafast & tune none
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+22-feec4bdf98663ac4
x265 [info]: build info [Windows][GCC 9.1.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2

...snip...

encoded 2400 frames in 166.78s (14.39 fps), 1002.53 kb/s, Avg QP:34.62

[StvG]

Maybe bottleneck for you is x265 so the speed difference between direct and piped is killed?
Here is x265 I built for the test above (https://www.upload.ee/files/9972743/x265.7z.html).

Edit:
tested with crf 18
pipeavs2yuv_x64 -depth 10 1.avs -o - | x265.exe --y4m - --ctu 32 --preset ultrafast -o nul --pools 10 --crf 18 --frame-threads 2 > pipe.txt 2>&1

y4m [info]: 1920x1080 fps 24000/1001 i420p10 unknown frame count
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+25-39b35ea86283
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

encoded 4316 frames in 50.39s (85.64 fps), 1629.90 kb/s, Avg QP:24.65
directx265.exe 1.avs --ctu 32 --preset ultrafast -o nul --pools 10 --crf 18 --frame-threads 2 > direct.txt 2>&1

lavf [info]:
Format : avisynth
Codec : rawvideo ( raw video )
PixFmt : yuv420p10le
Framerate : 24000/1001
Timebase : 1001/24000
Duration : 0:03:00
lavf [info]: 1920x1080 fps 24000/1001 i420p10 frames 0 - 4315 of 4316
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+25-39b35ea86283
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

encoded 4316 frames in 55.19s (78.21 fps), 1629.90 kb/s, Avg QP:24.65
tested with crf 18 and asm sse4.2
pipeavs2yuv_x64 -depth 10 1.avs -o - | x265.exe --y4m - --ctu 32 --preset ultrafast -o nul --pools 10 --crf 18 --frame-threads 2 > pipe.txt 2>&1 --asm sse4.2

y4m [info]: 1920x1080 fps 24000/1001 i420p10 unknown frame count
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+25-39b35ea86283
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2

encoded 4316 frames in 58.54s (73.73 fps), 1629.90 kb/s, Avg QP:24.65
directx265.exe 1.avs --ctu 32 --preset ultrafast -o nul --pools 10 --crf 18 --frame-threads 2 > direct.txt 2>&1 --asm sse4.2

lavf [info]:
Format : avisynth
Codec : rawvideo ( raw video )
PixFmt : yuv420p10le
Framerate : 24000/1001
Timebase : 1001/24000
Duration : 0:03:00
lavf [info]: 1920x1080 fps 24000/1001 i420p10 frames 0 - 4315 of 4316
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+25-39b35ea86283
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2

encoded 4316 frames in 61.19s (70.53 fps), 1629.90 kb/s, Avg QP:24.65
tested real encoding scenario where bottleneck is x265
pipeavs2yuv_x64 -depth 10 1.avs -o - | x265.exe --y4m - --ctu 32 --preset slower --crf 18 -o nul --frame-threads 2 --no-amp --no-sao --no-strong-intra-smoothing > pipe.txt 2>&1 --asm avx512

y4m [info]: 1920x1080 fps 24000/1001 i420p10 unknown frame count
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+25-39b35ea86283
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512

encoded 501 frames in 153.32s (3.27 fps), 10340.94 kb/s, Avg QP:22.84
directx265.exe 1.avs --ctu 32 --preset slower --crf 18 -o nul --frame-threads 2 --no-amp --no-sao --no-strong-intra-smoothing > direct.txt 2>&1 --asm avx512

lavf [info]:
Format : avisynth
Codec : rawvideo ( raw video )
PixFmt : yuv420p10le
Framerate : 24000/1001
Timebase : 1001/24000
Duration : 0:00:20
lavf [info]: 1920x1080 fps 24000/1001 i420p10 frames 0 - 500 of 501
raw [info]: output file: nul
x265 [info]: HEVC encoder version 3.0_Au+25-39b35ea86283
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512

encoded 501 frames in 153.33s (3.27 fps), 10340.94 kb/s, Avg QP:22.84

[qyot27]

Maybe bottleneck for you is x265 so the speed difference between direct and piped is killed?
On a Celeron J3455? Incredibly likely. The lack of AVX2 alone probably makes the biggest difference.

If you use FFmpeg with libx265, what fps does it pull? If it's close to the same speed as the LAVF input in x265, the difference is due to the way the libavformat AviSynth demuxer is written.

[StvG]

Almost the same speed as "direct"ffmpeg.exe -i 1.avs -c:v libx265 -preset ultrafast -x265-params ctu=32:pools=10:frame-threads=2 -f rawvideo - > x265.txt 2>&1

ffmpeg version 7211e1c Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 9.1.1 (GCC) 20190517
configuration: --enable-libx265 --enable-avisynth --extra-cflags='-march=core-avx2 -mtune=native -pipe -s -O2' --cpu=core-avx2 --disable-debug --enable-gpl --cross-prefix=x86_64-w64-mingw32- --target-os=mingw32 --arch=x86_64 --pkg-config-flags=--static --enable-zlib --disable-pthreads --enable-w32threads --enable-version3
libavutil 56. 22.100 / 56. 22.100
libavcodec 58. 35.100 / 58. 35.100
libavformat 58. 20.100 / 58. 20.100
libavdevice 58. 5.100 / 58. 5.100
libavfilter 7. 40.101 / 7. 40.101
libswscale 5. 3.100 / 5. 3.100
libswresample 3. 3.100 / 3. 3.100
libpostproc 55. 3.100 / 55. 3.100
Input #0, avisynth, from '1.avs':
Duration: 00:03:00.01, start: 0.000000, bitrate: 0 kb/s
Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le, 1920x1080, 23.98 fps, 23.98 tbr, 23.98 tbn, 23.98 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> hevc (libx265))
Press [q] to stop, [?] for help
x265 [info]: HEVC encoder version 3.0_Au+25-39b35ea86283
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2

Output #0, rawvideo, to 'pipe:':
Metadata:
encoder : Lavf58.20.100
Stream #0:0: Video: hevc (libx265), yuv420p10le, 1920x1080, q=2-31, 23.98 fps, 23.98 tbn, 23.98 tbc
Metadata:
encoder : Lavc58.35.100 libx265

encoded 4316 frames in 52.01s (82.98 fps), 313.44 kb/s, Avg QP:34.64

[filler56789]

x265.exe 3.0_Au+25-39b35ea86283
(x64, multilib, GCC 7.4.0)

http://www.mediafire.com/file/ezp7c7s00c6j664/x265_3.0_Au%2B25-39b35ea86283.7z

[Barough]

x265 v3.0_Au+27-3f4fb9a2ac68 (https://www.mediafire.com/file/4q6krzl1n07dqem/x265-3.0_Au+27-3f4fb9a2ac68_Win_GCC.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit-GCC v7.4.0 / 64bit-GCC v8.3.0)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[Barough]

x265 v3.0_Au+30-b9bef1a4c34a (https://www.mediafire.com/file/bcwgvqq5fjs35cy/x265-3.0_Au+30-b9bef1a4c34a_Win_GCC.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit-GCC v7.4.0 / 64bit-GCC v8.3.0)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[Barough]

x265 v3.0_Au+31-4583000db964 (https://www.mediafire.com/file/ems39bb8ct5gsjr/x265-3.0_Au+31-4583000db964_Win_GCC.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit-GCC v7.4.0 / 64bit-GCC v8.3.0)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[stax76]

@Barough

Thanks for the new built. Is there a change log somewhere?

[Barough]

@Barough

Thanks for the new built. Is there a change log somewhere?

YW stax76 :)

You need 2 check the commits comments on the Bitbucket URL above. There is only a changelog available for the v3.0 Stable version.

[stax76]

@Barough

Thanks, I found it now.


@Wolfberry

avs input is still not working for me:

[avs2 @ 0000000004623380] Format avs2 detected only with low score of 1, misdetection possible!
[avs2 @ 0000000004623380] Could not find codec parameters for stream 0 (Video: avs2, none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[NULL @ 0000000004635340] No codec provided to avcodec_open2()
[NULL @ 0000000004635340] No codec provided to avcodec_open2()
lavf [error]: could not find decoder for video stream
x265 [error]: unable to open input file <C:\Users\john&janedoe\Daten\Samples\test_temp\test.avs>

[Barough]

x265 v3.0_Au+32-a46ded2c1411 (https://www.mediafire.com/file/upisl3t05o71h0g/x265-3.0_Au+32-a46ded2c1411_Win_GCC.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (32bit-GCC v7.4.0 / 64bit-GCC v8.3.0)

https://bitbucket.org/multicoreware/x265/commits/branch/default

[benwaggoner]

Some interesting checkins in the last couple of weeks, with new parameters!

Has anyone played with the --fades command? As described, it sounds like a useful always-on feature to improve quality/efficiency of fades. x265 didn't seem to have a particular issue with those, but fades are a special case where efficiency and random access can be improved via the described mode.

[Wolfberry]

avs input is still not working for me

I did not build ffmpeg with avisynth support in that build.

Try this build (https://drive.google.com/open?id=1Ouvyj85LeV2yw0mYoz8SJgTb2ognaI7z)

x265 [info]: HEVC encoder version 3.0+2-b4b1d84566d7
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 8bit+10bit+12bit
x265 [info]: (libavcodec 58.52.102)
x265 [info]: (libavformat 58.27.103)
x265 [info]: (libavutil 56.28.100)
x265 [info]: (lsmash 2.16.1)

[stax76]

@Wolfberry

Works perfectly, awesome!

@everybody

There was a built that did show the estimated output file size and people have posted positive feedback about this feature, I like it too. Sadly I don't remember from where I got that built.

[vanden]

Sorry, I may not be in the right forum ...

Here is my problem, I have 2 rip bluray 2160p and the problem is that it must miss some info.

Blade Runner 2049:
Do not pass with "madMeasureDynamicClipping", I rename the file .measurements.incomplete in .measurements and I open it with "madMeasureDynamicClipping":
https://nsa40.casimages.com/img/2019/06/01/mini_190601120901821988.jpg (https://www.casimages.com/i/190601120901821988.jpg.html)
************** Texte de l'exception **************
System.OverflowException: The arithmetic operation caused an overflow.
à madMeasureDynamicClipping.FrmMain.ImportData()
à madMeasureDynamicClipping.FrmMain.StartAnalysis()
à System.Windows.Forms.Control.OnClick(EventArgs e)
à System.Windows.Forms.Button.OnClick(EventArgs e)
à System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
à System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
à System.Windows.Forms.Control.WndProc(Message& m)
à System.Windows.Forms.ButtonBase.WndProc(Message& m)
à System.Windows.Forms.Button.WndProc(Message& m)
à System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
**************
Reading it with "madMeasureDynamicClipping" I get this :
https://nsa40.casimages.com/img/2019/06/01/mini_190601120904919036.jpg (https://www.casimages.com/i/190601120904919036.jpg.html)
By removing all the .measurements.incomplete and .measurements files, I get this :
https://nsa40.casimages.com/img/2019/06/01/mini_190601120857262445.jpg (https://www.casimages.com/i/190601120857262445.jpg.html)
Pass in "Tone map HDR MadVR" :
https://nsa40.casimages.com/img/2019/06/01/mini_190601120623189391.jpg (https://www.casimages.com/i/190601120623189391.jpg.html)

Sully:
Do not pass with "madMeasureDynamicClipping" :
https://image.noelshack.com/minis/2019/22/4/1559206661-sully2.png (https://www.noelshack.com/2019-22-4-1559206661-sully2.jpg)
Pass in "Tone map HDR MadVR" :
https://nsa40.casimages.com/img/2019/06/01/mini_190601121906235835.jpg (https://www.casimages.com/i/190601121906235835.jpg.html)


Do you have a solution ?


T2 Trainspotting everything is ok :
Ok avec "madMeasureDynamicClipping" :
https://nsa40.casimages.com/img/2019/06/01/mini_19060112302966618.jpg (https://www.casimages.com/i/19060112302966618.jpg.html)
Reading it with "madMeasureDynamicClipping" I get this :
https://nsa40.casimages.com/img/2019/06/01/mini_190601121952878821.jpg (https://www.casimages.com/i/190601121952878821.jpg.html)
By removing all the .measurements files, I get this :
https://nsa40.casimages.com/img/2019/06/01/mini_190601123401283975.jpg (https://www.casimages.com/i/190601123401283975.jpg.html)
Pass in "Tone map HDR MadVR" :
https://nsa40.casimages.com/img/2019/06/01/mini_190601123539772152.jpg (https://www.casimages.com/i/190601123539772152.jpg.html)

[Wolfberry]

There was a built that did show the estimated output file size and people have posted positive feedback about this feature, I like it too. Sadly I don't remember from where I got that built.

The feature you are referring to is probably this commit: Cosmetic: x264-r2204 style progress indicator (https://github.com/msg7086/x265-Yuuki-Asuna/commit/daa22d26c2bd6a6a93e7f5107c00f62d964d4a4a)

@MeteorRain's signature contains the link (https://down.7086.in/x265-Yuuki-Asuna/)to the x265 binaries built from the Yuuki / Asuna branch of https://github.com/msg7086/x265-Yuuki-Asuna.

[stax76]

The zones feature is broken in 3.0+2, it encodes to the end but returns an error code, cmd/batch users normally don't check for the exit code, so they don't notice the problem but staxrip treats this a fatal and aborts further processing.

ffmpeg -i test.avs -f yuv4mpegpipe - | x265 --crf 22 --output-depth 10 --zones 0,200,b=1.2 --frames 300 --y4m --output test.hevc -

echo %ERRORLEVEL%
pause

Video encoding using x265 3.0+2 Wolfberry failed with exit code: -1073740940 (0xC0000374)

The exit code might be a system error code: A heap has been corrupted.

The feature you are referring to is probably this commit: Cosmetic: x264-r2204 style progress indicator

@MeteorRain's signature contains the link to the x265 binaries built from the Yuuki / Asuna branch of https://github.com/msg7086/x265-Yuuki-Asuna.

Probably yes, thanks.

[Jamaika]

I have a question.
Is it possible to use tune vmaf in x265 v3.1?
x265 has 5 tune modes (psnr, ssim, grain, zero-latency, animation) whereas SVT-HEVC
has only 3 tune modes (0 - visual quality, 1 - PSNR / SSIM and 2 - VMAF). Below
table shows the mapping of tune modes,

+-----------------------+---------------------------+
| x265 Tune Modes | SVT-HEVC Tune Modes |
+=======================+===========================+
| vmaf | 2 |
+-----------------------+---------------------------+
| psnr | 1 |
+-----------------------+---------------------------+
| ssim | 1 |
+-----------------------+---------------------------+
| grain | 0 |
+-----------------------+---------------------------+
| fastdecode | 0 |
+-----------------------+---------------------------+
| zerolatency | 0 |
+-----------------------+---------------------------+
| animation | 0 |

If so how to initiate it?
api->param_default_preset(p, preset, "vmaf");

[Barough]

x265 v3.1_RC1+1-10decf67c077 (https://www.mediafire.com/file/yddwwadpx8f859p/x265-3.1_RC1+1-10decf67c077_Win_GCC910.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 9.1.0)

https://bitbucket.org/multicoreware/x265/commits/branch/Release_3.1

[Natty]

Did you say you make .yuv raw using ffv1 lossless? So would that be an AVI file instead?

i love your yuuki x265 mod, i used the latest 3.1 version, but observed that it hides encoding settings in mediainfo. i didn't change any encoding setting while using your mod..

hoping to get regular gcc 9.1 build updates from you. :thanks:

[MeteorRain]

i love your yuuki x265 mod, i used the latest 3.1 version, but observed that it hides encoding settings in mediainfo. i didn't change any encoding setting while using your mod..

hoping to get regular gcc 9.1 build updates from you. :thanks:

Natty,

Thank you for the kind word. I have fixed the issue and slip-streamed to the same file.

I intentionally downgraded to gcc 8.3 as I expect a performance regression on newer gcc. But I might change my mind and upgrade to gcc 9.1 on final build. Suggestions welcome.

[Natty]

Natty,

Thank you for the kind word. I have fixed the issue and slip-streamed to the same file.

I intentionally downgraded to gcc 8.3 as I expect a performance regression on newer gcc. But I might change my mind and upgrade to gcc 9.1 on final build. Suggestions welcome.

it working fine :thanks:

[birdie]

Has the meaning of CRF wildly changed since version 2.4?

I'm encoding the same source video using 2.4 and 3.0 using absolutely the same options ( -preset veryslow -x265-params keyint=600:min-keyint=30:bframes=16:crf=22:no-sao=1 - it's relatively static which is why keyint is so high) and 3.0 produces the file which is significantly heavier.

104,103,181 bytes for v2.4
128,682,330 bytes for v3.0

Also encoding has become significantly slower:

2.4: encoded 10656 frames in 6302.40s (1.69 fps), 2229.94 kb/s, Avg QP:27.68
3.0: encoded 10656 frames in 11981.75s (0.89 fps), 2780.38 kb/s, Avg QP:27.30

Full output for 2.4:

x265 [info]: HEVC encoder version 2.4
x265 [info]: build info [Linux][GCC 6.3.1][64 bit] 8bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main profile, Level-3.1 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 2 / wpp(12 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 4 / 4
x265 [info]: Keyframe min / max / scenecut / bias: 30 / 600 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 40 / 16 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 5 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-22.0 / 0.60
x265 [info]: tools: rect amp limit-modes rd=6 psy-rd=2.00 rdoq=2 psy-rdoq=1.00
x265 [info]: tools: rskip limit-tu=4 signhide tmvp b-intra
x265 [info]: tools: strong-intra-smoothing deblock
Output #0, matroska, to '265.mkv':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.71.100
Stream #0:0(eng): Video: hevc (libx265), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 29.83 fps, 1k tbn, 29.83 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc57.89.100 libx265
Stream #0:1(eng): Audio: aac (LC) ([255][0][0][0] / 0x00FF), 48000 Hz, mono, fltp, 96 kb/s (default)
Metadata:
handler_name : SoundHandler
frame=10656 fps=1.7 q=-0.0 Lsize= 101663kB time=00:05:57.24 bitrate=2331.2kbits/s speed=0.0567x
video:97282kB audio:4187kB subtitle:0kB other streams:0kB global headers:2kB muxing overhead: 0.192047%
x265 [info]: frame I: 20, Avg QP:20.80 kb/s: 13832.49
x265 [info]: frame P: 2322, Avg QP:22.77 kb/s: 6661.60
x265 [info]: frame B: 8314, Avg QP:29.07 kb/s: 964.32
x265 [info]: Weighted P-Frames: Y:11.9% UV:9.3%
x265 [info]: Weighted B-Frames: Y:6.8% UV:4.4%
x265 [info]: consecutive B-frames: 7.3% 5.3% 8.5% 30.1% 16.4% 21.8% 6.9% 3.2% 0.2% 0.2% 0.1% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%


Full output for 3.0:

x265 [info]: HEVC encoder version 3.0
x265 [info]: build info [Linux][GCC 9.1.1][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main profile, Level-3.1 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 2 / wpp(12 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 4 / 5
x265 [info]: Keyframe min / max / scenecut / bias: 30 / 600 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 40 / 16 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 5 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-22.0 / 0.60
x265 [info]: tools: rect amp rd=6 psy-rd=2.00 rdoq=2 psy-rdoq=1.00 rskip
x265 [info]: tools: signhide tmvp b-intra strong-intra-smoothing deblock
Output #0, matroska, to '265.mkv':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: isommp42
com.android.version: 6.0
encoder : Lavf58.20.100
Stream #0:0(eng): Video: hevc (libx265), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 29.83 fps, 1k tbn, 29.83 tbc (default)
Metadata:
encoder : Lavc58.35.100 libx265
handler_name : VideoHandle
Stream #0:1(eng): Audio: aac (LC) ([255][0][0][0] / 0x00FF), 48000 Hz, mono, fltp, 96 kb/s (default)
Metadata:
handler_name : SoundHandle
frame=10656 fps=0.9 q=-0.0 Lsize= 125666kB time=00:05:57.24 bitrate=2881.6kbits/s speed=0.0298x
video:121284kB audio:4187kB subtitle:0kB other streams:0kB global headers:2kB muxing overhead: 0.155674%
x265 [info]: frame I: 20, Avg QP:20.69 kb/s: 16303.41
x265 [info]: frame P: 2322, Avg QP:22.30 kb/s: 8450.15
x265 [info]: frame B: 8314, Avg QP:28.71 kb/s: 1164.35
x265 [info]: Weighted P-Frames: Y:11.9% UV:9.3%
x265 [info]: Weighted B-Frames: Y:6.8% UV:4.4%
x265 [info]: consecutive B-frames: 7.3% 5.3% 8.5% 30.1% 16.4% 21.8% 6.9% 3.2% 0.2% 0.2% 0.1% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%

encoded 10656 frames in 11981.75s (0.89 fps), 2780.38 kb/s, Avg QP:27.30

[ChaosKing]

Different settings affect the CRF output: https://x265.readthedocs.io/en/default/releasenotes.html#encoder-enhancements

Preset: change param defaults for veryslow and slower preset. Replace slower preset with defaults used in veryslow preset and change param defaults in veryslow preset as per experimental results.
AQ: change default AQ mode to auto-variance

[Boulder]

Filesize is probably due to the default aq-mode being 2 now. It often causes a higher average bitrate with the same CRF than aq-mode 1.

[Blue_MiSfit]

The preset changes also mean veryslow is radically different relative to 2.4.

Just saw this was mentioned already :D

[katzenjoghurt]

The zones feature is broken in 3.0+2, it encodes to the end but returns an error code, cmd/batch users normally don't check for the exit code, so they don't notice the problem but staxrip treats this a fatal and aborts further processing.[...]


Could some kind soul please, please, pretty please investigate this bug?
It is so annoying. :(

It means that after every encoding with zones (with an affected tool) your would need to do the muxing manually afterwards
as the tool crashes away due to the returned x265 error code.

See also: https://bitbucket.org/multicoreware/x265/issues/490/x265-crash-when-using-zone

[birdie]

Different settings affect the CRF output: https://x265.readthedocs.io/en/default/releasenotes.html#encoder-enhancements

My settings were 100% identical.

[Boulder]

No they're not if you look at the output in your post. I still think the biggest difference comes from aq-mode changing from 1 to 2. The other changes should cause a much smaller change in the bitrate.

EDIT: as mentioned earlier, the presets have been changed. Veryslow in v2.4 is different from v3.0.

[Barough]

x265 v3.1_RC1+3-3bdf06e3c628 (https://www.mediafire.com/file/2y44i8i3hhybv76/x265-3.1_RC1+3-3bdf06e3c628_Win_GCC910.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 9.1.0)

https://bitbucket.org/multicoreware/x265/commits/branch/Release_3.1

[StvG]

Could some kind soul please, please, pretty please investigate this bug?
It is so annoying. :(

It means that after every encoding with zones (with an affected tool) your would need to do the muxing manually afterwards
as the tool crashes away due to the returned x265 error code.

See also: https://bitbucket.org/multicoreware/x265/issues/490/x265-crash-when-using-zone

Use VS20xx not GCC x.x build from here (http://msystem.waw.pl/x265/).

[katzenjoghurt]

Hey StvG,

in fact I already do. I'm using the VS2019 versions from there.

[katzenjoghurt]

Oh my.
Am I the only one having such a hard time encoding scenes with red light / red backgrounds?

[...]

Oh man. I THINK I finally found a solution for my neverending problem with red areas getting blurred and/or blocky.

People already gave me a hint to use 10bit encoding or playing around with the crqpoffs parameter. Both didn't really help.

This changed drastically now after I converted the source to YV24 colorspace and encoded it with Main 444 10.

Suddenly the crqpoffs parameter started to work wonders and setting it to -1 already brought back most of the details.


How to do it in StaxRip 1.7.0.6:
1) Click on the "AVS Filter" label. Click "Profiles".
2) Find the [Misc] section and add this line to the bottom: ConvertToYV24 = ConvertToYV24()
3) Add the Filter now via right-click -> Misc -> ConvertToYV24
4) Go into the x265 encoder settings
5) In "Basic" chose the Main 444 10 profile.
6) In "Rate Control 1" set CR QB Offset to -1.

[StvG]

Hey StvG,

in fact I already do. I'm using the VS2019 versions from there.
Hey. A week ago or so I tested VS2019 builds and I had no problems when using zones. GCC builds were crashing when using zones.

[birdie]

No they're not if you look at the output in your post. I still think the biggest difference comes from aq-mode changing from 1 to 2. The other changes should cause a much smaller change in the bitrate.

EDIT: as mentioned earlier, the presets have been changed. Veryslow in v2.4 is different from v3.0.

I thought CRF actually meant something. Not really a given bitrate but something close to it.

[mini-moose]

I thought CRF actually meant something. Not really a given bitrate but something close to it.

the lower the CRF value, the higher the bitrate. slowdown is due to preset changes, bitrate differences are due to AQ mode changing from AQ1 default to AQ2 default.

I found that AQ1 was giving a higher bitrate than AQ2, but I suppose it depends on source. I think AQ1 is constant while AQ2 is variable.

[LigH]

CRF means: Constant Rate Factor. It keeps an internal distortion metric in the encoding workflow (called "rate factor") below a threshold. But many details are part of the RF calculation, and not all represent your personal subjective quality impression perfectly. Yet, it is a much better one than PSNR and allows an easy adjustment of the quantization with some respect to the encoding complexity. But a few seemingly paradox results may happen. Heavier efforts do not always result in more efficient video stream code or visually more convenient pictures, exceptions from a general rule aren't impossible.

[jlpsvk]

welcome back LigH. Missing new builds from you. :) Haven't seen you for a longer time here active.

[katzenjoghurt]

Hey. A week ago or so I tested VS2019 builds and I had no problems when using zones. GCC builds were crashing when using zones.

Oh man. You put my hopes so high... :D
Tried out RC1+3 (built with VS2019) but... no luck - it still crashes for me.



Video encoding [...] failed with exit code: -1073740940 (0xC0000374)

The exit code might be a system error code: Ein Heap wurde beschädigt.

[...]
x265 [info]: HEVC encoder version 3.1_RC1+3-3bdf06e3c628
[...]

*sigh*

[benwaggoner]

So, I am trying to do a lossless HDR encoding test.

x265.exe - --y4m --colorprim bt2020 --transfer smpte2084 --colormatrix bt2020nc --hdr --hdr-opt -o 4K_LOSSLESS.hevc

And I get this error, which sounds like hdr-opt is getting deactivated because it is confused about the metadata?

Output #0, yuv4mpegpipe, to 'pipe:':
Metadata:
encoder : Lavf58.27.103
Stream #0:0: Video: wrapped_avframe, yuv420p10le, 3840x2160, q=2-31, 200 kb/s, 24 fps, 24 tbn, 24 tbc
Metadata:
encoder : Lavc58.53.100 wrapped_avframe
y4m [info]: 3840x2160 fps 24/1 i420p10 unknown frame count
raw [info]: output file: D:\Octarine\foo_4k_HDR.hevc
x265 [info]: HEVC encoder version 3.1_RC1+3-3bdf06e3c628
x265 [info]: build info [Windows][MSVC 1921][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [error]: Recommended Settings for HDR: colour primaries should be BT.2020,
transfer characteristics should be SMPTE ST.2084,
matrix coeffs should be BT.2020,
the input video should be 10 bit 4:2:0
Disabling offset tuning for HDR videos
x265 [warning]: Turning on repeat-headers for HDR compatibility

Everything seems set up just fine. Any thoughts about what might be going on here?

[LigH]

welcome back LigH. Missing new builds from you.

Ah, well ... had some issues with other projects compiled in MABS, and more work thus less spare time. And there are so many competitors now. But a "merge with stable" is a reason to publish another one.

[Blue_MiSfit]

@benwaggoner I'd suggest patching out the feature that disables hdr-opt in certain conditions. You certainly know what you're doing and it's a one-line fix ;)

[benwaggoner]

@benwaggoner I'd suggest patching out the feature that disables hdr-opt in certain conditions. You certainly know what you're doing and it's a one-line fix ;)
First off, I want to know what the conditions are. I don't dismiss the possibility that I don't know what I'm doing in some aspect :sly:.

[Barough]

x265 v3.1+2-b36c03e4e771 (https://www.mediafire.com/file/h2t7taqxxikptfr/x265-3.1+2-b36c03e4e771_Win_GCC910.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 9.1.0)


https://bitbucket.org/multicoreware/x265/commits/branch/default

[Blue_MiSfit]

@benwaggoner

https://bitbucket.org/multicoreware/x265/src/b36c03e4e7719da0c708218192357a17f0a3d42b/source/encoder/encoder.cpp#lines-3316


if (p->internalCsp != X265_CSP_I420 || p->internalBitDepth != 10 || p->vui.colorPrimaries != 9 ||
p->vui.transferCharacteristics != 16 || p->vui.matrixCoeffs != 9)


So, if any of the following are NOT true x265 will disable hdr-opt

4:2:0
10 bit
color primaries = 9 (bt2020)
transfer characteristics = 16 (smpte2084)
matrix coefficients = 9 (bt2020nc)

It does look like you're set up correctly, so no idea why this is happening.

I've found hdr-opt to be helpful for Dolby Vision Profile 5 encoding, which is incompatible with the required VUI to enable it, so I just patched out the above code block :devil:

[StvG]

Oh man. You put my hopes so high... :D
Tried out RC1+3 (built with VS2019) but... no luck - it still crashes for me.



Video encoding [...] failed with exit code: -1073740940 (0xC0000374)

The exit code might be a system error code: Ein Heap wurde beschädigt.

[...]
x265 [info]: HEVC encoder version 3.1_RC1+3-3bdf06e3c628
[...]

*sigh*

gcc:
avs2yuv_x64 -depth 10 1.avs -o - | x265-10b.exe --y4m - --level-idc 5.1 --high-tier --ref 6 --bframes 12 --rd 4 --me 3 --subme 5 --merange 57 --ipratio 1.2 --pbratio 1.1 --aq-mode 3 --aq-strength 0.95 --qcomp 0.70 --psy-rd 1.35 --psy-rdoq 1.20 --ctu 32 --rc-lookahead 60 --deblock -3:-3 --cbqpoffs 0 --crqpoffs 0 --qg-size 8 --no-rskip --no-rect --no-amp --no-sao --no-open-gop --no-early-skip --no-cutree --tu-intra-depth 4 --tu-inter-depth 4 --range limited --aud --repeat-headers --hrd --hdr-opt --colorprim bt2020 --colormatrix bt2020nc --transfer smpte2084 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,20)" --max-cll=0,0 --input-depth 10 --chromaloc 2 --preset slower -o gcc.mkv --asm avx512 --zones 2100,3000,b=1.22/9000,10500,b=1.3/13000,13500,b=1.1/22000,22320,b=1.11/25000,25300,b=1.15 --crf 17 > gcc.txt 2>&1y4m [info]: 1920x1080 fps 24000/1001 i420p10 unknown frame count
raw [info]: output file: gcc.mkv
x265 [info]: HEVC encoder version 3.1_RC1+3-3bdf06e3c628
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
...
30194 frames: 5.47 fps, 20856.40 kb/s
30198 frames: 5.47 fps, 20855.46 kb/s
30201 frames: 5.47 fps, 20854.86 kb/s
30206 frames: 5.47 fps, 20853.95 kb/s
30210 frames: 5.47 fps, 20852.97 kb/s
vs2019:
avs2yuv_x64 -depth 10 1.avs -o - | x265-10bvs.exe --y4m - --level-idc 5.1 --high-tier --ref 6 --bframes 12 --rd 4 --me 3 --subme 5 --merange 57 --ipratio 1.2 --pbratio 1.1 --aq-mode 3 --aq-strength 0.95 --qcomp 0.70 --psy-rd 1.35 --psy-rdoq 1.20 --ctu 32 --rc-lookahead 60 --deblock -3:-3 --cbqpoffs 0 --crqpoffs 0 --qg-size 8 --no-rskip --no-rect --no-amp --no-sao --no-open-gop --no-early-skip --no-cutree --tu-intra-depth 4 --tu-inter-depth 4 --range limited --aud --repeat-headers --hrd --hdr-opt --colorprim bt2020 --colormatrix bt2020nc --transfer smpte2084 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,20)" --max-cll=0,0 --input-depth 10 --chromaloc 2 --preset slower -o vs.mkv --asm avx512 --zones 2100,3000,b=1.22/9000,10500,b=1.3/13000,13500,b=1.1/22000,22320,b=1.11/25000,25300,b=1.15 --crf 17 > vs.txt 2>&1y4m [info]: 1920x1080 fps 24000/1001 i420p10 unknown frame count
raw [info]: output file: vs.mkv
x265 [info]: HEVC encoder version 3.1_RC1+3-3bdf06e3c628
x265 [info]: build info [Windows][MSVC 1921][64 bit] 10bit
...
30194 frames: 5.32 fps, 20856.40 kb/s
30197 frames: 5.32 fps, 20855.68 kb/s
30199 frames: 5.32 fps, 20855.23 kb/s
30201 frames: 5.32 fps, 20854.86 kb/s
30205 frames: 5.32 fps, 20854.18 kb/s
30207 frames: 5.33 fps, 20853.70 kb/s
30210 frames: 5.33 fps, 20852.97 kb/s

x265 [info]: frame I: 204, Avg QP:14.97 kb/s: 33917.45
x265 [info]: frame P: 3896, Avg QP:16.17 kb/s: 26657.95
x265 [info]: frame B: 26112, Avg QP:17.10 kb/s: 20900.09
x265 [info]: Weighted P-Frames: Y:2.8% UV:0.6%
x265 [info]: Weighted B-Frames: Y:5.6% UV:0.2%
x265 [info]: consecutive B-frames: 13.6% 1.0% 2.4% 9.0% 3.2% 7.3% 3.9% 24.4% 5.8% 4.8% 4.0% 14.9% 5.6%

encoded 30212 frames in 5669.34s (5.33 fps), 21730.49 kb/s, Avg QP:16.97
GCC build didn't output the last lines with the statistics but the last encoded frame is identical to the vs last frame. Also GCC as always is a bit faster than VS at least for me.

Edit: GCC MediaInfo (https://pastebin.com/M2wC2YTB) and VS MediaInfo (https://pastebin.com/aDp3fkjX). Both has zone-count=5

[MeteorRain]

Had anyone actually done any benchmark on x265 on GCC 8.3 and 9.1?
From past records 9preview was slower than 8 but how about now?

[StvG]

Couple days ago I tested x265 compiled with gcc-7/8/9-branch and all three builds gave me the same fps (<=1%).

[benwaggoner]

First off, I want to know what the conditions are. I don't dismiss the possibility that I don't know what I'm doing in some aspect :sly:.
It appears that the --lossless flag blocks --hdr-opt. Which makes total sense in retrospect, as lossless is QP 4.

So, right behavior, wrong error message.

[katzenjoghurt]

Oh my.
Am I the only one having such a hard time encoding scenes with red light / red backgrounds?

[...]

Oh man. I THINK I finally found a solution for my neverending problem with red areas getting blurred and/or blocky.

People already gave me a hint to use 10bit encoding or playing around with the crqpoffs parameter. Both didn't really help.

This changed drastically now after I converted the source to YV24 colorspace and encoded it with Main 444 10.

Suddenly the crqpoffs parameter started to work wonders and setting it to -1 already brought back most of the details.


How to do it in StaxRip 1.7.0.6:
1) Click on the "AVS Filter" label. Click "Profiles".
2) Find the [Misc] section and add this line to the bottom: ConvertToYV24 = ConvertToYV24()
3) Add the Filter now via right-click -> Misc -> ConvertToYV24
4) Go into the x265 encoder settings
5) In "Basic" chose the Main 444 10 profile.
6) In "Rate Control 1" set CR QB Offset to -1.

Meh.

And now I reallize that with Main 10 444...
a) ... the contrast is too high in VLC
b) ... the video won't play at all in Windows Media Player
c) ... while it works well in Kodi and Zoom Player.

:(

[LigH]

x265 3.1+2-b36c03e4e771 (https://www.mediafire.com/file/12qr86ocy2eox6d/x265_3.1+2-b36c03e4e771.7z/file) (MSYS2, GCC 9.1.0)

New options / changes:

--[no-]field Enable or disable field coding. Default disabled
--[no-]early-skip Enable early SKIP detection. Default enabled
--max-merge <1..5> Maximum number of merge candidates. Default 3
--limit-refs <0|1|2|3> Limit references per depth (1) or CU (2) or both (3). Default 1
--[no-]b-intra Enable intra in B frames in veryslow presets. Default enabled
--[no-]fades Enable detection and handling of fade-in regions. Default disabled
--max-cll <string> Specify content light level info SEI as "cll,fall" (HDR).
--[no-]cll Emit content light level info SEI. Default enabled

[Natty]

Had anyone actually done any benchmark on x265 on GCC 8.3 and 9.1?
From past records 9preview was slower than 8 but how about now?

it looks like the bug of hidden encoding settings is still there in x265-Yuuki-3.1_RC1+4-gf48f4d038+23.7z build :helpful:

[MeteorRain]

Well I observed some performance regression so I'm keeping my GCC 8.3 for now. 1-3% slower with GCC 9.1 than 8.3 on my workstation.
It could be measurement error though.

@Natty Yea 3.1 RC1+4 was not properly compiled. Will remove it soon. I've properly revised the mod branch and that should be fixed in 3.1 GA.

[katzenjoghurt]

[...]

I updated to the GCC build LigH posted a bit above (thx!) and it seems zones now aren't crashing for me as well any more. Hooray!!!!! :)

Can't explain why it worked for you but not for me before that.

[poisondeathray]

New options / changes:

--[no-]fades Enable detection and handling of fade-in regions. Default disabled


Thanks for the build;

Any more info or discussion on this somewhere?

Is it sort of like the --fade-compensate patch from x264 ?

Is there anyway to modulate the strength ?


1) The setting is not currently reflected in the encoder settings (as read by mediainfo)

2) on a quick test, it seems to correctly detect and increase the bitrate in the fade area... But there seems to be a slight abrupt transition as it exits out of the fade, almost like "keyframe popping", when compared to without --fades . There is an I frame placed on the --fades encode, where it was a B on the without .

Just one observation, one test, so I'll take do some other tests before before making any preliminary conclusions...

[LigH]

I updated to the GCC build LigH posted a bit above (thx!) and it seems zones now aren't crashing for me as well any more. Hooray!!!!! :)

Can't explain why it worked for you but not for me before that.

Probably depending on details, due to the nature of the problem (https://bitbucket.org/multicoreware/x265/commits/b4e38ce16d7c4b37a6482dc7ae61fd31071b6ff1):

Fix double free in zones

[benwaggoner]

2) on a quick test, it seems to correctly detect and increase the bitrate in the fade area... But there seems to be a slight abrupt transition as it exits out of the fade, almost like "keyframe popping", when compared to without --fades . There is an I frame placed on the --fades encode, where it was a B on the without .

Just one observation, one test, so I'll take do some other tests before before making any preliminary conclusions...
What preset were you using? I'd expect that with weighted B and P prediction, a fade itself shouldn't require an increase in bitrate. And with Open GOP I'd hope the keyframe pop wouldn't be quite so harsh.

[poisondeathray]

What preset were you using? I'd expect that with weighted B and P prediction, a fade itself shouldn't require an increase in bitrate. And with Open GOP I'd hope the keyframe pop wouldn't be quite so harsh.


Slower , plus a few other changes . 10bit, CRF rate control

As you probably are aware, x264 and x265 have had issues with fades for a long time (probably why this option was introduced) . So I'm guessing the point of --fades was to increase the bitrate and fix the fade region (it's desired) . Without it - the banding and fade looks worse (as expected), With it, the fade looks better, bitrate increased - except for the popping

I haven't had a chance to examine in more detail or run more tests, but it's a pretty clear "pop" because of the quality change. The I frame is lower in quality. The test sequence was Lighthouses of the Pacific at the beginning . You only need to encode about 50 frames or so, it occurs 46-47 .

[filler56789]

As you probably are aware, x264 and x265 have had issues with fades for a long time (probably why this option was introduced) . So I'm guessing the point of --fades was to increase the bitrate and fix the fade region (it's desired) . Without it - the banding and fade looks worse (as expected), With it, the fade looks better, bitrate increased - except for the popping.

IMHO x264, x265, and Xvid as well, should have gone the way of DivX and WMV3/WVC1, regarding ~scene detection~. In both DivX and WMV, the fades generate a sequence of I-frames, and this is a non-problem when the playback device of the user doesn't care about "excessive" bitrates. Sadly it seems the FOSS developers think «the less options for the end-user, the better» :-/

[SeeMoreDigital]

IMHO x264, x265, and Xvid as well, should have gone the way of DivX and WMV3/WVC1, regarding ~scene detection~. In both DivX and WMV, the fades generate a sequence of I-frames, and this is a non-problem when the playback device of the user doesn't care about "excessive" bitrates. Sadly it seems the FOSS developers think «the less options for the end-user, the better» :-/I could have done with something like that a few years ago when I had to generate an encode that began with a black to full colour transition over the first 125 frames (5 seconds).

In order to get rid of the horrendous blocks at the beginning of the encode, I ended up having encode the first 125 frames separately (using I and P frames).

[TomV]

IMHO x264, x265, and Xvid as well, should have gone the way of DivX and WMV3/WVC1, regarding ~scene detection~. In both DivX and WMV, the fades generate a sequence of I-frames, and this is a non-problem when the playback device of the user doesn't care about "excessive" bitrates. Sadly it seems the FOSS developers think «the less options for the end-user, the better» :-/

That is an extremely inefficient strategy for encoding dissolve transitions.

[benwaggoner]

IMHO x264, x265, and Xvid as well, should have gone the way of DivX and WMV3/WVC1, regarding ~scene detection~. In both DivX and WMV, the fades generate a sequence of I-frames, and this is a non-problem when the playback device of the user doesn't care about "excessive" bitrates. Sadly it seems the FOSS developers think «the less options for the end-user, the better» :-/
Actually, the VC-1 encoder (and I believe the stock WMV9) would turn a transition into a sequence of P frames. It didn't have weighted prediction for B-frames so that was a lot more efficient.

Since HEVC has b-frame weighted prediction as well, that shouldn't be necessary. But rate control and motion estimation for a cross-dissolve is irreducibly tricky. Ideally the "before" and "after" frames would be defined and then forward/back proprogated via some mbtree extension.

[filler56789]

That is an extremely inefficient strategy for encoding dissolve transitions.

I know that. But it seems you don't know that it WORKS :sly:

[benwaggoner]

I know that. But it seems you don't know that it WORKS :sly:
Up to a point. Maybe. But a second long cross dissolve would need either a really big VBV or really high QP. And weighted prediction is EXACTLY for this kind of use case.

[Barough]

x265 v3.1+4-4f6dde51a5db (https://www.mediafire.com/file/lct1u36fy1kpy1x/x265-3.1+4-4f6dde51a5db_Win_GCC910.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 9.1.0)


https://bitbucket.org/multicoreware/x265/commits/branch/default

[birdie]

x265 3.1.1 is out.

https://bitbucket.org/multicoreware/x265/downloads/x265_3.1.1.tar.gz

[MeteorRain]

From 9612a000cb26748e62833dcd9e47d890af0e3dec Mon Sep 17 00:00:00 2001
From: Xinyue Lu <i@7086.in>
Date: Tue, 9 Jul 2019 21:30:15 -0400
Subject: [PATCH] icc: fix compiling and linking issue under ICC

---
source/common/x86/asm-primitives.cpp | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/source/common/x86/asm-primitives.cpp b/source/common/x86/asm-primitives.cpp
index 3948c4b97..abd3665f8 100644
--- a/source/common/x86/asm-primitives.cpp
+++ b/source/common/x86/asm-primitives.cpp
@@ -5429,7 +5429,7 @@ void setupAssemblyPrimitives(EncoderPrimitives &p, int cpuMask) // Main
} // namespace X265_NS

extern "C" {
-#ifdef __INTEL_COMPILER
+#if defined(__INTEL_COMPILER) && EXPORT_C_API

/* Agner's patch to Intel's CPU dispatcher from pages 131-132 of
* http://agner.org/optimize/optimizing_cpp.pdf (2011-01-30)
@@ -5440,7 +5440,7 @@ int __intel_cpu_indicator = 0;
// CPU dispatcher function
void PFX(intel_cpu_indicator_init)(void)
{
- uint32_t cpu = x265::cpu_detect(false);
+ uint32_t cpu = X265_NS::cpu_detect(false);

if (cpu & X265_CPU_AVX)
__intel_cpu_indicator = 0x20000;
@@ -5467,7 +5467,7 @@ void PFX(intel_cpu_indicator_init)(void)
* that backs up all the registers. */
void __intel_cpu_indicator_init(void)
{
- x265_safe_intel_cpu_indicator_init();
+ PFX(intel_cpu_indicator_init)();
}

#else // ifdef __INTEL_COMPILER
--
2.19.1.windows.1


Fixes #487 and #488. For those who'd like to use ICC. Anyone is free to submit this patch to the official with any amount of modification.

[chenm001]

From 9612a000cb26748e62833dcd9e47d890af0e3dec Mon Sep 17 00:00:00 2001
From: Xinyue Lu <i@7086.in>
Date: Tue, 9 Jul 2019 21:30:15 -0400
Subject: [PATCH] icc: fix compiling and linking issue under ICC


Fixes #487 and #488. For those who'd like to use ICC. Anyone is free to submit this patch to the official with any amount of modification.

Thank you point out this bug, I have been forward message to project manager.

[Stereodude]

What happened to x265 from 2.9 to 3.x? The bitrate dropped (and the QP increased) and the visual output quality has suffered at the same crf with the same command line. I read the release notes here: https://x265.readthedocs.io/en/default/releasenotes.html but they don't shed a ton of light. I know they made the old veryslow = new slower and effectively made a new preset between placebo and the old veryslow, but I've been using placebo so that doesn't explain it.

My typical 1080p command line switches for 8 bit 4:2:0 input.
--crf 16.0 -p placebo --no-sao --aq-strength 1.15 --vbv-maxrate 25000 --vbv-bufsize 25000 --level 5.0 --keyint 120 --open-gop -D 10 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1

[LigH]

:eek: Preset "placebo" as your "typical" 1080p parameter ... seriously?! I hope you use solar power to operate your PC. :o

The meaning of the placebo preset didn't change in the last years. So it must be a change in the general default behaviour.

I can hardly imagine anyone actually seeing an obvious loss of quality in a CRF 16 result played at normal speed. I hope you don't compare single frames with a magnifier to produce this claim. Are you able to provide clips to compare?

[Wolfberry]

The default AQ mode has been changed to auto-variance (aq-mode 2) in version 3.0.

Try setting --aq-mode 1 (the old default) in your command line.

[Stereodude]

I can hardly imagine anyone actually seeing an obvious loss of quality in a CRF 16 result played at normal speed. I hope you don't compare single frames with a magnifier to produce this claim. Are you able to provide clips to compare?
It's quite easy to see in motion without magnification. Watch how fine grain dances and changes over time compared to the source (or doesn't dance and change like it should). With 2.9 the presets faster than placebo did not give satisfactory results even with significantly more bitrate.
The default AQ mode has been changed to auto-variance (aq-mode 2) in version 3.0.

Try setting --aq-mode 1 (the old default) in your command line.
Okay, thanks I will try this.

[Barough]

x265 v3.1+7-147fb92c5ed5 (https://www.mediafire.com/file/1a9q2qjfusbvi9p/x265-3.1+7-147fb92c5ed5_Win_GCC910.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 9.1.0)


https://bitbucket.org/multicoreware/x265/commits/branch/default

[Stereodude]

Why hasn't direct AVIsynth input been added to the windows builds of x265 like x264 has? I'm rather sick of having to pipe video into it.

[Natty]

x265-3.1.1+1-04b37fd-win64-static-multilib (https://drive.google.com/open?id=1iCOs9fD5i6tWh62-OOIH_6AGJhpSWv66)


x265 v3.1+7-147fb92c5ed5 (https://www.mediafire.com/file/1a9q2qjfusbvi9p/x265-3.1+7-147fb92c5ed5_Win_GCC910.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 9.1.0)


which one is latest ?:thanks:

[Barough]

V3.1+7 is from the Default channel and includes what the v3.1.1 have and some more

Skickat från min SM-G975F via Tapatalk

[qyot27]

Why hasn't direct AVIsynth input been added to the windows builds of x265 like x264 has? I'm rather sick of having to pipe video into it.
Use x265-Yuuki or ffmpeg.

[stax76]

Use x265-Yuuki or ffmpeg.

Or Wolfberry

[Stereodude]

Use x265-Yuuki or ffmpeg.
Is there some trick to making x265-Yuuki work with avisynth+?

C:\HDTV Tools\x265_Yuuki>x265-gcc-multilib-full.exe -f 1500 --crf 16.0 -p placebo --no-sao --aq-strength 1.15 --vbv-maxrate 25000 --vbv-bufsize 25000 --level 5.0
--keyint 120 --open-gop -D 10 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 -o HZLIP_p_1.15_16.0_nosao_D10_dith_p1_t_10in.265
"E:\TempBD\HZLIP\HZLIP_comb_1.avs"
[avs2 @ 00000000037d8300] Format avs2 detected only with low score of 1, misdetection possible!
[avs2 @ 00000000037d8300] Could not find codec parameters for stream 0 (Video: avs2, none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[NULL @ 0000000004e14700] No codec provided to avcodec_open2()
[NULL @ 0000000004e14700] No codec provided to avcodec_open2()
lavf [error]: could not find decoder for video stream
x265 [error]: unable to open input file <E:\TempBD\HZLIP\HZLIP_comb_1.avs>
Edit: Wolfberry works with the same command line (changing only the name of the executable). :confused:

[qyot27]

Yuuki is the patchset/branch (https://github.com/msg7086/x265-Yuuki-Asuna/commits/Yuuki) (the patch itself is just generic LAVF input); whether the particular build was linked against a libavformat that itself was built with --enable-avisynth is up to the one building it. 'avs2' is not AviSynth (it's the Chinese AVS2 format), and the only reason libavformat would jump to that when given an AviSynth script is that FFmpeg hadn't been built with --enable-avisynth.

[Stereodude]

So why doesn't the Wolfberry build of x265 honor affinity restrictions? Using START "Enc #1" /NORMAL /NODE 0 /AFFINITY 00FF in front of the x265 command line still results in the same number of threads being spawned as no affinity restrictions or /AFFINITY 000F. It always sees the full processor which is the same problem I have with piping.

[Stereodude]

--seek doesn't work correctly with the Wolfberry build with an avisynth+ input either.

x265.exe" --pools 4 -F 1 --crf 16.0 -p veryslow --no-sao --aq-mode 1 --aq-strength 1.15
--vbv-maxrate 40000 --vbv-bufsize 40000 --level 5.1 --keyint 120 --open-gop -D 10
--colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --seek 16609
--frames 16678 -o out_2.265 "source.avs"

I get this output in the console: (which says all the right things)
lavf [info]:
Format : avisynth
Codec : rawvideo ( raw video )
PixFmt : yuv420p10le
Framerate : 24/1
Timebase : 1/24
Duration : 2:18:31
lavf [info]: 1920x1080 fps 24/1 i420p10 sar 1:1 frames 16609 - 33286 of 199487
raw [info]: output file: out_2.265
x265 [info]: HEVC encoder version 3.1.1+1-04b37fdfd2dc
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main 10 profile, Level-5.1 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 1 / wpp(17 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 4 / 5
x265 [info]: Keyframe min / max / scenecut / bias: 12 / 120 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 40 / 8 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 5 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 1.1 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-16.0 / 0.60
x265 [info]: VBV/HRD buffer / max-rate / init : 40000 / 40000 / 0.900
x265 [info]: tools: rect amp rd=6 psy-rd=2.00 rdoq=2 psy-rdoq=1.00 rskip
x265 [info]: tools: signhide tmvp b-intra strong-intra-smoothing deblock
[0.3%] 54/16678 frames, 0.16 fps, 6617.67 kb/s, 1.77 MB, eta 29:46:47, est.size 548.21 MB
But the actual out_2.265 output file starts from frame 0 in the source, not frame 16609.

[MeteorRain]

AviSynth support came from LAVF input filter.
LAVF input filter was ported from x264.
x264 didn't support seeking.

So this input filter never had seeking feature.

As a side note, LAVF was never designed to take AVS script (and that's exactly the reason I did not include AVS support in my build).

I'd hope one day someone can add a proper AVS native input. But on the other hand my scripts are on 32-bit so I have no choice but to use piper anyway.

[Stereodude]

AFAIK, piping can't seek either. The workaround is a giant pain. I have to make a unique .avs for every segment the outputs the right frames using trim, and then I have to make a unique .chp for every segment/x265 call that's shifted by the correct amount. :(

[alex1399]

How about multiple piping? Just pipe the avs + lossless x264 encode ultrafast before the x265 ones

[Stereodude]

How about multiple piping? Just pipe the avs + lossless x264 encode ultrafast before the x265 ones
Can you elaborate a bit more? I'm not sure I'm following you.

[alex1399]

Why hasn't direct AVIsynth input been added to the windows builds of x265 like x264 has? I'm rather sick of having to pipe video into it.
Never mind. Doesn't saw this previous post before.

[MeteorRain]

AFAIK, piping can't seek either. The workaround is a giant pain. I have to make a unique .avs for every segment the outputs the right frames using trim, and then I have to make a unique .chp for every segment/x265 call that's shifted by the correct amount. :(

Can you shed more light on what you are trying to accomplish?

Why would a chp file be affected by seeking support? Suppose it's a chapter file, and x265 does not take chapter files as input.

Talking about my experience, if I want to backup a movie, usually I want to encode it in multiple segments. I have scripts which can produce "import().trim()" type of AVS files, and I encode them in different batch, then concatenate them together, attach the chapter, and call it a day.

[Boulder]

I see there are two recent commits related to hierachial motion estimation. I couldn't find anything specific as to what the use cases would be, does anyone have any insights? Or is this once again one of those things that just appear in the encoder out of the blue (probably requested by some paying client) and the devs leave it at that.

[microchip8]

I see there are two recent commits related to hierachial motion estimation. I couldn't find anything specific as to what the use cases would be, does anyone have any insights? Or is this once again one of those things that just appear in the encoder out of the blue (probably requested by some paying client) and the devs leave it at that.

Here is a rather technical explanation http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/AV0405/ZAMPOGLU/Hierarchicalestimation.html

[Stereodude]

Can you shed more light on what you are trying to accomplish?

Why would a chp file be affected by seeking support? Suppose it's a chapter file, and x265 does not take chapter files as input.

Talking about my experience, if I want to backup a movie, usually I want to encode it in multiple segments. I have scripts which can produce "import().trim()" type of AVS files, and I encode them in different batch, then concatenate them together, attach the chapter, and call it a day.
I'm encoding something in multiple segments to improve quality (less threads) and speed (multiple encodes at once). I'm talking about having to shift frames around in the qpfile. Each unique .avs needs a unique qpfile.

Seek support would simply this since the same .avs and qpfile could be used and only the seek point and frame number on the command line would need to change.

This would also be more simple if affinity restrictions from the START command in Windows 10 worked with x265. It seems that somewhere in the Windows 10 feature update chain MS changed the behavior. Affinity restrictions on a .bat file that called an executable used to result in the same affinity restrictions being placed on the executable started by the .bat file. Now it doesn't. Affinity restrictions placed directly on the x265 command line don't even work. (They do work on the x264 command line.)

Ideally I would run a batch file with these four lines and be done.
START "Enc #1" /NORMAL /NODE 0 /AFFINITY F000 "C:\HDTV Tools\x265\x265.exe" -F 1 --crf 16.0 -p veryslow ...
START "Enc #2" /NORMAL /NODE 0 /AFFINITY 0F00 "C:\HDTV Tools\x265\x265.exe" -F 1 --crf 16.0 -p veryslow ...
START "Enc #3" /NORMAL /NODE 0 /AFFINITY 00F0 "C:\HDTV Tools\x265\x265.exe" -F 1 --crf 16.0 -p veryslow ...
START "Enc #4" /NORMAL /NODE 0 /AFFINITY 000F "C:\HDTV Tools\x265\x265.exe" -F 1 --crf 16.0 -p veryslow ...
Each x265 instance would see only 4 logical cores and would be locked to those 4 cores. But instead each x265 instance sees all 16 logical cores and I have to add --pools 4 to the command line and then manually modify the affinity of each x265 process in the Task Manager once they're all started.

[Boulder]

Here is a rather technical explanation http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/AV0405/ZAMPOGLU/Hierarchicalestimation.html

Thanks - yes, quite technical. To me it looks somehow similar to MRecalculate in MVTools, but maybe someone can explain it in layman's terms or simply come up with some POC they must have performed at some point. Looks interesting anyway.

[Atak_Snajpera]

Affinity restrictions placed directly on the x265 command line don't even work. (They do work on the x264 command line.)
I can confirm this behaviour. Looks like that x265 unlike x264 resets affinity at launch. I would also prefer that x265 didn't change initial affinity set by user.

[LigH]

I believe it may be possible to use the --pools parameter to control the affinity of x265 instances if it was just documented verbosely enough (https://x265.readthedocs.io/en/default/threading.html#thread-pools)...

x265 creates one or more thread pools per encoder, one pool per NUMA node (typically a CPU socket). --pools specifies the number of pools and the number of threads per pool the encoder will allocate. By default x265 allocates one thread per (hyperthreaded) CPU core on each NUMA node.

This may be not sufficient without examples. Right away you will notice that the online documentation does not even mention some parameters in the help output:

--pools <integer,...> Comma separated thread count per thread pool (pool per NUMA node)
'-' implies no threads on node, '+' implies one thread per core on node

I would guess that calls with a similar result as the above mentioned "start /AFFINITY" might look similar to:

x265 --pools 4,"+---"
x265 --pools 4,"-+--"
x265 --pools 4,"--+-"
x265 --pools 4,"---+"

I hope Ma can enlighten us.

[Stereodude]

I believe it may be possible to use the --pools parameter to control the affinity of x265 instances if it was just documented verbosely enough (https://x265.readthedocs.io/en/default/threading.html#thread-pools)...



This may be not sufficient without examples. Right away you will notice that the online documentation does not even mention some parameters in the help output:

--pools <integer,...> Comma separated thread count per thread pool (pool per NUMA node)
'-' implies no threads on node, '+' implies one thread per core on node

I would guess that calls with a similar result as the above mentioned "start /AFFINITY" might look similar to:

x265 --pools 4,"+---"
x265 --pools 4,"-+--"
x265 --pools 4,"--+-"
x265 --pools 4,"---+"

I hope Ma can enlighten us.
Your example doesn't work. And, it contradicts what's in this: https://x265.readthedocs.io/en/default/cli.html

It has these examples:
"" - default, unspecified, all numa nodes are used for thread pools
"*" - same as default
"none" - no thread pools are created, only frame parallelism possible
"-" - same as "none"
"10" - allocate one pool, using up to 10 cores on all available nodes
"-,+" - allocate one pool, using all cores on node 1
"+,-,+" - allocate one pool, using only cores on nodes 0 and 2
"+,-,+,-" - allocate one pool, using only cores on nodes 0 and 2
"-,*" - allocate one pool, using all cores on nodes 1, 2 and 3
"8,8,8,8" - allocate four pools with up to 8 threads in each pool
"8,+,+,+" - allocate two pools, the first with 8 threads on node 0, and the second with all cores on node 1,2,3
There's considerably more verbage on it too.

I see no references to core restrictions, only node restrictions in the documentation.

[LigH]

Ah, I missed these examples... I would have given a better guess if I knew them.

So it seems to be:

x265 --pools "4,4,4,4,+,-,-,-"
x265 --pools "4,4,4,4,-,+,-,-"
x265 --pools "4,4,4,4,-,-,+,-"
x265 --pools "4,4,4,4,-,-,-,+"

Or am I still wrong? ... In this case, I may misunderstand the nodes in contrast to the specific cores.

P.S.: I asked in the x265 developer mailing list.

[Stereodude]

Ah, I missed these examples... I would have given a better guess if I knew them.

So it seems to be:

x265 --pools "4,4,4,4,+,-,-,-"
x265 --pools "4,4,4,4,-,+,-,-"
x265 --pools "4,4,4,4,-,-,+,-"
x265 --pools "4,4,4,4,-,-,-,+"

Or am I still wrong? ... In this case, I may misunderstand the nodes in contrast to the specific cores.

P.S.: I asked in the x265 developer mailing list.
I didn't try any of those, but this system only has 1 numa node (8C/16HT) and those command lines would be for a system with 4 numa nodes, and they're contradicting. "0,0,0,4" would put 0 thread pools on the first 3 Numa nodes and 4 on the 4th. "4,4,4,4" would put 4 thread pools on each of the 4 Numa nodes. "+,-,-,-" would put a thread pool with an number equal to the logical cores in the first node and none on the next 3. So, you've got conflicting instructions on the command line.

By my reading and messing with them (I do have a dual Numa node system) you have control over nodes, but not over cores.

I'll be curious to see what reply you get on the mailing list.

[MeteorRain]

I have never made x265 to set its affinity as I intended so I always use my modded version of avs4x264 to do the job, which overwrites the affinity after x265 launches.

Regarding to your qpfile usage, honestly I have no similar usage of those so I don't have good suggestions for you. Usually if I want to force an IDR I'll just cut that as a new segment.

I'll take a quick look at the source code and see if I can find a way to do a seek easily. If it's easy enough to implement I'll slipstream it into my patch set.

[MeteorRain]

And -- I might be wrong, but -- I don't see any code with qpfile that takes into account seek option. That means you'll have to shift your qpfile for any seek option you applied in the command line.

[MeteorRain]

https://down.7086.in/x265-10bit-avisynth-seek-test.zip

qpfile still needs manual shifting.

[Stereodude]

And -- I might be wrong, but -- I don't see any code with qpfile that takes into account seek option. That means you'll have to shift your qpfile for any seek option you applied in the command line.
So if I tell it to drop an I frame at frame 6000 and I tell it to seek to frame 5000 to start the encode it doesn't subtract 5000 from 6000 and know to place it at 1000 in the encode (which is frame 6000 in the source)?

:confused:

[MeteorRain]

That is correct as far as I know.

[mandarinka]

I see there are two recent commits related to hierachial motion estimation. I couldn't find anything specific as to what the use cases would be, does anyone have any insights? Or is this once again one of those things that just appear in the encoder out of the blue (probably requested by some paying client) and the devs leave it at that.

If I'm not mistaken, this method could perhaps improve the search by allowing it to find more distant motion vectors (large motion difference between frames say in anime action sequences?). I'm completely assuming, haven't tried yet.

Has anybody here tested it?

(Edit: also, is it finished/ready for testing yet, or are there more incoming patches to plumb it together?)

[tuanden0]

New commit with new aq-mode
--aq-mode <0|1|2|3|4>

Adaptive Quantization operating mode. Raise or lower per-block quantization based on complexity analysis of the source image. The more complex the block, the more quantization is used. This offsets the tendency of the encoder to spend too many bits on complex areas and not enough in flat areas.

disabled
AQ enabled
AQ enabled with auto-variance (default)

3. AQ enabled with auto-variance and bias to dark scenes. This is recommended for 8-bit encodes or low-bitrate 10-bit encodes, to prevent color banding/blocking. 4. AQ enabled with

[quietvoid]

If I'm not mistaken, this method could perhaps improve the search by allowing it to find more distant motion vectors (large motion difference between frames say in anime action sequences?). I'm completely assuming, haven't tried yet.

Has anybody here tested it?

(Edit: also, is it finished/ready for testing yet, or are there more incoming patches to plumb it together?)From my quick testing using --hme with --hme-search hex,star,hex is 3x slower than just --me star (subme 5) with --preset slower.
Even hex,hex,hex is still 10% slower, so not sure what the use case is when it's that slow.

[vpupkind]

From my quick testing using --hme with --hme-search hex,star,hex is 3x slower than just --me star (subme 5) with --preset slower.
Even hex,hex,hex is still 10% slower, so not sure what the use case is when it's that slow.

The point is having a much larger search range. On high resolutions (e.g. 4K) the current search range is very small and is not great at capturing fast motion. With HME, using 3 levels with range of 32 will get you to the range of 224. The current default for star, hex, et al is less than 64.
The other point is better coherency of motion field at high resolutions.

[mandarinka]

Any reason why --HME uses hex, umh, umh as its default motion searches, but generally x265 uses --me star in its presets?
(would Star be better than UMH in HME, and generally outside of HME?)

[microchip8]

Any reason why --HME uses hex, umh, umh as its default motion searches, but generally x265 uses --me star in its presets?
(would Star be better than UMH in HME, and generally outside of HME?)

I guess performance reasons. I use hme-search=star for all levels (L0,1,2)

[Boulder]

So basically the option has no practical use in encodes of sub-4K resolutions?

[microchip8]

So basically the option has no practical use in encodes of sub-4K resolutions?

I use it for Full HD movies. I find it looks a bit sharper compared to no hme

[LigH]

x265 3.1+8-21db162c8622 (https://www.mediafire.com/file/95xlvfj16xxpwrb/x265_3.1+8-21db162c8622.7z/file) (MSYS2, MinGW32 / MinGW64 + GCC 9.1.0)

--[no-]hme Enable Hierarchical Motion Estimation. Default disabled
--hme-search <string> Motion search-method for HME L0,L1 and L2. Default(L0,L1,L2) is 1,2,2
--aq-mode <integer> Mode for Adaptive Quantization - 0:none 1:uniform AQ 2:auto variance 3:auto variance with bias to dark scenes 4:auto variance with edge information. Default 2
--[no-]aq-motion Block level QP adaptation based on the relative motion between the block and the frame. Default disabled

[Morku]

@LigH
As silent reader I want to say thank you for the fresh, updated Encoders. Please never stop. Even when the board seems to fall asleep sometimes, we appreciate all the work. A Thank Button would be nice here. Every contribute is important :)

I hope Zathor is fine...?

[jlpsvk]

what exctly that aq-mode 4 is doing? :) is it that "HEVC" aq? or? that HEVC aq was not so good... currently i am using aq-mode 2.

[Jamaika]

x265 3.1+8-21db162c8622 (https://www.mediafire.com/file/95xlvfj16xxpwrb/x265_3.1+8-21db162c8622.7z/file) (MSYS2, MinGW32 / MinGW64 + GCC 9.1.0)

--[no-]hme Enable Hierarchical Motion Estimation. Default disabled
--hme-search <string> Motion search-method for HME L0,L1 and L2. Default(L0,L1,L2) is 1,2,2
--aq-mode <integer> Mode for Adaptive Quantization - 0:none 1:uniform AQ 2:auto variance 3:auto variance with bias to dark scenes 4:auto variance with edge information. Default 2
--[no-]aq-motion Block level QP adaptation based on the relative motion between the block and the frame. Default disabled
I add function qp zero, aq-mode doesn't work.
p->rc.aqMode = X265_AQ_EDGE;
p->bAQMotion = 1;
p->bEnableHME = 1;
p->hmeSearchMethod[1] = p->hmeSearchMethod[2] = X265_STAR_SEARCH;
x265 (build 178) - 3.1+8:[Windows][GCC 9.1.1][64 bit][noasm] 10bit - H.265/HEVC codec - Copyright 2013-2018 (c) Multicoreware, Inc - http://x265.org - options: cpuid=0 frame-threads=2 wpp no-pmode no-pme no-psnr no-ssim log-level=2 bitdepth=10 input-csp=2 fps=25/1 input-res=1568x1560 interlace=0 total-frames=1 level-idc=0 high-tier=1 uhd-bd=0 ref=1 no-allow-non-conformance repeat-headers annexb no-aud no-hrd info hash=0 no-temporal-layers no-open-gop min-keyint=1 keyint=1 gop-lookahead=0 bframes=0 b-adapt=0 no-b-pyramid bframe-bias=0 rc-lookahead=0 lookahead-slices=0 scenecut=0 radl=0 no-splice no-intra-refresh ctu=64 min-cu-size=8 rect amp max-tu-size=32 tu-inter-depth=4 tu-intra-depth=4 limit-tu=0 rdoq-level=2 dynamic-rd=0.00 no-ssim-rd signhide tskip nr-intra=0 nr-inter=0 no-constrained-intra strong-intra-smoothing max-merge=5 limit-refs=0 no-limit-modes me=2 subme=5 merange=92 temporal-mvp hme Level 0,1,2=1,3,3 no-weightp no-weightb no-analyze-src-pics deblock=-1:1 sao no-sao-non-deblock rd=6 no-early-skip no-rskip no-fast-intra no-tskip-fast no-cu-lossless b-intra no-splitrd-skip rdpenalty=0 psy-rd=2.00 psy-rdoq=0.00 no-rd-refine no-lossless cbqpoffs=0 crqpoffs=0 rc=cqp qp=0 aq-mode=0 aq-strength=0.00 no-cutree zone-count=0 no-strict-cbr qg-size=32 no-rc-grain qpmax=69 qpmin=0 no-const-vbv sar=0 overscan=0 videoformat=5 range=0 colorprim=2 transfer=2 colormatrix=2 chromaloc=0 display-window=0cll=0,0 min-luma=0 max-luma=1023 log2-max-poc-lsb=8 vui-timing-info vui-hrd-info slices=1 no-opt-qp-pps no-opt-ref-list-length-pps no-multi-pass-opt-rps scenecut-bias=0.05 no-opt-cu-delta-qp aq-motion no-hdr no-hdr-opt no-dhdr10-opt no-idr-recovery-sei analysis-reuse-level=5 scale-factor=0 refine-intra=0 refine-inter=0 refine-mv=0 refine-ctu-distortion=0 no-limit-sao ctu-info=0 no-lowpass-dct refine-analysis-type=33620483 copy-pic=1 max-ausize-factor=1.0 no-dynamic-refine no-single-sei no-hevc-aq no-svt no-field qp-adaptation-range=1.00
For what presets is it and additional functions? Can it be used for 10bit?:D

[_kermit]

what exctly that aq-mode 4 is doing? :) is it that "HEVC" aq? or? that HEVC aq was not so good... currently i am using aq-mode 2.

I have the same question

[Selur]

--aq-mode <integer> Mode for Adaptive Quantization - 0:none 1:uniform AQ 2:auto variance 3:auto variance with bias to dark scenes 4:auto variance with edge information. Default 2
--[no-]hevc-aq Mode for HEVC Adaptive Quantization. Default disabled
--aq-strength <float> Reduces blocking and blurring in flat and textured areas (0 to 3.0). Default 1.00
--qp-adaptation-range <float> Delta QP range by QP adaptation based on a psycho-visual model (1.0 to 6.0). Default 1.00
--[no-]aq-motion Block level QP adaptation based on the relative motion between the block and the frame. Default disabled source: x265 --fullhelp
-> hevc-aq is still there, so aq-mode 4 should be something different ;)

[mandarinka]

From the description, it looks both aqmode 3 and aqmode 4 are modifications of aqmode 2. Based on the descriptions, the first adds bias for darker scenes/parts and the second (#4) adds bias for detected(?) edges (to improve them?).

[jlpsvk]

But how... quality and speed impact??? I've done test on the whole 4K HDR movie. Same settings, one with AQ2, one with AQ4.

AQ2:
x265 [info]: HEVC encoder version 3.1+8-21db162c8622
x265 [info]: build info [Windows][GCC 9.1.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
x265 [info]: Main 10 profile, Level-5.1 (High tier)
x265 [info]: Thread pool created using 28 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 4 / wpp(25 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 3 / 3
x265 [info]: Keyframe min / max / scenecut / bias: 24 / 240 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 48 / 8 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 4 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 8 / 1
x265 [info]: Rate Control / qCompress : CRF-16.0 / 0.60
x265 [info]: VBV/HRD buffer / max-rate / init : 160000 / 160000 / 0.900
x265 [info]: tools: rect amp limit-modes rd=4 psy-rd=2.00 rdoq=2 psy-rdoq=1.00
x265 [info]: tools: signhide tmvp lslices=4

x265 [info]: frame I: 1572, Avg QP:19.24 kb/s: 36306.92
x265 [info]: frame P: 31790, Avg QP:20.31 kb/s: 29999.39
x265 [info]: frame B: 120253, Avg QP:23.15 kb/s: 17831.23
x265 [info]: Weighted P-Frames: Y:9.7% UV:1.7%
x265 [info]: consecutive B-frames: 17.9% 9.6% 8.6% 19.1% 7.5% 10.0% 6.4% 9.3% 11.7%

encoded 153615 frames in 81511.10s (1.88 fps), 20538.46 kb/s, Avg QP:22.53

AQ4:
x265 [info]: HEVC encoder version 3.1+8-21db162c8622
x265 [info]: build info [Windows][GCC 9.1.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
x265 [info]: Main 10 profile, Level-5.1 (High tier)
x265 [info]: Thread pool created using 28 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 4 / wpp(25 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 3 / 3
x265 [info]: Keyframe min / max / scenecut / bias: 24 / 240 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 48 / 8 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 4 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 4 / 1.0 / 8 / 1
x265 [info]: Rate Control / qCompress : CRF-16.0 / 0.60
x265 [info]: VBV/HRD buffer / max-rate / init : 160000 / 160000 / 0.900
x265 [info]: tools: rect amp limit-modes rd=4 psy-rd=2.00 rdoq=2 psy-rdoq=1.00
x265 [info]: tools: signhide tmvp lslices=4

x265 [info]: frame I: 1571, Avg QP:19.38 kb/s: 35277.33
x265 [info]: frame P: 31792, Avg QP:20.46 kb/s: 29185.39
x265 [info]: frame B: 120252, Avg QP:23.29 kb/s: 17402.88
x265 [info]: Weighted P-Frames: Y:9.6% UV:1.7%
x265 [info]: consecutive B-frames: 17.9% 9.6% 8.7% 19.1% 7.6% 10.0% 6.3% 9.3% 11.7%

encoded 153615 frames in 83129.38s (1.85 fps), 20024.18 kb/s, Avg QP:22.66

[Jamaika]

I just found out that "aq-mode" parameters don't apply to frames I i.e. pictures. For qp on for frames P, aq-mode on. For qp off for frames I, aq-mode off. :D

[vpupkind]

From the description, it looks both aqmode 3 and aqmode 4 are modifications of aqmode 2. Based on the descriptions, the first adds bias for darker scenes/parts and the second (#4) adds bias for detected(?) edges (to improve them?).
Both hevc-aq and aq=4 try to take into account edges passing through the block.

[Jamaika]

x265&svt-3.1+8
I don't know why no one adds x265 & svt. Is this a temporary fusion? I will risk something to write.
https://www.sendspace.com/file/mt3kue
Information about color space is also interesting. For SVT, this is currently the High Dymanic Range. For 8bit is Low Dynamic Range.


x265 [info]: SVT doesn't support input-csp param; Disabling it
x265 [info]: SVT doesn't support crf param; Disabling it
x265 [info]: SVT doesn't support info param; Disabling it
x265 [info]: SVT doesn't support aq-mode param; Disabling it
x265 [info]: SVT doesn't support colormatrix param; Disabling it
x265 [info]: SVT doesn't support colorprim param; Disabling it
x265 [info]: SVT doesn't support transfer param; Disabling it
x265 [info]: SVT doesn't support limit-refs param; Disabling it
x265 [info]: SVT doesn't support range param; Disabling it

if (params->color_space == BPG_CS_YCbCr_HDR) {
p->vui.matrixCoeffs = 9;
svtHevcParam->highDynamicRangeInput = 1;
if (params->bit_depth => 10) {
p->vui.bEnableVideoFullRangeFlag = 1;
} else {
p->vui.bEnableVideoFullRangeFlag = 0;
}// for 8bit is Low Dynamic Range
}

PS change files in x265
svt.h --> #define INPUT_SIZE_8K_TH 0xB71B00 // 12 Million
param.cpp --> svtHevcParam->vbvMaxrate = 0;
param.cpp --> svtHevcParam->vbvBufsize = 0;
param.cpp --> svtHevcParam->vbvBufInit = 90;
param.cpp --> svtHevcParam->hrdFlag = 0;

[DJATOM]

Can someone explain, what's the purpose of HME? Is it for better quality/accuracy of ME or just for making things faster?

[microchip8]

Can someone explain, what's the purpose of HME? Is it for better quality/accuracy of ME or just for making things faster?

the former. it's for better quality/accuracy. Here's an article that explains it. It is also low computational but highly efficient

http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/AV0405/ZAMPOGLU/Hierarchicalestimation.html

[benwaggoner]

the former. it's for better quality/accuracy. Here's an article that explains it. It is also low computational but highly efficient

http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/AV0405/ZAMPOGLU/Hierarchicalestimation.html
So, it can make things faster at the same quality, higher quality at the same speed, or a mix of the two.

[aegisofrime]

So, it can make things faster at the same quality, higher quality at the same speed, or a mix of the two.

Could you share your settings? By simply setting --hme, I'm seeing a reduction of around 40% in speed on my encodes. Thanks!

[microchip8]

Could you share your settings? By simply setting --hme, I'm seeing a reduction of around 40% in speed on my encodes. Thanks!

I don't see such a high penalty on my i7 7700K processor. I use hme=1 and hme-search=umh,umh,umh

[DJATOM]

My friend tried star,star,umh and caught a crash on frame 50, but umh,star,umh works fine (but slower).

[microchip8]

My friend tried star,star,umh and caught a crash on frame 50, but umh,star,umh works fine (but slower).

yes, I experienced the same crash when using star for all levels (L0,1,2) hme-search=star

I use umh because I find it preserves details better but is a bit slower than star

[benwaggoner]

From the description, it looks both aqmode 3 and aqmode 4 are modifications of aqmode 2. Based on the descriptions, the first adds bias for darker scenes/parts and the second (#4) adds bias for detected(?) edges (to improve them?).
Speculation: aq-mode 4 is the non-experimental implementation of --hevc-aq.

[jlpsvk]

@benwaggoner

i don't think so, as hevc-aq and aq-mode 4 coexists...

[mandarinka]

My friend tried star,star,umh and caught a crash on frame 50, but umh,star,umh works fine (but slower).

I can confirm it doesn't like star on the first instance, I got crash right at start with merange 92 and 1440x1080 source.

Could you share your settings? By simply setting --hme, I'm seeing a reduction of around 40% in speed on my encodes. Thanks!

To get higher speed, lower your merange. The value given is used as the distance of the first-stage search, so it is effectively quadrupled.

[Boulder]

I'm currently using merange 32 for my 720p encodes (max CTU and TU is 32 for better CPU utilization). Would it make sense to lower the value for example to 16-20 and enable HME (with umh,umh,umh), or is the first stage range so important that it will start to affect the final result too much?

[microchip8]

I'm currently using merange 32 for my 720p encodes (max CTU and TU is 32 for better CPU utilization). Would it make sense to lower the value for example to 16-20 and enable HME (with umh,umh,umh), or is the first stage range so important that it will start to affect the final result too much?

I use as ME range a value of 26 for 1080p encodes (ctu is 32 too). ME range is calculated as follows:

ctu size - 4(luma) - 2(chroma) (- 1 if me=hex is used)

You can lower it to 26 and enable HME like I do. Here the performance penalty is very minor, also considering that HME, as explained in the article I linked to a few posts earlier, has computationally low complexity

I wonder what options the poster above uses to hit a 40% reduction in speed when HME is enabled

[Boulder]

I use as ME range a value of 26 for 1080p encodes (ctu is 32 too). ME range is calculated as follows:

ctu size - 4(luma) - 2(chroma) (- 1 if me=hex is used)

You can lower it to 26 and enable HME like I do. Here the performance penalty is very minor, also considering that HME, as explained in the article I linked to a few posts earlier, has computationally low complexity

I wonder what options the poster above uses to hit a 40% reduction in speed when HME is enabled

Thanks, I need to test that change. I also noticed a severe slowdown when testing HME at its default values and with merange 32. This is with basically settings from preset 'slower' with only a few minor changes.

[microchip8]

Thanks, I need to test that change. I also noticed a severe slowdown when testing HME at its default values and with merange 32. This is with basically settings from preset 'slower' with only a few minor changes.

I don't use presets but my own settings which I'm satisfied with. Here are my libx265 ffmpeg settings


X265PARAMS="ref=4:me=umh:hme=1:hme-search=umh,umh,umh:bframes=6:rd=4:subme=4:merange=26:strong-intra-smoothing=0:ctu=32:sao=0:cu-lossless=0:cutree=1:fades=1:tu-inter-depth=3:tu-intra-depth=3:rskip=1:max-merge=1:rc-lookahead=60:aq-mode=1:aq-strength=1.0:rdoq-level=1:psy-rdoq=1.5:psy-rd=2.3:limit-modes=1:limit-refs=3:limit-tu=1:rd-refine=0:deblock=-3,-3:weightb=1:weightp=1:rect=1:amp=0:wpp=1:pmode=0:pme=0:b-intra=1:b-adapt=2:b-pyramid=1:tskip-fast=0:fast-intra=0:early-skip=0:min-keyint=24:keyint=240"

[Boulder]

Biggest differences seem to be ref=5, me star, rd=6, rd-refine, strong-intra-smoothing, max-merge=4 and amp in my settings. I don't think those should affect the search for motion though. I really need to test to get some rough values.

[aegisofrime]

I use as ME range a value of 26 for 1080p encodes (ctu is 32 too). ME range is calculated as follows:

ctu size - 4(luma) - 2(chroma) (- 1 if me=hex is used)

You can lower it to 26 and enable HME like I do. Here the performance penalty is very minor, also considering that HME, as explained in the article I linked to a few posts earlier, has computationally low complexity

I wonder what options the poster above uses to hit a 40% reduction in speed when HME is enabled

I was using the default ME range of 57. I guess it needs to be dropped once HME is enabled.

I can't wait to get a 3900X or a 16 core Threadripper!

[microchip8]

Biggest differences seem to be ref=5, me star, rd=6, rd-refine, strong-intra-smoothing, max-merge=4 and amp in my settings. I don't think those should affect the search for motion though. I really need to test to get some rough values.

you're maxing out a few things. rd=6 (combined with rd-refine) is IMHO overkill if you want to compromize between speed and quality

Motion Estimation is where the encoder spends most of its time and takes the biggest penalty the higher you get. rd=6 is very expensive. I'm fine with rd=4 (which is really rd=3 as 4 maps to it)

This is the best I can do on my i7 7700K when turning on HME. I don't want to wait days for an encode to finish and I won't see the difference when watching from afar on the TV

With my settings above and HME turned on, I can shave ~100 MiB of an encode. Encoded Blade Runner once with and once without HME. The result was 100 MiB in size reduction with better subjective quality when HME is on

[Boulder]

Here are my results with my Black Sails testclip, 2748 frames @ CRF 18:

rd 6, no hme - 4333,39 kbps - 2.91 fps
rd 6, hme umh - 4323,63 kbps - 2.16 fps
rd 4, no hme - 4149,59 kbps - 3.95 fps
rd 4, hme umh - 4154,60 kbps - 2.66 fps

So the difference with rd=4 is even bigger than with rd=6 + rd-refine. I'm using a Ryzen 1800X @ 3.8 GHz.

[microchip8]

Here are my results with my Black Sails testclip, 2748 frames @ CRF 18:

rd 6, no hme - 4333,39 kbps - 2.91 fps
rd 6, hme umh - 4323,63 kbps - 2.16 fps
rd 4, no hme - 4149,59 kbps - 3.95 fps
rd 4, hme umh - 4154,60 kbps - 2.66 fps

So the difference with rd=4 is even bigger than with rd=6 + rd-refine. I'm using a Ryzen 1800X @ 3.8 GHz.

I have the opposite experience. rd=6 is too costly here. But, the tests I did were a long time ago so I'll retest again soon

[Boulder]

Of course, it could be AVX2 which is making the difference here. On a first generation Zen CPU, it's more useful to disable it (as I've done). I'd expect an Intel CPU to benefit from AVX2.

[microchip8]

I've reported the segfault when using star for hme-search Level 0 to the x265 devs. One of them came with a patch which I tested and seems to work. Patch is not committed yet but you can find it at https://mailman.videolan.org/pipermail/x265-devel/2019-July/012601.html

[DJATOM]

Sweet, thanks.

[Natty]

using --dither and happy that it dithers properly. amazing results. but I want to know which dithering method it uses? and what does it use when this flag isn't enabled?

thanks.

[Barough]

x265 v3.1+10-459d3822c608 (http://www.mediafire.com/file/nnxk5mj74ww3741/x265-3.1%252B10-459d3822c608_Win_GCC910.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 9.1.0)
https://bitbucket.org/multicoreware/x265/commits/branch/default

[LigH]

There was a comment in the sources in previous versions. From a former issue report: (https://bitbucket.org/multicoreware/x265/issues/255/dither-option-in-the-command-line-produces)
The dithering algorithm is based on Sierra-2-4A error diffusion.

Without dither, it will probably just strip off less significant bits (but not sure).

[Natty]

There was a comment in the sources in previous versions. From a former issue report: (https://bitbucket.org/multicoreware/x265/issues/255/dither-option-in-the-command-line-produces)


Without dither, it will probably just strip off less significant bits (but not sure).

nice its same as x264 then. i read about this. its similar to floyd. clears my confusion why grain particles were looking finer than ordered, but not as fine as floyd. :thanks:

[Barough]

x265 v3.1.2+1-76650bab70f9 (https://www.mediafire.com/file/asn09rfxt1zaumj/x265-3.1.2+1-76650bab70f9_Win_GCC910.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 9.1.0)
https://bitbucket.org/multicoreware/x265/commits/

[LigH]

Excuse me, fellow developers, but to a half-wit like me, this graph looks like there might be parallel development for a while now: One branch with mainly code and one with mainly version tags.

https://www.ligh.de/pics/x265branches.png

[filler56789]

Excuse me, fellow developers, but to a half-wit like me, this graph looks like there might be parallel development for a while now: One branch with mainly code and one with mainly version tags.

https://www.ligh.de/pics/x265branches.png

Yeah, it's becoming a mess :-/

[LigH]

So, at the moment, users have to decide between either branch until a merge may happen...

[nevcairiel]

So, at the moment, users have to decide between either branch until a merge may happen...

From your screenshot of the history, I don't see any commit that is only on the 3.1 branch thats not also on the default branch. After the earlier fixes there was a merge back to default (through stable), and the later fixes are simply on both branches.
Of course you still have to decide which branch you want, thats why branches exist. If all were identical, what would be their point? The full development "default" branch, or the possibly more stable 3.1 release branch?

[Boulder]

I tested HME some more, and got quite interesting results.

Source 1080p filtered with my standard methods and downsized to 720p, merange 26 and my standard settings in x265. 1000 frames.

umh,umh,umh 2.05 fps / 5313,76 kbps
umh,umh,star 2.24 fps / 5314,31 kbps
umh,star,star 2.00 fps / 5327,91 kbps
star,star,star 2.01 fps / 5316,15 kbps
hex,umh,umh 2.03 fps / 5299,69 kbps

no HME, umh 2.64 fps / 5323,86 kbps
no HME, star 2.79 fps / 5325,69 kbps

I'll make the same tests with some different clip to see if the difference simply occurs because of the source.

[MeteorRain]

So, at the moment, users have to decide between either branch until a merge may happen...

It's dev & stable branches strategy, except that instead of constantly merge stable to dev, they cherry pick (or duplicate) bug fixes onto dev branch.

It's literally the same thing as before, just in a different way.

[LigH]

Okay, I may have missed that some patches were applied to both branches.

The only problem here for casual builders using automated build suites: How shall the suite know which branch the user prefers? It cannot rely on "tip" pointing to either. May the author of such a suite have to add a choice in its configuration?

[nevcairiel]

The concept of "tip" in mercurial is flawed anyway, would never rely on it. Someone pushed an experimental half-broken branch, and suddenly thats the tip? Clearly not what everyone wants to build off of. The "most recent change" seems like a particular worthless point of reference to use.
You should always be aware of what branch you build from.

[mandarinka]

I tested HME some more, and got quite interesting results.

Source 1080p filtered with my standard methods and downsized to 720p, merange 26 and my standard settings in x265. 1000 frames.

umh,umh,umh 2.05 fps / 5313,76 kbps
umh,umh,star 2.24 fps / 5314,31 kbps
umh,star,star 2.00 fps / 5327,91 kbps
star,star,star 2.01 fps / 5316,15 kbps
hex,umh,umh 2.03 fps / 5299,69 kbps

no HME, umh 2.64 fps / 5323,86 kbps
no HME, star 2.79 fps / 5325,69 kbps

I'll make the same tests with some different clip to see if the difference simply occurs because of the source.

I tested HME (with umh/umh/umh, merange 92, against umh with same merange) on some cel anime bluray footage with motion (both full camera motion and objects in scene only motion).

I got notably decreased SSIM score with HME, so be careful, at these low resolutions it looks like it is possible that HME doesn't help but harms.
Why? The first ME pass with merange 92 is done on 4X downscale. So if you have 1080p resolution, you are down on 240p. It is likely that small objects won't get tracked properly at this stage. After that, the second (480p) and last (1080p) ME stage only only lower merange (search distance), I assume? So if the large downscaling fools the first pass, the second and third stage might not be able to correct it.

My footage was 1440x1080 and you would have even worse situation at 720p. Basically this factor may cause HME to be counter productive unless you use it on 4K for which ts was intended (IIRC).

"n:\x265-3.1+8-21db162c8622.exe" - --input-depth 8 --input-res 1440x1080 --fps 24000/1001 --preset slower --output-depth 10 --ctu 32 --max-tu-size 16
--pass 2 --bitrate 10000 --tune ssim --ssim --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --me star --subme 7 --merange 92 --amp --rect --ref 6
--weightb --weightp --keyint 300 --min-keyint 1 --bframes 8 --rd 5 --no-sao --no-open-gop --rc-lookahead 80 --max-merge 5 --qcomp 0.7
--no-strong-intra-smoothing --no-limit-modes --limit-refs 0 --limit-tu 0 --frame-threads 2 --wpp --deblock -2:-2 --qg-size 8 --pbratio 1.2 --no-cutree --cu-lossless
--lookahead-slices 2 --sar 1:1 --range limited --chromaloc 0 --colormatrix bt709 --no-rskip --rd-refine --cbqpoffs -2 --crqpoffs -2

encoded 853 frames in 5108.02s (0.17 fps), 10064.24 kb/s, Avg QP:19.49, SSIM Mean Y: 0.9820175 (17.451 dB)


"n:\x265-3.1+8-21db162c8622.exe" - --input-depth 8 --input-res 1440x1080 --fps 24000/1001 --preset slower --output-depth 10 --ctu 32 --max-tu-size 16
--pass 2 --bitrate 10000 --tune ssim --ssim --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --me umh --subme 7 --merange 92 --amp --rect --ref 6
--weightb --weightp --keyint 300 --min-keyint 1 --bframes 8 --rd 5 --no-sao --no-open-gop --rc-lookahead 80 --max-merge 5 --qcomp 0.7
--no-strong-intra-smoothing --no-limit-modes --limit-refs 0 --limit-tu 0 --frame-threads 2 --wpp --deblock -2:-2 --qg-size 8 --pbratio 1.2 --no-cutree --cu-lossless
--lookahead-slices 2 --sar 1:1 --range limited --chromaloc 0 --colormatrix bt709 --no-rskip --rd-refine --cbqpoffs -2 --crqpoffs -2

encoded 853 frames in 11135.11s (0.08 fps), 10064.75 kb/s, Avg QP:19.49, SSIM Mean Y: 0.9820200 (17.452 dB)

"n:\x265-3.1+8-21db162c8622.exe" - --input-depth 8 --input-res 1440x1080 --fps 24000/1001 --preset slower --output-depth 10 --ctu 32 --max-tu-size 16
--pass 2 --hme --hme-search umh,umh,umh --bitrate 10000 --tune ssim --ssim --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --me umh --subme 7
--merange 92 --amp --rect --ref 6 --weightb --weightp --keyint 300 --min-keyint 1 --bframes 8 --rd 5 --no-sao --no-open-gop --rc-lookahead 80 --max-merge 5
--qcomp 0.7 --no-strong-intra-smoothing --no-limit-modes --limit-refs 0 --limit-tu 0 --frame-threads 2 --wpp --deblock -2:-2 --qg-size 8 --pbratio 1.2 --no-cutree
--cu-lossless --lookahead-slices 2 --sar 1:1 --range limited --chromaloc 0 --colormatrix bt709 --no-rskip --rd-refine --cbqpoffs -2 --crqpoffs -2

encoded 853 frames in 7319.99s (0.12 fps), 10051.98 kb/s, Avg QP:19.52, SSIM Mean Y: 0.9819607 (17.438 dB)

p.s. Ignore the fps values, they are not indicative of the speed due to background cpu usage, system sleeping for a time and so on.

BTW, it would be interesting to determine what is the speed hit of a particular HME setting and then determine how much do you need to increase merange for the normal search mode to get at the same FPS - and then comapre the quality/SSIM. Because obvioously the added computation time could be used to improve quality by just increasing merange too, so that is your reference point for comparisons.

Edit: one other thing. Doing two-pass testing smaples, I found that just switching ME method between star and umh changes content of each frame and block quite a bit due to various indeterministic things going on. Individual frames would look worse and better based on how the rate control stars aligned.
So don't compare the results just by bitrate they give you!
It is quite possible the two files are not the exactly same quality, which would skew your judgement completely.

[Boulder]

Isn't it the other way around so that the first pass is made at full resolution and then the result is used as a predictor for the next pass which is done after downscaling. I agree that small details could well be lost when downscaling. The code seems to set the limit to use HME at 540 pixels of vertical resolution.

What I was wondering with those results of mine was that umh,umh,star was noticably faster than any other combination.

[benwaggoner]

Is there a build which includes the --aq-mode 4 patch? I've tried the latest ones here, and I keep getting:
C:\Users\benwagg\Desktop\x265>C:\Users\benwagg\Desktop\x265\x265.exe --input D:\Rights_Cleared_Sources\TearsofSteel_1920x800p24.y4m --preset slower --crf 28 --aq-mode 4 -F 1 --min-keyint 72 --keyint 120 --no-open-gop D:\TearsOfSteel\ToS_aqm4_crf28.hevc
y4m [info]: 1920x800 fps 24/1 i420p8 frames 0 - 17619 of 17620
raw [info]: output file: D:\TearsOfSteel\ToS_aqm4_crf28.hevc
x265 [info]: HEVC encoder version 3.1.2+1-76650bab70f9
x265 [info]: build info [Windows][GCC 9.1.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [error]: Aq-Mode is out of range
x265 [error]: failed to open encoder

[mandarinka]

I think https://forum.doom9.org/showpost.php?p=1879566&postcount=6935 worked for me.

[pradeeprama]

The new aq-mode and HME is only present in the default branch (3.1)

3.1.1 and 3.1.2 from the Release_3.1 branch only include some bug fixes that are also on the default branch.

x265-3.1+11-de920e0-win64-static-multilib (https://drive.google.com/open?id=1_gqvRPHTWI4C7DQ-LEsU0RHaDJy1-wOL)

x265 [info]: HEVC encoder version 3.1+11-de920e0a3183
x265 [info]: build info [Windows][GCC 9.1.1][64 bit] 8bit+10bit+12bit
x265 [info]: (libavcodec 58.55.100)
x265 [info]: (libavformat 58.30.100)
x265 [info]: (libavutil 56.33.100)
x265 [info]: (lsmash 2.16.1)

These features will be part of the next release, which will have its own branch. Only bug fixes for v3.1 will go into the Release_3.1 branch

[benwaggoner]

The new aq-mode and HME is only present in the default branch (3.1)

3.1.1 and 3.1.2 from the Release_3.1 branch only include some bug fixes that are also on the default branch.

x265-3.1+11-de920e0-win64-static-multilib (https://drive.google.com/open?id=1_gqvRPHTWI4C7DQ-LEsU0RHaDJy1-wOL)

It worked! I'm going to knock out some aq-mode 2 versus 4 tests with Tears of Steel.

[jlpsvk]

@benwaggoner
any thought about aq-mode 2 vs aq-mode 4?

[benwaggoner]

@benwaggoner
any thought about aq-mode 2 vs aq-mode 4?
CPUs have been too busy doing other things! But I'll kick them off now.

[brumsky]

CPUs have been too busy doing other things! But I'll kick them off now.

I'm looking forward to your results as well. I tested hevc-qa and it was bad... I had weird banding and everything was smoothed out really bad.

[markiemarcus]

I'm looking forward to your results as well. I tested hevc-qa and it was bad... I had weird banding and everything was smoothed out really bad.

I've had positive results with HEVC-AQ on animation; in part due to it taking AQ strength out of the equation.

Live action not so much.

[benwaggoner]

I've had positive results with HEVC-AQ on animation; in part due to it taking AQ strength out of the equation.

Live action not so much.
I'm running a 2-pass veryslow Tears of Steel encodes at 1 Mbps ABR comparing aq-mode 2, 3, 4 and aq-hevc. I should have some clips to evaluate tomorrow.

[markiemarcus]

I'm running a 2-pass veryslow Tears of Steel encodes at 1 Mbps ABR comparing aq-mode 2, 3, 4 and aq-hevc. I should have some clips to evaluate tomorrow.

It has actually surprised me how much variation there is between them. I've only really tested extensively on animation, often at 720p where the problems are more noticeable. The trouble really starts when you disable SAO, but you often have to in order to preserve high frequency detail. The artifacts resemble a mixture of ringing and mosquito noise around dark lines.

Under these circumstances Aq mode 1 is by far the most prone to distortion and though it's subjectively more detailed, I don't like it, unless the source has a lot of grain where it can be quite useful. Aq mode 2 is much less prone to distortion, but grain can look soft and it's noticeably poor in low luma. Aq mode 3 generally works well, but it's often a sub optimal usage of bits, especially if only fleeting scenes are low luma. That's usually where Hevc-aq does rather well (though Cbq and Crq need a -1 nudge down). I find it to be the most predictable and least troublesome in motion.

Aq mode 4 I don't really know what to make of. With grainy animation it seems a little more detailed than Aq mode 2. Just to clarify again that the above is all for animation; I haven't looked at the metrics and I don't have much experience with live action.

Looking forward to your results! I'm a long time lurker.

[Boulder]

I'm running a 2-pass veryslow Tears of Steel encodes at 1 Mbps ABR comparing aq-mode 2, 3, 4 and aq-hevc. I should have some clips to evaluate tomorrow.

1 Mbps is quite low already, it would be nice to see some results with average bitrate around 7-8 Mbps or so.

[benwaggoner]

1 Mbps is quite low already, it would be nice to see some results with average bitrate around 7-8 Mbps or so.
I'd expect there's be a lot of convergence at high bitrates. I'm most interested in how the different modes do when they don't have enough bits to do it right.

I'm also replicating the procedure from my encoding challenge, for apples-to-apples

https://forum.doom9.org/showthread.php?t=175776

[tebugg]

hello everyone. i am seeking a little bit of help with cpu usage. i have an amd ryzen 3900x overclocked to 4200mhz. it is on an msi x570 godlike mobo. so the vrm's are the best. i currently can only get between 70-80% cpu usage on the first pass. on the second pass it drops down to 60% usage. this is encoding a 4K video file that was originally encoded with x265 from my cell phone. is there a way for me to squeeze out more usage from x265's settings?

edit: on crf the encoder maxes out all the cores. also on x264 with 2pass and crf both max out the cores.

[Forteen88]

i currently can only get between 70-80% cpu usage on the first pass. on the second pass it drops down to 60% usage..If you can, then you should do 2 encodes at the same time.

[LazyNcoder]

Hello guys,
Any specific commands to encode HDR10+?
--transfer smpte2094 instead of --transfer smpte2084? is it enough? Does it work at all? because I couldn't find smpte2094 on x265 documentations ...

Edit:
Nope.
x265 [error]: invalid argument: transfer = smpte2094
Is there something wrong with my build?

[kabelbrand]

Any specific commands to encode HDR10+?
There is a command line switch to insert HDR10+ metadata but you'll need a separate tool e.g. from Samsung to process your source file for metadata creation.
https://x265.readthedocs.io/en/default/cli.html#cmdoption-dhdr10-info

[LazyNcoder]

There is a command line switch to insert HDR10+ metadata but you'll need a separate tool e.g. from Samsung to process your source file for metadata creation.
https://x265.readthedocs.io/en/default/cli.html#cmdoption-dhdr10-info

Thank you.

1- What if the source we already have is HDR10+? Can we somehow extract or bypass the metadata into x265?

2- What tool from Samsung exactly? or any alternative

[kabelbrand]

I guess you'd have to be a HDR10+ adopter to get access to the Samsung tools (JsonFromVideo, Hdr10PlusGenerator) or instead use a software like Colorfront Transkoder.

[filler56789]

Maybe off-topic but probably a necessary notice...

«After much consideration, we’ve decided to remove Mercurial support from Bitbucket Cloud and its API. Mercurial features and repositories will be officially removed from Bitbucket and its API on June 1, 2020.»

https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket

[quietvoid]

Thank you.

1- What if the source we already have is HDR10+? Can we somehow extract or bypass the metadata into x265?

2- What tool from Samsung exactly? or any alternative

This might be useful for you: https://github.com/quietvoid/hdr10plus_parser

[LigH]

@filler56789: Good to get this notice way ahead ... enough time for the x265 team to possibly move the repository to either a different hoster or a different versioning system... (I would assume rather the first, since they love funny version tags which annoy some compiler suites not expecting their syntax).

[kabelbrand]

This might be useful for you: https://github.com/quietvoid/hdr10plus_parser

Thanks, good work! Even though the Samsung Hdr10PlusInjector tool does not like this JSON format I guess as long as x265 is happy that's good enough.

For reference I attached the raw JSON output from my test. There are some additional indexes and ids in the Samsung output but these might be optional.

[quietvoid]

Thanks, good work! Even though the Samsung Hdr10PlusInjector tool does not like this JSON format I guess as long as x265 is happy that's good enough.

For reference I attached the raw JSON output from my test. There are some additional indexes and ids in the Samsung output but these might be optional.

Yes, if I remember correctly x265 just adds the metadata for each frame encoded, scene related info is ignored.
At the time of development (possibly now too), HDR10+ titles also have metadata inserted at every frame, and not like 1 SEI message per scene.

After all, some of this is trial and error using x265 as "correct" implementation.
There is still some desync between source JSON and extracted where there is a metadata change (scene change?), but it's usually just 1-2 frames that are different.

[benwaggoner]

Yes, if I remember correctly x265 just adds the metadata for each frame encoded, scene related info is ignored.
At the time of development (possibly now too), HDR10+ titles also have metadata inserted at every frame, and not like 1 SEI message per scene.

After all, some of this is trial and error using x265 as "correct" implementation.
There is still some desync between source JSON and extracted where there is a metadata change (scene change?), but it's usually just 1-2 frames that are different.
You want to use --dhdr10-opt. That'll insert the SEI only on IDR frames and frames where the metadata changes.

Note that dynamic metadata isn't necessarily static across a shot; if there are dramatic changes in a shot, than there likely will be mid-shot metadata.

[quietvoid]

You want to use --dhdr10-opt. That'll insert the SEI only on IDR frames and frames where the metadata changes.

Note that dynamic metadata isn't necessarily static across a shot; if there are dramatic changes in a shot, than there likely will be mid-shot metadata.

I'm aware of x265's --dhdr10-opt, but I prefer to keep it the same as the source (which seems to always be for every frame).
Not sure if it would affect compatibility as well.