x265 HEVC Encoder

[LigH]

A patch for the Y4M pipe has just been mentioned in the mailinglist; waiting for commit...

[Selur]

or does the app developer (for RipBot264, StaxRip, Hybrid and others) will have to modify the "behind the scenes" command's.
Hybrid always used 'x265 --input - ' for pipe input, so nothing to adjust there. :)

A patch for the Y4M pipe has just been mentioned in the mailinglist; waiting for commit...
Nice!

Cu Selur

[jpsdr]

I've continued my build tests. Someone even suggest my to use "-Bstatic -l<lib> -l<lib> .. -Bdynamic", to force static link, but it didn't work.
Finaly, what worked was indeed to rename...
For build with mcf version, i renamed libstdc++ and libmcfgthread.
For build with clang just libc++.

[LigH]

New upload: x265 3.6+103-85f1e34+pipefix (https://www.mediafire.com/file/oei7rjjxiilvaur/x265_3.6+103-85f1e34+pipefix.7z/file)

[Windows][GCC 14.2.0][32/32XP/64 bit] 8bit+10bit+12bit

No serious changes in intel x86[-64] code. Added the patch to fix Y4M pipe handling prior to its commit.

[rwill]

What I do not understand is that there is code like this for the 'input' arg:

OPT("input") inputfn[0] = optarg;

So when one specifies --input something then inputfn[0] gets set to that.

Then there is new code like this:

#if !ENABLE_MULTIVIEW
if (optind < argc && !inputfn[0])
inputfn[0] = argv[optind++];
#endif

So having the input filename as one of the last trailing args on the command line is now only supported for non ENABLE_MULTIVIEW builds. I find this kind of strange as from a functionality point of view '--input' and having it 'trailing' is of no difference?

[LigH]

If you have multi-view enabled, and you actually want to encode MVC with separate input video sources for more than one camera angle, you must be able to tell apart one input from the other. That's why the implicit "last parameter is input" doesn't work, there can be only one last parameter.

[rwill]

Yes I know, but this is what their Multiview Config File is for. There can also only be one --input. Instead of breaking compatibility with a couple of GUIs and Scripts they could have used


if( !MultiViewConfigFile )
...


instead of


#if !ENABLE_MULTIVIEW

[LigH]

Telling us doesn't help. Tell MultiCoreWare.

[Z2697]

The pipe problem...
I did my fix weeks ago.
But now the patch in mailing list looks better to me.
(The "better" in the commit msg refers to me previously just remove " if (!pic_in[view]) " )
https://github.com/Mr-Z-2697/x265-Yuuki-Asuna/commit/3d72ad6a97be8d106c71e3e6e3a4121dbd49f793

[Selur]

I did my fix weeks ago.
And you did not post it to the dev list. why?

[Z2697]

And you did not post it to the dev list. why?

Hmm...
It's like, I don't know how. Sorry.

Besides, my code sucks.

[modus-ms325c]

Telling us doesn't help. Tell MultiCoreWare.

And you did not post it to the dev list. why?

so, how come have we, as a society, as the kind of people who just so happen to be grouped in some sort of space or bar or whatever, decided that giving this kind of response to literally anyone who has even the slightest of trust issues to code maintainers, is in any way normal?

or maybe dealing with code maintainers is just more trouble than it's worth being begged to acceptance, idk.

[Selur]

Sorry, but this thread is no where just some sort of space. + don't know what society you live in, since I have no clue about you.
Where LigH and I live (Germany), asking this isn't frowned upon like it seems in the society you live in.
Also posting a fix to an issue tracker or a mailing list is worth it, even if the maintainers do not accept it, others might.
A more philosophical discussion about societal development probably should be help in another environment.
Not in a thread about the development of the x265 encoder.

Cu Selur

[jpsdr]

Euh... It looks like a lot in the same place i put a fix on my github, as i said in post #9449, according rwill post.

Does my version also fix the issue ?

My commit is:
https://github.com/jpsdr/x265/commit/a70b9c0dd269154a92a86c3d6d3ded7556b96562
But no one told me if it fxes the issue or not, as i personnaly not using x265 in a way it produces the issue.

[LigH]

Added the patch to fix Y4M pipe handling prior to its commit.

This patch has been pushed to the master branch.

So now everyone can build (generic or modified) binaries without manually adding that specific patch, it is now included in official sources.

[jpsdr]

LOL ! It's exactly what i've done...! But still, thanks to the report of rwill.

[Selur]

Nice! Thanks for the update!

[rwill]

Now because people are so hesitant to contact MCW, they are a business trying to make money. They also still seem have to a development team so who are we to tell em how to do their job. I just write comments here so maybe people prep up their little forks locally.

Now regarding sending actual patches to x265-devel ... I think they require a signed contributor agreement to give them full copyright. Like, they are a business trying to make money. I personally don't have time for that and it is also a legal problem for me because I have my own AVC, HEVC, VVC, etc. encoders to take care of.

x264 was different, just hop on IRC (which had no account restrictions back then) and tell one of the devs of something you found. These were the good times..

[jpsdr]

Benchmark

Out of curiosity, i've made some benchmark of my builds.

I have a dedicated PC for video encoding/processing, it has no network cable connected, ethernet chipset is disabled in BIOS, all internet related services, printer, an not necessary stuff are disabled, only the mandatory services needed by the "core" are kept.
There is of course no firewall, antivirus or any kind of these stuff installed, so there is "nothing" else running on this PC.

mcf, win32 and clang version under msys2.
LLVM version integrated with Visual Studio.

I've made a multi-pass encode on a small 128 frames 4k HDR video.

mcf v14.2.1 (20240821)
Pass 1: 143.08s
Pass 2: 258.47s
Pass 3: 201.14s

win32 v13.1.0 (20230426)
Pass 1: 141.34s
Pass 2: 255.88s
Pass 3: 200.82s

clang v1.8.1
Pass 1: 139.64s
Pass 2: 254.95s
Pass 3: 198.47s

LLVM v1.8.1
Pass 1: 136.19s
Pass 2: 247.68s
Pass 3: 196.74s

Honnestly, it's not the results i expected... :(
With x264, i had mcf faster than win32 faster than posix.
I expected found at least here mcf faster than win32.

Also, with clang, i can add "-march=broadwell -mtune=broadwell" not supported with LLVM. So, i expected clang be at least as fast as LLVM.
But, i also for clang need to remove 90% of the options i put in gcc are they were either unknow (giving [Error]) or not supported (giving [Warning]). I still have to check with the VisualStudio command line for LLVM and also the site of clang to see if i can add some options on the clang command line, to at leat achieve the same speed than LLVM.

[LigH]

I don't understand why you expected any noticeable difference between binaries executing more than 90% of pre-optimized SIMD assembly code, so that the impact of remaining C/C++ code and compiler optimizations of not at all performance relevant parts remains negligible.

[jpsdr]

I wasn't clear enough on my "disapointment".
It's not the values, it's the order.
I expected clang faster than LLVM faster than mcf faster than win32.

[LigH]

The differences are so tiny, random delays caused by OS tasks and disk access hitting or missing the cache are way more relevant.

[jpsdr]

% between each version (mcf->win32, win32->clang, clang->llvm/mcf->llvm)
mcf v14.2.1 (20240821)
Pass 1: 143.08s
Pass 2: 258.47s
Pass 3: 201.14s

win32 v13.1.0 (20230426)
Pass 1: 141.34s [-1,27%]
Pass 2: 255.88s [-1,00%]
Pass 3: 200.82s [-0,16%]

clang v1.8.1
Pass 1: 139.64s [-1,21%]
Pass 2: 254.95s [-0,36%]
Pass 3: 198.47s [-1,17%]

LLVM v1.8.1
Pass 1: 136.19s [-2,47%]/[-4,82%]
Pass 2: 247.68s [-2,85%]/[-4,17%]
Pass 3: 196.74s [-1,01%]/[-2,19%]

Different are tiny but consistant.
I hoped i could switch from LLVM to clang, it would be faster for me to make build, but i'll stay with LLVM.

And about OS task, i've described the PC used, and how they are reduced to minimum. Disk is SSD. Even 4%, when you have 3 days for one pass, it's almost 3h, it's still interesting.

[Z2697]

I know this is very much a "source: trust me bro" situation but clang or the underlying LLVM will cause the built x265 to output different result compared to GCC built one, and in some situations this "difference" is significant decrease in quality.
So generally speaking, GCC is the way to go. For x265.
To be honest, encoding result being different happens very often. For example even FLAC will produce different result (but still perfectly lossless) when comparing Clang / GCC, static / shared. But for this specific software, x265, it is just worse.

By the way, I think the naming scheme is unclear.
Win32? An operating system API?
Clang? A compiler front-end?
LLVM? The "man in the chair" of Clang?
What is MCF?

[jpsdr]

win32 is the "win32" thread build of gcc by msystem.waw.
mcf is the "mcf" thread build of gcc by lhmouse.
clang is the clang version compiler for msys2 (mingw-w64-clang-x86_64-toolchain).
LLVM is the version you can install and use with Visual Studio (https://llvm.org/ or https://github.com/llvm/llvm-project/releases).

Otherwise, i'm using my LLVM builds for years, without issue noticed. And during these build tests, i've watched the .h265 raw streams created between each version, and didn't seen any difference. I didn't make a bit to bit compare.

Otherwise, to check speed stability, i run 3 more times the encode with LLVM version, results are :

Pass 1: 136.19s/135.94s/135.75s/135.98s
Pass 2: 247.68s/246.62s/247.58s/246.61s
Pass 3: 196.74s/193.37s/192.91s/193.71s

Edit:
I will tomorrow encode my test file with a gcc build and an llvm build and comparte the binary result file.

[Selur]

I've made a multi-pass encode on a small 128 frames 4k HDR video.
+
mcf v14.2.1 (20240821)
Pass 2: 258.47s

win32 v13.1.0 (20230426)
Pass 2: 255.88s [-1,00%]

clang v1.8.1
Pass 2: 254.95s [-0,36%]

LLVM v1.8.1
Pass 2: 247.68s [-2,85%]/[-4,17%]

that's like 2 seconds per frame (0.5 fps) :scared: during the 2nd pass.
Out of curiosity: Is your system that old, or are you using totally insane/placebo settings?

[Jamaika]

What do you think about the added corrections? There is no description of what this results from.
https://bitbucket.org/multicoreware/x265_git/diff/source/common/lowpassdct.cpp?diff2=7c190672efa6&at=master

https://bitbucket.org/multicoreware/x265_git/diff/source/common/lowpassdct.cpp?diff2=dd594f59d951&at=master

[Boulder]

What do you think about the added corrections? There is no description of what this results from.
https://bitbucket.org/multicoreware/x265_git/diff/source/common/lowpassdct.cpp?diff2=7c190672efa6&at=master

https://bitbucket.org/multicoreware/x265_git/diff/source/common/lowpassdct.cpp?diff2=dd594f59d951&at=master

They are related to this experimental feature: https://x265.readthedocs.io/en/master/cli.html#cmdoption-lowpass-dct. I don't think I've ever seen anyone use it.

[Jamaika]

They are related to this experimental feature: https://x265.readthedocs.io/en/master/cli.html#cmdoption-lowpass-dct. I don't think I've ever seen anyone use it.
So why is someone correcting it? Delete it.

How did it happen that first the value of 1 was changed to -1 and then only in one place 1 was differentiated to the value of -1.

[Boulder]

So why is someone correcting it? Delete it.
So you think the encoder should only have features that 99% of the average Joe user base uses? Just take a look at how many real-time encoding related things there are, why not ditch them as well then :p

[Jamaika]

I'm more interested in the train of thought. Does anyone do this on a whim?

Second thing. The https://bitbucket.org website seems to be a mirror without description.

[Boulder]

MCW is a commercial entity and I bet most of the work we see in the repo are results from paid or sponsored work. You can submit patches as a normal user but then you are allowing them to use those in commercial stuff.

[jpsdr]

@Selur
My system was the top when i made it... a few years ago. It's a 10 cores Broadwell overclocked to 3,8GHz.

You can see my settings in post #9368 (https://forum.doom9.org/showpost.php?p=2005481&postcount=9368).

[Z2697]

win32 is the "win32" thread build of gcc by msystem.waw.
mcf is the "mcf" thread build of gcc by lhmouse.
clang is the clang version compiler for msys2 (mingw-w64-clang-x86_64-toolchain).
LLVM is the version you can install and use with Visual Studio (https://llvm.org/ or https://github.com/llvm/llvm-project/releases).

Otherwise, i'm using my LLVM builds for years, without issue noticed. And during these build tests, i've watched the .h265 raw streams created between each version, and didn't seen any difference. I didn't make a bit to bit compare.

Otherwise, to check speed stability, i run 3 more times the encode with LLVM version, results are :

Pass 1: 136.19s/135.94s/135.75s/135.98s
Pass 2: 247.68s/246.62s/247.58s/246.61s
Pass 3: 196.74s/193.37s/192.91s/193.71s

Edit:
I will tomorrow encode my test file with a gcc build and an llvm build and comparte the binary result file.

Results from the different version of the same compiler and same flags is probably gonna be identical. That's not what I'm talking about. I was talking about different compilers.
These kind of difference is kinda common, at least in the encoders I have tested or seen someone tested. It's not necessarily mean something bad happened.

Now talk about the statement that the clang sometimes makes "that difference" of x265 to go bad. I must apology I didn't do a very comprehensive test. A friend told me and what I did was only confirming it.
After I did some more testing, I found that statement is probably not true. The only situation I can confirm is with very unusual encoding parameters. I can basically narrow it down to very low crf (e.g. 4), with negative cb/crqpoffset and rdoq enabled (rdoq-level 1 or 2).
The original example I got is with crf=4 and cb/crqpoffset=-2. Clang-built x265 will have severe blocking in chroma but GCC-built one is fine.
But if I give them more "extreme" values, like crf=-12 (10bit) and offsets=-12, GCC-built x265 will have severe problem too.
So Clang probably just shows this problem "earlier". But they have the same flaws under the hood.

As for the speed comparison, will you interested in doing some profile-guided optimization testing?

[Z2697]

@Selur
My system was the top when i made it... a few years ago. It's a 10 cores Broadwell overclocked to 3,8GHz.

You can see my settings in post #9368 (https://forum.doom9.org/showpost.php?p=2005481&postcount=9368).

Broadwell is almost a decade ago now😂

[tormento]

Broadwell is almost a decade ago now
I7-2600k here, daily overclocked to 4.6 GHz.

[jpsdr]

I was talking about different compilers.
So do I.

I've made an encode with a version build with llvm.
Then I've made an encode with a version build with gcc.
File were of different sizes.
But... to be sure, i redo the exact same encode with the gcc build version. Third file, third different size...
So, even with the exact same exe, running twice the exact same encode didn't even produce bit identical files, as they even are not of the same size. Of course, difference is small (around 2kB~4kB for a 20MB file).

For speed, thanks for the offer, but i will not be doing more testing.

[Z2697]

So, even with the exact same exe, running twice the exact same encode didn't even produce bit identical files, as they even are not of the same size. Of course, difference is small (around 2kB~4kB for a 20MB file).


That's not normal I'd say. Most times the result should be exactly the same, with same exact settings and executable file. (Even the 3.5 and 3.6 versions I compiled prodeces same results (except for header) with my daily settings.)
Something I can remember now is the HME feature, it will produce different result when combined with lookahead-slices.
I've also mainly tested 1-pass CRF only, not sure if it's the 3-pass ABR encoding that's responsible.

[Lucky38]

Just building: Latest commit adds support for Screen Content Coding (https://hevc.hhi.fraunhofer.de/scc), requires compile option enabled


__

New upload: x265 3.6+65-3c3d03746 (https://www.mediafire.com/file/5zoll6p8s1hvdnn/x265_3.6+65-3c3d03746.7z/file)

[Windows][GCC 14.2.0][32/32XP/64 bit] 8bit+10bit+12bit
-DENABLE_HDR10_PLUS=ON -DENABLE_LIBVMAF=ON -DENABLE_ALPHA=ON -DENABLE_MULTIVIEW=ON -DENABLE_SCC_EXT=ON


what is real benefit when SCC is ON?

[LigH]

Did you even read the linked explanation by Fraunhofer HHI?

The Screen Content Coding (SCC) extensions will improve compression capability for video containing a significant portion of rendered (moving or static) graphics, text, or animation rather than (or in addition to) camera-captured video scenes. Example applications include wireless displays, remote computer desktop access, and real-time screen sharing for videoconferencing.

[Lucky38]

Did you even read the linked explanation by Fraunhofer HHI?

i read it... so it is not something that can be used for movies, tv shows then...

[LigH]

Depends. If it is a movie with a lot of artificial screen overlays, maybe. Then test it with and without, compare, and then decide.

Please note, this is an extension to the basic HEVC standard. Not all decoders may support it.


PS: Another batch of AArch64 patches was just pushed. So x86* builds are probably not affected.

[tormento]

I have tried Patman’s build and SCC seems to be encoding but the result is an error about premature end of file and a black unreadable video. He confirmed the issue.

Someone had better luck?

[LigH]

Yes, a freshly built x265 v3.6+130 with SCC enabled encodes but MPC-HC with its internal LAV filters can't decode it properly. That's probably due the libavcodec HEVC decoder to not supporting HEVC SCC Profile bitcode. You would need to find another decoder which is able.

See also: Wikipedia: High Efficiency Video Coding | Version 3 and higher profiles (https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding#Version_3_and_higher_profiles)

[Boulder]

SCC is one those things related to streaming, like webinars with PowerPoint presentations etc. For example SVT-AV1 has had it for a long time already, but of course it's really not meant for your regular content.

[tormento]

Yes, a freshly built x265 v3.6+130 with SCC enabled encodes but MPC-HC with its internal LAV filters can't decode it properly.
Do you know a player that can actually reproduce it?

Why does StaxRip give a frame error at the end? I didn't try with other GUIs so I don't know if it's a x265 or StaxRip issue.

Moreover, when muxing the hevc, MKVToolnix-gui outputs a very short video.

[LigH]

Quite recent MP4Box and FFmpeg versions can multiplex it to MP4. But mkvmerge seems to recognise only HEVC packets known in basic profiles.

And the native HEVC decoder in libavcodec, core of ffmpeg, does not yet support decoding it.

[LigH]

New upload: x265 3.6+130-184bed4 (https://www.mediafire.com/file/pvybf780x601u03/x265_3.6+130-184bed4.7z/file)

[Windows][GCC 14.2.0][32/32XP/64 bit] 8bit+10bit+12bit

No serious changes in intel x86[-64] code.

[LigH]

MKVToolnix-gui outputs a very short video.

Reported to Moritz Bunkus (https://gitlab.com/mbunkus/mkvtoolnix/-/issues/3738) - low priority, there is no need to hurry without a decoder yet.

[LigH]

x265 Version 4.0 has been released:

The v4.0 release of x265 is out now. Below is the list of new features, optimization, and bug fixes in this version.


Version 4.0
===========

New feature
-----------
1. Alpha Channel feature.
2. Screen Content Coding (SCC).
3. MV-HEVC feature.

Enhancements to existing features
---------------------------------
1. Added support for the VMAF v3.x.

API changes
-----------
1. Add command line parameter for Alpha Channel feature :option:`--alpha`.
2. Add command line parameter for SCC feature :option:`--scc 1`.
3. Add command line parameters for the MV-HEVC feature :option:`--multiview-config "multiview_config.txt"`.

Optimizations
---------------------
1. Arm SIMD optimizations: Several time-consuming scalar C functions now have SIMD implementations on Arm platforms. Existing Arm SIMD implementations have also been optimized. These optimizations result in up to 57% faster encoding compared to release 3.6.
2. Arm SIMD optimizations include use of Armv8.4 DotProd, Armv8.6 I8MM, and Armv9 SVE2 instruction set extensions. The following algorithms now have optimized SIMD implementations: SAD, SSE, DCT, SAO, convolution, quantization, intra_planar, intraFilter, intrapred DC and IDCT16x16.

Bug fixes
---------
1. Fix for y4m pipe input broken.
2. Fix SCC crash on multipass encode.
3. Fix mcstf when :option:`--bframes` value was less than 5.
4. Fix lowpass DCT for high bit depth.
5. Added build support for Visual Studio 17.
6. Fix issue in default code flow and memory leak.
7. Framethreads tuning for Windows ARM devices.
8. Fix scc crash on multipass encode.

Thanks & Regrds,

Karam Singh