x265 HEVC Encoder

[benwaggoner]

It appears to have been split into two options: --qp-delta-ref and --qp-delta-nonref. The first is for REFERENCED p/b frames and defaults to 5.0, while the second is for NON-REFERENCED p/b frames and defaults to 6.5.

This is present in the CLI encoder's documentation, but not on x265.readthedocs.io; 3.4+20 (and all versions after 3.4+4) are development builds, rather than release. (Even if the difference between the two is usually kinda small.)
This seems like a good feature to have. Spending fewer bits on non-reference frames in order to spend more on reference frames so they provide better predictors for the frames they reference is a key feature of good rate control.

We already have that in the ip and pb ratios. But with HEVC you can have non-referenced blocks in all frame types.

Getting it tuned so there isn't any thresholding (visible seams when a contiguous area has some reference and some non-reference blocks) is the tricky part with such things.

[birdie]

They've migrated to a new repo and pretty much all the old open issues have been lost in transition. Sigh.

[benwaggoner]

They've migrated to a new repo and pretty much all the old open issues have been lost in transition. Sigh.
Well, we should go back and add new repros for bugs that still exist. I hope some of the older ones have gotten fixed in the backgroud.

[Majorlag]

ok.. seriously there's a problem with x265 and HDR10+ encoding...

tried another HDR10+ encode... source has 195593 frames, encode has 195585. Without HDR10+ (only HDR10) encode has 195593. It's clearly problem of x265, as with nVidia and Intel's encoders, using the same HDR10+ metadata file, output has right number of frames.

How are you feeding x265, hopefully with a frame accurate frame server?

What frame server are you using AviSynth+ or VaporSynth? Also which indexer are you using? I tend to get better consistent results with FFMSIndex. To many failed encodes with L-SMASH Works indexer.

Have you tried removing all x265 custom parameters and just use defaults with just hdr10+ options to eliminate any conflicting parameters?

[Barough]

x265 v3.4+24-g6ed1c24a2 (http://www.mediafire.com/file/x813hgzm896xpz7/x265-3.4+24-g6ed1c24a2_Win_GCC102.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 10.2.0)

https://bitbucket.org/multicoreware/x265_git/commits/branch/master

[jlpsvk]

How are you feeding x265, hopefully with a frame accurate frame server?

What frame server are you using AviSynth+ or VaporSynth? Also which indexer are you using? I tend to get better consistent results with FFMSIndex. To many failed encodes with L-SMASH Works indexer.

Have you tried removing all x265 custom parameters and just use defaults with just hdr10+ options to eliminate any conflicting parameters?

no matter what i try. :) still the same result, staxrip, ripbot264, command line.... still the same... with the same setting but just without dhdr, encoded frame number is identical to source

as soon as i use GPU encoder (intel or nvidia), encode is fine.

[benwaggoner]

no matter what i try. :) still the same result, staxrip, ripbot264, command line.... still the same... with the same setting but just without dhdr, encoded frame number is identical to source

as soon as i use GPU encoder (intel or nvidia), encode is fine.
Check your JSON file, and see if it has a different number of frames than your source or something. Or if the final frames that are cut off have all have the same metadata. My wild guess is that x265 is cutting off frames that it doesn't have metadata for, or unique metadata for. Which would be a bug.

[jlpsvk]

Check your JSON file, and see if it has a different number of frames than your source or something. Or if the final frames that are cut off have all have the same metadata. My wild guess is that x265 is cutting off frames that it doesn't have metadata for, or unique metadata for. Which would be a bug.

when using JSON for encode with intel or nvidia, everything is OK.... :( so i excluded wrong JSON.. i also tried with different movies and the results are the same.

[Barough]

x265 v3.4+26-ga82c6c7a7 (http://www.mediafire.com/file/errun2hhaj9u0mw/x265-3.4+26-ga82c6c7a7_Win_GCC102.7z/file) (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 10.2.0)

https://bitbucket.org/multicoreware/x265_git/commits/branch/master

[filler56789]

OK, I was bored, so I compiled x265.exe 3.4+30-g6722fce.

GIT command-line:
git clone --single-branch --branch Release_3.5 --depth 128 https://bitbucket.org/multicoreware/x265_git.git

Changes since commit ga82c6c7a7:
https://bitbucket.org/multicoreware/x265_git/commits/all

Download:
http://www.mediafire.com/file/8n5bnny3rc4915x/x265_3.4%252B30-g6722fce.7z/file

[benwaggoner]

And wow, this thread went three weeks without a post.

I see from the git that we're getting x265 3.5 features checked in.

[markiemarcus]

AQ needs some refactoring, for sure!
aq-mode 3=2 with low luma bias. hevc-aq is really aq-mode 5, and should map to that.

Aq-motion is a lot more experimental than hevc-aq, and I've not heard anyone suggest it'd be appropriate to have on by default. HEVC-AQ is a lot more mature at this point, althogh newer.



Yes the mapping is a little confusing! Also that Aq strength has no impact on hevc-aq could probably be better clarified.

I must say though, HEVC-AQ is by far my preferred method of encoding animation in instances with selective sao 0-2. The other aq modes typically produce more noise around dark lines. Even with 2 pass encodes and checking aq-strength in steps of 0.05, I haven't been able to replicate hevc-aq's results in this area with any of the other aq modes. There's also something about having RD greater than 4 and b-intra enabled that really seems to helps with this too. Slow though, but you can make compromises in other areas. Rect for instance seems to achieve nothing or very little. Tskip works well.

How have you found aq mode 4? With grainier or more detailed content, I'm finding the results extremely encouraging. Seems to fare well on flat areas with grain, especially in high motion.

[vpupkind]

hevc-aq and aq mode 4 are different: hevc-aq is based on quartile variance, while aq mode 4 is based on variance of edges

[markiemarcus]

hevc-aq and aq mode 4 are different: hevc-aq is based on quartile variance, while aq mode 4 is based on variance of edges

Very obviously so based on the results alone. Aq mode 4 is actually a fair bit slower too.

They both seem quite mature to me; but it's definitely very much source dependent. HEVC-AQ for instance I've had less than ideal results with live action, but it's rather good and my preferred mode with animation.

[vpupkind]

Aq mode 4 is actually a fair bit slower too.


AQ4, histogram-based scenecut, and rskip 2 all depend on an edge detection pre-pass.

[markiemarcus]

AQ4, histogram-based scenecut, and rskip 2 all depend on an edge detection pre-pass.

Ah that explains a lot.

[quietvoid]

I've created a new issue for the previously mentioned x265 HDR10+ crashes: https://bitbucket.org/multicoreware/x265_git/issues/574/x265-segmentation-fault-when-using-hdr10
If someone knows a bit around the code, it might be simple to fix.
Otherwise I hope it reaches someone over at MCW.

[jlpsvk]

I've created a new issue for the previously mentioned x265 HDR10+ crashes: https://bitbucket.org/multicoreware/x265_git/issues/574/x265-segmentation-fault-when-using-hdr10
If someone knows a bit around the code, it might be simple to fix.
Otherwise I hope it reaches someone over at MCW.

Hail to quietvoid!!! :)

[LigH]

I'm getting buried under NASM warnings for over a month now... patching them out like in x264 is not good enough for x265. They seem to need experienced contributors.

[filler56789]

https://bitbucket.org/multicoreware/x265_git/issues/559/warnings-when-assembling-with-nasm-215

so the problem is 3 months old already -_-

I keep using nasm 2.14, but I know the users of MABS have to use what MABS gives them :-/

[LeXXuz]

So many updates in the last couple of weeks/month. May I ask what version is recommended for daily use?

[Selur]

Can someone clear up why '--tune psnr' disables '--hevc-aq' ?

For detail how I come to this conclusion read:
https://forum.doom9.org/showpost.php?p=1925877
https://forum.doom9.org/showpost.php?p=1925878

Cu Selur

Ps.: also created an entry in the issue tracker (https://bitbucket.org/multicoreware/x265_git/issues/575/can-someone-clear-up-why-tune-psnr)

[vpupkind]

I think this is the expected behavior for tune=psnr.
If I am not mistaken, it always disabled adaptive quantization and psy-rd.

[Asmodian]

Can someone clear up why '--tune psnr' disables '--hevc-aq' ?

Because psnr is worse with --hevc-aq enabled?

[LigH]

Yes. Psycho-visual enhancements in general shall appear more convenient to a human, but will probably cause a bigger objective difference to the original.

[Selur]

If I am not mistaken, it always disabled adaptive quantization and psy-rd.
But shouldn't "--tune psnr --hevc-aq" and "--aq-mode 0 --no-psy-rd --psy-rdoq 0 --hevc-aq" then not have the same effect?

Because psnr is worse with --hevc-aq enabled?
but there shouldn't it be documented somewhere that "--tune psnr" disables "-hevc-aq" and "--aq-mode 0 --no-psy-rd --psy-rdoq 0" does not which according to https://bitbucket.org/multicoreware/x265_git/src/master/source/common/param.cpp (line 566-571) is what psnr does?

Cu Selur

[benwaggoner]

But shouldn't "--tune psnr --hevc-aq" and "--aq-mode 0 --no-psy-rd --psy-rdoq 0 --hevc-aq" then not have the same effect?

but there shouldn't it be documented somewhere that "--tune psnr" disables "-hevc-aq" and "--aq-mode 0 --no-psy-rd --psy-rdoq 0" does not which according to https://bitbucket.org/multicoreware/x265_git/src/master/source/common/param.cpp (line 566-571) is what psnr does?
Neither SSIM or PSNR are the "native" internal metric used by x265. Thus turning off all psychovisual tuning doesn't push encoding into PNSR.

And a really aggressive PSNR optimization has lots of unfortunate side effects. For example, optimizing for mean PSNR would mean subjective quality would be higher at the start of a GOP and lower at the end, as getting really good references early improves the whole GOP, while encoding a lot of intra on the last B-frame doesn't.

Pretty much any metric winds up being increasingly pathological subjectively the more closely its objective values are optimized for. There are plenty of tricks that can be played with VMAF even.

[Selur]

This wasn't really about metrics, but about that shortening the command line using options '--tune psnr' should cover has unexpected effects,..

Cu Selur

[LigH]

x264 --fullhelp
--tune <string> Tune the settings for a particular type of source
or situation
Overridden by user settings.
Multiple tunings are separated by commas.
Only one psy tuning can be used at a time.
...
- psnr (psy tuning):
--aq-mode 0 --no-psy
...

x265 --fullhelp is not as verbose. More information only online (https://x265.readthedocs.io/en/master/presets.html#tunings) (see blue Note).

Would you like some kind of x265 [info|warning]: PSNR tuning may result in subjectively lower quality console output?

[Selur]

According to the x265 source code:

if (!strcmp(tune, "psnr"))
{
param->rc.aqStrength = 0.0;
param->psyRd = 0.0;
param->psyRdoq = 0.0;
}
source: https://bitbucket.org/multicoreware/x265_git/src/master/source/common/param.cpp
"--tune psr" should be the same as using "--aq-strength 0 --no-psy-rd --psy-rdoq 0"
"--aq-strength 0" is the same as "--aq-mode 0" and "--psy-rdoq" shouldn't be used when "--no-psy-rd" is used
so:
1. --aq-mode 0 --psy-rdoq 0 --hevc-aq
should have the same effect as:
2. --psnr --hevc-aq
which is has not.
When using 1. --hevc-aq is used, but with 2. it is not;

[poisondeathray]

so:
1. --aq-mode 0 --psy-rdoq 0 --hevc-aq
should have the same effect as:
2. --psnr --hevc-aq
which is has not.
When using 1. --hevc-aq is used, but with 2. it is not;

typo?

--tune psnr is not the same thing as --psnr

--psnr just calculates metrics

[LigH]

Surely Selur meant --tune psnr

[Selur]

yes I meant '--tune psnr'

[Asmodian]

I agree, --hevc-aq is a new option that is and should be turned off by --tune psnr. The documentation simply needs updating to let the user know --tune psnr will turn it off like the other AQ options.

[benwaggoner]

I agree, --hevc-aq is a new option that is and should be turned off by --tune psnr. The documentation simply needs updating to let the user know --tune psnr will turn it off like the other AQ options.
--hevc-aq is a terrible parameter, as it introduces a lot of unneeded interdependency with other parameters. If it was just --aq-mode 5 everything would just work.

[Selur]

If it was just --aq-mode 5 everything would just work.
agree :)

[Boulder]

Has anyone made any comprehensive tests regarding the various rskip modes? I was just thinking if CPU time is not a factor, would --rskip 0 always produce the highest quality output, no matter if it's a high or low bitrate encode. Rskip mode 2 clearly produces less onion artifacts compared to mode 1 but I'm not sure if allowing full recursion would bring any improvement.

This question arises because I'm always having a hard time trying to understand what the x265 development is targeting. It often seems to be trying to do more things at very low bitrates, which usually means losing detail etc. while my needs are quite the opposite.

[-QfG-]

Has anyone an idea how i can make a single layer DV Encode? I mean, if i have both layers, HDR10 and DV - how can i encode one single Layer DV file, in case how makeMKV does this? I have seen an single Layer DV x265 Encode, so i think there is a functionality.

Many Thanks for response.

[LigH]

What does "DV" mean in this context?

[Majorlag]

Has anyone an idea how i can make a single layer DV Encode? I mean, if i have both layers, HDR10 and DV - how can i encode one single Layer DV file, in case how makeMKV does this? I have seen an single Layer DV x265 Encode, so i think there is a functionality.

Many Thanks for response.

I am going to guess your talking about DolbyVision. As per makeMKV forum, you would use mp4 muxer, instructions at https://makemkv.com/forum/viewtopic.php?t=18602

mp4muxer --dv-profile 7 --input-file (baselayername.hevc1) --input-file (dvlayername.hevc2) --input-file (audio.ac3) --media-lang eng --output-file (nameofmovie).mp4

[-QfG-]

dv profile 7 is Dual Layer. I need Profile 8.1 Single Layer DV + HDR Support. The newest makeMKV can create this Video Stream from a DV Dual Layer UHD, so i have one Single DV + HDR Videostream.
I search a Tool, that can convert 2 Streams (Video Layer and Dolby Vision Layer) in a Single Layer DV Profile 8.1.

[jlpsvk]

dv profile 7 is Dual Layer. I need Profile 8.1 Single Layer DV + HDR Support. The newest makeMKV can create this Video Stream from a DV Dual Layer UHD, so i have one Single DV + HDR Videostream.
I search a Tool, that can convert 2 Streams (Video Layer and Dolby Vision Layer) in a Single Layer DV Profile 8.1.

you want this...

https://www.makemkv.com/forum/viewtopic.php?f=12&t=18602&p=96282#p96282

[agressiv]

dv profile 7 is Dual Layer. I need Profile 8.1 Single Layer DV + HDR Support. The newest makeMKV can create this Video Stream from a DV Dual Layer UHD, so i have one Single DV + HDR Videostream.
I search a Tool, that can convert 2 Streams (Video Layer and Dolby Vision Layer) in a Single Layer DV Profile 8.1.

MakeMKV is doing their own thing by adding the Profile 7 EL on top of a the BL in a "piggyback" fashion, but this is now "officially" supported in Matroska. MKVToolnix is currently unaware of Dolby Vision, but the latest MediaInfo will see it.

If you truly want Profile 8.1, you'll need the RPU extracted as an elementary stream and can use x265.exe to encode it with this guide:

http://x265.org/x265-delivers-dolby-vision-streams/

The MakeMKV guide linked above is good if you just want to remux, but depending on your player - TV's tend to like dual-layer DV and players like NVidia Shield want single layer DV.

Muxing the video - well, let's just say that Dolby's mp4muxer doesn't like to mux encoded x265 Dolby Vision Profile 8.1 streams - it tends to just hang. I would advise using TSMuxeR until MKVToolnix has native support, assuming they handle other DV profiles when support is added. If you can get mp4muxer to work on encoded DV content, I'm all ears on how you got it to work.

[jlpsvk]

MakeMKV is doing their own thing by adding the Profile 7 EL on top of a the BL in a "piggyback" fashion, but this is now "officially" supported in Matroska. MKVToolnix is currently unaware of Dolby Vision, but the latest MediaInfo will see it.

If you truly want Profile 8.1, you'll need the RPU extracted as an elementary stream and can use x265.exe to encode it with this guide:

http://x265.org/x265-delivers-dolby-vision-streams/

The MakeMKV guide linked above is good if you just want to remux, but depending on your player - TV's tend to like dual-layer DV and players like NVidia Shield want single layer DV.

Muxing the video - well, let's just say that Dolby's mp4muxer doesn't like to mux encoded x265 Dolby Vision Profile 8.1 streams - it tends to just hang. I would advise using TSMuxeR until MKVToolnix has native support, assuming they handle other DV profiles when support is added. If you can get mp4muxer to work on encoded DV content, I'm all ears on how you got it to work.

MKVtoolnix 51 is aware of DoVi and preserves metadata...

With the guide o posted, you can encode pure HDR without cropping, the use encoded hevc to combine with EL+RPU to combine into PROFILE 8.1

[LigH]

New upload: x265 3.4+27-g5163c32d7 (https://www.mediafire.com/file/xzi9yyztn0wk5gi/x265_3.4+27-g5163c32d7.7z/file)

CLI changes since x265 3.4+14-gd419c7152:


Defaults changed from undef to unknown for:
--overscan
--colorprim
--colormatrix
New options:
--min-vbv-fullness <double> Minimum VBV fullness percentage to be maintained. Default 50.00
--max-vbv-fullness <double> Maximum VBV fullness percentage to be maintained. Default 80.00

--[no-]vbv-live-multi-pass Enable realtime VBV in rate control 2 pass.Default disabled


Apparently there is a new commit based on an older branch (+27) which only updates the documentation but skips the latest additions with the "ABR Ladder" (+30)...

[ghostshadow]

hello, on version 3.0 you have introduced the parameters:
–dolby-vision-profile <integer|float>
and
–dolby-vision-rpu’ File containing Dolby Vision RPU metadata.
Excluding as for:
--dhdr10-info
You do not provide anything to exploit them. Indeed how to extract the Dolby Vision RPU? So these parameters are not usable.
For HDR10 + (--dhdr10-info) a coder had to face a python script to extract the metadata to use the: --dhdr10-info.
So what do we do to exploit the: –dolby-vision-rpu
thank

[LigH]

New upload: x265 3.4+30-g6722fce1f (https://www.mediafire.com/file/sk7nni880d16eo8/x265_3.4+30-g6722fce1f.7z/file)

[benwaggoner]

hello, on version 3.0 you have introduced the parameters:
–dolby-vision-profile <integer|float>
and
–dolby-vision-rpu’ File containing Dolby Vision RPU metadata.
Excluding as for:
--dhdr10-info
You do not provide anything to exploit them. Indeed how to extract the Dolby Vision RPU? So these parameters are not usable.
For HDR10 + (--dhdr10-info) a coder had to face a python script to extract the metadata to use the: --dhdr10-info.
So what do we do to exploit the: –dolby-vision-rpu
thank
Normally one uses software that generates the RPU file, like Dolby's SDK or ColorFront Transkoder. Unless Profile 8.x is targeted, that's coupled with software that creates the non-backwards compatible base layer, and the RPU data is relative to that.

Dolby hasn't really invested in making Dolby Vision a hobbyist-friendly format. Although the support in the iPhone 12 Pro might change that some. But even that uses a new profile that's not widely supported by TVs (HLG + metadata), and doesn't have RPU data compatible with typical DoVi content. Making a DoVi Blu-ray from an iPhone DoVi file requires changing every pixel in the video and generating new dynamic metadata.

[Blue_MiSfit]

Yeah, how would one even do that? Convert the HLG into PQ and do a new analysis pass in Resolve or something, then encode UHD BD compliant Profile 5 or 7 from that?

[FranceBB]

Dolby hasn't really invested in making Dolby Vision a hobbyist-friendly format. Although the support in the iPhone 12 Pro might change that some. But even that uses a new profile that's not widely supported by TVs (HLG + metadata), and doesn't have RPU data compatible with typical DoVi content. Making a DoVi Blu-ray from an iPhone DoVi file requires changing every pixel in the video and generating new dynamic metadata.

Our of curiosity, how are the new iPhones recording?
I saw the "Dolby Vision" announcement days ago, but it was rather vague. Is it H.265 10bit UHD HDR HLG 60p with metadata? Does it have the FULL HD 10bit additional layer with info to make the 12bit final result? Or perhaps it's still one layer in 8bit? And most importantly, given that most mobile phones have a ridiculously tiny sensor whose stops are low, does all this even make sense? I mean, I read that it can achieve up to 12 stops, but I think that it's not gonna get even close. SDR in BT709 100 nits is 6 stops, if they manage to get 9 stops out of that it would already be a miracle...