x265 HEVC Encoder

[FranceBB]

Hi there,
I always forget who's maintaining those: http://msystem.waw.pl/x265/

but... can I ask for an AVX-512 Windows x64 build?

[DJATOM]

Afaik all builds supports AVX512, but you have to enable it explicitly (using --avx512 option). And it benefits only in 4k encodes, so it's not really useful for common user.

[FranceBB]

Originally Posted by DJATOM

Afaik all builds supports AVX512, but you have to enable it explicitly (using --avx512 option). And it benefits only in 4k encodes, so it's not really useful for common user.

Ah so that's why it's not using them by default, I have to specify it from the command line, got it, now I remember, it's:

--asm=avx512

sometimes I learn something and that pushes out something else I already knew xD

[FranceBB]

New issue: why are 2 pools being created (56 + 56 = 112) but only 1 CPU (56) is used?

https://i.imgur.com/eWyQo1R.png

#Indexing the HDR10 PQ BT2100 4:2:2 10bit planar source
video=FFVideoSource("\\10.64.6.11\avisynth\Scambio\WARNER_MOVIES\BatmanBegins\HDR\BATMANBEGINS_INTL_UHD_HDR_PQ_REC2020_2398_240_PR4444_ITA51_ITA20_EN51_EN20-20210508155521.mov")
ch1=FFAudioSource("\\10.64.6.11\avisynth\Scambio\WARNER_MOVIES\BatmanBegins\HDR\BATMANBEGINS_INTL_UHD_HDR_PQ_REC2020_2398_240_PR4444_ITA51_ITA20_EN51_EN20-20210508155521.mov", track=1)
ch2=FFAudioSource("\\10.64.6.11\avisynth\Scambio\WARNER_MOVIES\BatmanBegins\HDR\BATMANBEGINS_INTL_UHD_HDR_PQ_REC2020_2398_240_PR4444_ITA51_ITA20_EN51_EN20-20210508155521.mov", track=2)
ch3=FFAudioSource("\\10.64.6.11\avisynth\Scambio\WARNER_MOVIES\BatmanBegins\HDR\BATMANBEGINS_INTL_UHD_HDR_PQ_REC2020_2398_240_PR4444_ITA51_ITA20_EN51_EN20-20210508155521.mov", track=3)
ch4=FFAudioSource("\\10.64.6.11\avisynth\Scambio\WARNER_MOVIES\BatmanBegins\HDR\BATMANBEGINS_INTL_UHD_HDR_PQ_REC2020_2398_240_PR4444_ITA51_ITA20_EN51_EN20-20210508155521.mov", track=4)
ch5=FFAudioSource("\\10.64.6.11\avisynth\Scambio\WARNER_MOVIES\BatmanBegins\HDR\BATMANBEGINS_INTL_UHD_HDR_PQ_REC2020_2398_240_PR4444_ITA51_ITA20_EN51_EN20-20210508155521.mov", track=5)
ch6=FFAudioSource("\\10.64.6.11\avisynth\Scambio\WARNER_MOVIES\BatmanBegins\HDR\BATMANBEGINS_INTL_UHD_HDR_PQ_REC2020_2398_240_PR4444_ITA51_ITA20_EN51_EN20-20210508155521.mov", track=6)
audio=MergeChannels(ch1, ch2, ch3, ch4, ch5, ch6)
AudioDub(video, audio)

#From 4:2:2 10bit to 4:2:2 16bit planar
ConvertBits(16)

#From 4:2:2 16bit planar Narrow Range to RGB Planar 16bit Narrow Range
z_ConvertFormat(pixel_type="RGBP16", colorspace_op="2020:2020:2020:limited=>rgb:2020:2020:limited", dither_type="none")

#From PQ to HLG with 16bit precision
Cube("I:\Scambio\WarnerBros\WarnerBros_PQToHLG_MaxCLL_1000.cube", fullrange=true)

#From RGB 16bit planar Narrow Range to YUV420 16bit planar Narrow Range
z_ConvertFormat(pixel_type="YUV420P16", colorspace_op="rgb:2020:2020:limited=>2020:2020:2020:limited", dither_type="none")

#Spatial denoise with 16bit planar precision
dfttest(sigma=64, tbsize=1, lsb_in=false, lsb=false, Y=true, U=true, V=true, opt=0, dither=0)

ffmpeg.exe -i "D:\Temp\AVS Script.avs" -vf scale=out_color_matrix=bt2020nc:out_h_chr_pos=0:out_v_chr_pos=0 -pix_fmt yuv420p16le -strict -1 -an -f yuv4mpegpipe - | x265.exe --y4m - --dither --preset medium --level 5.0 --tune fastdecode --no-high-tier --ref 4 --profile main10 --bitrate 25000 --deblock -1:-1 --min-luma 64 --max-luma 940 --chromaloc 2 --range limited --videoformat component --colorprim bt2020 --transfer arib-std-b67 --colormatrix bt2020nc --overscan show --no-open-gop --min-keyint 1 --keyint 24 --repeat-headers --rd 3 --vbv-maxrate 25000 --vbv-bufsize 25000 --wpp --pass 1 --asm=avx512 -o NUL 

ffmpeg.exe -i "D:\Temp\AVS Script.avs" -vf scale=out_color_matrix=bt2020nc:out_h_chr_pos=0:out_v_chr_pos=0 -pix_fmt yuv420p16le -strict -1 -an -f yuv4mpegpipe - | x265.exe --y4m - --dither --preset medium --level 5.0 --tune fastdecode --no-high-tier --ref 4 --profile main10 --bitrate 25000 --deblock -1:-1 --min-luma 64 --max-luma 940 --chromaloc 2 --range limited --videoformat component --colorprim bt2020 --transfer bt2020-10 --colormatrix bt2020nc --overscan show --no-open-gop --min-keyint 1 --keyint 24 --repeat-headers --rd 3 --vbv-maxrate 25000 --vbv-bufsize 25000  --wpp --pass 2 --asm=avx512 -o "D:\Temp\raw_video.hevc"



C:\Program Files\MKVToolNix\mkvmerge.exe --output "D:\Temp\raw_video.mkv" --language 0:und ^"^(^" D:\Temp\raw_video.hevc ^"^)^"

ffmpeg.exe -i "D:\Temp\AVS Script.avs" -map 0:1 -af loudnorm=I=-24:LRA=14:tp=-2 -c:a ac3 -b:a 384k -ar 48000 "D:\Temp\audio.ac3"

ffmpeg.exe -i "D:\Temp\AVS Script.avs" -map 0:1 -af loudnorm=I=-24:LRA=14:tp=-2 -c:a libtwolame -b:a 192k -ar 48000 "D:\Temp\audio.mp2"

ffmpeg.exe  -i "D:\Temp\raw_video.mkv" -i "D:\Temp\audio.ac3" -i "D:\Temp\audio.mp2" -map 0:0 -map 1:0 -map 2:0 -c:v copy -c:a copy -c:a copy  -f mpegts -metadata provider_name="Sky" -metadata service_provider="Sky" -metadata service_name="Sky" -mpegts_original_network_id 0x1122 -mpegts_transport_stream_id 0x3344 -mpegts_service_id 0x5566 -mpegts_pmt_start_pid 0x1500 -mpegts_start_pid 0x150 -mpegts_service_type "digital_tv" -mpegts_flags "resend_headers" "D:\Temp\final.ts"

pause

[LeXXuz]

Originally Posted by FranceBB

Hi there,
I always forget who's maintaining those: http://msystem.waw.pl/x265/

Btw who IS maintaining those?

[FranceBB]

Why am I getting VBV violation? (the command line is the one above). x265 3.5+13-4c6880d

[Boulder]

Based on those QPs, your source is just too compressible and the encoder cannot keep up with the requested bitrate at those points?

[FranceBB]

Originally Posted by Boulder

Based on those QPs, your source is just too compressible and the encoder cannot keep up with the requested bitrate at those points?

Makes sense.
The movie is 1h and 30 and it looks like it's near the end, which is pure black with static credits, so it's very likely.
The CBR 1 Gbit/s ProRes is almost definitely zero filled towards that point.
Nobody asked for CBR, so VBR 25 Mbit/s with underflow should be ok. I'm gonna send those over anyway, wish me good luck :P

[Mr.Rippley]

Hi! I decided to learn how to efficiently encode video using HEVC. What is needed for this? Some special software or coding settings should be set "very slowly". Now I use StaxRip 2.8. I would like to achieve the quality and size of the video as on the PSA website, there are just magicians(IMHO). Thank you in advance for your help!

[rwill]

Originally Posted by FranceBB

Makes sense.
The movie is 1h and 30 and it looks like it's near the end, which is pure black with static credits, so it's very likely.
The CBR 1 Gbit/s ProRes is almost definitely zero filled towards that point.
Nobody asked for CBR, so VBR 25 Mbit/s with underflow should be ok. I'm gonna send those over anyway, wish me good luck :P

Please check one of the following:
[ ] I know what a VBV underflow means.
[ ] I do not know what a VBV underflow means.

[FranceBB]

Originally Posted by rwill

Please check one of the following:
[ ] I know what a VBV underflow means.
[X] I do not know what a VBV underflow means.

Checked, but I thought it was implied by the fact that I asked you, guys, about it xD
I get that that any vbv warning means that there's something off with the bitrate, however I generally used to get vbv errors when I was using like a very low CRF and there's a very complex scene like rain or snow etc and the bitrate wanted to skyrocket and fly high but I was limiting it with some constraints to stay much lower than it should, however I feel like this time it's a completely different situation, so if you could clarify what the encoder is actually telling me other than the fact that there's something wrong and it can't abide my constraints, that would be very helpful!

[rwill]

Video Buffer Verifier Model: Two buckets with one being above the other and having a hole in it. The buckets
hold bits. The encoder encodes pictures and puts them in the upper bucket. From there they flow through the
hole at a constant rate ( the vbv maxrate ) to the lower bucket. The lower bucket is the video buffer. When the
decoder decodes a picture it takes bits from the lower bucket ( video buffer ), most of the time this happens
at a constant frame rate.

A video buffer overflow happens when a decoder is not taking enough bits from the lower bucket and it flows over.
Now this is not really problematic unless its caused by encoder and decoder clocks drifting apart. If encoder
and decoder share a common clock the encoder should always put pictures at a framerate in its upper bucket
so that they can be removed at the same framerate from the lower bucket. Padding the flow from upper to lower
bucket when the upper bucket is empty may prevent that nothing flows but today thats not really needed anymore,
it was needed in the past though. When replacing the encoder and upper bucket by some instant access storage
( like a file ) one would just stop reading and so stop filling the lower bucket once it is full.

Now a problem arises where the encoder produces pictures that are so large that they cannot make it fast enough
through the hole to the lower bucket and once they made it there they are quicky removed by the decoder at the
fixed rate. There might arise a situation where the decoder tries to fetch a picture from the lower bucket but
the bucket does not contain a complete picture as it still needs some refill from the upper bucket. The past
pictures were to large and got removed previously -> the video buffer is empty, underrun, underflow -> "Buffering".

Video encoders try to prevent such a situation ( mostly at all costs because violating the VBV model will get your
stream get kicked from broadcast multiplex for example ). x265 tries to increase the QP within a picture when it
detects that encoding takes more bits than anticipated while encoding a picutre. I guess the pictures in question
( the ones with the underflows ) show lower quality in the lower pictures parts, maybe even complete breakage.
It appears to be not enough to prevent the underflow though. I think x265 just does not see it coming. Encoding
at such low QPs is quite delicate.

I wonder what the content looks like that x265 is able to maintain an average QP of ~3.2 at that rate. Are you
sure you got your bitdepth and clipping right ? I see in your script that you use
"-pix_fmt yuv420p16le -f yuv4mpegpipe". I never used y4m but wouldnt this flag the y4m as 16 bit depth ?
x265 gets no explicit input bit depth set too. And your .AVS script is doing 10 bit with limited range clipping.
I am confused.

Have you decoded the encoder output ? Does it look ok ?

[FranceBB]

First of all thank you so much for the detailed explanation, it makes a lot of sense now!
Still, I get that x265 just doesn't see it coming, but I kinda thought it would given that it's a two pass encode and this is the second pass.

Originally Posted by rwill

I wonder what the content looks like that x265 is able to maintain an average QP of ~3.2 at that rate. Are you
sure you got your bitdepth and clipping right ?

Given that it's an old digital movie upscaled by the studio who sent it for the HDR UHD Remaster, I think there are no details whatsoever so it's just very easy to encode for x265.

Originally Posted by rwill

I see in your script that you use
"-pix_fmt yuv420p16le -f yuv4mpegpipe". I never used y4m but wouldnt this flag the y4m as 16 bit depth ?
x265 gets no explicit input bit depth set too.

Yeah I forgot to post the final AVS, I have updated it now.
And yes, I tend to deliver 16bit planar to x265 and let it dither down to 10bit planar and use x265 itself to perform clipping at 64, 940 Limited TV range (not that HLG ever reaches the upper part of the available 940 anyway 'cause the movie ain't 1000 nits in the first place, but I just play safe). About the complexity, I don't like grain, like, at all, it doesn't look good on anything unless you have a very high bitrate (which I never have), therefore I always always always get rid of it, either spatially or temporally and sometimes both. This also makes things much easier for x265, hence the low qp.

Originally Posted by rwill

Have you decoded the encoder output ? Does it look ok ?

It looks ok from the RDP given that all this is going on in a server, however I'll be able to tell once the encode is over and the file has been moved to a shared RAID6 instead of a puny local RAID1.

[rwill]

Originally Posted by FranceBB

It looks ok from the RDP given that all this is going on in a server, however I'll be able to tell once the encode is over and the file has been moved to a shared RAID6 instead of a puny local RAID1.

I would look at least around timecode 00:34:21, 01:05:02, 01:21:16 and 01:29:24 and check
for visual problems.

Maybe lowering the average rate to 20Mbit or doing more passes will help. Or
increase minQP. x265 does not give many options in such a case, I think its
being too enthusiastic about reaching the average rate of 25Mbit and due to
local failure in frame size prediction it produces underflows.

[Barough]

x265 v3.5+16-7955046e7 (32 & 64-bit 8/10/12bit Multilib Windows Binaries) (GCC 10.3.0)

https://bitbucket.org/multicoreware/x265_git/commits/branch/master

[Mr.Rippley]

Thanks!

[FranceBB]

Originally Posted by Mr.Rippley

Please, help me!

We don't have enough info. Do you want device playback compatibility or not? 'cause if you don't and you just want to compress, then it's much easier. You can try with something like:

--preset placebo --ref 16 --me full --subme 7 --crf 22

to start and then you can work from there.

[Mr.Rippley]

Originally Posted by FranceBB

We don't have enough info. Do you want device playback compatibility or not? 'cause if you don't and you just want to compress, then it's much easier. You can try with something like:

--preset placebo --ref 16 --me full --subme 7 --crf 22

to start and then you can work from there.

Thanks for the answer! I tried the placebo preset, coded it for several days) And what kind of encoder program do you use?Or do you consider it the best?

[FranceBB]

Originally Posted by Mr.Rippley

Thanks for the answer! I tried the placebo preset, coded it for several days) And what kind of encoder program do you use?Or do you consider it the best?

x265.exe via command line called through a BAT file.
If you're looking for builds, you'll find them here: http://msystem.waw.pl/x265/

[LeXXuz]

'Placebo' preset is a little bit over the top, isn't it?

If encoding time and energy consumption is of no concern, well go for it then.

But I think for private every day use the 'Slower' or 'Very Slow' presets are more than sufficient enough. I highly doubt most people would see any difference, especially on consumer displays.