Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 15th April 2018, 22:03   #6041  |  Link
Ma
Registered User
 
Join Date: Feb 2015
Posts: 296
Bisection is one of the most common method of finding bugs.
In source file common/x86/asm-primitives.cpp it is function pointers assignment with avx512 code -- from line # 4696 to 5385.
We can turn off (comment) first half of this functions (x265-1.exe) and then second half (x265-2.exe).
You can download avx512-bisect1.7z VS 2015 binaries with diff files -- if x265.exe (from clean sources) hangs, please try x265-1.exe and then x265-2.exe -- it should be that one of them hangs and second not.
Please report back the results (which from the 3 binaries hangs and which not).
Ma is offline   Reply With Quote
Old 15th April 2018, 22:38   #6042  |  Link
Xizer
Registered User
 
Join Date: Sep 2008
Posts: 4
Quote:
Originally Posted by Ma View Post
Bisection is one of the most common method of finding bugs.
In source file common/x86/asm-primitives.cpp it is function pointers assignment with avx512 code -- from line # 4696 to 5385.
We can turn off (comment) first half of this functions (x265-1.exe) and then second half (x265-2.exe).
You can download avx512-bisect1.7z VS 2015 binaries with diff files -- if x265.exe (from clean sources) hangs, please try x265-1.exe and then x265-2.exe -- it should be that one of them hangs and second not.
Please report back the results (which from the 3 binaries hangs and which not).
All three binaries crash with the avx512 flag I'm afraid.

Is there additional information that can be provided for debug?
Attached Files
File Type: txt DxDiag.txt (75.4 KB, 6 views)
Xizer is offline   Reply With Quote
Old 15th April 2018, 22:56   #6043  |  Link
Ma
Registered User
 
Join Date: Feb 2015
Posts: 296
Quote:
Originally Posted by Xizer View Post
All three binaries crash with the avx512 flag I'm afraid.

Is there additional information that can be provided for debug?
Thanks for the info.

Could you post results of the command
x265 -V && x265 --asm avx2 -V && x265 --asm avx512 -V
For example in my system it looks like
Code:
F:\x265p\ma\avx512>x265 -V && x265 --asm avx2 -V && x265 --asm avx512 -V
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
This bug could be not related directly to avx512 code -- could you check if it hangs if you use '--asm avx2' instead of '--asm avx512' (it is important to use --asm avx2 option).
------------------------------------------------------
Your OS is Win 8.1 that do not support avx512. This bug in x265 is not technical but conceptual -- avx512 is not auto recognized by default so option '--asm avx512' should not turn on avx512 without any check.
You can test file avx512-patch.7z with x265 that check what is supported by CPU & OS up to avx512 if option '--asm avx512' is used.
On Win 8.1 it should work with '--asm avx512' exactly like without this option, on Win 10 it should turn on avx512 if you have CPU with avx512 and you use option '--asm avx512'.

Last edited by Ma; 15th April 2018 at 23:58.
Ma is offline   Reply With Quote
Old 17th April 2018, 07:31   #6044  |  Link
Bhavnahari
Registered User
 
Join Date: Nov 2016
Posts: 4
Quote:
Originally Posted by jlpsvk View Post
any suggested quality wise settings recommendation for 4K HDR encoding? with CRF ie 17?
If the data that is being reused comes from encoding a downscaled video (scale-factor=2), it does not make a lot of sense to use intra/inter refinement levels 0/1 as levels 0 and 1 reuse most of the information from the previous pass with no or minimal re-evaluation of analysis information. This can have a huge impact on the quality of the encode.

We have observed that --refine-inter=3 and --refine-intra=4 gives the best quality, even better than standalone x265 encodes in some cases, with a performance gain of up to 1.8X.

For 4K HDR content, you will have to modify the display settings based on the monitor. Please refer the docs for more information - http://x265.readthedocs.io/en/defaul...mation-options
Bhavnahari is offline   Reply With Quote
Old 18th April 2018, 19:48   #6045  |  Link
jlpsvk
Registered User
 
Join Date: Dec 2014
Posts: 162
Quote:
Originally Posted by Bhavnahari View Post
If the data that is being reused comes from encoding a downscaled video (scale-factor=2),
it does not make a lot of sense to use intra/inter refinement levels 0/1 as levels 0 and 1 reuse most of the information from
the previous pass with no or minimal re-evaluation of analysis information. This can have a huge impact on the quality of the encode.

We have observed that --refine-inter=3 and --refine-intra=4 gives the best quality, even better than standalone x265 encodes in some cases,
with a performance gain of up to 1.8X.
My settings then...

Code:
--crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock
--no-sao --no-strong-intra-smoothing --high-tier --refine-inter 3 --refine-intra 4
Of course, display settings are entered too...

Warning:
Code:
x265 [warning]: Intra refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling intra refine.
x265 [warning]: Inter refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling inter refine.
Can it be used with CRF? Or? Some parameters missing?

Last edited by jlpsvk; 18th April 2018 at 20:10.
jlpsvk is offline   Reply With Quote
Old 18th April 2018, 21:47   #6046  |  Link
Loomes
Registered User
 
Join Date: Nov 2003
Location: Germany, Berlin
Posts: 38
So far no success for me running the avx512 option on my 7820x on Windows 10 64it. I tried
Code:
ffmpeg -i source.mkv -f yuv4mpegpipe - | x265.exe --preset slow --asm avx512 --crf 21 --y4m - -o dest.h265
which crashes immediately. Using --asm avx2 works fine.

I ran
Code:
x265.exe -V && x265.exe --asm avx2 -V && x265.exe --asm avx512 -V
and it showed
Code:
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
Quote:
Originally Posted by Ma View Post
You can test file avx512-patch.7z with x265 that check what is supported by CPU & OS up to avx512 if option '--asm avx512' is used.
I tried from that package x265.exe, x265-1.exe and x265-2.exe. All of them crashed immediately. I am using the latest nightly version of ffmpeg.
Am I missing something?

Last edited by Loomes; 18th April 2018 at 22:16.
Loomes is offline   Reply With Quote
Old 18th April 2018, 22:45   #6047  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 3,200
Quote:
Originally Posted by Loomes View Post
I tried from that package x265.exe, x265-1.exe and x265-2.exe. All of them crashed immediately. I am using the latest nightly version of ffmpeg.
Am I missing something?
I had crashes using the builds foxyshadis posted but I was successful using LigH's build from this post. Win10 and an i9-7900X. Have you tried them?
__________________
madVR options explained
Asmodian is offline   Reply With Quote
Old 18th April 2018, 22:57   #6048  |  Link
Loomes
Registered User
 
Join Date: Nov 2003
Location: Germany, Berlin
Posts: 38
Quote:
Originally Posted by Asmodian View Post
I had crashes using the builds foxyshadis posted but I was successful using LigH's build from this post. Win10 and an i9-7900X. Have you tried them?
I did now but it also results in an instant crash. Are you using a ffmpeg pipe like me or what is your command line?
Loomes is offline   Reply With Quote
Old 18th April 2018, 23:22   #6049  |  Link
Ma
Registered User
 
Join Date: Feb 2015
Posts: 296
Quote:
Originally Posted by Loomes View Post
I tried from that package x265.exe, x265-1.exe and x265-2.exe. All of them crashed immediately. I am using the latest nightly version of ffmpeg.
Am I missing something?
x265-1.exe == x265-2.exe == x265.exe from package avx512-bisect1.7z (I commented out code for 8-bit encoding and compiled 10-bit version, sorry for that). So this package is to forget.

I want to precise: x265.exe from package avx512-patch.7z gives in your OS output:
x265 --asm avx512 -V
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
?
Ma is offline   Reply With Quote
Old 19th April 2018, 01:18   #6050  |  Link
Loomes
Registered User
 
Join Date: Nov 2003
Location: Germany, Berlin
Posts: 38
Quote:
Originally Posted by Ma View Post
I want to precise: x265.exe from package avx512-patch.7z gives in your OS output:
x265 --asm avx512 -V
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
?
Confirmed. I tried again on Windows 10 64bit with an 7820x using the command line shown above.

Last edited by Loomes; 19th April 2018 at 01:22.
Loomes is offline   Reply With Quote
Old 19th April 2018, 06:31   #6051  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 3,200
Quote:
Originally Posted by Loomes View Post
I did now but it also results in an instant crash. Are you using a ffmpeg pipe like me or what is your command line?
No, actually I was using avs2pipemod.exe (ver 1.1.1 Aug 15, 2016) piping to x265.exe with y4m:
C:\Tools\avs2pipemod.exe -y4mp=1:1 %1 | C:\Tools\x265.exe --input - --y4m -o "D:\Temp\%~n1.mkv" --asm avx512 --preset veryslow --crf 18.5 --output-depth 10

edit:
"x265 --asm avx512 -V" gives:
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][GCC 7.3.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
__________________
madVR options explained

Last edited by Asmodian; 19th April 2018 at 06:34.
Asmodian is offline   Reply With Quote
Old 19th April 2018, 11:38   #6052  |  Link
Ma
Registered User
 
Join Date: Feb 2015
Posts: 296
Quote:
Originally Posted by Loomes View Post
Confirmed. I tried again on Windows 10 64bit with an 7820x using the command line shown above.
Thanks for confirmation -- it looks like your CPU & OS supporting avx512.

I've prepared some binaries to narrow this bug -- avx512-bisect2.7z.
x265-0.exe -- 10-bit encoder with commented out all avx512 function pointers (it works on my i7 8700 with --asm avx512 option). It is for check if the bug is before avx512 code is running but the execution path is exactly like in avx512 case.

x265-01.exe -- assigned only first 1/4 function pointers with avx512 code.
x265-02.exe -- assigned only second 1/4 function pointers with avx512 code.
x265-03.exe -- assigned only third 1/4 function pointers with avx512 code.
x265-04.exe -- assigned only last 1/4 function pointers with avx512 code.

Could you test these 5 exe's and report back which works and which not?
Ma is offline   Reply With Quote
Old 19th April 2018, 13:39   #6053  |  Link
Loomes
Registered User
 
Join Date: Nov 2003
Location: Germany, Berlin
Posts: 38
Quote:
Originally Posted by Ma View Post
Could you test these 5 exe's and report back which works and which not?
I'm home in about 6 hours and will do. By the way: Is it correct that --asm avx512 will not work on Windows 7/8 64bit, no matter what?
Loomes is offline   Reply With Quote
Old 19th April 2018, 13:56   #6054  |  Link
nevcairiel
Registered Developer
 
Join Date: Mar 2010
Location: Hamburg/Germany
Posts: 9,312
Quote:
Originally Posted by Loomes View Post
Is it correct that --asm avx512 will not work on Windows 7/8 64bit, no matter what?
That would appear to be the case. If you force avx512 usage through --asm avx512 on an other OS it'll just crash.

Thats really the problem with using the --asm option, it overrides the CPU/OS feature detection, a different option to enable it when present would be better.
__________________
LAV Filters - open source ffmpeg based media splitter and decoders
nevcairiel is online now   Reply With Quote
Old 19th April 2018, 22:42   #6055  |  Link
Loomes
Registered User
 
Join Date: Nov 2003
Location: Germany, Berlin
Posts: 38
Quote:
Originally Posted by Ma View Post
Could you test these 5 exe's and report back which works and which not?
I am afraid, none of the x265-xx.exe worked in my Windows 10 and crashed instantly. But I found something else: I tried to use directly the libx265 of ffmpeg by
Code:
ffmpeg-2018-04-18.exe -i source.mkv -c:v libx265 -x265-params crf=21 -preset slow -pix_fmt yuv420p10le -bsf hevc_mp4toannexb -f rawvideo target.h265
and it also crashed instantly. Then I changed this new ffmpeg-2018-04-18.exe for an older one ffmpeg-2018-03-24.exe and it worked. Then I ran again your x265-xx.exe files with the older ffmpeg-2018-03-24.exe but they all crashed again with
Code:
ffmpeg-2018-03-24.exe -i source.mkv -f yuv4mpegpipe - | x265-0.exe --preset slow --crf 21 --y4m - -o target.h265
Even more strange that ffmpeg-2018-04-18.exe works flawlessly on Windows 7 and so the x265-0.exe of your files does (it even claims to make use of AVX512 when it starts). All the other x265-0x.exe crashed immediately on Windows 7.

This is really weird. Let me know if I can provide you any more info of my system or do some more tests.

Last edited by Loomes; 19th April 2018 at 23:23.
Loomes is offline   Reply With Quote
Old 20th April 2018, 00:06   #6056  |  Link
Ma
Registered User
 
Join Date: Feb 2015
Posts: 296
Thanks for the details. It is quite a riddle.
On Win7 everything is OK, x265-0 should work (all avx512 code is not active), x265-0x should crash.
On Win10 it is interesting -- in your 2 examples it is no '--asm avx512' option. It crashes without avx512 too?
Is something special with your Win10, for example it works in VirtualBox?
Ma is offline   Reply With Quote
Old 20th April 2018, 00:15   #6057  |  Link
Loomes
Registered User
 
Join Date: Nov 2003
Location: Germany, Berlin
Posts: 38
Quote:
Originally Posted by Ma View Post
On Win10 it is interesting -- in your 2 examples it is no '--asm avx512' option. It crashes without avx512 too?
Yes, I had --asm avx512 included in the first run and all crashed. Then I had the idea that it's about ffmpeg and let it run without the asm option but well, did not work either.

Quote:
Originally Posted by Ma View Post
Is something special with your Win10, for example it works in VirtualBox?
No, not at all. It is Windows 10 Professional 64bit, nothing special about it as far as I know. The system drive has 14GB free space, the drive where x265, ffmpeg etc. is running with 80G free space. As I said, it works fine with the older ffmpeg version or older x265.exe.
Is it possible that there is something special about the 7820x CPU?
Loomes is offline   Reply With Quote
Old 20th April 2018, 08:41   #6058  |  Link
Ma
Registered User
 
Join Date: Feb 2015
Posts: 296
Quote:
Originally Posted by Loomes View Post
Is it possible that there is something special about the 7820x CPU?
It is possible that there are hardware problems -- easiest way to check this out is Prime95 https://www.mersenne.org/download/
If you run this app with button 'Just Stress Testing' it should find memory/CPU problems (in file 'results.txt' should be only [date] and XXX passed! entries).

There was some changes in ffmpeg, so you can test ffmpeg release 3.4.2 -- it should be different approach to thread synchronization.

There was some reports about x265 hangs with pipe input from ffmpeg but it is hard to reproduce -- maybe there is bug in x265 that is not fixed yet.
Ma is offline   Reply With Quote
Old 20th April 2018, 10:17   #6059  |  Link
Bhavnahari
Registered User
 
Join Date: Nov 2016
Posts: 4
Quote:
Originally Posted by jlpsvk View Post
My settings then...

Code:
--crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock
--no-sao --no-strong-intra-smoothing --high-tier --refine-inter 3 --refine-intra 4
Of course, display settings are entered too...

Warning:
Code:
x265 [warning]: Intra refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling intra refine.
x265 [warning]: Inter refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling inter refine.
Can it be used with CRF? Or? Some parameters missing?

refine-inter and refine-intra features may be used with any rate control techniques including ABR, CRF and CQP. However, as indicated by the warnings, these features are enabled only for scaled save/load encodes. The "save" run will encode the content of resolution WxH (1920x1080, say) and generate an analysis.dat file. This analysis information is scaled up by the factor and used by the "load" run which works on the content of resolution 2Wx2H (3840x2160, say). Using the analysis information from a low-res encode to encode high-res content will impact the quality of the encode hence we try to minimize the repercussions by incorporating various intra and inter refinement strategies.

Your CLI should be:
HD encode : --crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-save <filename> --analysis-reuse-level 10 --scale-factor 2

UHD encode: -crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-load <filename> --analysis-reuse-level 10 --scale-factor 2 --refine-inter 3 --refine-intra 4
Bhavnahari is offline   Reply With Quote
Old 20th April 2018, 21:26   #6060  |  Link
jlpsvk
Registered User
 
Join Date: Dec 2014
Posts: 162
Quote:
Originally Posted by Bhavnahari View Post
refine-inter and refine-intra features may be used with any rate control techniques including ABR, CRF and CQP. However, as indicated by the warnings, these features are enabled only for scaled save/load encodes. The "save" run will encode the content of resolution WxH (1920x1080, say) and generate an analysis.dat file. This analysis information is scaled up by the factor and used by the "load" run which works on the content of resolution 2Wx2H (3840x2160, say). Using the analysis information from a low-res encode to encode high-res content will impact the quality of the encode hence we try to minimize the repercussions by incorporating various intra and inter refinement strategies.

Your CLI should be:
HD encode : --crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-save <filename> --analysis-reuse-level 10 --scale-factor 2

UHD encode: -crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-load <filename> --analysis-reuse-level 10 --scale-factor 2 --refine-inter 3 --refine-intra 4
Thanks!!! So it won't downscale and upscale video? Only stats? Right? So it's better for retain quality/details or not?

Last edited by jlpsvk; 20th April 2018 at 21:47.
jlpsvk is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 15:48.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.