Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th April 2018, 19:19   #6081  |  Link
jlpsvk
Registered User
 
Join Date: Dec 2014
Posts: 240
Quote:
Originally Posted by Selur View Post
wild guess: still got free space on c: ? (iirc analysis files etc can get really large)
yeah... 600GB free on C: SSD. It's happening in the second phase, READ, when the stats are reading... First, 1080p encode, with writing stats is OK.

Stats file 27GB big.
jlpsvk is offline   Reply With Quote
Old 26th April 2018, 19:25   #6082  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,752
Well, it's not completely impossible that encoding in AVX-512 mode might create a different output. But probably hard to confirm...
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 26th April 2018, 19:35   #6083  |  Link
jlpsvk
Registered User
 
Join Date: Dec 2014
Posts: 240
Quote:
Originally Posted by LigH View Post
Well, it's not completely impossible that encoding in AVX-512 mode might create a different output. But probably hard to confirm...
first phase was created using AVX-512 too... tried READ phase without it, result is the same. Probably will try to use first without also.
jlpsvk is offline   Reply With Quote
Old 27th April 2018, 00:35   #6084  |  Link
divxmaster
Registered User
 
Join Date: Mar 2015
Location: New Zealand
Posts: 45
As a follow up to my post a few months ago re spectre/meltdown patching and its influence on x265 encoding fps,
(where I found on my skylake quad it made virtually no difference),
I have now test my haswell system (4770k) with the microsoft microcode updates release two days ago (KB4091663 for 1703 I am running).

The latest KB patches the microcode for spectre variant two (branch target injection), at boot time only. GRC's Inspectre shows my system is now fully patched.
This latest KB DOES affect throughput fps. I have seen a drop of about 2.5%. Not a big deal luckily. I will just adjust my conservative OC
from 4.1ghz to 4.2ghz. (which is +2.5%).

Cheers,
Divxmaster
divxmaster is offline   Reply With Quote
Old 27th April 2018, 06:31   #6085  |  Link
jlpsvk
Registered User
 
Join Date: Dec 2014
Posts: 240
Quote:
Originally Posted by jlpsvk View Post
first phase was created using AVX-512 too... tried READ phase without it, result is the same. Probably will try to use first without also.
So... stats write encode (I am not writing here mastering display, options), spline36 downsized from 3840x1600 to 1920x800:
--crf 17 --keyint 240 --min-keyint 24 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star --max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-save c:\stats.dat --analysis-reuse-level 10 --scale-factor 2 --no-cutree

Stats read encode at full 3840x1600:
--crf 17 --keyint 240 --min-keyint 24 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star --max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-load c:\stats.dat --analysis-reuse-level 10 --scale-factor 2 --refine-inter 3 --refine-intra 4 --no-cutree

With or withou AVX-512, result is the same.
jlpsvk is offline   Reply With Quote
Old 3rd May 2018, 07:02   #6086  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,752
x265 2.7+346-69aafa6d70ad (stable merge)

AVX-512 cleanups and a few nits (getting a RC soon?)
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid

Last edited by LigH; 3rd May 2018 at 19:20.
LigH is offline   Reply With Quote
Old 4th May 2018, 05:06   #6087  |  Link
pradeeprama
Registered User
 
Join Date: Sep 2015
Posts: 48
Quote:
Originally Posted by divxmaster View Post
As a follow up to my post a few months ago re spectre/meltdown patching and its influence on x265 encoding fps,
(where I found on my skylake quad it made virtually no difference),
I have now test my haswell system (4770k) with the microsoft microcode updates release two days ago (KB4091663 for 1703 I am running).

The latest KB patches the microcode for spectre variant two (branch target injection), at boot time only. GRC's Inspectre shows my system is now fully patched.
This latest KB DOES affect throughput fps. I have seen a drop of about 2.5%. Not a big deal luckily. I will just adjust my conservative OC
from 4.1ghz to 4.2ghz. (which is +2.5%).

Cheers,
Divxmaster
Thanks for that. I typically classify any performance difference under 5% (in either direction) as noise when measured on a real system... so I suppose it is fair to say that Specrte and Meltdown don't affect us. This is not surprising as we barely make syscalls that require escalated privileges, which is where these bugs kick-in and safe guard!
pradeeprama is offline   Reply With Quote
Old 4th May 2018, 05:07   #6088  |  Link
pradeeprama
Registered User
 
Join Date: Sep 2015
Posts: 48
Quote:
Originally Posted by jlpsvk View Post
So... stats write encode (I am not writing here mastering display, options), spline36 downsized from 3840x1600 to 1920x800:
--crf 17 --keyint 240 --min-keyint 24 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star --max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-save c:\stats.dat --analysis-reuse-level 10 --scale-factor 2 --no-cutree

Stats read encode at full 3840x1600:
--crf 17 --keyint 240 --min-keyint 24 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star --max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-load c:\stats.dat --analysis-reuse-level 10 --scale-factor 2 --refine-inter 3 --refine-intra 4 --no-cutree

With or withou AVX-512, result is the same.
What CPU are you using?
pradeeprama is offline   Reply With Quote
Old 6th May 2018, 10:44   #6089  |  Link
jlpsvk
Registered User
 
Join Date: Dec 2014
Posts: 240
Quote:
Originally Posted by pradeeprama View Post
What CPU are you using?
Core i9-7940X
jlpsvk is offline   Reply With Quote
Old 9th May 2018, 15:40   #6090  |  Link
Barough
Registered User
 
Barough's Avatar
 
Join Date: Feb 2007
Location: Sweden
Posts: 480
x265 v2.7+348-0968a46d6ba4 (GCC 7.3.0, 32 & 64-bit 8/10/12bit Multilib Windows Binaries)

Code:
https://bitbucket.org/multicoreware/x265/commits/all
Barough is offline   Reply With Quote
Old 9th May 2018, 19:02   #6091  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,752
x265 2.7+348-0968a46d6ba4

fixes a build error on Mac and a calculation bug in scalefactor 0
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 11th May 2018, 23:48   #6092  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
Does anyone see where my mistake is?

I'm trying to convert a 8bit pc range clip to 10bit x265 using:
Code:
ffmpeg -y -threads 8 -i "C:\Users\Selur\Desktop\Color Range Test.MOV" -map 0:0 -an -sn -vsync 0 -strict -1 -pix_fmt yuv420p10le -f yuv4mpegpipe - | x265 --input - --output-depth 10 --y4m --profile main10 --no-open-gop --crf 18.00 --range full --colormatrix bt470bg --output "H:\Temp\x265.265"
Problem is the output histogram is compressed, see: https://s14.postimg.cc/pq865qnfj/compare.png
Uploaded the source to my google drive as Color Range Test.MOV.

for the comparision I used:
Code:
# Imports
import vapoursynth as vs
core = vs.get_core()
# Loading Plugins
core.std.LoadPlugin(path="G:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")


# Loading H:\Output\Color Range Test.MOV using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="H:/Output/Color Range Test.MOV")
# making sure input color matrix is set as 470bg
clip = core.resize.Point(clip, matrix_in_s="470bg")
# making sure frame rate is set to 30000/1001
clip = core.std.AssumeFPS(clip, fpsnum=30000, fpsden=1001)
# Making sure input color range is set to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
clip = core.text.Text(clip=clip, text="Original")
clip = core.hist.Levels(clip=clip)

clip2 = core.lsmas.LWLibavSource(source="H:/Temp/x265.265", format="YUV420P10", cache=0)
# making sure input color matrix is set as 470bg
clip2 = core.resize.Point(clip2, matrix_in_s="470bg")
# making sure frame rate is set to 30000/1001
clip2 = core.std.AssumeFPS(clip2, fpsnum=30000, fpsden=1001)
# Making sure input color range is set to PC (full) range.
clip2 = core.std.SetFrameProp(clip=clip2, prop="_ColorRange", intval=0)
clip2 = core.text.Text(clip=clip2, text="x265")
# adjusting output color from: YUV420P8 to YUV420P10
clip2 = core.resize.Bicubic(clip=clip2, format=vs.YUV420P8)
clip2 = core.hist.Levels(clip=clip2)


clip = core.std.StackHorizontal([clip, clip2])
# Output
clip.set_output()
My guess is that I need to change the ffmpeg call, but I don't see where.

Cu Selur

Ps.: Adding '-color_range 2' to the ffmpeg call doesn't help either.
__________________
Hybrid here in the forum, homepage

Last edited by Selur; 12th May 2018 at 00:18.
Selur is offline   Reply With Quote
Old 12th May 2018, 10:24   #6093  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
As you say this has nothing to do with x265.

You could try to "trick" ffmpeg by pretending the input was limited range.
-vf zscale=rangein=limited:range=limited -pix_fmt yuv420p10le

Maybe someone with more ffmpeg knowledge knows a "proper" solution ...

Last edited by sneaker_ger; 12th May 2018 at 10:27.
sneaker_ger is offline   Reply With Quote
Old 12th May 2018, 11:20   #6094  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
Quote:
As you say this has nothing to do with x265.
That is what I guessed, not sure whether ffmpeg does something wrong or x265 has some problem with y4m output or ffmpeg.

Quote:
You could try to "trick" ffmpeg by pretending the input was limited range.
-vf zscale=rangein=limited:range=limited -pix_fmt yuv420p10le
That still changes the luma histogram. (https://s31.postimg.cc/tnpvy7cq3/compare2.png)
using:
Code:
-vf zscale=rangein=full:range=full
is better, but there are still some changes. (https://s31.postimg.cc/9hmdzaehn/compare3.png)

Quote:
Maybe someone with more ffmpeg knowledge knows a "proper" solution ...
I hope so.

Cu Selur
__________________
Hybrid here in the forum, homepage

Last edited by Selur; 12th May 2018 at 11:23.
Selur is offline   Reply With Quote
Old 12th May 2018, 17:18   #6095  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
maybe there is a bug with full range ?

waveform/histograms only match if you use lossless (even compared to low crf like --crf 8, where you 'd only expect tiny differences) , regardless if you use vapoursynth input or ffmpeg scaling with rawvideo pipe

crf8 - waveform is compressed
Code:
ffmpeg -i "Color Range Test.MOV" -vf scale=iw:ih:in_range=pc:out_range=pc,format=yuv420p10le -f rawvideo - | x265_10bit.exe --fps 30000/1001 --input-res 1280x720 --input-depth 10 --output-depth 10 --input-csp i420 --crf 8.00 --range full --colormatrix smpte170m -o rawpipe_x265crf8.265 -
--lossless works with ffmpeg rawvideo pipe
Code:
ffmpeg -i "Color Range Test.MOV" -vf scale=iw:ih:in_range=pc:out_range=pc,format=yuv420p10le -f rawvideo - | x265_10bit.exe --fps 30000/1001 --input-res 1280x720 --input-depth 10 --output-depth 10 --input-csp i420 --lossless --range full --colormatrix smpte170m -o rawpipe_x265lossless.265 -

vspipe y4m --crf 8 waveform is compressed
Code:
vspipe --y4m 1.vpy - | x265_10bit.exe --y4m --crf 8.00 --colormatrix smpte170m --range full -o vs_x265.265 -

vspipe y4m --lossless waveform is ok
Code:
vspipe --y4m 1.vpy - | x265_10bit.exe --y4m --lossless --colormatrix smpte170m --range full -o vs_x265_lossless.265 -
vpy input
Code:
clip = core.lsmas.LibavSMASHSource(r'F:\Color Range Test.MOV')
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10)
clip.set_output()
poisondeathray is offline   Reply With Quote
Old 12th May 2018, 20:28   #6096  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
Quote:
waveform/histograms only match if you use lossless (even compared to low crf like --crf 8, where you 'd only expect tiny differences) , regardless if you use vapoursynth input or ffmpeg scaling with rawvideo pipe
I understand some small changes like when using '-vf zscale=rangein=full:range=full' and my lines from above might be a side effect of the video compression, but the compression of the histogram really looks like either a bug in x265 or all the other tools.

Cu Selur
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 12th May 2018, 23:03   #6097  |  Link
Ma
Registered User
 
Join Date: Feb 2015
Posts: 326
Quote:
Originally Posted by Selur View Post
I'm trying to convert a 8bit pc range clip to 10bit x265 using:
Code:
ffmpeg [...] -f yuv4mpegpipe - | x265 [...]
yuv4mpegpipe is for limited range only. Please use raw video for full range.
Ma is offline   Reply With Quote
Old 13th May 2018, 00:06   #6098  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by Selur View Post
I understand some small changes like when using '-vf zscale=rangein=full:range=full' and my lines from above might be a side effect of the video compression, but the compression of the histogram really looks like either a bug in x265 or all the other tools.
I agree . That's why I used crf 8, you'd expect only tiny tiny difference, not that large. It looks like CbCr are affected too

It affects x264 too . vs rawpipe too instead of y4m for fun
Code:
vspipe 1.vpy - | x264_10bit --demuxer raw --input-csp i420 --input-depth 10 --input-res 1280x720 --fps 30000/1001 --crf 8.00 --colormatrix smpte170m --input-range pc --range pc -o vs_raw_x264.264 -
But if it was a toolchain problem, you'd expect lossless to exhibit the same problem

Something is up...
poisondeathray is offline   Reply With Quote
Old 13th May 2018, 02:20   #6099  |  Link
jlpsvk
Registered User
 
Join Date: Dec 2014
Posts: 240
ok. x265 devs. what is the command line to output uhd bd compliant stream?
jlpsvk is offline   Reply With Quote
Old 13th May 2018, 08:44   #6100  |  Link
Jamaika
Registered User
 
Join Date: Jul 2015
Posts: 697
Quote:
Originally Posted by Ma View Post
yuv4mpegpipe is for limited range only. Please use raw video for full range.
Never heard about it. A few threads further users advise against the use of rawvideo.

I test conveters:
Code:
ffmpeg_080518.exe -i "Color Range Test.MOV" -f yuv4mpegpipe -vf scale=1280:720:in_color_matrix=470bg:in_range=full:out_color_matrix=470bg:out_range=full,format=yuv420p10,lutyuv=val:val:val -strict -1 - | 
x264-10bit.exe --demuxer y4m --muxer mp4 --input-csp i420 --input-depth 10 --input-range pc --output-csp i420 --threads 4 --preset veryslow --tune grain --crf 28 --fps 29.970 --keyint 60 --nal-hrd none 
--colormatrix bt470bg --colorprim bt709 --transfer bt709 --range pc 
--output "x264_420p10le_crf28_1.mp4" -
Code:
x264-10bit_2851.exe --muxer mp4 --input-csp i420 --input-depth 8 --input-range pc --output-csp i420 --threads 4 --preset veryslow --tune grain --crf 28 --fps 29.970 --keyint 60 --nal-hrd none 
--colormatrix bt470bg --colorprim bt709 --transfer bt709 --range pc 
--output "x264_420p10le_crf28_2.mp4" "Color Range Test.MOV"
Code:
bpgenc_0.9.8.exe -v -b 10 -f 420 -c ycbcr_bt709 -e x265 -m 9 -q 28 -o x265_420p10le_crf28.bpg rgba.png
Result of the test:
1)bad range,
2)bad range, no latest version 10bit 2901 (2018),
3)good.

Conclusion:
For me, the problem is in BT470bg.

Last edited by Jamaika; 13th May 2018 at 09:28.
Jamaika is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 10:07.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.