x265 HEVC Encoder

[littlepox]

Time to find proper settings for 1080p high quality archival then.


play around with :

--preset slower --ctu 32 --max-tu-size 16 --crf 19 --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --me 3 --subme 5 --merange 44 --no-rect --b-intra --no-amp --ref 5 --weightb --keyint 360 --min-keyint 1 --bframes 8 --aq-mode 1 --aq-strength 1.0 --rd 5 --psy-rd 1.5 --psy-rdoq 5.0 --rdoq-level 1 --no-sao --no-open-gop --rc-lookahead 80 --max-merge 4 --qcomp 0.8 --no-strong-intra-smoothing --deblock -2:-2 --qg-size 16 --pbratio 1.2

You can manually set crf to be within [18,20] for your preference. Too large or too small and this combination is not optimal anymore.



Do you feel it would be possible at this point to find settings for x265 which would result in an encoded 1080p/720p contents weighting 25% less than it's x264 counterpart or am I being delusional here?

2.5% is doable, our test suggests that on average x265 can save about ~10% of bitrate.

wait, 25%? don't even daydreaming:(...



Edit; Also encoding a shorter movie (1h23m) with the latest settings you gave me 3 posts ago, it seems to encode much faster, ~6 fps versus ~2.4 fps

I turned off --rect since you complained about the speed. --rect is a parameter which takes a lot of computation but the gain is negligible. It has little impact on quality or rate-control.

[pingfr]

play around with :

--preset slower --ctu 32 --max-tu-size 16 --crf 19 --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --me 3 --subme 5 --merange 44 --no-rect --b-intra --no-amp --ref 5 --weightb --keyint 360 --min-keyint 1 --bframes 8 --aq-mode 1 --aq-strength 1.0 --rd 5 --psy-rd 1.5 --psy-rdoq 5.0 --rdoq-level 1 --no-sao --no-open-gop --rc-lookahead 80 --max-merge 4 --qcomp 0.8 --no-strong-intra-smoothing --deblock -2:-2 --qg-size 16 --pbratio 1.2

You can manually set crf to be within [18,20] for your preference. Too large or too small and this combination is not optimal anymore.

Already running an encode with these settings as we speak:

--preset slower --ctu 32 --max-tu-size 16 --crf 18 --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --me 3 --subme 5 --merange 44 --b-intra --no-rect --no-amp --ref 5 --weightb --keyint 360 --min-keyint 1 --bframes 8 --aq-mode 1 --aq-strength 1.0 --rd 5 --psy-rd 1.5 --psy-rdoq 5.0 --rdoq-level 1 --no-sao --no-open-gop --rc-lookahead 80 --max-merge 4 --qcomp 0.8 --no-strong-intra-smoothing --deblock -2:-2 --qg-size 16 --pbratio 1.2

ETA used to be 6 hours'ish but now bumped up to 9 hours or so.

I'm exhausted and got a lot of work to do tomorrow, I will talk to you laters. You have no idea how grateful I am for your help.

Thank you, thank you, thank you.

[LigH]

One additional comment (I hope I did not miss it being mentioned already in the verbose discussion above):

If you have "perfect quality" original content (e.g. Blender render movies as PNG or even 16-bit-per-channel TIFF sequences), then you can compare the result one encoder created from the original content with the result another encoder created from the same original content. With some relaxed point of view, material encoded with a very generous bitrate and quantization fine enough to "guarantee visual transparency" may be considered as "quasi original"; Blu-ray content should match this criterion if the production studio did not mess it with nerdy blindness.

But if your so-called "original" content is not really original, but already compressed with a lossy format, then it doesn't make much sense to compare a re-encode of this already lossy content passing another lossy encode. This will introduce a cascading issue: The next encoder will have to spend more bitrate to resemble the already present encoding artifacts produced by the previous encoder due to its rate control limits. Thus, no big surprise that the re-encoded result can become larger and still look worse.

[pingfr]

@littlepox: The results are here.

Source first:

General
Unique ID : 40485131295552032955658755792816885178 (0x1E75271526187B453EC961B7C6A0CDBA)
Complete name : D:\source.mkv
Format : Matroska
Format version : Version 1
File size : 15.3 GiB
Duration : 1h 23mn
Overall bit rate : 26.1 Mbps
Encoded date : UTC 2016-03-11 02:00:12
Writing application : eac3to
Writing library : Haali DirectShow Matroska Muxer 1.13.138.14

Video
ID : 1
Format : VC-1
Format profile : Advanced@L3
Codec ID : V_MS/VFW/FOURCC / WVC1
Codec ID/Hint : Microsoft
Duration : 1h 23mn
Bit rate : 25.6 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 24.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.514
Stream size : 15.0 GiB (98%)
Default : No
Forced : No

Encoding parameters were:

@echo off
avs4x265.exe -P x265.exe --preset slower --ctu 32 --max-tu-size 16 --crf 18 --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --me 3 --subme 5 --merange 44 --b-intra --no-rect --no-amp --ref 5 --weightb --keyint 360 --min-keyint 1 --bframes 8 --aq-mode 1 --aq-strength 1.0 --rd 5 --psy-rd 1.5 --psy-rdoq 5.0 --rdoq-level 1 --no-sao --no-open-gop --rc-lookahead 80 --max-merge 4 --qcomp 0.8 --no-strong-intra-smoothing --deblock -2:-2 --qg-size 16 --pbratio 1.2 --output out.hevc %1
pause

The x265.exe output log:

avs [info]: AviSynth 2.60 (ICL10)
avs [info]: Video colorspace: YV12
avs [info]: Video resolution: 1920x800
avs [info]: Video framerate: 24/1
avs [info]: Video framecount: 120888
avs4x265 [info]: "x265.exe" - --frames 120888 --fps 24/1 --input-res 1920x800 --input-csp i420 --preset slower --ctu 32 --max-tu-size 16 --crf 18 --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --me 3 --subme 5 --merange 44 --b-intra --no-rect --no-amp --ref 5 --weightb --keyint 360 --min-keyint 1 --bframes 8 --aq-mode 1 --aq-strength 1.0 --rd 5 --psy-rd 1.5 --psy-rdoq 5.0 --rdoq-level 1 --no-sao --no-open-gop --rc-lookahead 80 --max-merge 4 --qcomp 0.8 --no-strong-intra-smoothing --deblock -2:-2 --qg-size 16 --pbratio 1.2 --output out.hevc
yuv [info]: 1920x800 fps 24/1 i420p8 unknown frame count
raw [info]: output file: out.hevc
x265 [info]: HEVC encoder version 1.9+73-6d06de58c316
x265 [info]: build info [Windows][GCC 5.3.0][64 bit] 8bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: frame threads / pool features : 5 / wpp(25 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 8
x265 [info]: Residual QT: max TU size, max depth : 16 / 2 inter / 2 intra
x265 [info]: ME / range / subpel / merge : star / 44 / 5 / 4
x265 [info]: Keyframe min / max / scenecut : 1 / 360 / 40
x265 [info]: Intra 32x32 TU penalty type : 2
x265 [info]: Lookahead / bframes / badapt : 80 / 8 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 5 / 1 / 0
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 1.0 / 16 / 1
x265 [info]: Rate Control / qCompress : CRF-18.0 / 0.80
x265 [info]: tools: limit-modes rd=5 psy-rd=1.50 rdoq=1 psy-rdoq=5.00 signhide
x265 [info]: tools: tmvp b-intra lslices=4 deblock(tC=-2:B=-2)

x265 [info]: frame I: 1037, Avg QP:14.18 kb/s: 28314.02
x265 [info]: frame P: 24299, Avg QP:16.47 kb/s: 15315.20
x265 [info]: frame B: 95552, Avg QP:18.73 kb/s: 7015.12
x265 [info]: Weighted P-Frames: Y:2.6% UV:1.4%
x265 [info]: Weighted B-Frames: Y:1.8% UV:1.2%
x265 [info]: consecutive B-frames: 13.7% 7.0% 8.6% 15.7% 8.9% 25.7% 7.7% 6.1% 6.7%

encoded 120888 frames in 31095.92s (3.89 fps), 8866.18 kb/s, Avg QP:18.23
Press a key to continue...

Results are:

General
Format : HEVC
Format/Info : High Efficiency Video Coding
File size : 5.20 GiB
Writing library : x265 1.9+73-6d06de58c316:[Windows][GCC 5.3.0][64 bit] 8bit
Encoding settings : wpp / ctu=32 / min-cu-size=8 / max-tu-size=16 / tu-intra-depth=2 / tu-inter-depth=2 / me=3 / subme=5 / merange=44 / no-rect / no-amp / max-merge=4 / temporal-mvp / no-early-skip / rdpenalty=2 / no-tskip / no-tskip-fast / no-strong-intra-smoothing / no-lossless / no-cu-lossless / no-constrained-intra / no-fast-intra / no-open-gop / no-temporal-layers / interlace=0 / keyint=360 / min-keyint=1 / scenecut=40 / rc-lookahead=80 / lookahead-slices=4 / bframes=8 / bframe-bias=0 / b-adapt=2 / ref=5 / limit-refs=2 / limit-modes / weightp / weightb / aq-mode=1 / qg-size=16 / aq-strength=1.00 / cbqpoffs=0 / crqpoffs=0 / rd=5 / psy-rd=1.50 / rdoq-level=1 / psy-rdoq=5.00 / no-rd-refine / signhide / deblock=-2:-2 / no-sao / no-sao-non-deblock / b-pyramid / cutree / no-intra-refresh / rc=crf / crf=18.0 / qcomp=0.80 / qpmin=0 / qpmax=51 / qpstep=4 / ipratio=1.40 / pbratio=1.20

Video
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main@L4@Main
Width : 1 920 pixels
Height : 800 pixels
Display aspect ratio : 2.40:1
Frame rate : 24.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Writing library : x265 1.9+73-6d06de58c316:[Windows][GCC 5.3.0][64 bit] 8bit
Encoding settings : wpp / ctu=32 / min-cu-size=8 / max-tu-size=16 / tu-intra-depth=2 / tu-inter-depth=2 / me=3 / subme=5 / merange=44 / no-rect / no-amp / max-merge=4 / temporal-mvp / no-early-skip / rdpenalty=2 / no-tskip / no-tskip-fast / no-strong-intra-smoothing / no-lossless / no-cu-lossless / no-constrained-intra / no-fast-intra / no-open-gop / no-temporal-layers / interlace=0 / keyint=360 / min-keyint=1 / scenecut=40 / rc-lookahead=80 / lookahead-slices=4 / bframes=8 / bframe-bias=0 / b-adapt=2 / ref=5 / limit-refs=2 / limit-modes / weightp / weightb / aq-mode=1 / qg-size=16 / aq-strength=1.00 / cbqpoffs=0 / crqpoffs=0 / rd=5 / psy-rd=1.50 / rdoq-level=1 / psy-rdoq=5.00 / no-rd-refine / signhide / deblock=-2:-2 / no-sao / no-sao-non-deblock / b-pyramid / cutree / no-intra-refresh / rc=crf / crf=18.0 / qcomp=0.80 / qpmin=0 / qpmax=51 / qpstep=4 / ipratio=1.40 / pbratio=1.20

Then results in a matroska container to grasp the average bitrate from mediainfo/mpc-hc:

General
Unique ID : 255088647150482398718149168969974570018 (0xBFE84951A4CB2B07B9B78C63B4105C22)
Complete name : D:\out.mkv
Format : Matroska
Format version : Version 4 / Version 2
File size : 5.20 GiB
Duration : 1h 23mn
Overall bit rate : 8 869 Kbps
Encoded date : UTC 2016-03-11 10:58:24
Writing application : mkvmerge v8.9.0 ('Father Daughter') 64bit
Writing library : libebml v1.3.3 + libmatroska v1.4.4
DURATION : 01:23:57.000000000
NUMBER_OF_FRAMES : 120888
NUMBER_OF_BYTES : 5582851354
_STATISTICS_WRITING_APP : mkvmerge v8.9.0 ('Father Daughter') 64bit
_STATISTICS_WRITING_DATE_UTC : 2016-03-11 10:58:24
_STATISTICS_TAGS : BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main@L4@Main
Codec ID : V_MPEGH/ISO/HEVC
Duration : 1h 23mn
Bit rate : 8 692 Kbps
Width : 1 920 pixels
Height : 800 pixels
Display aspect ratio : 2.40:1
Frame rate mode : Constant
Frame rate : 24.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Bits/(Pixel*Frame) : 0.236
Stream size : 5.10 GiB (98%)
Writing library : x265 1.9+73-6d06de58c316:[Windows][GCC 5.3.0][64 bit] 8bit
Encoding settings : wpp / ctu=32 / min-cu-size=8 / max-tu-size=16 / tu-intra-depth=2 / tu-inter-depth=2 / me=3 / subme=5 / merange=44 / no-rect / no-amp / max-merge=4 / temporal-mvp / no-early-skip / rdpenalty=2 / no-tskip / no-tskip-fast / no-strong-intra-smoothing / no-lossless / no-cu-lossless / no-constrained-intra / no-fast-intra / no-open-gop / no-temporal-layers / interlace=0 / keyint=360 / min-keyint=1 / scenecut=40 / rc-lookahead=80 / lookahead-slices=4 / bframes=8 / bframe-bias=0 / b-adapt=2 / ref=5 / limit-refs=2 / limit-modes / weightp / weightb / aq-mode=1 / qg-size=16 / aq-strength=1.00 / cbqpoffs=0 / crqpoffs=0 / rd=5 / psy-rd=1.50 / rdoq-level=1 / psy-rdoq=5.00 / no-rd-refine / signhide / deblock=-2:-2 / no-sao / no-sao-non-deblock / b-pyramid / cutree / no-intra-refresh / rc=crf / crf=18.0 / qcomp=0.80 / qpmin=0 / qpmax=51 / qpstep=4 / ipratio=1.40 / pbratio=1.20
Default : Yes
Forced : No

My conclusions are:

The source video was 15.3GB big (16.441.757.770 bytes) with an average bitrate of 25.6 Mbps, we are ending up with a x265 compressed video file which weights 5.19GB (5.582.852.322 bytes) so that's nearly a 1/3rd of the original so that's quite good, visual quality seemed rather decent in indoor scenes, looked rather "not so HD content looking anymore but acceptable still" in outdoor scenes. It took no less than 31095 seconds (8 hours and 38 minutes roughly, which is acceptable by my standards (once again though, the movie length is only 1h23m including intro & credits which both compress very fast and very well and the frame size is 1920x800 "only"), however the problem I see is:

Once the video gets merged with it's 910MB DTS track, we end up with a movie that's ready for "archiving" at the size of 6.08GB (6.534.470.512 bytes) while it's x264 counterpart yielded a file that was 6.56 GB (7.041.497.802 bytes).

My final conclusion is; Arguably the x264 encode seemed slightly better than the x265 in terms of visual quality on outdoor scenes and we computed this for roughly 8 hours 40 minutes only to shave off... 507MB (507.027.290 bytes).

Overall would I say this was this worth it? Hard to tell. It would be worth it if the quality would be on par and I would have shaved off at least 1GB of data compared to a x264 encode and of course kept identical quality.

In our case, I have over 2600 movies to re-encode and archive for our online digital library, given we could shave off 1GB per movie/encode, that's potentially 2.6TB of data saved up, right there... at the expense of power consumption bill and patience, lots of it.

I'm doing another test with the same parameters just with --preset veryslow to see if I can get the file slightly smaller. Will post results as soon as I have them.

Thank you again for your attention.

[littlepox]

I'm doing another test with the same parameters just with --preset veryslow to see if I can get the file slightly smaller. Will post results as soon as I have them.

Thank you again for your attention.

Be careful when you are switching to --preset veryslow because I have manually set a lot parameters.

This is what you should do for the --preset veryslow (otherwise the actual parameters shall be rather identical)

--preset veryslow --ctu 32 --max-tu-size 16 --crf 18 --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --keyint 360 --min-keyint 1 --bframes 8 --aq-mode 1 --aq-strength 1.0 --rd 5 --psy-rd 1.5 --psy-rdoq 5.0 --rdoq-level 1 --no-sao --no-open-gop --rc-lookahead 80 --max-merge 4 --qcomp 0.8 --no-strong-intra-smoothing --deblock -2:-2 --qg-size 16 --pbratio 1.2

Just be patient.

[pingfr]

Unfortunately I am out at the moment and have neither physical nor intranet access to the encoding box as we speak.

I shall return in a few hours, will do a test with your newest params once I get back.

Thanks!

[LigH]

Some more bitrate control enhancements in x265 1.9+88-b6d8e66e7f71 (https://www.mediafire.com/download/0kv17e8g4vmy2rj/x265_1.9+88-b6d8e66e7f71.7z)

[pingfr]

@littlepox: Just got back, will interrupt current encoding and restart a newer one with the latest parameters you gave me a few minutes ago.

@LigH: Looking at the source https://bitbucket.org/multicoreware/x265/commits/all I don't see any bitrate control related changes, which commit are you refering to?

[LigH]

Some are abbreviated, some a bit indirect ... e.g. "rc: change reencode position for cappedvbr" ("rc" = "rate control"; cappedvbr ~ VBV limited VBR with instantly re-encoded GOPs), or the tweaks for "--tune grain"; even "sao: Use qp of encoded CU, instead of slice qp" (using a better quantization base value should improve the bitrate distribution slightly).

[TalasNetrag]

Yes, a kind of "motion trails" with alternating distance, which seems to be related to mini-GOP ranges (means, the group of B frames between a pair of P, maybe I, frames).

I tried playing aroung with the minGOP and maxGOP settings. minGOP didnt do anything, maxGOP also didnt do much until I set it to 0, with also introduced a lot of noise.

[LigH]

Both options are mostly unrelated to this issue. They set the limits of distances between I frames. But the issue is in the B frames, I would guess. I doubt you can fix that easily with any available CLI options, maybe except some which are B frame related, e.g. limit the maximum number of consecutive B frames (but that would decrease the overall efficiency), or alter the use of B frame pyramids (that may be interesting).

[MeteorRain]

Once the video gets merged with it's 910MB DTS track, we end up ...

Hmm? Why using the crappy 1509k DTS?

If you are going to play with h/w decoding, go with 640k AC3.
If going s/w way, ~400k AAC or even OPUS.

Using DTS to me is like encoding your bluray with MPEG-1.

[benwaggoner]

Some are abbreviated, some a bit indirect ... e.g. "rc: change reencode position for cappedvbr" ("rc" = "rate control"; cappedvbr ~ VBV limited VBR with instantly re-encoded GOPs), or the tweaks for "--tune grain"; even "sao: Use qp of encoded CU, instead of slice qp" (using a better quantization base value should improve the bitrate distribution slightly).
And --tune grain now includes --rc-grain, a whole new rate control mode for grainy content. Seems like a "1.9.5" update just happened.

https://bitbucket.org/multicoreware/x265/commits/578c7f12b7f4ab51b5635fc0acfd89e057630623
https://bitbucket.org/multicoreware/x265/commits/54d2625ae6a18acd471708b78a3ee8d5357d2fbc
https://bitbucket.org/multicoreware/x265/commits/305a1272a412e6da50544c151820631b319de1cc

I'll try to test today or over the weekend, if I can get either of my 16-core workstations to start booting again (they were all good two days ago!).

Are those having quality issues using CRF and RC at the same time trying the new 2-pass CRF reencode mode? That should help quality when high complexity results in the VBV constraining CRF.

[benwaggoner]

...
This is based on the --tune film we have tested out. Again, do NOT modify any one of the above parameters otherwise you shall destroy the whole combination.
Has anyone tried doing a separate "tune animation." Based on x264, I'd think optimal cel animation settings would be substantially different than the film ones. Although there may be some overlap in constraining unit size and such.

[pingfr]

@littlepox: ETA is still roughly 8 hours, this encode test should be done around 4:00AM UTC+1. Until then, don't hold your breath. :p

(Regardless of the results, once this encode is done, I'll upgrade to the latest 1.9+88 as benwaggoner more or less tagged it as a "1.9.5").

[TalasNetrag]

Both options are mostly unrelated to this issue. They set the limits of distances between I frames. But the issue is in the B frames, I would guess. I doubt you can fix that easily with any available CLI options, maybe except some which are B frame related, e.g. limit the maximum number of consecutive B frames (but that would decrease the overall efficiency), or alter the use of B frame pyramids (that may be interesting).

*sails over the head* I'll just make lossless backup with x264 and try newer versions of x265 when they come out.

[littlepox]

@littlepox: ETA is still roughly 8 hours, this encode test should be done around 4:00AM UTC+1. Until then, don't hold your breath. :p

(Regardless of the results, once this encode is done, I'll upgrade to the latest 1.9+88 as benwaggoner more or less tagged it as a "1.9.5").

I've said this combination is tested with v1.9+1. It is NOT going to be optimal anymore for later versions.

[roo1234]

Has anyone tried doing a separate "tune animation." Based on x264, I'd think optimal cel animation settings would be substantially different than the film ones. Although there may be some overlap in constraining unit size and such.
I'm really interested in some animation tune for really low bitrates. I mean low, like 100-200kbps for mobile content. HEVC seems killer for that.

[pingfr]

@littlepox: The newest results are here. :)

The source is identical so we might as well refer to the other post if needed.

The newest parameters just so we agree on what we're talking about:

@echo off
avs4x265.exe -P x265.exe --preset veryslow --ctu 32 --max-tu-size 16 --crf 18 --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --keyint 360 --min-keyint 1 --bframes 8 --aq-mode 1 --aq-strength 1.0 --rd 5 --psy-rd 1.5 --psy-rdoq 5.0 --rdoq-level 1 --no-sao --no-open-gop --rc-lookahead 80 --max-merge 4 --qcomp 0.8 --no-strong-intra-smoothing --deblock -2:-2 --qg-size 16 --pbratio 1.2 --output newparams.hevc %1
pause

The console log output for reference:

avs [info]: AviSynth 2.60 (ICL10)
avs [info]: Video colorspace: YV12
avs [info]: Video resolution: 1920x800
avs [info]: Video framerate: 24/1
avs [info]: Video framecount: 120888
avs4x265 [info]: "x265.exe" - --frames 120888 --fps 24/1 --input-res 1920x800 --input-csp i420 --preset veryslow --ctu 32 --max-tu-size 16 --crf 18 --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --keyint 360 --min-keyint 1 --bframes 8 --aq-mode 1 --aq-strength 1.0 --rd 5 --psy-rd 1.5 --psy-rdoq 5.0 --rdoq-level 1 --no-sao --no-open-gop --rc-lookahead 80 --max-merge 4 --qcomp 0.8 --no-strong-intra-smoothing --deblock -2:-2 --qg-size 16 --pbratio 1.2 --output newparams.hevc
yuv [info]: 1920x800 fps 24/1 i420p8 unknown frame count
raw [info]: output file: newparams.hevc
x265 [info]: HEVC encoder version 1.9+73-6d06de58c316
x265 [info]: build info [Windows][GCC 5.3.0][64 bit] 8bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: frame threads / pool features : 5 / wpp(25 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 8
x265 [info]: Residual QT: max TU size, max depth : 16 / 2 inter / 2 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 4 / 4
x265 [info]: Keyframe min / max / scenecut : 1 / 360 / 40
x265 [info]: Intra 32x32 TU penalty type : 2
x265 [info]: Lookahead / bframes / badapt : 80 / 8 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 5 / 0 / 1
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 1.0 / 16 / 1
x265 [info]: Rate Control / qCompress : CRF-18.0 / 0.80
x265 [info]: tools: rect amp limit-modes rd=5 psy-rd=1.50 rdoq=1 psy-rdoq=5.00
x265 [info]: tools: signhide tmvp b-intra deblock(tC=-2:B=-2)

x265 [info]: frame I: 1035, Avg QP:14.19 kb/s: 28438.99
x265 [info]: frame P: 23470, Avg QP:16.46 kb/s: 15379.48
x265 [info]: frame B: 96383, Avg QP:18.74 kb/s: 7048.38
x265 [info]: Weighted P-Frames: Y:2.7% UV:1.4%
x265 [info]: Weighted B-Frames: Y:1.8% UV:1.2%
x265 [info]: consecutive B-frames: 12.4% 5.8% 7.7% 15.8% 9.2% 27.2% 8.1% 6.7% 7.1%

encoded 120888 frames in 43255.58s (2.79 fps), 8848.98 kb/s, Avg QP:18.26
Press a key to continue...

And the resulted file size: 5.18GB (5.572.021.384 bytes).

With --preset veryslow and the newest parameters, we merely shaved off 11MB extra over the previous encode with --preset slower, not that impressive.

Also, it took no less than 43255 seconds to encode, in human time, it's 720 minutes or 12 hours and 55 minutes, almost 13 hours only to shave off 11MB extra, that's negligible.

I think for now I'll stick to the previous encode with --preset slower over this --preset veryslow non-sense, the other encode took "only" 8 hours and half.

Also for what it's worth:

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main@L4@Main
Codec ID : V_MPEGH/ISO/HEVC
Duration : 1h 23mn
Bit rate : 8 675 Kbps
Width : 1 920 pixels
Height : 800 pixels
Display aspect ratio : 2.40:1
Frame rate mode : Constant
Frame rate : 24.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Bits/(Pixel*Frame) : 0.235
Stream size : 5.09 GiB (98%)
Writing library : x265 1.9+73-6d06de58c316:[Windows][GCC 5.3.0][64 bit] 8bit
Encoding settings : wpp / ctu=32 / min-cu-size=8 / max-tu-size=16 / tu-intra-depth=2 / tu-inter-depth=2 / me=3 / subme=4 / merange=57 / rect / amp / max-merge=4 / temporal-mvp / no-early-skip / rdpenalty=2 / no-tskip / no-tskip-fast / no-strong-intra-smoothing / no-lossless / no-cu-lossless / no-constrained-intra / no-fast-intra / no-open-gop / no-temporal-layers / interlace=0 / keyint=360 / min-keyint=1 / scenecut=40 / rc-lookahead=80 / lookahead-slices=0 / bframes=8 / bframe-bias=0 / b-adapt=2 / ref=5 / limit-refs=1 / limit-modes / weightp / weightb / aq-mode=1 / qg-size=16 / aq-strength=1.00 / cbqpoffs=0 / crqpoffs=0 / rd=5 / psy-rd=1.50 / rdoq-level=1 / psy-rdoq=5.00 / no-rd-refine / signhide / deblock=-2:-2 / no-sao / no-sao-non-deblock / b-pyramid / cutree / no-intra-refresh / rc=crf / crf=18.0 / qcomp=0.80 / qpmin=0 / qpmax=51 / qpstep=4 / ipratio=1.40 / pbratio=1.20
Default : Yes
Forced : No

Thank you for your time and efforts. Cheers.

Good night to you.

Edit: Oh also, the previous --preset slower encode was probably better as MPC-HC reported a bitrate of 8 692 Kbps versus 8 675 Kbps here. It's not that big of a difference but every little bit helps I guess (pun intended), that of course... and the 8h38m encode time vs. 12h55m, I can't possibly justify keeping a server's ressources maxed out for nearly 5 hours more just to shave off 11MB.

[djesteban]

@ foxy etc.: Can you confirm that zones don't work for n-pass, --pass 3|2?

I am having trouble with this too... and trouble finding answers.
I was able to make zones work in CRF mode, but it doesn't seem to have any effect on 2pass encodes.

Can anyone confirm this?!

[littlepox]

Has anyone tried doing a separate "tune animation." Based on x264, I'd think optimal cel animation settings would be substantially different than the film ones. Although there may be some overlap in constraining unit size and such.

We have that as well. For anime contents basically you wish to encourage bits to be invested in the edge areas, and that can be done by tweaking psy and aq.


Will probably do another extensive test and post our suggestions here after v2.0. Our current parameters are updated with v1.9 stable.

[pingfr]

@littlepox: No feedback or conclusions on the last encode that ended up a few hours ago? :p

[littlepox]

@littlepox: No feedback or conclusions on the last encode that ended up a few hours ago? :p

No. This was just a repeat of our previous tests suggesting higher presets shall only waste your time with little improvement.

[pingfr]

@littlepox: And it turned out we both came to the same conclusions, more or less. :p

I'm currently pondering whether should I request more funds from our IT department boss to upgrade to a dual Xeon E5-26xx v3 right away or if I should instead wait a few extra months/weeks for Intel to "release" their upcoming and highly anticipated newer Broadwell-EP v4 lines of Xeon CPUs.

It would help me decide if I knew what kind of fps crunching improvement I would get... but meh, I disgress.

Thanks for all the fish!

[Boulder]

Out of interest, do you have to keep everything at the original resolution? Upon playback, a huge amount of Blu-rays look just fine, if not the same as the original, when downscaled to 720p with some sharpening.

[x265_Project]

I'm currently pondering whether should I request more funds from our IT department boss to upgrade to a dual Xeon E5-26xx v3 right away or if I should instead wait a few extra months/weeks for Intel to "release" their upcoming and highly anticipated newer Broadwell-EP v4 lines of Xeon CPUs.

v4 Xeons (Broadwell) use the same basic CPU architecture as v3 (Haswell), so you shouldn't expect to see much in the way of instructions-per-clock improvement. You will be able to get up to 22 cores per chip with v4 Xeons, but it's likely that the price per core will probably be similar for a while after the v4 release. v5 Xeons (Skylake) will support AVX-512 and higher internal ring bandwidth, but I don't expect to see them until late 2016 or early 2017.

[pingfr]

@x265_Project: Hey Tom, good to have you here.

So basically what you're saying is, that a dual v3 encoding box will more or less yield identical encoding speed (fps crunching) to a dual v4 Broadwell-EP because there aren't any newer set of instruction introduced?

[x265_Project]

@x265_Project: Hey Tom, good to have you here.

So basically what you're saying is, that a dual v3 encoding box will more or less yield identical encoding speed (fps crunching) to a dual v4 Broadwell-EP because there aren't any newer set of instruction introduced?
Hey... I'm always here... but I'm letting our experts, nandaku2 and pradeeprama handle more of the technical questions.

That's right... Broadwell is basically a die shrink of Haswell, going from 22 nm to 14 nm. As with any new chip, there will be refinements in timings and power utilization... but the logic units and architecture are basically the same, and there are no new instructions in Broadwell. Because of the die shrink, however, (according to leaked information (http://wccftech.com/intel-broadwellep-xeon-e52600-v4-skus-leaked/)) there will be larger Xeons available with more cores.

On the consumer side there were more changes... primarily bigger, better GPUs (Intel Gen Graphics).

[LigH]

I am having trouble with this too... and trouble finding answers.
I was able to make zones work in CRF mode, but it doesn't seem to have any effect on 2pass encodes.

Can anyone confirm this?!

I just tested the Sintel trailer in a smaller size (640x272) in 3 passes with verbose CSV log files. Plotting the average QP per frame (in encoded order only, displayed order is too hard to reconstruct in Excel) reveals a quite confusing behaviour in "--pass 2", while it looked about as expected in "--pass 3". Developers got details via mailing list.

[aegisofrime]

Hey... I'm always here... but I'm letting our experts, nandaku2 and pradeeprama handle more of the technical questions.

That's right... Broadwell is basically a die shrink of Haswell, going from 22 nm to 14 nm. As with any new chip, there will be refinements in timings and power utilization... but the logic units and architecture are basically the same, and there are no new instructions in Broadwell. Because of the die shrink, however, (according to leaked information (http://wccftech.com/intel-broadwellep-xeon-e52600-v4-skus-leaked/)) there will be larger Xeons available with more cores.

On the consumer side there were more changes... primarily bigger, better GPUs (Intel Gen Graphics).

While we are on this topic, what would you say would be the performance increase upgrading from a i7-4770 to a i7-6700?

[x265_Project]

While we are on this topic, what would you say would be the performance increase upgrading from a i7-4770 to a i7-6700?
For most scenarios, the Skylake i7-6700 is about 20% faster, due to the higher clock speed. For 4K ultrafast it is more than 2x faster, due to higher internal memory bandwidth (which bottlenecks the Haswell i7-4770).

[pingfr]

Since we're talking hardware, I'll drop my experiments here:

In the lab we're using a 5960X (8C/16T) which has it's stock speed clocked at 3GHz with a Turbo speed of 3.5GHz, I instantly overclocked it to 4GHz with a very basic cooler and without any extra tweaks (I could most likely push it farther towards the 4.5GHz even 4.7GHz clamp with proper watercooling, which in my case can't be justified in a professionnal environment).

During our crunching tests at the 4GHz frequency, I use a reference short 5 minutes (11862 frames@50fps) Ultra HD clip with the nasty resolution of 3840x2160; with the following parameters on x265-1.9+88 (10 bits):

avs4x265.exe -P x265-10b.exe -D 10 --input-depth 10 --fps 50 --input-res 3840x2160 --preset ultrafast --no-rect --crf 18 --output output.hevc %1

We're getting a 8.80 fps crunching speed, rendering the 11862 frames encoded more or less under 22 minutes.

This sunday I was granted temporary access to a "supposedly" high-end machine running a single Xeon E5 2695v3 at stock 2.2GHz (could not be overclocked at all, wasn't allowed to play around with it's BIOS or anything) and let's just say that I was REALLY REALLY disappointed.

During my tests, the Xeon E5 2695v3 running at full speed (2.2GHz) with all cores saturated at 100% at all time, gave us crunching speeds of 4.5fps which seemed abysmally low for a production server that was far more costly than our little lab's i7 5960X.

I gathered that a CPU running stable at 4GHz will pretty much "own" a 2.2GHz clocked CPU, but at the same time, that Xeon is supposed to sport 14 cores and no less than 28 threads which I initially thought would grant faster speeds over an i7 HEDT sporting "only" 8 cores and "only" 16 threads and therefore "make up" for the slower clocking speeds.

Could it be related to the fact we only found out in the end that Xeon CPU was an Engineering Sample leaked from Intel and therefore could be slower/crippled/lacking features over a retail CPU?

Who knows.

As far as I'm concerned: i7 5960X (8C/16T) OC'ed at 4GHz > Xeon E5-2695v3 ((ES) 14C/28T) at 2.2GHz any time, any day, anywhere.

[djesteban]

I just tested the Sintel trailer in a smaller size (640x272) in 3 passes with verbose CSV log files. Plotting the average QP per frame (in encoded order only, displayed order is too hard to reconstruct in Excel) reveals a quite confusing behaviour in "--pass 2", while it looked about as expected in "--pass 3". Developers got details via mailing list.

LigH, I have a thread dedicated to this problem here (http://forum.doom9.org/showthread.php?t=173290).
Would be nice if you could post your result there also to attract more attention on the issue.

I'll make my own test on my side and post my result there. If there's indeed an issue, I'll log a bug on bitbucket I guess...

[x265_Project]

In the lab we're using a 5960X (8C/16T) which has it's stock speed clocked at 3GHz with a Turbo speed of 3.5GHz, I instantly overclocked it to 4GHz with a very basic cooler and without any extra tweaks...

During our crunching tests at the 4GHz frequency, I use a reference short 5 minutes (11862 frames@50fps) Ultra HD clip with the nasty resolution of 3840x2160; with the following parameters on x265-1.9+88 (10 bits):

avs4x265.exe -P x265-10b.exe -D 10 --input-depth 10 --fps 50 --input-res 3840x2160 --preset ultrafast --no-rect --crf 18 --output output.hevc %1

We're getting a 8.80 fps crunching speed, rendering the 11862 frames encoded more or less under 22 minutes.

This sunday I was granted temporary access to a "supposedly" high-end machine running a single Xeon E5 2695v3 at stock 2.2GHz (could not be overclocked at all, wasn't allowed to play around with it's BIOS or anything) and let's just say that I was REALLY REALLY disappointed.

During my tests, the Xeon E5 2695v3 running at full speed (2.2GHz) with all cores saturated at 100% at all time, gave us crunching speeds of 4.5fps which seemed abysmally low for a production server that was far more costly than our little lab's i7 5960X.

I gathered that a CPU running stable at 4GHz will pretty much "own" a 2.2GHz clocked CPU, but at the same time, that Xeon is supposed to sport 14 cores and no less than 28 threads which I initially thought would grant faster speeds over an i7 HEDT sporting "only" 8 cores and "only" 16 threads and therefore "make up" for the slower clocking speeds.

As far as I'm concerned: i7 5960X (8C/16T) OC'ed at 4GHz > Xeon E5-2695v3 ((ES) 14C/28T) at 2.2GHz any time, any day, anywhere.
You're seeing Amdahl's Law (https://en.wikipedia.org/wiki/Amdahl's_law) in action. While we've done everything possible to parallelize x265, there are certain functions that are inherently serial (CABAC encoding and decoding, for example). Even for the operations we've parallelized, there are often dependencies on other threads.

For example, if you are running 8 threads on a quad-core machine, most threads will be busy most of the time. Occasionally, one or more of the 8 threads will be waiting for the result from another thread, and so it will be stalled. If you double the power of your machine (run a 5960x with 8 cores/16 threads), you will not get 2x the performance, due to Amdahl's Law. Now, the number of threads that might be waiting for another thread to finish goes up. In x265 we have multiple frame encoders, which call row encoders, which call CTU encoders. Let's say one of the CTU encoders encounters some complex video, and it doesn't easily find good prediction matches in its reference frames. It will take longer than average to encode, and it will stall the row encoder, which may stall other row encoders that depend on that CTU being done (so that it can be referenced).

So, it is an ongoing challenge to get the highest possible effective utilization from many-core machines. We have a private commercial encoding library called UHDkit that can break the incoming video into chunks, encoding each chunk with a separate instance of x265. In this way we can gain higher effective utilization on many-core machines, such as the dual Xeon E5-2699 v3 (2 x 18 cores = 36 cores / 72 threads). The downside is added latency for live encoding, as you have to queue up enough video to keep all encoder instances busy.

[pingfr]

We have a private commercial encoding library called UHDkit that can break the incoming video into chunks, encoding each chunk with a separate instance of x265.

Hey Tom,

Sounds good but on our little test-bed project, I already fought hard with my department's manager and the IT department's boss/office chief to allocate funds to acquire a 5960X...

In the coming days I will have to fight again to convince them to either allocate even more extra funds or to grab a dual E5-26XX-v3 server from the next IT department...

So now if on top of that, I also have to convince them to acquire expensive licenses for a private commercial encoding library... my head will be on a pike, literally. :)

But yes, this definitely looks sexy:

http://www.multicorewareinc.com/wp-content/uploads/2015/04/x265_88FPS.jpg

:drool:

[stax76]

StaxRip should now be up-to-date in regard of x265, changes in the last build are:



larger custom command line TextBox
new switch added --rc-grain
tune grain defaults updated
the encoding options dialogs for x265, AMD, Intel and NVIDIA have now an option to display the full command line
removed one switch that was accidentally added twice :-)
x265 1.9+88


Download: http://forum.doom9.org/showthread.php?p=1760640#post1760640

http://oi64.tinypic.com/c26id.jpg

[pingfr]

@stax76: Congrats! (I'm more of a MeGUI guy myself but nevertheless, grats to you!). :p

Edit: Also good to see those settings littlepox gave us a few days ago are being added as a "custom" profile to StaxRip, way to go! :)

[benwaggoner]

Will probably do another extensive test and post our suggestions here after v2.0. Our current parameters are updated with v1.9 stable.
I don't know there's any indication that 1.9 will be followed by a "super major" 2.0 release, other than base 10 :). Next up might be 1.10 AFAIK. Or maybe 1.A if it turns out MCW was using hex all along...

1.9 is a good release, and I don't think there's any reason to wait to make animation tuning. And the earlier it is worked on, the more feedback can go into the next version. I'd love to see a proper --tune animation and --tune film in there .

[LigH]

@ benwaggoner:

The x265 team already documented that they will prefer a continuous decimal version progression. After v1.9 will quite certainly follow v2.0 without any "super major" speciality attribute, v1.0 wasn't special either after v0.9.

[pingfr]

@LigH: How about... x265-v2.0-reloaded-extended-unleashed-extreme-edition-super-combo-deluxe+1 ? :)

[littlepox]

I don't know there's any indication that 1.9 will be followed by a "super major" 2.0 release, other than base 10 :). Next up might be 1.10 AFAIK. Or maybe 1.A if it turns out MCW was using hex all along...

1.9 is a good release, and I don't think there's any reason to wait to make animation tuning. And the earlier it is worked on, the more feedback can go into the next version. I'd love to see a proper --tune animation and --tune film in there .

I mean currently they are doing quite a lot in RC adjustments. Our combination works quite different under 1.9 stable and the latest builds. So we might just wait until v2.0, when the change is more or less verified and settled.

For people interested in --tune animation, here it is:

--ctu 32 --max-tu-size 16 --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --weightb --keyint 360 --min-keyint 1 --aq-mode 1 --aq-strength 1.1 --rd 5 --psy-rd 1.5 --psy-rdoq 3.0 --pbratio 1.2 --rdoq-level 1 --no-sao --no-open-gop --rc-lookahead 80 --scenecut 40 --max-merge 4 --qcomp 0.8 --no-strong-intra-smoothing --deblock -2:-2 --qg-size 16

if your ref is <3, pls increase it by 1.
if your bframes is <4, pls increase it by 2.


BTW, the definition of "animation" is quite ambiguous. We assume you are referring to those with sharp edges, simple colors, and very tiny grains.
However, with developments recently in Japan, this is also "animation": http://img.2222.moe/images/2016/02/23/21114.png

Should you are dealing with anime of high quality (usually BD source) and you wish to retain the high quality (typically with x264_crf<17), you are more encouraged to use our --tune film.

[benwaggoner]

I mean currently they are doing quite a lot in RC adjustments. Our combination works quite different under 1.9 stable and the latest builds. So we might just wait until v2.0, when the change is more or less verified and settled.

For people interested in --tune animation, here it is:

--ctu 32 --max-tu-size 16 --tu-intra-depth 2 --tu-inter-depth 2 --rdpenalty 2 --ref 5 --weightb --keyint 360 --min-keyint 1 --bframes 8 --aq-mode 1 --aq-strength 1.1 --rd 5 --psy-rd 1.5 --psy-rdoq 3.0 --pbratio 1.2 --rdoq-level 1 --no-sao --no-open-gop --rc-lookahead 80 --scenecut 40 --max-merge 4 --qcomp 0.8 --no-strong-intra-smoothing --deblock -2:-2 --qg-size 16
No --qg-size? I would have guessed that 16 would be helpful to push down quant changes to small blocks that would contain lines.

BTW, the definition of "animation" is quite ambiguous. We assume you are referring to those with sharp edges, simple colors, and very tiny grains.
However, with developments recently in Japan, this is also "animation": http://img.2222.moe/images/2016/02/23/21114.png

Should you are dealing with anime of high quality (usually BD source) and you wish to retain the high quality (typically with x264_crf<17), you are more encouraged to use our --tune film.
I was thinking traditional cel animation. I agree that the introduction of continuous tones like in your image would make it more like film content in terms of adaptive quant requirements. The lack of motion blur and large areas that are identical between frames will remain a significant difference, although I'm not sure if they would impact tuning. More likely, they just will make encoding more efficient.

[littlepox]

No --qg-size? I would have guessed that 16 would be helpful to push down quant changes to small blocks that would contain lines.

check the last one.
Also I removed the hard coded --ref and --bframes


I was thinking traditional cel animation. I agree that the introduction of continuous tones like in your image would make it more like film content in terms of adaptive quant requirements. The lack of motion blur and large areas that are identical between frames will remain a significant difference, although I'm not sure if they would impact tuning. More likely, they just will make encoding more efficient.

We are actually doing the opposite types, which, at least in high quality, even for x264 we are using a tuning more close to --tune film.

[x265_Project]

@LigH: How about... x265-v2.0-reloaded-extended-unleashed-extreme-edition-super-combo-deluxe+1 ? :)

That's got my vote!

[pingfr]

That's got my vote!

Glad you like it, watch out tho: I might ask for royalties fees if you ever go with that one! :devil:

[pingfr]

@x265_Project: Contemplating the idea of grabbing 2x E5-2687W v3 (10C/20T at 3.1GHz with Turbo at 3.5GHz).

http://ark.intel.com/products/81909/Intel-Xeon-Processor-E5-2687W-v3-25M-Cache-3_10-GHz

But unsure if any other CPU combo would be any faster...

Also a test that might somehow be relevant: http://www.anandtech.com/show/8730/intel-haswellep-xeon-14-core-review-e52695-v3-and-e52697-v3/3

Any ideas on that matter?

[Atak_Snajpera]

@LigH: How about... x265-v2.0-reloaded-extended-unleashed-extreme-edition-super-combo-deluxe+1 ?
You forgot ULTRA and TURBO words.

[LigH]

*LigH prepares a "Bullsh.../Buzzword Bingo" sheet...*

[x265_Project]

You forgot ULTRA and TURBO words.
Not to mention "Titanium Edition"

[x265_Project]

@x265_Project: Contemplating the idea of grabbing 2x E5-2687W v3 (10C/20T at 3.1GHz with Turbo at 3.5GHz).

http://ark.intel.com/products/81909/Intel-Xeon-Processor-E5-2687W-v3-25M-Cache-3_10-GHz

But unsure if any other CPU combo would be any faster...

Also a test that might somehow be relevant: http://www.anandtech.com/show/8730/intel-haswellep-xeon-14-core-review-e52695-v3-and-e52697-v3/3

Any ideas on that matter?
Again, it depends on whether you are running a single x265 instance, or a transcode pipeline that supports multiple parallel instances. If you're running one x265 instance, lean towards faster single-threaded performance. If you're running UHDkit, get the most total compute power for your dollars.

The Anandtech review from 2014 is interesting, but it's a bit dated, given all of the AVX2 acceleration we did last year, and other algorithmic improvements.