10 hours for 21.8 GiB.

[mindphasar]

Hi guys,

I would like to reduce the size of this file: 10 hours for 21.8 GiB. The audio part (AAC) will be copied with ffmpeg.

This file is a live event recorded with youtube-dl.


Can anyone give me some settings for x265?



Thanks


Format : MPEG-4
Format profile : Base Media
Codec ID : isom (isom/iso2/avc1/mp41)
File size : 21,8 GiB
Duration : 10 o 8 min
Overall bit rate : 5.135 kb/s
Writing application : Lavf58.33.100

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.2
Format settings : CABAC / 1 Ref Frames
Format settings, CABAC : Si
Format settings, RefFrames : 1 frame
Format settings, GOP : M=1, N=30
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 10 o 8 min
Bit rate : 5.034 kb/s
Width : 1.920 pixel
Height : 1.080 pixel
Display aspect ratio : 16:9
Frame rate mode : Variabile
Frame rate : 42,547 FPS
Minimum frame rate : 0,016 FPS
Maximum frame rate : 62,500 FPS
Original frame rate : 60,000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bit
Scan type : Progressive
Bits/(Pixel*Frame) : 0.057
Stream size : 21,4 GiB (98%)
Codec configuration box : avcC

[Forteen88]

If you got a fast CPU, then this x265-setting is good,
--profile main10 --output-depth 10 --preset slower --crf 18 --no-sao --aq-mode 3 --rdoq-level 0 --psy-rdoq 0

Use the --crf 18 for high-quality, but if you want to compromise video-quality over file-size, then change to a higher crf-value, like 20.

Also, maybe use you should the Avisynth-script TemporalDegrain2 if the video-source is too grainy for your taste.

That video-source got variable frame-rate, I hope that it will cause no problem.

[RanmaCanada]

Youtube is already at optimal bitrate. You're not going to get a beneficial size reductions in regard to the amount of time it will take you to encode it. Just burn this download to a bluray and you're done.

[microchip8]

If you got a fast CPU, then this x265-setting is good,
--preset slower --crf 18 --no-sao --aq-mode 3 --rdoq-level 0 --psy-rdoq 0

Use the --crf 18 for high-quality, but if you want to compromise video-quality over file-size, then change to a higher crf-value, like 20.

Also, maybe use you should the Avisynth-script TemporalDegrain2 if the video-source is too grainy for your taste.

That video-source got variable frame-rate, I hope that it will cause no problem.

I find crf 18 rather high. The crf mappings between x264 and x265 are different. A crf 18 in x264 will correspound to a crf 20 or 21 for x265. Since x265 is more efficient in compressing for the same visual quality, for me personally a crf of 21 is visually indistinguishable comparet to crf 18 in x264

But to each his own...

PS: i do 10 bits encodes only so this has an effect too

[foxyshadis]

To save anything at all, since YouTube 1080p is already a fairly low 5mbps, you'd have to start with a crf of at least 25 and go up from there if it still isn't enough. (5mpbs will generally be between 18 and 24, though YMMV by video.) You're already working with mediocre Youtube-quality source, and the only way you have to really save bits is to murder quality even further. How far do you want to go?

Dropping it to 720p might actually help significantly, though.

I'd probably just put some time into editing the 10 hour video down, to save whole swaths of bits that don't need to be encoded at all.

[Forteen88]

PS: i do 10 bits encodes only so this has an effect tooTrue, 10 bits is the way to go with x265. Weird, I thought that 10 bits was the default --output-depth value in x265, but I just looked it up, and saw that it's not (at least not in the x265-build I use), default is 8!

You might want to add: --frame-threads 1
in the commandline for better picture-quality, but if you got many cores on your CPU, then it will take longer time to encode (since x265 don't use all cores then).

UPDATE: froggy1, OK, --frame-threads 2 might be a better value to set, for more CPU-core usage.

[microchip8]

True, 10 bits is the way to go with x265. Weird, I thought that 10 bits was the default --output-depth value in x265, but I just looked it up, and saw that it's not (at least not in the x265-build I use), default is 8!

You might want to add: --frame-threads 1
in the commandline for better picture-quality, but if you got many cores on your CPU, then it will take longer time to encode (since x265 don't use all cores then).

What's the point then of having a CPU with enough/many threads and then not being able to utilize it fully during encoding for the sake of quality improvement that is so minor you won't notice it unless you use a magnifying glass to look at still pictures?

[excellentswordfight]

True, 10 bits is the way to go with x265. Weird, I thought that 10 bits was the default --output-depth value in x265, but I just looked it up, and saw that it's not (at least not in the x265-build I use), default is 8!

You might want to add: --frame-threads 1
in the commandline for better picture-quality, but if you got many cores on your CPU, then it will take longer time to encode (since x265 don't use all cores then).
So he should encode an 10h 60p(!) clip with preset slower and frame-threads 1 with crf18? So he can come back to this thread after like 2months of encoding to say that he ended up with another 20Gb file?

[Forteen88]

So he should encode an 10h 60p(!) clip with preset slower and frame-threads 1 with crf18? So he can come back to this thread after like 2months of encoding to say that he ended up with another 20Gb file? & froggy1.
He could just encode something else (maybe even 3-4 encodes), simultaneously, if he wants 100% CPU-core use.

Bitrate on the video-source is only 5k, so it seems very compressible, not a high detail source...

TC (mindphasar) should probably better use 2-pass instead of CRF, to get the file-size he wants.

[microchip8]

& froggy1.
He could just encode something else (maybe even 3-4 encodes), simultaneously, if he wants 100% CPU-core use.

Bitrate on the video-source is only 5k, so it seems very compressible, not a high detail source...

That doesn't solve the problem of waiting a long time to finish the specific encode. Throwing other encodes too will not make them encode any faster even if the CPU is fully saturated because you run an X amount of them. It will take just as long for all of them since they'll all use 1 frame thread and the quality benefit is still very questionable.

You really need to re-think your strategy. Encoding is all about compromise between how much speed you can or want to give up and how much quality you want to get. The trick is to find the right balance between the two

If you don't mind encoding at 0.1 fps and wait a whole month for a single (or a few encodes if you use others simultaniously to saturate the CPU in case of 1 frame thread) and think this will somehow improve quality so much that it's obvious from an average distance from the TV/monitor, you're only deluding yourself

[Forteen88]

froggy1. Personally (with my 6-core CPU), I like setting --frame-threads 1
because if I use all cores, then the computer gets quite unusable (laggy) for anything else.
Setting --frame-threads 1 makes x265 take like 60-70% of my CPU.
I don't encode a lot though, and I've never tried x265-encoding on a 12+ core CPU.

EDIT: froggy1, yeah, thanks, I know about priority (crappy Windows10 that I use got that too), but I just don't bother with that (because I'm in no hurry when I encode, and then I don't need to stress out my CPU).

[microchip8]

froggy1. Personally (with my 6-core CPU), I like setting --frame-threads 1
because if I use all cores, then the computer gets quite unusable (laggy) for anything else.
Setting --frame-threads 1 makes x265 take like 60-70% of my CPU.
I don't encode a lot though, and I've never tried x265-encoding on a 12+ core CPU.

That's fine, but don't suggest others do it as well. You'll find virtually none here use 1 frame threads for all of their encodes. The quality benefit vs speed just isn't worth it.

I do encodes all the time (previously on an i7-7700K, now on a Ryzen 7 3700X) and spent a few days comparing different options and tweaking things in order to come up with something that improves quality (compared to the i7-7700K) and doesn't sacrifice too much speed on my new Ryzen 7 system. Using 1 frame threads wasn't even a consideration as there's no point in getting a Ryzen 7 for this

As for sluggishness, you can make the encode run at the lowest possible priority and still have a responsive system. Sure, it'll take a bit longer if you do other stuff that uses most or all threads, but it's worth it. As you might guessed from my avatar, I'm on Linux and always run encodes at nice priority of 19 (lowest possible). There is absolutely no sluggishness here. Windows might be different so I can't comment on that

[Boulder]

On Windows, I use the command start /belownormal encode.bat to run the batch file which contains my job list. Works nicely, no sluggishness and wins over any stuff that is set for idle priority but goes to the background when something more urgent comes up. I just add stuff to the end of the file so my computer is basically encoding 24/7.

[Forteen88]

By the way, not everyone here at Doom9 have high --frame-threads,
https://forum.doom9.org/showthread.php?p=1852904#post1852904

TC, you can also set --ctu 32 if you want more CPU-usage, --ctu 64 (default for --slower) should be set for UHD-video though.

[DotJun]

On windows you can remove a core using affinity while you are doing other things then re-enable it when you are done.

[mindphasar]

thank you so much for the great suggestions guys. I don't know which settings are recommended for this huge file. I use debian 10.with ffmpeg, x265 and mkvmerge.

where i find a examples of 2pass with x265?.

if I add --no-sao with CRF 19, it is not passed 19 but CRF-28.0 Why?

x265 [info]: Main 10 profile, Level-3 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 2 / wpp(8 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 3 / 3
x265 [info]: Keyframe min / max / scenecut / bias: 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 25 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 0 / 0
x265 [info]: References / ref-limit cu / depth : 4 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-28.0 / 0.60
x265 [info]: tools: rect limit-modes rd=4 psy-rd=2.00 rdoq=2 psy-rdoq=1.00
x265 [info]: tools: rskip signhide tmvp strong-intra-smoothing


my goal is to reduce the size of this file, if it is convenient to do a similar action, obviously I have an asus Bluray 16x recorder.

this script is ok?

#!/bin/bash
ffmpeg -i "10 hours.mp4" -pix_fmt yuv420p10 -s 720x408 -aspect 16:9 -strict -1 -an -f yuv4mpegpipe - | x265 --y4m - --preset slow --tune fastdecode --no-high-tier --ref 3 --rc-lookahead 5 -b 3 --profile main10 --crf 19 --deblock -4:-4 --output-depth 10 --dither --wpp -o "raw_video.hevc"

[RanmaCanada]

If you have a bluray burner, just burn the file to disc as I suggested. Or even use any of the free authoring programs to create an actual bluray. It will take 1 single layer bluray to keep the file as it is, which typically cost $0.50 when they are bought in a 50 pack. You will waste more money in electricity converting this than the price of a bluray.

[Forteen88]

...
this script is ok?
...
--rc-lookahead 5 ... --deblock -4:-4 ...That --rc-lookahead is too low! Set it to at least 25 (I set it to 48).

And you should rather set --deblock -3:-3 or as high as you accept (I usually set it to --deblock -2:-2), since the compression gets better then.

You can use an encoding-GUI like "Simple x264/x265-launcher" (available here at Doom9) to encode 2-pass, but don't forget to update x265 in that package, because that developer doesn't update that program often.

EDIT: Although you didn't really apply any of my advice, so IDC.

[MeteorRain]

10 hours live event 1920x1080 60fps at 5034kbps is fairly low.

I have been encoding some live events from TV broadcasting (source is at much higher bitrate -- less artifacts), and 10-bit 1440x1080 60fps CRF24 costs 7500kbps, and that's after a heavy denoising (which saves a good portion).

So a good chance is you'd come back at a much worse, terrible quality for an even bigger file.

EDIT: Just saw you're re-encoding it to 400p. That would make it small but still it would be at much worse quality.

[microchip8]

That --rc-lookahead is too low! Set it to at least 25 (I set it to 48).

And you should rather set --deblock -3:-3 or as high as you accept (I usually set it to --deblock -2:-2), since the compression gets better then.

You can use an encoding-GUI like "Simple x264/x265-launcher" (available here at Doom9) to encode 2-pass, but don't forget to update x265 in that package, because that developer doesn't update that program often.

EDIT: Although you didn't really apply any of my advice, so IDC.

A deblock of -3,-3 is a bit too low and will start to create banding in flat/clean areas. Keep in mind that the deblock filter in HEVC is much improved compared to the one in H.264. I use -1,-1 and while it creates a bit softer image, it masks banding while still preserving enough detail. Also, compression will be better the more you increase the deblocking since it will start to smooth things (= remove more detail) leading to the encoder to spend less bits. The more you lower deblocking, the more detail it'll preserve (and start to introduce banding) thus making the encoder spend more bits on the detail

As for rc-lookahead, the more you increase it, the more efficient adaptive quantization will be. rc-lookahead cannot be greater than the max keyframe. I use a value of 100 here (max-keyframe set to 240). Also rc-lookahead has an impact on latency. Lower values provide better latency at the expense of less effectiveness of adaptive quant (and less accurate scenecut decision)

[benwaggoner]

What's the point then of having a CPU with enough/many threads and then not being able to utilize it fully during encoding for the sake of quality improvement that is so minor you won't notice it unless you use a magnifying glass to look at still pictures?
-F 1 --pmode will saturate 12+ cores at 1080p. The quality benefit from turning off frame threading is a lot less than it was a few years ago, however.

[benwaggoner]

True, 10 bits is the way to go with x265. Weird, I thought that 10 bits was the default --output-depth value in x265, but I just looked it up, and saw that it's not (at least not in the x265-build I use), default is 8!
With an 8-bit source, there's no real reason to encode in 10-bit in x265. Encoding 8-bit will be significantly faster and the output will be more compatible with older mobile devices. Encoding 8-bit in 10-bit provided a measurable benefit with x264, but HEVC is designed to fix the issues in 8-bit that made that valuable with H.264.

You might want to add: --frame-threads 1
in the commandline for better picture-quality, but if you got many cores on your CPU, then it will take longer time to encode (since x265 don't use all cores then).

UPDATE: froggy1, OK, --frame-threads 2 might be a better value to set, for more CPU-core usage.
Yeah, 2 is pretty safe.

[benwaggoner]

I think there is a lot of boiling the ocean here with changing deblock, sao, bit depth, etcetera. Those really don't have nearly the same impact in x265 3.1+ compared to a few years ago.

I'd start with just --preset slower --crf 18 and see what comes out. Given the weakness of the original encode (only 1 ref frame, no b-frames, probably a HW encoder) I'd expect that x265 could deliver the same quality in less than half the bitrate. Find some interesting 5 min chunks and do test encodes of those. The big question is what CRF offers sufficient transparency to the source. Which certainly already has visible compression artifacts. Particularly detail loss, maybe some blocking as well. 5 Mbps for 1080p60 with that limited AVC feature set is WAY low. I'd want more like 80 with those settings to have a hope at reasonable source quality.

[Forteen88]

-F 1 --pmode will saturate 12+ cores at 1080p. The quality benefit from turning off frame threading is a lot less than it was a few years ago, however.So what is the recommended way for higher video-quality when doing ~6 core encoding? -F 1 --pmode or just -F 2?

[Boulder]

-F 1 --pmode will saturate 12+ cores at 1080p.

That doesn't happen based on my experiences. I'm currently encoding a 1920x956 stream with --pmode -F 4 on my Ryzen 3900X (12c/24t), and the CPU usage is ~90% and that also includes decoding and denoising in the Vapoursynth script. Using -F 1 would drop the usage quite a bit lower.

[benwaggoner]

That doesn't happen based on my experiences. I'm currently encoding a 1920x956 stream with --pmode -F 4 on my Ryzen 3900X (12c/24t), and the CPU usage is ~90% and that also includes decoding and denoising in the Vapoursynth script. Using -F 1 would drop the usage quite a bit lower.
I should have said 12 threads, not 12 cores.

And it depends on other parameters that also impact parallelization. Is -F 4 really materially faster than -F 3? There is threading overhead, so fps of encoding is more important than saturation. Doing processing at the same time to encoding can serialize things and also negatively impact L3 caching.

[benwaggoner]

So what is the recommended way for higher video-quality when doing ~6 core encoding? -F 1 --pmode or just -F 2?
What resolution? At higher presets at 1080p, just -F 1 is probably pretty close to optimal fps.

[Forteen88]

What resolution? At higher presets at 1080p, just -F 1 is probably pretty close to optimal fps.At 1080p, with --preset slower. Sometimes I do 720p too, but not at all as often.

[benwaggoner]

At 1080p, with --preset slower. Sometimes I do 720p too, but not at all as often.
What fps and utilization are you seeing?

I again caution people to focus more on fps than on utilization. The fastest encoding options for a given quality may be <<100% utilization.

[Boulder]

I did some tests, and in my case it looks like this:

1080p, ctu 32, F 1 - 1.55 fps - 15209.92 kbps
1080p, ctu 32, F 2 - 1.98 fps - 15160.69 kbps
1080p, ctu 32, F 3 - 2.00 fps - 15216.68 kbps
1080p, ctu 32, F 4 - 2.00 fps - 15218.20 kbps

1080p, ctu 64, F 1 - 0.96 fps - 15183.31 kbps
1080p, ctu 64, F 2 - 1.41 fps - 15195.30 kbps
1080p, ctu 64, F 3 - 1.58 fps - 15194.53 kbps
1080p, ctu 64, F 4 - 1.68 fps - 15176.67 kbps

720p, ctu 32, F 1 - 2.95 fps - 6035.36 kbps
720p, ctu 32, F 2 - 3.98 fps - 6038.02 kbps
720p, ctu 32, F 3 - 4.23 fps - 5942.17 kbps
720p, ctu 32, F 4 - 4.29 fps - 6038.01 kbps

I do use --limit-tu 0 and --limit-refs 1, otherwise pretty much close to --preset slower. The Vapoursynth part utilizes ~20-25% of CPU.

[Forteen88]

What fps and utilization are you seeing?

I again caution people to focus more on fps than on utilization. The fastest encoding options for a given quality may be <<100% utilization.I just want to know in general what I should use of those options in x265 using a 6-core CPU (I use TemporalDegrain2(grainLevel=false) avisynth script quite often with that), I never actually used pmode.
I care about power-consumption when doing encodes, but I still want very high quality encodes.

froggy1. Yeah, I always set --limit-tu 0 when I set --preset slower.

EDIT: OK, thanks Waggoner.

[benwaggoner]

I just what to know in general what I should use of those options in x265 using a 6-core CPU (I use TemporalDegrain2-avisynth script quite often with that), I never actually used pmode.
I care about power-consumption when doing encodes, but I still want very high quality encodes.

froggy1. Yeah, I always set --limit-tu 0 when I set --preset slower.--selective-sao 2 is a decent speedup without material quality impact. I am sure it'll be on by default in future presets.

--limit-tu 4 is a better quality/speed tradeoff in my experience.

If you are doing VBV limited and CRF, you can get the same quality at a slightly higher ABR with --rd 4 --dynamic-rd 3. That'll give you full "slower" efficiency at the VBV peak while being faster and a little less efficient when quality isn't limited by VBV.

--pmode is definitely power-inefficient, since it is doing things in parallel that will get thrown away. And often it'll increase utilization AND reduce fps if there aren't enough free cores available. Same with --pme.

Sent from my SM-T837V using Tapatalk

[excellentswordfight]

With an 8-bit source, there's no real reason to encode in 10-bit in x265. Encoding 8-bit will be significantly faster and the output will be more compatible with older mobile devices. Encoding 8-bit in 10-bit provided a measurable benefit with x264, but HEVC is designed to fix the issues in 8-bit that made that valuable with H.264.
Do you have any link for any research done on this? I get noticeable less banding when using main10 on 8bit content with x265. And those legacy devices isnt really an argument, cause if you need to it to be compatible with those devices, then you simply use 8bit... And if not (i.e if its not the target device for playback), why even think about those? 10bit decoders are in majority.

[benwaggoner]

Do you have any link for any research done on this? I get noticeable less banding when using main10 on 8bit content with x265. And those legacy devices isnt really an argument, cause if you need to it to be compatible with those devices, then you simply use 8bit... And if not (i.e if its not the target device for playback), why even think about those? 10bit decoders are in majority.How are you displaying your 10-bit video? Since truly 10 bit panels are pretty rare, encoding in 10-bit could well trigger some dithering mode that 8-bit doesn't go through. That would be a difference in postprocessing instead of encoding, and would be quite dependent on the playback system. Other times 10-bit can look worse due to LSB truncation or other bad conversion.

If you're only playing back on devices that do better with 10-bit and don't care about the speed difference, of course using 10-bit for 8-bit can make perfect sense. My focus is on how to make content look good across thousands of different devices (thanks, Android), including future ones.

Sent from my SM-T837V using Tapatalk

[Forteen88]

...
--pmode is definitely power-inefficient, since it is doing things in parallel that will get thrown away. And often it'll increase utilization AND reduce fps if there aren't enough free cores available. Same with --pme.Thanks. Is --frame-threads 1 more power-efficient compared to --frame-threads 2?

[excellentswordfight]

How are you displaying your 10-bit video? Since truly 10 bit panels are pretty rare, encoding in 10-bit could well trigger some dithering mode that 8-bit doesn't go through. That would be a difference in postprocessing instead of encoding, and would be quite dependent on the playback system. Other times 10-bit can look worse due to LSB truncation or other bad conversion.
I'm aware that 10bit panels are rare, and that all file based SDR media I watch is converted to 8bit RGB eitherway. Still, from the encoding i've done with x265, using main10 has given better results.

So here is a grab from ToS, one from an 8bit encode and one 10bit at 3Mbps, decoded with avisynth (FFMS2 source and converted to 8bit RGB with ConvertToRGB24)
8bit
https://ibb.co/7Qbxxx5
10bit
https://ibb.co/ChLFrTB
source
https://ibb.co/zxC48x6

If you're only playing back on devices that do better with 10-bit and don't care about the speed difference, of course using 10-bit for 8-bit can make perfect sense. My focus is on how to make content look good across thousands of different devices (thanks, Android), including future ones.
And ofc it makes sense to have greater compatibility when encoding for thousands of different devices... But I assume from TS request that he isnt re-encoding for amazon :)

[benwaggoner]

Thanks. Is --frame-threads 1 more power-efficient compared to --frame-threads 2?
I'd think so. It's a somewhat complex and empirical question, because a computer uses X power just doing nothing. More threading is generally more overhead and thus MIPS/watt goes down. But that'd delta watts from the minimum power state. So the best fps/watt is going to depend on specifics of settings, content, and system.

The answer will also be different between running a single encode and multiple encodes at the same time. Reducing threading with multiple instances is more beneficial.

[benwaggoner]

I'm aware that 10bit panels are rare, and that all file based SDR media I watch is converted to 8bit RGB eitherway. Still, from the encoding i've done with x265, using main10 has given better results.

So here is a grab from ToS, one from an 8bit encode and one 10bit at 3Mbps, decoded with avisynth (FFMS2 source and converted to 8bit RGB with ConvertToRGB24)
8bit
https://ibb.co/7Qbxxx5
10bit
https://ibb.co/ChLFrTB
source
https://ibb.co/zxC48x6
Yeah, I do see a little more detail here in the 10-bit. Of course, ToS is available in 16-bit RGB; which version did you use? Even an 8-bit RGB has more information than an 8-bit limited range YUV, as it is 4:4:4 0-255 going to 4:2:0 16-235. I was really talking about encoding from 8-bit limited range YUV being fine with 8-bit encoding.

Is the difference visible in motion?

[excellentswordfight]

Yeah, I do see a little more detail here in the 10-bit. Of course, ToS is available in 16-bit RGB; which version did you use? Even an 8-bit RGB has more information than an 8-bit limited range YUV, as it is 4:4:4 0-255 going to 4:2:0 16-235. I was really talking about encoding from 8-bit limited range YUV being fine with 8-bit encoding.

Is the difference visible in motion?
The source is 8bit limited, I have encoded an 1080p bluray compatable version from the 4k y4m version that I use.

[benwaggoner]

The source is 8bit limited, I have encoded an 1080p bluray compatable version from the 4k y4m version that I use.
Interesting and intriguing! That's a well designed test you did. I'll try to run some of my own with other content and see if I can replicate.