x264 Lossless question

[Dark Shikari]

Cool thanks Dark!

Well I did the YUV Dump anyway and got a big file (which is obvious) but I have no idea how to load it into AviSynth, sorry for the OT, but any ideas?No need, as the problem has already been pinpointed and resolved. It was an x264 bug.

(You'd use RawSource if you wanted to load it into AviSynth).

[Aktan]

Yep I already knew you've already pinpointed and resolved the issue, I was just curious as to how to load it, which you've told me how. Thanks!

[damian101]

Ultra-fast: --subme 1 --no-cabac --partitions none --me dia

Normal: --subme 5 --partitions all

Slow: --subme 6 --8x8dct --partitions all --ref 2

Slower: --subme 8 --8x8dct --partitions all --ref 4 --me umh --mixed-refs

Slowest: --subme 8 --8x8dct --partitions all --ref 16 --me esa --mixed-refs

Note that no-cabac loses about 10-15% compression but massively speeds up playback. Other things that hurt compression the most are subme=0 (vs other values) and partitions=none.

--partitions all is usually harmful on noisy camera content, especially with subme < 6

Also, --8x8dct should be default anyway with all presets except ultrafast.

[benwaggoner]

In her defense, Dark Shikari's answer was from over fifteen years ago.

[Tlen]

Just for the records,
because i had to make a lossless compression of a 4 hours film
I always used Lagarith as default,
so i tried x264 lossless to see what happens.

4 Hour, 1920 * 1080 YV12

x264 (qp 0 very slow)
210 GB @ 13.4 fps
Decoding speed ok (no frame lost) on a Low TDP CPU (i5) but CPU stuck at 100%

Lagarith
264 GB @ 120 fps
Decoding speed ok (no frame lost) on a Low TDP CPU (i5) - CPU 88%

Ssso I think that the only pro of x264 would be a compatibility side
(you can use Lag only on PC while you could teorically be able to run x264 on a NAS too)
and a little better compression but that costs an incredibly huge amount of time.

So all in all Lags remains the choice (generally).

[FranceBB]

Lagarith is ok but it's getting older and it doesn't support high bit depth which you don't need in this case as you're working in yv12 but if you'll ever need it then you're gonna be out of luck.
x264 is of course much slower in encoding time, but it does compress better and it also supports 10bit, while x265 supports up to 12bit and it's gonna be much slower again.
As to x264 vs x265, going back to summer 2013, the x265 lossless mode wasn't very optimized and it sometimes produced larger files compared to x264.
A lot of time has passed since then (11 years) so it probably improved but I haven't done a significant benchmark for a long time (although I feel like I should).
Another one that supports up to 10bit is UTVideo which has been a good alternative for a while.
If you need something that supports high bit depth up to and including 16bit planar, you can use HuffYUV which is my de facto choice for lossless encoding nowadays.
HuffYUV is still maintained and it's ridiculously fast on modern CPUs to both encode and decode.
If you want something that has the same bit depth support as HuffYUV (i.e up to 16bit) but has a better compression efficiency there's always FFV1 which is very very good and has CRC checks per slice.
I'll probably make a new comparison if I have time.

[jpsdr]

HuffYUV is still maintained...? I'll search... I thought it stopped a looong time ago at version 2.1.1
There is also MagicYUV.

[Tlen]

I remember that huffyuv was way beyond compression ratio than lagarith although very much faster,
but for archival needs, compression ratio has more weight in the equation.

I remember testing FFV1 long time ago and not remembering why i discarded ahaha
Same as MagicYUV which i tested but didn't choose.

Next time pheraps I'll re-evaluate MagicYUV which I vaguely remember had good ratio and high speed.

Concerning high bit depth, yes atm I don't deal with that so it's out of the requirements.

[benwaggoner]

I remember that huffyuv was way beyond compression ratio than lagarith although very much faster,
but for archival needs, compression ratio has more weight in the equation.

I remember testing FFV1 long time ago and not remembering why i discarded ahaha
Same as MagicYUV which i tested but didn't choose.

Next time pheraps I'll re-evaluate MagicYUV which I vaguely remember had good ratio and high speed.

Concerning high bit depth, yes atm I don't deal with that so it's out of the requirements.


IDR-only HEVC lossless is more efficient than anything you’ve listed, and has HW decode support.


Sent from my iPhone using Tapatalk

[FranceBB]

HuffYUV is still maintained...? I'll search... I thought it stopped a looong time ago at version 2.1.1

yeah, the original version did stop at 2.1.1 and it's 8bit only, but it got forked several years ago and led to the creation of ffvhuff which supports 8bit, 10bit, 12bit, 14bit and 16bit planar and it's still maintained to this very day. ;)

[Tlen]

IDR-only HEVC lossless is more efficient than anything you’ve listed, and has HW decode support.

I don't doubt it will be probably more efficient in terms of pure compression ratio.

But much probably it will be even slower than x264, so at present day
pheraps it's not an option, cause of the cpu / time cost,
unless the compression gain is so huge that can balance the cost.

Out of curiosity, command line for x265 lossless?

[excellentswordfight]

I remember that huffyuv was way beyond compression ratio than lagarith although very much faster,
but for archival needs, compression ratio has more weight in the equation.

I remember testing FFV1 long time ago and not remembering why i discarded ahaha
Same as MagicYUV which i tested but didn't choose.

Next time pheraps I'll re-evaluate MagicYUV which I vaguely remember had good ratio and high speed.

Concerning high bit depth, yes atm I don't deal with that so it's out of the requirements.
Just FYI it can be worth testing x264 in a faster preset, in my experience there isnt a drastic difference in filesize/compression in lossless mode between the presets.

Did a tests a few years back comparring a few lossless codecs (only tested codecs bundled with ffmpeg):

Did a quick test a year or two ago with a 1min sample from an 1080p24 bluray:

Original: 179 MiB
x264 (preset veryslow, crf 0): 733 MiB
x264 (preset fast, crf 0): 765 MiB
x265 (preset veryslow, lossless): 808 MiB
FFV1: 831 MiB
JPEG2000: 932 MiB
x265 (preset fast, lossless): 934 MiB
Ut Video: 1.30 GiB
HUFFYUV: 1.87 GiB
Uncompressed: 4.17GiB

Not sure if x265 lossless mode is "fixed", but when I tried it, it was worse than x264.

[poisondeathray]

Just for the records,
because i had to make a lossless compression of a 4 hours film
I always used Lagarith as default,
so i tried x264 lossless to see what happens.

4 Hour, 1920 * 1080 YV12

x264 (qp 0 very slow)
210 GB @ 13.4 fps
Decoding speed ok (no frame lost) on a Low TDP CPU (i5) but CPU stuck at 100%

Lagarith
264 GB @ 120 fps
Decoding speed ok (no frame lost) on a Low TDP CPU (i5) - CPU 88%

Ssso I think that the only pro of x264 would be a compatibility side
(you can use Lag only on PC while you could teorically be able to run x264 on a NAS too)
and a little better compression but that costs an incredibly huge amount of time.

So all in all Lags remains the choice (generally).


For the 8bit 4:2:0 Lossless AVC scenario - FYI lossless profile is HW accelerated supported by NVDec . A fantastic option for 8bit 4:2:0 IMO, high compression ratio, fast decode, low CPU .



Out of curiosity, command line for x265 lossless?

For x265 cli

--lossless

[Tlen]

Just FYI it can be worth testing x264 in a faster preset

This could be an interesting experiment.

If the the ultrafast (for example) gives me the same performance in terms of ratio/time computing compared to Lagarith, it would have the added benefit of compatibility and (suposedly) hardware accelleration.

Regarding to hw accelleration, MPCH with LavFilter reports that the i5 i'm using would be capable of h264 HW decoding,

but when i decoded the Lossless h264 Cpu was 100% stuck.
If it was really hw decoding capable i would expect totally differrent numbers.

Is it a separate accelleration for lossless?
Or there is problem with the Lav capabilities reporting?

lossless profile is HW accelerated supported by NVDec
Yes, but in my (and i think many) situation the video is played on a system without a dedicated GPU
but a shared soldered GPU (not Nvidia) and not with NVDec.

x265 cli

Thank you

[huhn]

while lossless h264 and h265 are hardware decodable you still need a decoder that actually supports that. there is a lot of codec that are hardware decodable but not added.

as far as i know lavfilter can't do that (have not checked this in years). maybe it was added for nvdec.
try mpv.

and only nvidia.

[Tlen]

Here's an update on compression tests

2hours 16min film 1920*1080 YV12

Lagarith
116 Gigs @ 140fps -> Decode (i5) 65% CPU

x264 Ultrafast
130 Gigs @240 fps -> Decode (i5) 45% CPU

x264 Almost 2 times faster with little difference in size and less CPU. Amazing!

I think i found the replacement.

Thank you excellentswordfight for the suggestion!

Ps.
What I don't understand is why LAV filter is hw accelerated (10% CPU on average) with standard x264 compression (ie. Bluray)
and not accelerated with lossless compression.

[poisondeathray]

Ps.
What I don't understand is why LAV filter is hw accelerated (10% CPU on average) with standard x264 compression (ie. Bluray)
and not accelerated with lossless compression.

Apparently Intel quicksync does not support lossless AVC decoding - this was confirmed by Intel developer in their forums . So you are using CPU in that case

[Tlen]

@poisondeathray
I got the reply in the LAV thread.
It seems lossless encoding works in a completely different way than lossy and is not implemented in (any?) chip, or just Intel?

[huhn]

as i said only nvidia has added it and that doesn't mean all decoder can use it.

it's a special profile.
nvidia can even hardware encode lossless.

[jpsdr]

Out of curiosity. Is there an FFV1 codec install like there is for UTVideo ?
Allowing use under VirtualDub.

[benwaggoner]

I don't doubt it will be probably more efficient in terms of pure compression ratio.
IDR only is maybe 30% higher bitrate for typical natural image content. The savings from inter frame compression are proportional to how noisy the content is, so really grainy stuff may not increase bitrate much at all, but clean computer-rendered cel animation might need 3x the bitrate for lossless.

To get the minimum possible lossless file size, x265 --lossless --preset placebo is the best you can get today. However, it is slow.

placebo doesn't make sense for most lossy encoding, it makes a bigger difference for lossless. I've seen placebo come out at a 10% smaller file size than veryslow.

But much probably it will be even slower than x264, so at present day perhaps it's not an option, cause of the cpu / time cost,
unless the compression gain is so huge that can balance the cost.
IDR-only is a lot faster than interframe encoding, particularly on systems with lots of cores. You can use an arbitrarily high frame threads count as there's no rate control or frame comparisons needed in this use case; each frame is an entirely independent encode. x265 can useful use more cores per frame than x264 can, so the speed delta might not be that bad, and I wouldn't be surprised if you can encode with x265 faster than x264 aiming for the same file size (x265 --preset medium should still deliver lower sizes than x264 --preset placebo)

Speed versus file size tradeoffs are really for you to figure out for yourself; it's complicated.

If you're worried about decode speed, lossless is a mandatory feature of x265 Main and Main10 profiles, and so you get HW decoder support on modern systems. x264 lossless isn't always accelerated (although is somewhat faster to decode in software).

I am not sure which if any HW encoders support lossless. If you can do it with an Nvidia GPU, you could get very fast encoding, but presumably with some bigger file size.

Out of curiosity, command line for x265 lossless?
You just add --lossless. Setting that causes a whole lot of other parameters to be ignored, like profile, level, crf, bitrate. Key remaining ones are --keyint and --profile. If not doing IDR I strongly recommend using a shorter --keyint than the default 250. It can take a really long time to decode all those huge frames in the reference structure for frame 247.

Full command line documentation is here: https://x265.readthedocs.io/en/master/cli.html

And some usefully in depth info about lossless encoding is here: https://x265.readthedocs.io/en/master/lossless.html

[damian101]

Not sure if x265 lossless mode is "fixed", but when I tried it, it was worse than x264.

I always assumed that x265's quite bad lossless performance was format-related. After all, H.264 has its own lossless profile, while H.265 does not. But a significant benefit for H.265 here is that a lossless H.265 stream should be decodable by hardware decoders.

[damian101]

Actually, it should (also?) be encoder-related to at least a significant degree. Because, iirc, preset superfast gave me the best lossless compression with x265, which is obviously not how it should be.

[damian101]

Actually, it should (also?) be encoder-related to at least a significant degree. Because, iirc, preset superfast gave me the best lossless compression with x265, which is obviously not how it should be.

Not true anymore. Apparently, preset medium has been optimized.

Also, I'm pretty sure preset superfast didn't give me "the best lossless compression", just the best that wasn't extremely slow.

But x264 still easily beats x265 at much higher speed.

[SeeMoreDigital]

But a significant benefit for H.265 here is that a lossless H.265 stream should be decodable by hardware decoders.Really!

[poisondeathray]

I found an 8bit420 example where x265 lossless outperforms x264 lossless - foreman_cif.y4m . It might be related to the cif resolution. Exceedingly rare for this to happen

x264 keyint 500 , preset veryslow
15,948,130 bytes

x265 keyint 500 , preset veryslow
15,917,148 bytes

yuvsoft
15,475,020 bytes

[SeeMoreDigital]

Actually, it should (also?) be encoder-related to at least a significant degree. Because, iirc, preset superfast gave me the best lossless compression with x265, which is obviously not how it should be.Really!I can confirm it's true!

Using VirtualDub2 I generated a couple of lossless HEVC encodes in 8-bit and 10-bit and they played fine on my old 2016 LG television.

[Balling]

I can confirm it's true!

Using VirtualDub2 I generated a couple of lossless HEVC encodes in 8-bit and 10-bit and they played fine on my old 2016 LG television.


Like this? Found this on internet, 1 deprecated option, it is not lossless though:

ffmpeg -i through-the-cracks.mp4 -c:v hevc_nvenc -pix_fmt yuv444p16 -profile:v main10 -preset slow -global_quality 6 -g 0 -c:a copy cwqasdcwsa1.mp4


BTW, there was a recent fix with all intra https://github.com/FFmpeg/FFmpeg/commit/c9151ea50715c4ce47ad1c8df519781565db01f6