So... x265 can't match x264 for lossless encoding?

[gdgsdg123]

No matter how hard I try...



"test_x265.mkv": 254,612 KB
ffmpeg -i "%X%" -c:v libx265 -preset placebo -x265-params "lossless=1:keyint=15:no-deblock=1:no-sao=1" -flags +bitexact -fflags +bitexact -map_metadata -1 "test_x265.mkv"

"test_x265_no-strong-intra-smoothing.mkv": 254,614 KB
ffmpeg -hide_banner -i "%X%" -c:v libx265 -preset placebo -x265-params "lossless=1:keyint=15:no-deblock=1:no-sao=1:no-strong-intra-smoothing=1" -flags +bitexact -fflags +bitexact -map_metadata -1 "test_x265_no-strong-intra-smoothing.mkv"

"test_x265_X.mkv": 254,535 KB
ffmpeg -hide_banner -i "%X%" -c:v libx265 -preset placebo -x265-params "lossless=1:keyint=15:me=4:subme=7:no-deblock=1:no-sao=1:no-strong-intra-smoothing=1" -flags +bitexact -fflags +bitexact -map_metadata -1 "test_x265_X.mkv"

"test_x264.mkv": 249,922 KB
ffmpeg -hide_banner -i "%X%" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1" -flags +bitexact -fflags +bitexact -map_metadata -1 "test_x264.mkv"

"test_x264_weightp=0.mkv": 250,526 KB
ffmpeg -hide_banner -i "%X%" -c:v libx264 -preset placebo -qp 0 -x264-params "keyint=15:no-deblock=1:weightp=0" -flags +bitexact -fflags +bitexact -map_metadata -1 "test_x264_weightp=0.mkv"

"test_x265_XX.mkv": 277,259 KB
ffmpeg -hide_banner -i "%X%" -c:v libx265 -preset placebo -x265-params "lossless=1:keyint=15:bframes=0:ref=8:me=4:subme=7:no-deblock=1:no-sao=1:no-strong-intra-smoothing=1" -flags +bitexact -fflags +bitexact -map_metadata -1 "test_x265_XX.mkv"

"test_x265_XXrev.mkv": 277,276 KB
ffmpeg -hide_banner -i "%X%" -c:v libx265 -preset placebo -x265-params "lossless=1:keyint=15:bframes=0:b-adapt=0:no-weightb=1:no-b-intra=1:ref=8:me=4:subme=7:no-deblock=1:no-sao=1:no-strong-intra-smoothing=1" -flags +bitexact -fflags +bitexact -map_metadata -1 "test_x265_XXrev.mkv"

"test_x265_XXFinal.mkv": 277,266 KB
ffmpeg -hide_banner -i "%X%" -c:v libx265 -preset placebo -x265-params "lossless=1:keyint=15:no-cutree=1:psy-rd=0:rdoq-level=0:bframes=0:ref=8:me=4:subme=7:no-deblock=1:no-sao=1:no-strong-intra-smoothing=1" -flags +bitexact -fflags +bitexact -map_metadata -1 "test_x265_XXFinal.mkv"

"test_x265_Finale.mkv": 277,259 KB
ffmpeg -hide_banner -i "%X%" -c:v libx265 -preset placebo -x265-params "qp=0:lossless=1:keyint=15:bframes=0:ref=8:me=4:subme=7:no-deblock=1:no-sao=1:no-strong-intra-smoothing=1" -flags +bitexact -fflags +bitexact -map_metadata -1 "test_x265_Finale.mkv"

"test_x265_FinaleX.mkv": 254,211 KB
ffmpeg -hide_banner -i "%X%" -c:v libx265 -preset placebo -x265-params "lossless=1:keyint=15:bframes=16:ref=16:me=4:subme=7:no-deblock=1:no-sao=1" -flags +bitexact -fflags +bitexact -map_metadata -1 "test_x265_FinaleX.mkv"

"test_x265_FinaleFinal.mkv": 254,335 KB
ffmpeg -hide_banner -i "%X%" -c:v libx265 -preset placebo -x265-params "allow-non-conformance=1:lossless=1:keyint=15:bframes=16:ref=16:me=4:no-deblock=1:no-sao=1" -flags +bitexact -fflags +bitexact -map_metadata -1 "test_x265_FinaleFinal.mkv"

Full command log attached.



Out of ideas... Any suggestion?

[gdgsdg123]

And mediainfo comparison on some of the outputs.



refine-analysis-type?.. what does this thing do?
(BTW "diff x265_Finale x265_XX.png" may shed some light on this (https://forum.doom9.org/showthread.php?p=1847890))

[nevcairiel]

Lossless is a rather special case, since thats not the primary usage target of codecs like H.264 or H.265, so many of their optimizations just don't apply.
You can probably get a much better result using a dedicated lossless codec.

[gdgsdg123]

Lossless is a rather special case, since thats not the primary usage target of codecs like H.264 or H.265, so many of their optimizations just don't apply.
You can probably get a much better result using a dedicated lossless codec.

While there's not that much can be done for optimizations in the lossless case... after all.

Dedicated lossless codec?.. Anything better than x264? Haven't noticed one though...

[Sharc]

Dedicated lossless codec?.. Anything better than x264? Haven't noticed one though...
Huffyuv, Lagarith, UTvideo to name a few.

[benwaggoner]

Huffyuv, Lagarith, UTvideo to name a few.
x265 will absolutely outperform Huffyuv and Lagarith! Interframe compression alone helps a bunch.

And it should be at least a little better than x264. Can you share the x264 command line and results?

[Sharc]

x265 will absolutely outperform Huffyuv and Lagarith! Interframe compression alone helps a bunch.
I mentioned these as examples for "dedicated" lossless codecs.
Isn't lossless = lossless? So x265, x264 lossless outperform the others in terms of filesize if this is taken as the performance criterion, as I understand.

[excellentswordfight]

I have always used ffv1 for lossless encoding. Never done an Apple to apples comparison to x264/x265 though. But seen some tests were it event beats x264, even though its far faster (but i would take that with some salt). But might be worth checking out.

[Stereodude]

Huffyuv, Lagarith, UTvideo to name a few.
Are you sure about that? Lossless x264 (--CRF 0 veryslow) is smaller than MagicYUV 2.0 (on 4:2:0 8-bit) which I found to be better than HuffYUV and Lagarith in prior comparisons.

Edit: Lagarith was smaller than MagicYUV on my 720x476 SD test clip, but x264 is smaller still. x265 (--lossless -p veryslow) was bigger than x264

Lagarith: 4.45GB (in AVI)
MagicYUV: 4.82GB (in AVI)
x264: 3.68GB (elementary stream)
x265: 3.81GB (elementary stream)

[benwaggoner]

I mentioned these as examples for "dedicated" lossless codecs.
Isn't lossless = lossless? So x265, x264 lossless outperform the others in terms of filesize if this is taken as the performance criterion, as I understand.



Yeah, it is hard to see how a codec that can do interframe compression wouldn’t beat a intraframe-only codec. HEVC in particular has intraframe prediction and other advanced tools that should squeeze even more bits out.

They aren’t all on even in —preset placebo, though! Using —cu-lossless and —tskip can help improve efficiency and quality of near-lossless encoding. Cu-lossless allows any given CTU to use lossless mode if it is more efficient; real lossless is just using that for the whole frame.

[Stereodude]

Updated post 9 with the x265 results. It is bigger than the x264 encode.

[benwaggoner]

Updated post 9 with the x265 results. It is bigger than the x264 encode.


Intriguing. Can you share the full command line and the console output? Something interesting is happening here.

[poisondeathray]

I did about 30 tests over the last few years (different sources, different genres), 100% of the time , x264 lossless compresses better than x265 lossless (at least for 8bit 4:2:0) with default settings. Typically a few % better .

FFV1 (in interframe mode) and x264 typically have the higher compression ratio . If source has identical content, duplicate frames, then lagarith can be better (null frame option)

[Stereodude]

Intriguing. Can you share the full command line and the console output? Something interesting is happening here.
x265:

x265.exe -f 34164 -p veryslow --lossless --keyint 150 --open-gop -D 8 --colorprim "smpte170m" --transfer "smpte170m" --colormatrix "smpte170m" --sar 32:27...

avs2pipemod[info]: writing 34164 frames of 30000/1001 fps, 720x476,
sar 0:0, YUV-420-planar-8bit progressive video.
y4m [info]: 720x476 fps 30000/1001 i420p8 sar 32:27 unknown frame count
raw [info]: output file: x265_ll_test_2.0.265
x265 [info]: HEVC encoder version 2.9+15-81373aab81df
x265 [info]: build info [Windows][MSVC 1916][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main profile, Level-8.5 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 4 / wpp(8 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 4 / 4
x265 [info]: Keyframe min / max / scenecut / bias: 15 / 150 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 40 / 8 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 5 / off / on
x265 [info]: Rate Control : Lossless
x265 [info]: tools: rect amp limit-modes rd=6 psy-rd=2.00 rdoq=2 psy-rdoq=1.00
x265 [info]: tools: rskip limit-tu=4 signhide tmvp b-intra
x265 [info]: tools: strong-intra-smoothing deblock sao
avs2pipemod[info]: finished, wrote 34164 frames [100%].a 0:00:22
avs2pipemod[info]: total elapsed time is 12722.433 sec.
x265 [info]: frame I: 534, Avg QP:4.00 kb/s: 34259.39
x265 [info]: frame P: 9718, Avg QP:4.00 kb/s: 32024.29
x265 [info]: frame B: 23912, Avg QP:4.00 kb/s: 27268.91
x265 [info]: Weighted P-Frames: Y:11.5% UV:9.0%
x265 [info]: Weighted B-Frames: Y:22.2% UV:17.3%
x265 [info]: consecutive B-frames: 15.8% 12.6% 25.7% 26.8% 8.0% 10.4% 0.5% 0.1% 0.1%
x265 [info]: lossless compression ratio 2.86::1

encoded 34164 frames in 12752.24s (2.68 fps), 28730.85 kb/s, Avg QP:4.00

x264:

x264.2935.x64.exe" --crf 0 --preset veryslow --tune film --threads 12 --lookahead-threads 2 --aq-strength 1.2 --keyint 150 --open-gop --colorprim "smpte170m" --transfer "smpte170m" --colormatrix "smpte170m" --sar 32:27...

avs [info]: 720x476p 32:27 @ 30000/1001 fps (cfr)
x264 [info]: using SAR=32/27
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x264 [info]: profile High 4:4:4 Predictive, level 4.0, 4:2:0, 8-bit
x264 [info]: frame I:1421 Avg QP: 0.00 size:138568
x264 [info]: frame P:32743 Avg QP: 0.00 size:114868
x264 [info]: mb I I16..4..PCM: 52.7% 0.0% 47.3% 0.0%
x264 [info]: mb P I16..4..PCM: 29.6% 0.0% 21.6% 0.0% P16..4: 34.1% 5.0% 7.4% 1.3% 0.8% skip: 0.3%
x264 [info]: 8x8 transform intra:0.0% inter:61.4%
x264 [info]: coded y,uvDC,uvAC intra: 99.6% 94.6% 94.5% inter: 96.9% 78.3% 78.3%
x264 [info]: i16 v,h,dc,p: 40% 25% 32% 4%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 42% 28% 10% 4% 4% 4% 3% 3% 2%
x264 [info]: i8c dc,h,v,p: 18% 27% 50% 5%
x264 [info]: Weighted P-Frames: Y:19.7% UV:15.4%
x264 [info]: ref P L0: 83.0% 4.8% 7.2% 2.1% 0.7% 0.5% 0.5% 0.2% 0.2% 0.1% 0.1% 0.1% 0.1% 0.1% 0.1% 0.1%
x264 [info]: kb/s:27777.17

encoded 34164 frames, 42.93 fps, 27777.17 kb/s

[excellentswordfight]

I have always used ffv1 for lossless encoding. Never done an Apple to apples comparison to x264/x265 though. But seen some tests were it event beats x264, even though its far faster (but i would take that with some salt). But might be worth checking out.
Did a quick test on a 1min 1080p Bluray sample:

Original: 179 MiB
x264 (preset veryslow, crf 0): 733 MiB
x264 (preset fast, crf 0): 765 MiB
x265 (preset veryslow, lossless): 808 MiB
FFV1: 831 MiB
JPEG2000: 932 MiB
x265 (preset fast, lossless): 934 MiB
Ut Video: 1.30 GiB
HUFFYUV: 1.87 GiB
Uncompressed: 4.17GiB

[Asilurr]

The same seems to happen with libvpx, it just got outmatched on a small "DVD rip" sample (4:3 576p50, 8-bit 4:2:0, ~700 frames):
x264 --preset veryslow --qp 0 || relative filesize 100.0%
x264 --preset fast --qp 0 || 100.6%
vpxenc --cpu-used 0 --qmin 0 --qmax 0 || 104.1%
vpxenc --cpu-used 3 --qmin 0 --qmax 0 || 116.6%

Frankly, the gaps are reminiscent of ESF's results above. Both x265 and libvpx use at least ~5% more bitrate than x264 in lossless scenarios, apparently.

[kolak]

Did a quick test on a 1min 1080p Bluray sample:

Original: 179 MiB
x264 (preset veryslow, crf 0): 733 MiB
x264 (preset fast, crf 0): 765 MiB
x265 (preset veryslow, lossless): 808 MiB
FFV1: 831 MiB
JPEG2000: 932 MiB
x265 (preset fast, lossless): 934 MiB
Ut Video: 1.30 GiB
HUFFYUV: 1.87 GiB
Uncompressed: 4.17GiB


Try 5 different source files with different nature.

[Stereodude]

Try 5 different source files with different nature.
Because everyone who has tested it so far with the same results just have happened to find the right footage where it fails to even match x264?

[excellentswordfight]

Try 5 different source files with different nature.
Did you see any claims in my post? No, just a quick test, with no opinion in regards to whats better then the other cause I know that there isnt enough data to do so. If you wanna do an more conclusive test go ahead and do it yourself.

But atm, it's not looking too promising for x265 for lossless, and thats even if you take speed out of the equation.

[kolak]

I'm not saying h265 has to be better. If it was confirmed by different people using random sources then then it's what it's. Lossless compression is special case for codecs like h264/5, so I'm somehow not specially surprised about end results.

[Asilurr]

A couple of tests on video content generated from still images.

#1 Lenna (preview: wikipedia.org/wiki/File:Lenna_(test_image).png; download: lenna.org), 512x512 RBG24 converted to 8-bit 4:4:4.
x264vs || relative filesize 100.0%
x265vs || 104.7%

#2 M87* (preview: wikipedia.org/wiki/File:Black_hole_-_Messier_87.jpg; download: eso.org/public/archives/images/original/eso1907a.tif), 7416x4320 RGB48 converted to 3708x2160 10-bit 4:4:4. Do note that the original TIFF is a gargantuan 183 MB file.
x264vs || 100.0%
x264f || 100.8%
x265vs || 103.2%
x265f || 104.2%

[gdgsdg123]

HEVC in particular has intraframe prediction and other advanced tools that should squeeze even more bits out.

They aren’t all on even in —preset placebo, though! Using —cu-lossless and —tskip can help improve efficiency and quality of near-lossless encoding. Cu-lossless allows any given CTU to use lossless mode if it is more efficient; real lossless is just using that for the whole frame.
The docs (https://x265.readthedocs.io/en/default/index.html) give somewhat different opinion.





About --tskip:

Lossless — x265 documentation (https://x265.readthedocs.io/en/default/lossless.html)

Transform Skip
A somewhat related feature, --tskip tells the encoder to evaluate transform-skip (bypass DCT but with quantization still enabled) when coding small 4x4 transform blocks. This feature is intended to improve the coding efficiency of screen content (aka: text on a screen) and is not really intended for lossless coding. This feature should only be enabled if the content has a lot of very sharp edges in it, and is mostly unrelated to lossless coding.


About --cu-lossless:

Command Line Options — x265 documentation (https://x265.readthedocs.io/en/default/cli.html)

--cu-lossless, --no-cu-lossless
For each CU, evaluate lossless (transform and quant bypass) encode of the best non-lossless mode option as a potential rate distortion optimization. If the global option --lossless has been specified, all CUs will be encoded as lossless unconditionally regardless of whether this option was enabled. Default disabled.

Only effective at RD levels 3 and above, which perform RDO mode decisions.



And 1 thing I must emphasize... near-lossless = lossy.

[foxyshadis]

Heavily filtered video might sometimes benefit from tskip, but natural video never will. And since x265 doesn't even implement the screen coding range extensions, the actual capabilities of tskip and other screen coding tunings are very nerfed.

[pookpooi]

The rumored Sony's new video codec XEVC which is based on HEVC version 2 claims 240Mbps peak bitrate lossless compressing 1920 x 1080, 12 bit depth, 24fps. I did a calculation and get 1:7.12 ratio.

source: https://www.eoshd.com/2018/12/sonys-next-codec-is-xevc-could-blow-the-doors-off-the-industry-with-8k-raw-at-240mbit/

[benwaggoner]

The rumored Sony's new video codec XEVC which is based on HEVC version 2 claims 240Mbps peak bitrate lossless compressing 1920 x 1080, 12 bit depth, 24fps. I did a calculation and get 1:7.12 ratio.

source: https://www.eoshd.com/2018/12/sonys-next-codec-is-xevc-could-blow-the-doors-off-the-industry-with-8k-raw-at-240mbit/
Visually lossless at that rate is certainly feasible, but mathematically lossless is impossible unless they're doing some sort of prefiltering.

[gdgsdg123]

FFV1 (in interframe mode)...
Hmm?..

FFV1 Video Codec Specification (https://www.ffmpeg.org/~michael/ffv1.html#introduction)

The FFV1 video codec is a simple and efficient lossless intra-frame only codec.




If source has identical content, duplicate frames, then lagarith can be better (null frame option)
I doubt it.

As Lagarith is sort of an old outdated codec... And more advanced codecs (like x264, x265) should have implemented similar capabilities in the algorithms.

[gdgsdg123]

If source has identical content, duplicate frames, then lagarith can be better (null frame option)
You might be correct... See this (https://forum.videohelp.com/threads/393903).

[poisondeathray]

Hmm?..


FFV1 Video Codec Specification

The FFV1 video codec is a simple and efficient lossless intra-frame only codec.



Scroll down


4.8.17 intra

intra indicates the relationship between frames. Inferred to be 0 if not present.
value relationship
0 frames are independent or dependent (keyframes and non keyframes)
1 frames are independent (keyframes only)
Other reserved for future use



https://trac.ffmpeg.org/wiki/Encode/FFV1
GOP size -g integer >= 1 For archival use, GOP-size should be "1".









I doubt it.

As Lagarith is sort of an old outdated codec... And more advanced codecs (like x264, x265) should have implemented similar capabilities in the algorithms.

Yes, but null frames is a special case with pure duplicates . And it's an intra codec, usually compression is worse on typical content


eg.

blankclip(length=1000, width=1280,height=720, pixel_type="YV12")

ie. 1001 blank frames, 1280x720 , 24fps, YUV 4:2:0

x264 --qp 0 mp4 container => 49.2kb
x264 --qp 0 --keyint 1001 mp4 container => 48.6kb
lagarith with null frames, avi container => 31.4kb



You might be correct... See this (https://forum.videohelp.com/threads/393903).


I have no idea what you're trying to say there

Maybe try google translate

[gdgsdg123]

Hmm?..

FFV1 Video Codec Specification (https://www.ffmpeg.org/~michael/ffv1.html#introduction)

The FFV1 video codec is a simple and efficient lossless intra-frame only codec.
Scroll down

4.8.17 intra (https://www.ffmpeg.org/~michael/ffv1.html#intra)
intra indicates the relationship between frames. Inferred to be 0 if not present.

value relationship
0    frames are independent or dependent (keyframes and non keyframes)
1    frames are independent (keyframes only)
Other reserved for future use


https://trac.ffmpeg.org/wiki/Encode/FFV1
GOP size -g integer >= 1 For archival use, GOP-size should be "1".
I doubt such behavior matches the definition of intra-frame (https://en.wikipedia.org/wiki/Intra-frame_coding) only...





You might be correct... See this (https://forum.videohelp.com/threads/393903).
I have no idea what you're trying to say there

Maybe try google translate
Another time then...

Hint: Those numbers are file sizes in Bytes.

[benwaggoner]

You might be correct... See this (https://forum.videohelp.com/threads/393903).
Although skip frames in the elementary stream are also possible in H.265 and HEVC. I'm not sure if x265 will use those with IDR-only encoding. It could be a useful option to add.

[gdgsdg123]

Although skip frames in the elementary stream are also possible in H.265 and HEVC. I'm not sure if x265 will use those with IDR-only encoding. It could be a useful option to add.
Merging such duplicates would be almost the most obvious way in exploiting redundancy.

And for people who came up with: general compression, motion estimation, psycho-visual optimization... such state-of-the-art stuffs. There's no possibility that they ain't aware of this.



So?..

On x264's Handling regards Consecutive Duplicate Frames (https://forum.videohelp.com/threads/393903)

Conclusion

Absurd, isn't it?
Bewildered too at first... But then soon sort of understood its design language behind this seeming absurdity.
(To be continued...)

[benwaggoner]

Merging such duplicates would be almost the most obvious way in exploiting redundancy.

And for people who came up with: general compression, motion estimation, psycho-visual optimization... such state-of-the-art stuffs. There's no possibility that they ain't aware of this.
Oh, I'm sure they do. But one advantage of a true skip frame is that it needed rest the maximum GOP duration encoder. So a 150 frame GOP could potentially encapsulate hundreds of playback frames in anime, for example.