FFMpeg next will likely support AVIF and HEIC/HEIF

[Zebulon84]

I have done some quick tests for one specific application : compress scans of various administrative documents (e.g. invoices), so quite low quality needed compared to what most people here may wish for, and that's not photos. Small color changes are not a problem, the criteria is only no big artifacts.
The best was AVIF (@40%), second Jpeg XL, then HEIF, WebP and JPG. I haven't kept the test files, but IIRC, AVIF was about 20% smaller than JXL, and quicker to encode. I did not do decode tests as I rarely need to open several documents in a row, and opening one is pretty quick (~1s on my i3-12100).
I used Skanlite (from KDE), I don't know precisely which encoder is used.

Of course, the results may be quite different for high quality photo encodings, screenshots...

[Z2697]

As a matter of fact, HEIF is just a container. The quality and speed is highly dependent on the encoder used, whether it's the HEVC variant (HEIC), AV1 variant (AVIF) or else.

Especially for these Video Codec Based variants, there are more choices and bigger differences than you typically get for image encoders (within same codec).

(WebP is also video codec (VP8) based but you don't have encoders to choose though, also the only supported pixel format is YUV(A)420P8, meh.)



Now the general situation is aomenc and svt-av1 struggles to match the quality of a tuned x265 in mid to high bitrate, for Intra only images, there's less encoding tool they can use, and the "advantage" of AV1 side is even less.

But AV1 might have more widely supported (decoding side) "SCC extensions", that could be a potential benefit but I don't think it's significant for anything not screen content.

My personal ranking is HEIC (x265) > JPEG XL (libjxl) > AVIF (aomenc or svt-av1) for lossy encoding. (WebP is excluded because it sucks, or somewhere at the bottom not even able to compete with JPEG because the lack of 4:4:4 support)

(By sucks I mean I always considered the VP series is half a generation off compared to MPEG series like VP9 is between AVC and HEVC, so that means VP8 is... similar to or less than AVC?)



Now if we need to talk about lossless encoding, well... WebP wins (kind of). (Assuming your input is RGB! If your input is YUV things might get more interesting but I didn't test that.)

Now the VCB formats or HEIF variants sucks because of the lack of specifically designed lossless mode.

JPEG XL is basically two parts (VarDCT for lossy and Modular for lossless) merged together and somtimes cooperating perhaps, WebP actually has a secret sauce called VP8L but only supports ARGB (32bpp / 8bpc) even for no-alpha images. (Despite the similarity of the name to VP8, it's actually better than just a quantization bypass or something like in HEVC or AV1 lossless, in terms of compression ratio, I didn't take a deeper look)

My personal ranking is JPEG XL (libjxl) > WebP > PNG > [VCB HEIF variants, you name it] for lossless encoding.

Be careful that some VCB HEIF encoding tools might be in fact not really lossless and give you illusions that they performed well in lossless tasks. (Like some early versions of a certain library that I'm not sure if it's libheif or not, they convert RGB pixels to YUV first.)



One step further, animations.

Video codec based formats wins again. Even in lossless mode. When compressing single image they are not as powerful, they are better at exploiting inter-frame compressions thus makes up for the shortcomming. (But your animations will need to have enough frames to make the benefits significant enough, or even get to outweight the shortcoming)
Other formats previously mentioned are basically "ranked" like in the single frame "ranks".

(DISHORNORABLE mention: WebP, while being VP8, a video codec based, seemingly not taking the advantage of better inter-frame compression. And VP8 is very old.)

[LigH]

L4cache is trying to put it in M-AB-S. So far there is a linker issue with libde265, so heif-dec contains no HEIC decoder yet ... but there is some progress.

[LigH]

Linking libde265 works now.

For the curious: libheif 1.19.5 c057050 (MSYS2/MinGW, GCC 14.2; different sets of codecs per bitness)

[Jamaika]

Originally Posted by LigH

L4cache is trying to put it in M-AB-S. So far there is a linker issue with libde265, so heif-dec contains no HEIC decoder yet ... but there is some progress.

Interesting. I didn't think anyone would be interested in libheif on this forum.

[Jamaika]

Originally Posted by LigH

Linking libde265 works now.

For the curious: libheif 1.19.5 c057050 (MSYS2/MinGW, GCC 14.2; different sets of codecs per bitness)

M-A-B-S reminds me of ffmpeg compatibility. Interesting that regular ffmpeg doesn't have it.
LigH. What plugins did you add and with what options in the latest ffmpeg? There are a lot of them on github.

[Jamaika]

Test libheif
heif-enc.exe image_2.png --verbose --vvc -q 100 -e vvenc --no-alpha --matrix_coefficients=0 --full_range_flag=1 -o image_vvenc.vvif
I don't like the action. Why is there no verbose information?

heif-info.exe imgae_uvg266.heif/imgae_kvazaar.heif
MIME type: unknown
main brand: vvic
compatible brands: mif1, vvic, miaf

image: 4000x3000 (id=1), primary
colorspace: RGB
bit depth: 8
color profile: nclx
alpha channel: no
depth channel: no
metadata:
none
transformations:
none
region annotations:
none
properties:

but in ffmpeg is...
Stream #0:0[0x1]: Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(pc, gbr/bt709/iec61966-2-1), 4000x3000, 1 fps, 1 tbr, 1 tbn (default)
https://imgur.com/oNLHtw4
Has LigH added uvg266/kvazaar 10bit?

What are the options for openh264?
heif-enc.exe image_2.png --verbose -q 100 -e openh264 --no-alpha --matr
ix_coefficients=0 --full_range_flag=1 -o image_openh264.heif

[LigH]

Originally Posted by Jamaika

Interesting. I didn't think anyone would be interested in libheif on this forum.

Actually it was someone in the German doom9/Gleitz board.

Originally Posted by Jamaika

What plugins did you add and with what options in the latest ffmpeg? There are a lot of them on github.

I did nothing more than running L4cache's fork which is supposed to get pulled into the official M-AB-S some day. Details in issue 2516.

So many times I already told you: I am not a very competent developer when it comes to C/C++, Linux style compilers, and Linux style shell scripts. I can only run them and make halfwit guesses about error messages. Definitely I can not add anything elaborate like a whole codec support chapter to the suite.

[benwaggoner]

Originally Posted by Jamaika

Interesting. I didn't think anyone would be interested in libheif on this forum.

Why not? Video codecs have been great still image codecs since AVC was invented, and HEVC heic became a really great one as tools like transform skip and lossless coding units finally made it possible to have high quality at high compression rates for both natural and synthetic content in the same frame.

Plus pretty much any computing device has a HW decoder for them, which can make heic more performant than JPEG. Being able to copy some compressed bits to the GPU where they get decoded and composited is so much better than doing a RGB or RGBA in software and pushing the whole bitmap over. Heck, just decoding to YV12 in software and copying that cuts memory bandwidth in half.

Apple has used heic over jpeg for default for years for these sorts of advantages. Having fast HW encoders as well certainly doesn't hurt (although Apple used a SW encoder for images last I checked).

[Z2697]

Originally Posted by Jamaika

Test libheif
heif-enc.exe image_2.png --verbose --vvc -q 100 -e vvenc --no-alpha --matrix_coefficients=0 --full_range_flag=1 -o image_vvenc.vvif
I don't like the action. Why is there no verbose information?

heif-info.exe imgae_uvg266.heif/imgae_kvazaar.heif
MIME type: unknown
main brand: vvic
compatible brands: mif1, vvic, miaf

image: 4000x3000 (id=1), primary
colorspace: RGB
bit depth: 8
color profile: nclx
alpha channel: no
depth channel: no
metadata:
none
transformations:
none
region annotations:
none
properties:

but in ffmpeg is...
Stream #0:0[0x1]: Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(pc, gbr/bt709/iec61966-2-1), 4000x3000, 1 fps, 1 tbr, 1 tbn (default)
https://imgur.com/oNLHtw4
Has LigH added uvg266/kvazaar 10bit?

What are the options for openh264?
heif-enc.exe image_2.png --verbose -q 100 -e openh264 --no-alpha --matr
ix_coefficients=0 --full_range_flag=1 -o image_openh264.heif

Wait, did you mean that the vvenc and .vvic specified in command line result in HEVC?
What command produced the corrupted result in the link?

[Jamaika]

I've been on this forum for 7 years. Before the pandemic, this was a time of a flood of codecs. Many of them are unlikely to see use.

Originally Posted by benwaggoner

Why not? Video codecs have been great still image codecs since AVC was invented, and HEVC heic became a really great one as tools like transform skip and lossless coding units finally made it possible to have high quality at high compression rates for both natural and synthetic content in the same frame.

To be honest, I haven't kept up with the latest technologies for a long time. Today I'm interested in the naming and purposes of the container. hevc is heic, vvc is vvic, and h264 is ???.

Originally Posted by benwaggoner

Plus pretty much any computing device has a HW decoder for them, which can make heic more performant than JPEG. Being able to copy some compressed bits to the GPU where they get decoded and composited is so much better than doing a RGB or RGBA in software and pushing the whole bitmap over. Heck, just decoding to YV12 in software and copying that cuts memory bandwidth in half.

hmmm... So lossless jpeg9 in heif container, what is its efficiency?
I understand that heif in hardware has potential but adding XXX generation jpegs to heif I don't understand it.

Originally Posted by benwaggoner

Apple has used heic over jpeg for default for years for these sorts of advantages. Having fast HW encoders as well certainly doesn't hurt (although Apple used a SW encoder for images last I checked).

HW encoders are a commercial for higher players. For me, an amateur can watch the work. Create a commercial for an encoder, but in truth, the encoder takes shape after adding useful functions for a specific video editor. For a programmer, it's 5 minutes of work.

[Z2697]

Originally Posted by Jamaika

Interesting. I didn't think anyone would be interested in libheif on this forum.

Well, not exactly libheif, but I even wrote an abomination of script to play with HEIF using FFmpeg (libx265 in particular) and mp4box.

[Jamaika]

Originally Posted by Z2697

Wait, did you mean that the vvenc and .vvic specified in command line result in HEVC?

No. Theoretically it starts to get funny because vvc in heif should be called differently. Just like for av1 it is avif.
For vvc you also have the uvg266 encoder.

[Jamaika]

Originally Posted by Z2697

What command produced the corrupted result in the link?

heif-enc.exe image_2.png --verbose -q 100 -e kvazaar --no-alpha --matrix_coefficients=0 --full_range_flag=1 -o image_kvazaar.heif

[Z2697]

Originally Posted by Jamaika

heif-enc.exe image_2.png --verbose -q 100 -e kvazaar --no-alpha --matrix_coefficients=0 --full_range_flag=1 -o image_kvazaar.heif

There must be problem(s) in kvazaar or the way m-ab-s builds it, I just checked FFmpeg recently built with m-ab-s, libkvazaar also produces broken output.

Oh, never mind. FFmpeg's problem is m-ab-s build 10bit kvazaar but FFmpeg just assumes default 8bit.
The problem with libheif is probably a different one.

10bit kvazaar.exe itself is OK.

But if we build 8bit kvazaar instead, both FFmpeg and libheif will be able to produce correct result.
Although they should be considered as bugs and need to be fixed, I wonder why m-ab-s chose to build 10bit kavzaar in the first place?

[Jamaika]

Originally Posted by Z2697

Although they should be considered as bugs and need to be fixed, I wonder why m-ab-s chose to build 10bit kavzaar in the first place?

No. LigH built m-a-b-s libheif correctly with kvazaar 8-bit. The problem is that ffplay displays nonsense.
Is it possible to add kvazaar/uvg266 10bit to libheif? I doubt it.
In ffmpeg it is impossible because plugins are only for 8bit.
You can't add two 8/10bit options for kvazaar/uvg266 like for x264/x265 because they have common functions.
Kvazaar and uvg266, vvenc do not have yuv422 and yuv444 options because they are commercials.
And there is another problem gnu ffmpeg does not support decoder with AVX2. I suppose it is during rebuilding.

[Z2697]

Originally Posted by Jamaika

No. LigH built m-a-b-s libheif correctly with kvazaar 8-bit. The problem is that ffplay displays nonsense.
Is it possible to add kvazaar/uvg266 10bit to libheif? I doubt it.
In ffmpeg it is impossible because plugins are only for 8bit.
You can't add two 8/10bit options for kvazaar/uvg266 like for x264/x265 because they have common functions.
Kvazaar and uvg266, vvenc do not have yuv422 and yuv444 options because they are commercials.
And there is another problem gnu ffmpeg does not support decoder with AVX2. I suppose it is during rebuilding.

https://github.com/m-ab-s/media-auto...mpile.sh#L1262

The default behaviour of m-ab-s is to build 10bit kvazaar.

[Jamaika]

Indeed you are right. Apparently frame I was saved in the heif container as 8bit.
Jamaika tests no mabs

I know how it is in mabs but can't save 8bit photo for heif with 10bit codec. I created two 8, 10bit libheifs and for each I changed the output_bit_depth accordingly. Creator has put 10bit output under vvenc.
I don't know what the h264 decoder is supposed to be used for.

heifenc_08bit_avx.exe image_48bit.png --verbose -q 100 -e kvazaar --no-alpha --matrix_coefficients=6 --full_range_flag=0 -o image_kvazaar_08bit.heic
--- construct_pipeline
from: colorspace=RGB chroma=RGB bpp(R)=8 alpha=no
to: colorspace=YCbCr chroma=420 bpp(R)=8 alpha=no matrix-coefficients=6 colour-primaries=1 transfer-characteristics=13 full-range=no

libheif can't be with functions matrix_coefficients=0, full_range_flag=1. He should have information and disabled functions

heifenc_10bit_avx.exe image_48bit.png --verbose -q 100 -e kvazaar --no-alpha --matrix_coefficients=6 --full_range_flag=0 -o image_kvazaar_10bit.heic
--- construct_pipeline
from: colorspace=RGB chroma=RRGGBBB_LE bpp(R)=10 alpha=no
to: colorspace=YCbCr chroma=420 bpp(R)=10 alpha=no matrix-coefficients=6 colour-primaries=1 transfer-characteristics=13 full-range=no
It's still bad for 10bit
https://imgur.com/0xDDiaN
Libheif errors:
heifenc_10bit_avx.exe image_48bit.png --verbose -q 100 --vvc -e vvenc --no-alpha --matrix_coefficients=6 --full_range_flag=0 -o image_10bit.vvic
--- construct_pipeline
from: colorspace=RGB chroma=RRGGBBB_LE bpp(R)=10 alpha=no
to: colorspace=YCbCr chroma=420 bpp(R)=10 alpha=no matrix-coefficients=6 colour-primaries=1 transfer-characteristics=13 full-range=no
final pipeline has 3 steps:
> 27Op_RRGGBBaa_swap_endianness
> 25Op_RRGGBBaa_BE_to_RGB_HDR
> 15Op_RGB_to_YCbCrItE
input spec: colorspace=RGB chroma=RRGGBBB_LE bpp(interleaved)=10
input spec: colorspace=RGB chroma=RRGGBB_BE bpp(interleaved)=10
input spec: colorspace=RGB chroma=444 bpp(R)=10
Could not encode HEIF/AVIF file: Encoder plugin generated an error: Unsupported image type: Bit depth not supported by vvenc <-- Why?

heifenc_10bit_avx.exe image_24bit.png --verbose -q 100 --vvc -e vvenc --no-alpha --matrix_coefficients=6 --full_range_flag=0 -o image_10bit.vvic
--- construct_pipeline
from: colorspace=RGB chroma=RGB bpp(R)=8 alpha=no
to: colorspace=YCbCr chroma=420 bpp(R)=8 alpha=no matrix-coefficients=6 colour-primaries=1 transfer-characteristics=13 full-range=no <--Why?

image4>heifenc_08bit_avx.exe image_24bit.png --verbose -q 100 --vvc -e vvenc -
-no-alpha --matrix_coefficients=6 --full_range_flag=0 -o image_8bit.vvic
--- construct_pipeline
from: colorspace=RGB chroma=RGB bpp(R)=8 alpha=no
to: colorspace=YCbCr chroma=420 bpp(R)=8 alpha=no matrix-coefficients=6 colour-primaries=1 transfer-characteristics=13 full-range=no <--Why? VVenc doesn't have 8bit output
https://www.sendspace.com/file/5xv4ho

ffplay_avx2.exe image_kvazaar08.heic
[mov,mp4,m4a,3gp,3g2,mj2 @ 000002129285c000] error reading header
image_kvazaar08.heic: Invalid data found when processing input
https://www.sendspace.com/file/m5vk82

https://github.com/strukturag/libheif/issues/1410

Edit:
https://www.sendspace.com/file/oiyj2m

[Z2697]

@Jamaika Thanks for the testing, but there's one problem: I don't understand some of what you said, I mean the grammar. I assume you are not native English speaker? No offense but I think it might be better to use a translator, or write in you native language and let us try to understand it ourselves.

[LigH]

@Z2697: Not only that. But on top of it, Jamaika's goal seems to be not the general average build, but a very special one, statically linking into ffmpeg even DLLs which were meant to be installed into the system by default, like Avisynth. That must be one of the reasons why not the most recent GCC is used, but one with the least linking issues for that specific goal. I did not understand that for several months, until recently, because it was either never documented clearly or I missed it because it was mentioned too long ago.