FFMpeg next will likely support AVIF and HEIC/HEIF

[birdie]

Support for both formats has been sorely missing for years.

[Jamaika]

Super. Has anyone compiled heif fix?
https://github.com/FFmpeg/FFmpeg/com...d74d2774c73afa

mov.c: In function 'avif_add_stream':
mov.c:4952:10: error: 'MOVContext' has no member named 'hvcc_offset'
 4952 |     if (c->hvcc_offset >= 0) {
      |          ^~
mov.c:4956:35: error: 'MOVContext' has no member named 'hvcc_offset'
 4956 |         if (avio_seek(c->fc->pb, c->hvcc_offset, SEEK_SET) != c->hvcc_offset) {
      |                                   ^~
mov.c:4956:64: error: 'MOVContext' has no member named 'hvcc_offset'
 4956 |         if (avio_seek(c->fc->pb, c->hvcc_offset, SEEK_SET) != c->hvcc_offset) {
      |                                                                ^~
mov.c:4960:65: error: 'MOVContext' has no member named 'hvcc_size'
 4960 |         ret = ff_get_extradata(c->fc, st->codecpar, c->fc->pb, c->hvcc_size);
      |                                                                 ^~
mov.c: In function 'mov_read_meta':
mov.c:5007:6: error: 'MOVContext' has no member named 'hvcC_offset'
 5007 |     c->hvcC_offset = -1;
      |      ^~
mov.c:5008:6: error: 'MOVContext' has no member named 'hvcC_size'
 5008 |     c->hvcC_size = 0;
      |      ^~
mov.c: In function 'mov_read_iprp':
mov.c:7899:14: error: 'MOVContext' has no member named 'hvcC_offset'
 7899 |             c->hvcC_offset = avio_tell(pb);
      |              ^~
mov.c:7900:14: error: 'MOVContext' has no member named 'hvcC_size'
 7900 |             c->hvcC_size = sub_size;
      |              ^~

[FranceBB]

Awesome news, especially for HEIF!

[benwaggoner]

Originally Posted by FranceBB

Awesome news, especially for HEIF!

Yeah, Apple has really demonstrated HEIC (HEIF with HEVC IDR frames) as a superior alternative to JPEG. Smaller file sizes, much less annoying artifacts, and much more efficient for synthetic images. For thinks like line art and with a well-tuned encode (using tools like transform skip and full flexibility for TU and CU sizes and shapes), I've seen HEIC outperform JPEG at <<10% the bitrate.

[FranceBB]

Yep yep, I've been trying to make it gain traction too and I have no problems displaying HEIF images, however for most people the fact that they have to install a decoder and that they're not displayed by whatever their OS is using by default is annoying enough.
In other words, despite using HEIF myself, most often than not I had to re-encode it to JPEG before sending it to other people.
As far as mobile apps are concerned, instead, they almost always downscale and re-encode to JPEG anyway, which is really a shame (I mean, don't get me wrong, JPEG served its own purpose for years, but I sort of consider it the mp3 of images nowadays).
Hopefully having a free open source decoder openly available in FFMpeg will make the adoption easier and if Android devices are gonna start decoding them automatically then I think we're gonna see an upward shift in its use.

[ShortKatz]

In the meantime, not only iPhones, also some mirrorless Sony cameras can save the images as HEIF images instead of JPEG.

Originally Posted by FranceBB

Yep yep, I've been trying to make it gain traction too and I have no problems displaying HEIF images, however for most people the fact that they have to install a decoder and that they're not displayed by whatever their OS is using by default is annoying enough.

But this depends on the OS. If you are on macOS, you have no issues opening HEIF images. On my Windows PC at work running Windows 10 I also have no issue, but I have no idea if our system admins have installed some add-on of some sort.

[birdie]

And it works!

I've checked some files from https://filesamples.com/formats/heic and https://aomediacodec.github.io/av1-a...tFiles/Link-U/

ffplay version N-113314-g564a15b2ee Copyright (c) 2003-2024 the FFmpeg developers
  built with gcc 13 (GCC)

[ShortKatz]

HEIF parsing was improved in FFmpeg now.

[Jamaika]

Test Jamaika {compatibility}
heifenc_avx.exe image.jpg --verbose -L -q 100 -e x265 --no-alpha -p chroma=444 -p tune=ssim --matrix_coefficients=0 --full_range_flag=1 -o image1.heif
Stream #0:0[0x1]: Video: hevc (Rext) (hvc1 / 0x31637668), gbrp(pc, gbr/bt709/iec61966-2-1), 280x420, 1 fps, 1 tbr, 1 tbn
[ffplay_buffer @ 0000019f96d3c410] filter context - w: 280 h: 420 fmt: 71 csp: unknown range: unknown, incoming frame - w: 280 h: 420 fmt: 71 csp: gbr range: pc pts_time: 0
[ffplay_buffer @ 0000019f96d3c410] Changing video frame properties on the fly is not supported by all filters.

heifenc_avx.exe image.jpg --verbose -L -q 100 -e kvazaar --no-alpha --matrix_coefficients=0 --full_range_flag=1 -o image2.heif
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000022259c0d9e0] iloc: extent_count > 1 not supported
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000022259c0d9e0] error reading header
image2.heif: Not yet implemented in FFmpeg, patches welcome

heifenc_avx.exe image.jpg --verbose --jpeg2000 -L -q 100 -e openjpeg --no-alpha --matrix_coefficients=0 --full_range_flag=1 -o image3.heif
Failed to open file 'image3.heif' or configure filtergraph

heifenc_avx.exe image.jpg --verbose --htj2k -L -q 100 -e openjph --no-alpha --matrix_coefficients=0 --full_range_flag=1 -o image4.heif
Failed to open file 'image4.heif' or configure filtergraph

MP4Box.exe -add-image output_uvg266.mp4: primary -ab heic -new image.vvic
Failed to open file 'image.vvic' or configure filtergraph

[benwaggoner]

Originally Posted by FranceBB

Yep yep, I've been trying to make it gain traction too and I have no problems displaying HEIF images, however for most people the fact that they have to install a decoder and that they're not displayed by whatever their OS is using by default is annoying enough.
In other words, despite using HEIF myself, most often than not I had to re-encode it to JPEG before sending it to other people.
As far as mobile apps are concerned, instead, they almost always downscale and re-encode to JPEG anyway, which is really a shame (I mean, don't get me wrong, JPEG served its own purpose for years, but I sort of consider it the mp3 of images nowadays).
Hopefully having a free open source decoder openly available in FFMpeg will make the adoption easier and if Android devices are gonna start decoding them automatically then I think we're gonna see an upward shift in its use.

It seems like a JavaScript HEIF parser that passes the media decode to hardware should be pretty straightforward as things go. Has no one implemented that as a fallback for browsers that don't have native support?

[SeeMoreDigital]

Originally Posted by FranceBB

... (I mean, don't get me wrong, JPEG served its own purpose for years, but I sort of consider it the mp3 of images nowadays).

I like the analogy

How is HEIF compared to JPEG 2000?

[FranceBB]

Originally Posted by SeeMoreDigital

How is HEIF compared to JPEG 2000?

Uhmmm I'm not sure, but I don't actually dislike JPEG2000 to be fair. If we think about it in terms of browser, consumer electronics etc, then it might not get quite into the main picture, however there's one simple reason why I can't dislike JPEG2000: it came in to defeat a far worse enemy -> MPEG-2.
Yes, although in images it didn't have much joy and never really got adopted, in videos it's a whole other story. Before the JPEG2000 adoption, every time we went to the cinemas to see a movie, we had an 8bit yv16 50-85 Mbit/s MPEG-2 Long GOP (up to FULL HD) encode which, projected on a very large screen like the ones we see in movie theatres, was awful. JPEG2000 came to the rescue and MJPEG2000 was standardized as an All Intra codec (basically each frame is a .j2k still that then becomes a bitstream muxed in an .mxf container). With that, we went to the new 2K (and later 4K) standards allowing 125 Mbit/s and 250 Mbit/s respectively of glorious MJPEG2000 All Intra 12bit XYZ 4:4:4 which made MPEG-2 8bit look like garbage and quickly became the de-facto standard for DCP. Nowadays, MPEG-2 is deprecated and any movie we've been seeing in a theatre has been MJPEG2000 12bit for a very long time, without any banding or other artifacts. To me, JPEG2000 is the hero of cinemas and the Wavelet Transform yields far better results than the Discrete Cosine Transform implemented in the old MPEG-2 standard. Back then, the fact of encoding a whole image at full resolution and treating it as one single macroblock rather than dividing it in blocks and macroblocks of 4x4 and 8x8 was seen as unfeasible 'cause it meant that there was no way of parallelizing the process and it took a long time to encode a stream, however modern CPUs are able to encode a single 4K image in a jiffy and by it being an All Intra codec, you can parallelize as much as you want, in fact, in my 56c/112th Intel Xeon from 2019, I generally encode 56 frames at the same time when I have to produce a DCP, making the whole process really fast. I have this in my heart 'cause not only I like to create DCP, but aside from my job, I'm also a consumer, so I do go to the cinema to watch movies, so this is something I'm able to experience the benefit of as a direct consumer.
As something indirect, however, there's also IMF. The story with IMF would need a whole chapter for itself and I don't wanna get into the nitty gritty details of it, but I'm a huge sustainer of IMF too and I'm proud to be part of the IMF Usergroup. The use of MJPEG2000 there too meant that we now get deliveries in high bit depth RGB Full Range with PQ or HLG BT2020 in it (or even just plain old BT709 SDR) with very high quality at reduced bitrate (i.e less than 1 Gbit/s), moving away from 1.2 TB Apple ProRes exports per movie (that are generally in the range of 1.3 Gbit/s for grainy movies) and most importantly there are lots of free open source encoders and decoders that can be used for JPEG2000 (and this is without mentioning the advantages of using a CPL instead of having to manually trim textless materials etc).

[Jamaika]

Originally Posted by FranceBB

Uhmmm I'm not sure, but I don't actually dislike JPEG2000 to be fair. If we think about it in terms of browser, consumer electronics etc, then it might not get quite into the main picture, however there's one simple reason why I can't dislike JPEG2000: it came in to defeat a far worse enemy -> MPEG-2.
Yes, although in images it didn't have much joy and never really got adopted, in videos it's a whole other story. Before the JPEG2000 adoption, every time we went to the cinemas to see a movie, we had an 8bit yv16 50-85 Mbit/s MPEG-2 Long GOP (up to FULL HD) encode which, projected on a very large screen like the ones we see in movie theatres, was awful. JPEG2000 came to the rescue and MJPEG2000 was standardized as an All Intra codec (basically each frame is a .j2k still that then becomes a bitstream muxed in an .mxf container). With that, we went to the new 2K (and later 4K) standards allowing 125 Mbit/s and 250 Mbit/s respectively of glorious MJPEG2000 All Intra 12bit XYZ 4:4:4 which made MPEG-2 8bit look like garbage and quickly became the de-facto standard for DCP. Nowadays, MPEG-2 is deprecated and any movie we've been seeing in a theatre has been MJPEG2000 12bit for a very long time, without any banding or other artifacts. To me, JPEG2000 is the hero of cinemas and the Wavelet Transform yields far better results than the Discrete Cosine Transform implemented in the old MPEG-2 standard. Back then, the fact of encoding a whole image at full resolution and treating it as one single macroblock rather than dividing it in blocks and macroblocks of 4x4 and 8x8 was seen as unfeasible 'cause it meant that there was no way of parallelizing the process and it took a long time to encode a stream, however modern CPUs are able to encode a single 4K image in a jiffy and by it being an All Intra codec, you can parallelize as much as you want, in fact, in my 56c/112th Intel Xeon from 2019, I generally encode 56 frames at the same time when I have to produce a DCP, making the whole process really fast. I have this in my heart 'cause not only I like to create DCP, but aside from my job, I'm also a consumer, so I do go to the cinema to watch movies, so this is something I'm able to experience the benefit of as a direct consumer.
As something indirect, however, there's also IMF. The story with IMF would need a whole chapter for itself and I don't wanna get into the nitty gritty details of it, but I'm a huge sustainer of IMF too and I'm proud to be part of the IMF Usergroup. The use of MJPEG2000 there too meant that we now get deliveries in high bit depth RGB Full Range with PQ or HLG BT2020 in it (or even just plain old BT709 SDR) with very high quality at reduced bitrate (i.e less than 1 Gbit/s), moving away from 1.2 TB Apple ProRes exports per movie (that are generally in the range of 1.3 Gbit/s for grainy movies) and most importantly there are lots of free open source encoders and decoders that can be used for JPEG2000 (and this is without mentioning the advantages of using a CPL instead of having to manually trim textless materials etc).

MJPEG2000 reminds me of cheap cameras from the 90s. Interestingly there seems to be no successors of the MHTJ2K or MJPEGXS type.
JPEG2000 is history. What about the successor JPEGXS which is already paid. It has a chance to exist on the market.

Part 14 specifies an XML representation of the JPEG 2000 file format and marker segments, along with methods to for accessing the internal data of a JPEG 2000 image.
Openjpeg isn't integrated with libxml2
Apparently libheif is integrated with librsvg.

Part 15 speeds-up JPEG 2000 by an order of magnitude at the expense of slightly reduced coding efficiency. The resulting HTJ2K system retains JPEG 2000's advanced features, with reduced quality scalability, while being faster and much more efficient than traditional JPEG. This is achieved by replacing the Part 1 block coder with an innovative block coder for today's vectorized computing architectures. This also allows mathematically lossless transcoding to/from legacy JPEG 2000. Part 15 is intended to be royalty-free.

Part 16 specifies the carriage of JPEG 2000 codestreams in ISO/IEC 23008-12, commonly referred to as HEIF. A revision is underway to support more flexible wrapping of all JPEG 2000 codestreams, including HTJ2K.
https://cdn.standards.iteh.ai/sample...08-12-2022.pdf

Part 17, currently under development, introduces alternate “breakpoint-dependent” spatial wavelet transforms that dependent on an auxiliary image component, known as a “breakpoint component", and scalable coding technologies for these breakpoint components. This improves the coding of media with hard discontinuities. An important example of such media is depth imagery, where each image sample is related to the length of the 3D line segment between the corresponding scene point and the camera. Depth imagery includes stereo disparity maps, where sample values are reciprocally related to depth. Another example of media with strong discontinuities is optical flow data, where each sample location is a two-dimensional vector. In these examples, discontinuities arise naturally at the boundaries of scene objects.

Choose whether pictures are recorded in JPEG or HEIF.

Option Description
JPEG Pictures are recorded in the widely-supported JPEG format.
HEIF Pictures are recorded in HEIF, a format with excellent compression but limited options for viewing and sharing.
JPEG is automatically selected in place of HEIF during filter-effect, panorama, multiple-exposure, and HDR photography.

Selecting HEIF disables CLARITY and sets COLOR SPACE to sRGB.

HEIF pictures are stored on the memory card as files with the extension “.HIF”. Before the pictures can be viewed on a computer, the extension must be changed to “.HEIC”. This occurs automatically when HEIF pictures are uploaded from the camera to a computer via USB.

[SeeMoreDigital]

I wonder how eager the cinema industry is about moving over from JPEG2000 to something else, whether it be HEIC or something else?!

[FranceBB]

Originally Posted by SeeMoreDigital

I wonder how eager the cinema industry is about moving over from JPEG2000 to something else, whether it be HEIC or something else?!

I'm not quite sure, but I think JPEG2000 is well cemented in today's standard and will be for a very long time.
To be fair, 250 Mbit/s for a 4K 12bit XYZ 4:4:4 encode which consumers will see at the theatres is probably more than we will ever need in a very long time. Last time around, when the 4K specs were updated, they even expanded the frame rates a cinema could play to allow 30p 48p 50p and 60p (on top of the old 23.976p and 24p) so I don't really see specs changing anytime soon. Changing codecs would mean changing all the systems out there which would make cinemas incur in added costs, so I really don't see that happening anytime soon for 4K. The real question will be for 8K but I wouldn't be surprised to see official JPEG2000 specs being extended again, if and when 8K comes around. I know that Japan has been a really big sustainer of 8K since late 2018 but here we are 6 years later with only a bunch of documentaries that you can count in your hands (all made in Japan and more as an example than anything) and no cinema standard nor actual contents yet.

[SeeMoreDigital]

Whenever there's a major change in specification that requires new equipment, such as providing support for 4K images, that's the golden opportunity to provide support for newer image formats too...

[kolak]

Originally Posted by FranceBB

... The real question will be for 8K but I wouldn't be surprised to see official JPEG2000 specs being extended again, if and when 8K comes around. I know that Japan has been a really big sustainer of 8K since late 2018 but here we are 6 years later with only a bunch of documentaries that you can count in your hands (all made in Japan and more as an example than anything) and no cinema standard nor actual contents yet.

There is new mode for Jpeg2000, which is called HTJ2K. It's a new, simplified mode which looses around 5-10% in efficiency but has 5-10x gain in encoding/decoding speed.
Current Jpeg2000 is overdone and problematic even today.
New mode changes it and makes it more acceptable. Support is already present in some pro tools. It may come to DCP servers I assume, so then same hardware should do 8K.
CineForm is what Jpeg2000 should be from the start (but they overdone it)- fairly efficient and fast.

[benwaggoner]

Originally Posted by SeeMoreDigital

I like the analogy

How is HEIF compared to JPEG 2000?

HEIC (the HEVC alternate) provides better compression efficiency than J2K. It doesn't support wavelet subbands, which is a tool some people use, but otherwise I've not seen anything that a well-tuned HEIC doesn't beat J2K.

The tskip and lossless cu modes that allow efficient encoding of synthetic regions can be very powerful for content like text and line hard that frequency-based techniques generally fail on.

[benwaggoner]

Originally Posted by FranceBB

I'm not quite sure, but I think JPEG2000 is well cemented in today's standard and will be for a very long time.
To be fair, 250 Mbit/s for a 4K 12bit XYZ 4:4:4 encode which consumers will see at the theatres is probably more than we will ever need in a very long time. Last time around, when the 4K specs were updated, they even expanded the frame rates a cinema could play to allow 30p 48p 50p and 60p (on top of the old 23.976p and 24p) so I don't really see specs changing anytime soon. Changing codecs would mean changing all the systems out there which would make cinemas incur in added costs, so I really don't see that happening anytime soon for 4K. The real question will be for 8K but I wouldn't be surprised to see official JPEG2000 specs being extended again, if and when 8K comes around. I know that Japan has been a really big sustainer of 8K since late 2018 but here we are 6 years later with only a bunch of documentaries that you can count in your hands (all made in Japan and more as an example than anything) and no cinema standard nor actual contents yet.

J2K for DCI supports 12-bit, which isn't widely supported in HW HEVC decoders yet (Nvidia GPUs do, but I can't think of anything else).

Which some file size savings would be possible with newer codecs, it's not like DCI package size has become a bigger problem in the last decade plus . Studios and theaters already took that hit.

It was a huge technical and financial effort to get existing DCI hardware into all the theaters, and there would have to be a huge reason to go though that again.

Getting rid of the whole film print manufacture, shipping, and storage saved huge amounts of money that funded the DCI conversion. There's really no big compelling technologies that would save that kind of money when switching up formats.

...as much as I like to think better compression changes the world...