Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > New and alternative video codecs

Reply
 
Thread Tools Search this Thread Display Modes
Old 2nd December 2020, 19:23   #21  |  Link
Scope
Registered User
 
Join Date: Feb 2020
Posts: 13
WebP 2 + dlls [3724e16] [Windows][GCC 10.2.0][64 bit]
WebP 2 + dlls [377656e] [Windows][GCC 10.2.0][64 bit]

Last edited by Scope; 8th December 2020 at 19:09.
Scope is offline   Reply With Quote
Old 3rd December 2020, 14:28   #22  |  Link
Marsu42
Huba Huba
 
Marsu42's Avatar
 
Join Date: Aug 2005
Location: Palumbian Jungle
Posts: 78
Quote:
Originally Posted by benwaggoner View Post
Do we care about WebP?
Personally, I do care because webp has the highest lossless compression I've seen - making it my photo archival format, esp. because it still has fast decoding (unlike flif and other specialized codecs). A common fallacy seems to be that webp uses yuv/vp8 for lossless - but it's rgba and afaik a separate implementation added independently from the video codec legacy.

That's why I'm very excited to read about "improved lossless compression" and "full 10bit architecture (HDR10)" in "What to expect?" and I am keen to see comparisons 1.0 vs. 2.0 lossless.

Webp is added to most browsers, a lot of image viewers - so it might be possible that these will be quickly upgrading to 2.0 (except for older Android)

Quote:
Originally Posted by VoodooFX View Post
I see up to ~50% better compression over PNG in my use scenarios.
I'm recompressing all png as webp, saving a lot of space even though they are only image formats. I'm using the built-in pre-processing, which some might see as a hack and of course this is somewhat possible with png, too - but webp is very efficient at it.

Re-compressing gif animated webp is still a bit tricky since results vary (esp. with dithering in the gif), recompression is slow, and some image viewers only support non-animated webp

Quote:
Originally Posted by skal View Post
Note that compressing losslessly photo material is not the main use-case case WebP2 is targetting
I hope you're still optimizing it, because imho webp lossless is terrific - I don't care that much about compression time, but decompression has to be rather swift even for larger images. At least webp seems to be designed with loessless hdr in mind, while avif results are rather disappointing atm because av1 isn't really tuned for intra-only compression - and lossless avif seems a bit like a hack.

I'd really like to stop looking for an image archival format that supports hdr (12bit would be better though for dslr images...), has lossless compression with a high ratio plus fast decompression _and_ is widely supported... and I've tried just about anything that is out there.

Quote:
Originally Posted by skal View Post
libwebp2 has been open-sourced
Is the bitstream finalized and the lib ready enough so we can begin asking devs of image viewers to upgrade?

Quote:
Originally Posted by Scope View Post
WebP 2 [b77ef4c] [Windows][GCC 10.2.0][64 bit]
This depends on a _lot_ of mingw shared dll - could you like them as static so this is stand-alone?
__________________
"The innocent have nothing to fear" :stupid:

Last edited by Marsu42; 3rd December 2020 at 15:04.
Marsu42 is offline   Reply With Quote
Old 3rd December 2020, 22:26   #23  |  Link
skal
Registered User
 
Join Date: Jun 2003
Posts: 121
Quote:
Originally Posted by Marsu42 View Post
I hope you're still optimizing it, because imho webp lossless is terrific - I don't care that much about compression time, but decompression has to be rather swift even for larger images.
Yes, lossless Photos are still on the radar, although probably less than -say- lossless coding of game assets (which are photo-like, sometimes!).

Tiles should help regarding decoding speed of large images, if multithreading is acceptable.

Quote:
Originally Posted by Marsu42 View Post
Is the bitstream finalized and the lib ready enough so we can begin asking devs of image viewers to upgrade?
It's still evolving a lot, don't use the format for anything precious!
(note that there's a 'vwp2' tool for visualization and for playing with the compression parameters in the library).
skal is offline   Reply With Quote
Old 3rd December 2020, 22:30   #24  |  Link
skal
Registered User
 
Join Date: Jun 2003
Posts: 121
Regarding WebP2's triangle-based compression for previews, i've set up a demo page you might find amusing here:

https://codepen.io/skal65535/project/full/ZwnQWM

You can visualize the base64-strings resulting from running 'mk_preview -b64 ...'
skal is offline   Reply With Quote
Old 3rd December 2020, 22:31   #25  |  Link
Jamaika
Registered User
 
Join Date: Jul 2015
Posts: 697
Sorry for no comment. I put it on quickly.
Code:
av1enc.cc: In function 'int main(int, const char**)':
av1enc.cc:243:46: error: 'env_vars' was not declared in this scope
  243 |   for (size_t i = 0; i < std::max((size_t)1, env_vars.size()); ++i) {
      |                                              ^~~~~~~~
Fix defined
Code:
#if !defined(_WIN32)
                  } else {
                    const char symbol = (std::abs(v) >= (norm * 0.50)) ? 'X'
                                      : (std::abs(v) >= (norm * 0.25)) ? 'x'
                                                                       : '.';
                    line += SPrintf("  %c ", symbol);
#endif
                  }
Jamaika is offline   Reply With Quote
Old 3rd December 2020, 23:15   #26  |  Link
skal
Registered User
 
Join Date: Jun 2003
Posts: 121
Quote:
Originally Posted by Jamaika View Post
Sorry for no comment. I put it on quickly.
Code:
av1enc.cc: In function 'int main(int, const char**)':
av1enc.cc:243:46: error: 'env_vars' was not declared in this scope
  243 |   for (size_t i = 0; i < std::max((size_t)1, env_vars.size()); ++i) {
      |                                              ^~~~~~~~

thanks for testing -> should be fixed now, hopefully!
skal is offline   Reply With Quote
Old 4th December 2020, 22:56   #27  |  Link
Jamaika
Registered User
 
Join Date: Jul 2015
Posts: 697
https://chromium.googlesource.com/co...a9f9594d3c5703

Compressing AV1 at quality 0.0...
aom_codec_decode(&codec, (const uint8_t*) bitstream.data(), (size_t) bitstream.size(), NULL) failed!

cwp2.exe image_21447.ppm image_21447.wp2 -summary -q 68 -nometadata -effort 9 -inplace
image_21447.ppm
Assertion failed: effort != 9 || angle_scores.back().score <= min_allowed_score, file predictor.cc, line 961

cwp2.exe image_21447.ppm -summary -q 68 -nometadata -effort 6 -csp 3 -o image_21447.wp2
Assertion failed: id < FeatureMap::kHistogramSize, file segment_enc.cc, line 415

cwp2.exe image_21447.ppm image_21447.wp2 -summary -q 68 -nometadata -effort 9 -csp 2 -inplace
image_21447.ppm
Error! Cannot encode 'image_21447.ppm' as WP2
Status: WP2_STATUS_INVALID_PARAMETER

Last edited by Jamaika; 6th December 2020 at 14:10.
Jamaika is offline   Reply With Quote
Old 5th December 2020, 00:07   #28  |  Link
skal
Registered User
 
Join Date: Jun 2003
Posts: 121
Quote:
Originally Posted by Jamaika View Post
https://chromium.googlesource.com/co...a9f9594d3c5703

cwp2.exe image_21447.ppm image_21447.wp2 -summary -q 68 -nometadata -effort 9 -inplace
image_21447.ppm
Assertion failed: effort != 9 || angle_scores.back().score <= min_allowed_score, file predictor.cc, line 961
...
Thanks! Could you upload image_21447.ppm somewhere please?
(or on the bug-tracker https://bugs.chromium.org/p/webp2, even)
skal is offline   Reply With Quote
Old 5th December 2020, 01:08   #29  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,752
Quote:
Originally Posted by skal View Post
Regarding WebP2's triangle-based compression for previews, i've set up a demo page you might find amusing here:

https://codepen.io/skal65535/project/full/ZwnQWM

You can visualize the base64-strings resulting from running 'mk_preview -b64 ...'
Very cool!
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 6th December 2020, 23:40   #30  |  Link
Marsu42
Huba Huba
 
Marsu42's Avatar
 
Join Date: Aug 2005
Location: Palumbian Jungle
Posts: 78
Quote:
Originally Posted by skal View Post
(tests welcome, too. Keep in mind the code is changing on a ~weekly basis, though!)
How do I enable 10-bit output, or is this auto-selected depending on 8/16 bit input, or not imlemented yet? Thanks.
__________________
"The innocent have nothing to fear" :stupid:
Marsu42 is offline   Reply With Quote
Old 7th December 2020, 00:14   #31  |  Link
skal
Registered User
 
Join Date: Jun 2003
Posts: 121
Quote:
Originally Posted by Marsu42 View Post
How do I enable 10-bit output, or is this auto-selected depending on 8/16 bit input, or not imlemented yet? Thanks.
The special tool extras/ewp2 can handle Y4M files with higher bit-depth samples.

Code:
./extras/ewp2 -h
Usage:
  ewp2 in_file [-o out_file] [options]

Options valid if 'out_file' has the .wp2 or .webp extension:
  -q <float> ........................ quality factor (0:small..100:big),
                                      default=75.0
  -effort <int> ..................... compression effort
                                      (0:fast..9:slower/better), default=5
  -csp <int> ........................ color space (wp2 only)

Options valid if 'out_file' has the .y4m extension (4:4:4 8b by default):
  -depth <int> ...................... output bit depth per plane per sample
                                      (8..12)
  -444 .............................. 4:4:4 output samples (full chroma
                                      resolution)
  -420 .............................. 4:2:0 output samples (half chroma
                                      resolution)

Options valid if 'in_file' and/or 'out_file' are 4:2:0:
  -sampling <filter> ................ algorithm to use for up- and/or chroma
                                      downsampling: smooth (default), nearest

Options valid if 'out_file' is specified:
  -[no]metadata ..................... include [or exclude] input metadata in
                                      output
  -[psnr|ssim] <file path> <float> .. error returned if output (or input if no
                                      given output) has a distortion worse
                                      than specified compared to a reference
                                      file

Other options:
  -v ................................ verbose
  -size ............................. print size and bits-per-pixel
  -rm [A|R|G|B|Y|U|V] ............... set specified input channel(s) values to
                                      opaque/black/grey
System options:
  -noasm ............................ disable all assembly optimizations
  -version .......................... print version number and exit
  -h / -help ........................ this help
skal is offline   Reply With Quote
Old 7th December 2020, 17:27   #32  |  Link
Marsu42
Huba Huba
 
Marsu42's Avatar
 
Join Date: Aug 2005
Location: Palumbian Jungle
Posts: 78
Quote:
Originally Posted by skal View Post
The special tool extras/ewp2 can handle Y4M files with higher bit-depth samples.
Um, so this preprocesses images to y4m (which in turn can be fed to the cwp2 compressor, to get a hdr .wp2 file), or is the y4m compressed but another container?

Sorry if I'm jumping the gun, I'm simply very excited to see webp lossless compression getting to 10 (and 12?) bit, it's not like I can use it a lot right now. In any case, I don't compile this myself atm, so someone needs to provide windows binaries @Jamaika, @Scope
__________________
"The innocent have nothing to fear" :stupid:

Last edited by Marsu42; 7th December 2020 at 17:36.
Marsu42 is offline   Reply With Quote
Old 7th December 2020, 19:24   #33  |  Link
Scope
Registered User
 
Join Date: Feb 2020
Posts: 13
Quote:
Originally Posted by Marsu42 View Post
I don't compile this myself atm, so someone needs to provide windows binaries @Jamaika, @Scope
Added ewp2, mk_preview and rd_curve to webp2-mingw64-3724e16.7z (but some dlls are also still required, since it takes me more time to compile completely static exe)

Quote:
Originally Posted by Marsu42 View Post
and I am keen to see comparisons 1.0 vs. 2.0 lossless.
I did a Lossless Image Formats Comparison, it's not completely finished yet, I want to update some results and add a more denser Jpeg XL compression mode (-s 9 -E 3, but it slows down the decoding a bit).

Some notes:
- there are Tabs for switching between different types of sets
- AVIF is lossless, but AVIF 444 is lossy (--cicp 1/13/1) the difference with the original looks like this https://i.imgur.com/MSCaY6E.png
- BMF is a closed format, slow for decoding, does not support alpha, but one of the most effective, selected only to understand the compression possibilities
- PNGs have been optimized (Pingo + ECT), the original PNGs are much larger (as can be seen in the PNG tab, where I made a comparison of the most effective and popular optimizers) https://i.imgur.com/Zj62IEi.png

Lossless Image Formats Comparison (Jpeg XL, AVIF, WebP 2, WebP, FLIF, PNG, ...)

Some general results (although it is better to compare by types of sets, as they are not equal in correlation):
Scope is offline   Reply With Quote
Old 7th December 2020, 20:23   #34  |  Link
skal
Registered User
 
Join Date: Jun 2003
Posts: 121
Quote:
Originally Posted by Scope View Post
- PNGs have been optimized (Pingo + ECT), the original PNGs are much larger (as can be seen in the PNG tab, where I made a comparison of the most effective and popular optimizers) https://i.imgur.com/Zj62IEi.png
That's quite interesting: does it look like PingoA+Brotli is the best tradeoff in size x time ?

thanks for spending time / watts testing this! Very useful.

skal/
skal is offline   Reply With Quote
Old 7th December 2020, 20:45   #35  |  Link
Scope
Registered User
 
Join Date: Feb 2020
Posts: 13
Quote:
Originally Posted by skal View Post
That's quite interesting: does it look like PingoA+Brotli is the best tradeoff in size x time ?
Yes, but this is uncompressed PNG + Brotli compression (it is supported by browsers, but not by applications), so this is only a theoretical option for comparison, in other cases it is better to use Pingo + ETC (which I used to optimize all PNGs)
Scope is offline   Reply With Quote
Old 7th December 2020, 21:44   #36  |  Link
jethro
Registered User
 
Join Date: Mar 2006
Posts: 102
Quote:
Originally Posted by skal View Post
The special tool extras/ewp2 can handle Y4M files with higher bit-depth samples.

Code:
./extras/ewp2 -h
Usage:
  ewp2 in_file [-o out_file] [options]


Options valid if 'out_file' has the .y4m extension (4:4:4 8b by default):
  -depth <int> ...................... output bit depth per plane per sample
                                      (8..12)
  -444 .............................. 4:4:4 output samples (full chroma
                                      resolution)
  -420 .............................. 4:2:0 output samples (half chroma
                                      resolution)
Hi Skal,
Will 4:4:4 sampling be included in the standard Webp2 decoder? What about 4:2:2? And why there is a split between normal and special encoders? Thanks.
jethro is offline   Reply With Quote
Old 7th December 2020, 22:41   #37  |  Link
skal
Registered User
 
Join Date: Jun 2003
Posts: 121
Jethro,

Quote:
Originally Posted by jethro View Post
Hi Skal,
Will 4:4:4 sampling be included in the standard Webp2 decoder? What about 4:2:2? And why there is a split between normal and special encoders? Thanks.
4:4:4 is pretty much the only (internal) working space.
webp2 is able to use a mixed per-block 4:2:0 / 4:4:4 switch right now. But considering that 4:2:0 was invented for reducing the storage of *reference frames* in video sequence, it's highly possible we'll just drop it from webp2 and just work everything as 4:4:4 with special quantization tricks for emulating 4:2:0 blocks locally.
Memory consumption of reference storage is less a problem when you're trying to blit everything on screen on a per-row basis!

And as of 4:2:2, well, it has too much a smell of the interlaced fields from the past to be very relevant nowadays.

Right now, there's a split between special encoder (ewp2) and the everyday-use encoder (cwp2) because most of casual users (i mean, outside of this specialized forum) are mostly concerned about just transcoding their good-old 8b jpeg or PNG, and would be confused by all this new stuff like HDR10, Y4M, PQ, etc.
cwp2 is supposed to work off-the-bat for 99% of everyday users, and ewp2 is supposed to empower the savvy curious ones.
Eventually they'll converge...
skal is offline   Reply With Quote
Old 8th December 2020, 12:38   #38  |  Link
Marsu42
Huba Huba
 
Marsu42's Avatar
 
Join Date: Aug 2005
Location: Palumbian Jungle
Posts: 78
Quote:
Originally Posted by Scope View Post
I did a Lossless Image Formats Comparison, it's not completely finished yet, I want to update some results and add a more denser Jpeg XL compression mode (-s 9 -E 3, but it slows down the decoding a bit).
That's great, thanks, that'll save me a lot of time doing it myself :-)

Why didn't you include heic (i.e. hevc-compression), as it's another currrent video codec with a widely used image spin-off? The performance seems to be somewhere between avif and webp, with the same problem for true rgb mode.

I'm currently trying to figure out how to do rgb-lossless heic: https://forum.doom9.org/showthread.p...94#post1930194

Quote:
Originally Posted by Scope View Post
- AVIF is lossless, but AVIF 444 is lossy (--cicp 1/13/1) the difference with the original looks like this https://i.imgur.com/MSCaY6E.png
Um, could anyonye please evaluate what this means - since 'real' lossless avif seems to be off the charts, is 444-avif (and probably the same for hevc-heic) very close or not?

Quote:
Originally Posted by skal View Post
cwp2 is supposed to work off-the-bat for 99% of everyday users, and ewp2 is supposed to empower the savvy curious ones.
Speaking as an everyday user :-) compressing 16bit-something (like dslr images) to more-than-8-bit webp2 seems to be an important use case, esp. since this is one of the main difference vs. webp1.
__________________
"The innocent have nothing to fear" :stupid:

Last edited by Marsu42; 8th December 2020 at 12:49.
Marsu42 is offline   Reply With Quote
Old 8th December 2020, 14:43   #39  |  Link
Scope
Registered User
 
Join Date: Feb 2020
Posts: 13
Quote:
Originally Posted by Marsu42 View Post
Why didn't you include heic (i.e. hevc-compression), as it's another currrent video codec with a widely used image spin-off? The performance seems to be somewhere between avif and webp, with the same problem for true rgb mode.[/url]
Because I wanted to compare mainly open, royalty-free lossless formats (preferably also web-oriented), HEIC (HEVC) has very little chance of being supported in browsers because of its patent problems, it has the same bad efficiency of true RGB-lossless image compression as AVIF, it is also less efficient than AVIF (the current difference may not be as noticeable due to less mature encoders).

Quote:
Originally Posted by Marsu42 View Post
Um, could anyonye please evaluate what this means - since 'real' lossless avif seems to be off the charts, is 444-avif (and probably the same for hevc-heic) very close or not?
https://www.reddit.com/r/AV1/comment...eg_xl/fl3y57j/
Quote:
jonsneyers
YUV444 is not actually lossless. Conversion of 8-bit RGB to 8-bit YCbCr reduces the number of colors from ~16m to ~4m. You're basically not encoding the least significant bits in the red and blue channels - and those are typically high-entropy bits, so you get a close to 2bpp advantage by not encoding them.
And https://www.reddit.com/r/AV1/comment..._webp/gbe5fw5/

Also HEIC without --matrix_coefficients=0 (heifenc -p chroma=444 --lossless -e x265 -p preset=placebo ...)
https://i.imgur.com/ysoWbAK.png
Visually the result may be the same, but it's not real lossless and other formats also have more effective near-lossless modes, when the image becomes much smaller and it would be unfair to compare such modes with real lossless.
Scope is offline   Reply With Quote
Old 8th December 2020, 17:37   #40  |  Link
skal
Registered User
 
Join Date: Jun 2003
Posts: 121
Quote:
Originally Posted by Marsu42 View Post
Speaking as an everyday user :-) compressing 16bit-something (like dslr images) to more-than-8-bit webp2 seems to be an important use case, esp. since this is one of the main difference vs. webp1.
duly noted!
skal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 08:06.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.