JPEGs, WebP, HEVC

[xooyoozoo]

I've noticed for a while now that Photoshop's SafeForWeb JPEGS were almost always superior to the ones by Libjpeg (used in a huge amount of projects), especially at lower q levels. To try to quantify the discrepancy in performance, I ran a series of encodes through IW-SSIM (http://www.ece.uwaterloo.ca/~z70wang/publications/IWSSIM.pdf), which should approximate still-image quality quite well. I made a similar post on the ImageMagick forums (http://www.imagemagick.org/discourse-server/viewtopic.php?f=22&t=22808&sid=b73139890f8dc48efcc63403000c0bfe) a couple days back, but some comments I got elsewhere made me tweak the tests a bit (lossless sources, better WebP parameters, threw in HEVC for the hell of it).

Reference lossless RGB pictures from the MMSPG: Bike (http://i.minus.com/ipy0jjCebgvyu.png), Cafe (http://i.minus.com/iCfLLKj5CNa8v.png), Car (http://i.minus.com/ibbh70jBSCGKt4.png), Office (http://i.minus.com/ix2t2MdFaClus.png), Plant (http://i.minus.com/ibqvzaKeo3j6fB.png), Woman (http://i.minus.com/iW2sxpZVtx8zj.png)


convert 6.8.0-10 $f -define jpeg:dct-method=float -quality $i
kdu_compress 7.1 -full -precise -i $f -rate $i
webpe 0.2.1 -m 6 -segments 4 -q $i -pass 3
CS6 ExportOptionsSaveForWeb(); [ ... ] optimized = true; [ ... ] quality = i;
HEVC10 Main_Intra -sr 128 -d 1 -dqd 3 -aq 1 --SAOLcuBoundary 1 -q $i

JPGs were also ran through ImageOptim ( JpegOptim + jpegrescan + jpegtran )


Setup notes:

SSIM only accounts for luma.
FFMPEG's RGB -> i420 yuv default conversion is just off. There's an odd hue in some pictures that's fixed by one or more of "-sws_flags +accurate_rnd+full_chroma_int+full_chroma_inp"
For HEVC, luma channel of yuv output was compared directly against luma channel of yuv input. Matlab's RGB2Gray is apparently in disagreement with FFMPEG's various conversions, so I decided to sidestep the issue.
Matlab could directly read all other formats except for WebP, where the official libwebp decoder was used to create a PNG for input


Results:

Edit Old set of 6 individual graphs for the 6 refs above. (http://imgur.com/a/61L9h#0)

Aggregate performance relative to Photoshop over 24 image Kodak set + 8 of XR set:
http://i2.minus.com/iFUn98JTZnONA.png

Comparison of 3 well-used JPEG2000 encoders versus libJPEG (aka why good tunings matter (http://imgur.com/a/3ayPz#0)):
http://i.imgur.com/tiiyhaI.png



Results notes:

It's perhaps best to read the graphs horizontally ("for this metric value, A needs X-percent less bpp than B")
Min-y value in a graph was chosen by Photoshop's jpegs at Quality=0
Max-x value was chosen by me trying to leave ~1/3 white space on the left
My own image checks agree with the larger (>5% bpp) gaps in the results
Anything less, where the lines are overlapping, my preferential ranking is then JP2000=WebP>psJPG>libJPG=HEVC. The first 3 have great visual weights, while libJPG's are non-optimal and HEVC has none to speak of.
IW-SSIM does surprisingly well in ranking the Woman picture, as faces can be tough. However, it doesn't sufficiently mark down HEVC for overdoing the 'makeup' effect.
HEVC really did do that well in everything but Cafe and Woman. It's going to be killer if they actually make a decent file format out of it and throw on some psy tweaks.
The main image formats' lines were interpolated out of an (unnecessary...) 59 points. HEVC's line was interpolated out of 6 points. PCHIP was used.


If anything particular point on a graph interests you, feel free to ask for image uploads. Just give the [ bpp*1280*1600/8 ] value. I should be able to find something close for all but HEVC (only 6 points).

Edit: Comparison pics in my post below (http://forum.doom9.org/showthread.php?p=1616467#post1616467).

[IgorC]

xooyoozoo,
Thank You for interesting comparsion and useful papers.

There was also the comparison between different jpeg2000 (kdu also) encoders vs jpeg, though pretty old. Not sure whether kdu is enough representative jpeg2000 encoder, anyway it was PSNR. http://compression.ru/video/codec_comparison/jpeg2000_codecs_comparison_en.html

Can I ask You to post lossy images in png also?

[xooyoozoo]

Oh, I forgot to mention that the test images were pulled off the MMSPG.


There was also the comparison between different jpeg2000 (kdu also) encoders vs jpeg, though pretty old. Not sure whether kdu is enough representative jpeg2000 encoder, anyway it was PSNR. http://compression.ru/video/codec_comparison/jpeg2000_codecs_comparison_en.html

Can I ask You to post lossy images in png also?

The reason I chose Kakadu over Jasper and OpenJPEG, the last of which is even conveniently included in most of my programs, is that the former comes with visual weights that have been known to dramatically improve subjective performance at low filesizes (http://mmspg.epfl.ch/files/content/sites/mmspl/files/shared/QoE/IQA/SPIE09.pdf). Psy tweaks aren't really important in JP2K's current market niches (research, medicine, storage|archival), but we probably want them here.

I won't delve too deeply into the MSU study you linked, as it's relatively old and focuses solely on PSNR, but KDU's visual weights are on by default and does hurt PSNR quite a bit. ( Actually, I've always wondered why KDU's binaries were also so vocal about its default psy tweaks and kept warning to turn them off when testing MSE/PSNR. I guess now we know... :D )

And yea, I'll include some comparisons. Give me a sec to upload them. I'll only include one bpp per image though, as there are too many pictures to sort through.

Edit: Concerning Kakadu JP2k's performance on objective metrics, I forgot about this paper (https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CEUQFjAB&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.170.5725%26rep%3Drep1%26type%3Dpdf&ei=UkAkUenvDYq5iwKFjYFg&usg=AFQjCNFeGDx9vMgC0AxwzH3i6rjoS138oQ&bvm=bv.42661473,d.cGE), which has this graph (http://i.imgur.com/e8QH4oI.png) detailing how changing objective metrics could affect relative ranking.

[xooyoozoo]

I wasn't happy with some of the needlessly high bpp points in the pic comparison, so I tweaked things a bit. Car and Plant are the same, though. The images are also now directly linked to make things easier(?) too.


bpp
0.53 Bike: psJPG (http://i6.minus.com/inAijkiURKqs6.jpg), libJPG (http://i3.minus.com/ipz8dc50eXNFh.jpg), JP2000 (http://i1.minus.com/ibaR1YIEgoX4ig.png), WebP (http://i5.minus.com/iP16rEr9Mplrm.png), HEVC (http://i7.minus.com/iZBQ04Dy5rh4m.png)
1.02 Cafe: psJPG (http://i5.minus.com/iw6aQUykWoIEV.jpg), libJPG (http://i5.minus.com/iQCNMLuPsS5gc.jpg), JP2000 (http://i4.minus.com/iCpsnO5Pn4Et7.png), WebP (http://i7.minus.com/iHJh5EZUs2E04.png), HEVC (http://i6.minus.com/iBIYIhlyTZltw.png)
0.36 Car: psJPG (http://i6.minus.com/iOj8ez0unn8nl.jpg), libJPG (http://i4.minus.com/iblE0WKWXfIijJ.jpg), JP2000 (http://i4.minus.com/ivodHoshMCk2N.png), WebP (http://i1.minus.com/iDc1fNlPH3FOG.png), HEVC (http://i3.minus.com/iNtJw6q75IN8s.png)
0.45 Office: psJPG (http://i3.minus.com/ixzIH84UqvZNo.jpg), libJPG (http://i1.minus.com/ibaQ6Ai1V70W5X.jpg), JP2000 (http://i2.minus.com/iEO2V9V2Oh9R2.png), WebP (http://i1.minus.com/ibfpmnVtkyIrC2.png), HEVC (http://i5.minus.com/ib2axB897JJ2vo.png)
0.38 Plant: psJPG (http://i3.minus.com/ibtn4JSRHnAQrr.jpg), libJPG (http://i4.minus.com/iWl92QJGN8CuS.jpg), JP2000 (http://i2.minus.com/iSOc76M8Nkru8.png), WebP (http://i5.minus.com/iH8pzfUQtHu0T.png), HEVC (http://i4.minus.com/iTHiQlLomGMJC.png)
0.41 Woman: psJPG (http://i4.minus.com/ijplqF9Sn4u2M.jpg), libJPG (http://i3.minus.com/iDuvsI9YxCzt6.jpg), JP2000 (http://i2.minus.com/i4OoLMbwyFvow.png), WebP (http://i5.minus.com/iB2iWBHiQu2Cj.png), HEVC (http://i5.minus.com/ib1MFpvsRIwyA0.png)


Refs: Bike (http://i.minus.com/ipy0jjCebgvyu.png), Cafe (http://i.minus.com/iCfLLKj5CNa8v.png), Car (http://i.minus.com/ibbh70jBSCGKt4.png), Office (http://i.minus.com/ix2t2MdFaClus.png), Plant (http://i.minus.com/ibqvzaKeo3j6fB.png), Woman (http://i.minus.com/iW2sxpZVtx8zj.png)

[xooyoozoo]

Ahem I swear I don't really like talking to myself that much. Last bit, I promise. :D

I was reminded of Dark Shikari's blog post on WebP (http://x264dev.multimedia.cx/archives/541) a couple years back that (rightfully) tore down the weak encoder. As I don't think anyone's try to put WebP and x264 head-to-head recently, I might as well tack on x264 here too.

First, from my own experience, x264's psy-rd is unoptimized for stress, in either video or image form. If anything, it exacerbates low bitrates rather than make it more bearable.

In this particular situation, both default settings and 'tune stillimage' at the low-but-not-excessively-so filesizes chosen by Photoshop's q=0 produced unacceptable results. Tune-ssim is then at the other unacceptable end of the spectrum (well there's always tune-psnr), so I did a few arbitrary reductions of psy-rd and settled on "tune stillimage -psy-rd 0.5:0.175". Manually tweaking x264 in ways that normal users wouldn't is incredibly unrepresentative, of course, so there's that.

Images to compare to the ones above:

0.53 Bike: x264 (http://i7.minus.com/i1srTtKrq1gwG.png)
1.02 Cafe: x264 (http://i5.minus.com/iR8VBS8dTdVGo.png)
0.36 Car: x264 (http://i5.minus.com/iV50WrFUvNH47.png)
0.45 Office: x264 (http://i5.minus.com/ib19MjFmkEnbGN.png)
0.38 Plant: x264 (http://i2.minus.com/iGZfiETV3MfUA.png)
0.41 Woman: x264 (http://i7.minus.com/iWkUOYhVFHvTl.png)

bpp measured using .264 bitstream. If you're curious about IW-SSIM results, here are the odd-numbered graphs (http://imgur.com/a/OEBh2#0).


My own opinion is that, except for Plant, I prefer x264 either more than or equally to the others (I did handpick the psy settings for myself) until bpp gets cut by at least another 1/3 or so from the samples above. Then sheer HEVC efficiency starts to win. Random example at 0.254 Bike: HEVC (http://i5.minus.com/iACpz0q6JR2Jg.png), x264 (http://i3.minus.com/ibtTvVd30TJLOP.png).

[naoan]

Very interesting, is VP9 the same as WebP or can you add it to the fray?

[IgorC]

WebP is VP8, intra profile. Google still hasn't mentioned anything about VP9 for still image compression.

[xooyoozoo]

I did a few quick tests with VP9 for myself, and I don't think there's really much to say that's meaningful.

[WebP's strength as an encoder] >> [VP9's strength as a format] at least for the time being.


Edit: Random addendum,

Lossless compression of the the first 6 graphs, the 6 ref pics, and the 6 HEVC pics (18 total):
36.60MB = PNG with pngout+advpng+optipng
28.82MB = WebP lossless on fastest "-m 0" (so fast I thought it didn't work...)
27.63MB = WebP lossless "-m 6" (ahhh that's better ;))

[xooyoozoo]

The problem with graphs in the first post is that the data they present is too discrete. Aggregation makes for much better visualization and generalization.

To that end, we're going to take the 24 768x512 images of the Kodak image set (http://r0k.us/graphics/kodak/) and the 8 1680x1200 images of the Jpeg-XR test set (http://mmspg.epfl.ch/iqa) ask the question "how much space would one save if a batch of Photoshop jpegs had each of its images encoded by something else?". The effect is that gains on each image is weighted against how much space the gains would actually save. Using Photoshop batches as a reference point also assumes that constant quality encoding is more important here than arbitrary size limits.

http://i2.minus.com/iFUn98JTZnONA.png

TL;DR: Jpegs are gonna stick around for a long while.

[IgorC]

Well, JPEG's patents have expired and it has quite good compression/quality ratio already. But new markets as of mobile devices have a challenges as a very low mobile bandwidth. Many people probably wish that a web pages with multimedia content would open faster on their phones. It's a big and fastly growing market

[xooyoozoo]

Don't get me wrong, I'm up for anything more efficient to replace JPEG, and just having lossy+alpha alone makes WebP a good candidate (tangent: they should have started with those features). However, I don't think the contenders so far have what it takes to offer much momentum for change. I'll take a page from the video encoding side of things, which is even less conservative if anything, and say "50% or bust" for changing an industry.

------

I remembered your earlier comment on KDU and got around to testing Kakadu vs OpenJPEG vs Jasper.

Same test sets and hypothetical scenario as my last post. However, we'll use libJPEG as reference this time, as it'll allow more freedom in bpp points and is significantly more relevant as long as we're also discussing OpenJPEG and Jasper in the same breath.


convert 6.8.0-10 $f -strip -define jpeg:dct-method=float -quality $i $g.jpg [...] +JpegOptim+jpegrescan+jpegtran
kdu_compress 7.1 -full -precise -i $f -rate $i -o $g.jp2
opj_compress 2.0.0 -i $f -I -r $i $g.jp2
convert 6.8.0-10 $f -define jp2:rate=$i $g.jp2


http://i.imgur.com/tiiyhaI.png

The results, unfortunately, are about what I expected based on past readings. If you think I made a mistake in encoding parameters though, feel free to correct me.

Also, if anyone's curious JPEG-XR (circa 2010, but I doubt much has changed) should perform similarly to Jasper, at least in comparison to JPEG. Vanilla JPEG comes with perceptual tunings (derived from contrast sensitivity function (https://docs.google.com/viewer?a=v&q=cache:r-kkAFALfWYJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.113.7853%26rep%3Drep1%26type%3Dpdf+&hl=en&gl=us&pid=bl&srcid=ADGEESjq_TcYvNkz6zHuOGSWzyrM_8gMSJl1j31LRipH030tgmraeBsYhi9bcJITq674Tlqq3BjuyfcFhltx7fv4B3yrrJCzJdPjEGxWu--1vAXSj9b93iTSzJOQLGg8_NRwQTWtYS11&sig=AHIEtbQrpBTFpuI0R8WpSnmmaKR5vPeZjA)) that, while a bit outdated by now, are still perceptual tunings! Competing reference encoders can't just tune for PSNR and call it a day (WebP was guilty of this way back then).

Here's a couple of comparison of the three JPEG2000 encoders on Kodak 17 (http://imgur.com/a/48B5W#0) and Kodak 21 (http://imgur.com/a/3rbYa#0).

Edit: Kodak 24 at ~35KB, libJPEG vs Jasper (http://imgur.com/a/3ayPz#0) (lol)

[NicolasRobidoux]

Thank you, very much, for posting this.

[NicolasRobidoux]

Do I remember correctly that the Kodak images are not "natively" sRGB, which means that they are somewhat unrepresentative of what's put on the web these days?

[xooyoozoo]

I believe the Kodak set actually predates the creation of sRGB. However, as far as being 'representative' goes, colorspace might be the least of our concerns, considering that a third of the set has artificial frames and overlays.

The set seems to still be in full use for compression and image research, though, so I wasn't about to argue with tradition :) and my own concerns were settled with the inclusion of the 8 two megapixels XR images.

[Jamaika]

New version codec WebP 0.5.0.
https://github.com/webmproject/libwebp/releases/tag/v0.5.0
- 12/17/2015: version 0.5.0
* miscellaneous bug & build fixes (issues #234, #258, #274, #275, #278)
* encoder & decoder speed-ups on x86/ARM/MIPS for lossy & lossless
- note! YUV->RGB conversion was speed-up, but the results will be slightly different from previous releases
* various lossless encoder improvements
* gif2webp improvements, -min_size option added
* tools fully support input from stdin and output to stdout (issue #168)
* New WebPAnimEncoder API for creating animations
* New WebPAnimDecoder API for decoding animations
* other API changes: - libwebp: WebPPictureSmartARGBToYUVA() (-pre 4 in cwebp) WebPConfig::exact (-exact in cwebp; -alpha_cleanup is now the default) WebPConfig::near_lossless (-near_lossless in cwebp) WebPFree() (free'ing webp allocated memory in other languages) WebPConfigLosslessPreset() WebPMemoryWriterClear()
- libwebpdemux: removed experimental fragment related fields and functions - libwebpmux: WebPMuxSetCanvasSize()
* new libwebpextras library with some uncommon import functions: WebPImportGray/WebPImportRGB565/WebPImportRGB4444

What am I missing?
Added export to PNG files. Unfortunately always deletes the alpha channel. There is no function. {dwebp.exe -png or -pnga}. There can still be saved in RGB compresji lossy.
http://i65.tinypic.com/2dwes1k.png