Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Announcements and Chat > General Discussion

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th February 2013, 00:58   #1  |  Link
xooyoozoo
Registered User
 
Join Date: Dec 2012
Posts: 199
JPEGs, WebP, HEVC

I've noticed for a while now that Photoshop's SafeForWeb JPEGS were almost always superior to the ones by Libjpeg (used in a huge amount of projects), especially at lower q levels. To try to quantify the discrepancy in performance, I ran a series of encodes through IW-SSIM, which should approximate still-image quality quite well. I made a similar post on the ImageMagick forums a couple days back, but some comments I got elsewhere made me tweak the tests a bit (lossless sources, better WebP parameters, threw in HEVC for the hell of it).

Reference lossless RGB pictures from the MMSPG: Bike, Cafe, Car, Office, Plant, Woman

Code:
convert 6.8.0-10 $f -define jpeg:dct-method=float -quality $i
kdu_compress 7.1 -full -precise -i $f -rate $i
webpe 0.2.1 -m 6 -segments 4 -q $i -pass 3
CS6 ExportOptionsSaveForWeb();  [ ... ]  optimized = true;   [ ... ]   quality = i;      
HEVC10 Main_Intra -sr 128 -d 1 -dqd 3 -aq 1 --SAOLcuBoundary 1 -q $i

JPGs were also ran through ImageOptim ( JpegOptim + jpegrescan + jpegtran )
Setup notes:
  • SSIM only accounts for luma.
  • FFMPEG's RGB -> i420 yuv default conversion is just off. There's an odd hue in some pictures that's fixed by one or more of "-sws_flags +accurate_rnd+full_chroma_int+full_chroma_inp"
  • For HEVC, luma channel of yuv output was compared directly against luma channel of yuv input. Matlab's RGB2Gray is apparently in disagreement with FFMPEG's various conversions, so I decided to sidestep the issue.
  • Matlab could directly read all other formats except for WebP, where the official libwebp decoder was used to create a PNG for input

Results:
Quote:
Edit Old set of 6 individual graphs for the 6 refs above.

Aggregate performance relative to Photoshop over 24 image Kodak set + 8 of XR set:


Comparison of 3 well-used JPEG2000 encoders versus libJPEG (aka why good tunings matter):

Results notes:
  • It's perhaps best to read the graphs horizontally ("for this metric value, A needs X-percent less bpp than B")
  • Min-y value in a graph was chosen by Photoshop's jpegs at Quality=0
  • Max-x value was chosen by me trying to leave ~1/3 white space on the left
  • My own image checks agree with the larger (>5% bpp) gaps in the results
  • Anything less, where the lines are overlapping, my preferential ranking is then JP2000=WebP>psJPG>libJPG=HEVC. The first 3 have great visual weights, while libJPG's are non-optimal and HEVC has none to speak of.
  • IW-SSIM does surprisingly well in ranking the Woman picture, as faces can be tough. However, it doesn't sufficiently mark down HEVC for overdoing the 'makeup' effect.
  • HEVC really did do that well in everything but Cafe and Woman. It's going to be killer if they actually make a decent file format out of it and throw on some psy tweaks.
  • The main image formats' lines were interpolated out of an (unnecessary...) 59 points. HEVC's line was interpolated out of 6 points. PCHIP was used.

If anything particular point on a graph interests you, feel free to ask for image uploads. Just give the [ bpp*1280*1600/8 ] value. I should be able to find something close for all but HEVC (only 6 points).

Edit: Comparison pics in my post below.

Last edited by xooyoozoo; 2nd March 2013 at 21:24.
xooyoozoo is offline   Reply With Quote
Old 20th February 2013, 01:46   #2  |  Link
IgorC
Registered User
 
Join Date: Apr 2004
Posts: 1,234
xooyoozoo,
Thank You for interesting comparsion and useful papers.

There was also the comparison between different jpeg2000 (kdu also) encoders vs jpeg, though pretty old. Not sure whether kdu is enough representative jpeg2000 encoder, anyway it was PSNR. http://compression.ru/video/codec_co...arison_en.html

Can I ask You to post lossy images in png also?

Last edited by IgorC; 20th February 2013 at 01:49.
IgorC is offline   Reply With Quote
Old 20th February 2013, 03:41   #3  |  Link
xooyoozoo
Registered User
 
Join Date: Dec 2012
Posts: 199
Oh, I forgot to mention that the test images were pulled off the MMSPG.

Quote:
Originally Posted by IgorC View Post
There was also the comparison between different jpeg2000 (kdu also) encoders vs jpeg, though pretty old. Not sure whether kdu is enough representative jpeg2000 encoder, anyway it was PSNR. http://compression.ru/video/codec_co...arison_en.html

Can I ask You to post lossy images in png also?
The reason I chose Kakadu over Jasper and OpenJPEG, the last of which is even conveniently included in most of my programs, is that the former comes with visual weights that have been known to dramatically improve subjective performance at low filesizes. Psy tweaks aren't really important in JP2K's current market niches (research, medicine, storage|archival), but we probably want them here.

I won't delve too deeply into the MSU study you linked, as it's relatively old and focuses solely on PSNR, but KDU's visual weights are on by default and does hurt PSNR quite a bit. ( Actually, I've always wondered why KDU's binaries were also so vocal about its default psy tweaks and kept warning to turn them off when testing MSE/PSNR. I guess now we know... )

And yea, I'll include some comparisons. Give me a sec to upload them. I'll only include one bpp per image though, as there are too many pictures to sort through.

Edit: Concerning Kakadu JP2k's performance on objective metrics, I forgot about this paper, which has this graph detailing how changing objective metrics could affect relative ranking.

Last edited by xooyoozoo; 20th February 2013 at 11:41. Reason: moar speling
xooyoozoo is offline   Reply With Quote
Old 20th February 2013, 10:57   #4  |  Link
xooyoozoo
Registered User
 
Join Date: Dec 2012
Posts: 199
I wasn't happy with some of the needlessly high bpp points in the pic comparison, so I tweaked things a bit. Car and Plant are the same, though. The images are also now directly linked to make things easier(?) too.

Code:
bpp  
0.53 Bike:   psJPG, libJPG, JP2000, WebP, HEVC
1.02 Cafe:   psJPG, libJPG, JP2000, WebP, HEVC
0.36 Car:    psJPG, libJPG, JP2000, WebP, HEVC
0.45 Office: psJPG, libJPG, JP2000, WebP, HEVC
0.38 Plant:  psJPG, libJPG, JP2000, WebP, HEVC
0.41 Woman:  psJPG, libJPG, JP2000, WebP, HEVC
Refs: Bike, Cafe, Car, Office, Plant, Woman

Last edited by xooyoozoo; 20th February 2013 at 11:05.
xooyoozoo is offline   Reply With Quote
Old 21st February 2013, 12:58   #5  |  Link
xooyoozoo
Registered User
 
Join Date: Dec 2012
Posts: 199
Ahem I swear I don't really like talking to myself that much. Last bit, I promise.

I was reminded of Dark Shikari's blog post on WebP a couple years back that (rightfully) tore down the weak encoder. As I don't think anyone's try to put WebP and x264 head-to-head recently, I might as well tack on x264 here too.

First, from my own experience, x264's psy-rd is unoptimized for stress, in either video or image form. If anything, it exacerbates low bitrates rather than make it more bearable.

In this particular situation, both default settings and 'tune stillimage' at the low-but-not-excessively-so filesizes chosen by Photoshop's q=0 produced unacceptable results. Tune-ssim is then at the other unacceptable end of the spectrum (well there's always tune-psnr), so I did a few arbitrary reductions of psy-rd and settled on "tune stillimage -psy-rd 0.5:0.175". Manually tweaking x264 in ways that normal users wouldn't is incredibly unrepresentative, of course, so there's that.

Images to compare to the ones above:
Code:
0.53 Bike:   x264
1.02 Cafe:   x264
0.36 Car:    x264
0.45 Office: x264
0.38 Plant:  x264
0.41 Woman:  x264

bpp measured using .264 bitstream. If you're curious about IW-SSIM results, here are the odd-numbered graphs.
My own opinion is that, except for Plant, I prefer x264 either more than or equally to the others (I did handpick the psy settings for myself) until bpp gets cut by at least another 1/3 or so from the samples above. Then sheer HEVC efficiency starts to win. Random example at 0.254 Bike: HEVC, x264.
xooyoozoo is offline   Reply With Quote
Old 21st February 2013, 13:05   #6  |  Link
naoan
Registered User
 
Join Date: Oct 2009
Posts: 152
Very interesting, is VP9 the same as WebP or can you add it to the fray?
naoan is offline   Reply With Quote
Old 22nd February 2013, 00:39   #7  |  Link
IgorC
Registered User
 
Join Date: Apr 2004
Posts: 1,234
WebP is VP8, intra profile. Google still hasn't mentioned anything about VP9 for still image compression.
IgorC is offline   Reply With Quote
Old 22nd February 2013, 04:02   #8  |  Link
xooyoozoo
Registered User
 
Join Date: Dec 2012
Posts: 199
I did a few quick tests with VP9 for myself, and I don't think there's really much to say that's meaningful.

[WebP's strength as an encoder] >> [VP9's strength as a format] at least for the time being.


Edit: Random addendum,

Lossless compression of the the first 6 graphs, the 6 ref pics, and the 6 HEVC pics (18 total):
36.60MB = PNG with pngout+advpng+optipng
28.82MB = WebP lossless on fastest "-m 0" (so fast I thought it didn't work...)
27.63MB = WebP lossless "-m 6" (ahhh that's better )

Last edited by xooyoozoo; 22nd February 2013 at 08:05.
xooyoozoo is offline   Reply With Quote
Old 25th February 2013, 00:20   #9  |  Link
xooyoozoo
Registered User
 
Join Date: Dec 2012
Posts: 199
The problem with graphs in the first post is that the data they present is too discrete. Aggregation makes for much better visualization and generalization.

To that end, we're going to take the 24 768x512 images of the Kodak image set and the 8 1680x1200 images of the Jpeg-XR test set ask the question "how much space would one save if a batch of Photoshop jpegs had each of its images encoded by something else?". The effect is that gains on each image is weighted against how much space the gains would actually save. Using Photoshop batches as a reference point also assumes that constant quality encoding is more important here than arbitrary size limits.



TL;DR: Jpegs are gonna stick around for a long while.

Last edited by xooyoozoo; 25th February 2013 at 00:50. Reason: clarifications
xooyoozoo is offline   Reply With Quote
Old 25th February 2013, 03:38   #10  |  Link
IgorC
Registered User
 
Join Date: Apr 2004
Posts: 1,234
Well, JPEG's patents have expired and it has quite good compression/quality ratio already. But new markets as of mobile devices have a challenges as a very low mobile bandwidth. Many people probably wish that a web pages with multimedia content would open faster on their phones. It's a big and fastly growing market
IgorC is offline   Reply With Quote
Old 25th February 2013, 11:44   #11  |  Link
xooyoozoo
Registered User
 
Join Date: Dec 2012
Posts: 199
Don't get me wrong, I'm up for anything more efficient to replace JPEG, and just having lossy+alpha alone makes WebP a good candidate (tangent: they should have started with those features). However, I don't think the contenders so far have what it takes to offer much momentum for change. I'll take a page from the video encoding side of things, which is even less conservative if anything, and say "50% or bust" for changing an industry.

------

I remembered your earlier comment on KDU and got around to testing Kakadu vs OpenJPEG vs Jasper.

Same test sets and hypothetical scenario as my last post. However, we'll use libJPEG as reference this time, as it'll allow more freedom in bpp points and is significantly more relevant as long as we're also discussing OpenJPEG and Jasper in the same breath.

Code:
convert 6.8.0-10 $f -strip -define jpeg:dct-method=float -quality $i $g.jpg  [...] +JpegOptim+jpegrescan+jpegtran
kdu_compress 7.1 -full -precise -i $f -rate $i -o $g.jp2
opj_compress 2.0.0 -i $f -I -r $i $g.jp2
convert 6.8.0-10 $f -define jp2:rate=$i $g.jp2


The results, unfortunately, are about what I expected based on past readings. If you think I made a mistake in encoding parameters though, feel free to correct me.

Also, if anyone's curious JPEG-XR (circa 2010, but I doubt much has changed) should perform similarly to Jasper, at least in comparison to JPEG. Vanilla JPEG comes with perceptual tunings (derived from contrast sensitivity function) that, while a bit outdated by now, are still perceptual tunings! Competing reference encoders can't just tune for PSNR and call it a day (WebP was guilty of this way back then).

Here's a couple of comparison of the three JPEG2000 encoders on Kodak 17 and Kodak 21.

Edit: Kodak 24 at ~35KB, libJPEG vs Jasper (lol)

Last edited by xooyoozoo; 26th February 2013 at 01:15.
xooyoozoo is offline   Reply With Quote
Old 11th March 2013, 03:22   #12  |  Link
NicolasRobidoux
Nicolas Robidoux
 
NicolasRobidoux's Avatar
 
Join Date: Mar 2011
Location: Copenhagen Denmark
Posts: 269
Thank you, very much, for posting this.
NicolasRobidoux is offline   Reply With Quote
Old 12th March 2013, 02:10   #13  |  Link
NicolasRobidoux
Nicolas Robidoux
 
NicolasRobidoux's Avatar
 
Join Date: Mar 2011
Location: Copenhagen Denmark
Posts: 269
Do I remember correctly that the Kodak images are not "natively" sRGB, which means that they are somewhat unrepresentative of what's put on the web these days?
NicolasRobidoux is offline   Reply With Quote
Old 12th March 2013, 04:46   #14  |  Link
xooyoozoo
Registered User
 
Join Date: Dec 2012
Posts: 199
I believe the Kodak set actually predates the creation of sRGB. However, as far as being 'representative' goes, colorspace might be the least of our concerns, considering that a third of the set has artificial frames and overlays.

The set seems to still be in full use for compression and image research, though, so I wasn't about to argue with tradition and my own concerns were settled with the inclusion of the 8 two megapixels XR images.
xooyoozoo is offline   Reply With Quote
Old 24th December 2015, 13:05   #15  |  Link
Jamaika
Registered User
 
Join Date: Jul 2015
Posts: 439
New version codec WebP 0.5.0.
https://github.com/webmproject/libwe...ses/tag/v0.5.0
- 12/17/2015: version 0.5.0
* miscellaneous bug & build fixes (issues #234, #258, #274, #275, #278)
* encoder & decoder speed-ups on x86/ARM/MIPS for lossy & lossless
- note! YUV->RGB conversion was speed-up, but the results will be slightly different from previous releases
* various lossless encoder improvements
* gif2webp improvements, -min_size option added
* tools fully support input from stdin and output to stdout (issue #168)
* New WebPAnimEncoder API for creating animations
* New WebPAnimDecoder API for decoding animations
* other API changes: - libwebp: WebPPictureSmartARGBToYUVA() (-pre 4 in cwebp) WebPConfig::exact (-exact in cwebp; -alpha_cleanup is now the default) WebPConfig::near_lossless (-near_lossless in cwebp) WebPFree() (free'ing webp allocated memory in other languages) WebPConfigLosslessPreset() WebPMemoryWriterClear()
- libwebpdemux: removed experimental fragment related fields and functions - libwebpmux: WebPMuxSetCanvasSize()
* new libwebpextras library with some uncommon import functions: WebPImportGray/WebPImportRGB565/WebPImportRGB4444


What am I missing?
Added export to PNG files. Unfortunately always deletes the alpha channel. There is no function. {dwebp.exe -png or -pnga}. There can still be saved in RGB compresji lossy.

Last edited by Jamaika; 24th December 2015 at 21:02.
Jamaika is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 18:45.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.