JpegSource – advanced JPEG decoder for Avisynth 2.6

[SEt]

As someone asked for JPEG decoder that does no colorspace conversions and provide YUV data directly, and I've written exactly such thing for some other project – here it is, JpegSource for Avisynth.

Current (2014.4.19) version: https://www.dropbox.com/s/rjnt0y3ead...ce_20140419.7z
Previous (2014.1.5) version: https://www.dropbox.com/s/qvm4kn27b4f7bb9/JpegSource.7z

Due to various chroma subsamplings of JPEG it's Avisynth 2.6 only plugin. And even current Avisynth 2.6 doesn't support all colorspaces that you can encounter in JPEG, like YUV440 and CMYK/YCCK, so to process them you'd have to use single-channel mode. To comply with other Avisynth restrictions multi-channel mode may also "un-crop" images a bit (to provide even width to YV16 etc.), while single-channel mode always returns exact dimensions.

Syntax: JpegSource(string file, int rec, int channel, int length, float fps_num, int fps_den)

• file – source file name
• rec – number of reconstruction passes. Default: 1.
• channel – load only one image channel with specified index (zero based). Default: -1 (load all channels).
• length – clip length in frames. Default: 1000.
• fps_num and fps_den – fps numerator and denominator. Default: 24 and 1.

MTMode to use: 1. (3 and 5 are ok too; don't use 2 or 4.)

Beside simply decoding JPEG files it can try to "reconstruct" the data. Shouldn't hurt any image. On high-quality JPEGs change is insignificant, on strongly compressed can produce quite impressive results, but won't help on recompressed files where artifacts were produced by previous compressions. Reconstruction requires SSSE3 capable CPU. For example try on this file.

Can also decode PJG-compressed JPEG files and JPEG files with arithmetic compression.

Remember that JPEGs use PC levels, Rec.601 colorspace and MPEG1 chroma position most of the time. Sadly, there is no way now for source filter to provide such information and you'd have to set correct parameters manually when converting to other colorspaces. (Actually, there is something about chroma position, but it doesn't work.)

License: free for non-commercial use, closed source. (Ad-infested software is not "non-commercial".)


Usage Notes

RGB JPEGs are decoded as YV24 etc. To get proper RGB combine them by

MergeRGB(last, UToY8(), VToY8(), "RGB24").

To correctly upsample MPEG1 chroma (most JPEGs) use

YToUV(UToY8().Spline36Resize(last.width, last.height), VToY8().Spline36Resize(last.width, last.height), last)

[Reel.Deel]

Thank you SEt. It's nice to finally be able to decode 4:4:4 jpegs natively in AviSynth (I was using jpeg2yuv and RawSource before but it was a hassle).
I quickly tried with the sample picture you linked, and I have to say the reconstruction pass does a very nice job of deblocking.

One last thing, I ran JpegSource.dll through Dependency Walker and I saw something odd, it lists "AVISYNTH.DLL" as a dependency. Is that normal? I've not seen that before.

[Groucho2004]

Originally Posted by Reel.Deel

One last thing, I ran JpegSource.dll through Dependency Walker and I saw something odd, it lists "AVISYNTH.DLL" as a dependency. Is that normal? I've not seen that before.

That happens when you statically link with avisynth.lib.

[SEt]

It's not just deblocking – it tries to remove all JPEG artifacts. For example text strongly compressed in JPEG should be way less painful to read after 3 passes of reconstruction. The interesting aspect of this "reconstruction" is that if you compress reconstructed image with exactly the same settings – you should get exactly the same JPEG file. So it's indeed kind of reconstruction or more accurate decoding unlike usual dumb blur-them-all postprocessings.

The funny thing is with such postprocessing JPEG can rival or even surpass JPEG2000 on low bitrates, what is even more true with more efficient final entropy coding like PJG.

You don't need to worry about missing runtimes with my builds. Yes, avisynth.dll as dependency is normal for 2.6 interfaces. (I know it's ugly, but that's how 2.6 interfaces work.)

[ultim]

Originally Posted by SEt

Yes, avisynth.dll as dependency is normal for 2.6 interfaces. (I know it's ugly, but that's how 2.6 interfaces work.)

There is zero need for plugins to link with the avisynth.dll binary in any way, not dynamically nor statically. It is not normal for a plugin, not even in 2.6, to work like that. If for some strange reason you do want to link with it on purpose, it's fine by me, but I felt the need to post this before other plugin authors start linking with avisynth.dll by mistake.

[Wilbert]

If for some strange reason you do want to link with it on purpose

It would be nice if it would be open source. Is there a reason that you don't want to open it?

Btw, what's YUV440? I read somewhere that it is rotated YUV422, but i have no idea what that means. Does it mean that chroma is shared between two vertical pixels (instead of horizontal)?

[innocenat]

Originally Posted by Wilbert

Btw, what's YUV440? I read somewhere that it is rotated YUV422, but i have no idea what that means. Does it mean that chroma is shared between two vertical pixels (instead of horizontal)?

Yes, chroma is shared between two vertical pixels.

[SEt]

ultim, global variable AVS_linkage is effectively linking. Be it static like here or dynamic as assignment in AvisynthPluginInit3 doesn't change the fact. If you think that assignment is the preferred way, ok, next version will link by that.

Wilbert, closed source because I don't want commercial usage of reconstruction code.
Yes, in YUV440 chroma is shared between two vertical pixels but not horizontal. It often appears as result of lossless rotation by 90 degrees of YUV422 data. Avisynth has enough flags to express such subsampling, but refuses to create frames of this type.
Btw, does anyone know how rotated YUV411 is called? I can't think of a way to express such subsampling in standard notation.

[foxyshadis]

Can you add a native VapourSynth interface? (Or have you?) The colorspace options there are more versatile.

You could also possibly take advantage of decoding into high bit depth - I've always wondered if using a floating-point iDCT directly into a high bit depth prior to transforming to RGB would solve some banding problems, even if it obviously wasn't created with more than 8 bits.

[wOxxOm]

Originally Posted by SEt

Btw, does anyone know how rotated YUV411 is called? I can't think of a way to express such subsampling in standard notation.

The standard notation describes two rows, so the only way is to use a non-standard 4-row 4-column description 4:4:0:0:0, also as an example: the non-rotated 420 would be 42020 (to convert any standard notation simply repeat the chroma numbers twice).

[SEt]

Originally Posted by foxyshadis

Can you add a native VapourSynth interface? (Or have you?) The colorspace options there are more versatile.

I have no interest in VapourSynth.

Originally Posted by foxyshadis

You could also possibly take advantage of decoding into high bit depth - I've always wondered if using a floating-point iDCT directly into a high bit depth prior to transforming to RGB would solve some banding problems, even if it obviously wasn't created with more than 8 bits.

Maybe. With standard decoding blocking is far more serious issue, but with reconstruction it might make sense to try to combat banding too. Can you post examples of JPEGs with banding problem?

[ajp_anton]

So with an unsupported colorspace, will it fail to open it, or silently convert it to a compatible format?

[ultim]

Originally Posted by SEt

ultim, global variable AVS_linkage is effectively linking. Be it static like here or dynamic as assignment in AvisynthPluginInit3 doesn't change the fact. If you think that assignment is the preferred way, ok, next version will link by that.

From the Avisynth's core view, I don't care. But it is your interest that you don't link with avisynth.dll directly, because if you do, Windows will only load your plugin if it can find avisynth.dll in the standard system paths. It will be in the system path for most installations, but not all.

[SEt]

Originally Posted by ajp_anton

So with an unsupported colorspace, will it fail to open it, or silently convert it to a compatible format?

No silent conversions here – it will fail. This plugin always provides raw data.

Originally Posted by ultim

But it is your interest that you don't link with avisynth.dll directly, because if you do, Windows will only load your plugin if it can find avisynth.dll in the standard system paths. It will be in the system path for most installations, but not all.

Not a problem if it's all you have against symbol importing. As it's plugin to Avisynth and will be loaded by Avisynth that means that avisynth.dll will already be loaded in memory. In that case import will be resolved by avisynth.dll in memory regardless of where the file is placed.

[ultim]

Originally Posted by SEt

Not a problem if it's all you have against symbol importing. As it's plugin to Avisynth and will be loaded by Avisynth that means that avisynth.dll will already be loaded in memory. In that case import will be resolved by avisynth.dll in memory regardless of where the file is placed.

To my knowledge, Windows uses the basename, and optionally the full path if supplied, when searching for already loaded libraries in memory. That means it wouldn't work if avisynth.dll is named differently, or if the symbols are supplied by a different library, like VapourSynth (as VS is able to load AVS plugins). I know you don't care about VS, but it is a real-life example of the problem I am trying to demonstrate.

Anyway, if not supporting those scenarios is fine by you, fine by me. I only brought up the issue because others might care about those few cases that you don't care about, and because of those few cases it is generally better not to link with avisynth.dll directly. With my original post I'm not trying to get you to change your code, it was just to let other developers know that there is an even better approach than linking to the core.

[SEt]

Indeed, this time you are correct, though the case of renamed avisynth.dll is kind of artificial. It was explicitly stated that it's Avisynth 2.6 plugin.

[Reel.Deel]

Originally Posted by SEt

Can you post examples of JPEGs with banding problem?

I took a picture of the sun and sky a while back and it made a good example of JPEG banding. Source is a raw photo taken with my Canon 7D and processed with Adobe Lightroom 5.

- Banding.7z - 4:2:0 JPEGs ranging from quality 0 - 100 (>54 are 4:4:4)
- Original.7z - 16-bit TIFF

Let me know if you want something with more detail.

[Nevilne]

Thanks for the filter, the reconstruction is very nice. What would be the safe number for good quality pictures? I've tested some with 3 and it didn't seem to do any harm.

Some bugs to report:
RGB jpegs open as YV24:
http://i1.someimage.com/kFS0P4R.jpg

YUV 4:1:1 chroma gets displaced (sane shamanism with avisynth chromaplacements is not enough, needs an offset)
http://i1.someimage.com/KXOlJZB.jpg
(compare red leaves of jpegsource with imagesource)

When you open the image with jpegsource, it blocks the source file. Other image source filters like ImageSource, FFImageSource don't block the file and even allow seeking after file deletion with end=999 repetition.

[SEt]

For good quality pictures doesn't really matter – filter limits itself to variation introduced by compression data loss, so change will be small anyway. I usually use 1 for realtime postprocessing in my image viewer to save time.
For highly compressed photos I prefer 1 level, as 2+ produce too soft image.
For highly compressed text 3-4 are nice for salvaging as much as possible.

RGB JPEGs are quite rare, most Avisynth filters now strongly prefer planar formats and there is no planar RGB in Avisynth yet. So that behavior is somewhat "by design". You can easily get correct RGB from that by

MergeRGB(last, UToY8(), VToY8(), "RGB24")

For correctly handling MPEG1 YUV chroma in conversions you can explicitly upsample chroma before it. For example to get correctly aligned YV24 you can use:

YToUV(UToY8().Spline36Resize(last.width, last.height), VToY8().Spline36Resize(last.width, last.height), last)

Also do note that current resizing functions always process chroma as MPEG1, while conversions process it as MPEG2 by default.

Indeed, it's possible to close file earlier – will do it in next version.


My new version of decoder can also support JPEGs with arithmetic compression – not sure if there is any interest here for working with such images.

[wOxxOm]

SEt, if it's possible at all, can you adapt your pp algorithm to make an MPEG-2 post-processor?