iip: Integrated Image Processor, e.g. DVD -> 720p HDTV

[Didée]

Hello folks,

this is an early version of iip: integrated image processor. (I don't like the name too much, but didn't come up with a better one.)

Its main purpose is upconversion from DVD resolutions to (pseudo-) HDTV resolutions. It should work, however, also for just processing at native input resolution, or even for traditional downsizing. The default parameters are chosen for upconversion, and probably should be adopted for smaller resolutions.
This script aims at natural sources only! For animated/cartoon content, you are probably better of with mf's mftoon & sharpresize.

I wrote this script for the following reasons:
- after denoising a clip, much detail is attenuated also, or even gone away completely.
- DVB broadcasting in Germany is often very poor, and this script is just what I needed for these sources.
- plain fun of fiddling around

So, what does it do exactly?

- it will test your patience!

1. Basic denoising of the clip
2. Sharpen the picture up, excluding already hard edges to avoid oversharpening. This step can be done in a small supersampled stage, to get (a little) finer granularity.
3. Big Supersampling, applying Xsharpen for better edge definition.
4. Now search for edges again. Areas without edges get the original, denoised, not-sharpened picture applied. This should give the best relative compressability for any level of detail enhancement.


The denoising is currently hard-coded to PixieDust. If you want to try any other denoiser, you must change the function itself for now - no hard job, its only one line in the function.
However, in my opinion the dust filters still are offering the best ratio of "compressability/preserved detail", and that's why I've build the whole function around PixieDust.

The basic usage looks like this:

LoadPlugin("path\to\undot.dll")           #
LoadPlugin("path\to\MaskTools_v1.4.9.dll") # These plugins
LoadPlugin("path\to\WarpSharp.dll")        #
LoadPlugin("path\to\LoadPluginEx.dll")     # are required!
LoadPlugin("path\to\dustv5.dll")           #

Removed. See update on this thread's page 6!

#

As you see, there are quite some parameters to play around with. And you MUST play around with them.
Everything depends in the quality of the input, the final resolution, your taste regarding image quality, and the bitrate you are willing to spend


Parameter description

dest_x, dest_y [int],[int]
Obviously, these are specifying the final output resolution.

duststr, dustweight, antiflicker [int],[float 0.0~1.0],[bool]
duststr is the denoising level of PixieDust. default is 2, try 1~8 depending on how noisy the source is.
dustweight: instead of using the "dusted" clip, you can mix it back with the raw input image by a percentage of dustweight.
antiflicker: I had other plans for it. For now, it simply calms the picture further down by a weak temporalsoften(). See if you like it better on or off.

ss1_x, ss1_y [float],[float]
These are the supersampling factors for the following sharpening stage. Default is 1.4, try 1.0~2.0. With 1.0 the supersampling is discarded.

detailcontr, contrast_radius, PreSharp [int],[int],[float]
These are the values for detail enhancement.
detailcontr is the strength of unsharp masking. Default is 112, try 40~80~160~255~512, this depends on the source quality and your taste.
contrast_radius is the radius for unsharp masking. Default is 2, try 3 and 4 also. This correlated with the supersampling factors above.
PreSharp is simply the value for a sharpen()-command after unsharp masking. Default is 0.8, try 0.0~1.0.
NOTE: to completely disable detail enhancement, set detailcontr=0 and PreSharp=0.0

ss2_x, ss2_y [float],[float]
These are the supersampling factors for the following Xsharpening stage. Default is 3.5 . For downsizing, go not under 4.0. For upsizing, 3.0 might be sufficient.

Xstren, Xlimit [int],[int]
These are the strength and limit factors for Xsharpen(). Default is (255,255). If you don't have the patience for big enough supersampling, e.g. for 2* supersampling only, try something like (128,23).

do_subpel [bool]
This envokes a sharpen(1.0) on the flat areas, and a blur(1.0) on the edges of the supersampled image, prior to XSharpen().

flatweight [int]
At the very end, the denoised-but-not-sharpened original clip is copied back into the flat areas of the full-processed clip. Through flatweight you can control the strength of this copying process. "0" will return the calmest picture with best compressability. Try values 0~32~64~128.


So, I suggest you play around with all the knobs, and see if you like it.

AviSynth virtuosos might want to play with the levels() commands for the masks. In this area there is probably room for improvement - you are invited.

perhaps upcoming features:
- integrated deringing
- perhaps a better flicker reduction

Have fun


- Didée

[Didée]

The script.

Copy&paste in you editor, save & import as "iip.avs".

EDIT:
Script removed. See updated version of iiP on page 6!

[Wilbert]

btw, there is still a bug in generalconvolution. (There is no division by the total number of weights ...) I hope Richard has time one day to correct this

congratulations: you win!

http://web00020.ipax.tk/video

it preserves very much detail and even adds some compared to just resizing. the sharpening is also not too strong - this should look really good viewed on a normal television.

drawback: i cannot use it as it is since i have to crop black borders from the source before doing anything (pvr250 always adds 8-10 black pixel left/right, even if it gets a normal 4:3 signal) and i almost never get a mod 16 by this. maybe you could integrate cropping in the function?

haha... was i speaking of practical usage?!
a divx5 1st-pass would need ~34 hours for one csi episode...

[Soulhunter]

A quick test... Hmmm... Yes, a nice script !!!

But I will take a closer look next weekend !!!


Bye

[Didée]

@ all

Keep in mind this is only the proof-of-concept version of iiP. The next version is almost ready, and delivers even better results. But sorry to kill your illusions - it will NOT deliver those results faster than now

The script has its name for a reason. This is not an image filter. It is an image processor (!).


Wilbert:

Yes, I'm aware of that missing feature. However, for the given purpose, it just doesn't matter - in contrary.
Of course I would *love* to use anything other than the (slow) GeneralConvolution for edge detection. But, with all the bunch of filters that are now delivering edge masks, not one of these is versatile. Either they deliver binary masks only (I need graduaded masks), or they are not pixel-centered (2*2 pseudo kernels of MaskTools), or they cannot consider all 8 surrounding pixels (vector-matrix implementation of Manao within MaskTools), or they are ... simply not implemented (Kurosu mentions the LaPlacian kernel in MaskTool's Readme, but that feature is not in the DLL!).
Perhaps it's time to post on the development forum ...


alky:

Thank you very much for testing, much appreciated. I didn't test it on analog captures at all - bought a DVB-S card some time ago
For the little problems you had - see further down.


Soulhunter:

Good timing. New version will definetly be online at weekend, perhaps even earlier. Perhaps.


Some little tips:

MOD16 issue
This is not too handy, I know. Next version will remove this limit.
For now, if you need other horizontal cropping, you should crop *after* iiP.
Example:
720ish frame, crop 4 at left, 10 at right, destination=640:
iiP(dest_x=654, ...).crop(4,top,-10,-bottom)
Yes, it's cumbersome, and it's not exact, ... it's a workaround for now. There are more evil things in the world.

Speed considerations
Shortly: good processing takes good time :P
You should try&see if, after all, you need Xsharpening + big supersampling. This alone takes huge time, and is most effective only if you are upsizing or keeping DVD resolution. For downsizing, you could probably live without it.
So, if you set ss2_x=ss2_y=1.0 and Xstren=Xlimit=0, things will go quite a bit faster. Still slow, but more bearable.

In case of a 2-pass scenario, it is not a bad idea to discard the ss2 supersampling + Xsharpening in the 1st pass, and do it only in the 2nd pass.

As of now, I am converting a 40min episode with (the upcoming) version, with the abovementioned fast 1st pass.
On my (only) Athlon1800, the first pass took 9.5 hours, second will crunch for about 16-19 hours.
Encoded framesize is 704*528 (anamorphic), upconverted from a 16:9 non-anamorphic source.
Destination ratio of bits/pixel*frame is >> 0.102 << (hello RealMedia ), and the test encodings looked ... well, not bad


For the next version, deringing will be integrated (and there are some scenes that definetly need this feature!), and the sometimes-appearing flickering of detail is mostly nailed down.

Stay tuned

- Didée

[Richard Berg]

Originally posted by Wilbert
btw, there is still a bug in generalconvolution. (There is no division by the total number of weights ...) I hope Richard has time one day to correct this

Done in the latest CVS. (Thought about doing multiplication too, but I think Overlay would be faster.)

Here's the new internal documentation:

    /** 
      * GeneralConvolution(PClip clip, int divisor=1, int bias=0, string matrix) 
      * clip     =  input video          
      * divisor  =  divides the output of the convolution (calculated before adding bias)
      * bias     =  additive bias to adjust the total output intensity
      * matrix   =  the kernel (3x3 or 5x5).  any kind of whitespace is ok, see example
      * 
      * clip.GeneralConvolution("1 2 3
      *                          4 5 6
      *                          7 8 9")
     **/

[SoonUDie]

I worked on something similar for a while, but I've given up for the time being. There are some filters I want to code before I give it another serious go, but I'm not a coder and I don't have a lot of free time right now :/

One thing I'd love but could never code myself would be a superresoultion filter - they regenerate a high resolution image from temporal information. There are a million papers available on the technique, but I haven't seen any effort to make one.

[Richard Berg]

Actually, that little example at the bottom is very old, and therefore wrong Now that there are more arguments you have to say 'matrix="..."' or prepend it with '1, 0, "..."'.

[mf]

Originally posted by Didée
[BAs of now, I am converting a 40min episode with (the upcoming) version, with the abovementioned fast 1st pass.
On my (only) Athlon1800, the first pass took 9.5 hours, second will crunch for about 16-19 hours.[/B]

Thought of VBLE yet? Or don't you have the HD space?

[Didée]

[off-topic]

Originally posted by mf
Thought of VBLE yet? Or don't you have the HD space?

Of course that's one of the common procedures. I am used to do things that way too, yes.

My excuses:

1. My HD's are currently cluttered with WAY too much unfinished projects and not-yet-touched DVB captures.

2. I'm short with HD space generally: it's only ~ 500 GB

3. Since this script is fairly new for me, too, I decided to do a conventional 1st-pass and analyze it thourougly afterwards, to see how the actual compressability will turn out, and to be able to perhaps adjust the parameters slightly for the 2nd pass.

I'm going to put 13 episodes of 40~42 minutes each, at 704*528, to one DVD+R. Hence I don't want to make (too) much mistakes with such a crawler script.

Rule#1: First know your tools exactly, *then* start doing the tricks.

BTW, I was almost sure I'll get a kick in the a** from you, mf, for the crap I am scripting here

[/off-topic]

- Didée

[mf]

Originally posted by Didée
BTW, I was almost sure I'll get a kick in the a** from you, mf, for the crap I am scripting here

Yeah, I didn't look too closely yet, but I was tickled for a moment to script a quick hack that *does* allow the denoiser to be chosen .

[Mug Funky]

hmm. i don't use generalconvolution for edge masking, but are there any advantages do just "subtract(last,last.blur(1))" and some leveling or overlaying?

i'm pretty new to edgemasking, but this method works for me in most instances (except for the odd white-dot effect when colours stray outside the CCIR range)

[mf]

Originally posted by Mug Funky
i'm pretty new to edgemasking, but this method works for me in most instances (except for the odd white-dot effect when colours stray outside the CCIR range)

Fix for that is using YV12Subtract with tol=0 twice and layering them.
Example (semi-avs cause I'm lazy):
dark = yv12subtract(tol=0)
light = invert.yv12subtract(tol=0).invert
layer/overlay/whatever(dark, light, op=lighten)

That worked for me. The idea is that when using tol=0 only the dark values come out for some wicked reason. So if you invert twice you get the light values and then you layer those over the dark ones.

[Didée]

@ mf:

...I was tickled for a moment to script a quick hack that *does* allow the denoiser to be chosen.

Now, this is a really BRILLIANT idea:

I'll do the technical art, and you take the punishment of making my stuff more user friendly!

Do we have a deal?

originally posted by Mug Funky
i don't use generalconvolution for edge masking, but are there any advantages do just "subtract(last,last.blur(1))" and some leveling or overlaying?

A good idea, Mug Funky - which I had already 2 years ago ... [edit: in November 2002]

Subtract() will leave you with an image centered around Y=127, spreading the differences towards 0 and 255 both. This is not suited for masking (most times).
Almost 2 years ago, I proposed more than once that subtract() badly needed another mode where it returns [no difference]=[0] & [full difference]=[255]. But I was talking to the wind.
Of course it is possible to adjust the 127-centered result of subtract() to what is needed; but these calculations (you need both overlaying and levelling) also need some CPU time, so that doing it from start by GeneralConvolution turns out to be about just the same effort.
Another possibility is to use "adjust.dll" for making subtract()'s output suitable ... works pretty good also, but requires YV12<->YUY2 conversion.
After all, with Convolution I have better control over what's exactly happening. (Keep dreaming of "UltraFastLaPlacianKernelConvolution.DLL" ...)


Regarding iip itself:

New version is ready. Deringing is in. MOD16 is no more required. Sharpening works a little more tasty now (for me). Speed has not improved. Custom denoising is waiting for mf

When I'll return home, I'll have to do a little more fine-tuning on the deringing part - deringing of natural sources with neither destroying nor "forgetting" something is probably a little more complicated than on animee ...

Should go online this night.

One more question:

Much overlaying is going on in this script, and ...

1. Overlay() is (yet) less optimized than MaskTool's functions, speak: it's noticeably slower.

2. MaskedMerge() of MaskTools seems to inheritely have small rounding errors: it may introduce slight color shifts.

What do you prefer ... ?

- Didée

[Didée]

Oops, didn't see that mf was replying while I was still typing ...

mf:

So one can get it to work that way, okay.

But then, how much faster is [2*subtract]+[invert]+[overlaying]+[SomeThingsMore] compared to GeneralConvolution()??

Plus, the latter still is more exact than this UnsharpMasking-like approach -- but: it is well suited to e.g. create simple deringing masks, agreed.

Should we create a new thread, about edge detection ...

- Didée

[mf]

Actually, I hate putting in the end-user stuff. I just like to release a filter and see the reactions, so I try to put in as much user-configurable stuff as possible. In that sense you could say I'm a fame junkie .

And you've given me an idea for an edge mask. It has to do with an old filter in MaskTools that's been a friend of mine for some time... Can you guess it?

[Didée]

I put (little) money on "MaskOverlay()" ? - I didn't try that one myself, not one single time

But I do remember the good old times, say a year ago, when we simply posted pure linear scripts, and simply stated "try this!".
Today, we shall produce full-featured functions, that catch every possible user error, are configurable from head to toe, do sing and dance, glow in the dark and defragment your HD in the background.
Oh my Lord. ( partly , partly )

- Didée

[mf]

It has to do with Inflate .