iip: Integrated Image Processor, e.g. DVD -> 720p HDTV

[Soulhunter]

SetMemoryMax(256)

Import("C:\Programme\AviSynth 2.5\plugins\IIP.avs")

LoadPlugin("C:\Programme\AviSynth 2.5\plugins\undot.dll")
LoadPlugin("C:\Programme\AviSynth 2.5\plugins\MaskTools.dll")
LoadPlugin("C:\Programme\AviSynth 2.5\plugins\WarpSharp.dll")
LoadPlugin("C:\Programme\AviSynth 2.5\plugins\LoadPluginEx.dll")
LoadPlugin("C:\Programme\AviSynth 2.5\plugins\dustv5.dll")

mpeg2source("C:\Dokumente und Einstellungen\Soulhunter\Desktop\Temporal\test4\test4.d2v")

iip( dest_x= 1024, dest_y= 576,
\ ss1_x = 1.414, ss1_y = 1.414,
\ duststr = 2, dustweight = 1.0, antiflicker1= true, antiflicker2= true,
\ detailcontr1=127, detailcontr2 = 255, contr_radius = 3, PixSharp=0.4,
\ ss2_x = 3.5, ss2_y = 3.5,
\ Xstren = 255, Xlimit = 255,
\ subpelstren= 1.0, flatweight = 64,
\ protect_floor= 4, protect_bias = 16,
\ dering = -60, dering_weight= 1.0, dering_floor = 16, dering_bias=8,
\ detail_floor= 1, EQ = 2,
\ warp_Y = true, warp_UV = false,
\ debug= "showall | compareH/V | protect | dering",
\ cropx=32, cropy=16
\ )


Script error: yv12subtract does not have a named argument "wideRange" -> IIP.avs, line 161...

[Didée]

masktools-v1.4.16.zip (http://www.geocities.com/manao47/Filters/)

[malkion]

Didee, great job! :cool:

The new routine looks cool. Much softer than previous editions of iip.
Further testing on its way now. Thanks.

[DarkNite]

I agree, nice additions, and great script overall.

Oh, and I now present you with the The Most Visually Appealing Hyperlink of the Doom9 Forums Award for 2004. Kudos ;)

[Didée]

I re-arranged this post for better readability of the thread.


Full function call for iiP v.0.5a:
iip( dest_x= XXX, dest_y= YYY,
\ ss1_x = 1.414, ss1_y = 1.414,
\ duststr = 2, dustweight = 1.0, antiflicker1= true, antiflicker2= true,
\ detailcontr1= 104, detailcontr2 = 208, contr_radius = 2, PixSharp=0.38,
\ ss2_x = 3.5, ss2_y = 3.5,
\ Xstren = 255, Xlimit = 255,
\ subpelstren= 1.58, flatweight = 0,
\ protect_floor = 0, protect_bias = 16,
\ dering = -80, dering_weight= 1.0, dering_floor = 8, dering_bias=12,
\ detail_floor = 10, EQ = 2,
\ warp_Y = false, warp_UV = false, exborder = false,
\ debug= "protect | dering | compareH|V | showall",
\ cropx=64, cropy=32
\ )


The script:
# iip : Integrated Image Processor, v.0.5a
#
#
# ---------------
# | It is slow. |
# ---------------
#
# New in v.0.5 :
#
# EQ [int] : Edge quality. 0=deaktivated (don't do that!!!), 1=standard (as formerly), 2=better'n'slower, 3=evenbetter'n'evenslower
# Note: when using EQ = 2|3, then it's a good idea to *DEACTIVATE* deringing in iiP (don't hit the dog twice).
#
# warp_Y, warp_UV [bool] additional Luma & Chroma Warping.
#
# Luma warping should make the picture more 'gracile' when upsizing to bigger resolution, e.g. DVD -> HDTV.
# For same resolution or downsizing, it may (?) help for sources with distorted detail. Or it may not - YMMV.
#
# Chroma warping helps on sources with bleeding colors (e.g. Enterprise: on the uniforms, the red stripes are bleeding into
# the blue cloth, which looks *very* ugly [like an aquarell that's gotten wet]. It looks much better with chroma warping.)
# On clean sources, chroma warping is a waste of CPU cycles.
#
# exborder [bool] : exclude outmost 8-pixel-border from sharpening (gradient masked)
#
# dering: old method: called with positive values for "dering"
# new method: called with negative values for "dering" (better, but slower)
#
# detail_floor [int] : was formerly hardcoded, now user-definable. default 20.
#

function Ylevels(clip clp, int a, float gamma, int b, int c, int d)
{ wicked="x "+string(a)+" - "+string(b)+" "+string(a)+" - / 1 "+string(gamma)+" / ^ "+string(d)+" "+string(c)+" - * "+string(c)+" +"
# Reminder: Yexpr = "x a - b a - / 1 gamma / ^ d c - * c +"
# return( clp.subtitle(wicked) )
return( clp.YV12LUT(Yexpr = wicked) )
}

function FineEdge( clip clp, int "div" )
{
logic( clp.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", setdivisor=true, divisor=div)
\ ,clp.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", setdivisor=true, divisor=div), "max", Y=3,U=1,V=1 )
}

function iip( clip clp,
\ int "dest_x", int "dest_y",
\ int "duststr", float "dustweight",
\ float "ss1_x", float "ss1_y",
\ int "detailcontr1", int "detailcontr2",
\ int "contr_radius", float "PixSharp",
\ float "ss2_x", float "ss2_y",
\ int "Xstren", int "Xlimit",
\ float "subpelstren", int "flatweight",
\ bool "antiflicker1", bool "antiflicker2",
\ int "protect_floor",int "protect_bias",
\ int "dering", float "dering_weight",
\ int "dering_floor", float "dering_bias",
\ int "detail_floor",
\ int "EQ", bool "exborder",
\ bool "warp_Y", bool "warp_UV",
\ string "debug",
\ int "cropx", int "cropy"
\ )
{
ox = clp.width
oy = clp.height
xcorr=(int(ox/16.0+.9999)*16)-ox
ycorr=(int(oy/ 8.0+.9999)* 8)-oy
dest_x = default(dest_x, clp.width)
dest_y = default(dest_y, clp.height)
ss1_x = default(ss1_x, 1.414)
ss1_y = default(ss1_y, 1.414)
ss2_x = default(ss2_x, 3.5)
ss2_y = default(ss2_y, 3.5)
duststr = default(duststr, 2)
dustweight = default(dustweight, 1.0)
antiflicker1= default(antiflicker1, true)
antiflicker2= default(antiflicker2, true)
Xlimit = default(Xlimit, 255) # SS<3 ? 23 : 255
Xstren = default(Xstren, 255) # SS<3 ? 216 : 255
detailcontr1= default(detailcontr1, 104)
detailcontr2= default(detailcontr2, 208)
contr_radius= default(contr_radius, 2)
PixSharp = default(PixSharp, 0.4)
subpelstren = default(subpelstren, 1.58)
flatweight = default(flatweight, 0)
protect_floor=default(protect_floor, 0)
protect_bias= default(protect_bias, 16)
dering = default(dering, -80)
dering_floor= default(dering_floor, 8)
dering_floor= dering<0 ? dering_floor*2 : dering_floor
dering_bias = default(dering_bias, 16)
dering_weight=default(dering_weight, 1.0)
detail_floor= default(detail_floor, 20)
EQ = default(EQ, 2)
EQ = (EQ>3) ? 3 : EQ
exborder = default(exborder, false)
warp_Y = default(warp_Y, false)
warp_UV = default(warp_UV, false)
debug = default(debug, "Mickey Mouse")
cropx = default(cropx, 40)
cropy = default(cropy, 20)

cropx=int(cropx/4)*4
cropy=int(cropy/4)*4
xx_ss1 = int(ox*ss1_x/16+.5)*16
yy_ss1 = int(oy*ss1_y/16+.5)*16
xx_ss2 = int(ox*ss2_x/16+.5)*16
yy_ss2 = int(oy*ss2_y/16+.5)*16
xx_small = int(ox/(abs(dering)/80+1.0)/16+.5)*16 + 32
yy_small = int(oy/(abs(dering)/80+1.0)/16+.5)*16 + 16

# Now I have the other idea - but it's too slow, yet
clp = antiflicker1 == true ? clp.temporalsoften(2,2,3,23,2) : clp

#---------------------------------------------------------------------------------------------------------------------
# Base denoising by PixieDust
#
dusted = clp.addborders(0,0,xcorr,ycorr).converttoyuy2().PixieDust(limit=duststr).converttoyv12().crop(0,0,-xcorr,-ycorr)
dusted = (dustweight==0.0) ? clp
\ : ( dustweight<1.00 ? Overlay(clp, dusted, opacity=dustweight)
\ : dusted )

soft=dusted.bicubicresize(xx_small,yy_small,.2,.4).bicubicresize(xx_ss1,yy_ss1,1.0,.0)

#---------------------------------------------------------------------------------------------------------------------
# Build EdgeMask to protect already sharp detail from oversharpening
#
YV12subtract( dusted.sharpen(0.6).sharpen(0.6), dusted.blur(1.0).blur(1.0) )

edge00 = bicubicResize(xx_ss1,yy_ss1,-1.0,1.0)
edge0=edge00
\ .YV12LUT(Yexpr="x 128 - abs "+string(protect_floor)+" - "+string(protect_bias)+" *")
\ .deflate(Y=3,U=1,V=1).inflate(Y=3,U=1,V=1).blur(1.58)
edge00 = edge00.yv12lut(yexpr="x 128 - 2 *").expand(Y=3,U=1,V=1).blur(1.58)

#---------------------------------------------------------------------------------------------------------------------
# Build DeRing'ing Mask: +++ OLD ROUTINE +++
#
dering > 0 ? expand(Y=3,U=1,V=1) : NOP
edge1b = dering > 0
\ ? yv12subtract( expand(Y=3,U=1,V=1).blur(1.58),
\ inpand(Y=3,U=1,V=1),tol=1,wideRange=true )
\ .bicubicResize(xx_ss1,yy_ss1,-1.0,1.0)
\ .yv12LUT(Yexpr="x 128 - abs "+string(dering_floor)+" - "+string(dering_bias)+" *")
\ .deflate(Y=3,U=1,V=1).blur(1.58)
\ : last
# .Ylevels(0,3.0,255,0,255)
#---------------------------------------------------------------------------------------------------------------------
# 1st supersampling stage to perform sharpening at
#

(ss1_x > 1.0 || ss1_y > 1.0) ? dusted.Lanczosresize(xx_ss1,yy_ss1)
\ : dusted

#---------------------------------------------------------------------------------------------------------------------
# Build DeRing'ing Mask & perform dering'ing: +++ NEW ROUTINE (Pre-processor) +++
#
tmpp = string(dering_floor)
edge1b = dering < 0
\ ? yv12subtract( last, last.xsharpen(255,255),tol=1,wideRange=true )
\ .yv12lut(yexpr="x 128 - abs "+tmpp+" *")
\ .expand().inflate().expand().inflate().FineEdge(dering_bias).blur(1.58)
\ .greyscale.Ylevels(19,1.6,208,0,255)
\ : edge1b
dering < 0 ? maskedmerge( last, soft, edge1b, Y=3,U=1,V=1, useMMX=true ) : NOP

#-------- iterative sharpening, currently only 3-fold: its slow enough ... --------

temp = last
shrpcnt = 0
mskcnt = 0
shrpcnt = (PixSharp != 0) ? shrpcnt + 1 : shrpcnt
shrpcnt = (detailcontr2 != 0) ? shrpcnt + 1 : shrpcnt
shrpcnt = (detailcontr1 != 0) ? shrpcnt + 1 : shrpcnt
mskcnt = (EQ>shrpcnt) ? shrpcnt : EQ

# Stage 1 : UnsharpMasking, wide radius -----------------------
detailcontr1 != 0
\ ? unsharpmask(detailcontr1,contr_radius,0)
\ : NOP
detailcontr1 != 0 && (shrpcnt==1 || mskcnt>1 && shrpcnt>=mskcnt) && (EQ>0)
\ ? MaskedMerge(last,temp,edge0.expand.inflate.inflate, Y=3,U=1,V=1,useMMX=true)
\ : NOP
mskcnt = (detailcontr1 != 0 && (shrpcnt==1 || mskcnt>1 && shrpcnt>=mskcnt))
\ ? mskcnt - 1
\ : mskcnt
shrpcnt = detailcontr1 != 0 ? shrpcnt - 1 : shrpcnt

# Stage 2 : UnsharpMasking, small radius ----------------------
detailcontr2 != 0
\ ? unsharpmask(detailcontr2, 1,0)
\ : NOP
detailcontr2 != 0 && (shrpcnt==1 || mskcnt>1&&shrpcnt>=mskcnt) && (EQ>0)
\ ? MaskedMerge(last,temp,edge0.inflate.inflate,Y=3,U=1,V=1,useMMX=true)
\ : NOP
mskcnt = (detailcontr2 != 0 && (shrpcnt==1 || mskcnt>1&&shrpcnt>=mskcnt))
\ ? mskcnt - 1
\ : mskcnt
shrpcnt = detailcontr2 != 0 ? shrpcnt - 1 : shrpcnt

# Stage 3 : per-pixel sharpening ------------------------------
PixSharp != 0.0
\ ? sharpen( PixSharp )
\ : NOP
PixSharp != 0.0 && shrpcnt > 0 && EQ>0
\ ? MaskedMerge(last,temp,edge0, Y=3,U=1,V=1,useMMX=true)
\ : NOP
undot()

#---------------------------------------------------------------------------------------------------------------------
# DeRing'ing of sharpen'ed clip: +++ OLD ROUTINE (Post-processor) +++
#
dering > 0 ? MaskedMerge(last,last.unfilter(-dering,-dering),edge1b.bicubicresize(xx_ss1,yy_ss1),Y=3,U=1,V=1,useMMX=true)
\ : NOP

# Now I have the other idea - but it's too slow, yet
antiflicker2 == true ? temporalsoften(1,2,3,23,2) : NOP

#---------------------------------------------------------------------------------------------------------------------
# 2nd supersampling stage to perform SubPel operation & XSharpening at
#
# Notes: Luma Warping - may help for sources that are hard to get a "clean" picure from.
# - Should help making the picture a little more "gracile" when doing DVD -> HDTV upsizing

xx_ss2 != ox || yy_ss2 != oy ? LanczosResize(xx_ss2,yy_ss2) : NOP

subpelstren != 0.0 ? blur(subpelstren) : NOP

warp_Y ? awarpsharp(depth=(ss2_x+ss2_y)/2*3,thresh=0.5,blurlevel=2,cm=0) : NOP

Xstren != 0 && Xlimit !=0 ? xsharpen(Xstren,Xlimit) : NOP

LanczosResize(dest_x,dest_y).undot()

#---------------------------------------------------------------------------------------------------------------------
# Build new EdgeMask of the enhanced clip
#
edge2 = bicubicresize(int(dest_x/1.75/16+.5)*16,int(dest_y/1.75/16+.5)*16,1.0,.0)
\ .DEdgeMask(0,255,0,255,"-5 -7 -5 -7 48 -7 -5 -7 -5",Y=3,U=1,V=1,useMMX=true,setdivisor=true,divisor=1)
bordermask = blankclip(edge2,width=dest_x-16,height=dest_y-16,color=$FFFFFF)
\ .addborders(2,2,2,2,color=$8F8F8F).addborders(2,2,2,2,color=$6F6F6F)
\ .addborders(2,2,2,2,color=$202020).addborders(2,2,2,2,color=$000000)
edge2 = edge2
\ .levels( (detail_floor+int(detailcontr1/15+detailcontr2/30+PixSharp*15)), 4.0,128,flatweight,255,false)
\ .bicubicResize(dest_x,dest_y,1.0,0)
edge2 = exborder ? edge2.logic(bordermask,"min") : edge2
edge2 = edge2
\ .inflate(Y=3,U=1,V=1).inflate(Y=3,U=1,V=1)
\ .blur(1.58).greyscale()
\ .inflate(Y=3,U=1,V=1).inflate(Y=3,U=1,V=1)
\ .temporalsoften(1,63,63,255,2)

#---------------------------------------------------------------------------------------------------------------------
# Put original de-noised & not-sharpened clip into areas still appearing flat, after all that sharpening
#
MaskedMerge(dusted.lanczosresize(dest_x,dest_y),last,edge2,Y=3,U=1,V=1,useMMX=true)

#---------------------------------------------------------------------------------------------------------------------
# Chroma Warping - often useful for sources where colors are "bleeding". Superfluid for clean sources.
#
warp_UV ? MergeChroma( aWarpSharp(depth = 8.5 * (dest_x/ox + dest_y/oy), thresh=0.5, cm=1, blurlevel=2) ) : NOP

#---------------------------------------------------------------------------------------------------------------------
# Visualizations
#
debug == "dering" ? overlay(last,blankclip(last,color=$00A000),mask=edge1b.bicubicresize(dest_x,dest_y),mode="blend",opacity=dering_weight)
\ : NOP
debug == "protect" ? overlay(last,blankclip(last,color=$00A000),mask=edge0.bicubicresize(dest_x,dest_y),mode="blend",opacity=0.75)
\ : NOP
debug == "detail" ? overlay(last,blankclip(last,color=$00A000),mask=edge2.bicubicresize(dest_x,dest_y),mode="blend",opacity=0.75)
\ : NOP
debug == "compareH" ? stackhorizontal(clp.lanczosresize(dest_x,dest_y).crop(cropx,0,-cropx,-0).addborders(0,0,4,0,color=$800000).subtitle("Original"),
\ last.crop(cropx,0,-cropx,-0).subtitle("iiP") )
\ : NOP
debug == "compareV" ? stackvertical(clp.lanczosresize(dest_x,dest_y).crop(0,cropy,-0,-cropy).addborders(0,0,0,4,color=$800000).subtitle("Original"),
\ last .crop(0,cropy,-0,-cropy).subtitle("iiP") )
\ : NOP
debug == "showall" ? stackvertical( stackhorizontal( edge0 .greyscale().lanczosresize(dest_x,dest_y).crop(cropx,cropy,-cropx,-cropy)
\ .addborders(0,0,4,4,color=$800000).subtitle("Protection of sharp edges"),
\ edge1b.greyscale().lanczosresize(dest_x,dest_y).crop(cropx,cropy,-cropx,-cropy)
\ .addborders(0,0,0,4,color=$800000).subtitle("DeRing'ing") ),
\ stackhorizontal( edge2 .greyscale().lanczosresize(dest_x,dest_y).crop(cropx,cropy,-cropx,-cropy)
\ .addborders(0,0,4,0,color=$800000).subtitle("Detail areas"),
\ last .crop(cropx,cropy,-cropx,-cropy).subtitle("iiP") ) )
\ : NOP

return(last)
}

#

[Didée]

Parameter description (only parameters new in iiP v.0.5)

EQ

Edge "quality" parameter.
Specifies how many of the sharpening steps will be protected by the protection mask.

0 = deaktivated. Don't ever choose that for encoding!! - but it's nice for visualizing why edge protection is needed at all ;)
1 = protect only the last sharpening step. "Standard", as in older versions.
2 = protect the first & the last sharpening step. Recommended.
3 = protect all three sharpening steps. Minimalizes any risk of producing oversharpening, but lessens the overall contrast gain.


warp_Y, warp_UV , [bool]

These will activate additional luma warping, resp. chroma warping. (Uses MarcFD's aWarpSharp plugin.)

Luma warping - should help in making the picture a little more 'gracile' when upsizing to bigger resolution, e.g. 1:1 PAR -> anamorphic PAR, DVD -> 720p, or such. For keeping same resolution or downsizing, perhaps it may help on sources with distorted detail by "straightening" it. Or it may not help - YMMV.
Luma warping is performed in 2nd supersampling stage, prior to XSharpening.

Chroma warping - should help a lot on sources with bleeding colors. I put this in, since I had big problems on my Enterprise DVB captures: on the starfleet uniforms, the red stripes are badly bleeding into the blue cloth. This looks *very* ugly, like an aquarell that's gotten wet. It looks much better with chroma warping.
On clean sources, chroma warping mostly is a waste of CPU cycles.

Note: aWarpSharp's parameters are hard-coded. Depth is automatically scaled by the supersampling factors (Y) resp. by the [original:destination] size ratio (UV).
If someone thinks more configuration is needed, tell me. For me it works fine the way it is.


[b]New deringing routine

Now you can choose between to routines for deringing:

The "old" routine can be used as formerly. Parameters have not changed. Note that this routine is a post-processor, after the sharpening, but prior to XSharpening.

The new routine uses another way to create the deringing mask (utilizes XSharpen to find areas-of-interest). The current processing method for the found areas is very sophisticated: "Downsize(softbicubic).Upsize(verysoftbicubic)".

It acts as a pre-processor, prior to the main sharpening steps, but already in the first supersampling stage.

To use the new deringing routine, assigning a negative value to the dering parameter.

The parameters dering_floor and dering_bias behave differently with this routine. It's a bit difficult to describe ...
dering_floor is sort of a sensitivity against edges: the bigger this parameter, the more areas around weaker edges will be considered.
dering_bias can be seen as some sort of multiplier, or normalizer ... how to tell? (Internally, it's the "divisor" parameter for MaskTool's "DEdgeMask" function.) The smaller dering_bias, the brighter the mask will get, and the more areas found by dering_floor will be kept. The bigger dering_bias, the more weaker areas will get cancelled from the mask again, and the darker the dering-mask will become.

To get a feeling for this, please play a little with these two parameters & the new routine, while having set debug="showall". A good starting point is floor=8, bias=12. It seems to me that bias should mostly be in the range bias = [floor, floor*2] - but who knows what different sources may require. This one is new to me, too ;)

dering = -200 ~ -1 tells how strongly the affected areas will be blurred through the bicubic resizers. Recommended range is from "-160" for strong processing to "-20" for weak processing.
Attention: If you use a too big number here, you'll get artefacts near edges! Try "dering = -350" to see what I mean. You should not exceed -160, perhaps evev -200. Look out when artefacting starts to appear. "dering = -120" gives already pretty strong processing.

The dering_weight parameter is not used with the new deringing routine.


exborder

This is a switch to exclude the outmost border from sharpening.

It is not uncommon, even for otherwise HiQ DVD sources, to have artefacts in the border areas. You now these bright and/or dark lines that often appear near the top/bottom letterboxing, and on the frame's sides as well.
But, not everyone is willing to crop these additional 2 ~ 8 pixels from the sides, to get rid of that crap. Now, if you're cropping so tight that source's border artefacts are kept, then at least they shouldn't be sharpened ;)
Set [i]exborder=true to do so. The used masking draws a "sinus sweep" (sort of) from 100% exclusion to ~15% exclusion, over the outmost 8 pixels on all sides of the frame.
There is nothing to configure. Tell me if you want something.


[b]detail_floor [int]

This is part of the controlling how weak a detail is still considered to be "detail", in order to be copied to the output frame from the sharpened clip. Remember: areas considered to be "no-detail" are copied from the PixieDust'ed, not-sharpened clip.
Look at the "Detail Mask" in debug mode to get a feeling for it. Former versions had this parameter hardcoded to "23". New default is "20". Negative values are possible, in case you consider them needed.

[Didée]

Explanations, thoughts, tips and (speed) considerations

- or -

Didée's usual, dreadfully boring blabla ;)



Regarding the EQ parameter:

iiP performs up to three chained sharpening steps: UnsharpMasking-wide + UnsharpMasking-narrow + PixelSharpen.
Generally, any unprotected sharpening endangers oversharpening of high-contrast edges. With chained sharpening, it gets even worse: the first sharpening greates artefacts, the second one sharpens also those artefacts, the third one ... you get the idea. Without any protection, you'll get those ugly halos, and you get them even with additional borders, themselves emphasized:
first sharpener gives halos, second sharpener gives those halos a hard border, the third makes everything even more annoying.

Former iiP versions internally created exactly such an over-oversharpened image, and used only one MaskedMerge() operation after all sharpening, to exclude hard edges from getting copied from the sharpened clip into the working clip.
Although multi-protection was on my mind from the very start, I wanted to avoid the additionally needed processing time. And I thought the one-step protection would be sufficient to exclude all oversharpening artefacts. Well, it is mostly, but not always. Previous iiP versions let some artefacts sneak through, especially when strong sharpening values were used.

With v.0.5, multiple protection may be specified through the EQ parameter.

EQ=1 is exactly the same as formerly. Everything is sharpened-up without protection, and then copied to the working clip according to the protection mask.

The most "exact" way to do the image sharpening is to protect *every* step by the protection mask. This is done by specifying EQ=3. The result should be close to totally free of oversharpening artefacts produced by iiP itself. Oversharpening already present in the source is anther story -> DeRinging.
With EQ=3, the working chain is " 1.sharpening -> MaskedMerge -> 2.sharpening -> MaskedMerge -> 3.sharpening -> MaskedMerge ".

However, EQ=3 lessens the possible gain of detail contrast quite a bit. From my experience so far, EQ=2 is a very good compromise.
EQ=2 will protect the first and the last sharpening step - whichever those might be, in case you are using only two of the three possible sharpening steps. (Sounds simple, and basically it IS simple. However, to make this work correctly for all possibilities of choosing only two of the three sharpeners, required some more lines of code than I expected...)
EQ=2 should give the best trade-off between speed and edge quality. There should hardly be any artefacts coming through, and the contrast gain is definetly better than with EQ=3. Hence I recommend to use EQ=2 - unless you figure out that your particular source needs EQ=3, or can get away with EQ=1.

EQ=0 is a major bad idea for encoding, of course. However, by setting EQ to zero, it gets blataneously obvious why all this protection stuff is needed at all. Isn't it?

Needless to say: the higher EQ you use, the slower the script will run. Each EQ increment requires another instance of MaskedMerge()'ing the luma plane over - and this is on the 1st supersampling stage, so this operation is not for free.

OTOH: the higher EQ you use, the better compressability you'll get for the output.



(New) Deringing Routine

The biggest problem with self-made deringing is: to create a mask that "fits" really good onto the problematic areas.

Firstly, we want to catch as much of the problematic areas as possible, preferebly all of them, only them, and nothing else.

Secondly, while catching all of them, it must be avoided to mask *across* any edges in the image. We want the mask to be present in the "ultimate neighborhood" of hard edges, to catch halos and mosquito noise. In areas where the mask *covers* the edge, we would force the edge itself to get blurring (depending on the used processing mthod, that is). This is probably not what we want to do.

Thirdly, there is the question of which kind of processing should be actually performed on the masked areas.

(1) Is still unsolved. There's no distinction between "ah, here are edge artefacts, we must apply deringing here" and "yup, this area is clean, no deringing is needed". At least for edge halos, there are ideas of how to do indeed *that*. But it's not possible to do that in a script - if you can code plugins, and have a little spare time, just ask me. Ideas is what I have PLENTY.

(1)+(2) - The new deringing routine utilizes XSharpen to find areas-of-interest, basically by taking the clip, and subtracting an XSharpen'ed version from itself. On this difference image, a quasi-Sobel edge detector is performed. Finally, everything gets extended and feathered a little - there is the new deringing mask.
It has a definetly better respose to edge areas than the old method, which was based on an UnsharpMask - difference image. Still the new method's response to hard edges is greater than to weaker edges. The danger of masking-across-edges is magnitudes lower than with the old method. However, the "tightness" at which the new mask is "fitting" to either side of the edges is not perfectly distributed - but I consider it quite usable as it is. Further improvement might be possible.

(3) The currently used processor for deringing is the most lame one can think of:
"BicubicResize(small,soft).BicubicResize(orig,verysoft)" is what actually is done to the dering'ed areas - and this works surprisingly effective, handling both halos and mosquitos pretty well, without getting all too obvious. There is danger, of course, of loosing small detail that is present in the direct neighborhood of strong edges. (How do you want to distinguish small detail from mosquito noise - to a machine it looks the very same).
The strength of dering-processing defines at what scale the image will be downsized, before upsizing again. "dering=-150" will scale down by a factor of ca. 2.8, "-100" by ca. 2.2, "-50" by ca. 1.6. Values numerically greater than 160 might cause artefacts. Try "-350" to force those artefacts. Then, knowing how they look like, you know what to look for if you're forced to use very high settings.
Values up to "-150" should be safe, and strong enough for almost all cases. "-40" to "-80" should be a good range for rather clean sources - but let your own eyes decide.


As already stated, the old routine can be used just as formerly. The new routine is invoked by specifying negative values to the dering parameter.

It should be mentioned that the old routine acted (and still acts) as a *post* processor. That is, after the main sharpening, but before the XSharpening. Together with the fact that it partly masks *across* edges, it was used to fight against source-artefacts as well as artefacts created by iiP itself (through too little edge protection). The blurring of some edges then was (partly) restored by the following XSharpening.

The new routine now acts as *pre* processor, i.e. before any sharpening is applied (but after the base denoising). That's the preferred way to do it, now that EQ is available.


EQ and New Deringing

The new deringing is *noticeably* slower than the old one. That's just how it is, and there's little that can be done.
You should carefully check your source, if it needs deringing at all. Since selfmade artefacts now can be avoided through higher EQ values, perhaps you simply don't need any deringing!
In particular, I would not recommend to use the new deringing with strong values, together with EQ=3. This will result in very slow processing, and will give an output that is ... smooooth. Of course there might be sources out there that require exactly this. But generally, you should not go higher than EQ=2 when using somewhat strong deringing. With low or medium deringing, I'd say EQ=2 is the one to use. With strong deringing, and/or when using some lower sharpening values or less sharpening steps, you might be well off with EQ=1 as well.

All I can advise you to do is this:
Watch out for different problematic scenes, and then figure out which combination of features copes the best with all of them, on average.


Luma Warping

Well, not much to say here - exept that it uses MarcFD's "aWarpSharp" plugin, which you should have available. Try it, and see for yourself if you like it or not. Or if you can see any difference at all.
The effect should be minor, and that's how it is supposed to be. Assumingly, some people would like to hit me for the mere fact that I'm using WarpSharping on natural video soures. However, it is performed in the "big" 2nd supersampling stage, with a relative small depth value.
The intended effect is to get some minor thinning/straightening of image features, and it is mostly meant to be used when you actually *upscale* your clip through iiP. When you're doing the old "anamorphic 720*yy --> 1:1 640*yy" game, then there is probably not much to gain with lume warping. But try and see yourself.

Thanks to the fact that MarcFD managed to get his aWarpSharp pretty fast, the impact on speed is not so bad.


Chroma Warping

This one is not intended for general usage. But you might come across sources where it can do small wonders!
I implemented it for sources with bleeding colors, and there it generally achieves very noticeable improvements. One example: For the ones that know the (new) Enterprise series (Cpt. Archer, T'Pol, Dr. Phlox ...), you might have noticed that the blue uniforms with red stripes are particular difficult - the stripes are "bleeding" into the blue cloth like ink flows into ink paper :( For me, chroma warping does a brilliant job there.

UV warping is performed on the final output clip, on destination resolution. It use chroma mode 1 - "warp chroma with luma bump map".

This one also is not too expensive, performance-wise.

I think it doesn't make much sense on souces with clean colors (but mf might disagree...)

***

Ah, I'm out of time now.

If you have any questions, go ahead. I'm all ears.


- Didée

[Chainmax]

Wouldn't it be better to just link to a premade avs instead of posting the whole script?

[malkion]

Chainmax, I dont think Didee wants to setup linkage space.. I find it insignificant either way.

Didee, is there any possible hints you could provide to help us narrow the parameters for the various floors and bias settings with the debug view? Also if it helps to set the various other parameters, like the newer dering parm, I mean, what do you personally use the debug view for? I noticed prior to this version you had coded some yuv analyzing which you could call on to check out the video (or was it on limitedsharpen). Also, what are the usage for the current script's optional coding? such as the following:

1) the #ylevels call on the old dering routine.

2) edge00 = edge00.yv12lut(yexpr="x 128 - 2 *").expand(Y=3,U=1,V=1).blur(1.58)

this line is found in the build edgemask for protect floor routine. edge00 is no longer called in the debugview section.

these 2 are from the new 0.5a version. I guess I saw some more from a old version no longer worth mentioning. :D

there are some similar #calls in the limitedsharpen script, but thats for a different thread. I'll be doing further tests of course. These questions arent that important since it has nothing to do with the actual video encoding.

[Soulhunter]

Originally posted by Didée
masktools-v1.4.16.zip (http://www.geocities.com/manao47/Filters/)
Sorry, it was late n' there are way too much MaskTools versions on my HD !!!


By a first small preview, I was able to spot out some ugly blocks... :(

Think they are related to PixieDust !!!


Btw, the next eye riddle... (http://forum.doom9.org/showthread.php?s=&threadid=78856)


Bye

[malkion]

You can try adding .BlindPP(quant=8,cpu2="xxxxoo",moderate_h=40,moderate_v=60)

after dusted. may not help during preview, but according to some people
it can catch most blocks during encoding.

[Soulhunter]

Yes, but this introduces too much quality degradation IMO... :(

Think Deen would be a good replacement for PixieDust !!!


Bye

[Didée]

Originally posted by Soulhunter
[B]Sorry, it was late n' there are way too much MaskTools versions on my HD !!!
Heheh, yes, same on my side. That's why on my HD, *all* versions of MaskTools are named with version number: "MaskTools_v1.4.16.dll" ;)

By a first small preview, I was able to spot out some ugly blocks... :(
Think they are related to PixieDust !!!
Yes and yes. Old "problem" of PixieDust - I don't want to discuss this all over again. I'd say, it's a trade-off you have to pay for the (currently) unreached, unbeaten, unbelievable best overall denoiser+smoother we have available for usage. /*bites lips to not start lenghty elaborating*/

You should actually try something like malkion suggested:
by malkion
You can try adding .BlindPP(quant=8,cpu2="xxxxoo",moderate_h=40,moderate_v=60)
Previously, I even had a similar line in the script, it was only outcommented.
Should really go in as a configurable option, so you can either use it or not, without having to mess with the script.

IMHO, s-th like BlindPP(Quant=12,cpu2="xx??oo",moderate_h=45|50|55, moderate_v=55|60|65" doesn't hurt the output noticeably (iiP still is to come after this line ;) ), but helps a lot against the blocking. It won't vanish, but it gets much better.

by Soulhunter
Think Deen would be a good replacement for PixieDust !!!
If you think so - go ahead and just try it! Very easy to do:

Just set "duststr=0", then PixieDust is completely disabled. As replacement, call your denoiser-of-choice immediately before iiP():

...
Deen(whatever)
iiP( 8064, 6048, duststr=0, ...)

I am curious what results you'll get, regarding quality, and not less regarding compressability :D
To have said that: I have close to zero experience with Deen. Why? Because its basic working principle is the same as Convolution3D. And the latter is one of the filters that disappeard (almost) completely out of my scripts, over time.

As said, right now I'm not in the mood to discuss PixieDust again ...
But I do have my certain reasons to use PixieDust exclusively, this you can believe.
Every heard something about "noise cutoff"? Tip: Think about the general principle of limiting thresholds ...


Preparing some more tips now.


- Didée

[malkion]

On my PC, I've noticed as well, that if you're encoding to anything greater than original DVD resolution (havent found the threshold yet) with pixiedust, the result clip is borked. I tried PixieDust on a HDTV source, and it didn't go too well. On normal DVD resolution source, it's fine. PixieDust doesn't like a source at 1280,720.

[Didée]

Chainmax:

Better it would be, yes.

In front of my mind's eye, I see a little website, holding all my scripts, nice colorful manuals for them, and preferably several dozens of MB's for offering demo clips to download.

You organize the webspace, and I'll upload. :D


malkion:

Indeed. All Dust flavours by Steady are limited by framesize. But remember, Steady appeared out of nowhere with his Dust package by a time when Conv3D was "the magic filter" - early 2003, that was. Times have changed a lot since then. By then, we thought not very much about processing HDTV sources. For sure we didn't think about *motion compensated* processing of such big framesizes ... not to speak about encoding ;)
A big pity is that you can't call Dust more than once on the same clip ranges, because of some internal variable's stuff, as it seems. Otherwise you could crop with a little overlap, blow the dust over it, and re-stack it together.
If you definetly want to do it, you have to create four avisynth scripts, each calling the same source, each cropping one quadrant+overlap, dust it, crop overlap, return it. Then you create a script that calls these four scripts by *AviSource*, and stack them together.
With a little luck, it might work.

Hmmh ... time to check out Manao's mvtools - although I've the feeling they still have some way to go to reach Dust's level, as far as denoising is concerned. Actually I can't tell.

[edit]

still to malkion:

Excuse me, but ...
I noticed prior to this version you had coded some yuv analyzing which
you could call on to check out the video (or was it on limitedsharpen)
I did ... WHAT ? :eek:

I have no idea what that should have been. Please explain closer. Do you by any chance mean an optional "colorYUV(analyze=true)" call to show the used level ranges?

Oh, and "edge00" is only a relict from developping, it is not used at all. Moreover, the usage of edge00 in the debug section was ... a li'l mistake ;)


- Didée

[Sirber]

Could you do a ZIP with all needed stuff? Thanks :D

[scharfis_brain]

didée:

jep, multiple calls of ...dust doens'Ät work, but what about this? :


a= crop(upperleft)
b= crop(lowerleft)
c= crop(upperright)
d= crop(lowerright)

a+b+c+d

pixiedust(..)

a= trim(upperleftframerange)
b= trim(lowerleftframerange)
c= trim(upperrightframerange)
d= trim(lowerrightframerange)

stackhorizontal(stackvertical(a,b),stackvertical(c,d))

[Didée]

Originally posted by Sirber
Could you do a ZIP with all needed stuff? Thanks
So that you can feed iiP's output into any sort of RealMedia encoder?

Do you think I'm CRAZY?

:D

Sure, Sirber. ;)


Scharfi:

Ah, interesting idea. Worth trying out. (But I put no money on it.)

Scharfi: The (failing) sample, please ...

- Didée

[Sirber]

RealVideo, RealVideo, RealVideo!!!!!!! :devil:

Want me to host you?

[Soulhunter]

Originally posted by Didée

Deen(whatever)
iiP( 8064, 6048, duststr=0, ...)

Hehe, this is what I did before... :D

But I wasn't sure if your script doest need PixieDust's processing to do some other funky stuff !!!

Originally posted by Didée
I am curious what results you'll get, regarding quality, and not less regarding compressabilityWhen I find the time, I will post some samples... ;)


Btw, progress with the new eye riddle ???


Bye

[Didée]

One correction of my own false comment, before you start out:

To disable Pixie, use dustweight = 0.0 ! When setting "duststr=0.0", it is also (effectively) disabled ... but PixieDust would first do its full work, changing nothing, and then get copied over ... a pretty slow No-Op. By zeroing dustweight, Pixie will actually be bypassed - that's what you want to do.

But remember:

The more *you* like the preview you're getting -

- the more your *encoder* will HATE it ;)


Me has taken some screenshots, wait a little.


- Didée

[Didée]

A case example.

Yesterday I did my first full encoding with iiP v.0.5a. You see, I myself am still learning to use it ;)

Target was Enterprise episode 1x09 - "Die Saat". (am too lazy now to browse for the original title. perhaps "The Seed" - but probably not)

Here are some screenshots comparing the naked RAW DVB stream to iiP's output. Note that this is an upscale from 4:3 letterboxed to 16:9 anamorphic:


source-1:
http://img111.imageshack.us/img111/3425/1x091asourcevh0.png

iiP-1:
http://img50.imageshack.us/img50/998/1x091biipbr2.png


source-2:
http://img486.imageshack.us/img486/7166/1x092asourcerq9.th.png (http://img486.imageshack.us/my.php?image=1x092asourcerq9.png)

iiP-2:
http://img486.imageshack.us/img486/4876/1x092biiplg6.th.png (http://img486.imageshack.us/my.php?image=1x092biiplg6.png)


source-3:
http://img50.imageshack.us/img50/1931/1x093asourceou8.th.png (http://img50.imageshack.us/my.php?image=1x093asourceou8.png)

iiP-3:
http://img50.imageshack.us/img50/4828/1x093biiplm2.th.png (http://img50.imageshack.us/my.php?image=1x093biiplm2.png)


source-4:
http://img41.exs.cx/img41/9159/1x09_4a_source.png

iiP-4:
http://img41.exs.cx/img41/7486/1x09_4a_iiP.png


source-8:
http://img45.imageshack.us/img45/9435/1x098asourceut5.th.png (http://img45.imageshack.us/my.php?image=1x098asourceut5.png)

iiP-8:
http://img275.imageshack.us/img275/3058/1x098biipzq3.th.png (http://img275.imageshack.us/my.php?image=1x098biipzq3.png)

source-9:
http://img107.imageshack.us/img107/4268/1x099asourcete6.th.png (http://img107.imageshack.us/my.php?image=1x099asourcete6.png)

iiP-9:
http://img490.imageshack.us/img490/6310/1x099biipua8.th.png (http://img490.imageshack.us/my.php?image=1x099biipua8.png)


The used script looked like this:
v1=mpeg2source("ENTERPRISE_0327_1.d2v", idct=7, cpu=4,moderate_h=45,moderate_v=60,showQ=false)

v1.trim(12,60029)
crop(8,72,-8,-72)

MyDelogo()

## version05a
iip( dest_x = 708, dest_y = 532,
\ duststr = 2, dustweight = 1.0,
\ ss1_x = 1.414, ss1_y = 1.414,
\ detailcontr1= 127, detailcontr2=255+40, contr_radius=2, PixSharp=0.38,
\ ss2_x=3.25, ss2_y=3.25,
\ Xstren=255, Xlimit=255,
\ subpelstren = 1.0, antiflicker1=false, antiflicker2=true,
\ flatweight = 0,
\ protect_floor = 0, protect_bias = 12,
\ dering = -80, dering_weight = 1.0,
\ dering_floor = 10, dering_bias = 12,
\ EQ = 2, detail_floor = 15,
\ warp_Y = true, warp_UV=true, exborder = true,
\ debug="-showall", cropx=36, cropy=56
\ )

crop(2,2,-2,-2)

colorYUV(cont_u=18,cont_v=12)

limiter()

return last

Notes:

Be sure to view those screenshots at an sufficient brightness/gamma level, in particular the dark ones. You know, all those parts that are invisible, "lost in the dark", in normal desktop viewing, spring to the eye very obviously when played through overlay in a dark room ...

As you can see from the output, I was careful with the sharpening & enhancing. One can very easily overdo things with iiP - but of course I don't want to do over-processing. My goal with this source is to bring it back to what it should have been in first place (and what it *was*, originally. But we dumb TV consumers are getting fed with such big crap sometimes, it's just ridiculous).
Moreover, my aim is to put each season onto 2 DVDs - that makes 13 episodes, each about ~42 min, per DVD. Given the resolution, you can do the math yourself ;) -- Below the summing line, I end up with an average bitrate of 1050 kbps, taken the overall for all episodes. This really isn't that impressing high for a framesize of 704*528 ;)
I'm ending up with an average P-quantizer of 4.0~4.2 using SixOfNine-HVS in XviD. "Translated" to the standard matrix, that would be a quant of ~2.6.

The source is somewhat noisy - not too bad, but definetly too much to simply pipe the source unfiltered into XviD. This results in annoying "floating wall" artefacts.

For my taste, the source is also too blurry. You see I've used relatively high sharpening values - "detailcontr2" is already in overdrive range. For cleaner sources, one could simply go higher with "detailcontr1", but on these particular sources, bitrate climbs way too much. 127 is my limit in this specific case.

protect_floor is down to zero - always a good choice to not enhance any remaining noise more than necessary.
protect_bias at 12 was figured by try'n'error. 16 gave not enough sharpening, at 10 some edge artefacting started to appear.

DeRinging was almost standard:floor-bias at 10-12 gave a good compromise at catching source artefacts even at weaker levels, while still masking everything strong enough.
I was teared between 8-10 and 12-12 for different scenes, so I just took the golden middle ;)
Now that I review it, I could have put it a little higher values for dering, perhaps -100 would have been better.
Moreover, now I have the impression I should feather-out the whole deringing mask a little bit more.

The ss2_x|y values were set a little lower. The source's details are not as dense as in a good DVD movie, so I could afford this.

detail_floor was lowered to 15 after inspection of several scenes with weak detail. However, this is more important for compression than for the visuals.

All warp options were used, for the formerly mentioned reasons. Same goes for border exclusion.

EQ mode was "2". The source comes along with an (un)fair amount of small, but quite annoying artefacts (best seen in motion, there's no so much to tell from the screenshots), which here makes EQ=1 to look not very nice. And EQ=3 would be far too strong for a source with such insufficient sharpness.

Lastly, I applied a weak color correction. Somehow the series seems to suffer from NTSC-television-color-flatness, or whatever damage they might have done. It just looks too "empty", occasionally.

Very lastly: No, the the removal of that wonderful logo was NOT done by iiP :D


Okay, that was the little walk-through, or better said, run-through.

Also, now you have an idea what crap I am producing all day long ;)


- Didée

[Inc]

Very lastly: No, the the removal of that wonderful logo was NOT done by iiP

What procedure did u use for removing the logo in such a nice quality? Yo want to keep it secret? ;)

Greets
Inc.

[Didée]

Secret? What secret? :confused:

1. The name is "Didée".

2. The filter is "DeLogo".

3. The search button is at the top of the page. :)

4. The remainder is hard work.


- Didée

[Soulhunter]

Originally posted by Didée
One correction of my own false comment, before you start out:

To disable Pixie, use dustweight = 0.0 ! When setting "duststr=0.0", it is also (effectively) disabled ... but PixieDust would first do its full work, changing nothing, and then get copied over ... a pretty slow No-Op. By zeroing dustweight, Pixie will actually be bypassed - that's what you want to do.
Ok, I set them both to zero... :)


THX n' Bye

[Soulhunter]

Ok, here is my "replace dust with deen" example...

iip( dest_x= 1024, dest_y= 576,
\ ss1_x = 1.414, ss1_y = 1.414,
\ duststr = 2, dustweight = 1, antiflicker1= true, antiflicker2= true,
\ detailcontr1=127, detailcontr2 = 255, contr_radius = 3, PixSharp=0.25,
\ ss2_x = 3.5, ss2_y = 3.5,
\ Xstren = 255, Xlimit = 255,
\ subpelstren= 1.0, flatweight = 64,
\ protect_floor= 4, protect_bias = 16,
\ dering = -60, dering_weight= 1.0, dering_floor = 16, dering_bias=8,
\ detail_floor= 1, EQ = 2,
\ warp_Y = false, warp_UV = false,
\ debug= "showall | compareH/V | protect | dering",
\ cropx=32, cropy=16
\ )

http://img21.exs.cx/img21/1483/IIP_2077.jpg


edeen(3,3,3,1,1,true)

iip( dest_x= 1024, dest_y= 576,
\ ss1_x = 1.414, ss1_y = 1.414,
\ duststr = 0, dustweight = 0, antiflicker1= true, antiflicker2= true,
\ detailcontr1=127, detailcontr2 = 255, contr_radius = 3, PixSharp=0.25,
\ ss2_x = 3.5, ss2_y = 3.5,
\ Xstren = 255, Xlimit = 255,
\ subpelstren= 1.0, flatweight = 64,
\ protect_floor= 4, protect_bias = 16,
\ dering = -60, dering_weight= 1.0, dering_floor = 16, dering_bias=8,
\ detail_floor= 1, EQ = 2,
\ warp_Y = false, warp_UV = false,
\ debug= "showall | compareH/V | protect | dering",
\ cropx=32, cropy=16
\ )

http://img37.exs.cx/img37/8587/eDeen_2077.jpg

[malkion]

Originally posted by Soulhunter
Ok, here is my "replace dust with deen" example...

It seems like pixiedust looks a tad sharper on still samples.

How does it look when in motion?

[DarkNite]

@Didée

When I first looked at that source (thinking DVB content would actually be much sharper) I thought it might've been one of those hazed over love scenes.

Those results looks amazingly better than that soft source.

@Soulhunter

Any particular reason for using eDeen the spatial monster, instead of the kinder, faster, more tweakable, and much less prone to access violations, Deen?

Looking at the upper left corner wall section, and the lateral obliques of Mr. "Russel Crowe beats my a%$!" there's some strange oil painting effect going on. Looks like more experimenting may be called for.

[Soulhunter]

Originally posted by DarkNite
Any particular reason for using eDeen the spatial monster, instead of the kinder, faster, more tweakable, and much less prone to access violations, Deen?
Not really, was only a quick "try n' see" attempt... :o

Originally posted by DarkNite
Looking at the upper left corner wall section, and the lateral obliques of Mr. "Russel Crowe beats my a%$!" there's some strange oil painting effect going on. Looks like more experimenting may be called for. Yeah, I will do further experiments !!!


Bye

[Soulhunter]

Now a "replace Dust with MipSmooth" example...

IIP as before:

http://img21.exs.cx/img21/1483/IIP_2077.jpg


MipSmooth MovieHQ:

http://img33.exs.cx/img33/4228/MipSmoothed.jpg


Bye

[tedkunich]

@Didée

I've been trying out your iip processor for my LD caps and have run into a bit of a snag that I'm hoping you might be able to suggest some guidance with:

with the following parameters:
iip( dest_x= 720, dest_y= 360,
\ ss1_x = 1.414, ss1_y = 1.414,
\ duststr = 2, dustweight = 1, antiflicker1= false, antiflicker2= false,
\ detailcontr1=127, detailcontr2 = 255, contr_radius = 2, PixSharp=0.4,
\ ss2_x = 3.5, ss2_y = 3.5,
\ Xstren = 255, Xlimit = 255,
\ subpelstren= 1.0, flatweight = 0,
\ protect_floor= 4, protect_bias = 16,
\ dering = -60, dering_weight= 1.0, dering_floor = 16, dering_bias=8,
\ detail_floor= 1, EQ = 2,
\ warp_Y = false, warp_UV = false,
\ debug= "",
\ cropx=32, cropy=16
\ )


I get some really ugly artifacts/grain/noise in the returned clip. I have experimented with the detailcontr1 settings without much success in returning a clean clip. Setting detailcontr2 along with all of the other requisite settings to zero, reduces the introduced noise somewhat. Setting the Dust strength higher (8) does nothing but lose detail - noise/artifacts in the returned clip are unaffected. Setting ss1_x and ss1_y to something like 5 will eliminate the artifacts, but then your script does not really sharpen up any edges. Any thoughts as to what to try next?

T


PS I'll try to get a sample of what I am talking about and get it hosted somewhere.....

[Didée]

tedkunich:

Yes, please make a short sample available. Without, everything is guesswork.

However, you definetly should raise detail_floor from 1 to something >>8. Realize that default is 20...
Check with "showall" that the "detail" debug frame appears with not too much noise in it (a _moderate_ noise level in this frame is okay). Raise detail_floor until it starts to appear cleaner.

Also, those two antiflicker switches could help to improve.

Lastly, are you sure about your dering settings? Default is -80/8/16. Your settings of -60/16/8 will push "deringing" on big amounts of frame area - and therefore also in areas of weak detail (real or artificial), that might tend to be spatially instable from the very start. That could lead to additional "artefacts", namely those transitions between "deringed" and not "deringed" parts, in areas where it was not intended to be used. (The actual [new] deringer is mostly designed to clean around hard edges.)

Perhaps your input is simply not clean enough and would require additional pre-processing. As mentioned time ago, iiP is not so much a "repairer". Its focus lies on enhancing. If source's artefact aren't fully catched by Dust & deringing, then something else must be done beforehand.

A sample would help.

[tedkunich]

@Didée

Here are two small samples... In clip 1, my settings distort the face, giving it almost a cartoonish look. Clip2, there is lots of artifacts in the sky background....

http://www.geocities.com/tedk_1/clips.htm

My POS ISP has no hosting, so I had to go thorugh a free Geocities account... right now, I get a message saying to check back later.... dunno what that is all about.... give it an hour or so and the clips should be available.


I did ramp up the detail_foor setting to 60 and was able to get much better results, although I get a lot of noise around the sharp transitions in detail...

Would it be to difficult to put together a usage guide of your script for us noobs? kind of a first things first approach with maybe a bit of a narrative as to settign up the parameters. You have descriptions of the parameters scattered about in this thread as the script has evolved, but I for one, still have not been able to determine what are the effects of each of the paramters. As an example, I was able to figure out the "DERING" debug view, but is there an prefered order in adjusting the parameter so that they are not skewed way off? I had no clue as to what the "DETAIL" view was telling me, prior to your reply - from what I gather now, the white mask areas are where there are details that will not be processed????


Thanks for all your efforts!!!


T

[Didée]

I love GeoCities :devil:

Now it tells me the site has exceeded its limit. Some minutes before, the links appeared, but could not be accessed ("... is temporarily unavailable").

>> Would it be to difficult to put together a usage guide

Yes. I'm currently a little out of shape, and have neither the energy nor the mood for such an effort.

Everything lies somewhere in the thread ... be adventureous, and read carefully ;)

[JanBing]

Wow, I am a bittle overwhelmed by this thread... any chances for some collected info? Simply as: Which plugins do I need, and what's the script *g*

There are so many version (and so many info in general) floating around in this thread, that I - as a newbie to AviSynth - am definintely knocked out.

So... to cut a long story short... (and this is where you finish this sentence :D )

[Didée]

tedkunich:

Finally geoCities delivered.

Well ... difficult source to get HD quality from (joke). The Han Solo sequence can't get good-looking at all: too much noise, too much detail exceeded by noise floor. And faces start looking ugly much earlier than anything else...

Perhaps more motion compensated pre-processing via MVTools + (e.g.) DeGrainMedian could prepare the source better.

For iiP alone, perhaps try the following:

- manually replace PixieDust with GoldDust
- settings suggestion:crop(0,0,-0,-4)
converttoyv12()
iip( dest_x= 720, dest_y= 288,
\ ss1_x = 1.414, ss1_y = 1.414,
\ duststr = 3, dustweight = 1.0, antiflicker1= true, antiflicker2= true,
\ detailcontr1= 127, detailcontr2 = 255, contr_radius = 3, PixSharp=0.3,
\ ss2_x = 3.5, ss2_y = 4.0,
\ Xstren = 255, Xlimit = 255,
\ subpelstren= 1.0, flatweight = 0,
\ protect_floor = 0, protect_bias = 8,
\ dering = -60, dering_weight= 1.0, dering_floor = 8, dering_bias=12,
\ detail_floor = 25, EQ = 2,
\ warp_Y = true, warp_UV = false, exborder = false,
\ debug= "-compareV",
\ cropx=40, cropy=16
\ )

I didn't play too much with it, but (together with GoldDust) the 2nd scene looks rather OK, and Harry looks not too bad when in motion. Not sure if warpY should be used or not, though.

BTW, Harry looked pretty awful when I fed the source without any filtering to XviD ;)

[tedkunich]

@Didée

Thanks, FYI I spent a bit of time trying to find some of the worst scenes in the capture for those clips... :D the rest is not that bad... especially considering that it is a LD capture with a $50 card...


I'm processing the clip now with some other settings I came up with.... It has about 4 more hours to go, so I'll let it finish, then I'll try your recomendations.

Any tips on the MVTools? I have never used it before. How would that work with DegrainMedian?


Thanks


Ted

[audioslave]

@Didée
Great work! :D
This just gets better and better. Any chance of getting a zip file with all filters + the script? Or, maybe there already is one that I missed... :(
Thanks again! ;)

[Jo_Gors]

May be I'm posting a boring question, but I tried iip several times with no success.
Whatever warsharp version I use, seems doesn't works.

With the version dideè linked in his post, I allways get a "hieroglyplical like" error message (japanese?) and with the other one I've found in the web, I get "..there's no function named unsharpmask".


The loadpluginex is present both in the script and the plugins' folder.
I'm running the latest 2.55 avisynth version (august,31)

Tried on 2 different PC with the same results.

Any suggestions?



greetings

Jo

[Didée]

msvcr70.dll/msvcr71.dll
msvcp70.dll/msvcp71.dll
--> %windir%/system32

Plus for safety, put WarpSharp.dll *NOT* in the pluginfolder, but load it manually. Autoloading may cause problems, too.

[Jo_Gors]

Thanks a lot Didèe, now it works.

One more thing, what's about that "MyDelogo()" in your script?
(I know,... your name is.... and the search button is.... :D )


greetings

Jo

[AllTimeSToneD]

Any chance we could get IIP to accept 1920x1080P sources? It works nice with 1280x720 but once feed with 1920x1080P i get "unrecognized expection" error. And "No" lol im not gone upsample it but i think iip does a damn nice work also on downsampling :D

I could probably set a resizer to1280x720 before iip but i would lose alot of image information this way that iip could use :D :D

[Didée]

/*holds hands against head, eyes go :eek:, thinking:*/

That guy wants to FEED 1920*1080 into iiP! He must be AllTimeSToneD ...

:)

No currently you can't. Dust is the limiting factor, since it supports 1440*1440 at max. And I didn't do any safety checkings wether the input is too big ...
So, as an compromise, you could resize your awfully shitty looking HDTV source to 1440^2 first, and then apply iiP to deliver 1280*720.

However, using Dust on such resolutions is no fun: Around christmas time, please remember to inform us how the encoding job is going and all.

Try and see if "duststrength=0, dustweight=0.0" will pass through, or if the process chain insists on actually building Dust's fork. If so, manually delete it from the script, or replace it with an easier & faster filter, like ... MipSmooth @ 2.0 (where is the "evil grin" smiley??)

Moreover, using supersamplings > 1.0 in iiP perhaps would be not so clever in this case.

Think I'll do some more changes at weekend - iiP is going to marry so, anyways.

[AllTimeSToneD]

hat guy wants to FEED 1920*1080 into iiP! He must be AllTimeSToneD ...
Yes I'm :D
Imagine actually there is a channel for insane ppl like myself and mf :D :D :D
No currently you can't. Dust is the limiting factor, since it supports 1440*1440 at max.
Yeah i realized that too.
Try and see if "duststrength=0, dustweight=0.0" will pass through...manually delete it from the script...

Tryed first and second option but it didn't work :rolleyes: I'm currently messing around with the replacing and somehow i can't get it to work :p Might be cuz its 2:30am :D
Think I'll do some more changes at weekend - iiP is going to marry so, anyways.
A nice! Tell me if you need a true 1920x1080 source(native) as sample to play around.

... i wonder how and upsampled 4000x2250 (IMAX) from a 1920x1080 movie would look like :D :D :D
Around christmas time, please remember to inform us how the encoding job is going and all.
Ah dont worry about that :D i have three comps standing around and idling. I could probably use them or even TCP Deliver IPP :D

[Didée]

Yes, setting duststr & dusweight to zero can't escape the framesize problem - the way the code is writtern, the dust chain internally will always be established, even if it is not used later on.

But modding is rather easy. Find the following part at the beginning:
|
#---------------------------------|
# Base denoising by PixieDust |
# |
dusted = clp.addborders(0,0,xcorr,|...
dusted = (dustweight==0.0) ? clp |
\ : ( dus|...
\ |
|
soft=dusted.bicubicresize(xx_small|...
|
#---------------------------------|
# Build EdgeMask to protect alread|...
|

And adjust it like that:
|
#---------------------------------|
# Base denoising by PixieDust |
# |
dusted = duststr>0 ? clp.addborder|...
\ : clp |
dusted = (dustweight==0.0) ? clp |
\ : ( dus|...
\ |
|
soft=dusted.bicubicresize(xx_small|...
|
#---------------------------------|
# Build EdgeMask to protect alread|...
|
Now the Dust chain should not be built up when you specify "duststr=0, dustweight=0.0). Use another denoiser of your choice then, before calling iiP (IF denoising is needed at all).

Or alternatively, you could hardcode another denoiser yourself, like
#---------------------------------
# Base denoising by PixieDust

dusted = clp

# WannaDenoise?
dusted = dusted.YourVeryDenoiser()

#---------------------------------
# Build EdgeMask to protect alread...

Avisynth is easy, isn't it?

[AllTimeSToneD]

>Avisynth is easy, isn't it?

Ic now its crystal clear :D though its 4:45pm ;)

Thx

[oo_void]

In related resizing issues ... I tried a couple of passes of sending the results of tritical's FastEDIUpsizer (http://forum.doom9.org/showthread.php?s=&postid=459732#post459732) into your new version of IIP, IIP_v6.0a I think it is (yeah, the new super-alpha, untested one). Insteading of an extremely nice, resized image though, I seem to be getting some strange results.

With the EDI upsize, the dimensions of the video going into IIP are 1408 x 544 (2x source size) and I'm trying to get 720 x 360 out of IIP (dest_x = 720, dest_y = 360). Instead of getting the nice, prestine image I've come to expect out of IIP ;), the image appears to have become currupted in the lower right hand portion ... A wierd interlace like corruption flipped on the x axis. I'd written the issue off as having something to do with Dust, but in hearing that Dust should be able to accept files up to 1440 x 1440, it seems a little strange.

I tried playing around with MemoryMax a bit to, but no avail. Things work fine if I run a Lanczos4 before the IIP call and knock it down to 720 x 544, but that's the extent of the testing I've been able to do to date.

If you're interested, I post a sample image sometime tonight (the power's out at home so I can RDP in to the video machine until everything is reset). Anyways ...

------
oo_void

[Socio]

This whole thread is very interesting, I am a real Avisynth supernoob in fact the only thing I have used Avisynth is re-sizing in ffdshow for DVD playback.

After seeing some of the results I am now wondering if it possible to use Didee's script at least in part for on the fly DVD viewing via ffdshow and Zoomplayer? If so, and before I go finding all the plugins I will need what would need to be omitted from this script, I assume the mpeg2source line would be because Avisynth can automatically use ffdshow as source.
SetMemoryMax(256)

Import("C:\Program files\AviSynth 2.5\plugins\IIP.avs")

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\LoadPluginEx.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\undot.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MaskTools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\WarpSharp.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\dustv5.dll")

#mpeg2source

iip( dest_x= 1280, dest_y= 720,
\ ss1_x = 1.414, ss1_y = 1.414,
\ duststr = 2, dustweight = 1, antiflicker1= true, antiflicker2= true,
\ detailcontr1=127, detailcontr2 = 255, contr_radius = 3, PixSharp=0.25,
\ ss2_x = 3.5, ss2_y = 3.5,
\ Xstren = 255, Xlimit = 255,
\ subpelstren= 1.0, flatweight = 64,
\ protect_floor= 4, protect_bias = 16,
\ dering = -60, dering_weight= 1.0, dering_floor = 16, dering_bias=8,
\ detail_floor= 1, EQ = 2,
\ warp_Y = false, warp_UV = false,
\ debug= "showall | compareH/V | protect | dering",
\ cropx=32, cropy=16
\ )




Thanks for any info

BTW: Just in case it might work I found all but one plugin, I can;t find the LoadPluginEx.dll (did find LoadPluginEx2.dll will this work?). Could someone point me in the right ditection?


Thanks

[oo_void]

If you don't mind watching stuff at 2fps ... sure you can use it for on the fly conversion ;).

Seriously, IIP is definately not for real-time processing ... Even Dust alone will knock you down to 50% real time. Once you add all the masks and sharpening, your down to about a day and a half of processing (Athlon 2800 speeds) for about 2 hours of video footage. Anyways ...

------
oo_void

[Socio]

Well I tried it and the movie starts for a sec and then Zoomplpayer crashes. To bad because if it were to work in real time it would have been awesome looking at pics of others results.