dehalo and sharpening help with soft noisy ringing DV source

[2Bdecided]

This source is aliased, soft, ringing, noisy, dark, with poor colours and trashed highlights. Pretty much like most of my indoor DV footage from a decade ago!

Now I'm trying to edit it with HD footage, and though it'll never look anything like HD, I've made good progress with some of the problems, but I can't fix the softness and ringing.

I've tried many combinations of dehaloalpha, limited sharpen faster, plain old sharpen etc at various points in the script - I either get mush (too much dehalo-ing!) or something that looks like a chiselled artoon (i.e. colour graduations which weren't edges are sharpened to look like edges).

I'm not expecting miracles, but something just slightly less bad would be nice!

Here's my current script, showing the bits that I think work OK. There's no dehaloing because everything I've tried makes it look worse.

AVISource("DV.2000-06-14 23-16.00 018 tragedy 16x9.avi",pixel_type="yv12")

# fix aliasing:
mt_convolution(horizontal= " -0.0616 0.2950 0.5829 0.2950 -0.0616 ", vertical= "1", u=3, v=3)

# deinterlace:
TempGaussMC_alpha3()

# fix levels:
levels(10,1.0,255,0,235,coring=false)
hdragc()

# "fix" colour:
tweak(sat=1.2,coring=false)
coloryuv(off_v=-15,off_u=-0)

# denoise:
smdegrain(tr=3)

# PAL 16x9 doesn't have nearly enough horizontal pixels for square pixel output, so use EEDI2 to help preventing stepping on diagonals during later resizing
turnright().eedi2(field=0).turnleft()

# "fix" chroma bleeding:
mergechroma(aWarpSharp(depth=20.0, thresh=0.75, blurlevel=3, cm=1))

# remove junk from edges of frame:
crop(0,8,-16,-2)

# getting desperate here - this doesn't really work:
sharpen(0.5)
grainfactory3(g1str=8, g2str=8, g3str=8)
addgrainc(0.2)

# resize to 16x9 - even the smallest of these still looks poor!
#spline36resize(1920,1080)
spline36resize(1280,720)
#spline36resize(1024,576)
#spline36resize(960,540)
#spline36resize(640,360)

Sample: http://www.mediafire.com/?kdyci4kowoz

screen shots (also attached):
original:
http://www.mediafire.com/file/wqyxuyzdytu/DV_tragedy_original.jpg
processed:
http://www.mediafire.com/file/4n0nmzlq2wl/DV_tragedy_proc.jpg

Any help with dehaloing or sharpening (or anything else!) would be greatly appreciated.

Cheers,
David.

[WorBry]

David,

Sorry I cant help with your issue, but, seeing your script, I'm interested in the mt_convolution approach to anti-aliasing that you are using.

Just wondering, how did you/does one derive the 'horizontal' parameter values?

[Dogway]

The aliasing thing intrigued me. How does it work? Source actually looked like pixelated more than aliased although aliasing is one issue related to pixelization/low sampling. Does it perform better than any other anti-aliasers solutions?
Also you may want to darken from gamma, as you are doing now you are crushing the darks, no recovery possible for hdargc.
You may also want to addgrain at last stage, to keep the grain size

[WorBry]

Found this earlier thread on the mt_convolution anti-aliasing:

http://forum.doom9.org/showthread.php?p=1300816#post1300816

and through it, this:

http://forum.doom9.org/showthread.php?p=1237938#post1237938

Seems like mp4 guy is the expert on the subject (FIR filtering) . Even with the program that he refers to for calculating FIR co-efficients, I wouldn't have a clue where to start.

Sorry to digress David; I know you are looking for an answer to your specific problem.

[2Bdecided]

Thinking about it, what I have in that source is closer to pixelation or microblocking(!) than aliasing. I got the idea for using mt_convolution in that thread from mp4guy. I can't remember where the filter above came from - it might be one of mp4guy's, but truncated, or it might be one I made myself. I made my own filter by drawing the frequency response I wanted in Cool Edit Pro's FFT filter (I'm an audio guy!), filtering an impulse, truncating it, normalising the coefficients (making them add up to one), and putting them into mt_convolution.

Anyway, that parts works really well on my source (all the footage shot with my old camcorder is like that! 16x9 mode is slightly worse than 4x3 mode) - and I'm very happy with the tgmc deinterlacing too.

But after that, it goes down hill - dubious colour correcting, and no idea about de haloing and sharpening - so any help would be appreciated!

Cheers,
David.

P.S. The crushed blacks are intentional - YMMV but I don't want HDRAGC getting its hands on the very bottom of the luma range - it looks like junk to me, as if there's nothing real there (try it without crushing the blacks - I think it looks horrible). As for grain: I don't think it's helping as it is, but putting it at the end, so I get perfect sharp 1280x720 grain on a 720x576 source (where the source seems more like 640x360 at best, in terms of actual sharpness/resolution!) just looked silly. IMO!

[Didée]

FIR filtering is a somewhat poor approach in this case. Look closely at the source: it has "half horizontal resolution", similar as in "pointresize(x2)".

What would you do to upsize an image by factor=2 ? Would you do "PointResize(width*2,height*2).blur(1)" ? Or would you do an ordinary "{Standard}Resize(width*2,height*2)? Or, would you use some E.D.I. resampling instead?

I'd guess you wouldn't use the PointResize Method. If so, you probably don't want to use mt_convolution here, either. ;)

[*.mp4 guy]

FIR filtering and avisynths internal interpolation are from a certain perspective the same thing. Fir filtering just lets you have much more control. I tend to think FIR filtering is safer when you don't really know exactly what is in the source, because it doesn't throw away information like using a standard interpolator would.

[2Bdecided]

FIR filtering is a somewhat poor approach in this case. Look closely at the source: it has "half horizontal resolution", similar as in "pointresize(x2)".It might look like that, but I've got other scenes filmed in brighter conditions (so they're a little sharper) where it's clear the pixellation isn't simple pixel repeating. I tried nnedi2(), and it lost resolution - i.e. a fairly fine repeating pattern was wiped out to flat. Whereas the FIR filtering above maintained the pattern, but removed all the pixellation.

Anyway, now both you bright guys have joined the thread, how about the softness + haloing problem? ;)

Cheers,
David.

[Didée]

What i had in mind was the average(nnedi-top,nnedi-bottom)+contrasharp method. See here (http://forum.doom9.org/showthread.php?p=1360219#post1360219). Of course, use that in a turnleft/../turnright sandwich. Vdub in front of me shows a good result, you should try it, too. ;)

[WorBry]

Anyway, now both you bright guys have joined the thread, how about the softness + haloing problem? ;)
Cheers,
David.

So who is the other one ? :D

..... but seriously though, Didee beat me to it; I was about to suggest that you try the merged ('doubled') EDI approach - works well with various forms of aliasing, and no haloing/ringing (as a result).

[Didée]

Also ... with the given source material, the following might put a smile on your face:

boring_sharp_and_blur = tehclip.sharpen(0.4,0).blur(1,0)
impressing_hi_tec_filter = tehclip.mt_convolution(horizontal= " -0.0616 0.2950 0.5829 0.2950 -0.0616 ", vertical= "1", u=3, v=3)

interleave( tehclip, impressing_hi_tec_filter, boring_sharp_and_blur )

[2Bdecided]

What i had in mind was the average(nnedi-top,nnedi-bottom)+contrasharp method. See here (http://forum.doom9.org/showthread.php?p=1360219#post1360219). Of course, use that in a turnleft/../turnright sandwich. Vdub in front of me shows a good result, you should try it, too. ;)Thanks, that's a big improvement. Silly thing is, I saw that original thread in 2006, and was there when it was used more recently with WorBry's footage too!

I'm clearly getting old and forgetful.

Now, since you've been so kind, dare I mention those halos again? :)

Cheers,
David.

P.S. Both video samples (tragedy = DV.2000-06-14 23-16.00 018 tragedy 16x9 part.avi; seaside = DV.2002-07-19 13-31.40 027 sample.avi) are here...

http://www.mediafire.com/?sharekey=605ee2b5f85656f5e62ea590dc5e5dbbd97c0f0ae180fcd7b8eada0a1ae8665a

...with pictures showing the improved result of Didée's suggestion vs the alternatives on the "seaside" one (more dramatic that the one linked at the top of this thread), if anyone else wants to be impressed.

[2Bdecided]

btw, with the double nnedi2 approach, I have another question...

I need to deinterlace (tgmc), and I need to anti-alias (nnedi2 merged).

If I anti-alias first, the combing makes the anti-aliasing sub-optimal (it can't find the edges for the combing).

If I deinterlace first, the alias makes the deinterlacing sub-optimal (the motion compensation gets "stuck" on the pixelation edges sometimes).

Or I can anti-alias on separated fields, and then deinterlace properly.


I know I can try it (I have tried it!) - strangely, the biggest difference seems to be in the chroma (yv12 interlaced chroma taking a hit?) - but as I'll probably end up doing this to all my footage, and I can hardly try it on all my footage to check for obscure problems, it would be cool if anyone knows if there's a good reason why one order or the other of deinterlace/anti-alias should be theoretically better, or "correct".

Cheers,
David.

[Didée]

If I anti-alias first, ....

If I deinterlace first, ....
Good point. Neither way is "perfect" ... but then, since both steps are working in perpendicular directions, there's not too much difference to expect. Without having tried it, I'd guess that it should be rather safe to first do the horizontal-AA on the separated fields, then deinterlace. (It's very uncertain if the way round is better - but this way round is faster.)


Breaking News from the dehaloing front: the war is lost.

Seriously ... I toyed a bit with the "seaside" screen, and found it's a good example to show up shortcomings of halo-remover-scripts. It's not that hard to remove the halos, but it's much easier to create a dull, poor looking mess.

I came up with something that might be acceptable, or might be not ... a matter of taste, as usual. The result is okay, but I am not impressed either.

Source:
http://img714.imageshack.us/img714/6281/2bdsource.th.png (http://img714.imageshack.us/i/2bdsource.png/)

3-fold dehalo_alpha /w different radii, constrained by edgemask, with some re-sharpening:
http://img339.imageshack.us/img339/5213/2bddehalosharpnessfree.th.png (http://img339.imageshack.us/i/2bddehalosharpnessfree.png/)

As before, but sharpening limited acc. to previous removal:
http://img441.imageshack.us/img441/9039/2bddehalosharpnesslimit.th.png (http://img441.imageshack.us/i/2bddehalosharpnesslimit.png/)

Is that something, or is it nothing ... can't tell. It's not bad overall, but the question is if it's really worth the effort.

[2Bdecided]

Hi Didée,

That certainly maintains more sharpness that anything I've managed. It edges towards "cartoony" - but without halos this is a very low resolution source, so what can you do?! I can maybe combat this a bit by selectively adding a little of the original back in. I know, swings and roundabouts, but I'll see.

So yes, please share. If you can explain your thinking behind the 3-fold approach, that would probably be useful to myself and others too.

(I know it might not be worth the effort, but for blowing things up to edit with HD, I've got to do something!)

Cheers,
David.

[Didée]

Oh! Script! - I knew I had forgotten something ... :p

imagereader("seaside_12140_nnedi_merged.tif").converttoyv12()
o=last

dh3 = o.dehalo_alpha_2BD(rx=2.0,ry=1.7,darkstr=1.00,brightstr=1.00*0.8)
\ .dehalo_alpha_2BD(rx=2.5,ry=1.9,darkstr=0.85,brightstr=0.85*0.6)
\ .dehalo_alpha_2BD(rx=3.0,ry=2.1,darkstr=0.74,brightstr=0.74*0.4)
e3 = mt_lutxy(dh3.mt_expand().mt_expand(),dh3.mt_inpand().mt_inpand(),"x y - 10 - 4 *").mt_expand()
mrg = o.mt_merge(dh3,e3,U=2,V=2)
x1 = mrg.removegrain(11)
fin1 = mrg.mt_adddiff(mt_makediff(x1,x1.rg114()).mt_lut("x 128 - 1.501 * 128 +"),U=2,V=2)
x2 = x1.sbr()
fin2 = fin1.mt_adddiff(mt_makediff(x2,x2.rg114()).mt_lut("x 128 - 1.499 * 128 +"),U=2,V=2)

mrgD = mt_makediff(mrg,o)
fin2D= mt_makediff(mrg,fin2)
DD = fin2D.repair(mrgD,1)
fin3 = mrg.mt_makediff(DD,U=2,V=2)

interleave(o,dh3,fin2,fin3) # dh3 = 3-fold dehalo_alpha / fin2 = dh3 with 2-step mediansharpen (free) / fin3 = fin2 with sharpness limiting
# "visually best" is fin2, the others are just for comparison

return( last )

#================================================

# "use highpass of a blur's difference" - function
function sbr(clip c) {
rg11D=mt_makediff(c,c.removegrain(11,-1))
rg11DD=mt_makediff(rg11D,rg11D.removegrain(11,-1)).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
c.mt_makediff(rg11DD,U=2,V=2) }


# "rg411" -- use removegrain(11), but maximally as much as removegrain(4) did in the 3x3 neighborhood

function rg114(clip c) {
d=mt_makediff(c,c.removegrain(11)).repair(mt_makediff(c,c.removegrain(4)),1)
c.mt_makediff(d,U=2,V=2) }


# modified dehalo_alpha: here with increased search-radius for validation (marked red)

function DeHalo_alpha_2BD(clip clp, float "rx", float "ry", float "darkstr", float "brightstr", float "lowsens", float "highsens", float "ss")
{
rx = default( rx, 2.0 )
ry = default( ry, 2.0 )
darkstr = default( darkstr, 1.0 )
brightstr = default( brightstr, 1.0 )
lowsens = default( lowsens, 50 )
highsens = default( highsens, 50 )
ss = default( ss, 1.5 )

LOS = string(lowsens)
HIS = string(highsens/100.0)
DRK = string(darkstr)
BRT = string(brightstr)
ox = clp.width()
oy = clp.height()
uv = 1
uv2 = (uv==3) ? 3 : 2

halos = clp.bicubicresize(m4(ox/rx),m4(oy/ry)).bicubicresize(ox,oy,1,0)
are = mt_lutxy(clp .mt_expand(U=uv,V=uv).mt_expand(U=uv,V=uv).mt_expand(U=uv,V=uv),clp .mt_inpand(U=uv,V=uv).mt_inpand(U=uv,V=uv).mt_inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
ugly = mt_lutxy(halos.mt_expand(U=uv,V=uv).mt_expand(U=uv,V=uv).mt_expand(U=uv,V=uv),halos.mt_inpand(U=uv,V=uv).mt_inpand(U=uv,V=uv).mt_inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
so = mt_lutxy( ugly, are, "y x - y 0.001 + / 255 * "+LOS+" - y 256 + 512 / "+HIS+" + *" )
lets = mt_merge(halos,clp,so,U=uv,V=uv)
remove = (ss==1.0) ? clp.repair(lets,1,0)
\ : clp.lanczosresize(m4(ox*ss),m4(oy*ss))
\ .mt_logic(lets.mt_expand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"min",U=uv2,V=uv2)
\ .mt_logic(lets.mt_inpand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"max",U=uv2,V=uv2)
\ .lanczosresize(ox,oy)
them = mt_lutxy(clp,remove,"x y < x x y - "+DRK+" * - x x y - "+BRT+" * - ?",U=2,V=2)

return( them )
}

[cretindesalpes]

fin1 = mrg.mt_adddiff(mt_makediff(x1,x1.rg114()).mt_lut("x 128 - 1.501 * 128 +"),U=2,V=2)
x2 = x1.sbr()
fin2 = fin1.mt_adddiff(mt_makediff(x2,x2.rg114()).mt_lut("x 128 - 1.499 * 128 +"),U=2,V=2)

Something related to RemoveGrain, I guess ?

[Didée]

Oops, I forgot to include that little function. Sorry.

function rg114(clip c) {
d=mt_makediff(c,c.removegrain(11)).repair(mt_makediff(c,c.removegrain(4)),1)
c.mt_makediff(d,U=2,V=2) }

It's now also included in the script above.

[2Bdecided]

Didée, my sincere thanks. That has worked wonders. It's amazing how "flat" it goes after the basic dehalo before you resharpen it - I wouldn't have had a clue how to do that.


To reduce the slight "cartoon" look, I've added a little of the original back in using...
fin3.overlay(o,mode="Lighten",opacity=1.0)
...to add a few highlights back (e.g. it makes streaks of blonde hair look quite strange and flat without this). I suppose I could have played with the parameters of dehalo to do the same, but just having that single opacity value to change above seemed simpler.

Vertical dehaloing seems somehow too strong (removes eyes sometimes) yet too weak (leaves halos sometimes), but I can't do any better. I might back it off to leave eyes in place!


I'm now quite happy, except I still have huge amounts of chroma noise left - even after smdegrain(tr=3) and the awarpsharp chroma part.

What do people use for killing really horrible chroma noise? I even tried Neat Video with a suitable noise print, but at the strength needed to have a useful effect, it was doing very strange things to small areas of legitimate colour.

Cheers,
David.

[2Bdecided]

I finally finished this. I wanted to thank everyone, especially Didée, for all their help.

FWIW my final script was this...

AVISource("DV.2000-06-14 23-16.00 018 tragedy 16x9.avi",pixel_type="yv12")

separatefields()

turnright()

o = last
o.NNEDI2(field=-2)
dbl = Merge(SelectEven(),SelectOdd())
dblD = mt_MakeDiff(o,dbl,U=3,V=3)
shrpD = mt_MakeDiff(dbl,dbl.RemoveGrain(11),U=3,V=3)
DD = shrpD.Repair(dblD,13)

dbl.mt_AddDiff(DD,U=3,V=3)

turnleft()

assumefieldbased()

weave()

TempGaussMC_alpha3()

levels(10,1.0,255,0,235,coring=false)
hdragc()

tweak(sat=1.2,coring=false)
coloryuv(off_v=-12,off_u=-0)

smdegrain(tr=3)

o=last

dh3 = o.dehalo_alpha_2BD(rx=2.0,ry=1.7,darkstr=1.00,brightstr=1.00*0.8)
\ .dehalo_alpha_2BD(rx=2.5,ry=1.9,darkstr=0.85,brightstr=0.85*0.6)
\ .dehalo_alpha_2BD(rx=3.0,ry=2.1,darkstr=0.74,brightstr=0.74*0.4)
e3 = mt_lutxy(dh3.mt_expand().mt_expand(),dh3.mt_inpand().mt_inpand(),"x y - 10 - 4 *").mt_expand()
mrg = o.mt_merge(dh3,e3,U=2,V=2)
x1 = mrg.removegrain(11)
fin1 = mrg.mt_adddiff(mt_makediff(x1,x1.rg114()).mt_lut("x 128 - 1.501 * 128 +"),U=2,V=2)
x2 = x1.sbr()
fin2 = fin1.mt_adddiff(mt_makediff(x2,x2.rg114()).mt_lut("x 128 - 1.499 * 128 +"),U=2,V=2)

mrgD = mt_makediff(mrg,o)
fin2D= mt_makediff(mrg,fin2)
DD = fin2D.repair(mrgD,1)
fin3 = mrg.mt_makediff(DD,U=2,V=2)

fin3.overlay(o,mode="Lighten",opacity=1.0)

#addborders(16,16,16,16)
#mergechroma(aWarpSharp(depth=20.0, thresh=0.75, blurlevel=2, cm=1))
#crop(16,16,-16,-16)

# Make larger version to remove jaggies from subsequent crop+resize
turnright().nnedi2(field=0,dh=true).TurnLeft().nnedi2(field=0,dh=true)

# remove junk from edges of frame:
crop(0,14,-16,-4)

# resize to 16x9 - even the smallest of these still looks poor!
#spline36resize(1920,1080)
#spline36resize(1280,720)
spline36resize(1024,576)
#spline36resize(960,540)
#spline36resize(640,360)

#limitedsharpenfaster()

sharpen(0.35)

o=last
hdragc()
overlay(o,opacity=0.5)

tweak(sat=0.9,coring=false)


#================================================

# "use highpass of a blur's difference" - function
function sbr(clip c) {
rg11D=mt_makediff(c,c.removegrain(11,-1))
rg11DD=mt_makediff(rg11D,rg11D.removegrain(11,-1)).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
c.mt_makediff(rg11DD,U=2,V=2) }


# "rg411" -- use removegrain(11), but maximally as much as removegrain(4) did in the 3x3 neighborhood

function rg114(clip c) {
d=mt_makediff(c,c.removegrain(11)).repair(mt_makediff(c,c.removegrain(4)),1)
c.mt_makediff(d,U=2,V=2) }


# modified dehalo_alpha: here with increased search-radius for validation (marked red)

function DeHalo_alpha_2BD(clip clp, float "rx", float "ry", float "darkstr", float "brightstr", float "lowsens", float "highsens", float "ss")
{
rx = default( rx, 2.0 )
ry = default( ry, 2.0 )
darkstr = default( darkstr, 1.0 )
brightstr = default( brightstr, 1.0 )
lowsens = default( lowsens, 50 )
highsens = default( highsens, 50 )
ss = default( ss, 1.5 )

LOS = string(lowsens)
HIS = string(highsens/100.0)
DRK = string(darkstr)
BRT = string(brightstr)
ox = clp.width()
oy = clp.height()
uv = 1
uv2 = (uv==3) ? 3 : 2

halos = clp.bicubicresize(m4(ox/rx),m4(oy/ry)).bicubicresize(ox,oy,1,0)
are = mt_lutxy(clp .mt_expand(U=uv,V=uv).mt_expand(U=uv,V=uv).mt_expand(U=uv,V=uv),clp .mt_inpand(U=uv,V=uv).mt_inpand(U=uv,V=uv).mt_inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
ugly = mt_lutxy(halos.mt_expand(U=uv,V=uv).mt_expand(U=uv,V=uv).mt_expand(U=uv,V=uv),halos.mt_inpand(U=uv,V=uv).mt_inpand(U=uv,V=uv).mt_inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
so = mt_lutxy( ugly, are, "y x - y 0.001 + / 255 * "+LOS+" - y 256 + 512 / "+HIS+" + *" )
lets = mt_merge(halos,clp,so,U=uv,V=uv)
remove = (ss==1.0) ? clp.repair(lets,1,0)
\ : clp.lanczosresize(m4(ox*ss),m4(oy*ss))
\ .mt_logic(lets.mt_expand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"min",U=uv2,V=uv2)
\ .mt_logic(lets.mt_inpand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"max",U=uv2,V=uv2)
\ .lanczosresize(ox,oy)
them = mt_lutxy(clp,remove,"x y < x x y - "+DRK+" * - x x y - "+BRT+" * - ?",U=2,V=2)

return( them )
}


I dropped the awarpsharp chroma - it made a clear improvement on some edges, but flattened some areas of colour (especially skin tones) that weren't supposed to be flat.

It's debatable whether the dehaloing is worth it - you lose almost all halos, but you lose some eyes and flecks of hair too! I left it in because some of the halos were so bad.

I couldn't improve the denoising, so left it as it is.

I didn't run the above script in one go - I dropped a "return last" line in after TGMC, saved the result, then put an AVIsource line in to load the result into the right place in the script.

In any case, it was rather slow, but it got there in the end (4.5 minutes footage over a couple of days!).

I'm so glad I've got an HD camcorder now!

Cheers,
David.

EDIT: P.S. the result is here: http://www.youtube.com/watch?v=xvxw62Fa3xA - it could still be 10x better, but its good enough to embarrass the people involved ;)