Vine, cutting edge quality de-halo filter and morphology stuff

[brucethemoose]

Originally Posted by feisty2

it should work on cartoon stuff also I guess, but probably not worth it, Vine is pretty slow, and other lower quality but levels faster dehalo filters might be good enough for cartoons since getting cartoonized is one of the last things to worry about on literally cartoon vids

Some toons have more detailed, painted backgrounds that you have to worry about, but that's still true.

In fact, my problem is that Vine isn't strong enough... I've tried tweaking parameters, but it barely puts a dent in big sharpening halos like this:

http://i.cubeupload.com/YAZVwX.png

[Mystery Keeper]

Originally Posted by brucethemoose

Some toons have more detailed, painted backgrounds that you have to worry about, but that's still true.

In fact, my problem is that Vine isn't strong enough... I've tried tweaking parameters, but it barely puts a dent in big sharpening halos like this:

http://i.cubeupload.com/YAZVwX.png

You need to learn that miracles don't happen. And be skeptic about halo removing in general. AFAIR, there has been no method to remove them that wasn't destructive to other parts of video. What you've got is a horrible multi-peaks ringing. No hope to fight it. Better to not cause it.

[Nevilne]

"you can't expect this halo remover to remove these halos"

[brucethemoose]

Originally Posted by Mystery Keeper

No hope to fight it. Better to not cause it.

Unfortunately, that's the raw NTSC DVD rip. Yeah, it's that bad .

And yeah, I know... but there is a compromise somewhere between the Vine settings I've tried and DeHalo_Alpha, I just haven't found it yet.

[Mystery Keeper]

Originally Posted by Nevilne

"you can't expect this halo remover to remove these halos"

These multi-peaks? No, you can't. Narrow one peak halos - sure, you can. But expect it to screw up something else.

[feisty2]

Have you tried to raise "h" and "radius" up like a bit insane, especially "h", like "h=24"

[brucethemoose]

Originally Posted by feisty2

Have you tried to raise "h" and "radius" up like a bit insane, especially "h", like "h=24"

Not yet, thanks.


But holy moly... now I'm running this:

 Vine.Dehalo(clip, radius=[8, 4], a=32, h=24, sharp=1.0, sigma=0.6, alpha=0.36, beta=32, cutoff=4, show=False)

And its taking like 8 seconds per 720x480 frame to encode. I'll get back to you when I get results.

[brucethemoose]

Originally Posted by feisty2

Have you tried to raise "h" and "radius" up like a bit insane, especially "h", like "h=24"

24 is way too strong, this is 20 with radius 4,2 (with a straight spline36 scale for reference):

http://www.screenshotcomparison.com/comparison/185123

Look at the guy's eyes, in particular.

I'm messing with it now, and 12-16 is looking better... the higher order halos aren't completely gone, but I might be able to mask them with another filter now.


This is some great code though, thanks feisty2.

[feisty2]

Now I think a large "h" is not it, something like "h=8" might be enough already
You need to disable all mask stuff in Vine cuz the halos are roaming VERY far from the exact edges, pointless tryna "mask" them out

[feisty2]

I'll add such option(disable mask stuff) after college

[brucethemoose]

Originally Posted by feisty2

Now I think a large "h" is not it, something like "h=8" might be enough already
You need to disable all mask stuff in Vine cuz the halos are roaming VERY far from the exact edges, pointless tryna "mask" them out

Sounds good.

However, if I'm looking at it right, I think the mask is actually reaching WAY out to the halos as it should.

http://u.cubeupload.com/brucethemoos...phavpymask.png

I don't think disabling it would help. There are other scenes with much more intricate painted backgrounds, and it looks like the mask will help preserve those.

EDIT: I could be wrong though, maybe I'm misunderstanding what the mask does...

So for now, what parameters should I adjust for the mask?

[feisty2]

radius=[10, None], sigma=1.6, alpha=0.48 works on your first screenshot
I'm on cellphone right now so...

[brucethemoose]

Originally Posted by feisty2

radius=[10, None], sigma=1.6, alpha=0.48 works on your first screenshot
I'm on cellphone right now so...

Sorry for not responding earlier, I went down the deinterlacing/IVTC rabbit hole.

What h value did you use?

[feisty2]

r5:
new parameter "masking"

[feisty2]

@brucethemoose

clp = Vine.Dehalo(clp, h=6, masking=False, cutoff=1, sharp=0.4)
clp = Vine.Dehalo(clp, h=4, masking=False, cutoff=1, sharp=0.6)
clp = Vine.Dehalo(clp, h=2, masking=False, cutoff=1, sharp=0.8)

[feisty2]

and of course you can go for a custom mask for this specific vid and protect soft features a bit more

[brucethemoose]

Fantastic, thanks! I'm playing with this now, might upload some before/after screenshots/clips later to show Vine off, as those Avatar DVDs are a DeHalo torture test.

[brucethemoose]

Originally Posted by feisty2

@brucethemoose

clp = Vine.Dehalo(clp, h=6, masking=False, cutoff=1, sharp=0.4)
clp = Vine.Dehalo(clp, h=4, masking=False, cutoff=1, sharp=0.6)
clp = Vine.Dehalo(clp, h=2, masking=False, cutoff=1, sharp=0.8)

Out of curiosity, why call Dehalo 3 times instead of just increasing h? Does it help with higher order halos or something?

[brucethemoose]

In certain material, Vine has trouble correctly detecting the actual edges:

Look at the edges of the cards here:

http://www.screenshotcomparison.com/comparison/185510

In that crack between the 2 black lines, it's so bad that the halos themselves are detected as edges, and it ends up blurring the actual edges. The mask and strength settings don't seem to affect that.



Feisty2, I know this is a live-action filter, but could you add an option that gives priority to black lines during edge detection? This would help oversharpened animation dramatically.

[feisty2]

there's no edge detection with "masking=False"
you do a halo/edge mask for your specific video on your own