Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 19th September 2016, 17:51   #21  |  Link
brucethemoose
Registered User
 
Join Date: Sep 2016
Posts: 67
Quote:
Originally Posted by feisty2 View Post
it should work on cartoon stuff also I guess, but probably not worth it, Vine is pretty slow, and other lower quality but levels faster dehalo filters might be good enough for cartoons since getting cartoonized is one of the last things to worry about on literally cartoon vids
Some toons have more detailed, painted backgrounds that you have to worry about, but that's still true.

In fact, my problem is that Vine isn't strong enough... I've tried tweaking parameters, but it barely puts a dent in big sharpening halos like this:

http://i.cubeupload.com/YAZVwX.png
brucethemoose is offline   Reply With Quote
Old 19th September 2016, 19:30   #22  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
Quote:
Originally Posted by brucethemoose View Post
Some toons have more detailed, painted backgrounds that you have to worry about, but that's still true.

In fact, my problem is that Vine isn't strong enough... I've tried tweaking parameters, but it barely puts a dent in big sharpening halos like this:

http://i.cubeupload.com/YAZVwX.png
You need to learn that miracles don't happen. And be skeptic about halo removing in general. AFAIR, there has been no method to remove them that wasn't destructive to other parts of video. What you've got is a horrible multi-peaks ringing. No hope to fight it. Better to not cause it.
__________________
...desu!
Mystery Keeper is offline   Reply With Quote
Old 19th September 2016, 19:31   #23  |  Link
Nevilne
Registered User
 
Join Date: Aug 2010
Posts: 134
"you can't expect this halo remover to remove these halos"
Nevilne is offline   Reply With Quote
Old 19th September 2016, 20:21   #24  |  Link
brucethemoose
Registered User
 
Join Date: Sep 2016
Posts: 67
Quote:
Originally Posted by Mystery Keeper View Post
No hope to fight it. Better to not cause it.
Unfortunately, that's the raw NTSC DVD rip. Yeah, it's that bad .

And yeah, I know... but there is a compromise somewhere between the Vine settings I've tried and DeHalo_Alpha, I just haven't found it yet.
brucethemoose is offline   Reply With Quote
Old 19th September 2016, 22:44   #25  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
Quote:
Originally Posted by Nevilne View Post
"you can't expect this halo remover to remove these halos"
These multi-peaks? No, you can't. Narrow one peak halos - sure, you can. But expect it to screw up something else.
__________________
...desu!
Mystery Keeper is offline   Reply With Quote
Old 19th September 2016, 23:23   #26  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Have you tried to raise "h" and "radius" up like a bit insane, especially "h", like "h=24"
feisty2 is offline   Reply With Quote
Old 20th September 2016, 00:06   #27  |  Link
brucethemoose
Registered User
 
Join Date: Sep 2016
Posts: 67
Quote:
Originally Posted by feisty2 View Post
Have you tried to raise "h" and "radius" up like a bit insane, especially "h", like "h=24"
Not yet, thanks.


But holy moly... now I'm running this:

Code:
 Vine.Dehalo(clip, radius=[8, 4], a=32, h=24, sharp=1.0, sigma=0.6, alpha=0.36, beta=32, cutoff=4, show=False)
And its taking like 8 seconds per 720x480 frame to encode. I'll get back to you when I get results.

Last edited by brucethemoose; 20th September 2016 at 00:09.
brucethemoose is offline   Reply With Quote
Old 20th September 2016, 05:37   #28  |  Link
brucethemoose
Registered User
 
Join Date: Sep 2016
Posts: 67
Quote:
Originally Posted by feisty2 View Post
Have you tried to raise "h" and "radius" up like a bit insane, especially "h", like "h=24"
24 is way too strong, this is 20 with radius 4,2 (with a straight spline36 scale for reference):

http://www.screenshotcomparison.com/comparison/185123

Look at the guy's eyes, in particular.

I'm messing with it now, and 12-16 is looking better... the higher order halos aren't completely gone, but I might be able to mask them with another filter now.


This is some great code though, thanks feisty2.

Last edited by brucethemoose; 20th September 2016 at 05:39.
brucethemoose is offline   Reply With Quote
Old 20th September 2016, 05:49   #29  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Now I think a large "h" is not it, something like "h=8" might be enough already
You need to disable all mask stuff in Vine cuz the halos are roaming VERY far from the exact edges, pointless tryna "mask" them out
feisty2 is offline   Reply With Quote
Old 20th September 2016, 05:51   #30  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
I'll add such option(disable mask stuff) after college
feisty2 is offline   Reply With Quote
Old 20th September 2016, 06:31   #31  |  Link
brucethemoose
Registered User
 
Join Date: Sep 2016
Posts: 67
Quote:
Originally Posted by feisty2 View Post
Now I think a large "h" is not it, something like "h=8" might be enough already
You need to disable all mask stuff in Vine cuz the halos are roaming VERY far from the exact edges, pointless tryna "mask" them out
Sounds good.

However, if I'm looking at it right, I think the mask is actually reaching WAY out to the halos as it should.

http://u.cubeupload.com/brucethemoos...phavpymask.png

I don't think disabling it would help. There are other scenes with much more intricate painted backgrounds, and it looks like the mask will help preserve those.

EDIT: I could be wrong though, maybe I'm misunderstanding what the mask does...

So for now, what parameters should I adjust for the mask?

Last edited by brucethemoose; 20th September 2016 at 06:36.
brucethemoose is offline   Reply With Quote
Old 20th September 2016, 06:48   #32  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
radius=[10, None], sigma=1.6, alpha=0.48 works on your first screenshot
I'm on cellphone right now so...
feisty2 is offline   Reply With Quote
Old 21st September 2016, 03:20   #33  |  Link
brucethemoose
Registered User
 
Join Date: Sep 2016
Posts: 67
Quote:
Originally Posted by feisty2 View Post
radius=[10, None], sigma=1.6, alpha=0.48 works on your first screenshot
I'm on cellphone right now so...
Sorry for not responding earlier, I went down the deinterlacing/IVTC rabbit hole.

What h value did you use?
brucethemoose is offline   Reply With Quote
Old 21st September 2016, 13:28   #34  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
r5:
new parameter "masking"
feisty2 is offline   Reply With Quote
Old 21st September 2016, 13:33   #35  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
@brucethemoose
Code:
clp = Vine.Dehalo(clp, h=6, masking=False, cutoff=1, sharp=0.4)
clp = Vine.Dehalo(clp, h=4, masking=False, cutoff=1, sharp=0.6)
clp = Vine.Dehalo(clp, h=2, masking=False, cutoff=1, sharp=0.8)
feisty2 is offline   Reply With Quote
Old 21st September 2016, 13:40   #36  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
and of course you can go for a custom mask for this specific vid and protect soft features a bit more
feisty2 is offline   Reply With Quote
Old 21st September 2016, 18:22   #37  |  Link
brucethemoose
Registered User
 
Join Date: Sep 2016
Posts: 67
Fantastic, thanks! I'm playing with this now, might upload some before/after screenshots/clips later to show Vine off, as those Avatar DVDs are a DeHalo torture test.

Last edited by brucethemoose; 21st September 2016 at 18:24.
brucethemoose is offline   Reply With Quote
Old 22nd September 2016, 02:48   #38  |  Link
brucethemoose
Registered User
 
Join Date: Sep 2016
Posts: 67
Quote:
Originally Posted by feisty2 View Post
@brucethemoose
Code:
clp = Vine.Dehalo(clp, h=6, masking=False, cutoff=1, sharp=0.4)
clp = Vine.Dehalo(clp, h=4, masking=False, cutoff=1, sharp=0.6)
clp = Vine.Dehalo(clp, h=2, masking=False, cutoff=1, sharp=0.8)
Out of curiosity, why call Dehalo 3 times instead of just increasing h? Does it help with higher order halos or something?
brucethemoose is offline   Reply With Quote
Old 23rd September 2016, 22:22   #39  |  Link
brucethemoose
Registered User
 
Join Date: Sep 2016
Posts: 67
In certain material, Vine has trouble correctly detecting the actual edges:

Look at the edges of the cards here:

http://www.screenshotcomparison.com/comparison/185510

In that crack between the 2 black lines, it's so bad that the halos themselves are detected as edges, and it ends up blurring the actual edges. The mask and strength settings don't seem to affect that.



Feisty2, I know this is a live-action filter, but could you add an option that gives priority to black lines during edge detection? This would help oversharpened animation dramatically.
brucethemoose is offline   Reply With Quote
Old 24th September 2016, 04:43   #40  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
there's no edge detection with "masking=False"
you do a halo/edge mask for your specific video on your own
feisty2 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 13:50.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.