Good Halo treatment?

[Codex0nz]

Yesterday my Avatar The last airbender dvd's arrived from usa, after popping one in and watching i noticed straight away horrible halo'ing (so much for having a good source -_-)

At the moment im trying out dehalo_alpha which does seem to do an average job for me

Before touching:
http://img75.imageshack.us/img75/3988/halotn3.jpg

After dehalo_alpha:
http://img150.imageshack.us/img150/8622/halofixattemptyz5.jpg

Yeah i also noticed that it shouldn't be resizing it :S but i can sort that out later

The source appears to be hybrid (although im not positive...)

Sample:
http://www.megaupload.com/?d=RH0LFP05

Script:
DGDecode_mpeg2source("H:\VOB\Avatar\Book 1\0 Test\vts_03_1.d2v",info=3)
ColorMatrix(hints=true, interlaced=true)

animeivtc(mode=5, aa=0, iregion="NTSC", oregion="NTSC")
DeHalo_Alpha(darkstr=0.4)

crop( 4, 2, -2, -2)

BicubicResize(656,480,0,0.5) # Bicubic (Neutral)

Question:
I guess the main question im asking is

Am i using a good halo treatment method? Should i be using a different script for my situation? Am i being a fool and doing something wrong?

Any help is highly appreciated :)

[thetoof]

There were already a few discussions about Avatar's haloing issue. One of 'em: http://forum.doom9.org/showthread.php?t=141433

[Codex0nz]

Yeah i did look around but couldn't find an avatar related thread that had an actual solution, like that one ended with what seemed to be no solution :S

[thetoof]

Post 9 and 17 of that thread have some good tips... but it's true that it ended without any "fully satisfying" solution. That being said, this is as far as I can help you, so let's hope somebody will jump in this thread with some good scripts!

[Nightshiver]

I don't think it's hybrid. I see no 3:2 pattern.

animeivtc(mode=2, aa=0)

For the halo's, this worked rather well: DeHalo_alpha(brightstr=1.3,darkstr=1.3,rx=2.1,ry=2.1,lowsens=50,highsens=20)
Resize to 640x480:

Crop(2,0,-0,-0)
spline36Resize(640,480)

You will also want to sharpen and give it a slight color boost as dehalo_alpha kind of "mushes" the picture and takes a little bit of saturation away.

Also, thetoof, for some reason avsp is giving me the error

System exception - Access Violation
(, line 148 of animeivtc)

[martino]

After a fast play around with it...

BlindDeHalo3(PPmode=-2,strength=200,rx=5,ry=5)
EdgeCleaner(14) #might nuke some of those "rings" or use plain aWarpSharp

I haven't played too much with the rx/ry/strength settings, might be worth trying out different values (mainly PPmode). EdgeCleaner can be found below.

# EdgeCleaner() v1.03 (06/08/2008)
# - a simple edge cleaning and weak dehaloing function
#
# Description:
# Functions have been briefly tested to work with MT on mode 1 and 2 without any problems
#
# Requirements:
# aWarpSharp, mt_masktools, Repair (optional), RemoveGrain (optional) and Deen (optional) plugins required
# YV12 input required and mod16 or even mod32 input is preferred since aWarpSharp borks sometimes
#
# Parameters:
# strength (float) - specifies edge denoising strength (8.0)
# rep (boolean) - actives Repair for the aWarpSharped clip (true; requires Repair)
# rmode (integer) - specifies the Repair mode; 1 is very mild and good for halos,
# 16 and 18 are good for edge structure preserval on strong settings but keep more halos and edge noise,
# 17 is similar to 16 but keeps much less haloing, other modes are not recommended (17; requires Repair)
# smode (integer) - specifies what method will be used for finding small particles, ie stars; 0 is disabled,
# 1 uses RemoveGrain and 2 uses Deen (0; requires RemoveGrain/Repair/Deen)
# hot (boolean) - specifies whether removal of hot pixels should take place (false)
# fix (boolean) - fixes an aWarpSharp bug by overlaying a healthy pixel from the source clip;
# good idea to set to false when over-cropping afterwards (true)

function EdgeCleaner(clip c, float "strength", bool "rep", int "rmode", int "smode", bool "hot", bool "fix") {

strength = default(strength, 8.0)
rep = default(rep, true)
rmode = default(rmode, 17)
smode = default(smode, 0)
hot = default(hot, false)
fix = default(fix, true)

c = (c.isYV12()) ? c : c.ConvertToYV12()
strength = (smode==0) ? strength : strength+4

main = c.aWarpSharp(strength,1)
main = (rep) ? Repair(main,c,rmode) : main

mask = c.mt_edge("prewitt",4,32,4,32).mt_invert().mt_convolution()

final = (!hot) ? mt_merge(c,main,mask) : Repair(mt_merge(c,main,mask),c,2)
final = (fix) ? Overlay(final,c.ConvertToRGB24().Crop(0,1,-c.width+1,-c.height+2),x=0,y=1) : final
final = (smode != 0) ? mt_merge(final,c,c.StarMask(smode)) : final

return final

}

function StarMask(clip c, int "mode") {

mode = default(mode, 1)

clean = (mode==1) ? c.RemoveGrain(17) : Repair(c.Deen("a3d",4,12,0),c,15).RemoveGrain(21)
diff = (mode==1) ? mt_makediff(c,clean) : NOP

final = (mode==1) ? diff.Greyscale().Levels(40,0.350,168,0,255).removegrain(7,-1).mt_edge("prewitt",4,16,4,16) : \
Subtract(mt_merge(clean,c,c.mt_edge("roberts",0,2,0,2).mt_expand(mode=mt_circle(1)).mt_invert()),c).mt_edge("roberts",0,0,0,0).mt_deflate()

return final

}

# Changelog:
# 06/08/2008 v1.03
# - improved mask that leaves less warping and more original line structure, therefore higher strengths are now safe to use
# - improved StarMask()
# - removed super mode
# - removed srep, sshiqloc, some smodes and VD_SmartSmoothHiQ() due to StarMask() changes
# 01/06/2008 v1.02
# - added srep parameter
# - improved particle masking
# 01/06/2008 v1.01
# - added masking for particles with two parameters; smode and sshiqloc
# 12/05/2008 v1.00
# - removed line darkening, mode 2 mask, RemoveGrain
# - assert changed to colorspace conversion to yv12
# - fixed some logic problems
# - "fixed" the aWarpSharp black pixel bug
# - added Repair

[thetoof]

There is a 3:2 pattern, but it is broken/reset/whatever-you-may-call-it at scene changes since it looks like hard pulldown was applied prior to editing. There'll be a major docs update in the next release to prevent such confusion...
Avatar is a known hybrid source, so even if this section doesn't have 30p motion, you should keep using hybrid decimation just in case.

Nightshiver, your dehaloing line works quite well and combined with a little edge enhancement/anti-aliasing/warpsharpening, it looks somewhat good.
SharpAAMCmod() #have fun tweaking

However, there is much work to do to because the problem described by Mug Funky cannot be solved easily
http://img139.imageshack.us/my.php?image=ouchqa2.jpg
(The top of their collar and Sokka's mouth are a good example of what's hard to solve)

Something that'd "bring back" the field to its original position would be best... because for now it only looks like a big ol' blur.
http://img411.imageshack.us/my.php?image=test0kf2.jpg

Maybe a mask that'd select the edges that are similar and a few pixels above to then use the neighbour pixels to estimate what it should be instead?

Oh, and I just tried martino's script and it leads to similar results...

[Nightshiver]

Alright, I guess I just didn't have a long enough clip to see that it was hybrid. When is this next update to animeivtc of yours?

[Sagekilla]

Your source looks like it could use some warp sharpening too. That might help a bit.

[zee944]

You should try Didée's YAHR:

# Y'et A'nother H'alo R'educing script

function YAHR(clip clp)
{
b1 = clp.minblur(2).removegrain(11,-1)
b1D = mt_makediff(clp,b1)
w1 = clp.aWarpSharp(depth=32,blurlevel=2,thresh=0.5)
w1b1 = w1.minblur(2,1).removegrain(11,-1)
w1b1D = mt_makediff(w1,w1b1)
DD = b1D.repair(w1b1D,13)
DD2 = mt_makediff(b1D,DD)
clp.mt_makediff(DD2,U=2,V=2)
}

From this thread: http://forum.doom9.org/showthread.php?p=1205653#post1205653

You should also try aWarpSharp, too. In the same thread above. I'm just guessing, but a smart combination of these three tools would almost totally eliminate the halos.

[*.mp4 guy]

mt_convolution(horizontal = " 1 ", vertical = " 12 -24 0 63 127 63 0 -24 12 ", u=2, v=2)

separatefields()

mt_convolution(horizontal = " 1 ", vertical = " 0 -6 10 0 -25 127 -25 0 10 -6 0 ", u=2, v=2)
mt_convolution(horizontal = " 0 -24 80 -112 255 -112 80 -24 0 ", vertical = " 0 -24 0 112 255 112 0 -24 0 ", u=2, v=2)

weave()

Yahr()

MT_Merge(mt_convolution(horizontal = " -5 -20 -5 192 -5 -20 -5 ", vertical = " -10 -20 -10 192 -10 -20 -10 ", u=2, v=2).mergechroma(last, 0), last.blur(0.3).mergechroma(last, 0), last.halomask1(ss=1.75, rad=6).frfun7(1.01, 256, 256))

[Adub]

Holy jimminychristmas. I hope that I will be able to understand that line in the next few years, and not the next few decades. I read your explanation in the right edge halos thread, but I think I need to go over it again. And the masktools manual.

[thetoof]

Yeah... if it's not much of a bother, could you give a *very* dumbed down explanation of how you ended up with those lines + what they do?

Also, even if it does remove the halos, the lines become "wavy" (very first frame of the sample)... though a good part of that can be corrected with sharpaamcmod()

[Codex0nz]

Man there are some intelligent people on this site o_O what req's are there for your script mp4 guy?

EDIT: Don't worry i managed to find them all :)

EDIT 2: Shitmyselfsideways you are inhuman mp4

EDIT 3:

The script gets rid of the halo's from what i see (most of them anyway) but i think needs tweaking to get a better result, as the final seems over sharpened / blurred :(

Source:
http://img296.imageshack.us/img296/3995/testfirstsx2.jpg

After MP4's suggested addition to the script (I wish i knew what the hell it meant and how to tweak it :S):
http://img296.imageshack.us/img296/164/testfp8.jpg

[thetoof]

Have you tried adding sharpaamcmod() as I suggested?

[*.mp4 guy]

Most of the blur is from the first mt_convolution line, it applies a vertical lowpass to remove the aliasing of certain lines. If you remove it, the output of the script will be sharper then the input, so feel free to replace it with a better vertical antialiaser if you know of one.

[Codex0nz]

Have you tried adding sharpaamcmod() as I suggested?
I can't find it, i searched the forums and google came up with Japanese sites :S If you link me to it then i will happily have another crack at it :)

Most of the blur is from the first mt_convolution line, it applies a vertical lowpass to remove the aliasing of certain lines. If you remove it, the output of the script will be sharper then the input, so feel free to replace it with a better vertical antialiaser if you know of one.

I have faith in your lines lol if you can write that then im sure you know what you are doing.

AnimeIVTC has it's own anti alias feature does it not? Or would that just be a silly suggestion

Thanks for the great suggestions so far and they are all certainly an improvement over the source it's-self

[thetoof]

Nothing in AnimeIVTC can properly deal with this kind of thing without the help of something external.
However, when I was talking about edge enhancement/anti-aliasing/warpsharpening as a post-processor to dehaloing, I was referring directly to sharpaamcmod, which is aa=4. So, as long as you have AnimeIVTC in your plugins folder, you can call it.

I was actually hoping for *mp4 guy to bump in this thread with his crazy mt_convolution scripts, as I'll start encoding Avatar soon and I was looking for some neat dehaloing :p

[Codex0nz]

Im desperately hoping you'll share any success you have with me toof please

should i call aa=4 again after those lines?

like AnimeIVTC(aa=4) after mp4 guys lines?

[thetoof]

No.
As I said previously in this thread as well as the docs, aa=4 can be called externally as sharpaamcmod(). You should not use it internally as it is meant, in your case, to be a dehaloing post-processor, meaning that you must run it after the mt_convolution lines.

I mentioned "aa=4" in the previous post to answer your question about using animeivtc's aa approaches. Yes, you can use them, but not alone.

ivtc
other filters
dehalo
sharpaamcmod
other filters

[Codex0nz]

No.
As I said previously in this thread as well as the docs, aa=4 can be called externally as sharpaamcmod(). You should not use it internally as it is meant, in your case, to be a dehaloing post-processor, meaning that you must run it after the mt_convolution lines.

I mentioned "aa=4" in the previous post to answer your question about using animeivtc's aa approaches. Yes, you can use them, but not alone.

ivtc
other filters
dehalo
sharpaamcmod
other filters

Ok so i read the documentation regarding your suggestions, and did the script as you instructed and got this result:

http://img242.imageshack.us/img242/9844/testingspg2.jpg

http://img242.imageshack.us/img242/5599/testings2zp3.jpg

So now i am wondering if i should attempt running dehaloalpha when it gets to this point to wipe out the leftovers?

[thetoof]

Well, try it! The more you test, the more you'll learn!
At this point it's more or less a matter of personal preferences ;)
You could also try tweaking sharpaamcmod as it may be over-sharpening and introducing new halos.

[xxxyzzzz]

@Codex0nz

Can you please post the you got to obtain those last results? I have solved all the hybrid/deinterlacing problems with Avatar (with excellent help in this forum,) but I am still hung up on the halo-ing/over-sharpening in season 1.

Thanks,
Z

[Codex0nz]

There is still haloing in those last results

I cant figure it out past there

Dehalo_alpha destroys the detail and detailed lines become really bold etc -_-

[Nightshiver]

Such is the drawback of using something as good as dehalo_alpha.

[Codex0nz]

yeah

well im going to keep on trying

but is there any other way to deal with this kind of thing? as some one said before a filter to realign the fields perhaps? (Could be my misinterpretation :S)

I cant get any better then what ive been getting so far, and what i have so far is thanks to everyone who helped, i appreciate it :)

[Codex0nz]

I Don't mean to bump my topic, but mp4 guy, i was fiddling with your script last night and saw the separate fields call in that awesomely epic script your wrote.

If the fields can be separated is it possible to realign them and sew them back together to get rid of these effects? or am i out of my mind and have completely failed to understand anything so far -_-

[*.mp4 guy]

I actually tried that a while ago, its probably possible to fix the artifacts through some sort of field juju, but it isn't a simple sub-pixel shift, I can't figure out what exactly to do to fix it, everything I've tried only makes it worse.

I would guess that the fields were put through some sort of convolution separatley, and this caused the pseudo field misalignment artifacts, but without knowing exactly what happened to the source, I don't know what to do to fix it.

[Codex0nz]

I actually tried that a while ago, its probably possible to fix the artifacts through some sort of field juju, but it isn't a simple sub-pixel shift, I can't figure out what exactly to do to fix it, everything I've tried only makes it worse.

I would guess that the fields were put through some sort of convolution separatley, and this caused the pseudo field misalignment artifacts, but without knowing exactly what happened to the source, I don't know what to do to fix it.

I can upload some more clips for you to play with if you want? I spent all weekend seeing if different things worked but can't get anything to produce any results even close to what you did

Someone in this thread suggested it was from final cut pro? Maybe we should start there? :S If we can figure this one out im shouting you a beer, lol

[Nightshiver]

Why do you want to go through all this trouble even if it is possible when you can already do a decent alternative?

[Codex0nz]

Why do you want to go through all this trouble even if it is possible when you can already do a decent alternative?

Because it's hardly decent on small details, i'll post screenshots later if i get a psu for my vcr going -_-

And if we can sort this out then it could help with a lot of thigns that have suffered the same fate

[ShocWave]

I'd hate to bump an old thread, but this is the most recent thread I found using the search feature.

I have a bunch of older videos, some VHS captures and eairly DVD releases which suffer from extreme haloing like shown in this thread.

What *.mp4 guy has done is absolutely amazing. I've poked through some of the other halo threads and found that there is some sort of masks applied and certian values can be calculated with a program. Still, I have no clue how this works.

I've tried pretty much every dehalo filter out there, and I couldn't come anywhere close to producing the results that *.mp4 guy managed. Is anyone able to shed some light on how his script works?

[*.mp4 guy]

Which particular parts don't make sense? I can explain how I am applying certain ideas, but I can't, for instance, adequately explain how a FIR filter works and how to make one.

I've been doiung some work on making stuff like this more generally applicable (less hand tweaking) but it still needs work.

[saint-francis]

I've been doiung some work on making stuff like this more generally applicable (less hand tweaking) but it still needs work.

Interesting. Do you have an update to the script you posted earlier int he thread then?

[ShocWave]

Which particular parts don't make sense? I can explain how I am applying certain ideas, but I can't, for instance, adequately explain how a FIR filter works and how to make one.

I've been doiung some work on making stuff like this more generally applicable (less hand tweaking) but it still needs work.

I was wondering about the FIR filter, but I suppose it's much too complicated to explain. :(

[Guest]

http://en.wikipedia.org/wiki/Finite_impulse_response

[*.mp4 guy]

They aren't exactly complicated, I just find them hard to explain. There are a lot of ways to conceptualize them, but it doesn't matter that much, just using guess and check you can usually do pretty well.

Interesting. Do you have an update to the script you posted earlier int he thread then?
I don't have the avatar on dvd, so no. However I might be able to make a gneral purpose script at some point, assuming I can fix a few more issues.

[travolter]

Im looking for some dering or dehalo script to work in realtime with ffdshow..

I can sacrifice a bit of quality if Ill gain speed

Anyone know a fast script?