How to soften aliased edges after IVTC (anime)?

[Eud]

I'm looking for a way to smooth/soften these kinds of edges that show mainly around the hair:

http://min.us/mbb51RzntJlEWT

They aren't really that noticeable at source resolution, but when the video gets upscaled to screen resolution they start to get really annoying.

I tried some denoisers and smoothers but they eat a lot of detail, more than I'm willing to lose. Maybe I can't set them up properly (I tried the recommended "best" settings first and then lowered them a bit: the former eats too much, the latter doesn't smooth enough), so the results aren't what I'm wishing for.

The avisynth script is really short, just IVTC and some debanding:

Telecide()
Decimate(5)
AssumeFPS(24000, 1001)
Spline64Resize(1280, 720)
GradFun3(smode=3)

Anything you recommend or must I just give up and keep the aliasing or lose the details?

[Xaser]

hmm well you are lucky that you got the footage deinterlaced with IVTC, most footage I got my hand on underwent the weirdest processing...

Anyway, one filter you should give a try is MSmooth (http://neuron2.net/msmooth/msmooth.html). It was developed especially with animated footage in mind, I got some real improvement on my dirty interlaced (QTGMC deinterlaced) footage. Also have a look at MSharpen.

A good resource on how to process animes can be found here:
http://www.animemusicvideos.org/guides/avtechold/index.html

Regards,
Max


PS: doing some research on the filters you used... I'm still learning as well and as a non-native I'm having trouble understanding the meaning of debanding.. can somebody post a link real quick or send me a pm?
EDIT: nvm, I found an explanation of banding noise here (http://blog.topazlabs.com/denoise-5s-debanding-technology/)

[Asmodian]

For AA I hear good things about DAA (http://avisynth.nl/index.php/DAA) but I am not experianced with it. I found some more options here (http://pastebin.com/Y42s9hRZ) but I don't know how well any of them work.

This image from the GradFun2DBmod thread is a good example of banding. Look at the night sky in the input.
http://latoninf.free.fr/d9/GF/show.png

Edit:
EDIT: nvm, I found an explanation of banding noise here (http://blog.topazlabs.com/denoise-5s-debanding-technology/)

That is not the kind of banding we are talking about, we are talking about encoder/processing introduced banding not camera introduced banding noise.

[Xaser]

That is not the kind of banding we are talking about, we are talking about encoder/processing introduced banding not camera introduced banding noise.


That's why I never delete stuff from my posts, always better to be corrected than confusing stuff. :thanks:

At first I didn't even see the difference in the original picture, but after I turned my desk-light off, it became perfectly visible.

[Motenai Yoda]

can u try with this, also there is a santiag(mod) aa that works likewise

function nnediaa(clip input)
{
mask = input.mt_edge("sobel",7,7,5,5,u=1,v=1).mt_inflate(u=1,v=1)
aa_clip=input.nnedi3(field=1,dh=true,nsize=4,u=false,v=false).TurnRight().nnedi3(field=1,dh=true,nsize=4,u=false,v=false).TurnLeft().spline36resize(input.width,input.height,-0.5,-0.5,2*input.width,2*input.height)
return mt_merge(input,aa_clip,mask,u=2,v=2)
}

@Xaser, MSmooth? MSharpen? Seriously?
They aren't bad filters, but now there are more that outclass them. (probably a neuron2's post appear right below this)

Banding is when and where a lossy pass generates bands of uniform color instead of gradients
http://reymendoza.com/wp-content/uploads/2013/04/Colour_banding_example01.png

[Xaser]

Thanks, yoda, I hope I didn't drive this thread too much OT.

For AA MSmooth certainly may not be the best filter, but for the artefacts described on the MSmooth page, it does real wonders (at least of the (few) filters I tried it got the best result). If you have any others I should try, please tell me, I'm always eager to learn more ;)

[Eud]

can u try with this, also there is a santiag(mod) aa that works likewise

function nnediaa(clip input)
{
mask = input.mt_edge("sobel",7,7,5,5,u=1,v=1).mt_inflate(u=1,v=1)
aa_clip=input.nnedi3(field=1,dh=true,nsize=4,u=false,v=false).TurnRight().nnedi3(field=1,dh=true,nsize=4,u=false,v=false).TurnLeft().spline36resize(input.width,input.height,-0.5,-0.5,2*input.width,2*input.height)
return mt_merge(input,aa_clip,mask,u=2,v=2)
}


That didn't really seem to work that much:

http://i6.minus.com/irhOAGvfFBcPC.png

Aside from things like the credits, and even then it's not noticeable:

http://i1.minus.com/iAbJweTAjwGw.png

Another thing I noticed going frame by frame are these kind of deinterlacing artifacts:

A: http://i7.minus.com/i7JZXtgM3468s.png
B: http://i7.minus.com/ikJHjvd0On1bC.png
C: http://i3.minus.com/ibtQ37fpWu4DHZ.png

A: http://i5.minus.com/iV5VGIaPyKfJu.png
B: http://i1.minus.com/ibaamlth2lRYVb.png

Isn't there a way to blend the frames without that combing effect? Or to selectively use the previous frame if the change is practically non-influential like in the second example?

Actually nevermind, I just noticed that in the second example the frame moves. I'd still like to know if there's a way to fix that, possibly without heavy smoothing.

(Using this for the comparisons: http://forum.doom9.org/showpost.php?p=1683804&postcount=2)

[osgZach]

You could try ripping the anti-aliasing functions out of AnimeIVTC, etc and running them through one of them.

I always liked SharpAACMod, but it is pretty CPU intensive.

Also those artifacts (Lavi's back) are likely bad field matches. You might get better results by using TFM (TIVTC) in 2-pass to generate a VFR file with timecodes, btw.
I think Telecide has similar overrides for fixing field matches as well.

[Motenai Yoda]

That didn't really seem to work that much:
then go with maa/maa2, nnedi(3)aa is too gentle.
http://forum.doom9.org/showthread.php?t=168315

also as osgZach said, try with tfm(pp=6).tdecimate(mode=1) instead of telecide+decimate (set pp=0 to disable postprocessing)

[Eud]

I decided to address the IVTC problem first. I did some tests with different settings:

Just using AnimeIVTC:
AnimeIVTC(4, omode=1)

http://i5.minus.com/iI9FnPZ6IipmT.png
http://i2.minus.com/iPLbkdAAlYQe5.png

Cutting the IVTC part out of the script to avoid mixing in other unneeded filters:
tfm(slow=2, pp=6, clip2=clip.tdeint(2, edeint=clip.nnedi3(-2), emask=clip.tmm(1)))
tdecimate(mode=1, hybrid=1)

http://i1.minus.com/iMdcMzfFUHZVm.png
http://i1.minus.com/ibaVNvsdFXqi5T.png

Motenai Yoda's suggested settings:
tfm(pp=6).tdecimate(mode=1)

http://i1.minus.com/ibfZiFwG6zcJyk.png
http://i7.minus.com/iEJJyZxecW3JI.png

And back to Decomb:
Telecide()
Decimate(5)

http://i3.minus.com/ibzQuEHSkNX5Wi.png
http://i6.minus.com/ibj04YqSPFNriF.png

Then I went back and looked at the original material:

http://i5.minus.com/iT3RqPyYiBoac.png
http://i1.minus.com/ibeBOobLR68lRs.png

I'm scratching my head here, thinking that if those are marked as progressive frames that might be the result of a bad deinterlaced source that was then telecined for the blu-ray release, and if that's the case I've no idea how to solve it.

[ChiDragon]

I assumed you were upsizing a DVD, not downsizing an upscaled 1080i Blu-ray. I would say you're fighting against artifacts from the 480i -> 480p -> 1080p -> 1080i process used to create the disc then.

One of the fields in the second raw image you posted (http://i1.minus.com/ibeBOobLR68lRs.png) looks reasonable, but the other one is filled with jaggies and/or upsized combs (view with SeparateFields).

[Eud]

Yeah, sorry, I should've specified that.
The main reason I was using the Spline64Resize(1280, 720) was because doing so most of the interlacing artifacts seemed to disappear or become almost unnoticeable (I guess because some alternate fields got fused, smoothing the picture), but since now I'm trying to tackle again the problem from the beginning I removed it and I'm working at source resolution.

[Motenai Yoda]

seems for me an R1 or an upsampled BD relase, right?
most of them were processed with no or bad ivtc then upscaled and re-telecined.

this filter maybe can do something... http://forum.doom9.org/showthread.php?t=158696

[foxyshadis]

That's a pity, I have the Last Exile DVD box set and was looking into the BDs.

I'm fairly certain these were upscaled from the DVDs (without any kind of filtering either) based on some of the encoding artifacts I see, though I'd have to go back to mine to compare frames. The DVDs are wretched to IVTC, because they're full of first-generation CGI added post-telecine at 60i (that I'm sure was only rendered at 480i, so there's no high-def master in the first place), and it's obvious they half-assed the Bluray instead of recreating the CGI from scratch.

Using Vinverse and strong filtering is likely the only way to save these, short of going back to the DVDs and making your own HD remaster.

[mandarinka]

I'm looking for a way to smooth/soften these kinds of edges that show mainly around the hair:

http://min.us/mbb51RzntJlEWT

This sort/extent of aliasing should be well handled by this script (http://forum.doom9.org/showthread.php?t=167480).

Somebody suggested Daa, but that filter isn't very suitable here IMHO. It is meant to only anti-alias in the vertical direction (and your pictures need both directions), because it is made for effects of interlacing. You also need to run Daa at higher-than-source resolution, because it causes a lot of artifacts if you don't.

[osgZach]

That's a pity, I have the Last Exile DVD box set and was looking into the BDs.

I'm fairly certain these were upscaled from the DVDs (without any kind of filtering either) based on some of the encoding artifacts I see, though I'd have to go back to mine to compare frames. The DVDs are wretched to IVTC, because they're full of first-generation CGI added post-telecine at 60i (that I'm sure was only rendered at 480i, so there's no high-def master in the first place), and it's obvious they half-assed the Bluray instead of recreating the CGI from scratch.

Using Vinverse and strong filtering is likely the only way to save these, short of going back to the DVDs and making your own HD remaster.

Yeah this crap really pisses me off.. I think a lot of Companies have learned their lesson, but there are a lot of bad upscaled Blu Rays out there from the initial switch from SD to HD.

It's a shame to learn Last Exile got the same treatment, and I know Chobits is one to avoid as well. Gundam Seed is a mixed bag, but its mostly upscale issues as oppose to butchering everything.

I actually need to find out if my Blu Ray player upscales DVD's or not and see how good it is. Now that I've got a capture card and a means to bypass HDCP it might be easier to just record that and see what kind of output I get.... Although I guess I could invest in a hardware scaler too, but that's almost as much as my capture card cost..:(

OP - any chance of providing a source clip?

[Eud]

seems for me an R1 or an upsampled BD relase, right?
most of them were processed with no or bad ivtc then upscaled and re-telecined.

this filter maybe can do something... http://forum.doom9.org/showthread.php?t=158696
Yeah, R1.
Unfortunately that filter didn't seem to do anything. Not that I was expecting it to work since this didn't look like it was upscaled from 720p.

That's a pity, I have the Last Exile DVD box set and was looking into the BDs.

I'm fairly certain these were upscaled from the DVDs (without any kind of filtering either) based on some of the encoding artifacts I see, though I'd have to go back to mine to compare frames. The DVDs are wretched to IVTC, because they're full of first-generation CGI added post-telecine at 60i (that I'm sure was only rendered at 480i, so there's no high-def master in the first place), and it's obvious they half-assed the Bluray instead of recreating the CGI from scratch.

Using Vinverse and strong filtering is likely the only way to save these, short of going back to the DVDs and making your own HD remaster.
Indeed. I also own a PAL version of the DVDs but I never looked at them closely on the computer. Maybe I'll take a look at them just in case they weren't re-encoded from the NTSC DVDs but taken directly from the source. Probably it'll be pointless though.

This sort/extent of aliasing should be well handled by this script (http://forum.doom9.org/showthread.php?t=167480).

Somebody suggested Daa, but that filter isn't very suitable here IMHO. It is meant to only anti-alias in the vertical direction (and your pictures need both directions), because it is made for effects of interlacing. You also need to run Daa at higher-than-source resolution, because it causes a lot of artifacts if you don't.


In the end I realized that IVTC-wise I get the best results with my original script, using Decomb to IVTC then resizing to 720p: http://i1.minus.com/ibba1rnGKxtWo6.png

Vinverse seemed to help a bit in hiding the combing artifacts so I'll be adding it to my chain: http://i1.minus.com/ibb62LSumxSPo5.png

Now about the aliasing: I tried nnediaa, mrdaahq and a combination of the two. As previously mentioned, nnediaa didn't seem to affect anything noticeably, whereas mrdaahq's only effect was a slight sharpening of the picture: before (http://i3.minus.com/i0u9UWRo2f86U.png) and after (http://i6.minus.com/ibrEMXXuD1dBNa.png).
I actually like it better now, but it didn't really help diminishing the aliasing.
I couldn't find SharpAACMod.

MAA2 gave me pretty good results: http://i5.minus.com/iM0XsXN9cOzog.png
It doesn't seem to have eaten that much detail either.
I'll be looking into tweaking it to remove the little aliasing still left.

[osgZach]

I left out a letter - It's near the end of AnimeIVTC's script file
if you've installed AnimeIVTC you should be able to call the function directly.

You may need to change AnimeIVTC from a *.avs to a *.avsi extension.

Alternatively open it in notepad and search for "sharpaamcmod"

You can then copy the function definition and paste it into a new blank text file and save it as a *.avsi file so you can call it.

You still should look into using an overrides file to fix those bad field matches though. They stand out like a sore thumb. I'd really like to play with a sample clip of that section (and any others) though just to be sure and see if I can't come up with something better for you.

[Motenai Yoda]

Yeah, R1.
I mean R1 DVD or any BD (a lot of bd look poorly even in r2j version too)

Indeed. I also own a PAL version of the DVDs but I never looked at them closely on the computer. Maybe I'll take a look at them just in case they weren't re-encoded from the NTSC DVDs but taken directly from the source. Probably it'll be pointless though.
Usually pal will be even worst, some of them derive from r1.

[Eud]

OP - any chance of providing a source clip?
http://www.mediafire.com/download/juaclzajkrxnc8a/LE1.m2ts
http://www.mediafire.com/download/so7031vrkoj2yoh/LE2.m2ts

Notice the "things" on the sides of the ship in the fourth cut of the second clip. That's just horrible.

[osgZach]

Initial Impression: looks butchered to me. Like Chi mentioned earlier some of these are likely artifacts from all the screwed up processing they did and I wouldn't know how/if its possible to fix some of those particular things. Some of them DO look like bad field matches (i.e proceeding frame is a dupe of current frame, but is cleaner than current frame) but other than that I'm probably out of my element.

I will see if I can't help with the aliasing when I have more free time to look at it.

edit:
So far I seem to get the best results by calling SharpAAMCMod() after the Spline64Resize()

It cleans up a lot of the shimmering, but not 100%, but in some cases like just plain bad image or artifacts already present, or bad field matches, it won't help too much. If I owned these BD's and wanted to do this, I'd either resolve myself to do it by hand through YATTA, or not do it at all and rely on playback deinterlacing, etc. Personally there are far too many trouble spots for me to do it manually by hand without some centralized visual aid (YATTA)

[osgZach]

Also, not 100% sure, but I think this source might have field blending in it.