Antialiasing, Vertical Upscaling, and Sharpening

[Avisynth_challenged]

Hi all,

I'm looking to finish up a laserdisc conversion project I've been working on throughout the year. Thanks to Avisynth and this awesome forum, I've achieved some amazing results. So thanks to everyone.

I need to perform the following tasks to the video for encoding it to 16:9 MPEG2:

1. Antialias
2. Vertical upscaling from 720x300 to 720x378
3. Sharpening
4. Final checking, and tweaking as necessary, of brightness, contrast, gamma, and saturation
5. Addborders(0,51,0,51)
6. Colorspace conversion from YV12 to YUY2


With respect to the first 3 items, I'm looking at doing the following:

1. Antialias: I'm experimenting with the script that was recommended to me in this post (http://forum.doom9.org/showthread.php?p=1208276#post1208276). I've obtained good results with some initial testing, but since this script originally came from Didée (http://forum.doom9.org/showthread.php?p=553304#post553304) that's hardly surprising :)

2. Vertical upscaling from 720x300 to 720x378: I'm not sure if I'll use Lanczos or Spline for vertical upscaling. Is there a consensus as to which method is better? Or, alternatively, would it be better to upscale and sharpen simultaneously using LimitedSharpen?

3. Sharpening: I'm definitely going to try LimitedSharpen first. If maximum quality is desired, then is the original LimitedSharpen (LS) script better than LimitedSharpenFaster (LSF), which,as I understand it, was designed for real-time video postprocessing during playback? And...as previously mentioned in #2 above...should I consider using LS for upscaling and sharpening at the same time? Hmmmm, WWDD (What Would Didée Do?)

All the above assumes that Antialias comes before Vertical Upscaling, and Vertical Upscaling comes before Sharpening. Does this order seem reasonable?

Thanks for reading and for any comments and feedback.

[Sagekilla]

1) I would go with Spline16, Spline is about as sharp as Lanczos but exhibits less ringing artifacts. I did some testing and Lanczos tends to be more bitrate hungry than Spline, while offering more or less the same quality. IIRC, sampling in LSF is done only to help sharpening a bit. You're better off doing ressampling outside.

2) I don't think LS has any real quality advantage over LSF right now. The latest LSF is faster and better than LS I believe.

3) Yes, I would go with AA --> SS --> Sharp

4) Why the conversion to YUY2? Does your encoder require YUY2 input?

[Undead Sega]

Would this happen to be for a Star Wars project may i ask?

[kemuri-_9]

1) I would go with Spline16, Spline is about as sharp as Lanczos but exhibits less ringing artifacts.

AVS 2.5.6+ also has Spline36, and then AVS 2.5.8 has Spline64 and Blackman (the latter of which is lanczos altered to reduce ringing)
so there's always plenty of resizer options

[Sagekilla]

Indeed there are, but generally speaking, in terms of bitrate required:

Lanczos < Lanczos4
Spline 16 < Spline36 < Spline64

It's all dependent on the user too: If you think one resizer is too soft, try the next grade up, but be aware that they're more bitrate hungry. I've noticed a variance of 200 - 300 kbps in CRF mode for x264 depending on which resizer is used, which is primarily why I use Spline16 over any of the "Sharper" methods.

[kemuri-_9]

that's natural based on the concept of sharpening.
frames with higher levels of sharpening will consume more bitrate than ones with less.

[mikeytown2]

I pick my Resize depending on the content/target. I prefer Spline16/Bilinear for web deployment, Spline36 for animated content and Blackman/Spline36 (flip a coin) as my general one; general used to be Lanczos. If doing something nuts like a 5x zoom-in then I'll go for Spline64. For more info on resizing ImageMagick has a great page on the subject.
http://www.imagemagick.org/Usage/resize/

As Sagekilla said, the sharper you go, the more bits it takes.

[Avisynth_challenged]

Wow, thanks for all the responses.


4) Why the conversion to YUY2? Does your encoder require YUY2 input?

Yes, I'm using CCE Basic 2.70


Would this happen to be for a Star Wars project may i ask?

No, but I do appreciate those who labored over and exchanged ideas for the many Star Wars laserdisc conversion projects that happened during the last 6 years or so. I obtained many ideas from Star Wars related discussions. True pioneers, they are :)


Okay, here are two screenshots, basically "before" and "after" for a particular script, which follows the images:


Source:
http://i252.photobucket.com/albums/hh18/FoolsTalk/source01.png


Result:
http://i252.photobucket.com/albums/hh18/FoolsTalk/result01.png


Script:
################
Import("C:\program files\avisynth 2.5\plugins\limitedsharpenfaster.avs")
Import("C:\program files\avisynth 2.5\plugins\seesaw.avs")


# This line loads the ffdshow decoded source
source=DirectshowSource("c:\...\source.avi").addborders(0,1,0,1)


function antialiasing( clip orig, int "th_luma", int "th_croma", string "type", int "order", int "aath", bool "HQedge")
{

# "th_luma" and "th_croma" affect directly the edge detection: higher values = more edges filtered
# set "order = 0" for Top Field First; order = 1 --> Bottom Field First
# aath = anti-aliasing strenght (default should be fine)


th_luma = Default(th_luma, 20)
th_croma = Default(th_croma, 20)
type = Default(type, "sobel")
order = Default(order, 1)
aath = Default(aath, 48)
HQedge = Default(HQedge, false)
ox = orig.width
oy = orig.height
dx = orig.width * 2
dy = orig.height * 2

orig.convertToYV12()
a=last
b=lanczos4Resize(dx, dy).TurnLeft().SangNom(order,aath).TurnRight().SangNom(order,aath)

# native chroma edges:
#c=b.lanczosresize(ox,oy).EdgeMask(0, th_luma, 0, th_croma, type,Y=3,U=3,V=3)

# use luma edges for chroma:
c=b.lanczosresize(ox,oy).EdgeMask(0, th_luma, 0, th_croma, type,Y=3,U=1,V=1).FitY2UV()

d= logic( b.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=1,V=1)
\ ,b.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=1,V=1)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false).FitY2UV().lanczosresize(ox,oy)
b=b.lanczosresize(ox,oy)

HQedge ? MaskedMerge(a,b,d,Y=3,U=3,V=3) : MaskedMerge(a,b,c,Y=3,U=3,V=3)
}

a=source.antialiasing(th_luma=20, th_croma=20, type="sobel", order=1, aath=18, HQedge=false)

final1=a.limitedsharpenfaster(strength=40).spline36resize(720,378).limitedsharpenfaster(strength=100).Levels(0,0.95,255,0,255).tweak(0,1.6,0,1.0,false,false).addborders(0,51,0,51)
return final1
################

What does everyone think? And thanks again for all the responses. :thanks: :thanks: :thanks:

[Didée]

IMO a purely spatial AA approach is not fully sufficient here. In the posted sample, look at the railing (in the background, above the stairs). They are flickering quite nervous in the source, and after applying "antialiasing()", they are still flickering quite nervous.
Seems some temporal filtering would be indicated.


The source is full of ringing. When sharpening without caring for that, the present artifacts get pronounced.

Related, the usual lanczos* or spline* resizers introduce ringing in each resizing step. Now, when you do

[orig.size] .LSF() .upsize() .LSF() ,

realize that this chain does

[orig.size] .upsize().sharp().downsize() .upsize() .upsize().sharp().downsize()


That's a total of *five* resizing steps, which is *lots* of opportunity to emphasize existing ringing in the source ...


Looking closer at finer detail, the usual look of "upscaled+sharpened" becomes obvious. E.g. in the screenshot, the wrinkles on the man's forehead lost much of their "3D" appearance, and have turned into more like flat ribbons.
The "basic characteristic" of this is introduced by the E.D.I. scaling (here: SangNom, EEDI2 would be similar in this respect, while NNEDI is slower, but nicer), then is solidified by LSF.
Hint: edge's gradient is only one side of the coin. Gradient's magnitude is the other side. Many of the popular sharpening practices care too much for the former, and too little for the latter...


Quite some problems to solve. The fact that the original capturing was set up too sharp doesn't make things easier.

[Avisynth_challenged]

I've finally managed to lure Didée into the discussion, and boy, there's a lot of new information from him to consider...



IMO a purely spatial AA approach is not fully sufficient here. In the posted sample, look at the railing (in the background, above the stairs). They are flickering quite nervous in the source, and after applying "antialiasing()", they are still flickering quite nervous.
Seems some temporal filtering would be indicated.


Are you aware of any antialiasing functions that operate temporally? I scoured the forum with search terms like antialias(ing), jaggies, alias(ing), etc. and could only find that one antialiasing function that appears in the script I posted earlier.



The source is full of ringing. When sharpening without caring for that, the present artifacts get pronounced.

Related, the usual lanczos* or spline* resizers introduce ringing in each resizing step. Now, when you do

[orig.size] .LSF() .upsize() .LSF() ,

realize that this chain does

[orig.size] .upsize().sharp().downsize() .upsize() .upsize().sharp().downsize()


That's a total of *five* resizing steps, which is *lots* of opportunity to emphasize existing ringing in the source ...


I've been plagued by the ringing problem throughout this project. Intuitively, I know it is being exacerbated by all the resize/sharpening steps. My plan, as of this time, is to do all the resize and sharpening to reveal all the ringing, then to attempt de-ringing afterwards. Is this plan reasonable?



Looking closer at finer detail, the usual look of "upscaled+sharpened" becomes obvious. E.g. in the screenshot, the wrinkles on the man's forehead lost much of their "3D" appearance, and have turned into more like flat ribbons.
The "basic characteristic" of this is introduced by the E.D.I. scaling (here: SangNom, EEDI2 would be similar in this respect, while NNEDI is slower, but nicer), then is solidified by LSF.
Hint: edge's gradient is only one side of the coin. Gradient's magnitude is the other side. Many of the popular sharpening practices care too much for the former, and too little for the latter...


From these statements, I'm concluding that I should experiment with the antialiasing script, so that it uses NNEDI instead of SangNom, or to skip antialiasing altogether (to avoid the obvious "upscaled + sharpened" look).



Quite some problems to solve. The fact that the original capturing was set up too sharp doesn't make things easier.

Today, I find that what I want to hear and what I need to hear are two different things. Didée, thanks for responding. Please do feel free to continue telling me what I need to hear. :)

[thetoof]

You can have fun trying SharpAAMC(mod). The original can be found in the "fine anime antialiasing" thread and the other within AnimeIVTC. I might add a option to use nnedi for aa.

Some denoising (dfttest, mvdegrainx...) might help deringing, though I don't have a lot of experience dealing with that.

[martino]

at_denoise helps deringing a lot, but I think you might want to take a look at aWarpSharp too for that purpose.

[Sagekilla]

I suppose for the flickering portions, you could mask out that portion of the image and apply temporal averaging to it, like MVDegrain or any one of the various filters available (http://avisynth.org/mediawiki/External_filters#Temporal_Denoisers).

I'm not quite sure how to mask out those portions, but I suppose you could look for areas with huge differences between two frames. The only downside with this approach would be that scene changes would be detected too and would be filtered as well.