Sharpening problem, blending problem

[pablobhz]

Hello.
I'm tryng to encode a video, but i'm having some bizarre issues.
My source, is the japanese version of Lucky Star DVD.
I'm using some simple things, in order to make a nice video.
But i'm having some sharpening issues, and some blending issues.
Look this picture:
(http://img143.imageshack.us/my.php?image=notsharpenedyz8.png)

It looks normal yep. But you can see some sharpening problems, of course. Now, look at this picture with a 300% zoom:
[IMG]http://img404.imageshack.us/img404/4373/problempngcn1.th.png (http://img404.imageshack.us/my.php?image=problempngcn1.png)
You can see with no problems some borders around the hair, the sharpening problems. Ok, its a picture with zoom, but i used to encode the video in h264, using CQ18 leaving those problems, it was horrible.
About the blending, you can't see it...but i'm sure there will be certain scenes where it will be detected. Also, i'm a raw provider, i'm sure that fansubs who download this video will detect those problems.

Well, if anyone needs a video sample (sample from dvd), just tell me. Thanks for any help, and if there's any other information i cna provide just tell me.
P.s: I'm a beginner on using avisynth.


LoadPlugin("D:\AVSPlugins\TIVTC\TIVTC.dll")
LoadPlugin("D:\AVSPlugins\TDeint\tdeint.dll")
LoadPlugin("D:\AVSPlugins\EEDI2\EEDI2.dll")
LoadPlugin("D:\AVSPlugins\DGIndex\DGDecode.dll")
LoadPlugin("D:\AVSPlugins\ColorMatrix2\ColorMatrix\ColorMatrix.dll")
LoadPlugin("D:\AVSPlugins\FFT3DFilter\FFT3DFilter.dll")
#MaskTools 1.5.8
LoadPlugin("D:\AVSPlugins\MaskTools\mt_masktools.dll")
#MaskTools 2.0
LoadPlugin("D:\AVSPlugins\MaskTools2\mt_Masktools.dll")
LoadPlugin("D:\AVSPlugins\WarpSharp\WarpSharp.dll")
LoadPlugin("D:\AVSPlugins\RemoveGrain\RemoveGrain.dll")
Import("D:\AVSPlugins\VMToon\vmToon-v0.74.avsi")
Import("D:\AVSPlugins\LimitedSharpenFaster\LimitedSharpenFaster.avs")
MPEG2Source("D:\KABA_2707_DVD\VIDEO_TS\Lucky Star EP13.d2v")
#TDeint()
interp = separatefields().selecteven().EEDI2(field=1)
deinted = tdeint(edeint=interp,order=1,field=1)
tfm(clip2 = deinted,order=1)
LimitedSharpenFaster(strength=255)
#Fft3dfilter()
#vmToon()
ColorMatrix()
Crop(2, 4, -2, -0)

Yeah, i have many "LoadPlugin". Its because if i use one folder for each plugin =]

[Dark Shikari]

Also, i'm a raw provider, i'm sure that fansubs who download this video will detect those problems.While we all love Lucky Star, everyone's favorite 24-minute Haruhi advertisement... Rule 6 is still in effect.

[DarkT]

Hmmm, AFAIK, rule #6 concerns how the source was OBTAINED, not what will happen with the source after it was obtained, no? His source is a dvd, and nothing(as far as I've read) indicates that the source was obtained illegaly.

Anyway, If I'm wrong, please delete my message guys, or soemthing, but according to the rules, I don't really see a problem...

AT ANY RATE, Pablo, what's it called is "halos" I belive, it's due to over-sharpening.

What you want is something like...DeHalo_alpha, by didee I think, don't remember... But also try to see if perhapes you aren't oversharpening it?

Instead of:
LimitedSharpenFaster(strength=255)
Try:
LimitedSharpenFaster(smode=4, strength=120) - or something like that...

You might also want to apply anti-aliasing to it... Say, something like this?

AssumeTFF()
SeparateFields()
merge(SelectEven().nnEDI(dh=true,field=1),SelectOdd().nnEDI(dh=true,field=0),0.5)

And you MIGHT want to drop eedi2 in favour of nnedi... As well as tweak tfm a bit more... Meow...

Meow, at any rate...

[Dark Shikari]

Hmmm, AFAIK, rule #6 concerns how the source was OBTAINED, not what will happen with the source after it was obtained, no? His source is a dvd, and nothing(as far as I've read) indicates that the source was obtained illegaly.

Anyway, If I'm wrong, please delete my message guys, or soemthing, but according to the rules, I don't really see a problem...I don't care personally, just edit it out of the post.

You could look at a script someone posted here a bit ago--he was testing various clean-up filters for an extremely low-quality anime source to sharpen and clean the lines and flat areas. For Lucky Star, you won't need something nearly as strong, but given the very clear flat areas throughout Lucky Star, such a filter is likely to be successful.

[pablobhz]

Hmmm, AFAIK, rule #6 concerns how the source was OBTAINED, not what will happen with the source after it was obtained, no? His source is a dvd, and nothing(as far as I've read) indicates that the source was obtained illegaly.

Anyway, If I'm wrong, please delete my message guys, or soemthing, but according to the rules, I don't really see a problem...

AT ANY RATE, Pablo, what's it called is "halos" I belive, it's due to over-sharpening.

What you want is something like...DeHalo_alpha, by didee I think, don't remember... But also try to see if perhapes you aren't oversharpening it?

Instead of:
LimitedSharpenFaster(strength=255)
Try:
LimitedSharpenFaster(smode=4, strength=120) - or something like that...

You might also want to apply anti-aliasing to it... Say, something like this?

AssumeTFF()
SeparateFields()
merge(SelectEven().nnEDI(dh=true,field=1),SelectOdd().nnEDI(dh=true,field=0),0.5)

And you MIGHT want to drop eedi2 in favour of nnedi... As well as tweak tfm a bit more... Meow...

Meow, at any rate...
Hey guys.
Changed to nnEDI, take a look on what happened =P
http://img90.imageshack.us/img90/7277/blendpm4.th.png (http://img90.imageshack.us/my.php?image=blendpm4.png)

Maybe make a VFR with it would solve the problem ?
btw, look with a 300% zoom. Things got a bit better, but i don't think its perfect... maybe there's something more ?
http://img147.imageshack.us/img147/1343/changedow6.th.jpg (http://img147.imageshack.us/my.php?image=changedow6.jpg)

BTW, i'm using a Forced Film d2v. But, i couldn't see much difference from 24fps to 29fps. Same problems.

Thanks for the help.
Sorry for asking this, but...where can i dl dehalo alpha ? thx ^^

[kumi]

DeHalo_Alpha (for old MaskTools):
http://forum.doom9.org/showpost.php?p=738264&postcount=43

DeHalo_Alpha (for MaskTools 2.0):
http://forum.doom9.org/showpost.php?p=777956&postcount=70

From the looks of those screenshots, you have "double-haloing"...

[pablobhz]

DeHalo_Alpha (for old MaskTools):
http://forum.doom9.org/showpost.php?p=738264&postcount=43

DeHalo_Alpha (for MaskTools 2.0):
http://forum.doom9.org/showpost.php?p=777956&postcount=70

From the looks of those screenshots, you have "double-haloing"...

Thanks dude.
So, i'm supposed to dehalo_alpha().dehalo_alpha() ?
I've added dehalo to the script. Take a look on the result:
http://img152.imageshack.us/img152/3483/0031tk6.th.jpg (http://img152.imageshack.us/my.php?image=0031tk6.jpg)
300% zoom. Got better, also. I can see that, if you don't consider the fact that with 300% zoom there always is some sharpening, you can see its only the default sharpening that is showed, not the one from bad encode. Or i'm wrong and there should be no sharpening ? I'm using dehalo alpha with some parameters:
dehalo_alpha(brightstr=1,darkstr=.1,highsens=80)
I took it from a post around Doom9.
oh, i decided to change the d2v. Now i'm using one with the option "Honor pulldown flags".


More screenshots:
http://img293.imageshack.us/img293/5598/0032yg9.th.png (http://img293.imageshack.us/my.php?image=0032yg9.png)

Oh god, there's still some sharpening. Look konata's hair. If there's no more sharpening sorry it may be a problem with my eyes =P

http://img178.imageshack.us/img178/9889/0033ht2.th.png (http://img178.imageshack.us/my.php?image=0033ht2.png)

But, this one looks like its nice. For now, i plan go and encode it, to see final results. Prolly i'll encode a piece of 1 min with no audio ^^

Latest pic:
http://img185.imageshack.us/img185/8378/0034an5.th.png (http://img185.imageshack.us/my.php?image=0034an5.png)

And, my script now:

LoadPlugin("D:\AVSPlugins\TIVTC\TIVTC.dll")
LoadPlugin("D:\AVSPlugins\TDeint\tdeint.dll")
LoadPlugin("D:\AVSPlugins\EEDI2\EEDI2.dll")
LoadPlugin("D:\AVSPlugins\DGIndex\DGDecode.dll")
LoadPlugin("D:\AVSPlugins\ColorMatrix2\ColorMatrix\ColorMatrix.dll")
LoadPlugin("D:\AVSPlugins\FFT3DFilter\FFT3DFilter.dll")
#MaskTools 1.5.8
LoadPlugin("D:\AVSPlugins\MaskTools\mt_masktools.dll")
#MaskTools 2.0
LoadPlugin("D:\AVSPlugins\MaskTools2\mt_Masktools.dll")
LoadPlugin("D:\AVSPlugins\WarpSharp\WarpSharp.dll")
LoadPlugin("D:\AVSPlugins\RemoveGrain\RemoveGrain.dll")
LoadPlugin("D:\AVSPlugins\nnedi\nnedi.dll")
Import("D:\AVSPlugins\LimitedSharpenFaster\LimitedSharpenFaster.avs")
Import("D:\AVSPlugins\VMToon\vmToon-v0.74.avsi")
Import("D:\AVSPlugins\DehaloAlpha\DeHalo_alpha.avsi")
MPEG2Source("D:\KABA_2707_DVD\VIDEO_TS\Lucky Star DVD07 - Not Forced.d2v",cpu=4,info=3)
ColorMatrix(hints=true,interlaced=true)
AssumeTFF()
SeparateFields()
merge(SelectEven().nnEDI(dh=true,field=1),SelectOdd().nnEDI(dh=true,field=0),1.0)
#interp = separatefields().selecteven().EEDI2(field=1)
#deinted = tdeint(edeint=interp,order=1,field=1)
#tfm(clip2 = deinted,order=1)
LimitedSharpenFaster()
dehalo_alpha(brightstr=1,darkstr=.1,highsens=80)
FFT3DFilter()
Crop(2, 4, -2, -0)
Lanczos4Resize(720,480)

[kumi]

I'm not sure what you should do in the case of double halos. Maybe you should adjust rx/ry and increase darkstr; that might be the reason why the dark "double halo" in this (http://img185.imageshack.us/img185/8378/0034an5.png) picture is still present (for example, look at the vertical edge of the yellow hair ribbon.)

Also, from those last three screenshots, the dark edges look really discontinuous. I doubt that's in the source... are you sure you're not doing something wrong? Maybe you could upload a short, unprocessed clip for us to play around with.

[pablobhz]

I'm not sure what you should do in the case of double halos. Maybe you should adjust rx/ry and increase darkstr; that might be the reason why the dark "double halo" in this (http://img185.imageshack.us/img185/8378/0034an5.png) picture is still present (for example, look at the vertical edge of the yellow hair ribbon.)

Also, from those last three screenshots, the dark edges look really discontinuous. I doubt that's in the source... are you sure you're not doing something wrong? Maybe you could upload a short, unprocessed clip for us to play around with.

Well dude.
dehalo_alpha(rx=1.0,ry=2.0,brightstr=1.2,darkstr=1.0,highsens=60,ss=2.0)
There it is. Look my results:

http://img20.imageshack.us/img20/8397/doom901gi4.th.png (http://img20.imageshack.us/my.php?image=doom901gi4.png)

Screenshot 01. Seems to be very nice

http://img178.imageshack.us/img178/2245/doom902xj0.th.png (http://img178.imageshack.us/my.php?image=doom902xj0.png)

Screenshot 02. Seems to be nice too, no problems...
But screenshot 03...

http://img178.imageshack.us/img178/9125/doom903op5.th.png (http://img178.imageshack.us/my.php?image=doom903op5.png)

Damn dude. Looks like everyone's face has problems.
I'm getting CRAZY /o\
Im freankin' testing rx, ry, darkstr...those were the best values i could found.

Also, there's a sample of the video here. Extracted with DGIndex:
http://rapidshare.com/files/80202244/sample.demuxed.m2v.html
70MB Piece.
One minute file. I hope its sufficient.

Thanks for all the help.
Pablo

EDIT: Decided to use another resizer, HybridFuPP(720,480,Resizer=6)
And... even LimitedSharpenFaster can't remove the sharpening that is showed on last screenshots.
Is there another thing stronger than it to remove ? THx

[kumi]

Can you post the script you used to obtain those screenshots?

[pablobhz]

Sure
There it goes:

LoadPlugin("D:\AVSPlugins\TIVTC\TIVTC.dll")
LoadPlugin("D:\AVSPlugins\TDeint\tdeint.dll")
LoadPlugin("D:\AVSPlugins\EEDI2\EEDI2.dll")
LoadPlugin("D:\AVSPlugins\DGIndex\DGDecode.dll")
LoadPlugin("D:\AVSPlugins\ColorMatrix2\ColorMatrix\ColorMatrix.dll")
LoadPlugin("D:\AVSPlugins\FFT3DFilter\FFT3DFilter.dll")
#MaskTools 1.5.8
LoadPlugin("D:\AVSPlugins\MaskTools\mt_masktools.dll")
#MaskTools 2.0
LoadPlugin("D:\AVSPlugins\MaskTools2\mt_Masktools.dll")
LoadPlugin("D:\AVSPlugins\WarpSharp\warpsharp.dll")
LoadPlugin("D:\AVSPlugins\RemoveGrain\RemoveGrain.dll")
LoadPlugin("D:\AVSPlugins\nnedi\nnedi.dll")
Import("D:\AVSPlugins\LimitedSharpenFaster\LimitedSharpenFaster.avs")
Import("D:\AVSPlugins\VMToon\vmToon-v0.74.avsi")
Import("D:\AVSPlugins\DehaloAlpha\DeHalo_alpha.avsi")
Import("D:\AVSPlugins\HybridFuPP\HybridFuPP.avsi")
LoadPlugin("D:\AVSPlugins\Deen\Deen.dll")
LoadPlugin("D:\AVSPlugins\HybridFuPP\unfilter_25_dll_20030116\UnFilter_Avisynth_2.5_Alpha\UnFilter.dll")
LoadPlugin("D:\AVSPlugins\RemoveGrain\Repair.dll")
LoadPlugin("D:\AVSPlugins\HybridFuPP\AddGrain\AddGrain.dll")
LoadPlugin("D:\AVSPlugins\HybridFuPP\mvtools-v1.8.5.1\mvtools.dll")
Import("D:\AVSPlugins\Toon081\Toon081.avsi")
LoadPlugin("D:\AVSPlugins\aWarpSharp\aWarpSharp.dll")
MPEG2Source("D:\KABA_2707_DVD\VIDEO_TS\Lucky Star DVD07 - Not Forced.d2v",cpu=4,info=3)
#ColorMatrix(hints=true,interlaced=true)
AssumeTFF()
SeparateFields()
merge(SelectEven().nnEDI(dh=true,field=1),SelectOdd().nnEDI(dh=true,field=0),1.0)
dehalo_alpha(rx=1.0,ry=2.0,brightstr=1.2,darkstr=1.0,highsens=60,ss=2.0)
LimitedSharpenFaster()
Crop(2, 4, -2, -0)
HybridFuPP(720,480,Resizer=6)

There he is. Now i'm taking a look on Toon 0.81, and VmToon()

[pablobhz]

Sorry for double posting.
Well, this is the script i've decided to use:

LoadPlugin("D:\AVSPlugins\ws\warpsharp.dll")
LoadPlugin("D:\AVSPlugins\TIVTC\TIVTC.dll")
LoadPlugin("D:\AVSPlugins\TDeint\tdeint.dll")
LoadPlugin("D:\AVSPlugins\DGIndex\DGDecode.dll")
LoadPlugin("D:\AVSPlugins\ColorMatrix2\ColorMatrix\ColorMatrix.dll")
LoadPlugin("D:\AVSPlugins\FFT3DFilter\FFT3DFilter.dll")
#MaskTools 1.5.8
LoadPlugin("D:\AVSPlugins\MaskTools\mt_masktools.dll")
#MaskTools 2.0
LoadPlugin("D:\AVSPlugins\MaskTools2\mt_Masktools.dll")
LoadPlugin("D:\AVSPlugins\RemoveGrain\RemoveGrain.dll")
LoadPlugin("D:\AVSPlugins\nnedi\nnedi.dll")
Import("D:\AVSPlugins\LimitedSharpenFaster\LimitedSharpenFaster.avs")
Import("D:\AVSPlugins\VMToon\vmToon-v0.74.avsi")
Import("D:\AVSPlugins\DehaloAlpha\DeHalo_alpha.avsi")
Import("D:\AVSPlugins\HybridFuPP\HybridFuPP.avsi")
LoadPlugin("D:\AVSPlugins\Deen\Deen.dll")
LoadPlugin("D:\AVSPlugins\HybridFuPP\unfilter_25_dll_20030116\UnFilter_Avisynth_2.5_Alpha\UnFilter.dll")
LoadPlugin("D:\AVSPlugins\RemoveGrain\Repair.dll")
LoadPlugin("D:\AVSPlugins\HybridFuPP\AddGrain\AddGrain.dll")
LoadPlugin("D:\AVSPlugins\HybridFuPP\mvtools-v1.8.5.1\mvtools.dll")
LoadPlugin("D:\AVSPlugins\aWarpSharp\aWarpSharp.dll")
Import("D:\AVSPlugins\FastLineDarkenMOD\FastLineDarkenMOD.avsi")
MPEG2Source("D:\KABA_2707_DVD\VIDEO_TS\Lucky Star DVD07 - Not Forced.d2v",cpu=4,info=3)
ColorMatrix(hints=true,interlaced=true)
AssumeTFF()
SeparateFields()
merge(SelectEven().nnEDI(dh=true,field=1),SelectOdd().nnEDI(dh=true,field=0),1.0)
dehalo_alpha(rx=1.0,ry=2.0,brightstr=1.2,darkstr=1.0,highsens=60,ss=2.0)
LimitedSharpenFaster()
vmtoon(sharpen=false,thinning=0,strength=50,luma_cap=203,threshold=8)
#FFT3DFilter()
FastLineDarkenMOD()
Crop(2, 4, -2, -0)
HybridFuPP(720,480,Resizer=6)
#vmtoon(strength=35,luma_cap=203,sharpen=false,thinning=0)



A sample picture from it:
http://img297.imageshack.us/img297/1470/verdeyw2.th.png (http://img297.imageshack.us/my.php?image=verdeyw2.png)

Yep, i've saw the problem on the green hair, i fixed it.
btw its the best result i could reach...
Any suggestions, i would thank a lot ^^

[kumi]

OK, well, right off the bat your merge() call is the wrong thing to use. It seems to be standard 3:2 pulldowned stuff. You can tell by running this script, which show a pattern of 3:2 field repetitions:

assumetff().separatefields()


So you need to IVTC it, like this:

loadplugin("c:\program files\dgmpgdec\dgdecode.dll")
loadplugin("C:\Program Files\AviSynth 2.5\plugins\TIVTC\TIVTC.dll")
mpeg2source("g:\temp\sample.demuxed.d2v", idct=6) # Use "Honor Pulldown Flags" when creating .d2v
assumetff()
tfm(d2v="g:\temp\sample.demuxed.d2v").tdecimate(mode=1)


That yields a clean 23.976 FPS progressive clip.

[pablobhz]

Well
There it is, the final result:
http://img212.imageshack.us/img212/8766/perfectoz6.th.png (http://img212.imageshack.us/my.php?image=perfectoz6.png)

Screenshot 01. Seems to be no problems.

http://img181.imageshack.us/img181/608/perfect2oi9.th.png (http://img181.imageshack.us/my.php?image=perfect2oi9.png)

Screenshot 02. Am i seeing halo, or i should go sleep =P

http://img180.imageshack.us/img180/969/perfect3as8.th.png (http://img180.imageshack.us/my.php?image=perfect3as8.png)
Screenshot03. Well, i think its nice ^^

Here goes the script...
I'm gonna post only what changed, in order to don't make a longer page.

MPEG2Source("D:\KABA_2707_DVD\VIDEO_TS\Lucky Star DVD07 - Not Forced.d2v",cpu=4,info=3)
ColorMatrix(hints=true,interlaced=true)
tfm(d2v="D:\KABA_2707_DVD\VIDEO_TS\Lucky Star DVD07 - Not Forced.d2v").tedecimate(mode=1)
dehalo_alpha(rx=1.0,ry=2.0,brightstr=1.2,darkstr=1.0,highsens=60,ss=2.0)
LimitedSharpenFaster()
vmtoon(sharpen=false,thinning=0,strength=50,luma_cap=203,threshold=8)
FastLineDarkenMOD() // Tryng to find out about him =]
Crop(2, 4, -2, -0)
HybridFuPP(720,480,Resizer=6)

Well, there it is.
Thanks for your help dude.

EDIT:
Changed the script. Fixed some gradient problems, and another color issues.
At least now it looks ok. Also, decided to don't use vmtoon, since i was losing some details.
The halo i had, i checked with a zoom'ed picture, and it was not present anymore.

Thanks for all help guys
http://img265.imageshack.us/img265/1302/post04do7.th.png (http://img265.imageshack.us/my.php?image=post04do7.png)
Screen01

http://img227.imageshack.us/img227/9048/post01pq6.th.png (http://img227.imageshack.us/my.php?image=post01pq6.png)
Screen02

http://img245.imageshack.us/img245/333/post02xm2.th.png (http://img245.imageshack.us/my.php?image=post02xm2.png)
Screen03

http://img245.imageshack.us/img245/7908/post03ql1.th.png (http://img245.imageshack.us/my.php?image=post03ql1.png)
Screen04

Again, thanks for everything
And sorry for any incovenience
Happy new year & stuff for you all
From Brazil =)

Pablo