Anime upsizer and shadowing? [was: "...and ghosting?"]

CAFxX · 10th July 2006, 15:58

Hi all, I am currently trying to produce a 720p version of Mononoke Hime out of my 576p DVD.
The script I'm currently using is producing quite an impressive output, but has a few side effects I'd like to discuss with you in order to get rid of them.
Enough with the chat, let's have a look at the screenshots:
as a reference, this is a straight 576->720 conversion using lanczos

and this is a 576->720 conversion using my script

the two main problems are
1) that black border on the left of the frame that i can't figure out why does it appear
2) the dark haloes around black lines. they can be found also in the original version so it's not really a "side effect", but in my version they are more annoying since the lines are thinner. any idea on how to get rid of these? I thought about the possibility to mask out the haloed zone, interpolating the chroma channels of the masked zone, then running awarpsharp on those zones and finally merge back the obtained chroma channels on the "edges" clip (see the code below) but it doesn't seem to work.

this is the script i'm using. it's annoyingly slow, i know, but speed is not a concern to me: i'm doing this for quality.
all code is written from scratch based partially on ideas found on the forum (e.g. warpsharpresize).

Code:

function WarpSharpResize(src, width, height, passes) {
	d_w = Float(width - src.width) / Float(passes)
	d_h = Float(height - src.height) / Float(passes)
	return WarpSharpResizer(src, d_w, d_h, passes, Float(src.width), Float(src.height))
}

# this is the recursive internal function - you should not call this (at least, there's no need to do so)
function WarpSharpResizer(src, d_w, d_h, passes, c_w, c_h) {
	p_w = Round(c_w + d_w) - (Round(c_w + d_w) % 4) # width should always be mod 4 in YV12
	p_h = Round(c_h + d_h) - (Round(c_h + d_h) % 2) # height should always be mod 2 in YV12
	return passes > 0 ? WarpSharpResizer(LanczosResize(src, p_w, p_h).aWarpSharp(5, 2), d_w, d_h, passes-1, c_w + d_w, c_h + d_h) : src
}

function CAFtoon(clip src) {
	base = spline36resize(src, src.width*4, src.height*4)

	edges = WarpSharpResize(src, src.width*4, src.height*4, 8)

	lines = generalconvolution(edges.converttorgb32(), 0, "1 2 1 2 -12 2 1 2 1").converttoyv12().greyscale().blur(0.5).tweak(bright=-60, cont=3).blur(0.5)

	edgesmask = generalconvolution(src.converttorgb32(), 0, "1 2 1 2 -12 2 1 2 1").converttoyv12().greyscale().blur(1).spline36resize(src.width*4, src.height*4).levels(10, 1, 50, 0, 255)

	out = overlay(overlay(base, edges, mask=edgesmask), lines, mode="subtract")

	#return lines
	#return edges
	#return edgesmask
	return out
}

setMemoryMax(256)
DirectShowSource("dvd\MH.m2v", fps=25)
AssumeFPS(24)
crop(10, 8, -8, -14)
CAFtoon()
tweak(bright=12)
LanczosResize(1280, 720)

final note: the source is the italian PAL DVD. the frame in the screenshot is number 80259 (text-less opening program chain)

check · 10th July 2006, 16:34

EDIT: original post was using unresized imageshack image. I've rewritten the post to actually be relevant and not embarrassing to me

Well, that looks very sharp, in my opinion perhaps oversharp. No, i'm not trying to salvage my previous position, I feel that the lines are far too clinical and artificial for my liking - definately not what the director was wanting you to see. If you absolutely insist on such sharp lines, I'd throw in a slight luma blur to somewhat offset it.

Other than that, I think you will also need some denoising, although the reasonably strong warpsharp removes much of the noise, the face still has lots of noise on it, you'll need to add a denoiser at the end of your script. The shadowing will be hard to clean up, a strong spatial denoiser is probably your best shot, but watch you don't blur the background. Temporal blurring will be helpful if there is any grain, I'd suggest you try hqdn3d and the rest of the usual suspects.

CAFxX · 10th July 2006, 16:49

And btw, as far as bitrate is concerned: my target is a DVD5. The film is about 140 minutes. This means about 4.5 mbit/s.
4 mbit/s for 720-24p animated content with H264 should be enough (my guess, i may be wrong)
In the case it ain't enough, I would go for a DVD9. That should definitely be enough.

check, btw "Western Australia" where? My sister will spend the next five months there.

CAFxX · 10th July 2006, 17:56

Quote:

you'll need to add a denoiser at the end of your script

I was actually looking for a good anime denoiser, but I was thinking to place it right after the directshowsource(). Do you say it would be better to have it at the end?
Another problem is that at the moment I'm using directshowsource to let ffdshow apply post-processing to the input video, since in high-motion scenes blocking is clearly visible. Can anybody suggest an alternative to this?

Quote:

a strong spatial denoiser is probably your best shot, but watch you don't blur the background

I designed my script in such a way that the background undergoes a simple splineresize(). no warpsharping, no blurring, nothing.
I'll try hqdn3d on the input and we'll see.

check · 10th July 2006, 18:19

fyi, hqdn3d is the same as the dn3d option in ffdshow with hq checked. I dislike denoising before warpsharping because warpsharp is a bit like a nuke, it changes everything. Personally, I dont like the effect it produces so I don't use it much, but you will definately need a denoiser at the end.

CAFxX · 10th July 2006, 18:19

hqdn3d() -- default
this is way too strong, the bg gets badly blurred

hqdn3d(2, 1.5, 6, 4.5) -- halved spatial strenght
way better, bg still gets (very) slightly blurred, no noticeable side-effects on the efficiency of my filter

*.mp4 guy · 10th July 2006, 18:27

It would be easier to help if you provided an uncompressed source clip, or at the very least an unaltered image.

Awatef · 10th July 2006, 19:48

CAF, what you're doing is a waste of time IMHO.
Upsizing won't create extra details, and when you recompress a DVD you'll be even LOOSING details so...

BigDid · 10th July 2006, 20:40

Quote:

Originally Posted by CAFxX

I was actually looking for a good anime denoiser ...

Hi,

Have you tried Frfun7 ?
Thread here: http://forum.doom9.org/showthread.ph...604#post825604
Some pics here: http://forum.doom9.org/showthread.ph...817#post821817

Did

CAFxX · 10th July 2006, 20:44

Quote:

It would be easier to help if you provided an uncompressed source clip, or at the very least an unaltered image.

Working on that. Just give me the time to pick a few different scenes and to upload them somewhere.

Quote:

CAF, what you're doing is a waste of time IMHO.
Upsizing won't create extra details, and when you recompress a DVD you'll be even LOOSING details so...

Yep, we all know what good ol' Nyquist said.
But you know, if I had to choose which one I like most between those two pictures in my first post, I'd say the second one. And since my screen is 1280x1024 (pretty small I know), my projector supports 720p, I got plenty of bitrate and there's no filter on earth capable of doing that during playback I think I might give it a try.

Trixter · 10th July 2006, 21:57

Quote:

Originally Posted by CAFxX

Yep, we all know what good ol' Nyquist said.
But you know, if I had to choose which one I like most between those two pictures in my first post, I'd say the second one. And since my screen is 1280x1024 (pretty small I know), my projector supports 720p, I got plenty of bitrate and there's no filter on earth capable of doing that during playback I think I might give it a try.

The great irony is that you're actually losing detail by doing this. When you get a filter setting that you think looks fantastic, do it on a PAL test plate (resolution test, chroma test, etc.) and you'll see just how much you're mangling the footage.

I've never understood why people want to "sharpen" some things. You can't create detail that isn't there, and any attempt to do so loses whatever fine detail that is there.

CAFxX · 10th July 2006, 22:28

Quote:

The great irony is that you're actually losing detail by doing this. When you get a filter setting that you think looks fantastic, do it on a PAL test plate (resolution test, chroma test, etc.) and you'll see just how much you're mangling the footage.

Again, I am not trying to sell a filter that does create 1080p razor-sharpen images out of a QCIF. I just asked if someone could help resolving those issues I mentioned in the first post. I do perfectly know I can't raise the resolution by means of (such) a filter; however, since resizing has to be done sooner or later, I thought that maybe I could do something better than plain bicubic.
And, btw, what's the point in saying "try that on a PAL test and you'll see how much your filter sucks". It's not meant for that as much as jpg compression isn't meant to deal with audio. We're not talking about objective quality here (whatever this may or may not mean), but just personal taste. I like it best. Everything else is - I beg your pardon for the term - bullshit, since it's OT.

CAFxX · 10th July 2006, 22:31

Now for those who are still interested in helping: here's the screenshot of frame 80259 in all its 576p beauty. (Yes, the DVD has that black border).
The only difference from the original source is that, being a PNG, the colorspace has been converted to RGB.

Tomorrow I'll post the video. Be patient.
Right now I'm pretty busy watching the huge celebration we're having down in Rome. There are 700000 (and counting) people out there.

Mug Funky · 11th July 2006, 02:18

Quote:

final note: the source is the italian PAL DVD

i might have to grab a copy of this one... looks to be a new transfer (and actually better than the japanese one).

the one i've got at the moment is NTSC->PAL converted and multiangled, so action sequences can look a little off... in spite of about a hundred segment re-encodes that went into it.

CAFxX · 11th July 2006, 07:03

[OT]

Quote:

i might have to grab a copy of this one...

I bought it on a online store for 9.90€, if you're interested.
[/OT]

*.mp4 guy · 11th July 2006, 09:01

Script:

Code:

ImageSource("42jf.png").Converttoyv12(matrix="PC.709")

MultiSWAR(1280, 720, warper=false, StepRatio=8, Sharpen=8, Smoothing=true, Sthresh=8, Sstrength=.9, EDetail=false, NthPass=true)

Result (Q100 JPG)
Result (PNG)

Script (sharper):

Code:

ImageSource("42jf.png").Converttoyv12(matrix="PC.709")

MultiSWAR(1280, 720, warper=false, StepRatio=8, Sharpen=8, Smoothing=true, Sthresh=8, Sstrength=.9, EDetail=true, Esharp=0.5, FDetail=0, NthPass=true)

Result (Q100 JPG)
Result (PNG)

Script (smoother):

Code:

ImageSource("42jf.png").Converttoyv12(matrix="PC.709")

MultiSWAR(1280, 720, warper=false, StepRatio=8, sharper=true, Sharpen=8, Smoothing=true, Sthresh=8, Sstrength=1.2, EDetail=true, Esharp=0.5, FDetail=0, NthPass=true)

(Q100 JPG)
Result (PNG)

Script (smoothest):

Code:

ImageSource("42jf.png").Converttoyv12(matrix="PC.709")

MultiSWAR(1280, 720, warper=true, StepRatio=8, sharper=true, Sharpen=16, Smoothing=true, Sthresh=8, Sstrength=1, EDetail=false, Esharp=0.5, FDetail=0, NthPass=true)

(Q100 JPG)
Result (PNG)

foxyshadis · 11th July 2006, 10:24

You probably can't really get rid of those shadows, because they're present in the original (it looks like a lot of color bleed and a littel luma bleed is the main deal). You might be able to minimize it by first running dehalo_alpha on it, before attempting to sharpen, but that also softens some details.

I wonder how you can live with such aliased results, but to each their own.

CAFxX · 11th July 2006, 10:56

I left my PC doing a little test encode on a different sequence using hqdn3d(2, 1.5, 6, 4.5) and it's definitely not good. The BGs gets devastated. And i think that, if hqdn3d managed to keep at least a good amount of detail on the BG of the screenshot above and produces such a disaster in this other case, MultiSWAR will behave even worse (your first PNG screenshot shows less detail than mine in the BG, and the second shows too much artifacts on the face). Anyhow, I'll give it a try. Who knows, maybe I'm wrong.
Next test will be hqdn3d with spatial denoising disabled.

CAFxX · 11th July 2006, 10:58

Quote:

I wonder how you can live with such aliased results, but to each their own.

Are you talking about my results? In the case, I perfectly know they're pretty far from being (at least) acceptable.

CAFxX · 11th July 2006, 11:41

Tried deactivating spatial denoising and BGs are just little (almost unnoticeably) more blurry than without hqdn3d().
Still, it has problems with moving BGs. Will try to lower the temporal strenght as well (and maybe reactivating sliglthly spatial denoising).
Other anime denoisers? No-one ever attempted to write a filter/function able to identify flat color areas?

BTW, I tried to output at 720x404 and that already looks really great.

EDIT: I'm currently uploading about 2 minutes of the raw mpeg2 stream to my site. It is around 130mb, and will likely be ready to be downlaoded in an hour. http://www.n-side.it/public/CAFxX/MH.mpg. I won't leave it there forever, so fetch it ASAP.

EDIT2: Done uploading.

10th July 2006, 15:58	#1 \| Link
CAFxX Stray Developer Join Date: Mar 2003 Location: Italy Posts: 82	Anime upsizer and shadowing? [was: "...and ghosting?"] Hi all, I am currently trying to produce a 720p version of Mononoke Hime out of my 576p DVD. The script I'm currently using is producing quite an impressive output, but has a few side effects I'd like to discuss with you in order to get rid of them. Enough with the chat, let's have a look at the screenshots: as a reference, this is a straight 576->720 conversion using lanczos and this is a 576->720 conversion using my script the two main problems are 1) that black border on the left of the frame that i can't figure out why does it appear 2) the dark haloes around black lines. they can be found also in the original version so it's not really a "side effect", but in my version they are more annoying since the lines are thinner. any idea on how to get rid of these? I thought about the possibility to mask out the haloed zone, interpolating the chroma channels of the masked zone, then running awarpsharp on those zones and finally merge back the obtained chroma channels on the "edges" clip (see the code below) but it doesn't seem to work. this is the script i'm using. it's annoyingly slow, i know, but speed is not a concern to me: i'm doing this for quality. all code is written from scratch based partially on ideas found on the forum (e.g. warpsharpresize). Code: function WarpSharpResize(src, width, height, passes) { d_w = Float(width - src.width) / Float(passes) d_h = Float(height - src.height) / Float(passes) return WarpSharpResizer(src, d_w, d_h, passes, Float(src.width), Float(src.height)) } # this is the recursive internal function - you should not call this (at least, there's no need to do so) function WarpSharpResizer(src, d_w, d_h, passes, c_w, c_h) { p_w = Round(c_w + d_w) - (Round(c_w + d_w) % 4) # width should always be mod 4 in YV12 p_h = Round(c_h + d_h) - (Round(c_h + d_h) % 2) # height should always be mod 2 in YV12 return passes > 0 ? WarpSharpResizer(LanczosResize(src, p_w, p_h).aWarpSharp(5, 2), d_w, d_h, passes-1, c_w + d_w, c_h + d_h) : src } function CAFtoon(clip src) { base = spline36resize(src, src.width4, src.height4) edges = WarpSharpResize(src, src.width4, src.height4, 8) lines = generalconvolution(edges.converttorgb32(), 0, "1 2 1 2 -12 2 1 2 1").converttoyv12().greyscale().blur(0.5).tweak(bright=-60, cont=3).blur(0.5) edgesmask = generalconvolution(src.converttorgb32(), 0, "1 2 1 2 -12 2 1 2 1").converttoyv12().greyscale().blur(1).spline36resize(src.width4, src.height4).levels(10, 1, 50, 0, 255) out = overlay(overlay(base, edges, mask=edgesmask), lines, mode="subtract") #return lines #return edges #return edgesmask return out } setMemoryMax(256) DirectShowSource("dvd\MH.m2v", fps=25) AssumeFPS(24) crop(10, 8, -8, -14) CAFtoon() tweak(bright=12) LanczosResize(1280, 720) final note: the source is the italian PAL DVD. the frame in the screenshot is number 80259 (text-less opening program chain) __________________ CAFxXcrossway, a collection of my projects CAFxX@strayorange, my blog Last edited by CAFxX; 10th July 2006 at 22:45.

10th July 2006, 16:34	#2 \| Link
check phjbdpcrjlj2sb3h Join Date: Sep 2005 Location: Western Australia Posts: 1,691	EDIT: original post was using unresized imageshack image. I've rewritten the post to actually be relevant and not embarrassing to me Well, that looks very sharp, in my opinion perhaps oversharp. No, i'm not trying to salvage my previous position, I feel that the lines are far too clinical and artificial for my liking - definately not what the director was wanting you to see. If you absolutely insist on such sharp lines, I'd throw in a slight luma blur to somewhat offset it. Other than that, I think you will also need some denoising, although the reasonably strong warpsharp removes much of the noise, the face still has lots of noise on it, you'll need to add a denoiser at the end of your script. The shadowing will be hard to clean up, a strong spatial denoiser is probably your best shot, but watch you don't blur the background. Temporal blurring will be helpful if there is any grain, I'd suggest you try hqdn3d and the rest of the usual suspects. __________________ CCCP \| MeWiki \| techouse's x264 builds \| Kovensky's mplayer builds Last edited by check; 10th July 2006 at 17:19.

10th July 2006, 16:49	#3 \| Link
CAFxX Stray Developer Join Date: Mar 2003 Location: Italy Posts: 82	And btw, as far as bitrate is concerned: my target is a DVD5. The film is about 140 minutes. This means about 4.5 mbit/s. 4 mbit/s for 720-24p animated content with H264 should be enough (my guess, i may be wrong) In the case it ain't enough, I would go for a DVD9. That should definitely be enough. check, btw "Western Australia" where? My sister will spend the next five months there. __________________ CAFxXcrossway, a collection of my projects CAFxX@strayorange, my blog Last edited by CAFxX; 10th July 2006 at 16:59.

10th July 2006, 18:19	#5 \| Link
check phjbdpcrjlj2sb3h Join Date: Sep 2005 Location: Western Australia Posts: 1,691	fyi, hqdn3d is the same as the dn3d option in ffdshow with hq checked. I dislike denoising before warpsharping because warpsharp is a bit like a nuke, it changes everything. Personally, I dont like the effect it produces so I don't use it much, but you will definately need a denoiser at the end. __________________ CCCP \| MeWiki \| techouse's x264 builds \| Kovensky's mplayer builds

10th July 2006, 18:19	#6 \| Link
CAFxX Stray Developer Join Date: Mar 2003 Location: Italy Posts: 82	hqdn3d() -- default this is way too strong, the bg gets badly blurred hqdn3d(2, 1.5, 6, 4.5) -- halved spatial strenght way better, bg still gets (very) slightly blurred, no noticeable side-effects on the efficiency of my filter __________________ CAFxXcrossway, a collection of my projects CAFxX@strayorange, my blog

10th July 2006, 18:27	#7 \| Link
*.mp4 guy Registered User Join Date: Feb 2004 Location: USA Posts: 1,348	It would be easier to help if you provided an uncompressed source clip, or at the very least an unaltered image.

10th July 2006, 19:48	#8 \| Link
Awatef A CORE Media Player User Join Date: May 2002 Location: Africa Posts: 789	CAF, what you're doing is a waste of time IMHO. Upsizing won't create extra details, and when you recompress a DVD you'll be even LOOSING details so... __________________ Stereo better than 128kbps MP3? Ogg Vorbis at -q2! (~96kbps) / Get the LANCER releases for max conversion speed (at least twice as fast as Lame 3.97!)

10th July 2006, 22:31	#13 \| Link
CAFxX Stray Developer Join Date: Mar 2003 Location: Italy Posts: 82	Now for those who are still interested in helping: here's the screenshot of frame 80259 in all its 576p beauty. (Yes, the DVD has that black border). The only difference from the original source is that, being a PNG, the colorspace has been converted to RGB. Tomorrow I'll post the video. Be patient. Right now I'm pretty busy watching the huge celebration we're having down in Rome. There are 700000 (and counting) people out there. __________________ CAFxXcrossway, a collection of my projects CAFxX@strayorange, my blog Last edited by CAFxX; 10th July 2006 at 22:47.

11th July 2006, 10:24	#17 \| Link
foxyshadis Angel of Night Join Date: Nov 2004 Location: Tangled in the silks Posts: 9,569	You probably can't really get rid of those shadows, because they're present in the original (it looks like a lot of color bleed and a littel luma bleed is the main deal). You might be able to minimize it by first running dehalo_alpha on it, before attempting to sharpen, but that also softens some details. I wonder how you can live with such aliased results, but to each their own.

11th July 2006, 10:56	#18 \| Link
CAFxX Stray Developer Join Date: Mar 2003 Location: Italy Posts: 82	I left my PC doing a little test encode on a different sequence using hqdn3d(2, 1.5, 6, 4.5) and it's definitely not good. The BGs gets devastated. And i think that, if hqdn3d managed to keep at least a good amount of detail on the BG of the screenshot above and produces such a disaster in this other case, MultiSWAR will behave even worse (your first PNG screenshot shows less detail than mine in the BG, and the second shows too much artifacts on the face). Anyhow, I'll give it a try. Who knows, maybe I'm wrong. Next test will be hqdn3d with spatial denoising disabled. __________________ CAFxXcrossway, a collection of my projects CAFxX@strayorange, my blog

11th July 2006, 11:41	#20 \| Link
CAFxX Stray Developer Join Date: Mar 2003 Location: Italy Posts: 82	Tried deactivating spatial denoising and BGs are just little (almost unnoticeably) more blurry than without hqdn3d(). Still, it has problems with moving BGs. Will try to lower the temporal strenght as well (and maybe reactivating sliglthly spatial denoising). Other anime denoisers? No-one ever attempted to write a filter/function able to identify flat color areas? BTW, I tried to output at 720x404 and that already looks really great. EDIT: I'm currently uploading about 2 minutes of the raw mpeg2 stream to my site. It is around 130mb, and will likely be ready to be downlaoded in an hour. http://www.n-side.it/public/CAFxX/MH.mpg. I won't leave it there forever, so fetch it ASAP. EDIT2: Done uploading. __________________ CAFxXcrossway, a collection of my projects CAFxX@strayorange, my blog Last edited by CAFxX; 11th July 2006 at 15:20.