Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th February 2006, 23:53   #1  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,564
Dithering gradients to yv12

I captured a pure-CG show recently and was shocked by how bad the banding was in it, and moving to yv12 to process just made it slightly worse. The problem is that there's almost no noise, and it's not a very colorful show at all, so you're right up against the color resolution limits of yv12.

Now, the best observation I can make is that this always happens on the gradients, especially noticeable on shallow gradients. Well, the main gradient filter I know of it TBilateral, and with an enormous radius it'd probably be able to find find the gradients, but even if internally it could use a high bit-depth it all gets truncated to yv12 in the end. If a version of TBilateral came out that could dither that would be hotter than hot (I've poked at it, but my C is too rusty). I bet yv24 would be great for this.

I've toyed with adding and removing noise, although I haven't yet tried a friend's good suggestion of adding and removing a little noise many times. If I could make a mask of the gradient areas, that would be awesome, then you can do all kinds of things without affecting main detail. So I thought I'd ask and see what nifty ideas you guys have.

Also, I'm open to any AE plugins you know of, but even if it looks great in rgb it still has to be able to end up looking good in yv12 in the end.
foxyshadis is offline   Reply With Quote
Old 26th February 2006, 23:58   #2  |  Link
Zarxrax
Registered User
 
Join Date: Dec 2001
Posts: 1,190
Maybe you could try using some smoother that will smooth out the gradients, and then overlay this onto the original clip trying different levels of opacity. I think this should allow you to keep some detail, but still make the banding less apparent.
Zarxrax is offline   Reply With Quote
Old 27th February 2006, 01:51   #3  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,564
The problem is that it's the smoothers themselves that cause the banding. Try something like this on a fairly clean source with some banding in low-contrast areas:
Code:
TBilateral(diameterL=7,diameterC=7,sDevL=2.00,sDevC=2.00,iDevL=50.200,iDevC=48.800, \
  csL=1.000,csC=1.000,d2=false,chroma=true,kernS=4,kernI=9,resType=0)
Or an equivalent smoother at extremely strong smoothing levels. You'll see that instead of smoothing away banding, it'll actually increase it, because it removes the dithering noise that hides the small colorspace of yv12 (at least as opposed to rgb32), esp. with tv-levels.
foxyshadis is offline   Reply With Quote
Old 27th February 2006, 02:01   #4  |  Link
DryFire
Perpetually Lost
 
Join Date: Sep 2005
Posts: 93
I don't know if this would be of any use in such a case but this might be of worth a shot.
DryFire is offline   Reply With Quote
Old 27th February 2006, 02:04   #5  |  Link
Zarxrax
Registered User
 
Join Date: Dec 2001
Posts: 1,190
Have you tried Deen? It's w2d/w3d modes are specifically designed to preserve gradients.
Zarxrax is offline   Reply With Quote
Old 27th February 2006, 02:09   #6  |  Link
Soulhunter
Bored...
 
Soulhunter's Avatar
 
Join Date: Apr 2003
Location: Unknown
Posts: 2,812
Code:
Function DeBand(Clip Clp,Int"Dither")
{
Dither=Default(Dither,5)
Result=Clp.AddGrain(Dither,0,0)
Return(Result)
}
Just kidding... ^^


Bye
__________________

Visit my IRC channel
Soulhunter is offline   Reply With Quote
Old 27th February 2006, 02:34   #7  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,564
I will have to meditate on how to apply Didée's method in this case (where the conversion has already been done), but as usual he comes to my rescue. Zar, I will also give Deen a shot, although I haven't used it in ages, because if that's true then it may well have the dithering built-in.

Soulhunter, oddly enough, AddGrain(128) completely removes the banding! Thank you so much!
foxyshadis is offline   Reply With Quote
Old 27th February 2006, 02:52   #8  |  Link
Soulhunter
Bored...
 
Soulhunter's Avatar
 
Join Date: Apr 2003
Location: Unknown
Posts: 2,812
You have a sample of ya CG show for us?


Bye
__________________

Visit my IRC channel
Soulhunter is offline   Reply With Quote
Old 27th February 2006, 03:20   #9  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,564
Okay, these are all from the .ts capture:

1, 2, 3, 4, 5, 6, 7

JPEGs for the bandwidth impaired: (Note the jpegs actually look slightly better thanks to the ringing noise introduced.... that's an interesting idea.)

1, 2, 3, 4, 5, 6, 7

Either the CG was just entirely smooth before broadcast and it was the mpeg-2 that hurt it, or they ran it through an extra smooth filter pre-broadcast.
foxyshadis is offline   Reply With Quote
Old 27th February 2006, 04:43   #10  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,394
Quote:
Originally Posted by foxyshadis
I will have to meditate on how to apply Didée's method in this case
Ah, Soul & foxy practice telepathy.

If the banding is already in the source, restoring obviously can't work.

Make a min-max edgemask of radius 2 or 3. Consider only areas where this min-max is, say, 3 or below. To only these areas, add enough noise, and mt_clamp by the formerly calculated min's and max's.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 27th February 2006, 06:00   #11  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,564
You're the man, as usual. I gave mt_luts a shot the way you said and put together a noise+smoothing method that's starting to give very good results. It's very slow, I think I need to use a dumber smoother without concepts like "detail preservation". I'm going to try Deen at a high radius, or maye msmooth or mipsmooth. I'll post what I end up with later.
foxyshadis is offline   Reply With Quote
Old 27th February 2006, 10:44   #12  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,564
This is quite slow (1.5 fps on my dual-core yonah), but very effective against extreme banding. More moderate settings would probably be faster and more useful more other sources.
Code:
x=Source()
m=x.deband_makemask(4)
b=x.deband_blur(7)
deband=mt_merge(x,b,m,luma=true)
deband

function deband_makemask(clip c, int "radius") {
	radius=default(radius,3)
	return mt_luts(c,c,mode="range",pixels = mt_square( radius ), expr = "y" ) \
		.mt_lut(expr = "400 x 2.5 - abs 1.7 ^ /" ).removegrain(mode=20)
}

function deband_blur(clip c, int "noise",bool "extra") {
	noise=default(noise,8)
	extra=default(extra,true)
	noisec=noise*2/3
	g1=c.addgrain(noise,0,0)
	grainu=c.utoy().addgrain(noisec,0,0)
	grainv=c.vtoy().addgrain(noisec,0,0)
	grain=ytouv(grainu,grainv).mergeluma(g1)
	tb=grain.TBilateral(diameterL=7,diameterC=7,sDevL=7.00,sDevC=7.00, \
	  iDevL=255.000,iDevC=255.000,csL=0.000,csC=0.000,d2=false,chroma=true,kernS=4,kernI=4,resType=0)
	f=extra ? tb.addgrain(1,0,0).removegrain(mode=2).addgrain(1,0,0).degrainmedian(limity=5,limituv=0,mode=1) : tb
	return f
}

function deband(clip c, clip "m", clip "b") {
	m=default(m,c.deband_makemask())
	b=default(b,c.deband_blur())
	return mt_merge(c,b,m,luma=true)
}
There might be a way to put the lut+removegrain 20 in the luts, but I'm not familiar enough with advanced masktools. TBilateral is a big source of slowdown but it's the only filter that can blur as much as it does. The three denoisers all work differently, they look nicer comined than any one alone.
foxyshadis is offline   Reply With Quote
Old 28th February 2006, 01:34   #13  |  Link
Soulhunter
Bored...
 
Soulhunter's Avatar
 
Join Date: Apr 2003
Location: Unknown
Posts: 2,812
Someone else's try... ^^

1, 2, 3, 4, 5, 6, 7 / Script and DLLs

Bye
__________________

Visit my IRC channel

Last edited by Soulhunter; 28th February 2006 at 01:44.
Soulhunter is offline   Reply With Quote
Old 28th February 2006, 01:43   #14  |  Link
DryFire
Perpetually Lost
 
Join Date: Sep 2005
Posts: 93
Soulhunter do you have the readme for that broadsword dll?

Also I see someone is working on Ergo Proxy.
DryFire is offline   Reply With Quote
Old 28th February 2006, 01:49   #15  |  Link
Soulhunter
Bored...
 
Soulhunter's Avatar
 
Join Date: Apr 2003
Location: Unknown
Posts: 2,812
Quote:
Originally Posted by DryFire
Soulhunter do you have the readme for that broadsword dll?
Nope... ;P

But you only need the 1st param -> threshold!


Bye
__________________

Visit my IRC channel

Last edited by Soulhunter; 28th February 2006 at 02:29.
Soulhunter is offline   Reply With Quote
Old 28th February 2006, 07:46   #16  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,564
Thanks Soulhunter! Very interesting. It blurs detail way too much, but combined with a mask like the one I use, it would be an excellent replacement for the deband_blur function I use, esp. if it's a lot faster. I already applied my filtering and I'm ready to compress, but since it's almost guaranteed the next episodes are going to look the same (since the teasers were), I'll probably use yours later.

1, 2, 3, 4, 5, 6, 7

They are grainier, which I'm not sure whether I like or not. I might just refilter it all with soulhunter's tools. Are you willing to release the source at all? I'm quite interested in how broadsword works (I suspect the other works like ffdshow's hq upsampling).

Last edited by foxyshadis; 28th February 2006 at 11:04.
foxyshadis is offline   Reply With Quote
Old 28th February 2006, 17:45   #17  |  Link
Soulhunter
Bored...
 
Soulhunter's Avatar
 
Join Date: Apr 2003
Location: Unknown
Posts: 2,812
Sorry, no source-code... Im not the author of this plugins! But they work like this: BroadSword does hard thresholding of a weighted overcomplete 8x8 DCT transform with error diffusion when quantizing back to 8 bits, and YV12ToRGB24HQ is a 16 bit YV12->RGB24 conversion with error diffusion when quantizing back to 8 bits... same logic ^^


Bye
__________________

Visit my IRC channel

Last edited by Soulhunter; 28th February 2006 at 17:47.
Soulhunter is offline   Reply With Quote
Old 1st March 2006, 02:02   #18  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,564
Wow, soul, you should have warned me; I got .25 fps per core, even after significantly speeding up the masking. o.O;; It looks so good I can live with that though. However, they also have a killer memory leak that chewed up 3.5 gigs in less than 600 frames before crashing. Now I really wish I had the source code, I'd like to fix it and continue.
foxyshadis is offline   Reply With Quote
Old 1st March 2006, 04:24   #19  |  Link
Soulhunter
Bored...
 
Soulhunter's Avatar
 
Join Date: Apr 2003
Location: Unknown
Posts: 2,812
Quote:
Originally Posted by foxyshadis
However, they also have a killer memory leak that chewed up 3.5 gigs in less than 600 frames before crashing.
SetMemoryMax(256) or so?


Bye
__________________

Visit my IRC channel
Soulhunter is offline   Reply With Quote
Old 1st March 2006, 04:51   #20  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,564
That only works for AVISynth's cache, unfortunately. It loses about 5 megs per frame, which is weird because each frame is only 1.3 megs in yv12 (or 2.6 in rgb).

A little more investigation indicates that yv12torgb24hq is the leak, which is unfortunate, because it's the fast half and really improves the output a lot. Hmm, maybe broadsword can be replaced by fft3dfilter somehow.
foxyshadis is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 06:07.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.