Dithering gradients to yv12

[foxyshadis]

I captured a pure-CG show recently and was shocked by how bad the banding was in it, and moving to yv12 to process just made it slightly worse. The problem is that there's almost no noise, and it's not a very colorful show at all, so you're right up against the color resolution limits of yv12.

Now, the best observation I can make is that this always happens on the gradients, especially noticeable on shallow gradients. Well, the main gradient filter I know of it TBilateral, and with an enormous radius it'd probably be able to find find the gradients, but even if internally it could use a high bit-depth it all gets truncated to yv12 in the end. If a version of TBilateral came out that could dither that would be hotter than hot (I've poked at it, but my C is too rusty). I bet yv24 would be great for this.

I've toyed with adding and removing noise, although I haven't yet tried a friend's good suggestion of adding and removing a little noise many times. If I could make a mask of the gradient areas, that would be awesome, then you can do all kinds of things without affecting main detail. So I thought I'd ask and see what nifty ideas you guys have.

Also, I'm open to any AE plugins you know of, but even if it looks great in rgb it still has to be able to end up looking good in yv12 in the end.

[Zarxrax]

Maybe you could try using some smoother that will smooth out the gradients, and then overlay this onto the original clip trying different levels of opacity. I think this should allow you to keep some detail, but still make the banding less apparent.

[foxyshadis]

The problem is that it's the smoothers themselves that cause the banding. Try something like this on a fairly clean source with some banding in low-contrast areas:

TBilateral(diameterL=7,diameterC=7,sDevL=2.00,sDevC=2.00,iDevL=50.200,iDevC=48.800, \
  csL=1.000,csC=1.000,d2=false,chroma=true,kernS=4,kernI=9,resType=0)

Or an equivalent smoother at extremely strong smoothing levels. You'll see that instead of smoothing away banding, it'll actually increase it, because it removes the dithering noise that hides the small colorspace of yv12 (at least as opposed to rgb32), esp. with tv-levels.

[DryFire]

I don't know if this would be of any use in such a case but this might be of worth a shot.

[Zarxrax]

Have you tried Deen? It's w2d/w3d modes are specifically designed to preserve gradients.

[Soulhunter]

Function DeBand(Clip Clp,Int"Dither")
{
Dither=Default(Dither,5)
Result=Clp.AddGrain(Dither,0,0)
Return(Result)
}

Just kidding... ^^


Bye

[foxyshadis]

I will have to meditate on how to apply Didée's method in this case (where the conversion has already been done), but as usual he comes to my rescue. Zar, I will also give Deen a shot, although I haven't used it in ages, because if that's true then it may well have the dithering built-in.

Soulhunter, oddly enough, AddGrain(128) completely removes the banding! Thank you so much!

[Soulhunter]

You have a sample of ya CG show for us?


Bye

[foxyshadis]

Okay, these are all from the .ts capture:

1, 2, 3, 4, 5, 6, 7

JPEGs for the bandwidth impaired: (Note the jpegs actually look slightly better thanks to the ringing noise introduced.... that's an interesting idea.)

1, 2, 3, 4, 5, 6, 7

Either the CG was just entirely smooth before broadcast and it was the mpeg-2 that hurt it, or they ran it through an extra smooth filter pre-broadcast.

[Didée]

Originally Posted by foxyshadis

I will have to meditate on how to apply Didée's method in this case

Ah, Soul & foxy practice telepathy.

If the banding is already in the source, restoring obviously can't work.

Make a min-max edgemask of radius 2 or 3. Consider only areas where this min-max is, say, 3 or below. To only these areas, add enough noise, and mt_clamp by the formerly calculated min's and max's.

[foxyshadis]

You're the man, as usual. I gave mt_luts a shot the way you said and put together a noise+smoothing method that's starting to give very good results. It's very slow, I think I need to use a dumber smoother without concepts like "detail preservation". I'm going to try Deen at a high radius, or maye msmooth or mipsmooth. I'll post what I end up with later.

[foxyshadis]

This is quite slow (1.5 fps on my dual-core yonah), but very effective against extreme banding. More moderate settings would probably be faster and more useful more other sources.

x=Source()
m=x.deband_makemask(4)
b=x.deband_blur(7)
deband=mt_merge(x,b,m,luma=true)
deband

function deband_makemask(clip c, int "radius") {
	radius=default(radius,3)
	return mt_luts(c,c,mode="range",pixels = mt_square( radius ), expr = "y" ) \
		.mt_lut(expr = "400 x 2.5 - abs 1.7 ^ /" ).removegrain(mode=20)
}

function deband_blur(clip c, int "noise",bool "extra") {
	noise=default(noise,8)
	extra=default(extra,true)
	noisec=noise*2/3
	g1=c.addgrain(noise,0,0)
	grainu=c.utoy().addgrain(noisec,0,0)
	grainv=c.vtoy().addgrain(noisec,0,0)
	grain=ytouv(grainu,grainv).mergeluma(g1)
	tb=grain.TBilateral(diameterL=7,diameterC=7,sDevL=7.00,sDevC=7.00, \
	  iDevL=255.000,iDevC=255.000,csL=0.000,csC=0.000,d2=false,chroma=true,kernS=4,kernI=4,resType=0)
	f=extra ? tb.addgrain(1,0,0).removegrain(mode=2).addgrain(1,0,0).degrainmedian(limity=5,limituv=0,mode=1) : tb
	return f
}

function deband(clip c, clip "m", clip "b") {
	m=default(m,c.deband_makemask())
	b=default(b,c.deband_blur())
	return mt_merge(c,b,m,luma=true)
}

There might be a way to put the lut+removegrain 20 in the luts, but I'm not familiar enough with advanced masktools. TBilateral is a big source of slowdown but it's the only filter that can blur as much as it does. The three denoisers all work differently, they look nicer comined than any one alone.

[Soulhunter]

1, 2, 3, 4, 5, 6, 7 / Script and DLLs

Bye

[DryFire]

Soulhunter do you have the readme for that broadsword dll?

Also I see someone is working on Ergo Proxy.

[Soulhunter]

Originally Posted by DryFire

Soulhunter do you have the readme for that broadsword dll?

Nope... ;P

But you only need the 1st param -> threshold!


Bye

[foxyshadis]

Thanks Soulhunter! Very interesting. It blurs detail way too much, but combined with a mask like the one I use, it would be an excellent replacement for the deband_blur function I use, esp. if it's a lot faster. I already applied my filtering and I'm ready to compress, but since it's almost guaranteed the next episodes are going to look the same (since the teasers were), I'll probably use yours later.

1, 2, 3, 4, 5, 6, 7

They are grainier, which I'm not sure whether I like or not. I might just refilter it all with soulhunter's tools. Are you willing to release the source at all? I'm quite interested in how broadsword works (I suspect the other works like ffdshow's hq upsampling).

[Soulhunter]

Sorry, no source-code... Im not the author of this plugins! But they work like this: BroadSword does hard thresholding of a weighted overcomplete 8x8 DCT transform with error diffusion when quantizing back to 8 bits, and YV12ToRGB24HQ is a 16 bit YV12->RGB24 conversion with error diffusion when quantizing back to 8 bits... same logic ^^


Bye

[foxyshadis]

Wow, soul, you should have warned me; I got .25 fps per core, even after significantly speeding up the masking. o.O;; It looks so good I can live with that though. However, they also have a killer memory leak that chewed up 3.5 gigs in less than 600 frames before crashing. Now I really wish I had the source code, I'd like to fix it and continue.

[Soulhunter]

Originally Posted by foxyshadis

However, they also have a killer memory leak that chewed up 3.5 gigs in less than 600 frames before crashing.

SetMemoryMax(256) or so?


Bye

[foxyshadis]

That only works for AVISynth's cache, unfortunately. It loses about 5 megs per frame, which is weird because each frame is only 1.3 megs in yv12 (or 2.6 in rgb).

A little more investigation indicates that yv12torgb24hq is the leak, which is unfortunate, because it's the fast half and really improves the output a lot. Hmm, maybe broadsword can be replaced by fft3dfilter somehow.