Integer DCT/iDCT filter questions

[trbarry]

Just out of curiousity I'm writing a filter that will do an integer DCT (Discreet Cosine Tranform), reduce some of the high frequency noise components, and then do an iDCT. I'm hoping it will help things compress better and maybe get rid of some noise. And I just wanted to play with filtering in the frequency domain.

I'm using the code from Xvid though I think most of our utilities use the same DCT/iDCT routines from DVD2AVI, etc. But it is not obvious from the code of these utilities what the range of values is returned from DCT and I would like to scale my input parameters so they don't overflow.

Anybody happen to know how the returned values range from integer DCT? They look like maybe 9-10 bit signed values.

- Tom

edit: I released it at www.trbarry.com/DctFilter.zip , see later post below.

[SansGrip]

Anybody happen to know how the returned values range from integer DCT? They look like maybe 9-10 bit signed values.

I have no idea, but I'm really looking forward to trying this filter as I was contemplating the same thing myself :).

[trbarry]

We talked about it here somewhere months ago but for some reason I got up this morning and started working on it.

I've got the DCT/IDCT part working but I'm trying to figure out how I want to handle the parms, and what I can get away with without having to deal with overflow problems.

I'd probably have it done tonight except Buffy & Smallville are on and even a nerd must have priorities. ;)

- Tom

[SansGrip]

We talked about it here somewhere months ago but for some reason I got up this morning and started working on it.

I'm investigating wavelet transforms myself. They look interesting.

I'd probably have it done tonight except Buffy & Smallville are on and even a nerd must have priorities. ;)

Especially a nerd with an HDTV :D.

[trbarry]

Especially a nerd with an HDTV

Sadly that is not true this week. Last Wed. my friendly local TV repairman took the convergence board for my RPTV back to the shop. He likely won't be back until Thursday afternoon.

And anyway Buffy is not in HDTV (drat!!!!!) and, while Smallville is, the WB channel here in Detroit is still having licensing problems about being able to broadcast HDTV. Something about Canadian interference across the river.

So I can't watch either of my 2 favorite shows in HDTV yet.

I think it is a conspiracy. :(

I usually like to do my web browsing and board posting from my recliner in front of a 55" screen, but I'm typing this now on a 20 year old NTSC TV that I'd kept around only to display the stock ticker. I've had to move my chair forward a few feet to even be able to read this.

Okay, end of OT rant. Back to Avisynth programming.

- Tom

[SansGrip]

Something about Canadian interference across the river.

I'm in Canada. Trust me, it's jealousy :D.

I usually like to do my web browsing and board posting from my recliner in front of a 55" screen

That's something I've yet to try. But I think if I had a cordless keyboard, mouse etc. and my TV out hooked up I'd never leave the house.

Back to Avisynth programming.

Good idea ;).

[Psyche]

Theoretically, DCT coefficients should fall in the [-2048, 2048] range, that is, 12 bits (11 bits plus sign).
BTW, I don't think removing noise *should* have an important impact in compression, because the quantizer does just that (well, MPEG quant at least) remove high frequency. BUT, maybe the motion estimation stage (which is done before getting into frequency domain) can benefit from the removed noise.
I don't know, let's wait a bit for the filter to be finished and then start making tests galore! Thanks to you both for the useful filters.

[-h]

Anybody happen to know how the returned values range from integer DCT? They look like maybe 9-10 bit signed values.

They will always be in the range [-2048, 2047].

Unless they're not, in which case they won't be :)

It may be possible to create coefficients outside that range with the implementation in XviD, however I strongly doubt that it will happen in pratcise. A couple pminsw/pmaxsw pairs will solve any problem with that however.

-h

[SansGrip]

A couple pminsw/pmaxsw pairs will solve any problem with that however.

You mean to say there are min/max instructions in MMX? Now I really want to learn :).

[-h]

You mean to say there are min/max instructions in MMX? Now I really want to learn :).

Well pminsw/pmaxsw are part of I-SSE, which doesn't exist on things like K6-2's or really old celerons (I think). It's not terribly hard to perform a min/max operation with vanilla MMX though, just using saturated addition/subtraction.

-h

[Psyche]

They will always be in the range [-2048, 2047].
Unless they're not, in which case they won't be
:D Very funny man!
BTW, we replied at the same time.

[trbarry]

Okay, thanks guys.

I guess it looks like the results might then use a signed 12 bits, and I'll save one more for good luck or -h's joke, which has now been cursed by Murphy's law. ;)

But out of 16 that still leaves me 3 bits to play with so I guess I can multiply things by 8 and still get away with it.

- Tom

[trbarry]

I finally released a first version of this, for YV12 Avisynth 2.5 only.

It currently will work only on size multiples of 16x16, so you will have to crop or add borders as needed.

See:

www.trbarry.com/Readme_DctFilter.txt for the readme and
www.trbarry.com/DctFilter.zip for the dll and source.

I'm not sure yet about the application of this or any recommended parms (8 of them). I've attached an image (Help, mod :o ) of using

DctFilter(1,1,1,.7,.3,0,0,0)

which is probably too strong but I wanted to show the effect.

With some very quick testing, using that above set of parms Xvid will save more than 30% at an Xvid fixed quant=2 but that drops to a little over 25% at quant 3, and likely even less at higher quants.

I'm only starting to study this type of filtering but think it is very interesting.

Beware that if you crank the parms up too much the results will be both soft and blocky.

Somebody please let me know what you think of all this, or any recommended applications. I'm also curious if this can have application to anime since it is often full of hard edges, too high spatial frequencies. But I haven't tested any anime yet.

And, BTW, thanks to the Xvid guys, since I stole their DCT code. ;)

- Tom

[trbarry]

I seem to be in a coffee driven manic state this morning, but just had another interesting idea.

If there were any demand for this type of filtering MPEG2DECx (or even ffdshow) could be made to perform this service almost for free, by gimmicking the DCT values straight from the file before finishing the decoding process. It would easily run real time.

But I don't know how that would interact yet with the motion estimation stuff, or what order all that is performed.

Wonder if it would work? ;)

- Tom

[-h]

Wonder if it would work? ;)

I think Bad Things would happen. Smearing over time and such. Then again there are only what, 5 or 6 consecutive P-frames in standard MPEG-2 encodes?

There'd be weird snap-back effects every I-frame and other odd things. What kind of filtering are you performing on the coefficients? The same filter could be used inside xvidcore after its own DCT is performed.

-h

[SansGrip]

@Tom

I'm getting an "illegal instruction" exception with the following script:


LoadPlugin("mpeg2dec3.dll")
LoadPlugin("dctfilter.dll")
Mpeg2Source("g:\ap\ap.d2v")
DctFilter(1, 1, 1, 1, 1, 1, 0.5, 0)
Levels(0, 1.5, 255, 16, 255)
ConvertToYUY2()
ConvertToRGB()


The frame size is 720x480. I get the exception whatever parameters I use, but no exception when I comment out DctFilter :(.

I'm on an Athlon XP, and am just about to fire up the debugger to give you more info.

[SansGrip]

The exception occurs at 'punpcklqdq' in fdct_mmx.asm. That's an SSE2 instruction and so won't work on my Athlon :(.

[trbarry]

SansGrip -

It sounds like I'm calling sse2 forward DCT when I should use sse. I'll either find the problem quickly or post a version that uses mmx only for now. But if that is the problem then the current version probably only runs on P4's, which is the only place I've tested it so far. But easy to fix.

-h

All I'm doing is multiplying each of the 64 values returned from FDCT by x/8 with a different x for each position, depending on the row/col parms.

I was thinking that if this was useful Xvid could even do it as a prefiltering, doing something similar. It would just apply it after the fdct but before the idct used to create the image for motion search.

But again, I haven't thought this through properly and even on decode I don't really know the order of applying dct vs motion estimation info. So there might be a clash somewhere that would make it harder to do.

Anyway I guess it's better to play with it for awhile as a standalone filter and see what works. (besides P4's)

- Tom

[trbarry]

Due to a couple of brain-dead mistakes I was doing the CPU type checking backwards. Also I've turned off sse2 support until I get a chance to look at it further. It hopefully should now run on all mmx or higher boxes.

So please try another copy: www.trbarry.com/DctFilter.zip

- Tom

[SansGrip]

Works great now! Even with quite aggressive settings it's hard to spot a difference between the two, even though compression would (theoretically) be greatly enhanced!

Is it my imagination, or do settings like:

DctFilter(1, 1, 1, 1, 0.5, 0, 0, 0)

actually sharpen details?

Edit: Here's a grab (http://www.jungleweb.net/~sansgrip/DctFilter_1.png) (1.1mb) to show what I mean. The top image is the original, the bottom is with the above settings. Certain areas show a very distinct sharpening effect, notably the pattern on the drapes, the fabric on the circular-back chair in the right hand side of the frame, and the right edge of the picture frame.

Edit 2: Now Tom's attachment has been approved I can see the same effect in it, too.

[trbarry]

Is it my imagination, or do settings like:

DctFilter(1, 1, 1, 1, 0.5, 0, 0, 0)

actually sharpen details?

I don't think so, except maybe for macro block boundaries. But I'm not sure yet exactly what it does. I wrote it to find out. ;)

The only time I thought it sharpened was earlier when I did not realize that in 2.5 Avisynth StackVertical puts the first clip on the BOTTOM, and I had them backwards.

So I put subtitles on them to be sure. I think I already commented on that somewhere today.

But I'm glad it works now. I'm really sort of surprised how much info we can throw away before it starts to obviously affect the PQ. I know much of it would be pitched out during the quantization process anyway but this is different. Instead of quantizing a frequency component for lesser accuracy we are just keeping the accuracy of a diminished component. (which is quantized later if needed)

I'd almost be willing to bet you could store 6x6 macro blocks and supply a boundary of 2 rows and columns of zeros later and nobody would notice. But maybe those rows and columns usually get quantized to zeros anyway, especially with the H263 quant matrix. This just helps it along a little.

- Tom

[SansGrip]

The only time I thought it sharpened was earlier when I did not realize that in 2.5 Avisynth StackVertical puts the first clip on the BOTTOM, and I had them backwards.

Ah hah! That's the same mistake I made :D.

I'm really sort of surprised how much info we can throw away before it starts to obviously affect the PQ.

I'm going to have to run some more tests now that I know which way is up :rolleyes: ;).

You're right, though -- this is a very "natural" way to improve compressibility. If you see what I mean.

[iago]

@Tom

I just want to say (based on a few constant quant 2 tests with XviD and mpeg quantization) that this filter seems like a magical tool! ;)

~30% compressibility gain and almost no quality loss with
----------------------------
DctFilter(1,1,1,1,0.5,0,0,0)
----------------------------
in the script. However this setting resulted in terrible EE together with LanczosResize(576,304), which I'll try to get rid of from either with UnFilter or UnDot following the DctFilter in the script.

Thanks for all your great work,

best regards,
iago

[trbarry]

iago -

I didn't mention it before but this filter should probably be run LAST, after everything else.

The reason is that it creates values that a subsequent DCT operation will use to create lots of zeros and low values and we probably wouldn't want to mess that up.

Did you run it after resize? If so, maybe you will just have to tone it down a bit.

Or maybe run UnFilter(-,-) or C3D right before it. Or even after, who knows? Nobody knows how this works yet. :)

- Tom

[iago]

@Tom,

Yes, DctFilter was the last chain in my script ;). Now I'm trying with some less agressive settings as below and with h263 quantization type:

----------------------------------------------
LoadPlugin("C:\FILTERS-YV12\MPEG2Dec3.dll")
LoadPlugin("C:\FILTERS-YV12\UnFilter.dll")
LoadPlugin("C:\FILTERS-YV12\DctFilter.dll")
mpeg2source("D:\RIP\RIP.d2v")
crop(2,16,-2,-16)
BicubicResize(576,304,0,0.7)
LumaFilter(-2)
UnFilter(-5,-5)
DctFilter(1,1,1,1,.7,.5,.2,0)
-----------------------------------------------

With the above script, I'll run a two pass encode of a pretty hard-to-compress movie aiming for 1CD, even without b-frames! ;)

I'll report back the results.

ciao,
iago

[SansGrip]

I just tried this with a very noisy source and it doesn't really touch the noise, at least in any noticible way. But that's what smoothers are for, right? ;)

[SansGrip]

I'm harbouring some hope that this filter might help reduce mosquito noise on resulting low bitrate MPEG-1 encodes. Since the Gibbs effect is a high-frequency phenomenon, maybe we'll be able to pull down the high-frequency information enough to at least subdue it somewhat...

[trbarry]

I just tried this with a very noisy source and it doesn't really touch the noise, at least in any noticible way. But that's what smoothers are for, right?

Actually that surprises me a lot. I would have thought that much noise would appear as high frequency stuff this filter would get rid of. But I haven't tried it.

- Tom

[trbarry]

For mosquito noise you might also want to try UnDot first.

- Tom

[SansGrip]

Actually that surprises me a lot. I would have thought that much noise would appear as high frequency stuff this filter would get rid of.

I tested briefly with something like 1,1,1,1,0,0,0,0 (and even with 1,1,1,0,0,0,0,0) but the noise was still there. Maybe I'm misunderstanding how the parameters work, but wouldn't those get rid of a lot of high-frequency information, including noise?

[SansGrip]

For mosquito noise you might also want to try UnDot first.

I might be mistaken, but I was under the impression that UnDot can help existing mosquito noise. I'm looking for a way to filter "proactively" to reduce the appearance of mosquito noise in the encode (sort of like what Blockbuster does for DCT blocks).

[trbarry]

You are probably correct there. I misunderstood what you were trying to do.

- Tom

[iago]

@SansGrip

Such a compressibility gain without losing the details and without losing the source noise; and _you_ still complain about it!? :D

iago

[milan]

I've just added DCT/iDCT filter to ffdshow. GoFilterIt method is ported without any change and it uses unmodified XviD DCT/IDCT functions. Only luma plane is filtered for now. No integration with libavcodec decoding routines yet, I'll try to implement this later.
Simple configuration page is present too, just allowing to enter 8 floating point numbers. Maybe later I'll add possibility to enter/load/save full 8x8 matrix.

One question to trbarry: is your filter able to handle parameters higher than 1?

[serbersan]

Maybe a stupid question. I've recently studed about filters in the frequency domain....

And I don't understand why those type of filters doesn't be used until now?? Difficult to implement???

Good Work Tom¡¡

[trbarry]

One question to trbarry: is your filter able to handle parameters higher than 1?

milan -

Don't know yet. ;)

From what I've been told above in this thread about the max/min values from DCT is correct then nothing should overflow until parms get > 1.414. I suppose that would also depend upon whether the iDCT had problems with larger values.

But I haven't had a chance to test it and find out what it does. I'd wondered if it would act like adding noise in post processing to do something like:

DctFilter(1,1,1,1,1,1,1.3,1.3)

Maybe making things look more realistic after upscaling but that's just a guess. And possibly it would also have a sharpening effect but I really can't predict that one at all.

- Tom

[Guest]

Originally posted by serbersan
Maybe a stupid question. I've recently studed about filters in the frequency domain....

And I don't understand why those type of filters doesn't be used until now?? Difficult to implement??? Not difficult to implement but highly CPU intensive. Such filters tend to be slow and Doom9'ers do not like to wait. :)

Also, most of the things we need to do don't require frequency domain processing, although it's certainly possible that some things could be done better there, at the expense of time.

[trbarry]

I have a question myself for anyone who understands DCT/IDCT etc.

Say, for a too extreme example, you had decided that using

DctFilter(1,1,1,1,0,0,0,0)

seemed to give acceptable results on some clip. That is, representing the clip with all the higher frequencies zeroed out in 3/4 of the 8x8 DCT output block. This would leave non-zero values only in the top left 4x4 corner.

But there is nothing magical about an 8x8 macro block except we have fast code for the 8x8 DCT & IDCT. So if you had a 4x4 IDCT then it could be performed on the intermediate results of the above, giving a clip with only 1/4 the number of pixels. That is, do the 8x8 DCT, zero the extra values (or pretend you did) and then do a 4x4 IDCT resulting in a downsized clip with only half the H and V dimensions. Encode that clip with Xvid. Then reverse the process at display time.

Since not counting integer round off's and quantizations DCT is a reversible transformation then it seems that if the results of running DctFilter above was acceptable then so should the downsize/upsize be okay in that fashion.

Anybody know about this sort of thing? Am I asking for too much of a free lunch?

- Tom

[-h]

But there is nothing magical about an 8x8 macro block except we have fast code for the 8x8 DCT & IDCT. So if you had a 4x4 IDCT then it could be performed on the intermediate results of the above, giving a clip with only 1/4 the number of pixels. That is, do the 8x8 DCT, zero the extra values (or pretend you did) and then do a 4x4 IDCT resulting in a downsized clip with only half the H and V dimensions. Encode that clip with Xvid. Then reverse the process at display time.

I was initially going to say this wouldn't work due to frequency mismatch, but then remembered that the top 3/4 of the coefficients represent signals whose resolution is too high for a 4x4 block to represent. So yes this reconstruction would "work" for performing downsizing.

I'm not so sure that it's a good idea though :) XviD will encode the 4x4-blocked image with its native 8x8 DCT, destroying a fair bit of the signal along the way. If the clip produced by only leaving 1/4 of the coefficients is watchable, you would get similar quality by just resizing to a quarter of the image size and blowing it back up on playback.

-h

[trbarry]

I'm not so sure that it's a good idea though XviD will encode the 4x4-blocked image with its native 8x8 DCT, destroying a fair bit of the signal along the way. If the clip produced by only leaving 1/4 of the coefficients is watchable, you would get similar quality by just resizing to a quarter of the image size and blowing it back up on playback.

Yep. But I was betting that since there would only be 1/4 as many pixels Xvid could encode it at a MUCH lower average quant, not to mention how much faster it would go at only 1/4 res. (or maybe 9/16 res)

Really this would just be another way of downscaling to make the file size smaller. But I was wondering if this method would somehow be more reversible if we eventually intended to upscale again at display time.

In principle there is probably no reason now that you couldn't double the size of any clip at display time by doing a 4x4 fDCT, padding it with 0's like DctFilter, and then doing an 8x8 iDCT.

But I have no idea if it would look any better than the hardware scaling on the average Radeon card and it would certainly take longer.

But if we had a more reversible scaling method it might be worth it. That is, optimize the downscale method with the intention of upscaling again at display time with a known method optimized for that special purpose. Playing with DctFilter can maybe show something of what we could get away with here.

- Tom

[SansGrip]

Such a compressibility gain without losing the details and without losing the source noise; and _you_ still complain about it!? :D

I'm not complaining! I think it's great! :D

The reason I was eager to test it on noisy sources is that I was actually just about to try this kind of frequency filtering myself (I'm investigating wavelets, too) because I figured it would work well on noise.

Though this just reinforces my developing belief that spatial noise reduction is a flawed concept.

And before you say anything else, that doesn't mean I think _this_ filter is flawed, because it's not a spatial denoiser :p.

[SansGrip]

Anybody know about this sort of thing? Am I asking for too much of a free lunch?

I don't really know about this sort of thing, but I don't see how one would gain anything from doing it, since you already got the compressibility gain from the zigzag RLE of all those zeroes.

Or perhaps I'm misunderstanding what you wrote. I'll go read it again :).

[jarthel]

how about a yuv2 version for people who sticked with pre-2.5 avisynth? :)

jayel

[trbarry]

how about a yuv2 version for people who sticked with pre-2.5 avisynth?

jarthel -

Probably not right away. I consider this still experimental and I'm not ready to be maintaining 2 versions. And even YUY2 users can just convert to YV12 once they are using a production version of Avisynth 2.5, once that gets here. There shouldn't be any quality loss doing that because it is going to happen as soon as you get to Xvid anyway.

I find I like working in YV12 format better, it gives better and faster results, and most of my input data is in that format anyway. So as long as there are no real show stoppers I'll probably stick mostly to that.

- Tom

[jarthel]

thanks Tom :)

I also await for the production or even beta release of Avisynth.

[Marc FD]

hi ^^

sorry, i think i'm very dumb because i don't see any interrest in this filter. reminds me the vp3 internals.

BTW, please don't speak of quant 2 compressibility.
it means nothing when you play with spacial freq.
you can lower it 2x in real time. that's not hard.

i think you should never use anything DCT-based out of a DCT-based codec, but for ultra hardcore denosing. (btw, wavelets are really much better for that)

just my 2 euros cents ;)

[SansGrip]

Here's an Avisynth 2.0 YUY2 version for those of us still stuck in the stone age ;). I didn't update the readme, but the usage is the same.

I've sent the new code to Tom and am awaiting his judgement on it, so this may or may not make it into his release. It works for me, though, and at least the 2.0 crowd has something new to play with :).

You can get it here (http://www.jungleweb.net/~sansgrip/DctFilter_YUY2.zip) until the attachment is modded.

Have fun.

[trbarry]

BTW, please don't speak of quant 2 compressibility.
it means nothing when you play with spacial freq.
you can lower it 2x in real time. that's not hard.

i think you should never use anything DCT-based out of a DCT-based codec, but for ultra hardcore denosing. (btw, wavelets are really much better for that)

Marc -

Dunno yet.

It looks like it can save a bunch when using low quants and probably not as much with higher quants.

But with a normal 2 pass encode the actual quants used probably wanders around quite a bit. So I'm hoping that for high quality encodings what I save on the low quants will lower the average quant of the other blocks and frames.

Still playing, but I'm hoping this will be good for more than just noise removal (which is nice too). It is easily possible that this won't be anything that could not already be accompliched in Xvid with modulated/custom quants, but it's another tool in the box and more visible to tinker with. ;)

- Tom

[SansGrip]

trbarry: It is easily possible that this won't be anything that could not already be accompliched in Xvid with modulated/custom quants

Well, let's not forget that not everyone is using XViD. This allows one to monkey around with the frequency domain even when the encoder's quant matrix is hardcoded.

What's more, it might even help compressibility wrt encoders that don't use the DCT model. I can't think of any right now, but that's not the point ;).

[trbarry]

SansGrip -

I'm still testing to see where/if it helps compressibility. Hopefully does.

BTW, my last post crossed with yours and I didn't even notice that you had released the YUY2 version above.

All -

Hopefully I'll have SansGrip's YUY2 version also included in mine by later tonight. I know some folks have been asking for it.

- Tom