Integer DCT/iDCT filter questions

[trbarry]

Just out of curiousity I'm writing a filter that will do an integer DCT (Discreet Cosine Tranform), reduce some of the high frequency noise components, and then do an iDCT. I'm hoping it will help things compress better and maybe get rid of some noise. And I just wanted to play with filtering in the frequency domain.

I'm using the code from Xvid though I think most of our utilities use the same DCT/iDCT routines from DVD2AVI, etc. But it is not obvious from the code of these utilities what the range of values is returned from DCT and I would like to scale my input parameters so they don't overflow.

Anybody happen to know how the returned values range from integer DCT? They look like maybe 9-10 bit signed values.

- Tom

edit: I released it at www.trbarry.com/DctFilter.zip , see later post below.

[SansGrip]

Anybody happen to know how the returned values range from integer DCT? They look like maybe 9-10 bit signed values.

I have no idea, but I'm really looking forward to trying this filter as I was contemplating the same thing myself .

[trbarry]

We talked about it here somewhere months ago but for some reason I got up this morning and started working on it.

I've got the DCT/IDCT part working but I'm trying to figure out how I want to handle the parms, and what I can get away with without having to deal with overflow problems.

I'd probably have it done tonight except Buffy & Smallville are on and even a nerd must have priorities.

- Tom

[SansGrip]

We talked about it here somewhere months ago but for some reason I got up this morning and started working on it.

I'm investigating wavelet transforms myself. They look interesting.

I'd probably have it done tonight except Buffy & Smallville are on and even a nerd must have priorities.

Especially a nerd with an HDTV .

[trbarry]

Especially a nerd with an HDTV

Sadly that is not true this week. Last Wed. my friendly local TV repairman took the convergence board for my RPTV back to the shop. He likely won't be back until Thursday afternoon.

And anyway Buffy is not in HDTV (drat!!!!!) and, while Smallville is, the WB channel here in Detroit is still having licensing problems about being able to broadcast HDTV. Something about Canadian interference across the river.

So I can't watch either of my 2 favorite shows in HDTV yet.

I think it is a conspiracy.

I usually like to do my web browsing and board posting from my recliner in front of a 55" screen, but I'm typing this now on a 20 year old NTSC TV that I'd kept around only to display the stock ticker. I've had to move my chair forward a few feet to even be able to read this.

Okay, end of OT rant. Back to Avisynth programming.

- Tom

[SansGrip]

Something about Canadian interference across the river.

I'm in Canada. Trust me, it's jealousy .

I usually like to do my web browsing and board posting from my recliner in front of a 55" screen

That's something I've yet to try. But I think if I had a cordless keyboard, mouse etc. and my TV out hooked up I'd never leave the house.

Back to Avisynth programming.

Good idea .

[Psyche]

Theoretically, DCT coefficients should fall in the [-2048, 2048] range, that is, 12 bits (11 bits plus sign).
BTW, I don't think removing noise *should* have an important impact in compression, because the quantizer does just that (well, MPEG quant at least) remove high frequency. BUT, maybe the motion estimation stage (which is done before getting into frequency domain) can benefit from the removed noise.
I don't know, let's wait a bit for the filter to be finished and then start making tests galore! Thanks to you both for the useful filters.

[-h]

Anybody happen to know how the returned values range from integer DCT? They look like maybe 9-10 bit signed values.

They will always be in the range [-2048, 2047].

Unless they're not, in which case they won't be

It may be possible to create coefficients outside that range with the implementation in XviD, however I strongly doubt that it will happen in pratcise. A couple pminsw/pmaxsw pairs will solve any problem with that however.

-h

[SansGrip]

A couple pminsw/pmaxsw pairs will solve any problem with that however.

You mean to say there are min/max instructions in MMX? Now I really want to learn .

[-h]

You mean to say there are min/max instructions in MMX? Now I really want to learn .

Well pminsw/pmaxsw are part of I-SSE, which doesn't exist on things like K6-2's or really old celerons (I think). It's not terribly hard to perform a min/max operation with vanilla MMX though, just using saturated addition/subtraction.

-h

[Psyche]

They will always be in the range [-2048, 2047].
Unless they're not, in which case they won't be

Very funny man!
BTW, we replied at the same time.

[trbarry]

Okay, thanks guys.

I guess it looks like the results might then use a signed 12 bits, and I'll save one more for good luck or -h's joke, which has now been cursed by Murphy's law.

But out of 16 that still leaves me 3 bits to play with so I guess I can multiply things by 8 and still get away with it.

- Tom

[trbarry]

I finally released a first version of this, for YV12 Avisynth 2.5 only.

It currently will work only on size multiples of 16x16, so you will have to crop or add borders as needed.

See:

www.trbarry.com/Readme_DctFilter.txt for the readme and
www.trbarry.com/DctFilter.zip for the dll and source.

I'm not sure yet about the application of this or any recommended parms (8 of them). I've attached an image (Help, mod ) of using

DctFilter(1,1,1,.7,.3,0,0,0)

which is probably too strong but I wanted to show the effect.

With some very quick testing, using that above set of parms Xvid will save more than 30% at an Xvid fixed quant=2 but that drops to a little over 25% at quant 3, and likely even less at higher quants.

I'm only starting to study this type of filtering but think it is very interesting.

Beware that if you crank the parms up too much the results will be both soft and blocky.

Somebody please let me know what you think of all this, or any recommended applications. I'm also curious if this can have application to anime since it is often full of hard edges, too high spatial frequencies. But I haven't tested any anime yet.

And, BTW, thanks to the Xvid guys, since I stole their DCT code.

- Tom

[trbarry]

I seem to be in a coffee driven manic state this morning, but just had another interesting idea.

If there were any demand for this type of filtering MPEG2DECx (or even ffdshow) could be made to perform this service almost for free, by gimmicking the DCT values straight from the file before finishing the decoding process. It would easily run real time.

But I don't know how that would interact yet with the motion estimation stuff, or what order all that is performed.

Wonder if it would work?

- Tom

[-h]

Wonder if it would work?

I think Bad Things would happen. Smearing over time and such. Then again there are only what, 5 or 6 consecutive P-frames in standard MPEG-2 encodes?

There'd be weird snap-back effects every I-frame and other odd things. What kind of filtering are you performing on the coefficients? The same filter could be used inside xvidcore after its own DCT is performed.

-h

[SansGrip]

@Tom

I'm getting an "illegal instruction" exception with the following script:

LoadPlugin("mpeg2dec3.dll")
LoadPlugin("dctfilter.dll")
Mpeg2Source("g:\ap\ap.d2v")
DctFilter(1, 1, 1, 1, 1, 1, 0.5, 0)
Levels(0, 1.5, 255, 16, 255)
ConvertToYUY2()
ConvertToRGB()

The frame size is 720x480. I get the exception whatever parameters I use, but no exception when I comment out DctFilter .

I'm on an Athlon XP, and am just about to fire up the debugger to give you more info.

[SansGrip]

The exception occurs at 'punpcklqdq' in fdct_mmx.asm. That's an SSE2 instruction and so won't work on my Athlon .

[trbarry]

SansGrip -

It sounds like I'm calling sse2 forward DCT when I should use sse. I'll either find the problem quickly or post a version that uses mmx only for now. But if that is the problem then the current version probably only runs on P4's, which is the only place I've tested it so far. But easy to fix.

-h

All I'm doing is multiplying each of the 64 values returned from FDCT by x/8 with a different x for each position, depending on the row/col parms.

I was thinking that if this was useful Xvid could even do it as a prefiltering, doing something similar. It would just apply it after the fdct but before the idct used to create the image for motion search.

But again, I haven't thought this through properly and even on decode I don't really know the order of applying dct vs motion estimation info. So there might be a clash somewhere that would make it harder to do.

Anyway I guess it's better to play with it for awhile as a standalone filter and see what works. (besides P4's)

- Tom

[trbarry]

Due to a couple of brain-dead mistakes I was doing the CPU type checking backwards. Also I've turned off sse2 support until I get a chance to look at it further. It hopefully should now run on all mmx or higher boxes.

So please try another copy: www.trbarry.com/DctFilter.zip

- Tom

[SansGrip]

Works great now! Even with quite aggressive settings it's hard to spot a difference between the two, even though compression would (theoretically) be greatly enhanced!

Is it my imagination, or do settings like:

DctFilter(1, 1, 1, 1, 0.5, 0, 0, 0)

actually sharpen details?

Edit: Here's a grab (1.1mb) to show what I mean. The top image is the original, the bottom is with the above settings. Certain areas show a very distinct sharpening effect, notably the pattern on the drapes, the fabric on the circular-back chair in the right hand side of the frame, and the right edge of the picture frame.

Edit 2: Now Tom's attachment has been approved I can see the same effect in it, too.