Integer DCT/iDCT filter questions

[trbarry]

Just out of curiousity I'm writing a filter that will do an integer DCT (Discreet Cosine Tranform), reduce some of the high frequency noise components, and then do an iDCT. I'm hoping it will help things compress better and maybe get rid of some noise. And I just wanted to play with filtering in the frequency domain.

I'm using the code from Xvid though I think most of our utilities use the same DCT/iDCT routines from DVD2AVI, etc. But it is not obvious from the code of these utilities what the range of values is returned from DCT and I would like to scale my input parameters so they don't overflow.

Anybody happen to know how the returned values range from integer DCT? They look like maybe 9-10 bit signed values.

- Tom

edit: I released it at www.trbarry.com/DctFilter.zip , see later post below.

[SansGrip]

Anybody happen to know how the returned values range from integer DCT? They look like maybe 9-10 bit signed values.

I have no idea, but I'm really looking forward to trying this filter as I was contemplating the same thing myself :).

[trbarry]

We talked about it here somewhere months ago but for some reason I got up this morning and started working on it.

I've got the DCT/IDCT part working but I'm trying to figure out how I want to handle the parms, and what I can get away with without having to deal with overflow problems.

I'd probably have it done tonight except Buffy & Smallville are on and even a nerd must have priorities. ;)

- Tom

[SansGrip]

We talked about it here somewhere months ago but for some reason I got up this morning and started working on it.

I'm investigating wavelet transforms myself. They look interesting.

I'd probably have it done tonight except Buffy & Smallville are on and even a nerd must have priorities. ;)

Especially a nerd with an HDTV :D.

[trbarry]

Especially a nerd with an HDTV

Sadly that is not true this week. Last Wed. my friendly local TV repairman took the convergence board for my RPTV back to the shop. He likely won't be back until Thursday afternoon.

And anyway Buffy is not in HDTV (drat!!!!!) and, while Smallville is, the WB channel here in Detroit is still having licensing problems about being able to broadcast HDTV. Something about Canadian interference across the river.

So I can't watch either of my 2 favorite shows in HDTV yet.

I think it is a conspiracy. :(

I usually like to do my web browsing and board posting from my recliner in front of a 55" screen, but I'm typing this now on a 20 year old NTSC TV that I'd kept around only to display the stock ticker. I've had to move my chair forward a few feet to even be able to read this.

Okay, end of OT rant. Back to Avisynth programming.

- Tom

[SansGrip]

Something about Canadian interference across the river.

I'm in Canada. Trust me, it's jealousy :D.

I usually like to do my web browsing and board posting from my recliner in front of a 55" screen

That's something I've yet to try. But I think if I had a cordless keyboard, mouse etc. and my TV out hooked up I'd never leave the house.

Back to Avisynth programming.

Good idea ;).

[Psyche]

Theoretically, DCT coefficients should fall in the [-2048, 2048] range, that is, 12 bits (11 bits plus sign).
BTW, I don't think removing noise *should* have an important impact in compression, because the quantizer does just that (well, MPEG quant at least) remove high frequency. BUT, maybe the motion estimation stage (which is done before getting into frequency domain) can benefit from the removed noise.
I don't know, let's wait a bit for the filter to be finished and then start making tests galore! Thanks to you both for the useful filters.

[-h]

Anybody happen to know how the returned values range from integer DCT? They look like maybe 9-10 bit signed values.

They will always be in the range [-2048, 2047].

Unless they're not, in which case they won't be :)

It may be possible to create coefficients outside that range with the implementation in XviD, however I strongly doubt that it will happen in pratcise. A couple pminsw/pmaxsw pairs will solve any problem with that however.

-h

[SansGrip]

A couple pminsw/pmaxsw pairs will solve any problem with that however.

You mean to say there are min/max instructions in MMX? Now I really want to learn :).

[-h]

You mean to say there are min/max instructions in MMX? Now I really want to learn :).

Well pminsw/pmaxsw are part of I-SSE, which doesn't exist on things like K6-2's or really old celerons (I think). It's not terribly hard to perform a min/max operation with vanilla MMX though, just using saturated addition/subtraction.

-h

[Psyche]

They will always be in the range [-2048, 2047].
Unless they're not, in which case they won't be
:D Very funny man!
BTW, we replied at the same time.

[trbarry]

Okay, thanks guys.

I guess it looks like the results might then use a signed 12 bits, and I'll save one more for good luck or -h's joke, which has now been cursed by Murphy's law. ;)

But out of 16 that still leaves me 3 bits to play with so I guess I can multiply things by 8 and still get away with it.

- Tom

[trbarry]

I finally released a first version of this, for YV12 Avisynth 2.5 only.

It currently will work only on size multiples of 16x16, so you will have to crop or add borders as needed.

See:

www.trbarry.com/Readme_DctFilter.txt for the readme and
www.trbarry.com/DctFilter.zip for the dll and source.

I'm not sure yet about the application of this or any recommended parms (8 of them). I've attached an image (Help, mod :o ) of using

DctFilter(1,1,1,.7,.3,0,0,0)

which is probably too strong but I wanted to show the effect.

With some very quick testing, using that above set of parms Xvid will save more than 30% at an Xvid fixed quant=2 but that drops to a little over 25% at quant 3, and likely even less at higher quants.

I'm only starting to study this type of filtering but think it is very interesting.

Beware that if you crank the parms up too much the results will be both soft and blocky.

Somebody please let me know what you think of all this, or any recommended applications. I'm also curious if this can have application to anime since it is often full of hard edges, too high spatial frequencies. But I haven't tested any anime yet.

And, BTW, thanks to the Xvid guys, since I stole their DCT code. ;)

- Tom

[trbarry]

I seem to be in a coffee driven manic state this morning, but just had another interesting idea.

If there were any demand for this type of filtering MPEG2DECx (or even ffdshow) could be made to perform this service almost for free, by gimmicking the DCT values straight from the file before finishing the decoding process. It would easily run real time.

But I don't know how that would interact yet with the motion estimation stuff, or what order all that is performed.

Wonder if it would work? ;)

- Tom

[-h]

Wonder if it would work? ;)

I think Bad Things would happen. Smearing over time and such. Then again there are only what, 5 or 6 consecutive P-frames in standard MPEG-2 encodes?

There'd be weird snap-back effects every I-frame and other odd things. What kind of filtering are you performing on the coefficients? The same filter could be used inside xvidcore after its own DCT is performed.

-h

[SansGrip]

@Tom

I'm getting an "illegal instruction" exception with the following script:


LoadPlugin("mpeg2dec3.dll")
LoadPlugin("dctfilter.dll")
Mpeg2Source("g:\ap\ap.d2v")
DctFilter(1, 1, 1, 1, 1, 1, 0.5, 0)
Levels(0, 1.5, 255, 16, 255)
ConvertToYUY2()
ConvertToRGB()


The frame size is 720x480. I get the exception whatever parameters I use, but no exception when I comment out DctFilter :(.

I'm on an Athlon XP, and am just about to fire up the debugger to give you more info.

[SansGrip]

The exception occurs at 'punpcklqdq' in fdct_mmx.asm. That's an SSE2 instruction and so won't work on my Athlon :(.

[trbarry]

SansGrip -

It sounds like I'm calling sse2 forward DCT when I should use sse. I'll either find the problem quickly or post a version that uses mmx only for now. But if that is the problem then the current version probably only runs on P4's, which is the only place I've tested it so far. But easy to fix.

-h

All I'm doing is multiplying each of the 64 values returned from FDCT by x/8 with a different x for each position, depending on the row/col parms.

I was thinking that if this was useful Xvid could even do it as a prefiltering, doing something similar. It would just apply it after the fdct but before the idct used to create the image for motion search.

But again, I haven't thought this through properly and even on decode I don't really know the order of applying dct vs motion estimation info. So there might be a clash somewhere that would make it harder to do.

Anyway I guess it's better to play with it for awhile as a standalone filter and see what works. (besides P4's)

- Tom

[trbarry]

Due to a couple of brain-dead mistakes I was doing the CPU type checking backwards. Also I've turned off sse2 support until I get a chance to look at it further. It hopefully should now run on all mmx or higher boxes.

So please try another copy: www.trbarry.com/DctFilter.zip

- Tom

[SansGrip]

Works great now! Even with quite aggressive settings it's hard to spot a difference between the two, even though compression would (theoretically) be greatly enhanced!

Is it my imagination, or do settings like:

DctFilter(1, 1, 1, 1, 0.5, 0, 0, 0)

actually sharpen details?

Edit: Here's a grab (http://www.jungleweb.net/~sansgrip/DctFilter_1.png) (1.1mb) to show what I mean. The top image is the original, the bottom is with the above settings. Certain areas show a very distinct sharpening effect, notably the pattern on the drapes, the fabric on the circular-back chair in the right hand side of the frame, and the right edge of the picture frame.

Edit 2: Now Tom's attachment has been approved I can see the same effect in it, too.

[trbarry]

Is it my imagination, or do settings like:

DctFilter(1, 1, 1, 1, 0.5, 0, 0, 0)

actually sharpen details?

I don't think so, except maybe for macro block boundaries. But I'm not sure yet exactly what it does. I wrote it to find out. ;)

The only time I thought it sharpened was earlier when I did not realize that in 2.5 Avisynth StackVertical puts the first clip on the BOTTOM, and I had them backwards.

So I put subtitles on them to be sure. I think I already commented on that somewhere today.

But I'm glad it works now. I'm really sort of surprised how much info we can throw away before it starts to obviously affect the PQ. I know much of it would be pitched out during the quantization process anyway but this is different. Instead of quantizing a frequency component for lesser accuracy we are just keeping the accuracy of a diminished component. (which is quantized later if needed)

I'd almost be willing to bet you could store 6x6 macro blocks and supply a boundary of 2 rows and columns of zeros later and nobody would notice. But maybe those rows and columns usually get quantized to zeros anyway, especially with the H263 quant matrix. This just helps it along a little.

- Tom

[SansGrip]

The only time I thought it sharpened was earlier when I did not realize that in 2.5 Avisynth StackVertical puts the first clip on the BOTTOM, and I had them backwards.

Ah hah! That's the same mistake I made :D.

I'm really sort of surprised how much info we can throw away before it starts to obviously affect the PQ.

I'm going to have to run some more tests now that I know which way is up :rolleyes: ;).

You're right, though -- this is a very "natural" way to improve compressibility. If you see what I mean.

[iago]

@Tom

I just want to say (based on a few constant quant 2 tests with XviD and mpeg quantization) that this filter seems like a magical tool! ;)

~30% compressibility gain and almost no quality loss with
----------------------------
DctFilter(1,1,1,1,0.5,0,0,0)
----------------------------
in the script. However this setting resulted in terrible EE together with LanczosResize(576,304), which I'll try to get rid of from either with UnFilter or UnDot following the DctFilter in the script.

Thanks for all your great work,

best regards,
iago

[trbarry]

iago -

I didn't mention it before but this filter should probably be run LAST, after everything else.

The reason is that it creates values that a subsequent DCT operation will use to create lots of zeros and low values and we probably wouldn't want to mess that up.

Did you run it after resize? If so, maybe you will just have to tone it down a bit.

Or maybe run UnFilter(-,-) or C3D right before it. Or even after, who knows? Nobody knows how this works yet. :)

- Tom

[iago]

@Tom,

Yes, DctFilter was the last chain in my script ;). Now I'm trying with some less agressive settings as below and with h263 quantization type:

----------------------------------------------
LoadPlugin("C:\FILTERS-YV12\MPEG2Dec3.dll")
LoadPlugin("C:\FILTERS-YV12\UnFilter.dll")
LoadPlugin("C:\FILTERS-YV12\DctFilter.dll")
mpeg2source("D:\RIP\RIP.d2v")
crop(2,16,-2,-16)
BicubicResize(576,304,0,0.7)
LumaFilter(-2)
UnFilter(-5,-5)
DctFilter(1,1,1,1,.7,.5,.2,0)
-----------------------------------------------

With the above script, I'll run a two pass encode of a pretty hard-to-compress movie aiming for 1CD, even without b-frames! ;)

I'll report back the results.

ciao,
iago

[SansGrip]

I just tried this with a very noisy source and it doesn't really touch the noise, at least in any noticible way. But that's what smoothers are for, right? ;)

[SansGrip]

I'm harbouring some hope that this filter might help reduce mosquito noise on resulting low bitrate MPEG-1 encodes. Since the Gibbs effect is a high-frequency phenomenon, maybe we'll be able to pull down the high-frequency information enough to at least subdue it somewhat...

[trbarry]

I just tried this with a very noisy source and it doesn't really touch the noise, at least in any noticible way. But that's what smoothers are for, right?

Actually that surprises me a lot. I would have thought that much noise would appear as high frequency stuff this filter would get rid of. But I haven't tried it.

- Tom

[trbarry]

For mosquito noise you might also want to try UnDot first.

- Tom

[SansGrip]

Actually that surprises me a lot. I would have thought that much noise would appear as high frequency stuff this filter would get rid of.

I tested briefly with something like 1,1,1,1,0,0,0,0 (and even with 1,1,1,0,0,0,0,0) but the noise was still there. Maybe I'm misunderstanding how the parameters work, but wouldn't those get rid of a lot of high-frequency information, including noise?

[SansGrip]

For mosquito noise you might also want to try UnDot first.

I might be mistaken, but I was under the impression that UnDot can help existing mosquito noise. I'm looking for a way to filter "proactively" to reduce the appearance of mosquito noise in the encode (sort of like what Blockbuster does for DCT blocks).

[trbarry]

You are probably correct there. I misunderstood what you were trying to do.

- Tom

[iago]

@SansGrip

Such a compressibility gain without losing the details and without losing the source noise; and _you_ still complain about it!? :D

iago

[milan]

I've just added DCT/iDCT filter to ffdshow. GoFilterIt method is ported without any change and it uses unmodified XviD DCT/IDCT functions. Only luma plane is filtered for now. No integration with libavcodec decoding routines yet, I'll try to implement this later.
Simple configuration page is present too, just allowing to enter 8 floating point numbers. Maybe later I'll add possibility to enter/load/save full 8x8 matrix.

One question to trbarry: is your filter able to handle parameters higher than 1?

[serbersan]

Maybe a stupid question. I've recently studed about filters in the frequency domain....

And I don't understand why those type of filters doesn't be used until now?? Difficult to implement???

Good Work Tom¡¡

[trbarry]

One question to trbarry: is your filter able to handle parameters higher than 1?

milan -

Don't know yet. ;)

From what I've been told above in this thread about the max/min values from DCT is correct then nothing should overflow until parms get > 1.414. I suppose that would also depend upon whether the iDCT had problems with larger values.

But I haven't had a chance to test it and find out what it does. I'd wondered if it would act like adding noise in post processing to do something like:

DctFilter(1,1,1,1,1,1,1.3,1.3)

Maybe making things look more realistic after upscaling but that's just a guess. And possibly it would also have a sharpening effect but I really can't predict that one at all.

- Tom

[Guest]

Originally posted by serbersan
Maybe a stupid question. I've recently studed about filters in the frequency domain....

And I don't understand why those type of filters doesn't be used until now?? Difficult to implement??? Not difficult to implement but highly CPU intensive. Such filters tend to be slow and Doom9'ers do not like to wait. :)

Also, most of the things we need to do don't require frequency domain processing, although it's certainly possible that some things could be done better there, at the expense of time.

[trbarry]

I have a question myself for anyone who understands DCT/IDCT etc.

Say, for a too extreme example, you had decided that using

DctFilter(1,1,1,1,0,0,0,0)

seemed to give acceptable results on some clip. That is, representing the clip with all the higher frequencies zeroed out in 3/4 of the 8x8 DCT output block. This would leave non-zero values only in the top left 4x4 corner.

But there is nothing magical about an 8x8 macro block except we have fast code for the 8x8 DCT & IDCT. So if you had a 4x4 IDCT then it could be performed on the intermediate results of the above, giving a clip with only 1/4 the number of pixels. That is, do the 8x8 DCT, zero the extra values (or pretend you did) and then do a 4x4 IDCT resulting in a downsized clip with only half the H and V dimensions. Encode that clip with Xvid. Then reverse the process at display time.

Since not counting integer round off's and quantizations DCT is a reversible transformation then it seems that if the results of running DctFilter above was acceptable then so should the downsize/upsize be okay in that fashion.

Anybody know about this sort of thing? Am I asking for too much of a free lunch?

- Tom

[-h]

But there is nothing magical about an 8x8 macro block except we have fast code for the 8x8 DCT & IDCT. So if you had a 4x4 IDCT then it could be performed on the intermediate results of the above, giving a clip with only 1/4 the number of pixels. That is, do the 8x8 DCT, zero the extra values (or pretend you did) and then do a 4x4 IDCT resulting in a downsized clip with only half the H and V dimensions. Encode that clip with Xvid. Then reverse the process at display time.

I was initially going to say this wouldn't work due to frequency mismatch, but then remembered that the top 3/4 of the coefficients represent signals whose resolution is too high for a 4x4 block to represent. So yes this reconstruction would "work" for performing downsizing.

I'm not so sure that it's a good idea though :) XviD will encode the 4x4-blocked image with its native 8x8 DCT, destroying a fair bit of the signal along the way. If the clip produced by only leaving 1/4 of the coefficients is watchable, you would get similar quality by just resizing to a quarter of the image size and blowing it back up on playback.

-h

[trbarry]

I'm not so sure that it's a good idea though XviD will encode the 4x4-blocked image with its native 8x8 DCT, destroying a fair bit of the signal along the way. If the clip produced by only leaving 1/4 of the coefficients is watchable, you would get similar quality by just resizing to a quarter of the image size and blowing it back up on playback.

Yep. But I was betting that since there would only be 1/4 as many pixels Xvid could encode it at a MUCH lower average quant, not to mention how much faster it would go at only 1/4 res. (or maybe 9/16 res)

Really this would just be another way of downscaling to make the file size smaller. But I was wondering if this method would somehow be more reversible if we eventually intended to upscale again at display time.

In principle there is probably no reason now that you couldn't double the size of any clip at display time by doing a 4x4 fDCT, padding it with 0's like DctFilter, and then doing an 8x8 iDCT.

But I have no idea if it would look any better than the hardware scaling on the average Radeon card and it would certainly take longer.

But if we had a more reversible scaling method it might be worth it. That is, optimize the downscale method with the intention of upscaling again at display time with a known method optimized for that special purpose. Playing with DctFilter can maybe show something of what we could get away with here.

- Tom

[SansGrip]

Such a compressibility gain without losing the details and without losing the source noise; and _you_ still complain about it!? :D

I'm not complaining! I think it's great! :D

The reason I was eager to test it on noisy sources is that I was actually just about to try this kind of frequency filtering myself (I'm investigating wavelets, too) because I figured it would work well on noise.

Though this just reinforces my developing belief that spatial noise reduction is a flawed concept.

And before you say anything else, that doesn't mean I think _this_ filter is flawed, because it's not a spatial denoiser :p.

[SansGrip]

Anybody know about this sort of thing? Am I asking for too much of a free lunch?

I don't really know about this sort of thing, but I don't see how one would gain anything from doing it, since you already got the compressibility gain from the zigzag RLE of all those zeroes.

Or perhaps I'm misunderstanding what you wrote. I'll go read it again :).

[jarthel]

how about a yuv2 version for people who sticked with pre-2.5 avisynth? :)

jayel

[trbarry]

how about a yuv2 version for people who sticked with pre-2.5 avisynth?

jarthel -

Probably not right away. I consider this still experimental and I'm not ready to be maintaining 2 versions. And even YUY2 users can just convert to YV12 once they are using a production version of Avisynth 2.5, once that gets here. There shouldn't be any quality loss doing that because it is going to happen as soon as you get to Xvid anyway.

I find I like working in YV12 format better, it gives better and faster results, and most of my input data is in that format anyway. So as long as there are no real show stoppers I'll probably stick mostly to that.

- Tom

[jarthel]

thanks Tom :)

I also await for the production or even beta release of Avisynth.

[Marc FD]

hi ^^

sorry, i think i'm very dumb because i don't see any interrest in this filter. reminds me the vp3 internals.

BTW, please don't speak of quant 2 compressibility.
it means nothing when you play with spacial freq.
you can lower it 2x in real time. that's not hard.

i think you should never use anything DCT-based out of a DCT-based codec, but for ultra hardcore denosing. (btw, wavelets are really much better for that)

just my 2 euros cents ;)

[SansGrip]

Here's an Avisynth 2.0 YUY2 version for those of us still stuck in the stone age ;). I didn't update the readme, but the usage is the same.

I've sent the new code to Tom and am awaiting his judgement on it, so this may or may not make it into his release. It works for me, though, and at least the 2.0 crowd has something new to play with :).

You can get it here (http://www.jungleweb.net/~sansgrip/DctFilter_YUY2.zip) until the attachment is modded.

Have fun.

[trbarry]

BTW, please don't speak of quant 2 compressibility.
it means nothing when you play with spacial freq.
you can lower it 2x in real time. that's not hard.

i think you should never use anything DCT-based out of a DCT-based codec, but for ultra hardcore denosing. (btw, wavelets are really much better for that)

Marc -

Dunno yet.

It looks like it can save a bunch when using low quants and probably not as much with higher quants.

But with a normal 2 pass encode the actual quants used probably wanders around quite a bit. So I'm hoping that for high quality encodings what I save on the low quants will lower the average quant of the other blocks and frames.

Still playing, but I'm hoping this will be good for more than just noise removal (which is nice too). It is easily possible that this won't be anything that could not already be accompliched in Xvid with modulated/custom quants, but it's another tool in the box and more visible to tinker with. ;)

- Tom

[SansGrip]

trbarry: It is easily possible that this won't be anything that could not already be accompliched in Xvid with modulated/custom quants

Well, let's not forget that not everyone is using XViD. This allows one to monkey around with the frequency domain even when the encoder's quant matrix is hardcoded.

What's more, it might even help compressibility wrt encoders that don't use the DCT model. I can't think of any right now, but that's not the point ;).

[trbarry]

SansGrip -

I'm still testing to see where/if it helps compressibility. Hopefully does.

BTW, my last post crossed with yours and I didn't even notice that you had released the YUY2 version above.

All -

Hopefully I'll have SansGrip's YUY2 version also included in mine by later tonight. I know some folks have been asking for it.

- Tom

[FuPP]

Don't know for xvid, but using this filter with mpeg2 compression (svcd), I gain about 10% ( using DctFilter(1,1,1,1,1,.5,.5,0) ) without visible loss of details.

Thx a lot Tom :) !

Regards

FuPP.

[fraatz]

Hi!
I recently made some tests with the DctFilter and have to say it's quite cool. Thanks TBarry!
I used C3D in my tests with the MovieHQ- preset and I found that both together limit reasonable values for DctFilter a lot.
I tried to find the best settings for keeping the Iimage as original as possible while increasing compressibility as much as possible.
These where my settings:

LoadPlugin("D:\Programme\DVDRip\AVSPlugins\MPEG2Dec3.dll")
LoadPlugin("D:\Programme\DVDRip\AVSPlugins\Convolution3D.dll")
LoadPlugin("D:\Programme\DVDRip\AVSPlugins\DctFilter_YUY2.dll")
LoadPlugin("D:\Programme\DVDRip\AVSPlugins\MSharpen.dll")
MPEG2source("E:\jhb.d2v",lumoff=-3,lumgain=138)
crop(1,75,717,423)
Convolution3d(preset="movieHQ")
#MSharpen(20,50,true,false,false)
BilinearResize(608,240)
#DctFilter(1,1,1,1,1,.5,.5,0) #2
#DctFilter(1,1,1,1,.7,.5,.2,0) #1
#DctFilter(1,1,1,1,.5,0,0,0) #3
#DctFilter(1,1,1,1,1,.7,.5,0) #4
#DctFilter(1,1,1,1,1,1,.5,0) #5
#DctFilter(1,1,1,1,1,1,.7,0) #6

Compressibility gain was huge, about 30%, especially with dctfilterset #3. But imagequality suffered a lot, many details were lost..
So I decreased DctFilter- Influence until loss of detail became neglectable. This happened with filterset #5 but i still got a compressibillity gain of 10%. Amazingly good with a already *very* clean source!!
Now I tried to get more detail to the image, to fill the "saved" bits with information. MSharpen(20,50,true,false,false) together with DctFilter(1,1,1,1,1,1,.5,0) was very good. These settings gave me a little lower filesize as without DctFilter/MsSharpen at all. But the picture was as sharp as BicubicResize 0,0.7 !!

All test were made with xvidapi3 without new features.

I made a three twopass encodes with about 4250 frames an I used xvid with bframes 3/150/100, qpel, chroma me and lumi. to compare quant redistribution:

1. Avs- Script as above:

Q:2:1
Q:3:62
Q:4:2574
Q:5:2077
Q:6:52

2. + MSharpen(20,50,true,false,false) together with DctFilter(1,1,1,1,1,1,.5,0) :

Q:2:1
Q:3:30
Q:4:2072
Q:5:2546
Q:6:117

3. Script as above only + DctFilter(1,1,1,1,1,1,.5,0) :

Q:2:1
Q:3:164
Q:4:3282
Q:5:1295
Q:6:24

Number 3 gave very good quality, interestingly the mosquitonoise was reduced quite well!!

If you are interested in my test I can attach the screenshots I made....
Thanks again for this filter !!!
Try DctFilter(1,1,1,1,1,1,.5,0) with C3D. You will not be dissapointed.

-f

[TheUnforgiven]

big thanks to tom.
great filter

[iago]

Tom,

I think with the help of DctFilter now it's possible to put "any" movie on 1CD with a reasonable resolution of 576*... (maybe even more) and a 128 Kbps audio track, without ruining the encode and keeping decent/acceptable quality.

However (you may/should also have noticed that) the only thing that I still cannot eliminate is what I can call the "staircase" effect when DctFilter is used. Using BilinearResize, UnFilter(-,-), UnDot(), etc. have not worked for me yet. (Though I must admit that this effect is one of the most acceptable artifacts for my eyes ;))

Any ideas or recommendations from the author about that? ;)

And thanks once again for this amazing filter!

iago

[trbarry]

However (you may/should also have noticed that) the only thing that I still cannot eliminate is what I can call the "staircase" effect when DctFilter is used. Using BilinearResize, UnFilter(-,-), UnDot(), etc. have not worked for me yet. (Though I must admit that this effect is one of the most acceptable artifacts for my eyes )

iago -

I'm still playing with this and don't really understand all the ramifications yet. It seems DctFilter does something similar but not quite the same as raising quants towards the edges of a custom quant matrix, much as the difference between the mpeg and H263 matrices. But it is reducing the value of the higher frequency components, not the precision.

I think if you over do it you can effectively create macroblocks, in sizes that are 2x2, 4x4, 8x8, etc. And offhand I can't think of any fix for that besides backing off a bit on the DctFilter parms or maybe softening even more first with something like C3D, possibly in conjunction with lowering the resolution a bit. Still no magic bullet.

Just a guess.

- Tom

[Marc FD]

okay guys. i intensively tested DCT lowering in XviD.
i've hacked XviD a very clean way to have 192 config paramters + quant/movement/vop adaptive dct-filtering.

i've run hours of test, and i need to admit that the idea looked attractive (-30% at quant2) but in fact it's not good : using bigger quantisation makes much crisper image (more noisy too) and heavy prefiltering makes images smoother with lower quants.

i can provide a hacked build for testing, but it really doesn't worth it.
i tested the idea to ill ^^.

topic close.

Regards,
MarcFD

[-h]

Perhaps better than filtering on fixed 8x8 boundaries (remember that motion compensation will be copying blocks from across those boundaries, so the filtering will not have the intended outcome all the time), would be to run a lowpass filter over the entire image.

-h

[Marc FD]

no, i used that between fdct and quantisation. (on inter coefs too)
using a lowpass filter is far too destructive for low-mo.

[-h]

Perhaps use a transformation other than the DCT, on blocks larger than 8x8 (i.e. the entire image).

-h

[trbarry]

Marc -

Those are disappointing results. :(

I think all that is maybe because Xvid is already very good at jiggering quants if you let it (don't fix quants), so it is harder to compete.

But I'm still playing with things based upon these principles and I certainly haven't given up yet. I'll post some test results if I get anything useful.

As I said on earlier posts here, this is largely for curiousity and I'm not sure what the recommended usage is yet. But it is another tool that probably can be useful sometimes, and not just for spinning off a high quality fixed quant=2 work file.

Too bad Xvid doesn't support quant 1. I'll bet the savings would look even more dramatic. ;)

I wonder how DctFilter would perform in (-h)uffYV12?
(a mythical non-dct based compression method) ;)

Along the same lines right now I'm trying to make a quickie resize filter now based upon an integer transform that works similar to DctFilter.

- Tom

[MfA]

Block based filtering of DCT coefficients should be left to the codec IMO. Normal low pass filtering is easier to do in the spatial domain.

BTW the MPEG matrix does not exactly lowpass filter the coefficients ... high frequency coefficients will more often fall in the deadzone and be filtered out so much is true, but in the cases where they fall outside of the deadzone they will tend to only do so just. Coefficients just outside of the deadzone tend to actually be amplified from their true value after dequantization.

[trbarry]

Coefficients just outside of the deadzone tend to actually be amplified from their true value after dequantization.

I think that is one of the main differences between DctFilter and normal quantization.

- Tom

[MfA]

BTW this thesis (http://www-isl.kaist.ac.kr/Papers/PhDTheses/phd_sdkim.zip) presents some result for DCT domain pre-filtering inside the codec. There it makes more sense to use the DCT representation, although even there the majority of pre-filtering papers have used spatial filters.

[trbarry]

MfA -

Thanks. Great paper. It not only gives a much more sophisticated analysis of DctFilter than I would have been capable of, it also contains very good explanations of both the deblocking and deringing process for anyone interested in those.

And I like the idea that for performance you could actually build the DctFilter scaling right into the scaling coefficients of the DCT itself, for "marginal" extra overhead.

- Tom

[Taranli Maren]

I was curious, and tried this filter on an anime source, a clean source with many crisp lines. I'm sure you all expect this, but it created a sort of light aura around the black edges at low levels (1,1,1,1,1,1,.5,0) and (1,1,1,1,1,1,1,0). At higher levels it created normal mosquito noise around those edges. It did have this affect more around sharp high contrast edges than weaker ones. This is all in preview before encoding. I've not tested the compressibility difference, but will be getting to that next.

[Marc FD]

>BTW this thesis presents some result for DCT domain pre-filtering inside
>the codec. There it makes more sense to use the DCT representation,
>although even there the majority of pre-filtering papers have used
>spatial filters.

do you think it's a good idea to try this one into XviD ?
i read the paper and it seems great.

>Thanks. Great paper. It not only gives a much more sophisticated
>analysis of DctFilter than I would have been capable of, it also
>contains very good explanations of both the deblocking and deringing
>process for anyone interested in those.

tom, you rewrote the dering part in nic's pp, can you say me if it's the proposed one or the vm8 one in nic's implementation ?

>And I like the idea that for performance you could actually build the
>DctFilter scaling right into the scaling coefficients of the DCT itself,
>for "marginal" extra overhead.

that's what i tried, but my approach was too basic.
i'm gonna try this method into XviD now, i wonder how well it'll work ^_^

[mikeson]

@Marc FD:

i'm gonna try this method into XviD now, i wonder how well it'll work ^_^

Any progress or even some results? ;)

So please let us know when you have something. I'm very interested in this filter and its implementation inside XviD (if it is worth it).

[trbarry]

do you think it's a good idea to try this one into XviD ?
i read the paper and it seems great.

>Thanks. Great paper. It not only gives a much more sophisticated ....

tom, you rewrote the dering part in nic's pp, can you say me if it's the proposed one or the vm8 one in nic's implementation ?

Marc -

It would be interesting in Xvid but Xvid has usually tried more for standard MPEG4 compliance so I'm not sure how happy they would be complicating the core functions for a non-complient mod, especially if we are not sure of the benefits yet.

I think the real savings in something like this would maybe be if a decoder also had an option for a 6x6 or a 4x4 DCT/iDCT. Then it could encode in a downsized format that was optimized to be upsized again at display time. This process would take no extra overhead if done by the codec. But it would be non-standard. I'm writing a filter to test this. But it would be non-standard to do it more efficiently in the codec.

I'm afraid I only skimmed the first part of the deringing section in that paper. All I did in Nic's decoder was create the MMX optimized asm version from the C code. I didn't write the orig code. I think it was based upon the old OpenDivx code.

- Tom

[Marc FD]

i studied it a bit...
it could be good to encode in a very noisy environnement in real-time for very low bitrate/res. that's not what XviD is designed for.

RRVs will do the job better ^_^

[MfA]

RRVs is a mostly seperate issue to filtering, and I would not be so fast to sing it praises until you try it out. Besides, it is not part of the advanced simple profile ...

Wether you do your filtering in the DCT domain or in the spatial domain I think one lesson from the paper is pertinent :

"Even in an experiment based on the assumption that quantization noise is absent, it is also shown that pre-filtering for motion-compensated error blocks is more desirable than spatial domain filtering for original image blocks."

I think filters which adapt to the quantizers and/or motion should always work on the DFD, otherwise you run the risk of filtering being counterproductive. Say we have a MB which happens to move fast in a given frame with lots of motion, and a corresponding high quantizer, but which is perfectly predictable by motion compensation ... filtering the original block will increase both rate and distortion, the exact opposite of what you want to accomplish. If you filter the DFD (displaced frame difference, what is left of the frame after motion compensation) however the MB will simply not be coded, a filtered 0 signal stays a 0 signal, no distortion and the optimum number of bits used to code it.

Other kinds of filters (denoising for instance) can safely use the original images of course ... but I think you could almost always roll them into a DFD filter.

[Marc FD]

>RRVs is a mostly seperate issue to filtering, and I would not be so
>fast to sing it praises until you try it out.

i just think it'll be better if you _really_ want to compress more.
or go for 1000 kbps encodes, and any preprocessing other than light denoinsing is useless IMHO.

>Besides, it is not part
>of the advanced simple profile ...

if it works with XviD it's enough for me ^^

>but I think you could almost always roll them into a DFD filter.

you think DFD denoising will be better ?
but waht if you prefilter _before_ me ??

[MfA]

If the motion estimation was done accurately (which is not a given inside the codec given rate restraints, and the sensitivity of ME for the noise you are trying to remove) the DFD tells you a lot about what is noise and what is signal ... from a single image you can only guess what is noise.

That said though image based filtering (where the filter strength is independent of motion and quantizers) can improve ME (SAD is noise sensitive) and it will usually improve compressibility ... so it can be usefull. It just cant do the things motion compensated spatiotemporal filtering and adaptive R-D optimized pre-filtering can do, it is a lot easier though.

If you want to go that route you will probably want to consider non-linear filters, they can handle edges better than linear filters ... something like this (http://133.23.229.11/~ysuzuki/Proceedingsall/ICASSP95/pdf/ic952193.pdf) (this is old, and might be totally outdated).

Here (http://ivpl.ece.nwu.edu/Research/Current/PrePost/PrePost.html) is another bit of research which went the DFD route, but it uses spatial instead of DCT domain filtering.

Marco

PS. it working for XviD is fine and well, but if XviD is going to ignore profiles it might as well ignore strict standard compliance for things as not following a VOL header with an I-frame too ... lots of hardware decoders support nothing beyond advanced simple profile.

[Marc FD]

>DFD tells you a lot about what is noise and what is signal

that's what i thought too. i'll try an approach were i optimise pre-filtering using DFD.

> It just cant do the things motion compensated spatiotemporal filtering
> and adaptive R-D optimized pre-filtering can do, it is a lot easier
> though.

that's why i'm playing with that ^^

>If you want to go that route you will probably want to consider
>non-linear filters, they can handle edges better than linear filters
>... something like this (this is old, and might be totally outdated).

i'll look at it.

>Here is another bit of research which went the DFD route, but it uses
>spatial instead of DCT domain filtering.

i personnaly prefer a spatial approach.

thx for ressources, i'll code a bit on XviD today ^^

>PS. it working for XviD is fine and well, but if XviD is going to
>ignore profiles it might as well ignore strict standard compliance for
> things as not following a VOL header with an I-frame too ... lots of
>hardware decoders support nothing beyond advanced simple profile.

that's the problem, advanced simple profile is a bloated part of the standart. only this profile allow bvops/qpel/gmc :-/
BTW, what's the profile for H.264 tools ?? maybe they added more powerfull profiles in MPEG-4 VM 10 ??

marc ^^

[MfA]

Originally posted by Marc FD
i'll try an approach were i optimise pre-filtering using DFD.

Whichever way you do it, if you work on the original image blocks unless you check the DFD before and after filtering (and just throw the filtered results away if they "increased" the DFD) it is very hard to determine what the effect of your filter will be.

A problem you wont have with non adaptive image based filters such as the nonlinear one I mentioned (which on reflection probably wasnt a very good idea, I doubt it will do better than smartsmoother) because they filter all frames equally, if the reference images were identically filtered there is no problem. But these kind of filters dont really need to be tightly bound with XviD.

i personnaly prefer a spatial approach.

Shrug, as long as it gets better results ... it is much more work than simply scaling the DCT coefficients though.

thx for ressources, i'll code a bit on XviD today ^^

XviD needs a plugin API more than filters BTW :)

that's the problem, advanced simple profile is a bloated part of the standart. only this profile allow bvops/qpel/gmc :-/
BTW, what's the profile for H.264 tools ?? maybe they added more powerfull profiles in MPEG-4 VM 10 ??

It has an entirely new set of profiles which I do not know from memory, it will have the same "issues" though ... it will have simple and advanced profiles, and while from a software point of view it is attractive to just mix and match that would make interoperability with hardware solutions impossible.

[Marc FD]

>Whichever way you do it, if you work on the original image blocks
>unless you check the DFD before and after filtering (and just throw
>the filtered results away if they "increased" the DFD) it is very hard
>to determine what the effect of your filter will be.

that's what i do. the filtering is selected with DFD, and filtering is spatial only (to avoid dumb ghosting).

>Shrug, as long as it gets better results ... it is much more work than
>simply scaling the DCT coefficients though.

i already tried DCT scaling and i don't like the result at all.
IMHO, we should not touch fDCT/Q for fine pre-processing.

BTW, i tried a bit basic DFD filtering (3x3 blur), and it gives 10% comp at q2/ 5% at q10. not bad, but the image is not really better (nor worse ^^), so i'll try my first idea. (who's just harder to implement)

[MfA]

Originally posted by Marc FD
that's what i do. the filtering is selected with DFD, and filtering is spatial only (to avoid dumb ghosting).

That is not what I meant ... just knowing that there is an error before filtering is not enough. If you filter the original image block with a "filter strength" determined by the error before filtering you have no idea what is going to happen to the error after filtering, if it takes less bits to code it is by sheer luck :)

BTW, i tried a bit basic DFD filtering (3x3 blur), and it gives 10% comp at q2/ 5% at q10. not bad, but the image is not really better (nor worse ^^), so i'll try my first idea. (who's just harder to implement)

Well unless it was noisy to start with it isnt supposed to look better.

Marco

[Marc FD]

>That is not what I meant ... just knowing that there is an error before
>filtering is not enough. If you filter the original image block with a
>"filter strength" determined by the error before filtering you have no >idea what is going to happen to the error after filtering, if it takes
>less bits to code it is by sheer

i'm not sure i understand. i DFD-optimise my smoothing (filtering is like avisynth spatialsoften), so the smoothing is _always_ reducing DFD after filtering in intensity. of course, it doesn't mean it would have the same inpact en frequency, but of course it will help.

>Well unless it was noisy to start with it isnt supposed to look better.

yes of course. but my source is clean DivX-SBC (300 Ko/s) and i get now 20% compressibility improvement with slightly reduced ringing at quant2. not bad, when i think using 2dcleaner (3x3,16) on the same source give only 5% compressibility improvement.
at quant 10, i gain only 5% compressibility.

BTW, i'm testing on anime, it's of course very different from movies.
but i don't have movies at home so i think i'll MMX-optimise my stuff and make some "XviD with preprocessing" builds for friends ^_^

anybody want to test here ??

[MfA]

Originally posted by Marc FD
i'm not sure i understand. i DFD-optimise my smoothing (filtering is like avisynth spatialsoften), so the smoothing is _always_ reducing DFD after filtering in intensity.

That is only true if your filtering works on the DFD itself. If you smooth the present frame and subtract the motion compensated reference again you have no idea wether the new DFD will actually be smoother ...

Say that after motion compensation you have a DC level shift for a MB, with a high SAD, coding that without filtering would just give you a DC component to code. If you filter the image however and calculate a new DFD it will actually become more complex taking more bits to encode.

Marco

[Marc FD]

>That is only true if your filtering works on the DFD itself. If you
>smooth the present frame and subtract the motion compensated reference
>again you have no idea wether the new DFD will actually be smoother ...

no, i don't work on the DFD.
i soften in a temporary location, and then i compare current and cleaned current vs compensated reference. i always keep the best.

this way i get the smallest DFD intensity possible, but DFD is still "natural" : i don't filter it.

>Say that after motion compensation you have a DC level shift for a MB,
>with a high SAD, coding that without filtering would just give you a DC
>component to code. If you filter the image however and calculate a new
>DFD it will actually become more complex taking more bits to encode.

i clean _during_ motion compensation.

i've almost finished, i'm just going to do the MMX stuff, because using 3x3 smartsoften on entires CCIR images (i'm testing on DVD now) is too slow, i get 50% speed decrease in XviD encoding...

[MfA]

Oh I misunderstood ... are you checking the error just by SAD or also by other means?

[Marc FD]

absolute difference for each pixel. no sad.
no adaptive settings (only quant-adaptive)

i've finished the MMX algo, i just want to make a 3d avisynth filter with for fun. i'm going to implement it. i hope it'll improve a _lot_ speed.

[MfA]

You make the choice per pixel?

[Marc FD]

yes, i know i could do more complex things but i feel per-pixel is good ^^

[zyrill]

have i missed a post or is there still no version for non-p4? if not: :( (athlon) if there is: :D and pls drop me a line.

[trbarry]

have i missed a post or is there still no version for non-p4? if not: (athlon) if there is: and pls drop me a line.

No version of DctFilter?

Mine should have no processor requirements above mmx though it will do special code for higher ones. Note my version supports Avisynth 2.5a YV12 only though SansGrip posted a more general one above for YUY2 and other releases. So if it's crashing make sure you've got the right one.

- Tom

[seewen]

You said that you didn't add "init2" to DctFilter beccause Sansgrip have worked on it, and it will be "better".


But DctFilter works already with 2.5 alpha. So is it possible that you just add "init2" to the 2.5alpha ?

So we can use this filter without having to wait "too long" for Sansgrip ;)

[trbarry]

It seems more likely that you are not using the later 2.5 alpha or beta that requires the init2.

Or else you didn't get it from my site.

- Tom

[Boulder]

I think Seewen asked if you could compile the old (current) YV12 version with init2 so that he (we:D ) could use it with AVS2.5 beta.

[seewen]

I get it from your site ( trbarry.com/dctfilter ), but there's no "init2" in it.

SO it works very well with Avisynth 2.5 from 18.1.03 , but not with older versions

[trbarry]

I'll try to work on DctFilter today. I don't want to lose the YUY2 support but it really shouldn't be much work to make it function properly with VS6, 2.5 beta, etc.

And I had some other ideas I'd like to add to it later so it's time for me to get back to it.

Hopefully I'll post one by tonight if nothing else comes up.

- Tom

[seewen]

Originally posted by trbarry
I'll try to work on DctFilter today. I don't want to lose the YUY2 support but it really shouldn't be much work to make it function properly with VS6, 2.5 beta, etc.

And I had some other ideas I'd like to add to it later so it's time for me to get back to it.

Hopefully I'll post one by tonight if nothing else comes up.

- Tom

That's really nice ;)

Beccause this filter is really missing ;) ( DctFilter & MpegDecoder-working-with-Mpeg1, and it will be great )

[trbarry]

... but get it at www.trbarry.com/DctFilter.zip .

It has support for Avisynth 2.5 beta, VS6, YV12, and SansGrip's YUY2 code. (Thank's SansGrip :) )

No other new features or changes yet.

- Tom

[Bulletproof]

I have a question, which I think is related. DCT is used in many video compression standards, such as MPEG-4 and MPEG-2. My question is, couldn't the MPEG-4 compressor benefit from the MPEG-2 files already containing quantizing information. Isn't there alot of efficiency lost in making a locked quantizer 2 file on scenes that have high quantization done on them. Couldn't there be a way to read the MPEG-2 file's quantization/DCT information per frame so that better decisions could be made in the MPEG-4 compression process? For instance, if a frame in the original MPEG-2 file has high quantization and you've used a locked 2x quantizer, that increases the filesize for no reason. I realize that you can achieve proper compression where quantizer 2 will be used in the appropriate situation with the right settings, however I don't think it would be "dead-on". What I'm saying I guess is to add quantizer decisions based on the MPEG-2 frames quantization/DCT information.

[seewen]

Thank you very much

You've done it very fast, and so far it works perfectly ;)

[trbarry]

optimist (falling past 20'th floor): "So far, so good." ;)

Bulletproof -

Dunno. Maybe someone else can answer that. But there is a thread now in the development forum about transcoding and shrinking MPEG2 files.

- Tom

[Boulder]

Thank you maestro;)

[iago]

Thanks Tom!
I'm just starting a 120min 1CD encode keeping the 350mb ac3 audio :D.

[trbarry]

I'm afraid to ask. ;)

[iago]

Ah, of course I was kidding Tom ;), but I just want to report that, with the script below,

-----------------------------------
mpeg2source("C:\movie\movie.d2v")
crop(12,16,-8,-16)
Trim(16000,17000) # a short bright scene from U-Turn
BicubicResize(512,272,0,0.5)
DctFilter(1,1,1,1,1,.5,.5,0)
-----------------------------------

and using constant quantizer 2 with XviD Koepi's latest (and AviSynth 2.5 beta), the filesize is 20774kb whereas without DctFilter in the same script the filesize is 25062kb.

Using b-frames 3-100-200, the filesize drops to 16658kb, without losing much from its visual quality.

I don't know what if you add some UnFilter to the script to remove the stairstepping artifacts, etc. ?! ;)

regards,
iago

Btw, why are you afraid Tom? The photo is taken after the last plastic surgery I had. I dared not send a formerly taken one :D.

[mikeson]

@iago:

Is this really you? ;)

[MrBunny]

@Bulletproof

I think I understand what you're saying. I guess it's somewhat based on how people believe DVD2One works in which it takes the source's existing motion vectors and manipulates them without decoding the frames themselves and then reencoding them like other encoders would. I don't think this could work within avisynth since it seems the first step is always to decode the mpeg2 stream (mpeg2dec). As a standalone encoder, it might be possible.

However, I believe that the ratio of I/P/B frames in DVD mpeg2 streams is also very different than that for high compression mpeg4 streams. I think MPEG2 has 1 I-frame for every say 15-20 frames or so, as compared to about 1/250+ for mpeg4. In order to create a P/B frame out of the source's I-frame, you would be forced to decode that frame and one or two neighboring frames before preforming motion analysis.

Finally, and this one I'm really not sure about, since MPEG-2 is field based, I would assume that the fields themselves are compressed (using DCT/quantization) rather than frames. As a result, to convert it to a frame based format like mpeg4, I would believe that you'd need to decode the fields and process them. In contrast, with something like DVD2One, since it's going from fieldbased to fieldbased, they don't have to worry about frames.

Without decoding the image, it would become near impossible to filter the image (at least temporally or spatially, since the encoder won't "see" the frames), and I think most people here agree that filtering can be a very important factor on the quality of any encode.

Please please correct me if I'm wrong in anything (or everything) I've said. I'm just basing this on what I think I know and the information I've been picking up here and there.

Mr. B

[iago]

@Tom,

There is something I really wonder a lot about DctFilter, which I still haven't figured out:

Is it reliable/suitable to use DctFilter in a 2-pass encode, or would it be better/safer to use it for 1-pass (CBR or constant quant) encodes? ;)

edit - well, might sound a bit weird, that question of mine; but I have a couple of question marks on my mind, which is difficult to put more clearly into words for me now, even in this insomniac mood! ;)

[MattO]

trbarry,

The following is the main part of my VCD .AVS:
----------
AssumeFrameBased().SeparateFields().SelectOdd()
#DctFilter(1,1,1,1,1,.5,.5,0)
Crop(11,36,698,216).cnr2()
SimpleResize(336,160)
FluxSmooth(15,8)
MergeLuma(Sharpen2(0.2))
DctFilter(1,1,1,1,1,.5,.5,0)
LegalClip()
Blockbuster( method="noise", variance=.3, seed=1 )
AddBorders(8,64,8,64)
Sampler()
----------

Encoding as it is I get better compression but 2 groove-like lines appear at the top of the image.
If I encode using the other DCTFilter line compression is nowhere near as good, but the 2 lines are gone.
Do you have any ideas as to why this occurs?
Is it something I am doing wrong?

MattO

[MfA]

The problem with filtering in the DCT domain is that the filtering isnt translation independent, so when something moves from one frame to the next the filter will have a slightly different effect ... this of course causes extra errors after motion compensation.

[MaTTeR]

Originally posted by MattO
Is it something I am doing wrong? You should always crop using even numbers, worth a shot.

@Tom,
DCTfilter is very impressive from what I can see. I stole iago's settings posted above and honestly can't see any visual difference but the compressibilty is certainly welcomed;) Great work as always!

[trbarry]

MattO -

Try to narrow it down a bit more to see the minimum number of filters that cause the problem. Also try cropping in multiples of 4 to see if it makes a difference.

Finally, I think that AssumeFrameBased().SeparateFields().SelectOdd() will not have exactly the same effect in YV12 as in YUY2, at least if the material is not really progressive.

- Tom

[trbarry]

The problem with filtering in the DCT domain is that the filtering isnt translation independent, so when something moves from one frame to the next the filter will have a slightly different effect ... this of course causes extra errors after motion compensation.

Certainly true. But I don't think there is anything particularly I can do about it. And when it's the last filter it should be working on the same 8x8 pixel blocks that Xvid is going to have to use anyway. So I think that's a general problem of all block based codecs.

DctFilter was just an experiment. It has it's most dramatic improvements using Xvid fixed quant=2 which maybe isn't realistic for most uses. Using other Xvid methods it loses much of it's effectiveness since Xvid or Divx are pretty good at quantizing those same high frequencies anyway when you let them. And you can even make custom quant matrices that may have much the same effect.

But reports are that DctFiler also helps with some other codecs so I'll probably continue to play with the idea.

- Tom

[spyder]

I got a dramatic reduction in filesize with no visible qulaity loss using:

DctFilter(1,1,1,.75,.5,.5,.25,0)

The constant quant=2 xvid of the unfiltered clip was 10.6MB and the filtered one was 7.93MB. The average bitrate of the unfiltered was 241.989KB/s and the filtered was 180.979KB/s(accoriding to windows property dialogs). According to ffdshow, at one point in the clip the bitrate reaches 1900-2000kbps in the unfiltered clip. At the same spot in the filtered clip it is reaching 1500-1600kbps. I find this filter very useful. As my latest attempts at 1CD encodes with convolution3d have a very small first pass size, sometimes just barely over my desired size, I think it's a great filter. Keep up the good work.

[Bulletproof]

trbarry I know this isn't the right thread, but I think it would be better if I just asked about this here.

It seems Deen/eDeen is possibly causing a problem with your plugin UnFilter. I seem to be getting access violations when using it. Donald has said that it initiates an MMX mode which it never lets go and causes other plugins after it to mess up, he was able to make a workaround for his Dup filter, can you do the same for UnFilter?

[trbarry]

Bulletproof -

I know nothing about Deen but that sort of problem is typically when someone leaves out a mmx emms instruction. Filters that use floating point can protect against it by doing an emms instruction on entry.

But do you really mean UnFilter? I just did a quick scan and UnFilter does not seem to use any floating point instructions so it should not be vulnerable to this.

And if you mean DctFilter, it uses float only doing setup and that part looks like it is already protected.

- Tom

[Bulletproof]

Hmm, I seem to have trouble replicating the error again this afternoon. What was happening last night was that when I added UnFilter at the bottom of my script, the video would encode for a couple of seconds and then avisynth would give me an access violation. Sometimes when scrolling between the video it would give an access violation too, but when I took out UnFilter it stopped doing that. It's possible that my computer was in a sort of unstable/bad state causing it to give access violations too, I did do a reboot this morning. If I'm able to replicate the problem again I'll let you know, thanks for your time.

[tangent]

Hi, DctFilter seems pretty interesting and can be really useful for low bitrate rips. Currently using it in my valiant attempt to encode the extended edition of LotR onto 1 CD :)

Anyway, I talked to SysKin about the possibilties of integrating a lowpass DCT filter into XviD since it would be much faster doing it during the encoding without needing an extra DCT+iDCT cycle pair. He got an XviD compile ready (in only 5 mins) which zeros 15 coefficients from the 8th row and 8th column.

I suggested another way of zeroing coefficients which may provide more compressibility efficiency. Instead of zeroing from 8th row/column, zero the diagonals starting from the F diagonal, E diagonal inwards.
If you zero diagonals A-E, that would be 15 coefficients too.


12345678
23456789
3456789A
456789AB
56789ABC
6789ABCD
789ABCDE
89ABCDEF


I believe this way may be better because if I am not wrong, MPEG encodes the 8x8 block in this diagonal order, starting from diagonal 1, 2, 3 all the way until F. When the remaining coefficients are all 0, the block encoder puts in an "End of Block" signal and doesn't encode anymore coefficients. So doing a low pass filtering this way would simply chop off the tail of the block stream and placing the "End of Block" signal in place.

SysKin also compiled a binary which uses the second method. I haven't had the time to test them myself, but according to SysKin, the first compile gave savings of 3% without perceptible quality loss, the second compile gave savings of 2.5% without perceptible quality loss (and supposedly less loss). It might be possible to zero the 9th diagonal for a better space saving and also keep the loss imperceptible.

I haven't obtained permission from SysKin to distribute the compile, but if you are interested, you could PM me your email and I'll email it to you if SysKin is fine with it. Not sure if it works correctly on all architectures though, someone with a P4 had crashing problems with it.

[MrBunny]

@tangent

I agree with most of what you said, however as Tom has noted earlier the DCTfilter isn't really that useful for Xvid. This is because it already has the option to create custom quantization matrices, in which you can apply any DCT based filtering you want (like scaling or zeroing certain components). As well, another reason is that although when using the DCT filter, it looks good on "paper" (the compressibility rate can go way up), but trying to get xvid to compress it further is harder (and uglier) since you've already removed all the high frequency components that it would be dropping to lower the bitrate. Basically, you're doing exactly what xvid would have done to try to scale it's bitrate anyways. Although I guess having a lower average quantizer might be of some help (any lurking Xvid experts want to comment on that? );) At the same time, when you start getting really agressive with the high coefficient filtering, you really start to erode fine details, especially around the edges, just toss some test scripts into TheWEF's great avscompare tool and you should see the difference. Of course, I've been told that I'm excessively picky about barely noticeable "flaws" when filtering.
That having been said, I would agree completely that changing the DCT filter to work diagonally, as MPEG does is probably a better idea than column/row based as it is now.

Mr. B

[CruNcher]

someone with a P4 had crashing problems with it

hmmm who could that be :rolleyes:

indeed a nice filter keep on :)

[MfA]

The effect of the quantization matrix is different from a filter, using larger entries in the quantization matrix on average decreases the size of corresponding coefficients because it truncates a lot of them to 0 ... but the one which do get through are still large, just with a large error (with corresponding ringing/mosquito-noise).

You cannot use the quantization matrix to get the exact same results as that of a denoiser/smoother.

[MrBunny]

I agree that a custom quant matrix can't do the same as a denoiser/smoother, but I was talking about comparing custom quant to the DCTfilter. I believe using it should give the same results as using appropriately chosen custom quant matrices. The DCT matrix scales the resulting coefficients, whereas using a custom quant matrix could also scale them in the same way. Using DCTfilter(1,1,1,1,1,1,1,0.5) should be the same as taking the default matrix and doubling every coefficient in the 8th row and column. Of course this doesn't hold for xvid's h263, but if you want to be playing with the quantization, you might as well use a mpeg-custom quant matrix.

Mr. B

[MfA]

It scales the coefficients, and some people are suggesting to add thresholding ... both these things are radically different from what quantization does.

Remember, quantization is followed by dequantization ... you cannot use the quantization matrix to get the same result as a denoiser/smoother, and the proposed filter DOES act like a denoiser/smoother.

[trbarry]

I haven't always been able to predict exactly what it does. That's why I wrote it and am playing around with it.

I'm pretty sure that it tends to make more high frequency coefficent go to zero, just like quantization. But the difference with DctFilter is once they go to zero they stay there. With high quants but a very stong high frequency signal the darn thing will come back to haunt you, causing mosquito noise. If you want that signal then you put up with the echos but if you can live without it then it compresses a little bit better with DctFilter and maybe does not come back with the artifacts. So there might still be some advantage here.

That is, if it was going to generate very high quants then you might prefer to just zero it. But this is just wild speculation on my part.

I think I do agree about using the diagonals, subject to a bit of testing.

I remember someone else earlier in this thread suggested that integrating DctFilter type processing might screw up motion estimation somehow. But I don't know enough about that to be able to say.

However I assume SysKin does. I breathlessly await further results. ;)

- Tom

[trbarry]

BTW, I had early problems reported with P4's and DctFilter, though I developed it on one. So I turned off SSE2 DCT/iDCT.

I seem to remember it had something to do with optimization options but I never really figured it out so DctFilter still does not use the SSE2 calls.

- Tom

[MrBunny]

@MfA

I've kinda lost the flow of the thread, by "the proposed filter", are you talking about the DCTfilter or some other idea someone threw into the middle of the thread. Are you even talking to me? ;)

I think I do see what you're saying about quantization being followed by dequantization. While modifying the custom quant matrix to force the encoded quantized coefficients to be the same as a DCTfiltered approach is possible, those coefficients would then be dequantized by different matrices. The DCTfilter approach would return the DCTfiltered coefficients, whereas the custom quant method would return something near the original minus those coefficients rounded to zero. A custom quant matrix could only mimic the behavior of DCTfilter whose arguments are set to 1 or 0.

Silly of me to only think of half of the equation :o

Mr B.

[tangent]

Oh yeah, SysKin also mentioned that it would be quite easy to make the DctFilter adaptive to the amount of motion (more motion = more agressive filter) since motion vectors are available.

Attached here is the XviD build which chops 15 coefficients from the 8th row/column.

[tangent]

And this one does the diagonal cut.

[trbarry]

"And this one does the diagonal cut."

Pics aren't approved yet.

But some of my HDTV captures seem to have more pixels than resolution. That is, they have a huge resolution of 1920x1080 but have been so filtered or lost so much in compression that there isn't really that much information there.

I have one case now where I don't see much or any difference after using:

DctFilter(1,1,1,1,0,0,0,0)

This would actually zero 48 of the 64 components in the 8x8 DCT block (leaving only the top left corner), yet doesn't seem to matter except removing a little haze.

I'm not sure how DctFilter could become adaptive inside Xvid. That would seem like the kind of case that would mess up motion comp since more high frequency components would exist in one frame to the next. But maybe there's some way to deal with that. I don't know how they do lumi masking either.

- Tom

[cinghio]

Hi TrBarry, hello all guys.

I wonder if your filter could be used (as it is or modified)
as a bridge for the task of transcoding from MPEG2 (without/before iDCT) to MPEG4 (without doing DCT nor re-quantization)

I suppose free resize would be impossible, maybe crop of and integer number of 8x8 [16x16] blocks could be done.

Any way, I only think about a simple transcoding without any resizing/filtering, e. g. for a tipical Personal Video Recorder application.

Maybe I'm going OT, in this case I apologize.

This topic could also be related to direct Xvid,DivX to MPEG4 transcode, and maybe accomplishing that task with little (is it true???) modifications also MPEG2=>MPEG4 would be possible.

I have two questions:
1) In terms of computational times, which is the weight of iDCT(MPEG2) + DCT(MPEG4) + quant(MPEG4) over the total MPEG2 to MPEG4 process?
2) Is there a "MPEG4 encoder" accepting DCT (& quantized) domain as input?

Thanks for any information.

[trbarry]

cinghio -

I hoped someone else would jump in here and answer since I can't answer that myself. I'm not even sure I understand your questions. ;)

I'm not really sure how DctFilter would help with transcoding.

I did do one test last year with a performance monitor on DVD2AVI re-encoding to Xvid. FWIW, it turned out that the DVD2AVI iDCT was using about 5% of the job total.

I'm not sure what you are asking about with "free resize". I did create a resize based upon a similar idea that rapidly doubles the size of a clip. But it was based upon a (non-standard) 4x4 integer transform, not DCT. I'll release it someday, after some more tinkering.

- Tom

[cinghio]

Hi Trbarry,

Thank you for the reply and the precious information about the CPU costs of iDCT.

With "free resize" I mean a not "integer magnitude" one,
and would not be possible assuming information is available in
Discrete Cosine Transformed domain, with 8x8 blocks.
"Constrainted resize" would be possible, i.e. integer magnitude resize (2x for example or 1x, that is no resize) with a "dumb cropping" made discarding entire rows or coloumns of 8x8 blocks.

However, for immediate storage of videos, the idea is to do no transformation and process the DCT domain information of the MPEG2 and assume they were created by the DCT preprocessor of XviD.

Is this kind of approach possible?
Maybe for my purpose a new filter is needed, capable of dealing with DCT domain data before the complete MPEG2 decoding has been done.


If I understood how it works, your filter does a complete DCT, then some transformations in DCT domain and after that an iDCT.

However your code could be integrated in XviD DCT preprocessor (or in Mpeg2dec before doing iDCT from MPEG2 source if you don't need resize),
so that the entire process of DCT filtering and re-encoding, will not cause double stages of DCT and iDCT.

Is XviD capable of accepting DTC domain data in some way?

[trbarry]

hank you for the reply and the precious information about the CPU costs of iDCT.

With "free resize" I mean a not "integer magnitude" one,
and would not be possible assuming information is available in
Discrete Cosine Transformed domain, with 8x8 blocks.
"Constrainted resize" would be possible, i.e. integer magnitude resize (2x for example or 1x, that is no resize) with a "dumb cropping" made discarding entire rows or coloumns of 8x8 blocks.


Okay, I see what you meant. In that Integer transform resizer I actually also implemented 4:6 and 6:4 resizing. If the underlying basis functions are continuously defined (like cosine) then I think you can always IDCT to a different block size. Of course all final results are integer dimensions, but maybe not integer ratios.


However, for immediate storage of videos, the idea is to do no transformation and process the DCT domain information of the MPEG2 and assume they were created by the DCT preprocessor of XviD.

Is this kind of approach possible?


I think you are talking about a transocder here. There is a thread in the Xvid forum where -h is working on one I think.


Maybe for my purpose a new filter is needed, capable of dealing with DCT domain data before the complete MPEG2 decoding has been done.

If I understood how it works, your filter does a complete DCT, then some transformations in DCT domain and after that an iDCT.


For pure filtering purposes it could be added to decode, like in DVD2AVI. I thought Marc FD mentioned some experiments like this above in this thread somewhere. (not too favorably for some reason) But it would be faster.


However your code could be integrated in XviD DCT preprocessor (or in Mpeg2dec before doing iDCT from MPEG2 source if you don't need resize),
so that the entire process of DCT filtering and re-encoding, will not cause double stages of DCT and iDCT.

Is XviD capable of accepting DTC domain data in some way?

I don't think anything like that is anticipated.

- Tom

[trbarry]

For any of you that have been following this long thread, I noted way above that it might be possible to achieve the benefits of DctFilter at no CPU cost in Xvid simply by using custom quant tables.

Well, I've been playing with that a bit and think it does seem to be mostly true. See my Free Filtering with custom quants (http://forum.doom9.org/showthread.php?s=&threadid=45695) thread in the Xvid forum. The example I posted there is a bit extreme (over soft) but I think it's possible to duplicate just about any DctFilter effect.

That is, remove a bit of capture or mosquito noise, soften a bit, and improve compressibility, all without running any extra code.

- Tom

[MrBunny]

Hi Tom,

I believe you are drawing the same incomplete conclusion I made back on page 6, which MfA corrected me on. Your conclusion is only correct if you're zeroing the coefficients (which you were in your example in the other thread). If you're scaling the coefficients (in DCTfilter), then a custom quant matrix would not do the same task as DCTfilter since dequantization must occur after quantization, and so unless the custom matrix zeroed the coefficient, dequantizing that value will result in the original value (with a greater error) rather than the scaled value DCTfilter would give.

Mr B.

[trbarry]

You're right, but it seems to have the desired effect. It's a bit more complicated but I'm basically setting (more than) the lower right half of the quant tables to all 99. And I'll use 255 if I find it's legal and seems to work but I haven't tested it yet that way.

But after DCT there are rarely any high numbers in the DCT results for the higher frequency components anyway, regardless of quants. I noticed that while debugging DctFilter. I believe this means that 99 or higher will really drive almost everything to zero in those values.

I'm sure there are contrived examples where it would fail and cause artifacts but I hope they are very rare in real life (Murphy's Law and some anime textures notwithstanding).

- Tom

[tangent]

I would feel much better about zeroing those coefficients. At least there would be no chance of a rogue non-zero coefficient destroying the macroblock.

[trbarry]

I'd feel better about zeroing them too if I could, but I'm still getting good results with the custom quants. I don't see any particular extra ringing type artifacts. And I still haven't experimented with how high a value I can put in a quant matrix before it croaks.

..... changing horses in mid stream

What does everyone think about a different flavor of DctFilter that would work on diagonals instead of rows & columns like now. The logic above was fairly persuasive that any block based codec using a diagonal or zigzag scan would probably compress better that way.

I could probably woop one up in an hour or 2. Would that be a useful option for anybody?

- Tom

edit: Moderator? If you happen to read this could you correct the spelling on my thread title ("quiestions"). It's annoyed me for months now and I can't change it. :(

[iago]

Originally posted by trbarry
What does everyone think about a different flavor of DctFilter that would work on diagonals instead of rows & columns like now. The logic above was fairly persuasive that any block based codec using a diagonal or zigzag scan would probably compress better that way.

I could probably woop one up in an hour or 2. Would that be a useful option for anybody? Though I am quite pleased with the results I can get with the current version of DctFilter (together with some extra filtering), especially when encoding "really hard-to-compress movies" and aiming for low file sizes, I would be more than willing to test a new version! :)

Kind regards and many thanks one more time for this miraculous toy called DctFilter ;).

iago

[Guest]

Originally posted by trbarry
edit: Moderator? If you happen to read this could you correct the spelling on my thread title ("quiestions"). It's annoyed me for months now and I can't change it. :( Your wish is my command.

I didn't even know I could do it!

Look for rule violating titles to disappear very fast from now on. :devil:

[MfA]

It would be easy to adapt xvid to interpret a 0 in the custom matrix as an indicator it should zero a given coefficient.

[trbarry]

I would love to see that but I thought someone else posted in my Free Filtering (http://forum.doom9.org/showthread.php?s=&threadid=45695) thread that the quants are stored in only 8 bits. If this is so then you can't really store a large enough value to be guaranteed zeroing every single component. And doing logic on each pixel would maybe be prohibitive.

- Tom

[MfA]

The assembly versions of the quantization routines implement the divisions by multiplication and shift, so that is not an issue, but Ill not continue this discussion here ... that thread you mentioned seems a better place.

[Kaiousama]

@TrBarry
Recently i've done some test with your DCTfilter and the results are not as i've expected, check this link for all the Test's details (http://forum.doom9.it/download.php?id=86).

As you can see there is (as expected) a direct correlation between psnr and filter's aggressiveness.
The strange is the fluttuating filesizes, more noticeable if you touch the last value, less noticeable if you change the 7th coefficient.
Another strange thing is that an all 1 settings don't give the original filesize&psnr (if i've well understood an all 1 setting don't scale anithing)
Another not expected behaviour is that a (in my opinion) not to aggressive settings like DctFilter(1,1,1,1,1,.500,0,0)gives an output PSNR like if you increse the conpression's uantitizer by 1, but the filesize is much more reducted if you increase the quantitizer in front of using the DCTfilter

What do you think about?

[trbarry]

Kaiousama -

Great series of tests.

Mostly it's as expected, with gradually declining file sizes and PSNR's. And I can believe some random variance due maybe to strange interactions with motion search etc.

But I can't explain the degree of the INCREASE in file size twice as the parms went all the way to zero. And I really did think I set it up so a parm set of all 1's would leave things unchanged.

So there may be a round off error somewhere when I first set up the tables. I was using floating point during setup and in the conversion it is always possible that something gets truncated. I'll go look for that.

Another not expected behaviour is that a (in my opinion) not to aggressive settings like DctFilter(1,1,1,1,1,.500,0,0)gives an output PSNR like if you increse the conpression's uantitizer by 1, but the filesize is much more reducted if you increase the quantitizer in front of using the DCTfilte

Sorry, I'm not sure what you are saying here.

- Tom

[Kaiousama]

Originally posted by trbarry
So there may be a round off error somewhere when I first set up the tables. I was using floating point during setup and in the conversion it is always possible that something gets truncated. I'll go look for that.
Mhhh.. yes, if you watch only filesizes it seems that one setting every 2 it's bad rounded and leads to unpredictable consequences, but if you look the psnr all things are ok (i've inceased the values by 1/8 every time as you said in the filter's guide)


Sorry, I'm not sure what you are saying here.
I was thinking about the filter's efficience, look:

NO-filter quant.4 size:24.287.232 PSNR:42.2381
DctFilter(1,1,1,1,1,.500,0,0) quant.3 size:30.709.760 PSNR:42.0057

If you look for the same psnr results using DCTfilter the encoding brings you a 6Mb higher filesize in front of increasing the codec quantitizer to reach the same PSNR.

I'm thinking if this behaviour is a sporadic case in my test or if it's the same with some other tests (yuhhuuuu... plz someone test one of your clips to check this results)

Or it's possible that the bitrate saving made by DCTfilter is not well quantitized by the PSNR algorithm... do you know a way to valuate (with a quantitative value) the Sharpness (only that) of a clip?

[trbarry]

I'm not the PSNR expert but I don't know of any objective reliable method that is really any better.

But both with DctFilter and my experiments with custom quant tables the idea is actully to discard some information that we might consider less trustworthy or useful. The fact that we discard it will probably hurt PSNR calculations but it's up to the eyeball to decide whether it would have been worth keeping it. That's really the same process as lossy coding using quantization. This just adds another option.

I think most of the human visual system experiments are more targeted at what the human eye can see. But here we are doing many rips and caps where we also have to consider getting rid of what they eye might not like to see.

For instance, PSNR may measure how well your encoding can faithfuly represent mosquito noise and blocks from a previous encoding. But if a certain level of detail is not reliably represented in the source you are working with then it's maybe better to just discard it and save the space in the process.

- Tom

[tangent]

Is the diagonal cut version of DCTfilter available soon? I would definitely use that in preference to the original DCTfilter. It should be more efficient.

Also, I don't think using values between 0 to 1 (such as 0.5) is a good idea. You're going to get very little bits saving for some quality loss because as long as a coefficient is not zeroed, many more bits would be used to encode it. You may save 1 bit per coefficient, but that's hardly significant.

[trbarry]

Hi tangent -

Your post made me go back and look for your previous questions. That in turn pointed out I'd previously totally misunderstood what you'd attached.

I thought it was just pics of the results, not then yet approved, and not a modified Xvid. And I'd promptly forotten about them, before they got approved. So I'll check that out now.

I haven't made the diagonal version yet, but will soon, possibly today.

As far as using fractional values, I agree as far as custom quants but maybe not for DctFilter parms. I'm not sure about that one yet. It may be okay to just increase the probability that a given coeficient goes to zero but let extreme cases through. I can't even guess how much that helps compression but it doesn't seem it would create the high quant artifacts that doing the same thing with a custom quant table might do.

But for DctFilter anyone can just use 1's and 0's anyway if they wish. Increasingly I do.

- Tom

[bilu]

@trbarry

Originally posted by trbarry

I haven't made the diagonal version yet, but will soon, possibly today.




Got news? :)

Bilu

[trbarry]

I haven't done much programming for a few days but this morning I did make a new version of Dctfilter.

This includes a new function DctFilterD(DiagCt) that works on diagonals. It is simpler and either zeroes diagonals or leaves them alone.

In a 8x8 DCT result there are 15 possible diagonals (visualize a chess board). So if you specify:

DctFilterD(4)

then the 4 diagonals in the lower right corner of the DCT result will be set to 0. The DiagCt parm may be any integer value from 1-14.

I haven't tested this much but, like my results with custom quant tables, it appears if you set DiagCt very large you will start to get edge noise, something like ringing. You can see this in the DctFilterD(10) part of the sample picture I put out at www.trbarry.com/DctFilterD.jpg .

Like usual, get the files (source and dll) at:

www.trbarry.com/Readme_DctFilter.txt and
www.trbarry.com/DctFilter.zip

The diagonal version is certainly easier to specify, and runs a tiny bit faster but I'd be interested in any feedback about whether it really does a better job. I'm still very uncertain about this.

- Tom

[tangent]

Thanks Tom. Time to restart my FotR-in-one-CD experiment :)

[Didée]

I lost a little the track of this thread, and DCT-filter was lying in the corner quite some time now (day has only 24 hours...)

Now, I mad some short tests on the Kung-Fu scene of "Matrix".
Have yet to check visuals, but for sure I don't understand the filesizes I got:
dct(1,..,0,0,0) 32876 kB
dctD(5) 32934 kB
dctD(3) 32874 kB
dctD(7) 32520 kB
plain encode 32279 kB
dctD(8) 32240 kB
dctD(10) 31080 kB
dctD(12) 29036 kB
How can it be that filesize grows, when zeroing out some neglectable high frequencies?
I used XviD with nothing but I-P and mpeg quant @ q2. Oh, and qpel.
Three shots with q7 showed same behaviour.

:confused:

[iago]

@Tom,

I can confirm the paradoxical results about filesizes that Didée mentioned.

h263 / q2 / DctFilterD(6)

plain: 22120kb
DctFilterD(6): 22228kb

??

iago

[tangent]

Weird... could there be some misalignment in the 8x8 blocks?
Perhaps this is really best implemented in the codec itself...

[MfA]

DCT transform is not translation invariant, what this means is that "signal energy" present in one coefficient might be in another coefficient if you shift the image.

One possible explanation of the growth in number of bits is that when there is motion then for each frame the filter is changing the image in a slightly different way, and this happens between each and every frame, after motion compensation that means we get new errors which would not have been there had we not applied the filter. If this effect is greater than the saved number of bits because of smoothing it is a net loss. When performing the zeroing of coefficients in the codec through custom quants this effect should be less of an issue, but this has problems of its own.

Personally I think transform based denoisers/smoothers should really use translation invariant transforms (undecimated wavelets for instance).

[trbarry]

For lack of anything better I guess I'll go with MfA's explanation above.

But I don't know how to code an efficient mmx implementation of undecimated wavelets. ;) :(

OTOH, the other possibility is I messed up something with this last release. This might be supported by the difference between my Quant 2 test results for my first release and this recent one. I'll go do some more testing.

- Tom

[MfA]

This paper (http://home.student.utwente.nl/m.f.al/E006851770.pdf) describes how to implement a 1D undecimated wavelet transform using relatively short filters, translating that to a MMX implementation shouldnt be too hard :)

If anyone wants to try ... only section II-1 is of import, the rest deals with deblocking and application specific optimizations which wouldnt work for general denoising, although I think performing filtering first entirely in one direction and then in another is a good idea ... the direction sensitivity of the wavelet transform can give annoying artifacts when you use tresholding with a 2D transform. Apart from the transform you still need to apply thresholding, theres a million different ways of doing that though ... easy enough to find a couple of ways on the web (it is sometimes also called coring or shrinkage).

[trbarry]

MfA -

Thanks for the paper but I'm afraid I really have no intuitive grasp of how wavelets work. It took me long enough to understand DCT, to the extent that I even do.

Hopefully one of these days I'll take the time to sit down and figure it out but I suspect it won't be real soon.

How about a nice simple paper explaining why wavelets are better than (i)DCT's. ;)

- Tom

PS - make it even simpler. I think I'm having a stupid week.

[SILICON]

One posible cause for grow filesizes.

The zeroes matrices can make more hard detect the motion.

The more size decresase in the compresion are for the motion vectors, isn't it?

[tangent]

Up.

Any progress with bug finding?
Or perhaps this filter thing should be best done within XviD?

[trbarry]

Sorry, I got involoved with some other things. I'll try to get back to it soon.

- Tom

[odysseus]

as a general rule how does this kind of thing interact with a deinterlacer.

will a smoothing deinterlacer make this filter unnecessary,

and if I run DCTFilter before a deinterlacer will DCTFilter remove some information that the deinterlacer needs, that is, will DCTFilter do some deinterlacing of its own, making the job harder for the "real" deinterlacer?

I've made some test runs putting DCTFilter before and after DG's kernel deinterlacer & can't see a difference on my crappy monitor.

some guidance please.

[trbarry]

I don't know how DG's kernal deint works but generally it is a bad idea to do almost anything before deinterlacing. This would include DctFilter.

- Tom