Integer DCT/iDCT filter questions

[FuPP]

Don't know for xvid, but using this filter with mpeg2 compression (svcd), I gain about 10% ( using DctFilter(1,1,1,1,1,.5,.5,0) ) without visible loss of details.

Thx a lot Tom :) !

Regards

FuPP.

[fraatz]

Hi!
I recently made some tests with the DctFilter and have to say it's quite cool. Thanks TBarry!
I used C3D in my tests with the MovieHQ- preset and I found that both together limit reasonable values for DctFilter a lot.
I tried to find the best settings for keeping the Iimage as original as possible while increasing compressibility as much as possible.
These where my settings:

LoadPlugin("D:\Programme\DVDRip\AVSPlugins\MPEG2Dec3.dll")
LoadPlugin("D:\Programme\DVDRip\AVSPlugins\Convolution3D.dll")
LoadPlugin("D:\Programme\DVDRip\AVSPlugins\DctFilter_YUY2.dll")
LoadPlugin("D:\Programme\DVDRip\AVSPlugins\MSharpen.dll")
MPEG2source("E:\jhb.d2v",lumoff=-3,lumgain=138)
crop(1,75,717,423)
Convolution3d(preset="movieHQ")
#MSharpen(20,50,true,false,false)
BilinearResize(608,240)
#DctFilter(1,1,1,1,1,.5,.5,0) #2
#DctFilter(1,1,1,1,.7,.5,.2,0) #1
#DctFilter(1,1,1,1,.5,0,0,0) #3
#DctFilter(1,1,1,1,1,.7,.5,0) #4
#DctFilter(1,1,1,1,1,1,.5,0) #5
#DctFilter(1,1,1,1,1,1,.7,0) #6

Compressibility gain was huge, about 30%, especially with dctfilterset #3. But imagequality suffered a lot, many details were lost..
So I decreased DctFilter- Influence until loss of detail became neglectable. This happened with filterset #5 but i still got a compressibillity gain of 10%. Amazingly good with a already *very* clean source!!
Now I tried to get more detail to the image, to fill the "saved" bits with information. MSharpen(20,50,true,false,false) together with DctFilter(1,1,1,1,1,1,.5,0) was very good. These settings gave me a little lower filesize as without DctFilter/MsSharpen at all. But the picture was as sharp as BicubicResize 0,0.7 !!

All test were made with xvidapi3 without new features.

I made a three twopass encodes with about 4250 frames an I used xvid with bframes 3/150/100, qpel, chroma me and lumi. to compare quant redistribution:

1. Avs- Script as above:

Q:2:1
Q:3:62
Q:4:2574
Q:5:2077
Q:6:52

2. + MSharpen(20,50,true,false,false) together with DctFilter(1,1,1,1,1,1,.5,0) :

Q:2:1
Q:3:30
Q:4:2072
Q:5:2546
Q:6:117

3. Script as above only + DctFilter(1,1,1,1,1,1,.5,0) :

Q:2:1
Q:3:164
Q:4:3282
Q:5:1295
Q:6:24

Number 3 gave very good quality, interestingly the mosquitonoise was reduced quite well!!

If you are interested in my test I can attach the screenshots I made....
Thanks again for this filter !!!
Try DctFilter(1,1,1,1,1,1,.5,0) with C3D. You will not be dissapointed.

-f

[TheUnforgiven]

big thanks to tom.
great filter

[iago]

Tom,

I think with the help of DctFilter now it's possible to put "any" movie on 1CD with a reasonable resolution of 576*... (maybe even more) and a 128 Kbps audio track, without ruining the encode and keeping decent/acceptable quality.

However (you may/should also have noticed that) the only thing that I still cannot eliminate is what I can call the "staircase" effect when DctFilter is used. Using BilinearResize, UnFilter(-,-), UnDot(), etc. have not worked for me yet. (Though I must admit that this effect is one of the most acceptable artifacts for my eyes ;))

Any ideas or recommendations from the author about that? ;)

And thanks once again for this amazing filter!

iago

[trbarry]

However (you may/should also have noticed that) the only thing that I still cannot eliminate is what I can call the "staircase" effect when DctFilter is used. Using BilinearResize, UnFilter(-,-), UnDot(), etc. have not worked for me yet. (Though I must admit that this effect is one of the most acceptable artifacts for my eyes )

iago -

I'm still playing with this and don't really understand all the ramifications yet. It seems DctFilter does something similar but not quite the same as raising quants towards the edges of a custom quant matrix, much as the difference between the mpeg and H263 matrices. But it is reducing the value of the higher frequency components, not the precision.

I think if you over do it you can effectively create macroblocks, in sizes that are 2x2, 4x4, 8x8, etc. And offhand I can't think of any fix for that besides backing off a bit on the DctFilter parms or maybe softening even more first with something like C3D, possibly in conjunction with lowering the resolution a bit. Still no magic bullet.

Just a guess.

- Tom

[Marc FD]

okay guys. i intensively tested DCT lowering in XviD.
i've hacked XviD a very clean way to have 192 config paramters + quant/movement/vop adaptive dct-filtering.

i've run hours of test, and i need to admit that the idea looked attractive (-30% at quant2) but in fact it's not good : using bigger quantisation makes much crisper image (more noisy too) and heavy prefiltering makes images smoother with lower quants.

i can provide a hacked build for testing, but it really doesn't worth it.
i tested the idea to ill ^^.

topic close.

Regards,
MarcFD

[-h]

Perhaps better than filtering on fixed 8x8 boundaries (remember that motion compensation will be copying blocks from across those boundaries, so the filtering will not have the intended outcome all the time), would be to run a lowpass filter over the entire image.

-h

[Marc FD]

no, i used that between fdct and quantisation. (on inter coefs too)
using a lowpass filter is far too destructive for low-mo.

[-h]

Perhaps use a transformation other than the DCT, on blocks larger than 8x8 (i.e. the entire image).

-h

[trbarry]

Marc -

Those are disappointing results. :(

I think all that is maybe because Xvid is already very good at jiggering quants if you let it (don't fix quants), so it is harder to compete.

But I'm still playing with things based upon these principles and I certainly haven't given up yet. I'll post some test results if I get anything useful.

As I said on earlier posts here, this is largely for curiousity and I'm not sure what the recommended usage is yet. But it is another tool that probably can be useful sometimes, and not just for spinning off a high quality fixed quant=2 work file.

Too bad Xvid doesn't support quant 1. I'll bet the savings would look even more dramatic. ;)

I wonder how DctFilter would perform in (-h)uffYV12?
(a mythical non-dct based compression method) ;)

Along the same lines right now I'm trying to make a quickie resize filter now based upon an integer transform that works similar to DctFilter.

- Tom

[MfA]

Block based filtering of DCT coefficients should be left to the codec IMO. Normal low pass filtering is easier to do in the spatial domain.

BTW the MPEG matrix does not exactly lowpass filter the coefficients ... high frequency coefficients will more often fall in the deadzone and be filtered out so much is true, but in the cases where they fall outside of the deadzone they will tend to only do so just. Coefficients just outside of the deadzone tend to actually be amplified from their true value after dequantization.

[trbarry]

Coefficients just outside of the deadzone tend to actually be amplified from their true value after dequantization.

I think that is one of the main differences between DctFilter and normal quantization.

- Tom

[MfA]

BTW this thesis (http://www-isl.kaist.ac.kr/Papers/PhDTheses/phd_sdkim.zip) presents some result for DCT domain pre-filtering inside the codec. There it makes more sense to use the DCT representation, although even there the majority of pre-filtering papers have used spatial filters.

[trbarry]

MfA -

Thanks. Great paper. It not only gives a much more sophisticated analysis of DctFilter than I would have been capable of, it also contains very good explanations of both the deblocking and deringing process for anyone interested in those.

And I like the idea that for performance you could actually build the DctFilter scaling right into the scaling coefficients of the DCT itself, for "marginal" extra overhead.

- Tom

[Taranli Maren]

I was curious, and tried this filter on an anime source, a clean source with many crisp lines. I'm sure you all expect this, but it created a sort of light aura around the black edges at low levels (1,1,1,1,1,1,.5,0) and (1,1,1,1,1,1,1,0). At higher levels it created normal mosquito noise around those edges. It did have this affect more around sharp high contrast edges than weaker ones. This is all in preview before encoding. I've not tested the compressibility difference, but will be getting to that next.

[Marc FD]

>BTW this thesis presents some result for DCT domain pre-filtering inside
>the codec. There it makes more sense to use the DCT representation,
>although even there the majority of pre-filtering papers have used
>spatial filters.

do you think it's a good idea to try this one into XviD ?
i read the paper and it seems great.

>Thanks. Great paper. It not only gives a much more sophisticated
>analysis of DctFilter than I would have been capable of, it also
>contains very good explanations of both the deblocking and deringing
>process for anyone interested in those.

tom, you rewrote the dering part in nic's pp, can you say me if it's the proposed one or the vm8 one in nic's implementation ?

>And I like the idea that for performance you could actually build the
>DctFilter scaling right into the scaling coefficients of the DCT itself,
>for "marginal" extra overhead.

that's what i tried, but my approach was too basic.
i'm gonna try this method into XviD now, i wonder how well it'll work ^_^

[mikeson]

@Marc FD:

i'm gonna try this method into XviD now, i wonder how well it'll work ^_^

Any progress or even some results? ;)

So please let us know when you have something. I'm very interested in this filter and its implementation inside XviD (if it is worth it).

[trbarry]

do you think it's a good idea to try this one into XviD ?
i read the paper and it seems great.

>Thanks. Great paper. It not only gives a much more sophisticated ....

tom, you rewrote the dering part in nic's pp, can you say me if it's the proposed one or the vm8 one in nic's implementation ?

Marc -

It would be interesting in Xvid but Xvid has usually tried more for standard MPEG4 compliance so I'm not sure how happy they would be complicating the core functions for a non-complient mod, especially if we are not sure of the benefits yet.

I think the real savings in something like this would maybe be if a decoder also had an option for a 6x6 or a 4x4 DCT/iDCT. Then it could encode in a downsized format that was optimized to be upsized again at display time. This process would take no extra overhead if done by the codec. But it would be non-standard. I'm writing a filter to test this. But it would be non-standard to do it more efficiently in the codec.

I'm afraid I only skimmed the first part of the deringing section in that paper. All I did in Nic's decoder was create the MMX optimized asm version from the C code. I didn't write the orig code. I think it was based upon the old OpenDivx code.

- Tom

[Marc FD]

i studied it a bit...
it could be good to encode in a very noisy environnement in real-time for very low bitrate/res. that's not what XviD is designed for.

RRVs will do the job better ^_^

[MfA]

RRVs is a mostly seperate issue to filtering, and I would not be so fast to sing it praises until you try it out. Besides, it is not part of the advanced simple profile ...

Wether you do your filtering in the DCT domain or in the spatial domain I think one lesson from the paper is pertinent :

"Even in an experiment based on the assumption that quantization noise is absent, it is also shown that pre-filtering for motion-compensated error blocks is more desirable than spatial domain filtering for original image blocks."

I think filters which adapt to the quantizers and/or motion should always work on the DFD, otherwise you run the risk of filtering being counterproductive. Say we have a MB which happens to move fast in a given frame with lots of motion, and a corresponding high quantizer, but which is perfectly predictable by motion compensation ... filtering the original block will increase both rate and distortion, the exact opposite of what you want to accomplish. If you filter the DFD (displaced frame difference, what is left of the frame after motion compensation) however the MB will simply not be coded, a filtered 0 signal stays a 0 signal, no distortion and the optimum number of bits used to code it.

Other kinds of filters (denoising for instance) can safely use the original images of course ... but I think you could almost always roll them into a DFD filter.

[Marc FD]

>RRVs is a mostly seperate issue to filtering, and I would not be so
>fast to sing it praises until you try it out.

i just think it'll be better if you _really_ want to compress more.
or go for 1000 kbps encodes, and any preprocessing other than light denoinsing is useless IMHO.

>Besides, it is not part
>of the advanced simple profile ...

if it works with XviD it's enough for me ^^

>but I think you could almost always roll them into a DFD filter.

you think DFD denoising will be better ?
but waht if you prefilter _before_ me ??

[MfA]

If the motion estimation was done accurately (which is not a given inside the codec given rate restraints, and the sensitivity of ME for the noise you are trying to remove) the DFD tells you a lot about what is noise and what is signal ... from a single image you can only guess what is noise.

That said though image based filtering (where the filter strength is independent of motion and quantizers) can improve ME (SAD is noise sensitive) and it will usually improve compressibility ... so it can be usefull. It just cant do the things motion compensated spatiotemporal filtering and adaptive R-D optimized pre-filtering can do, it is a lot easier though.

If you want to go that route you will probably want to consider non-linear filters, they can handle edges better than linear filters ... something like this (http://133.23.229.11/~ysuzuki/Proceedingsall/ICASSP95/pdf/ic952193.pdf) (this is old, and might be totally outdated).

Here (http://ivpl.ece.nwu.edu/Research/Current/PrePost/PrePost.html) is another bit of research which went the DFD route, but it uses spatial instead of DCT domain filtering.

Marco

PS. it working for XviD is fine and well, but if XviD is going to ignore profiles it might as well ignore strict standard compliance for things as not following a VOL header with an I-frame too ... lots of hardware decoders support nothing beyond advanced simple profile.

[Marc FD]

>DFD tells you a lot about what is noise and what is signal

that's what i thought too. i'll try an approach were i optimise pre-filtering using DFD.

> It just cant do the things motion compensated spatiotemporal filtering
> and adaptive R-D optimized pre-filtering can do, it is a lot easier
> though.

that's why i'm playing with that ^^

>If you want to go that route you will probably want to consider
>non-linear filters, they can handle edges better than linear filters
>... something like this (this is old, and might be totally outdated).

i'll look at it.

>Here is another bit of research which went the DFD route, but it uses
>spatial instead of DCT domain filtering.

i personnaly prefer a spatial approach.

thx for ressources, i'll code a bit on XviD today ^^

>PS. it working for XviD is fine and well, but if XviD is going to
>ignore profiles it might as well ignore strict standard compliance for
> things as not following a VOL header with an I-frame too ... lots of
>hardware decoders support nothing beyond advanced simple profile.

that's the problem, advanced simple profile is a bloated part of the standart. only this profile allow bvops/qpel/gmc :-/
BTW, what's the profile for H.264 tools ?? maybe they added more powerfull profiles in MPEG-4 VM 10 ??

marc ^^

[MfA]

Originally posted by Marc FD
i'll try an approach were i optimise pre-filtering using DFD.

Whichever way you do it, if you work on the original image blocks unless you check the DFD before and after filtering (and just throw the filtered results away if they "increased" the DFD) it is very hard to determine what the effect of your filter will be.

A problem you wont have with non adaptive image based filters such as the nonlinear one I mentioned (which on reflection probably wasnt a very good idea, I doubt it will do better than smartsmoother) because they filter all frames equally, if the reference images were identically filtered there is no problem. But these kind of filters dont really need to be tightly bound with XviD.

i personnaly prefer a spatial approach.

Shrug, as long as it gets better results ... it is much more work than simply scaling the DCT coefficients though.

thx for ressources, i'll code a bit on XviD today ^^

XviD needs a plugin API more than filters BTW :)

that's the problem, advanced simple profile is a bloated part of the standart. only this profile allow bvops/qpel/gmc :-/
BTW, what's the profile for H.264 tools ?? maybe they added more powerfull profiles in MPEG-4 VM 10 ??

It has an entirely new set of profiles which I do not know from memory, it will have the same "issues" though ... it will have simple and advanced profiles, and while from a software point of view it is attractive to just mix and match that would make interoperability with hardware solutions impossible.

[Marc FD]

>Whichever way you do it, if you work on the original image blocks
>unless you check the DFD before and after filtering (and just throw
>the filtered results away if they "increased" the DFD) it is very hard
>to determine what the effect of your filter will be.

that's what i do. the filtering is selected with DFD, and filtering is spatial only (to avoid dumb ghosting).

>Shrug, as long as it gets better results ... it is much more work than
>simply scaling the DCT coefficients though.

i already tried DCT scaling and i don't like the result at all.
IMHO, we should not touch fDCT/Q for fine pre-processing.

BTW, i tried a bit basic DFD filtering (3x3 blur), and it gives 10% comp at q2/ 5% at q10. not bad, but the image is not really better (nor worse ^^), so i'll try my first idea. (who's just harder to implement)

[MfA]

Originally posted by Marc FD
that's what i do. the filtering is selected with DFD, and filtering is spatial only (to avoid dumb ghosting).

That is not what I meant ... just knowing that there is an error before filtering is not enough. If you filter the original image block with a "filter strength" determined by the error before filtering you have no idea what is going to happen to the error after filtering, if it takes less bits to code it is by sheer luck :)

BTW, i tried a bit basic DFD filtering (3x3 blur), and it gives 10% comp at q2/ 5% at q10. not bad, but the image is not really better (nor worse ^^), so i'll try my first idea. (who's just harder to implement)

Well unless it was noisy to start with it isnt supposed to look better.

Marco

[Marc FD]

>That is not what I meant ... just knowing that there is an error before
>filtering is not enough. If you filter the original image block with a
>"filter strength" determined by the error before filtering you have no >idea what is going to happen to the error after filtering, if it takes
>less bits to code it is by sheer

i'm not sure i understand. i DFD-optimise my smoothing (filtering is like avisynth spatialsoften), so the smoothing is _always_ reducing DFD after filtering in intensity. of course, it doesn't mean it would have the same inpact en frequency, but of course it will help.

>Well unless it was noisy to start with it isnt supposed to look better.

yes of course. but my source is clean DivX-SBC (300 Ko/s) and i get now 20% compressibility improvement with slightly reduced ringing at quant2. not bad, when i think using 2dcleaner (3x3,16) on the same source give only 5% compressibility improvement.
at quant 10, i gain only 5% compressibility.

BTW, i'm testing on anime, it's of course very different from movies.
but i don't have movies at home so i think i'll MMX-optimise my stuff and make some "XviD with preprocessing" builds for friends ^_^

anybody want to test here ??

[MfA]

Originally posted by Marc FD
i'm not sure i understand. i DFD-optimise my smoothing (filtering is like avisynth spatialsoften), so the smoothing is _always_ reducing DFD after filtering in intensity.

That is only true if your filtering works on the DFD itself. If you smooth the present frame and subtract the motion compensated reference again you have no idea wether the new DFD will actually be smoother ...

Say that after motion compensation you have a DC level shift for a MB, with a high SAD, coding that without filtering would just give you a DC component to code. If you filter the image however and calculate a new DFD it will actually become more complex taking more bits to encode.

Marco

[Marc FD]

>That is only true if your filtering works on the DFD itself. If you
>smooth the present frame and subtract the motion compensated reference
>again you have no idea wether the new DFD will actually be smoother ...

no, i don't work on the DFD.
i soften in a temporary location, and then i compare current and cleaned current vs compensated reference. i always keep the best.

this way i get the smallest DFD intensity possible, but DFD is still "natural" : i don't filter it.

>Say that after motion compensation you have a DC level shift for a MB,
>with a high SAD, coding that without filtering would just give you a DC
>component to code. If you filter the image however and calculate a new
>DFD it will actually become more complex taking more bits to encode.

i clean _during_ motion compensation.

i've almost finished, i'm just going to do the MMX stuff, because using 3x3 smartsoften on entires CCIR images (i'm testing on DVD now) is too slow, i get 50% speed decrease in XviD encoding...

[MfA]

Oh I misunderstood ... are you checking the error just by SAD or also by other means?

[Marc FD]

absolute difference for each pixel. no sad.
no adaptive settings (only quant-adaptive)

i've finished the MMX algo, i just want to make a 3d avisynth filter with for fun. i'm going to implement it. i hope it'll improve a _lot_ speed.

[MfA]

You make the choice per pixel?

[Marc FD]

yes, i know i could do more complex things but i feel per-pixel is good ^^

[zyrill]

have i missed a post or is there still no version for non-p4? if not: :( (athlon) if there is: :D and pls drop me a line.

[trbarry]

have i missed a post or is there still no version for non-p4? if not: (athlon) if there is: and pls drop me a line.

No version of DctFilter?

Mine should have no processor requirements above mmx though it will do special code for higher ones. Note my version supports Avisynth 2.5a YV12 only though SansGrip posted a more general one above for YUY2 and other releases. So if it's crashing make sure you've got the right one.

- Tom

[seewen]

You said that you didn't add "init2" to DctFilter beccause Sansgrip have worked on it, and it will be "better".


But DctFilter works already with 2.5 alpha. So is it possible that you just add "init2" to the 2.5alpha ?

So we can use this filter without having to wait "too long" for Sansgrip ;)

[trbarry]

It seems more likely that you are not using the later 2.5 alpha or beta that requires the init2.

Or else you didn't get it from my site.

- Tom

[Boulder]

I think Seewen asked if you could compile the old (current) YV12 version with init2 so that he (we:D ) could use it with AVS2.5 beta.

[seewen]

I get it from your site ( trbarry.com/dctfilter ), but there's no "init2" in it.

SO it works very well with Avisynth 2.5 from 18.1.03 , but not with older versions

[trbarry]

I'll try to work on DctFilter today. I don't want to lose the YUY2 support but it really shouldn't be much work to make it function properly with VS6, 2.5 beta, etc.

And I had some other ideas I'd like to add to it later so it's time for me to get back to it.

Hopefully I'll post one by tonight if nothing else comes up.

- Tom

[seewen]

Originally posted by trbarry
I'll try to work on DctFilter today. I don't want to lose the YUY2 support but it really shouldn't be much work to make it function properly with VS6, 2.5 beta, etc.

And I had some other ideas I'd like to add to it later so it's time for me to get back to it.

Hopefully I'll post one by tonight if nothing else comes up.

- Tom

That's really nice ;)

Beccause this filter is really missing ;) ( DctFilter & MpegDecoder-working-with-Mpeg1, and it will be great )

[trbarry]

... but get it at www.trbarry.com/DctFilter.zip .

It has support for Avisynth 2.5 beta, VS6, YV12, and SansGrip's YUY2 code. (Thank's SansGrip :) )

No other new features or changes yet.

- Tom

[Bulletproof]

I have a question, which I think is related. DCT is used in many video compression standards, such as MPEG-4 and MPEG-2. My question is, couldn't the MPEG-4 compressor benefit from the MPEG-2 files already containing quantizing information. Isn't there alot of efficiency lost in making a locked quantizer 2 file on scenes that have high quantization done on them. Couldn't there be a way to read the MPEG-2 file's quantization/DCT information per frame so that better decisions could be made in the MPEG-4 compression process? For instance, if a frame in the original MPEG-2 file has high quantization and you've used a locked 2x quantizer, that increases the filesize for no reason. I realize that you can achieve proper compression where quantizer 2 will be used in the appropriate situation with the right settings, however I don't think it would be "dead-on". What I'm saying I guess is to add quantizer decisions based on the MPEG-2 frames quantization/DCT information.

[seewen]

Thank you very much

You've done it very fast, and so far it works perfectly ;)

[trbarry]

optimist (falling past 20'th floor): "So far, so good." ;)

Bulletproof -

Dunno. Maybe someone else can answer that. But there is a thread now in the development forum about transcoding and shrinking MPEG2 files.

- Tom

[Boulder]

Thank you maestro;)

[iago]

Thanks Tom!
I'm just starting a 120min 1CD encode keeping the 350mb ac3 audio :D.

[trbarry]

I'm afraid to ask. ;)

[iago]

Ah, of course I was kidding Tom ;), but I just want to report that, with the script below,

-----------------------------------
mpeg2source("C:\movie\movie.d2v")
crop(12,16,-8,-16)
Trim(16000,17000) # a short bright scene from U-Turn
BicubicResize(512,272,0,0.5)
DctFilter(1,1,1,1,1,.5,.5,0)
-----------------------------------

and using constant quantizer 2 with XviD Koepi's latest (and AviSynth 2.5 beta), the filesize is 20774kb whereas without DctFilter in the same script the filesize is 25062kb.

Using b-frames 3-100-200, the filesize drops to 16658kb, without losing much from its visual quality.

I don't know what if you add some UnFilter to the script to remove the stairstepping artifacts, etc. ?! ;)

regards,
iago

Btw, why are you afraid Tom? The photo is taken after the last plastic surgery I had. I dared not send a formerly taken one :D.

[mikeson]

@iago:

Is this really you? ;)