Integer DCT/iDCT filter questions

[MrBunny]

@Bulletproof

I think I understand what you're saying. I guess it's somewhat based on how people believe DVD2One works in which it takes the source's existing motion vectors and manipulates them without decoding the frames themselves and then reencoding them like other encoders would. I don't think this could work within avisynth since it seems the first step is always to decode the mpeg2 stream (mpeg2dec). As a standalone encoder, it might be possible.

However, I believe that the ratio of I/P/B frames in DVD mpeg2 streams is also very different than that for high compression mpeg4 streams. I think MPEG2 has 1 I-frame for every say 15-20 frames or so, as compared to about 1/250+ for mpeg4. In order to create a P/B frame out of the source's I-frame, you would be forced to decode that frame and one or two neighboring frames before preforming motion analysis.

Finally, and this one I'm really not sure about, since MPEG-2 is field based, I would assume that the fields themselves are compressed (using DCT/quantization) rather than frames. As a result, to convert it to a frame based format like mpeg4, I would believe that you'd need to decode the fields and process them. In contrast, with something like DVD2One, since it's going from fieldbased to fieldbased, they don't have to worry about frames.

Without decoding the image, it would become near impossible to filter the image (at least temporally or spatially, since the encoder won't "see" the frames), and I think most people here agree that filtering can be a very important factor on the quality of any encode.

Please please correct me if I'm wrong in anything (or everything) I've said. I'm just basing this on what I think I know and the information I've been picking up here and there.

Mr. B

[iago]

@Tom,

There is something I really wonder a lot about DctFilter, which I still haven't figured out:

Is it reliable/suitable to use DctFilter in a 2-pass encode, or would it be better/safer to use it for 1-pass (CBR or constant quant) encodes? ;)

edit - well, might sound a bit weird, that question of mine; but I have a couple of question marks on my mind, which is difficult to put more clearly into words for me now, even in this insomniac mood! ;)

[MattO]

trbarry,

The following is the main part of my VCD .AVS:
----------
AssumeFrameBased().SeparateFields().SelectOdd()
#DctFilter(1,1,1,1,1,.5,.5,0)
Crop(11,36,698,216).cnr2()
SimpleResize(336,160)
FluxSmooth(15,8)
MergeLuma(Sharpen2(0.2))
DctFilter(1,1,1,1,1,.5,.5,0)
LegalClip()
Blockbuster( method="noise", variance=.3, seed=1 )
AddBorders(8,64,8,64)
Sampler()
----------

Encoding as it is I get better compression but 2 groove-like lines appear at the top of the image.
If I encode using the other DCTFilter line compression is nowhere near as good, but the 2 lines are gone.
Do you have any ideas as to why this occurs?
Is it something I am doing wrong?

MattO

[MfA]

The problem with filtering in the DCT domain is that the filtering isnt translation independent, so when something moves from one frame to the next the filter will have a slightly different effect ... this of course causes extra errors after motion compensation.

[MaTTeR]

Originally posted by MattO
Is it something I am doing wrong? You should always crop using even numbers, worth a shot.

@Tom,
DCTfilter is very impressive from what I can see. I stole iago's settings posted above and honestly can't see any visual difference but the compressibilty is certainly welcomed;) Great work as always!

[trbarry]

MattO -

Try to narrow it down a bit more to see the minimum number of filters that cause the problem. Also try cropping in multiples of 4 to see if it makes a difference.

Finally, I think that AssumeFrameBased().SeparateFields().SelectOdd() will not have exactly the same effect in YV12 as in YUY2, at least if the material is not really progressive.

- Tom

[trbarry]

The problem with filtering in the DCT domain is that the filtering isnt translation independent, so when something moves from one frame to the next the filter will have a slightly different effect ... this of course causes extra errors after motion compensation.

Certainly true. But I don't think there is anything particularly I can do about it. And when it's the last filter it should be working on the same 8x8 pixel blocks that Xvid is going to have to use anyway. So I think that's a general problem of all block based codecs.

DctFilter was just an experiment. It has it's most dramatic improvements using Xvid fixed quant=2 which maybe isn't realistic for most uses. Using other Xvid methods it loses much of it's effectiveness since Xvid or Divx are pretty good at quantizing those same high frequencies anyway when you let them. And you can even make custom quant matrices that may have much the same effect.

But reports are that DctFiler also helps with some other codecs so I'll probably continue to play with the idea.

- Tom

[spyder]

I got a dramatic reduction in filesize with no visible qulaity loss using:

DctFilter(1,1,1,.75,.5,.5,.25,0)

The constant quant=2 xvid of the unfiltered clip was 10.6MB and the filtered one was 7.93MB. The average bitrate of the unfiltered was 241.989KB/s and the filtered was 180.979KB/s(accoriding to windows property dialogs). According to ffdshow, at one point in the clip the bitrate reaches 1900-2000kbps in the unfiltered clip. At the same spot in the filtered clip it is reaching 1500-1600kbps. I find this filter very useful. As my latest attempts at 1CD encodes with convolution3d have a very small first pass size, sometimes just barely over my desired size, I think it's a great filter. Keep up the good work.

[Bulletproof]

trbarry I know this isn't the right thread, but I think it would be better if I just asked about this here.

It seems Deen/eDeen is possibly causing a problem with your plugin UnFilter. I seem to be getting access violations when using it. Donald has said that it initiates an MMX mode which it never lets go and causes other plugins after it to mess up, he was able to make a workaround for his Dup filter, can you do the same for UnFilter?

[trbarry]

Bulletproof -

I know nothing about Deen but that sort of problem is typically when someone leaves out a mmx emms instruction. Filters that use floating point can protect against it by doing an emms instruction on entry.

But do you really mean UnFilter? I just did a quick scan and UnFilter does not seem to use any floating point instructions so it should not be vulnerable to this.

And if you mean DctFilter, it uses float only doing setup and that part looks like it is already protected.

- Tom

[Bulletproof]

Hmm, I seem to have trouble replicating the error again this afternoon. What was happening last night was that when I added UnFilter at the bottom of my script, the video would encode for a couple of seconds and then avisynth would give me an access violation. Sometimes when scrolling between the video it would give an access violation too, but when I took out UnFilter it stopped doing that. It's possible that my computer was in a sort of unstable/bad state causing it to give access violations too, I did do a reboot this morning. If I'm able to replicate the problem again I'll let you know, thanks for your time.

[tangent]

Hi, DctFilter seems pretty interesting and can be really useful for low bitrate rips. Currently using it in my valiant attempt to encode the extended edition of LotR onto 1 CD :)

Anyway, I talked to SysKin about the possibilties of integrating a lowpass DCT filter into XviD since it would be much faster doing it during the encoding without needing an extra DCT+iDCT cycle pair. He got an XviD compile ready (in only 5 mins) which zeros 15 coefficients from the 8th row and 8th column.

I suggested another way of zeroing coefficients which may provide more compressibility efficiency. Instead of zeroing from 8th row/column, zero the diagonals starting from the F diagonal, E diagonal inwards.
If you zero diagonals A-E, that would be 15 coefficients too.


12345678
23456789
3456789A
456789AB
56789ABC
6789ABCD
789ABCDE
89ABCDEF


I believe this way may be better because if I am not wrong, MPEG encodes the 8x8 block in this diagonal order, starting from diagonal 1, 2, 3 all the way until F. When the remaining coefficients are all 0, the block encoder puts in an "End of Block" signal and doesn't encode anymore coefficients. So doing a low pass filtering this way would simply chop off the tail of the block stream and placing the "End of Block" signal in place.

SysKin also compiled a binary which uses the second method. I haven't had the time to test them myself, but according to SysKin, the first compile gave savings of 3% without perceptible quality loss, the second compile gave savings of 2.5% without perceptible quality loss (and supposedly less loss). It might be possible to zero the 9th diagonal for a better space saving and also keep the loss imperceptible.

I haven't obtained permission from SysKin to distribute the compile, but if you are interested, you could PM me your email and I'll email it to you if SysKin is fine with it. Not sure if it works correctly on all architectures though, someone with a P4 had crashing problems with it.

[MrBunny]

@tangent

I agree with most of what you said, however as Tom has noted earlier the DCTfilter isn't really that useful for Xvid. This is because it already has the option to create custom quantization matrices, in which you can apply any DCT based filtering you want (like scaling or zeroing certain components). As well, another reason is that although when using the DCT filter, it looks good on "paper" (the compressibility rate can go way up), but trying to get xvid to compress it further is harder (and uglier) since you've already removed all the high frequency components that it would be dropping to lower the bitrate. Basically, you're doing exactly what xvid would have done to try to scale it's bitrate anyways. Although I guess having a lower average quantizer might be of some help (any lurking Xvid experts want to comment on that? );) At the same time, when you start getting really agressive with the high coefficient filtering, you really start to erode fine details, especially around the edges, just toss some test scripts into TheWEF's great avscompare tool and you should see the difference. Of course, I've been told that I'm excessively picky about barely noticeable "flaws" when filtering.
That having been said, I would agree completely that changing the DCT filter to work diagonally, as MPEG does is probably a better idea than column/row based as it is now.

Mr. B

[CruNcher]

someone with a P4 had crashing problems with it

hmmm who could that be :rolleyes:

indeed a nice filter keep on :)

[MfA]

The effect of the quantization matrix is different from a filter, using larger entries in the quantization matrix on average decreases the size of corresponding coefficients because it truncates a lot of them to 0 ... but the one which do get through are still large, just with a large error (with corresponding ringing/mosquito-noise).

You cannot use the quantization matrix to get the exact same results as that of a denoiser/smoother.

[MrBunny]

I agree that a custom quant matrix can't do the same as a denoiser/smoother, but I was talking about comparing custom quant to the DCTfilter. I believe using it should give the same results as using appropriately chosen custom quant matrices. The DCT matrix scales the resulting coefficients, whereas using a custom quant matrix could also scale them in the same way. Using DCTfilter(1,1,1,1,1,1,1,0.5) should be the same as taking the default matrix and doubling every coefficient in the 8th row and column. Of course this doesn't hold for xvid's h263, but if you want to be playing with the quantization, you might as well use a mpeg-custom quant matrix.

Mr. B

[MfA]

It scales the coefficients, and some people are suggesting to add thresholding ... both these things are radically different from what quantization does.

Remember, quantization is followed by dequantization ... you cannot use the quantization matrix to get the same result as a denoiser/smoother, and the proposed filter DOES act like a denoiser/smoother.

[trbarry]

I haven't always been able to predict exactly what it does. That's why I wrote it and am playing around with it.

I'm pretty sure that it tends to make more high frequency coefficent go to zero, just like quantization. But the difference with DctFilter is once they go to zero they stay there. With high quants but a very stong high frequency signal the darn thing will come back to haunt you, causing mosquito noise. If you want that signal then you put up with the echos but if you can live without it then it compresses a little bit better with DctFilter and maybe does not come back with the artifacts. So there might still be some advantage here.

That is, if it was going to generate very high quants then you might prefer to just zero it. But this is just wild speculation on my part.

I think I do agree about using the diagonals, subject to a bit of testing.

I remember someone else earlier in this thread suggested that integrating DctFilter type processing might screw up motion estimation somehow. But I don't know enough about that to be able to say.

However I assume SysKin does. I breathlessly await further results. ;)

- Tom

[trbarry]

BTW, I had early problems reported with P4's and DctFilter, though I developed it on one. So I turned off SSE2 DCT/iDCT.

I seem to remember it had something to do with optimization options but I never really figured it out so DctFilter still does not use the SSE2 calls.

- Tom

[MrBunny]

@MfA

I've kinda lost the flow of the thread, by "the proposed filter", are you talking about the DCTfilter or some other idea someone threw into the middle of the thread. Are you even talking to me? ;)

I think I do see what you're saying about quantization being followed by dequantization. While modifying the custom quant matrix to force the encoded quantized coefficients to be the same as a DCTfiltered approach is possible, those coefficients would then be dequantized by different matrices. The DCTfilter approach would return the DCTfiltered coefficients, whereas the custom quant method would return something near the original minus those coefficients rounded to zero. A custom quant matrix could only mimic the behavior of DCTfilter whose arguments are set to 1 or 0.

Silly of me to only think of half of the equation :o

Mr B.

[tangent]

Oh yeah, SysKin also mentioned that it would be quite easy to make the DctFilter adaptive to the amount of motion (more motion = more agressive filter) since motion vectors are available.

Attached here is the XviD build which chops 15 coefficients from the 8th row/column.

[tangent]

And this one does the diagonal cut.

[trbarry]

"And this one does the diagonal cut."

Pics aren't approved yet.

But some of my HDTV captures seem to have more pixels than resolution. That is, they have a huge resolution of 1920x1080 but have been so filtered or lost so much in compression that there isn't really that much information there.

I have one case now where I don't see much or any difference after using:

DctFilter(1,1,1,1,0,0,0,0)

This would actually zero 48 of the 64 components in the 8x8 DCT block (leaving only the top left corner), yet doesn't seem to matter except removing a little haze.

I'm not sure how DctFilter could become adaptive inside Xvid. That would seem like the kind of case that would mess up motion comp since more high frequency components would exist in one frame to the next. But maybe there's some way to deal with that. I don't know how they do lumi masking either.

- Tom

[cinghio]

Hi TrBarry, hello all guys.

I wonder if your filter could be used (as it is or modified)
as a bridge for the task of transcoding from MPEG2 (without/before iDCT) to MPEG4 (without doing DCT nor re-quantization)

I suppose free resize would be impossible, maybe crop of and integer number of 8x8 [16x16] blocks could be done.

Any way, I only think about a simple transcoding without any resizing/filtering, e. g. for a tipical Personal Video Recorder application.

Maybe I'm going OT, in this case I apologize.

This topic could also be related to direct Xvid,DivX to MPEG4 transcode, and maybe accomplishing that task with little (is it true???) modifications also MPEG2=>MPEG4 would be possible.

I have two questions:
1) In terms of computational times, which is the weight of iDCT(MPEG2) + DCT(MPEG4) + quant(MPEG4) over the total MPEG2 to MPEG4 process?
2) Is there a "MPEG4 encoder" accepting DCT (& quantized) domain as input?

Thanks for any information.

[trbarry]

cinghio -

I hoped someone else would jump in here and answer since I can't answer that myself. I'm not even sure I understand your questions. ;)

I'm not really sure how DctFilter would help with transcoding.

I did do one test last year with a performance monitor on DVD2AVI re-encoding to Xvid. FWIW, it turned out that the DVD2AVI iDCT was using about 5% of the job total.

I'm not sure what you are asking about with "free resize". I did create a resize based upon a similar idea that rapidly doubles the size of a clip. But it was based upon a (non-standard) 4x4 integer transform, not DCT. I'll release it someday, after some more tinkering.

- Tom

[cinghio]

Hi Trbarry,

Thank you for the reply and the precious information about the CPU costs of iDCT.

With "free resize" I mean a not "integer magnitude" one,
and would not be possible assuming information is available in
Discrete Cosine Transformed domain, with 8x8 blocks.
"Constrainted resize" would be possible, i.e. integer magnitude resize (2x for example or 1x, that is no resize) with a "dumb cropping" made discarding entire rows or coloumns of 8x8 blocks.

However, for immediate storage of videos, the idea is to do no transformation and process the DCT domain information of the MPEG2 and assume they were created by the DCT preprocessor of XviD.

Is this kind of approach possible?
Maybe for my purpose a new filter is needed, capable of dealing with DCT domain data before the complete MPEG2 decoding has been done.


If I understood how it works, your filter does a complete DCT, then some transformations in DCT domain and after that an iDCT.

However your code could be integrated in XviD DCT preprocessor (or in Mpeg2dec before doing iDCT from MPEG2 source if you don't need resize),
so that the entire process of DCT filtering and re-encoding, will not cause double stages of DCT and iDCT.

Is XviD capable of accepting DTC domain data in some way?

[trbarry]

hank you for the reply and the precious information about the CPU costs of iDCT.

With "free resize" I mean a not "integer magnitude" one,
and would not be possible assuming information is available in
Discrete Cosine Transformed domain, with 8x8 blocks.
"Constrainted resize" would be possible, i.e. integer magnitude resize (2x for example or 1x, that is no resize) with a "dumb cropping" made discarding entire rows or coloumns of 8x8 blocks.


Okay, I see what you meant. In that Integer transform resizer I actually also implemented 4:6 and 6:4 resizing. If the underlying basis functions are continuously defined (like cosine) then I think you can always IDCT to a different block size. Of course all final results are integer dimensions, but maybe not integer ratios.


However, for immediate storage of videos, the idea is to do no transformation and process the DCT domain information of the MPEG2 and assume they were created by the DCT preprocessor of XviD.

Is this kind of approach possible?


I think you are talking about a transocder here. There is a thread in the Xvid forum where -h is working on one I think.


Maybe for my purpose a new filter is needed, capable of dealing with DCT domain data before the complete MPEG2 decoding has been done.

If I understood how it works, your filter does a complete DCT, then some transformations in DCT domain and after that an iDCT.


For pure filtering purposes it could be added to decode, like in DVD2AVI. I thought Marc FD mentioned some experiments like this above in this thread somewhere. (not too favorably for some reason) But it would be faster.


However your code could be integrated in XviD DCT preprocessor (or in Mpeg2dec before doing iDCT from MPEG2 source if you don't need resize),
so that the entire process of DCT filtering and re-encoding, will not cause double stages of DCT and iDCT.

Is XviD capable of accepting DTC domain data in some way?

I don't think anything like that is anticipated.

- Tom

[trbarry]

For any of you that have been following this long thread, I noted way above that it might be possible to achieve the benefits of DctFilter at no CPU cost in Xvid simply by using custom quant tables.

Well, I've been playing with that a bit and think it does seem to be mostly true. See my Free Filtering with custom quants (http://forum.doom9.org/showthread.php?s=&threadid=45695) thread in the Xvid forum. The example I posted there is a bit extreme (over soft) but I think it's possible to duplicate just about any DctFilter effect.

That is, remove a bit of capture or mosquito noise, soften a bit, and improve compressibility, all without running any extra code.

- Tom

[MrBunny]

Hi Tom,

I believe you are drawing the same incomplete conclusion I made back on page 6, which MfA corrected me on. Your conclusion is only correct if you're zeroing the coefficients (which you were in your example in the other thread). If you're scaling the coefficients (in DCTfilter), then a custom quant matrix would not do the same task as DCTfilter since dequantization must occur after quantization, and so unless the custom matrix zeroed the coefficient, dequantizing that value will result in the original value (with a greater error) rather than the scaled value DCTfilter would give.

Mr B.

[trbarry]

You're right, but it seems to have the desired effect. It's a bit more complicated but I'm basically setting (more than) the lower right half of the quant tables to all 99. And I'll use 255 if I find it's legal and seems to work but I haven't tested it yet that way.

But after DCT there are rarely any high numbers in the DCT results for the higher frequency components anyway, regardless of quants. I noticed that while debugging DctFilter. I believe this means that 99 or higher will really drive almost everything to zero in those values.

I'm sure there are contrived examples where it would fail and cause artifacts but I hope they are very rare in real life (Murphy's Law and some anime textures notwithstanding).

- Tom

[tangent]

I would feel much better about zeroing those coefficients. At least there would be no chance of a rogue non-zero coefficient destroying the macroblock.

[trbarry]

I'd feel better about zeroing them too if I could, but I'm still getting good results with the custom quants. I don't see any particular extra ringing type artifacts. And I still haven't experimented with how high a value I can put in a quant matrix before it croaks.

..... changing horses in mid stream

What does everyone think about a different flavor of DctFilter that would work on diagonals instead of rows & columns like now. The logic above was fairly persuasive that any block based codec using a diagonal or zigzag scan would probably compress better that way.

I could probably woop one up in an hour or 2. Would that be a useful option for anybody?

- Tom

edit: Moderator? If you happen to read this could you correct the spelling on my thread title ("quiestions"). It's annoyed me for months now and I can't change it. :(

[iago]

Originally posted by trbarry
What does everyone think about a different flavor of DctFilter that would work on diagonals instead of rows & columns like now. The logic above was fairly persuasive that any block based codec using a diagonal or zigzag scan would probably compress better that way.

I could probably woop one up in an hour or 2. Would that be a useful option for anybody? Though I am quite pleased with the results I can get with the current version of DctFilter (together with some extra filtering), especially when encoding "really hard-to-compress movies" and aiming for low file sizes, I would be more than willing to test a new version! :)

Kind regards and many thanks one more time for this miraculous toy called DctFilter ;).

iago

[Guest]

Originally posted by trbarry
edit: Moderator? If you happen to read this could you correct the spelling on my thread title ("quiestions"). It's annoyed me for months now and I can't change it. :( Your wish is my command.

I didn't even know I could do it!

Look for rule violating titles to disappear very fast from now on. :devil:

[MfA]

It would be easy to adapt xvid to interpret a 0 in the custom matrix as an indicator it should zero a given coefficient.

[trbarry]

I would love to see that but I thought someone else posted in my Free Filtering (http://forum.doom9.org/showthread.php?s=&threadid=45695) thread that the quants are stored in only 8 bits. If this is so then you can't really store a large enough value to be guaranteed zeroing every single component. And doing logic on each pixel would maybe be prohibitive.

- Tom

[MfA]

The assembly versions of the quantization routines implement the divisions by multiplication and shift, so that is not an issue, but Ill not continue this discussion here ... that thread you mentioned seems a better place.

[Kaiousama]

@TrBarry
Recently i've done some test with your DCTfilter and the results are not as i've expected, check this link for all the Test's details (http://forum.doom9.it/download.php?id=86).

As you can see there is (as expected) a direct correlation between psnr and filter's aggressiveness.
The strange is the fluttuating filesizes, more noticeable if you touch the last value, less noticeable if you change the 7th coefficient.
Another strange thing is that an all 1 settings don't give the original filesize&psnr (if i've well understood an all 1 setting don't scale anithing)
Another not expected behaviour is that a (in my opinion) not to aggressive settings like DctFilter(1,1,1,1,1,.500,0,0)gives an output PSNR like if you increse the conpression's uantitizer by 1, but the filesize is much more reducted if you increase the quantitizer in front of using the DCTfilter

What do you think about?

[trbarry]

Kaiousama -

Great series of tests.

Mostly it's as expected, with gradually declining file sizes and PSNR's. And I can believe some random variance due maybe to strange interactions with motion search etc.

But I can't explain the degree of the INCREASE in file size twice as the parms went all the way to zero. And I really did think I set it up so a parm set of all 1's would leave things unchanged.

So there may be a round off error somewhere when I first set up the tables. I was using floating point during setup and in the conversion it is always possible that something gets truncated. I'll go look for that.

Another not expected behaviour is that a (in my opinion) not to aggressive settings like DctFilter(1,1,1,1,1,.500,0,0)gives an output PSNR like if you increse the conpression's uantitizer by 1, but the filesize is much more reducted if you increase the quantitizer in front of using the DCTfilte

Sorry, I'm not sure what you are saying here.

- Tom

[Kaiousama]

Originally posted by trbarry
So there may be a round off error somewhere when I first set up the tables. I was using floating point during setup and in the conversion it is always possible that something gets truncated. I'll go look for that.
Mhhh.. yes, if you watch only filesizes it seems that one setting every 2 it's bad rounded and leads to unpredictable consequences, but if you look the psnr all things are ok (i've inceased the values by 1/8 every time as you said in the filter's guide)


Sorry, I'm not sure what you are saying here.
I was thinking about the filter's efficience, look:

NO-filter quant.4 size:24.287.232 PSNR:42.2381
DctFilter(1,1,1,1,1,.500,0,0) quant.3 size:30.709.760 PSNR:42.0057

If you look for the same psnr results using DCTfilter the encoding brings you a 6Mb higher filesize in front of increasing the codec quantitizer to reach the same PSNR.

I'm thinking if this behaviour is a sporadic case in my test or if it's the same with some other tests (yuhhuuuu... plz someone test one of your clips to check this results)

Or it's possible that the bitrate saving made by DCTfilter is not well quantitized by the PSNR algorithm... do you know a way to valuate (with a quantitative value) the Sharpness (only that) of a clip?

[trbarry]

I'm not the PSNR expert but I don't know of any objective reliable method that is really any better.

But both with DctFilter and my experiments with custom quant tables the idea is actully to discard some information that we might consider less trustworthy or useful. The fact that we discard it will probably hurt PSNR calculations but it's up to the eyeball to decide whether it would have been worth keeping it. That's really the same process as lossy coding using quantization. This just adds another option.

I think most of the human visual system experiments are more targeted at what the human eye can see. But here we are doing many rips and caps where we also have to consider getting rid of what they eye might not like to see.

For instance, PSNR may measure how well your encoding can faithfuly represent mosquito noise and blocks from a previous encoding. But if a certain level of detail is not reliably represented in the source you are working with then it's maybe better to just discard it and save the space in the process.

- Tom

[tangent]

Is the diagonal cut version of DCTfilter available soon? I would definitely use that in preference to the original DCTfilter. It should be more efficient.

Also, I don't think using values between 0 to 1 (such as 0.5) is a good idea. You're going to get very little bits saving for some quality loss because as long as a coefficient is not zeroed, many more bits would be used to encode it. You may save 1 bit per coefficient, but that's hardly significant.

[trbarry]

Hi tangent -

Your post made me go back and look for your previous questions. That in turn pointed out I'd previously totally misunderstood what you'd attached.

I thought it was just pics of the results, not then yet approved, and not a modified Xvid. And I'd promptly forotten about them, before they got approved. So I'll check that out now.

I haven't made the diagonal version yet, but will soon, possibly today.

As far as using fractional values, I agree as far as custom quants but maybe not for DctFilter parms. I'm not sure about that one yet. It may be okay to just increase the probability that a given coeficient goes to zero but let extreme cases through. I can't even guess how much that helps compression but it doesn't seem it would create the high quant artifacts that doing the same thing with a custom quant table might do.

But for DctFilter anyone can just use 1's and 0's anyway if they wish. Increasingly I do.

- Tom

[bilu]

@trbarry

Originally posted by trbarry

I haven't made the diagonal version yet, but will soon, possibly today.




Got news? :)

Bilu

[trbarry]

I haven't done much programming for a few days but this morning I did make a new version of Dctfilter.

This includes a new function DctFilterD(DiagCt) that works on diagonals. It is simpler and either zeroes diagonals or leaves them alone.

In a 8x8 DCT result there are 15 possible diagonals (visualize a chess board). So if you specify:

DctFilterD(4)

then the 4 diagonals in the lower right corner of the DCT result will be set to 0. The DiagCt parm may be any integer value from 1-14.

I haven't tested this much but, like my results with custom quant tables, it appears if you set DiagCt very large you will start to get edge noise, something like ringing. You can see this in the DctFilterD(10) part of the sample picture I put out at www.trbarry.com/DctFilterD.jpg .

Like usual, get the files (source and dll) at:

www.trbarry.com/Readme_DctFilter.txt and
www.trbarry.com/DctFilter.zip

The diagonal version is certainly easier to specify, and runs a tiny bit faster but I'd be interested in any feedback about whether it really does a better job. I'm still very uncertain about this.

- Tom

[tangent]

Thanks Tom. Time to restart my FotR-in-one-CD experiment :)

[Didée]

I lost a little the track of this thread, and DCT-filter was lying in the corner quite some time now (day has only 24 hours...)

Now, I mad some short tests on the Kung-Fu scene of "Matrix".
Have yet to check visuals, but for sure I don't understand the filesizes I got:
dct(1,..,0,0,0) 32876 kB
dctD(5) 32934 kB
dctD(3) 32874 kB
dctD(7) 32520 kB
plain encode 32279 kB
dctD(8) 32240 kB
dctD(10) 31080 kB
dctD(12) 29036 kB
How can it be that filesize grows, when zeroing out some neglectable high frequencies?
I used XviD with nothing but I-P and mpeg quant @ q2. Oh, and qpel.
Three shots with q7 showed same behaviour.

:confused:

[iago]

@Tom,

I can confirm the paradoxical results about filesizes that Didée mentioned.

h263 / q2 / DctFilterD(6)

plain: 22120kb
DctFilterD(6): 22228kb

??

iago

[tangent]

Weird... could there be some misalignment in the 8x8 blocks?
Perhaps this is really best implemented in the codec itself...

[MfA]

DCT transform is not translation invariant, what this means is that "signal energy" present in one coefficient might be in another coefficient if you shift the image.

One possible explanation of the growth in number of bits is that when there is motion then for each frame the filter is changing the image in a slightly different way, and this happens between each and every frame, after motion compensation that means we get new errors which would not have been there had we not applied the filter. If this effect is greater than the saved number of bits because of smoothing it is a net loss. When performing the zeroing of coefficients in the codec through custom quants this effect should be less of an issue, but this has problems of its own.

Personally I think transform based denoisers/smoothers should really use translation invariant transforms (undecimated wavelets for instance).