different approach to deblocking

[dukey]

The xvid deblocker does a good job of removing blocks from colour graduation gradients and scenes which are generally starved of bitrate. But it all too often destroys the details in videos and leaves them looking very washed out, even when the original picture contains no visible blocks rendering it pretty much useless. The problem is, I think that the deblocker deblocks everything, even the parts which are not really visible as blocky.

My suggestion for an xvid deblocker, is thus:
ignore the boundaries between the mpeg blocks and focus on the spacial resolution of the image within the block. From my observations most of the time when I see artifacts, the spacial resolution of the parts of the image are often very low. If you could plot a histogram of the of the blocks you would probably see just a few large spikes.

I did a few 'test' images by taking a frame, one copy was the original. The 2nd copy was deblocked. Then i cut out the parts which were very low in detail from the deblocked image and pasted these over the original undeblocked image, and was pretty pleased with the results i was getting. The parts of the video which were in focus were kept sharp, and the parts of the video which were out of focus and blocky were nicely smoothed. I'll post some images later if i can find some good results. I am not sure if or how well this would scale to a whole video, but i think perhaps its worth a try :)

[dukey]

okay..
some piccys to try and explain what I am on about

First pic shows what happens when I enable deblocking, on a part of a video which really doesn't need deblocking. It is much more obvious when the video is playing but if you look carefully at the screenshot you will see his neck looks somewhat smoothed out, a bit like plastic.

http://img443.imageshack.us/img443/5348/unnecessarydeblock1aq.png

Next image is one which shows blockiness. Our eyes making the blocking in this image worse, because they are detecting the edges on the solid colour boundaries and enhancing them. I think this is called mach banding. It is this we really want to eliminate using normal deblocking, but without wiping uncessary details like in the screenshot above.

http://img248.imageshack.us/img248/4031/original18gs.png

Next image shows the deblocked version of this. (using normal deblocker). The very low detail parts of the image have been nicely smoothed out, but also areas with some mpeg noise but higher detail have been smoothed out too. Most noticably her face.

http://img75.imageshack.us/img75/9527/deblocked11pw.png

The final image is my proposed deblocking idea. The areas of the picture with the lowest spatial resolution, ie the blocks containing only a few colours have been deblocked. And blocks which contain more detail, regardless of the mpeg block noise have been left untouched. As a result her face still contains a little bit of mpeg noise. It is not too noticable but the detail remains. And the areas of the image which have practally no detail (the muddy water) are nicely smoothed out.

http://img75.imageshack.us/img75/1483/mydeblock16oe.png

I've tried this with a few screenshots and generally pleased with the results im getting. I think it would be interesting to see a video filter do this. It has got to be better than the current deblock everything filter :p

[Guest]

It's a worthwhile idea. Why don't you implement it?

[Didée]

@dukey:

Feed the search engine with "deblock_qed", and try out what you'll find. ;)

It's not really a "new" deblocker. The script uses the "deblock()" filter from DGDecode (or MVTools) as base filter, and modifies its result in exactly that way: remove blocking on block boundaries only, while keeping as much detail as possible.

The Avisynth function probably is too slow to be used for realtime processing, but for offline-processing the results are quite nice, as long as the blocking is not disastrous.
Implemented in a plugin or a decoder, speed should be no problem at all.
The script implementation was only meant as proof-of-concept, hence the "qed" appendix. I wonder if anyone ever feels like putting the idea into something that can be fed into a compiler ...

[Inventive Software]

Seems a good idea, though it's a little hard to distinguish what you're on about. Glad to see somebody likes Invasion though!!

[dukey]

I could be wrong
but from looking at the xvid source, i think the xvid deblocker is a direct implimentation of the follow paper:
http://neuron2.net/library/deblocking.pdf

A quote from the paper which I think gives a valuable clue in explaining why the deblocker kills fine details.

When we consider a smoothing operation to remove blocking artifactsin video coding, we find three interesting observations. First,the HVS is more sensitive to blocking artifacts in flat regions than in complex regions [9]. Therefore, a strong smoothing filter is required on those regions. In complex regions, how-ever, smoothing of a few pixels around the block boundaryis enough to achieve a desired deblocking effect. Second,smoothing operations tend to introduce more undesirable blur in complex regions than in flat regions. Hence, adaptive smoothing to preserve image details is desirable in complex regions. Third, because of motion compensation, blocking artifacts are propagated, and the propagated artifacts are more visible in flat regions than in complex regions. Therefore,smoothing in flat regions must cover the inside of a block as well as block boundaries.From the observations above, it is found that the use oftwo separate filters depending on local image characteristics ispreferable for effective deblocking. The filter for flat regionsshould provide a strong smoothing effect inside a block aswell as on block boundaries. On the other hand, the filter forcomplex regions is required to work only on block boundaries.

So, i think the xvid deblocker has 2 modes.

1 in flat regions to smooth the entire block.
2 for detailed regions only smooth the block boundaries.

The problem i can see is all too often it recognises areas as 'flat' and smooths the entire block regardless of whether it actually needs smoothing or not, wiping it of very fine details leaving a very plastic look. Instead of the 'flatness' test
a simple unique colour counter across the block boundary might work. The less colours there are the more likely it is the block has very little detail. Where as the normal 'flatness' test probably marks blocks to be smoothed when the colours are very close together.

[foxyshadis]

The problem i can see is all too often it recognises areas as 'flat' and smooths the entire block regardless of whether it actually needs smoothing or not, wiping it of very fine details leaving a very plastic look. Instead of the 'flatness' test
a simple unique colour counter across the block boundary might work. The less colours there are the more likely it is the block has very little detail. Where as the normal 'flatness' test probably marks blocks to be smoothed when the colours are very close together.
I'm pretty sure the current xvid deblocker only uses the block's quantizer when estimating "detail" level. Just for fun I checked the sources, and though it's a little over my head, the checks on which filter mode to apply are all based on the quant.

[dukey]

you sure you are looking at the same source code as me ? :)

heres a quote from the link i pasted earlier..

http://img471.imageshack.us/img471/6462/untitled7kr2.png

then the xvid source ..

/* First, decide whether to use default or DC-offset mode */ \
\
eq_cnt = 0; \
\
eq_cnt += tbls->xvid_thresh_tbl[s[0] - s[1] + 255]; \
eq_cnt += tbls->xvid_thresh_tbl[s[1] - s[2] + 255]; \
eq_cnt += tbls->xvid_thresh_tbl[s[2] - s[3] + 255]; \
eq_cnt += tbls->xvid_thresh_tbl[s[3] - s[4] + 255]; \
eq_cnt += tbls->xvid_thresh_tbl[s[4] - s[5] + 255]; \
eq_cnt += tbls->xvid_thresh_tbl[s[5] - s[6] + 255]; \
eq_cnt += tbls->xvid_thresh_tbl[s[6] - s[7] + 255]; \
eq_cnt += tbls->xvid_thresh_tbl[s[7] - s[8] + 255]; \
\
if(eq_cnt < THR2) { /* Default mode */ \

}

else { /* DC-offset mode */ \

}



I am not sure I fully understand what the look up table there is doing.