dancing blocks problem (DCT?) dark scenes only (anime)

[EcchiNut]

Not sure which forum this should go in, but since it happens whenever I encode it to Xvid, I figure its probably one of the causes (Not noticing it in Huffy).
Been tearing my hair out on this one for the last couple days. The source looks fine, and despite whatever filters, or Xvid Settings, or even compressions, this problem still persists. Dancing blocks in dark scenes (mainly brown-ish). All other scenes, bright, glowing, whichever, look perfectly fine. But as soon as it gets to these dark scenes, its spits out huge dancing blocks that's driving me nuts.

Source:
http://img78.photobucket.com/albums/v347/EcchiNut/source.png


Encode:
http://img78.photobucket.com/albums/v347/EcchiNut/xvid-blocks.png
(look at the pants and shirt, its really noticable when in motion, since the blocks change positions/patterns).
Suggestions?

[RadicalEd]

Well, the change in color and brightness is already there in the source, it's just very slight and therefore mostly unnoticable. However, it's slight variations like these that don't survive quantization, and thus you get as a biproduct blocks with more noticable boundaries. If you ask me, it looks like a problem inherent to the compression. However, if there were a way to selectively average DC components of blocks that have no AC components with neighboring blocks that have no AC components it might be possible to solve the problem without any severe side-effects.

[EcchiNut]

Yah I think I see it. From a filter standpoint, its already running some de-noisers and smoothers - any particular suggestions? (in "english" preferrably :p )

[Manao]

Post your encoding settings ( AVS script, XviD's settings, especially, did you use adaptive quantization ? )
but since it happens whenever I encode it to Xvid, I figure its probably one of the causes (Not noticing it in Huffy).It's not exactly XviD's fault, it's more an inherent default of the MPEG-4 encoding scheme.

I would try to add some noise at playback, with FFDShow. You shouldn't notice the default anymore.

Some pointers, also ( thnanks to the search button, this subject is a recurent one ) :

http://forum.doom9.org/showthread.php?s=&threadid=76741&highlight=dark+blocks

http://forum.doom9.org/showthread.php?s=&threadid=45431&highlight=dark+blocks

Edit : saw your post, RadicalEd, I agree with you except on that point : However, if there were a way to selectively average DC components of blocks that have no AC components with neighboring blocks that have no AC components it might be possible to solve the problem without any severe side-effects.It would work on an I-frame, but on a p-frame, you also have to take into account the motion compensation, so it won't work anymore.

[RadicalEd]

Oh yeah. Stupid motion compensation :|

[Leak]

Originally posted by EcchiNut
Yah I think I see it. From a filter standpoint, its already running some de-noisers and smoothers - any particular suggestions? (in "english" preferrably :p )

ffdshow's SPP deblocking works great for such artefacts, but the problem is that it eats CPU for breakfast... :( (50%-80% on my P4 2.4 GHz)

np: Vladislav Delay - Anima 3 (http://www.huumeclothing.com/nohuume)

[Video Dude]

I found that using AddGrain in the AviSynth script clears up this "dancing blocks" problem.

AddGrain(3,0,0) is good starting point.

If there is too much grain try AddGrain(2,0,0) and so on, or increase it if you any more grain.

Of course it is a trade off between grain and "dancing blocks".

Some people would rather have grain over "dancing blocks", while others hate grain. It comes down to what you like.

[lordadmira]

To really characterize how much Xvid is hurting u here u should encode the problem scenes at constant quant 1 and quant 2 and compare them with ur standard encode. If the blockiness goes away then u know the point of failure. My first impression is that ur encoding to too small a file size. Do a histogram equalize on the output frame to get a better handle on the blockiness (photoshop etc.). U might be able to solve the problem by setting a constant quant 2 zone for these areas. Enabling Chroma Optimizer should also help some.

LA

[EcchiNut]

Manao: I've done probably 80+ tests on this scene, maybe more, just with Xvid setting changes alone (turning something off, and adding more bitrate). It really shouldnt matter at this point, but by default, I always have AQ off, I did turn it on during one of the test-runs, without any visual improvements. Litterally I have done every imaginable outcome on this scene, GMC off/on, qpel off/on, AQ off/on.. etc.. at variety of bitrates.. 500, 1000, 1500, 2000, Q2, Q1.5, Q1, etc. Nothing had any real specific bonus. But by default Im usually running:
H263, GMC, 2pass, default b-vop, packbitstream OFF, unrestricted profile, Chroma Optimizer enabled, motion-6, vhq4, chroma motion, max I-frame interval 350, 2-31 for all min/max frame types and trellis.
Again it doesnt really matter in this case, since I have tested each one now off/on with varying bitrates with no success.

Filters also shouldnt matter much, since I've done them with and without avisynth filters. The last batch was a Lascosz Resize (source is bigger then encode and is a mpeg2 thats been done in huffy with all these (on and off - again it depends, i will sometimes frameserve directly from the original)), VagueDenoiser, Mipsmooth, and some Deen. All with varying strengths.

Ive been playing with Blockbuster, BlindPP, and DCTFilter - but these filters are kinda useless since the source isnt really that "blocky" - its the outcome that has it.

This is the first time this has really been an issue for me. Past encodes with anime, even dark scenes seemed fine (back then). Its really noticable with that brown color for some reason. *shrugs*

[lordadmira]

Well if it persists at quant 1 then the only thing left is to go to another matrix. U need one that gives u enough of a quality boost to get rid of the blocks but doesn't blow ur bit budget. Try Six-of-Nine, HVS-Best, others. Fox Home Entertainment is probably the highest quality matrix I know of.

LA

[Manao]

BlockBuster isn't meant to eliminate blocks which are in the source, but to avoid blocks while encoding.

Don't use AQ & treillis ( both will try to drop DCT coefficient on simple textures such as your brown area ), nor H263.

If you can't afford noise at playback, or if you really want to try to make the moving blocks disappear from the encoding, then try what Video Dude proposed : AddGrain or BlockBuster. Moreover, you can apply them just to the dark areas by using the Masktools :noisyclip = source.AddGrain(3,0,0)
darkmask = source.Binarize(threshold = 20, upper = true, Y = 3, U = 1, V = 1)
result = source.MaskedMerge(noisyclip, darkmask, Y = 3, U = 2, V = 2)Play with the value of threshold in binarize to set the zone on which you want to apply AddGrain.

[MasterYoshidino]

here's my suggestion.put all your ivtc, deinterlace, blurs, softeners etc. here
#at the very last line, after resize and crops, and i mean the BOTTOM of your script
lumafilter(lumoff=-2).unfilter(5,5)if you got good results in your uncompressed raw, then unfilter should add bytes (grain) at the compressor to reduce this DCT artifact. if your pre processing script is bad then it won't help as much. just don't try high quants i.e. >4

the reason that lumafilter and unfilter must be at the bottom is to maximize grain and minimize potential to accidentially filter it our prior to DCT, thus minimizing potential to over DCT your source and spit out blocky frames. i usually encode constant quant 3 with no b-frames and if my preprocessing is right, never a macroblock sees day.

[drcl]

Maybe I am being to simple but surely adding grain to the source reduces available bitrate and will cause a greater problem whilst also lowering the sharpness. the lines in Line Animation are quite demanding of dct methods are they not?

Isn't this the moving walls problem?

The simplest and I've found the best way to combat this problem which also occurs when using libavcodec mpeg1/2/4 is to use a temporal smoother. temporal smoothers are perfect for line animation as there is not supposed to be any texture in the plain areas so You dont get the tell-tale ghosting you may get with natural sources.

If you set long enough ranges on the temporal variable and low enough thresholds on the luminance you should be able to acheive a perfectly even pixel value across the frames thus removing the chance of this artifact cropping up.

I've used the above method on many USA/UK produced animations. Is anime different in some way?

[MasterYoshidino]

all anime short of those that are done with pure pc cells (not handdrawn) will contain film grain. if you are using mpeg2dec3 then I recommend you decode using zero PP unless you notice that the source is heavily compressed (test for high Q like above 8 by setting the showq=true flag in the mpeg2source line). the PP tends to increase temporal instability imho. try to encode with it off.MPEG2source("source.d2v",cpu=0,cpu2="000000",iPP=true,idct=3)
#or idct=0 to use whatever you used in dvd2aviif you really need to spatially blur your source I recommend something soft like deen("m2d",4,4,4). don't use _2dcleanyuy2 as it blocks up anything guaranteed. you should not notice any major macroblock if you preserve the original grain prior to encoding (provided you use Q2). At Q3 and above you do risk some blockiness to remain due to increased compression. The grain helps prevent overcompression, and tends to get blurred/lost through compression. That type of blockiness (flat area) is hard to avoid but if you don't do any type of spatial softening at all (not even PP) then as long as you use Q2 there should be no luma blocks that big. lumafilter + unfilter helps give better looking pans in anime imho, especially on flat color areas. also try gmc enabled. temporal filters are definately going to help though. for simple temporal filtering with little chance of ghosting or funny looking luma forward predicted ghosting (ala temporalsoften) try temporalcleaner(2,2).

if you want my opinion your source frame is way too softened to stand a chance of coming out without blocks. grain = good. overblur = bad.