Bad Blocking in Flat Colored Sky

[Heaud]

I have this clip where there is a dark blue sky and it blocks badly. Here is the link for the encoded problem clip.
http://www.mediafire.com/?d4tv32mmtyw

I used this avisynth script:
DGDecode_mpeg2source("night sky.d2v", info=3)
ColorMatrix(hints=true)
setmemorymax(225)
source=last
backward_vectors = source.MVAnalyse(isb = true, delta = 1, idx = 1)
forward_vectors = source.MVAnalyse(isb = false, delta = 1, idx = 1)
forward_compensation = source.MVFlow(forward_vectors, idx=1)
backward_compensation = source.MVFlow(backward_vectors, idx=1)
interleave(forward_compensation, source, backward_compensation)
dfttest(sigma =1.07, tbsize=3, sosize=6) #increase sosize for overlap strength and sigma for denoise strength
dull=last
sharp = dull.LimitedSharpenFaster(smode=3, strength = 20, undershoot = 8, ss_x = 1, ss_y = 1)
Soothe(sharp, dull)
selectevery(3,1)
gradfunkmirror(1.3)
Crop(2,0,-2,0)

and encoded in MeGUI with x264 today's build with:
--pass 2 --bitrate 1300 --stats ".stats" --ref 16 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --ratetol 4.0 --me umh --merange 32 --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input" --sar 32:27

Here is the link to the split vob file with avisynth script if you want to try to find a solution yourself. http://www.mediafire.com/?aummdm2z1jn

I thank you in advance for helping me out with this.

[TheRyuu]

dfttest might be oversmoothing it which is creating the gradiants.

If you're using the latest build of x264 then AQ is enabled by default so at least you have that part covered.

[Dark Shikari]

Looks fine to me... you really can't expect any encoder to keep a perfect gradient in that kind of sky. You could try setting qpmin and raising AQ even stronger but there's no magic bullet.

[SPiKA]

I'm not alone... xD

[Heaud]

I did some tweaking with the script and did some changes like Dark suggested and it removed as much blocking as possible.

DGDecode_mpeg2source("night sky.d2v", info=3, cpu=4)
ColorMatrix(hints=true)
setmemorymax(225)
source=last
backward_vectors = source.MVAnalyse(isb = true, delta = 1, idx = 1)
forward_vectors = source.MVAnalyse(isb = false, delta = 1, idx = 1)
forward_compensation = source.MVFlow(forward_vectors, idx=1)
backward_compensation = source.MVFlow(backward_vectors, idx=1)
interleave(forward_compensation, source, backward_compensation)
dfttest(sigma =1.1, tbsize=3, sosize=6) #increase sosize for overlap strength and sigma for denoise strength
dull=last
sharp = dull.LimitedSharpenFaster(smode=3, strength = 20, undershoot = 4, ss_x = 1, ss_y = 1)
Soothe(sharp, dull)
selectevery(3,1)
gradfunkmirror(1.3)
Crop(2,0,-2,0) I added some deblocking on the mpeg2source part of the script in order to help removing any real blocking going on. Dfttest was tuned up a bit since I noticed there was still a small amount of noise left.

--pass 2 --bitrate 1500 --stats ".stats" --ref 16 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --qpmin 1 --ratetol 5.0 --me tesa --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input" --sar 32:27 --aq-strength 1.5 I tried out strength 1.6 and 1.4 on the video. 1.6 gave a slightly more noticeable bit artifacting on the female's hair than 1.5 and 1.4 still had a bit of blocking left. I also switched to SATD exhaustive since it also helped with removing the blocking. Bitrate was also raised to help.

Here's a link to how the video looks with the settings in this post. http://www.mediafire.com/?qz0m5yc2jty

[thetoof]

You could add grain right after gradfun and then stabilize it with GrainOptimizer with a high tdist. This will add some dithering to your video and, because it's been stabilized temporally, the encoder should keep it. (and AQ will help a lot with this)

[dbzgundam]

I haven't taken a look at the sample myself, but dfttest has always been notorious for posterization in whatever I've used it with (though more notoriously in anime).

This could possibly be leading to even further blocking in the compression stage.

[Heaud]

I guess I should try it out with another denoiser. Which one would you recommend for anime? The source isn't noticeably noised but it does have some mosquito noise in high animation areas.

[thetoof]

From what I've seen, the negative impacts of dfttest are not that bad... I've gotten very good results on anime by using smode=0, sbsize=5, tbsize=5 and a low sigma.

You could give FFT3DFilter a try, but it's slightly worse regarding haloing.

Also, the suggestion I gave before can help a lot to have good looking color gradients. Try loading your video with something like this (it's a copy-paste of a script I used on an anime with a LOT of banding) :

dfttest(sigma=1.7,smode=0,sbsize=5,tbsize=5) # ajust sigma accordingly with your source to remove all the noise without losing detail
tomerge=last
gradfun2db(tomerge,3) # ajust the threshold to remove all the "blocks" (i. e. have nice transitions between colors), but be careful as it may screw up your image if you use inappropriate settings
dull=last
sharp=dull.limitedsharpenfaster(strength=175) # have fun tweaking
soothe(sharp,dull)
mt_average(AddGrainC(2),AddGrainC(2,seed=10)) # somewhat stable dithering
grainy=last
clean=grainy.dfttest(sigma=8,smode=0,sbsize=5,tbsize=5) # ajust sigma... it should remove ALL the grain (you will have banding here, but it doesn't matter since it's only a reference clip for the next filter. You can also use a cheaper denoiser to improve speed.
opt=grainoptimizer(grainy,clean,tdist=999) # with a crazy tdist like that, the grain you added before will look like almost static dithering = the transition between colors will not be abrupt and your sky will look better
mergechroma(opt,tomerge,0.95) # all those filters will dull your colors, so with this you'll recover your original good looking colors (and I'd recommend using tweak before all this to improve them) and, since all that was done was mostly for luma, it won't nullify the effects of the filters.

edit: Forgot to mention that all this + AQ will greatly reduce blocking and banding, because since the grain is stable, it's more likely to be kept by the encoder, even at low bitrates.

[Heaud]

Thetoof, I just tried out your script and it locks up any program I try to run the avisynth script with. I will try to work something similiar to what you made.

[thetoof]

It's probably because of your comp... I've got a Quad-core with 4GB of RAM and the complete script (with deinterlacing, deblending, decimating, color correction, crop and resize) runs @ a bit less than 1fps

Try changing the 2nd dfttest with something cheaper like TemporalSoften()

You can check this (http://forum.doom9.org/showthread.php?t=51181) thread to help you find an alternative to dfttest.