De-blocking issues with x264 and animated material

[Dr.Khron]

After a lot of trial and error, I've developed an encoding template for my Simpsons episodes. This encludes using x264 deblocking set to 1, 1.
I was cranking through the episodes when one of them displayed some blocking issues.

I think it has to do with the poor quality of the source they made the DVD from. While there is no blocking in the AVS output, the scene in question suffers from color fluctuations and some flicker. This must give x264 some problems.

So, I made a ten second D2V file for the part with the issues, and encoded it 5 ways: 1-1, 1-2, 2-1, 2-2, and 0,0 (all other encoder settings remain the same)
.
The blocking problem looks exactly the same in the output for ALL of these runs! (thats why I checked the huffy uncompressed file: I wanted to make sure it was an encoder issue, not an AVS issues).

I'm afraid of reducing the image quality for the rest of the clip, so I don't want to turn the deblocking up past 2-2.

Comments, suggestions welcomed.

[check]

are you decoding with coreavc, perchance? That can cause problems on occasion.

[Dr.Khron]

Nope, I'm a big fan the CCCP.

[Manao]

Screenshots ?

[Dr.Khron]

Well, I tried to make a screenshot... but its VERY faint, you almost need to see it in motion to see the blocks. You probably couldn't see it all on a TV, but I have a very expensive monitor.

Basically its a wall of uniform color. I'll get a screenshot up soon.

[Manao]

If you can only see it in motion, post the screenshots of two consecutive frames.

[Dr.Khron]

Here ya go.

http://img178.imageshack.us/img178/4242/test1uy3.jpg


Here it is, blown up and adjusted.

http://img178.imageshack.us/img178/5615/test2cn7.jpg

Thanks again for any help.

[bob0r]

CCCP includes ffdshow or coreavc no?

[burfadel]

The aq-patch seems to help in dark areas like that (afterall, thats what its programmed for!).

[Dr.Khron]

Bob-
Coreavc is not free. Therefore, its not included in CCCP, which uses FFDshow.

Never heard of this AQ patch.. how do I use it?
A search on the forums here wasn't very enlightening.

[chainring]

AQ patch is included in Sharktooth's x264 builds, which is only up to r635. SVN builds are up to r651, or thereabouts. Cef just did a x264 r648 build that includes it as well.

Use this on the "Zones" tab in "Custom Commandline Options" box:
--aq-strength 0.3 Someone else will have to clarify the strength values.

[Dr.Khron]

Ah, very cool, thank you for the direction.
I will DL Sharktooths build and start monkeying with this right now.

It took a lot of searching, but I finally found some good info on how this patch works:

--aq-strength
Usage: --aq-strength <float> (default=0.0)
One part of the settings for Adaptive Quantisation. Adaptive Quantisation allows different parts of a frame to use different quantisers, as opposed to the whole frame using the same quantizer. The common use for this in XviD was that it would compress light and dark areas more, thus freeing bits for parts that were more noticable. Adaptive Quantisation works differently in x264 because it was added by Haali to address a problem that large blue areas could sometimes look blocky. Adaptive Quantisation in x264 assigns more bits to dark and blue areas to help stop them blocking up. This particular setting instructs the amount to adjust QP per Macroblock. Setting 0.0 is no Adaptive Quantisation, and 1.1 is considered strong. It is recommended that you only use this setting if your source is particularly blocky in blue areas only wheras the rest of the image looks fine.

--aq-sensitivity
Usage: --aq-sensitivity <float> (default=15.0)
This switch is to be used in conjunction with --aq-strength, and will not work if --aq-strength is set to 0.0, or not specified (since it is disabled by default). This controls the "flatness" threshold at which to apply AQ (Adaptive Quantisation). Setting it to 5 will apply AQ to almost all blocks, and setting it to 22 will apply it only to flat blocks. A flat block would be described as an area of solid colour or little detail.


Shameless plug: this guy's page had a ton of good information.

http://aflux.deltaanime.net/Zero1/MP4/x264.html

[burfadel]

Use these builds:

http://cef.neuf.fr/x264/

They have the aq patch and are much faster. They're a few versions behind but I'm sure he'll update it soon. 651 was released only a couple of days ago (21st April) and today its already 654. The main benefit would be a slight speed increase... but these builds are so much faster than anything else the 650 would be faster than the 654 from x264.nl :)

[fofwp]

Use these builds:

http://cef.neuf.fr/x264/

They have the aq patch and are much faster. They're a few versions behind but I'm sure he'll update it soon. 651 was released only a couple of days ago (21st April) and today its already 654. The main benefit would be a slight speed increase... but these builds are so much faster than anything else the 650 would be faster than the 654 from x264.nl :)


sorry for the newb questions, but how do builds differ from each other?

Would the video quality depend on the build used?

I use Megui, so I don't know who's build it downloads whenever it updates, but I am guessing Sharktooth's.

[Dark Shikari]

sorry for the newb questions, but how do builds differ from each other?

Would the video quality depend on the build used?

I use Megui, so I don't know who's build it downloads whenever it updates, but I am guessing Sharktooth's.
The builds differ in three ways:

a) Speed, based on optimization options in the compiler, etc.
b) The version of the source code they are based on. Newer is better, of course: both quality and speed-wise, usually.
c) Some may have patches, like the AQ patch.

[Dr.Khron]

Ah, but thats the beauty of open source software development. This is how it works: you've got a group of people who work on the core encoder. The developers for x264 are constantly improving it, to point of some builds being released daily.

However, becuase its open source, people are free to take the vanilla x264 builds and add whatever tweaks or addons they desire. The patch in question was written by Sharktooth, but since he has the patch posted on his webspace, anybody can include it in their build, like Cef did.


Anyway, back to my Simpsons encodes...
I tried out the patch, and did the trick, but I had to turn it up all the way to .9, which I thought was kinda high. However, it didn't seem to effect the rest of the video, so I may go with that.

Meanwhile though, I decided to experiment with some new AVS filters for sharpening. Hopefully, a cleaner input will help a bit with the blocking. Once I get that part figured out, I'll go back to experimenting with the AQ Patch again.

Thanks for all your help. And thanks to Sharktooth for the patch.

[KoD]

I thought the AQ patch was Haali's work.

[burfadel]

I think the aq patch was haali's work! Was it Cef's build that you tried or the older Sharktooth build?

There is a threshold setting with the aq patch. --aq-sensitivity
The default is 15, but in your case a lower value closer to 10 may have been better, or even lower! A setting that works basically :)

[Dr.Khron]

DOH! Yes, my mistake, the documentation clearly states that its Haali's patch. I just got confused becuase I downloaded the patch from Sharktooths site. Extra props to Haali then, even though I don't think I've ever seen him post here.

Burfadel- It was Cefs build that I used. Thank you for the link.

When I get back to the encoding stage I'll try fooling around the the sensitivity option. As I mentioned, right now I'm working on cleaning up the source. I'm happy with Fluxsmooth for denoise, but I can't decide between Msharpen or VMtoon for sharpening. And to complicate things more, if I get some free time today I might trying fooling around with See-Saw a bit.

[Terranigma]

I think the aq patch was haali's work!
Read This Quote (http://forum.doom9.org/showthread.php?p=992795#post992795)

[burfadel]

Have you tried --no-fast-pskip

Its a standard option, but by default its off (that is fast pskip). Basically it was an option added when a new much faster method was found for a certain function but was less accurate? so the option uses the more accurate function... Something along those lines...

But the main point I guess is to try that too. The speed difference isn't massive...

[Dr.Khron]

I've used --no-fast-pskip (box checked in MeGUI) on every iteration of my encoder template, if only because I have most of the "slow" options turned on. I know that most high quality/slow templates use it , and I've read that its especially imortant for animated content.

[foxyshadis]

I'm surprised AQ helped, but I guess it must lower the quant quite a bit in the flattest areas. I'm curious about whether this would be significantly improved by 10-bit DCT alone, or if it would require a dithered conversion to RGB or YUV deband, without having to lower the quant. (Full 10+ bit intermediate YUV stages could help, but you know, so could encoding in rgb and we don't have that either so. :p) My tests with Quenc's 10-bit DC mode are mostly inconclusive.

I'm beginning to wish that dithered conversion to RGB was considered more important than raw speed, since it makes such a huge difference in noiseless encodes.

[akupenguin]

Isn't 10-bit DC equivalent to a CQM? MPEG-1/2 needed a special case because the range of the quantized coefs was limited, but H.264 has no such restriction.

[Manao]

It's a bit different. Even with the highest precision ( 11 bits ), the range is enough. However, higher precision means more bits, so the precision acts like a quantizer.

Intra DC dequantization in mpeg2 is DC * alpha. Alpha = 8 when the DC precision is 8, 4 when it's 9, 2 for 10 and 1 for 11. And alpha is constant whatever the quantizer.

So it's equivalent to a CQM, but with the DC value in that CQM depending on the quantizer used.

For example, at linear quantizer 4, with a 8 bits DC precision, the DC coefficient in the equivalent CQM would be 16. At linear quantizer 2, it would be 32, and at quantizer 8, it would be 8. And a 10 bits DC precision would be equivalent to 4 at quantizer 4, 2 at quantizer 8 and 8 at quantizer 2.

[Dr.Khron]

I'm surprised AQ helped,

Well, in the final analysis, it didn't. I guess I was just in deluding myself.

After two days of fooling around with this, I've decided that AQ doesn't really fix the problem. It does smooth out the blocking considerably, but as I said in the original post, its the MOTION of the blocks thats annoying, no matter how faint they are. I guess the original clip is just super hard to encode with x264, beuase of the color flux.

Test1-no AQ
http://img168.imageshack.us/img168/9562/test1noaqpy7.jpg

Test2-AQ 0.3
http://img243.imageshack.us/img243/3865/test2aq03nf2.jpg

Test3-AQ 0.9
http://img168.imageshack.us/img168/1576/test3aq09hn7.jpg

Test4-AQ 1.2
http://img168.imageshack.us/img168/9796/test4aq12xs4.jpg

Test5-AQ 0.9 Sensitivity 12
http://img168.imageshack.us/img168/1298/test5aq0912mr5.jpg


Perhaps cranking up the FluxSmooth might with the rapid color changes?

[check]

have you tried turning on deband in ffdshow on playback?

[foxyshadis]

Perhaps cranking up the FluxSmooth might with the rapid color changes?

A heavy 3x3x3 filter will only make the problem worse. (Try it.) You're fighting quantization from the colorspace and the codec, and you can't really win by brute force. AQ works because it lightens the codec's quantization, but you still have to deal with YV12, which limits the granularity of the colors; by using levels, you're just highlighting the dark side of video compression and why it's so easy to overfilter things. When you kill all the dithering (which x264 will do in these situations), you make the gradients obvious - those are present in the original film stock and not easily removable, the low bit depth makes them splotchy instead of smooth, and the codec quantization reduces the splotches to 8x8 squares. You can do a 32x32 average filtering in double precision and still have breaks and splotches, just because every value gets quantised to an 8-bit value in the end.

Something you might look at is Clouded's CelForeground, designed to restore small gradients present in all film back down to solid, flat CG surfaces. A warning though, if the gradient is steep enough you'll get an ugly tear in the middle of what should be a solid object, and raising the thresholds to compensate will turn background detail into the same featureless flat surface.

After that long exposition, I agree with check that re-introducing dithering in the output is the best fix when playing on computer.

Isn't 10-bit DC equivalent to a CQM? MPEG-1/2 needed a special case because the range of the quantized coefs was limited, but H.264 has no such restriction.

I was hoping a DC could more precisely represent the average, but that might not be the case. (I realized afterward that without dithering in the decoding you'd just end up truncating the value anyway, gaining nothing in this DC-only case.)

[DarkZell666]

Actually, if it's "moving-blocks-in-the-background" that bother's you, hqdn3d is your friend :) The worse you'll get is obvious banding on gradients, but you won't see any "moving backgrounds" anymore (tested and approved :rolleyes:). Be warned, the default params are quite strong (and you can avoid the banding by turning down the spatial params) ^^

I'm not aware of too much gradients in the simpsons anyway, they appear much more in japanimation if you ask me ... ;)

[Dr.Khron]

CelForeground-
Holy crap, that looks complicated. However, I love what it does to clean up the example pics. When I get some time I'll take a deeper look at that.

Ok, I throw down the gauntlet: I challenge someone else to get a get a good x264 encode out of these clips.
I've uploaded two small sets of frames (in huffy AVI):

http://rapidshare.com/files/28302620/Simpson_Clips.rar.html

The second is pretty funny, watch Bart as he shakes around and changes colors. I'm not sure if the original animation was this way, or if tapes they produced this DVD from were really bad.

Here is the Avisynth script I produced them with:


DGDecode_mpeg2source("D:\test.d2v")

edeintted = AssumeTFF().SeparateFields().SelectEven().EEDI2(field=-1)
tdeintted = TDeint(edeint=edeintted,order=1)
tfm(order=1,clip2=tdeintted).tdecimate(hybrid=1)

crop( 8, 0, -8, 0)

LanczosResize(640,480)

FluxSmoothST(7,9)

limitedsharpenfaster()

Good luck.

[Dr.Khron]

I couldn't get the simplier modes of CelForeground to work, so I gave up and tried HQDN3D. I got some very good results with this! This is what I liked so far: adding HQDN3D() and encoding with AQ 0.6 Sensitivity 13. I'm going to run the full episode tonight so I can look to see if I've introduced artifacts elsewhere.

FluxSmoothST(7,9)

hqdn3d()

limitedsharpenfaster()

Now, I know that mixing denoisers is kind of a voodoo art, but maybe I should go with this instead?

hqdn3d()

FluxSmoothST(7,9)

limitedsharpenfaster()

[DarkZell666]

Now, I know that mixing denoisers is kind of a voodoo art,
Muahaha this is the funniest thing I've read in a while :D. There's a part of truth in that sentence too, so I suppose some people around here aren't called guru's for nothing :p.

Nice to hear you managed to get a decent picture now, but actually, I'm not sure fluxsmooth is going to have much effect if you place if after hqdn3d (for me, fluxsmooth hardly had any effect at all even with high parameter values btw ..., mind you I never zoomed on the picture to check :/).

[Dr.Khron]

The Fluxsmooth is there mostly to improve compressability, which is why I put it first.

It does clean up a decent amount of noise, but I would have to crank it up really high in order to take out all the noise, to the point where features disapear from background characters. This clip has a ton of noise, such as scratches and mystery dots, particularly in the opening sequence. I should probably increase the Spatial number in the fluxsmooth a bit as some of the dot noise is really big.

The other funny thing is the arguments for hqdn3d() and LimitedSharpenFaster()... I really did mess around with the arguments for both of those quite a bit, but in the end, the default settings were the best overall. I could tweak the settings to make a particular scene marginally better then another one, but the stock settings won in the end.

[R3Z]

I dont know mate, this one;

http://img168.imageshack.us/img168/1298/test5aq0912mr5.jpg

Looks really good. Just remember mpeg2 does have alot of macro blocks as well, its just you dont always see them because the codec is good at keeping noise and crap in the encode to hide the blocks.

I have even tried mainconcepts 2.1.0 h264 encoder and it has the same problems as x264 but without AQ :eek:

The h264 codec in Sonic Cinevision seems to not have this issue at all, but then again it cant encode SD material without fooking the frame rate up and making it unplayable.

[CruNcher]

The h264 codec in Sonic Cinevision seems to not have this issue at all, but then again it cant encode SD material without fooking the frame rate up and making it unplayable.


That is strange as Mainconcepts\Elecard Encoder is used in Cinevision i think hmm did you tried Elecards Converter Studio also ?

[R3Z]

That is strange as Mainconcepts\Elecard Encoder is used in Cinevision i think hmm did you tried Elecards Converter Studio also ?

The mainconcept encoder in cinevision is different from the one available in 2.1.0 (sept 2006). Havent tried elecards encoder as i was told it was the same as the 2.1.0 release.

[Dr.Khron]

Wow, my thread got hijacked into in an interesting place...
I'll just jack it into a new direction.

Since I've now solved my encoding de-blocking problems, i now realize that I've got some blocking/errors in my original source material. For example,

DGDecode_mpeg2source("D:\S03E04.d2v")

Produces this:
http://img412.imageshack.us/img412/7277/samp11hg9.jpg

Running my normal AVS script and encode, here is the same frame from the Huffy AVI and then the finished MKV:

http://img147.imageshack.us/img147/5378/samp12on2.jpg

Here is some ugly crap on Krabobble's neck, in Direct D2V, Huffy AVS output, and final x264 MKV:

http://img147.imageshack.us/img147/9572/samp2eh8.jpg

Yecch. Looks like someone puked on the teacher.

Whats wierd is that if you play the D2V or the ripped vob files directly in MPC, you can clearly see the errors. However, if you play the DVD IFO file with MPC, you can barely see the errors at all! (I have a complete "file mode" rip of the original DVD on my HD, in addition to the "PGC mode" rip)

Ok, so since this is clearly not an ecoder issue, maybe I shouldn't be posting about this issue here. But where should I go? Any advice appreciated.


EDIT:
Ok, I figured it out. It was a bad rip, which I didn't know was possible.

I keep all of the original DVDs in a box in the closet, since I long ago ripped them (in file mode) to the HD of my living room PC. When I needed to do a PGC rip with DVDD, here are the shenanigans I went through to avoid having to dig out the original DVDs:

1. Build an image file from the VIDEO_TS files with ImgBurn.
2. Mount the ISO image with Daemon tools.
3. Rip the PGCs from the mounted ISO image with DVDD.

Somewhere in that process, those errors occured.
I dug out the discs and did a fresh PGC rip with DVDD directly from the original discs, and everything looks fine now, no errors in the source D2V at all. I'm not going to bother trying to figure out exactly where the errors were introduced, since making clean rips is an easy fix (now that I've unearthed the original box sets of discs).