Does x264 inherently de-noise/de-dither DVD video and reveal blocks?

[Morte66]

A while back, I bought a Dell 2405FPW LCD monitor, which is extremely bright and contrasty compared to my 8 year old 21" Iiyama CRT. On the whole it's great -- hello to 3D video -- but it sure does reveal blocking. And I've begun to notice lots of blocking with x264 encodes from DVD, mostly in deep shadow areas but also on things like foreheads in lower bitrate source.

I've tried all sorts of stuff to get rid of this. My source (playing back the avisynth script) is not blocky, but it has far more noise than the x264 encode. In particular, the problem areas that become blocky once encoded with x264 are covered in noise on the DVD. So I'm wondering if x264 is denoising the video, and incidentally removing dither to reveal blocks in the underlying MPEG2. x264 certainly seems to denoise things overall. And if I run a fairly strong denoiser on the MPEG2 (without encoding) I do see hints of blocking emerge through the mush.

So, I wonder, does x264 come with a built in denoiser that's going to de-dither the source? If so, is there anything I can do, apart from using another encoder?

My typical encodes would run from a script like:# Set DAR in encoder to 141 : 74. The following line is for automatic signalling
global MeGUI_darx = 141
global MeGUI_dary = 74
video=DGDecode_mpeg2source("D:\MeGUI\BLANC\BLANC.d2v",info=3)
audio=nicac3source("BLANC Track 1 Français DELAY 0ms.ac3",2)
audiodub(video,audio)
ColorMatrix(hints=true)
#Not doing anything because the source is progressive
AssumeFPS(24,1,true)
SSRC(48000,False)

My encoder settings are usually something like:--crf 22 --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct temporal --filter -2,-2 --subme 6 --trellis 1 --analyse all --8x8dct --threads 2 --thread-input --cqmfile "C:\Program Files\x264\eqm_avc_hr.cfg" --progress --no-psnr --output "D:\MeGUI\BlancSample\blanc.mp4" "D:\MeGUI\BlancSample\blanc.avs"


Here are some things I've tried:

- I jacked the bitrate up. This had negligible effect on blocking.

- I varied the x264 deblocking options between -2-2 and +1+1. The effects were minor on the problem areas, basically the big blocky clumps got 10% smaller. The block-free areas varied as expected.

- I used Sharktooth's eqm_avc_hr custom matrix. This seems to prevent blocking on sources that never had it (like some encodes I made from uncompressed yuv), but it does nothing much for DVDs.

- I've disabled ColorMatrix(). No effect on blocking.

- I adjusted the DGIndex/Decode deblocking. It seems to introduce blocks where there were none before without deblocking the ones that that already existed, so I turned it off.

- I stopped using B frames. This helped a little, but only really scratched the surface.

- I used the blockbuster(method="noise") AviSynth filter to make x264 throw bitrate at the shadow areas. I saw the noise it added clearly when playing the avisynth script, but x264 seemed to remove it and gave me the same blocky encodes.

- I used Xvid at q4. It encoded the noise and gave me output that looked rather like the input, except the noise in the potentially-blocky areas sort of clumped and the clumps sort of crawled around.

- I use Xvid at q2. It gave output basically the same as the input, with no obvious blocking. A satisfactory encode, but rather large.

[Mug Funky]

you might want to try blindpp instead of dgdecode's internal deblocking. it's not optimal, but it's more stable (dgdecode will deblock more agressively on macroblocks with a higher quantiser, which can cause detail to flicker in and out between b-frames and p-frames).

this is a desperate measure, but you can run levels(...) on you source to clip the dark areas to pure black. you can't have visible block edges when adjacent blocks are pure black :)

also, check the stats x264 spits out - what percentage of blocks are being skipped?

also, direct prediction should be set to "auto" (i think that's the name of the param, but check the docs), as temporal prediction might not always be optimal. auto will use temporal and spatial where appropriate. this might just have a positive effect on the skipped blocks (which are likely to be where you're seeing blocky artefacts).

you might also want to try it without --no-fast-pskip, as that will also have an (unknown to me) effect.

but FYI, yes h.264 will remove grain, and lots of it. this may or may not be a good thing (i like to keep a bit), but h.264 does allow for film grain simulation on decoding based on side information on encoding. this will certainly hide blocking if it's there, but obviously the more elegant solution would be for the blocking to not arise at all :)

[Morte66]

you might want to try blindpp instead of dgdecode's internal deblocking. it's not optimal, but it's more stable (dgdecode will deblock more agressively on macroblocks with a higher quantiser, which can cause detail to flicker in and out between b-frames and p-frames).

OK, I'll try that when the queue clears.

this is a desperate measure, but you can run levels(...) on you source to clip the dark areas to pure black. you can't have visible block edges when adjacent blocks are pure black :)

If you'd seen my Seinfeld encodes, you'd know DGIndex does that for me. You get a big black blob with pulsating crinkly edges instead of an assortment of smaller black blobs with pulsating crinkly edges. At least it stays put, instead of dancing around and shouting "hey, I'm over here, ignore that guy Jerry and look at the shadows". :)

If I'm desperate, i.e. I can't crack it with x264, I expect I'll use Xvid at 1 or 2 movies per DVD5 to backup and give up on the media server idea. Hmm, actually, DVDShrink can be quite good at 1 movie/DVD5...

also, check the stats x264 spits out - what percentage of blocks are being skipped?

I can't make head or tail of this, but for a short clip the log says...

avis [info]: 720x576 @ 25.00 fps (481 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!
mp4 [info]: initial delay 2 (scale 25)
x264 [info]: slice I:2 Avg QP:19.00 size: 28962
x264 [info]: slice P:152 Avg QP:20.92 size: 5833
x264 [info]: slice B:327 Avg QP:22.64 size: 1118
x264 [info]: mb I I16..4: 40.4% 53.2% 6.3%
x264 [info]: mb P I16..4: 1.4% 1.8% 0.2% P16..4: 74.0% 5.4% 10.0% 0.0% 0.0% skip: 7.3%
x264 [info]: mb B I16..4: 0.0% 0.1% 0.0% B16..8: 18.0% 0.2% 0.4% direct: 0.1% skip:81.3%
x264 [info]: 8x8 transform intra:55.9% inter:88.4%
x264 [info]: ref P 66.1% 13.6% 12.5% 3.4% 4.4%
x264 [info]: ref B 81.7% 11.2% 4.3% 1.4% 1.4%
x264 [info]: kb/s:544.8

encoded 481 frames, 8.94 fps, 546.12 kb/s

also, direct prediction should be set to "auto" (i think that's the name of the param, but check the docs), as temporal prediction might not always be optimal. auto will use temporal and spatial where appropriate. this might just have a positive effect on the skipped blocks (which are likely to be where you're seeing blocky artefacts).

OK, I'll give that a spin.

you might also want to try it without --no-fast-pskip, as that will also have an (unknown to me) effect.

Tried it before. Enabling no-fast-pskip removes certain blocks when encoding from a clean source (yuv), especially if you're making sharp encodes (deblocking -2-2), but it seems to have no particular effect re-encoding a DVD which already has plenty of blocks.

but FYI, yes h.264 will remove grain, and lots of it.

Ah, then I guess I'm up against it.

but h.264 does allow for film grain simulation on decoding based on side information on encoding.

As best I understand it (not so well), that's a feature of the h264 standard but not something that's implemented in the x264 encoder as yet?

[Morte66]

Here's a sample (http://www.joel-benford.co.uk/temp/BlancSample.zip), with mpeg2/x264/xvid. It's 18MB. Best watched full screen in a dark room with the contrast turned up, to experience its full glory.

[Morte66]

Ok, tried BlindPP() instead of dgdecode's deblocking. It seems to put a wide blur with graduated edges over the blocks, so I get "blobbing" instead of "blocking". I think I might prefer it, since the smooth blobs draw attention less than the sharp/jagged blocks. But it's a small thing compared to losing the noise/dither.

I tried "auto" instead of "temporal" for the prediction, and I thought it made things worse. Mostly it looked different without being better/worse, but once in a while there'd be a an obvious jump with the blocks moving around. I didn't get that in temporal mode.

I really ought to do more samples to be sure how these compare to my regular options, but so far it doesn't look like they're going to resolve the basic issue.

[shon3i]

I watched clips, and i can only say that x264 use quart smaller bitrate than XviD, and have smaler filesize.

[Morte66]

I watched clips, and i can only say that x264 use quart smaller bitrate than XviD, and have smaler filesize.

But the blocking doesn't change if you increase the x264 file size. Other things change, but not the blocking. I've been up to crf 14 with various DVDs, getting encodes that are bigger than the original MPEG2, and the block issue is essentailly the same as at crf 22. Sometimes the higher bitrate x264 encodes look worse, because they have sharper edges on the blocks.

So far as I can tell, if I find a solution to this it won't (just) be bitrate.

[shon3i]

Other things change, but not the blocking. I've been up to crf 14 with various DVDs, getting encodes that are bigger than the original MPEG2, and the block issue is essentailly the same as at crf 22I saw that few months ago but i almost always use two pass encoding to prevent this, maybe you try to switch off deblock.

[Morte66]

I saw that few months ago but i almost always use two pass encoding to prevent this

Really? How would that help?

Do you have any suggestions for settings?

maybe you try to switch off deblock.

I'm not sure how to do that, but why do you think it would help?


I eventually managed to make encodes from yuv at crf 22 deblocking -2-2 and avoid blocks in the output (thanks to Sharktooth's CQM and no-fast-pskip). So far as I can tell, the problem isn't that x264 necessarily introduces blocking. I'm coming to think that x264's denoiser reveals blocking that was hidden by noise/dithering in the source, then the encoder reproduces those blocks as faithfully as it can at a given bitrate.

So what I'd really need is either:
- a way to turn off the de-noising and get x264 to encode the dither
- some other encoder that encodes noise (e.g. Xvid)
- some other way to properly dither the original MPEG2 blocks on playback of the final h264

[Latexxx]

These are my thoughts and it isn't sure whether everything I say is true.

X264 doesn't de-noise the source but AVC's in-loop deblocker first tries to blur and when it can't anymore it starts to block. This might appear as if it had denoised the source and revealed blocks.

So use X264 and use higher bitrate and/or two-pass encoding. 2-pass encoding gives hard scenes more bits than easier scenes so you should end up with less blocks as there is more bits to be used in hard scenes.

Here is a small example.
You are doing 1000 kbps encoding. When you do it as cbr, every second gets 1000 kilobits. When using 2-pass, easier (less high frequency content = details) get something likee 500 kbps and harder-to-code parts get 1500 kbps. NB this is an over simplification.

There should be an easy way to determine where the blocks originate from. Take your mpeg-2 source and resize it before coding to avc. If blocks are 16 pixels wide then they originate from avc but if they are some other size then they are from your mpeg-2 source. That is because mpeg-2 blocks are 16x16 pixels as are avc's block usually.

[Morte66]

So use X264 and use higher bitrate and/or two-pass encoding. 2-pass encoding gives hard scenes more bits than easier scenes so you should end up with less blocks as there is more bits to be used in hard scenes.

I thought crf was meant to do that, by a different route. But I'll certainly give it a spin.

There should be an easy way to determine where the blocks originate from. Take your mpeg-2 source and resize it before coding to avc. If blocks are 16 pixels wide then they originate from avc but if they are some other size then they are from your mpeg-2 source. That is because mpeg-2 blocks are 16x16 pixels as are avc's block usually.

OK, will do. Thanks for the idea.

[Mug Funky]

mb B I16..4: 0.0% 0.1% 0.0% B16..8: 18.0% 0.2% 0.4% direct: 0.1% skip:81.3%

that seems a high number of skipped blocks... my low bitrate stuff tends to be around the 70% mark (and seems to skip more on p-frames). for a grainy source that seems a bit much. hmmm. have you tried it without any CQM? sharktooth's should be fine, but it's all i can think of for now (the matrix used influences bitrate but not RC decisions like block skipping).

[smok3]

or maybe try callibrating the monitor first..., i have a dell thingy as well, and there was no way to set this thing properly (1905FP)

[*.mp4 guy]

The blocks you are seeing in low contrast areas are cause by quantisization, not by skipped blocks (skipped blocks create a different kind of blocking, it looks a lot different) the easiest solution is to use a cqm that doesn't drop as many frequencies.

If you wan't to try a cqm, this one (http://www.yousendit.com/transfer.php?action=download&ufid=F3AD65A7559EA004) should work well, though you should change Bframe predict mode to none to stop X264 from behaving badly.

(the matrix used influences bitrate but not RC decisions like block skipping).
Thats actually not true, the matrix does effect RD decisions, usually its not a problem, but sometimes X264 can get a bit wierd about matrices.

[juerginst]

Any other recommendations for this matrix? Low, mid or high bitrates?

[*.mp4 guy]

Mid-low, to mid-high, about the same amount of ringing as the standard matrix, a lot less blocking in low contrast areas. Better preservation of faint details, a bit worse preservation of prominent details, more texture-noise artifacts and smearing then the standard matrix.

[Revgen]

A while back, I bought a Dell 2405FPW LCD monitor, which is extremely bright and contrasty compared to my 8 year old 21" Iiyama CRT. On the whole it's great -- hello to 3D video -- but it sure does reveal blocking. And I've begun to notice lots of blocking with x264 encodes from DVD, mostly in deep shadow areas but also on things like foreheads in lower bitrate source.


I have the same monitor and the same experience. LCD's are so bright that blocks that you never saw on a CRT are revealed on LCD's. CRT's are far more superior in terms of image quality, yet they are too expensive now that manufacturers have switched to LCD's. Try working with the brightness and contrast levels as much as possible to improve it.

The other thing to do is to use better denoisers than you did in the past.

[CAFxX]

On my Vaio laptop I could spot those blocks even on good DVDs.
The only way you can get rid of these blocks is by calibrating the monitor. Lower the gamma and brightness, ensure you have the proper profile loaded in the color manager.
Eventually use an utility such as Adobe Gamma.

[Morte66]

OK, replying to most of this...

@Shon31/Latexxx re 2-pass vs crf:

A 2-pass encode at the same bitrate as the crf encode (which varies the bitrate as it goes along to hit a quality goal) looks essentially the same. Doubling or quadrupling the bitrate made only minor differences to the blocky areas, maybe the big groups of blocks got 10% smaller.


@Latexx re resized encode:

I did a resize to 1440x1152 before encoding, then compared it to a 720x576 encode resized after encoding. The block structure is still predominantly the same, which indicates that the blocks are predominantly in the MPEG2. The result does seem a bit less crinkly around the edges though, which suggests that x264 is adding a little mischief of its own, on a smaller scale.


@MugFunky re not using any CQM:

That's what I was doing at the start. Sharktooth's CQM has helped a little with blocking (it is, as I understand it, specifically an anti-blocking matrix). I do do the occasional "control check" with the x264 default matrix, and it's not a lot different.


@Smok3/Revgen/CAFxX re monitor cal:

My monitor is very carefully calibrated -- I'm a photographer and I use it for proofing. Never mind Adobe Gamma, I own a hardware calibrator that measures test patterns on screen. :) I see what you're saying: by using the full luminosity range of the monitor to get a punchy and revealing picture, I'm revealing flaws in that picture. Sure, I can squash the picture to hide the flaws (I've tried this), but then I hide the good stuff too. The key point here is that the blocking did not bother me on the DVD because it was hidden by noise/dither (unobtrusive at a suitable viewing distance). But the blocking does becomes a problem on the x264 encode using the same monitor setup, whether x264 reveals it or creates it.


Thanks for all the help and suggestions, everybody.


@*.mp4 guy

You're next, when the queue clears. :)

[GodofaGap]

If you ever find a better solution than adjusting levels, please let me know. I've been annoyed by this for quite a while but never found anything that solves it (yes, I tried CQMs, but perhaps not the right ones...)

[juerginst]

Mid-low, to mid-high, about the same amount of ringing as the standard matrix, a lot less blocking in low contrast areas. Better preservation of faint details, a bit worse preservation of prominent details, more texture-noise artifacts and smearing then the standard matrix.

Thanks *.mp4 guy! And thanks Morte66 for your tests.

[Morte66]

The blocks you are seeing in low contrast areas are cause by quantisization, not by skipped blocks (skipped blocks create a different kind of blocking, it looks a lot different) the easiest solution is to use a cqm that doesn't drop as many frequencies.

If you wan't to try a cqm, this one (http://www.yousendit.com/transfer.php?action=download&ufid=F3AD65A7559EA004) should work well, though you should change Bframe predict mode to none to stop X264 from behaving badly.

OK, I gave that a go. The results are substantially as before, the gross blocking is still intact. It does look a little bit nicer though, the blocky clumps are smoother around the edges and generally less attention-grabbing.

I repeated the process suggested by Latexx, comparing "resize then encode" to "encode then resize". The two versions seem somewhat closer than before, so I think your CQM is doing a good job of knocking down the blocks introduced by h264 and just leaving those that were there in the original MPEG2.

This is definitely the best result I've had so far with x264, but the basic problem is still substantially there.

Oh well, let's see if I can get anywhere with post-encode dithering.

[Morte66]

OK, I tried running blockbuster(method="noise") on the output of the x264 encode, instead of using it before encoding. This is not what blockbuster is meant for but it's a start, and the results were most intriguing.

It substantially cured the perceived blocking problem. The blocks can be found if I look for them, but they don't draw attention to themselves and if I hadn't seen them a dozen times before (so I know where they are), I think I'd hardly notice them on normal viewing.

Alas it's not all gravy because the blockbuster noise doesn't look quite right. I think it's just luma noise, or at least it has far less chroma than the noise on the original DVD, and it's more fine grained. So it does look a bit false.

I need a better noise generator... I guess the ideal would be the h264 film grain simulation feature, which characterises the original noise during encode. But I think the x264 team don't see that as a priority. I've tried adding noise in ffdshow, but it's not as good as blockbuster on the settings I used. Also, I can't get ffdshow to co-exist with CoreAVC Pro in ZoomPlayer, which is a big problem. Does anybody have any ideas?

The really neat thing about post-encode dithering is that it might work for those three hundred encodes I've already made, the ones that I thought were block-free until I bought the new monitor.

OK, time to search and mess around again...

[check]

To get ffdshow working with coreavc, just tell ffdshow to decode raw video. It will then insert itself into every video graph, which could be good or bad for you.

[CAFxX]

The key point here is that the blocking did not bother me on the DVD because it was hidden by noise/dither (unobtrusive at a suitable viewing distance).
Good postprocessing and some noise adder will do the trick then.

[Didée]

I need a better noise generator... I guess the ideal would be the h264 film grain simulation feature,
Did you try gradfun2bd(), or its "[] DeBand" incarnation in recent ffdshow versions? While it's actually aiming at banding artefacts, it should/could also improve on blocking artefacts, since the respective characteristics are quite similar.

Can't tell anything from experience in this particular case, though ... my belief in glowing phosphor is much bigger than in liquid crystal. (Awaitening SED displays.) :)

Oh, and pre-processing with the "deblock_qed()" script might be worth a try, too. ;)

[Blue_MiSfit]

I will never give up my G500 21" Trinitron :) 1080p at 72 Hz, and 720p at 96 Hz. That's beautiful.

[Mug Funky]

ex-gov CRTs FTW!

if you ebay for it, you can pick up 19" or even 21" CRTs for as little as 20 bux.

i got mine at a swap meet for a fair bit more than that, but still pretty cheap for a trinitron.

only problem is space.

[Morte66]

gradfun2bd(), or its "[] DeBand" incarnation in recent ffdshow versions? While it's actually aiming at banding artefacts, it should/could also improve on blocking artefacts, since the respective characteristics are quite similar.

Hmm. It does make the blocked areas less blocky. But their edges stand out more than before. And it seems to have a problem with motion, there's a sort of on/off "pop" effect as blocky areas move out from under it.

A thing I've noticed is that treatments which add uncorrelated noise, usually center-weighted, subjectively desaturate the colours. It's especially obvious with global noise, rather than block-seeking noise. You can turn up the global saturation to compensate, but it's not quite right. :(

Ach, DVD Shrink is pretty good up to 5GB, and DVD9s are getting cheaper. Maybe I'll do less encoding and more burning...

(Awaitening SED displays.) :)

I want a SED panel or a projector, preferrably 1080p if the software selection becomes adequate. I've a nasty feeling that SED will be well established by the time they release more than a handful of HD discs with films worth watching...

[elguaxo]

The blocks you are seeing in low contrast areas are cause by quantisization, not by skipped blocks (skipped blocks create a different kind of blocking, it looks a lot different) the easiest solution is to use a cqm that doesn't drop as many frequencies.

If you wan't to try a cqm, this one (http://www.yousendit.com/transfer.php?action=download&ufid=F3AD65A7559EA004) should work well, though you should change Bframe predict mode to none to stop X264 from behaving badly.
I was using EQM AVC-HR rev.1 to encode an old and noisy movie (1500 Kbps, non anamorphic). The overall quality was good but there were too many 'dancing blocks'. I used your matrix and the results are really much better! In my case I needed just a little more bitrate with your matrix to keep the same ratefactor.

Any recommended settings for deblocking? I used -2,-1 but I wonder what you are using for a sharp picture.

Thanks!

[*.mp4 guy]

Deblocking settings are completely subjective, If it looks good to you then it is good. That being said the matrix is rather smooth, and is meant to emulate the look of mpeg2 video, so you probably won't get much more sharpness out of it with less then -3:-3.

[dbzgundam]

I'm not saying this is the absolute possibility, but has anyone considered that either:

- Your CRT was incredibly incalibrated

- Your LCD is too bright/has contrast set too high

The blocks you are seeing in low contrast areas are cause by quantisization, not by skipped blocks (skipped blocks create a different kind of blocking, it looks a lot different) the easiest solution is to use a cqm that doesn't drop as many frequencies.

If you wan't to try a cqm, this one (http://www.yousendit.com/transfer.php?action=download&ufid=F3AD65A7559EA004) should work well, though you should change Bframe predict mode to none to stop X264 from behaving badly.


Thats actually not true, the matrix does effect RD decisions, usually its not a problem, but sometimes X264 can get a bit wierd about matrices.

What matrix was this before the link died (in case I have it)? Otherwise, can you repost it somewhere?

[elguaxo]

M4G-V3
INTRA4X4_LUMA =
4,9,18,45,
9,16,35,180,
18,35,225,255,
45,180,255,255

INTRA4X4_CHROMAU =
14,22,32,72,
22,24,41,144,
32,41,96,200,
72,144,200,255

INTRA4X4_CHROMAV =
14,22,32,72,
22,24,41,144,
32,41,96,200,
72,144,200,255

INTER4X4_LUMA =
4,18,18,18,
18,18,18,18,
18,18,18,18,
18,18,18,18

INTER4X4_CHROMAU =
16,22,22,22,
22,22,22,56,
22,22,72,96,
22,56,96,128

INTER4X4_CHROMAV =
16,22,22,22,
22,22,22,56,
22,22,72,96,
22,56,96,128

INTRA8X8_LUMA =
5,8,11,15,18,21,24,35,
8,9,12,17,18,22,25,70,
11,12,18,19,22,30,40,125,
15,17,19,24,35,50,100,155,
18,18,22,35,70,135,160,180,
21,22,30,50,135,175,200,205,
24,25,40,100,160,200,225,230,
35,70,125,155,180,205,230,255

INTER8X8_LUMA =
5,14,16,18,18,18,19,19,
14,15,17,17,17,18,18,18,
16,17,17,17,17,18,18,18,
18,17,17,17,17,18,18,19,
18,17,17,17,17,18,18,19,
18,18,18,18,18,18,18,19,
19,18,18,18,18,18,18,20,
19,19,19,19,19,19,20,20

[surreal120]

Morte66, thanks for all your hard work on this topic!! I also recently upgraded from a CRT to a LCD and am having the same exact problem - it's SOOOO agravating! 2 questions for you:

Were you ever able to get ffdshow working with CoreAVC and ZoomPlayer? I recently did a clean install of Windows to get rid of any old codecs lying around and really don't want to install ffdshow again only to have to remove it later if it won't work with CoreAVC.

Also, when you mentioned using Blockbuster on the x264 output, I assume you meant using it through ffdshow. Did you try just putting it at the end of your avs script (i.e., after resizing)? I'm not much of an avisynth expert, but I've definitely noticed a big difference between applying filters before resizing as opposed to after (not sure if this is quite the same as postprocessing, though).

Anyway, thanks for all the research you've put into this already, it's been a huge help.

[surreal120]

Got a little antsy and when ahead and installed ffdshow again... Wow, what a difference Blockbuster makes! Certainly much better than anything else I've tried.

One issue I'm having now, though, is that the first 30 seconds or so of video look a bit choppy (it's slight, but definitely noticeable). The problem straightens itself out after a bit, but it doesn't make a lot of sense because I'm showing CPU usage of only around 35% and I have a pretty decent processor (AMD 64 3700+). Did you experience this problem? I tried running the video in Media Player Classic and it showed no frames dropped, but jitter was 2ms.

Could the problem have something to do with trying to run CoreAVC and ffdshow together? Or could there be an i-frame "missing" between the movie intro (e.g., the miramax logo) and the actual start of the video?

[j7n]

I ran onto (http://img142.imageshack.us/img142/2660/b1893bu2.png) this problem today. The effect appears as if the output color depth had been reduced by 1 bit. The overall quality of this video and all previous series was good, as there were no or few such fine gradients present.

Source:
http://img125.imageshack.us/img125/9924/quant19noskiphrsourcecc1.png

Then did some testing using HR, MP4 Guy's matrices and low quantizers.

CRF10, M4G-V3, Global PSNR: 51.057
http://img109.imageshack.us/img109/2380/quant10noskipm4gv3jw7.png

CRF16, M4G-V3, Global PSNR: 49.430
http://img109.imageshack.us/img109/8308/quant16noskipm4gv3fu2.png

CRF19, M4G-V3
http://img109.imageshack.us/img109/3776/quant19noskipm4gv3ii5.png

CRF19, eqm_avc_HR, Global PSNR: 49.653
http://img109.imageshack.us/img109/7114/quant19noskiphrpz8.png

Did you try gradfun2bd(), or its "[] DeBand" incarnation in recent ffdshow versions? While it's actually aiming at banding artefacts, it should/could also improve on blocking artefacts, since the respective characteristics are quite similar.

DeBand actually did a very good job in this case, actually "improving" the original. I'm sure I wouldn't have noticed that a post-process is applied if I didn't knew:
http://img137.imageshack.us/img137/4926/quant19noskiphrdebandid2.png

My command line
--keyint 500 --min-keyint 20 --bframes 3 --b-pyramid --ref 4 --nf --pbratio 1.25
--stats "c:\temp\x264_2pass.log" --analyse "all" --direct "auto" --weightb --me umh --subme 7
--b-rdo --mixed-refs --trellis 0 --bime --8x8dct --progress -B 1889 --pass 2

for testing:
--keyint 500 --min-keyint 20 --bframes 3 --b-pyramid --ref 4 --nf --pbratio 1.25
--analyse "all" --direct "auto" --weightb --me umh --subme 7 --b-rdo --mixed-refs --trellis 0
--bime --8x8dct --progress --no-fast-pskip --cqmfile d:\plugins\media\x264\matrices\m4g-v3.cfg --crf 10

The question is: What more could be done (apart from raising the bitrate up to a good MPEG-2 level) to actually encode the noise/gradient, assuming there won't be a ffdshow on the future system.

[woah!]

try taking b-frames out of your script and see if that helps..


--pass 2 -B 1889 --bframes 0 --ref 3 --nf --analyse "all" --direct "auto" --me umh --subme 5 --trellis 2 --8x8dct --progress --no-fast-pskip --cqmfile d:\plugins\media\x264\matrices\m4g-v3.cfg


crf 10 is very high as 18 is about the same as Q2 in xvid by the way...

[*.mp4 guy]

Don't get rid of Bframes, that shouldn't help (though I would recommend setting predict mode to none if you are using a cqm). Instead try using addgrain to add just a bit of grain to make the gradients seam a bit more important to the codec. Try adding this to the end of your script.
Addgrain(4, 0.2, 0.3).(blur0.33).blur(-0.25)

[elguaxo]

(though I would recommend setting predict mode to none if you are using a cqm)
You mean: --direct auto -> --direct none
or leave out this: --weightb

[Manao]

Won't work, you'll only complicates the codec's job. Debanding / adding noise during the playback is almost the only way to go ( at the moment - when adaptive quantization will work effectively in x264, it might solves the issue in a more satisfactory way )

Edit : that was for mp4 guy's remark

[*.mp4 guy]

You mean: --direct auto -> --direct none
or leave out this: --weightb
I meant the first one.
Won't work, you'll only complicates the codec's job. Debanding / adding noise during the playback is almost the only way to go ( at the moment - when adaptive quantization will work effectively in x264, it might solves the issue in a more satisfactory way )
The noise will help if deblocking is disabled, and you use a cqm, it would be rather useless if you had the deblocker enabled, and used the standard matrix. I wouldn't recommend adding noise normally, but the problem here is that that gradient is to fine to be adequately preserved in yv12 and x264 is killing the dithering, adding more noise will provide stronger dithering that x264 is more likely to keep.

[j7n]

I tried adding grain. The improvement was very subtle. Noise didn't spread across the whole color band, but only made its edges "shimmer" quickly and didn't look like real dither at all. The source is very clean and perhaps that is its "problem".

ATM I'll stick to debanding, and of course keep the source MPEG-2.

Does AVC include provision for generating noise on the decoder side only when needed without actually trying to encode the exact dither? Adding more noise to the whole picture is not an option.

[Manao]

Does AVC include provision for generating noise on the decoder sideYesonly when neededNot quite. It allows to configure the noise parameters for each pictures. And - iirc - the noise parameters can vary with the luma of the pixel ( i.e, more noise on black pixels than white ones ). That's not quite exactly what you'd like, but it's alredy a start. The drawback, of course, is that the noise generated is inherently random, and not temporally stable, while what you actually want is a proper dithering.Adding more noise to the whole picture is not an optionWhy ? Even studios add noise to CGI movies. It looks (almost) always better.

[j7n]

the noise generated is inherently random, and not temporally stable
Yes, that's the right description of why AddGrain() didn't look suitable.

I meant more than Addgrain(4, 0.2, 0.3).

[foxyshadis]

Maybe you could try something like:
mt_average(AddGrainC(2),AddGrainC(2,seed=10))

Add in hcorr and vcorr if you really want them. Seed makes it constant across a video. This way the noise is somewhat stable and somewhat not. Or use the average plugin for finer weighting. (A plain screen pattern, ie no temporal randomness, just looks weird though.)

There's a lot of very interesting things you can do with noise - search for noise factory for one of them. The noise should have different weightings depending on if it's a light or dark area, and detailed or smooth, and AVC grain deals with that.

[bananacreamandpeca]

If you wan't to try a cqm, this one (http://www.yousendit.com/transfer.php?action=download&ufid=F3AD65A7559EA004) should work well, though you should change Bframe predict mode to none to stop X264 from behaving badly..

Are you talking about the "weighted B-Prediction" option?

Seems like these blocks are a issue for lots of others as well.
(thought I was the only one)
And thank you all for trying all the other options I jotted down to try, before I did (and prob made me yell and
curse at everyone because it didn't work)
Saves a hell lot of time :D

[bananacreamandpeca]

ex-gov CRTs FTW!

if you ebay for it, you can pick up 19" or even 21" CRTs for as little as 20 bux.

i got mine at a swap meet for a fair bit more than that, but still pretty cheap for a trinitron.

only problem is space.


That last line was funny.

I also use a trinitron monitor. Don't know about the quality.
But seems alright to me. (CTX PR-711FL prob a cheaper trinitron model)

[Morte66]

Are you talking about the "weighted B-Prediction" option?

He means using --direct-none to change Bframe predict mode to none with that matrix. B-frame prediction can act funny with strong CQMs. This costs a lot of bitrate, only use it if it looks obviously better. [Most people use --direct-auto unless they have a reason not to.]

This is an old thread, there's an updated version of that matrix which I linked in the other thread you just started.

Seems like these blocks are a issue for lots of others as well. (thought I was the only one)


It's a quiet split in the Doom9 community. Some of us are really bothered about blocking on DVD encodes, others can't see what the fuss is about. I don't know if it's about perception or it's just that some people have display set ups that reveal blocking -- LCDs, dark rooms, high contrast, close viewing etc.

[nm]

It's a quiet split in the Doom9 community. Some of us are really bothered about blocking on DVD encodes, others can't see what the fuss is about. I don't know if it's about perception or it's just that some people have display set ups that reveal blocking -- LCDs, dark rooms, high contrast, close viewing etc.
It's probably a bit about all those things. For example, I have a couple of CRTs, a laptop LCD and an LCD projector. Blocking is not a big problem with any of these particular displays. I can see it if I look for it, but it doesn't bother normal viewing at all. However, a friend of mine has an older LCD panel which makes x264-encoded video look horrible. That's because the panel only has 18-bit colors (I think), and even worse, it displays dark colors with peculiar shades that can be told apart very easily. This makes blocking really stand out. The result looked like something one would expect to see at YouTube rather than in a DVD rip.

[R3Z]

It's a quiet split in the Doom9 community. Some of us are really bothered about blocking on DVD encodes, others can't see what the fuss is about. I don't know if it's about perception or it's just that some people have display set ups that reveal blocking -- LCDs, dark rooms, high contrast, close viewing etc.

I agree wholeheartedly here. Frankly, its quite frustrating that one needs to resort to using custom matrices (especially those that actually smooth the whole picture) when most of us are trying to keep all the precious details we can.

I find that adding grain during playback totally eliminates the blocking and i would really like to see some sort of option in the encoder to add grain during playback. I know this would mean that the decoder must be able to handle that feature?? and not sure how one would go about it.

I am just very lucky that i find grain and noise as a great thing. Some people hate it !

[Mug Funky]

h.264 supports "film grain simulation" which does what you ask.

x264 doesn't support it, and most (all?) decoders don't. but it's likely to be supported in the future, when it becomes less fashionable to overfilter encodes.

[check]

troll? It must be.
In any case, the nitpicker within me wants to point out that the feature is actually named "film grain modelling" (at least, that's what I've seen it referred to as).

[bananacreamandpeca]

Am trying a few things of my own to reduce those nasty looking things in the output.

I was wondering. If turning deblocking to off. Would that make things worse when trying to reduce those blocks?
Some say I should turn deblocking up
Some say to lower the value to a negative value

[Morte66]

I am just very lucky that i find grain and noise as a great thing. Some people hate it !

You certainly are lucky, I don't like noise much.

[Morte66]

Am trying a few things of my own to reduce those nasty looking things in the output.

I was wondering. If turning deblocking to off. Would that make things worse when trying to reduce those blocks?
Some say I should turn deblocking up

Easy to try.

Some say to lower the value to a negative value

For what it's worth, I think of the Deblocking settings in x264 as a sharpness control. They have a strong effect on sharpness, -2-2 looks like Xvid 1.2 and +0+0 looks like DivX3 (IMVHO, YMMV, taste is personal, etc). Whilst they do also effect blocking overall, it's minor compared to the blocks on the DVD you're encoding from and it has a minor effect on smooth/dark areas compared to a CQM.

If I'm encoding from a clean uncompressed source without any blocking (as I have done once or twice to experiment), x264 deblocking matters a lot more.

[Single]

I think all people who tries to do a details looks at coding results mast do a monitor calibration at first of all. Most of LCD monitors has a very very bad linearity compared to a cheap CRT. Calibration is a gamma at first than britness and contrast.

[bananacreamandpeca]

Can 'enabling CABAC' cause these blocks in x264?
No one ever mentions it. It says it can decrease bitrate with no Q. loss.
So how does it do that?

[akupenguin]

how it does that (http://en.wikipedia.org/wiki/Arithmetic_coding)

[Sharktooth]

Can 'enabling CABAC' cause these blocks in x264?
No one ever mentions it. It says it can decrease bitrate with no Q. loss.
So how does it do that?
Does zipping a file decrease its size without loosing data? ... (answer: yes)
CABAC does the same with the bitstream (well, cabac is a bit more complex than the zip/deflate algo though...) so it doesnt affect image quality at all.

[shon3i]

Yes, but zip is lossless process, and nobody can confirm that is CABAC 100% lossless.

[Sergey A. Sablin]

Yes, but zip is lossless process, and nobody can confirm that is CABAC 100% lossless.

I do.

[Manao]

...

CABAC is lossless. So is CAVLC. The lossy stage in h264/avc is the DCT / quantization stage.

[shon3i]

I do.
Ok, then, we clear that. Thanks.

What about PCM coding, it is usefull for something, and it is lossles?

[Manao]

It's not lossless ( 0 or 255 ( I don't remember which one ) ) can't be coded.

It's largely useless. It allows a resetting of the cabac contexts, so it's mostly an error resilience tool. It costs a lot of bitrate : a 64x64@25 fps video completely in PCM has a bitrate of 1.25 mbps.

[Sharktooth]

@Manao: is it possible to modify the bitstream data to maximize the CABAC efficiency but keeping the h.264 decoded data very similar to the original result?

[akupenguin]

It's not lossless ( 0 or 255 ( I don't remember which one ) ) can't be coded. 0 can't be coded in PCM in main profile (I'm gessing so that the decoder can just use the next 384 bytes instead of running startcode emulation prevention?), but it is allowed in high profile. Otherwise you couldn't use PCM in the only case where it makes sense: lossless compression. (PCM is not compressed, but by Pidgeonhole Principle some macroblocks would be even bigger if you try to compress them losslessly with inter/intra prediction + cabac.)

is it possible to modify the bitstream data to maximize the CABAC efficiency but keeping the h.264 decoded data very similar to the original result?
That's called rate distortion optimization.

[Sharktooth]

thanks.

[ToS_Maverick]

@morte66: did you try AQ?

i know it's been a while since this topic started. i, like morte, always had problems with big ugly blocks. i recently tried out AQ and got great results, you can see them here (http://forum.doom9.org/showthread.php?p=996829#post996829). as i'm really excited about it, i just wanted to mention it here ;)

for slight blocking i'd recommend --aq-strength 0.3 to 0.6, if you got severe blocking, 0.9 to 1.2 works great. be warned that the bitrate in the critical parts of the video will increase dramatically!