x264 - GUIDE to make high motion scenes look better at low bitrates

[simps]

I was discussing this issue in another thread, but since I decided to investigate it further, I am opening this thread for it.
I believe I can improve current situation, by raising --qcomp to 0.8-0.85 (from 0.7 at this test). I will do it, and report results.

My point: As you can see, there is too much quality on the static and low motion scene frames, even at 240kb/s. I want to move some bits to the high motion scenes, where x264 fails at low bitrates with regular settings. Since there are too much quality on the static and low motion frames, I believe it is possible to adjust it correctly to compensate the problems on the high motion parts. My goal here is to find the correct settings in x264 to do it. And I think it is doable even at 240kb/s. It has to be done at low bitrates, since with massive bitrates, I can't detect the problem on high motion scenes anyway.

Here we go, this is my settings for this test, and why I used it.

1) Default x264 settings will give a too much soft looking picture at low bitrates.
--trellis 2 and --psy-rd 1.02:0.4 (this is the optimal for me after lots of tests) adjust from 1.0:0.2 to 1.08:0.5
--deblock -1:-1 (this will give sharper picture with more details, but might give blocks too) adjust from -1:-1 to -2:-2
--aq-strength 1.05 (this is the correction I use to prevent blocks, from using --deblock -1:-1) adjust from 1.02 to 1.08

2) High motion scenes are hurt too much with default settings at low bitrates.
--qcomp 0.7 (this is provide more quality to high motion frames) adjust from 0.7 to 0.85
--no-fast-pskip (this helps a bit too) use it

3) b-frames tweak
--b-adapt 2 (this is the best for quality) use it
--bframes 8 (Overkill??? Well, from my tests arround 20% of my bframes were 5, 6, 7 consecutive) adjust from 8 - 10
--b-pyramid (this is good stuff) use it
--weightb (this is good stuff) use it

4) Other settings
--ref 16 (Overkill??? x264 reported ~98% of refs between 1 to 9) adjust from 9 - 12
--partitions all
--me umh
--merange 16
--subme 9
--mixed-refs
--8x8dct

5) --direct
I am using "auto" here. Since I asked and DS said "auto" would be best, since the encoder is smart enough to handle it properly.

6) --no-dct-decimate
I found this was bad for low bitrates. It increases the file size, and hurt compression / quality. It can preserve more detail under massive bitrates, and I would use it there, but under low bitrates, it is best not to use it.



Results at 640x352 with 2pass 240kb/s with above listed settings:
I was impressed, remember this is with --qcomp 0.7, I bet with --qcomp 0.8-0.85, the high motion scenes would look much better, without decreasing too much the rest of the video. I will give it a try, and post back results.

* I encoded the entire movie with the above listed settings. No such thing as encoding only high motion parts with one set of settings, and other parts with other sets. No cheating. 2h movie resulted in a 200MB video file.

** Frames included here are .png



Static scene 1: - results are very good
Source
http://img37.imageshack.us/img37/7530/7368staticsource.png
x264 - 2pass 240kb/s
http://img195.imageshack.us/img195/2023/7368staticx264.png



Static scene 2: - results are very good
Source
http://img32.imageshack.us/img32/8442/84970staticsource.png
x264 - 2pass 240kb/s
http://img38.imageshack.us/img38/3117/84970staticx264.png



Low motion scene 1: - results are good
Source
http://img29.imageshack.us/img29/4485/86540lowmotionsource.png
x264 - 2pass 240kb/s
http://img37.imageshack.us/img37/2224/86540lowmotionx264.png



Low motion scene 2: - results are good
Source
http://img196.imageshack.us/img196/5084/31793lowmotionsource.png
x264 - 2pass 240kb/s
http://img40.imageshack.us/img40/1228/31793lowmotionx264.png



High motion scene 1: - results are bad (will try --qcomp 0.8-0.85 and update)
Source
http://img195.imageshack.us/img195/7022/151137highmotionsource.png
x264 - 2pass 240kb/s
http://img29.imageshack.us/img29/5950/151137highmotionx264.png



High motion scene 2: - results are bad (will try --qcomp 0.8-0.85 and update)
Source
http://img190.imageshack.us/img190/2228/85374highmotionsource.png
x264 - 2pass 240kb/s
http://img38.imageshack.us/img38/8620/85374highmotionx264.png



I might post the video file (.264) if anyone is interested.

Thanks,
Simps

[juGGaKNot]

3) b-frames tweak
--b-adapt 2 (this is the best for quality) use it
--bframes 8 (Overkill??? Well, from my tests arround 20% of my bframes were 5, 6, 7 consecutive) adjust from 8 - 10
--b-pyramid (this is good stuff) use it
Simps

b-pyramid and --bframes 8 ? so no dxva/ps3 ?

[simps]

Yes. I am doing this in steps. First looking for the correct settings without thinking about hardware compatibility. I am using full x264 potential. That would be step 2 I think.

[Adub]

If you want to squeeze everything out of this encode, you may try going with --me esa, as it is considered the "exhaustive" motion search.

NOTE: --me tesa is something different, and is often considered a placebo. Don't use it. Try ESA instead.

[Dark Shikari]

If you want to squeeze everything out of this encode, you may try going with --me esa, as it is considered the "exhaustive" motion search.

NOTE: --me tesa is something different, and is often considered a placebo. Don't use it. Try ESA instead.esa and tesa are both placebos.

[simps]

esa and tesa are both placebos.

I was always curious about tesa. Do you mean it won't improve anything / anything relevant from umh, or do you mean it can actually produce worst results?

[Chengbin]

I was always curious about tesa. Do you mean it won't improve anything / anything relevant from umh, or do you mean it can actually produce worst results?

It won't improve anything most of the time.

[Dark Shikari]

I was always curious about tesa. Do you mean it won't improve anything / anything relevant from umh, or do you mean it can actually produce worst results?They're both better, it's just that generally the difference is generally very small from umh -> esa and from esa -> tesa on most sources.

[CruNcher]

im always amazed seeing the grain layer x264 nowdays can preserve thx Dark really really thx for that, but im not sure if it wouldn't be better in this case to try to lower it a little you could try higher deadzones this should save some bits and be a + for the high motion scenes also these fuzzy edges look like AQ is to heavy here the same problem joost has to fight with visually i guess you should maybe even take into account disabling AQ :) (if your goal is really 100% stability you have to exchange details for that)
Dark did you gave up on the smooth quantization for AQ ? or is that part of the new VAQ ?
In my tests i made i always ended up that Film grain utilizes around 30% (bad Mpeg-2 Blu-Ray add another 10-20% quantization noise) of the whole bitrate so there is the most to save especially as psy-rd and the other stuff was invented to preserve it, it currently backfires @ your goal and you have to adapt accordingly to your scenario so imho i would take into account also trying to disable all PSY stuff and maybe even take --nr into consideration or remastering it with mvdegrain :( (but first of course try to compensate with qcomp all possible bits away from the low motion scenes if that helps in your case perfect)

[Sagekilla]

If you're using Trellis (and Psy-Trellis with it, so it's biased towards a perceptively sharper image) then you can't do deadzones though. Personally, I'd rather use Trellis 2 + Psy-Trellis any day though. I'll have to pull out my copy of 300 again (last time was around r700), but I've never really ran into issues with getting a reasonably sharp (Read: If I'm watching it and can't tell the difference) encode using the new Psy-RD/Trellis.

Like you said though, props to Dark/Aku/Everyone else for getting x264 to where it is now ;)

Edit: On a side note, the fact that x264 even used a significant number (up to 7) of consecutive B's seems to make sense to me. IIRC, they provide more benefits at low bitrates than high. [b]Also, one big question: did you encode at 240 kb/s (kiloBYTES per second) or 240 kbps (kiloBITS per second)? If it's 240 kb/s (~ 2 mbps), I'm not really surprised since I regularly hit 2 mbps @ crf 18.

[Dark Shikari]

Edit: On a side note, the fact that x264 even used a significant number (up to 7) of consecutive B's seems to make sense to me. IIRC, they provide more benefits at low bitrates than high.x264's frametype decision is bitrate-agnostic.

[Sagekilla]

Wasn't meaning to say that x264 favors frametypes differently based on bitrate. Just that if you're going lossless, B-Frames don't help at all, but they seem to be more useful at lower bitrates.

[Dark Shikari]

Wasn't meaning to say that x264 favors frametypes differently based on bitrate. Just that if you're going lossless, B-Frames don't help at all, but they seem to be more useful at lower bitrates.B-frames are probably just as useful at QP1 as they are at QP18; lossless is just a special case.

[simps]

im always amazed seeing the grain layer x264 nowdays can preserve thx Dark really really thx for that, but im not sure if it wouldn't be better in this case to try to lower it a little you could try higher deadzones this should save some bits and be a + for the high motion scenes also these fuzzy edges look like AQ is to heavy here the same problem joost has to fight with visually i guess you should maybe even take into account disabling AQ :) (if your goal is really 100% stability you have to exchange details for that)

It could be AQ, since I have never ever tryed without AQ, but I really seriouly doubt that, especially because I am using --aq-strength 1.05 only. Remeber this results so far was with --qcomp 0.7, I so far I have tested --qcomp 0.8, --qcomp 0.9, and from what I've seem from it, the problem is exclusive the --qcomp parameter. Once you get it right, the problems with high motion are gone, even at this low bitrate, and the rest of the movie will look just as good.

About the detail preservation, it is because I found the sweet spot for this source. Might change a little from source to source, but for this, it was --deblock -1:-1 with --psy-rd 1.02:0.4. With default settings on this 2 parameters, the result was bad, very soft like frames, losing too much detail.

I am finishing second pass with higher --qcomp now, and I will post the results here, with a higher --qcomp, and I believe will guys will be VERY impressed.

Question: Can I upload the .264 file too? Is it going to be an issue, violation of rules, copyright etc? I am encoding my own DVD copy of the movie, would it be ok if I would share the .264 with you guys? I would really like you too see it during playback...

[simps]

[b]Also, one big question: did you encode at 240 kb/s (kiloBYTES per second) or 240 kbps (kiloBITS per second)? If it's 240 kb/s (~ 2 mbps), I'm not really surprised since I regularly hit 2 mbps @ crf 18.

This is --bitrate 240. So it is about (0.2 mbps). You will be surprised when I upload the higher --qcomp frames with this bitrate...

[Guest]

Question: Can I upload the .264 file too? Is it going to be an issue, violation of rules, copyright etc? I am encoding my own DVD copy of the movie, would it be ok if I would share the .264 with you guys? I would really like you too see it during playback... You can't distribute the whole movie, but a small enough sample would be considered fair use.

[jsquare]

Simps, could you post the whole command line(CL) for your x264 settings?
I'm relatively new to AVC and trying to move from Xvid to x264, but find the whole process a bit painful as far as so many variables are involved to achieve a good/low-bit-rate encode like yours.

[Sagittaire]

I am finishing second pass with higher --qcomp now, and I will post the results here, with a higher --qcomp, and I believe will guys will be VERY impressed

... will not prove anything with screenshoot. Simply because detail level in high motion part (and grain conservation) is less important than detail level in low motion by really large margin. Very high value is not good value for general case.

Moreover qcomp effect is relative to high/low motion frame proportion and bitrate value. You can't generalise with only one source and one bitrate. Aku choose qcomp at 0.6 for good reason.

[Audionut]

Aku choose qcomp at 0.6 for good reason.

You must have missed the other thread.

This was discussed to death and explained by numerous people. But some tend not to listen.

[lexor]

b-pyramid and --bframes 8 ? so no dxva/ps3 ?

Why not? b-pyramid and bframes (upto 16) work just fine with both. It's the ref frames that you must limit. I don't know if 16 is supported by either, but you don't need them this high anyway. 11 should be enough.

[Guest]

You must have missed the other thread. In which you make unsupported claims and accuse people of being slow minded.

[desta]

That wasn't Audionut.

[Audionut]

lol. ;)

To stay on topic. :p

4) Other settings
--ref 16 (Overkill??? x264 reported ~98% of refs between 1 to 9) adjust from 9 - 12


Beyond overkill. If 98% of refs are between 1-9 why waste time with 9 and over. :confused:


6) --no-dct-decimate
I found this was bad for low bitrates.

Because it's one of those options enabled to squeeze every last little bit of quality out of an encode. But this option will increase filesize.

http://forum.doom9.org/showthread.php?t=115604

--no-fast-pskip (this helps a bit too) use it

Fail to see how you came to that conclusion. As it's another option like above. And will increase fill size.

I realize I'm slow minded, but I'm fairly sure when your encoding at such a low bitrate, you do not want to enable options that increase file size.

[kemuri-_9]

Beyond overkill. If 98% of refs are between 1-9 why waste time with 9 and over. :confused:

that also depends on your source material...
as i primarily deal with animated content, both 16 bframes (on b-adapt 2) and 16 refs often have non insignificant %s in the high up counts.

but real life content usually tops out like your statement implies.

[Audionut]

Also forgot to mention that 16 refs virtually destroys the ability for the file to be played on hardware.

[cogman]

Heck, if you want some good encodes, try applying a noise filter on the source as well. You lose the grainy look (if that isn't what you want, don't do this) but generally see some nice benefits in quality (at a given bitrate).

I suggest MV Temporal Denoise (http://forum.doom9.org/showthread.php?t=139766) It works pretty well. (Slow as cold molasses though :))

[Guest]

That wasn't Audionut. oops, sorry, I meant simps.

[daWsOn_s]

I tried an encoding from BD 1080p of Lost, I tried with the best possible low motion scenes, it's unwatchable :)

You should add that you might use those settings to be shocked only with certain categories of movies

[10L23r]

how much of a size/quality efficiency increase would --me esa give at low bitrates? i know that anything above umh is useless at high bitrates, but what about the low bitrates in these encodes?

[Sagekilla]

I'm not 100% sure but I think it's as useful at low bitrates as in high bitrates: Not much at all. You'd have to do some testing to verify that though.

[amgeex]

Is the qcomp setting useful if I'm using an aq_strength = 0.8?

[10L23r]

I'm not 100% sure but I think it's as useful at low bitrates as in high bitrates: Not much at all. You'd have to do some testing to verify that though.

actually, i recall doing some testing comparing me tesa and me umh with big buck bunny at 640*360, 200kbps

tesa was somewhat better. but my question is, how much of a bitrate increase on umh would make it roughly equivalent to tesa?

[JohannesL]

Use umh if you care about encoding speed, tesa if you don't. Simple as that.

[cogman]

actually, i recall doing some testing comparing me tesa and me umh with big buck bunny at 640*360, 200kbps

tesa was somewhat better. but my question is, how much of a bitrate increase on umh would make it roughly equivalent to tesa?

Its almost completely going to depend on the source. IIRC the gains where somewhere along the lines of 0.3%.

As was said before, if you really don't care about the final file size, then there isn't much reason to avoid using tesa, 16 reference frames, and 16 b-frames (with b-adapt 2). However, they provide minimal gains, so a higher encoding speed could easily be achieved with something like 6 reference frames, 6 b-frames and umh. with very little loss in quality.

[nm]

As was said before, if you really don't care about the final file size, then there isn't much reason to avoid using tesa, 16 reference frames, and 16 b-frames (with b-adapt 2).
Um, if I don't care about the final file size, I think there is much reason to avoid such insane parameters.

However, they provide minimal gains, so a higher encoding speed could easily be achieved with something like 6 reference frames, 6 b-frames and umh. with very little loss in quality.
Indeed.

[cogman]

Um, if I don't care about the final file size, I think there is much reason to avoid such insane parameters.


Whoops. That's what I meant to say. If encoding speed doesn't matter, then there isn't much reason not to max out the settings. If it does, the more moderate settings will do fine with minimal losses.

[infoeater]

Unfortunately topic http://forum.doom9.org/showpost.php?p=1290084&postcount=74 (Why is --qcomp 0.6 default for 2passes?) is now closed, because I made same limited tests that show that for PSNR and SSIM qcomp=0.6 or lower is optimal. However visually problem is somehow similar to low-quality B-frames.

[benwaggoner]

Whoops. That's what I meant to say. If encoding speed doesn't matter, then there isn't much reason not to max out the settings. If it does, the more moderate settings will do fine with minimal losses.
Well, they do have an impact on decode speed and compatible with ASIC decoders.

[benwaggoner]

Heck, if you want some good encodes, try applying a noise filter on the source as well. You lose the grainy look (if that isn't what you want, don't do this) but generally see some nice benefits in quality (at a given bitrate).

I suggest MV Temporal Denoise (http://forum.doom9.org/showthread.php?t=139766) It works pretty well. (Slow as cold molasses though :))
Yes, noise reduction is often the key to good low-bitrate video. Trying to encode noise raises the QPs quite a bit, which can hurt the remaning detail more than the grain masks it.

The built-in -nr noise reducer is really quite effective and very fast. Since it's in-loop to the compressor, it seems to do an excellent job of figuring out which details to preserve.

When I'm getting bad looking video and can't increase the bitrate, I'll often do a test run with -nr 1000 to see if noise is the problem, and then tune the value down from there if the detail loss is too much.

[*.mp4 guy]

That's good advice, though I haven't tested the nr in x264 much.

Before I do anything else, I make sure to prepare the source so that it will look acceptable at the level of compression I am shooting for (removing overshaprness, or shaprening can be very important), and that I am wasting as few bits as possible on extraneous elements (for a given level of quality, hitting high quality levels always entails large amounts of waste). You should aim to lose little detail during compression, and for the detail that must be lost to hit your target bitrate, lose it doing something useful, IE denoising, or removing artifacts.

If the source isn't perfect, denoising is essentially mandatory for hitting low bitrates.

[ckmox]

If the source isn't perfect, denoising is essentially mandatory for hitting low bitrates.


The built-in -nr noise reducer is really quite effective and very fast. Since it's in-loop to the compressor, it seems to do an excellent job of figuring out which details to preserve.


noted! ill go for --nr 1000 from now on since im always encoding for ultra low bitrates

[movmasty]

make high motion scenes look better at low bitrates
There is another thing to consider
with high motion delta frames are bigger respect the I

So you may try to lower the P-B compression ratio and B number.

In low motion almost all the frames are made with blocks from I frames,
whereas in fast motion most of the blocks belong to delta frames.

[movmasty]

On the other hand artifacts could depend from an overcompression of I-frames,
then try to raise I/P ratio and lower P/B, or use QP

[movmasty]

Look who made the last posts.....:D i forgot....

I came here to say that i am experimenting with --qcomp of 30,

i need quality for slow scenes at low bitrate, and fast scenes are always ok with higer curve compression

Good quality in slow parts at low bitrates in x264 depends from deblock and AQ i think, but still i need some more,

So if something needs higer, someone likes lower,

And the default is ok :p