Unusually small x264 output [Archive]

asarian

3rd June 2010, 17:14

I recently upgraded from x264 version 1093 to the latest 1627 build from two days ago. Some parameters had been removed, others slightly changed. No big deal, I thought, so I used the following parameters on several encodings:

--crf 16 --sar 1:1 --level 4.1 --aud --nal-hrd none --vbv-bufsize 70000 --vbv-maxrate 35000 --filter 0,0 --ref 3 --b-pyramid normal --mixed-refs --bframes 8 --b-adapt 2 --weightb --no-fast-pskip --direct auto --subme 9 --trellis 1 --partitions all --8x8dct --me tesa --threads auto --merange 16 --thread-input

The result is phenomenal; a bit too phenomenal, it would seem, as the resultant output file is exceedingly small compared to what I'm used to. For example, re-coding my entire Blu-Ray 2010 disc (VC-1) results in a file length of < 5 GB! (of the original 19 GB). Used to be around the same size, also ~19 GB.

So, what's happening here? Did I mess up a new parameter? Does x264 mess up? (less likely, but not impossible, of course). Or is the result really all that spectacularly better? Only thing I truly changed is that I set bframes to 8 (instead of 3), as my PS3 can apparently handle it (as long as you limit refs to 3).

I prepared a 2 minutes sample from the output here (below). The low bit-rate and VERY small output length notwithstanding, the output video itself really looks fine to me: SAMPLE (http://rapidshare.com/files/394854108/2010-sample.rar.html)

creamyhorror

3rd June 2010, 17:38

1. What was the measured SSIM of the encode? That should give some assurance that what you got was of sufficient quality.

2. Do zoomed frame-matched comparisons with AvsP(mod) and see if you can tell the differences between the source and encode.

Ultimately, if it looks good to you even under close inspection, you shouldn't worry.

Keiyakusha

3rd June 2010, 17:58

actually I think 19 -> 5 gb is fine and if it "Used to be around the same size" then there was some problems before and now they gone

P.S.
you probably want to adapt your commandline to new preset system and get rid of stuff like --me tesa

nurbs

3rd June 2010, 18:05

Or at least you can get rid of the stuff that's on by default anyway:
--nal-hrd none --filter 0,0 --ref 3 --b-pyramid normal --mixed-refs --weightb --trellis 1 --8x8dct --threads auto --merange 16 --thread-input

asarian

3rd June 2010, 18:07

1. What was the measured SSIM of the encode? That should give some assurance that what you got was of sufficient quality.

I'm not entirely sure how useful computing SSIM in this case woul be, as I used a hefty TemporalDegrain filter to clean up the picture quality (for a Blu-Ray, the source is actually rather crappy, tbh).

The x264 log doesn't seem to have skipped any specified features, though, so I'm still both surprised (and a little worried) about the small output size. Yesterday I reduced a ~25 GB Anime to around also ~ 5 GB (although sizes like that are not necessarily uncommon for certain Anime's).

dstln

3rd June 2010, 18:47

This doesn't seem overly unreasonable for certain sources that are being denoised.

Blue_MiSfit

3rd June 2010, 21:45

Does it look good? If so, don't worry about it :)

If it doesn't look good, use a lower CRF number.

/thread

burfadel

3rd June 2010, 21:58

Keep --b-adapt 2 and change --subme to 10 instead of 9 :). Also use --me UMH instead of tesa.

Although tesa is minisculally more precise than umh, its speed degradation isn't worth it. I see in your command line using subme 10 to be more worthwhile than subme 9, and --aq-mode 2 instead of 1 (default)

asarian

3rd June 2010, 23:28

If it doesn't look good, use a lower CRF number.

I won't profess to be an x264 expert here (far from it, actually), but this advice doesn't seem to make a lot of sense. There's nothing of value to be gained from lowering CRF below 16, really. Even using 'tesa,' as people pointed out, is already more in the realm of being somewhat OCD about the highest possible setttings, rather than doing something real useful.

Also, keep in mind that I'm basically comparing the output to the result I received from using x264 1093, on which I also always use CRF 16.

Keep --b-adapt 2 and change --subme to 10 instead of 9 :). Also use --me UMH instead of tesa.

Doesn't 'subme 10' require trelis 2? I thought trelis 2 was still listed as 'experimental'?

I see in your command line using subme 10 to be more worthwhile than subme 9, and --aq-mode 2 instead of 1 (default)

Interesting tip on the aq-mode; I'll use that. :)

Dark Shikari

3rd June 2010, 23:48

Doesn't 'subme 10' require trelis 2? I thought trelis 2 was still listed as 'experimental'?Never has been. And really, just use the presets.

asarian

4th June 2010, 00:42

Never has been. And really, just use the presets.

Yep, appears I read it wrong; it's '--psy-rd 2' (which requires Trellis) which is listed as 'experimental'. Sorry.

And I'm leaning towards 'me umh'. Maybe my eyes just aren't good enough, but I have yet to detect a real difference to esa or tesa (except that the latter two increase encoding speed manifold).

Thanks for your time, Dark.

MatLz

4th June 2010, 01:14

subme 10 requires trellis == 2 and AQ on

psy-rd x:y :
to be used x requires m>=6 and y requires trellis in use (1 or 2)

All is in the --fullhelp, good way to start.

Blue_MiSfit

4th June 2010, 03:34

I won't profess to be an x264 expert here (far from it, actually), but this advice doesn't seem to make a lot of sense. There's nothing of value to be gained from lowering CRF below 16, really. Even using 'tesa,' as people pointed out, is already more in the realm of being somewhat OCD about the highest possible setttings, rather than doing something real useful.

Also, keep in mind that I'm basically comparing the output to the result I received from using x264 1093, on which I also always use CRF 16.

Doesn't 'subme 10' require trelis 2? I thought trelis 2 was still listed as 'experimental'?

Interesting tip on the aq-mode; I'll use that. :)

It's pretty simple, actually..

I'd suggest just taking a step back and asking yourself "does my output look good enough?"

If it doesn't look good enough, you can either spend more bits (reduce CRF number, or use more bitrate in 2pass mode), or you can "increase" encoding settings. You're already near the slowest possible settings, so there's not much more to do there.

SO, if you're not happy with the quality, increase bitrate :)

BUT!! You said you ARE happy with the quality, so I'd say leave it as-is :)! For academic purposes, you might try re-encoding with more typical CRF numbers - say 18 or so, and see if your eyes are happy with it.

Also, please do use presets and tunings. They make life infinitely simpler and are literally impossible to screw up.

Try JUST this:

x264.exe input.avs --level 41 --crf 18 --vbv-maxrate 35000 --vbv-bufsize 70000 --ref 3 --tune animation --preset veryslow --output output.mp4

BTW, those VBV settings seem a little unusual. Are these PS3 specific settings for USB playback? You would want different values for BD25/BD50 or BD5/BD9 (BluRay on a DVD+R(DL))

L4 limits maxrate to 25mbps, and L4.1 allows up to 62.5mbps.

Also.. going from x264 rev 1093 to 1627... well... there's been a LOT of improvement, to say the very least!!

Derek

Lam3rD

4th June 2010, 05:07

The source doesn't have any grain and no details past 720p -> the size is pretty normal. And yea you should check the preset/tune combo and maybe drop tesa in favor of umh

asarian

4th June 2010, 13:15

I'd suggest just taking a step back and asking yourself "does my output look good enough?"

That's always a bit of a tricky call, of course. For instance, in rev r1155 I read "Broke --me esa/tesa due to forgetting to add handling for x264_cost_mv_fpel." So it's always possible something inadvertently dropped back to a lower setting on the new version somehow (wasn't 'tesa' here, because it ran at tesa speed, LOL). Something that may not be immediately obvious. I did some more testing, though; and I'm now convinced the new x264 simply really is a LOT better than the old 1093 release.

So, if you're not happy with the quality, increase bitrate :)

Well, you can admire the output in the sample above. :) I really can't detect anything out of the ordinary with it, except that the output is much smaller now.

Try JUST this:

x264.exe input.avs --level 41 --crf 18 --vbv-maxrate 35000 --vbv-bufsize 70000 --ref 3 --tune animation --preset veryslow --output output.mp4

Those presets sound like a good idea. Thanks. And it seems I can still override certain requirements for my PS3 (like limit ref frames to 4 on 1920x1080p). So it's all good.

BTW, those VBV settings seem a little unusual. Are these PS3 specific settings for USB playback? You would want different values for BD25/BD50 or BD5/BD9 (BluRay on a DVD+R(DL))

L4 limits maxrate to 25mbps, and L4.1 allows up to 62.5mbps.

Yeah, I know; but I figured 35Mb/s for a x264 output file is already unusually high as it is; but I can crank it up a bit, for sure.

Also.. going from x264 rev 1093 to 1627... well... there's been a LOT of improvement, to say the very least!!

It would appear so, yes. I thought improvement had been pretty much maxed out already at 1093; but this newer version really seems to kick some major ass. :) Looks like I really should upgrade more often.

nurbs

4th June 2010, 14:09

Those presets sound like a good idea. Thanks. And it seems I can still override certain requirements for my PS3 (like limit ref frames to 4 on 1920x1080p). So it's all good.
You don't need to do that. As long as you don't manually specify the number of reference frames the --level 4.1 command will automatically limit the number used by the preset to the maximum that's allowed at the level you requested according to the resolution.

Trahald

4th June 2010, 19:20

The degrain filter definately didnt hurt, more b-frames than the original would have had to work on this degrained input, longer gops. All of that helps. Having said that, x264 of course is an excellent encoder(imho)

Blue_MiSfit

4th June 2010, 23:15

Yep, I'd suggest upgrading as frequently as possible. Little bugs get fixed all the time, and you missed out on some really good things like excellent 1 pass VBV, mb-tree (which is huge for anime) etc..

Derek

asarian

12th June 2010, 15:50

I'll say! :) Just re-encoded my entire Macross Frontier collection (9 Blu-Rays); this time also with max settings, but properly tuned for Anime with the new x264 settings. The result? I shaved over 20 GB off the total!

And last night I re-encoded Evangalion 1.11 again; also with '--tune animation' and otherwise max settings. The resultant h264 file is -- hang onto something! -- only 2 GB!! (out of ~25 GB). Looks like I'm just going to have to re-encode everything, as the results are simply too spectacular to ignore. Dark Shikari, you're the man!!

creamyhorror

12th June 2010, 16:51

Blue_MiSfit

13th June 2010, 00:21

Chances are, you probably used too much bitrate and chose bad seettings last time :) Honestly x264 is farking amazing, but it was also farking amazing back at rev ~1000.

IMO, x264 became the "best" option for most folks doing dvd / bluray backup when D_S coded in VAQ, and later Psy RDO. These eliminated the nasty issues of blocking static backgrounds which had plagued every H.264 implementation I'd seen until then (and kept me on Xvid for far too long). I think this all happened about 2.5 years ago..

Derek

mp3dom

13th June 2010, 13:27

And last night I re-encoded Evangalion 1.11 again; also with '--tune animation' and otherwise max settings. The resultant h264 file is -- hang onto something! -- only 2 GB!! (out of ~25 GB).

Allow me to highly doubt that 2 GB is enough to store that movie (at least if your intention was to re-encode the movie in a transparent way, keeping it also in FullHD)

Blue_MiSfit

14th June 2010, 01:30

Perfectly doable, especially with compressible anime!

mp3dom

14th June 2010, 02:10

Keiyakusha

14th June 2010, 07:13

If you mean small grain that is not visible easily, than this is matter of preference probably. Without turning of psy optimizations, there should be enough of that grain. So far I never saw any anime source that needs 15-20 mbps to look good with x264. Something around 10 max usually works for me (if source true HD and not some lame 720p->1080p upscale). Also even if some of that grain will be lost, I see no problem in that, because for those who like it there is gradfun, by using which you can get similar look or even better. So there is no need in wasting bitrate for some noise/grain. Also there is people who dislikes this grain. Personally, if there was a way to turn luma into something as flat as this (http://imgur.com/bK75O.png) without destroying some of the edges and increasing processing time very much - thats would be the first that I would done.

asarian

14th June 2010, 07:31

Allow me to highly doubt that 2 GB is enough to store that movie (at least if your intention was to re-encode the movie in a transparent way, keeping it also in FullHD)
Which brings me back to the title of this thread: "Unusually small x264 output." :) For the (transparent, 1080P) re-encoding of Evangalion 1.11 I used the exact following x264 parameters:

--crf 16 --sar 1:1 --level 4.1 --aud --nal-hrd none --vbv-bufsize 70000 --vbv-maxrate 60000 --filter 0,0 --ref 4 --b-pyramid normal --mixed-refs --bframes 8 --b-adapt 2 --weightb --no-fast-pskip --direct auto --subme 10 --trellis 2 --partitions all --8x8dct --me tesa --threads auto --merange 24 --thread-input --aq-mode 2 --psy-rd 0.4:0 --aq-strength 0.6 --deblock 1:1

The part in bold is just the equivalent of '--tune animation' (but without allowing x264 to double the ref frames: a thing my PS3 wouldn't like). Those settings yield a ~2 GB output file. It has, at times, a ridiculously low birate (< 1 Mb/s), but I can't detect *anything* wrong with it, visually.

mp3dom

14th June 2010, 09:10

So far I never saw any anime source that needs 15-20 mbps to look good with x264. Something around 10 max usually works for me (if source true HD and not some lame 720p->1080p upscale).
No doubt about your statement, but it depends on what's your final goal. Look good is a bit different from 'transparent'. At those bitrates the results is no transparent at all.

Also even if some of that grain will be lost, I see no problem in that, because for those who like it there is gradfun, by using which you can get similar look or even better.

This means that you'll only watch the movie on PC. Standalone players are not supposed to have gradfun built-in.

So there is no need in wasting bitrate for some noise/grain. Also there is people who dislikes this grain.
I think the main goal for a good work is to be as close as possible to the master source for every user with every equipment. So, basically, you prefer to denoise a movie if it contains grain.

The part in bold is just the equivalent of '--tune animation' (but without allowing x264 to double the ref frames: a thing my PS3 wouldn't like). Those settings yield a ~2 GB output file. It has, at times, a ridiculously low birate (< 1 Mb/s), but I can't detect *anything* wrong with it, visually.
Since I've that movie, I'm just curious to use the same settings to see the results. Are you watching on an LCD monitor or CRT? Is it calibrated? Because too often the artefacts are no visible due to wrong luma/contrast settings on the monitor.

asarian

14th June 2010, 09:42

Since I've that movie, I'm just curious to use the same settings to see the results. Are you watching on an LCD monitor or CRT? Is it calibrated? Because too often the artefacts are no visible due to wrong luma/contrast settings on the monitor.
I'm watching the result on my 40" Sony Bravia LCD TV (1080P). It's very well calibrated; I'm very fussy when it comes to quality (I tend to encode with higher x264 settings than most people, I think).

creamyhorror

14th June 2010, 11:43

--crf 16 --sar 1:1 --level 4.1 --aud --nal-hrd none --vbv-bufsize 70000 --vbv-maxrate 60000 --filter 0,0 --ref 4 --b-pyramid normal --mixed-refs --bframes 8 --b-adapt 2 --weightb --no-fast-pskip --direct auto --subme 10 --trellis 2 --partitions all --8x8dct --me tesa --threads auto --merange 24 --thread-input --aq-mode 2 --psy-rd 0.4:0 --aq-strength 0.6 --deblock 1:1

The part in bold is just the equivalent of '--tune animation' (but without allowing x264 to double the ref frames: a thing my PS3 wouldn't like).
Does anyone else think that these psy-rd and aq-strength values might be a little low for detailed/grainy animation? They make sense for clean anime, but (film) grain does well with higher aq-strength and psy-rd, so grain on anime movies might as well...

mp3dom

14th June 2010, 11:55

Keiyakusha

14th June 2010, 16:31

No doubt about your statement, but it depends on what's your final goal. Look good is a bit different from 'transparent'. At those bitrates the results is no transparent at all.
Well maybe but this depends on situation. I can't tell the difference, maybe someone can. So for me they transparent.

This means that you'll only watch the movie on PC. Standalone players are not supposed to have gradfun built-in.
Oops. For some reason I forgot about those who prefer standalone players over HTPC. By the way, do you know what is film grain modeling, which is part of h264? Isn't this is something that can give results similar to gradfun? (this is actually a question, I have very little knowledge about this FGT)

I think the main goal for a good work is to be as close as possible to the master source for every user with every equipment. So, basically, you prefer to denoise a movie if it contains grain.

I think this is doubtful. This noise added artificially. There is no such thing when anime just created/drawn. For example what if picture will be badly upscaled? Like... lets say AiR? We also don't want to leave it like it is. Also the "flat" image I posted is actually a histogram which shows luma of the vector-based image that I created. It uses colors and have some gragients. By converting it back to RGB, in looks exactly like just rendered RGB source (except chroma subsampling of course) but histogram still loks like have banding. Because this is normal, because there is also chroma which makes gradients look good and with adding this noise, result actually looks farther from the original. So as long as our chroma is not killed (by overcompression or something) and looks OK, maybe we don't need this additional noise, thats why it looks different but transparent.

No I don't prefer to denoise. I'm saying that if some of that noise is lost due to encoding (not prefiltering) this is fine. Generally I do denoise only if grain easily visible and not when I need to go frame by frame to see it.

Does anyone else think that these psy-rd and aq-strength values might be a little low for detailed/grainy animation? They make sense for clean anime, but (film) grain does well with higher aq-strength and psy-rd, so grain on anime movies might as well...

Thats depends on situation AQ - yes. I prefer mode 2 with default settings which should adjust its strength on frame basis. But PSY - no. No, if you want to keep source close to the original. Because at higher values it is possible to see distortion. For example recently I saw like at strength 0.8 small objects like falling leaves started to change their shape.

mp3dom

14th June 2010, 16:58

Well maybe but this depends on situation. I can't tell the difference, maybe someone can. So for me they transparent.

Ok, but here we go in a personal-subjective choice. Anyway it's quite irrealistic that a 3mbps avg and true 1080p could be really objectively transparent. x264 is good above all in the low-bitrate field but it's not a holy grail.

By the way, do you know what is film grain modeling, which is part of h264? Isn't this is something that can give results similar to gradfun? (this is actually a question, I have very little knowledge about this FGT)

I don't know but I don't think that film grain modeling adds dithering grain to the picture. Probably its use is to preserve an already-dithered image.

This noise added artificially. There is no such thing when anime just created/drawn.

It's called dithering. Due to the facts that the master comes in 10bit 4:2:2 (and up), if you don't do a proper dithering you'll end up with an image with absurd color banding (especially in flat/cg/color gradients) due to the fact that you're truncating data from 10bit to 8bit and going from 4:2:2 to 4:2:0. If the banding is too strong there's no gradfun here that could effectively resolve the problem. It's the same as 24bit audio to 16bit. You add dithering noise. If the dithering is made in the proper way, the noise is nearly as invisible and you preserve the smooth gradients (with proper bitrate). Otherwise you have an image completely damaged by posterization/banding which deviate even more from the original source. Try to encode the first 1000 frames of Elephants Dream from uncompressed PNG and you'll see what I'm saying.

For example what if picture will be badly upscaled? Like... lets say AiR? We also don't want to leave it like it is.

The difference here is that dithering is the right step when you're working on a master, upscale is the *wrong* step to do something :)

Keiyakusha

14th June 2010, 17:15

Probably its use is to preserve an already-dithered image.
Well as I understand FGT is something that created on the fly during decoding. Yes, it probably uses some additional info, maybe something like SBR in AAC but isn't this even better? Anyway grain is grain, so by using FGT we all (HTPC and Hardware guys) should be happy - thats was my idea :)

And about dithering, I don't want to go deeper in it now. By the way I can't seem to find Elephants Dream in PNG on xiph.org :(

mp3dom

14th June 2010, 17:22

The sources of Elephants Dream are here (http://media.xiph.org/ED/)

14th June 2010, 17:50

Well as I understand FGT is something that created on the fly during decoding. Yes, it probably uses some additional info, maybe something like SBR in AAC but isn't this even better? Anyway grain is grain, so by using FGT we all (HTPC and Hardware guys) should be happy - thats was my idea :)

If properly implemented, film grain extraction and synthesis might work, but Thomson's FGT died with HD DVD (at least for now) and it's not supported by current hardware decoders.

Blue_MiSfit

14th June 2010, 20:39

Correct. FFM / FGT isn't implemented anywhere. Some decoders may support it, but no encoders do AFAIK :)

Derek

asarian

15th June 2010, 16:27

That combination to me is low even for clean anime. I've found out that often it creates macroblocks on uniform colors even at high bitrates so in general I raise psy-rd to something near 1.0 and aq-strength to 0.7/0.8
Hmm, your findings are a mite disturbing. :( Maybe I should forget about "--psy-rd 0.4:0 --aq-strength 0.6" altogether for Anime, and just revert to a regular max profile. As the saying goes: when something looks too good to be true, it usually is.

For regular moviev, btw, does '--aq-mode 2' make me lose detail too? Or is it simply always better than 1?

asarian

15th June 2010, 18:22

Could it be, but that movie is not that compressible. It has lots of subtle grain practically in every scene to keep the color gradients as smooth as possible. With that verylow bitrate (what's the average, 3 Mbps? For a 1080p?) I bet that all the smooth gradients goes away. To start to preserve something you need to go at least at 15-20 Mbps on that movie (and it could be not enough).
Okay, I did some close-up testing; and, regret to say, you're right: the RESULT (http://rapidshare.com/files/399354618/goodbad.jpg.html) is bad. Very bad, actually. Can't believe I didn't see it at first (wishing is believing, I guess), but the resultant 2GB output is simply horrific, and completely unacceptable. So much gradient is gone; and so much macro-blocking in return! I'll chalk it up as a good experiment; but no more running with '--tune animation' for me. I'm sure it may work better on cleaner sources, but Evangelion is sadly not among those.

asarian

17th June 2010, 23:33

Okay, I'm getting a little worried now. :( Thinking "--psy-rd 0.4:0 --aq-strength 0.6 --deblock 1:1" was the culprit in my earlier rendition of Evangelion 1.11, which turned out quite bad, this time I re-encoded without that stuff:

--crf 16 --sar 1:1 --level 4.1 --aud --nal-hrd none --vbv-bufsize 70000 --vbv-maxrate 60000 --aq-mode 2 --rc-lookahead 60 --filter 0,0 --ref 4 --b-pyramid normal --mixed-refs --bframes 8 --b-adapt 2 --weightb --no-fast-pskip --direct auto --subme 10 --trellis 2 --partitions all --8x8dct --me tesa --threads auto --merange 24 --thread-input

And low and behold, I get the same crappy output! A ~2.3 GB file, that just looks horrible (borked gradients, huge macro-blocks). Look at this still photo sample (http://rapidshare.com/files/400185702/alsobad.jpg.html), please. It's very bad!

I attached the x264 log for inspection. I'm telling ya, something is fishy.

AlekseiV

18th June 2010, 00:46

Alt-host of the second image here (http://sovietpride.su/alsobad.jpg). First image is down.

I don't see anything wrong but my monitor is covered in dust and fingerprints.

What is "--filter 0,0"? I've never seen that before and the x264 Settings wikiarticle doesn't seem to have it.

mp3dom

18th June 2010, 00:47

To me there's too big difference between I frame QP (~12) and B frame QP (~20). Try to use tune grain and overwrite the psy-rd to 0.6 and aq-strength to 1.0. Also set qpmin to 0. Dithering are really fine details and qpmin=10 (default) is not enough. Also set weightp=2 if you can (your standalone supports it) and --no-dct-decimate . Also I would change --me tesa since it's too slow to --me umh and I would raise --merange to 64. Anyway keep in mind that preserving dithering is very difficult and above all, it require a lot of bitrates. If your goal is to 'shrink' the movie from 25 GB down to 3-4 GB AND keep all the smooth gradients, you're probably wasting your time. There's a reason why the original movie has an average bitrate of ~32 Mbps and it's not because the encoder used sucks.

I don't see anything wrong but my monitor is covered in dust and fingerprints.
It's simply awful :). The hairs, the car and the seat are all full of color banding/posterization

kemuri-_9

18th June 2010, 01:43

What is "--filter 0,0"? I've never seen that before and the x264 Settings wikiarticle doesn't seem to have it.

--filter is an alias for --deblock

asarian

18th June 2010, 02:15

Mp3dom, your priceless suggestions are much appreciated! :) I've implemented your changes, and am now running a new process with the following settings:

--level 4.1 --aud --nal-hrd none --vbv-bufsize 70000 --vbv-maxrate 60000 --ref 4 --b-pyramid normal --mixed-refs --bframes 8 --b-adapt 2 --weightb --no-fast-pskip --direct auto --subme 10 --trellis 2 --partitions all --8x8dct --me umh --threads auto --merange 64 --pbratio 1 --aq-mode 2 --qcomp 0.8 --deadzone-inter 6 --deadzone-intra 6 --rc-lookahead 60 --deblock -2:-2 --ipratio 1.1 --psy-rd 0.6:0.25 --aq-strength 1.0 --no-dct-decimate --qpmin 0 --weightp 2 --thread-input

I had encoded this a few days earlier (but didn't like the size of the subs, after all), so I decided to re-encode. For that earlier encode I had used the following params:

--level 4.1 --aud --nal-hrd none --vbv-bufsize 70000 --vbv-maxrate 60000 --filter 0,0 --ref 4 --b-pyramid normal --mixed-refs --bframes 8 --b-adapt 2 --weightb --no-fast-pskip --direct auto --subme 9 --trellis 1 --partitions all --8x8dct --me tesa --threads auto --merange 16 --aq-mode 1 --rc-lookahead 40 --thread-input

It's somewhat less heavy; and most noticeble difference is '--aq-mode 1' instead of 2, and 'trellis 1' instead of 2.

Thing of it is, though, that with those settings, the same movie looks much, much better (same still photo sample (http://little-albatross.net/notsobad.jpg), looking much better); it's not perfect yet, but hugely better than what I got from the 2 GB output file (this old one was around 10 GB) on the encode that was supposed to be even better. So, I'm still thinking there's something in the 'heavy' settings that causes the extreme deterioration. Dunno what, of course; could be the --aq-mode 2.

Audionut

18th June 2010, 03:14

What horrible command lines.
I'm also fairly positive that your vbv settings exceed level 4.1.
Are you using 4 refs for hardware compatibility? Again, I don't think you'll find a device on the planet that will take those bitrates.
Also, with the lowish bitrates you're getting, the bitrate isn't going to get anywhere near those vbv values. And even, if you had a crazy scene that "maybe" could reach those bitrates, imo, it would be frames of just grain like the hbo logo and would be better being limited in bitrate to spare more for the rest of the movie.

--qcomp 0.0???? Leave it alone!!!!!!!

That 1st command line can be reduced to this, with a few tweaks I think might help your case.

--preset placebo --tune grain --level 4.1 --ref 4 --me umh --merange 64 --aq-mode 2 --aq-strength 0.6 --psy-rd 0.4 --qpmin 0 --aud --nal-hrd none --vbv-bufsize 40000 --vbv-maxrate 40000

edit: what are you using to capture the still. Compared to the second still, the first one is de-saturated and has less brightness.
edit2: if you don't need to limit the refs for hardware compatibility, just remove the --refs 4 from the command line I posted.

asarian

18th June 2010, 03:46

What horrible command lines.
I'm also fairly positive that your vbv settings exceed level 4.1.
Dunno about that. L4.1 allows for max 62,500kb/s; and I'm fair certain --vbv-bufsize is still within bounds, too.

Are you using 4 refs for hardware compatibility?
Yes, for my PS3. Max allowed value for 1080P content is 4.

--qcomp 0.0???? Leave it alone!!!!!!!
That was me making a typo copying it erroneously from the '--tune grain' line. Should be 0.8 that.

edit: what are you using to capture the still. Compared to the second still, the first one is de-saturated and has less brightness.
The somewhat desaturated stills were taken from KMplayer; the last one with GOM player (as KMplayer kept choking on it)

asarian

18th June 2010, 14:14

Some news.

Mp3dom, I tried it with your settings, and there's still significant colorbanding:

still colorbanding (http://little-albatross.net/aq2.jpg)

Then I disabled something which I suspected all along: aq-mode 2, and let it use mode 1 instead. The result? Already a vast improvement! Here:

huge improvement with aq-mode 1 (http://little-albatross.net/aq1.jpg) (!)

So, I still say there's something fishy with --aq-mode 2; at least for this type of Anime.

Lastly, I cranked psy-rd up to 0.9; even slightly better, I think:

doing --psy-rd 0.9:0.25 (http://little-albatross.net/psy09.jpg)

It's not perfect yet, though. Are there other things I can do to stay closest to the original?

mp3dom

18th June 2010, 15:25

The difference with psy-rd is not that high (assuming the frame is exactly the same). Since too high psy could create other kind of artefacts (ringing or some image distortion) I would prefer to keep psy a bit lower.
Are you still keep CRF at 16?

burfadel

18th June 2010, 15:31