Please help a desperate x264 encoder

[radius]

Hi,

I've been trying to encode 4GB 720p stuff for my ps3 / 360 in x264 with bluray sources,
I really tried A LOT of profiles / settings but I don't seem to find out how to properly encode,
as a comparison here's what I was able to do with megui HQ Insane profile (I changed contrast so you can see how bad it is)

http://r4dius.free.fr/sup_b.jpg

and here's the same frame from a x264 release from the net at approxim 4500 kbps like me

http://r4dius.free.fr/sup_o.jpg

there's a BIG quality gap ..., I always tested 2 pass encodings, with and without x264 loop deblocking settings,
none of my attempts would keep such original noise and quality,
btw my h264 decoder is coreavc and the avs is just as clean as it could be so the problem is with x264 for sure,
I tried some command line encodings with the latest rev 721 too without much success,
please you guys that know how to use this encoder of choice let me know why I suck so much :confused:
Thanks

[Dark Shikari]

From what I know about the Scene groups that encode such releases:

1. They're probably using a custom quantization matrix, like Prestige. I'm not sure how much this helps, but you can try that.

2. They often use lower deadzone settings instead of trellis in order to increase grain retention.

3. They usually use adaptive quantization of some sort. You may want to try my adaptive quantization (http://forum.doom9.org/showthread.php?t=132760) if you're having trouble retaining grain.

[Sagekilla]

I think you mean "grain" and not noise. Noise retention = I have no idea who would want to do it. If I were you, I'd just remove the grain @ 1080p, downsize to 720p and then encode since I could eke out MUCH more quality after it's been filtered. That's just me though. Try what Dark Shikari said and play with deadzone and the new AQ

[radius]

I think you mean "grain" and not noise. Noise retention = I have no idea who would want to do it. If I were you, I'd just remove the grain @ 1080p, downsize to 720p and then encode since I could eke out MUCH more quality after it's been filtered. That's just me though. Try what Dark Shikari said and play with deadzone and the new AQ

Yes I meant grain not noise ^^, I'll try your recommendations Dark Shikari, actually I remember getting better results when I disabled trellis (or something from it I can't remember), can you tell me more about this deadzone thing ?

[Dark Shikari]

Yes I meant grain not noise ^^, I'll try your recommendations Dark Shikari, actually I remember getting better results when I disabled trellis (or something from it I can't remember), can you tell me more about this deadzone thing ?Deadzone and trellis both control how x264 makes decisions in quantization.

In quantization, x264 has to decide whether to round each coefficient up or down. Rounding up costs more bits (especially since most coefficients are either zero or one), but rounding down loses that data. Trellis quantizes in a manner to minimize mean squared error relative to the bit cost of the quantization.

Deadzone is a simple bias to the rounding. Low deadzone = rounds up more often, high deadzone = rounds up less often. So lowering deadzones forces x264 to round up, keeping that fine detail (at the cost of more bits). Note deadzone and trellis are different algorithms, so only one can be active at a time.

[radius]

Ahah in a couple of minutes you made my encoding look a HELL lot better ^^,

I ran this command (found in your thread)
x264 --pass 2 --bitrate 4500 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --analyse all --8x8dct --me umh --threads auto --thread-input --cqmfile "prestige.cfg" --progress --trellis 0 --aq-strength 0.8 --no-psnr --no-ssim --output "superbad_prestige2.mp4" "superbad.avs"

I'll try to check the difference betwen high and low --aq-strength (tried 0.3 first), please let me know any recommendation about this command :)

it's not as clean as the release one but it's not that far (and it's a 2000 frames encode only)

http://r4dius.free.fr/sup_2.jpg

Thanks again !!

[Sharktooth]

try not using the custom matrix and maybe lowering the intra deadzone a bit (by 1 or 2 from the default)

[Sagekilla]

Prestige is not a good matrix to use at all.. It has very bad and weird artifacting (which I can see in that image) Try using higher deblock settings because -3:-3 is a bit extreme. You'd be better off with -2:-1 or -2:-2 and lower deadzones.

[R3Z]

Prestige is not a good matrix to use at all.. It has very bad and weird artifacting (which I can see in that image) Try using higher deblock settings because -3:-3 is a bit extreme. You'd be better off with -2:-1 or -2:-2 and lower deadzones.

Prestige is not a good matrix, yet proffesional studios use it ???

I have never had any issues with prestige and crf encodes. In fact the prestige encodes show less blocking and more detail / grain retention.

[Dark Shikari]

Prestige is not a good matrix, yet proffesional studios use it ???

I have never had any issues with prestige and crf encodes. In fact the prestige encodes show less blocking and more detail / grain retention.Professional studios use a lot of bad encoding software and methods in general. They're actually quite clueless about what they're doing, in many cases, as they're simply relying on the hope that the expensive encoders they paid for are doing their job.

This cluelessness is why one finds frameblended video/similar.

[R3Z]

Professional studios use a lot of bad encoding software and methods in general. They're actually quite clueless about what they're doing, in many cases, as they're simply relying on the hope that the expensive encoders they paid for are doing their job.

This cluelessness is why one finds frameblended video/similar.

Some do, some dont. I dont think blatant generalizations (negative or positive) are helpfull.

Having not had any noticeable artifacts or issues with prestige, i will still fly its flag and encourage others to at least give it a go.

[Dark Shikari]

Some do, some dont. I dont think blatant generalizations (negative or positive) are helpfull.I don't think anyone has ever claimed that all studios are clueless, but its a pretty well known fact that many are.

If one looks at the quality of the H.264 encoders used at for the first HD-DVD/Blu-Ray discs, one realizes how little it means that "something is used by the studios."

[R3Z]

I don't think anyone has ever claimed that all studios are clueless, but its a pretty well known fact that many are.

If one looks at the quality of the H.264 encoders used at for the first HD-DVD/Blu-Ray discs, one realizes how little it means that "something is used by the studios."

Yeah i am aware my selection of words wasnt perfect, hence why i am human :p

[shon3i]

You can aslo use --level 4.1 and turn off b pyramids, to, make better decoder compactibility if you have some of newer graphich cards, you can watch 1080p on even very slow CPU processor <2.0ghz

[Sharktooth]

prestige matrix has a weird coefficient distribution. it creates artifacts in certain areas. it's definatly not good for general use.

[foxyshadis]

Prestige is not a good matrix, yet proffesional studios use it ???

I have never had any issues with prestige and crf encodes. In fact the prestige encodes show less blocking and more detail / grain retention.

And much more noise. If you already have a lot of noise/grain, you won't notice, but on a clear photographic scene (or CG) it almost looks like xvid. Never use it on cartoons/anime. It does farge up x264's rate control (and presumably other encoders') and change inloop behavior because it's not a 'balanced' matrix, but if you don't care about any of that, then it works.

[Atak_Snajpera]

You can aslo use --level 4.1 and turn off b pyramids, to, make better decoder compactibility if you have some of newer graphich cards, you can watch 1080p on even very slow CPU processor <2.0ghz

If you have latest r721 you can enable b pyramids. Tested. 1080p plays without problem on PS3.

@radius
Try RipBot264. It contains latest x264 encoder with Dark's New AQ.

[*.mp4 guy]

And much more noise. If you already have a lot of noise/grain, you won't notice, but on a clear photographic scene (or CG) it almost looks like xvid. Never use it on cartoons/anime. It does farge up x264's rate control (and presumably other encoders') and change inloop behavior because it's not a 'balanced' matrix, but if you don't care about any of that, then it works.
It will also introduce noticible bluring on very sharp sources, the inter matrix is wheighted so that it aways aplies a pseudo lowpass, which makes it very unsuited for use on high quality content (IE not blury like most studio's "profesionaly" encoded material). Of course on your average lets remove half the frequenxy spectrum for no good reason DVD/HD-DVD/Blu-Ray movie you won't have any problems with this.

[Sagekilla]

Take a look here (http://forum.doom9.org/showpost.php?p=995971&postcount=34) if you still don't believe us. Look at the source, then the output image from using prestige matrix. Ignoring the blatant color shifts, the prestige has a horizontal mice teeth pattern scattered all over the video. You would do better using M4G's high detail matrix, since it accomplishes much the same purpose minus the annoying artifacting.

[radius]

Hi guys :)
sorry I'm a bit occupied with one of my 300GB hdd death :o,
I've been compressing a sample multiple times and comparing again and again each version ... actually I'm preferring the m4g matrice most part of the time (less blocky and more details)

I don't mind the cpu use as it's mainly for ps3 / 360 or > 2.0 cpu ^^, so i'll prefer the quality to cpu use.

here's a part of what I tested, my preferred one is the 4th (with m4g matrice), I'll post some comparison pics



x264_AQ_RDRC_0.45.exe --pass 2 --bitrate 4500 --stats ".stats" --ref 5 --deadzone-inter 10 --deadzone-intra 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -2:-1 --subme 7 --analyse all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/Prestige.cfg" --progress --trellis 0 --aq-strength 0.3 --no-psnr --no-ssim --output "superbad_prestige_01.264" "superbad.avs"

x264_AQ_RDRC_0.45.exe --pass 2 --bitrate 4500 --stats ".stats" --ref 5 --deadzone-inter 10 --deadzone-intra 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -2:-1 --subme 7 --analyse all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/Prestige.cfg" --progress --trellis 1 --aq-strength 0.3 --no-psnr --no-ssim --output "superbad_prestige_02.264" "superbad.avs"

x264_AQ_RDRC_0.45.exe --pass 2 --bitrate 4500 --stats ".stats" --ref 5 --deadzone-inter 10 --deadzone-intra 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -2:-1 --subme 7 --analyse all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/Prestige.cfg" --progress --trellis 0 --aq-strength 0.7 --no-psnr --no-ssim --output "superbad_prestige_03.264" "superbad.avs"

x264_AQ_RDRC_0.45.exe --pass 2 --bitrate 4500 --stats ".stats" --ref 5 --deadzone-inter 10 --deadzone-intra 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -2:-1 --subme 7 --analyse all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/Prestige.cfg" --progress --trellis 1 --aq-strength 0.7 --no-psnr --no-ssim --output "superbad_prestige_04.264" "superbad.avs"



x264_AQ_RDRC_0.45.exe --pass 2 --bitrate 4500 --stats ".stats" --ref 5 --deadzone-inter 10 --deadzone-intra 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --no-deblock --subme 7 --analyse all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/Prestige.cfg" --progress --trellis 0 --aq-strength 0.3 --no-psnr --no-ssim --output "superbad_prestige_05.264" "superbad.avs"

x264_AQ_RDRC_0.45.exe --pass 2 --bitrate 4500 --stats ".stats" --ref 5 --deadzone-inter 10 --deadzone-intra 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --no-deblock --subme 7 --analyse all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/Prestige.cfg" --progress --trellis 1 --aq-strength 0.3 --no-psnr --no-ssim --output "superbad_prestige_06.264" "superbad.avs"

x264_AQ_RDRC_0.45.exe --pass 2 --bitrate 4500 --stats ".stats" --ref 5 --deadzone-inter 10 --deadzone-intra 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --no-deblock --subme 7 --analyse all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/Prestige.cfg" --progress --trellis 0 --aq-strength 0.7 --no-psnr --no-ssim --output "superbad_prestige_07.264" "superbad.avs"

x264_AQ_RDRC_0.45.exe --pass 2 --bitrate 4500 --stats ".stats" --ref 5 --deadzone-inter 10 --deadzone-intra 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --no-deblock --subme 7 --analyse all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/Prestige.cfg" --progress --trellis 1 --aq-strength 0.7 --no-psnr --no-ssim --output "superbad_prestige_08.264" "superbad.avs"

[radius]

Hi,
here's what i was about to keep as command line with Dark Shikari 0.47 x264 (I'm using prestige matrix again as it's the one giving me the better looking)

--level 4.1 --vbv-maxrate 25000 --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-2 --subme 7 --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/Prestige.cfg" --progress --deadzone-inter 16 --deadzone-intra 8 --aq-strength 0.8 --aq-sensitivity 0 --no-dct-decimate

I'm not activating trellis as i think it's adding blockyness, I've done a 3 pass encoding and the global quality is quite good and better than my reference release but there's a problem with edges, they sometimes get blocky compared to the release

an example
http://r4dius.free.fr/sup_3.jpg

do you guys have an idea about what I should change to fix this ?
Thanks

[Dark Shikari]

Try --aq-strength 0.6 --qcomp 1.0 --aq-sensitivity 15, using those settings on all three passes.

[shon3i]

I don't mind the cpu use as it's mainly for ps3 / 360 or > 2.0 cpu ^^, so i'll prefer the quality to cpu use.level 4.1 and 4 reference frames will not disturb quality. Even on dual core procesors decoding 1080 material is hard, and use much cpu resources during play. Aslo using this setting you will make better future hardware compatibility.

[radius]

Thanks Dark Shikari it's looking better but there's another problem now, on a 4000 frames avs the encoded version is missing frames ^^ (there's only 3836) for exemple this exact frame I took as example is missing :o, is it something with my command or a bug ?

[Dark Shikari]

Thanks Dark Shikari it's looking better but there's another problem now, on a 4000 frames avs the encoded version is missing frames ^^ (there's only 3836) for exemple this exact frame I took as example is missing :o, is it something with my command or a bug ?Is your source a file with variable framerate, maybe? What's your avisynth script?

[radius]

DirectShowSource("F:\HD\Superbad BR H264\BDMV\STREAM\00011.m2ts",fps=23.976,audio=false)
crop( 0, 20, 0, -20)
#trim(0,20000)
#trim(6028,8028)
LanczosResize(1280,688)
ConvertToYV12()

I don't think it's variable it's a normal BR (?),
it's not the first time I'm experiencing this in my recent tests, it looks like some x264 options combination produce this (my blocky encoding has all the frames)

[Dark Shikari]

I don't think it's variable it's a normal BR (?),
it's not the first time I'm experiencing this in my recent tests, it looks like some x264 options combination produce this (my blocky encoding has all the frames)x264 uses one-frame-in-one-frame-out for its API. It is physically impossible for x264 to drop frames.

[radius]

Hum I'm encoding some samples right now, I'll redo a test after that, all I can say is it was loosing frames using the same avs depending on the command I ran ^^ and the final filesize is the same either it's missing 164 frames or not (with a 4000 frame sample)

[Dark Shikari]

Hum I'm encoding some samples right now, I'll redo a test after that, all I can say is it was loosing frames using the same avs depending on the command I ran ^^ and the final filesize is the same either it's missing 164 frames or not (with a 4000 frame sample)x264 will tell you at the start of its encoding how many frames are going to be in the output file. This shouldn't change based on settings.

What setting are you finding is "changing the number of frames"?

[radius]

Ahah my bad it was something with mp4box, I verified the x264 output was saying 4000 frames then remuxed it and it's ok :rolleyes:

[squid_80]

x264 uses one-frame-in-one-frame-out for its API. It is physically impossible for x264 to drop frames.
No it doesn't. It's possible for frames to go in and nothing comes out - b-frame lag. The missing output frames must be flushed out when the input queue is empty.

[Dark Shikari]

No it doesn't. It's possible for frames to go in and nothing comes out - b-frame lag. The missing output frames must be flushed out when the input queue is empty.That conflicts with something pengvado said on #x264dev the other day; I know what you mean (yes, there is B-frame lag) but that doesn't conflict with the concept of "one in, one out."

[squid_80]

You do get an encoded frame back for every uncompressed frame fed to the encoder. But because of the lag the returned encoded frame might not be the frame you just fed it, and when you run out of input frames there may be some encoded frames still waiting to be returned.
Yes it's technically "one frame in/one frame out" but people associate that with vfw which has additional restrictions. When a b-frame lag occurs it does drop frames, the host app has to be smart enough to flush the encoding queue at the end to pick them up again.

[Inventive Software]

So the correct term is "buffer of frames in, ordered frames out"?.....

[squid_80]

Now you're getting technical :)
Let's just say the input/output is asynchronous (can happen independently) versus synchronous (i.e. vfw - an output frame must be returned for each input frame, and output can't occur without input).

And I'd like to hand the OP back his thread, sorry 'bout that. :devil:

[radius]

Eheh no problem guys ^^, as I said it was something with the h264 -> mp4 process (either mp4box or x264 implementation) so the 264 file at least was good :), btw I'm testing some --me tesa encoding, remembers me my pentium4 encoding speeds :'(

[radius]

Hi again :),
I finally found these options to produce a nice quality (with Dark Shikari 0.46 compil):
--level 4.1 --vbv-maxrate 25000 --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --deblock -2:-1 --deadzone-inter 10 --deadzone-intra 0 --subme 7 --8x8dct --merange 24 --me umh --threads auto --cqmfile "matrix/Prestige.cfg" --progress --aq-strength 0.5 --aq-sensitivity 0 --no-dct-decimate --no-psnr --no-ssim

but there's a "but" again, I've been compressing some stuff and i was quite happy with it till I tried with Disney's Cars
for some reason my encoding skips a lot of "colors" in gradients, I checked a release rip again and there's no such problem with it, then I tried a lot of options combination and could not get the gradient colors back :/, here's the release header

core 56 svn-675 - H.264/MPEG-4 AVC codec - Copyleft 2005 - http://www.videolan.org/x264.html - options: cabac=1 ref=5 deblock=1:-3:-3 analyse=0x3:0x133 me=umh subme=7 brdo=0 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=2 deadzone=21,11 chroma_qp_offset=0 threads=8 nr=0 decimate=1 mbaff=0 bframes=3 b_pyramid=1 b_adapt=1 b_bias=0 direct=1 wpredb=1 bime=1 keyint=240 keyint_min=24 scenecut=40(pre) rc=2pass bitrate=4729 ratetol=1.0 rceq='blurCplx^(1-qComp)' qcomp=0.60 qpmin=10 qpmax=51 qpstep=4 cplxblur=20.0 qblur=0.5 ip_ratio=1.40 pb_ratio=1.30 zones=0,1,b=0.1

http://r4dius.free.fr/c1.jpg http://r4dius.free.fr/c2.jpg

So I tried removing the matrix, changing intra and inter values enabling / disabling a lot of the switchs nothing would do, anyone having an idea ?
Thanks

[Adub]

First of all, update to Shikari's newest patch version, there are builds all over the place.

Second, are you serving this to x264 with Avisynth? If yes, what does your script look like.

Third, when you say release, are you talking about a scene release?

[neuron2]

@radius

Your punishment for violating rule 6 (discussing illegimate content) is thread closure.