Fade-in ugly blocking problem, any progress? [Archive]

Dr.D

15th September 2008, 04:38

Quark.Fusion

15th September 2008, 04:40

What decoder you use?

LoRd_MuldeR

15th September 2008, 04:42

AFAIK the new "B-Frame Decision" patch should allow x264 to handle fades MUCH better ...

Dark Shikari

15th September 2008, 04:51

Couple of years ago I've tried to move from XviD to x264. Finally I didn't. One of the reasons - subj. Now I've tried again - same thing. The forum search shows that I'm not alone, somebody politely calls that "x264 is not very good with fade-ins", but for me it's just unacceptable, blocking is so-o-o-o ugly that ruins whole video.
This is doesn't make any sense for me - why it's so complicated and hard to fix? In the Wiki H264 article I see:

Weighted prediction, allowing an encoder to specify the use of a scaling and offset when performing motion compensation, and providing a significant benefit in performance in special cases—such as fade-to-black, fade-in, and cross-fade transitions.

as the ADVANTAGE of the H264. I understand that in this areas codec increases quantizer to decrease bitrate. So, for hell sake, just do NOT do that!

Anyway, I'm looking for suggestions to minimize this problem, at least make the blockiness not so ugly. Or I have to don't use x264 for videos with fade-ins?Posting about a specific problem relating to x264 and fade-ins is fine--making general statements without posting streams of the problem is totally meaningless and makes it impossible to help you or improve the code to fix a specific problem.
I understand that in this areas codec increases quantizer to decrease bitrate. So, for hell sake, just do NOT do that!You seem to understand wrong. Xvid is far more likely to break on fades than x264 is, given the combination of a few factors:

1. No adaptive quantization (unless you're using a VAQ patched version) so the darkest frames of a fade will look awful.
2. The same B-frame decision as x264 (to some extent). x264's improved B-frame decision will be patched into the official version within a few days, which will perform even better on fades.
3. No weighted prediction.
4. No deblocking filter.

If you have found a specific case where x264 has problems with fades, post settings and an output stream. Whining makes it less likely that you will be helped.

foxyshadis

15th September 2008, 07:29

as the ADVANTAGE of the H264. I understand that in this areas codec increases quantizer to decrease bitrate. So, for hell sake, just do NOT do that!

x264 has an option to turn this on or off already, it's --weightb, and it's even off by default! But you plainly don't understand it, so I wouldn't do that, as it can only increase quality. It doesn't increase quantizer, it just reduces the residual that has to be encoded, like nearly every option that increases quality. Somehow I'm sure that it's more likely your encoding settings are bad, or even that you've completely turned in-loop deblocking off, use 1-pass ABR, or use too low a bitrate in general.

Manao

15th September 2008, 09:20

foxyshadis : not quite : weightb != explicit weighted prediction. Though weightb improves fades, it also improves everything else (except anime), and is actually not explicitely meant to handle fades. Explicit weighted prediction is the tool for that.

Dr.D

15th September 2008, 12:22

Posting about a specific problem relating to x264 and fade-ins is fine--making general statements without posting streams of the problem is totally meaningless and makes it impossible to help you or improve the code to fix a specific problem.
Oh-Oh. Your post:
http://forum.doom9.org/showpost.php?p=1039573&postcount=107
"That video seems to be a very bad example overall, for two reasons:
1. Tons of fades, which x264's frame decision algorithm is horrible with."

So, the problem exists, you know about the problem, it's not stream specific and well-known. How my stream can help? For the common well-known problem should be common ways for solution.

But I'll post the stream and screenshots soon.

Dr.D

15th September 2008, 12:54

What decoder you use?
CoreAVC.

Dr.D

15th September 2008, 13:03

AFAIK the new "B-Frame Decision" patch should allow x264 to handle fades MUCH better ...
I use this build:
http://forum.doom9.org/showpost.php?p=1181245&postcount=1008
The new B-frame decision in.

Sharktooth

15th September 2008, 13:21

you should set --b-adapt 2 then

Sagekilla

15th September 2008, 13:29

Can you post us a sample that's proving to be troublesome, and settings you used for x264? We can't really help your situation if all you're doing is telling us "it looks bad"

Dr.D

15th September 2008, 13:29

Dr.D

15th September 2008, 13:48

you should set --b-adapt 2 then
Thank you, but doesn't help too much.

See the problem, x264 decreases bitrate a lot in fades, thanks, it's "smart", but I don't need that! :)

Just a minute, I'm going to try this setting in combination with:
--qpmax 35 --qpstep 20

I've tried this before, helped a bit, but without --b-adapt 2.

Gabriel_Bouvigne

15th September 2008, 14:17

It's strange: it looks like your deblocking is off

Dr.D

15th September 2008, 14:35

Another "clue":
http://forum.doom9.org/showthread.php?p=1062149#post1062149

"...And Mainconcept/Elecard SDK AVC is by far better than x264 for that:
- Fade scene with wpred for pframe..."

So, 1) the problem exists, it's not a my specific problem, it's by the x264 design; 2) the problem can be fixed (Mainconcept did that).
Why (almost) everybody, including developers, says opposite?

I can understand answers like:
"oh, yes, we know about the problem, but:
- it so hard to fix
- it's just a minor problem, we even don't care to fix that"

but anwsers like - "hmm... fades problem... strange, never heard, fades are fine in x264, it must be your stream-settings-decoder-etc. problem" - are really weird for me.

Dr.D

15th September 2008, 14:57

It's strange: it looks like your deblocking is off

If you mean encoder settings - I've tried both, on and off.

If you mean decoder settings -sure it is, because:
1) deblocking blurs.
2) deblocking uses CPU a lot.
3) I need movie what looks good everywhere, on any software or hardware player. I don't like to have a following dialog with everybody whom I'll give a movie - "don't forget to turn on deblocking - what the hell is that?".

I know I shouldn't tell this here :). Using deblocking is super-hyper-mandatory on this forum :).
I'm not going to discuss about that now, may be I'll open a separate topic. I'm just saying - x264 is great codec, I'm completely satisfied of my result with deblocking off, way better than XviD. EXCEPT fading spots. Because x264 lower bitrate too much in this spots, I don't need this.

Comatose

15th September 2008, 15:08

3) I need movie what looks good everywhere, on any software or hardware player. I don't like to have a following dialog with everybody whom I'll give a movie - "don't forget to turn on deblocking - what the hell is that?".
Deblocking is on by default...

Manao

15th September 2008, 15:09

Dr.D

15th September 2008, 15:21

Dr. D : you're confusing the issue on deblocking. Let's separate inloop-deblocking, which, if used, must be used on both encoder and decoder, and deblocking (as a postprocessing), which is used only on the decoder.

I do hope that in the screenshot you posted, inloop deblocking was on.
Ok, inloop-deblocking is on on both encoder and decoder (I guess, I don't see an option on the CoreAVC decoder);
Decoder's Deblocking oprion is off.

Sharktooth

15th September 2008, 15:22

WTF, inloop deblocking should be NEVER disabled! so here's your problem...

Dr.D

15th September 2008, 15:24

Deblocking is on by default...
Not neccessary, on ffdshow postprocessing is off.

fields_g

15th September 2008, 15:28

Not neccessary, on ffdshow postprocessing is off.

I would venture to guess that most people here don't use post processing deblocking here. When you see deblocking mentioned, it is safe to assume they are talking about inloop deblocking.

Dr.D

15th September 2008, 15:37

WTF, inloop deblocking should be NEVER disabled! so here's your problem...
Oh, again, "your problem"...
Please read the thread - it's not a my problem.
Ok, please make your encoding with crf 25 and post the settings and result.

BTW, you can see my settings, inloop deblocking is not disabled.

15th September 2008, 15:45

Ok, inloop-deblocking is on on both encoder and decoder (I guess, I don't see an option on the CoreAVC decoder);
Decoder's Deblocking oprion is off.
What do you mean by "Decoder", CoreAVC? If so, you have turned inloop deblocking off in the decoder. The screenshot pretty much confirms this too since such corruption does not happen otherwise. Switch deblocking on and see if it helps.

Terranigma

15th September 2008, 15:53

please make your encoding with crf 25 and post the settings and result.

Here's (http://www.mediafire.com/?j4nw08dcmwu) my go at it with a modified revision (968) by TheRyuu.

I've used a cqm, because personally, I always felt that the flat matrix didn't handle fades well, even with psy-rdo and aq.

some notable settings used were:
--nf
--crf 20 (25? wth :eek:)

Conclusion: For fades, Explicit weighted prediction or a cqm would be the way to go. :devil:

Comatose

15th September 2008, 15:55

Yeah, 25 is really high :/

Ranguvar

15th September 2008, 16:00

The screenshot does indeed look very much like inloop is disabled. Considering inloop is H.264's biggest advantage over Xvid, except perhaps CABAC...

I too prefer slight blocking over blurring. If you want a more "Xvid-esque" feel to your encodes, play with the deblock settings during encode. A good guide is here (http://forum.doom9.org/showthread.php?p=810932#post810932). But turning off deblocking is suicidal. Almost all players support deblocking (pretty much everything except the original Xbox with XBMC, and maybe some PDAs/smartphones... iPod *might * work, PSP works, Archos works, 360 and PS3 work, Blu-ray/HD-DVD work, Popcorn Hour works, etc.). Plus, you make it seem like it's hard to use the inloop during playback? ALL decoders I know off use inloop by default.

And one last thing, inloop does not take a ton of power at all... try and see :)

Sorry if we seem indignant, but you appear to likely be using insane settings and then are accusing x264 of bugs? :) At least hold off on that until it's pretty obvious nothing's going wrong. I'll play your encoded video with deblocking *for sure* on when I get back home and see if I get better results than you.

--b-adapt 2 and --weightb should solve all but the most resistant fade problems without the need for Seppuku-by-CQM ;)

Gabriel_Bouvigne

15th September 2008, 16:08

*Yes, x264 is not very efficient when encoding fades. Explicit weighted prediction is not implemented, so it tries to do as best as possible with the other tools (including implicit wpred on B frames), and it could possibly do a better job using explicit wpreds.

*Yes, there seems to be a problem within your settings: you are encoding with deblocking enabled within the encoder, but you manually disabled it within the decoder ? You should make your choice: either you don't touch default deblocking settings (which is enabled by default, for a good reason), or you disable deblocking within the encoder (which will flag the stream as not using deblocking), but you should NOT have deblocking enabled within the encoder and manually disable it within the decoder.

Dark Shikari

15th September 2008, 16:16

Looks like a simple case of deblocking off during decoding blamed on x264. I haven't seen one of these in a while :p

Manao

15th September 2008, 16:20

Note to all : his commandline doesn't disable deblocking. I raised the issue because his vocabulary was confusing, but I think it did enable inloop deblocking, on both sides

15th September 2008, 16:25

Note to all : his commandline doesn't disable deblocking. I raised the issue because his vocabulary was confusing, but I think it did enable inloop deblocking, on both sides
Yes, but it seems that he has turned inloop deblocking off in CoreAVC settings.

Manao

15th September 2008, 16:28

He did say :"Ok, inloop-deblocking is on on both encoder and decoder (I guess, I don't see an option on the CoreAVC decoder);"
But then, I (or he, or you, or us) may be confused...

Quark.Fusion

15th September 2008, 16:30

CoreAVC.

…the fade in/out bug we fixed as in the new internal version I am using does not have the bug shown.
Maybe it related? Try another decoder.

Dark Shikari

15th September 2008, 16:31

He did say :
But then, I (or he, or you, or us) may be confused...You can tell quite clearly from the image that inloop deblocking is off.

Also, as I've said before, its not a bug in CoreAVC because the bug in CoreAVC is only triggered by weighted P-frame prediction, which x264 doesn't use yet.

Dr.D

15th September 2008, 17:23

Here's (http://www.mediafire.com/?j4nw08dcmwu) my go at it with a modified revision (968) by TheRyuu.

I've used a cqm, because personally, I always felt that the flat matrix didn't handle fades well, even with psy-rdo and aq.

some notable settings used were:
--nf
--crf 20 (25? wth :eek:)

Conclusion: For fades, Explicit weighted prediction or a cqm would be the way to go. :devil:
Thanks!
This encoding looks fine for me (even with deblocking off on the decoder).
Could you please do the same, but with crf 25? The rest of the movie (without fades) looks fine for me with crf 25, so I don't need to double-size the file only for a few fading spots.

P.S. Just made encoding with my settings, but with --crf 20 - fine too. But, again, I don't need a huge file size increase to fix 1% of movie with fades.

P.P.S. But your encode looks promising - file size is 1.5MB (mine only 500KB). It's exactly what I'm looking for - don't do this stupid "smart" dropping of bitrate in fades!
Could please post or PM your settings and the cqm?

Audionut

15th September 2008, 17:38

You can tell quite clearly from the image that inloop deblocking is off.

http://img511.imageshack.us/img511/6295/testfadesdflt0000380733kx4.jpg

An encode at crf 25 with inloop deblocking disabled.
And you wonder why it's a blockfest.

Regardless of what you think the problem might be, I find it highly disrespectful that you tell the devs they should be adjusting their baby because you decide you want it this way or another.

How bout doing the coding yourself, then you can have the encoder doing whatever you want.

Sharktooth

15th September 2008, 17:57

again.. dont turn off the inloop deblocking... neither in the encoder (you didnt) or the decoder (you did).
here's again YOUR problem...

Manao

15th September 2008, 18:00

Dr. D : please, can you confirm (using a screenshot of CoreAvc or FFdshow's panel configuration) that you didn't force the decoder to disable inloop deblocking. The issue has been so much confused that nobody will believe you if you don't, which would be a shame since if inloop is enable, then you remark is perfectly valid.

Dr.D

15th September 2008, 18:12

I would venture to guess that most people here don't use post processing deblocking here. When you see deblocking mentioned, it is safe to assume they are talking about inloop deblocking.
Yes, you are right, inloop deblocking is on by default on ffdshow decoder.

cyberbeing

15th September 2008, 19:11

One thing that I have noticed that hasn't been touched on is that the higher strength/threshold you set deblocking the encoder, the more often x264 purposefully blocks in large areas (usually flat) with the expectation of it being deblocked. With a low deblocking setting (say -3,-3) x264 usually seems to try to block in smaller, less obvious areas, likely in attempt to keep PSNR up and maintain perceptive quality. This may or may not be the case in the sample here considering that with a major lack of bitrate there is only so much x264 can do.

To sum it up, at least on anime content, if you are disabling in-loop deblocking in the decoder (for whatever reason) using a low deblock setting or even -no-deblock will usually give you a perceptually less-blocky output at the same bitrate in my experience when doing a multi-pass encode.

Only thing you may want to try is sticking a I frame right before a fade-out, during the 50/50 point of a fade-out/fade-in, and another right after the fade-in is complete with a qpfile and hacked stats (I really wish there was a way to keep b-adapt and scenecut while using a qpfile without having to hack your stats file). At least on anime content, even with scenecut maxed out x264 will forgot to place an I frame after a fade-in/fade-out and just wait for keyint to be triggered which usually results in a lack of quality in those areas. (Off-topic: The same thing also happens with large panning scenes and can cause a huge very abrupt quality jump right in the middle of pan from keyint being triggered no matter what the deblocking setting is) You could also use zones to increase the bitrate in those areas until the problem is solved.

Manao

15th September 2008, 19:16

the higher strength/threshold you set deblocking the encoder, the more often x264 purposefully blocks in large areas (usually flat) with the expectation of it being deblocked.You give too much credit to the encoder. x264 ignores deblocking when it decides the type/motion vector of a macroblock. What you see might instead be a side effect of having a deblocked reference frame. Since deblocking improve the reference frame, it might increases the likeliness for skipping the macroblock which in turn might give the impression of larger blocked area (while the edge of that area would not be blocky, the area itself would be squareish).

cyberbeing

15th September 2008, 19:28

...What you see might instead be a side effect of having a deblocked reference frame...

I don't see myself giving x264 too much credit because that is exactly what I expect is the reason for that happening. It still doesn't change that x264 behaves in this fashion.

kemuri-_9

15th September 2008, 19:42

Only thing you may want to try is sticking a I frame right before a fade-out, during the 50/50 point of a fade-out/fade-in, and another right after the fade-in is complete with a qpfile and hacked stats (I really wish there was a way to keep b-adapt and scenecut while using a qpfile without having to hack your stats file).

true --qpfile disables scenecut and b-adapt, but it has no qualms about you re-enabling it after you call --qpfile
i.e.
--qpfile x.txt --scenecut 50 --b-adapt 1
x264 accepts that and chugs away with a scenecut of 50 and --b-adapt of 1.

granted this should be used at a large risk with the ever increasing specificity of the qpfile:
just specifying a few frames as IDR has only a little risk.
more forced frames -> more risk

cyberbeing

15th September 2008, 19:47

true --qpfile disables scenecut and b-adapt, but it has no qualms about you re-enabling it after Ah, I usually stuck qpfile at the end of the command line, I didn't know you had to stick it before scenecut and b-adapt... Thanks for the heads up kemuri.

And yes, I usually only add a few IDR frames in problem areas like fade-in/fade-outs and long pans where x264 starts producing blocky/distorted P and B frames from lack of a good reference frame. I know that excessive use of a qpfile will degrade quality.

Dr.D

15th September 2008, 19:57

Ok, here are the samples:
Original video (http://www.mediafire.com/?hzllgl43rl1)

Default settings (http://www.mediafire.com/?en72ddewwio)
x264.exe --progress --fps 29.97 --crf 25.0 --output "Test Fades Dflt.mkv" "Main.avs" 1280x720

Main settings (http://www.mediafire.com/?vyhq8dx62mn)
x264.exe --progress --threads auto --no-fast-pskip --no-dct-decimate ^
--8x8dct --me umh --subme 7 --bime --level 4.1 ^
--mixed-refs --ref 4 --bframes 3 ^
--b-rdo --psy-rd 1.0:1.0 --trellis 2 ^
--fps 29.97 --aq-strength 1.0 --crf 25.0 ^
--no-psnr --no-ssim ^
--output "Test Fades.mkv" "Main.avs" 1280x720

Screenshot:
http://img255.imageshack.us/img255/4452/testfadesdflt0000380733ij0.th.png (http://img255.imageshack.us/my.php?image=testfadesdflt0000380733ij0.png)

In the movie it looks worse - dancing squares are awful.
Ok, I think I have to clarify this little mess about inloop deblocking. Actually, I was a bit confused myself.
1. Personally I use inloop deblocking OFF for both encoding and decoding. I know it's very risky to say so here, same as post good steak recipe in vegetarian forum :). Please do not discuss about this here, I think I'll open a separate thread why, after long experiments, I've decided that for my tasks it's better.

2. Encodes above have inloop deblocking ON. Why? Because if not, I'm 100% sure that only advise will be - "you should not ever-never-forever turn inloop deblocking off". Correct? :)

3. Screenshot above is from Default Settings encode and Decoder inloop deblocking OFF. Why? To show that encoder decreases bitrate way too much. So much, that Decoder's inloop deblocking can't handle it properly. Try to download encoded clip and play with Decoder's inloop deblocking ON. You will see very blurred picture with dancing artefacts. Like "good" old VCD encodes. It's understandable, impossible to make a candle from so low bitrate shit.
Sorry for any misleading caused by not providing all the details at first.

Guys, I'm sure you are looking in wrong direction. With or without inloop deblocking, the logic of encoding fade-ins is wrong - it's lower bitrate too much.
See an another proof.
I just made encoding of 2 clips.
First one (http://www.mediafire.com/?hzllgl43rl1) is the original clip, 55 frames.
Second one (http://www.mediafire.com/?cgzlyu916eg) is the frames 25 - 54 from the original clip. So, the dark and biggest part of fade-in is cut off. 30 frames, just bit more than HALF of the original clip.
What do you think about the sizes of encoded clips (don't worry, with inloop deblocking ON)? Second will be smaller then first by? 50%? 20%? 5%? 1%? No, all wrong.
Encoded second clip (what is the about half of the first) is BIGGER by 37%! 392529 against 286845 bytes. And the quality of the second clip, not surprise, is better, because almost no fade there.
http://www.mediafire.com/?qm3yx9bniyu
http://www.mediafire.com/?sbkvnaiqgcz
Again:

Encoded whole clip: 55 frames, 286845 bytes
Encoded part of the clip: 30 frames, 392529 bytes, 37% more.
Deblocking is ON.

Do you smell a rat? I do the big, fat, stinky one. If you don't, I give up...

My settings:
x264.exe --progress --threads auto --no-fast-pskip --no-dct-decimate ^
--8x8dct --me umh --subme 7 --bime --level 4.1 ^
--mixed-refs --ref 4 --bframes 3 ^
--b-rdo --psy-rd 1.0:1.0 --trellis 2 ^
--fps 29.97 --aq-strength 1.8 --crf 25.0 ^
--no-psnr --no-ssim ^
--output "Fade Test Cut.mkv" "Main.avs" 1280x720

cyberbeing

15th September 2008, 20:03

2. Encodes above have inloop deblocking ON. Why? Because if not, I'm 100% sure that only advise will be - "you should not ever-never-forever turn inloop deblocking off". Correct? :)
Well as I said in my post, if you or anybody else viewing your encode never plans to use the in-loop deblocker when decoding, you may get better results turning deblocking down or off in the encoder.

Sharktooth

15th September 2008, 20:15

--aq-strength 1.8.. way too much. --no-dct-decimate ... useless... --no-fast-pskip... useless... AND clip i too short...
also... again... DONT DISABLE the inloop deblocker.
you cant whine about blocks while disabling the deblocker...

Dark Shikari

15th September 2008, 20:19

3. Screenshot above is from Default Settings encode and Decoder inloop deblocking OFF. Why? To show that encoder decreases bitrate way too muchThat's a completely invalid argument. In loop deblocking is just that--in loop. Every frame depends on the output of the previous frame, so even if the bitrate is enormous, gigantic artifacts can accumulate from frame to frame if you disable the inloop deblocker. It has nothing to do with the quality of the encoding.

Terranigma

15th September 2008, 20:20

I see that some of you guys are pointing to in-loop being a problem, and that it should always be used. Well, my sample was encoded with --nf, so there's no in-loop (for the sharpest result possible).

With the progression that's been made to x264 over the past year, it's now possible you can configure x264 so that in-loop is practically useless (imho anyways). :p

Sharktooth

15th September 2008, 20:23

wrong... completely. inloop is adaptive, so its not a problem at all. also you can change the adaptiveness bias with the strength and threshold options...

Terranigma

15th September 2008, 20:25

wrong... completely. inloop is adaptive, so its not a problem at all. also you can change the adaptiveness bias with the strength and threshold options...

Useless as in, extra cpu usage for less sharper/pleasing results.

Sharktooth

15th September 2008, 20:26

and a blockfest without it... inloop is part of the h.264 specs for a reason...

Terranigma

15th September 2008, 20:27

and a blockfest without it...
Haha, not true. Maybe for you and your settings, but not mines. ;)

Sharktooth

15th September 2008, 20:29

are you sure? unless quants are really low three will be blocks more or less visible... but blocks.
you cant encode for example at quant 30 and have no blocks without inloop...

Terranigma

15th September 2008, 20:32

are you sure?
Yes I'm sure.

EDIT:
you cant encode for example at quant 30 and have no blocks without inloop...

That's true. I guess in-loop would help there, but i'm speaking from a range between 16-20; sometimes up to 22.

Sharktooth

15th September 2008, 20:39

in that range the deblock is not doing much... expecially with negative strength and/or threshold...IIRC @ Q15 it is always disabled (with 0:0).

Dr.D

15th September 2008, 22:33

Dark Shikari

15th September 2008, 22:43

Thanks, it's exactly what I do normally (but tsss..., it's a big secret, I don't want to be electrocuted for that :)), and I like the results very much. Except fade-in spots.Obviously it isn't what you do since you just said you disabled deblocking in the decoder... :rolleyes:

cyberbeing

15th September 2008, 22:59

Thanks, it's exactly what I do normally (but tsss..., it's a big secret, I don't want to be electrocuted for that :)), and I like the results very much. Except fade-in spots.

Well even though I said that, just to make sure you understand, you should still always avoid disabling in-loop deblocking in the decoder and control the deblocking strength/threshold in x264 when encoding.

The only time I disable deblocking in the decoder is when I'm trying to see where blocking is occurring and in effect where there will be detail loss / blurring when in-loop kicks in. Different deblocking strengths/thresholds will have blocking in different areas of a frame and sometimes different frames altogether. Since I like detailed encodes, my goal is to end up with minimal detail loss and not have any obvious blocking which is a bit of a see-saw. Sometimes this means higher in-loop and other times it means lower in-loop.

akupenguin

16th September 2008, 00:58

To sum it up, at least on anime content, if you are disabling in-loop deblocking in the decoder (for whatever reason) using a low deblock setting or even -no-deblock will usually give you a perceptually less-blocky output at the same bitrate in my experience when doing a multi-pass encode.
That's trivially true, but unrelated to your previous paragraph. If you encode with deblocking and decode without it, then you have encode-decoder desync, which is far uglier than any possible compression artifact. The weaker the deblocking, the less the desync in that case.

cyberbeing

16th September 2008, 01:26

That's trivially true, but unrelated to your previous paragraph. If you encode with deblocking and decode without it, then you have encode-decoder desync, which is far uglier than any possible compression artifact. The weaker the deblocking, the less the desync in that case.

This is the first time I've heard of encoder-decoder desync so I'm a bit confused by this. Are you saying that the blocking you see when disabling in-loop in the decoder doesn't actually exist in the stream? I don't see how that's possible but maybe I don't fully understand how h.264 in-loop deblocking works.

Can someone show me an example of what "encode-decoder desync" blocking looks like compared to "compression artifact" blocking?

Dark Shikari

16th September 2008, 01:32

I'm a bit confused by this. So are you saying that the blocking you see when disabling in-loop in the decoder doesn't actually exist in the stream?Correct.I don't see how that's possible but maybe I don't fully understand how h.264 in-loop deblocking works.

I'll give a simple example then:

Frame 0 is decoded, then deblocked. We'll call this output frame 0A.

Frame 1 is decoded, using Frame 0A as its reference.

If you turn off deblocking:

Frame 0 is decoded.

Frame 1 is decoded, using Frame 0 as its reference.

Since it uses a different frame as its reference, the output is going to be different--even if you deblock frame 1 and not frame 0, it'll look quite different than what the encoder intended.

Now imagine this error stacking up over 200 frames straight.

LoRd_MuldeR

16th September 2008, 01:38

I don't see how that's possible but maybe I don't fully understand how h.264 in-loop deblocking works.

In H.264 the deblocking filter is a mandatory part of the decoder. That's different from MPEG-2 or MPEG-4 ASP.
Hence the encoder relies on the decoder to do a proper deblocking. The encoder even references to deblocked frames!
So if you disable the deblocking in your H.264 decoder, you actually break the decoding and you'll get corrupted output.
You will not only see more blocking, you also get more serious artifacts that result from invalid reference frames...

cyberbeing

16th September 2008, 01:50

If I'm understanding you correctly, I'm assuming that means you can't judge compression artifacts in an encode by disabling in-loop in the decoder. How would you judge where compression artifacts are occurring and in turn what parts of a frame deblocking is being applied? When in-loop deblocking is turned on in the decoder it would be deblocking the blocks which defeats the purpose or identifying problem areas. Right?

Dark Shikari

16th September 2008, 01:56

When in-loop deblocking is turned on in the decoder it would be deblocking the blocks which defeats the purpose or identifying problem areas. Right?Huh? Deblocking is part of the encoding and decoding process--compression artifacts can only be measured when the entire encoding and decoding process is done correctly. If you do the process wrong, you'll get false results, obviously.

cyberbeing

16th September 2008, 02:00

LoRd_MuldeR

16th September 2008, 02:01

You might use h264visa (http://h264visa.com/index.html) or a similar tool to analyze your encode. It can show tree different "pictures" for each frame:
Predicted (result of reference+mc), Residual (difference between "predicted" and original), Pre-LoopFilter (result of predicted+residual, without deblocking) and Final (deblocked image)

Dark Shikari

16th September 2008, 02:05

While I understand what you're trying to say, I'm still under the belief that even with encoder-decoder desync, those areas are actually where blocking is occurring even if it may be worse then the encoder intended. This seems to be supported by these area getting blurred when in-loop is turned back on in the decoder. Is this wrong?The proper way to do this is to have a decoder that displays frames before they are deblocked, but still does the deblocking behind the scenes to avoid errors accumulating.

But I still don't understand your point--the entire purpose of the deblocking filter is to compensate for the weakness of block-based DCT formats by subtly blurring the edges of each block based on data from the stream. A block is not a "problem area" unless its problematic after the filter has had its way with it, too.

The entire purpose of the filter is so that the encoder doesn't have to worry about limiting blocking and can instead rely on the sane method of the loopfilter to deal with it. Also note that the purpose of the deblocking filter is first and foremost not visual; block edges tend to make motion compensation much less efficient, so resolving them increases coding efficiency even if you don't display the deblocked frame.

If you want to identify "problem areas", I'd say the best method is to take your eyes, watch the freaking video, and ask yourself what areas look problematic. Not try to make an analysis based on decoded frames that aren't supposed to exist.

cyberbeing

16th September 2008, 02:24

The proper way to do this is to have a decoder that displays frames before they are deblocked, but still does the deblocking behind the scenes to avoid errors accumulating.

Unfortunately there doesn't seem to be a free or affordable tool for this. h264visa sounds interesting but it's really expensive at $270 or $450 for the useful versions.

A block is not a "problem area" unless its problematic after the filter has had its way with it, too.
If a block appears over important detail in a frame, when de-blocked detail is lost. I view all blocking as problematic, just some less so then others.

If you want to identify "problem areas", I'd say the best method is to take your eyes, watch the freaking video, and ask yourself what areas look problematic. Not try to make an analysis based on decoded frames that aren't supposed to exist.
What I was doing was comparing the normal deblocked output, the output with deblocking disabled, and the source video to get a general idea for where in a frame blocking and hence detail loss was happening. It now seems this is not the best way to go about it so I guess I'll eliminate disabled decoder deblocking from the mix or at least keep in mind the decoding errors and take it's output for a grain of salt.

Thank you everybody for clearing up my misunderstanding.

LoRd_MuldeR

16th September 2008, 02:26

Unfortunately there doesn't seem to be a free or affordable tool for this. h264visa sounds interesting but it's really expensive at $270 or $450 for the useful versions.

You can test it for free. The unrestricted trial version works for 30 days and you can re-install as often as you like. No registration needed.

Dark Shikari

16th September 2008, 02:34

What I was doing was comparing the normal deblocked output, the output with deblocking disabled, and the source video to get a general idea for where in a frame blocking and hence detail loss was happening.Blocking and detail loss are completely orthogonal. In fact, most psychovisual optimizations I know of that bias in favor of more detailed images tend to increase blocking.

cyberbeing

16th September 2008, 02:34

you can re-install as often as you like
Oh, I thought they would have some sort of trial protection to prevent use even after a re-install, guess not. Thanks for the info LoRD_MuldeR.

Blocking and detail loss are completely orthogonal. In fact, most psychovisual optimizations I know of that bias in favor of sharper, more detailed images tend to increase blocking.
I don't mind a bit of blocking as long as it's not overly obvious, it's the blocking being blurred to hell with deblocking that can sometimes be an issue which is why I stick with negative in-loop settings. Hence higher in-loop in x264 leads to a softer, potentially less detailed output on an encode that has lots of blocking. All of a sudden your high detail source becomes a low detail x264 encode because of a high in-loop deblocking setting blurring the blocks which seem to tend to appear on detailed areas and sharp edges.

LoRd_MuldeR

16th September 2008, 02:36

Oh, I thought they would have some sort of trial protection to prevent use even after a re-install, guess not. Thanks for the info LoRD_MuldeR.

Obviously they do not, even though the product is quite expensive. I was surprised myself...

Ranguvar

16th September 2008, 03:16

I don't mind a bit of blocking as long as it's not overly obvious, it's the blocking being blurred to hell with deblocking that can sometimes be an issue which is why I stick with negative in-loop settings. Hence higher in-loop in x264 leads to a softer, potentially less detailed output on an encode that has lots of blocking. All of a sudden your high detail source becomes a low detail x264 encode because of a high in-loop deblocking setting blurring the blocks which seem to tend to appear on detailed areas and sharp edges.

Me, too - but I use *reasonably* low settings, like -2:-1. Killing deblocking, especially in only the encode or decode, or setting abnormally low values, is going to be really bad. And at such high quants, especially so.

A blur is better than a block, IMO. But I've been very surprised by the quality of the inloop deblocker. It's *far* better than post-processing deblockers. Blurs do not look like blurs. And besides, the only reason I am okay with some blockiness is because I'd rather have that than loose detail, like you. Well, x264 does a great job of being adaptive with its deblocking. Higher deblocking tends to do well at eliminating edge noise/dirt, and overall doesn't blur much at all.

All I'm saying is, do some tests. You may be surprised.

cyberbeing

16th September 2008, 03:54

LoRd_MuldeR

16th September 2008, 04:11

I agree with you that x264 does have a good in-loop deblocker

x264 does not "have" an in-loop deblocker. The inloop-deblocker is part of the H.264 specifications - all encoders and decoders use it in the same way...

The filter is applied to the vertical and horizontal edges of 4×4 chroma or luma blocks in a macroblock. The amount of filtering is measured by boundary strength (bS), which is decided by the quantiser parameter, coding modes of the adjacent blocks and the gradient of the sample cross the edge. Two thresholds (α and β) defined in the standard make decision on whether to perform filtering on the current boundary or not. α and β depend on the average QP of the two blocks adjacent to the edge: they increase with the increase of QP and vice versa. In the design of the deblocking filter, two controllable parameters α_offset and β_offset are added to provide a wider range of control of the filter along with QP. When α_offset and β_offset are set to zero, α and β depend on QP only. The values of α and β for any given QP have been chosen based on rate-distortion performance results.

akupenguin

16th September 2008, 04:33

Can someone show me an example of what "encode-decoder desync" blocking looks like compared to "compression artifact" blocking?
Desync artifacts don't necessarily look like blocks.

source (http://img140.imageshack.us/img140/7281/sourcefh5.png)
encode with deblock (http://img247.imageshack.us/img247/7414/deblockyt4.png)
encode without deblock (http://img247.imageshack.us/img247/2972/nodeblockhz0.png)
encode with deblock, decode without (http://img213.imageshack.us/img213/6914/desyncxm7.png)

cyberbeing

16th September 2008, 04:43

Wow, I've never seen anything like that before, must be lucky. Why do decoders even expose options to completely disable deblocking if it can cause such nasty side-effects? Performance shouldn't even be an excuse. Thanks akupenguin for the examples.

woah!

16th September 2008, 05:03

lets remember here he is giving this encode about 1500kbps on a 720 res fade-in source aswell... i would like to see the mainconcept result please, i bet it looks like ass aswell... also crf 25 for you isnt going to be the same for me if i use different settings, if i use crf 25 right now it gives me 3400kbps to play and of course looks way better... cant expect miracles mate...

try a 2pass fixed bitrate until you see a pleasing result and than adjust your crf to suit, i bet its way lower than 25...

heres my go at crf 28 which gives me a file the same size as your posted one : http://img524.imageshack.us/my.php?image=clipboard02ub3.png

Quark.Fusion

16th September 2008, 05:35

And yes, I usually only add a few IDR frames in problem areas like fade-in/fade-outs and long pans where x264 starts producing blocky/distorted P and B frames from lack of a good reference frame. I know that excessive use of a qpfile will degrade quality.
Maybe more refs and b-frames with --b-adapt 2 will help?

cyberbeing

16th September 2008, 06:16

Maybe more refs and b-frames with --b-adapt 2 will help?

It doesn't, already been tried on a few sources with no improvement. Once the current frame has nothing in common with the previous I frame quality starts degrading quickly and sometimes produces ugly blocking. Since x264 doesn't detect fade-in/fade-out or long pans as a scene change you have to add them manually.

Dark Shikari

16th September 2008, 06:49

Since x264 doesn't detect fade-in/fade-out or long pans as a scene change you have to add them manually.Yes, x264 detects fades; that's one of the prime purposes of b-adapt 2.

Also, note that putting I-frames on fades is exactly what you don't want to do. Fades need to be mostly B-frames, and in an ideal encoder, B-frames with weighted P-frame prediction between the P-frames.

cyberbeing

16th September 2008, 06:56

Yes, x264 detects fades; that's one of the prime purposes of b-adapt 2.

Also, note that putting I-frames on fades is exactly what you don't want to do. Fades need to be mostly B-frames, and in an ideal encoder, B-frames with weighted P-frame prediction between the P-frames.

Who ever said anything about putting I-frames on fades? You misunderstood me, I'm talking about scene changes not the fades themselves. Once a fade-out/fade-in completes x264 doesn't add an I-frame for the scenechange.

Dark Shikari

16th September 2008, 07:02

Who ever said anything about putting I-frames on fades? You misunderstood me, I'm talking about scene changes not the fades themselves. Once a fade-out/fade-in completes x264 doesn't add an I-frame for the scenechange.This is also a bad idea; its a complete waste of bits. Such a high-quality frame should be a P-frame, not an I-frame.

If you mean a higher quality frame to improve the quality of a very static scene after the fade has completed, RDRC (http://pastebin.com/f777a71fc) will do exactly that (in fact, it will do even better than your naive approach; it'll improve quality on a sliding scale towards the end of the fade, so you never see a sudden jump in quality). The macroblock tree patch that failed as a GSOC project would do even better (and without the massive speed hit of RDRC), but unfortunately nobody has volunteered to do the patch since the student bailed, despite it being probably the easiest of the four GSOC projects.

cyberbeing

16th September 2008, 07:13

That is what I was talking about. A slowly moving static image fading out while another slowly moving static image is fading in. When I encoded it with r938 last month I had a drop in quality for these areas after the fade until keyint got triggered. When I manually added a I-frame quality improved.

What is RDRC? A new patch? It sounds useful but I don't know how to interpret the diff you posted.

Dark Shikari

16th September 2008, 07:33

What is RDRC? A new patch? It sounds useful but I don't know how to interpret the diff you posted.New? No, its very very very old, and very bruteforce. I keep a version updated so as to work on recent versions of x264 (original is from >400 revisions ago).

All it does is for every single frame, encode X future frames depending on that frame for a whole bunch of frame quantizers in order to choose the optimal frame quantizer based on how much a frame's quality affects future frame quality.

Audionut

16th September 2008, 07:41

That sounds good. And slow.

Dark Shikari

16th September 2008, 07:52

That sounds good. And slow.It isn't as slow as you'd think, because it does the lookahead with ultra-fast encoding settings and skips bitstream coding, too.

If you're already a nut using ref=16, me=tesa, and similar, its not actually much slower.

But yes, its totally impractical, which is why we need MBtree.

cyberbeing

16th September 2008, 07:57

Would you be willing to compile a current build with the patch or should I field it for someone else in another thread? Compiling is something I'm not set-up to do.

And from the sound of it, you don't plan to pick up and develop the "macroblock tree" patch and unless someone suddenly volunteers it could stay undeveloped for quite a long time. Correct?

Audionut

16th September 2008, 07:58

which is why we need MBtree.

What's the difficulty level and care factor for that.

And also, weighted P-frame prediction.

Dark Shikari

16th September 2008, 07:59

And from the sound of it, you don't plan to pick up and develop the "macroblock tree" patch and unless someone suddenly volunteers it could stay undeveloped for quite a long time. Correct?I may do it at some point if I get pissed off enough at it not being done; that's what happened to the project from this summer which I considered to be most important, the B-frame decision project, for which the student literally disappeared on the first week (don't worry, the other two projects were successful, they weren't all this pathetic). I felt this was a bit insulting, as it was my pet project, so to speak. So I spent a week in August and just did the damn thing to get it over with.

I've already received a few volunteers to do the MB tree project, but I haven't actually seen anyone start working on it... :p
What's the difficulty level and care factor for that.

And also, weighted P-frame prediction.MB tree is not that difficult--if you're familiar with graph traversal algorithms it should not be too much of a challenge. Basically its just making a tree of how much "data" each macroblock references from every other and using that tree measure how much each macroblock is referenced in the future, and weight quality based on that number.

Weighted P-frame prediction already has a patch, but there's two catches. First, if I recall correctly, the patch is over 800 revisions old. Second, when it was originally made it didn't help significantly, so clearly it needed a better algorithm.

gizzin

16th September 2008, 08:39

I think I ran in to the same problem as this guy maybe a cut a clip, and upload it. I wouldn't consider it a fade in though.

foxyshadis

16th September 2008, 09:58

I very seldom encode with what would be considered high quants where blocking can become a major issue, so that is not a problem for me. I have done plenty of tests of my own and I agree with you that x264 does have a good in-loop deblocker, but it still can blur detail if you're not careful even at relatively high bitrates. I'm of a bit different opinion where I usually prefer razor sharp and slightly blocky (no horrid macroblocking please) over soft and slightly blurry. If it becomes too blocky, I aim for lower quants and throw more bitrate at it considering I encode for archiving purposes.

What I don't understand is, why not use a very low deblocking to catch the few blocks that inevitably do slip through? Something like -4:-4 or less. Performance and blurring shouldn't be an issue, because the whole deblocking step will be skipped for almost all blocks anyway. Unless you restrict the encoder to a very low qpmax, it's just there to lessen any visual problems wherever quants jump up, without having to re-encode with a much higher bitrate. It may only save you that trouble on 1 in 3 or 5 of what you now re-encode, would that be worth it?

Ranguvar

16th September 2008, 11:11

Desync artifacts don't necessarily look like blocks.

source (http://img140.imageshack.us/img140/7281/sourcefh5.png)
encode with deblock (http://img247.imageshack.us/img247/7414/deblockyt4.png)
encode without deblock (http://img247.imageshack.us/img247/2972/nodeblockhz0.png)
encode with deblock, decode without (http://img213.imageshack.us/img213/6914/desyncxm7.png)

What anime is that, please?
</offtopic>

Dark Shikari

16th September 2008, 11:12

What anime is that, please?
</offtopic>Ghost in the Shell 2: Innocence

(I'm also pretty sure that somewhere along the line it crossed the threshold of "this contains too high a percentage of CGI to be an anime anymore".)

cyberbeing

16th September 2008, 19:14

What I don't understand is, why not use a very low deblocking to catch the few blocks that inevitably do slip through? Something like -4:-4 or less. Performance and blurring shouldn't be an issue, because the whole deblocking step will be skipped for almost all blocks anyway. Unless you restrict the encoder to a very low qpmax, it's just there to lessen any visual problems wherever quants jump up, without having to re-encode with a much higher bitrate. It may only save you that trouble on 1 in 3 or 5 of what you now re-encode, would that be worth it?

I guess I should have made this clear earlier, I don't usually go below -3:-3 on the in-loop.

Ranguvar

16th September 2008, 21:57

The proposal is to use a low value like that WITH decoder deblocking. Even the insane -6:-6 with decoder deblocking will look better than that blockfest of a screenshot.

Dr.D

17th September 2008, 01:39

The problem is solved!

Details:
My point was that the codec lowers bitrate too much in fades. For example, size of my encoded test clip was 280K, what is way too small for a good quality. With or without deblocking. Deblocking, with all respect, can't do a magic. Ranguvar rang_x264_r0965-2 bulid has been used for testing.
Then I've downloaded skystrife x264.973.modified build and ... the size magically increased to [/B]730K[/B]! It gave a chance, I played the file and, no surprise, the quality was fine!

Then I've compared the sizes and qualities of different builds. For fair comparison I've used default settings with --crf 25.0:

Official....720.....519,548...bad
Ranguvar.950.....653,958...average
Ranguvar.965.....653,932...average
Ranguvar.965-2..324,079...disaster
Ranguvar.977.....810,358...good

Conclusion (based on 1 sample though, I'm going to make more tests): quality of fades encoding was slightly improved in the last 2 years (except one build what I was "lucky" to get for the initial testing :)), improved a lot in the newest 977 build.

The newest Ranguvar.977 build works fine with the rest of my clip, I'm very impressed, my big thanks to the developers! :thanks:

Question: what can cause the disaster in the Ranguvar.965-2 build and the good improvement in the Ranguvar.977 one?

Ranguvar

17th September 2008, 01:41

My 965-2 build used VAQ2mod, among many other odd patches. Output would be drastically different. Please read the notes I post :) Default settings are not default across builds/revisions.
As for the recent build, CRF was adjusted to give more bitrate at a given CRF, due to offset caused by AQ.
Glad to see you sorted the problem!

Dark Shikari

17th September 2008, 02:30

Dr.D

17th September 2008, 05:02

Official....720.....519,548...bad
Ranguvar.950.....653,958...average
Ranguvar.965.....653,932...average
Ranguvar.965-2..324,079...disaster
Ranguvar.977.....810,358...good
Surprise surprise, totally useless test is totally useless! :rolleyes:

If your eyes exactly follow the bitrate curve, that is "high bitrate looks better than low bitrate," your test says absolutely nothing because we already know that.

Always do comparisons at the same bitrate, anything else is dishonest and pointless.
Useless and pointless for whom?

Dark Shikari, I'm not looking for the best (or satisfactory) encoder for fades only. It's not a my business to encode fades. Instead of that I'm looking for satisfactory encoder for whole movies with (possibly) fades. Also I always use crf mode and decided to use some value, let say 25.0.
For people, like me, my test is not useless. It says - don't use builds which destroy fades. For example, Ranguvar.965-2 bulid is good for the rest of movie, but I have to refuse it.

Your suggestion - "Always do comparisons at the same bitrate" - is definetely correct in common, but useless for this thread. Let say the Ranguvar.965-2 bulid is the best for my faded testing clip for bitrate 3000. And what? What conclusion can I make? This build the best for whole movie for crf 25? Why?
I can only make conclusion, that the build is best for fades only. But, again, it's useless for me.

Audionut

17th September 2008, 05:49

Sagekilla

17th September 2008, 05:55

Dark Shikari

17th September 2008, 06:46

Useless and pointless for whom?

Dark Shikari, I'm not looking for the best (or satisfactory) encoder for fades only. It's not a my business to encode fades. Instead of that I'm looking for satisfactory encoder for whole movies with (possibly) fades. Also I always use crf mode and decided to use some value, let say 25.0.
For people, like me, my test is not useless. It says - don't use builds which destroy fades. For example, Ranguvar.965-2 bulid is good for the rest of movie, but I have to refuse it.

Your suggestion - "Always do comparisons at the same bitrate" - is definetely correct in common, but useless for this thread. Let say the Ranguvar.965-2 bulid is the best for my faded testing clip for bitrate 3000. And what? What conclusion can I make? This build the best for whole movie for crf 25? Why?
I can only make conclusion, that the build is best for fades only. But, again, it's useless for me.I'm not 100% sure what you're trying to say--are you saying you used 2pass mode for the whole clip, and the numbers you're posting are the bitrate allocated to the fade?

Dr.D

17th September 2008, 06:52

All those builds are different in more ways then one. ie: patches applied, code changes etc.

You didn't seem to comprehend something like inloop deblocking, so i gather that you are unaware of the difference that the different patches etc are making to the code.
Patches and code changes, by their design, change the output.
:D. Thank you for a few minutes of good laughing, Mr. Smart Guy!

Comparing crf of different builds is completely useless.
It is so pointless, it's like having a conversation about physics with a blonde.
Oh, that's even better!
Ok, my Lord, please teach me.
What should I do to decide - choose a new build or keep the old one. Step by step.
Keep in mind that I use crf only, the value is 25.0 and I'm not going to change the value without a serious reason.

Dr.D

17th September 2008, 07:02

Dr.D

17th September 2008, 07:57

@Dr. D: The reason why your test was useless was because of: (1) The varying bitrate (2) the different builds and (3) the use of CRF instead of bitrate.

Try encoding using 2-pass bitrate using the same bitrate for all encodes to ensure each build gets a fair number of bits to allocate to the clips, then you'll see which one is doing well on fades.
Well, it makes sense, but only if 1) I will test whole clip (or a combined - part with fade-in and large non-faded part); 2) use regular bitrate based encodes.
It makes some sense if I test whole or combined clip and use regular crf encodes.
It doesn't make any sense if I test faded piece only.

Please keep in mind that:
1) I use crf only.
2) Testing clip is fade-in only piece.

Dr.D

17th September 2008, 08:41

I'm not 100% sure what you're trying to say--are you saying you used 2pass mode for the whole clip, and the numbers you're posting are the bitrate allocated to the fade?
I use crf always.
The numbers are filesizes for encoded cut-off fade-in piece mentioned in this post: http://forum.doom9.org/showthread.php?p=1183889#post1183889.

I really can't understand what is so complicated or specific in my logic.
I thought is very simple, so may be my explanations were bad.
Ok, another try.

I've encoded my clip (30 minutes) with Ranguvar.965-2 build and --crf 25. Few fade-ins were terrible (less than 1 minute total), the rest of the clip was fine.
I cut off one fade-in and used for testing.
Encoded (with crf 25) cut-off had very low bitrate, it was a clue.
I've played with parameters - just minor improvement.
This allowed me to say that fades handling is bad.
Then I've updated to Ranguvar.977, encoded (with crf 25) cut-off was fine.
As usually, before choosing another buils, I've encoded the few my testing clips - was fine (quality not worse, size not bigger).
This allowed me to say that in this build fades handling is good.
Then for curiosity I've tested another builds (with crf 25) and posted the results.

What's wrong here?
Why I need same-bitrate testing for cut-off fade only piece?
Why I need same-bitrate testing testing at al, if (for particular type of sources)
- I used to use crf 25
- I'm using crf 25 now
- I will use crf and I will use the value 25 until very serious reason to change it
- my children will use crf 25
- my grandchildren will use crf 25?

Dark Shikari

17th September 2008, 08:43

I use crf always.
The numbers are filesizes for encoded cut-off fade-in piece mentioned in this post: http://forum.doom9.org/showthread.php?p=1183889#post1183889.

I really can't understand what is so complicated or specific in my logic.
I thought is very simple, so may be my explanations were bad.
Ok, another try.

I've encoded my clip (30 minutes) with Ranguvar.965-2 build and --crf 25. Few fade-ins were terrible (less than 1 minute total), the rest of the clip was fine.
I cut off one fade-in and used for testing.
Encoded (with crf 25) cut-off had very low bitrate, it was a clue.
I've played with parameters - just minor improvement.
This allowed me to say that fades handling is bad.
Then I've updated to Ranguvar.977, encoded (with crf 25) cut-off was fine.
As usually, before choosing another buils, I've encoded the few my testing clips - was fine (quality not worse, size not bigger).
This allowed me to say that in this build fades handling is good.
Then for curiosity I've tested another builds (with crf 25) and posted the results.

What's wrong here?
Why I need same-bitrate testing for cut-off fade only piece?
Why I need same-bitrate testing testing at al, if (for particular type of sources)
- I used to use crf 25
- I'm using crf 25 now
- I will use crf and I will use the value 25 until very serious reason to change it
- my children will use crf 25
- my grandchildren will use crf 25?So if I made a patch that made it so that all CRF bitrates doubled completely arbitrarily (and quality doubled along with it), you'd say that patch was the greatest thing ever? :rolleyes:

You cannot compare two encodes at different bitrates. End of story.

gizzin

17th September 2008, 08:56

Whats the recommended tool for cutting x264 in a mp4 container?

cyberbeing

17th September 2008, 10:29

So if I made a patch that made it so that all CRF bitrates doubled completely arbitrarily (and quality doubled along with it), you'd say that patch was the greatest thing ever? :rolleyes:

You cannot compare two encodes at different bitrates. End of story.

I think you're missing his point. CRF is based on rate control of some sort. The rate control decides where and how many bits to allocate to each frame to meet a certain metric. Now lets say you have two builds each with a different variation of CRF rate control. With the two builds you encode a clip at CRF25. Lets for arguments sake say they come out at exactly the same size and avg bitrate (not likely but bare with me). One of the two clips had average quality throughout. Lets call this "average quality", "CRF25 quality". The other of the two had above average quality (CRF20 quality) in 95% of the video but below average (CRF30 quality) in 5% (fades). Only average "CRF25 quality" was needed so the one with both above average "CRF20 quality" and below average "CRF30 quality" gets junked. Basically he has a certain quality expectations of CRF25. If one build's CRF rate control gives more consistent quality then another he views it as better.

If you theoretically made a patch which doubled CRF bitrate that doesn't solve his dilemma. All the means is that in order for him to meet his quality expectations he would end up using a higher CRF because CRF25 all of a sudden started producing "CRF10 quality" and that's not what he wants. If the build isn't keeping what he sees as the quality expected from a given CRF even in areas where a disproportionate amount of bitrate may be required to maintain that quality he sees it as a failure. In other words he is unhappy when encoding in CRF mode and x264 decides to starve bits from complex areas (lets say because 4x the average bitrate is needed to maintain quality and it only allocates 2x) and in turn quality drops.

Now this opinion is a bit flawed in many respects, but not totally unwarranted if you look at it from his point of view. Bitrate doesn't matter to him, only the average "CRF25 quality" he expects maintained throughout the whole clip not just most of the clip.

Dark Shikari

17th September 2008, 10:40

I think you're missing his point. CRF is based on rate control of some sort.I don't think you're one to be lecturing me on how CRF works, given that I wrote a large portion of it.

CRF is scaled in a completely arbitrary manner to be somewhat similar to QP in terms of filesize. It has at least 2 or three completely arbitrary constants thrown in for this purpose.

Also, the entire point of constant ratefactor is the entire video has the exact same ratefactor, the exact same CRF. If you specify CRF25, the video will be all CRF25. There is no part which is magically 20 or 30.

cyberbeing

17th September 2008, 11:07

I don't think you're one to be lecturing me on how CRF works, given that I wrote a large portion of it.
Thank you for taking my quote way out of context. :rolleyes: Those two sentences were completely unrelated to each other (I should have made a new paragraph).

Also I don't know how you got the idea I was trying to lecture you. I was trying my best to talk in theoretical terms and trying to be as vague as possible. I was in no way claiming that was how CRF works. It was just written that way to try to explain the logic from what I see as his point-of-view, not to be accurate.

That's also why I said the opinion (my above post is not my own opinion but what I view his as) was flawed in many respects, but I think that's how he is viewing CRF as more of a subjective quality metric and not how it actually works.

Audionut

17th September 2008, 11:08

only the average "CRF25 quality" he expects maintained throughout the whole clip not just most of the clip.

There is no part which is magically 20 or 30.

So just to expand on that, if a scene requires qp of 20 to look decent, with encoding at crf 25, you're shit out of luck.

Doesn't matter how many times you guys want to explain your reasoning, the result is still the same.

You cannot compare two encodes at different bitrates. End of story.

Dark Shikari

17th September 2008, 11:12

So just to expand on that, if a scene requires qp of 20 to look decent, with encoding at crf 25, you're shit out of luck.But QP isn't CRF.

CRF varies the QP on a block-by-block and scene-by-scene basis to match a certain metric.

cyberbeing

17th September 2008, 11:26

So, trying to be solution oriented, if Dr. D wants to avoid subjective quality drops in parts of a clip but doesn't in the process want to add extra bitrate to parts he thinks look fine, what is the best way for him to achieve that?

17th September 2008, 11:49

Zones, if manual adjustment is acceptable.

But I doubt his problems were really in rate control but the two things: missing decoder deblocking and the differences in CRF value mapping between builds.

I would suggest this test to Dr D., if he's interested in a valid experiment.
1. Select a clip with a fade and a minute or two of normal content.
2. Make a CRF=25 encode with a build that you claim encodes fades poorly (Ranguvar.965-2). Check the resulting bitrate or file size.
3. Make a CRF=X encode with a "good" build. Select X so that the resulting bitrate is about the same as in the first encode (you'll need to make a few encodes to find the correct X).
4. Compare the encodes visually.

Quark.Fusion

17th September 2008, 12:12

then i've compared the sizes and qualities of different builds. For fair comparison i've used default settings with --crf 25.0:

Official....720.....519,548...bad
ranguvar.950.....653,958...average
ranguvar.965.....653,932...average
ranguvar.965-2..324,079...disaster
ranguvar.977.....810,358...good

surprise surprise, totally useless test is totally useless! :rolleyes:

If your eyes exactly follow the bitrate curve, that is "high bitrate looks better than low bitrate," your test says absolutely nothing because we already know that.

Always do comparisons at the same bitrate, anything else is dishonest and pointless.
I think Dr. D wanted to say that metric used for CRF in ranguvar.965-2 failed to provide requested quality and ranguvar.977 do the job. Now it needs to see what patches was applied and what is difference in CRF handling in those builds.
When you request CRF control from x264 you want consistent quality across the clip.

cyberbeing

17th September 2008, 12:34

I think Dr. D wanted to say that metric used for CRF in ranguvar.965-2 failed to provide requested quality and ranguvar.977 do the job. Now it needs to see what patches was applied and what is difference in CRF handling in those builds.

This is what I was trying to explain in my previous post a couple up (#111 (http://forum.doom9.org/showpost.php?p=1184751&postcount=111)) in a rather roundabout manner.

When you request CRF control from x264 you want consistent quality across the clip.After some of Dark Shikari's posts in this thread I'm starting to have doubts that this is how CRF is supposed to work, unless he has been misunderstanding the questions. From his posts it sounds like, from a technical standpoint, CRF's goal isn't to produce consistent quality.

Quark.Fusion

17th September 2008, 12:54

Sharc

17th September 2008, 13:00

cyberbeing

17th September 2008, 13:13

I'm also starting to suspect that CRF is supposed to give consistent bitrate instead of quality after some posts in this forum… But if CRF is supposed to replace QP then first is wrong, else last is wrong…

Well how does x264 define consistent quality considering it is objective from person to person? I'm assuming this is impossible 100% of the time without some sort of AI or intelligent adaptive algorithms that are able to accurately judge quality for every possible situation.

It also probably goes back to a that a given QP does not directly represent quality over a number of frames and since it sounds like CRF is in some way based on QP it suffers from the same problem.

Quark.Fusion

17th September 2008, 13:13

1. Clip can contains only fades —*think about slideshow of photos.
2. In this case you compare quality-to-bitrate, but you may want to compare quality only. (and it will be oranges with oranges, not apples with apples)
3. If CRF means quality-based encode and you want to compare how it do this job, then you shouldn't adjust it to size. Equal file sizes provides 2-pass or CRF is meant to replace 2-pass with trial-and-error?

Quark.Fusion

17th September 2008, 13:19

well how do you define consistent quality considering it is objective from person to person? I'm assuming this is impossible 100% of the time without some sort of ai or intelligent adaptive algorithms that are able to accurately judge quality for every possible situation.

It also probably goes back to a that a given qp does not directly represent quality over a number of frames and since it sounds like crf is in some way based on qp it suffers from the same problem.
1) I'm define consisnet quality as when clip don't breaks to blocks suddenly accross the clip and when you encode two diferent clips quality drop from source is same, not that in first case you get only blocks and in second transparent video.

2) Yes, it what I'm understand as psy-optimisation purpose.

Sharktooth

17th September 2008, 13:22

the CRF value is not a quality factor...
it has some relations to a certain level of quality but it changes between revisions, builds and options used for encoding.

Inventive Software

17th September 2008, 13:23

CRF uses a range specified between QPmin and QPmax to achieve xx.x ratefactor, that is nearly equivalent to QPxx in file size. Nearly being the operative word. Do a comparison between QP18 and CRF18 or QP20 and CRF20 and see which you prefer. I can't do it, cos I'm at work, but note the file sizes. They should be nearly equal.

17th September 2008, 13:38

1. Clip can contains only fades —*think about slideshow of photos.
But in this case the question was whether rate control fails in some builds so that fades don't get enough bits compared to the rest of the encode. You can't test that without including both fades and normal content in the same video.

2. In this case you compare quality-to-bitrate, but you may want to compare quality only. (and it will be oranges with oranges, not apples with apples)
To compare two different encodes, either the quality or the bitrate needs to be anchored. If both vary, conclusions can only be made if one of the encodes has both better quality and lower average bitrate. In practice, it is easiest to anchor the average bitrate and compare visual quality.

3. If CRF means quality-based encode and you want to compare how it do this job, then you shouldn't adjust it to size. Equal file sizes provides 2-pass or CRF is meant to replace 2-pass with trial-and-error?
No, just pick a CRF that pleases you and use that for (almost) all sources. But as Sharktooth said, that CRF value may need to be adjusted when you update x264 or change the encoding options. Equal file sizes and trial-and-error are needed for comparisons between CRF encodes.

Sharktooth

17th September 2008, 13:46

also ratecontrol didnt fail. the problem was experimental patches used and the flawed testing method.

Dr.D

17th September 2008, 15:21

You cannot compare two encodes at different bitrates. End of story.
Ok, simple question for you.
Let say I have 2 builds, build1 and build2 and encode same clip with crf 25. The clip is 30 min long and has 1 min faded part.
Build1 gives bitrate 200 and awful quality for the faded part and bitrate 2000 for the rest.
Build2 gives bitrate 900 and good quality for the faded part and bitrate 2000 for the rest.
But Build1 encodes faded part better with 2-pass bitrate 900 encoding (as you suggest to test).

Which build handles fades better?

Sharktooth

17th September 2008, 15:23

such a build doesnt exist... unless you completely screw x264 by adding experimental patches...

poisondeathray

17th September 2008, 15:39

This was already suggested back on page 1, but I would encode with zones (e.g. crf20 for the 1min fade section, crf25 for the rest) - it's setup so it's very easy to do in MeGUI

Dr.D - Are you planning to run these tests on every patch and revision? or find a good revision to stick with and never upgrade?

Audionut

17th September 2008, 16:16

I'm also starting to suspect that CRF is supposed to give consistent bitrate instead of quality after some posts in this forum… But if CRF is supposed to replace QP then first is wrong, else last is wrong…

It was often thought that QP was constant quality.
However, 1 frame might look "good" with a quantizer of 18 but another frame might look, "not so good".

CRF mode was designed to alleviate that to a certain extent by varying the quantizer of frames in an encode while still maintaining an average of the quantizer set.

So as pointed out by Inventive above, if you do encode 1 at QP 20 and encode 2 at CRF 20, they both should have around the same bitrate. (although, this could have changed with the recent updates)

However, while encode 1 is blind quantizer 20 throughout, encode 2 uses a little bit of look ahead and variation to produce a theoretically more pleasing result for the viewer.

At least, that's how I recall it.

Dr.D

17th September 2008, 16:48

So just to expand on that, if a scene requires qp of 20 to look decent, with encoding at crf 25, you're shit out of luck.

Doesn't matter how many times you guys want to explain your reasoning, the result is still the same.
It's not the case.
The case is - at crf 25 fade scene requires qp 30 to look decent, encoder (some builds) give it qp 40.
Doesn't matter how many times I tried to explain that, you and some other do not understand.

Sharktooth

17th September 2008, 17:00

no. you do not understand... dont use experimental patched builds for final "production" or for bugreports.

Dr.D

17th September 2008, 17:08

Zones, if manual adjustment is acceptable.
No, thank you.
I don't need to spend hours to tweak parameters for each clip.
I'm expecting a constant quality for whole clip and for each clip.
Situation, when fades looks like shit and the rest of the clip looks fine is not a constant quality. In my understanding, at least.
So, I try to find a combination of parameters or a proper build. If I can't, I'm looking for another encoder. Plane and simple.

I would suggest this test to Dr D., if he's interested in a valid experiment.
1. Select a clip with a fade and a minute or two of normal content.
2. Make a CRF=25 encode with a build that you claim encodes fades poorly (Ranguvar.965-2). Check the resulting bitrate or file size.
3. Make a CRF=X encode with a "good" build. Select X so that the resulting bitrate is about the same as in the first encode (you'll need to make a few encodes to find the correct X).
4. Compare the encodes visually.
This definetely makes sense.
But, again, not for me.
Let say, I've found that X=22 and the quality is better. And what? Anyway I'll encode at CRF=25, for this CRF the quality could be worse than with the "bad" build.

Quark.Fusion

17th September 2008, 17:12

I was extreme to some extend in my previous posts to emphasize my point. I'm just don't understand why when developers introduce new features to CRF they want to make it same bitrate as before instead of quality or avg QP at least or don't care at all.

Quark.Fusion

17th September 2008, 17:15

So, I try to find a combination of parameters or a proper build. If I can't, I'm looking for another encoder. Plane and simple.You should try recent official GIT build at first.

Audionut

17th September 2008, 17:17

I'm just don't understand

The changes make the algorithm theoretically better "overall".

If people don't have the ability to adjust the crf value themselves to maintain the bitrate they want, :stupid: well, developer can't help that. ;)

Quark.Fusion

17th September 2008, 17:21

The changes make the algorithm theoretically better "overall".

If people don't have the ability to adjust the crf value themselves to maintain the bitrate they want, well, developer can't help that.

I think you misunderstood me. I'm perfectly happy if algorithm is better overall. And my point if you want same bitrate — use two pass or CRF2size program.

Sharktooth

17th September 2008, 17:33

exactly. so i cant understand all those complaints. CRF is not about bitrate... if you want a specific bitrate use a bitrate RC mode...

17th September 2008, 17:42

I'm expecting a constant quality for whole clip and for each clip.
Situation, when fades looks like shit and the rest of the clip looks fine is not a constant quality. In my understanding, at least.
Indeed, but you haven't yet shown us anything that supports such observations.

Let say, I've found that X=22 and the quality is better. And what?
Then you have proved that the build (or the encoding options) produce better, or in this case, more constant quality. However, we would need to see the encodes to believe your results. ;)

Dr.D

17th September 2008, 17:53

For a valid and fair comparison of crf based encodes I would suggest
1. The test clip should include not only the fade scene, but a pre-and post "normal" scene of a few seconds each.
2. Comparison has to made on equal file size = equal average bitrate in order to compare apples with apples.
3. Assuming crf mode, this means that for different patches and encoding parameters one needs to adjust (trial-and-error !) the CRF such as to obtain equal file sizes of the encoded test clip. Keeping crf at a fixed value (like crf 25) producing different filesizes is pointless.
Well, this is good point, but I still don't agree.
CRF values should mean something. Like, for example - CRF 18 (with best another options) gives transparent quality. If this value jumps from build to build - I think it's not normal.

Here is my logic (simlified) to decide - move to a new build or not.
I encode a few test clips with a new build and same parameters (or, if a new feature-parameter has been introduced, I do both - with and without this parameter).
If filesizes are bigger - I stop testing and keep the old build.
If smaller, I make a visual comparison.
If noticeable worse - stay with old, otherwise - move to the new build.

Audionut

17th September 2008, 18:07

CRF values should mean something. Like, for example - CRF 18 (with best another options) gives transparent quality.

Why should they mean something?

Read the forum rules. There is no best, and what is transparent to you, could be something totally different to another.

17th September 2008, 18:18

Well, this is good point, but I still don't agree.
CRF values should mean something. Like, for example - CRF 18 (with best another options) gives transparent quality. If this value jumps from build to build - I think it's not normal.
It doesn't usually jump significantly, but there have been some patches and changes that have caused larger variations. Like the VAQ2mod patch seems to do in Ranguvar's build that you used, and the AQ changes caused in r968. Have you read this thread: http://forum.doom9.org/showthread.php?t=141124

Still, making quality comparisons at the same CRF value is not a valid approach unless the average bitrates match, as has been repeated over and over again.

Dr.D

17th September 2008, 18:53

This was already suggested back on page 1, but I would encode with zones (e.g. crf20 for the 1min fade section, crf25 for the rest) - it's setup so it's very easy to do in MeGUI
Thanks, but it's a very last suggestion what I would to try. You see, I need "find and forget" technology and ready to spend extra time to find a good one. I don't need to spend time with each clip then.

Dr.D - Are you planning to run these tests on every patch and revision? or find a good revision to stick with and never upgrade?
Well, now I'm happy with the rang_977 build.
My strategy for future: once per month (or earlier, if I'll found some serious issue; or some WOW feature will be introduced) test the newest build as described here: http://forum.doom9.org/showpost.php?p=1184980&postcount=142

Dr.D

17th September 2008, 19:09

such a build doesnt exist... unless you completely screw x264 by adding experimental patches...
It does.
Not exact numbers, of course, but the same idea:

build.....faded piece..."normal" piece...ratio
Off 720......519,548....5,437,664.......10.5
Rang965-2..324,079....4,299,049.......13.2
Rang977.....822,764....4,106,181........5.0

This numbers tell me, that Rang977 build handles fades better than Off 720 and much better than Rang965-2.

Dr.D

17th September 2008, 19:20

I'm expecting a constant quality for whole clip and for each clip.
Situation, when fades looks like shit and the rest of the clip looks fine is not a constant quality. In my understanding, at least.

Indeed, but you haven't yet shown us anything that supports such observations.

Sure I can. But why you don't believe me? Otherwise, why I've started this thread? To blame developers? No, I don't need that. I need good encodes only.

Dr.D

17th September 2008, 19:36

the CRF value is not a quality factor...
it has some relations to a certain level of quality but it changes between revisions, builds and options used for encoding.
What I can't understand, why the developers don't keep it same, instead of this force millions user to spend a lot of time to find proper CRF value again and again...

LoRd_MuldeR

17th September 2008, 19:42

What I can't understand, why the developers don't keep it same, instead of this force millions user to spend a lot of time to find proper CRF value again and again...

Because when the behavior of the encoder changes, the meaning of the CRF value changes inherently. And if you want the encoder to improve (which I think you want ^^), the behavior must change. Of course the CRF value could be multiplied with some "magic" factor internally, so the CRF values specified by the user doesn't change it's meaning over different revisions. But it would be really hard to find that "magic" factor and it even might vary between different sources. So we simply have to life with changing CRF behavior. The CRF mode does an awesome job anyway...

Sharktooth

17th September 2008, 19:59

It does.
Not exact numbers, of course, but the same idea:

build.....faded piece..."normal" piece...ratio
Off 720......519,548....5,437,664.......10.5
Rang965-2..324,079....4,299,049.......13.2
Rang977.....822,764....4,106,181........5.0

This numbers tell me, that Rang977 build handles fades better than Off 720 and much better than Rang965-2.
numbers tell me you're a noob and as such you should not play with experimental builds that do not represent the x264 true encoding efficiency and quality potential.
that said you have no rights to criticize the encoder and jump to wrong conclusions if you dont even know the builds you tested have experimental patches and some of them have not even been wrote by the x264 devs.

cyberbeing

17th September 2008, 20:14

...you have no rights to criticize the encoder if you dont even know the builds you tested have experimental stuff and some of it has not been made by the x264 devs.

You are making it sound like it's utterly impossible for x264 to have a problem that is not caused by experimental patches.

Dr D. can you please just make some encodes with vanilla builds so experimental patches can be ruled out or confirmed as the cause? This thread is really going around in circles and unless it can be proven one way or another that some vanilla x264 builds have an issue with CRF it's really pointless in debating it any further.

Sharktooth

17th September 2008, 20:16

noob number 2... FIRST use the vanilla builds (with correct settings), THEN talk...
this thread is 8 pages of useless discussion since modified builds and unoptimal settings have been used to do the comparisons and tests...

cyberbeing

17th September 2008, 20:19

noob number 2... FIRST use the vanilla builds, THEN talk...
this thread is 8 pages of useless discussion since modified builds and unoptimal settings have been used to do the comparisons and tests...

noob number 3... If you read my whole post I just said that.

Sharktooth

17th September 2008, 20:21

right... i dont know what i read. i interpreted your post in a completely different way.

Dr.D

17th September 2008, 21:20

Sharktooth, your behavior becames funny.

Here is my original question:

Ok, simple question for you.
Let say I have 2 builds, build1 and build2 and encode same clip with crf 25. The clip is 30 min long and has 1 min faded part.
Build1 gives bitrate 200 and awful quality for the faded part and bitrate 2000 for the rest.
Build2 gives bitrate 900 and good quality for the faded part and bitrate 2000 for the rest.
But Build1 encodes faded part better with 2-pass bitrate 900 encoding (as you suggest to test).

Which build handles fades better?

You didn't answer, instead you wrote:

such a build doesnt exist... unless you completely screw x264 by adding experimental patches...

First of all, it shows that you are not in good relationship with logic (I guess it's obvious for everybody, but I can provide an explanation personally for you, if you need).

Second, "this build" (more exactly, a pair of builds) exists. I gave the example for you:

It does.
Not exact numbers, of course, but the same idea:

build.....faded piece..."normal" piece...ratio
Off 720......519,548....5,437,664.......10.5
Rang965-2..324,079....4,299,049.......13.2
Rang977.....822,764....4,106,181........5.0

Even if we don't count Rang965-2 (as "completely screwed x264 by adding experimental patches"), the pair is Off 720 and Rang977.

So, you said a pair with described behavior doesn't exist, I gave you the pair. So, you are wrong. Plain and simple.

Instead of confirming that, you wrote:

numbers tell me you're a noob and as such you should not play with experimental builds that do not represent the x264 true encoding efficiency and quality potential.
that said you have no rights to criticize the encoder and jump to wrong conclusions if you dont even know the builds you tested have experimental patches and some of them have not even been wrote by the x264 devs.
Well, seems to be you finally realized that you are wrong, became nervous (it's the sign) and used a last argument in a discussion, when no another left - blame somebody as a noob.

BTW, aboot noobs. Let say I'm a noob. Nothing wrong with that, I can learn. You - can't, what is much worse. It's impossible to learn without ability to admit own mistakes.

Dr.D

17th September 2008, 21:51

You are making it sound like it's utterly impossible for x264 to have a problem that is not caused by experimental patches.

Dr D. can you please just make some encodes with vanilla builds so experimental patches can be ruled out or confirmed as the cause? This thread is really going around in circles and unless it can be proven one way or another that some vanilla x264 builds have an issue with CRF it's really pointless in debating it any further.
See my post http://forum.doom9.org/showthread.php?p=1185024#post1185024.
Can I call 720 build from http://mirror01.x264.nl/x264 as vanilla build? It has the issue.

What exactly are you doubt about? That x264 was not good with fades? If you don't believe me (actually you don't need to believe, you can know if you read my posts and use logic), believe the developer:
http://forum.doom9.org/showpost.php?p=1039573&postcount=107

Dark Shikari

17th September 2008, 22:00

See my post http://forum.doom9.org/showthread.php?p=1185024#post1185024.
Can I call 720 build from http://mirror01.x264.nl/x264 as vanilla build? It has the issue.

What exactly are you doubt about? That x264 was not good with fades? If you don't believe me (actually you don't need to believe, you can know if you read my posts and use logic), believe the developer:
http://forum.doom9.org/showpost.php?p=1039573&postcount=107
Its been over a year now, and --b-adapt 2 --weightb will result in correct detection of fades.

Its rather bad form to quote posts from so long ago, especially since:

1. x264's development is ongoing
2. When I said that, I wasn't an x264 developer ;)

cyberbeing

17th September 2008, 22:03

Sharktooth

18th September 2008, 01:14

Sharktooth, your behavior becames funny.

Here is my original question:

You didn't answer, instead you wrote:

First of all, it shows that you are not in good relationship with logic (I guess it's obvious for everybody, but I can provide an explanation personally for you, if you need).

Second, "this build" (more exactly, a pair of builds) exists. I gave the example for you:

Even if we don't count Rang965-2 (as "completely screwed x264 by adding experimental patches"), the pair is Off 720 and Rang977.

So, you said a pair with described behavior doesn't exist, I gave you the pair. So, you are wrong. Plain and simple.

Instead of confirming that, you wrote:

Well, seems to be you finally realized that you are wrong, became nervous (it's the sign) and used a last argument in a discussion, when no another left - blame somebody as a noob.

BTW, aboot noobs. Let say I'm a noob. Nothing wrong with that, I can learn. You - can't, what is much worse. It's impossible to learn without ability to admit own mistakes.
i wont even comment apart the fact you ignore the advices from develpers and other ppl that tell you how to configure the encoder...
you use modified builds with experimental patches...
you fail to see where you're doing wrong comparisons since ppl told you to compare at the same bitrate...
for what concerns me, i have nothing to add except you can stop your crusade against a non existent problems (caused by your misconfiguration and/or the use of old or "modified" builds).
your conclusions and test methods are completely wrong... so you can either:
1 - use vanilla builds and compare with the correct test methodology
2 - shut up
chose one.

Ranguvar

18th September 2008, 02:06

I agree... and my r965-2 build was not "completely screwed", it was just very different from most custom builds (lots of experimental patches, some of which were on by default, such as VAQ2mod). Of course the results will be drastically different. Using the same CLI across builds does NOT guarantee the same settings. Revisions change defaults, as do patches. I make my builds with the expectation that the user will do their homework and check out the patches I used, to see what the changes are. If you aren't willing to do that, use vanilla builds.

Please do your homework next time.

But moving on, I believe I am right in believing what you're trying to do is show that x264 is inefficient with fades, by not using deblocking, as that would "cover up" the problem? This is not _entirely_ unreasonable. However, you must see that between the developers who make x264 and a person who has just started using it, we would believe the former more readily. By all means, if you see a better way to work with fades, submit a patch, or point it out :) But when using sane settings (this even includes low deblocking, like -2:-1, if you like sharpness! But enable deblocking on both ends.), and --b-adapt 2 and --weightb, I see no problem with fades. They are never smudged to me, which would indicate that the deblocker is working overtime, unless you take a very quick fade and grab a frame in the middle, in which case the encoder is right to drop quality there, as it will be unnoticeable.

Use a vanilla build, with the options I mentioned, using encode a slow fade, and show us evidence of the deblocker working overtime (blurry). Make sure to give the source, exact settings used, and another frame with source to show that other frames are not hurt, which means you chose a sane bitrate. Do all this, and I think I can assure you you will be heard.

cyberbeing

18th September 2008, 02:27

...unless you take a very quick fade and grab a frame in the middle, in which case the encoder is right to drop quality there, as it will be unnoticeable.

To the x264 devs, is the above statement by Ranguvar true? Is x264 designed to "drop quality" where "unnoticeable" when using CRF?

Please no hostility. I am just trying to fully understand how x264 works so then I don't have to make inaccurate assumptions.

As an abstract follow-up question, is unnoticeable a valid reason to drop quality? What if what was thought to be unnoticeable was noticeable after all?

Ranguvar

18th September 2008, 02:31

I'm just saying it might, and if so it'd probably be good! Not that it will! I have no clue! I just said that to explain why there may be a quality loss if you have an extremely quick fade, in case it came up :)

The goal of every encoder, after all, is to reduce filesize by compressing where you don't notice it to give more bitrate where you do notice it.

Dr.D

18th September 2008, 02:55

Dr. D, I understand what your trying to say from a logical point of view but you're missing something. What Sharktooth is trying to say is that unless all outside variables are eliminated you can't be drawing conclusions. Even Rang977 has patches applied. Now if the limited patches on that build could cause something like this is up for debate, but that's not the point. The inconsistencies from encoding with "unoptimal settings" as Sharktooth put it could be causing what you're seeing as well.
I afraid I've started to miss you point. Or you misses mine.

See the name of the thread - "Fade-in ugly blocking problem, any progress?". I answered few pages ago and said "the case is closed". The answer is "yes". Good progress in the last builds, except disaster in Rang.965-2, and very good progress in Rang.977.

How is your, probably correct theoretical construction, can help to answer my question?

Which exactly part of my conclusion you disagree?

The only way to truly know is to test with no patches whatsSolid proof is needed not the theoretical proof we have so far. If you want the x264 devs to fix a problem, you need to prove that there is a problem with methods they approve of. For all you know, if you follow what they say to do you may realize that the problem never existed and you were just looking at it the wrong way. On the other hand there may be a surprise that there is in fact a "bug". So far we can't say one way or another. Everybody fighting with each other isn't going to change that.
Again, few pages ago I said "the problem is solved".
So, I don't want the x264 devs to fix a problem - already fixed.
I've stopped any fighting since, I'm only defending.
Situation is crystal clear for me.
If somebody has this problem with fades and follow my advice - he will good chance to win.
If somebody will follow Sharktooth or Dark Shikari advice - test a faded piece with same bitrate - he will good chance to loose. But it will be this person's problem.

cyberbeing

18th September 2008, 02:59

@Dr D. That post was more that if the problem could be reproduced on a vanilla build it would be interesting. Were Sharktooh and Dark Shikari really telling you to use the same bitrate on just faded portion instead of the whole clip? I don't even know any longer but if you're testing rate control decisions it should be the same avg bitrate for the entire video not just the faded parts.

I'm just saying it might, and if so it'd probably be good! Not that it will! I have no clue! I just said that to explain why there may be a quality loss if you have an extremely quick fade, in case it came up :)

I'm wasn't trying to target you Ranguvar. I just felt it was a good opportunity to throw the question out there.

The goal of every encoder, after all, is to reduce filesize by compressing where you don't notice it to give more bitrate where you do notice it.
Every person is different though and may notice quality in different unlikely areas. What suits one person may not suit another. On that note, something that may be interesting would be fully tweakable presets for rate-control to better suit the types of frames where you want quality to be and have the ability to give things different weights of importance (dark areas, light areas, medium areas, detailed areas, flat areas, gradients, fades, fast moving scenes, slow moving scenes etc). I think AQ does some of this in a generalized fashion, but a streamlined and specialized weight based rate control based on different frame characteristics would be interesting. It would give a level of control beyond what we have today.

Ranguvar

18th September 2008, 03:02

I'm wasn't trying to target you Ranguvar. I just felt it was a good opportunity to throw the question out there.
I know :) I wasn't trying to be defensive.

Every person is different though and may notice quality in different unlikely areas. What suits one person may not suit another. On that note, something that may be interesting would be fully tweakable presets for rate-control to better suit the types of frames where you want quality to be and have the ability to give things different weights of importance (dark areas, light areas, medium areas, detailed areas, flat areas, gradients, fades, fast moving scenes, slow moving scenes etc). I think AQ does some of this in a generalized fashion, but a streamlined and specialized weight based rate control based on different frame characteristics would be interesting. It would give a level of control beyond what we have today.
True, but generalizations can, do, and have to be made, based on the average reception different settings get.

cyberbeing

18th September 2008, 03:24

True, but generalizations can, do, and have to be made, based on the average reception different settings get.
Nothing wrong with that, but a fully customizable rate control like I described would still be interesting and have it's uses for excessive setting tweakers in maximizing quality where you want it at a given bitrate.

akupenguin

18th September 2008, 03:25

To the x264 devs, is the above statement by Ranguvar true? Is x264 designed to "drop quality" where "unnoticeable" when using CRF?
Yes. Which is perfectly equivalent to trying to add quality where it's most noticeable. AQ, as well as all x264 ratecontrol modes other than CQP, do this.
However, note that ratecontrol has no fade detection. B-adapt=2 helps in fades, but even it doesn't know which frames are fades; it's just an algorithm which is closer to RD-optimal in general. There is no parameter you can tune that's "how much to raise/lower quality in fades".

cyberbeing

18th September 2008, 03:33

Yes. Which is perfectly equivalent to trying to add quality where it's most noticeable. AQ, as well as all x264 ratecontrol modes other than CQP, do this.
However, note that ratecontrol has no fade detection. B-adapt=2 helps in fades, but even it doesn't know which frames are fades; it's just an algorithm which is closer to RD-optimal in general. There is no parameter you can tune that's "how much to raise/lower quality in fades".

Thank you for the reply akupenguin. Is adding fade detection for ratecontrol impossible? How feasible would it be to develop a full customizable weight based (sort of like a "debug" mode) rate control like I described here (http://forum.doom9.org/showpost.php?p=1185200&postcount=164)?

Dr.D

18th September 2008, 05:00

i wont even comment apart the fact you ignore the advices from develpers and other ppl that tell you how to configure the encoder...
you use modified builds with experimental patches...
you fail to see where you're doing wrong comparisons since ppl told you to compare at the same bitrate...
for what concerns me, i have nothing to add except you can stop your crusade against a non existent problems (caused by your misconfiguration and/or the use of old or "modified" builds).
your conclusions and test methods are completely wrong... so you can either:
1 - use vanilla builds and compare with the correct test methodology
2 - shut up
chose one.
Just few advises.
Don't go deeper to the idiotic situation what you made by yourself for yourself.
Don't act like a sort of winner, you completely loose this part of discussion.
Don't masquerade your unability to comment my posts by talking about something completely different.

I'm still waiting your comment of that:

So, you said a pair (of builds) with described behavior doesn't exist, I gave you the pair. So, you are wrong. Plain and simple.

Try to turn on logic (after you answers it's hard to believe you can do that, but try your best) and realize, that I can choose encoder parameters, test clips, decoder parameters, etc. to disprove your statement.

so you can either:
1 - admit you are wrong
2 - shut up
chose one.

Sagekilla

18th September 2008, 05:09

Dr.D

18th September 2008, 05:13

Dark Shikari, i'm still waiting the answer on my question:

You cannot compare two encodes at different bitrates. End of story.

Ok, simple question for you.
Let say I have 2 builds, build1 and build2 and encode same clip with crf 25. The clip is 30 min long and has 1 min faded part.
Build1 gives bitrate 200 and awful quality for the faded part and bitrate 2000 for the rest.
Build2 gives bitrate 900 and good quality for the faded part and bitrate 2000 for the rest.
But Build1 encodes faded part better with 2-pass bitrate 900 encoding (as you suggest to test).

Which build handles fades better?

Hope you are not so bad with logic as Sharktooth and will not answer "this bulids don't exist" :).

Sagekilla

18th September 2008, 05:16

Dark Shikari

18th September 2008, 05:30

Dark Shikari, i'm still waiting the answer on my question:

Ok, simple question for you.
Let say I have 2 builds, build1 and build2 and encode same clip with crf 25. The clip is 30 min long and has 1 min faded part.
Build1 gives bitrate 200 and awful quality for the faded part and bitrate 2000 for the rest.
Build2 gives bitrate 900 and good quality for the faded part and bitrate 2000 for the rest.
But Build1 encodes faded part better with 2-pass bitrate 900 encoding (as you suggest to test).

Which build handles fades better?

Hope you are not so bad with logic as Sharktooth and will not answer "this bulids don't exist" :).Such a case surely couldn't exist, since CRF and 2pass use the same ratecontrol equation ;)

Dr.D

18th September 2008, 05:45

Guys, let's cool down a bit please. I don't think we want anyone getting struck for a rule or having this thread closed for whatever reason.

@Dr. D: I think what Sharktooth is trying to say is that you should try to really narrow any variables down to make sure you're getting the complete apples to apples comparison. The more variation (differing bitrate, for example) you have in different aspects, the less reliable your results will be.
Well, if you follow the Sharktooth's part of the discussion you will see only 2 points:
1. I'm a noob, so I even don't have rights to use non-vanilla builds, use non-deafult parameters, make conclusions and, of course, discuss with The Man From Heaven.
2. Compare at the same bitrate in any situation, always, forever and ever.

About 1 you can read the thread and make an own conclusion.

About 2 - it's wrong.
See my unanswered question to Dark Shikari and try to answer yourself:

Ok, simple question for you.
Let say I have 2 builds, build1 and build2 and encode same clip with crf 25. The clip is 30 min long and has 1 min faded part.
Build1 gives bitrate 200 and awful quality for the faded part and bitrate 2000 for the rest.
Build2 gives bitrate 900 and good quality for the faded part and bitrate 2000 for the rest.
But Build1 encodes faded part better with 2-pass bitrate 900 encoding (as you suggest to test).

Which build handles fades better?

Isn't it obvious for you that comparing faded part at the same bitrate here is plain stupid? Because this comparing says - Build1 is better. What is wrong - Build2 produses just microscopic larger clip with much better faded part.

Now see - it's not an imaginary situation, it's exactly mine situation. And, I guess, it's typical situation for many CRF users.

Dr.D

18th September 2008, 05:58

Dr. D, what you describe is more or less a pathological case. Unless you know of a source (and I'm sure all of us would love to see such a source, since it would help improve what's wrong with x264) that exhibits such a horrific variation between two builds, this is more or less a absolute-worst-case-one-in-a-million scenario.
Everybody could see the source a long time ago :) :
http://forum.doom9.org/showthread.php?p=1183889#post1183889
Just an usual fade-in piece.

Sagekilla

18th September 2008, 05:59

Sagittaire

18th September 2008, 06:03

IMO you use the new bframe algo for build 1: imply that fade detection are better. build 1 will use massive bframe number for fade but with high default ratio for bframe. That's mean lower bitrate and lower local quality for fade.

1) change the bframe ratio for better quality
2) use zone option if you want better quality in specific part

crf mode is not constant quality level. You can change the bitrate with the same crf for same build (ratio for frame or psy-rdo can change dramaticaly the bitrate).

Dr.D

18th September 2008, 06:04

Such a case surely couldn't exist, since CRF and 2pass use the same ratecontrol equation ;)
The statement "couldn't exist" means doesn't exist unconditionally, so I can choose everything - builds, clips, settings, etc. for disprove.
Correct?

Dr.D

18th September 2008, 06:10

I have actually never seen your case. I don't change the in loop deblocker settings (it's enabled and at defaults of 0:0 for the encoder, and CoreAVC has the setting enabled) for the record, and I've not once seen a blocking problem as bad as yours.

Edit: Thank you for directing me to the source. I'll see what I can do to help you out with your situation.
Thank you, but I already solved the problem :) :
http://forum.doom9.org/showthread.php?p=1184605#post1184605
The rest of discussion is the attemts to some people to prove that I'm a noob, made everything wrong and have to shut up :).

But, of course, if you will find something interesting, please let me know.

Dr.D

18th September 2008, 06:21

I have actually never seen your case. I don't change the in loop deblocker settings (it's enabled and at defaults of 0:0 for the encoder, and CoreAVC has the setting enabled) for the record, and I've not once seen a blocking problem as bad as yours.
Try to encode my sample with your settings, --crf 25 and Ranguvar 965-2 build http://www.rapidspread.com/redirect?link=495874&hash=rqt1.

Sagekilla

18th September 2008, 06:31

I'll get to this tomorrow when I have the time. I need to sleep for now.

Dr.D

18th September 2008, 07:24

Let say I have 2 builds, ...

Such a case surely couldn't exist, since CRF and 2pass use the same ratecontrol equation ;)
Hm... hope you are just kidding... If yes, just say that and don't read the rest of the post.

Otherwise you have serious problems with logic too.
I clearly see this scene from you childhood:

Teacher. Let say you have 5 apples. Let say you gave 3 apples to Susan. How many apples left with you?
DS. 5 apples.
Teacher. Why?
DS. I will not give any apples to Susan.

Ok, I slightly modified the question:

Let say I have 2 builds, build1 and build2 and encode 2 clips with crf 25. The 1st clip is faded 2 sec long and 2nd is 8 sec non-faded.
Build1 gives bitrate 1400 and awful quality for the 1st and bitrate 4300 for the 2nd.
Build2 gives bitrate 3600 and good quality for the 1st and bitrate 4100 for the 2nd.
But Build1 encodes faded 1st better with 2-pass bitrate 3600 encoding (as you suggest to test).

Which build handles fades better?

Dark Shikari

18th September 2008, 07:53

Hm... hope you are just kidding... If yes, just say that and don't read the rest of the post.CRF and unrestricted 2pass use the same ratecontrol equation.

I cannot imagine a case where they could give any significantly different bitrate distribution. If you can give me such a case, I may find it interesting.

However, since it seems that you cannot understand the most basic of statements, I am not going to bother any more with this ridiculous thread.

Quark.Fusion

18th September 2008, 08:05

Lets say build1 uses only b-frames on fades and build2 uses only p-frames on fades, --pbratio defaults at 1.3 — which build handles fades better? :) In CRF mode quality and size depends of how many b-frames you have, not counting AQ. I think it's why devs ask for same bitrate.

Now lets say build1 uses experimental psy-optimisations and build2 not experimental — where you should complain? Note that build1 will produce completely diffirent quality and bitrate at same CRF value.
If you jump from 250 revisions old build to current experimental branch and compare both with some other not-so-experimental branch — what point in it? Either compare current official with old official or current official with current experimental.

Because many ppl try to do that irrelevant compares devs shout at you. And when you shout you don't see all picture. Why others should do correct tests to prove that problem exists if there may not be problem at all?
All that can be easily compared is quality-to-bitrate ratio — all other is heavily relevant on settings and source. Get two related builds (see above) and do two two-pass encodes with same settings, then post x264 log from encode, the source used, produced result and frames that shows the difference. If source is to big — cut out problematic GOPs (try Avidemux).

Sharktooth

18th September 2008, 19:51

Just few advises.
Don't go deeper to the idiotic situation what you made by yourself for yourself.
Don't act like a sort of winner, you completely loose this part of discussion.
Don't masquerade your unability to comment my posts by talking about something completely different.

I'm still waiting your comment of that:

Try to turn on logic (after you answers it's hard to believe you can do that, but try your best) and realize, that I can choose encoder parameters, test clips, decoder parameters, etc. to disprove your statement.

so you can either:
1 - admit you are wrong
2 - shut up
chose one.
so if it makes you happy, you won... you won the prize of idiot of the year.
so yes, you win and no, im not unable to comment your posts, i just dont want coz i dont like to play with childish ppl, expecially with you, since you seem to not even know what are you talking about.
so, as i already said, i wont comment anymore. enjoy your prize and continue to show it to the rest of the ppl on the forum as you''re already doing.
thank you... drive thru...

foxyshadis

18th September 2008, 20:36

Thread's going nowhere but insults and logic games. Please ignore useless hypothetical questions instead of flaming. Closed.

Some commandments for x264 bug reporting, and this applies to most open-source projects in general:

1. Don't ever report any bug in any patched build. Reproduce it in the latest unpatched build, and report that.
2. If you can't, report the bug to the patch's author, not the main developers (unless they're the same, of course), unless you want to be flamed.

The commandments for discussing any x264 bug report here:

1. Verify that they followed #1 first. Don't flame, just ignore, if they didn't.
2. Don't let it get so far off topic! Open a new thread if you really want to discuss related things that aren't bugs.
3. If something is definitely verified broken, or has already been fixed, stop beating a dead horse (other than following a dev's request to help fix or verify it) and just appreciate it.