Visible Picture-Refresh at every new Keyint

[mbcd]

Sorry for my bad english, but i need a short adwise.

I Encode in x264 with a maximum Keyframeinterval (-keyint 25) of 25 frames to be compliant,

With that I notice every second that picture get refreshed. I can see Keyintchange very visible, because of changing pixel. Its like a overall noise which changes each second.

MPEG2 has nothing of that, i didnt realized it. But in x264 its extreme visible.

I also tried Mainconcepts h264 and there its much better. X264 is extreme.

Whats wrong ?

set CRF=--crf 25
set LEVEL=--level 3.1
set KEYINT=--keyint 25
set KEYMIN=--min-keyint 2
set REF=--ref 3
set PSY=--psy-rd 1.0:1.0
set TRELLIS=--trellis 1
set PSKIP=--no-fast-pskip
set FILTER=--filter 0,0
set SUBME=--subme 7
set PARTITIONS=--partitions p8x8,b8x8,i8x8
set DCT=--8x8dct
set VBVBUFSIZE=--vbv-bufsize 17500
set VBVMAXRATE=--vbv-maxrate 15000
set MIXEDREFS=--mixed-refs
set BADAPT=--b-adapt 2
set BFRAMES=--bframes 3
set DIRECT=--direct auto
set WEIGTHB=--weightb
set SCENECUT=--scenecut 40
set ME=--me umh
set MERANGE=--merange 25
set SAR=--sar 12:11
set THREADS=--threads auto
set THREADINPUT=--thread-input
set PROGRESS=--progress
set NOPSNR=--no-psnr
set NOSSIM=--no-ssim
set NR=--nr 0
set AUD=--aud
set FPS=--fps 25000

[Conquerist]

I've noticed in some tests of mine that keyframes become noticeable when using psy-trellis.

Try running the same encode, but change the psy argumnt to --psy-rd 1.0:0.0. That's also the default, so adding it is actually superfluous.

[Dark Shikari]

You can try lowering ipratio if you don't like the defaults. Part of the way that some other encoders like Mainconcept get around it is they just completely destroy the detail, so you can't see the detail changing at I-frames since there's no detail anyways.

Also, avoid using CRF with VBV. This could also be the cause of the problem (ratecontrol issues tend to manifest themselves in the form of I-frame flashes).

Could you post a sample?

[mbcd]

Thanks

Ive tried now anything but only raising ipratio-value takes help.
Lovering ipratio to "0.7" make it really pumping.

I changed keyint to standard "250", then I get a 10 times higher frequency. So its definitivly the keyint.

Now I have ipratio on "4.2" and changes are less visible (with keyint 25)

Is VBV not needed ? I have to limit maximum bitrate in CRF, otherwise the output may not be playable on SA-Players.

Sample is not possible, because I :readrule:
I have no other testing material on hand.

[Dark Shikari]

Is VBV not needed ? I have to limit maximum bitrate in CRF, otherwise the output may not be playable on SA-Players.No, I mean use 2-pass mode if you have to use VBV.

Since you had to raise keyint as high as 4.2 to fix it, I suspect your issue is caused by VBV; in a normal situation raising it that high would make it far, far worse.

Mainconcept likely looked better because you used a less restrictive VBV. Also, your bufsize/maxrate don't match any SA I know of.

[JohannesL]

Also try lowering the CRF to 22. 25 is quite high and often produces a garbled mess even without VBV.

[Adub]

I second this notion. Frankly, I don't know how people can stand crf above 23. Maybe for artificial sources (think Shrek), but for real world applications crf 25 looks like a bad youtube encode.

[Sagittaire]

Sorry for my bad english, but i need a short adwise.

I Encode in x264 with a maximum Keyframeinterval (-keyint 25) of 25 frames to be compliant,

With that I notice every second that picture get refreshed. I can see Keyintchange very visible, because of changing pixel. Its like a overall noise which changes each second.

MPEG2 has nothing of that, i didnt realized it. But in x264 its extreme visible.

I also tried Mainconcepts h264 and there its much better. X264 is extreme.

Whats wrong ?

it's simply hard quantizer transition generaly between consecutive Iframe and Bframe for low motion and no scene change. Change ratio for that: --ipratio 1.10 --pbratio 1.10.

It will be good idea for x264 developpement to set (for short gop scenario) no scene change and IDR with really lower ratio ... something like --idrratio 1.10 for default setting for example or add ratio variable at keyint --keyint 25:1.10

[nurbs]

@mbod:
Either use trellis 2 or lower psy-trellis to 0.2 . That fixed similar problems for me. If that doesn't help try without psy-trellis.

IIRC in my case the P-frame right before the I-frame looked incredibly bad. If anyone of the developers is interested in a sample and settings I can post them sunday evening.

[Sagittaire]

@mbod:
Either use trellis 2 or lower psy-trellis to 0.2 . That fixed similar problems for me. If that doesn't help try without psy-trellis.

IIRC in my case the P-frame right before the I-frame looked incredibly bad. If anyone of the developers is interested in a sample and settings I can post them sunday evening.

No it's not the problem ... problem is short keyint with high ratio ...

[CruNcher]

There are many ways to somehow stabilize this via encoding options the most influential i found seems to be deadzones adapting those also influences this i-frame pulsing maybe some clever combination can lower it even more ?, though that's also a interesting thing about the ON2 VP8 Demo there is no pulsing @ all visible on their example while for H.264 you quite heavily see it (x264) Ateme and Mainconcept are better visually in those short gop Broadcast scenarios (Live Streaming) (without needing to change options).
Ateme and Mainconcept seem to have balanced their Encoders out to keep a uniform visual Appearance for Broadcasts and accepting some loses for non live scenarios that is still better then x264 forceing maximum quality everywhere not trying to take on these visual issues (at least for not experienced users).
The Question i guess here is when should details be exchanged for stability (finding the right balance) and in a short GOP Scenario with low bitrate i would go for stability all the way, because this pulsing is heavily annoying for the viewer then some small uniform detail lose, i like Sagittaires idea of giving the user some better easier options here for the Broadcast case, though i guess better would be adapting the settings automatically between both scenarios so it doesn't even require much user intervention @ all (accepting the loses).
Even if you can't agree with this i guess we can agree on the fact that there is no perfect way of doing this either you fully visually optimize for one scenario or you find a nice balanced way and don't fear some loses on 1 of the sides and x264 decided for the non Broadcast side of things which is fully understandable and acceptable for most so you then need to tweak around a little to find a nice live result or in your case AVCHD Sap compatible one with 1 pass scenario ? first disable scenecut and yeah these vbv settings seem odd ;).
I wonder what Gabriel can say about these stuff and Joosts low bitrate live streaming experiences and how they try to lower this side effect of the Encoder ? (also seeing that he had to fix masses of VBV issues over the years)

[nurbs]

No it's not the problem ... problem is short keyint with high ratio ...

Then why does setting trellis 2 or lowering psy-trellis fix it for me with all other settings being the same?
I'm pretty sure I played around with ip- and pb-ratio as well and I don't think it helped. However it has been almost a year ago. I'll have a look at it again tomorrow.

[Dark Shikari]

it's simply hard quantizer transition generaly between consecutive Iframe and Bframe for low motion and no scene change. Change ratio for that: --ipratio 1.10 --pbratio 1.10.But notice how he had to increase ipratio to make it look better--not decrease. This makes me suspect it's a VBV issue (with VBV restricting the size of I-frames) rather than one of "real" I-frame flashing due to higher quality in I-frames.

[Dethis]

@Mbcd

Please give us some more info.. like video resolution, intended and achived bitrate. Does this "flashing" happens while you are playing it with your SA-player or your software player (which one ?, DXVA enabled or not ?) or both ??

May i also ask why do you restrict your settings this way ...
I mean ... you set level=3.1 while 4.1 is permitted. Remember that for level 3.1 max bitrate must be 14000kbps (a litle less than your "vbv-maxrate" setting).
Moreso if your max bitrate is < 15000 you could set the GOP size at 2 secs (50 frames). And more ... isn't your SA-Player's buffsize upper limit at 30000 ??.

[benwaggoner]

Is there a lot of noise in the source? Other codecs may use dynamic motion vector costs to keep the grain pattern relatively static within a GOP, but then with a new IDR, that pattern "jumps" to a new one. Open GOP can help here by allowing the leading B-frames to reference both GOPs.

That's a different issue than VBV causing I-frames to be relatively starved of bits. But I'd imagine Open GOP would help there as well for the same reason. 2-pass encoding or Lookahead ratecontrol is the biggest help here, of course, since the encoder can pick up when a better I-frame makes for easier matches for following frames, and so can be allocated more bits.

Is this issue why the x264 default minimum GOP length is 25 frames? That alwasy seemed really high to me; I've got plenty of content with a half-dozen scene changes in a second which would yield a whole buch of intra-coded "I frame in all but name" frames that wouldn't offer any bitrate savings.

In PEP, we had a VC-1 "keypop reduction" mode that would help the "popping with grain" issue (HD DVD had 0.6 sec max GOPs), but with some loss in detail. We never got it working well enough that it was a safe "always on" feature.

[akupenguin]

Is this issue why the x264 default minimum GOP length is 25 frames? That always seemed really high to me; I've got plenty of content with a half-dozen scene changes in a second which would yield a whole buch of intra-coded "I frame in all but name" frames that wouldn't offer any bitrate savings.
x264 will then yield a bunch of "I frame in all including name". It will also keep earlier frames in the reference list, in case your rapid-fire scenechanges reuse a background which multiref can help predict. Which makes those I-frames not IDR.
I have hypothesized a 2pass I/IDR placement algorithm which would make such decisions optimally without missing any potential free seekpoints. But I haven't gotten around to implementing it.

[benwaggoner]

x264 will then yield a bunch of "I frame in all including name". It will also keep earlier frames in the reference list, in case your rapid-fire scenechanges reuse a background which multiref can help predict.
I wish that were true in practice, but crazy Buckheimer/Bay style cutting rarely offers anything for multiref to grab on to.

Which makes those I-frames not IDR. I have hypothesized a 2pass I/IDR placement algorithm which would make such decisions optimally without missing any potential free seekpoints. But I haven't gotten around to implementing it.
That could be useful for some content, but probably not for the stuff I'm thinking about.

For film/video content, multiref doesn't seem that useful that often. Cel animation and motion graphics, sure, but other than a shot-through-spinning-fan-blades ala early Scott, I rarely see it make much of a difference.

[Dark Shikari]

I wish that were true in practice, but crazy Buckheimer/Bay style cutting rarely offers anything for multiref to grab on to.But camera-flashes are a great example where there's plenty.

[benwaggoner]

But camera-flashes are a great example where there's plenty.
Multi-ref does those well, certianly. But for 1-2 frame flashes , BI frames (B-frame with just intra-coding) can do the job just fine as well without multiref.

[mbcd]

Thanks for your reply

I try to do as good as I can and understand.

I made a 22mb package with all needed things. A small sample of source is included. Also all scripts I used.

It includes 4 Test-Encodes and a small settings.txt which describes a little bit.

My source is SD-PAL, and crf=25 is enough for this very bad material, there is no blocking, picturequality is very fine for this ugly lowdetail-source. I dont want to take lower CRF or 2 pass.
I tried 2pass, but I got exactly the same problem. Its no difference between CRF and 2pass in my case.

As you can see, in TEST1.mkv you can notice a high "pulsing", especialy in blue background. TEST2.mkv (with a very higher keyint) background is very fine until next keychange, just as I would like to have. But such a high keyint is mostly not allowed, BLURAY has a keyint of about max 50 for this "bitrate" as I know.

In TEST3.mkv and TEST4.mkv you can see IPRATIO.
With a low ipratio (in TEST3.mkv) you have a very noticable pulsing. With a high ipratio (ipratio=12 in TEST4.mkv) it is not so hard.


Download TEST-PACKAGE (http://netload.in/dateiClw6HrVl26/test.rar.htm)

I played back on PC with VLC and oh StandAlone Player (DUNE). Dune was connected to an old analog-TV but always there I noticed every keyframe.

I also checked filtered output with VirtualDub. There of course is a lot of noise, but my problem is definitvly keyint in x264, filtered source is "OK".

It seems to be a general Problem in x264 and h264. MPEG2 does not have this, its smoother.
I did look into a Bluray, there was also this "flashing", but much lower, it was hard to recognize.

@Dethis:

For SD-PAL Level 4.1 is not allowed, as I remember 3.2 is highest.

Wikipedia says in Level 3.1 (at HighProfile) a max Bitrate of 17.5 Mbit/s. So I think its OK, SAs mostly can handle HighProfile, especialy in Level 3.1 .

I could set keyint to max 50, but that would not solve my Problem.

[Guest]

@mbcd

Maybe you will find something of interest in this thread, which gets into I frame flicker:

http://forum.doom9.org/showthread.php?t=143996

[benwaggoner]

@mbcd,

What kind of QP do you see in your strobing I-frames compared to the QP before/after?

[audyovydeo]

@mbcd,

What kind of QP do you see in your strobing I-frames compared to the QP before/after?

I remember getting this effect when I first started using H.264, QuickTime's implementation, and also when I moved to x264, using --qp 26 and -I 25

The explanation I was given then was that the bitrate was too low relative to the QP, so the I-frame would be significantly worse-looking than the other frames, inducing the once-a-second (in my PAL case) flicker.
Indeed, lowering the QP value or, more pertinently, leaving the keyframe value at defaults got rid of the problem.

cheers
audyovydeo

[Guest]

leaving the keyframe value at defaults got rid of the problem. No, it just makes it less frequent.

[Gabriel_Bouvigne]

I made a 22mb package with all needed things. A small sample of source is included. Also all scripts I used.

As you can see, in TEST1.mkv you can notice a high "pulsing", especialy in blue background. TEST2.mkv (with a very higher keyint) background is very fine until next keychange, just as I would like to have. But such a high keyint is mostly not allowed, BLURAY has a keyint of about max 50 for this "bitrate" as I know.

In TEST3.mkv and TEST4.mkv you can see IPRATIO.
With a low ipratio (in TEST3.mkv) you have a very noticable pulsing. With a high ipratio (ipratio=12 in TEST4.mkv) it is not so hard.

In TEST1, I frames are using qp25 and P frames qp26. The difference is too low to prevent visible flickering (especially on non-stable backgrounds, like in your sample). Something like qp24/qp28 would have been more suitable.

In TEST3, I frames are using qp31 and P frames qp26. With I frames using an higher qp than P frames, obviously you're going to have some very visible flickering.

In TEST4, I frames are using qp10 and P frames qp28. The qp difference is overkill, and of course you are not going to see flickering on mostly static parts, but you are then going to end up with flickering on new objects which appeared before an I frame.

edit: in TEST4, you also significantly increased bitrate, thus biasing results

After further testing, ipratio does not really improve the situation. As there is noise, even the I frames are different enough to create flickering. Thus it's not only an issue of P frames vs I frames.

[audyovydeo]

No, it just makes it less frequent.

Maybe so, but I also assume that a larger keyint allows the bitrate to be spread more efficiently (ie less quality difference between the best-looking and worst-looking frames).

[Guest]

Maybe so, but I also assume that a larger keyint allows the bitrate to be spread more efficiently (ie less quality difference between the best-looking and worst-looking frames). What is your basis for assuming that?

The basis for my position is test results. Have you tested it?

[mbcd]

I agree with Gabriel_Bouvigne. IPRATIO does not solve the problem generally.

I have lots of this videoparts, on some of them you can reduce flickering, but in some you cant.
So I have to try and recorrect every part by hand. Thats not acceptable and possible.

This thing is very tricky, I think there is no general solution for this problem with one simple setting. It seems it has to solve internaly.

I tried something more last days. I took Mainconcepts Reference-Encoder and Canopus ProCoder and tried my sampleclip.
In all cases you have this noticeable flicker. It seems to be a general h264-Problem. All encoders seem to produce it.

I cant follow this conversation here in detail, because im not a codec-""freak"", im only a normal user.

But I personaly hate this flicker, because it doesn`t appear in MPEG2 and XVID.
It disturbs me. I hope that X264-Developers find a way to reduce it.

BTW: I also tried VC1 with Mainconcept Reference, and there you have same issue. Flickering on every IDR-Change

[CruNcher]

mbcd did you tried the new Autovaq yet ? i see a very nice HVS improvement in terms of i-frame pulsing :)
didn't tried other samples yet and this scene is very special in terms of compensating bad ratecontroll via AQ

http://forum.doom9.org/showthread.php?t=147067&page=4

The only ones i know that have this very good under control are Ateme i guess beneficial for them is their lookahead here

[Trahald]

But I personaly hate this flicker, because it doesn`t appear in MPEG2 and XVID.

any video format with different frame types (i,p) is capable of flicker. ive seen it in mpeg2 and xvid when the bitrates were low enough.

the source is very bad. alot of things you see in a lot of captures, very noisy, colors are off at some edges. i think crf 25 is too much to ask.

make an mpeg2 at 2,246kb (like your sample 1) for comparison.

[audyovydeo]

What is your basis for assuming that?

The basis for my position is test results. Have you tested it?

Hello neuron2.

no I didn't test it. Intuitively (at least to me) given the two hard constraints "low bitrate" and "whatever keyint" that x264 has to respect for a given run, "smaller keyint" imposes a stricter constraint to x264 >> it has less leeway in bitrate allocation.
But that's just my non-mathematical me.

In practice, I base myself on this old thread on the "flickering" problem, particularly :

http://forum.doom9.org/showthread.php?p=993203#post993203

and

http://forum.doom9.org/showthread.php?p=993481#post993481



cheers
audyovydeo

[akupenguin]

no I didn't test it. Intuitively (at least to me) given the two hard constraints "low bitrate" and "whatever keyint" that x264 has to respect for a given run, "smaller keyint" imposes a stricter constraint to x264 >> it has less leeway in bitrate allocation.
Extra I-frames waste some bits that could otherwise have been used to improve quality. But the don't pose any particular restrictions on the distribution of the remaining bits.

I recommend setting keyint high enough that there won't be any I-frames other than on scenecuts. Then you don't waste any bits on I-frames, and there's no visible flicker. But bluray doesn't like that, *shrug*.

[onesloth]

Is the "problem" that the seemingly static pictures aren't sufficiently static for the DCT->quantization to give the same results from one I-frame to the next?

[Dark Shikari]

Is the "problem" that the seemingly static pictures aren't sufficiently static for the DCT->quantization to give the same results from one I-frame to the next?No, the problem is that inter prediction and intra prediction don't create the same sort of artifacts. In particular, if you use inter prediction to create detail (like x264 does with psy-RD), the intra prediction won't match it that well.