x264 won't always give an I-frame where needed

[laserfan]

I've noticed that if an I-frame is asserted for the first frame Prior To a scene change,the first frame in the new
scene won't always be made an I-frame. For instance, there's a sequence in the BD movie Wall-E, where if I've set
--keyint 24, it doesn't matter what setting of --min-keyint I try (1 or zero or 2...) or --scenecut variations,
I can't get x264 to give me an I-frame in the correct position of the output.

If you have the disc, look at frames 30213 thru 30481. Right after Wall-E has been mauled by the iron pipes, frame
30453 is the first frame in the next scene, where Eve is shining her blue x-ray under the hood of a yellow truck
(00:21:10.144). x264, with --keyint 24 and --min-keyint (anything) will always choose 30452 as an I-frame, and will
never make 30453 an I-frame. I found this because 30453 is supposed to be a chapter mark, and x264 refused to give
me an I-frame in this chapter mark position, despite that it's a very distinct scene change.

After much trial/error, I found that if I set --keyint to something other than 24, say 240 (or even 14) then I can
get an I-frame at that position. But I suspect some problem in the --min-keyint math that WHATEVER you set keyint
as, if an I-frame is forced because of that, and the next frame happens to be a scene change, well it won't be made
an I-frame as it should (should it not?).

Here are my typical command lines:

"C:\Program Files\x264\x264.exe" --pass 1 --bitrate %IN_BITRATE% --stats "%IN_TITLE%.stats" --level 4.1
--keyint 24 --min-keyint 1 --bframes 3 --weightb --direct auto --subme 2 --partitions none --vbv-bufsize 14500
--vbv-maxrate 17500 --me dia --threads auto --thread-input --progress --no-psnr --no-ssim --output NUL
"%IN_TITLE%.avs" --aud --sar 1:1

"C:\Program Files\x264\x264.exe" --pass 2 --bitrate %IN_BITRATE% --stats "%IN_TITLE%.stats" --level 4.1
--keyint 24 --min-keyint 1 --ref 3 --mixed-refs --bframes 3 --weightb --direct auto --subme 7 --trellis 1
--partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 14500 --vbv-maxrate 17500 --me umh --threads auto
--thread-input --progress --no-psnr --no-ssim --output "%IN_TITLE%.264" "%IN_TITLE%.avs" --aud --sar 1:1

[cacepi]

I can't get x264 to give me an I-frame in the correct position of the output.
Then force an I-frame with --qpfile:
FrameNumber FrameType(I,P,B) DesiredQP
30453 I 18

[Sagekilla]

@laserfan: x264 doesn't always place I-Frames at scene-cuts. IIRC, it places it at points where it would help compress, which may not necessarily be at a new chapter or scene change.

[Sagittaire]

Then force an I-frame with --qpfile:
FrameNumber FrameType(I,P,B) DesiredQP
30453 I 18

1) Well x264 don't choose Iframe for good reason. For the core (in pure efficiency term) it's not necessary to place Iframe at this scene change. Anyway you can change the Iframe sensivity for better scene change detection.

2) Better way is to force Iframe for each chapter point. Use qpfile for make that only for the first pass.

[kemuri-_9]

you can use the qpfile to set IDR frames manually, but it currently doesn't work when scenecut is enabled.
when using --qpfile, b-adapt is set to 0 and scenecut is set to -1, which is the values to disable both of them.

re-enabling b-adapt still works fine (1 or 2),
however re-enabling scenecut causes the qpfile's frame decisions to be discarded (the specified QPs are kept however).

and if you want to let x264 decide what the QP for the frame is (you just want to tell it the type), put -1 as the desiredQP for that frame:
instead of

30453 I 18

do

30453 I -1

Take note that in x264, I = IDR frame and i = non-IDR I frame

non-IDR I frames are inserted instead of IDR frames when it's been less than min-keyint frames from the last IDR frame
so if min-keyint is 1, you'll never get non-IDR I frames inserted by scenecut.

without a sample of the problematic section, there's not much else that can be done to help on this.

[laserfan]

x264 doesn't always place I-Frames at scene-cuts. IIRC, it places it at points where it would help compress, which may not necessarily be at a new chapter or scene change.

1) Well x264 don't choose Iframe for good reason. For the core (in pure efficiency term) it's not necessary to place Iframe at this scene change. Anyway you can change the Iframe sensivity for better scene change detection.There's (obviously) a lot I don't understand about this, but:

1. Aren't the subsequent frames following an I-frame based upon the image in the I-frame immediately preceding? So how can subsequent images that are based upon an I-frame from a completely different scene be "better compressed" or "more efficient" as a result?

2. If the first frame of the new scene is the right choice for an I-frame using keyint of 14 or 240 or in fact ANY keyint other than 24, what rationale can there be for it NOT being the right choice for keyint exactly 24? :confused:

BTW Sagittaire as I said I DID try numerous variations on the standard scenecut 40 parameter, to no different effect in this scenario. Only getting-off of keyint 24 gives the expected result.

I will try to provide a sample, which is not easy given my internet provider...

[ajp_anton]

you can use the qpfile to set IDR frames manually, but it currently doesn't work when scenecut is enabled.
when using --qpfile, b-adapt is set to 0 and scenecut is set to -1, which is the values to disable both of them.Would it be possible to modify the stats file instead?

[kemuri-_9]

Would it be possible to modify the stats file instead?

yes, that was what people did before partial qpfiles was allowed... of course this only works when doing 2pass modes (because you need a .stats to edit)

however, doing this is risky and can lead to some side effects (possibly corrupting the decisions for surrounding frames)

@laserfan
since this is only happening apparently with keyint 24 (for hardware playback compatibility i'm presuming)
where's the most recent IDR frame before frame 30452?

[Dark Shikari]

yes, that was what people did before partial qpfiles was allowed... of course this only works when doing 2pass modes (because you need a .stats to edit)

however, doing this is highly risky and can lead to some side effects (possibly corrupting the decisions for surrounding frames)Solution: run a third pass.

(Note: you still have to change P/B decisions accordingly, as x264 does not support open GOP).

[laserfan]

@laserfan
since this is only happening apparently with keyint 24 (for hardware playback compatibility i'm presuming)
where's the most recent IDR frame before frame 30452?It was 24 frames earlier, i.e. frame 30452 was the 25th following the earlier I-frame. Then of course the next frame is the scene change.

Well, I uploaded the clip--thanks in advance to any who take a look at this, it's about 72Mb in size:

http://www.mediafire.com/file/txhxwz34mmh/video.264

I used tsMuxeR to output the clip from the original m2ts, then DGAVCIndex to make a .dga file, then this .avs

LoadPlugin("C:\Program Files\RipBot264\Tools\DGAVCDEC\DGAVCDecode.dll")
AVCSource("E:\test\clip.dga")

as input to my passes.cmd (with CRs for readability, not paste-ability):

@echo OFF
@SET IN_TITLE=clip
@SET IN_BITRATE=4400
@echo Beginning Pass1
echo.
"C:\Program Files\x264\x264.exe" --version
DATE /T
TIME /T

@echo ON
"C:\Program Files\x264\x264.exe" --pass 1 --bitrate %IN_BITRATE% --stats "%IN_TITLE%.stats" --level 4.1
--keyint 24 --min-keyint 1 --bframes 3 --weightb --direct auto --subme 2 --partitions none --vbv-bufsize 14500
--vbv-maxrate 17500 --me dia --threads 2 --thread-input
--progress --no-psnr --no-ssim --output NUL "%IN_TITLE%.avs" --aud --sar 1:1
@echo OFF

echo.
@echo Beginning Pass2
echo.
"C:\Program Files\x264\x264.exe" --version
DATE /T
TIME /T

@echo ON
"C:\Program Files\x264\x264.exe" --pass 2 --bitrate %IN_BITRATE% --stats "%IN_TITLE%.stats" --level 4.1
--keyint 24 --min-keyint 1 --ref 3 --mixed-refs --bframes 3
--weightb --direct auto --subme 7 --trellis 1
--partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 14500 --vbv-maxrate 17500 --me umh --threads 2 --thread-input
--progress --no-psnr --no-ssim --output "%IN_TITLE%.264" "%IN_TITLE%.avs" --aud --sar 1:1
Finally, to ID the I-P-Bs I opened the encoding with Avidemux which shows I at the "wrong" frame. Excuse me, it's not the "wrong" frame, it's the 25th frame. It's just that the NEXT frame needs ALSO to be an I-frame (IMO).

Sorry I'm not at my desk w/that PC right now--can't tell you the exact frame numbers to look for. Well, let me walk out there and look... I'll edit this post.

EDIT: To clarify, the above link is to the original; the challenge is to make an x264 copy with --keyint 24 that yields an I-frame at frame 542. When I run the above .cmd, I get I-frame at 541 and P-frame at 542.

BTW if anyone knows an easier way to see I-P-B-per-frame than opening with Avidemux, please let me know.

[kemuri-_9]

ah ha, i was suspecting that was the case!
because it had to put an IDR frame at the frame prior to the scene change,
i'm believing x264 isn't finding it efficient to put another IDR frame right after.

BTW if anyone knows an easier way to see I-P-B-per-frame than opening with Avidemux, please let me know.

x264 writes the frame decisions to the .stats file:
the most important ones to most people is:
in:# out:# type:X

in is the frame number taken from the input
out is the frame number it becomes in the output stream
X is the frame type it is:
I = IDR
i = non IDR I
P = P
B = referenced B
b = non-ref'd B

the file is sorted by ascending output frame #

but an other note, i think i have a fix to have the qpfile's decisions be accepted...
it turns out that you couldn't do all 3 of
A. (partial) qpfile
B. b-adapt
C. scenecut

at the same time...

i'll be on irc after i get some rest and go to classes (boo!) to talk with D_S about the fix i have.

[Sagittaire]

For the future best way must be to use zone option to impose Iframe at each first frame zone.

[laserfan]

...BTW if anyone knows an easier way to see I-P-B-per-frame than opening with Avidemux, please let me know.

...x264 writes the frame decisions to the .stats file:
the most important ones to most people is:
in:# out:# type:X

in is the frame number taken from the input
out is the frame number it becomes in the output stream
X is the frame type it is:
I = IDR
i = non IDR I
P = P
B = referenced B
b = non-ref'd B

the file is sorted by ascending output frame #Thanks for that explanation, but I was asking about a way to SEE the visual image of the frame-in-question, along with the type of frame that it is. I have only found Avidemux to do this, and in fact it seems to count frames differently than Avisynth/avs/Vdub does (Vdub only ever shows K). So I like a visual way to confirm the frame # vs. the actual image on-screen.

So kemuri_9 you have looked at the clip, and don't think x264's inability to place an I-frame there is a bug?

[Dark Shikari]

So kemuri_9 you have looked at the clip, and don't think x264's inability to place an I-frame there is a bug?The fact that scenecut decision is not absolutely perfect is not a "bug."

x264's algorithms are not perfect, and x264 does not take into the number of future references to a frame in determining whether or not it should be an I-frame.

[Sagekilla]

Would it even be possible for the scene change algo to be modified (in it's current state) to allow for that?

[laserfan]

The fact that scenecut decision is not absolutely perfect is not a "bug."

x264's algorithms are not perfect, and x264 does not take into the number of future references to a frame in determining whether or not it should be an I-frame.You haven't addressed though why x264 gives this frame an I-frame in most (every?) scenario other than keyint 24.

Guess I shouldn't have used the "b" word, though it seems to me that other developers around here (like maybe that brain-cell guy ;)) actually *appreciate* when users raise flags to them. I didn't post this to yank anyone's chain, and I do have better things to do with my time...

[Dark Shikari]

You haven't addressed though why x264 gives this frame an I-frame in most (every?) scenario other than keyint 24.Because the weighting of the scenecut threshold is based on the distance to the next keyframe. If there was just a keyframe, x264 strongly weights against adding yet another I-frame right after it. If we're almost all the way to the next keyframe as required by keyint, x264 weights strongly in favor of a scenecut.

Thus, depending on the keyint you chose, that scenecut may be very soon after a previous keyframe or very long after a previous keyframe. This will affect the scenecut decision.

[shon3i]

What will happen if you take this command line?

--pass 2 --bitrate 5835 --stats ".stats" --level 4 --keyint 24 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 3 --weightb --direct auto --deblock -3:-3 --subme 7 --trellis 1 --psy-rd 0.8:0.8 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 8000 --vbv-maxrate 15000 --me umh --threads 6 --thread-input --aq-strength 0.8 --aud --progress --no-dct-decimate --no-psnr --no-ssim --output "output" "input" --sar 1:1 --mvrange 511 --aud --nal-hrd --qpfile "F:\HaroldKumar\frames.txt" --b-adapt 1 --scenecut 40 --pre-scenecut

frames.txt
8142 I -1
16762 I -1
28407 I -1
34876 I -1
42445 I -1
52746 I -1
61238 I -1
68187 I -1
77692 I -1
85872 I -1
92813 I -1
98025 I -1
106850 I -1
115620 I -1
127650 I -1

I alredy encoded and when i muxed in tsmuxer, chapters are in place. Is that a coincidence

[Sagekilla]

Well if you take that command line and run it, you'd get video out I should hope. You should be getting I frames where you specified in the qpfile, too. Also, --pre-scenecut is implied by --threads > 1, and --scenecut 40 is a default (as well as --b-adapt 1). So there's no point in including any of those three.

[laserfan]

Because the weighting of the scenecut threshold is based on the distance to the next keyframe. If there was just a keyframe, x264 strongly weights against adding yet another I-frame right after it. If we're almost all the way to the next keyframe as required by keyint, x264 weights strongly in favor of a scenecut.

Thus, depending on the keyint you chose, that scenecut may be very soon after a previous keyframe or very long after a previous keyframe. This will affect the scenecut decision.Ok, well that answers my question, thank you.

The chapter mark issue is truly a sidebar for me, it was only the fact that I WAS LOOKING AT THEM CAREFULLY that I found this repeatable instance of "no I-frame at the scene change". Re: creating chapter marks accurately, I will try kemuri-_9's suggestion.

But another reason I was concerned is that I have on rare occasions experienced frame corruptions at the beginning of a scene, when that scene does not start with an I-frame. I sought neuron2's advice in this post (http://forum.doom9.org/showthread.php?p=1235313#post1235313) but since I was not able to duplicate the issue in a snippet from that movie, it was not pursue-able. Indeed in this Wall-E case, the scene-with-no-I-frame-up-front encodes fine, I can't get it to fail.

So I still don't know what's causing my infrequent frame corruptions, but I thought this oddity re: I-frame selection was worth reporting to this forum. I learned long ago "if you fix known problems, sometimes the hardest ones go away by themselves".

[kemuri-_9]

sorry for being misinforming, but only when finding a possible solution to the qpfile problem did i find that the problem is not with scenecut but with b-adapt.

so after --qpfile which sets scenecut to -1 and b-adapt to 0, re-enabling b-adapt to 1 or 2 will discard the qpfile decisions.
re-enabling scenecut is still fine.

[shon3i]

So there's no point in including any of those three. I put it because kemuri-_9 said so after --qpfile which sets scenecut to -1 and b-adapt to 0, re-enabling b-adapt to 1 or 2 will discard the qpfile decisions.
re-enabling scenecut is still fine.

That's why asking, and i alredy make two encoders with reenabled b-adapt and scenecut, and look's like chapters are in their places :)

[ajp_anton]

Will this qpfile+b-adapt issue be fixed?

[G_M_C]

It might be a good feature request to make x264 so that you can put in "chapter points" via a chapter list HCEnc does for instance. The mechanism seems to exist already (more or less) via the --qpfile, a renewed implementation of this might be possible.

Also, i'd like to request more energy go into making x264 a viable encoder for real BD compliant streams (patches like the --nal-hrd patch, or the slices problem).I ask this because it is evident that more and more people are making their own "BD compliant" projects, creating the need for the best BD compliance possible. The evidence of this can be found all over this board, take BD-rebuilder as example or MultiAVCHD or all the other treads about BD9. And in the near future BD-R projects will come, creating even bigger numbers of questions and thus more need for good bd compliancy. The chapter-points is just one thing needed to make backing up (with menu's) better possible, but more work has to be done. I'm no programmer, just an observer that sees the questions about this subject coming faster and faster on our board.

[laserfan]

For the future best way must be to use zone option to impose Iframe at each first frame zone.I'm a rank amateur at this, and dunno about the "zone option", but it does seem to me that somehow for a multi-pass encoding one ought to be able to tell x264 to "first find all the scene changes and apply I-frames to these, then assure (e.g.) no greater than 24- or x-length GOPs, and finally assign extra I-frames as needed for quality".

Note I said that because I'm not a programmer. Therefore it is, as the saying goes, just "a simple matter of software"! :p :D

[J_Darnley]

somehow for a multi-pass encoding one ought to be able to tell x264 to "first find all the scene changes and apply I-frames to these, then assure (e.g.) no greater than 24- or x-length GOPs, and finally assign extra I-frames as needed for quality".
Sounds like --pass 1 to me

If you misunderstood what Sagittaire said: zones do not force frame types. One (possible) hack to use zone to try to force a keyframe might be to use a one frame zone to set keyint to 1.

[kemuri-_9]

Sounds like --pass 1 to me

If you misunderstood what Sagittaire said: zones do not force frame types. One (possible) hack to use zone to try to force a keyframe might be to use a one frame zone to set keyint to 1.

zones don't support changing the keyint values, so even if you did specify it in a zone, it's ignored.
however, someone is working on having the SAR be able to be changed midstream and is looking to use zones to do so.
from what he's been saying from his tests, it looks like it'll need to force an IDR frame when the zone/SAR change starts.

but, it looks like he's currently facing the same problem as the qpfile:
b-adapt is ignoring all pre-determined frame decisions....

[J_Darnley]

Oh yes, silly me. I have seen the discussion about SAR changes but I wasn't paying it much attention because I have no use for that. I will need to watch it more if it also "fixes" frame type troubles.

[gatewaymastering]

I asked this question about I frames a while back. In DVD world to get around this I used to encode each chapter seperately then concatenate them together. By encoding each chapter seperately you always get an I frame at the beginning. Can you string together multiple .264 files?

[J_Darnley]

Yes, with some limitations. Using identical settings throughout will be fine but don't try to increase refs on a later part, it broke decoding last time I did that.

[gatewaymastering]

What app or command do you use to piece the files together?

[J_Darnley]

Windows copy /b, Unix cat, mkvmerge (or rather mmg) append feature.

[kemuri-_9]

Going back to the topic at hand...

a few commits hit repository just now....
be a bit before x264.nl updates to r1077

one (r1076) fixes the qpfile functionality with b-adapt
also... --qpfile no longer disables scenecut or b-adapt when called... so you don't have to re-enable them back now.

[laserfan]

Going back to the topic at hand...Actually, the topic at hand is "how to assure I-frames at scene changes" but I'm glad you found something to fix k9.

So I guess the answer to my issue is "there's no way". Any time an I frame is created the frame before a scene change, no new I-frame can be assured for the first frame in the new scene.

[LoRd_MuldeR]

Any time an I frame is created the frame before a scene change, no new I-frame can be assured for the first frame in the new scene.

That won't happen, as x264's scenecut detection will put an I-Frame at the first frame of a new scene. Not at the last frame right before a scene change.

It may not put an I-Frame at the first frame of a scene in case that scene change is too close to the previous scene change (min_keyint) or when the scenecut detection has failed.

...but then it still won't put an I-Frame right before the scene change. That can only happen if you run into the maximum keyframe interval.

[Dark Shikari]

That won't happen, as x264's scenecut detection will put an I-Frame at the first frame of a new scene. Not at the last frame right before a scene change.Not true at all. There's no reason that x264 won't end up putting an I-frame right before a scene change--what if it's the end of keyint? Then it's required to.

[LoRd_MuldeR]

...but then it still won't put an I-Frame right before the scene change. That can only happen if you run into the maximum keyframe interval.

Not true at all. There's no reason that x264 won't end up putting an I-frame right before a scene change--what if it's the end of keyint? Then it's required to.

:confused:

[Guest]

That won't happen

That can...happen

:confused:

[LoRd_MuldeR]

So if you grab fragments of my post and put them out of the context then the sense is lost. So what?

Fact remains that the only case where x264 puts the I-Frame right before the scene change (instead of putting an I-Frame at the beginning of the new scene) is when the max_keyint limit is reached.

So in the majority of all cases it won't happen, when using reasonable min_keyint/max_keyint values...

[laserfan]

...the only case where x264 puts the I-Frame right before the scene change (instead of putting an I-Frame at the beginning of the new scene) is when the max_keyint limit is reached.

So in the majority of all cases it won't happen, when using reasonable min_keyint/max_keyint values...When using keyint of 24 (as is the case here with the clip in question) for Blu-ray compatibility, you've got thousands of opportunities for this to happen. With an HD-DVD target of only 14, the chances are even greater.

Maybe I'm making a big deal out of this for the wrong reason, i.e. maybe the corrupted frames I've seen when a scene is not begun with an I-frame are caused by something else, but I reported this because it just seems wrong--new scenes should start with a new I-frame.

Anyone here have a 264 encoder that is not x264? Wonder how pro tools would handle this.

[LoRd_MuldeR]

When using keyint of 24 (as is the case here with the clip in question) for Blu-ray compatibility, you've got thousands of opportunities for this to happen.

BluRay uses 24 as maximum keyframe interval? :eek:

[kemuri-_9]

BluRay uses 24 as maximum keyframe interval? :eek:

from what i've read here on doom9 it needs 1 IDR frame for every second in the video, so for ~24fps, that would be 1 every 24 frames.

[G_M_C]

BluRay uses 24 as maximum keyframe interval? :eek:

from what i've read here on doom9 it needs 1 IDR frame for every second in the video, so for ~24fps, that would be 1 every 24 frames.

Yep, the BD standard uses a GOP length defined as 1 second. That means the max keyframe interfal can always be set equal to the framerate, rounded to the nearest whole number (i.e 24,25,30,50 or 60). The minimum keyframe interval is defined as 1.

Based on those rules, it easily forseable that there will be more than one instance where you'll see concequtable keyframes (in an average length movie that is).

[Guest]

It's not just bluray. For broadcast applications, you need to allow for fast channel hopping, and so there are frequent keyframes.

In fact what LoRd_MuldeR thinks are reasonable keyframe intervals are actually the unreasonable ones. Even for error recovery, if you have a glitch in your stream and you have 300 frame GOPs, you can be staring at garbage for 10 seconds.

[G_M_C]

[...]
In fact what LoRd_MuldeR thinks are reasonable keyframe intervals are actually the unreasonable ones[...]

The only reasonable thing about long GOPs i can think of, would be compressabillity; But somehow i doubt that it'll make that big a difference, that it outweighs the negative side-effects you mention for instance (i.e it isn't really efficient).

But back on the subject; I would like a feature like HCEnc has for chapterpoints. This would make backing up (and authoring) using x264 better, cause you can set your chapters back on exactly the same spot they were in the original (or exactly at the point you chose).

[fields_g]

Not true at all. There's no reason that x264 won't end up putting an I-frame right before a scene change--what if it's the end of keyint? Then it's required to.

Wouldn't it be more efficient (compressibility-wise and error correction) to split the last 24 frame GOP into two smaller GOPs to allow the I frame to be placed on the scene change. That is... have a GOPs lengths of 13, 12, then 24 for example (scene change at start of 24). Probably would require the encoder to step back and burn more time though.....

[LoRd_MuldeR]

from what i've read here on doom9 it needs 1 IDR frame for every second in the video, so for ~24fps, that would be 1 every 24 frames.

Then you will run into the may_keyint limit more often, then I had expected...


The only reasonable thing about long GOPs i can think of, would be compressabillity; But somehow i doubt that it'll make that big a difference, that it outweighs the negative side-effects you mention for instance (i.e it isn't really efficient).

Yes, compressibility is a big point! Any x264 front-end I can think of uses the following rule of thumb for keyframe intervals by default: min_keyint = fps, max_keyint = fps * 10. So for 25 fps material that would be 25 and 250. I know that "short" keyframe intervals make sense for broadcasting, because a lot of data loss is expected. But we are talking about a local media here. If BluRay really had an error rate that high, people would see a lot of one second flashes of corrupted picture (and BluRay had to be considered a faulty design). But I don't see people report about that. Guess they sacrificed compressibility with short GOPs, because BluRay has enough bitrate to waste anyway. After all it initially was designed with the bitrate requirements of MPEG-2 in mind...

[Guest]

Bluray needs to support decent trick modes, which requires frequent seek points, aka keyframes.

[laserfan]

If BluRay really had an error rate that high, people would see a lot of one second flashes of corrupted picture (and BluRay had to be considered a faulty design). But I don't see people report about that...
Perhaps you are referring to my posts above where I'd found an instance of corrupt frames when a new scene did not start with an I-frame.

But as I explained, although the corrupt frames caused me to observe that the I-frame seemed misplaced, I have no way of knowing whether the misplaced I-frame was the *cause* of the corruption, and the corruption was not repeatable.

I started this thread in the hope that someone would say "the settings you should be using are these" so I could at least eliminate the "I-frame at one frame too early" question. But now I'm hearing "that's the best x264 can do" and I'm hoping someone on the dev team will decide that it's just not quite good enough! ;)

[Dark Shikari]

Perhaps you are referring to my posts above where I'd found an instance of corrupt frames when a new scene did not start with an I-frame.If you actually have an issue with "corrupt frames", go and report the bug by uploading a small sample of the resulting output video containing and post the settings you used to encode it, along with an exact description of what you see as the symptoms of the problem and in which frames they occur.

What not to do: spend 3 pages complaining about it! :rolleyes:

(Also, don't misuse the word "corrupt". If you actually have a case of x264 producing a corrupt bitstream, that's a serious bug and needs to be fixed, and is completely orthogonal to the thread topic.)

[laserfan]

Good grief, how much clearer can I be than having said x times that I don't know WHAT caused the corrupt frames!?

Sensitive, much? Sheesh... :rolleyes:

Indeed, I've suggested every time that the Decoder was causing them.

[Dark Shikari]

Good grief, how much clearer can I be than having said x times that I don't know WHAT caused the corrupt frames!?I'm not asking you what caused them, I'm asking for a sample.

[G_M_C]

Then you will run into the may_keyint limit more often, then I had expected...




Yes, compressibility is a big point! Any x264 front-end I can think of uses the following rule of thumb for keyframe intervals by default: min_keyint = fps, max_keyint = fps * 10. So for 25 fps material that would be 25 and 250. I know that "short" keyframe intervals make sense for broadcasting, because a lot of data loss is expected. But we are talking about a local media here. If BluRay really had an error rate that high, people would see a lot of one second flashes of corrupted picture (and BluRay had to be considered a faulty design). But I don't see people report about that. Guess they sacrificed compressibility with short GOPs, because BluRay has enough bitrate to waste anyway. After all it initially was designed with the bitrate requirements of MPEG-2 in mind...


Compressabillity isn't the most important factor, you say that yourself. More important to Blu-rays, and other "fixed media" is the feature-set. The standard they chose has to account for the desired features, i'll give an example;

I've got at least one BD where you can a specific scene of the movie, for an aphabetically ordered list. For instance, you can choose a scene where "storm blows roof of the house". When you select that scene you are instantly transported to that point in the movie where that happens (+- 1 second before the event). The event itself only lasts 2 or 3 seconds or so, and isn't a regular chapter point, just a scene.

The example i gave, and all the 1 or 2 hundered other entrys in the alphabetical list, are only workable when there are frequent i-frames in the stream; Since the events they point to are short. GOPs of 10 seconds, like you purpose, would not be workable.

So you see, standards arent chosen for compressabilly only, other factors play a role too.

The same example shows why it would be desirable to have x264 be able to put i-frames on points chosen by the encoding person/application. In this this case the i-frame points can best be held placed on the exact same place as in the original, only that way can you achieve a full BD-50 -> BD25 backup, with including menus.

[shon3i]

Something is wrong here

I just tryed to encode Crank HDDVD to BD5, with skyfire x264 r1077 founded on MeGUI Autoupdate, and after completly first pass i always get message

[Error] Log
-[Information] Versions
--[NoImage] MeGUI Version : 0.3.1.1010
--[NoImage] OS : Microsoft Windows XP Professional SP3 (5.1.196608.2600)
--[NoImage] Framework used : 2.0 SP1 (2.0.50727.3053)
-[Information] Hardware
--[NoImage] CPU : AMD Phenom(tm) 9550 Quad-Core Processor
-[Error] Log for job1 (video, crank.avs -> )
--[Information] [18/01/2009 12:33:26 AM] Started handling job
--[Information] [18/01/2009 12:33:26 AM] Preprocessing
--[NoImage] Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 1 --bitrate 5436 --stats "F:\Crank\crank.stats" --level 4 --keyint 24 --min-keyint 1 --bframes 3 --weightb --direct auto --deblock -3:-3 --subme 2 --partitions none --vbv-bufsize 8000 --vbv-maxrate 15000 --me dia --threads 6 --thread-input --aq-strength 0.8 --sar 1:1 --progress --no-dct-decimate --no-psnr --no-ssim --output NUL "F:\Crank\crank.avs" --sar 1:1 --mvrange 511 --aud --nal-hrd --qpfile "frames.txt"
--[Information] [18/01/2009 12:33:29 AM] Encoding started
--[Error] An error occurred: x264 [error]: can't parse qpfile for frame 7697
--[NoImage] Standard output stream
--[NoImage] Standard error stream
---[NoImage] avis [info]: 1920x1080 @ 23.98 fps (135479 frames)
---[NoImage] x264 [info]: using SAR=1/1
---[NoImage] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSEMisalign LZCNT
---[NoImage] x264 [info]: profile Main, level 4.0
---[NoImage]
---[NoImage] x264 [info]: slice I:9153 Avg QP:20.72 size: 84125
---[NoImage] x264 [info]: slice P:81689 Avg QP:23.69 size: 32691
---[NoImage] x264 [info]: slice B:44637 Avg QP:25.45 size: 7456
---[NoImage] x264 [info]: consecutive B-frames: 39.6% 38.0% 5.7% 16.7%
---[NoImage] x264 [info]: mb I I16..4: 65.1% 0.0% 34.9%
---[NoImage] x264 [info]: mb P I16..4: 34.4% 0.0% 0.0% P16..4: 35.8% 0.0% 0.0% 0.0% 0.0% skip:29.8%
---[NoImage] x264 [info]: mb B I16..4: 2.8% 0.0% 0.0% B16..8: 14.9% 0.0% 0.0% direct: 6.4% skip:75.9% L0:33.2% L1:47.1% BI:19.7%
---[NoImage] x264 [info]: final ratefactor: 24.33
---[NoImage] x264 [info]: direct mvs spatial:97.5% temporal:2.5%
---[NoImage] x264 [info]: kb/s:5342.1
---[NoImage] encoded 135479 frames, 19.44 fps, 5346.99 kb/s
--[Information] [18/01/2009 2:29:39 AM] Job completed


whole pass is finished but i don't know what is problem about 7697 frame??

qpfile:
7696
21954
28903
38001
45175
57111
71038
81168
97966
121612
126339

[Dark Shikari]

That isn't how a qpfile is formatted. Read --longhelp.

[shon3i]

OMFG :)

Thanks Dark Shikari :) my mistake lol

[laserfan]

...it would be desirable to have x264 be able to put i-frames on points chosen by the encoding person/application. In this this case the i-frame points can best be held placed on the exact same place as in the original, only that way can you achieve a full BD-50 -> BD25 backup, with including menus.Well, all I asked about was getting I-frames at scene changes, and since this apparently can't be assured, instead of setting keyint to 24 I've set it instead to 240. Should be very rare I'd think to not have an I-frame where you might need one (without having to go back and re-encode the whole video, that is).

Fortunately I have a BD player that doesn't seem to mind about the 24 GOP "standard" tho others might not be so lucky.

[turbojet]

Fortunately I have a BD player that doesn't seem to mind about the 24 GOP "standard" tho others might not be so lucky.

I've tested quite a few players in store and they all played 240 as well as 24 keyint files. I've also never heard of anyone report 240 not working but 24 as working.

Depending on the compression and target 240 can help a lot or not really much at all. At CRF20 I normally see anywhere from a 3-10% smaller file with 240 vs 24. With a typical BD50 to BD25 encode this is a rather meaningless gain. With DVD5/9 targets it can be quite a noticable difference.

I also would really appreciate the ability to insert a keyframe where I specify it and I know for sure it will be there. CCE, HCEnc, divx3, xvid are just some encoders I've used that have this ability and it works 100% of the time.

[Dark Shikari]

I also would really appreciate the ability to insert a keyframe where I specify it and I know for sure it will be there.Would you appreciate it more if I remind you that it's already there?

[turbojet]

Indeed, I will try it with latest build soon and see if it does place them in the frame specified.

[laserfan]

Indeed, I will try it with latest build soon and see if it does place them in the frame specified.I found another instance this morning of an I-frame being created immediately prior to the scene change (I have keyint at 240 but this occurred at 175 so I dunno why it missed this one).

In any case I too will use the qpfile method next, to assure I-frames at desired chapter locations.

[burfadel]

Would a forward scenecut detection provide any benefit in keyframe placement, such that the keyframe may be extended a few frames in that instance? in all other instances the standard keyframe could apply?... I could imagine this could be fractionally beneficial?...

In addition to that, a while back a ME prepass was tested but found not to be benefiical, which is understandable for CRF and 2+ pass modes, BUT would ME-prepass be of any benefit for CBR mode, which people seem to preference for streaming?

[shon3i]

I made one small app which convert eac3to or chapter grabber chapters to x264 frame based ascii file, aslo can convert between framerates, for example for HDDVD you need to convert from 24 to 23.976 to get actual times. Suggestions are welcome.

http://tom.niko.users.sbb.co.yu/qpfilegen.rar

[laserfan]

I made one small app...I just went thru this a couple hours ago, and in comparing your app's output to my own it is perfect! My, this is so much better than opening w/Vdub, finding every frame number by copy/paste of timecode, then noting frameno and adding I -1.

Excellent work, many thanks! :)

[Sagekilla]

In any case, thanks for the app shon3i! I've wanted to try forced I-Frames at chapter points or scene changes, but as laserfan said it was too cumbersome for me to bother.

[laserfan]

My first attempt at using the --qpfile function worked swimmingly well; thanks kemuri-_9 for laying it out here. Now with shon3i's app (and accurate indexing using neuron2's tools) it's easier to assert chapter marks "up-front" than it was to do it after encoding.

@shon3i--it would be nice too if your qpfilegen would output frame numbers in tsMuxeR .meta file format also, e.g.:

00:00:00.000;00:07:53.056;00:13:37.150;00:18:15.136;00:22:07.701;00:24:47.736;00:30:07.764;00:33:26.379;
00:36:04.871;00:41:58.474;00:47:11.245;00:52:57.257;00:59:00.829;01:03:03.613;01:07:40.265;01:15:19.932;
01:17:41.407;01:21:27.257;01:25:43.096;01:30:44.856;01:34:36.254

though, yes, I know a word processor could do this lickety-split as well.

Thanks again kemuri-_9, and for making the fix(es) to 1077... :)

[shon3i]

@laserfan, done ;) use same download link to redownload app :). btw i using qpfile and i newer have problems with chapter points.

[laserfan]

@laserfan, done ;) use same download link to redownload app :). btw i using qpfile and i newer have problems with chapter points.Ah, ha! You've been holding-out on us shon3i! ;)

Very excellent; many thanks! :)

[shon3i]

I update my small app :), now can extract chapters directly from Blu-Ray or HDDVD, of course via eac3to (thanks to madshi :)) You only need to set path to eac3to exe in settings, and choose root of Blu-Ray/HDDVD and little wait :). It's little slower than other metods like ChapterGrabber but works :)

Download (http://tom.niko.users.sbb.co.yu/qpfilegen.rar) (it's same link as above :))

[laserfan]

As an aside, I just did a blu-ray mux where qpfile and I-frames and Chapter marks et. al. were perfect, but fixclpi or not my version of PowerDVD didn't seek to the chapters perfectly on the resulting mux. I did another disc completely different (OTA MPEG2 to HD DVD with Ulead MF6+ and chapters) and same thing--PDVD came-in to the chapters a little early. The discs seek perfectly with TMT and my stb player.

That PowerDVD is unreliable in testing chapter marks is certainly an unhappy discovery for me, but I'm glad to report the "I-frame assignment by qpfile" method appears perfect, and I'm safely using keyint 24 again.