multi-threads break hd dvd compliance (bug?)

[vmrsss]

Hello.

While attempting to I learn to encode x264+ac3 for a HD-DVD player (Toshiba HD-E1) by adapting MeGUI's x264:SA-HD-DVD profile to my command-line MacOSX environment, I bumped into a strange behaviour triggered by "--thread auto" as opposed to "--thread 1" which may well be a bug.

I can't say much more that when burned to HD-DVD format the encodes with "--threads 1" work fine, while for those with "--thread auto" (which is 3 in my case) playback stops abruptly, systematically at several and frequent spots, always the same (and it took me much longer than I'd like to admit to get to this stage). If I compare the x264 -v logs, the encodes are almost identical, the GOPs appear to be the same, while frame QPs, sizes and bitrates differ very, very marginally. Yet, when I play the disc, "--threads 1" work, "--threads 3" aborts playback with no message whatsoever, as if the end of the stream had occurred.

I have verified this behaviour on winxp to, using MeGUI. Has anybody came across this? Is it likely to be a bug in the concurrency management? It'd be good to fix it, it's no fun to encode HD material with a single thread... :-(

A few details: For x264, I have pulled it from git, patched with x264_hrd_pulldown.09_interlace.diff and with x264_psyrd-0.5.diff, nothing else, compiled (static, 64bit, gcc-4.2). Here is what I get:

[macosx] x264_hrd --version
x264 0.60.928M 4816918

[winxp virtual machine] "c:\Program Files\megui\tools\x264\x264.exe" --version
x264 0.60.928M 4816918

The commandline I was playing with is below, though I have been able modify several parameters and still replicate the --thread 3 failure.

$ "c:\Program Files\megui\tools\x264\x264.exe" test.avs --crf 21 
--threads 3 --level 4.1 --keyint 14 --min-keyint 2 --ref 3 --mixed-refs 
--bframes 2 --b-rdo --bime --weightb --direct auto --filter -2,-1 
--subme 6 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --ipratio 1.1 
--pbratio 1.1 --vbv-bufsize 14745 --vbv-maxrate 29400 --qcomp 0.5 
--direct-8x8 1 --me hex --progress --no-psnr --no-ssim --mvrange 511 
--aud --nal-hrd --output test.264

(test.avs only contains DirectShowSource("z:\Desktop\test.m2v"), this is a fragment of a TV MPEG2 capture of no special importance)

For this fragment the summaries look something like this

x264 [info]: slice I:162   Avg QP:19.43  size: 18069
x264 [info]: slice P:970   Avg QP:19.83  size: 10402
x264 [info]: slice B:919   Avg QP:21.47  size:  6438
x264 [info]: consecutive B-frames: 20.5% 26.2% 53.4%
x264 [info]: mb I  I16..4: 13.6% 75.9% 10.5%
x264 [info]: mb P  I16..4:  2.4% 12.7%  0.6%  P16..4: 43.7% 17.2%  8.1%  0.0%  0.0%    skip:15.4%
x264 [info]: mb B  I16..4:  0.2%  1.6%  0.0%  B16..8: 54.2%  1.6%  2.7%  direct:12.1%  skip:27.6%  L0:34.9% L1:50.9% BI:14.3%
x264 [info]: 8x8 transform  intra:78.9%  inter:71.0%
x264 [info]: direct mvs  spatial:78.7%  temporal:21.3%
x264 [info]: ref P L0  72.2% 18.4%  9.4%
x264 [info]: ref B L0  83.5% 16.5%
x264 [info]: kb/s:1846.3

encoded 2051 frames, 13.98 fps, 1851.60 kb/s

I could upload the fragment somewhere, but I suspect it won't be necessary: the problem appears to be quite easy to replicate; it occurred to me in several tv captures and also on a h264 -> h264 re-encode: the problem is not the source or its codecs.

Regards, and thanks for any help you might be able to give.
(currently travelling, I might answer slowly).

(PS. It occurred to me that this might be a bug in one of the patches hrd or psy-rdo, for a long while I suspected a problem with vbv and later on with interlace)

[Manao]

It's not quite a bug, it's rather a known subobtimality. x264 is close to useless in one pass encoding with VBV constraints. Things gets worse with multithreading which complicates x264's job in regard to VBV constraints. You're better off using two passes, even if you don't care about bitrate. Do the first pass in CRF without VBV constraints, and the second one at the obtained bitrate with VBV constraints. If you can afford it, don't multithread the second pass.

[Dark Shikari]

Originally Posted by Manao

It's not quite a bug, it's rather a known subobtimality. x264 is close to useless in one pass encoding with VBV constraints. Things gets worse with multithreading which complicates x264's job in regard to VBV constraints. You're better off using two passes, even if you don't care about bitrate. Do the first pass in CRF without VBV constraints, and the second one at the obtained bitrate with VBV constraints. If you can afford it, don't multithread the second pass.

This is rather doubtful since his bitrate is an order of magnitude lower than the max. If there was a VBV underflow, x264 would print it as a warning; x264 doesn't silently violate VBV. I'd say there's likely more difference between the two streams from the change from regular scenecut to pre-scenecut than from anything VBV-related. Add to this the fact that his maxrate is (I think?) lower than the HD DVD maxrate.

I'd also say its rather obnoxious (and perhaps intentionally misleading, given that your company produces a competing product...) to claim that x264 is "close to useless" in onepass mode considering that my company's entire business model successfully depends on onepass mode working properly under heavy VBV constraints with large numbers of threads on hundreds of realtime encoders simultaneously.

"Not multithreading the second pass" seems particularly absurd since the second-pass VBV algorithm is run before the encoding process as part of the 2pass ratecontrol--threading has no effect on this.

I would, of course, recommend 2pass with VBV whenever possible, since it will give strictly better results.

[Sagittaire]

Well seem not a vbv problem:
- If you use HDDVD player you use a HDDVD authoring software and there are vbv check at the mux.
- The part use really low bitrate (local bitrate is 1850 Kbps with max at 29400 Kbps).

There are another problem ... perhaps GOP compliancy or motion vector compliancy in multithread mode.

[vmrsss]

Hi again. I did not mean to ignite a school-of-thought debate (I am just as amateur, a great fan of CRF who uses 2pass when a specific size is required). I just think I accidentally bumped into what look like a nasty bug, which may in principle manifest itself in other contexts and might therefore be worth a look.

No, I don't think it is a VBV problem. The bitrates are small (this is just a DVB I am using as a test). The only difference between the encodes appear in fact to be the pre-scenecut parameter. Inspecting the logs frame by frame, the differences that this induces on the QPs and bitrates are relatively minute, we are talking hundreds of bytes, and the largest the frames become is below 25KB.

GOPs indeed differ with or without multithreads. But they do not exceed the limit of 14 frames (though they reach it in both versions). Sagittaire, are you saying that --keyint should be lower than 14? Hmm, I am not sure this is the way, because also the single-threaded versions has several GOPs running up to the limit, with no problems.

Regarding the move vectors, sounds intriguing but I am not competent to guess. Is it a real possibility? What test might reveal it?

Thx

PS. I had forgotten to add the most important observation that I did of course run some of the over 100 tests I did with 2 passes, and the result is the same: both with my macosx-compiled x264 and with winxp's meGUI precompiled one, --threads 1 is complaint, --threads auto or 3 breaks it. (Incidentally, in doing this I notice that MeGUI's gui environment seems to ignore the --threads auto setting, which confused me for while.)

[Dark Shikari]

Originally Posted by vmrsss

Hi again. I did not mean to ignite a school-of-thought debate (I am just as amateur, a great fan of CRF who uses 2pass when a specific size is required). I just think I accidentally bumped into what look like a nasty bug, which may in principle manifest itself in other contexts and might therefore be worth a look.

No, I don't think it is a VBV problem. The bitrates are small (this is just a DVB I am using as a test). The only difference between the encodes appear in fact to be the pre-scenecut parameter. Inspecting the logs frame by frame, the differences that this induces on the QPs and bitrates are relatively minute, we are talking hundreds of bytes, and the largest the frames become is below 25KB.

GOPs indeed differ with or without multithreads. But they do not exceed the limit of 14 frames (though they reach it in both versions). Sagittaire, are you saying that --keyint should be lower than 14? Hmm, I am not sure this is the way, because also the single-threaded versions has several GOPs running up to the limit, with no problems.

Regarding the move vectors, sounds intriguing but I am not competent to guess. Is it a real possibility? What test might reveal it?

Thx

Have you tried turning on --pre-scenecut in singlethreaded mode and seeing if that worked or not?

[foxyshadis]

It might be worth uploading the fragments, if only so that they can be analysed in streameye.

[Sagittaire]

My intuition say problem with motion vector length. Try to reduce the mvrange at 255 and see the result ...

[Shinigami-Sama]

Originally Posted by Sagittaire

My intuition say problem with motion vector length. Try to reduce the mvrange at 255 and see the result ...

wasn't that fixed 3 months ago?

[Manao]

I have overlooked the low final bitrate, so indeed VBV shouldn't be an issue. I don't see how threading could increase MV length, since, if anything, threading put more constraints on MVs. I would search for a bug in HRD patch, but then so many people have used it successfully that I don't see why it wouldn't work for you.

Dark Shikari : I don't like gratuitous attack on my probity, especially when you yourself explicitely bash 1 pass VBV handling : http://forum.doom9.org/showthread.ph...60#post1167360

Not multithreading the second pass (or reducing the number of threads) was perhaps overdoing it, but I think in with small VBV (in duration) and average bitrate around half the maximum bitrate, it might still be necessary.

[Dark Shikari]

Originally Posted by Manao

Dark Shikari : I don't like gratuitous attack on my probity, especially when you yourself explicitely bash 1 pass VBV handling : http://forum.doom9.org/showthread.ph...60#post1167360

Oh, I never said 1pass VBV handling was great, but its not useless either; its moreso of the case wher if you can avoid it, you should; obviously, in realtime encoding, you can't avoid it.

My reasons for recommending against it are much the same reasons I wouldn't recommend 1pass ABR.

[vmrsss]

I have not tested "--thread 1 --pre-scenecut", good point, I will do and let you know. I will also test "--thread 2 --mvrange 255". I can't burn and do a full test on my SA, because I am travelling at the moment (I'll be home only at the end of next week), but I'll run a few encodes and log at the logs anyway.

As for upload, yes, any advice on the site to upload to?

[Dark Shikari]

Originally Posted by vmrsss

I have not tested "--thread 1 --pre-scenecut", good point, I will do and let you know. I will also test "--thread 2 --mvrange 255". I can't burn and do a full test on my SA, because I am travelling at the moment (I'll be home only at the end of next week), but I'll run a few encodes and log at the logs anyway.

As for upload, yes, any advice the site to upload to?

Mediafire is a great site if the size of the file is under 100MB.

[vmrsss]

Hello.

I have uploaded a zip file containing the same sequence encoded with --threads 1 and --thread 3, and the corresponding output from x264 -v.

http://www.mediafire.com/?imsk2xfsgv1

playback on SA player stops at a frame 390 <= F <= 420 for --thread 3, goes fine for --thread 1. (The speed of my current connection advices again uploading the YUV source, though I will try if that is of the essence.)

I have run a few more tests with --pre-scenecut, and although I cannot be completely certain of the reaction of my SA player, it really look as though there is where the problem lies. In particular, I have noticed that in all encodes with --pre-scenecut, whether forced by multithreading or explicitly added, whether with with --mvrange 255 or --mvrange 511, there are frame sequences I-I which do not occur without --pre-scenecut and that appear to break the command line setting --min-keyint 2. (These occurrences do not seem to be near the place where things go wrong, but seems to tell something is not quite right with --pre-scenecut...) (In the logs there is also a message about setting the move vectors range to 88 whose extent I don't think I understand.)

Cheers.

[Dark Shikari]

Min-keyint limits the placement of IDR frames, not I-frames.

[vmrsss]

Originally Posted by Dark Shikari

Min-keyint limits the placement of IDR frames, not I-frames.

Received, thanks. I am looking into the code to understand the debug line better.

Could you look at the sequences I posted? Found out anything relevant?

regards

[Dark Shikari]

I notice nothing particularly odd about the sequences except for the fact that somehow they're constructed in a manner that prevents my stream analyzer from seeking correctly, which made it rather aggravating to do any sort of analysis at all.

[vmrsss]

Originally Posted by Dark Shikari

I notice nothing particularly odd about the sequences except for the fact that somehow they're constructed in a manner that prevents my stream analyzer from seeking correctly, which made it rather aggravating to do any sort of analysis at all.

Ouch. What do you mean "constructed"? It's just the output of x264 -v, I haven't instrumented them otherwise. Would you prefer some other form?

[Dark Shikari]

Originally Posted by vmrsss

Ouch. What do you mean "constructed"? It's just the output of x264 -v, I haven't instrumented them otherwise. Would you prefer some other form?

I was referring to the .h264 files. Maybe NAL-HRD breaks Elecard Streameye or something?

[vmrsss]

Originally Posted by Dark Shikari

I was referring to the .h264 files. Maybe NAL-HRD breaks Elecard Streameye or something?

Oh, I see now. NAL-HRD and PSY-RDO are the only "non-official" patches applied. It should be easy to answer this specific question by encoding with the same settings excluding --nal-hrd. More generally, could this all be a problem with nal-hrd? Is it influenced at all by the number of running threads? Perhaps nal-hrd's author is reading...

[vmrsss]

Hello. I have now been able to run a few more tests on this and I think I can confirm it seems to be all down to pre-scenecut: the playback problems seems to be triggered by having --pre-scenecut when making frame-type decisions, more precisely scenecuts (but I still don't know what it is that actually causes the crashes). In fact:

(a) encodes with --threads 1 and --pre-scenecut (as well as encodes with --threads 3) crash (at the same frame), as long as scenecut > 0, both with --crf and with --pass 1 --bitrate;

(b) encodes with: --threads 1 and no --pre-scenecut or with: --scenecut 0 and any number of threads, work well.

(c) 2-pass encodes work well if their --pass 1 falls in case (b), crash if their --pass 1 falls in case (a); in particular after "--pass 1 --threads 1" it is OK to run "--pass 2 --threads auto", if the first pass has --pre-scenecut, the second pass will produce a faulty stream.

I have put these examples at

http://www.mediafire.com/?c9d96s2t5iy

with and without --nal-hrd, as Dark Shikari mentioned the latter makes analysis more difficult for him. (In the filenames T1, PS, T3, SOT1, SOT3 mean respectively --threads 1, --pre-scenecut, --threads 3, --scenecut 0 --threads 1, and --scenecut 0 --threads 3.) The problem frame is around frame 155 in PS and T3.

I have no clue about what is actually happening, so I would like to compare the --scenecut and --pre-scenecut code to see if I can find out anything. Dark Shikari, would you have time to interact on this this week?

Regards.

[Dark Shikari]

The only real difference is in the section where you report breakage, there are more scenecuts in the threaded version. It sounds like your device doesn't like IDR-frames close together.

[vmrsss]

Originally Posted by Dark Shikari

The only real difference is in the section where you report breakage, there are more scenecuts in the threaded version. It sounds like your device doesn't like IDR-frames close together.

Hmm. From your point of view, anything that could be thought as x264 not honoring the command line?, or at least not doing the optimal thing? Anything crazy with frame 155?

Technically, how is it that x264 decides on fewer IDR-frames without threads? If your conclusion is right, then the breakage will happen with all sequences with quick scene changes: that --threads 1 avoid troubles might be just a lucky happening with this particular sequence which won't work in general... Or will it..?

I have learned from one of Sagittaire's posts that --min-keyint must be 2 for HD-DVD. (Perhaps the information is incorrect, perhaps we are in the presence of a decoder bug.) I will now test a few more values for that parameter and then post again. Incidentally, I had already removed --min-keyint in some of my tests and, most surprisingly, an entire section of some 30 odd frames is dropped on playback around the "difficult" frame. This buggers me. Logically, what could cause that behaviour? At such low bitrates, I thought no sequence could challenge a HD player... And then, can a sequence of IDR-frames be considered "complex"?

It will also be interesting to check whether the problem is in the authoring software which remuxes x264's output in .evo streams.

Thanks for now.

[Quark.Fusion]

pre-scenecut

Default: disabled when --threads 1, else enabled

Enables a faster scenecut mechanism. It's enabled by default when you use more than one thread, since regular (not pre) scenecut doesn't work with threads. Normal IDR frame encoding works by encoding the frame normally, then if the frame passes the scenecut test, re-encoding it as an IDR frame. Pre-scenecut detects before encoding if the frame should be an IDR frame and acts accordingly. Pre-scenecut is theoretically not as good as normal scenecut, although the cost of this is usually low. See this thread for more information about the quality loss from bad IDR frame placement.

Recommendation: Default

See also: keyint, min-keyint, scenecut, threads

http://mewiki.project357.com/wiki/X264_Settings

[Dark Shikari]

Originally Posted by vmrsss

Hmm. From your point of view, anything that could be thought as x264 not honoring the command line?, or at least not doing the optimal thing? Anything crazy with frame 155?

Technically, how is it that x264 decides on fewer IDR-frames without threads? If your conclusion is right, then the breakage will happen with all sequences with quick scene changes: that --threads 1 avoid troubles might be just a lucky happening with this particular sequence which won't work in general... Or will it..?

I have learned from one of Sagittaire's posts that --min-keyint must be 2 for HD-DVD. (Perhaps the information is incorrect, perhaps we are in the presence of a decoder bug.) I will now test a few more values for that parameter and then post again. Incidentally, I had already removed --min-keyint in some of my tests and, most surprisingly, an entire section of some 30 odd frames is dropped on playback around the "difficult" frame. This buggers me. Logically, what could cause that behaviour? At such low bitrates, I thought no sequence could challenge a HD player... And then, can a sequence of IDR-frames be considered "complex"?

It will also be interesting to check whether the problem is in the authoring software which remuxes x264's output in .evo streams.

Thanks for now.

Just set min keyint to 14 and max keyint to 14. Sure, efficiency will drop a bit, but since your player seems to dislike closely-spaced IDR frames, perhaps that will fix it.

[vmrsss]

Originally Posted by Quark.Fusion

http://mewiki.project357.com/wiki/X264_Settings

Thanks, nice tutorial and web page. (When you say "this is because pre-scenecut does not work with b-adapt", do you mean "without b-adapt"?)

[vmrsss]

Hi DS

Originally Posted by Dark Shikari

Just set min keyint to 14 and max keyint to 14. Sure, efficiency will drop a bit, but since your player seems to dislike closely-spaced IDR frames, perhaps that will fix it.

Apparently min-keyint = 4 is sufficient to avoid breakage (whilst 3 is not). I start to think that the problem might just be with the authoring software (which sadly is Toast9) which muxes x264's rawvideo output files to evo containers. For instance, the muxer mysteriously appears to drops frames from the encodes as follows:

  h264 sequence: 376 frames; problem sequence 32 frames

  after HD-DVD authoring to evo containers:

  min-keyint  2: frames 362; problem sequence 32f; playback broken
  min-keyint  4: frames 325; problem sequence 24f; playback ok
  min-keyint  6: frames 312; problem sequence 20f; playback ok
  min-keyint 12: frames 290; problem sequence 09f; playback ok

Sounds as though some GOPs are dropped, which doesn't sound right to me. I can't be sure though, nor of course identify a pattern in this, because I am pretty useless at analysing evo files. Are there suitable tools available?

If you can spare time to give a look, I have packed a few h264 streams with the corresponding evos generated by authoring at

http://www.mediafire.com/?iudl62uk3xy

Hopefully, you might be able to spot something resolutive.
Cheers.

[vmrsss]

Originally Posted by Dark Shikari

Sure, efficiency will drop a bit, but since your player seems to dislike closely-spaced IDR frames, perhaps that will fix it.

PS. In what sense is it that larger min-keyint values imply lower efficiency?

[kemuri-_9]

Originally Posted by vmrsss

PS. In what sense is it that larger min-keyint values imply lower efficiency?

DS was saying that having max-keyint and min-keyint being the same values will cause a decrease in efficiency,
since that means that IDR frames will occur at exact increments of that value, giving x264 no relaxation at all in IDR placement
i.e. potentially putting them in spots that really shouldn't deserve them.

[LoRd_MuldeR]

Originally Posted by vmrsss

PS. In what sense is it that larger min-keyint values imply lower efficiency?

If you have two or even more scene-changes within your min-keyint, then the encoder is not allowed to place an IDR frame at each scene-change, although this would be most efficient way to encode these scene changes. Using a larger min-keyint obviously increases the probability that this happens...

[Sagittaire]

If you don't have problem only with --threads 1 it's IMO problem for GOP compliancy with x264. Use simply --threads 1 for fast first pass and use --threads 3 for slow second pass.

[Dark Shikari]

Originally Posted by Sagittaire

If you don't have problem only with --threads 1 it's IMO problem for GOP compliancy with x264.

No, it isn't. x264 is obeying the GOP restrictions passed to it on commandline; its not x264's fault if the restrictions you pass to it are wrong.

If you have a case of x264 violating the --min-keyint parameter, I would like a sample. However, I have never seen such a thing, and doubt I ever will.

[vmrsss]

Hi everybody.

Thanks to all for taking an interest in this. Let me retrace the key steps in this:

I too started from the point of view that this ought to be x264's fault, because I just couldn't conceive how varying the number of threads could have any effect whatsoever on any decoder at all (that is, logically it shouldn't). Removing all patches and counting all GOPs from the -v logs (on several different short sequences showing the problem) is the first thing I did, and --keyint is not broken. I thought --min-keyint was, but DS explained i was confused between I and IDR frames. So, that leaves us with the following options:

(1) --min-keyint 2 is after all not compliant with HD-DVD specs; this seems to be ruled out by several sources on the net, as well as the fact that "--threads 1 --min-keyint 2" works just fine.

(2) it is x264's fault, but definitely something subtler than GOP compliancy. This would be something that happens only with --prescenecut and causes the hd-dvd authoring software to shed some "problem" GOPs when remuxing, apparently sufficiently many for the player to work when --min-keyint > 3, not enough when --min-keyint =2,3. I don't think we can discard this possibility, yet I don't believe I have a serious chance to catch it by running random experiments (eg, haven't tried --bframes 0 or --no-cabac or...) and without any analysis tools. That is why I started to post samples hoping that DS or some x264 cognoscenti of his calibre could give some help.

(3) like 2 above, but not because x264 does something wrong, but because we don't fully understand HD-DVD compliance and pass it the "wrong" command line, if you see what I mean. In this case, it would be even more interesting to understand what triggers the problem.

(4) the hd-dvd author software that is buggy, the other side of the coin of point 2 above. I have verified that going from .h264 files to .evo in HVDDVD_TS folders it drops frames if --pre-scenecut is set when frame-type decisions are made (and doesn't otherwise). I find this bizarre, there is no audio here, what reason are frames being dropped for? It could simply be a bug, and possibly it may be exactly such dropping which breaks compliance! I find this option rather attractive, which is why I packed some of the .evo files hoping for DS to give them a look.

Originally Posted by Dark Shikari

If you have a case of x264 violating the --min-keyint parameter, I would like a sample. However, I have never seen such a thing, and doubt I ever will.

DS, please give a look to my latest samples, in particular the .evo files it looks to me as if something goes wrong at that stage.

Originally Posted by Sagittaire

Use simply --threads 1 for fast first pass and use --threads 3 for slow second pass.

Yes, I will look into --pass 1 parameters to see whether it is faster to encode "--threads 1 --crf 20" or "--threads 1 --pass 1 ; --threads 3 --pass 2". (Only got two cores, it's not obvious the latter will be overall quicker.) In any case I am still entirely curious to understand what goes wrong here.

Regards.

[Dark Shikari]

Originally Posted by vmrsss

(1) --min-keyint 2 is after all not compliant with HD-DVD specs; this seems to be ruled out by several sources on the net, as well as the fact that "--threads 1 --min-keyint 2" works just fine.

Do you still not understand? The normal scenecut is not very aggressive, so in the clips you gave me, it never put any IDR frames within two frames of each other anyways, even when min-keyint was equal to 2. Pre-scenecut did. You're blaming this, now, on a "bug in x264," when in reality x264 did nothing except exactly what you told it to do.

Not a single one of your samples has shown a case where x264 violates the --min-keyint rule.

[Quark.Fusion]

Originally Posted by vmrsss

Thanks, nice tutorial and web page. (When you say "this is because pre-scenecut does not work with b-adapt", do you mean "without b-adapt"?)

full quote is

Note that if you want to force a specific number of B-frames in threaded mode, scenecut must also be disabled, by setting it to -1: this is because pre-scenecut does not work with b-adapt.

thats not my words, and I'm not expert in x264 — can't imagine why pre-scenecut should depends on b-adapt.

[Dark Shikari]

Originally Posted by Quark.Fusion

full quote is
thats not my words, and I'm not expert in x264 — can't imagine why pre-scenecut should depends on b-adapt.

Because pre-scenecut relies on B-adapt to not choose B-frames across a scenecut.

Also, that quote is wrong, the correct quote is

Note that if you want to force a specific number of B-frames in threaded mode, scenecut must also be disabled, by setting it to -1: this is because pre-scenecut does not work with no-b-adapt.

[tetsuo55]

I may be misunderstanding here but.

All other settings equal:

Threads1> File works
Threads3> File breaks.

All threads those is either spread, or not spread the work over several cores/hw threads.

So its obvious that either threads1 is broken or threads3, with the latter being more obvious.

So what we need is a sample file with threads1 and a sample file with threads3 + confirmation that 1 works and 3 is broken.

(On the other hand its obvious that threading is not working 100% yet, might be better not to use it for anything but testing? It's difficult to troubleshoot it when it's only partially able to thread tasks)

[Dark Shikari]

Originally Posted by tetsuo55

I may be misunderstanding here but.

All other settings equal:

Threads1> File works
Threads3> File breaks.

All threads those is either spread, or not spread the work over several cores/hw threads.

So its obvious that either threads1 is broken or threads3, with the latter being more obvious.

No, again, you are making false assumptions and thus deducing false conclusions.

Just because a setting triggers a problem does not mean the setting is at fault for the problem. As I have said a dozen times already, what's happening is pre-scenecut is more aggressive on scenecut and puts IDR frames closer together.

Your device cannot handle this, yet you still refuse to stop using a min-keyint that is incompatible with it!

I don't care if you are incapable of listening to basic instructions to solve your own problems, but I don't want you spreading FUD about "bugs in x264"--we have enough real bugs and don't need people to come up with fake new ones just to confuse people.

[Quark.Fusion]

Originally Posted by Dark Shikari

Because pre-scenecut relies on B-adapt to not choose B-frames across a scenecut.

It's not to break previously encoded B-frames or what wrong will be?

I edited wiki page. Whats will happen with --threads 2 --no-b-adapt? last will be ignored?

[Dark Shikari]

Originally Posted by Quark.Fusion

It's not to break previously encoded B-frames or what wrong will be?

I edited wiki page. Whats will happen with --threads 2 --no-b-adapt? last will be ignored?

Yes, it will be ignored unless you use --scenecut=-1.

Having no-b-adapt with scenecut doesn't make that much sense, since if you want a single number of B-frames everywhere, scenecut will stop that (so you will lose the ability to have temporal scalability, which would be the primary reason to have no-b-adapt, I'd think).

[tetsuo55]

Originally Posted by Dark Shikari

No, again, you are making false assumptions and thus deducing false conclusions.

Just because a setting triggers a problem does not mean the setting is at fault for the problem. As I have said a dozen times already, what's happening is pre-scenecut is more aggressive on scenecut and puts IDR frames closer together.

Your device cannot handle this, yet you still refuse to stop using a min-keyint that is incompatible with it!

I don't care if you are incapable of listening to basic instructions to solve your own problems, but I don't want you spreading FUD about "bugs in x264"--we have enough real bugs and don't need people to come up with fake new ones just to confuse people.

It's not my problem(i just keep an eye on all settings that break HW compatibility)

Damn and i read it twice too, i didn't see his min-keyint in the first post. If he stops using that the file should work.

[Sharktooth]

... ill update the megui HD-DVD presets with min-keyint 4 so ppl will stop bitching ...

[tetsuo55]

Originally Posted by Sharktooth

... ill update the megui HD-DVD presets with min-keyint 4 so ppl will stop bitching ...

I will do the same

[Quark.Fusion]

Originally Posted by Dark Shikari

if you want a single number of B-frames everywhere, scenecut will stop that (so you will lose the ability to have temporal scalability, which would be the primary reason to have no-b-adapt, I'd think).

I was think so at first, but not come with idea why you may need that.

Is there any reason to have no-b-adapt with scenecut, maybe no-b-adapt should be no-b-adapt-and-scenecut?

[vmrsss]

Originally Posted by Dark Shikari

Do you still not understand? The normal scenecut is not very aggressive, so in the clips you gave me, it never put any IDR frames within two frames of each other anyways, even when min-keyint was equal to 2. Pre-scenecut did. You're blaming this, now, on a "bug in x264," when in reality x264 did nothing except exactly what you told it to do.

I do understand. You haven't read my message carefully, I am certainly not blaming x264...

[Sharktooth]

so whats the problem?

you said:

While attempting to I learn to encode x264+ac3 for a HD-DVD player (Toshiba HD-E1) by adapting MeGUI's x264:SA-HD-DVD profile to my command-line MacOSX environment, I bumped into a strange behaviour triggered by "--thread auto" as opposed to "--thread 1" which may well be a bug.

the answer is:
no it's not a bug. rise the min-keyint if you use more than 1 thread.

[Dark Shikari]

Originally Posted by Sharktooth

so whats the problem?

you said:

the answer is:
no it's not a bug. rise the min-keyint if you use more than 1 thread.

No, he should raise it even if he's only using one thread, because the fact that one thread worked fine is sheer luck/coincidence.

[vmrsss]

Originally Posted by tetsuo55

I may be misunderstanding here but.

All other settings equal:

Threads1> File works
Threads3> File breaks.

All threads those is either spread, or not spread the work over several cores/hw threads.

So its obvious that either threads1 is broken or threads3, with the latter being more obvious.

No, it doesn't follow: the streams generated with --threads 1 and --threads 3 don't need to be equal to be both correct, and their differences may just trigger bugs in the muxing software or in the decoder. As I tried to write carefully in my post, logically at the outset it could be one of four things: (1) the command line is not HD-DVD complaint (I don't believe so); (2) it is a non trivial bug in x264 (I don't believe so, DS hasn't found anything wrong with a number of samples); (3) it is a bug in the muxer (quite possible, there is some evidence of dropping frames); (4) it is a bug in the player (possible, less likely than 3).

I believe 3 and 4 are much more likely than the rest; to understand what really is the problem may lead to an improved SA-HD-DVD profile (besides addressing my vested interest with my SA player)

Originally Posted by tetsuo55

So what we need is a sample file with threads1 and a sample file with threads3 + confirmation that 1 works and 3 is broken.

I have provided several some samples in my previous posts.

Regards

[Sharktooth]

good. well, i've done it for him some minutes ago. he just need to update the megui presets.
however min-keyint 2 always worked here... bah...

[vmrsss]

Originally Posted by Dark Shikari

Note that if you want to force a specific number of B-frames in threaded mode, scenecut must also be disabled, by setting it to -1: this is because pre-scenecut does not work with no-b-adapt.

Which is exactly what I said, in a different form.

[vmrsss]

Originally Posted by Dark Shikari

No, again, you are making false assumptions and thus deducing false conclusions.

Just because a setting triggers a problem does not mean the setting is at fault for the problem.

Yes, I understand and agree entirely with that!

Originally Posted by Dark Shikari

Your device cannot handle this, yet you still refuse to stop using a min-keyint that is incompatible with it!

Well, where have you read that? I am not refusing to do anything, I am just trying to understand what's going on; I have tried all your suggestions, reported that it appears to work with --min-keyint 4, but I am not yet convinced this is the final solution, because with --min-keyint > 3 some frames are mysteriously omitted by the muxer. I conjecture there might be a bug in the muxer, not in x264! (in one of my previous posts I asked explicitly whether it is the muxer that by removing frames breaks compliance!) and asked your opinion and help to look inside those .EVO files.

Originally Posted by Dark Shikari

I don't care if you are incapable of listening to basic instructions to solve your own problems, but I don't want you spreading FUD about "bugs in x264"--we have enough real bugs and don't need people to come up with fake new ones just to confuse people.

Please, calm down! I do not wish to upset you. I am not sure you are referring to me with that; if so, please accept my apologies, as you misunderstood my statements: when did I spread FUD about bugs? I am simply trying to investigate this issue with the highest regard for x264. And I think we are making good progress towards clarifying it...

Friendly,
-vmrsss

[Dark Shikari]

Originally Posted by vmrsss

Yes, I understand and agree entirely with that!



Well, where have you read that? I am not refusing to do anything, I am just trying to understand what's going on; I have tried all your suggestions, reported that it appears to work with --min-keyint 4, but I am not yet convinced this is the final solution, because with --min-keyint > 3 some frames are mysteriously omitted by the muxer. I conjecture there might be a bug in the muxer, not in x264! (in one of my previous posts I asked explicitly whether it is the muxer that by removing frames breaks compliance!) and asked your opinion and help to look inside those .EVO files.



Please, calm down! I do not wish to upset you. I am not sure you are referring to me with that; if so, please accept my apologies, as you misunderstood my statements: when did I spread FUD about bugs? I am simply trying to investigate this issue with the highest regard for x264. And I think we are making good progress towards clarifying it...

Friendly,
-vmrsss

I apologize, I was not paying enough attention to usernames and I though it was all one person

[vmrsss]

Originally Posted by tetsuo55

It's not my problem(i just keep an eye on all settings that break HW compatibility)

Exactly. As I said, this could be a problem with HW compatibility of a particular command line, or silly bug of silly Toast9 which while muxing breaks the stream generated by x264.

Originally Posted by tetsuo55

Damn and i read it twice too, i didn't see his min-keyint in the first post. If he stops using that the file should work.

Why do you say so? It sounds as though you knew that 2 is the wrong value for that parameter, which cannot be the case. Please read my recent post, the more I raise --min-keyint the more frames are dropped by the muxer. Is that normal/expected?

That is where I started to think that this may all be a red-herring, and after all there might be nothing wrong with --min-keyint 2 nor with my device, and that is the conjecture I am trying to test at the moment...

PS. Again, lest we misunderstand each other, I am not saying x264 is at fault. I am investigating an issue with HW compatibility.

[Sharktooth]

ok, so try using a different muxer and see if the problems persists

[vmrsss]

Originally Posted by Sharktooth

so whats the problem?

the answer is:
no it's not a bug. rise the min-keyint if you use more than 1 thread.

Just trying to understand you: you speak as though you knew I was obviously doing something wrong in --min-keyint 2, which I am not as I took it from MeGUI's SA-HD-DVD profile (which is yours, if I am not mistaken).

If instead your sentence is because of what you read in this thread, I would like to remark that the statement that --min-keyint 4 solves the problem is not a proven fact, it is just a conjecture of mine which I am still in the process of checking out.

I would be very happy however to agree that this is not a bug with x264 (can I change the thread's title?), so that nobody has to feel defensive in this discussion.

[vmrsss]

Originally Posted by Dark Shikari

No, he should raise it even if he's only using one thread, because the fact that one thread worked fine is sheer luck/coincidence.

Agreed.

[Sharktooth]

i was going by logic. 2 doesnt work, 4 does... keep 4.
however, have you access to another muxer? coz if not, we wont be able to estabilish if that's a muxer problem or not.

[vmrsss]

Originally Posted by Sharktooth

ok, so try using a different muxer and see if the problems persists

Trying with the tools I have. I am having problems with mplayer, seem to be having some success with ffmpeg and VLC. Unfortunately, I don't seem able to find an alternative to generate the HDDVD_TS folder with Toast, which is indeed the weakest link in the chain...

[vmrsss]

Originally Posted by Sharktooth

however, have you access to another muxer? coz if not, we wont be able to estabilish if that's a muxer problem or not.

If I fail to find one, I will try again to look inside the EVO files generated by Toast. Incidentally, do you know of any tool for doing that?

[tetsuo55]

Originally Posted by vmrsss

If I fail to find one, I will try again to look inside the EVO files generated by Toast. Incidentally, do you know of any tool for doing that?

Do you have access to a windows(dualboot or VM)

[vmrsss]

Originally Posted by tetsuo55

Do you have access to a windows(dualboot or VM)

Yes, via Parallels.

[Sagittaire]

1) You use certainely Sonic scenarist for the mux and Sonic Cinevision encoder profil use 4 frames for minimum interval between Iframe. Perhaps simply muxer limitation here.

2) For HDDVD compliant stream all the Iframe must be IDR.

[tetsuo55]

maybe this helps??

http://forum.doom9.org/showthread.php?t=140459

[Sharktooth]

as i did for blu-ray megui presets, i ensured they will work on ALL HD-DVD compatible HW/SW, so i will keep min-keyint 4 in the HD-DVD presets.

[tetsuo55]

Originally Posted by Sharktooth

as i did for blu-ray megui presets, i ensured they will work on ALL HD-DVD compatible HW/SW, so i will keep min-keyint 4 in the HD-DVD presets.

Are you sure its needed? Dark Shikari said x264 already respects the limits required.
Also vmrsss said that although it fixes the first problem it causes a new one

[Sharktooth]

i dont care. 4 is ok and seems to work well with hd-dvd sw/hw.

[vmrsss]

Originally Posted by Sagittaire

2) For HDDVD compliant stream all the Iframe must be IDR.

is that so?, i didn't realise it before. this is not the way x264 operates, can it be controlled from the command line?

[Quark.Fusion]

Correct me if this is wrong, but:

min-keyint

Integer: 25

Sets the minimum length between IDR frames. See keyint for an explanation of IDR frames. Very small IDR ranges can reflect incorrect frame placement (for example, a strobing scene). This option limits the minimum length after each IDR frame before another can be placed.

Recommendation: Default, or whatever your framerate is.

See also: keyint, scenecut

Why you set min-keyint so small? It's not about I-frames versus non-I-frames — it's about normal-I-frame vs IDR-I-frame.

Originally Posted by Sagittaire

2) For HDDVD compliant stream all the Iframe must be IDR.

Where it from?


P.S. is IDR stays for "I Delete References"?

[kemuri-_9]

The major reason to reduce the min-keyint/max-keyint values is to increase the frequency of IDR frames to improve seeking.

since i'm not all that good with names of things, wikipedia had the answer of Instantaneous Decoding Refresh

as for x264 producing non-IDR i-frames, afaik the only way it produces them is when it's told to via a qpfile.
Would need akupenguin or DS's confirmation on this though.

[Dark Shikari]

Originally Posted by kemuri-_9

The major reason to reduce the min-keyint/max-keyint values is to increase the frequency of IDR frames to improve seeking.

since i'm not all that good with names of things, wikipedia had the answer of Instantaneous Decoding Refresh

as for x264 producing non-IDR i-frames, afaik the only way it produces them is when it's told to via a qpfile.
Would need akupenguin or DS's confirmation on this though.

If scenecut triggers but min-keyint hasn't been reached, it uses an I-frame instead of IDR.

[Quark.Fusion]

So on first frame of strobing scene x264 deny references to previous frames and drops new IDR every min-keyint? Or that case can be only with pre-scenecut?

[vmrsss]

Originally Posted by Sharktooth

i dont care. 4 is ok and seems to work well with hd-dvd sw/hw.

Unfortunately, I can't confirm yet. I spent a lot of time on this, without any real result. I downloaded all the muxer/burners with a trial period I could find, but for one petty reason after another I couldn't get any of them to accept the h264 generated by x264. So, no actual burning, no actual tests on the HW. Very frustrating.

[vmrsss]

Originally Posted by Dark Shikari

If scenecut triggers but min-keyint hasn't been reached, it uses an I-frame instead of IDR.

Does this mean that --min-keyint 1 forces all I-frames to be IDR-frames?

[Dark Shikari]

Originally Posted by vmrsss

Does this mean that --min-keyint 1 forces all I-frames to be IDR-frames?

Yes, it does.

[F J Walter]

Is it at all possible that there could be an HD-DVD compliance issue caused by using I frames which are not IDR?

Sagittaire mentioned this above but sofar nobody has confirmed or denied it.

If that is plausible, it might be worth checking vmrsss' "threads 1 without pre-scenecut" output to see if it just happens to have no I frames which are not IDR. It might also be interesting to do an encode with min-keyint of 1 (just for troubleshooting; I know it's off-spec) to see if that causes any problems with the muxer or hardware player.

[vmrsss]

Originally Posted by F J Walter

Sagittaire mentioned this above but sofar nobody has confirmed or denied it.

I found another reference to this in the context of a posting about main concept as a compliant HD-DVD encoder. I've been reading pages and pages, I dont yet have a view myself.

Originally Posted by F J Walter

If that is plausible, it might be worth checking vmrsss' "threads 1 without pre-scenecut" output to see if it just happens to have no I frames which are not IDR.

How do I tell whether or not an I-frame is an IDR-frame? Can anybody kindly explain the detailed meaning of eg:

x264 [debug]: frame=   0 QP=17.42 NAL=3 Slice:I Poc:0   I:6120 P:0    SKIP:0    size=41769 bytes
x264 [debug]: frame=   1 QP=19.90 NAL=2 Slice:P Poc:4   I:0    P:262  SKIP:5858 size=478 bytes
x264 [debug]: frame=   2 QP=25.00 NAL=0 Slice:B Poc:2   I:0    P:151  SKIP:5969 size=163 bytes

Originally Posted by F J Walter

It might also be interesting to do an encode with min-keyint of 1 (just for troubleshooting; I know it's off-spec) to see if that causes any problems with the muxer or hardware player.

Will do. (I am currently experimenting with Nero; I finally managed to get it to accept my h264 raw stream, but still can't prevent it from re-encoding them.)

[kemuri-_9]

Originally Posted by vmrsss

How do I tell whether or not an I-frame is an IDR-frame?

in the .stats generated by x264
i.e.

in:500 out:500 type:I q:16.45 itex:165538 ptex:0 mv:23069 misc:337 imb:1350 pmb:0 smb:0 d:s;

a type of upper case 'I' is an IDR frame
a type of lower case 'i' is a regular i-frame

so can run grep/find on the .stats looking for "type:i" and if any hits occur, it has i-frames:
windows: find "type:i" xxx.stats
linux (or with gnuwin32 tools for windows): grep type:i xxx.stats

[vmrsss]

I think we can write the bottom line of this thread.

I have finally managed to feed another muxer with x264's output, and guess what: the muxer likes x264 and the toshiba hd-e1 plays the resulting hd-dvd perfectly, with --min-keyint = 1, 2, 3, ..., 14 (and --bframes 3, --ref 4, --b-pyramid and all the rest), 720x576, 1280x720, 1440x1080 and 1920x1080.

I suppose this finally proves that it was all due to a bug with buggy Roxio Toast; not the first time I'm forced to make this observation after having wasted precious days. Indeed, the fact that it autonomously dropped frames when muxing was a strong indication that something wasn't quite right. Quite why then Toast appears to mux well h264 streams generated without --pre-scenecut and fails hopelessly with --pre-scenecut I will probably never know. Yet, quite frankly, I don't give a toss. I am going to buy something else anyway. (A just in time decision, as next week it is back to work, and at least till Xmas I won't be able to spend this sort of time on my encoding hobby :-)

Thanks everybody for paying attention to me, and apologies for wasting your time too. Please feel free to round this up, if there is anything you want to add/ask.

-vmrsss

[Sharktooth]

cool, so i can set min-keyint back to 2 into the megui hd-dvd presets... damn toast...

[vmrsss]

Originally Posted by Sharktooth

cool, so i can set min-keyint back to 2 into the megui hd-dvd presets... damn toast...

Indeed. Mind you, we have not really learned more about what is and what is not HD-DVD compliant: also --min-keyint 1 (as suggested in a post by Sagittaire) works well (and I am using --ref 4 --bframes 3 --b-pyramid just as well).

[Sharktooth]

i wont include b-pyramid in any hardware compatible presets until it gets properly fixed.