Reporting Strange behaving in X264. Possible bug. Please read

[OvejaNegra]

OK
First sorry for the english.
Now the problem:

I was very happy encoding with some settings, until y decided make two encodes using some new settings.

First, i encoded Gladiator and i looks good. I was in a hurry for low HD space so i deleted the vobs and all the AVS script and the stuff.
Then, encoded Robocop.
This time, since robocop is very noisy, i was checking the final product, and noticed something wierd:
I'm having LOTS of frame drops and high CPU spikes. FFdshow drops lots of frames. This is new for me. I'm not doing anything new (exept one thing)

For avoiding innecesary posts:
Using MPC:
Change the renderer does not works
Using internal MKV splitter does not works
I have all internal filters disabled

FFDshow: Last version updated today, nothing
Disable queue, nothing
Disable subtitles, nothing
Disable h264 deblock, nothing
Disable some color spaces , nothing
Disable drop frame on delay / skip deblock on delay only slows down the movie (not enough CPU), no frame drops, everything goes slower.

I'm using last version of haali splitter
Finally, using mplayer : nothing, frame drops or slow video (if i disable frame drops)

I have intel p4 HT (3GHz * 2). 512 MB ram. Radeon 9250 (!!PCI!! not PCI Xpress) video card but works fine (many videos tested)

Now: Encoding side:
I'm encoding in a HP Proliant server (of course during nights), P4 HT (2 cpus), 1 GB Ram.

Using last stable AVSynth with last stable DGIndex.
Using x264 667 (Yes i know, i forgot it) but i have other videos encoded with that and are OK.

Now, i always used auto thereads (x264 uses the 4 cores), that's not new.

THE ONLY NEW thing i'm doing is: Raise the number of bframes to 16. Until yestarday i was using 2, but some one told me (here, i dont remember were) that using 16 is better because gives all freedom to the codec to make better b-frame insertions.

Now the commmand line.
This are the seetings i use always (the only new was raise the b frames to 16 and increase merange to 32, but tha's only for in the encoder):

--pass 2 --bitrate 2795 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-rdo --bime --weightb --subme 7 --analyse all --8x8dct --qpmin 12 --me esa --merange 32 --threads auto --thread-input --progress --no-psnr --no-ssim --output "C:\BCK\On Process\Chaps\ROB\paso2.mkv" "C:\BCK\On Process\Chaps\ROB\MainMovie\DVD_VOLUME\VIDEO_TS\LOAD.avs"

Robocop is rolling VERY bad, and sadly, Gladiator too.
Tellme any suggestions. I'dont erase anything or redo the encodecs until know what's happening.

If i need provide some sample of the encoded material jus't tell me how.

If a capture with the FFdshow OSD on may help, just tellme.

As side note: The playback perfomance of my machine stills the same, i don't think is on my side. I think is some encoding issue.

Thanks

If this post should not be here, just tell me were.
Until this i was encoding with out problems.
Sorry for the english.
Thanks


EDIT:
HEY!! new!!
Mplayer is reporting this:
Too many video packets in the buffer: (461 in 8407881 bytes).
Maybe you are playing a non-interleaved stream/file or the codec failed?
For AVI files, try to force non-interleaved mode with the -ni option.

And this:

Cannot seek in this file.

Does it helps?

[Sagekilla]

It could very well be that your system is not up to decoding the video files. Unfortunately P4's are not exactly very "good" processors overall so it seems kind of natural you're having decoding issues.

Can you upload a short clip (1 minute long) where you experience a lot of frame dropping so I can see if I have the same problems? Like I said, it may very well be that your system is not up to the task to decoding the videos, and not the actual encodes. Even though, as you've said, you have videos that play perfectly fine, Robocop and Gladiator could be two instances where there's a lot of frame referencing and therefore high CPU usage. So, I'd like to see a small clip to confirm this possibility.

[OvejaNegra]

Sure, how i cut the sample?

I have some othe encodes that play just fine (movies, animes, etc)
Even in my old celeron 1.2 GHz, socket 370, 512 MB ATi rage 128 (Directx7) those videos play FINE. I use always the same settings.
The only differences are,
a- the new settings
b- These movies were done in 2 pass, the rest of my encodes are 1 pass CRF 16 or 18. (with higher bit rate)

How i cut the clip?

[CruNcher]

--bframes 16 <- i would bet this is the problem :P the decoding complexity is for sure higher now couseing the decoding problems, now you would have to compensate this by useing a faster decoder like CoreAVC (or Hardware accelleration Nvidia/Ati) or reduce the complexity try --bframes 3 (my advice is --bframes 1), people really underestimate this to much imho, for sure before these settings you have been on the edge of decoding possibilities of your system and that might have been only the case for the Video part you still have to calculate Audio decoding complexity and parseing overhead to this and voila to complex for playback (frame drops) with libavcodec so with the new settings your complexity seems to high, it's better now to rebalance it to the encoding time (slower encoding time == higher decoding speed) reduceing bidirectional predecition grant's this encoding will be slower tough 2 other possibilities to lower the complexity of a AVC stream would be --no-cabac avoiding the 8x8 transform (no High profile) or lowering the reference frames.

try this
--pass 2 --bitrate 2795 --stats ".stats" --ref 3 --mixed-refs --no-fast-pskip --bframes 1 --b-rdo --weightb --subme 7 --analyse all --8x8dct --qpmin 12 --me esa --merange 32 --threads auto --thread-input --progress --no-psnr --no-ssim --output "C:\BCK\On Process\Chaps\ROB\paso2.mkv" "C:\BCK\On Process\Chaps\ROB\MainMovie\DVD_VOLUME\VIDEO_TS\LOAD.avs"

[Sagekilla]

Well, you're using mkv for the container, correct? I believe the mkvmerge GUI allows you to cut the files. Just open the mkv file, goto "Global," enable splitting. It's a bit odd though, since the way it works it'll output it in chunk sizes if you choose "After this duration"

Edit: Nevermind, if you simply put in the two times (beginning, end) when you're experiencing it you can cut the mkv. I.e. use 00:32:00, 00:33:00 to cut 1 minute of video. You'll end up getting a bunch of files depending on the length of the video (If it's at 1 minute in the movie, you'll get over 50 mkv files for a 1 hour movie,) so you can just cancel it early to prevent this.

[Manao]

--bframes 16 <- i would bet this is the problem :P the decoding complexity is for sure higher now couseing the decoding problems, now you would have to compensate this by useing a faster decoder like CoreAVC (or Hardware accelleration Nvidia/Ati) or reduce the complexity try --bframes 3It's a myth. BFrames are barely more complicated than PFrames to decode. And there certainly won't be any difference in speed between -b 16 and -b 3.
Robocop and Gladiator could be two instances where there's a lot of frame referencing and therefore high CPU usageAnother myth. Reference frames increases the needed memory bandwidth, but not directly the CPU usage ( indirectly, the video will be slower to decode though ). So they make it slower to decode especially if you have a slow RAM.

What's the resolution of the movie ? A P4 should be more than able to handle 720x576 @ 3mbit, but it might choke if the bitrate gets too high locally ( lets say 10 mbit might be a problem ). You can check the local bitrate with FFDShow's OSD.

If the whole film isn't properly being decoded, the problem is elsewhere ( neither in the encoder nor the decoder ). Failing video drivers ? Failing harddrive ? Disabled video acceleration ?

[imcold]

Cannot seek in this file.
Give remuxing a try.

[OvejaNegra]

OK:
I'll try to answer everybody:
Thanks for the atention:

Cruncher: I'll do another encoding but with two Bframes (as always ) only to check that.
Manao:
I checked all what you say and it's fine. I can play onther movies encoded with those settings and everithing is fine. The reslolution
is 704X480 (normal anamorphic encoding) FILM Dvd.

Still having frame drops
The OSD:
Accurate dblockin: yes
Frame type B QPEL
CPU load 66%
Mean Quantizer 19
Input bitrate 2717
Need more info?
Te video only drops is some places, the rest is fine
Imcold: I remuxed several times, even without audio or subtitles. Even the original MKV from X264 have the same problem

I'm a little worried about this:
"Too many video packets in the buffer: (461 in 8407881 bytes)."

How works the video buffer? I'm using default settings. Remember please, that when i encode animes or series i use CRF. But this time i'm using 2 pass. Does the buffer influence here?

Thanks

[OvejaNegra]

Sample Here:
http://www.sendspace.com/file/4qdc7w
The part when the title comes closer have problems.
The part of the flying screens allways have problems.

[Dark Shikari]

Sample Here:
http://www.sendspace.com/file/4qdc7w
The part when the title comes closer have problems.
The part of the flying screens allways have problems.
No problem here... plays perfectly fine with CoreAVC.

[OvejaNegra]

Posting the log of X264:
Looking for job processor for job...
Processor found!


------------------------------------------------------

Starting job job1 at 09:38:39 p.m.
Starting preprocessing of job...
Preprocessing finished!
encoder commandline:
--pass 1 --bitrate 2795 --stats "C:\BCK\On Process\Chaps\ROB\MainMovie\DVD_VOLUME\VIDEO_TS\LOAD.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-rdo --bime --weightb --subme 7 --analyse all --8x8dct --qpmin 12 --me esa --merange 32 --threads auto --thread-input --zones 139614,150705,q=24 --progress --no-psnr --no-ssim --output NUL "C:\BCK\On Process\Chaps\ROB\MainMovie\DVD_VOLUME\VIDEO_TS\LOAD.avs"
successfully started encoding
Processing ended at 05:19:37 p.m.
----------------------

Log for job job1

avis [info]: 704x480 @ 23.98 fps (150706 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2
x264 [info]: slice I:1756 Avg QP:15.44 size: 42801
x264 [info]: slice P:67051 Avg QP:17.48 size: 20060
x264 [info]: slice B:81899 Avg QP:19.09 size: 9007
x264 [info]: mb I I16..4: 30.0% 44.3% 25.7%
x264 [info]: mb P I16..4: 5.9% 11.6% 5.1% P16..4: 26.4% 17.6% 7.4% 0.7% 0.5% skip:24.8%
x264 [info]: mb B I16..4: 0.5% 1.1% 0.6% B16..8: 27.0% 3.2% 7.8% direct: 6.2% skip:53.5%
x264 [info]: final ratefactor: 15.91
x264 [info]: 8x8 transform intra:50.6% inter:41.9%
x264 [info]: ref P 67.6% 14.6% 9.0% 4.8% 3.9%
x264 [info]: ref B 84.5% 7.6% 3.9% 2.2% 1.9%
x264 [info]: kb/s:2746.3

encoded 150706 frames, 2.13 fps, 2746.44 kb/s

----------------------
Starting postprocessing of job...
Job completed successfully and deletion of intermediate files is activated
Postprocessing finished!
End of log for job1
-------------------------------------------------------

What is skip:53.5%???

Forget the slow encoding, i have time.

[Dark Shikari]

x264 [info]: mb B I16..4: 0.5% 1.1% 0.6% B16..8: 27.0% 3.2% 7.8% direct: 6.2% skip:53.5%
This means that 53.5% of all B-macroblocks were "skipped". There are three ways of encoding a block:

1. Intra--a totally new block.
2. Inter--based on a previous block, perhaps using a motion vector to represent where that block has moved from.
3. Skip--exactly the same as the block in the previous reference frame. No change at all.

1 uses the most bitrate, and 3 uses the least (none), so skipping blocks is advantageous whenever possible without sacrificing image quality.

[nm]

Sample Here:
http://www.sendspace.com/file/4qdc7w
The part when the title comes closer have problems.
The part of the flying screens allways have problems.
Works here too with libavcodec-based players (MPlayer and VLC), on an Athlon XP 2400+. CPU usage is less than 40%.
Looks like you have a decoding problem.

[OvejaNegra]

My cpu stays in less 30% (using ffdshow)
When a frame drop occurs only reach 60%
Never have 100 or even 80%

With Mplayer th cpu stay 25 % +- and the frame drop start with +60%. If i disable frame droping the video plays slower in that part and i lose audio synch.

If i have a playbak problem. Ok. Maybe i have an old machine. But why was ok before with those settings?

What is the buffer. I know what is a buffer but how it is related to x264? Or video playback?

The default settings are ok?

[Atak_Snajpera]

I checked your sample and everything was fine.

[OvejaNegra]

Please tell me what software (ffdshow + ? or mplayer)
And your hardware
Thanks

UPDATE:
I'm doing constant quantizer testings with Haruhi Suzumiya and i have the same problem, but this time i changed som things and now is better:

Lowering the b-frames to 2 again reduced the frame dropping
Disabling Bidirectional M.E ALLMOST solved the problem. But still happening.
Disable B-Frames Solved the problem.


Of course, all my previous encodes have 2 b frames and the same settings and roll well. Maybe the problem is related to the b frames.

Now i have a more precise diagnosis:

PANNIGNS KILL THE CPU:

When a pannig or a zooming scene start, the cpu (30% or less) start to climb (60% and more), sometimes when reaches +60% i have the frame drop. Sometimes i have the frame drop when reaches +80%. And during a VERY large pannig it reaches 100%. Of course i have a complete freezed picture.

The ROBOCOP title coming to the screen is like a zoom (in the sample i posted) and the part with the flying screens beguins with a zoom in all teh screens.

X264 does not have GMC.

Any suggestion?
I'm going to disable HT in the bios and test the samples in other computer (much older).

[nm]

My cpu stays in less 30% (using ffdshow)
When a frame drop occurs only reach 60%
Never have 100 or even 80%
Are those per-virtual-CPU figures or overall load? Since you had HT enabled and you're using a single-threaded decoder, 60% CPU usage could mean that one virtual CPU is under full load and the other runs some less-intensive threads. You could check the per-virtual-CPU loads to see if this is what really happens, or just disable HT as you said.

If i have a playbak problem. Ok. Maybe i have an old machine. But why was ok before with those settings?
The machine is otherwise fine (I can easily decode 720p with my Athlon XP), but the video card may be a bottleneck. Usually PCI cards do work fine with SD video, but maybe there is an acceleration issue of some kind. Or you have a pure software problem. Which MPlayer and ffdshow versions did you try?

[foxyshadis]

Experiment with the VBV settings until you something that your CPU can reliably handle, and the stuttering will go away. You can see why here:

http://foxyshadis.slightlydark.com/random/robosize.png

xxl made a few ffdshow builds with the experimental avc multithreading, those could help, and of course coreavc will. Your video is just barely on the playable side for your cpu with single-threaded ffdshow, so if you use it you need to enforce VBV so that it doesn't spike like that.

[Taurus]

@ OvejaNegra
Just tested your sample on an old AMD Athlon Xp 2200+
Win XP latest directx updates.
No stutters, fluid motion, cpu @ about 50%.
Latest clsid ffdshow compile and Haali Media Splitter.
With CoreAVC about 10 - 15% less cpu load.
Just to mention: Nvidia Geforce4 TI 4200 AGP8x Video Card.
Old stable drivers.
So, there is something burked at your side, either hardware or software side.

Cheers

Taurus

[OvejaNegra]

Well:
I think the problem is solved (need more testing):

In my old celereon 1.2 GHz all samples play just fine:
The only difference: ATI Rage 128 AGP4x. I think it was an issue in the video card. But i don't know why. I know PCI is slow, but we are talking about a common resolution video, not HD or something like that.
I'm using last haali and ffdshow on both machines (but VISTA in the one with the ATI9250, so i'm using the windows drivers).
Mplayer is 1.0 RC1.3.4.2
Disable HT was not usefull, the CPU behavior is the same.
Maybe FFDSHOW is not using both cores.

Recommend some settings for the buffer please
(And some lessons about the buffer in X264 will be fine)

The Core AVC page does not open in my country. I'll talk to my uncle. Core AVC must be very fast. Worth the money? If yes, i'll buy it.

foxyshadis: is that Chi/ Freya from chobits? :-)

Why the scenes with pannig or zooming are so hard? H264 uses some special tool in those cases?

I'll try the HT ffdshow. Thanks.

[Sagekilla]

Yes, CoreAVC is VERY fast especially when compared to a hardware decoder. It's well worth purchasing, as it let me decode 1080p and some difficult 720p content.

Also, I don't know why panning/zooming is difficult.. It may because there's fewer skipped macroblocked and more referenced frames but as someone pointed out, more referencing usually isn't that huge of a deal, even though it is more computationally expensive then simply saying "Don't do anything to this block."

Sorry, I'm not an expert on it so I can't say why it is.

[foxyshadis]

Since a P4 should otherwise be powerful enough to play that, you're probably on the right track about the drivers. They probably use an inordinate amount of CPU to do color conversion, bus transfer, or something like that. CoreAVC would definitely be fast enough to play everything at full speed, if ffdshow only stutters occasionally. Something else I forgot, though, is you might look into the ffdshow settings and make sure all the filters are turned off, some of them can really soak the CPU.

You can try --vbv-maxrate 4000 to start with, that's just a guess. That's pretty close to your average, at least in that scene, so you may see some wavering in quality if you do that. (The peak of that clip is over 6000 kbps, though.)

It probably happens because panning and zooming might take up a lot of extra bitrate for good quality on that movie, just because the matching is imperfect, especially if anything is moving or it's grainy, and lots of extra bitrate = lots of extra cpu when you use cabac.

(Close, it is actually this (http://kei-garou.net/blog/archives/2006/10/4_2.html) angel. My sig would make it obvious, if I could have it. She still frightens me.)

[OvejaNegra]

Problem Solved:
One RAM Broken. Changed and now everything is working. With 16 B-Frames and all the stuff.
Thanks to everyone for the help!!!!

If someone have a problem like this one, try checking the Ram. Disable the superfetch/ prefetch service give me the idea (because everything improved). You don't have to change your Ram to make a test. Just go to the Bios and disable Any FAST BOOT setting. That will test the memory during the startup.

:-)

Now i can continue with my dup-frames-in-anime-tests.
PD: The CPU use is now under 30% with FFDSHOW.
Who was talking bad about my CPU :-] ?
But i'm gonna play a little with the settings of hte buffer, to avoid those peaks. Because the cpu raise to 50% - 60%. It does not drop frames (like before) but will drop on less powerfull CPU like my 1.2GHz or my 1.4 GHz laptop (maybe not).

Thanks to all!!!!!!!
Mina Arigatto!!!

[Sagekilla]

That seems odd, because even though the fast boot option in BIOS typically disables some of the more time consuming options, like memory check, I don't see why that would make it work properly.

[OvejaNegra]

Sorry for my terrible english
Disable fast boot options enable (in my board) some aditional testing like full memory testing. That was the way to discover the problem (broken memory), it's changed (the DDR memory) and now it's all fine.

[foxyshadis]

Do you have a board with ECC memory? That's pretty interesting, I guess your cpu use went way up trying to correct it when it partly failed, and low other times. Usually things just crash left and right. Cool.

[R3Z]

Memtest86 and a floppy drive is a swiss army knife :)

[OvejaNegra]

"Memtest86 and a floppy drive is a swiss army knife"

You are right

Well, i'm not sure right now about if DDR or DDR2. It's 533MHz so must be DDR2.

Everything its ok now. The memory is now on my museum (wit my old 486, pentium 133 and Pentium 200 MMX. Those 3 still works)