open-gop patch for x264 (committed)

[Trahald]

Does this patch solve "frame pulsing problem" with small --keyint (for example 24) and low bitrate?

In my (limited) testing, yes.

[jpsdr]

I've tested with .20 patched version, 1rst pass finished without problem. I've stopped the 2nd pass at around 50%, because i needed my PC to do more urgent work, but, in my concern, the problem seems solved.

[jpsdr]

Any patched version with v1570 ?

[Midzuki]

I'd rather wait until the patch has become "official". :)

[Trahald]

@shon3i

where you able to do that test

[shon3i]

I been busy these days, but i am almost finish all task i want to test, i finished with 1080p24 source with qpfile used, and it's pass ok, now i doing same with different source which require Interlaced and Pulldown.

[shon3i]

Ok here is one problem. Now i use "Long" (2 second) GOP because my max bitrate is under 15000kbps i use keyint 48 for 24p source, MUI Generator on end show "The number of frames displayed in GOP exceeds the limit"

[Trahald]

Only with open GOP? I'll look at it when I get home.
Do you have open GOP source not from the patch with 2 second gops that you can check mui generator against?

[shon3i]

Only with open GOP? I'll look at it when I get home. Yes, stream with OpenGOP and 2Sec GOP fail to index into MUIGenerator.

Yes i have stream with 1 Second GOP i will send you both samples to examine, when i get home.

[jpsdr]

I think Trahald is asking for (if i've not misunderstood) :
1) A 2 seconds open-gop source comming from somewhere else than x264+open-gop, and wich is working fine with MUIGenerator.
2) If a 2 second gop source encoded with the same version of x264 but without the open-gop is working fine

[Trahald]

I think Trahald is asking for (if i've not misunderstood) :
1) A 2 seconds open-gop source comming from somewhere else than x264+open-gop, and wich is working fine with MUIGenerator.
2) If a 2 second gop source encoded with the same version of x264 but without the open-gop is working fine
yes.. thank you..

if you can test something like those it would be great. mui generator is known not to be perfect. (not to say its not the patch)

[Trahald]

Welp.. I made a quick sample with closed gop on unpatched x264 (23.976 fps/48 frames a gop) and got the same error on mui generator. oddly, 23.976/47 frames works. Then i tried 24 fps and used gop of 48 frames and it worked. It tells me its not related to the open-gop patch , however its something id like to solve. does anyone have any short h264 encodes (preferably from a bluray but if not just anything other than made by x264) that has 48 frames between gops and is 23.976 fps? it can be opened or closed gop.

[shon3i]

I made a quick sample with closed gop on unpatched x264 (23.976 fps/48 frames a gop) and got the same error on mui generator.Hmm, i never have problem with this, i always use 2 sec GOP for mine BD5-9 backups, with recent unpatched builds from x264.nl. Please test again, even Dark's x264 Demo Blu-Ray use 2 second gop without problem.

I am right now upload sampes to mediafire.

Anyway i can made encode with Ateme encoder.

[shon3i]

Ok, here is archive http://www.mediafire.com/?yd2zqy5zgo2, that contains samples

1 sec closed GOP = PASS
1 sec open GOP = PASS
2 sec closed GOP = PASS
2 sec open GOP = FAIL!

I am currently not able to encode with Ateme, i must finish some work, i will encode tonight.

[Trahald]

Tried again. same error. (used a different file name to make sure i didnt test the wrong file)

BTW the demo disk has no 23.976 fps tracks... any other frame rates are not effected (24 fps is fine as 2*24=48)

[shon3i]

BTW the demo disk has no 23.976 fps tracks... any other frame rates are not effected (24 fps is fine as 2*24=48)
Yes but mine are prue 23.976 or 24000/1001.

Not only me, here many ppl use 2 sec closed GOP fine with scenarist, sony blu-print.

Btw, i use last Rack04 buld, maybe that is different from yours?

[Trahald]

Ugh.. i see.. its because i set 23.976 instead of 24000/1001 in my tests. mui generator couldnt deal with the odd framerate.

OK.. i think i see the issue. I use the actual frame-number to figure out when to put a i-frame. This results in the POC being exactly 48 frames apart but with a variable b (b-adapt > 0) the physical frames can end up more than 48 frames apart. I didnt have any streams that were open gop and variable-b length so I assumed POC counted. (I asked around a bit and the consensus was POC.) I can change it to physical distance.

do you have access to streams that are open gop and variable b? id like to know see what they do.

[Trahald]

This patch accounts for written distance between key frames w/opengop and a variable b . Should pass mui generator. Please test to make sure i didnt break anything.

[moviefan]

I cannot apply the patch to the r1583. "Hunk #2 FAILED at 1226", "1 out of 2 hunks FAILED -- saving rejects to file common/common.c.rej"

Is your correction especially relevant for Blu-ray encodes or was open-gop generally broken (so also for software decoders)?

[Trahald]

its really only an issue if you do 48 frame encodes and even then probably not. (it could effect 24 frame encodes but chances are the local bitrate was under the 2 second threshhold and you were safe) Having said that, the hard limits are enforced by muxers. it is for assumed real limitations in hardware , but usually the playback hardware is more resilient. This would not effect software at all.

remember any encodes made with the patch before its fully tested are experimental.

[Trahald]

I hand patched 1583 and rediffed it for you. (i was working from 1564)

[shon3i]

Ok, i am now wait for build. Btw, tomorow i will send you samples, i get all required things.

[jpsdr]

Tested on a 158000 frames video, 720px23.967fps with keyint of 48, pass Megui Ok. Used the last x64 rack04 build.

[Blue_MiSfit]

What kind of subjective quality or metric improvements does Open-GOP give in typical 1-2 second GOPs?

I capture with a 1 second GOP, and some of my clients need 2 second GOPs for deliverables. Oh, and there's that whole BluRay thing :D! I'm very excited by this patch polishing up!

Thanks guys
~MiSfit

[Trahald]

I really havent done alot of testing. As i went along i did some metric tests (read that tossed in --ssim and tested against a vanilla x264 w/closed gop) and generally open gop scored better than closed. A couple of visual tests showed reduction in keyframe pulsing. Once its done I may do some more tests but so far Im happy with the results. Anyone is free to make some metrics now of course but the final patch may change again.

[Atak_Snajpera]

Is this normal that seeking on PC is alot slower than with closed gop keint=fps*10 ?

[Guest]

Is this normal that seeking on PC is alot slower than with closed gop keint=fps*10 ? It depends on how the seeking is implemented. For example, if you want to be able to seek to the leading B frames, then you have to go back one GOP to start decoding and with that keyint at 30fps, you have to decode 300 extra frames to get started. On the other hand, if the seek skips leading B frames then there is no reason it should be slower.

[Trahald]

In addition... Some playback software and muxing software will need to adapt to open GOP. The program streams x264 creates muxes the iframes as keyframes but other software may not acknowlege it. The stream will appear as one large GOP. This shouldnt happen with ts or m2ts (bluray) since open GOP isn't unusual there but some mp4 and mkv muxing software isn't use to it.

[moviefan]

If I have muxed open-gop x264 streams already to mkv, will the raw stream - if I demux again to raw - broken so that I cannot remux to an updated mkvmerge version that can handle open-gop correctly? The demuxed raw stream is not bit-identical to the originally encoded raw stream for Blu-ray compliant streams (that's the reason why I'm asking).

[Trahald]

v26 some small changes, mostly cosmetic.

[jpsdr]

Bad news unfortunately.
I've encoded 12 files 720px23.976fps with the following command lines :

@echo off

SET E_SRC=%6%1.avs
SET E_DST=%3%1.264
SET STAT_FILE=%1.stats
SET TUNING=%4
SET LOG_FILE_1=%1_log_1.txt
SET LOG_FILE_2=%1_log_2.txt

REM Set of max bitrate (ici le bitrate max)
set MAX_BR=15000

REM Set of Buffer (ici le buffer)
set BUF_BR=30000

REM 1ère passe
x264_x64.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 1 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 48 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --subme 9 --trellis 1 --me "umh" --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --videoformat "ntsc" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output NUL %E_SRC% 2> %LOG_FILE_1%

REM 2ème passe
x264_x64.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 2 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 48 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --videoformat "ntsc" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output %E_DST% %E_SRC% 2> %LOG_FILE_2%


Note : Bitrate was 4529.

The 7 first were done with the 1629M Rack 04 version v22 open gop.
The 5 last were done with the 1629M Rack 04 version v26 open gop.
For the 7 first :
4 are Ok with Scenarist MUI Gen.
3 have "Error : The number of fields (frames) displayed in a GOP exceeds the limit".
For the 5 last :
2 are Ok.
3 display an error.

[shon3i]

@jpsdr it's already known, Trahald make possible solution but isn't work every time.

problem is in combination of keyint 48 and openGOP.

The demuxed raw stream is not bit-identical to the originally encoded raw stream for Blu-ray compliant streams (that's the reason why I'm asking). And you will never get same stream, because calcualtions of SPS, AUD, HRD data depend on container. If you target is Blu-Ray, never pull stream through mkv or mp4, because will destroy or change calculated data.

[jpsdr]

Euh... Why are you talking about mkv/mp4 when i'm doing raw (.264) streams ? Or maybe this part was not an answer for me ?
Wasn't the point of the fix to make it working, or do you imply that it will never be possible to combine opengop and keyint 48 in any encoder (not specialy x264) whatever the solution/algorithm implemented ?
My purpose here is to notify of the problem, but now, if there is no solution...

[shon3i]

Or maybe this part was not an answer for me ?yes this part is for moviefan :) i quote him :)

Wasn't the point of the fix to make it working, or do you imply that it will never be possible to combine opengop and keyint 48 in any encoder (not specialy x264) whatever the solution/algorithm implemented ?It will be possible, but need more time, until that use it with short gop (24)

[jpsdr]

MUI Gen show pop-up GOP error always at 99% or 100% of the progress bar. I don't know if MUI Gen show pop-up always after doing all the file, or when error is found.
To check this, i'l encode a small file (around 1500 frames) with keyint of 100, to see if the progress bar information is relevant to the position of the error. But i'll not be able to test this before several hours. If in the meantime someone (maybe you shon3i ?) can confirm that MUI Gen report the error as soon as it's found, and so the progress bar information is relevant, it could help me.
If the problem occurs in fact on the end of the file, here some information for you Trahald, you could eventualy use to check :
- Maybe there is simply some problem when hitting the end of a file.
- The end of my files are fadding black, with if i remember properly, black going for around 50 frames. So, maybe, when video is all black, and maybe (or not) combined with being at the end of the file, it triggers something.
- I'll test this evening to encode the last 2000 frames of my files, see if problem can be reproducted on small files, wich will allow me to produce sample i'll be able to provide, because providing 4GB files is not something realy easy...

[Trahald]

The current version was updated by kemuri_9 A few days ago (let's call it v27) it is the one being considered for committing. I'll see if I can post the patch but I'm at work. As far as temporary fixes go, you can use something like 48-whatever your bframe setting is. If you have bframes 4 then use 44.

[jpsdr]

Ok.
First, i've been able to reproduce the problem on a small file, but... Problem occurs only when the qpfile is used.
Here you can get sample (both results with and without qpfile) :
http://rapidshare.com/files/396744909/Sample_x264.rar.html
If you can post the patch it will be VERY usefull, as i can very quickly test it.

[Trahald]

Thank you very much for testing. I was able to duplicate the issue as well. And yes the problem is the qpfile. if the requested I-frame placement was just past the end of a GOP it could cause a long gop. This fixes it in my tests.

[shon3i]

So that basicly mean we cover all situations :)

Thanks Trahald.

[jpsdr]

I just get the rack04 builds, results will be in 3-4 days, when encodes will be finished.

[moviefan]

Trahald: Did you change anything between revision 22 and revision 26 that might have fixed the artefacts that randomly appear after seeking, shown in the following picture?

http://a.imagehost.org/t/0654/screenshot.jpg (http://a.imagehost.org/view/0654/screenshot)

When I mux x264 r1629 (+ open-gop r26) streams to mkv with mkvtoolnix 4.0 there are no artefacts after seeking anymore. I use keyint 240, min-keyint 24, preset placebo and a qpfile to insert I-frames at chapter marks. Encoding has always worked fine without any errors or so. Remuxing with the latest mkvmerge hasn't fixed the issue for previously encoded movies, so I wonder if it was you who fixed this and thus whether I have to reencode all "broken" ones or not.

[jpsdr]

The encode of the 4 first of my 12 files with rack04 1629M (og v28) pass MUI Gen Ok.
Still going (with rack04 1643M this time).

[Trahald]

[QUOTE=moviefan;1407214]Trahald: Did you change anything between revision 22 and revision 26 that might have fixed the artefacts that randomly appear after seeking, shown in the following picture?

http://a.imagehost.org/t/0654/screenshot.jpg (http://a.imagehost.org/view/0654/screenshot)

edit -- actually there was a change. intra-refresh uses the code from the earlier open-gop patches but changed a little. I didnt adapt for that fully until rev 23. so thats probably it. its not a problem with the bitstream really , so give me a week. i'll whip up something that will rewrite the recovery points leaving everything else alone so you dont have to reencode.

[moviefan]

If that's not too much of an effort for you, I would really appreciate it! Would you still like to have a sample that contains a chapter mark? If so, how can I you cut the mkv file (or the raw stream) so that the sample starts with an I-frame (I think this is necessary, right?)?

[Trahald]

http://www.mediafire.com/?2b0nn2nztjj is the file and the unoptimized and ugly source http://pastebin.org/325230 (name it blah.c ) if you want to compile it. Usage 'blah raw_src.264 raw_dest.264'

this only works on raw h264 files so demux/repair rp/mux

and.. Nah.. i dont need the clip anymore.. but.. here is a way to cut it.. You demux it to raw h264. Load it into dgavcindex. just load it. select a small viewing range around a problem spot (you use the [ ] brackets to set your start / end). and hit save project and demux video.

[moviefan]

Hm, something is wrong! I still had some of the raw streams encoded with x264 r1602 (open-gop r22) and muxed them with mkvtoolnix 4.0, but it showed artefacts. Also running a demuxed stream through your tool didn't change anything. Do you have any clue what's wrong? I'm currently encoding one of the broken movies with the latest x264 revision patched with the latest open-gop revision (which produced a perfect output last time...). It'll take a few days but I'll report on the output when it's finished.

[rack04]

Hm, something is wrong! I still had some of the raw streams encoded with x264 r1602 (open-gop r22) and muxed them with mkvtoolnix 4.0, but it showed artefacts. Also running a demuxed stream through your tool didn't change anything. Do you have any clue what's wrong? I'm currently encoding one of the broken movies with the latest x264 revision patched with the latest open-gop revision (which produced a perfect output last time...). It'll take a few days but I'll report on the output when it's finished.

Look though the current patches thread and see if the artifact encodes were done with the patched version that I had a typo on the patch.

[jpsdr]

The encode of my 12 videos files of around 160000 frames each finished yesterday, and were all accepted on MUI Gen, and no error during MUX.

[Trahald]

The encode of my 12 videos files of around 160000 frames each finished yesterday, and were all accepted on MUI Gen, and no error during MUX.
Excellent. Thank you for testing.

[moviefan]

Look though the current patches thread and see if the artifact encodes were done with the patched version that I had a typo on the patch.

It might well be the case that I used a patch with a typo. I have successfully encoded a second video with open-gop and muxed it with mkvtoolnix 4.0. There are no seeking issues anymore. Thanks for everyone's support and of course Trahald's great patch (soon official feature :))!