Tool to set keyframes for ogg chapters

[zulu]

Hi Everybody!

As most of you know there is an issue when seeking to chapters
in an ogg container: You import dvd chapters in your ogg file to be able to
jump to exactly the same movie positions as on dvd.
But unfortunately you'll rarely get to the expected point, due to the fact
that the frame at this point has to be a keyframe.
Since the codec you encoded the video with decides where to place keyframes
you are pretty much lost. Until now. :)


I've written a little tool to solve the problem described above (at least for XviD users).

It works as follows:

1. Run the 1st pass of the encoding process with vdub.
2. Start my program and specify the statsfile of the 1st pass as inputfile and a new statsfile as outputfile.
3. Add the desired chapterpoints to the chapterlist by hand or import them from a chapterXtractor file.
4. Select the framerate of your movie.
5. Press "start" to write the new statsfile.
6. Replace the old statsfile (or back it up ;)) with the new one.
7. Go to VDub and start the 2nd pass.

Thats it. If the 2nd pass has finished start Koepis' Oggmux, import the chapters
(yes, again, and use the same chapters you used with my tool ;)) and convert the avi to ogg.
Now you should be able to seek to exactly the same positions as on the dvd.
Since i live in a R2 region, i tested this with a pal movie (25 fps), only.

There is also a little drawback:
This won't work with XviD 0.9 Stable, this build simply ignores the new keyframes of the 1st-pass statsfile.
Currently this is only working with the latest builds with bframes _disabled_ (-1), because there is a bug/feature in XviDs 2nd-pass IPB-Decision code.
Probably this will be fixed in the future.

Please test KeyframeEnforcer (http://zuludev.de.vu) and report bugs here.


kind regards,

zulu


PS:

This program is written in C#, so if you're not happen to be on winxp+sp1 you'll need the .net clr. Get it here (http://msdn.microsoft.com/downloads/default.asp?url=/downloads/sample.asp?url=/msdn-files/027/001/829/msdncompositedoc.xml).

EDIT: updated the url.

EDIT 2:
KFE works with B-frame encodes now!
Now, there's also a commandline version available which doesn't require the .NET framework and features drag & drop support.
XviD 1.0 zones are supported in both, the GUI and the CLI version.

[Emp3r0r]

great job zulu, I'll test immediately since this is something I've been meaning to try for a long time now.

[Ashman]

I could be dumb but i think last version of Statsreader (2.0) add the possibility to insert keyframes where you want ;)

[zulu]

I could be dumb but i think last version of Statsreader (2.0) add the possibility to insert keyframes where you want


Sure, but this tool was created before statsreader 2.0, koepi can confirm this ;)
I just wanted to hold it back until the problem with bframes gets fixed,but now i decided to release it anyway, after all its working with IP-frame encodes..

[Ashman]

Ok, no pb :p
I haven't use it yet, will try it when i need it :D

[zulu]

I uploaded a new build.
Beside minor changes a bug has been fixed which caused frame 1 to be written as IFrame.
Chapters can be imported from Smartripper *INFO.txt files as well (No need to use chapterXtractor).

Anyone tested the program with NTSC movies?

[Koepi]

SmartRipper's INFO-files are mostly flawed (at least here on my end).
That's why I added support for ChapterXtractor, as that program gives reliable results :)

Regards
Koepi

[ookzDVD]

I think Koepi can easily merge this feature to insert the keyframe
for each chapter from the ChapterXtractor output into his StatReader
version 2.1 ;)

[zulu]

I think Koepi can easily merge this feature to insert the keyframe
for each chapter from the ChapterXtractor output into his StatReader
version 2.1


many thanks for your support :confused:

[ookzDVD]

@zulu,

I don't mean to say that your program is useless,
your program is good especialy for the original idea to "automate"
the keyframe insertion from the ogg chapter to the 1st pass file.

I use your program for my 1st pass file before I input it to the
StatReader.

Thank you for your great job. I really apreciate that.

[Koepi]

Yes, and I'm sorry that I included that feature - but you asked me why it doesn't work to insert keyframes in the statsfiles, and i had to check that and thus coded that as well (quite an easy hack within statsreader I assure you).

No offense, but I don't mean to be a "concurent", be assured.

Regards
Koepi

[zulu]

heyheyhey, everythings' easy, friends :D

@ookzdvd
sorry, i missunderstand you.
nice to hear you like it :)

@koepi
i know and have absolutely no problem with that;)

[duartix]

I usually rip at KFrame=Q2, IFrame=Q3, so this tool is very nice but still can't help people like me who do 1-Pass constant Quality/constant Quantizer rips...
How could I force those key frames?
How could I generate a dummy first pass without actually doing it, so I could force those keyframes?

[zulu]

Sorry, but i'm afraid there is no solution for 1-pass encoding yet.
Creating a dummy 1st pass statsfile and then doing just the second pass won't work. The 1st-pass file contains essential video analysis information like framesizes, information the 2nd pass relies on. if you simply store dummy values in this file, the second pass would completely mess up..

[Snakeisthestuff]

i have never expierenced the problem, that a chapter extracted with ChapterExctractor was not working, maybe the option that you can open an avi and save the txt file then is doing that, dont know. but i have another question, must it really be an Keyframe where your chapter is, cause i thought the ogm container eliminetes this contra, or why can i jump to every point on my avi with only 200 i frames in it?

maybe someone can explain it to me :confused: .

i am away for minimum 1 week so if someone has an answer please write me an email: Snakeisthestuff@gmx.li

[Emp3r0r]

Ogg streams seek instantly because OggDS can (and does by default) seek only to keyframes. This is the reason why you would want to force a keyframe exactly where chapters start. BTW, you can change this behavior in the OggDS filter properties page.

[Peters]

Originally posted by Emp3r0r
Ogg streams seek instantly because OggDS can (and does by default) seek only to keyframes. This is the reason why you would want to force a keyframe exactly where chapters start. BTW, you can change this behavior in the OggDS filter properties page.

Changing this behavior modifies only seeking.
Reading begins allways with a KF

[zulu]

I just found a bug that caused total keyframe misplacement on systems where a period (".") is used as decimal number seperator (e.g. USA).
I uploaded a new build that has nothing new but a fix to this bug.

[sandworm]

I've been encoding avi's with chapter splits on 1 pass by doing each chapter separately in virtualdub, recombining the avi segments at the end and then muxing in the ac3. Virtualdub and virtualdubmod (at least all the versions I've tried) introduce a 1 frame deficiency when combining avi segments so I've had to encode the chapter split frame twice, once at the end of the previous chapter and again at the beginning of the next chapter. For ogm's it should be easy enough to just use vdumbod to save as ogm after loading in all the avi segments. Incidentally, this is also the only way I've been able to force iframes while using bframes. Hope this helps.

[duartix]

Wellcome sandworm!
This looks like too much work. Do you have a fast way of encoding the chapters separately in VDubMod?

[sandworm]

Actually, yes there is, though I have yet to fully implement it. A simple program to parse a chapter frame listing and also a virtualdub jobs file would be needed. The VirtualDubMod.jobs (or -Mod if using vanilla vdub) has the general format:

// VirtualDub job list (Sylia script format)
// This is a program generated file -- edit at your own risk.
//
// $numjobs 1
//

// $job "Job 1"
// $input "E:\br\bb4.avs"
// $output "e:\br\bb4.00.avi"
// $state 0
// $start_time 0 0
// $end_time 0 0
// $script

VirtualDub.SetNandubCompatibility(0);
VirtualDub.Open("E:\\br\\bb4.avs",0,0);
encoding settings from a .vcf file pasted here
VirtualDub.SaveAVI("e:\\br\\bb4.00.avi");
VirtualDub.Close();

// $endjob
//
//--------------------------------------------------
// $done

So you'd need to have 2 .vcf's with your encoding settings saved (one for normal and one for credits, since the xvid credits option borks when slicing and dicing like this). The parser, which should read in the bolded lines above, would produce a new jobs file with $numjobs changed to reflect the segments, adding in a new job entry for each chapter. Also, if required, the credits start frame needs to be added in manually in the chapterxtractor output and the parser would have to load in the credits .vcf when adding in the final job.

Basically, for the end user, the steps are:
Save the avi as normal with 1 pass, checking the Don't run this job now box, and it'll save it to the .jobs file. Run the parser, load in the new outputted jobs, start the encoding, recombine and mux.
Sorry if any part of this is unclear, as I'm rushing off to work. If anyone's interested I can post a more detailed breakdown of the jobs file or some code to do it if I ever get around to hammering it out.

[Aktan]

the 1 frame deficiency... are you sure u set the end and start at the same exact frame?

[sandworm]

If the end and start are set to the exact same frame, there's no deficiency. It's only when a movie is split from say frame 1-999 and then from 1000-2000, and you try to recombine in vdub, the final movie will only have 1999 frames.

[mgerald21]

@zulu,

very nice program ... but I just found one discrepancy.
I manually added some keyframes after 1st pass and
did the 2nd pass.

Checking the resulting AVI file VirtualDub
reports a keyframe exactly one frame later than
specified position.

Wondering, if this only happens to me?!

greets, GM

[zulu]

@mgerald21:
Sorry for the late reply, i just returned from a vacation.
The one-frame-delay is done intentionally, because there is a bug in Ogg DSF's seeking algorithm. If the chapter keyframe is not 1 frame delayed, Ogg DSF won't find it and seek to the next keyframe. I already reported this to tobias, he will have a look at it and probably fix it in his next release. Chapter accurate seeking in movies you encode now (with a delayed chapter keyframe) will of course still work then.

[tigre]

I'll be HAPPY if it works the way I expect it to (downloading .net ATM, then I'll see). Thanx for this nice proggie.

I just wanted to mention that the download link doesn't work with Opera 6, I had to ask someone with IE to download it for me.

[zulu]

Originally posted by tigre
I'll be HAPPY if it works the way I expect it to (downloading .net ATM, then I'll see). Thanx for this nice proggie.

you're welcome! :)
]Originally posted by tigre
I just wanted to mention that the download link doesn't work with Opera 6, I had to ask someone with IE to download it for me.

i'll try to fix it..

[cfm]

Maybe the following will work for a single pass. I use this method for a multipass encode.

Reduce the keyframe rate to something usuable. (I use 20 frames for a 25 fps movie). Extract the chapter list and save this to a txt file. Open the move in something like VirtualDub where you can see where the keyframes are. Open the chapter txt file. Now edit the chapter entry point to the closest key frame.

The biggest mistake that you will make using this method will be half of your keyframe interval. (In my case 10 frame, or 400 ms.) Acceptable for me.

[duartix]

@cfm:

Thanks for the idea but for me (and probably many others) setting a 20-25 KF interval it's like killing a mosquito with a shotgun.

4-5 % keyframes it's a bit overkill...

[zulu]

Hi,
i uploaded a new build.
I made the following changes/additions:

Chapters can be imported directly from IFO files now!
This feature requires vStrip.dll. I'll include it when i release the sourcecode under gpl, until then you can get it here (http://www.maven.de/code/index.html#vstrip).
Support for Divx 5.03+ logfiles (2-pass only, no B-Frames) (don't hit me! :eek:;))
Added proper exception handling
Based on VESLib
GUI has XP style now (only when running on win xp of course..)
New icon
Other minor changes i can't remember

I guess this was the last update to this tool.
Let's see if i can find something more useful to work on ;)

regards,
zulu

[zulu]

Oh..yeah...i got a little bored today and coded a commandline version in plain C... Just for fun...
Might be useful for ppl who want to use kfe in their encoding apps/frontends and of course for people who fear to download that biiig fat evil .net-framework-elephant thingy ;)

It works like this:
kfec -i input.stats -o output.stats -cl chapterxtractor_list.txt -fps 29.970

regrds,
zulu

[Pen-Pen]

great idea

I'm testing it right now :)

[bilu]

@zulu

About the B-frame issue: I know this is not a perfect solution, but could you just avoid B-frames and force keyframes over P-frames? :confused:

This way every movie would be supported, with a maximum delay of the consecutive b-frames, less than 3 frames for most users (I think).

I'd like to use your CLI version like this. ;)

Thanks in advance,
Bilu

[TaZ4hvn]

I found a little bug in the CLI version: when encoding a very short part of a movie (say 3min) with Chapterlist of the whole movie, KFE CLI will discard hour information and set keyframe according to minutes, for instance:

Clip is 00:03:00.000

> Chapter 01 (00:04:17.160) skipped - chapter is out of range.
> Chapter 02 (00:06:33.840) skipped - chapter is out of range.
...
> Chapter 15 (01:00:56.680) Set to intra.
> Chapter 16 (01:08:58.320) skipped - chapter is out of range.
...

Also I'm with Bilu :D

[zulu]

Originally posted by TaZ4hvn
> Chapter 15 (01:00:56.680) Set to intra.
:eek: i'll look into it


Regarding the b-frame issue:
The last time i tried to force keyframes in a statsfile with b-frames,
it failed because XviD's dynamic IBP-Decision code was ignoring them. I don't know if this was fixed in the meantime.
Have you tried it with a b-frame featured statsfile? If the enforced keyframes are still ignored i'm afraid _I_ can't do nothing. If they are respected, but the encoded video shows artifacts/errors, then its time for me to look into it again and make sure that it won't break anything like DivX5 BVOP compatibility.

regards,

zulu

[zulu]

Arrgghhh...
Thanks to Taz i found a major bug (typo) in the frame calculation routine which causes keyframe misplacing of chapters > 1 hour.
If you're using the CLI version please update to the latest build.

regards,
zulu

[bilu]

Haven't tested your tool yet but found something important:

http://forum.doom9.org/showthread.php?s=&postid=334316#post334316

Quoted from Koepi:

some people prefer the old IP-decision. It respects frametypes from the statsfile - if you add them with one of the tools around (i.e. statsreader :) ) they get inserted in the final encoding.

Bilu

[zulu]

Thanks for the info, even if i knew that already :D
BTW: More about this issue can be found here (http://forum.doom9.org/showthread.php?s=&threadid=50295&perpage=20&pagenumber=1) (starting at post ~12).

zulu

[zulu]

I over-worked KeyframeEnforcer to support XviD 1.0 RC1+ zones,
meaning you can use it to set up keyframes zones (zones starting with a keyframe, e.g. chapter positions)
in a very quick and comfortable way.

It works as follows:

Load Keyframes from a DVD IFO or ChapterXtractor file (click "import list").

http://www.shockshit.net/zuludev/screenshots/kfe3.jpg

Select "Extra" -> "Export keyframelist to XviD 1.0 config".

http://www.shockshit.net/zuludev/screenshots/kfe2.jpg

Now, open the XviD VfW configuration and you'll see a list of new zones, starting with a keyframe each.
Credits for the original idea go to Emp3r0r.
Many thanks to theoneyouknowleast for testing!

Please post your results! :)

Full changelog:

31.01.04 0.8
- Added: Keyframes can be exported to XviD 1.0 zones (Registry)
- Added: An option to turn off the 1-frame-delay (required for ogm chapters)
- Added: [maven's] vStrip.dll included (required for IFO chapter import)
- Changed: sourcecode released under GPL
- Changed: Keyframes can be inserted by time or framenumber
- Changed: significant code optimizations resulting in ~100% speed increase when processing Divx5 or XviD-Dev-Api-3 stat files
- Fixed: A minor bug in the file dialogs
- Fixed: A worker thread/gui thread threading issue (made it impossible to debug the app)
- Fixed: A bug in the IFO chapter import (KfE was refusing IFOs with more than one PGC)
- Fixed: A bug in the IFO chapter import (chapter times of NTSC IFOs were sometimes slightly off)
- ...


The non-.net CLI version got updated too, though XviD 1.0 isn't supported (yet).

Changelog:

31.01.04 0.6
- Added: -kfl <keyframefile> switch - read keyframes from a keyframelist file (every line contains a keyframe number)
- Added: -delay <ON/OFF> switch - specifies wheter keyframes should be delayed by 1 (required for OGM chapters)


regards,
zulu

[zulu]

BTW, i changed the thread topic to a more suitable one, but it doesn't show up in the forum.
I guess only mods are privileged to do that. So, if a mod could change that please i'd be thankful :)

[zulu]

I fixed a bug that caused inserted keyframes to be off by one sometimes. Thanks to theoneyouknowleast for helping tracking this down. :)

Changelog:
29.02.04 0.8.1
- Fixed: A bug that caused keyframes to be off by one sometimes
- Added: An option to flush XviD zone registry settings

regards,
zulu

[ssjkakaroto]

hey zulu any chance that you'll update the CLI to this current version? ;)

[Joe Fenton]

I run XP64, and MS doesn't have the .Net framework for 64-bit yet, and the 32-bit .Net framework won't even install on XP64. The CLI version would be the only one I could use, so an update on that would be nice.

[zulu]

ok guys, you convinced me :)
I'll see if i find time next weekend.

[zulu]

Hi!
I updated the NON-.NET commandline version with full XviD 1.0 zones support.
New switched are -zonesexport and -noconfirm.

Example:
kfec -zonesexport -cl c:\matrix_chapters.txt -delay ON -fps 25.00

Although KFEC is a commandline app, this release features drag & drop support! :D
Just drag & drop chapterfiles on the included script and you're done
(make sure you set the APPROPRIATE FRAMERATE in the script).

http://shockshit.net/zuludev/screenshots/kfec2.jpg

Changelog
14.03.04 0.7
- Added: Support for XviD 1.0 zones
- Added: Drag & drop support script
- Fixed: Roundig bug

regards,
zulu


P.S.: Since this tool isn't restricted to OGM anymore and is useful for matroska as well,
could someone with the necessary rights change the thread topic to 'Tool to set keyframes (e.g. for OGM/MKV chapters)' ?

[darkpepe]

Hi!
I have a question regarding the usage of this app:
Is the guide in the first posting up to date? Someone told me that adding the reg file to registry is enough.

Thanks!

[zulu]

hi darkpepe,
no, the guide on page 1 isn't up to date, it's refering to an old xvid build where the zone feature wasn't introduced yet.

To enforce chapterkeyframes with current xvid builds you have to..

start kfe
hit "import list" and choose an ifo or chapterxtractor file to load the chapterpoints from
select "export to xvid zones" in the extras menu


regards,
zulu