DGPulldown with matroska timecode?

[Malow]

i was looking for a way to do vfr mkv>dvd, but, i can't find... or cant see...

i was thinking if dgpulldown can work like mkvmerge, generating pulldown flags to make a cfr avi (extracted form matroska) and timecodes, making easy a vfr dvd creation...

or aim being silly?

[foxyshadis]

You could load the mkv into avisynth with

directshowsource("video.mkv",fps=59.94,convertfps=true)
separatefields().selectevery(4,0,3)
weave()

...I think, I'm not very up to speed on reinterlacing stuff. I can't remember if 4,0,3 is tff or if it's 4,1,2.

That means hard telecining only, but at least it's a way to do it without needing dgpulldown modified. I'd be interested in hearing if a soft telecine solution is feasible though.

[Malow]

but my point is not process the video, just put pulldown in right places like mkvmerge do with cfr mp4 + timecodes...

[Mug Funky]

that's kinda a cool idea. it would also mean adaptive soft telecine would be possible with avisynth + any encoder you like. just encode 29.97, and use DGpulldown + timecode file to make a "80% film" style m2v. that would be quite cool.

[Malow]

that's kinda a cool idea. it would also mean adaptive soft telecine would be possible with avisynth + any encoder you like. just encode 29.97, and use DGpulldown + timecode file to make a "80% film" style m2v. that would be quite cool.


sure... i hope neuron is lintening us......

"vfr dvd > vfr mkv > vfr dvd" is my dream.... only 50% of it is done... :D

[robU*4]

my dream too ;)

[neuron2]

What is the format of the timecodes file you'd like to have DGPulldown read? Can you provide a sample stream with corresponding timecodes file? How do you make these timecodes files?

[check]

There are two timecode formats that mkv use. They are easily convertible, so you could support only one and make the user convert, or I guess even implement an internal converter.
timecodes Version1:
# timecode format v1
Assume 29.970000
500,750,23.976000
1366,1368,23.976000
2100,2350,11.988000
frames not specified are assumed to be at the assumed rate.
timecodes Version2:
# timecode format v2
0.000000
33.366700
66.733400
100.100100
133.466800
166.833500
the value is simply the duration of that frame (in milliseconds). All frames must be given a value.

I can upload a sample video for you, but it's reasonably easy to create your own with dedup() if you wish. Just set a high threshold to arbitrarily drop random frames.

[Malow]

neuron2, here u can find a simple example with timecode v1:

http://rapidshare.de/files/35487130/vfr_example.zip.html

edit: i think timecode v1 is more useful, because tivtc generate v1, and the mkv2vfr save as v1 too...

Tritical's have a conversor:

http://www.missouri.edu/~kes25c/
or direct link:
http://www.missouri.edu/~kes25c/avi_tc_package.zip

[dbloom]

I'm working on adding MKV v1 timecode support to the DGPulldown source code. I'm not very familiar with the DGPulldown source code (or with C programming in general) so my modification will be a quick and dirty hack, but I'm hoping it will compile and may be even work...

Basically, for those of you who are familiar with the source code, I'm modifying the generate_flags function so that it will open a timecode file and generate flags that vary based upon the framerates specified in that file. Fun stuff.

[Malow]

I'm working on adding MKV v1 timecode support to the DGPulldown source code.

yeahhh dude!!!!!

if u need someone to test, "happy to help" :)

Question: the framerate variation is only ntsc/film or may have a "decimation" between these two?

[JetlagMk2]

# timecode format v1
Assume 29.970000
500,750,23.976000
1366,1368,23.976000
2100,2350,11.988000


I just wanted to point out that this particular example would be impossible to pull-down, considering pull-down can only achieve a framerate 1.5 times your starting framerate.

[dbloom]

Jetlag is right. Is there any way to force the filter that makes the timecode to not go below a certain FPS?

[Malow]

Jetlag is right. Is there any way to force the filter that makes the timecode to not go below a certain FPS?

i never came across a movie with so low fps... the lower to me is always 23,976... even in some cartoon/draw scenes... i don't think this is a problem... i just concern about "between" "momentanium framerates" that may appear in some "harcoded" movies... (here in brazil we have the worst movies makers in the planet... ;) )

anyway, im sure tritical can update the tivtc package if something is needed... he is one cool guy.. ;

[dbloom]

How do I compile this thing?

I am using the MS Visual C++ Toolkit 2003 compiler and I have the latest Windows Platform SDK installed. When I try to compile, I get this:
/out:dgpulldown.exe
dgpulldown.obj
dgpulldown.obj : error LNK2019: unresolved external symbol __imp__EndDialog@8 re
ferenced in function _DlgProc@16
dgpulldown.obj : error LNK2019: unresolved external symbol __imp__SetWindowTextA
@8 referenced in function _DlgProc@16
dgpulldown.obj : error LNK2019: unresolved external symbol __imp__GetOpenFileNam
eA@4 referenced in function _DlgProc@16

What am I missing?

[dbloom]

Malow:

It should be fine if the framerate gets too low occasionally and briefly, as long as the low framerates do not last for more than a few frames. When this happens, the pulldown will slow down as much as it can for as long as it needs to to regain sync.

[neuron2]

What am I missing? You need to link in:

user32.lib
comdlg32.lib

[dbloom]

w00t!, got it working :)

my modified dgpulldown is attached to this post (source included).

read TIMECODE_README.txt before you use it.

update: i also uploaded it here so you don't have to wait for attachment approval: http://bloomd.home.mchsi.com/dgpulldown_mkv_timecode.zip

update 2: changed array sizes so that a maximum of 8192 instead of 1024 framerate ranges can be specified, due to the fact that my 3 minute music video test clip specified about 111 framerate ranges. i made the array larger so that it should work properly with those longer files (it should work fine with files as long as 4 hours, possibly longer)

[Malow]

w00t!, got it working :)

my modified dgpulldown is attached to this post (source included).

read TIMECODE_README.txt before you use it.

wow! tnks! i will test at home!!!

[dbloom]

Is there anyone who wants to try getting the VFR functionality added to the DGPulldown GUI?

Also, I'm thinking about adding the ability to add TMPGEnc manual-VBR textfile generation to the DGPulldown mod. This way, you could achieve CBR when using variable frame rates (I did something similar for my previous MPEG-2 VFR program (http://davidbloom.home.mchsi.com)). I think this would be useful for encoding video with a restrictive maximum bitrate, such as SVCD.

[Malow]

Is there anyone who wants to try getting the VFR functionality added to the DGPulldown GUI?

i don't care about gui (i will use in a batch), but i can grantee, a lot will say: "UH? NO GUI? NO WAY!" so, if possible... wold be welcome...

Also, I'm thinking about adding the ability to add TMPGEnc manual-VBR textfile generation to the DGPulldown mod. This way, you could achieve CBR when using variable frame rates

I'm sure this will be nice. please, don't stop the evolution... ;)

[dbloom]

Maybe there should be a standalone general-purpose VFR timecode --> VBR birtrates utility that can output files in the appropriate format for various encoders (TMPG, CCE, etc....), and provide provide options for CBR/2pass VBR/etc output.

I think something like this will be necessary because obviously changing a constant framerate file to VFR will affect bitrate...

[Malow]

yeah, but is not enough a simple calc based on the timecode to show the "variation" = XXX = maximun/medium/minimun bitrate = XXXX?

or a simple variation percentage? to help bitrate calculation?

[dbloom]

I think something more useful would be average bitrate instead of maximum variation. I think that control of this could be achieved by reverse-engineering the 1st-pass VBR log file formats used by various MPEG-2 encoders...read the logfile then rewrite it wiht modified bitrates that compensate for VFR before the 2nd pass.

[Malow]

I think something more useful would be average bitrate instead of maximum variation. I think that control of this could be achieved by reverse-engineering the 1st-pass VBR log file formats used by various MPEG-2 encoders...read the logfile then rewrite it wiht modified bitrates that compensate for VFR before the 2nd pass.

yeap, u are right..

man, this will be amazing... :D

now i see a lack in procoder, it does not save 1pass log file, or even allow a stop betwen passes... :(

but, i guess CCE, HC, Tmpeg, and QEnc can use this method wright? tnks for your nice work.. ;) :thanks:

[Malow]

OWWW MOMMY, works flawless!!!

my test:

created a 29.97fps mpeg2 video with some film-telecined video (NO PULLDOWN flags - a.k.a. hardcoded - "always 29.97fps" -just to "force" a little bit.. )

dgindex > tivtc > x264 > mp4 + v1 timecodes = mkv_vfr!

tested wiht MPC + coreavc + fdshow as video filter, to see the framerate duration variation (the vfr was nice)

the problem was i can't extract with mkv2vfr... i got a "No Video Tracks Found"... but no problem, i just convert the mkv direct to a progressive mpeg2 (procoder - using assumefps to avoid processing) and using the original timecodes, not the extracted from mkv... :(

but, ok, working nice ... then....

dgpulldown video_conv.m2v -bff -destfps 29.970 -tcv1 timecodes_v1.txt

and guess what? a nice mpeg2 with the correct pulldown "in places"

then, make a dvd with scenarist... no problem...

tested with nvidia mpeg2 decoder, to see the film/video variation... and... PERFECT!!! the dvd was perfect!!!

the frame count:

original mpeg = 1575 frames +- 52sec @29.97fps
converted mpg2 = 1442 frames +- 48sec @29.97fps
pulldown converted mpg2 = 1575 frames +- 52sec @29.97fps

not satisfied enough, (kkk) i open the dvd in dgindex to confirm film/video parts... perfect...

ow my god, im gonna cry!!!!

neuron2 and dbloom: :thanks: :thanks: :thanks:


edit: i forgot to encode the first time using a 12 frame gop... good to remember... ;)
edit2: ive tested with 3 videos, a 52sec, 4 min, and 30 min.... i will test with a full movie as soon as i can! ;)
edit3: i guess when x264_interlaced+coreavc works "friendly" together (and in the optimized way) we wold not have to deinterlace the video parts, keeping the movie as 100% original! (did i hear a "HURRA?")

[Mug Funky]

maaaan, this is very cool.

makes me wish i did more stuff for NTSC land (and to date everything i've done for NTSC land has either been pure film all through, or completely un-pulldown-able).

i'm now going to figure out if it's possible to use this to make 20fps PAL stuff when scenes are still enough - could be a bitrate saver, and in theory will be perfectly acceptible PAL.

[Malow]

i'm now going to figure out if it's possible to use this to make 20fps PAL stuff when scenes are still enough - could be a bitrate saver, and in theory will be perfectly acceptible PAL.

i put a video part filmed on a tripod, very static, and yes, tivtc go to film on that part... ive used the default threshold for duplicate detection.... in that part, the little motion it has, was nice... i guess it is a good thing.. ;)

[Mug Funky]

excellent. means in still scenes (but possibly very detailed), you're effectively allowed a max bitrate of 12250 kbps instead of 9800 kbps...

you never know when you'll need to turn the bitrate up to 11 after all. especially with anime.

[dbloom]

excellent. means in still scenes (but possibly very detailed), you're effectively allowed a max bitrate of 12250 kbps instead of 9800 kbps...

Actually, with PAL, you could turn the framerate all the way down to 16.6666667 FPS (if all frames are RFF). So you could get an even higher bitrate :D

[foxyshadis]

You'd need a customized Dedup to get the best out of it. You could limit it to every other frame now, but it would have to max out at one in three.

Of course, here mug funky's going to create a whole new generation of pain for australian backer-uppers. ;)

[dbloom]

You'd need a customized Dedup to get the best out of it. You could limit it to every other frame now, but it would have to max out at one in three.

TDecimate could do the trick, however.

[foxyshadis]

TDec can only do one out of 5 (mode 5 limitation), I just meant if you really needed to drop an extra frame per cycle you'd need something else. For your average hybrid conversion, this patch+TDec will work wonders.

[dbloom]

TDecimate was labeling parts (that lasted about 2-5 frames) of my music video as 17.[something] FPS. Perhaps this is just a rounding error in the conversion to timecode format, however...

[dbloom]

This is somewhat unrelated to the whole VFR idea, but I could probably significantly improve the time it takes dgpulldown to run by making it use a TMPGEnc encode logfile (which indicates the compressed size of each picture and therefore how far forward to seek to the next picture). I've never run dgpulldown on a large, movie-size file, so I don't know how long that normally takes...but it could probably run ten times as fast if it could read an encode log.

[foxyshadis]

I think DGPulldown already uses start codes, which are about as cheap to read and follow on the fly as an external size table. Optimizing the read speed of small seeks is somewhat pointless, since modern hard drives read and buffer sequential seeks best, but of course it's a waste for I-frames that can be up to 250K for a single frame. (They seem to be around 10-80K at normal bitrates.) Not sure if it'd gain anything for other picture types, especially on raid drives. Worth testing!

[dbloom]

I think DGPulldown already uses start codes, which are about as cheap to read and follow on the fly as an external size table. Optimizing the read speed of small seeks is somewhat pointless, since modern hard drives read and buffer sequential seeks best, but of course it's a waste for I-frames that can be up to 250K for a single frame. (They seem to be around 10-80K at normal bitrates.) Not sure if it'd gain anything for other picture types, especially on raid drives. Worth testing!

But it won't have to seek to every single frame. For example, in 24fps, only every other frame (in display order) would have to be seeked to (and the frames that are preceded by a GOP header which would have to have the timecode header updated).

[foxyshadis]

DGPulldown rewrites the flags on every frame as it is. On a totally unflagged m2v, it would be an optimization though.

[dbloom]

Whether the file is already flagged for 2:3 could be determined using the TMPG encode log. Also, the encode log could be verified by matching its last-modified date with that of the file (as well as verifying the order/size of the first few frames of the file to verify that they match the info in the encode log).

Basically, the encode log could be loaded automatically when it is usable (since it is the same filename as the m2v except with a .log instead of .m2v extension).

[dbloom]

I thought I'd revive this thread...I'm not sure if everyone got a chance to check this out because I'm sure there are plenty of folks who could use it.

Can anyone help me add this feature to the DGPulldown GUI and get the GUI to compile?