Hybrid VFR with 24, 30 and 60FPS

[thecoreyburton]

I have a couple of DVDs that I'd like to mux into mkv, but I'm trying to work out the best way to do it. The DVDs are all NTSC and play at 29.97fps. They have telecined film sections, progressive 29.97 sections, and interlaced 29.97 sections.

Previously I've handled this with TFM & TDecimate's mode 4 and 5 functions and it's done the job pretty well. The only catch is in the deinterlacing - I'm stuck with a VFR clip that runs at either 23.976fps or 29.97. In some of the video clips there's quite a bit of fast scene swapping and the best way to capture that would be to bob the clips when they're interlaced, QTGMC being my preference here.

Is there a way to field match the film sections, not touch the progressive sections and merge them all down into a VFR clip?

I've had three somewhat successful attempts so far:
1: TDecimate gave me a 24/30 VFR.
2: TFM, ChangeFPS QTGMC and Dedup combined with gave me the correct frames - but because of the nature of exact dedup the 24fps sections were played as if they were at 30fps.
3: TDecimate gave me a 60/48 VFR. This is probably the best outcome I've had so far as all the timings are correct, there were just a lot of frames that I didn't really need in the file and I'd like to get rid of those if possible.

I've heard good things about AnimeIVTC being able to handle 60/30/24 but as far as I can tell (I have 2.00) I can only achieve 30/24.

If you'd like a sample you're welcome to request one.

[ChiDragon]

I've never really tried VFR myself, but my understanding is this requires a 120fps intermediate step rather than 60fps.

[kuchikirukia]

60fps doesn't capture anything about a 30fps clip better than 30fps.

[ChiDragon]

With regard to actual interlaced content, you're mistaken. 30fps is really 60Hz = 60 unique moments in time captured with half vertical resolution.

[thecoreyburton]

60fps doesn't capture anything about a 30fps clip better than 30fps.

It depends on the content. If it's 30fps content without interlacing you're correct, as is true if it's a bad transfer - in this case ChiDragon's spot on. Some of the frames are truly interlaced, meaning when they're properly bobbed I can have 60 unique frames a second (often 30fps is acceptable but on the off-chance there's faster paced content or you want the full experience then there's definitely a reason to get the footage to 60).

I've never really tried VFR myself, but my understanding is this requires a 120fps intermediate step rather than 60fps.

That was my goal with the dedup attempt. I was thinking perhaps TFM with no post processing & tdecimate modes 4 and 5. That'd create a VFR clip with both 30i and 24p. I'd output that to an intermediate AVI which could load that up and use VFRtoCFR with 120000/1001 (120fps). The only problem now would be the interlacing, as deinterlacing wouldn't work with all the duplicate frames.

To counteract this I could use a IsCombedIVTC and use clip2 as follows: load the clip up again with VFRtoCFR 30000/1001, apply QTGMC and then ChangeFPS to 120000/1001. Essentially that'd be the same clip, but bobbed and it'd only replace the frames in the original that have combing. Then do a dedup (or exactdedup would be better in this case).

There are two large issues with this though.
1) The process requires four passes.
2) It's inaccurate, sometimes it works, other times the footage is jumpy and bobs either incorrectly or replaces the wrong frames.

I'm thinking there has to be a simpler way to do this and I've over-complicated it, but I certainly agree that getting it to 120fps seems to be the likely way to go as it's the lowest common multiple of all 3.

[ChiDragon]

The TDecimate documentation mentions 120fps with regards to mode 6. Surely there must be posts somewhere about the rest of this workflow.

[thecoreyburton]

I've found some, but mostly it's about re-encoding VFR (for example after loading a 24/30 VFR clip, applying the timecodes in avisynth and setting the clip to 120fps, it'll work in the same way ExactDedup would and would remove any exact duplicates). Given it can do that and also create 24/30 clips I'd assume there'd be a way to get it to do what we're asking of it. I'll keep looking into tdecimate documentation and other potential cases of the same problem.

I was working on an idea based off the following:

Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("audio.ac3")
Source=AudioDub(Movie, Audio)
IVTC=Source.QTGMC()
EvenFrames=Source.TFM(PP=2, clip2=IVTC.SelectEven)
OddFrames=Source.TFM(PP=2, clip2=IVTC.SelectOdd)
Interleave(EvenFrames, OddFrames)
ChangeFPS(Movie.Framerate*4)

With TDecimate using modes 4 and 6 on each respective pass. Those lines are emitted simply to keep this post shorter but would be at the end.

It's good in theory and decimates all the duplicates - but because it doesn't use any TFM input (due to having called TFM twice), the TFM sections are left in the form of abcdd with a double at the end.

A closer examination of the clip without any TDecimate functions shows that the two last frames are similar but not identical.

Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("audio.ac3")
Source=AudioDub(Movie, Audio)
IVTC=Source.QTGMC(FPSDivisor=2)
Source.TFM(PP=2, clip2=IVTC.SelectEven, output="TFM.txt")

Same deal about the TDecimate lines being omitted, but this time with modes 4 and 5 respectively.

This decimates those sections perfectly, the abcdd sections become simply abcd and everything lines up perfectly.

Inspection of the clip without any TDecimate functions shows that the last two frames are once again very similar but not identical. Because this method does use a TFM input though, the duplicate is removed on the second pass.

[thecoreyburton]

I figured that the only reason the mode 6 script above didn't work is because the TFM sections weren't perfect duplicates and I was thinking about it for a bit. I came to the conclusion that if perhaps I did TFM and mode 5, then loaded the VFR clip back up the frames would have to duplicate, then I could do mode 6. I attempted a four pass shot at this:

Pass #1: Ran this as an analysis pass in Virtual Dub.
Movie=DGDecode_MPEG2Source("video.d2v", cpu=4, info=3)
Audio=FFAudioSource("audio.ac3")
AudioDub(Movie,Audio)
TFM(PP=0, Mode=4, output="TFM-Pass1.txt")
TDecimate(mode=4, output="TDecimate-Pass1.txt")

Pass #2: Saved this as a lossless AVI.
Movie=DGDecode_MPEG2Source("video.d2v", cpu=4, info=3)
Audio=FFAudioSource("audio.ac3")
AudioDub(Movie,Audio)
TFM(PP=0, Mode=4, input="TFM-Pass1.txt")
TDecimate(mode=5, hybrid=2, vfrDec=1, input="TDecimate-Pass1.txt", tfmIn="TFM-Pass1.txt", mkvOut="Timecodes-Pass2.txt")

I converted the timecodes file from v1 to v2 to make it compatible with the next script. I named the file "Timecodes-Pass3.txt" for reference.

Pass #3: Ran this as an analysis pass in Virtual Dub.
AVISource("intermediate.avi")
Source=VFRToCFR(times="Timecodes-Pass3.txt",numfps=30000,denfps=1001)
IVTC=Source.QTGMC()
EvenFrames=Source.TFM(PP=2, clip2=IVTC.SelectEven, output="TFM-Pass3-Even.txt")
OddFrames=Source.TFM(PP=2, clip2=IVTC.SelectOdd, output="TFM-Pass3-Odd.txt")
Interleave(EvenFrames, OddFrames).ChangeFPS(Source.Framerate*4)
TDecimate(mode=4, output="TDecimate-Pass3.txt")

Pass #4: Simply had a scrub through the frames in Virtual Dub and saw the problem here, otherwise this would have been the final output.
AVISource("intermediate.avi")
Source=VFRToCFR(times="Timecodes-Pass3.txt",numfps=30000,denfps=1001)
IVTC=Source.QTGMC()
EvenFrames=Source.TFM(PP=2, clip2=IVTC.SelectEven, output="TFM-Pass3-Even.txt")
OddFrames=Source.TFM(PP=2, clip2=IVTC.SelectOdd, output="TFM-Pass3-Odd.txt")
Interleave(EvenFrames, OddFrames).ChangeFPS(Source.Framerate*4)
TDecimate(mode=6, hybrid=2, vfrDec=1, input="TDecimate-Pass3.txt", mkvOut="Timecodes-Final.txt")

The results of this weren't what I hoped though. It worked a little worse than mode 6 on it's own. Without the TFM in the now-identical frames of the film section still weren't decimated. Four passes felt a bit tedious AND because I had decimated and loaded up the file again, the frame timing and placement varied slightly. This caused stuttering back and forth in that parts which used QTGMC like what you'd see if you had the field order wrong. Because of the nature of the VFR clip though, the stuttering wasn't consistent. In some cases it'd stutter whereas in others it'd deinterlace properly.

After typing all this up it's obvious to me why it didn't work, but hopefully it gives someone an idea on potential solutions, perhaps. Experimenting with TDecimate's advanced options in the hopes one of them may help.

[ChiDragon]

Personally I would just encode a 60fps file with 3:2 pulldown intact (http://forum.doom9.org/showpost.php?p=1713315&postcount=16) and forget about VFR. Unless you actually have a display that you run with a 120Hz source.

As a VFR test I tried that tweaked TFM script in place of yours, but I couldn't get TDecimate to recognize the 3:2 pulldown sections as 24fps no matter what I did. Even when I fed it as 60fps instead of 120fps, the actual decimation of the film sections was mostly correct, but the timecodes file just switched between 59.94 and 39.96 every frame.

[thecoreyburton]

I'm seriously considering it. Or even manually analyzing each source individually and determining whether 60/30 or 30/24 would better suit it and doing the appropriate one. I do have a high refresh rate display that I use for both PC and TV though and some of the clips I'm trying to do are what I'd classify as my all time favorites, so the passion and want is there - I'm just not sure if there is a way to do what I'm asking and therefore I'm not if it's worth persisting or just giving up for now.

That's about where I'm at with this too. Best case scenario I can get 2 of the 3 desired rates displaying correctly.

I'm a bit of a novice when it comes to this though, I feel like there's something I'm overlooking or an obvious way to do this. I should probably ask, is TDecimate capable of detecting 3 different frame rates? Perhaps the issue is that it's capped at two per file and it starts at the highest. In cases of 60/30/24 it'd get to 60/30 and then try and decide between them for each scene, in 30/24 it'd only have those two to choose from.

Edit: On that note, if we use your script for keeping the pulldown intact and call TDecimate modes 4 and 5 afterwards we get a 60/48 clip with occasional bursts of 36fps.

[ChiDragon]

If you want to keep trying, I never attempted this combo:

My TFM+QTGMC + Dup (make the "D" in ABCDD an exact dupe) + ExactDedup for VFR (instead of TDecimate)

[thecoreyburton]

I think it'd work with decimation, but the frame playback times would be incorrect. ExactDedup would have the frames play back as ABCD-ABCD-, the dash representing where the a frame is held for an extra frame. TDecimate detects these sections and manually sets them to 24fps.

Is there a way to use that method but somehow get it to 120fps with the field-matched sections playing at the correct times? Eg: AAAAABBBBBCCCCCDDDDD rather than AAAABBBBCCCCDDDDDDDD.

[ChiDragon]

You're right. I realized today that's also the problem with TDecimate mode 6 for this purpose. It's the one mode that is designed not to change frame timings, since the idea is to input a 120fps file that already uses only integer frame repetitions (5*24, 4*30, 2*60) with no messy 3:2 or 6:4 pulldown patterns.

And the problem with the other TDecimate modes is as you say, they're only designed to handle two frame rates (base decimation rate and hybrid passthrough rate).

[thecoreyburton]

I guess that means this is a bit beyond my scope at the moment. What are the chances of being able to get in contact with Tritical? TIVTC would work with some slight modifications to be able to handle this sort of thing.

[ChiDragon]

I believe he stopped developing anything to do with Avisynth years ago.

[thecoreyburton]

That's not very fortunate in this case! I guess I'll have to try various obscure methods and hope something gets it right at this point. I have a possible solution, but I'll have a crack at it before I post it here.

Edit: Here's my newest attempt, I actually got pretty close.

Pass #1:
Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("audio.ac3")
AudioDub(Movie,Audio)
TFM(PP=0, output="video-tfm1.txt")
TDecimate(mode=4, output="video-tdecimate1.txt")

Pass #2: I saved this pass as "Step2.avi"
Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("audio.ac3")
AudioDub(Movie,Audio)
TFM(PP=0, input="video-tfm1.txt")
TDecimate(mode=5, hybrid=2, vfrDec=1, input="video-tdecimate1.txt", tfmin="video-tfm1", mkvOut="video-temporarycodes.txt", tcfv1=false)

Pass #3
Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("video.ac3")
Original=AudioDub(Movie,Audio)
Step2=AVISource("step2.avi").ConvertToYV12().VFRtoCFR(times="video-temporarycodes.txt",numfps=120000,denfps=1001)
IVTC=Original.QTGMC().ChangeFPS(Original.Framerate*4)
Step2.TFM(PP=2, clip2=IVTC, output="video-TFM-Pass3.txt")

Pass 3 (and then 4) would have been identical except for the use of ExactDedup.
By loading in the already decimated clip at 120fps the timings would be correct even for the film sections and frames would simply be duplicated. Since deinterlacing this clip is out of the question, I called the original clip back in and attempted to deinterlace that instead and use it to replace the combed frames in the already decimated clip. The frames weren't played at the same times though it seems. Adding StackHorizontal(Step2,IVTC) makes the problem more apparent and I can see why it's going wrong, I'm just not sure how to fix it.

[ChiDragon]

As I recall, the process to create a 120fps CFR file involved using a crazy combination of poorly documented and/or untranslated Japanese tools that may not even be available anymore.
http://forum.doom9.org/showthread.php?p=286172#post286172

I'm not entirely sure they even handled 60Hz sections by bobbing. Most anime only used pure interlacing on credits, and even the hardcores don't care much about perfectly smooth motion there.

[thecoreyburton]

Wow, 2003. Thank you, that's a hell of a lot of digging you did to help me out. I'll get to work on acquiring all those plugins and testing out various methods of getting this to work. It does look like it's designed specifically for 30/24 as opposed to 60/30/24, but depending on how the plugins work there may be a way of incorporating bobbed footage. I'll work with bob in place of QTGMC until I get somewhere substantial I think, to save time.

[osgZach]

2: TFM, ChangeFPS QTGMC and Dedup combined with gave me the correct frames - but because of the nature of exact dedup the 24fps sections were played as if they were at 30fps.


So why not take the output from that, and then rewrite the timecodes in V2 format? You can specify which frame ranges should run at what FPS.


Any chance of getting a sample of the source file (with all 3 framerates) ?

[thecoreyburton]

I have several sources with varying rates at varying points, that's part of the problem. Some are simply credit sections which are very easy to do manually, others are not so simple and have many moments flicking back and forth in a short time. Manual selection on some of them would be too tedious. I'll have a flick through the sources and see if I can find a portion that has all 3, or even interlaced and film (so 60/24). There's probably a case like this on music DVD I have, I'd assume.

[thecoreyburton]

Excuse the shortness of the clip, it's compressed losslessly with the Lagarith codec in order to retain the original data. From what I can tell the mouth sections are film and the flashing sections are mostly interlaced, although at some points (not present in this particular sample due to the size of a lossless file) the flashing sections are at 29.97 and are progressive. This is a good example of where setting the frame rate manually would be a little too tedious for me.

60mb @ Mediafire (http://www.mediafire.com/download/hp50pmqsom7l2zl/sample2.avi)

[osgZach]

yeah that does look pretty ugly..
Well I suppose its about how much work you really want to put into it. You always have to pick a trade-off between time and quality of result in these situations.

[thecoreyburton]

I'm willing to put quite a lot of hours / days into doing this and figuring out a way to do it automatically, but to do a couple hundred clips like that manually would probably drive me to insanity.

[thecoreyburton]

I may have devised a complicated theoretical way of doing this, I'll be looking into it shortly (when I rip another source that has similar problems).

File #1, Pass #1:
Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("audio.ac3")
AudioDub(Movie, Audio)
IVTC=Last.QTGMC(preset="very slow").SelectEven()
TFM(PP=2, clip2=IVTC, output="TFM1.txt")
TDecimate(mode=4, output="TDecimate1.txt")

File #1, Pass #2
Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("audio.ac3")
AudioDub(Movie, Audio)
IVTC=Last.QTGMC(preset="very slow").SelectEven()
TFM(PP=2, clip2=IVTC,input="TFM1.txt")
TDecimate(mode=5, hybrid=2, vfrDec=1, input="TDecimate1.txt", tfmin="TFM1", mkvOut="Timecodes1.txt", tcfv1=false)

Make sure to save this file as Video1.avi or something you can easily remember and that you can pair with Timecodes1.txt.

File #2, Pass #1:
Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("audio.ac3")
AudioDub(Movie, Audio)
IVTC=Last.QTGMC(preset="very slow").SelectOdd()
TFM(PP=2, clip2=IVTC, output="TFM2.txt")
TDecimate(mode=4, output="TDecimate2.txt")

File #2, Pass #2
Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("audio.ac3")
AudioDub(Movie, Audio)
IVTC=Last.QTGMC(preset="very slow").SelectEven()
TFM(PP=2, clip2=IVTC,input="TFM2.txt")
TDecimate(mode=5, hybrid=2, vfrDec=1, input="TDecimate2.txt", tfmin="TFM2", mkvOut="Timecodes2.txt", tcfv1=false)

Make sure to save this file as Video2.avi or something you can easily remember and that you can pair with Timecodes2.txt. The file you have here is almost identical, however SelectOdd() was used in place of SelectEven(). This means that you now have two video files, with identical timings and identical 24fps sections, but alternate frames in the 30fps sections.

The last bit here, will load each video up with their corresponding timecodes at 120fps (which is divisible by both 30 and 24) meaning the timings will be correct. You then interleave frames from the two videos (and potentially double the FPS to take this into account) and you have a 240fps video that plays each frame at the correct time and otherwise leaves behind a string of identical duplicates - which can be easily handled by ExactDedup and then muxed into a 60/24 VFR clip.

Final clip, pass #1:
Clip1=AVISource("video1.avi").VFRToCFR(times="Timecodes1.txt",numfps=120000,denfps=1001)
Clip2=AVISource("video2.avi").VFRToCFR(times="Timecodes2.txt",numfps=120000,denfps=1001)
Interleave(Clip1,Clip2).AssumeFPS(numerator=240000,denominator=1001)
ExactDedup(firstpass=true, dupinfo="info.txt", times="Timecodes.txt")

Final clip, pass #2:
Clip1=AVISource("video1.avi").VFRToCFR(times="Timecodes1.txt",numfps=120000,denfps=1001)
Clip2=AVISource("video2.avi").VFRToCFR(times="Timecodes2.txt",numfps=120000,denfps=1001)
Interleave(Clip1,Clip2).AssumeFPS(numerator=240000,denominator=1001)
ExactDedup(firstpass=false, dupinfo="info.txt", times="Timecodes.txt")

I haven't had time to test this out properly, but I'm almost certain it will work despite how tedious it may be to do so many passes. All the frames should display at the correct times in any of the sections.

[StainlessS]

thecoreyburton,

I have not followed entire thread, but here an attempt to get all working in single script.
Untested, and probably at least a bit broken.


# FILE COMMON
Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("audio.ac3")
AudioDub(Movie, Audio)
IN = Last
QTG = Last.QTGMC(preset="very slow")
# EVEN
IVTC_E = QTG.SelectEven()
# EVEN PASS 1
Last.TFM(PP=2, clip2=IVTC_E, output="TFM1.txt")
Last.TDecimate(mode=4, output="TDecimate1.txt")
Last.ForceProcessAVI() # Force Write log files
# EVEN PASS 2
IN
Last.TFM(PP=2, clip2=IVTC_E, input="TFM1.txt")
Last.TDecimate(mode=5, hybrid=2, vfrDec=1, input="TDecimate1.txt", tfmin="TFM1", mkvOut="Timecodes1.txt", tcfv1=false)
Last.TWriteAVI("Video1.Avi",Overwrite=True,FourCC="ULY0")
Last.ForceProcessAVI() # Force Write Lossless
IN
# ODD (I AM ASSUMING that both input video.d2v's and audio.ac3's are actually the SAME files)
IVTC_O = QTG.SelectOdd()
# ODD PASS 1
Last.TFM(PP=2, clip2=IVTC_O, output="TFM2.txt")
Last.TDecimate(mode=4, output="TDecimate2.txt")
Last.ForceProcessAVI() # Force Write log files
# ODD PASS 2
IN
Last.TFM(PP=2, clip2=IVTC_O,input="TFM2.txt")
Last.TDecimate(mode=5, hybrid=2, vfrDec=1, input="TDecimate2.txt", tfmin="TFM2", mkvOut="Timecodes2.txt", tcfv1=false)
Last.TWriteAVI("Video2.Avi",Overwrite=True,FourCC="ULY0")
Last.ForceProcessAVI() # Force Write Lossless
#######
####### FINAL CLIP
#######
Clip1=AVISource("video1.avi").VFRToCFR(times="Timecodes1.txt",numfps=120000,denfps=1001)
Clip2=AVISource("video2.avi").VFRToCFR(times="Timecodes2.txt",numfps=120000,denfps=1001)
Interleave(Clip1,Clip2).AssumeFPS(numerator=240000,denominator=1001)
ILEAVE=Last
ExactDedup(firstpass=true, dupinfo="info.txt", times="Timecodes.txt")
Last.ForceProcessAVI() # Force Pass 1
ILEAVE # Not sure what ExactDedup returns so use ILEAVE (same)
ExactDedup(firstpass=false, dupinfo="info.txt", times="Timecodes.txt")
### Here, Could use TWriteAVI and ForceProcessAVI, and return eg MessageClip("ALL DONE")


Good luck.

EDIT: I AM ASSUMING that both input video.d2v's and audio.ac3's are actually the SAME files.
EDIT: Cannot use TwriteAVI with MT, it will crash.
EDIT: When working, you might want to try creating a DGIndex Template file, and create output and temp file names based on
input video filename, and maybe skip audio dub until last step to avoid writing audio multiple times to lossless files.

[StainlessS]

Below as for last EDIT in above post


VideoFileName="video.d2v"
AudioFileName="audio.ac3"
AudioDelay=0.0
####################### DGIndex TEMPLATE (UNCOMMENT and delete above)
#VideoFileName = "__vid__"
#AudioFileName = LCase("__aud__")
#AudioDelay = Value("__del__")
####################### Ensure Full Path Names (instead of relative)
VideoFileName=RT_GetFullPathName(VideoFileName)
AudioFileName=RT_GetFullPathName(AudioFileName)
#######################
FOURCC = "ULY0" # UtVideo
####
TFM1 = RT_String("%s.TFM1.txt",VideoFileName)
TD1 = RT_String("%s.TDecimate1.txt",VideoFileName)
TIMECODES1 = RT_String("%s.Timecodes1.txt",VideoFileName)
LOSSLESS1 = RT_String("%s.Video1.AVI",VideoFileName)
#
TFM2 = RT_String("%s.TFM2.txt",VideoFileName)
TD2 = RT_String("%s.TDecimate2.txt",VideoFileName)
TIMECODES2 = RT_String("%s.Timecodes2.txt",VideoFileName)
LOSSLESS2 = RT_String("%s.Video2.AVI",VideoFileName)
#
DUPINFO = RT_String("%s.Info.txt",VideoFileName)
TIMECODES = RT_String("%s.Timecodes.txt",VideoFileName)
FINAL = RT_String("%s.FINAL.AVI",VideoFileName)
###
myName = "Thecoreyburton: "
#######################
# FILE COMMON
Movie=DGDecode_MPEG2Source(VideoFileName)
Audio=FFAudioSource(AudioFileName)
AudioDub(Movie, Audio)
DelayAudio(AudioDelay)
##### Un-Comment if AudioDubbing at end of script (avoid write audio multiple times to lossless), Keep above audio stuff (find out now if works ok)
#KillAudio()
#####
#
IN = Last
QTG = Last.QTGMC(preset="very slow")
# EVEN
IVTC_E = QTG.SelectEven()
# EVEN PASS 1
Last.TFM(PP=2, clip2=IVTC_E, output=TFM1)
Last.TDecimate(mode=4, output=TD1)
RT_DebugF("EVEN FORCE PASS 1",name=myName)
Last.ForceProcessAVI() # Force Write log files
# EVEN PASS 2
IN
Last.TFM(PP=2, clip2=IVTC_E, input=TFM1)
Last.TDecimate(mode=5, hybrid=2, vfrDec=1, input=TD1, tfmin=TFM1, mkvOut=TIMECODES1, tcfv1=false)
Last.TWriteAVI(LOSSLESS1,Overwrite=True,FourCC=FOURCC)
RT_DebugF("EVEN FORCE PASS 2",name=myName)
Last.ForceProcessAVI() # Force Write Lossless
IN
# ODD (I AM ASSUMING that both input video.d2v's and audio.ac3's are actually the SAME files)
IVTC_O = QTG.SelectOdd()
# ODD PASS 1
Last.TFM(PP=2, clip2=IVTC_O, output=TFM2)
Last.TDecimate(mode=4, output=TD2)
RT_DebugF("ODD FORCE PASS 1",name=myName)
Last.ForceProcessAVI() # Force Write log files
# ODD PASS 2
IN
Last.TFM(PP=2, clip2=IVTC_O,input=TFM2)
Last.TDecimate(mode=5, hybrid=2, vfrDec=1, input=TD2, tfmin=TFM2, mkvOut=TIMECODES2, tcfv1=false)
Last.TWriteAVI(LOSSLESS2,Overwrite=True,FourCC=FOURCC)
RT_DebugF("ODD FORCE PASS 2",name=myName)
Last.ForceProcessAVI() # Force Write Lossless
#######
####### FINAL CLIP
#######
Clip1=AVISource(LOSSLESS1).VFRToCFR(times=TIMECODES1,numfps=120000,denfps=1001)
Clip2=AVISource(LOSSLESS2).VFRToCFR(times=TIMECODES2,numfps=120000,denfps=1001)
Interleave(Clip1,Clip2).AssumeFPS(numerator=240000,denominator=1001)
ILEAVE=Last
ExactDedup(firstpass=true, dupinfo=DUPINFO, times=TIMECODES)
RT_DebugF("EXACTDUP FORCE PASS 1",name=myName)
Last.ForceProcessAVI() # Force Pass 1
ILEAVE # Not sure what ExactDedup returns so use ILEAVE (same)
ExactDedup(firstpass=false, dupinfo=DUPINFO, times=TIMECODES)
### Here, Could use TWriteAVI and ForceProcessAVI (write final LOSSLESS) , and return eg MessageClip("ALL DONE")
#AudioDub(Last, Audio) # Avoided writing audio to temp lossless
#DelayAudio(AudioDelay)
#Last.TWriteAVI(FINAL,Overwrite=True,FourCC=FOURCC)
#RT_DebugF("FINAL FORCE WRITE LOSSLESS",name=myName)
#Last.ForceProcessAVI() # Force Write Final AVI file
#MessageClip("ALL DONE")
Return Last

Still untested.
EDIT Req RT_Stats and TWriteAVI v2.0

[StainlessS]

Assuming that previous script works ok, then try below and compare timings (Attempted to re-use QTGMC cached frames for EVEN/ODD)


VideoFileName="video.d2v"
AudioFileName="audio.ac3"
AudioDelay=0.0
####################### DGIndex TEMPLATE (UNCOMMENT and delete above)
#VideoFileName = "__vid__"
#AudioFileName = "__aud__"
#AudioDelay = Value("__del__")
####################### Ensure Full Path Names (instead of relative)
VideoFileName=RT_GetFullPathName(VideoFileName)
AudioFileName=RT_GetFullPathName(AudioFileName)
#######################
FOURCC = "ULY0" # UtVideo
####
TFM1 = RT_String("%s.TFM1.txt",VideoFileName)
TD1 = RT_String("%s.TDecimate1.txt",VideoFileName)
TIMECODES1 = RT_String("%s.Timecodes1.txt",VideoFileName)
LOSSLESS1 = RT_String("%s.Video1.AVI",VideoFileName)
#
TFM2 = RT_String("%s.TFM2.txt",VideoFileName)
TD2 = RT_String("%s.TDecimate2.txt",VideoFileName)
TIMECODES2 = RT_String("%s.Timecodes2.txt",VideoFileName)
LOSSLESS2 = RT_String("%s.Video2.AVI",VideoFileName)
#
DUPINFO = RT_String("%s.Info.txt",VideoFileName)
TIMECODES = RT_String("%s.Timecodes.txt",VideoFileName)
FINAL = RT_String("%s.FINAL.AVI",VideoFileName)
###
myName = "Thecoreyburton: "
#######################
# FILE COMMON
Movie=DGDecode_MPEG2Source(VideoFileName)
Audio=FFAudioSource(AudioFileName)
AudioDub(Movie, Audio)
DelayAudio(AudioDelay)
##### Un-Comment if AudioDubbing at end of script (avoid write audio multiple times to lossless), Keep above audio stuff (find out now if works ok)
#KillAudio()
#####
#
IN = Last
QTG = Last.QTGMC(preset="very slow")
IVTC_E = QTG.SelectEven()
IVTC_O = QTG.SelectOdd()
# PASS 1 (BOTH Even and ODD), try to re-use cached frames from QTGMC()
TFM_E1 = IN.TFM(PP=2, clip2=IVTC_E, output=TFM1)
TFM_O1 = IN.TFM(PP=2, clip2=IVTC_O, output=TFM2)
TD_E1 = TFM_E1.TDecimate(mode=4, output=TD1)
TD_O1 = TFM_O1.TDecimate(mode=4, output=TD2)
RT_GraphLink(TD_E1,TD_O1) # Similar to Echo() in v2.6 (Link BOTH EVEN and ODD chains in filter graph)
RT_DebugF("FORCE PASS 1 (Write LOG files)",name=myName)
Last.ForceProcessAVI() # Force Write both EVEN and ODD log files
# PASS 2 (BOTH Even and ODD)
TFM_E2 = IN.TFM(PP=2, clip2=IVTC_E, input=TFM1)
TFM_O2 = IN.TFM(PP=2, clip2=IVTC_O, input=TFM2)
TD_E2 = TFM_E2.TDecimate(mode=5, hybrid=2, vfrDec=1, input=TD1, tfmin=TFM1, mkvOut=TIMECODES1, tcfv1=false)
TD_O2 = TFM_O2.TDecimate(mode=5, hybrid=2, vfrDec=1, input=TD2, tfmin=TFM2, mkvOut=TIMECODES2, tcfv1=false)
TD_E2 = TD_E2.TWriteAVI(LOSSLESS1,Overwrite=True,FourCC=FOURCC) # Write Even Lossless
TD_O2 = TD_O2.TWriteAVI(LOSSLESS2,Overwrite=True,FourCC=FOURCC) # Write Odd Lossless
RT_GraphLink(TD_E2,TD_O2) # Similar to Echo() in v2.6 (Link BOTH EVEN and ODD chains in filter graph)
RT_DebugF("FORCE PASS 2 (Write LOSSLESS files)",name=myName)
Last.ForceProcessAVI() # Force Write both EVEN and ODD LOSSLESS files
#######
####### FINAL CLIP
#######
Clip1=AVISource(LOSSLESS1).VFRToCFR(times=TIMECODES1,numfps=120000,denfps=1001)
Clip2=AVISource(LOSSLESS2).VFRToCFR(times=TIMECODES2,numfps=120000,denfps=1001)
Interleave(Clip1,Clip2).AssumeFPS(numerator=240000,denominator=1001)
ILEAVE=Last
ExactDedup(firstpass=true, dupinfo=DUPINFO, times=TIMECODES)
RT_DebugF("EXACTDUP FORCE PASS 1",name=myName)
Last.ForceProcessAVI() # Force Pass 1
ILEAVE # Not sure what ExactDedup returns so use ILEAVE (same)
ExactDedup(firstpass=false, dupinfo=DUPINFO, times=TIMECODES)
#AudioDub(Last, Audio) # IF Avoided writing audio to temp lossless
#DelayAudio(AudioDelay)
#####
### Here, Could use TWriteAVI and ForceProcessAVI (write final LOSSLESS) , and return eg MessageClip("ALL DONE")
#Last.TWriteAVI(FINAL,Overwrite=True,FourCC=FOURCC)
#RT_DebugF("FINAL FORCE WRITE LOSSLESS",name=myName)
#Last.ForceProcessAVI() # Force Write Final AVI file
#MessageClip("ALL DONE")
###
Return Last

[thecoreyburton]

Thank you for all your effort, I looked over all three of your scripts and adjustments and I'm very pleased. I'll hopefully have a chance to rip the source again and test it within a week, it depends on how much free time I get for this sort of thing. Either way, your work is greatly appreciated! I do have a few questions though:

1) Just to double check I understand correctly, changing the file name changes the name of the log files and such, so if I run this script for two differently named files in the same directory there would be no clashes?
2) Assuming 1 is true and it works the way we expect, could it be turned into an .avsi function for easy repetitive use?
3) How big are the files written by with TWriteAVI in comparison with other lossless codecs / what codec is used?

[StainlessS]

1) Yes. All log files based on VideoFileName, eg for 'fred.d2v' then "fred.d2v.TDecimate1.txt".
Files will be created in the directory of the VideoFileName, and if "fred.d2v" is currently in "D:\BullPen\" then name converted to "D:\BullPen\fred.d2v.TDecimate1.txt".
(assuming VideoFileName does not already have an explicit path).

2) Yes, although was intended that you (when happy that it works as it is), just delete the hard coded filenames and uncomment the DGIndex template stuff,
then can simply use DGIndex to create the avs files.

ie DGIndex Template stuff EDIT: DGIndex fills in the stuff below in BLUE, Choose TEMPLATE in DGIndex Menu

####################### DGIndex TEMPLATE (UNCOMMENT and delete above)
VideoFileName = "__vid__"
AudioFileName = "__aud__"
AudioDelay = Value("__del__")

And the filename Full path stuff

####################### Ensure Full Path Names (instead of relative)
VideoFileName=RT_GetFullPathName(VideoFileName)
AudioFileName=RT_GetFullPathName(AudioFileName)


You could create AVSI function with Args VideoFileName, AudioFileName and AudioDelay.

3), In above scripts using FourCC "ULY0" ie UT_Video YV12 codec, same size as written by VDub (internal source of TWriteAVI v2.0 is vdub source).
Change FourCC to "LAGS" or whatever [EDIT: Lagarith].

FOURCC = "ULY0" # UtVideo


EDIT: Suggest rip just 5 or 10 minutes to test that it does not crash after hours of work.
Has had ZERO testing done.

EDIT: Below, inserts VideoFileName String where the string insertion marker '%s' is, giving eg "fred.d2v.TFM1.txt".

TFM1 = RT_String("%s.TFM1.txt",VideoFileName)

'%d' would insert an int var, '%f' a float var. (args following the format string [ie "%s.TFM1.txt"] MUST match the insertion
marker types in the format string).

EDIT: I think that the only error I spotted in your original script was here:-

Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("audio.ac3")
AudioDub(Movie, Audio)
IVTC=Last.QTGMC(preset="very slow").SelectEven()
TFM(PP=2, clip2=IVTC,input="TFM1.txt")
TDecimate(mode=5, hybrid=2, vfrDec=1, input="TDecimate1.txt", tfmin="TFM1.txt", mkvOut="Timecodes1.txt", tcfv1=false)

Missing ".txt", same for matching ODD TFM2.txt.

[thecoreyburton]

I left a preliminary test on an hour long video with my own scripts on when I left this morning and it just finished, the result was a success (I corrected the TFM oversight too, when it popped up). Some portions of the video play at 60fps and some portions play at 24fps. Edit: Some portions jump backwards and forwards by a frame, see 1) below.

I'll try the same source again with your latest script and I'll let you know how it goes, though I don't see why the results would be any different.

The only thought I have left on this is handling the non-interlaced 30fps portions. I'm quite happy to leave it where it is in regards to 60/24 as the frames all display at the correct times and the motion looks as smooth as it should, but the perfectionist in me is thinking away. I have three thoughts at this point:

1) A potential flaw in my original logic (and therefore the idea as a whole) is that it imports at 120fps to make sure the 24fps and 60fps moments all occur at the specific moment in time that they should. Because of the import, in the 60fps sections each moment of time occurs exactly twice. By interleaving this with the other clip, technically it'll be going moment1, moment2, moment1, moment2. It may be better to interleave the frames in groups of two, eg: 1a,2a,1b,2b,3a,4a,3b,4b. The letters being the clip they come from and the numbers being the frame numbers from that clip. I'm not sure, it could even happen more than once seeing the source is 30fps loaded up as 120fps.

2) Following up on that last point. When the two clips are imported in the final step, they're imported at 120fps. This is divisible by all of the desired rates (24,30,60). Because they're interleaved, there are twice as many frames and therefore the clip needs to be played twice as fast, hence the need for 240fps. To save time, would it be better to cut the number of frames in half instead, seeing as it's just redundant data?

3) This script is good for detelecining the 24fps parts, and deinterlacing the interlaced parts, but there's always the very likely change that there will be some portions of the video at 30fps that are progressive. To decimate these properly in the final step, they need to be identical duplicates, which if they've been passed through QTGMC, they're not. I've never used IsCombedTIVTC in my own scripts so I'm not fully familiar with it, but could something like as an alternative to the first passes this solve the final issue:


Movie=DGDecode_MPEG2Source("video.d2v")
Audio=FFAudioSource("audio.ac3")
AudioDub(Movie, Audio)
#Set up the reference footage for where deinterlacing should occur.
#Change the fps to 60, duplicating every frame including combed frames.
FrameCheck=Last.ChangeFPS(Last.FPS*2)
#Set up the deinterlaced footage
IVTC=Last.QTGMC(preset="very slow").SelectEven()
#Set up the final replacement footage by checking and combining the two above.
#I'm not sure how to use this function correctly, but I'd like to check FrameCheck for combed frames and if so, replace the frame with the same frame from IVTC.
ReplacementFootage=FrameCheck.IsCombedTIVTC(IVTC)
TFM(PP=2, clip2=ReplacementFootage, output="TFM1.txt")
TDecimate(mode=4, output="TDecimate1.txt")


In theory, I'm hoping that ReplacementFootage will have only deinterlaced the frames that need deinterlacing but there's every chance I'm overthinking, overcomplicating or missing something in this. It's not even necessary as the 60fps captures the motion of the 30fps sections perfectly, it's just a bit of a nitpick in regards to compression and consistency.

[StainlessS]

Cant help I'm afraid, never touched that kind of stuff and dont really want to :)

I am though curious as to whether or not the final script gave same results as you original script,
and whether it was faster than the one prior to it (because of QTGMC frame cache re-use).
The only reason I posted, was just to save you the trouble of having to do half a dozen or so passes.
Anyway, I hope you can figure out what the final script was doing.
(I have no idea what the underlying script ie TFM/TDEcimate et al, is doing, never touched VFR).

EDIT: "frame cache re-use",
we used simultaneous EVEN and ODD metrics writing (and same for Lossless writing) due to linking both filter sub
graphs together (RT_GraphLink), so that adjacent even/odd frames would be accessed at about the same time,
and due to caching of QTGMC frames would hopefully not have to do MC stuff twice.
(although it would still of course need both metrics writing and lossless writing passes without caching between
those passes).

EDIT: And here as DGIndex template, and other suggested mods also enabled.

####################### DGIndex TEMPLATE
VideoFileName = "__vid__"
AudioFileName = "__aud__"
AudioDelay = Value("__del__")
####################### Ensure Full Path Names (instead of relative)
VideoFileName=RT_GetFullPathName(VideoFileName)
AudioFileName=RT_GetFullPathName(AudioFileName)
#######################
FOURCC = "ULY0" # UtVideo
####
TFM1 = RT_String("%s.TFM1.txt",VideoFileName)
TD1 = RT_String("%s.TDecimate1.txt",VideoFileName)
TIMECODES1 = RT_String("%s.Timecodes1.txt",VideoFileName)
LOSSLESS1 = RT_String("%s.Video1.AVI",VideoFileName)
#
TFM2 = RT_String("%s.TFM2.txt",VideoFileName)
TD2 = RT_String("%s.TDecimate2.txt",VideoFileName)
TIMECODES2 = RT_String("%s.Timecodes2.txt",VideoFileName)
LOSSLESS2 = RT_String("%s.Video2.AVI",VideoFileName)
#
DUPINFO = RT_String("%s.Info.txt",VideoFileName)
TIMECODES = RT_String("%s.Timecodes.txt",VideoFileName)
FINAL = RT_String("%s.FINAL.AVI",VideoFileName)
###
myName = "Thecoreyburton: "
#######################
# FILE COMMON
Movie=DGDecode_MPEG2Source(VideoFileName)
Audio=FFAudioSource(AudioFileName)
AudioDub(Movie, Audio)
DelayAudio(AudioDelay)
##### AudioDubbing at end of script (avoid write audio multiple times to lossless), Keep above audio stuff (find out now if works ok)
KillAudio()
#####
#
IN = Last
QTG = Last.QTGMC(preset="very slow")
IVTC_E = QTG.SelectEven()
IVTC_O = QTG.SelectOdd()
# PASS 1 (BOTH Even and ODD), try to re-use cached frames from QTGMC()
TFM_E1 = IN.TFM(PP=2, clip2=IVTC_E, output=TFM1)
TFM_O1 = IN.TFM(PP=2, clip2=IVTC_O, output=TFM2)
TD_E1 = TFM_E1.TDecimate(mode=4, output=TD1)
TD_O1 = TFM_O1.TDecimate(mode=4, output=TD2)
RT_GraphLink(TD_E1,TD_O1) # Similar to Echo() in v2.6 (Link BOTH EVEN and ODD chains in filter graph)
RT_DebugF("FORCE PASS 1 (Write LOG files)",name=myName)
Last.ForceProcessAVI() # Force Write both EVEN and ODD log files
# PASS 2 (BOTH Even and ODD)
TFM_E2 = IN.TFM(PP=2, clip2=IVTC_E, input=TFM1)
TFM_O2 = IN.TFM(PP=2, clip2=IVTC_O, input=TFM2)
TD_E2 = TFM_E2.TDecimate(mode=5, hybrid=2, vfrDec=1, input=TD1, tfmin=TFM1, mkvOut=TIMECODES1, tcfv1=false)
TD_O2 = TFM_O2.TDecimate(mode=5, hybrid=2, vfrDec=1, input=TD2, tfmin=TFM2, mkvOut=TIMECODES2, tcfv1=false)
TD_E2 = TD_E2.TWriteAVI(LOSSLESS1,Overwrite=True,FourCC=FOURCC) # Write Even Lossless
TD_O2 = TD_O2.TWriteAVI(LOSSLESS2,Overwrite=True,FourCC=FOURCC) # Write Odd Lossless
RT_GraphLink(TD_E2,TD_O2) # Similar to Echo() in v2.6 (Link BOTH EVEN and ODD chains in filter graph)
RT_DebugF("FORCE PASS 2 (Write LOSSLESS files)",name=myName)
Last.ForceProcessAVI() # Force Write both EVEN and ODD LOSSLESS files
####### FINAL CLIP
Clip1=AVISource(LOSSLESS1).VFRToCFR(times=TIMECODES1,numfps=120000,denfps=1001)
Clip2=AVISource(LOSSLESS2).VFRToCFR(times=TIMECODES2,numfps=120000,denfps=1001)
Interleave(Clip1,Clip2).AssumeFPS(numerator=240000,denominator=1001)
ILEAVE=Last
ExactDedup(firstpass=true, dupinfo=DUPINFO, times=TIMECODES)
RT_DebugF("EXACTDUP FORCE PASS 1",name=myName)
Last.ForceProcessAVI() # Force Pass 1
ILEAVE # Not sure what ExactDedup returns so use ILEAVE (same)
ExactDedup(firstpass=false, dupinfo=DUPINFO, times=TIMECODES)
AudioDub(Last, Audio) # Avoided writing audio to temp lossless
DelayAudio(AudioDelay)
Last.TWriteAVI(FINAL,Overwrite=True,FourCC=FOURCC)
RT_DebugF("FINAL FORCE WRITE LOSSLESS",name=myName)
Last.ForceProcessAVI() # Force Write Final AVI file
MessageClip("ALL DONE")
Return Last

[thecoreyburton]

No problem, I'll do a run through with your script and how fast it runs and what result it produces and go from there. I do have a pretty decent idea on how to fix my scripts up to account for most of the problems I mentioned, so I'll just incorporate those fixes into yours if it's successful one by one.

[StainlessS]

"I do have a pretty decent idea"

Yep, I can see that, as can anyone else, to knock up about 1/2 dozen scripts without testing and only mistake being
to miss out ".txt" a couple of times, not bad going at all.

I cannot guarantee my scripts have as few errors, let me know if there were any.

[thecoreyburton]

The script's running on my home PC as we speak, I should be home on Monday to have a look at the end result.

I managed to make a table of the potential flaw in my original script, I'm not sure how to go about solving it or if it even can be solved without the aid of a new plugin that's specifically designed to do what TFM and TDecimate do but to also factor in 60fps in the VFR side of things.

http://i65.tinypic.com/34pykhi.jpg

The first result shows the fluctuations I noticed when I had a look at the output of my original script. Not to mention the fields get reversed after the 24fps section. The second result was a solution I was hopeful for, but due to the nature of the 24fps sections again, it is thrown out depending on their length. I'd also like to point out to anyone else reading this, that this pattern is not consistent, the duration of each section is entirely dependent on the source.

Edit: I had a shower-thought moment; perhaps this could be done the following way. This isn't actually a script but rather an idea I'll expand upon when I get time. I just thought I'd pop it up here in case you or anyone else see any potential flaws in it before I put the effort into making and testing it.

Load video clip
TFM with no post processing (so frames that aren't field matched remain interlaced).
Two-pass TDecimate as before, leaving the clip VFR with both 24fps progressive segments and 30fps interlaced segments.
Write this clip to a temporary file, videoname-clip1.

Load video clip
QTGMC with appropriate settings
Write this clip to a temporary file, videoname-clip2.
(provided we use a modified version of your script, writing won't be necessary, it can just be used as is for the next step)

Load both clips up at 120fps (use the original timecode method for clip1, double frame rate of clip2).
Use IsCombedIVTC* to check for combed frames in clip1, replace those frames with frames from clip 2.
Two pass ExactDedup to write timecodes and output final file.

*Potentially could be repaced TFM with pp=2 (because it'll have no fields to match, it'll work the same way and I'm more familiar with it).

This has many advantages, as the 30fps sections that are not interlaced won't be replaced with the QTGMC'd frames, meaning the clip would be able to display 30fps, 24fps and 60fps frames as they all should be. It should also be somewhat faster as there are less passes.

Field order in clip1 shouldn't matter either, because the clip's replacing frames from an already deinterlaced clip. That being said, there could be alignment or timing problems and I've been hopeful before about an idea on this topic so I'm not holding my breath.

[StainlessS]

thecoreyburton,
Just spotted a stupid omission in the TWriteAVI scripts.
We should write the QTGMC result to lossless (TWriteAVI and ForceProcessAVI) prior to anything else, and then reload
and assign to eg "QTG", and then use that for writing metrics and Lossless passes.

Probably even be better to

QTG = Last.QTGMC(preset="very slow").TWriteAVI (QTG_FILE,...etc) # without ForceProcessAVI)


and then this

IVTC_E = QTG.SelectEven()
IVTC_O = QTG.SelectOdd()
# PASS 1 (BOTH Even and ODD), try to re-use cached frames from QTGMC()
TFM_E1 = IN.TFM(PP=2, clip2=IVTC_E, output=TFM1)
TFM_O1 = IN.TFM(PP=2, clip2=IVTC_O, output=TFM2)
TD_E1 = TFM_E1.TDecimate(mode=4, output=TD1)
TD_O1 = TFM_O1.TDecimate(mode=4, output=TD2)
RT_GraphLink(TD_E1,TD_O1) # Similar to Echo() in v2.6 (Link BOTH EVEN and ODD chains in filter graph)
RT_DebugF("FORCE PASS 1 (Write LOG files)",name=myName)
Last.ForceProcessAVI() # Force Write both EVEN and ODD log files AND QTG_FILE
# PASS 2 (BOTH Even and ODD)

QTG=AviSource(QTG_FILE)
IVTC_E = QTG.SelectEven()
IVTC_O = QTG.SelectOdd()

TFM_E2 = IN.TFM(PP=2, clip2=IVTC_E, input=TFM1)
TFM_O2 = IN.TFM(PP=2, clip2=IVTC_O, input=TFM2)
# etc


Simultaneous creation of QTGMC file and metrics, and then for Lossless creation, re-use QTG file.

EDIT: Possible problem, after the first code block, ALL requests to QTGMC().TwriteAVI() need to be strictly sequential,
not sure how RequestLinear() works but that might solve it [EDIT: maybe between the two code blocks], otherwise need to use 1st suggestion
and force write complete QTGMC file before metrics created.

[thecoreyburton]

What's happens if we don't do that? Are frames dropped / accuracy lost? I'm not sure about RequestLinear() either, but I don't see a problem in writing the QTG file. I agree with you on the placement, between the two code blocks looks like it should work nicely.

I'll have a few days coming up where I can try get a 20 minute video to work properly. It has segments of all three frame rates, so whilst it's long at least the test will be thorough.

[StainlessS]

"What's happens if we don't do that? Are frames dropped",

I'm kinda curious bout that myself :)

Most of the given script should be ok, have given the EVEN chain first in order to RT_GraphLink,
but dont know how TFM accesses IVTC_E and IVTC_O.
Hopefully RequestLinear would sort probs out (and additionally insert an extra cache buffer between code blocks code),
but I have posted before about peculiar behavior of the new 2.6 cache where it seems to not cache every 30th frame.

TWriteAVI must have sequential frames requested from it (you cannot write an AVI file with out-of-order frames
and expect it to play in correct order).


n = (n<0) ? n : (n >= vi.num_frames) ? vi.num_frames-1 : n; // Range limit n to valid frames
PVideoFrame frame = child->GetFrame(n, env);
if(out != NULL && lpBuffer != NULL && n == NextFrame) {

// WRITE FRAME STUFF HERE

NextFrame = n + 1;
if(NextFrame>=vi.num_frames) {
DPRINTF("GetFrame Complete")
Close(false,true,false); // Last frame, Write no more frames, allow re-open of written file for read.
}
}


Above, will ignore frames requested that are non strictly sequential, and wait for next in sequence, or end of requests and destructor.
Might be prudent to remember initial FrameCount, and compare with result of QTG_File=AviSource() : QTG_File.FrameCount().
Throw an error if not same.

EDIT: I guess that TWriteAVI could be changed to write all frames missing in sequence up to requested frame, and then return
requested frame to following filter.

EDIT: JFYI, ForceProcessAVI() requests ALL frames [EDIT: and audio samples] in strictly sequential order, it is a function (not filter) and returns only on completion.

EDIT: Just spotted a NASTY bug in above code, in RED, ie

n = (n<0) ? n : (n >= vi.num_frames) ? vi.num_frames-1 : n;

should be

n = (n<0) ? 0 : (n >= vi.num_frames) ? vi.num_frames-1 : n;


Arh well, gotta do bug fix version anyway so will implement as suggested above.

[StainlessS]

TWriteAVI v2.03, should sort out all previously mentioned problems.

http://forum.doom9.org/showthread.php?t=172837

[thecoreyburton]

Thank you for that and all your help so far too, I'll grab the updated version now.

I managed to do a lot of trial-and-error last night and have finally got a series of scripts that produce a pretty accurate result without all of the bugs I mentioned earlier. It's not entirely without problems at this point but with a bit of tweaking it should be good. I tried to remove some of the more redundant components in the individual scripts this time too. For assumptions sake in these scripts

First pass of the reference clip that I'll be calling "clip01". It uses TFM to field-match all sections that should be at 24fps and leaves the remaining frames interlaced. This doesn't perform any decimation.
DGDecode_MPEG2Source("video.d2v")
TFM(PP=0, output="TFM.txt")
TDecimate(mode=4, output="Tdecimate.txt")

Second pass of "Clip01". Same as before, but this time it actually performs the frame decimation where appropriate and outputs timecodes to go with this. This is saved losslessly as Clip01.avi.
DGDecode_MPEG2Source("video.d2v")
TFM(PP=0, input="TFM.txt")
TDecimate(mode=5, hybrid=2, vfrDec=1, input="TDecimate.txt", tfmin="TFM.txt", mkvOut="Clip01TC.txt", tcfv1=false)

This is the deinterlaced clip that I'll be referring to as "Clip02". It interlaces the entire video to 60fps. This is saved losslessly as Clip02.avi.
DGDecode_MPEG2Source("video.d2v")
QTGMC(Preset="Very Slow")

First pass of the output clip. This loads up both clips at 120fps and replaces the interlaced frames in Clip01 with the deinterlaced frames from Clip02. Because the check only affects combed frames, it means that 30fps sections and the field-matched 24fps sections from earlier are unaffected and remain as a string of duplicates allowing for easy decimation by ExactDedup. IsCombedTIVTC could potentially replace TFM in this script but it does work like this too.
Clip01=AVISource("Clip01.avi").VFRToCFR(times="Clip01TC.txt",numfps=120000,denfps=1001).ConvertToYV12()
Clip02=AVISource("Clip02.avi").ChangeFPS(Clip01.Framerate).ConvertToYV12()
Audio=FFAudioSource("audio.ac3")
Clip01.TFM(pp=2, clip2=Clip02)
Audiodub(Last,Audio)
ExactDedup(firstpass=true, dupinfo="Duplicates.txt", times="Timecodes.txt")

This is the same as the one above, but as a second pass. This is the clip that will be outputted.
Clip01=AVISource("Clip01.avi").VFRToCFR(times="Clip01TC.txt",numfps=120000,denfps=1001).ConvertToYV12()
Clip02=AVISource("Clip02.avi").ChangeFPS(Clip01.Framerate).ConvertToYV12()
Audio=FFAudioSource("audio.ac3")
Clip01.TFM(pp=2, clip2=Clip02)
Audiodub(Last,Audio)
ExactDedup(firstpass=false, dupinfo="Duplicates.txt", times="Timecodes.txt")

The resulting clip, after being passed through my scripts had frame rates of 24, 30 and 60 and all timings were in the correct order. I might have a shot at putting it all into a single script using your earlier one as a template, but no guarantees on my skill set there haha. There are also two bugs at this point.

Bug #1: To reach 120fps, each frame is duplicated. In 60fps it's duplicated once, in 30 four times. In 24fps it's duplicated five times. Because of this, after a string of 24fps sections it may end on an odd frame instead of an even frame. This means that the next frame will be loaded instantly, meaning for that split second it plays at 120fps. I'm not sure what negative effects this may have on players or the final output. My PC handles it fine and the timing doesn't look off, but I don't know about hardware restrictions. I'm currently looking for a way to hard-cap this or work around it.

Bug #2: For similar reasons, the clips' durations may be off by several frames (I don't see this being over 10). Because the QTGMC clip is an exact double of the actual footage, after loading both clips at 120fps I used this as the reference and added 10 blank frames to the other clip, then a trim command using the QTGMC's framecount. I'm not sure if a better way exists and for that purpose I didn't include this in the scripts above. When using those scripts in a real world situation you'll get an error about the fact they don't match in duration unless you add something like this.

Still a little bit of a way to go, but I'm glad this is actually shaping up to be something worthwhile. Thank you again for all your assistance and apologies if there are any errors in either my scripts or my writeup in general!

[StainlessS]

Is it necessary to dub audio for both passes to ExactDeDup ?, I would have thought doing audioDub at very last stage would be ok.

Hope you get the bugs 1 & 2 sorted, I'll leave it to you to insert the TWriteAVI stuff, nice little exercise for you.
If you have any problems though, just holla.

Groucho2004 ICL10 DGDecode v1.5.8, might be worth a shot, I've DL'ed it [EDIT: and using] but not done any speed comparisons.

[Katie Boundary]

They have telecined film sections, progressive 29.97 sections, and interlaced 29.97 sections.

That's... literally not possible. Do you mean that it's a mix of Film content flagged for pulldown, 30fps interlaced content, and 60fps interlaced content?

[poisondeathray]

That's... literally not possible. Do you mean that it's a mix of Film content flagged for pulldown, 30fps interlaced content, and 60fps interlaced content?

It's possible with hard telecined film sections 3:2 (ie. not flags with soft telecine)

ie. 23.976p content, 29.97p content, 59.94 fields/sec content . All in a 59.94 fields/sec interlaced stream

[hello_hello]

I'd not have said it's possible. I'd have gone with "probable". ;)

I think Katie's still determined to confuse pull-down with interlacing and interlacing with some imaginary 60fps interlaced format.
60fps interlaced content? On a DVD? Anyone seen a 60fps DVD? Anyone?

[Katie Boundary]

It's possible with hard telecined film sections 3:2 (ie. not flags with soft telecine)

ie. 23.976p content, 29.97p content, 59.94 fields/sec content . All in a 59.94 fields/sec interlaced stream

Well... the content may have originally been 29.97p, but it gets interlaced upon conversion to NTSC, so the "progressive 29.97" part is still impossible when talking about DVDs.

[hello_hello]

Well... the content may have originally been 29.97p, but it gets interlaced upon conversion to NTSC, so the "progressive 29.97" part is still impossible when talking about DVDs.

DVDs are 29.970fps. The video can be 29.970fps interlaced, or it can be 29.970fps progressive. It can be 23.976fps with pulldown flags. It can be 29.970fps with hard-coded pulldown. It can be lots of things, including 29.970fps progressive.

In theory interlaced video and progressive video can be basically the same, only an interlaced frame consists of two fields that are different moments in time, while a progressive frame can consist of two fields that are the same moment in time. Both are 29.970 frames per second.

How would you "interlace" progressive video? You can encode it as interlaced, but in the end, the fields are still the same moment in time and it's still progressive.

Ever heard of progressive-scan DVD players?

[poisondeathray]

Well... the content may have originally been 29.97p, but it gets interlaced upon conversion to NTSC, so the "progressive 29.97" part is still impossible when talking about DVDs.

You should be more clear when you say "it gets interlaced"

You need to distinguish "content", with "type of encoding"

29.97p content can be contained in a 59.94 fields/sec stream. Its essentially 2:2 pulldown. Each field pair represents the same moment in time. In this case, hard pulldown instead of soft pulldown with flags. Progressive content encoded as interlaced 59.94 field/sec stream

[Katie Boundary]

Whatever... the point is that there never has been and never will be a single DVD in the world with 29.97 progressively encoded frames per second, which is why the OP's choice of words was extremely strange, wrong, and confusing.

I mean, technically you could create this kind of content, but it would require shifting all the top fields or all the bottom fields ahead or behind by one frame prior to encoding... and it would require the sort of insane mindset that does these sorts of things just for lulz.

[wonkey_monkey]

That's... literally not possible. Do you mean that it's a mix of Film content flagged for pulldown, 30fps interlaced content, and 60fps interlaced content?

No, that's literally not possible. You can't have a mix of film content flagged for pulldown and 30/60fps material in the same video.

You can have 30fps progressive content (yes, encoded as interlaced), 24fps hard telecined to 30fps interlaced, and 60fps interlaced.

which is why the OP's choice of words was extremely strange, wrong, and confusing.

To you, maybe, but everyone else seems to understand perfectly well.

Whatever... the point is that there never has been and never will be a single DVD in the world with 29.97 progressively encoded frames per second

Progressively encoded, or progressive content? I'm pretty sure everyone here is well aware that we're talking about progressive content. Please stop nitpicking, because you're not adding anything to the discussion and you're just sowing needless confusion.

(Progressive encoding at 30p is in the DVD spec, by the way)

As for 30fps progressive content never appearing on DVD, I give you:

https://www.amazon.com/H-G-Wells-Worlds-Anthony-Piana/dp/B0009PW4D2

As someone who was obsessed for about five minutes with listing every problematic DVD in the world it seems naive of you to assume that 29.97p progressive content has never been put to disc. Assuming that is what you're doing, rather than pointless nitpicking over content and encoding.

I mean, technically you could create this kind of content, but it would require shifting all the top fields or all the bottom fields ahead or behind by one frame prior to encoding...

...what?

[real.finder]

so far, this still an issue, but my idea to solve it in tivtc was https://github.com/pinterf/TIVTC/issues/26 and in yatta https://github.com/myrsloik/yatta-ivtc/issues/8

and I think the last hope https://github.com/Asd-g/AviSynthPlus-Scripts/issues/15#issuecomment-4366531429