Matroska VFR Proof of Concept.

[gizmotech]

Good Day everyone.

I will appologize in advance for any typo's, bad grammar, and spelling mistakes as I type this.

My name is GizmoTech, an encoder and lurker on doom9. I recently came accross the Excel Saga source to encode and identified that the episodes are riddled with 24fps and 30fps scenes. Until this week I was at a complete loss as to how to encode this files efficiently.

Thanks to recent efforts by the MKV Team, I have now massively simplified the act of creating VFR MKV files through manual FPS identification and timecode modification at time of muxing. I will now demonstate how this file was created to show proof that VFR encoding can be accomplished relatively painlessly through current VFW technologies and a little hard work.

NOTE: All information provided here is provided AS IS. The author is not responsible for it being unable to do what you want it to, and is not about to provide technical support or further documentation on how this was done.

Tools used:
VirtualDubMod (1.5.4.1 release)
AVS 2.5 October 11th CVS
MKVMerge 0.7.2 Build 20031027-1
Notepad <-- Never leave home without it.

Process (in brief):
Identify 24/30fps sections by frame numbers.
Creates AVS 30fps speedup script with ranged decimation
Encode 30FPS script (or huffy) into destination file size
Create VFR timecode file based on information allready acquired
Load encoded file and VFR Timecode file into mkvmerge

End Result: VFR MKV file

Next Post: The Actual Process

[gizmotech]

The Process:

Identification of 24/30fps sections:
The easiest way to identify 30fps content is actually thanks to Decomb.dll's telecide function. 30fps content, in show=true mode, is 99% of the time matched ccccc, and as such allows you to easily identify 30fp sections.

Watching a quickly encoded file with the following AVS will allow you to record frame numbers on 30fps content:

function FOrder(clip c) { return GetParity(c)? 1:0 }
LoadPlugin("mpeg2dec3dg.dll")
LoadPlugin("decomb.dll")

mpeg2source("wherever.d2v")

Telecide(order=FOrder(),guide=0,show=true,post=0)

Once these times are recorded a little bit of math is involved to make this process clean.

What needs to be caculated is mod/5 on the portions that are not 30fps. Because decimate can only have accurately predicted results on mod/5, ie total frame count can be predicted), a little bit of number manipulation comes into play.

The user must identify ranges for decimation which can be divisible by mod/5.

Example:
#Pre decimation Frames
0,768,29.970
769,843,23.976 <-- mod/5 capable.
844,921,29.970

The total frame count for the 30fps sections is largely irrelevent, but doing accurate mod/5 calculations are INCREDIBLY important for later on.

Create AVS w/ 30fps speedup:

What happens now is through manipulations of an AVS script, largely inspired by bilu in the VFR/DShow thread, allows the user to create a 30fps speedup file, which can be later time code adjusted back to proper fps's.

function FOrder(clip c) { return GetParity(c)? 1:0 }
Loadplugin("mpeg2dec3dg.dll")
Loadplugin("decomb.dll")

mpeg2source("wherever.d2v")

Telecide(order=FOrder(),guide=0,post=0)

p1=trim(0,768)
p2=trim(769,843).Decimate(cycle=5,quality=3)
     .AssumeFPS(last.framerate)
p3=trim(844,0)
assembly=p1+p2+p3
return assembly

This AVS essentially creates small sections of 30fps and 24fps content, increases the speed on the 24fps areas upto 30fps, and joins them together and returns something useable to the VFW compatible encoding application. In this case, a script similar to this was loaded into VDubMod for encoding into XVID.

Encode 30fps script/huffy into destination file size:
Pretty self explinatory. If you can't figure this one out, the rest of this must be WAY over your head.

Create VFR Timecode File:

Using the information from above we now create the modified time file.
Using the following syntax, and performing mod/5 calculations on 23.976 fps sections we can now feed a timecode file to mkv merge which will allow it to modify our encoded file back to correct fps.

Reference format:

# timecode format v1
Assume 29.970
# Entries Start Here
# 29.970 entries do not need to be entered. 
# If frames are not mention they are assumed 
# to be the fps above
769,829,23.976
# (843+1-769)*.8 = 60 frames

Though yes this is a VERY simple file, more complex ones can be created, using ANY fps the user wants, given the current format of Start_Frame,End_Frame,FPS(Float).

Load Encoded file and VFR Timecode file into mkvmerge:

Using a line like this, one can load the source file, the timcode file, and output the VFR Modified file.

mkvmerge -o "output.mkv" -d 0 -A --timecode "0:vfrtimecode.txt" 
  -S  "xvid1pass2.avi"

This will load the vfrtimecode.txt file, parse it, and modify the timecode entries on xvid1pass2.avi and output the TimeCode correct VFR encoded output.mkv file.

Proof of Concept:

Using the methods listed above I have created a 4 minute, 320x,240 XviD, VFR MKV file. The entries listed above were completly fictional, and not the real numbers used on this encode.

The proof of concept can be aquired either through bittorrent or e2dk. Please note that this requires the latest 1.0.1.8 mkv splitter from the Guliverkli project http://sourceforge.net/projects/guliverkli/ for playback.

Please also note if you use mpc to disable the built in MKV splitter.

http://www.anime-legion.net/bt/a-l.e...id.mkv.torrent

ed2k://|file|a-l.excel.vfr.test.xvid.mkv|22458776|EC68FC4B55DBD64E2535F1105C30D2EA|/

I would like all downloaders to note this VFR Xvid MKV is provided as a proof of concept for video manipulation techniques in modern containers and is not an attempt to perform any form of DVD backup distribution.

At any rate, I believe that is all that is required ATM. If you have any specific questions, feel free to ask them here and I will answer as best as I can.

GizmoTech,
#Anime-Legion

[Atamido]

Wow.

I knew that this was possible, and I saw the discussions about it, but I did not think that it would be implemented so quickly. I do think that there are better ways to get this result, but the result IS the same, so congradulations. This opens a whole new world of possibilities for using AVISynth with VFR clips.

[Zarxrax]

This sounds extremely impressive! Good job!

But er... the clip crashes when I try to play it back ^^;

[gizmotech]

a) added bit about MPC.

b) Bolded relevent areas.

Have a nice day.

[mf]

Am I stupid or should your framerate thingies either ALL say:
29.976
23.976
OR
29.970
23.970
?

My guess is having the film parts playing fractionally faster than the video parts could cause desync in the long run.

[gizmotech]

I'll vote for stupid and tell you to open up a 24fps stream and a 30 fps stream in vdubmod and examine the FPS vdub tells you about the clips. You'll notice it will identify the 30fps content as 29.970 and the 24fps content as 23.976.

But that's ok... I didn't actually expect you to research the subject matter before hand anyways.

Gizmo.

[mf]

Originally posted by gizmotech
I'll vote for stupid and tell you to open up a 24fps stream and a 30 fps stream in vdubmod and examine the FPS vdub tells you about the clips. You'll notice it will identify the 30fps content as 29.970 and the 24fps content as 23.976.

But that's ok... I didn't actually expect you to research the subject matter before hand anyways.

Gizmo.

Edit: Oh wait. Now that is one strange thing. And there I was thinking it was simply decimating one out of 5 frames. Telecining sure is creepy. Oh and btw, I'm excused cuz I live in PAL land .

[gizmotech]

Taken from one of my many d2v files kicking around on my system.

Field_Operation=0 (0:None 1:ForcedFILM 2:SwapOrder)
Frame_Rate=29970
Location=0,0,1,6ABC

Gizmo

[gizmotech]

Process #2:
Perform ITVC through YATTA
Mark 30fps sections in YATTA
Save AVS and timecode file from YATTA
Encode AVS into Speedup 30fps file
Mux encoded sped up avi into mkvmerge w/ YATTA generated timecode to receive time code correct mkv.

Now as Pamel mentioned earlier, my previous process remained largely manual operations and alot of unnescessary calculations. To do a project this way would require an intensive effort on the encodes part to perform calculations to achieve a proper vfr encode w/o audio/sub desync.

So for this we now have the YATTA application.

YATTA (Yet Another Telecide Tool for Anime), is a program developed my an aquantance of mine for performing intensive override files which can be fed to Decombs Telecide function to correct many common comb errors, and a number of other interesting features as well. The tool also has a number of range functions, one of which is a no decimation feature.

Using the no decimation feature, combined with its AVS generating properties (only basic generation for the task at hand), it allows a user to playback the contents of an encode, identify the 30fps sections manually, and save mass amounts of time on calculations. Also by using this tool eliminates a large amount of user error by determining in advance the final frame count of decimated sections (by ensuring that all decimated areas are mod/5) and creating the timecode file for use in mkvmerge.

So the new process being:
Perform IVTC through YATTA:

Yatta essentially operates in 2 steps. The first is a gathering run where it runs a telecide on the source based on configuration information you specify, and then provides the user with an interface to adjust the results that telecide provided yatta, and allowing the other useful features such as FreezeFrame assistance, ApplyRange assistance, Various advanced patern matching assistance tools, and the recent ranged NoDecimation tool.

Mark 30fps Sections in YATTA:

Using the start range key (e) set the beginning of your range at the start of the identified 30fps section. **To understand how to ID 30fps content see earlier posts**. Navigate through the scene until the conclusion and hit the NoDecimation (o) key. This will mark the range (+1-4 frames) as 30fps. This erorr on marking is to compensate for the decimated areas which surround it.


Save the AVS from YATTA:

Using the FreezeFrame/ApplyRange window, hit the Save VFR to save the timecode file used by mkvmerge, and hit the save button to generate an AVS script.

An AVS Script similar to this one massively simplifies mass encoding:

loadplugin("mpeg2dec3dg.dll")
loadplugin("decomb510.dll")

mpeg2source("episode2.d2v")

telecide(order=1,guide=0,ovr="episode2.tel.txt")

Import("../filtersettings.avs")

Import("episode2.freezeframe.avs")

When using the save all option within Yatta you will receive a file called Freezeframe.avs. This file also contains the VFR information if you specify it in YATTA.

an example of what the episode2.freezeframe.avs file would contain:
--- Code
FreezeFrame(4829,4829,4830)
DClip = Decimate(cycle=5,quality=3,ovr="").assumefps(last.framerate)
trim(0,3459)+DClip.trim(2768,3155)+trim(3945,4089)+DClip.trim(3272,3611)+trim(4515,4559)+DClip.trim(3648,3859)+trim(4825,4874)+DClip.trim(3900,4231)+trim(5290,5374)+DClip.trim(4300,4355)+trim(5445,5524)+DClip.trim(4420,4475)+trim(5595,5919)+DClip.trim(4736,4975)+trim(6220,6254)+DClip.trim(5004,5035)+trim(6295,6344)+DClip.trim(5076,0)
--- /Code

Encode File:
See above

Muxing into VFR MKV:
See Previous Process.

Now as you can see this greatly reduces the total time required for VFR file creation, and is obviously aimed at Anime VFR encoding, however with a little bit of creative thinking on the users part they can use this tool for ANY VFR encoding project.

I will be making a more indetail posting about the YATTA application in the Avisynth Usage forum tomorrow afternoon, where you will be able to download the application and try it for yourself. A reference link will be included here.

Just to put things into perspective for time reduction. I managed to produce the current VFR proof of concept clip in approximately 10 minutes or preparation work with YATTA, and this included fixing the little IVTC errors that riddled the source.

Feel free to post questions and I'll answer as many of them as I can. (And this time I'll even do it nicely too :P)

Gizmo

EDIT:
YATTA details can be found here:
http://forum.doom9.org/showthread.php?s=&threadid=64440

[Kurosu]

I also thought of something quite alike:
- have an app check for duplicated frames (my guess is to use only one field to avoid interpolation artifacts)
- the log produced by that app can be used by an avisynth filter to discard all frames.
- have a tool (possibly the very first application to convert from the log to mkvmerge's droplist - or the output could simply be a mkvmerge's droplist, but that's more troublesome to parse)

One could do all inside only one avisynth filter, but you'd have to run many passes so as to:
- determine the exact frame number on output
- have the proper file to encode in an VfW application (so as to properly set credits encoding settings)

There are several variants (like the droplist produced by the avisynth filter, a window in the filter's debug mode to undrop frames...) that are worth considering if the previous solution isn't satisfying.

[Suiryc]

Originally posted by Zarxrax
This sounds extremely impressive! Good job!

But er... the clip crashes when I try to play it back ^^;

You should update your Matroska filter. This clip was made using a new feature available in the recent libmatroska and that can reduce overhead in Matroska even more.
Unlike other modifications (that shouldn't be visible and shouldn't trouble old filters / programs) this one will likely crash filters and programs using older libmatroska versions (but the space gain that can be achieved in many cases really worth it).

[gizmotech]

Originally posted by Kurosu
I also thought of something quite alike:
- have an app check for duplicated frames (my guess is to use only one field to avoid interpolation artifacts)
- the log produced by that app can be used by an avisynth filter to discard all frames.
- have a tool (possibly the very first application to convert from the log to mkvmerge's droplist - or the output could simply be a mkvmerge's droplist, but that's more troublesome to parse)

One could do all inside only one avisynth filter, but you'd have to run many passes so as to:
- determine the exact frame number on output
- have the proper file to encode in an VfW application (so as to properly set credits encoding settings)

There are several variants (like the droplist produced by the avisynth filter, a window in the filter's debug mode to undrop frames...) that are worth considering if the previous solution isn't satisfying.

These are some interesting ideas, and could quite possibly produce better resuls, however duplicate frame detection, even if done on one field, will still generate bad results on slow moving areas unless there is some very acurate way to determine duplication. The only problem being a slow moving pan (ie very slow) will generate the same lvl of duplication, assuming a threshold like sytem is used, as poor source. IE: Using it on a source like Lain for detection of duplication would also result in some of the slow pans developing jerkiness (ie jumps).

Now if a user could correctly read back the actual frame rates from the mpeg2 source then we could get somewhere. As I was recently enlightened mpeg 2 PS content is natively VFR if read correctly as progressive frames (which none of our current tools do), and if we could somehow produce a VFR MPEG2 stream into an application, record the time stamps on it, then in-theory we could generate frame rate correct mpeg2 duplicates into another medium (ie mpeg4 vfr)

I'm going to be doing some research into this and attempt to verify this information, however if I can verify it it might be of interest to look at better parsing of the mpeg2 file as opposed to other such implementations. The main benefit of this would be in IVTC, or specifically if we could grab the VFR mpeg2 PS stream w/o parsing whatever interlace technology is used in the mpeg2 control stream to create the IVTC'd content then in theory we wouldn't need to perform IVTC again unless it was TC'd directly into the source content.

But like I said, more research is nescessary, and I will have to read the ISO specs on the mpeg2 streams. 13818-2 and 13818-2.

Gizmo.

[Kurosu]

Well, what you describe is coming from the hope encoders (as in: the software or the hardware doing the encoding) bother to properly set flags. But if the material is reencoded (R1 dvds from whatever source, or even original DVDs but from broadcast material), [edit]those flags may not be maintained[/edit]. What I describe is a general mean to actually detect duplicate frames (as for dupe by Neuron2, thresholds are required - and the filter dropping the frames should be put last in the filter chain).

The two are indeed usefull, but they serve different purpose. You want to achieve what the repeat flags in MPEG-2 do, I would like to achieve something like the dropedude filter used in RealProducer.

As for the technical implementation of the difference detection, I shouldn't have described it: one can set modes and behaviours, and design whatever mechanisms fitting one's needs. But that shouldn't have been part of my goal.

[bilu]

@Gizmo

Haven't had the time to download it yet, but it won't stop me from saying it: Great Job

Just wanted to remember this post from Bidoche, it's particularly important for the YATTA filter.
It again validates your MOD5 rule and kills any Dropdupe-like VFR filtering in Avisynth.

http://forum.doom9.org/showthread.ph...164#post390164

Clips publish the frames count they intend to output, they can't decide to drop a frame in the middle. That frame is expected.
What they can do is mark it to be dropped later (the duration=0 thing), not drop it.

EDIT: "Dropdupe" filters could be done outside Avisynth, between the AVS script and the encoder. They would drop frames and change timecodes in a file before providing the stream to the encoder.

Best regards,
Bilu

[bilu]

Originally posted by gizmotech
Now if a user could correctly read back the actual frame rates from the mpeg2 source then we could get somewhere. As I was recently enlightened mpeg 2 PS content is natively VFR if read correctly as progressive frames (which none of our current tools do), and if we could somehow produce a VFR MPEG2 stream into an application, record the time stamps on it, then in-theory we could generate frame rate correct mpeg2 duplicates into another medium (ie mpeg4 vfr)

@Gizmo

It can't help on hard-telecined MPEG-2 streams, these have telecine parts with no RFF flags.

Bilu

[Atamido]

Originally posted by gizmotech
These are some interesting ideas, and could quite possibly produce better resuls, however duplicate frame detection, even if done on one field, will still generate bad results on slow moving areas unless there is some very acurate way to determine duplication. The only problem being a slow moving pan (ie very slow) will generate the same lvl of duplication, assuming a threshold like sytem is used, as poor source. IE: Using it on a source like Lain for detection of duplication would also result in some of the slow pans developing jerkiness (ie jumps).

This has been discussed before in several theads. For still scenes in Anime, it is common for the frame to be the same across greater than 3 frames. So, you could use a drop dupe filter that required a minimal amout of change spanning 3+ frames to reduce the chances of a duped detected for a slow pan/zoom.

Now if a user could correctly read back the actual frame rates from the mpeg2 source then we could get somewhere. As I was recently enlightened mpeg 2 PS content is natively VFR if read correctly as progressive frames (which none of our current tools do), and if we could somehow produce a VFR MPEG2 stream into an application, record the time stamps on it, then in-theory we could generate frame rate correct mpeg2 duplicates into another medium (ie mpeg4 vfr)

You can read some of the discussion we had along these lines on the Matroska.devel mailing list. Advance through the conversation by clicking the next in thread link at the top.

Part of the idea is to store the MPEG-2 as VFR after removing the pulldown flags. However spyder is violently opposed to this as a lot of MPEG-2 filters would likely b0rk, and its not technically legal according to the MPEG-2 specs. I intend to try it personally to see how it does once we the MPEG-2 parsing filters are finished. If it works, then you would just need a DirectShowSource() filter in AVISynth that can write the correct timecodes to the VFR Timecode file.

But like I said, more research is nescessary, and I will have to read the ISO specs on the mpeg2 streams. 13818-2 and 13818-2.

Feel free to respond to the ML listed above to get more information.

[Atamido]

Originally posted by Pamel
[BThis has been discussed before in several theads. For still scenes in Anime, it is common for the frame to be the same across greater than 3 frames. So, you could use a drop dupe filter that required a minimal amout of change spanning 3+ frames to reduce the chances of a duped detected for a slow pan/zoom.

After talking with bilu I realize I misunderstood the intent. Normally an MPEG-2 decoder will turn this series of progressive frames:
[A] [B] [C] [D]
Into this series of interlaced frames:
[Aa] [Bb] [Cc] [CD] [dD]
if the RFF flag is set.

A hard telecine would be if the encoding software actually stored it like [Aa] [Bb] [Cc] [CD] [dD] instead of storing progressive frames and setting the RFF flag. I think that the best way to detect if there is a hard telecine would be to compare fields. Create a filter that did something like this:

If
(
Even field of Frame 4 is >= in similarity to Even field of Frame 3 than Even field of Frame 5
AND
Odd field of Frame 4 is >= in similarity to Odd field of Frame 5 than Even field of Frame 3
)
Then Sequence is hard telecined.

[spyder]

@Pamel: This is what Telecide already does, only it's algorithm is a little better than yours :P

PS: About me being violently opposed to the VFR MPEG2 thing, it is true :P Why should we b0rk the stream when a decoder can simply decode it VFR anyway. Many of the good decoders do this. See Gabest's MPEG2 decoder in MPC...

[gizmotech]

Edited out a typo or two.

Linked yatta thread in avisynth usage.

Have a nice Day

Gizmo.