VirtualDub: A/V sync with VFR FLV file

[KeyMs92]

Hi,

For a while now, I've been editing VFR flv files with VirtualDub. In short, this is the procedure:

1. Open the flv files in Ultra Video Splitter and split them up in rough pieces (I must open them in UVS anyway because Vdub crashes with the original container). The output files are VFR flv files (like the input).
2. Open the pieces in Vdub and and set the framerate for all of them to 25 (nominal framerate).
3. Encode the bits outside the keyframes, using the VP6 6.4.2 VFW codec. Join them with the appropriate non-encoded bits.
4. Synchronize the audio and apply some fade-outs using Audacity. The delay is usually around 150ms.

What I've been noticing is that audio seemed slightly off here and there. As I found out VirtualDub (and avi) doesn't support VFR and handles the frames as CFR, so that explains the problem. My question is, is there a way I could edit the files in VFR? Or would it be better to convert the files to CFR (using Avisynth)? It's also important that I can use the plugin, I'm not sure if it supports VFR...

EDIT: I'm also very confused about the timecodes. When I extract the v2 timecodes with FLV extract, all the timecodes reflect 25fps except the first few. Yet, FLV Extract as well as Vdub report 25.002 (with the first frames excluded) as the average framerate. When I convert the timecodes to v1, I get an assumed framerate of 29.970 and an average framerate of 24.975! Lastly, when I make the file smaller towards the ending, the reported framerate increases and the timecodes along with it. In other words, e.g. the last 100 timecodes are different for each different split.
Original file info:

Format : Flash Video
File size : 460 MiB
Duration : 2h 7mn
Overall bit rate : 504 Kbps
_audiochannels : 1.000
_audiocodecid : .mp3
_audiodatarate : 96.000
_audiodevice : Microsoft DV Camera and VCR
_audioinputvolume : 75.000
_audiosamplerate : 44100.000
_creationdate : Thu Oct 13 13:23:20 2011
_duration : 7654.205
_framerate : 25.000
_height : 288.000
_presetname : Custom
_videocodecid : VP62
_videodatarate : 400.000
_videodevice : Microsoft DV Camera and VCR
_videokeyframe_frequency : 5.000
_width : 512.000

Video
Format : VP6
Codec ID : 4
Duration : 2h 7mn
Bit rate : 379 Kbps
Width : 512 pixels
Height : 288 pixels
Display aspect ratio : 16:9
Frame rate mode : Variable
Bit depth : 8 bits
Stream size : 346 MiB (75%)

Audio
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 3
Codec ID : 2
Codec ID/Hint : MP3
Duration : 2h 7mn
Bit rate mode : Constant
Bit rate : 96.0 Kbps
Channel(s) : 1 channel
Sampling rate : 44.1 KHz
Compression mode : Lossy
Stream size : 87.6 MiB (19%)

[hello_hello]

I assume you don't necessarily want to split the files, it's just because of the crashing problem?

Until someone with more experience working with flv files and/or VFR video comes along, what if you open the entire video via directshow, then get AVIsynth to add/delete frames as necessary to give you a constant frame rate?

If you create a basic AVIsynth script (I guess I'm assuming you have AVIsynth installed) such as this (it's just a text file with an avs extension):

DirectShowSource("D:\video.flv")
ChangeFPS(25)

Then open the script with VirtualDub, it should open with the same duration as the original video but with Avisynth feeding VirtualDub uncompressed video at 25fps. If the majority of the original video is basically 25fps it shouldn't be jerky. From there you should be able to use VirtualDub to re-encode both the audio and video (or leave the audio uncompressed if you want to edit it later) and with any luck the output will be CFR 25fps video with the audio still in sync.

More info here: http://avisynth.org/mediawiki/AssumeFPS

[sneaker_ger]

DirectShowSource("D:\video.flv")
ChangeFPS(25)

That way is not viable for VFR -> CRF. Since AviSynth is always CFR, the VFR->CFR conversion has to be done by the source filter, i.e.:
DirectShowSource("D:\video.flv", fps=25, convertfps=true)
or
ffvideosource("D:\video.flv", fpsnum=25, fpsden=1)

But in this case (flv, constant 25 fps, except the first two (?) frames) I would assume a constant framerate anyways. I don't know where these two first timecodes in FLV files come from, but when catting several of them while ignoring the timecodes and instead assumung CFR, it never resulted in any desync for me, so I'd personally just ignore them.

I don't know any VFR editing app out of my head. If it's some hybrid source (23.976 and 29.976) you could split it and edit the parts separately and join again after editing. If it's some "crazy" VFR, converting to CFR would probably be a lot easier.

[hello_hello]

http://avisynth.org/mediawiki/AssumeFPS#ChangeFPS
"ChangeFPS changes the frame rate by deleting or duplicating frames."

I just opened a video which I know to have a variable frame rate. MediaInfo reports a variable frame rate and a frame rate of 29.857 fps. The duration is 11 min, 38 sec.
I created two basic directshow scripts and added ChangeFPS(25) to one and ChangeFPS(29.970) to the second. I opened them both with MPC-HC and they both still had the same duration as the original but 25 and 29.970 frame rates respectively. I even ran a quick encode of the 25fps version to make sure I wasn't going mad, but the encode came out at 25fps. What am I missing?

Your method also works. I tried it at 25fps and compared it with my script using MPC-HC and couldn't see any difference in the video. Even CPU usage was pretty much the same.

Two ways of doing the same thing maybe?

[sneaker_ger]

Two ways of doing the same thing maybe?

No. ChangeFPS() has no way of knowing the timecodes, it will be feed by the source filter with a constant framerate.

Imagine a hybrid source, which has 23.976 fps and 29.97 fps parts. If you open it with DirectShowSource (without the "convertfps=true" parameter) it will read all source frames, without dropping or adding anything and it will report a constant framerate to AviSynth, depending on the splitter. Let's say it assumes 23.976 fps. At this point you're already screwed, because all timecodes are lost and the duration is now different from the real duration of the VFR source (it has increased, because 23.976 is slower than 29.97). Now you add ChangeFPS(24000, 1001) to try to make a constant 23.976 fps - what will happen? Answer: Nothing will be changed by ChangeFPS(24000, 1001), because the whole clip already appears as 23.976 fps to the function, changefps() has no idea that some parts were originally 29.97 fps. The duration is still wrong.
If you type in ChangeFPS(30000, 1001) you will essentially do a pulldown of all parts of the file, even those that were originally 29.97 fps already. The duration is still wrong.

ChangeFPS() can only convert from one constant framerate to another constant framerate. It has no timecode input option and internally AviSynth doesn't pass any timecode information (except a single, constant rate).

[sneaker_ger]

I've made a sample that can be played around with:
http://www.mediafire.com/?653eou18b26n1ed

The first 1000 frames run at 23.976 fps, the rest of the file runs at 29.97 fps.

Compare the a/v sync of these scripts:
audio=FFAudioSource("vfr_sample.mkv")
video=FFVideoSource("vfr_sample.mkv")
AudioDub(video, audio)
ChangeFPS(24000, 1001)
audio=FFAudioSource("vfr_sample.mkv")
video=FFVideoSource("vfr_sample.mkv", fpsnum=24000, fpsden=1001)
AudioDub(video, audio)

Of course you can tinker with the constant framerate, e.g. if you use the average framerate the duration will be constant and it may be in sync here or there but you cannot achieve a/v sync over the course of the complete file with the ChangeFPS() function. It's a lost cause.

[KeyMs92]

Thanks all for you comments :)

I'm still in doubt whether I should convert to CFR. I forgot to mention that the flv file is a recording of a live stream. Basically I get to choose from two options:

1. Recording the stream with WM Recorder, which already converts the VFR stream to CFR. I'm pretty sure it really converts the framerate since the audio remains in sync right away, something which doesn't happen when it assumes CFR. I can now just use 25 fps.

2. Recording the stream with Jaksta, which preserves the VFR. Advantage is that I have VFR and can still decide to convert it to CFR. Until now I've just assumed CFR and choose the framerate which suites the video best.

Both options, give good results and it's really hard to decide which option is the best. Option 2 seems to have the edge for me, so I've chosen that one mostly.

[hello_hello]

ChangeFPS() can only convert from one constant framerate to another constant framerate. It has no timecode input option and internally AviSynth doesn't pass any timecode information (except a single, constant rate).

Well the sample I tested prior to my last post was definitely a VFR video. I actually opened it with MPC-HC, used ffdshow to decode, enabled it's on screen display and looked for frame rate changes.
I found a section where the frame rate constantly changed, wondering around between about 28 and 30 fps.
That was the section I cut and encoded using ChangeFPS(25). The resulting encode had a 25fps frame rate, the same duration as the original cut, and the frame rate was 25fps.

When I opened the script with MPC-HC, despite ChangeFPS(25), ffdshow's OSD still reported the original VBR while it was playing. Then again, it did the same thing when using your method.

Anyway, I'm downloading your sample as I type. I'll play around with it and report back.

[hello_hello]

Okay.... well you're correct when it comes to your sample. I'll have to think about it again later on when I've got time. Obviously there's a big difference in the way the frame rate conversion is handled, I just don't understand why in my example ChangeFPS seemed to work perfectly. I'll have to try it again later as well to see if I missed something. I know I've successfully used ChangeFPS on VFR video before....

But anyway, when I encoded your sample your way everything was fine. When I did it my way the final video had exactly the same number of frames, exactly the same duration, only the video was running faster and it was truncated (it finished before the end).

[sneaker_ger]

I found a section where the frame rate constantly changed, wondering around between about 28 and 30 fps.

So you cut out a part with a framerate oscillating between 28 fps and 30 fps? I'd say that can be simplified to a constant 29(.97?) fps without loosing sync. You basically created a CFR file, where using ChangeFPS() wouldn't pose any problem.

Ask yourself the following question: how would the ChangeFPS() function know which section is 23.976 fps (and shouldn't be touched) and which section is 29.97 fps (where frames would have to be dropped)? Answer: it has no way to get these informations.
Put "info()" between the source filter and ChangeFPS() to see what AviSynth thinks about the current framerate - maybe that helps understanding the problem.

I know I've successfully used ChangeFPS on VFR video before....

Did you only rely on MediaInfo to say "variable framerate" or did you analyze the timecodes? Sometimes VFR files can be very close to CFR files or MediaInfo's detection might be wrong - in either case there is little harm done treating them as CFR.

[sneaker_ger]

Thanks all for you comments :)

I'm still in doubt whether I should convert to CFR. I forgot to mention that the flv file is a recording of a live stream. Basically I get to choose from two options:

1. Recording the stream with WM Recorder, which already converts the VFR stream to CFR. I'm pretty sure it really converts the framerate since the audio remains in sync right away, something which doesn't happen when it assumes CFR. I can now just use 25 fps.

2. Recording the stream with Jaksta, which preserves the VFR. Advantage is that I have VFR and can still decide to convert it to CFR. Until now I've just assumed CFR and choose the framerate which suites the video best.

Both options, give good results and it's really hard to decide which option is the best. Option 2 seems to have the edge for me, so I've chosen that one mostly.

Encoding or remuxing while preserving VFR is not that hard, editing/cutting is. AviDemux nightlies (http://www.avidemux.org/nightly/win32/) are supposedly able to handle VFR, but I don't know if it will be able to cope with your files.

[KeyMs92]

Encoding or remuxing while preserving VFR is not that hard, editing/cutting is.
Indeed that seems the case. Interestingly, I noticed that Ultra Video Splitter seems to drop frames at some points. By default it drops them in the first few seconds and than it might drop some frames later on. Not that that is a bad thing though: without the framedrop syncing the audio would've been impossible when assuming CFR.

I was under the impression that UVS preserves VFR, mainly because whenever I split a different part of the video in UVS the avarage framerate differs for each fragment. So I'm a bit confused now.. :(