Help needed with VFR Framerate Conversion?

[aegisofrime]

Hi all. I have been stumped at trying to re-encode a certain MP4 file.

Here's the MediaInfo for it:

Video
ID                          : 1
Format                      : AVC
Format/Info                 : Advanced Video Codec
Format profile              : Main@L4
Format settings, CABAC      : Yes
Format settings, ReFrames   : 1 frame
Format settings, GOP        : M=1, N=60
Codec ID                    : avc1
Codec ID/Info               : Advanced Video Coding
Duration                    : 2h 10mn
Bit rate                    : 5 779 Kbps
Width                       : 1 920 pixels
Height                      : 1 080 pixels
Display aspect ratio        : 16:9
Frame rate mode             : Variable
Frame rate                  : 29.102 fps
Original frame rate         : 29.970 fps
Minimum frame rate          : 9.990 fps
Maximum frame rate          : 30.000 fps
Color space                 : YUV
Chroma subsampling          : 4:2:0
Bit depth                   : 8 bits
Scan type                   : Progressive
Bits/(Pixel*Frame)          : 0.096
Stream size                 : 5.25 GiB (97%)
Language                    : English
Encoded date                : UTC 2015-08-19 01:48:20
Tagged date                 : UTC 2015-08-19 01:48:20

I would like to apply Interframe (a frame double) and encode it in 59.94FPS. If I do this, regardless of the source filter the audio is always out of sync with the video.

So I Googled and the Avisynth Wiki suggested using mp4fpsmod to extract the timecode and mux it with the resulting file.

So, I went ahead and used MKVMerge to mux the MP4, along with the timecode, and the resultant mkv file plays in sync.

However, if I were to do my 60fps encoding, MKVMerge throws up an error when I try to mux the timecode file with the resulting 60fps mkv, complaining that the amount of frames in the timecode is lesser than the file.

What would be the proper way to handle this? I hope that I'm making sense here, I know it's a complex issue. Thanks in advance

tl;dr Trying to convert VFR file to 60fps-ish. Result has video/audio sync problems.

Edit: I have tried using DirectShowSource in conjunction with convertfps=true and this does not work as well.

[Overdrive80]

Invoke video with:

LSMASHVideoSource("xxxx.mp4").assumefps(30000,1001.0)
#Now interframe

[aegisofrime]

Originally Posted by Overdrive80

Invoke video with:

LSMASHVideoSource("xxxx.mp4").assumefps(30000,1001.0)
#Now interframe

Thanks Overdrive. I have given your suggestion a go and unfortunately the result is still out of sync. I noticed that the length for the audio track as read by L-Smash is 02:09:58; the video length is 02:06:14. Without the AssumeFPS part the length is 02:10:00.

Is there anything else I can try?

[sneaker_ger]

a=LSMASHAudioSource("xxxx.mp4", av_sync=true)
v=LSMASHVideoSource("xxxx.mp4", fpsnum=30000, fpsden=1001)
AudioDub(v, a)

If it is still out of sync add DelayAudio() and try different values until it's in sync. Then after that add all the interframe stuff you want to use.

[Motenai Yoda]

You have to open it with

DirectShowSource("F:\Hybrid\vfr.mkv", fps=29.97, convertfps=true)
#interframe

(in case dds2("F:\Hybrid\vfr.mkv", fps=29.97) will work too)

or extract the timecode from mp4 and use some tool to convert the vfr script or avi to a cfr one like tc2cfr

otherwise use normal lsmash/ffmpeg/dss2 source filter, interframe it and use the extracted timecode when encoding or muxing, but I don't know if there is a tool to easely edit timecodecs for a different framerate.

[aegisofrime]

Originally Posted by sneaker_ger

a=LSMASHAudioSource("xxxx.mp4", av_sync=true)
v=LSMASHVideoSource("xxxx.mp4", fpsnum=30000, fpsden=1001)
AudioDub(v, a)

If it is still out of sync add DelayAudio() and try different values until it's in sync. Then after that add all the interframe stuff you want to use.

Interesting about the av_sync flag, I was scouring the L-Smash docs but missed it. Will give this a go!

Originally Posted by Motenai Yoda

You have to open it with

DirectShowSource("F:\Hybrid\vfr.mkv", fps=29.97, convertfps=true)
#interframe

(in case dds2("F:\Hybrid\vfr.mkv", fps=29.97) will work too)

or extract the timecode from mp4 and use some tool to convert the vfr script or avi to a cfr one like tc2cfr

otherwise use normal lsmash/ffmpeg/dss2 source filter, interframe it and use the extracted timecode when encoding or muxing, but I don't know if there is a tool to easely edit timecodecs for a different framerate.

Yeah, I have the timecode extracted by mp4fpsmod which if I muxed back would actually enable the resultant MKV to be played in sync (If I just muxed the mp4 to mkv it would be out of sync). However I too have no idea how to change the timecode for a different framerate.

Thanks guys will give your suggestions a go as soon as my current encode finishes.

[sneaker_ger]

Demuxing vfr timecodes and remuxing them later is trivial if you don't touch frame count/timings (sharpening, resizing, denoising etc.), doing things like interframe is a completely different beast.

[Music Fan]

Another possibility that may work : open your video in Virtual Dub with the ffmpeg plugin, convert in lossless (Lagarith for example), open this video (that should have a 29.97 CFR and be played in sync) with Avisynth and use Interframe.
Virtual Dub with the ffmpeg plugin was the only method that worked with a VFR h264 video I had to convert in CFR.
https://fr.osdn.jp/projects/sfnet_vi...eginputplugin/
In Virtual Dub, go in File, open video file, files of type, choose FFMpeg supported files, select your video and open it.

The problem being that a 2h10mn 1080p video in Lagarith will be huge.

[colours]

Motenai Yoda's post is, thus far, the only reply in this thread that actually makes sense. If you want to stick to AviSynth processing, you should convert to CFR first (because there's no concept of VFR within AviSynth) and then do motion interpolation. (Note: "convert", not just "assume as CFR".) Using one of those duplicate frame removing thingamajigs will probably improve results further.

This is not a perfect solution, but it's certainly an easy one. If you want the frame timings to be handled 100% properly, one way to do this is to write a script to parse the extracted timecodes, calculate the interpolation position for every frame, then dump this out in the form of an AviSynth script.

Alternatively, you could run InterFrame as usual, and modify the timecodes so that you insert a new timecode between every two old timecodes by averaging them, then mux the result with these new timecodes. This is probably a bit easier than the above suggestion.

[Music Fan]

Originally Posted by colours

Motenai Yoda's post is, thus far, the only reply in this thread that actually makes sense

To you.
Mine makes sense too and works (maybe not with all videos but no solution works every time), the problem being the size of the intermediate file. But it's easy to do, no need for timecode extraction and remux.

[foxyshadis]

Every suggestion I've seen so far will lead to broken interpolation, by duplicating or dropping frames and confusing the hell out of Interframe. How about this: Use DSS to import a Graphedit .grf instead, one in which you start with your mp4 source and connect SVP. Let the VFR be taken care of in a VFR domain instead of importing it into avisynth.

If Interframe is ever ported to VapourSynth, it would just work, since VS natively supports VFR. But it's not available yet.

[AzraelNewtype]

Originally Posted by foxyshadis

Every suggestion I've seen so far will lead to broken interpolation, by duplicating or dropping frames and confusing the hell out of Interframe. How about this: Use DSS to import a Graphedit .grf instead, one in which you start with your mp4 source and connect SVP. Let the VFR be taken care of in a VFR domain instead of importing it into avisynth.

If Interframe is ever ported to VapourSynth, it would just work, since VS natively supports VFR. But it's not available yet.

colours' final suggestion of running interframe as normal and then just making a new timecodes file for it would actually work just fine. If it's mainly one framerate with a couple oddball segments, a v2->v1 converter and just doubling the rates would be pretty easy. If it's messier you can just read every timecode line, divide it by two, and insert that twice into a new file. Maybe alternate rounding up and down for safety. It's easy to script, though lol please don't do it by hand. Vapoursynth would surely make it trivial, but it's not necessary.

[Music Fan]

Originally Posted by foxyshadis

Every suggestion I've seen so far will lead to broken interpolation, by duplicating or dropping frames and confusing the hell out of Interframe.

The principle of VFR is that some frames are duplicated when the video is played. Thus anyway, when it's converted to CFR correctly (which means it stays in sync), the video will contain duplicated frames, but the result is supposed to be the same than if the video had been encoded directly in CFR mode. The only difference is that the duplicated frames are hard coded and not created in real time by the player (same principle than soft pulldown and hard pulldown).
I don't understand why Interframe would be more confused if there is a VFR step before, it's not supposed to know it if CFR is created before the Interframe step (whatever the VFR to CFR conversion is made with Avisynth in the same script than the Interframe process or with Virtual Dub and an intermediate avi).

[vivan]

Originally Posted by Music Fan

The principle of VFR is that some frames are duplicated when the video is played.

Imagine playing back video that has 24 and 30 fps parts on 120 hz display. Normally it will not have stutter, but if you convert it to 30 fps first then it will.
Should you convert it 120 instead? Maybe even using so-called "null frames"? Oh, wait, we've been there before.

Originally Posted by Music Fan

I don't understand why Interframe would be more confused if there is a VFR step before, it's not supposed to know it if CFR is created before the Interframe step (whatever the VFR to CFR conversion is made with Avisynth in the same script than the Interframe process or with Virtual Dub and an intermediate avi).

It won't be confused. It just won't interpolate.
Avisynth can't handle VFR, and thus can't Interframe. Whatever method you use to convert it to CFR will have same result - it just won't interpolate between dublicated frames at best (and with blended frames it will be even more sad).
Imagine having 10 fps video that was converted to 30 and then interpolated to 60.

Originally Posted by foxyshadis

How about this: Use DSS to import a Graphedit .grf instead, one in which you start with your mp4 source and connect SVP.

I don't know if it can handle it now, but 3 years ago it handled VFR by looking at fps ffdshow reported. It took 10+ seconds just to switch from one constant framerate to another... Also it will require processing everything in realtime, or am I misunderstanding something?

If video is switching between different constant framerates it should be possible to splice those parts and use Interframe on each part separately... With v1 timecodes it shouldn't be hard.

[Music Fan]

Originally Posted by vivan

Imagine playing back video that has 24 and 30 fps parts on 120 hz display. Normally it will not have stutter, but if you convert it to 30 fps first then it will.

Ok, but you are talking about a special case where two framerates are mixed together, but most of VFR videos have only one original framerate and the VFR encoding is only used to discard the duplicated frames and lower the bitrate.

Originally Posted by vivan

It won't be confused. It just won't interpolate.
Imagine having 10 fps video that was converted to 30 and then interpolated to 60.

I understand. In this case, if it doesn't work with hard coded duplicated frames, it won't work better with a VFR video.

[Motenai Yoda]

You are all rights, (even if I don't know how interframe can handle vfr streams), but at 90% the source of the mp4 was a 30p one.

[aegisofrime]

Wow, thanks for the very enlightening discussion so far guys! There's so much that I don't know. The point about using Vapoursynth is interesting. Wasn't there a way to use Avisynth plugins in Vapoursynth?

Unfortunately I haven't gotten a chance to try your suggestions yet; QTGMC'ing a Blu-Ray is no joke!

If it's mainly one framerate with a couple oddball segments, a v2->v1 converter and just doubling the rates would be pretty easy. If it's messier you can just read every timecode line, divide it by two, and insert that twice into a new file. Maybe alternate rounding up and down for safety

This actually sounds interesting because I know that the timecode I extracted works, if I give MKVMerge the MP4 and timecode file the result plays in sync. To clarify, is this what you are saying?

# Example for 10 consecutive frames at 25fps:
0
40.0
80.0
120
160
200
240
280
320
360

# Example for 10 consecutive frames at 50fps:
0
20
40.0
60.0
80.0
100.0
120
140
160
180
200
220
240
260
280
300
320
340
360

If so I can probably whip up a quick program to do the job.

Once again, I appreciate everyone's input in this! It would probably help others who have to deal with VFR content as well!

[Music Fan]

Originally Posted by Motenai Yoda

I don't know how interframe can handle vfr streams

I also wonder how it does handle CFR videos with some short parts without any movement (as it happens sometimes in movies when the camera don't move and film fixed objects).

[colours]

Originally Posted by foxyshadis

If Interframe is ever ported to VapourSynth, it would just work, since VS natively supports VFR. But it's not available yet.

That's the exact reason I didn't suggest VapourSynth here. The problem of writing a shim to handle VFR correctly is as hard as writing a script to output a script with pre-calculated interpolation fractions (as in the time parameter of MFlow/MFlowInter).

[foxyshadis]

Originally Posted by Music Fan

I understand. In this case, if it doesn't work with hard coded duplicated frames, it won't work better with a VFR video.

The problem is that they're not hard coded, but flattening to a 30fps constant frame rate in DSS creates hard coded duplicates where they didn't exist before. That's why I said frame rate conversion should occur where VFR can run free. Interframe is an amazing script, but it just slapping timecodes on its output isn't sufficient when the input is already garbage.

If SVP really can't switch on real-time framerate -- apparently it's still tied too close to MVTools -- then I guess there are no good easy solutions. VS would require some heavy lifting, not just translating but changing the script. Like vivan says, for best quality you're stuck for now with chopping the video up with v1 timecodes and separately Interframing each.