FFMpegSource2 VFR to CFR Out of sync

[ENunn]

I have some Shadowplay footage I'm trying to edit in AVSPmod. Because of how it records I have to do a color matrix conversion and a vfr to cfr conversion so it'll look better and such. My problem here is that a lot of my recordings are way out of sync when I import them with FFMPegSource2. This particular recording shows an extra 9 seconds added (although it doesn't show up in AVSPmod, only the final encoding) + audio sync issues.

Is there any way I can fix this?

FFMPegSource2("F:\Shadowplay\Forza Horizon 4\Forza Horizon 4 2019.09.21 - 15.34.13.11.mp4",atrack=-1, fpsnum=60000, fpsden=1001)
ColorMatrix(mode="Rec.601->Rec.709")
ChromaShiftSP(-.5,.2)

[FranceBB]

Out of curiosity, have you tried with

video=LWLibavVideoSource("F:\Shadowplay\Forza Horizon 4\Forza Horizon 4 2019.09.21 - 15.34.13.11.mp4")
audio=LWLibavAudioSource("F:\Shadowplay\Forza Horizon 4\Forza Horizon 4 2019.09.21 - 15.34.13.11.mp4")
AudioDub(video, audio)

to see whether it's in sync or not?
Can you check it through VirtualDub BEFORE encoding it?

[ENunn]

Originally Posted by FranceBB

Out of curiosity, have you tried with

video=LWLibavVideoSource("F:\Shadowplay\Forza Horizon 4\Forza Horizon 4 2019.09.21 - 15.34.13.11.mp4")
audio=LWLibavAudioSource("F:\Shadowplay\Forza Horizon 4\Forza Horizon 4 2019.09.21 - 15.34.13.11.mp4")
AudioDub(video, audio)

to see whether it's in sync or not?

That makes it in sync.

Originally Posted by FranceBB

Can you check it through VirtualDub BEFORE encoding it?

I've checked through VirtualDub and the external player option in AVSPmod and they're both out of sync (when I use FFMpegSource).

[FranceBB]

Well, alright then, use LSMASH instead of ffms2.
Unfortunately, sometimes, those things can happen with ffms2 even if you specify fpsnum and fpsden.
For instance, it actually happened to me on a few dodgy footages recorded from a phone.
Technically, it should automatically detect the frame rate changes, set the output framerate automatically to the average and output something for you that is in sync, but sometimes it doesn't, which is why we have different indexers.
Feel free to use LWLibav as indexer instead of FFMpegSource2 this time, you're gonna be fine.

Cheers,
Frank.

[ENunn]

Thanks for the suggestion on LSMASH, France.

It's working quite well, however I've found that on some of my footage the video is much more choppier on LSMASH than on FFMpeg, even with both having the fpsnum/den command at the end.

I can use FFMpegSource for that but what if I still have that audio sync problem? What should I do there?

[FranceBB]

Well it is choppy 'cause the only thing that Avisynth indexers do is to find the highest fps, the lowest fps, sum them and divide the value by 2 to get the average.
So, if your video is at 60fps for most parts and only for a couple of seconds it dropped to 20fps, the indexer will still output 40fps by dropping frames in the 60p part and duplicating frames in the 20p part.
This is far from being ideal. Ideally, you'd want your frameserver to output 60p with blended frames when it doesn't have enough of them to preserve the 60p portion and not make the 20p one look like garbage.
Unfortunately, since Avisynth only works with CFR, it can be hard to achieve this, but it can be done with ffmpeg.

ffmpeg -i "input.mp4" -vf tblend=all_mode=average -r 60 -c:v huffyuv -pix_fmt rgb24 -f avi  "output.avi"

then you can index the newly created lossless .avi file in Avisynth and do whatever you want since it should now be 60p CFR with blended frames.
You could actually give a shot to

-filter:v minterpolate

which should apply linear interpolation on the low fps parts, but it might create artifacts.
Oh and for the records, I didn't try those commands, I wrote them as I remembered them, let me know if they work.

[richardpl]

tblend without any parameters does not do anything much, except losing one final frame in output. You probably wanted to use -vf tblend=all_mode=average.
minterpolate filter is not optimized, but you can use fast mode that does not do motion interpolation.

[hello_hello]

Originally Posted by FranceBB

Well it is choppy 'cause the only thing that Avisynth indexers do is to find the highest fps, the lowest fps, sum them and divide the value by 2 to get the average.
So, if your video is at 60fps for most parts and only for a couple of seconds it dropped to 20fps, the indexer will still output 40fps by dropping frames in the 60p part and duplicating frames in the 20p part.
This is far from being ideal. Ideally, you'd want your frameserver to output 60p with blended frames when it doesn't have enough of them to preserve the 60p portion and not make the 20p one look like garbage.

I have to disagree a little there. The indexer does output the average frame rate for VFR sources, but it's the real average frame rate. If most of a source is 23.976 fps (for example) and a very small section of it is 29.97 fps, it'll be decoded at 24.8 fps, or whatever the average frame rate happens to be.

Maybe I'm misunderstanding what you wrote, but without frame rate conversion the indexer should decode without repeating or dropping any frames. If the average is 40 fps, the 60 fps sections would run at a lower frame rate and the 20 fps sections would run faster, making a mess of the A/V sync, but no frames shouldn't be duplicated or dropped.

For a variable frame rate source you'd probably want to convert to a constant frame rate that's the same or close to the maximum frame rate, if possible, so the indexer should only duplicate frames, although if the majority of a source is 29.97 fps with a couple of tiny peaks at 31.4 fps, for example, you're probably better off converting to 29.97fps instead.

Choppiness can sometimes be caused (I think) by "jitter" in the timecodes (for MKV they're rounded to the nearest millisecond).
If you have a constant frame rate source at 23.976 fps, and convert the frame rate to 23.976 fps (fpsnum=24000, fpsden=1001) in theory the indexer should only output the original frames, but sometimes it'll cause it to duplicate or drop frames when it shouldn't, making motion choppy. Remuxing the video and audio to another container and using it as the source might help, or it might not.

ENunn,
When you wrote that the following keeps the audio in sync, did you forget to include the frame rate conversion or is the A/V sync okay without it? It's just that if it is VFR, it's very unlikely the audio/video would stay synced without frame rate conversion.

video=LWLibavVideoSource("F:\Shadowplay\Forza Horizon 4\Forza Horizon 4 2019.09.21 - 15.34.13.11.mp4")
audio=LWLibavAudioSource("F:\Shadowplay\Forza Horizon 4\Forza Horizon 4 2019.09.21 - 15.34.13.11.mp4")
AudioDub(video, audio)

This is one of those situations where it'd probably help to upload a sample.

[FranceBB]

Originally Posted by hello_hello

Maybe I'm misunderstanding what you wrote, but without frame rate conversion the indexer should decode without repeating or dropping any frames. If the average is 40 fps, the 60 fps sections would run at a lower frame rate and the 20 fps sections would run faster, making a mess of the A/V sync, but no frames shouldn't be duplicated or dropped.

Well that would be bad.

Originally Posted by hello_hello

For a variable frame rate source you'd probably want to convert to a constant frame rate that's the same or close to the maximum frame rate, if possible, so the indexer should only duplicate frames

Well absolutely, my point was that in Avisynth when you duplicate, if you don't do anything else, it might be less smooth than applying other techniques like blending. For instance, I would personally get the highest frame rate, output that framerate but instead of duplicating, I would blend the parts with a lower fps. The fact of it being not "ideal" is that if I gotta duplicate first, detect the dups, remove the dups and replace them with blended frames it would be a mess. I would actually love ffms2 / LSMASH to be able to perform blending as an option when specifying fpsnum and den.

[hello_hello]

Originally Posted by FranceBB

Well that would be bad.

If you have a VFR source, it's easy enough to re-encode as VFR. FFMS2 can write a timecodes file as it indexes, and you can add it to the x264 (and maybe x265) command line and encode as VFR. The old version of lsmash I'm using can't write a timecodes file, but maybe more recent versions can. For that sort of thing though, their behaviour is good.

Originally Posted by FranceBB

Well absolutely, my point was that in Avisynth when you duplicate, if you don't do anything else, it might be less smooth than applying other techniques like blending. For instance, I would personally get the highest frame rate, output that framerate but instead of duplicating, I would blend the parts with a lower fps. The fact of it being not "ideal" is that if I gotta duplicate first, detect the dups, remove the dups and replace them with blended frames it would be a mess. I would actually love ffms2 / LSMASH to be able to perform blending as an option when specifying fpsnum and den.

I do see your point. Converting to a constant frame rate using blending would no doubt be useful at times. Maybe one day someone clever will create a plugin for it. Thinking about it, there's already a couple for reading a timecodes file and converting to a constant frame rate. For someone who knows what they're doing, I can't imagine adding frame blending logic would be an insurmountable challenge, and it'd no doubt be useful, given VFR seems to be common for video shot with smartphoness.

If ENunn hasn't had any success with lsamsh or FFMS2, maybe he'll have more luck with one of these:

http://avisynth.nl/index.php/VfrToCfr

https://github.com/jojje/VfrToCfr-th...ases/tag/1.1.1

When adding duplicate frames works as it's supposed to it can still be useful though. I've re-encoded VFR sources that obviously started out as combinations of film and video, but the film sections were decimated for 23.976 fps. It's easy enough to convert to a constant 29.97fps, then use TDecimate to decimate the duplicates and frame blend for a CFR, but I've no idea if there's an automated tool for working with non-standard frame rates.

Cheers.

[manolito]

FWIW a good while ago I made this note from a post of one of the DVD gurus at this forum (forgot who it was...)

Foolproof VFR to CFR

If no frame rate is available, or even if one is but it's nonstandard, I'd just open with DSS's convertfps=true with exactly double whatever the final framerate of the DVD is, weave the fields, and encode interlaced. That handles all VFR as optimally as possible without either blending or mo-comp.

More advanced analysis could reveal where progressive with soft pulldown is possible, but you have to parse the timecodes for that and pass the ranges on. Encoding interlaced just works.

Implementation for AVStoDVD:

...Your_Source_Filter...Use "convertfps=true" or "fpsnum/fpsden" and
specify twice the final DVD framerate...


# Insert after the resizer

Last = Video
AssumeTFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()
Video = Last

# The encoder must be set to "Interlaced".