Variable framerate

[deama]

So basically I used a frame extractor in a video to get all it's frames, then I used an image enhancer to enhance the images, and then I used ffmpeg to make a video out of them again. However, the problem is that the sound doesn't match up with it anymore.

Everytime I try to correct the sound, it slowly diverges over a minute or two and it's back to being not in sync anymore. Is there a way I could capture the variable framerate from the original and put it on the new one? Or perhaps try to sync up the audio by hand without losing quality?

[poisondeathray]

Extract the timecodes (timestamps) from the source, mux timecodes into the new video

What tools you might use depends on what type of video and containers you have . eg you might use mkvextract and mkvmerge for mkv , or mp4fpsmod for mp4 etc...

If the framecount is the same as the source, you're done. Otherwise you have to edit the timecodes

[deama]

Extract the timecodes (timestamps) from the source, mux timecodes into the new video

What tools you might use depends on what type of video and containers you have . eg you might use mkvextract and mkvmerge for mkv , or mp4fpsmod for mp4 etc...

If the framecount is the same as the source, you're done. Otherwise you have to edit the timecodes
I have MKVTool. I'm using the GUI and I can see a field that allows me to provide a timestamp file, but I donno how to extract the timestamp from the other video, any ideas?

EDIT: Oh ok, nvm, apparentely you need to use the mkvextract.exe via the command line.

[deama]

Ok so I have another problem related to this.

I took the video in question and used interpolation algorithms to double it's framerate. It came out well and all, but now the timestaps doesn't work for it, so the audio is out of sync because the timestamp file is for the 24fps version instead of the 48fps version I made.

Anyone know how to fix this? I tried doubling all the entries in the timestamp, but that ended up making the video 12fps, I tried halfing all the timestamps, which did it, but then the video cut off half way because I guess the timestamps ran out?

[hello_hello]

Does the frame rate vary all over the place or is it a combination of film and video, or that sort of thing. If the answer is yes, can you easily tell which section is which? If the answer is yes, you could go through the video and note the start and end frame numbers for each video section, or each film section, then make a timecodes file to match. A basic timecodes file looks like this:

# timecode format v1
Assume 59.940060
0, 3539, 23.976024
3540, 3699, 23.976024
5998, 6342, 23.976024
19263, 20258, 23.976024
22887, 23268, 23.976024

x264 still seems to need
# timecode format v1
as the first line when being fed a timecodes file for VFR encoding, but for muxing I vaguely remember MKVToolNix switched to
# timestamp format v1
Although it might still accept either.

By the way. MKVCleaver and gMKVExtractGUI both tend to make extracting stuff from MKVs a fair bit less painful.

[deama]

I think the framerate varies all over the place. When I used the extractor it gave me timestamp v2. I think I might be able to tell in what section the framerate switches, but that would be too painful, I tried doing it and I think it did it every 1-2 minutes and the video is like 30-35 minutes long, and I have no experience with this timecode stuff, before that video, I didn't even think it was possible to switch framerate in the middle of the video just like that, pretty annoying.

[hello_hello]

Somehow you need to double the number of frames in the timecodes file while halving their duration. You can extract the timecodes file from your encode, which we know are wrong, but would at least have the correct number of frames. Then taking the timecodes from the source file you'd have to use each frame's duration for two frames in the new timecodes.

0 0
21
42 42
63
83 83
104
125 125
146
167 167

I can't think of an automated way to do that, but converting your v2 timecodes to v1 timecodes might make it easier because they should be grouped (similar to the example in my previous post), and in theory all you should need to do is double the number of frames in each group and double each group's frame rate. At least that's how it works in my head.

I don't think there's a way to extract timecodes as v1 but there's two 7zip files attached to this post and they both contain timecode converters.
https://forum.doom9.org/showthread.php?p=1445996#post1445996

If they spit out your v2 timecodes, change timestamp to timecode in the first line.

[deama]

I'm not quite sure what to do here. I just extracted and converted to v1, here are the results:

original:
# timecodes format v1
Assume 29.970
56,572,23.976
573,573,-21453
574,574,-21453
607,607,-21453
608,608,-21453
609,701,23.976
707,1090,23.976
1096,1107,23.976
1113,1124,23.976
1130,1540,23.976
1541,1541,-21453
1542,1544,23.976
1572,1572,-21453
1573,1573,-21453
1574,1575,23.976
1576,1576,-21453
1577,1577,-21453
1585,1587,23.976
1588,1588,-21453
1589,1592,23.976
1593,1593,-21453
...


2x framerate version:
# timecodes format v1
Assume 29.970
1,83987,47.952


What would I do in this situation?

[hello_hello]

The timecodes from the 2 x encode indicate a constant frame rate of 47.952 was used. They're not relevant because you're going to replace them.

I'm not quite sure what to make of the frames with negative frame rates in the v1 timecodes from the source. You might need to check the v2 timecodes to see if you can work out what they mean, so I'll ignore them for the moment. Are there a lot of frames with negative frame rates and do you get the same v1 timecodes using both the converters I linked to? I'd possibly start by removing any frames with negative frame rates completely, and after converting the rest, test the new timecodes to see if you're at least in the A/V sync ballpark.

Assume 29.970
56,572,23.976
609,701,23.976
707,1090,23.976

If the video is exactly as you described, I think all you should need to do is multiply every number you see by 2 for the new timecodes. The frames that aren't grouped will display at 59.940 instead of 29.970, and each group of frames will contain twice as many, displaying at 47.952 instead of 23.976. Without the frames with a negative frame rate, the first four lines above would become:

Assume 59.940
112,1144,47.952
1218,1402,47.952
1414,2180,47.952

The negative frame rates might be due to "jitter" in the original timecodes. At 23.976fps the frame duration is 41.70833ms, but the MKV timecodes are usually rounded to the nearest ms, so what you end up with for a 23.976 section is timecodes alternating between 41ms and 42ms in some sort of pattern. Something similar would apply to the 29.97fps sections. At the end of each 23.976 section there might be a few ms the converter can't compensate for, or I simply don't know what it's trying to say, but given those negative frame rates appear to occur in a regular pattern, I think it's likely that's the cause. If they only represent a few ms each time, or a fraction of a ms, then you should be able to simply delete them.

Could you do me a favour and let me know how that works out? I'm curious.

Edit. If the negative frame rates do represent a frame rate, then logically those frames would display for -0.0467ms (1000 / -21453), so if they're the way the converter is handling jitter, I think it'd mean when you delete one, you're making that frame display for 16.684ms instead (59.94fps), which means the A/V sync would change by 16.7307ms.
The frames at 47.952fps display for 20.854ms, so in my head it works out every time you delete a frame with a negative frame rate, you'd have to remove four frames from a group displaying at 47.952fps to make them display at 59.94fps instead (or two frames when they're 23.976fps), correcting the audio sync by 4.17ms for each frame, and by 16.68ms in total, making up for deleting a frame with a negative frame rate.

That seems to be logical, or it could be a load of crap, but it's the sort of thing to keep in mind if the audio sync is wrong when you simply delete those frames. Nothing's ever easy.....

[poisondeathray]

You can try tcConv from tritical; it will probably do a better/cleaner job of the v2 to v1 conversion (and you can set a max_diff to filter out values if you need to)

But I'd approach it the same way hello_hello outlined above . There might be some excel macro or trick to automatically do it (basically multiply entries by 2) , not sure

[deama]

Alright so... it doesn't work.
I used a different converter and it removed the negative stuff, works fine with original.
If I try to double all the numbers, it becomes very off-sync with the audio.
If I try to double just the framerate, gets pretty close, but seems to be off by 2-3 seconds, I'll try playing with the delay.
I made a python script to double the numbers, and upon looking at them, they seem fine.

EDIT: Addin a delay doesn't really help as some parts are off by +1 second, whereas other parts are off by -1 seccond, I guess due to the variability of the fps? Idk.

[hello_hello]

How many frames are there in total for each version? Using the v2 timecodes it's easy to tell. If you open them with any text editor that displays the current line number in the status bar and you scroll to the end, the line number should be the same as the number of frames, minus the header line, I guess.
If the number of frames for the encoded version isn't double the original, then it's not quite as "simple" as it appeared it should be.

Which I guess leads to another question. How many images were there and did the same number of frames come out (or double) after converting them to video? It seems kind of odd that the original timecodes are almost okay if you simply double the frame rate.

[deama]

For the original there's 45769 linse (v2).
For the new one there's 83989 lines (v2).

For the new one, I simply doubled the amount of frames there were, so it should work, quite confused myself.

[poisondeathray]

How did you "extract" the original frames, and how many were there? (what was the framecount)

[deama]

I used mkextract to get the timecodes of the original frames. It gave me 83,989 lines of those numbers + the #timestamp v2 header.