InterFrame 2.8.2 - Framedoubling/60FPS/HFR conversion script

[poisondeathray]

Originally Posted by markfilipak

How can MKV affect PTS? PTS (and DTS) are part of the elemental stream, not part of the container.

Am I misinterpreting things?

2 separate issues are being referred to

1) One is your original 120000/1001 vs. 120.0 fps issue . The input in my little test is clearly 119.88fps

Stream #0:0, 11, 1001/120000: Video: rawvideo (I420 / 0x30323449), yuv420p(progressive), 640x480, 119.88 fps, 120 tbr, 119.88 tbn, 119.88 tbc

But the output is rounded to 120fps -this is a null stream, no container

  Stream #0:0, 0, 1/120: Video: wrapped_avframe, yuv420p(progressive), 640x480, q=2-31, 200 kb/s, 120 fps, 120 tbn

exerpt from -report

[graph 0 input from stream 0:0 @ 0000000afe895840] Setting 'frame_rate' to value '120/1'

Why is ffmpeg setting frame rate to 120/1 ? Over to Paul...



2) Second is the general issue of MKV container default 1/1000s timebase resulting in rounded timestamps. It's fine for playback. Not ok for other workflows with intermediates - potential for propogating errors

eg.

The same 24000/1001 video is encoded using MP4 container vs. MKV container

The resulting timestamps look like this

MKV

0
42
83
125
167
.
.
.

MP4

0
41.7083333333333
83.4166666666667
125.125
166.833333333333
.
.
.

Why is MP4 more precise? - Because of the MP4 timebase scale

Read this for the math
https://www.nmm-hd.org/newbbs/viewto...art=290#p10428

[markfilipak]

Originally Posted by poisondeathray

2) Second is the general issue of MKV container default 1/1000s timebase resulting in rounded timestamps. It's fine for playback. Not ok for other workflows with intermediates - potential for propogating errors

I confess that I don't know MKV, but what about this: http://forum.doom9.org/showthread.php?p=1936167?

Read this for the math
https://www.nmm-hd.org/newbbs/viewto...art=290#p10428

To me, that looks like lipstick on a pig.

If, in addition to the PES's 'PES_HEADER_EXTENSION', the MKV container also stores PTS (but truncated to milliseconds), then MKV is seriously flawed. But I find it hard to believe that MKV (or any container TS) does that. I can't say it doesn't because, as I said, I don't know MKV formatting, but I can say that it would be really strange.

Edit: I would say, stick with the PTSs as stored in the PES and ignore what MKV has in the TS (if, indeed, it does that).

[markfilipak]

Originally Posted by poisondeathray

vspipe --y4m 1.vpy - | ffmpeg -f yuv4mpegpipe -i - -an -f null NUL

Would you post '1.vpy' so I can replicate your findings?

[poisondeathray]

Originally Posted by markfilipak

what about this: http://forum.doom9.org/showthread.php?p=1936167?

Not sure. How does it cover VFR situations ? e.g. smartphones video, webcams, etc.. Currently they are handled by external timestamps ("container") , not in the ES.

Originally Posted by markfilipak

Would you post '1.vpy' so I can replicate your findings?

It's the same thing as 2.vpy without selecteven, or you can comment it out

import vapoursynth as vs
core = vs.get_core()
clip = core.std.BlankClip(format=vs.YUV420P8, length=240, color=[16, 128, 128])
clip = core.text.FrameNum(clip)
clip = core.std.AssumeFPS(clip, fpsnum=120000, fpsden=1001)
clip.set_output()

[poisondeathray]

For the 120fps vs. 120000/1001 - you can encode 1.vpy with other tools like x264cli, x265cli, it results in the proper 120000/1001 output. You can check the encoded framerate and timestamps with other tools, they all agree, even ffprobe/ffmpeg says the x264cli output is 119.88fps not 120fps

The ffmpeg " Setting 'frame_rate' to value '120/1' " I think is not intended behaviour

[markfilipak]

Regarding an MPEG2 PES,

Originally Posted by poisondeathray

Not sure. How does it cover VFR situations ? e.g. smartphones video, webcams, etc.. Currently they are handled by external timestamps ("container") , not in the ES.

Hmmm... I checked a bunch of BDs & DVDs. Some extras and previews are VFR. A few DVD features are VFR if they start with a studio splash at 30fps -- assumed: That it was a "Home Video" splash jammed on for DVD but that wasn't seen in theaters.

The change is, of course, perfectly okay if the fps change happens at a keyframe. In fact, it can happen solely at keyframes.

Currently, I transcode for the main part of the feature or extra, not the opener or splash. I don't care about them. Often, I cut them out.

What I infer from what you write is that ffmpeg (and other tools) don't allow for VFR coming out of the decoder. Am I correct? If so, that's too bad and aught to be fixed. Putting framerate in the transport stream is a poor choice from an architectural point of view. Oh, well.

I understand what you mean now when you say that MKV is a bad choice. And I agree. I picked it solely because it's popular, everthing seems to play it, Handbrake & MKVToolNix, and people said to avoid MP4.

[markfilipak]

Originally Posted by poisondeathray

The ffmpeg " Setting 'frame_rate' to value '120/1' " I think is not intended behaviour

I'm no longer surprised by what passes for "intended behavior". I submitted the following

tinterlace broken - SAR & DAR wrong

ffmpeg -report -i source.mkv -vf "tinterlace=mode=merge" -c:a copy -c:s copy -dn tinterlace.mkv

I made source.mkv from a VOB:
ffmpeg -ss 1.00 -to 2.00 -i g:\VIDEO_TS\VTS_06_1.VOB -vf separatefields -c:a copy -c:s copy -dn source.mkv
That's why source.mkv is MPEG-4 AVC & 720x240 @ 60/1.001 FPS:
Stream #0:0: Video: h264 (High), yuv420p(tv, smpte170m, progressive), 720x240 [SAR 8:9 DAR 8:3], 59.94 fps, 59.94 tbr, 1k tbn, 119.88 tbc (default)

Therefore, tinterlace.mkv should be 720x480 @ 30/1.001 FPS [SAR 8:9 DAR 4:3] but it's not. tinterlace.mkv is:
Stream #0:0: Video: h264 (High), yuv420p(tv, smpte170m, top coded first (swapped)), 720x480 [SAR 16:9 DAR 8:3], 29.97 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default)

Though I've never done it before, I think I can force SAR & DAR, but tinterlace really needs to be fixed.

Thanks,
Mark.

and Paul said it was intended.

He is diddling SAR based, not on differential resampling of source pixels at a different sample aspect, but on what makes other things work -- at least, that's what I think (sort of offsetting penalties in various use-cases).

[poisondeathray]

Originally Posted by markfilipak

I understand what you mean now when you say that MKV is a bad choice. And I agree. I picked it solely because it's popular, everthing seems to play it, Handbrake & MKVToolNix, and people said to avoid MP4.

I never said "bad choice" ; I said

Originally Posted by poisondeathray

I would avoid mkv if you want more accurate timestamps. mkv uses a timebase of 1/1000s so there is rounding and some jitter

It's perfectly fine for final playback purposes. Don't overlook the other advantages that MKV has over other containers in stream support, A/V format support, sub support, attachments, ordered chapters. There are pros/cons to everything.

[markfilipak]

Originally Posted by poisondeathray

I never said "bad choice" ...

Oops! Indeed, you did not. Kindly excuse me. I'm still attempting to determine the properties of the various TS containers. It's not an easy task as there's much contradictory info/opinions.

[markfilipak]

Originally Posted by poisondeathray

[MKV is] perfectly fine for final playback purposes. Don't overlook the other advantages that MKV has over other containers in stream support, A/V format support, sub support, attachments, ordered chapters. There are pros/cons to everything.

You are so far down the road I'm trudging... May I ask: What if I transcode to MP4 and then remux to MKV? Would that be the best of both worlds? Would that be the best solution in my quest for perfection?

[poisondeathray]

Originally Posted by markfilipak

You are so far down the road I'm trudging... May I ask: What if I transcode to MP4 and then remux to MKV? Would that be the best of both worlds? Would that be the best solution in my quest for perfection?

Remux with mkvmerge or similar? - No difference in terms of timestamps, the end result will be the same without the MP4 step

[markfilipak]

Originally Posted by poisondeathray

Remux with mkvmerge or similar? - No difference in terms of timestamps, the end result will be the same without the MP4 step

Oh! I get it (I think). The problem isn't just that MKV has a 1 kHz timebase. It's that the ffmpeg filter pipeline also uses a 1 kHz TB. So, using a PES that has a 90 kHz TB wouldn't make any difference -- the resolution just isn't there.

Is that what you demonstated by writing to NULL? Do I have that right?

Oh, dear. I guess 1 kHz TB is baked into the decoder, right? If so, then there is no 'better' tool, or is there? Do all transcoding apps use the same decoders?

[poisondeathray]

Originally Posted by markfilipak

Oh! I get it (I think). The problem isn't just that MKV has a 1 kHz timebase. It's that the ffmpeg filter pipeline also uses a 1 kHz TB. So, using a PES that has a 90 kHz TB wouldn't make any difference -- the resolution just isn't there.

Is that what you demonstated by writing to NULL? Do I have that right?

Oh, dear. I guess 1 kHz TB is baked into the decoder, right? If so, then there is no 'better' tool, or is there? Do all transcoding apps use the same decoders?

What you start with doesn't matter in this MKV tbn regard ("tbn" what ffmpeg labels the container timebase), as soon as you put it into MKV, you use the 1K tbn. It doesn't matter what you use to parse the MKV . It does not matter if you non-ffmpeg/libavcodec tools either

You would expect (or I expected from prior tests years past) there to be no difference with using MP4 step - and I retested it again today, then read the timestamps in the output MKV file, and it confirms no difference. Those little rounding differences are not a problem for regular end playback situations. You have to be superhuman to perceive some difference. But it can be an issue for intermediates and some types of workflows


NUL is partly to help avoid all the other variables. It's part of the "basic scientific method (TM)". Eliminate all the other confounding variables like filters, containers, audio etc.. eg. Maybe adding of subtitles screws up timing; Maybe muxing audio screws something else up, etc... Strip it down to the smallest elements that can reproduce the issue. The issue was the 120000/1001 to 120 fps change. So that's what you test. Don't go testing other stuff or adding extraneous things unless you find out it's contributing to the problem. For debugging nobody want to wait for libx265 to complete. Nobody wants to download a big file to test, unless it's actually part of the problem. If you can simplify everything, each test run observation goes faster. If you can reproduce with blankclip or something like testsrc2 in ffmpeg, all the better. Do some of the legwork - so if there is a real bug - the developer does not have to spend extra time working things out

[markfilipak]

Because of excellent help here -- thank you ChaosKing & lansing & poisondeathray -- I succeeded piping vspipe to ffmpeg.

Now, I need to go in the opposite direction. I did a ton of reading and head scratching, then I guessed (and failed twice).

What's wrong?

Failure #1

ECHO from vapoursynth import core>Mark's.vpy
ECHO video = core.ffms2.Source(source=-)>>Mark's.vpy
ECHO import havsfunc as havsfunc>>Mark's.vpy
ECHO video = havsfunc.InterFrame(video, Preset="medium", Tuning="smooth", InputType="2D", NewNum=60000, NewDen=1001, GPU=True)>>Mark's.vpy
ECHO video.set_output()>>Mark's.vpy
ffmpeg -i INPUT.MKV -vf "...works.skipping..." -f rawvideo - | vspipe --y4m Mark's.vpy OUTPUT.MKV
...
Output #0, rawvideo, to 'pipe:':
  Metadata:
    encoder         : Lavf58.65.101
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p(tv, smpte170m, progressive), 240x236 [SAR 32:27 DAR 640:531], q=2-31, 16296 kb/s, 23.98 fps, 23.98 tbn (default)
    Metadata:
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      BPS-eng         : 642744
      DURATION-eng    : 00:00:05.739000000
      NUMBER_OF_FRAMES-eng: 172
      NUMBER_OF_BYTES-eng: 461089
      _STATISTICS_WRITING_APP-eng: mkvmerge v51.0.0 ('I Wish') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-12-21 04:14:21
      DURATION        : 00:00:05.739000000
      encoder         : Lavc58.119.100 rawvideo
Script evaluation failed:
Python exception: invalid syntax (Mark's.vpy, line 2)

Traceback (most recent call last):
  File "src\cython\vapoursynth.pyx", line 2241, in vapoursynth.vpy_evaluateScript
  File "Mark's.vpy", line 2
    video = core.ffms2.Source(source=-)
                                      ^
SyntaxError: invalid syntax

av_interleaved_write_frame(): Broken pipe
Error writing trailer of pipe:: Broken pipetime=00:00:00.04 bitrate=16296.2kbits/s speed=0.453x
frame=    1 fps=0.3 q=-0.0 Lsize=      83kB time=00:00:00.04 bitrate=16296.2kbits/s speed=0.0121x
video:83kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
Conversion failed!

Failure #2

ECHO from vapoursynth import core>Mark's.vpy
ECHO video = core.ffms2.Source(source=-)>>Mark's.vpy
ECHO import havsfunc as havsfunc>>Mark's.vpy
ECHO video = havsfunc.InterFrame(video, Preset="medium", Tuning="smooth", InputType="2D", NewNum=60000, NewDen=1001, GPU=True)>>Mark's.vpy
ECHO video.output(OUTPUT.MKV)>>Mark's.vpy
ffmpeg -i INPUT.MKV -vf "...works.skipping..." -f rawvideo - | vspipe --y4m Mark's.vpy 
...
Output #0, rawvideo, to 'pipe:':
  Metadata:
    encoder         : Lavf58.65.101
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p(tv, smpte170m, progressive), 240x236 [SAR 32:27 DAR 640:531], q=2-31, 16296 kb/s, 23.98 fps, 23.98 tbn (default)
    Metadata:
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      BPS-eng         : 642744
      DURATION-eng    : 00:00:05.739000000
      NUMBER_OF_FRAMES-eng: 172
      NUMBER_OF_BYTES-eng: 461089
      _STATISTICS_WRITING_APP-eng: mkvmerge v51.0.0 ('I Wish') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-12-21 04:14:21
      DURATION        : 00:00:05.739000000
      encoder         : Lavc58.119.100 rawvideo
av_interleaved_write_frame(): Invalid argument
Error writing trailer of pipe:: Invalid argument00:00:00.04 bitrate=16296.2kbits/s speed= 0.4x
frame=    1 fps=0.0 q=-0.0 Lsize=      83kB time=00:00:00.04 bitrate=16296.2kbits/s speed=0.18x
video:83kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
Conversion failed!

[poisondeathray]

Originally Posted by markfilipak

Failure #1
[CODE]ECHO from vapoursynth import core>Mark's.vpy
ECHO video = core.ffms2.Source(source=-)>>Mark's.vpy
ECHO import havsfunc as havsfunc>>Mark's.vpy
ECHO video = havsfunc.InterFrame(video, Preset="medium", Tuning="smooth", InputType="2D", NewNum=60000, NewDen=1001, GPU=True)>>Mark's.vpy
ECHO video.set_output()>>Mark's.vpy
ffmpeg -i INPUT.MKV -vf "...works.skipping..." -f rawvideo - | vspipe --y4m Mark's.vpy OUTPUT.MKV

ffmpeg raw pipe to vpy can be done with vsrawsource
https://github.com/HolyWu/vsrawsource/releases/

But vapoursynth usually is not used to write an output file directly without other libraries, typically you'd pipe to something else to write the final file (such has ffmpeg or similar)

It would look like this

ffmpeg -i "input.mkv" -vf..... -f rawvideo - | vspipe --y4m script.vpy - | ffmpeg -f yuv4mpegpipe -i - ..... "output.mkv"

Where the source filter in the vpy would look something like this. A rawvideo pipe require you to enter dimensions, pixel format, and framerate. yuv4mpegpipe does not. Since you're sending rawvideo from ffmpeg into the vpy script, you have to enter those parameters

video = core.raws.Source("-", width=1920, height=1080, src_fmt="YUV420P8", fpsnum=24000, fpsden=1001)

It would usually make more sense to do the filtering in vapoursynth since you're already using it for interframe, then to ffmpeg to write the final file . Piping back and forth adds overhead (slower, more memory consumption). A simplified workflow, fewer steps is generally preferred, unless there are some ffmpeg operations that you cannot emulate in vapoursynth

And if your desired output is in MKV container , it's still going to have a timebase of 1/1000s

[markfilipak]

Originally Posted by poisondeathray

ffmpeg raw pipe to vpy can be done with vsrawsource ...

Thanks for the push. Maybe I can get the rest of the way out of the ditch on my own. I'll try. I'm doing a lot of processing because the source was telecined, then a frame dropped back to 24fps, then a frame repeated back to 30fps... not by me, but I assume by a video engineer who did not have the intermediates -- originals had probably been discarded -- and the engineer needed to integrate the result in with a mix of other telecined material and NTSC video and 30fps progressive. It's a mess, but I can get the original 24fps with just a single frame that's field repeated at 60fps. Whew! I'll need to review the VS functions to see whether there's coverage of all the gymnastics I'm doing in ffmpeg. I doubt I'll be able to do it as, in the ffmpeg filter chain, I have to do a lot of intermediate 'setpts's and 'fps's to get the whole process to work out without mixing up the frame order.

And if your desired output is in MKV container , it's still going to have a timebase of 1/1000s

I would prefer to avoid 1kHz TB until the very end, and a suggestion how I can do that would be very welcome, but I suspect that the ffmpeg filter chain also uses 1kHz TB. I've not been successful getting an answer to the question of decoder/pipeline TB from the ffmpeg devs, and I don't know how to ascertain that answer on my own: I don't read 'C', and I don't know how to probe the filter pipeline.

I've noticed that, with all the processing I do (the gymnastics), the ffmpeg TB winds up being '23.98 tbn' in the stage just prior to interpolating to 60fps.

[poisondeathray]

Originally Posted by markfilipak

I'm doing a lot of processing because the source was telecined, then a frame dropped back to 24fps, then a frame repeated back to 30fps... not by me, but I assume by a video engineer who did not have the intermediates -- originals had probably been discarded -- and the engineer needed to integrate the result in with a mix of other telecined material and NTSC video and 30fps progressive. It's a mess, but I can get the original 24fps with just a single frame that's field repeated at 60fps. Whew! I'll need to review the VS functions to see whether there's coverage of all the gymnastics I'm doing in ffmpeg. I doubt I'll be able to do it as, in the ffmpeg filter chain, I have to do a lot of intermediate 'setpts's and 'fps's to get the whole process to work out without mixing up the frame order.

It suspect it would be easier to do in vapoursynth or avisynth without all the pts gymnastics.

I would prefer to avoid 1kHz TB until the very end, and a suggestion how I can do that would be very welcome, but I suspect that the ffmpeg filter chain also uses 1kHz TB. I've not been successful getting an answer to the question of decoder/pipeline TB from the ffmpeg devs, and I don't know how to ascertain that answer on my own: I don't read 'C', and I don't know how to probe the filter pipeline.

I've noticed that, with all the processing I do (the gymnastics), the ffmpeg TB winds up being '23.98 tbn' in the stage just prior to interpolating to 60fps.

"-i INPUT.mkv"

When you start with MKV, you start with that 1/1000s timebase. If you're doing other operations that require precise timing instead of rounded timestamps....don't start with MKV. In ffmpeg, a vob from DVD-video , or .m2ts from BD has a tbn of 1/90000s. You can check/verify the timestamps after each step, each filter, by inserting a -vf showinfo . You can set null output and use -report -loglevel quiet

[markfilipak]

Originally Posted by poisondeathray

When you start with MKV, you start with that 1/1000s timebase. If you're doing other operations that require precise timing instead of rounded timestamps....don't start with MKV.

Oooooohhhh! I see. I get it now. FFmpeg is using what it's given, eh?

Well, I've been using MKV sources just for development. My eventual sources will all be VOBs and M2TSs.

BTW, ChaosKing included vsrawsource in his distro (which I jumped on), so I can use it 'out of the box' so to speak.

You guys have saved my life. I would have committed 'sewercide' by now if it wasn't for y'all.

[flossy_cake]

Is this bit correct in InterFrame2.avsi?

# Create SuperString
Preset == "Fastest" ? Eval("""
SuperString = "{pel:2,"
""") : Preset == "Fast" || Preset == "Faster" ? Eval("""
SuperString = "{pel:1,"
""") : Eval("""
SuperString = "{"
""")

Because...

Originally Posted by http://avisynth.nl/index.php/SVPflow

"pel:" The accuracy of the motion estimation. Value can only be 1, 2 or 4. 1 means a precision to the pixel, 2 means a precision to half a pixel, 4 - to quarter pixel (not recommended to use).

It sounds like 4 is highest quality, 1 is lowest. I'm assuming by "quarter pixel" they mean subpixel precision?

By the way to anyone reading this who might be wondering how to get InterFrame working in 64-bit Avisynth+ without requiring SVPManager to be running just use the v4.2.0.142 dlls from the link above. InterFrame is basically just presets for SVPFlow - not that this diminishes its usefulness.

Then you can go something like...

InterFrame(GPU=true, tuning="film", preset="medium", Cores=8).Prefetch(8)

I'm not sure if I'm doing the multithreading correctly though, only tested with 576p files so far. My CPU and GPU barely breaking a sweat though - would like to crank up the quality more to see if artefacts can be mitigated further.

edit: okay increasing cores & prefetch to 8 makes 1080p possible now but still quite CPU heavy - I thought this was going to be mostly done on the GPU. Oh well it's still very cool, makes 576p25 streaming service Top Gear episodes look more like broadcast version.

[DTL]

"It sounds like 4 is highest quality, 1 is lowest. I'm assuming by "quarter pixel" they mean subpixel precision?"

Yes. Also performance greatly depends on pel value so fastest is pel=1 and slowest is pel=4 (quater size of MV length in the inter-samples virtual distance).