FFMS2000 - The experimental future of FFMS2

[dipje]

I have issues with .MTS files (avchd so they contain h264 and ac3).

The files have some weird timing . The audio starts sooner than the video most of the time. It seems when I press record on my camera it starts capturing audio immediately, but video takes half a second or so to get going. So the timecodes of the videostream don't start at 00:00:00. What ffms2000 seems to do is freeze-frame the first videoframe until a/v sync is achieved.
But halfway through the files the frames seem to jump backward and forward , weird glitches. Glitches that aren't there with ffplay or playing through mpc-hc and lavfilters.

The thing is, I don't think this is a regression from ffms2, I think ffms2 also has the same issue, and the issue comes from the libavformat demuxer for MTS which is 'not recommended' I believe.

In older ffms2 versions (2.1 or earlier even ?) You could disable the libav demuxer during indexing, so it would use another one (don't know which, haali or something from matroska that could read MTS ? ). Or maybe I remuxed the MTS files to mkv to prevent issues.

Anyway, lsmash libav reads them ok. It uses a different MTS parser right ?

All other files I use recently (magicyuv 10bit RGB, dnxhr 444, Cineform RGB ) seem to work just fine. (Only tested through Vapoursynth x64)

[Myrsloik]

I have issues with .MTS files (avchd so they contain h264 and ac3).

The files have some weird timing . The audio starts sooner than the video most of the time. It seems when I press record on my camera it starts capturing audio immediately, but video takes half a second or so to get going. So the timecodes of the videostream don't start at 00:00:00. What ffms2000 seems to do is freeze-frame the first videoframe until a/v sync is achieved.
But halfway through the files the frames seem to jump backward and forward , weird glitches. Glitches that aren't there with ffplay or playing through mpc-hc and lavfilters.

The thing is, I don't think this is a regression from ffms2, I think ffms2 also has the same issue, and the issue comes from the libavformat demuxer for MTS which is 'not recommended' I believe.

In older ffms2 versions (2.1 or earlier even ?) You could disable the libav demuxer during indexing, so it would use another one (don't know which, haali or something from matroska that could read MTS ? ). Or maybe I remuxed the MTS files to mkv to prevent issues.

Anyway, lsmash libav reads them ok. It uses a different MTS parser right ?

All other files I use recently (magicyuv 10bit RGB, dnxhr 444, Cineform RGB ) seem to work just fine. (Only tested through Vapoursynth x64)

Do you have a sample MTS file? It could be timestamp discontinuities (or something else) we're trying to fix right now.

I have no idea what l-smash uses. All I know is that it does everything very differently.

Haali splitter support was removed several years ago because haali stopped developing it looooooooong ago (and lavf got better)

[zub35]

Add the ability to convert vfr to cfr using two methods: duplicate or blend.
Only DSS2 does the correct conversion, by duplication.

[Myrsloik]

Add the ability to convert vfr to cfr using two methods: duplicate or blend.
Only DSS2 does the correct conversion, by duplication.

Duplication already exists. Also lrn2vfr. It's 2017.

[zub35]

Duplication already exists. Also lrn2vfr. It's 2017.

This is not right. ffms averages fps and discards frames that exceed it.
Dss2 works with the top bar of VFR and duplicates frames to it, without discarding.
p.s. Or maybe I do not understand something ...

Also that's when fps jumps to the insane 1000, and then all the other frames are duplicated to it, which is also bad.
It is necessary that ffms find the average maximum value, discarding too large bursts.

[Myrsloik]

This is not right. ffms averages fps and discards frames that exceed it.
Dss2 works with the top bar of VFR and duplicates frames to it, without discarding.
p.s. Or maybe I do not understand something ...

No frames are discarded umless you use the cfr mode. You're wrong.

[zub35]

sample: https://cloud.mail.ru/public/BX68/Cxg3rF7iX (sorry, only erotic)
dss2 30fps and 20970 frames [11:38.999]
ffms2000 20.220fps and 14113 frames [11:37.958]

Because of what, the video in ffms is accelerating, then slowing down

[Myrsloik]

Don't even have to look at it. It's because it has dropped frames (duplicates/nvops) and only returns actually coded frames. The file is vfr and you're using the vfr mode. Everything works as expected.

[zub35]

The video is not behaving naturally. Rides with acceleration and deceleration. This not normal
Also, it would still be great ffms2000 worked with timecodes_v2 in this conversion vfr to cfr, loading them from file. This was-would be useful for raw streams.

[Myrsloik]

The video is not behaving naturally. Rides with acceleration and deceleration. This not normal

Lrn2vfr. The dropped frames do exactly that.

[TheFluff]

Use the parameters fpsnum and fpsden if you want CFR by duplicating frames. You could try reading the manual, you know. It's really not bad at all.

(the default behavior is to output coded frames and optionally a timecodes file so you can do the right thing instead of this silly CFR business)

[dipje]

Do you have a sample MTS file? It could be timestamp discontinuities (or something else) we're trying to fix right now.

You have a private message

[Myrsloik]

Test4 released. Fixes some minor issues in the avisynth source.

[madshi]

Is the HDR metadata included in this build? How exactly is it transported?

[Myrsloik]

Is the HDR metadata included in this build? How exactly is it transported?

It's exported as new fields in FFMS_Frame. And set as the frame propertiesMDMDisplayPrimariesX
MDMDisplayPrimariesY
MDMWhitePointX
MDMWhitePointY
MDMMinLuminance
MDMMaxLuminance

[madshi]

Hmmmm... I tried this:

video = core.ffms2.Source('LG_Chess_HDR.mp4')

const VSFrameRef *src = vsapi->getFrameFilter(n, d->node, frameCtx);
const VSMap *props = vsapi->getFramePropsRO(src);
int error;
int transfer = (int) vsapi->propGetInt(props, "_Transfer", 0, &error); if (error) transfer = -1;
int maxLum = (int) vsapi->propGetInt(props, "MDMMaxLuminance", 0, &error); if (error) maxLum = -1;
It works for "_Transfer", but I get error "1" (which seems to mean "peUnset") when asking for "MDMMaxLuminance". I also tried "propGetFloat()", but same error. What am I doing wrong?

Btw, which data format does "MDMDisplayPrimariesX/Y" use?

[Myrsloik]

Float, array of 3 values. Just like ffmpeg exports it.

[madshi]

Ok, tried "vsapi->propGetFloat(props, "MDMDisplayPrimariesX", 0, &error)", but I also get error 1.

[Myrsloik]

Ok, tried "vsapi->propGetFloat(props, "MDMDisplayPrimariesX", 0, &error)", but I also get error 1.

It's not flagged for all frames. Request a 100 or so and one will have it.

[sneaker_ger]

For HEVC (and probably AVC) it should apply to all frames of a GOP, I think.
https://forum.doom9.org/showthread.php?p=1804070#post1804070

[Myrsloik]

Applying and being flagged are different things. This is how the ffmpeg api handles things.

[madshi]

Sorry, but still doesn't work. All first 100 frames in 2 different HDR demos I've tried (one MKV, one MP4) report error 1. LAV -> madVR during video playback report metadata just fine, so the metadata is definitely there.

But even if it worked after a couple dozen frames, I'm not sure it's a good idea for a VapourSynth filter to request 100 frames in advance just to be able to get metadata? Of course it would be possible to work without the metadata first, and then switch to a different processing mode once the metadata is available, but that doesn't sound really nice, either.

[Myrsloik]

Sorry, but still doesn't work. All first 100 frames in 2 different HDR demos I've tried (one MKV, one MP4) report error 1. LAV -> madVR during video playback report metadata just fine, so the metadata is definitely there.

But even if it worked after a couple dozen frames, I'm not sure it's a good idea for a VapourSynth filter to request 100 frames in advance just to be able to get metadata? Of course it would be possible to work without the metadata first, and then switch to a different processing mode once the metadata is available, but that doesn't sound really nice, either.

LG_Chess_HDR.mp4 has it on the first frame for me. I guess adding logic to repeat the metadata for every frame could be useful but that one appears to work. Can I have a sample that doesn't work?

(it does however have some kind of seek issue so that'll be fun to debug)

[madshi]

LG_Chess_HDR.mp4 is one of the files I tested with, and I weren't able to read any MDM metadata in the first 100 frames... :( Maybe I did something wrong? VapourSynth\plugins64\ffms2.dll is definitely from today, though.

Remembering/repeating the metadata would probably be useful, at least until you know for sure that the content is no longer HDR (e.g. next GOP?).

[Myrsloik]

LG_Chess_HDR.mp4 is one of the files I tested with, and I weren't able to read any MDM metadata in the first 100 frames... :( Maybe I did something wrong? VapourSynth\plugins64\ffms2.dll is definitely from today, though.

Remembering/repeating the metadata would probably be useful, at least until you know for sure that the content is no longer HDR (e.g. next GOP?).

Even tested exactly the dll I uploaded. Still works. I use clip.text.FrameProps() to show all properties and it's there. Odd

[madshi]

I didn't know "clip.text.FrameProps()" existed, it's quite useful! It helped me find the issue:

I had tested with the VapourSynth Editor, and for some reason it had loaded an old ffms2.dll from a python subfolder instead of the regular location. When running the vpy script directly, metadata transport works fine!

Sorry for the false alarm, and thanks for adding HDR metadata support. It would still be nice to have it working from frame 1, and to have it remember the metadata, if possible.

[sneaker_ger]

But it is working on the first frame? At least on the LG sample. But probably most others as well.

[madshi]

I've tested with 2 different samples and it seems to work on the first frame for both. So it seems there's no issue there.

[Myrsloik]

Due to a few huge coincidences and madshi's transport stream (with mp4 extension, just that took me a while to realize) we may have found a better strategy for seeking in mpeg and mpegts. Grab test5 from the first post and report your results relative to test4. It should have a lot less seeking issues in mpeg and transport streams now. Download link in the first post.

It would be very helpful if you throw all your mpeg and transport streams at it.

[sneaker_ger]

Yes, looking good.

Don't forget to let test5 create new .ffindex files, people.

[Myrsloik]

Yes, looking good.

Don't forget to let test5 create new .ffindex files, people.

Yes, delete all old ones. I forgot to bump the index version and using an old index=>old issues.

[dipje]

The weird things I had at the start of my MTS files seem fixed now. Stepping through the files (forward) reveal nothing out of the ordinary an all good.

Stepping backwards still starts acting weird, but you might just call that 'by design and do not seek backwards' I don't know.
If I start stepping backwards (after having stepped forward through the whole video) the first few frames seek very quick (cached I guess) then the frames seek back slower (to be expected, seeking backwards in a file that is not All-I).
But after around 80 frames the picture 'freezes'. I can step back a few frames but they all look the same (they didn't while seeking forward through the file). This lasts for around 10 frames (GOP length?). Then it goes to slowly seeking backwards frame by frame again.

Now, if I start stepping forward at this moment, those 10 'freeze frames' are still there. I have to step back a whole lot more and then step forward ago and now the exact same frames step forward OK. So is there actually some seeking gone bad here, or is the Vapoursynth cache being naughty? Or both?

What appears to be happening is that after seeking +/- 80 frames backwards ffms2 produces 10 frames of 'freeze frame', and Vapoursynth caches those 10 frames, so stepping them forward again will not fix it while the bad frames are still in the cache.

@Myrsloik: You still have my MTS example file right? Create a simple script that opens it with ffms2 with nothing more, open it in Vapoursynth Editor. Go to the last frame, then start holding down the arrow-down key to step backward. Around frame 192 the 'freeze frame' occurs. Also happens in VDFilterMod (to rule out a bug in Vapoursynth Editor or something).
If I open the file and don't seek to the end but instead seek to frame +/- 220 and start holding down the arrow key, the glitch once again occurs at frame +/- 192 (same spot) so it does seem related to that position in the file, not the amount of frames I stepped backwards or anything.

PS (And yes, I deleted .ffindex file before I started)
PS2 (And I'm talking about test5, just to be clear)

[stax76]

It would be very helpful if you throw all your mpeg and transport streams at it.

How would I handle audio, dump or audiodub?

I want to encode this sample:

http://www.mediafire.com/file/t51xbk6wsb5kvdf/480p__AVC__59%2C940fps__AC3_2.0__Growing_Pains.ts

Some script code and command lines to get me started would be helpful.

[Myrsloik]

The weird things I had at the start of my MTS files seem fixed now. Stepping through the files (forward) reveal nothing out of the ordinary an all good.

...

Interesting. It at least seems to improve and probably exposes a few different issues. So many old hacks to re-evaluate in there...

I wish I'd kept a bigger collection of samples.

[Atak_Snajpera]

Is this normal that in latest ffms2000 test5 all frames in VC-1 are detected as keyframes?
Example
# keyframe format v1
fps 0
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

FFMS 2.20
# keyframe format v1
fps 0
0
1
25
49
53
77
101
106
130
154
178
202
226
250
274
298
322
346
370
394
418
442
466
490

[Myrsloik]

Is this normal that in latest ffms2000 test5 all frames in VC-1 are detected as keyframes?
...

No, can I have a small sample?

[Atak_Snajpera]

No problem give me 15 minutes...

[Atak_Snajpera]

ok done
http://www.mediafire.com/file/98z5ynb5m6dwb0r/VC-1.mkv

[Myrsloik]

ok done
http://www.mediafire.com/file/98z5ynb5m6dwb0r/VC-1.mkv

Must be a broken mkv/ffmpeg bug. The parser really does report all the frames as keyframes. I'll poke it a bit more but definitely not my fault.

[FranceBB]

It seems that the latest ffms2000 breaks BePipe -> NeroAACEnc compatibility.
Although the audio stream is detected by BePipe, NeroAACEnc can't encode it and fails.

[Myrsloik]

It seems that the latest ffms2000 breaks BePipe -> NeroAACEnc compatibility.
Although the audio stream is detected by BePipe, NeroAACEnc can't encode it and fails.

Audio handling wasn't changed at all. Oddly enough.

[FranceBB]

Uhm... Well, with the stable FFMpegSource2 I was able to open a file in avisynth, and pipe the audio using BePipe to NeroAACEnc, but I can't with ffms2000.
I think you can reproduce my error pretty easily with any source.

Sample:

FFMpegSource2("Ep1.avi", fpsnum=24000, fpsden=1001, atrack=-1)

BePipe.exe --script "Import(^AVS Script.avs^)" | neroAacEnc.exe -lc -br 320000 -if - -of "audio.m4a"

[FranceBB]

Hi,
I have an .mka audio track which contains two tracks: the first one is a FLAC lossless audio track with 6 channels, and the second one is an AC3 stereo audio track.
When I use FFAudioSource() it takes the second track (AC3) instead of the very first one.
I think it should use the first available audio track, so FLAC.
Is this a bug? Do you want me to upload the .mka file?


General
Unique ID : 174555563715229058114849233714037036678 (0x83522DC00F66393A809D00CAF83E9686)
Complete name : O:\episodi\[VCB-Studio] Shingeki no Bahamut Genesis [Ma10p_1080p]\bahamutep2audiodolby.mka
Format : Matroska
Format version : Version 4 / Version 2
File size : 664 MiB
Duration : 23 min 52 s
Overall bit rate mode : Variable
Overall bit rate : 3 890 kb/s
Encoded date : UTC 2016-11-12 14:21:14
Writing application : mkvmerge v9.5.0 ('Quiet Fire') 64bit
Writing library : libebml v1.3.4 + libmatroska v1.4.5

Audio #1
ID : 1
Format : FLAC
Format/Info : Free Lossless Audio Codec
Codec ID : A_FLAC
Duration : 23 min 52 s
Bit rate mode : Variable
Bit rate : 3 249 kb/s
Channel(s) : 6 channels
Channel positions : Front: L C R, Side: L R, LFE
Sampling rate : 48.0 kHz
Frame rate : 11.719 FPS (4096 spf)
Bit depth : 24 bits
Stream size : 555 MiB (84%)
Writing library : libFLAC 1.2.1 (UTC 2007-09-17)
Language : Japanese
Default : Yes
Forced : No

Audio #2
ID : 2
Format : AC-3
Format/Info : Audio Coding 3
Format settings, Endianness : Big
Codec ID : A_AC3
Duration : 23 min 52 s
Bit rate mode : Constant
Bit rate : 640 kb/s
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 kHz
Frame rate : 31.250 FPS (1536 spf)
Bit depth : 16 bits
Compression mode : Lossy
Stream size : 109 MiB (16%)
Language : Japanese
Service kind : Complete Main
Default : No
Forced : No

[Myrsloik]

Are both tracks manually selectable?

[FranceBB]

Yes, sure. I can select them using potplayer. If I just open the file, it plays the FLAC track as default.

[Myrsloik]

Released test6. Download link in first post.

[Mystery Keeper]

Test6 finally opens VP9 properly. Thank you!

[stax76]

@Myrsloik

I believe there is a memory leak with the x64 build, I've not examined it in detail, in staxrip for every job memory grew 50-100 MB, I always thought it's staxrip and I've been too careless with resources, forgetting to detach event handlers and things, I changed my design and had major problems with it, by accident I noticed that it don't happen with l-smash, I've tried with VirtualDub, after every opening and closing circle memory grows 10 MB, for GUIs this is very problematic because they don't keep one script open but rather work with multiple scripts opening and closing them on demand. I've currently an issue on the tracker by somebody that is trying to batch process 3200 files in one go. :-)

[manolito]

That's what you get from including experimental versions of helper applications in a stable version of your own software... :devil:

[stax76]

I knew it was coming. ;)