FFmpegSource

[Myrsloik]

Quick question - awhile back someone was regularly making builds with ffmpeg-mt. Do the standard ffms2 builds come with ffmpeg-mt? If not, is someone making builds with this?

Also, is there any reason to NOT use ffmpeg-mt? The speedup for H.264 decoding (among other things) is quite spectacular, as we all know :)

~MiSfit

There are in some rare cases bugs only in ffmpeg-mt (very rare, actually) but the biggest reason is that you need to link it to pthreads and that's just no fun on windows. So get compiling and contribute a build. There are no known issues with it.

[Blue_MiSfit]

Indeed. I should get things up and running, but compiling software is one of my least favorite activities, especially using cygwin etc.... :(

If I get adventurous I might give it a whirl...

~MiSfit

[TheRyuu]

Indeed. I should get things up and running, but compiling software is one of my least favorite activities, especially using cygwin etc.... :(

If I get adventurous I might give it a whirl...

~MiSfit

I'm still alive. I'll have a build soon.

Pthread linking has become easy on windows now thanks to Ramiro with his auto-initialization (http://www.speedyshare.com/files/21117693/pthreads-autostatic-msc_and_gcc-mingw32-2.tar.bz2). Furthermore, because of this we can now remove unreferenced data in msvc as well (save <0.5MB yay!).
I guess I can commit that simple build change since the only reason that was put in was because of the pthreads stuff on windows.

[kemuri-_9]

I'm still alive. I'll have a build soon.

Pthread linking has become easy on windows now thanks to Ramiro with his auto-initialization (http://www.speedyshare.com/files/21117693/pthreads-autostatic-msc_and_gcc-mingw32-2.tar.bz2). Furthermore, because of this we can now remove unreferenced data in msvc as well (save <0.5MB yay!).
I guess I can commit that simple build change since the only reason that was put in was because of the pthreads stuff on windows.

ramiro's patch is only specifically for mingw builds of pthread that stay within the realm of mingw...
from conversations about this with a few people,
this patch uses specialized gcc function attributes that fail to work properly when msvc gets involved into the situation.

[TheRyuu]

ramiro's patch is only specifically for mingw builds of pthread that stay within the realm of mingw...
from conversations about this with a few people,
this patch uses specialized gcc function attributes that fail to work properly when msvc gets involved into the situation.

And following a month long back and fourth e-mail conversation he fixed the msvc part :) (tested it and it works).

[kemuri-_9]

And following a month long back and fourth e-mail conversation he fixed the msvc part :) (tested it and it works).

i don't recall ever seeing a new patch for this, where is it at?

[TheRyuu]

i don't recall ever seeing a new patch for this, where is it at?

Here you go. (http://paste2.org/p/688870)

Guess it never left my inbox. :p
Patch is off of pthreads svn for the date in there.

As far as (stupid) license stuff is concerned, perhaps I should make a note of it, AFAIK, it has some LGPL code from libavutil/log.c and the rest is GPL'd I think.

[TheRyuu]

ffms2-r292.7z (http://ffmpegsource.googlecode.com/files/ffms2-r292.7z)

Includes both vanilla ffmpeg and ffmpeg-mt, both patched with windows utf patch (and NOT the pthreads patch!)

Has the auto-initialization pthreads so give it a try. :)
Probably only going to notice the threading on the ffmpeg-mt build but both are built with pthreads (and both should work!).

[Atak_Snajpera]

MT version crashes
http://img46.imageshack.us/img46/4051/new1q.png

source is interlaced 50i (Canon hf200)

Script
video=FFVideoSource("C:\pal.m2ts",cachefile="C:\temp\index.ffindex")
return video


normal version works ok.

[TheRyuu]

normal version works ok.

The normal version in the r292 package? or normal as in v2.13?
Sample?

[Atak_Snajpera]

The normal version in the r292 package?
Yes. Your single threaded version. Only MT crashes immediately at the beginning

sample http://www.mediafire.com/file/z3qydwkn0it/pal.m2ts

[TheRyuu]

Yes. Your single threaded version. Only MT crashes immediately at the beginning

sample http://www.mediafire.com/file/z3qydwkn0it/pal.m2ts

Well I'm pretty sure it's not related to the auto-initialization pthreads, the old method pthreads also crashes.

May be a problem in ffmpeg-mt, works with threads=1.

[Mr VacBob]

Do you have a backtrace?

[rack04]

I'm still alive. I'll have a build soon.

Pthread linking has become easy on windows now thanks to Ramiro with his auto-initialization (http://www.speedyshare.com/files/21117693/pthreads-autostatic-msc_and_gcc-mingw32-2.tar.bz2). Furthermore, because of this we can now remove unreferenced data in msvc as well (save <0.5MB yay!).
I guess I can commit that simple build change since the only reason that was put in was because of the pthreads stuff on windows.

Can pthreads be statically linked to 64-bit ffmpeg with mingw32?

[TheRyuu]

Can pthreads be statically linked to 64-bit ffmpeg with mingw32?

I think you'd have to build pthreads with that patch on mingw64 and then build ffmpeg on mingw64 as well linked against that 64bit build of pthreads.

[TheFluff]

@TheFluff, about this bug:



Removing the index file between runs helped for a few files, but not for all. I could reproduce it with the same file everytime. To pin things down I first simplified the script. A simple call was enough to trigger the error.
file="C:\TEMP\Bas.avi"
A = FFAudioSource(file)
V = FFVideoSource(file)
AudioDubEx(V, A)

Then I started to simplify the batch script that I use. Small differences could make that the error didn't occur. Wierd... Then I generated a very small AVI-file with the same name I used for testing. It generated the error. But... renaming the Avisynth file was enough to make that the error didn't occur.

So the batch environment used had the biggest influence. Every script did open normally in Virtualdub. You can find the files used at: http://www.roelofs-coaching.nl/downloads/Bas.zip

I think I may have fixed this, try this build: http://www.mod16.org/ffms2/ffms2-r309.7z
(probably forgot UTF8 support left on, so don't try to open any files with funny names with it)

I didn't actually test your stuff by the way; dark shikari was yelling at me about some segfault some of his gsoc students were having and I realized this bug was most likely related to that.

[jmartinr]

I think I may have fixed this, try this build: http://www.mod16.org/ffms2/ffms2-r309.7z
(probably forgot UTF8 support left on, so don't try to open any files with funny names with it).
Thanks, will test tomorrow.

[jmartinr]

I didn't actually test your stuff by the way; dark shikari was yelling at me about some segfault some of his gsoc students were having and I realized this bug was most likely related to that.

Tested 8 small encodes without problems, so it seems OK now. :thanks:

[rack04]

Is it possible to cross compile ffmpegsource using MinGW in MSYS environment under WIN32? I have no problems cross compiling ffmpegsource under WIN64 but I get the following error in WIN32:

configure:3347: error: in `/c/x264/ffmpegsource-src':
configure:3351: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.


Here are the commands that I use to cross compile ffmpegsource:

./configure --prefix=/c/ffms2/x86_64-pc-mingw32 --build=x86_64-pc-mingw32 --host=x86_64-pc-mingw32

[kemuri-_9]

try specifying only --host=x86_64-pc-mingw32 instead of both --host and --build.

[Great Dragon]

What is output colorimetry by default? BT.709 or BT.601?

[Great Dragon]

Stephen R. Savage, sorry my bad. So, FFMPEG uses BT.601 for convert and HD video colors isn't correct, right? What I need to add to script to see colors like in DGIndexNV or MPC Home-Cinema in HD?

[kemuri-_9]

RGB/YUV conversions can occur, but that's source and user dependent, e.g.

- user selects to output RGB from a YV24 source
- user selects to output YV12 from a RGB32 source

when this occurs it uses the libswscale default,
which seems to be Rec./BT.601

[Groucho2004]

I just noticed that the content of the index file created by ffmsindex.exe is completely different to the index file that gets created automatically when I just run an Avisynth script with FFVideosource.

Can one of the developers shed some light on this?

I'm using ffms2 r292 from http://code.google.com/p/ffmpegsource

Edit:
However, the generated files (AVS -> uncompressed AVI) are bit-identical.

[Myrsloik]

Did you make sure to index the same tracks?

[Groucho2004]

Did you make sure to index the same tracks?

Yes I did. I have uploaded a sample and the 2 resulting index files here:
http://www.iol.ie/~schubert/Test.zip

"test_dll.m2v.ffindex" is the index file generated with the simple Avisynth script.
"test_ffmsindex.m2v.ffindex" is the index file generated with ffmsindex.exe.

[Myrsloik]

Can you explain why this is an issue? It sounds like everything works to me.

[Groucho2004]

Can you explain why this is an issue? It sounds like everything works to me.

It's not an issue. I was just curious why the index files are so different since I assumed that ffmsindex.exe uses the same functions in ffms2.dll.

[asarian]

I would like to use FFVideoSource to solve the following (still unsolved) problem of non-frame accurate encoding:

http://forum.doom9.org/showthread.php?t=147616

I tried things like:

FFVideoSource("D:\Temp\job1\video.mkv").ConvertToYV12().TemporalDegrain(degrain=3, ov=4, blksize=16)

But I keep getting this error:

avis [error]: unsupported input format (DIB )

The mkv is made with the latest mkvmerge, and contains just one simple 1080p elementary VC1 stream. And is being properly set to the ConvertToYV12() color space. It simply should work. But it doesn't.

I'm really still looking for a way to do a hefty (slow) TemporalDegrain, without frames getting out of sequence or being skipped. There's gotta be other people who do TemporalDegrain stuff and who get a clean encoding. I don't care if it takes 3x as long, as long as frames stay in sequence.

Thanks.

[kemuri-_9]

avis [error]: unsupported input format (DIB )

this particular error is that you're trying to give x264 RGB video which is unsupported in the old VFW interface.
this error is also caused when there is a script error as avisynth creates a RGB video with the error message overlayed onto it when used through VFW.

these issues are only generated by x264 builds that are still using VFW to interact with avisynth which has not been the way x264 does it for some time now. you should update your build and try again.
from the looks of your script, looks like you have an error in the script somewhere and so using a recent x264 build will tell you what the error is...
virtualdub could also tell you what the error is.

[asarian]

this particular error is that you're trying to give x264 RGB video which is unsupported in the old VFW interface.
this error is also caused when there is a script error as avisynth creates a RGB video with the error message overlayed onto it when used through VFW.

these issues are only generated by x264 builds that are still using VFW to interact with avisynth which has not been the way x264 does it for some time now. you should update your build and try again.
from the looks of your script, looks like you have an error in the script somewhere and so using a recent x264 build will tell you what the error is...
virtualdub could also tell you what the error is.

Thanks, Kemuri! This was precisely the sort of answer I was looking for. :)

Usually I work from a "Don't fix it if it ain't broken." adage, but in this case I guess the old x264 (1093) really needed updating after all.

Seems CoreAVC is borked with the new ffdshow and/or new Haali media splitter (odd artifacts, occasionally garbled output); but fortunately ffdshow itself handles the H264 decoding just fine. I'll let you know what the results are on the updated install.

[asarian]

Kemuri-san, doumo arigatou gozaimashita! :)

Yep, it's solved now. Temporal degraining is not exactly fast (0.82 fps on a VM core duo), but the output using FFVideoSource is perfectly frame consistent and no more skipping and jittery stuff.

[Blue_MiSfit]

Hi guys!

I'm seeing REALLY slow indexing speeds for my 80mbps intra-only 1080p MPEG-2 mezzanine files. Specifically, on the order of ~8-10MB per second. At this rate, indexing your average 2 hour feature basically runs real-time AT BEST.

I thought ffmsindex.exe was supposed to usually be I/O bound.

Here's a sample MediaInfo dump from one of these mezzanine files. Anything stand out as an obvious PITA?



General
Complete name : RussiansAreComing_80mbps_NTSC_235HD.mpg
Format : MPEG-PS
File size : 71.4 GiB
Duration : 2h 5mn
Overall bit rate : 81.3 Mbps

Video
ID : 224 (0xE0)
Format : MPEG Video
Format version : Version 2
Format profile : High@High
Format settings, BVOP : No
Format settings, Matrix : Default
Format settings, GOP : N=1
Duration : 2h 5mn
Bit rate mode : Constant
Bit rate : 80.0 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 23.976 fps
Color space : YUV
Chroma subsampling : 4:2:2
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 1.609
Stream size : 69.6 GiB (98%)

Audio
ID : 192 (0xC0)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Duration : 2h 5mn
Bit rate mode : Constant
Bit rate : 384 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Stream size : 345 MiB (0%)


Here's another offender, this time a TS at 50mbps, with AES3 PCM audio, and hard telecine :devil:!!!


General
ID : 0
Complete name : OxfordMurders_60i_50mbps_NTSC_235HD.mpg
Format : MPEG-TS
File size : 43.3 GiB
Duration : 1h 48mn
Overall bit rate : 57.1 Mbps

Video
ID : 481 (0x1E1)
Menu ID : 1 (0x1)
Format : MPEG Video
Format version : Version 2
Format profile : High@High
Format settings, BVOP : No
Format settings, Matrix : Default
Format settings, GOP : N=1
Duration : 1h 48mn
Bit rate mode : Constant
Bit rate : 50.0 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 29.970 fps
Color space : YUV
Chroma subsampling : 4:2:2
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Top Field First
Bits/(Pixel*Frame) : 0.805
Stream size : 36.7 GiB (85%)

Audio
ID : 482 (0x1E2)
Menu ID : 1 (0x1)
Format : PCM
Format profile : AES3
Duration : 1h 48mn
Bit rate : 5 760 Kbps
Channel(s) : 6 channels
Channel positions : Front: L C R, Side: L R, LFE
Sampling rate : 48.0 KHz
Bit depth : 16 bits
Stream size : 4.36 GiB (10%)


Any thoughts? DGIndex seems to blow ffmsindex.exe out of the water in these cases! I'm so confused as to why :(

Thanks folks,
Derek

PS - DGIndex can't extract the AES3 audio from files like these into a WAV. For this, I have to use eac3to, which does it flawlessly - and gives me free AC3 encoding while I'm at it! Of course, this means I have to basically read the entire file TWICE, before I can even start encoding it. This makes me a saaaad panda :(

[dansrfe]

Doesn't dss2 work best? Just wondering. Also, I must say that I am very impressed with those files. Love those bitrates but I think I'm already starting to get nightmares of the sizes lol.

[Blue_MiSfit]

Baaah those sizes are tiny :) ProRes HQ is ~220mbps for the same frame size and rate!!

I'm working on getting every setting dialed in for x264 capture instead of MPEG-2. At 50-80mbps (ok CRF so it's gonna vary a lot) it should be close to ProRes quality.

DSS2 does decode the fastest, but unfortunately (with the muxes I get anyway) it consistently returns a different number of frames than DGIndex. It's usually ~5 frames off, which can obviously lead to a substantial audio desync!

Regardless, my indexing issue is pretty nasty. Any thoughts on why FFMSIndex can't run these files through at closer to I/O bound speeds? This was on a fresh 2TB, 7200rpm hard drive with almost nothing else, so it wasn't a storage issue...

Thanks,

Derek

[SledgeHammer_999]

A programming question. I am not sure if this is the correct place.

Is there any way to use the FFMS API and get the SAME (fake)framerate that ffms reports to avisynth? If yes, please point me to the correct function, since a quick reading of the docs didn't yield a definitive answer. Thank you.

[Groucho2004]

Any thoughts on why FFMSIndex can't run these files through at closer to I/O bound speeds?

The problem might be the introduction of compressed index files a few months ago. I seem to remember that the indexing was faster before. I never had a problem with the size of the index files, even uncompressed.

[kemuri-_9]

A programming question. I am not sure if this is the correct place.

Is there any way to use the FFMS API and get the SAME (fake)framerate that ffms reports to avisynth? If yes, please point me to the correct function, since a quick reading of the docs didn't yield a definitive answer. Thank you.

the framerate is corrected to NTSC rationals for any and all sources that are opened through the API.
looking at FFMS_VideoProperties from a FFMS_VideoSource will have the corrected FPSNumerator and FPSDenominator values - the same ones that avisynth gets.

nothing usually cares about this correction except avisynth since it is cfr and can't use the vfr timebase & timestamps.

[SledgeHammer_999]

Thank you. The corrected framerate is useful when you want to "transform" the sub timings through aegisub. (when opening a vfr vid and hardsubbing through avisynth).

[SledgeHammer_999]

I have a question about the indexing. Shouldn't the ffindex file be the same regardless of under which OS it was created?

I compiled svn-ffms2 + git-ffmpeg-mt under Ubuntu 64bit and used ffmsindex to index a file. Then I booted in Windows XP (32bit) and compiled the same sources. I created a simple avs file using FFVideoSource(). But when I open the avs in mpc the ffindex gets recreated instead of using the one I created earlier. Is this normal? Can I avoid it? (I want to batch index(among other things) my files under linux and then switch to windows for the encoding part).

[kemuri-_9]

I have a question about the indexing. Shouldn't the ffindex file be the same regardless of under which OS it was created?

I compiled svn-ffms2 + git-ffmpeg-mt under Ubuntu 64bit and used ffmsindex to index a file. Then I booted in Windows XP (32bit) and compiled the same sources. I created a simple avs file using FFVideoSource(). But when I open the avs in mpc the ffindex gets recreated instead of using the one I created earlier. Is this normal? Can I avoid it? (I want to batch index(among other things) my files under linux and then switch to windows for the encoding part).

since you're using the same source of ffms2/ffmpeg, that excludes the possibility of incompatible versions.

but i see some use of size_t in the the index file data, so it appears index files from 64bit and 32bit platforms are quite possibly incompatible with each other.
see if you can replicate the re-indexing on your windows system if you use a 32bit version of ffmsindex on your ubuntu one.

[TheFluff]

I have a question about the indexing. Shouldn't the ffindex file be the same regardless of under which OS it was created?

I compiled svn-ffms2 + git-ffmpeg-mt under Ubuntu 64bit and used ffmsindex to index a file. Then I booted in Windows XP (32bit) and compiled the same sources. I created a simple avs file using FFVideoSource(). But when I open the avs in mpc the ffindex gets recreated instead of using the one I created earlier. Is this normal? Can I avoid it? (I want to batch index(among other things) my files under linux and then switch to windows for the encoding part).

The indexing "file format" is basically constructed by fwriting various internal data structures to disk, so it's extremely unportable not only between different operating systems, but also between different compilers, different compiler versions and other stuff. There is really no guarantee a given index file will work with any other binary except the one that you created it with. Someone should really fix that one of these days...

[SledgeHammer_999]

The indexing "file format" is basically constructed by fwriting various internal data structures to disk, so it's extremely unportable not only between different operating systems, but also between different compilers, different compiler versions and other stuff. There is really no guarantee a given index file will work with any other binary except the one that you created it with. Someone should really fix that one of these days...

That explains a lot...

I compiled a 32-bit version of ffms2 in Ubuntu and used it under the 64-bit. Same results as previously.

I will create a bug report upstream so someone may fix this.

[Snowknight26]

Is there any way to get dubugging output during normal usage? I'm battling ffms2 crashing in one of my avisynth scripts but it's hard to do with no info other than Windows saying ffms2.dll is the culprit.

[LoRd_MuldeR]

Is there any way to get dubugging output during normal usage? I'm battling ffms2 crashing in one of my avisynth scripts but it's hard to do with no info other than Windows saying ffms2.dll is the culprit.

Did you try using a debugger?

(The output from a debugger may be of limited use when using a non-debug build though)

[kemuri-_9]

Is there any way to get dubugging output during normal usage? I'm battling ffms2 crashing in one of my avisynth scripts but it's hard to do with no info other than Windows saying ffms2.dll is the culprit.

There is FFSetLogLevel(int level) for use in avs scripts...
but the normal 'release' builds of ffms2 don't properly set the logging callback from the standard one so you won't see any messages, you'll need to get a debug version that outputs messages to DebugView (http://technet.microsoft.com/en-us/sysinternals/bb896647.aspx)

the logging feature is naturally ffmpeg's so the log levels are as follows:

#define AV_LOG_QUIET -8
#define AV_LOG_PANIC 0
#define AV_LOG_FATAL 8
#define AV_LOG_ERROR 16
#define AV_LOG_WARNING 24
#define AV_LOG_INFO 32
#define AV_LOG_VERBOSE 40
#define AV_LOG_DEBUG 48

[Snowknight26]

Thanks for the info kemuri-_9. Now to find a debug build.

[Snowknight26]

Seems like there's memory leak in ffms2.dll. Tried debugging it but all VS showed me for the call stack was
> ffms2.dll!_ff_mpeg4_find_frame_end() + 0x1408 bytes
ffms2.dll!_ff_mpeg4_find_frame_end() + 0x1621 bytes
002cee18()

I still have it paused in the debugger so if there's anything else I can provide, by all means..

As far as I can tell, it doesn't like the M2TS source in my avs (http://pastebin.com/KcFrV9h3).

[kemuri-_9]

ff_mpeg4_find_frame_end is a ffmpeg function.
to my knowledge ffmpeg is known to have issues with ts/m2ts

[Blue_MiSfit]

@kemuri-_9:

Any thoughts on slllooooowww indexing performance?

Derek