native ffmpeg avisynth filter plugin

[jordanh]

Hey, i just want to ask who who votes for a native ffmpeg-avisynth audio (and/or) video filter plugin.
Currently the ffms2 source plugin can be used for decoding and ffmpeg can load .avs files. The negative side is that the ffms2 plugin is just not as good as ffmpeg itself and we always need to write lots of avs code to make sure that all audios and the video is forwarded to ffmpeg.
It is a lot of work to develop such plugin, would it be worth the efforts? What could we really win besides making the process of using ffmpeg+avisynth co-work a few steps simpler?

I imagine the usage could look like that:
ffmpeg -i c:\sourcefile.mkv -vf avisynth="c:\rescale.avs" c:\output.mp4

and the avisynth script like that:

liveffmpegsource()
... rest of the script

What you think?
Cheers!
Harry

[FranceBB]

You basically mean using ffmpeg to decode the source, pipe the decoded uncompressed stream to Avisynth and then pipe Avisynth back to ffmpeg to encode?
I think that piping back and forth would be less efficient than using an indexer in Avisynth.

The negative side is that the ffms2 plugin is just not as good as ffmpeg itself and we always need to write lots of avs code to make sure that all audios and the video is forwarded to ffmpeg.
More than the negative side, it's the fact that Avisynth only supports a single audio stream with n channels in it, so if you have CH.1-2 Stereo German, CH.3-4 Stereo English, you need to index them both, merge them together, create a 4ch audio stream in Avisynth and then split them back using ffmpeg (or whatever encoder you are using).
I do it at work all the time and IT IS annoying, BUT I would actually prefer to see Avisynth with the possibility to have multiple audio streams and the possibility to handle that using an indexer, rather than piping back and forth using ffmpeg.

I.e

Indexing CH.1-2 Language 1, CH.3-4 Language 2 and merging them together to make a single audio stream:

video=FFVideoSource("example.mxf")
ch1=FFAudioSource("example.mxf", track=1)
ch2=FFAudioSource("example.mxf", track=2)
ch3=FFAudioSource("example.mxf", track=3)
ch4=FFAudioSource("example.mxf", track=4)
audio=MergeChannels(ch1, ch2, ch3, ch4)
AudioDub(video, audio)

Splitting the single audio stream with 4 channels back to mono channels to make an XDCAM:

avs2yuv.exe "W:\Production\05487\AVS Script.avs" -csp AUTO -o - | ffmpeg.exe -i - -pix_fmt yuv422p -vcodec mpeg2video -s 1920:1080 -aspect 16:9 -vf setfield=tff -flags +ildct+ilme -r 25 -b:v 50000k -minrate 50000k -maxrate 50000k -bufsize 36408333 -g 12 -bf 2 -profile:v 0 -level:v 2 -color_range 1 -color_primaries 1 -color_trc 1 -colorspace 1 -f mxf "\\VBOXSVR\Share_Windows_Linux\output_video.mxf"

ffmpeg.exe -i "\\VBOXSVR\Share_Windows_Linux\output_video.mxf" -i "W:\Production\05487\AVS Script.avs" -map 0:0 -c:0:0 copy -map 1:1 -map_channels 1.1.0:0.1.0 -map 1:1 -map_channels 1.1.1:0:2.0 -map 1:1 -map_channels 1.1.2:0.3.0 -map 1:1 -map_channels 1.1.3:0.4.0 -c:a pcm_s24le -ar 48000 -ac 1 -f mxf "\\VBOXSVR\Share_Windows_Linux\output_final.mxf"


This could actually be avoided by letting Avisynth handling more than 1 audio stream and - eventually - passing metadata along with the stream.

Same applies if you wanna encode them as stereo tracks to make an H.264 with AAC audio, for instance: you still have to merge CH.1-2 and CH.3-4 together in Avisynth and then split them into two stereo streams in ffmpeg (rather than mono streams), which is still annoying.

[sneaker_ger]

Yeah, passing on audio isn't really needed. You can just use multiple input and then map video from avs input and audio+subtitles+metadata from the mkv file. Still, I somehow like the idea.

[jordanh]

@FranceBB, that is exactly what i was referring to. But i guess as ffmpeg is more mighty than avisynth in filtering audio, a video filter plugin is the only thing of interest here....

@sneaker_ger, maybe you find the time to think about why you somehow like the idea ;-)

On the positive side for a video filter plugin, inspired by sneaker; we would not loose all other tracks than video and audio, am i wrong?

[18fps]

I'd love such a filter, mainly because still there isn't a simple way to read a dpx sequence in 16 bits in avisynth+

[sneaker_ger]

@sneaker_ger, maybe you find the time to think about why you somehow like the idea ;-)
ease of use because it's a simple command
video track metadata could be copied automatically (like track language)
maybe it could handle vfr (at least as long as the frame numbers don't change). IVTC and the like would be difficult. It could have an option to take constant fps from script or fps from source container.

But I'm not qualified to program such a thing.

[jordanh]

OK, so currently we have for a video filter plugin the following pros:

-) ease of use (not sure about that because the ffmpeg folks will probably not like me to devliver the avisynth dll itself compiled into ffmpeg) - you will need to have avisynth installed (alone in order to use the avisynth version of your choice)
-) audio tracks are handled by ffmpeg, one does not loose track layout etc...
-) hanles vfr
-) support much more input formats (image sequences etc...) than ffms 2 plugin does

That already sounds like a good pro list. @sneaker_ger: no worries about your qualification, i believe i am qualified, that is why i asked.

Still, on the negative side i see the following:
-) it is a lot of work, about 2 full weeks for the first basic version and one more until ffmpeg folks accept the code and allow merging
-) the user base would be extremely small. Maybe the time would be better spent on something more useful like a native opencv integration

Also, i would like to get somewhat more response, i dislike the idea to start this without having lots of input and requirements.

Any comment is welcome, thanks for anyone investing brains into this.

By the way:
You basically mean using ffmpeg to decode the source, pipe the decoded uncompressed stream to Avisynth and then pipe Avisynth back to ffmpeg to encode?
I think that piping back and forth would be less efficient than using an indexer in Avisynth.

No, i am not talking about piping, piping would require avisynth to run in a separate process paralell to ffmpeg so it would require an additional executeable. What i am talking about is a native plugin just like any other video filter in ffmpeg (e.g. ffmpeg links the avisynth dll). Sure this only works on windows as avisynth requires all the directshow and vfw stuff.

Actually the final product would consist of 2 parts: an additional filter in the libavfilter source code (this will probably require a special config switch when building ffmpeg e.g. --enable-native-avisynth-filter - zeranoe would need to enable this in their builds) plus an avisynth source plugin dll which you need to load in your avs script load like any other non native avisynth plugin.