Freaky question of the decade: frame-serving from ffmpeg to Avisynth

[fvisagie]

Is there a way of frame-serving ffmpeg output to Avisynth?

I foresee doing some processing in ffmpeg, and then continuing in Avisynth. This project is huge and where possible I'd like to avoid intermediate files. Is Avisynth's TCPSource() compatible with any of the ffmpeg output protocols, and if so, in this case what ffmpeg command line would be needed for that protocol, especially '-f format'?

Thanks,
François

[raffriff42]

After a few tries, I got ffmpeg to stream to VLC Player:

ffmpeg -i %file% -c:v mpeg2video -q:v 1 -qmin 1 -pix_fmt yuv420p -c:a ac3 -q:a 0.35 -f mpegts "tcp://127.0.0.1:2000?listen"

(EDIT Regarding your format question, -f mpegts seems to be mandatory here)

But TCPSource...

TCPSource("tcp://127.0.0.1", port=2000, compression="none")

...always failed with "Access Violation." I tried deleting the compression argument, but it made no difference.
(EDIT I just realized compression here refers to over-the-network compression, not video compression.)

Checking the source code, I have to conclude that TCPSource is designed to work with TCPServer only - they are a matched pair.

  SendRequest(CLIENT_CHECK_VERSION, &ccv, sizeof(ccv));
  GetReply();

  if (reply->last_reply_type != REQUEST_CONNECTIONACCEPTED) {
    env->ThrowError("TCPSource: Version Check failed! (Ensure Client and Server are same version)");
  }

EDIT I can't even get the matched pair to work. "Access Violation."
Now I'm wondering if it's a Windows 8.1 compatibility issue.
EDIT about that...scroll down to post #19.


...If you want to do ffmpeg processing of some kind (what does it do that Avisynth filters can't?) with frameserving, you need some kind of ^DirectShow wrapper for ffmpeg.^ Fortunately a very good one exists, namely, ffdshow tryouts.

[fvisagie]

Thanks for your trouble, much appreciated.

Originally Posted by raffriff42

EDIT I can't even get the matched pair to work. "Access Violation."
Now I'm wondering if it's a Windows 8.1 compatibility issue.

On my Windows 7 SP1 machine TCPServer() and TCPSource() work together correctly. IIRC, initially I had an issue with the firewall popping up an authorisation request for the host application, so that might play a role in your case too.

Regrettably TCPSource()'s success did not extend to working with this particular ffmpeg command. I got the default unacceptable response:

TCPSource: Version Check failed! (Ensure Client and Server are same version)

...If you want to do ffmpeg processing of some kind (what does it do that Avisynth filters can't?)

Works with hundreds of sources without crashing, and produces audio ;-).

with frameserving, you need some kind of ^DirectShow wrapper for ffmpeg.^ Fortunately a very good one exists, namely, ffdshow tryouts.

I'm not sure DirectShow support is necessarily required, in the sense that the frame-serving host application interface would probably determine that (e.g. VirtualDub uses the VfW interface)? Or does ffdshow-tryouts provide its own frame-serving host? Its documentation is better than most but still sparse!

In any event, I'm looking into ffdshow-tryouts also for its FFV1 support.

[raffriff42]

Originally Posted by fvisagie

(e.g. VirtualDub uses the VfW interface)? Or does ffdshow-tryouts provide its own frame-serving host?

You can use VirtualDub's decoders and frameserving, with the ffdshow plugin for video effects:

(source) >> Vdub >> "ffvdub" plugin >> Vdub >> frameserve

...but it's easier to access ffdshow's decoders and effects in one step:

(source) >> DirectShowSource calling ffdshow >> Avisynth >> frameserve

You just need to enable the appropriate decoders in ffdshow options. If you have problems with this ("DirectShow video decoders are not required to support frame-accurate seeking"), you can fall back to the VirtualDub method.

EDIT I was thinking in terms of video effects filters. I wasn't thinking of audio processing at all, so this might not help you.

[fvisagie]

Originally Posted by raffriff42

You can use VirtualDub's decoders and frameserving

Thanks for explaining that. I was hoping that I could frame-serve to Avisynth from ffmpeg instead, because the latter can concatenate directly from non-AVI files without requiring intermediate conversion (and storage). However, reality seems to dictate otherwise...

[fvisagie]

Originally Posted by fvisagie

Regrettably TCPSource()'s success did not extend to working with this particular ffmpeg command.

On the premise that Avisynth would output via TCPServer() and input via TCPSource() data in similar format than it produces via frame-serving, I noted its video and audio formats when serving to ffmpeg and specified those codecs for the most likely-looking ffmpeg protocol, tcp.

Then came the issue of which format to specify for that protocol. ffmpeg reports Avisynth's input format as 'avisynth', which the former does not support as an output format. I tried a couple of likely-looking candidates, mostly the 'raw' ones. With some, e.g. 'rawvideo' or 'dnxhd':

ffmpeg -y -i 00000.MTS -c:v rawvideo -c:a pcm_s16le -f dnxhd tcp://127.0.0.1:22050?listen

ffmpeg and Avisynth would connect and ffmpeg would in fact start serving. After a short while, however, Avisynth would crash out and ffmpeg would terminate with

av_interleaved_write_frame(): Unknown error

This seems to indicate at least format incompatibility, and perhaps also initial connection set-up as raffriff42's source extract above would indicate.

If only I had the time, I would get into coding and adapt TCPSource() to FFTCPSource() (or whichever ffmpeg protocol most suitable in terms of seeking etc.)...

[Wilbert]

What happens if you frameserve a video only clip? Does that work?

[fvisagie]

Nothing exciting to note. Without audio, Avisynth seems to exit with the "TCPSource: Version Check failed! (Ensure Client and Server are same version)" message more often instead of crashing, compared to before. And for what it's worth, with two protocols Avisynth exited with the error message without aborting, AND the ffmpeg server continued running now. The protocols are 'mpegts' and 'rtsp'. With audio the ffmpeg server would exit with an error condition each time.

[raffriff42]

I've been Googling for alternatives and gotten nowhere...

MPC-HC Frameserver [cccp-project.net, 2013
http://www.cccp-project.net/forums/i...p?topic=6853.0

VLC backend server questions [2012]
http://forum.doom9.org/showthread.php?t=166460

Pipe data to VLC [videolan.org, 2010]
https://wiki.videolan.org/Uncommon_uses/

Mencoder as realtime frameserver [MPlayer-users, 2009]
https://lists.mplayerhq.hu/pipermail...er/078407.html

IDEA: A new universal way for frameserving [virtual drive emulation, 2006]
http://forum.doom9.org/showthread.php?t=108653

Originally Posted by fvisagie

This project is huge and where possible I'd like to avoid intermediate files.

I'd say it's very likely not possible. Just get a huge RAID array or whatever, and start working on your actual project, instead of this side project...

[fvisagie]

Originally Posted by raffriff42

Just get a huge RAID array or whatever, and start working on your actual project, instead of this side project...

Hehe, point taken, but in my defence, I put high value on some measure of preparation. If there's one thing I dislike more than working hard, it's working unnecessarily hard .

Thanks for your trouble, much appreciated.

[feisty2]

write a special source filter that reads raw videos directly from RAM

[jmartinr]

It would require some footwork, but you might try this: http://forum.doom9.org/showthread.php?t=170590

[Wilbert]

Originally Posted by jmartinr

It would require some footwork, but you might try this: http://forum.doom9.org/showthread.php?t=170590

Looks like great stuff. Did anyone play with this? Does it support audio too?

[poisondeathray]

Could you maybe modify the workflow so some of the ffmpeg processing is done in avisynth or vice-versa, or even another program ? What types of operations are being performed ?

[fvisagie]

Originally Posted by jmartinr

It would require some footwork, but you might try this: http://forum.doom9.org/showthread.php?t=170590

Wow! Well spotted, thanks.

[fvisagie]

Originally Posted by poisondeathray

Could you maybe modify the workflow so some of the ffmpeg processing is done in avisynth or vice-versa, or even another program ? What types of operations are being performed ?

Assuming your question refers to the first post, concatenation of M2TS input files at least, followed by transcoding to FFV1 (which seems to be the lossless codec most easily added to my existing workflow) for further improvement and editing in Avisynth. Transcoding to intermediates for lack of a known reliable way of frame-serving from ffmpeg to Avisynth, and FFV1 for ffmpeg's lack of Lagarith encoding.

However, following an analysis of M2TS and FFV1 support in the tools concerned it appears that that approach has issues of its own which require work-arounds that at least cancel the benefits. Therefore, for this project I'll stick with Lagarith intermediate AVIs and concatenating those in VirtualDub.

What I found when looking into M2TS and FFV1 support:

• ffmpeg has a bug with concatenating M2TS (H.264 in general it seems) files
• Avisynth intermediate files can overcome that, but at the expense of added complexity and processing
• On my system Lagarith decodes (read: editing use) around 3 times faster than intra-frame-only encoded FFV1. The newer version 3 FFV1 codec creates files that decode marginally faster than those created with the older version 1 codec.
• ffdshow-tryout's DirectShow, VfW and Virtualdbug plugin FFV1 interfaces all only read and write the FFV1 version 1 codec
• The only way (with these tools) to write FFV1 codec version 3 files is with recent ffmpeg versions, which have the buggy M2TS support above
• The only way for Avisynth and VirtualDub to read FFV1 codec version 3 files is with LWLibavVideoSource(), which has FFV1 issues of its own
• Lagarith files used in the above comparison are only ~10% larger than their FFV1 counterparts

[poisondeathray]

Originally Posted by fvisagie

Assuming your question refers to the first post, concatenation of M2TS input files at least, followed by transcoding to FFV1 (which seems to be the lossless codec most easily added to my existing workflow) for further improvement and editing in Avisynth. Transcoding to intermediates for lack of a known reliable way of frame-serving from ffmpeg to Avisynth, and FFV1 for ffmpeg's lack of Lagarith encoding.

However, following an analysis of M2TS and FFV1 support in the tools concerned it appears that that approach has issues of its own which require work-arounds that at least cancel the benefits. Therefore, for this project I'll stick with Lagarith intermediate AVIs and concatenating those in VirtualDub.

What I found when looking into M2TS and FFV1 support:

• ffmpeg has a bug with concatenating M2TS (H.264 in general it seems) files
• Avisynth intermediate files can overcome that, but at the expense of added complexity and processing
• On my system Lagarith decodes (read: editing use) around 3 times faster than intra-frame-only encoded FFV1. The newer version 3 FFV1 codec creates files that decode marginally faster than those created with the older version 1 codec.
• ffdshow-tryout's DirectShow, VfW and Virtualdbug plugin FFV1 interfaces all only read and write the FFV1 version 1 codec
• The only way (with these tools) to write FFV1 codec version 3 files is with recent ffmpeg versions, which have the buggy M2TS support above
• The only way for Avisynth and VirtualDub to read FFV1 codec version 3 files is with LWLibavVideoSource(), which has FFV1 issues of its own
• Lagarith files used in the above comparison are only ~10% larger than their FFV1 counterparts

If these are camcorder files, you run a higher risk of "glitches" , or sometimes AV sync , or crackle issues at the seams when appending individual file spanned clips when using avisynth, or ffmpeg or tsmuxer. The most reliable way is to copy the entire folder structure over, because there is accessory meta data that some programs use to seamlessly join the the clips. Usually the camera comes with crappy (but reliable) software to do this, or look at the manufacturer website. Most decent NLE's also do it that way with the entire folder structure - but they decode the files , not concatenate or stream copy

UT Video and x264 are other possibilities for lossless intermediates, that are also available in ffmpeg. You can adjust x264 parameters so it's has temporal compression like ffv1; you can tweak the parameters so it can decode faster or slower; higher or lower compression. UT usually is less compressed than lagarith, but decodes much faster. So if editing speed is a concern, it's probably the better choice. UT is also very compatible in NLE, x264 lossless isn't

Be aware Lagarith can have it's own issues for some people with random corrupt frames. A few threads discuss this at doom9. I've never personally had it happen to me

[fvisagie]

Thanks for that advice. Fortunately the clips as input to this workflow have already been joined by the camera's export software .

As potential back-ups should I run into issues with Lagarith (which I've been using for a long time) I've been keeping UT Video and H.264 in mind; I appreciate your inputs on those too.

[Wilbert]

Originally Posted by raffriff42

After a few tries, I got ffmpeg to stream to VLC Player: (EDIT Regarding your format question, -f mpegts seems to be mandatory here)

But TCPSource...

...always failed with "Access Violation." I tried deleting the compression argument, but it made no difference.
(EDIT I just realized compression here refers to over-the-network compression, not video compression.)

Checking the source code, I have to conclude that TCPSource is designed to work with TCPServer only - they are a matched pair.

  SendRequest(CLIENT_CHECK_VERSION, &ccv, sizeof(ccv));
  GetReply();

  if (reply->last_reply_type != REQUEST_CONNECTIONACCEPTED) {
    env->ThrowError("TCPSource: Version Check failed! (Ensure Client and Server are same version)");
  }

EDIT I can't even get the matched pair to work. "Access Violation."
Now I'm wondering if it's a Windows 8.1 compatibility issue.


...If you want to do ffmpeg processing of some kind (what does it do that Avisynth filters can't?) with frameserving, you need some kind of ^DirectShow wrapper for ffmpeg.^ Fortunately a very good one exists, namely, ffdshow tryouts.

For future reference. raffriff42 should have used (without the tcp:// in front of the ip number):

TCPSource("127.0.0.1", port=2000, compression="none")

[raffriff42]

Thanks Wilbert, my use of "tcp://" (copied from the ffmpeg line I guess) was the problem.