Fastpipe: Utility to boost the performance piping video frames

[leiming2006]

Hello, when I noticed that the default buffer size of windows' anonymous pipe is not large enough for pipe high resolution video frames, I make a utility to enlarge it and get a positive result.

It also injects a DLL into target processes, to enlarge the buffer used by fread and fwrite.

Following is my test:

Without my utility

D:\tools\ffmpeg\ffmpeg-n9.0\bin>ffmpeg -i "C:\Users\SoraYuki\Videos\2025-09-24 00-10-28.mp4" -t 600 -f rawvideo -y - | ffmpeg -f rawvideo -pix_fmt yuv420p10le -s 1920x1200 -i - -f rawvideo -y nul
(...)
frame=18000 fps=284 q=-0.0 Lsize=121500000KiB time=00:12:00.00 bitrate=1382400.0kbits/s speed=11.4x elapsed=0:01:03.42


With my utility

D:\tools\ffmpeg\ffmpeg-n9.0\bin>fp ffmpeg -i "C:\Users\SoraYuki\Videos\2025-09-24 00-10-28.mp4" -t 600 -f rawvideo -y - "|" ffmpeg -f rawvideo -pix_fmt yuv420p10le -s 1920x1200 -i - -f rawvideo -y nul
(...)
frame=18000 fps=417 q=-0.0 Lsize=121500000KiB time=00:12:00.00 bitrate=1382400.0kbits/s speed=16.7x elapsed=0:00:43.14


Note that I don't have an arm64 PC, so I didn't test if it works with arm64 tools.

Usage

fp <CommandLine1> "|" <CommandLine2> "|" <CommandLine3> ...



Download page:
https://github.com/sorayuki/Fastpipe/releases/latest

Source code:
https://github.com/sorayuki/Fastpipe

[Z2697]

I think the main reason is FFmpeg writing output in 32 KiB chunks.
It's been like this for 25 years :eek:
https://github.com/FFmpeg/FFmpeg/blob/f1a000d41f3b5de2c305b5abb6e2718d070fedcc/libavformat/aviobuf.c#L36
https://github.com/FFmpeg/FFmpeg/blob/ef52e1cc3850846987edc792c9583103e977e3b2/libavformat/avio.c#L40

My coding skill lead me nowhere by reading the code, so just FAFO, for piping rawvideo (from and to ffmpeg) the one in avio.c affects writing and avuibuf.c one affects reading.
By modifying these macros the same or better speed as your utility can be achieved. (and somehow y4m/yuv4mpegpipe is even faster)
I tried increasing both to, 0.5, 1, 2, 4, 8, and 16 MiB, 4 MiB gives best result on my machine.
But this is of course just FFmpeg, if it's impractical to modify every program, this utility is a nice thing to have.
And FFmpeg being a massive codebase, I'm not sure if some side effect might present or not, for such a "low level" modification.

[GeoffreyA]

FFmpeg has a massive regression-testing suite. When one submits a PR, it runs the entire suite before being reviewed. It's likely that an issue will be picked up.

[leiming2006]

This is intended as a general-purpose utility, not something FFmpeg-specific.

reduces the number of times a thread switches between running, blocking and ready states, which involves costly user/kernel mode transitions. When the buffer is empty or full, OS turns the thread who write/read the pipe into blocked state which introduces cpu cost. A process cannot easily enlarge its own stdin pipe buffer (the OS-level one) after it has already been created.

And for the setvbuf, it's motivated by google's av1 encoder aomenc. The tool has a very limited stdin (c runtime layer) buffer. I tried submitting a patch but it failed to pass their CI due to a license signature issue, even after I sent the signed license.

[Z2697]

What I was trying to say is that if you move the default 32k buffer of FFmpeg out of the equation the speed gap is not that big.
Not to dismiss the use of this utility.

[leiming2006]

I tested the scene that pipe video frames into latest aomenc release just now, resulted in no performance boost.

I don't know if there is something changed in aom, ffmpeg or vc runtime / Windows kernel.

I do remember when I increasing the stdin buffer size by setvbuf in previous half year, the performance boost a lot...


D:\tools\ffmpeg\ffmpeg-n9.0\bin>fp ffmpeg -i "C:\Users\SoraYuki\Videos\2025-09-24 00-10-28.mp4" -t 30 -pix_fmt yuv420p10le -f rawvideo - "|" aomenc --transfer-characteristics=smpte2084 --color-primaries=bt2020 --matrix-coefficients=bt2020ncl -w 1920 -h 1200 --fps=30/1 --target-bitrate=4000 -t 24 -u 1 --cpu-used=12 -b 10 --input-bit-depth=10 --fp-mt=1 --webm -o av1.webm -
Pass 1/1 frame 900/900 15007282B 133398b/f 4001940b/s 10191 ms (88.31 fps)

D:\tools\ffmpeg\ffmpeg-n9.0\bin>ffmpeg -i "C:\Users\SoraYuki\Videos\2025-09-24 00-10-28.mp4" -t 30 -pix_fmt yuv420p10le -f rawvideo - | aomenc --transfer-characteristics=smpte2084 --color-primaries=bt2020 --matrix-coefficients=bt2020ncl -w 1920 -h 1200 --fps=30/1 --target-bitrate=4000 -t 24 -u 1 --cpu-used=12 -b 10 --input-bit-depth=10 --fp-mt=1 --webm -o av1.webm -
Pass 1/1 frame 900/900 15008140B 133405b/f 4002150b/s 10095 ms (89.15 fps)

[Z2697]

aomenc or any encoder that's not absolute-speed-first is just not fast enough to reach the bottleneck.
But if you change the input to a blank video to save the analysis time, it might work better.