Windows pipe weirdness

[LoRd_MuldeR]

So, I have a command-line program that generates a stream of pseudo-random bytes (using simple "xorshift" method) and writes them to the STDOUT.

Now I wanted to measure the throughput (output rate) of the generator. On Linux, I would have used pv (https://linux.die.net/man/1/pv) for that purpose, but I was on Windows and couldn't find a native Win32 port of pv.

So, I ran the following command in an MSYS2 (https://www.msys2.org/) shell, where pv can be installed from the package manager:
./generator.exe | pv -br > /dev/null

This resulted in a measured throughput of about ~1.2 GiB/s.

Then I though I could write a "native" Win32 pv-like program with a few lines of code. So I did exactly that (https://github.com/lordmulder/pipe-utils/releases), and ran it from the Windows command prompt:
generator.exe | native_pv.exe > NUL

This time, however, I got a throughput of only ~500 MiB/s :eek:

Interesting observation: The "native_pv.exe" process was constantly using 12.5% of the CPU (on a Quadcore with HT that is), whereas the "generator.exe" process was idling at ~5% CPU usage.

Normally "generator.exe" uses 12.5% of the CPU, i.e. the maximum a single-threaded process can use.

But what's really bugging me is that, if I run the following from within the MSYS2 shell, then I suddenly get the full ~1.2 GiB/s again:
./generator.exe | ./native_pv.exe > /dev/null

This means that, apparently, not my "native_pv.exe" program sucks, but the pipe behaves quite differently in the MSYS2 shell than it does in the native Windows command prompt :confused:

Any ideas what is going on here?

(I would have expected that, if at all, the MSYS2 shell should be slower than the native command prompt, because it adds another emulation layer, i.e. msys-2.0.dll, on top of the Win32 API)

[nevcairiel]

Did you open the pipe in binary mode, to avoid any crazy overhead from Windows processing the stream in text mode?

[LoRd_MuldeR]

Did you open the pipe in binary mode, to avoid any crazy overhead from Windows processing the stream in text mode?

It's actually native Win32 code, so I'm using ReadFile() and WriteFile() directly, thus bypassing the C library (and any text/binary conversion) completely.

Anyway, today I create a little helper application (https://github.com/lordmulder/pipe-utils/releases) that launches two sub-processes and connects them via pipe. Because, this way, the creation of the pipe is under my control, I can choose the buffer size.

Turns out that this command, in the Windows command prompt, results in ~1.1 GiB/s:
mkpipe.exe generator.exe "|" native_pv.exe > NUL

(Pipe buffer size was set to 1 MiB)