Fast C/C++ file I/O routines?

[jeanl]

Hi guys,
I'm writing a small app on windows which requires shuffling around large VTS files (1GB), from 1 disk to another 1.
I"ve been using fopen(), fread(), fwrite(), to read chunks, then write them back to the output file. I've noticed that performance depends a lot on the chunk size, hence my question:
- For the simple task for copying 1 file from C: to E: doing reads and writes, how should I choose the chunk size (the size of each read() write() operation) to maximize throughput? (informal test show I can use 4MB chunks, but that might depend on the amount of system RAM?).
- Are there other (lower level) functions that fopen, fread, fwrite that might be faster to do that kind of things?

Of course, I want to do more than just copy (otherwise I would use a dos command), but I'm trying to find the optimum setting to minimize processing time.

Any pointers?
Thanks!
Jeanl

[dimzon]

the BEST method for Win32 Applications:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/file_mapping.asp

[jeanl]

dimzon,
Did you actually make tests? I have tried file mapping before, but didn't find that it was much faster, although I concur that it makes the coding much easier...
Jeanl

[dimzon]

dimzon,
Did you actually make tests? I have tried file mapping before, but didn't find that it was much faster, although I concur that it makes the coding much easier...
Jeanl
Keep optimal buffer size for OS! Just create 2 file mapping (source and target) and use memcpy() for file copy :)

[jeanl]

OK, I'll do some more tests then! Thanks!
Jeanl

[GZZ]

what you should do jeanl is to make a buffer. Because you read and write speed will be depending on HD read/write speed. So you might end up with the read disk are slower then the white disk. Properly because its heavily fragmentet. So making 2 thread and a buffer with a chritical section you can have one thread to read chunks into buffer (I have heard from a delphi group on irc that 4k and 8k chunks are the fastest to read). So a thread to read 4k or 8k chunk into buffer and a thread to read from buffer. I think thats the best way to do it.

[jeanl]

mmm, well, if I'm limited by the slower disk, then I'm not sure what I would gain from having 2 threads, unless there's something I can do while I wait for the next read/write (but for that there are asynchronous read/write function in the win32 skd)... OK I see, I could start reading the second disk while I write the first one, that makes sense... Thanks for the suggestion!
jeanl

[hank315]

Hi Jean,

Some remarks from someone (Fortran programmer) who doesn't use fopen(), fread(), fwrite().

Using the standard Fortran read/write routines I get the same speed as a normal copy/paste command.
Even performing some simple operations on the data between read and write doesn't really slow it down because CPU usage is low (10 - 20%).
If the data chunks are too small (<4K) it slows down a bit...

[jeanl]

In my (limited) tests, when using fopen() fread() fwrite(), the throughput is somewhat smaller than copy/paste, unless I use large buffers (in the order or 1MB) at which points it becomes just about as fast.
jeanl

[hank315]

IMHO it can't be faster as copy/paste.
I'm using chunks of 2K (packetsize) to scan and analyse VOBs, this gives me a read speed of 31 MB/s, using larger chunks raises it to 34 MB/s which seems a reasonable speed for my HD's.
Writing it back again to another physical HD doesn't slow it down, writing it to the same disk, speed goes down to 15 MB/s.

[jeanl]

You're right, I don't think it can be faster than copy/paste. I'm trying to get that kind of speed using small 2K packets, which is apparently what you're able to do (but not me). But I have to make more tests...
jeanl

[LIGHTNING UK!]

I expect windows buffers all the I/O anyway so it wont be reading (or writing) 2k from your hdd, it'll read (write) 32k or 64k.

Working in this way (single thread, reading, processing, writing, reading...) is fine if you're reading from a hdd. If you ever read from a cd/dvdrom, you must 'thread' the I/O or it'll never be as fast as it could be. Latency between reading, processing and reading again kills the transfer rate.

btw, I always use the CreateFile, ReadFile and WriteFile APIs. How that effects the speed is anyones guess ;)

I will say one thing though... if you read in 2k chunks from a network drive, it'll be slow as hell - because Windows caching doesn't come into play there.

[jeanl]

Thanks LUK! That's good info. In fact, from the doc, the _open(), _read() and _write() functions are not cached (although I find it hard to believe) and there are also options to disable caching with CreateFile()... I ran some tests yesterday, but they were a bit inconclusive, and I'm concerned that the results would depend on the machine... In any case, there does not seem to be any clear winner betwee all these functions, in general, although I have consistently seen much slower transfers when using 2K buffers, as opposed to for example 2MB buffers, with any of the read() _read() ReadFile() functions... But again, this might be specific to my machine.

jeanl

[mpucoder]

Are you sure you mean caching, and not read-ahead? Caching saves data that has been transferred from disk to a program, and will speed up the second and later access to the same data. Networked machines do cache the data received, but all they receive is what the program needs. The machine with the hard drive will cache the entire block of data read from the disk (if block transfers are enabled, which is almost 100% these days). This gives the effect of reading ahead, so later requests for data from the same block are really fast. Modern drives also actually read ahead and have the predicted next block in the drive's cache memory hopefully before the request arrives.

[LIGHTNING UK!]

Who knows what I mean.... I don't! lol

I'd just noticed that even though you can be reading 2k chunks, the phyiscal I/O transfers from the HDD will be 32k or whatever. So windows is doing something - be it read-ahead or buffering (caching, whatever you want to call it).

DVD Dec used to work in 2k chunks for reading the files back in ISO Write mode (don't know why, it just did), and it was fine if the file was on a local hdd - you'd never notice it being slow. Read that same file from a network share using 2k reads and you'd get awful transfer rates. Up it to 32k and you could burn in real time - rather than the programs buffer always being empty. Again, I don't know if that's readahead, buffering, caching, packet overhead, bandwidth limits etc. I just know that's what I discovered.