Raw video data IO from disk issue

[jasonme]

Hello all,

I don't know where my question should belong to, so I try my luck here. I have a video sequence which are HD raw dpx file data. Each frame is a seperate dpx file, which is about 8M byte a file (for one frame). Then I have thousands of frames. Now, the problem is, I have to load these raw data to process and compress, although it seems the disk is not fast enough. I can only reach about 8 frames per second only on fopen and fread.

In concept, SATA should have 3.0Gb/s bandwidth, in that case, I can read more than 50 frames per second. I only have normal RAID 0 disk now, how could I make it to real-time(25fps above)? If this is absolutely impossible, then what kind of hard disk should be using to make real-time IO possible?

It would be helpful even if you just point me some directions to study on. Thank you very much!

[JohnnyMalaria]

What language and OS are you using? On Windows you can use overlapped I/O and other techniques to read data from more than one file at a time. Also, how big a chunk are you reading at a time for each file? Is that 8 frames a second just for opening and reading (i.e., without any of the compression etc)? 8 x 8 MB/s = 64 MB/s = 512 Mb/s which is approx. the limit of IEEE-1394b and USB2.0. Is this an internal SATA drive? Are you reading and writing to the same physical drive? If so, having a separate drive for reading and writing will help tremendously because with a single drive, it has to seek to the input file and then to the output file which wastes a LOT of time. What happens if you do a simple copy using the OS - how long does it take to copy say 100 files? Compare this to your process to see if you need to optimize your program to leverage your OS's file handling capabilities. Also check to see how much CPU time is being used. If it is quite low, it suggests much of the delay is due to the disk I/O and not your other processing.

[jasonme]

Thank you for your reply!

I'm using C++ under Windows. Right now, I'm reading 1 frame at a time from SATA drive (which bandwidth I believe is about 3000Gb/s). Lucky for me, I only have to read from the disk but don't write (otherwise, it would be slower for sure).

What I'm experiencing may prove your theory. When I read only about 100 frames, then the first time I read all those files, it's very slow (5~8fps) and CPU usage is very low. But when reading data for the second time, it's very fast (53~55fps), which is about to reach the speed limit of SATA. So I'm guessing at the second time, the data is being stored in some cache where the disk is happy to read. However if I'm reading 300 frames, I'll be getting slow performance no matter how many times I try to load them, probably because the cache is not large enough for that many data, and cache miss is dragging the IO down.

I'll try overlapped IO today to find out whether I can have better performance. Thanks!

[MfA]

SATA is not the problem, your HD is ... to get a reliable 200 MB/s I'd say you would need at least 8 7.2K RPM drives in RAID 0.

With some of the faster drives (10K/15K/SSD) you can make do with a lot less of course, but at a much higher price per MB.