can we speedup rawsource?

[hanfrunz]

Hello everyone,

i have a large uncompressed file (mxf, avid1:1, 4:2:0 -> ~4mb/frame) on an external raid-system. The avid application can play it in realtime without problems.

But if i play my script with wmp i only get ~5 fps(!). Is there a way to speed this up? Reading larger blocks, not use windows api...?

The script is a very simple only one line:
RawSource("file.mxf",1920,1080,"UYVY", index="0:6192 1:4159536")

Any ideas?

Thanks hanfrunz

[Fizick]

did you look to source code?

(how it is implemented)

[hanfrunz]

yes it uses _open, _lseeki64 and _read. It then reads line by line. In tried my IUF-filter which reads the whole frame at once, but this doesn't increase speed. So i think there must be a totaly different way to access the file.

[IanB]

dst = env->NewVideoFrame(vi);
...
pdst = dst->GetWritePtr();

//read and copy all lines
for (i=0; i<vi.height; i++) {
memset(rawbuf, 0, maxwidth * 4);
ret = _read(h_rawfile, rawbuf, bytes_per_line);
for (j=0; j<(bytes_per_line/mapcnt); j++) {
for (k=0; k<mapcnt; k++) {
pdst[j*mapcnt + k] = rawbuf[j*mapcnt + mapping[k]];
}
}
pdst = pdst + dst->GetPitch();
}This is the code for the non-special cases. And it will be slow. For the avisynth native pixel formats the code I have marked blue would be a nop it should be absent for those cases.

dst should use a forced pitch (-ve) that matches the file so line shuffling is not needed.
For maximum speed there should be 1 big read directly into pdst.
the 1 big read should be disk cluster aligned.
dst should be requested already big enough to allow for this.
Use subframe to trim any crap off the beginning and/or end.
Have a no code case for formats that do not need byte reordering
Use MMX/SSE to do any byte reordering and do it in place.

Rules to live by :-
setting a pointer is 100,000's of times faster than actually moving bytes around in memory.
Using cluster aligned I/O means the OS can DMA directly into your codes buffer, so no hidden OS memcpy's. Even Windows 3 did this.
In place byte shuffling is faster than cross buffer shuffling.

The above code breaks all these rules.

[hanfrunz]

Hi IanB,

thanks for your reply. Great info. I have some questions:

*how can i force a pitch what do you mean with "(-ve)"
*do you think it is enough to read one frame at once (~4mb) or should that buffer be even bigger?
*how can i read disk cluster aligned? Is this possible with "normal" file io commands or do i need a special library?

i would do the changes but i have not enough knowledge (yet) about fast file access :-)

can i "steal" code from somewhere?

[IanB]

thanks for your reply. Great info. I have some questions:

*how can i force a pitch what do you mean with "(-ve)"This is the declaration for env->NewVideoFrame() :-virtual PVideoFrame __stdcall NewVideoFrame(const VideoInfo& vi, int align=FRAME_ALIGN) = 0;it has an optional argument align which defaults to 16. For positive values the true alignment is meddled with to keep minimum alignment constraints. For negative values you get the ABS of whatever you specified (let the programmer beware).

Making the PVideoFrame pitch match the pitch of the data in the file means you can do 1 single read directly into the PVideoFrame's buffer. You still might have to do a memcpy to make the pitch and alignment compatible with the requirements of the filters you wish to use, but doing this with overlaping memmove's within the existing buffer will still be faster than cross buffer memcpy's. RGB24 has no constraints, RGB32 and YUY2 need to be 4 byte aligned, YV12 needs to be 16 byte aligned.

*do you think it is enough to read one frame at once (~4mb) or should that buffer be even bigger?You do not want to memcpy data around, it is slow. Arranging for the OS to DMA the data directly into an Avisynth controlled buffer is as fast you will get. Yes it might be very slightly faster to do a single 64MB read than 16 x 4Mb reads, but with the former you would have to do 16 x 4MB memcpy's.

*how can i read disk cluster aligned? Is this possible with "normal" file io commands or do i need a special library?You just use normal Win32 IO calls. It is all about position and transfer size. If the file is positioned on a disk sector boundary and you transfer an integer number of sectors then the OS does not need to use an intermediate buffer.

So if your data starts at 0x100028 and end at 0x1FFF42, the disk controller must transfer from 0x100000 to 0x200000 to achieve this (i.e. only physical sectors). If your IO request matches this then no OS buffer is required. If not then an OS buffer is used and a memcpy to your buffer is performed. Better to doctor your IO request and just access your data starting at buff[0x28] and stopping at buff[0xFFF42] with buff[] needing to be 0x10000 long. Dubious use of SubFrame() can achieve this in an Avisynth context.

i would do the changes but i have not enough knowledge (yet) about fast file access :-)

can i "steal" code from somewhere?You might look at Avery Lee's code in the VirtualDub capture modules.

[hanfrunz]

this looks promissing:

http://msdn.microsoft.com/en-us/library/aa366548(VS.85).aspx

i'll investigate and test...

[hanfrunz]

i need a little help:
-i can now open a file and map it to memory (file mapping)
-The align of NewVideoFrame() is set to 0.
-I have a pointer to a portion of the file (one frame) ---> char * pData
-And i have my PVideoFrame dst

how do i get my data to dst? Do i have to use memcpy() or can i somehow set the pointer of dst to the adress of pData?
BTW i never understood this whole pointerstuff 100% :)

[Fizick]

align to 0 ? it may be 16, 8, (may be 1) IMHO.

probably you can use env->BitBlt.
(but i do not know much about "file mapping" an how is raw format similar to avisynth internal YUY2)

[hanfrunz]

align to 0 ? it may be 16, 8, (may be 1) IMHO.

probably you can use env->BitBlt.
(but i do not know much about "file mapping" an how is raw format similar to avisynth internal YUY2)

mmh yes i set it to 0 and it works :confused: but i'll better set it to 4 :)

my file is UYVY (damm it) but for the case of an "real" yuy2 file i'd like to know.

[IanB]

@hanfrunz,

You have not understood one word I have written? Okay I get that a lot :( Keep asking me to simplify until it makes sense.

File mapping will not help here. It just uses the OS page fault mechanism to read the file contents into memory, 1 page at a time for you as you touch the first byte in each page. Great for random access of an unpredictable data structure, i.e. a database volume file. Not so great for 1 time sequential access of large blocks of data. And you still would be memcpy'ing the data into the PVideoFrame buffer.

To make this easy, I'll spec an 85% solution. :-

Forget about the aligned reads stuff, it's complicated and will only save an average of 0.5 cluster of I/O and the associated OS memcpy.

Concentrate on doing the file I/O as 1 single contiguous read directly into the PVideoFrame buffer. This will be easiest to give a substantial performance boost.

The PVideoFrame you get from env->NewVideoFrame(vi, -1), i.e. with align=-1, will have a pitch equal to vi.RowSize() and the buffers total size will be at least vi.height*{pitch}. vi.RowSize() is vi.width*{bytes per pixel}

You can then use env->SubFrame() to adjust the final height, width and pitch to suit the true image size.

Do any byte twiddling in place, preferably with an MMX/SSE routine, or at least with some dedicated tight C code. The current general purpose code, although convenient, is slow. Code explicit fast routines for the cases you like and fall back to the general purpose code for those you do not care for.

If your files structure results in a non pixel modulus pitch, dummy up the PVideoFrame to be slightly larger than needed and use a reverse env->BitBlt() to repack the data in place or customise the byte twiddling routine to do it as a side effect. Then use env->SubFrame().


Here is some prototype MMX to in place swap bytes, i.e. UYVY -> YUY2. It does 16 bytes per loop, i.e. 8 UYVY pixels.mov eax, srcbuff
mov ecx, count16
loop123:
movq mm0, [eax]
movq mm2, [eax+8]
movq mm1, mm0
psrlw mm0, 8
psllw mm1, 8
movq mm3, mm2
psrlw mm2, 8
psllw mm3, 8
por mm0, mm1
por mm2, mm3
movq [eax], mm0
movq [eax+8], mm2
add eax, 16
dec ecx
jnz loop123

[hanfrunz]

Hi IanB,
thanks for your help. I'm not a very advanced programmer, so you have to be patient with me. :)
Okay the trick is to read a complete frame (or more) directly to my PVideoFrame.
I'll try that with this commands: _open, _lseeki64 and _read, right?
I'll experiment without byte shuffling first to see if the file access is fast enough, and come back later to optimize that :)

regards,
hanfrunz

[tritical]

Are you sure the bottleneck is actually rawsource? Testing it on my system (Q6600@2.8, 4 GB RAM, 7200 rpm sata hd) with a 1920x1080 UYVY file I get ~14 fps (load script in vdub, select file->run video analysis pass). However, if I try to play the script in mpc it gets about ~1-2 fps. If I add assumefps(13.0) after rawsource() then try to play the script in mpc it plays at 13fps.

Btw, I quickly wrote a rawsource clone which reads directly into the frame buffer and has a much faster byte shuffling routine, but it was not significantly faster than rawsource. Strangely (or maybe not strangely... I'm not an i/o guru) I found it to be faster to read the data line by line into the frame buffer than to read entire planes or entire frames at a time. Also, using _read() was just slightly faster than using fread() in combination w/ setvbuf() and a buffer size of 2MB. I didn't try to do disk cluster aligned reads.

[sh0dan]

My feeling is exactly as tritical, you are wasting time by doing MMX/SSE. You can read/write memory at 10GB/sec, and disk IO is very likely not faster than 100MB/sec, so the only source of speed-ups of any significance will be if you do async IO. That is, read and prepare next frame, while the rest of the filter chain is running.

Windows has built-in functions for async IO, but they obviously take more work to get working.

[hanfrunz]

what makes me wonder is, that the avid can play the video in realtime. If i export an quicktime-ref and play it with QT-player or read the mxf file directly or copy/paste the big file i never get "realtime". I get ~1/4x realtime to copy the file from one raid volume to another. So what "tricks" does avid use to get realtime access?

EDIT:
new numbers:
VirualDub - Run video analysis pass - original rawsource -> 2fps
VirualDub - Run video analysis pass - my rawsource v.0.1 -> 13fps *

*no byte shuffling yet, filter reads 10 frames at once and uses SubFrame() to return the right part.
code looks like this:

// dst and dst2 are similar, but dst2 is 10 times higher
_read(fh,dstp2,width*height*10);
dst=env->Subframe(dst2,0,dst_pitch2,dst_width2,dst_height); // return first frame

[hanfrunz]

mov eax, srcbuff
mov ecx, count16
loop123:
movq mm0, [eax]
movq mm2, [eax+8]
movq mm1, mm0
psrlw mm0, 8
psllw mm1, 8
movq mm3, mm2
psrlw mm2, 8
psllw mm3, 8
por mm0, mm1
por mm2, mm3
movq [eax], mm0
movq [eax+8], mm1
add eax, 16
dec ecx
jnz loop123

@IanB
this code nearly works, colors are fine, but there is a vertical black bar pattern over the image...
scrbuff is my dstp and count16=height*width*2/16

[squid_80]

@IanB
this code nearly works, colors are fine, but there is a vertical black bar pattern over the image...
scrbuff is my dstp and count16=height*width*2/16
I think I see the problem:
mov eax, srcbuff
mov ecx, count16
loop123:
movq mm0, [eax]
movq mm2, [eax+8]
movq mm1, mm0
psrlw mm0, 8
psllw mm1, 8
movq mm3, mm2
psrlw mm2, 8
psllw mm3, 8
por mm0, mm1
por mm2, mm3
movq [eax], mm0
movq [eax+8], mm2
add eax, 16
dec ecx
jnz loop123

[tritical]

@hanfrunz
What are the specs of the system you are using? raid type? type of hard drives? connection type? processor/ram?

2fps in vdub seems ridiculously slow... that's only ~8MB/s. My computer achieves ~54-56MB/s sustained on uncached reading of large files (>1GB, single 7200rpm 3.0Gb/s 16MB cache sata hd), and rawsource, as it currently is, hits that mark. It is interesting that reading huge chunks (10 frames in this case would be 10*1920*1080*2 = ~39.5 MB !) is faster for you, as using chunks greater than 8-16KB only slows things down here.

If your file is 25fps it would require ~100MB/s sustained transfer speed to play real time... if its 30fps that goes up to ~120MB/s.

[IanB]

Opps sorry about the typo in the sample code., thanks Squid.


Yes there is a great disparity between disk and memory speed, and an awful lot of repeated slack memcpy'ing of data buffers around can be tolerated. Doesn't mean it is okay though.


Yes, on an even lightly fragmented NTFS disk, big I/O's do tend to choke. My preference still is to use a clean FAT32 partition for big work files. I believe Avery still hold to this as well. There is no substitute for having the disk controller do really big contiguous transfers, trick is to con the OS and file system into requesting these. Seems the VFAT driver tries to allocate contiguous clusters when doing big disk writes, the NTFS driver seems to have other agendas, maybe the user level write size info is not available to the disk block allocater routine. :confused:


Perhaps doing a lossless compression pass on the raw source file would be an overall time saver. Do it once, save N*(A-B) time on N passes of the work file.


To test the I/O performance of your various disk partitions you could run Avery Lee's Auxsetup Benchmark (You need a 1.6.x VDub or earlier, Avery removed the benchmark from 1.7 onwards) You can play with transfer size, Windows OS buffering and total file size.

[hanfrunz]

@tritical:
System specs: HP Workstation xw8200, 2x Intel(R) Xeon(TM) CPU 3.40GHz, 2GB RAM, Avid Symphony Nitris Hardware, Storage: Media Vault U320 RX (http://www.hugesystems.com/solutions-mediavault_U320-RX.html) (dual scsi320 Raid3, windows stripe to one big disk with 8 partitions)

@all:
if i read one frame at once directly + shuffling i get now ~19fps which is pretty good, but still not realtime...

[IanB]

1920x1080 UYVY @ 19fps is ~75 MegaBytes per second.

You don't specify the disks you are using but typical scsi320 10k disk seem to be rated at ~80 MB/s, 15k at ~120 MB/s.

So as a brute force 1st cut you are doing pretty well.

How fast does Avery Lee's Auxsetup Benchmark (cache disabled) say your disks are? i.e. Do we have a chance without resorting to async I/O and double buffering?If your file is 25fps it would require ~100MB/s sustained transfer speed to play real time... if its 30fps that goes up to ~120MB/s.

[hanfrunz]

i get 780488kb/s -> 762 MB/s with Avery Lee's Auxsetup Benchmark on my System drive and the exact same value for my raid drive...
maybe that's why avery has deleted this benchmark from newer versions?

[tritical]

It would probably be easiest to just write a simple commandline app that sequentially reads a user specified file in chunks of 'n' bytes, n would also be user specified. In the program time how long it takes (using time(NULL), rdtsc, etc...), and it should give a good idea of what your system can achieve transfer wise. That way you could also play around with the chunk size, _open/_read vs fopen/fread/setvbuf, etc... and see what works best. Make sure to use large files (>2GB), and alternate between multiple files so caching doesn't mess things up.

If you can't achieve the required sustained read speed that way, then no way rawsource will ever be able to play your file in real time.