Indexing performance over a network (SMB/CIFS)

[Blue_MiSfit]

Hi folks,

I'm wondering if anything can be done to tune indexing performance (in DGIndex and DGIndexNV) for better throughput across a network connection - specifically SMB/CIFS.

When indexing over the network, I have basically two options described below.

For reference, the file was a 14.5mbps MPEG Transport stream containing 1080i MPEG-2 video and AC3 audio. Total size was about 6GB.

The file was stored on a fairly high performance NAS running NexentaStor (OpenSolaris for kernel CIFS), over a fairly busy gigabit ethernet network.

The client machine was a Dell PE R710 system with dual Xeon x5570 CPUs (8 cores, 16 threads total) and 4GB of RAM running Windows Server 2003 R2 Standard (32 bit). The example was performed using DGIndex 1.5.8

1) Copy the file to a local scratch disk and index from there.
A) Copy from network to local: 4 minutes
B) Index: About 4 minutes, 30 seconds
Total time - about 8.5 minutes to have indexed files ready for processing

2) Do all work over network
A) Index directly: Roughly 15 minutes!!!

Obviously this isn't a high performance SAN, and even so I'm not expecting times perfectly in line with locally stored files. Still, the huge difference here is rather surprising (to me, anyway)!

Keep in mind this was on a dedicated NAS. A production NAS would be much busier.

Is there anything that can be done to tune performance for this type of scenario? The reality of building large workflows almost always means that centralized storage of some form is essential.

Thanks,
Derek

[TinTime]

Well, my knowledge of networks is limited to plugging one end of a cable into a switch, the other end into a PC and hoping for the best. Anyway a couple of thoughts.

If you want to store everything centrally but it's faster to process locally then you could stick with your first method and then edit the local .d2v index to point at the original file. Then copy the .d2v back to the NAS.

Or can you index the files before putting them on the NAS, or are they created directly there?

Finally have you tried running the test again? Maybe it was just a freak occurrence.

As you can see, I'm a fan of workarounds rather than solutions :)

[Ghitulescu]

I understand that the requirement that the file should reside on a NAS than on the workstation is rather essential (workgroup project).

I haven't yet used dgindex, I can only assume that the network traffic it produces consists mainly in queries and seeks rather than in data transfer (this also occurs), because meanwhile someone else, a colleague, accesses himself too that NAS. If the file is on the local HDD all these queries are managed by the OS (maybe by the HDD FW too) using intelligent caching and addressing algorithms (you're the only user on your PC). I hope I was clear.

I would suggest the copy of the file down, process it, than up again, maybe with the aid of a semaphore (Hey guys I'm working on this file leave it me alone!)

[The Scientist]

The example was performed using DGIndex 1.5.8

1) Copy the file to a local scratch disk and index from there.
A) Copy from network to local: 4 minutes
B) Index: About 4 minutes, 30 seconds
Total time - about 8.5 minutes to have indexed files ready for processing

2) Do all work over network
A) Index directly: Roughly 15 minutes!!!
Interesting........ Maybe there's another way........ Your post got me thinking, obviously over the network is almost twice as long because the data has got to come to your PC to be read by DGIndex then put back over to your networked drive when done, basically all I/O is being networked.

So I thought what about an option to control what's going on with DGIndex remotely? I don't have such a thing but, I wrote a little app some time ago called 'index.NET' which is like a batch indexer GUI, will extract all the audio/subs/cc e.t.c and obviously index the movie file and can do hundreds of files in it's batch. What if that was amended to run the actual DGIndex file remotely, on the same drive as your source files? maybe this would work? All I/O would then be on the same drive cutting out the network journey.

...edit
Scrub that, I'm talking rubbish, it's amazing what a strong coffee will do. If I ran a program from PC-A that was located over the network on PC-B, then the process is obviously going to load into the memory of PC-A which kind of defeats the point.

[Blue_MiSfit]

I think the issue may have something to do with how neuron2 does disk caching internally. That's just a guess, but AFAIK he does this himself, and doesn't rely entirely on Windows :)

Derek

[mastrboy]

Just i tip for general network performance related to win2003:
Test with TCP Offloading disabled on windows 2003 and consider installing the SNP hotfix, after SP2 for win2003 there was a massive drop in network performance due to SP2 enabled SNP pr default, it's basically a must for busy SQL servers or you are going to get many TCP retransmits

http://support.microsoft.com/kb/948496

If you want to keep SNP on you should at least install the following hotfix: http://support.microsoft.com/kb/950224

(You could also install wireshark or similar and look for tcp retransmits)

Related to DGIndex you have to talk to the Dev, it might have cache functionality that does not work well with CIFS...

You could also monitor your NAS for iowait to see if you are saturating the drives, maybe a SSD ZIL is the solution since you are using Nexenta ;)


I did some quick tests against my own home NAS server and it confirms reduced performance when compared to file-copy, but not as much deviation as you are getting:


Copy:

iostat:
avg-cpu: %user %nice %system %iowait %steal %idle
8.17 0.19 9.37 12.89 0.00 69.38
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
md0 0.00 0.00 4771.93 0.00 94447.47 0.00 19.79 0.00 0.00 0.00 0.00
md0 0.00 0.00 5459.00 0.13 109470.40 1.07 20.05 0.00 0.00 0.00 0.00
md0 0.00 0.00 5883.41 0.00 111497.40 0.00 18.95 0.00 0.00 0.00 0.00
md0 0.00 0.00 5885.07 0.40 110784.53 3.20 18.82 0.00 0.00 0.00 0.00
md0 0.00 0.00 5837.73 0.00 112067.73 0.00 19.20 0.00 0.00 0.00 0.00

vnstat:
rx | tx
--------------------------------------+----------------------------------------
bytes 43.11 MB | 4.52 GB
--------------------------------------+----------------------------------------
max 575.35 kB/s | 58.21 MB/s
average 455.10 kB/s | 47.75 MB/s
min 18.43 kB/s | 19.51 kB/s
--------------------------------------+----------------------------------------
packets 578064 | 3210705
--------------------------------------+----------------------------------------
max 7271 p/s | 40354 p/s
average 5959 p/s | 33100 p/s
min 30 p/s | 30 p/s
--------------------------------------+----------------------------------------
time 1.62 minutes


DGNVIndex:

iostat:
avg-cpu: %user %nice %system %iowait %steal %idle
1.69 0.00 5.07 10.43 0.00 82.80
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
md0 0.00 0.00 4408.80 0.00 69973.33 0.00 15.87 0.00 0.00 0.00 0.00
md0 0.00 0.00 5563.87 0.00 85467.73 0.00 15.36 0.00 0.00 0.00 0.00
md0 0.00 0.00 5625.33 0.00 86011.20 0.00 15.29 0.00 0.00 0.00 0.00
md0 0.00 0.00 5655.00 0.00 84772.27 0.00 14.99 0.00 0.00 0.00 0.00
md0 0.00 0.00 5929.38 0.00 87544.84 0.00 14.76 0.00 0.00 0.00 0.00
md0 0.00 0.00 5710.47 0.00 85761.60 0.00 15.02 0.00 0.00 0.00 0.00
md0 0.00 0.00 5807.67 0.00 85437.87 0.00 14.71 0.00 0.00 0.00 0.00

vnstat:
rx | tx
--------------------------------------+----------------------------------------
bytes 35.76 MB | 4.53 GB
--------------------------------------+----------------------------------------
max 371.41 kB/s | 47.30 MB/s
average 288.36 kB/s | 36.54 MB/s
min 8.29 kB/s | 20.01 kB/s
--------------------------------------+----------------------------------------
packets 493300 | 3273420
--------------------------------------+----------------------------------------
max 4882 p/s | 33384 p/s
average 3884 p/s | 25774 p/s
min 22 p/s | 25 p/s
--------------------------------------+----------------------------------------
time 2.12 minutes


62715 <- difference in nr of packets

[Blue_MiSfit]

Thanks for your data, mastrboy!

Derek