DGIndex - performance [Archive]

Malcolm

14th February 2008, 00:11

Hi all,
i'd like to ask if someone could give me some performance numbers when using DGIndex. i'm having some performance issues and i need something for comparison. The following screenshot is from Process Explorer (http://www.google.com/search?q=process+explorer&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a) monitoring DGIndex.exe. I made the snapshot while creating the .d2v file. Demuxing video and audio is OFF. (Update speed is 1 second. - This is important since it determines how the numbers in the I/O bytes graph have to be interpreted)
http://img225.imageshack.us/img225/1416/dgindex1sa9.th.png (http://img225.imageshack.us/my.php?image=dgindex1sa9.png) http://img228.imageshack.us/img228/4382/hdtachwd750raid0wd320is4.th.png (http://img228.imageshack.us/my.php?image=hdtachwd750raid0wd320is4.png)
(Second screenshot shows the transfer rate of my harddisks (2nd is a RAID 0))

As you can see, there's a spike at the beginning in I/O bytes but then it drops to 21.5 MB/s although cpu usage is low (20%). So, i would expect the speed to be like the first spike over the whole time of creating the .d2v file. My harddisk is fast enough to handle sustaind transfer rates of >50 MB/s. My cpu is a Core2Duo @3.15GHz.

So please, could you post some numbers doing the same thing with DGIndex together with some infos about your HD/CPU setup?
A screenshot from Process Explorer would be great!

Thanks a lot,
Malcolm

Inventive Software

14th February 2008, 18:15

Just reminding you about rule 8: no cross-posting. :readrule:

I find DGIndex responds quite well to caching the file in memory (hit right for a few seconds and then index to see what I mean). It's like an enormous speed boost!

I also find the first part of the file takes ages to index, then about half-way through, it absolutely flies to the end. I dunno why that is though...

Malcolm

16th February 2008, 14:58

I have posted about this topic here because i did not get any answer to my question in the hardware/software forum. Also it was about other programs there (besides DGIndex). so for me, this is no cross-posting but the attempt to get some help finally!
Up to now nearly 200 people (DGIndex users?!) have read this thread but not even ONE has posted something helpful. Is it so hard to look into Process Explorer for a second to see how fast I/O is?!
Why are you people here in the forum? To get get help yourself but without giving it to someone else? Great!
Also your answer didn't help me. i have the opposite experience with DGIndex - exactly like the I/O graph shows. Fast in the first 3 seconds but the _very_ slow.

I only wanted to know: What are your numbers???

Thanks.

Guest

16th February 2008, 15:44

Maybe people don't know what Process Explorer is and how to use it. Or maybe they don't want to install something just to answer your question. That's why I haven't answered.

jeffy

16th February 2008, 18:49

DGIndex 1.5.0 RC2
IDCT: 32bit SSE2 MMX
Audio Output Method: Disabled
Process Priority: Normal
I/O bytes: 15-40 MB/s approx.
http://img165.imageshack.us/img165/3250/dgnormal1540wf0.th.png (http://img165.imageshack.us/my.php?image=dgnormal1540wf0.png) http://img20.imageshack.us/img20/6149/dgnormalbj5.th.png (http://img20.imageshack.us/my.php?image=dgnormalbj5.png)

Process Priority: High
I/O bytes: 14-50 MB/s approx.
http://img165.imageshack.us/img165/3507/dghigh1450gb4.th.png (http://img165.imageshack.us/my.php?image=dghigh1450gb4.png) http://img20.imageshack.us/img20/2407/dghighqu5.th.png (http://img20.imageshack.us/my.php?image=dghighqu5.png)

HDTach - the video file is on the same drive as the project, the drive does not contain neither Windows, nor the paging file
http://img137.imageshack.us/img137/1052/hdtachtestut5.th.png (http://img137.imageshack.us/my.php?image=hdtachtestut5.png)

As you can see, the time for the project creation was the same, this is not always true. Sometimes, high priority = faster.

EDIT: C2D E6600 3.51 GHz, 2 GB DDR2-800 MHz RAM (780 MHz) 4-5-4-11
EDIT 2: Process Explorer 11.04 does not need to be installed, but you have to agree to EULA before first start, download here
http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

Malcolm

16th February 2008, 19:49

@Jeff,
thanks A LOT! :thanks:

Meanwhile, i'm nearly convinced that my hd setup is o.k. i guess it's DGIndex that behaves weird in some circumstances. i have cases where DGIndex runs at maximum speed (limited by max. possible hd transfer rate) - see the following first screenshot. And there are cases where the speed drops after a while from >70 MB/s to 20-25 MB/s - see the second screenshot. Since this happens sometimes after more than 2 or 3 gigabytes have been processed, i don't this it has something to do with caching. (btw. hd is defragmented)
http://img166.imageshack.us/img166/3405/dgindex3jk8.th.png (http://img166.imageshack.us/my.php?image=dgindex3jk8.png) http://img166.imageshack.us/img166/7039/dgindex4ou6.th.png (http://img166.imageshack.us/my.php?image=dgindex4ou6.png)

My PC: C2D 4400 @ 3.15GHz, 2 GB DDR2-800 RAM (@840 MHz) 4-4-4-12

@neuron2,
o.k. it's likely that a lot of people don't know Process Explorer. (btw. i have linked it in my first post). I also understand your point, not everyone wants to answer this question. I thought maybe at least one of them would... :rolleyes: (so thanks again Jeff!)
Apart from this: i'm surprised that someone like you doesn't use Process Explorer... ;)

jeffy

16th February 2008, 22:45

@Malcolm
Yes, I come here for help and to help. I would like to say that I have read both your threads and am sorry that I wasn't able to reply earlier. The trouble with Process Explorer I/O bytes graph is: if DGIndex is done with indexing, it goes to 0 (Z-E-R-O, so it tells nothing, except for the graph history).

(btw. hd is defragmented)
Is it fully defragmented? Better worded: is the file you are indexing fully defragmented = (if possible) does it have 1 fragment (or lowest possible number of fragments)? You probably know that when the free space on the drive is less than 15% of the total space, Windows Defragmenter won't defragment the drive fully. Use something better: Diskeeper (commercial sw) or Contig (http://technet.microsoft.com/en-us/sysinternals/bb897428.aspx) & PowerDefragmenter GUI (http://www.majorgeeks.com/Power_Defragmenter_GUI_d4647.html) (both free and you can defrag a single file instead of the whole drive).

Command line example if GUI fails:
analyse number of fragments
contig -a filename.ext

defragment a single file:
contig filename.ext

Note: I use Windows XP Pro SP2

Question:
When you have more than 70 MB/s I/O throughput, could you please tell me what IDCT you selected and what are the properties of a clip and settings? (eg. 29.97 fps, honor pulldown flags, idct - simple mmx, no audio, 1920x1080, 50201 frames). A simple screenshot like mine above would be sufficient, thank you.

Malcolm

16th February 2008, 23:53

@Malcolm
Yes, I come here for help and to help. I would like to say that I have read both your threads and am sorry that I wasn't able to reply earlier. Thanks, i appreciate that. :)

Is it fully defragmented? Better worded: is the file you are indexing fully defragmented = (if possible) does it have 1 fragment (or lowest possible number of fragments)? The files from the last 2 screenshots had 1 fragment and 5 fragments respectively. 5 fragments shouldn't hurt overall I/O performance. maybe some sharp drops when the hd has to seek the next fragment. I have defragementet the partition with JkDefrag (filehippo (http://www.filehippo.com/download_jkdefrag/) and JkDefrag Homepage (http://www.kessels.com/JkDefrag/index.html))

Question:
When you have more than 70 MB/s I/O throughput, could you please tell me what IDCT you selected and what are the properties of a clip and settings? (eg. 29.97 fps, honor pulldown flags, idct - simple mmx, no audio, 1920x1080, 50201 frames). A simple screenshot like mine above would be sufficient, thank you. Here are the 2 screenshots:
http://img135.imageshack.us/img135/722/dgindex3amv9.th.png (http://img135.imageshack.us/my.php?image=dgindex3amv9.png) http://img135.imageshack.us/img135/5933/dgindex4aed3.th.png (http://img135.imageshack.us/my.php?image=dgindex4aed3.png)
Demuxing of video and audio was deactivated. DGIndex only had to scan the file and create the .d2v file.
As far as my understanding goes, the IDCT should be irrelevant at indexing time. IDCT is only performed when frames are decoded by dgdecode.dll. I don't know if 'Honor pulldown flags' affects scanning speed. Nevertheless 'IDCT Algorithm' is set to 'Skal SSE MMX'. 'Field operation' is set to 'Honor Pulldown Flags'.
I just did a quick check: 'Ignore Pulldown Flag' does not alter scanning speed. Still on some (not all) video files speed drops to 21 MB/s in the middle of the process and stays there. I suspect something more low-level. Either a fault in DGIndex itself or hd access by Windows (caching, blocksize). Particularly block size affects transfer speeds enormously. Try HD Tune (http://www.hdtune.com/) and set blocksize to 2kb when doing a benchmark. Maybe DGIndex switches to small blocksizes at some point... But that's all speculation.

Zep

19th February 2008, 09:30

Apart from this: i'm surprised that someone like you doesn't use Process Explorer... ;)

yeah once you do you will NEVER go back to the crappy task manager. The ability to see vars of an app and EACH THREAD an app is running and how much CPU each thread is using is gold and that is just the tip of the iceberg!

As for defragmenting , it is mostly worthless unless the defragger app moves the whole file to the outer edge of the platens. Even if just 1 fragment, your speed will be cut in half if as an example, the first part of video data is on the outside tracks and the second half on the very inside tracks. On smaller hard drives like 80 gigs or less, even a non fragmented large video file will end up being closer to the inner most area of the platens somewhere in the file and you will see a large slow down as data is read from this area.

UPDATE: yup sure enough that is what happens. I just now forced the first vob to the outside and the second vob to the very inside of a totally empty hard drive using ultimatedefrag and as soon as it hit the 2nd vob on the inner track the read speed was cut in half. So take a closer look as to exactly where the video data is on your hard drive(s)

Malcolm

19th February 2008, 11:40

UPDATE: yup sure enough that is what happens. I just now forced the first vob to the outside and the second vob to the very inside of a totally empty hard drive using ultimatedefrag and as soon as it hit the 2nd vob on the inner track the read speed was cut in half. So take a closer look as to exactly where the video data is on your hard drive(s)

Thanks for the hint. But i don't think this is the reason here. My HD can deliver >70 MB/s even in the outer areas of the platters. See the screenshot from HD Tach in my first post. Also the video files are placed on the inner area. But thanks nonetheless.

squid_80

19th February 2008, 12:30

Are you saving the .d2v file to a separate drive, or is it being saved to the same drive which holds the source vob?

Malcolm

19th February 2008, 13:27

Are you saving the .d2v file to a separate drive, or is it being saved to the same drive which holds the source vob?

I save the .d2v file to the same partition. But since the filesize is tiny, i'd guess that actual HD access is completely delayed by Window's write-cache. So that shouldn't hurt performance. But i'll do some test with a different target partition just to be sure.

squid_80

19th February 2008, 13:49

I save the .d2v file to the same partition. But since the filesize is tiny, i'd guess that actual HD access is completely delayed by Window's write-cache. So that shouldn't hurt performance. But i'll do some test with a different target partition just to be sure.
Unfortunately it's not the way I see it working here. It starts off fast with high cpu usage then the cache seems to run out (even though the input file should not be cached due to _O_SEQUENTIAL flag) and the drive starts seeking between reading the input file and writing the output. Reading and writing to separate drives works a lot better.

burfadel

19th February 2008, 15:59

Malcolm

19th February 2008, 16:54

What motherboard chipset have you got? Nvidia have performance sata drivers for all their controllers, and Intel has them for their higher chipsets - The ich9r southbridge controller for intel in combination with the p35 chipset for example, not the plain ich9 controller.

The combination of the intel turbo memory driver etc is supposed to help in certain circumstances? so I've heard anyway

I have a P965 chipset MoBo with the plain ICH9 chip. AFAIK the Intel Turbo Memory driver is supposed to work with an NAND chip ('Turbo Memory') to speed up boot time. Does it help too with 'normal' MoBos? Can you provide a link with additional informations?

Besides of that: Even if this driver speeds up HD access, i don't think this is the solution for DGIndex behaviour.

squid_80

19th February 2008, 18:11

No matter what sort of transfer rate your drives can achieve, if they're seeking like I explained performance is going to suck.

Malcolm

20th February 2008, 21:57

No matter what sort of transfer rate your drives can achieve, if they're seeking like I explained performance is going to suck.O.k. i did some further tests. Writing the d2v file to another HD makes no difference. Only 20 - 25 MB/s. There's no seeking on my HD (i can hear my HD if the head moves rapidly). I even created a ram disk to write the d2v file to, to completely eliminate additional work performed by the driver/controller. No difference.
Simply i suspect DGIndex / Windows file API to have some severe issues. There's simply no reason why speed should be that low....

burfadel

21st February 2008, 00:07

The readme file for Intel Turbo Memory driver can be found here:
http://downloadmirror.intel.com/14824/ENG/readme.txt

Although it may be primarily for the function of NAND, since it replaces the current 'inf' based driver (that is, it uses the internal windows driver), with an actual driver there may be some difference due to performance optimisations?

hkazemi

26th February 2008, 02:34

Try creating a ramdisk:
http://www.ramdisk.tk/
http://www.mydigitallife.info/2007/05/27/free-ramdisk-for-windows-vista-xp-2000-and-2003-server/

Then try putting the video file on it.
Then re-run the above tests.

This should help clarify whether it is a hard drive or DGIndex performance issue.

I think caching (in DGIndex or in the drive) is playing a role in the first spike.

hkazemi

26th February 2008, 02:42

In your Vista screenshot where the performance nose dives partially in, perhaps you are seeing some automatic lowering of process I/O priority. I don't know, but maybe the OS is limiting drive I/O by application to prevent the monopolization of resources. Or maybe an antivirus scan is kicking in and competing for I/O.

Zep

26th February 2008, 15:24

Thanks for the hint. But i don't think this is the reason here. My HD can deliver >70 MB/s even in the outer areas of the platters. See the screenshot from HD Tach in my first post. Also the video files are placed on the inner area. But thanks nonetheless.

your drive can read at 70 megs on the outside and half that on the very inside. My drives (heck all drives) are the same because the RPM of the drive is constant so twice the surface area is passed over on the outside of the platens in the same amount of time as the very inside.

Zep

26th February 2008, 15:27

Malcolm

5th March 2008, 13:51

Zep

6th March 2008, 20:08

o.k. thank you guys (hkzemi and zep).
But i'm 100% sure that this is not an harddrive issue. As you can see on the HD Tach screenshot in my first post, my HD can deliver a MINIMUM of 70 MB/s.
@zep: You are right, it's on the inner area of the platter, not on the outer area of the platter where speed is the lowest.

There is absolutely no reason, why DGIndex should only process 21 MB/s when my HD can deliver >70 MB/s and the processor usage is below 20%.
The first 'spike' cannot be due to caching since i didn't used the file before. Also with some other video files, the speed is constantly over 70 MB/s for the whole file!

For me, this is clearly an issue of DGIndex or some other software/driver related problem.

np this stuff is interesting to me :)

so what happened when you did the ram drive test? you should give it a go to see what happens. If DGindex goes faster and steady then we know it is not DGindex and also not some memory bound thing. That does not mean it is a hard drive issue but at least you will have a better idea of what it isn't lol

signal

10th March 2008, 02:21

C2D E6600 3.51 GHz, 2 GB DDR2-800 MHz RAM (780 MHz) 4-5-4-11
No offensive about your overclocking skills there, but I would seriously back off on that E6600 overclock and better yet, test it without any overclocking before hitting DGIndex as the source of the problem.

To me it looks like your having problems with your Southbridge based on the overclock.

Here's sample shot I get consistent performance with using:

DGIndex 150rc3: Force Film, Audio Disabled, iDCT Skal SSE MMX, Playback Speed Maximum, Priority Normal
Source: 720x480 VOB NTSC Progressive 48 minutes
System: E6850 (3Ghz, no overclocking), 4Gig DDR2800 5-5-5-15 (333/400 5:6 ratio), P35 Chipset, x2 Hitachi 250GB Raid 0 on ICH9R, OS on 74GB Raptor

http://img301.imageshack.us/img301/5788/dgtest2eg1.pnghttp://img404.imageshack.us/img404/6075/dgtest3os2.png

Note: No CPU or IO Spike, Consistent 50% CPU utilization and 116.9MB Read.

48:51 minutes -> 13 Seconds ain't all bad :) so I won't complain about the "only" 50% CPU utilization. :p

Malcolm

23rd March 2008, 19:37

No offensive about your overclocking skills there, but I would seriously back off on that E6600 overclock and better yet, test it without any overclocking before hitting DGIndex as the source of the problem.

To me it looks like your having problems with your Southbridge based on the overclock.

Hi signal,
thank you for sharing your numbers & screenshots.
The overclocked system you were referring to was jeffs. My system is a C2D 4400 @ 3.15GHz, 2 GB DDR2-800 RAM (@840 MHz) 4-4-4-12.
O.k. it's overclocked too, but not that much. Still, i checked the results with a non-overlocked cpu/ram (2GHz, 800MHz) as you have proposed but i got the same results (around 20 MB/s). So i don't think the problem is because of overclocking. You see, i have transferrates of over 80 MB/s when copying from one hd to the other. Or as a different example when multiplexing i also get 'normal' I/O rates in the range what i would expect from my hd (55MB/s reads + 55MB/s writes).

Nonetheless it's totally beyond me why DGIndex is so slow. Or the other way, why it's so fast on your system. Are you sure the file wasn't coming from window's cache instead of the hd?? I had the case myself that the first part of a file was in the filecache and I/O was that high. So would you please do me the favor and take a longer/bigger file like 2 hours / 5GB and feed this to DGIndex while watching Process Explorer? That would be great! What numbers do you get in the last third when processing the file?

A second question: what did you choose in Process Explorer under 'View -> Update Speed'? 1 second??

When playing back with F5 or F6, DGIndex decodes the stream for display. In this case cpu usage is at 50% (1 core). The speed is then limited by the cpu, not the hd. (Btw. i get around 150 fps (Information windows) and around 3 MB/s in Process Explorer). To max out I/O you have to choose F4! Because then DGIndex does *not* decode the stream for display (cpu usage <20%).

Thanks a lot and greetings,
Malcolm

Guest

23rd March 2008, 23:19

I tried the experiment using 1.5.0 RC3.

My destination drive HD Tach result is 55MB/s -> 35MB/s.

My sustained IO rate was 45 MB/s and remained steady during the entire Save Project operation.

Malcolm

24th March 2008, 13:07

@neuron2
Thanks :thanks:

I've made an important discovery yesterday: After scratching my head for the umpteenth times, i remembered a presumption i had from the very beginning: Maybe this all has to do with blocksize when accessing the data on hd.
So i dug out a tool i know for long time from german magazine c't. This tool is able to monitor exactly what's going on when accessing your harddisk. It measures & displays among other things current requested blocksize as well as the (logical) position of the head on the hd! (This is a really cool tool if you ever have the need to look for things like that) Here's the link: HDiskPerf 1.0.6 (http://translate.google.com/translate?u=http%3A%2F%2Fwww.withopf.com%2Ftools%2Fhdiskperf%2F&langpair=de%7Cen)
Here's a screenshot when running DGIndex:
http://img80.imageshack.us/img80/3618/dgindexhdiskperfmm3.th.png (http://img80.imageshack.us/my.php?image=dgindexhdiskperfmm3.png)
You can see, there's a sharp drop in the green graph (current blocksize) in the upper area. This is where blocksize dropped from 64KB to 4KB (current blocksize is displayed in the middle green bar).
(Btw. You can see in the lower area that the file is unfragmented, since the head is moving sequentially over the hd surface.)
This drop *exactly* corresponds to the drop of the transferrate in the screenshot of Process Explorer i showed before where it goes from >70MB/s to 23MB/s. Here's the screenshot:
http://img166.imageshack.us/img166/7039/dgindex4ou6.th.png (http://img166.imageshack.us/my.php?image=dgindex4ou6.png)

So, all this is because of the blocksize changing from 64KB to 4KB. Why?

I have looked into the sources of dgindex 1.4.9. Most of the times files are opened with _open(input_file, _O_RDONLY | _O_BINARY | _O_SEQUENTIAL) (initial_parse.cpp). There are several places in gui.cpp where you open the files with _open(cwd, _O_RDONLY | _O_BINARY) - so, no SEQUENTIAL flag. I don't know if this plays a role. Actually i don't know *what* exactly influences the blocksize when accessing data on hd. Since there are benchmark apps like HD Tune where you can choose the blocksize explicitly, you *can* influence this in your application. But then, maybe you can only read raw sectors from disk. Other possibility: It's determined by the disk.sys driver of windows.
So my finding today is, that this is not a problem of my system/hd/whatever. But it's application or driver behaviour.

Is someone willing to to test DGIndex again, when monitoring your hd with HDiskPerf and share the screenshot? 'signal' maybe? I'd be very interested to see if blocksize changes like on my pc.
Oh and btw.: I have few cases (1 out of 10) where the complete file is processed with a speed of >70MB/s. I suppose that blocksize stays at 64KB then...

@neuron2: Do you have an idea of how hd access in DGIndex can be optimized? (hd access with blocksize 64KB is ALWAYS faster than with 4KB). So if you could force this, probably every user of DGIndex would benefit.

Guest

24th March 2008, 14:10

So my finding today is, that this is not a problem of my system/hd/whatever. But it's application or driver behaviour. DGIndex opens input files with _O_SEQUENTIAL. I doubt it's relevant, though. DGIndex requests data in read operations of size 2048 bytes. It's entirely up to the operating system and HDD driver to decide how much to read at once from the disk. My guess is that its disk cache fills and then it has only to refill its cache with each subsequent operation. I would expect that to be the same on all systems and so I'm not persuaded by your theory of the cause for the slowdown. But that's all speculation of course.

Do you have an idea of how hd access in DGIndex can be optimized? (hd access with blocksize 64KB is ALWAYS faster than with 4KB). So if you could force this, probably every user of DGIndex would benefit. I could try reading a larger amount on each read operation. It may not be trivial though, because the design inherited from DVD2AVI has places where the buffer size is assumed to be the size of a DVD sector (2048 bytes). I'll experiment with it and report back.

Guest

24th March 2008, 20:29

I made a version with a 1MByte buffer. So it will request 1MByte chunks from the file system. Try it out and let us know if it changes anything.

http://neuron2.net/guest/DGIndex.zip

burfadel

24th March 2008, 20:43

Hi signal,
thank you for sharing your numbers & screenshots.
The overclocked system you were referring to was jeffs. My system is a C2D 4400 @ 3.15GHz, 2 GB DDR2-800 RAM (@840 MHz) 4-4-4-12.
O.k. it's overclocked too, but not that much.

I take it you mean 2.15Ghz? 3.15Ghz is a very large overclock for that CPU!

burfadel

24th March 2008, 20:51

I made a version with a 1MByte buffer. So it will request 1MByte chunks from the file system. Try it out and let us know if it changes anything.

http://neuron2.net/guest/DGIndex.zip

I tried indexing a m2v file (output from Projectx) with this version and your normal 1.50 RC3 from which this one is based.

The 1mb buffer version was faster on every run I did. which I did 3 each with both versions. The worst performance benefit was 14 percent over the normal version. Is it possible to give test versions with 2mb and 4mb buffers? I realise with the bigger buffer you get diminishing returns, but just out of interest.

Also, how does the audio and d2v file actually get written to the disk? I would assume the d2v file is saved to memory until the end since its small for maximum performance? (and if not maybe this can be a consideration). Whats the size of the audio buffer before its written to the disk, increasing this may decrease the time taken to index vob files for example? These are questions more than statements, just thought they may be of benefit!

Guest

24th March 2008, 21:31

Thank you for your preliminary results. They show that it will be worthwhile to embark upon a quest to answer your questions and others that occur to me. I'll work on it and report here.

Malcolm

25th March 2008, 01:09

@neuron2
thanks a thousand times!! :thanks::thanks::thanks:

It's so great that you created this special version of DGIndex!
And i didn't expected this so quickly! Otherwise i would have checked the forum earlier - that's for sure! :)

The results of my very first test with this version are already outstanding! Blocksize stayed at 64KB for the whole 6GB file. Also the transferrate fluctuated at around 51MB/s. that's a speed increase of factor 2.5! Great! Thanks again!
Here are the screenshots:
http://img527.imageshack.us/img527/7862/dgindex1mbbufferhdiskpemv0.th.png (http://img527.imageshack.us/my.php?image=dgindex1mbbufferhdiskpemv0.png) http://img413.imageshack.us/img413/6982/dgindex1mbbufferprocessmn5.th.png (http://img413.imageshack.us/my.php?image=dgindex1mbbufferprocessmn5.png)

I also had files where the transferrate was lower (45 MB/s). Since cpu usage is still below 35% and because hd access is purely sequential (see HDiskPerf screenshot) i can imagine that there is still room for improvement. (My hd has sequential transferrates of >90MB/s @64KB blocksize). So i'm anxious to see how a 2mb / 4mb version will perform! :rolleyes:

Edit: Meanwhile i also had files where the speed was over 80MB/s. That's factor 4!
Memory usage was *exactly* 23.8MB regardless of the file i was currently processing. The normal version of DGIndex uses nearly the same amount (21.8 MB).
http://img413.imageshack.us/img413/5125/dgindex1mbbufferprocessft7.th.png (http://img413.imageshack.us/my.php?image=dgindex1mbbufferprocessft7.png)

Cheers!
Malcolm

Malcolm

25th March 2008, 01:21

I take it you mean 2.15Ghz? 3.15Ghz is a very large overclock for that CPU!

Hi burfadel,
no, it's really 3.15GHz. Actually it even runs stable at 3.25GHz (i had this running for 6 months). 3.15GHz is the 'conservative' configuration. :D
Even under heavy load (both cores maxed out (encoding, gaming)) the cpu temperature does not go beyond 60° C. The cpu fan turns between 900 and 1100 rpm then and believe me, you cannot hear it!

Malcolm

25th March 2008, 01:33

In your Vista screenshot...
I'd like to point out that i'm using Windows XP! It's only a desktop scheme, not Vista what you see in the screenshots!
Maybe the OS this is of importance when examining/comparing the behaviour of DGIndex.

Guest

25th March 2008, 02:20

I've been experimenting and I find the greatest performance increase for my system (single core FX-55) is when the buffer size is exactly 64KBytes. As I make it bigger strange interactions start to occur with the OS and disk caching and the performance suffers, and when I make it 4MBytes it becomes unstable and hangs.

So I'll likely regression test this thoroughly and then make a new RC4 at 64KB buffer size.

burfadel

25th March 2008, 02:28

Whats the current buffer size of dgindex? maybe it depends on the OS, driver, chipset etc etc? all I know is that the 1mb version is faster than the current RC3, I've done some more tests!

This is on a C2D E6600/p35 chipset on Vista x64.

Guest

25th March 2008, 02:38

Whats the current buffer size of dgindex? http://forum.doom9.org/showthread.php?p=1116577#post1116577

Please re-download the test version. It now has a 64K buffer. Your test results will be appreciated.

http://neuron2.net/guest/DGIndex.zip

burfadel

25th March 2008, 03:18

Malcolm

25th March 2008, 10:50

Please re-download the test version. It now has a 64K buffer. Your test results will be appreciated.http://neuron2.net/guest/DGIndex.zip

Hi neuron2,
i will test the new version as soon as i can. I'm at work right now, so it has to wait until evening.
Maybe it's a good idea to make the buffer size configurable. Preferably in the gui under options. :cool:

Guest

25th March 2008, 13:19

Just did 3 tests each with the normal and 64kb buffer version on the same file as before. The 64kb buffer was faster, but only slightly. The 1mb version was still faster! (I even retested with it again). Things get very slow for me with large buffers and I don't know why. But I have a single core. That may be significant. I will explore making the buffer size configurable.

Guest

25th March 2008, 14:01

Redownload the test version linked above. It now has a configurable buffer size.

First open the new DGIndex and immediately close it. Then edit the last line of the INI file as desired before opening DGIndex for use with your new buffer size.

I'll be interested in your results.

Note: Excessively large values may adversely impact program stability and GUI usability.

Guest

25th March 2008, 14:21

Oops. Download it again if you already have. It had a bug.

BTW, powers of 2 recommended for the buffer size.

Taurus

25th March 2008, 17:14

mmh, just a quick and dirty test.
I've opened a 2 gig mpeg2 file.
No audio extraction, no other fancy stuff, just plain d2v project.
Old single core Athlon XP 2600.
@ 64kb buffer I get almost the same timings as in RC3.
Everything above or much under this threshold gave me slowdowns or hickups:p.
Summarize: No advantage by changing the buffer @ my side :confused:
Should I test with audio and video demuxing?

Guest

25th March 2008, 17:17

I don't get it either. I do get a small improvement of about 10% with the 64KB buffer.

I think it may have something to do with the threading on dual core processors.

Malcolm

25th March 2008, 21:33

Ok, here are my results so far with RC4:
I used 4 DVB-S recordings which were already demuxed to pure m2v streams. First result is from RC3 for comparison. Other results are RC4 with 64k buffer, 1mb buffer and 4mb buffer. The reported numbers are the time for processing that DGIndex prints in its information window.

video A: 5.80GB
RC3: 3:13 (min:sec)
64k: 1:44
1mb: 1:51
4mb: 1:44

video B: 4.27GB
RC3: 2:50
64k: 1:11
1mb: 0:58
4mb: 1:00

video C: 4.57GB
RC3: 2:52
64k: 1:21
1mb: 1:19
4mb: 1:11

video D: 4.33GB
RC3: 2:45
64k: 1:14
1mb: 1:24
4mb: 1:18

- Processing speed has doubled when going from RC3 to RC4
- buffer size doesn't influence processing speed much
- With buffer size 64kb, Process Explorer reported a fairly constant speed of around 60MB/s
- With higher buffer sizes, Process Explorer reported fluctuating speeds between 45MB/s and 85MB/s. (frequency 1 second).
- IIRC HDiskPerf reported higher 'I/O usage' (firt blue bar) of around 90% with 1mb/4mb buffer than with 64kb buffer (35% usage). But this has changed later. So i'm unsure if this has really something to do with buffer size.

I couldn't find any instabilities, slowdowns or similar with buffer sizes of 1mb and 4mb.

My conclusion:
64kb buffer size seems to be a very good 'save' setting.

Thanks neuron2 for your work! :thanks:

Guest

25th March 2008, 22:05

My conclusion:
64kb buffer size seems to be a very good 'save' setting. Yup, that was my conclusion too. Thank you for your testing.

Taurus

25th March 2008, 23:03

My findings:
I really don't get it:confused::p
So I did the same tests on an AMD Athlon XP 2200+
Filesize: 4.5 Gigs vob files
RC3: 2.29 sec
RC4: 2.28 sec @64k buffer (just normal fluctuation)
Yes, and I double and triple checked the results :cool:

What makes me wonder:
Why is Malcolm's machine running so slow?
This one: WinXP SP2, 512 RAM, slow harddisk, Antivirus running and many background tasks, so just a normal User frontend.
His machine should crunch the numbers in 1/4 of the time or even less.
I guess more observation is needed.
No benefit in the 64k tweak for older machines..