DGIndex - performance [Archive]

Malcolm

14th February 2008, 00:11

Hi all,
i'd like to ask if someone could give me some performance numbers when using DGIndex. i'm having some performance issues and i need something for comparison. The following screenshot is from Process Explorer (http://www.google.com/search?q=process+explorer&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a) monitoring DGIndex.exe. I made the snapshot while creating the .d2v file. Demuxing video and audio is OFF. (Update speed is 1 second. - This is important since it determines how the numbers in the I/O bytes graph have to be interpreted)
http://img225.imageshack.us/img225/1416/dgindex1sa9.th.png (http://img225.imageshack.us/my.php?image=dgindex1sa9.png) http://img228.imageshack.us/img228/4382/hdtachwd750raid0wd320is4.th.png (http://img228.imageshack.us/my.php?image=hdtachwd750raid0wd320is4.png)
(Second screenshot shows the transfer rate of my harddisks (2nd is a RAID 0))

As you can see, there's a spike at the beginning in I/O bytes but then it drops to 21.5 MB/s although cpu usage is low (20%). So, i would expect the speed to be like the first spike over the whole time of creating the .d2v file. My harddisk is fast enough to handle sustaind transfer rates of >50 MB/s. My cpu is a Core2Duo @3.15GHz.

So please, could you post some numbers doing the same thing with DGIndex together with some infos about your HD/CPU setup?
A screenshot from Process Explorer would be great!

Thanks a lot,
Malcolm

Inventive Software

14th February 2008, 18:15

Just reminding you about rule 8: no cross-posting. :readrule:

I find DGIndex responds quite well to caching the file in memory (hit right for a few seconds and then index to see what I mean). It's like an enormous speed boost!

I also find the first part of the file takes ages to index, then about half-way through, it absolutely flies to the end. I dunno why that is though...

Malcolm

16th February 2008, 14:58

I have posted about this topic here because i did not get any answer to my question in the hardware/software forum. Also it was about other programs there (besides DGIndex). so for me, this is no cross-posting but the attempt to get some help finally!
Up to now nearly 200 people (DGIndex users?!) have read this thread but not even ONE has posted something helpful. Is it so hard to look into Process Explorer for a second to see how fast I/O is?!
Why are you people here in the forum? To get get help yourself but without giving it to someone else? Great!
Also your answer didn't help me. i have the opposite experience with DGIndex - exactly like the I/O graph shows. Fast in the first 3 seconds but the _very_ slow.

I only wanted to know: What are your numbers???

Thanks.

Guest

16th February 2008, 15:44

Maybe people don't know what Process Explorer is and how to use it. Or maybe they don't want to install something just to answer your question. That's why I haven't answered.

jeffy

16th February 2008, 18:49

DGIndex 1.5.0 RC2
IDCT: 32bit SSE2 MMX
Audio Output Method: Disabled
Process Priority: Normal
I/O bytes: 15-40 MB/s approx.
http://img165.imageshack.us/img165/3250/dgnormal1540wf0.th.png (http://img165.imageshack.us/my.php?image=dgnormal1540wf0.png) http://img20.imageshack.us/img20/6149/dgnormalbj5.th.png (http://img20.imageshack.us/my.php?image=dgnormalbj5.png)

Process Priority: High
I/O bytes: 14-50 MB/s approx.
http://img165.imageshack.us/img165/3507/dghigh1450gb4.th.png (http://img165.imageshack.us/my.php?image=dghigh1450gb4.png) http://img20.imageshack.us/img20/2407/dghighqu5.th.png (http://img20.imageshack.us/my.php?image=dghighqu5.png)

HDTach - the video file is on the same drive as the project, the drive does not contain neither Windows, nor the paging file
http://img137.imageshack.us/img137/1052/hdtachtestut5.th.png (http://img137.imageshack.us/my.php?image=hdtachtestut5.png)

As you can see, the time for the project creation was the same, this is not always true. Sometimes, high priority = faster.

EDIT: C2D E6600 3.51 GHz, 2 GB DDR2-800 MHz RAM (780 MHz) 4-5-4-11
EDIT 2: Process Explorer 11.04 does not need to be installed, but you have to agree to EULA before first start, download here
http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

Malcolm

16th February 2008, 19:49

@Jeff,
thanks A LOT! :thanks:

Meanwhile, i'm nearly convinced that my hd setup is o.k. i guess it's DGIndex that behaves weird in some circumstances. i have cases where DGIndex runs at maximum speed (limited by max. possible hd transfer rate) - see the following first screenshot. And there are cases where the speed drops after a while from >70 MB/s to 20-25 MB/s - see the second screenshot. Since this happens sometimes after more than 2 or 3 gigabytes have been processed, i don't this it has something to do with caching. (btw. hd is defragmented)
http://img166.imageshack.us/img166/3405/dgindex3jk8.th.png (http://img166.imageshack.us/my.php?image=dgindex3jk8.png) http://img166.imageshack.us/img166/7039/dgindex4ou6.th.png (http://img166.imageshack.us/my.php?image=dgindex4ou6.png)

My PC: C2D 4400 @ 3.15GHz, 2 GB DDR2-800 RAM (@840 MHz) 4-4-4-12

@neuron2,
o.k. it's likely that a lot of people don't know Process Explorer. (btw. i have linked it in my first post). I also understand your point, not everyone wants to answer this question. I thought maybe at least one of them would... :rolleyes: (so thanks again Jeff!)
Apart from this: i'm surprised that someone like you doesn't use Process Explorer... ;)

jeffy

16th February 2008, 22:45

@Malcolm
Yes, I come here for help and to help. I would like to say that I have read both your threads and am sorry that I wasn't able to reply earlier. The trouble with Process Explorer I/O bytes graph is: if DGIndex is done with indexing, it goes to 0 (Z-E-R-O, so it tells nothing, except for the graph history).

(btw. hd is defragmented)
Is it fully defragmented? Better worded: is the file you are indexing fully defragmented = (if possible) does it have 1 fragment (or lowest possible number of fragments)? You probably know that when the free space on the drive is less than 15% of the total space, Windows Defragmenter won't defragment the drive fully. Use something better: Diskeeper (commercial sw) or Contig (http://technet.microsoft.com/en-us/sysinternals/bb897428.aspx) & PowerDefragmenter GUI (http://www.majorgeeks.com/Power_Defragmenter_GUI_d4647.html) (both free and you can defrag a single file instead of the whole drive).

Command line example if GUI fails:
analyse number of fragments
contig -a filename.ext

defragment a single file:
contig filename.ext

Note: I use Windows XP Pro SP2

Question:
When you have more than 70 MB/s I/O throughput, could you please tell me what IDCT you selected and what are the properties of a clip and settings? (eg. 29.97 fps, honor pulldown flags, idct - simple mmx, no audio, 1920x1080, 50201 frames). A simple screenshot like mine above would be sufficient, thank you.

Malcolm

16th February 2008, 23:53

@Malcolm
Yes, I come here for help and to help. I would like to say that I have read both your threads and am sorry that I wasn't able to reply earlier. Thanks, i appreciate that. :)

Is it fully defragmented? Better worded: is the file you are indexing fully defragmented = (if possible) does it have 1 fragment (or lowest possible number of fragments)? The files from the last 2 screenshots had 1 fragment and 5 fragments respectively. 5 fragments shouldn't hurt overall I/O performance. maybe some sharp drops when the hd has to seek the next fragment. I have defragementet the partition with JkDefrag (filehippo (http://www.filehippo.com/download_jkdefrag/) and JkDefrag Homepage (http://www.kessels.com/JkDefrag/index.html))

Question:
When you have more than 70 MB/s I/O throughput, could you please tell me what IDCT you selected and what are the properties of a clip and settings? (eg. 29.97 fps, honor pulldown flags, idct - simple mmx, no audio, 1920x1080, 50201 frames). A simple screenshot like mine above would be sufficient, thank you. Here are the 2 screenshots:
http://img135.imageshack.us/img135/722/dgindex3amv9.th.png (http://img135.imageshack.us/my.php?image=dgindex3amv9.png) http://img135.imageshack.us/img135/5933/dgindex4aed3.th.png (http://img135.imageshack.us/my.php?image=dgindex4aed3.png)
Demuxing of video and audio was deactivated. DGIndex only had to scan the file and create the .d2v file.
As far as my understanding goes, the IDCT should be irrelevant at indexing time. IDCT is only performed when frames are decoded by dgdecode.dll. I don't know if 'Honor pulldown flags' affects scanning speed. Nevertheless 'IDCT Algorithm' is set to 'Skal SSE MMX'. 'Field operation' is set to 'Honor Pulldown Flags'.
I just did a quick check: 'Ignore Pulldown Flag' does not alter scanning speed. Still on some (not all) video files speed drops to 21 MB/s in the middle of the process and stays there. I suspect something more low-level. Either a fault in DGIndex itself or hd access by Windows (caching, blocksize). Particularly block size affects transfer speeds enormously. Try HD Tune (http://www.hdtune.com/) and set blocksize to 2kb when doing a benchmark. Maybe DGIndex switches to small blocksizes at some point... But that's all speculation.

Zep

19th February 2008, 09:30

Apart from this: i'm surprised that someone like you doesn't use Process Explorer... ;)

yeah once you do you will NEVER go back to the crappy task manager. The ability to see vars of an app and EACH THREAD an app is running and how much CPU each thread is using is gold and that is just the tip of the iceberg!

As for defragmenting , it is mostly worthless unless the defragger app moves the whole file to the outer edge of the platens. Even if just 1 fragment, your speed will be cut in half if as an example, the first part of video data is on the outside tracks and the second half on the very inside tracks. On smaller hard drives like 80 gigs or less, even a non fragmented large video file will end up being closer to the inner most area of the platens somewhere in the file and you will see a large slow down as data is read from this area.

UPDATE: yup sure enough that is what happens. I just now forced the first vob to the outside and the second vob to the very inside of a totally empty hard drive using ultimatedefrag and as soon as it hit the 2nd vob on the inner track the read speed was cut in half. So take a closer look as to exactly where the video data is on your hard drive(s)

Malcolm

19th February 2008, 11:40

UPDATE: yup sure enough that is what happens. I just now forced the first vob to the outside and the second vob to the very inside of a totally empty hard drive using ultimatedefrag and as soon as it hit the 2nd vob on the inner track the read speed was cut in half. So take a closer look as to exactly where the video data is on your hard drive(s)

Thanks for the hint. But i don't think this is the reason here. My HD can deliver >70 MB/s even in the outer areas of the platters. See the screenshot from HD Tach in my first post. Also the video files are placed on the inner area. But thanks nonetheless.

squid_80

19th February 2008, 12:30

Are you saving the .d2v file to a separate drive, or is it being saved to the same drive which holds the source vob?

Malcolm

19th February 2008, 13:27

Are you saving the .d2v file to a separate drive, or is it being saved to the same drive which holds the source vob?

I save the .d2v file to the same partition. But since the filesize is tiny, i'd guess that actual HD access is completely delayed by Window's write-cache. So that shouldn't hurt performance. But i'll do some test with a different target partition just to be sure.

squid_80

19th February 2008, 13:49

I save the .d2v file to the same partition. But since the filesize is tiny, i'd guess that actual HD access is completely delayed by Window's write-cache. So that shouldn't hurt performance. But i'll do some test with a different target partition just to be sure.
Unfortunately it's not the way I see it working here. It starts off fast with high cpu usage then the cache seems to run out (even though the input file should not be cached due to _O_SEQUENTIAL flag) and the drive starts seeking between reading the input file and writing the output. Reading and writing to separate drives works a lot better.

burfadel

19th February 2008, 15:59

Malcolm

19th February 2008, 16:54

What motherboard chipset have you got? Nvidia have performance sata drivers for all their controllers, and Intel has them for their higher chipsets - The ich9r southbridge controller for intel in combination with the p35 chipset for example, not the plain ich9 controller.

The combination of the intel turbo memory driver etc is supposed to help in certain circumstances? so I've heard anyway

I have a P965 chipset MoBo with the plain ICH9 chip. AFAIK the Intel Turbo Memory driver is supposed to work with an NAND chip ('Turbo Memory') to speed up boot time. Does it help too with 'normal' MoBos? Can you provide a link with additional informations?

Besides of that: Even if this driver speeds up HD access, i don't think this is the solution for DGIndex behaviour.

squid_80

19th February 2008, 18:11

No matter what sort of transfer rate your drives can achieve, if they're seeking like I explained performance is going to suck.

Malcolm

20th February 2008, 21:57

No matter what sort of transfer rate your drives can achieve, if they're seeking like I explained performance is going to suck.O.k. i did some further tests. Writing the d2v file to another HD makes no difference. Only 20 - 25 MB/s. There's no seeking on my HD (i can hear my HD if the head moves rapidly). I even created a ram disk to write the d2v file to, to completely eliminate additional work performed by the driver/controller. No difference.
Simply i suspect DGIndex / Windows file API to have some severe issues. There's simply no reason why speed should be that low....

burfadel

21st February 2008, 00:07

The readme file for Intel Turbo Memory driver can be found here:
http://downloadmirror.intel.com/14824/ENG/readme.txt

Although it may be primarily for the function of NAND, since it replaces the current 'inf' based driver (that is, it uses the internal windows driver), with an actual driver there may be some difference due to performance optimisations?

hkazemi

26th February 2008, 02:34

Try creating a ramdisk:
http://www.ramdisk.tk/
http://www.mydigitallife.info/2007/05/27/free-ramdisk-for-windows-vista-xp-2000-and-2003-server/

Then try putting the video file on it.
Then re-run the above tests.

This should help clarify whether it is a hard drive or DGIndex performance issue.

I think caching (in DGIndex or in the drive) is playing a role in the first spike.

hkazemi

26th February 2008, 02:42

In your Vista screenshot where the performance nose dives partially in, perhaps you are seeing some automatic lowering of process I/O priority. I don't know, but maybe the OS is limiting drive I/O by application to prevent the monopolization of resources. Or maybe an antivirus scan is kicking in and competing for I/O.

Zep

26th February 2008, 15:24

Thanks for the hint. But i don't think this is the reason here. My HD can deliver >70 MB/s even in the outer areas of the platters. See the screenshot from HD Tach in my first post. Also the video files are placed on the inner area. But thanks nonetheless.

your drive can read at 70 megs on the outside and half that on the very inside. My drives (heck all drives) are the same because the RPM of the drive is constant so twice the surface area is passed over on the outside of the platens in the same amount of time as the very inside.

Zep

26th February 2008, 15:27

Malcolm

5th March 2008, 13:51

Zep

6th March 2008, 20:08

o.k. thank you guys (hkzemi and zep).
But i'm 100% sure that this is not an harddrive issue. As you can see on the HD Tach screenshot in my first post, my HD can deliver a MINIMUM of 70 MB/s.
@zep: You are right, it's on the inner area of the platter, not on the outer area of the platter where speed is the lowest.

There is absolutely no reason, why DGIndex should only process 21 MB/s when my HD can deliver >70 MB/s and the processor usage is below 20%.
The first 'spike' cannot be due to caching since i didn't used the file before. Also with some other video files, the speed is constantly over 70 MB/s for the whole file!

For me, this is clearly an issue of DGIndex or some other software/driver related problem.

np this stuff is interesting to me :)

so what happened when you did the ram drive test? you should give it a go to see what happens. If DGindex goes faster and steady then we know it is not DGindex and also not some memory bound thing. That does not mean it is a hard drive issue but at least you will have a better idea of what it isn't lol

signal

10th March 2008, 02:21

C2D E6600 3.51 GHz, 2 GB DDR2-800 MHz RAM (780 MHz) 4-5-4-11
No offensive about your overclocking skills there, but I would seriously back off on that E6600 overclock and better yet, test it without any overclocking before hitting DGIndex as the source of the problem.

To me it looks like your having problems with your Southbridge based on the overclock.

Here's sample shot I get consistent performance with using:

DGIndex 150rc3: Force Film, Audio Disabled, iDCT Skal SSE MMX, Playback Speed Maximum, Priority Normal
Source: 720x480 VOB NTSC Progressive 48 minutes
System: E6850 (3Ghz, no overclocking), 4Gig DDR2800 5-5-5-15 (333/400 5:6 ratio), P35 Chipset, x2 Hitachi 250GB Raid 0 on ICH9R, OS on 74GB Raptor

http://img301.imageshack.us/img301/5788/dgtest2eg1.pnghttp://img404.imageshack.us/img404/6075/dgtest3os2.png

Note: No CPU or IO Spike, Consistent 50% CPU utilization and 116.9MB Read.

48:51 minutes -> 13 Seconds ain't all bad :) so I won't complain about the "only" 50% CPU utilization. :p

Malcolm

23rd March 2008, 19:37

No offensive about your overclocking skills there, but I would seriously back off on that E6600 overclock and better yet, test it without any overclocking before hitting DGIndex as the source of the problem.

To me it looks like your having problems with your Southbridge based on the overclock.

Hi signal,
thank you for sharing your numbers & screenshots.
The overclocked system you were referring to was jeffs. My system is a C2D 4400 @ 3.15GHz, 2 GB DDR2-800 RAM (@840 MHz) 4-4-4-12.
O.k. it's overclocked too, but not that much. Still, i checked the results with a non-overlocked cpu/ram (2GHz, 800MHz) as you have proposed but i got the same results (around 20 MB/s). So i don't think the problem is because of overclocking. You see, i have transferrates of over 80 MB/s when copying from one hd to the other. Or as a different example when multiplexing i also get 'normal' I/O rates in the range what i would expect from my hd (55MB/s reads + 55MB/s writes).

Nonetheless it's totally beyond me why DGIndex is so slow. Or the other way, why it's so fast on your system. Are you sure the file wasn't coming from window's cache instead of the hd?? I had the case myself that the first part of a file was in the filecache and I/O was that high. So would you please do me the favor and take a longer/bigger file like 2 hours / 5GB and feed this to DGIndex while watching Process Explorer? That would be great! What numbers do you get in the last third when processing the file?

A second question: what did you choose in Process Explorer under 'View -> Update Speed'? 1 second??

When playing back with F5 or F6, DGIndex decodes the stream for display. In this case cpu usage is at 50% (1 core). The speed is then limited by the cpu, not the hd. (Btw. i get around 150 fps (Information windows) and around 3 MB/s in Process Explorer). To max out I/O you have to choose F4! Because then DGIndex does *not* decode the stream for display (cpu usage <20%).

Thanks a lot and greetings,
Malcolm

Guest

23rd March 2008, 23:19

I tried the experiment using 1.5.0 RC3.

My destination drive HD Tach result is 55MB/s -> 35MB/s.

My sustained IO rate was 45 MB/s and remained steady during the entire Save Project operation.

Malcolm

24th March 2008, 13:07

@neuron2
Thanks :thanks:

I've made an important discovery yesterday: After scratching my head for the umpteenth times, i remembered a presumption i had from the very beginning: Maybe this all has to do with blocksize when accessing the data on hd.
So i dug out a tool i know for long time from german magazine c't. This tool is able to monitor exactly what's going on when accessing your harddisk. It measures & displays among other things current requested blocksize as well as the (logical) position of the head on the hd! (This is a really cool tool if you ever have the need to look for things like that) Here's the link: HDiskPerf 1.0.6 (http://translate.google.com/translate?u=http%3A%2F%2Fwww.withopf.com%2Ftools%2Fhdiskperf%2F&langpair=de%7Cen)
Here's a screenshot when running DGIndex:
http://img80.imageshack.us/img80/3618/dgindexhdiskperfmm3.th.png (http://img80.imageshack.us/my.php?image=dgindexhdiskperfmm3.png)
You can see, there's a sharp drop in the green graph (current blocksize) in the upper area. This is where blocksize dropped from 64KB to 4KB (current blocksize is displayed in the middle green bar).
(Btw. You can see in the lower area that the file is unfragmented, since the head is moving sequentially over the hd surface.)
This drop *exactly* corresponds to the drop of the transferrate in the screenshot of Process Explorer i showed before where it goes from >70MB/s to 23MB/s. Here's the screenshot:
http://img166.imageshack.us/img166/7039/dgindex4ou6.th.png (http://img166.imageshack.us/my.php?image=dgindex4ou6.png)

So, all this is because of the blocksize changing from 64KB to 4KB. Why?

I have looked into the sources of dgindex 1.4.9. Most of the times files are opened with _open(input_file, _O_RDONLY | _O_BINARY | _O_SEQUENTIAL) (initial_parse.cpp). There are several places in gui.cpp where you open the files with _open(cwd, _O_RDONLY | _O_BINARY) - so, no SEQUENTIAL flag. I don't know if this plays a role. Actually i don't know *what* exactly influences the blocksize when accessing data on hd. Since there are benchmark apps like HD Tune where you can choose the blocksize explicitly, you *can* influence this in your application. But then, maybe you can only read raw sectors from disk. Other possibility: It's determined by the disk.sys driver of windows.
So my finding today is, that this is not a problem of my system/hd/whatever. But it's application or driver behaviour.

Is someone willing to to test DGIndex again, when monitoring your hd with HDiskPerf and share the screenshot? 'signal' maybe? I'd be very interested to see if blocksize changes like on my pc.
Oh and btw.: I have few cases (1 out of 10) where the complete file is processed with a speed of >70MB/s. I suppose that blocksize stays at 64KB then...

@neuron2: Do you have an idea of how hd access in DGIndex can be optimized? (hd access with blocksize 64KB is ALWAYS faster than with 4KB). So if you could force this, probably every user of DGIndex would benefit.

Guest

24th March 2008, 14:10

So my finding today is, that this is not a problem of my system/hd/whatever. But it's application or driver behaviour. DGIndex opens input files with _O_SEQUENTIAL. I doubt it's relevant, though. DGIndex requests data in read operations of size 2048 bytes. It's entirely up to the operating system and HDD driver to decide how much to read at once from the disk. My guess is that its disk cache fills and then it has only to refill its cache with each subsequent operation. I would expect that to be the same on all systems and so I'm not persuaded by your theory of the cause for the slowdown. But that's all speculation of course.

Do you have an idea of how hd access in DGIndex can be optimized? (hd access with blocksize 64KB is ALWAYS faster than with 4KB). So if you could force this, probably every user of DGIndex would benefit. I could try reading a larger amount on each read operation. It may not be trivial though, because the design inherited from DVD2AVI has places where the buffer size is assumed to be the size of a DVD sector (2048 bytes). I'll experiment with it and report back.

Guest

24th March 2008, 20:29

I made a version with a 1MByte buffer. So it will request 1MByte chunks from the file system. Try it out and let us know if it changes anything.

http://neuron2.net/guest/DGIndex.zip

burfadel

24th March 2008, 20:43

Hi signal,
thank you for sharing your numbers & screenshots.
The overclocked system you were referring to was jeffs. My system is a C2D 4400 @ 3.15GHz, 2 GB DDR2-800 RAM (@840 MHz) 4-4-4-12.
O.k. it's overclocked too, but not that much.

I take it you mean 2.15Ghz? 3.15Ghz is a very large overclock for that CPU!

burfadel

24th March 2008, 20:51

I made a version with a 1MByte buffer. So it will request 1MByte chunks from the file system. Try it out and let us know if it changes anything.

http://neuron2.net/guest/DGIndex.zip

I tried indexing a m2v file (output from Projectx) with this version and your normal 1.50 RC3 from which this one is based.

The 1mb buffer version was faster on every run I did. which I did 3 each with both versions. The worst performance benefit was 14 percent over the normal version. Is it possible to give test versions with 2mb and 4mb buffers? I realise with the bigger buffer you get diminishing returns, but just out of interest.

Also, how does the audio and d2v file actually get written to the disk? I would assume the d2v file is saved to memory until the end since its small for maximum performance? (and if not maybe this can be a consideration). Whats the size of the audio buffer before its written to the disk, increasing this may decrease the time taken to index vob files for example? These are questions more than statements, just thought they may be of benefit!

Guest

24th March 2008, 21:31

Thank you for your preliminary results. They show that it will be worthwhile to embark upon a quest to answer your questions and others that occur to me. I'll work on it and report here.

Malcolm

25th March 2008, 01:09

@neuron2
thanks a thousand times!! :thanks::thanks::thanks:

It's so great that you created this special version of DGIndex!
And i didn't expected this so quickly! Otherwise i would have checked the forum earlier - that's for sure! :)

The results of my very first test with this version are already outstanding! Blocksize stayed at 64KB for the whole 6GB file. Also the transferrate fluctuated at around 51MB/s. that's a speed increase of factor 2.5! Great! Thanks again!
Here are the screenshots:
http://img527.imageshack.us/img527/7862/dgindex1mbbufferhdiskpemv0.th.png (http://img527.imageshack.us/my.php?image=dgindex1mbbufferhdiskpemv0.png) http://img413.imageshack.us/img413/6982/dgindex1mbbufferprocessmn5.th.png (http://img413.imageshack.us/my.php?image=dgindex1mbbufferprocessmn5.png)

I also had files where the transferrate was lower (45 MB/s). Since cpu usage is still below 35% and because hd access is purely sequential (see HDiskPerf screenshot) i can imagine that there is still room for improvement. (My hd has sequential transferrates of >90MB/s @64KB blocksize). So i'm anxious to see how a 2mb / 4mb version will perform! :rolleyes:

Edit: Meanwhile i also had files where the speed was over 80MB/s. That's factor 4!
Memory usage was *exactly* 23.8MB regardless of the file i was currently processing. The normal version of DGIndex uses nearly the same amount (21.8 MB).
http://img413.imageshack.us/img413/5125/dgindex1mbbufferprocessft7.th.png (http://img413.imageshack.us/my.php?image=dgindex1mbbufferprocessft7.png)

Cheers!
Malcolm

Malcolm

25th March 2008, 01:21

I take it you mean 2.15Ghz? 3.15Ghz is a very large overclock for that CPU!

Hi burfadel,
no, it's really 3.15GHz. Actually it even runs stable at 3.25GHz (i had this running for 6 months). 3.15GHz is the 'conservative' configuration. :D
Even under heavy load (both cores maxed out (encoding, gaming)) the cpu temperature does not go beyond 60° C. The cpu fan turns between 900 and 1100 rpm then and believe me, you cannot hear it!

Malcolm

25th March 2008, 01:33

In your Vista screenshot...
I'd like to point out that i'm using Windows XP! It's only a desktop scheme, not Vista what you see in the screenshots!
Maybe the OS this is of importance when examining/comparing the behaviour of DGIndex.

Guest

25th March 2008, 02:20

I've been experimenting and I find the greatest performance increase for my system (single core FX-55) is when the buffer size is exactly 64KBytes. As I make it bigger strange interactions start to occur with the OS and disk caching and the performance suffers, and when I make it 4MBytes it becomes unstable and hangs.

So I'll likely regression test this thoroughly and then make a new RC4 at 64KB buffer size.

burfadel

25th March 2008, 02:28

Whats the current buffer size of dgindex? maybe it depends on the OS, driver, chipset etc etc? all I know is that the 1mb version is faster than the current RC3, I've done some more tests!

This is on a C2D E6600/p35 chipset on Vista x64.

Guest

25th March 2008, 02:38

Whats the current buffer size of dgindex? http://forum.doom9.org/showthread.php?p=1116577#post1116577

Please re-download the test version. It now has a 64K buffer. Your test results will be appreciated.

http://neuron2.net/guest/DGIndex.zip

burfadel

25th March 2008, 03:18

Malcolm

25th March 2008, 10:50

Please re-download the test version. It now has a 64K buffer. Your test results will be appreciated.http://neuron2.net/guest/DGIndex.zip

Hi neuron2,
i will test the new version as soon as i can. I'm at work right now, so it has to wait until evening.
Maybe it's a good idea to make the buffer size configurable. Preferably in the gui under options. :cool:

Guest

25th March 2008, 13:19

Just did 3 tests each with the normal and 64kb buffer version on the same file as before. The 64kb buffer was faster, but only slightly. The 1mb version was still faster! (I even retested with it again). Things get very slow for me with large buffers and I don't know why. But I have a single core. That may be significant. I will explore making the buffer size configurable.

Guest

25th March 2008, 14:01

Redownload the test version linked above. It now has a configurable buffer size.

First open the new DGIndex and immediately close it. Then edit the last line of the INI file as desired before opening DGIndex for use with your new buffer size.

I'll be interested in your results.

Note: Excessively large values may adversely impact program stability and GUI usability.

Guest

25th March 2008, 14:21

Oops. Download it again if you already have. It had a bug.

BTW, powers of 2 recommended for the buffer size.

Taurus

25th March 2008, 17:14

mmh, just a quick and dirty test.
I've opened a 2 gig mpeg2 file.
No audio extraction, no other fancy stuff, just plain d2v project.
Old single core Athlon XP 2600.
@ 64kb buffer I get almost the same timings as in RC3.
Everything above or much under this threshold gave me slowdowns or hickups:p.
Summarize: No advantage by changing the buffer @ my side :confused:
Should I test with audio and video demuxing?

Guest

25th March 2008, 17:17

I don't get it either. I do get a small improvement of about 10% with the 64KB buffer.

I think it may have something to do with the threading on dual core processors.

Malcolm

25th March 2008, 21:33

Ok, here are my results so far with RC4:
I used 4 DVB-S recordings which were already demuxed to pure m2v streams. First result is from RC3 for comparison. Other results are RC4 with 64k buffer, 1mb buffer and 4mb buffer. The reported numbers are the time for processing that DGIndex prints in its information window.

video A: 5.80GB
RC3: 3:13 (min:sec)
64k: 1:44
1mb: 1:51
4mb: 1:44

video B: 4.27GB
RC3: 2:50
64k: 1:11
1mb: 0:58
4mb: 1:00

video C: 4.57GB
RC3: 2:52
64k: 1:21
1mb: 1:19
4mb: 1:11

video D: 4.33GB
RC3: 2:45
64k: 1:14
1mb: 1:24
4mb: 1:18

- Processing speed has doubled when going from RC3 to RC4
- buffer size doesn't influence processing speed much
- With buffer size 64kb, Process Explorer reported a fairly constant speed of around 60MB/s
- With higher buffer sizes, Process Explorer reported fluctuating speeds between 45MB/s and 85MB/s. (frequency 1 second).
- IIRC HDiskPerf reported higher 'I/O usage' (firt blue bar) of around 90% with 1mb/4mb buffer than with 64kb buffer (35% usage). But this has changed later. So i'm unsure if this has really something to do with buffer size.

I couldn't find any instabilities, slowdowns or similar with buffer sizes of 1mb and 4mb.

My conclusion:
64kb buffer size seems to be a very good 'save' setting.

Thanks neuron2 for your work! :thanks:

Guest

25th March 2008, 22:05

My conclusion:
64kb buffer size seems to be a very good 'save' setting. Yup, that was my conclusion too. Thank you for your testing.

Taurus

25th March 2008, 23:03

My findings:
I really don't get it:confused::p
So I did the same tests on an AMD Athlon XP 2200+
Filesize: 4.5 Gigs vob files
RC3: 2.29 sec
RC4: 2.28 sec @64k buffer (just normal fluctuation)
Yes, and I double and triple checked the results :cool:

What makes me wonder:
Why is Malcolm's machine running so slow?
This one: WinXP SP2, 512 RAM, slow harddisk, Antivirus running and many background tasks, so just a normal User frontend.
His machine should crunch the numbers in 1/4 of the time or even less.
I guess more observation is needed.
No benefit in the 64k tweak for older machines..

Malcolm

26th March 2008, 00:15

@Taurus,
DGIndex really is only I/O (=hd speed) limited when creating the d2v file. the cpu is only used to a lower degree. So i have no advantage over your system even with a C2D @3.15GHz.

If you compare your numbers with mine you can see that with RC3 i need approximately the same time like you for a file of around 4.5GB (2.5 minutes). So i'm not really (that much) slower!
But with RC4 i only need 1.1 minutes for 4.5GB. I would assume that my hd is roughly double as fast as yours when doing sequential reads. (Western Digital 750GB).

Taurus

26th March 2008, 09:15

@Malcolm: Yeah, you're right.
I was just wondering, why a state of the art (day) machine is not running like a turbo compared to this really old one of mine.

Thanks

Underground78

26th March 2008, 19:55

I made a quick test on a 2 GB video (a ts file) (CPU : Athlon 64 X2 6000+) :

RC3 : 1 min 44
RC4 64k : 0 min 33
RC4 1024k : 0 min 35
RC4 2048k : 0 min 45

The speed improvement is really good I think !

hank315

27th March 2008, 12:20

Maybe it's just me but I'm getting errors with buffer sizes > 16k.
It seems to run OK but the generated d2v file contains errors.
Tests were done on a regular DVD rip with 7 VOBs, 6 GB total size.

2048 buffer:
900 5 5 1071550464 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1072138240 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1072668672 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1073201152 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1073713152 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 505856 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 1019904 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 1542144 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 2121728 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2

65536 buffer:
900 5 5 1071550464 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1072138240 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1072668672 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1073201152 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 -26624 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 505856 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 1019904 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 1542144 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 2121728 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2

Guest

27th March 2008, 14:06

Thanks for pointing that out, Hank. I'll investigate. As I mentioned, there were some design aspects that relied on a 2048B buffer. I may not have caught them all.

Taurus

27th March 2008, 17:41

About hank's observation:
Re-tested 3 DVD rips about 4.3 gigs each:
No difference/discrepancy between @64k buffer and default RC3 in d2v's here.

hank315

27th March 2008, 18:09

I'll investigate.
Thanks :)

Some more info, when the buffer size increases the number of negative values increases and it will happen earlier but always at a VOB switch.

My timings for the 6GB source:
2k 1:44
4k 1:46
8k 1:44
16k 1:44
32k 1:46
64k 1:49
128k 1:55
256k 1:50
512k 3:13
1024k 6:12
2048k 6:30
4096k 5:12

Intel C2D E6600, 2GB memory, Seagate 500GB, Windows XP SP2
Seems a larger buffer doesn't improve speed on my system...

@Taurus
With a 64k buffer I got the first error at the 6-->7 VOB switch, before that the d2v file was perfectly OK.
Could you try a larger source and buffer?

Zep

27th March 2008, 20:06

Guest

27th March 2008, 20:11

I understand the cause for this. It's actually a pre-existing design deficiency. I think that the differing buffer size just changes the statistics of it. I'll have to cogitate on the right solution.

The problem occurs when there is a PES packet start near the end of the file and then the I frame of that packet is on the next file.

Hank, would it be possible for you to give me cuts of the end of VOB x and the start of VOB y, where the x->y file transition is the problematic one? Thank you.

BTW, the only symptom of this would be bad random access to the affected GOP. Linear decoding through it will proceed normally.

Taurus

27th March 2008, 20:31

Bingo!
@Hank
You are right. Look here:
@64k
900 5 5 1072445440 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 5 1073012736 0 1 28 32 32 92 b2 b2 a2
900 5 5 1073260544 0 1 28 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 6 2048 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 b2 a2
900 5 6 587776 0 1 28 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 6 1083392 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2

@1024k
900 5 5 1071818752 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 5 1072445440 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 6 -630784 0 1 28 32 32 92 b2 b2 a2
900 5 6 -382976 0 1 28 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 6 2048 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 b2 a2
900 5 6 587776 0 1 28 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
At 64k everything is fine at my side.
1024k (the only one I tested @ higher buffer) is burked.
6 gigs/7vobs

@Zep: My hd's are formated to 4k. This might explain why I'm not getting a speed advantage by switching to 64k.

burfadel

28th March 2008, 04:27

what did everyone format their hard drives block size too? It makes a huge difference on large video files and why I have my video edit and demux drives at 64K and my other drives at 4K (the drives that have a lot of little files. i.e. my main and backup boot drives)

reading and writing is faster straight up but also you get a speed improvement because file fragmentation is a lot less, especially when short on disk space due to all the .ts files like in my case.

My guess is if DGindex is at 64k and you format at 64K you hit a sweet spot. Of course testing is needed to make sure :D

NTFS default block size is 4kb, which is still the case at 500gb. Not sure without looking it up at which point it increases, but most people would find its 4kb unless they decided to use a different block size.

I notice that with exfat you can have a block size of 32mb, now thats some serious wastage!

Zep

28th March 2008, 19:58

@Hank

@Zep: My hd's are formated to 4k. This might explain why I'm not getting a speed advantage by switching to 64k.

I saw a good speed up with R4 and 64K buffer on the 64K format drives and a smaller speed up on the 4k drives. All speeds are better with 64K buffer so it is a win all around.

Zep

28th March 2008, 20:08

NTFS default block size is 4kb, which is still the case at 500gb. Not sure without looking it up at which point it increases, but most people would find its 4kb unless they decided to use a different block size.

I notice that with exfat you can have a block size of 32mb, now thats some serious wastage!

yes you have to use CLI format and pass the -A option IIRC. I just wondered how many did that like me cause everyone seems to be into speed tweaks :D

yeah 32mb is a bit much lol However, on a drive that is just used to demux and edit large video files it may be worth it. Dunno never tried it.

burfadel

28th March 2008, 20:52

Ummmm, has anyone noticed that the windows cacheing system may be affecting the results...! I did some testing, then some more testing and now always on the first run no matter what file or buffer size is selected its slow, and then the second run, same or different buffer size, its significantly faster! ...

Guest

28th March 2008, 20:56

Yes, of course. That's why I always do it a couple times and then time it.

Malcolm

28th March 2008, 21:32

Yes, of course. That's why I always do it a couple times and then time it.

Hmm,
because of the caching, i read the files only once(!) for testing. So i can be sure they are NOT in the cache! Else you won't measure the hd access but only the main memory access! That way you'll never see a difference between buffer sizes in DGIndex!

Taurus

28th March 2008, 21:35

Yes, of course. That's why I always do it a couple times and then time it.
Same here.
But on real big files there should not be a speedup, cos OS can't catch up all the data from memory and must read from HD again.
My 2 Cents, correct me if I'm wrong.
How to flush the cache without rebooting?

hank315

28th March 2008, 21:38

Neuron, here's a link with the last 8 MB and first 8 MB of the VOB transition which is problematic.
Also included all d2v files with buffer sizes in the range of 2K to 4096k.
It shows at larger buffer sizes all VOB transitions have those negative values.
BTW, source is Troy, DVD PAL version.

link: http://www.mediafire.com/?ss3ym1yzygd

Guest

28th March 2008, 21:43

That's great, Hank. Thank you. I've been ripping DVDs for hours trying to make a bad file transition, without success. This will hopefully let me duplicate the issue.

burfadel

28th March 2008, 22:19

Maybe Windows cacheing can be used to the advantage of Dgindex? instead of relying on buffered reading, it could be cached reading from the Windows cache? So have the d2v file written to memory first, the audio written to the Windows cache and then always have say, 128mb of the current file in memory? then index from the memory? The indexing can continue in the memory as the current audio data is written to the disk via another thread...

That would probably be too complex to implement I know, but it sounds good in principle!

Guest

29th March 2008, 02:46

It's too complex, because I'd need different buffering schemes for play/preview versus indexing. Also, even for indexing, we've seen that reading large buffers actually slows thing down.

Malcolm

29th March 2008, 02:50

How to flush the cache without rebooting?
If you process 2 MPEG files with DGIndex which are larger than 1GB, then reading the second file will push the first one out of the cache.
The memory that Windows spends for caching disk access depends on your RAM size. It can get very large if you have no other app running, but normally it's <500MB.

Taurus

29th March 2008, 10:28

As a workaround I've used three different files at almost 4.5gigs each and did the testing circling between them :).
So I made sure that the results are correct.
@neuron:
If you're in need for another broken sample, just call me.
It's LotR SE PAL Version 6gigs/7vobs, see 13 posts above.
Out for the weekend, back tomorrow afternoon.

burfadel

29th March 2008, 11:28

What about a size of 4096, 8192, 16384 ... :)

Inventive Software

29th March 2008, 23:02

1.7 GB elementary MPEG-2 file took 1 minute to index. Same file with RC3 took 1m30. Good work. :)

Guest

31st March 2008, 23:01

While it's nice to have a performance improvement, we can't get it at the expense of incorrect indexing. It turns out that the existing design only works correctly at 2048 buffer size.

What happens is that with a larger buffer you now have the opportunity for the buffer to span the end of one file and the start of the next. Then the indexing goes haywire because it assumes that the entire contents correspond to only one file. In practice files that are used with DGIndex are all multiples of 2048 bytes, so this never was a possibility before. Even if you ran into a file that wasn't a multiple of 2048, the portion of the buffer that would be indexed incorrectly is 1024 bytes on average, and the likliehood of an indexable event being in there is small. So, in practice, it just doesn't ever happen. With a bigger buffer, it becomes likely, as we have seen.

There is no easy fix to the existing design. I'm considering possible solutions. But I'm thinking of releasing 1.5.0 with the existing scheme and saving this revision for the next release.

There may be a little kludge that will work, though. When I fill the buffer and don't get a full buffer out of the file, I can temporarily pretend that the buffer size is the size of the data I read, and then reset it back to the configured buffer size for the next read. Then we would never have two files together in the buffer. If this works out, I'll include it in 1.5.0.

Malcolm

1st April 2008, 09:58

But I'm thinking of releasing 1.5.0 with the existing scheme and saving this revision for the next release.
I second that. But if your workaround (looks foolproof to me) works, it would be great if it's in 1.5.0.

kypec

1st April 2008, 11:52

What happens is that with a larger buffer you now have the opportunity for the buffer to span the end of one file and the start of the next. Then the indexing goes haywire because it assumes that the entire contents correspond to only one file.
Is it to say that as long as I work with single VOB file projects this buffer size issue is not concerning me at all? Is it affecting only multiple VOB files projects?

Guest

1st April 2008, 13:33

Is it to say that as long as I work with single VOB file projects this buffer size issue is not concerning me at all? Is it affecting only multiple VOB files projects?
Yes. If you have only a single file, you can't have the bad case of multiple files in the buffer at once.

puffpio

1st May 2008, 08:04

i dont know what language dgindex is written in, but if the files are read as streams and then loaded into the 2k buffers, could you just wrap the filestreams in a bufferedstream? i'm sure there exists some class library for bufferedstreams in whatever language is used. in that way your code doesn't need to change much

puffpio

1st May 2008, 08:31

i started looking at your code

it looks like all the 2k reads are made via _donread which in turn calls the _read function to grab the 2k blocks.

instead of changing buffer sizes, perhaps _donread can be internally modified to have a 64kb or 1Mb buffer that _read's from the file.

masta_g_86

2nd May 2008, 14:17

Same here.
But on real big files there should not be a speedup, cos OS can't catch up all the data from memory and must read from HD again.
My 2 Cents, correct me if I'm wrong.
How to flush the cache without rebooting?

Taurus seems to be on the right track here.

I've been doing some episodic rips from TV shows lately. Each episode is just over 1GB. I have 2GB of RAM, so the ripped VOB stays resident in my system's RAM after the rip until I rip another episode, run a program that uses the VOB's RAM address space, or reset my machine.

If I jump straight over to DGIndex after ripping, the whole indexing process flies by. If I reset my machine and THEN index, I get comparatively poor results.

The reason for the poor results is that since the VOB is no longer also available in RAM, it must be read exclusively from the hard drive. Also, as it's being read from the hard drive, DGIndex is writing the .d2v file, slowing the process.

Here are some methods to improve performance:

1. Index your ripped movie IMMEDIATELY after ripping it.
2. Rip your movie to a secondary hard drive and save your .d2v to another hard drive that is also on a separate channel.
3. Recode DGIndex such that it only writes the .d2v file to the user's hard drive AFTER the indexing process is complete.

These can potentially provide the biggest speed improvements in my experience. Take note though that a heavily fragmented VOB file will still slow the process if you didn't immediately index after rip/indexed after a reset. Also take note that if the VOB is larger than your available system RAM the last portion of it will generally still exist in the address space, so while the indexing process may start slow, it should finish quickly.