DGIndex - performance

[Taurus]

@Malcolm: Yeah, you're right.
I was just wondering, why a state of the art (day) machine is not running like a turbo compared to this really old one of mine.

Thanks

[Underground78]

I made a quick test on a 2 GB video (a ts file) (CPU : Athlon 64 X2 6000+) :

RC3 : 1 min 44
RC4 64k : 0 min 33
RC4 1024k : 0 min 35
RC4 2048k : 0 min 45

The speed improvement is really good I think !

[hank315]

Maybe it's just me but I'm getting errors with buffer sizes > 16k.
It seems to run OK but the generated d2v file contains errors.
Tests were done on a regular DVD rip with 7 VOBs, 6 GB total size.

2048 buffer:
900 5 5 1071550464 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1072138240 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1072668672 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1073201152 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1073713152 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 505856 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 1019904 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 1542144 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 2121728 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2


65536 buffer:
900 5 5 1071550464 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1072138240 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1072668672 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 5 1073201152 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 -26624 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 505856 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 1019904 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 1542144 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2
900 5 6 2121728 0 2 15 72 72 d2 f2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2

[Guest]

Thanks for pointing that out, Hank. I'll investigate. As I mentioned, there were some design aspects that relied on a 2048B buffer. I may not have caught them all.

[Taurus]

About hank's observation:
Re-tested 3 DVD rips about 4.3 gigs each:
No difference/discrepancy between @64k buffer and default RC3 in d2v's here.

[hank315]

I'll investigate.
Thanks :)

Some more info, when the buffer size increases the number of negative values increases and it will happen earlier but always at a VOB switch.

My timings for the 6GB source:
2k 1:44
4k 1:46
8k 1:44
16k 1:44
32k 1:46
64k 1:49
128k 1:55
256k 1:50
512k 3:13
1024k 6:12
2048k 6:30
4096k 5:12

Intel C2D E6600, 2GB memory, Seagate 500GB, Windows XP SP2
Seems a larger buffer doesn't improve speed on my system...

@Taurus
With a 64k buffer I got the first error at the 6-->7 VOB switch, before that the d2v file was perfectly OK.
Could you try a larger source and buffer?

[Zep]

what did everyone format their hard drives block size too? It makes a huge difference on large video files and why I have my video edit and demux drives at 64K and my other drives at 4K (the drives that have a lot of little files. i.e. my main and backup boot drives)

reading and writing is faster straight up but also you get a speed improvement because file fragmentation is a lot less, especially when short on disk space due to all the .ts files like in my case.

My guess is if DGindex is at 64k and you format at 64K you hit a sweet spot. Of course testing is needed to make sure :D

[Guest]

I understand the cause for this. It's actually a pre-existing design deficiency. I think that the differing buffer size just changes the statistics of it. I'll have to cogitate on the right solution.

The problem occurs when there is a PES packet start near the end of the file and then the I frame of that packet is on the next file.

Hank, would it be possible for you to give me cuts of the end of VOB x and the start of VOB y, where the x->y file transition is the problematic one? Thank you.

BTW, the only symptom of this would be bad random access to the affected GOP. Linear decoding through it will proceed normally.

[Taurus]

Bingo!
@Hank
You are right. Look here:
@64k
900 5 5 1072445440 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 5 1073012736 0 1 28 32 32 92 b2 b2 a2
900 5 5 1073260544 0 1 28 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 6 2048 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 b2 a2
900 5 6 587776 0 1 28 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 6 1083392 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2

@1024k
900 5 5 1071818752 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 5 1072445440 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 6 -630784 0 1 28 32 32 92 b2 b2 a2
900 5 6 -382976 0 1 28 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 6 2048 0 1 28 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 b2 a2
900 5 6 587776 0 1 28 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
At 64k everything is fine at my side.
1024k (the only one I tested @ higher buffer) is burked.
6 gigs/7vobs

@Zep: My hd's are formated to 4k. This might explain why I'm not getting a speed advantage by switching to 64k.

[burfadel]

what did everyone format their hard drives block size too? It makes a huge difference on large video files and why I have my video edit and demux drives at 64K and my other drives at 4K (the drives that have a lot of little files. i.e. my main and backup boot drives)

reading and writing is faster straight up but also you get a speed improvement because file fragmentation is a lot less, especially when short on disk space due to all the .ts files like in my case.

My guess is if DGindex is at 64k and you format at 64K you hit a sweet spot. Of course testing is needed to make sure :D

NTFS default block size is 4kb, which is still the case at 500gb. Not sure without looking it up at which point it increases, but most people would find its 4kb unless they decided to use a different block size.

I notice that with exfat you can have a block size of 32mb, now thats some serious wastage!

[Zep]

@Hank

@Zep: My hd's are formated to 4k. This might explain why I'm not getting a speed advantage by switching to 64k.

I saw a good speed up with R4 and 64K buffer on the 64K format drives and a smaller speed up on the 4k drives. All speeds are better with 64K buffer so it is a win all around.

[Zep]

NTFS default block size is 4kb, which is still the case at 500gb. Not sure without looking it up at which point it increases, but most people would find its 4kb unless they decided to use a different block size.

I notice that with exfat you can have a block size of 32mb, now thats some serious wastage!

yes you have to use CLI format and pass the -A option IIRC. I just wondered how many did that like me cause everyone seems to be into speed tweaks :D

yeah 32mb is a bit much lol However, on a drive that is just used to demux and edit large video files it may be worth it. Dunno never tried it.

[burfadel]

Ummmm, has anyone noticed that the windows cacheing system may be affecting the results...! I did some testing, then some more testing and now always on the first run no matter what file or buffer size is selected its slow, and then the second run, same or different buffer size, its significantly faster! ...

[Guest]

Yes, of course. That's why I always do it a couple times and then time it.

[Malcolm]

Yes, of course. That's why I always do it a couple times and then time it.

Hmm,
because of the caching, i read the files only once(!) for testing. So i can be sure they are NOT in the cache! Else you won't measure the hd access but only the main memory access! That way you'll never see a difference between buffer sizes in DGIndex!

[Taurus]

Yes, of course. That's why I always do it a couple times and then time it.
Same here.
But on real big files there should not be a speedup, cos OS can't catch up all the data from memory and must read from HD again.
My 2 Cents, correct me if I'm wrong.
How to flush the cache without rebooting?

[hank315]

Neuron, here's a link with the last 8 MB and first 8 MB of the VOB transition which is problematic.
Also included all d2v files with buffer sizes in the range of 2K to 4096k.
It shows at larger buffer sizes all VOB transitions have those negative values.
BTW, source is Troy, DVD PAL version.

link: http://www.mediafire.com/?ss3ym1yzygd

[Guest]

That's great, Hank. Thank you. I've been ripping DVDs for hours trying to make a bad file transition, without success. This will hopefully let me duplicate the issue.

[burfadel]

Maybe Windows cacheing can be used to the advantage of Dgindex? instead of relying on buffered reading, it could be cached reading from the Windows cache? So have the d2v file written to memory first, the audio written to the Windows cache and then always have say, 128mb of the current file in memory? then index from the memory? The indexing can continue in the memory as the current audio data is written to the disk via another thread...

That would probably be too complex to implement I know, but it sounds good in principle!

[Guest]

It's too complex, because I'd need different buffering schemes for play/preview versus indexing. Also, even for indexing, we've seen that reading large buffers actually slows thing down.

[Malcolm]

How to flush the cache without rebooting?
If you process 2 MPEG files with DGIndex which are larger than 1GB, then reading the second file will push the first one out of the cache.
The memory that Windows spends for caching disk access depends on your RAM size. It can get very large if you have no other app running, but normally it's <500MB.

[Taurus]

As a workaround I've used three different files at almost 4.5gigs each and did the testing circling between them :).
So I made sure that the results are correct.
@neuron:
If you're in need for another broken sample, just call me.
It's LotR SE PAL Version 6gigs/7vobs, see 13 posts above.
Out for the weekend, back tomorrow afternoon.

[burfadel]

What about a size of 4096, 8192, 16384 ... :)

[Inventive Software]

1.7 GB elementary MPEG-2 file took 1 minute to index. Same file with RC3 took 1m30. Good work. :)

[Guest]

While it's nice to have a performance improvement, we can't get it at the expense of incorrect indexing. It turns out that the existing design only works correctly at 2048 buffer size.

What happens is that with a larger buffer you now have the opportunity for the buffer to span the end of one file and the start of the next. Then the indexing goes haywire because it assumes that the entire contents correspond to only one file. In practice files that are used with DGIndex are all multiples of 2048 bytes, so this never was a possibility before. Even if you ran into a file that wasn't a multiple of 2048, the portion of the buffer that would be indexed incorrectly is 1024 bytes on average, and the likliehood of an indexable event being in there is small. So, in practice, it just doesn't ever happen. With a bigger buffer, it becomes likely, as we have seen.

There is no easy fix to the existing design. I'm considering possible solutions. But I'm thinking of releasing 1.5.0 with the existing scheme and saving this revision for the next release.

There may be a little kludge that will work, though. When I fill the buffer and don't get a full buffer out of the file, I can temporarily pretend that the buffer size is the size of the data I read, and then reset it back to the configured buffer size for the next read. Then we would never have two files together in the buffer. If this works out, I'll include it in 1.5.0.

[Malcolm]

But I'm thinking of releasing 1.5.0 with the existing scheme and saving this revision for the next release.
I second that. But if your workaround (looks foolproof to me) works, it would be great if it's in 1.5.0.

[kypec]

What happens is that with a larger buffer you now have the opportunity for the buffer to span the end of one file and the start of the next. Then the indexing goes haywire because it assumes that the entire contents correspond to only one file.
Is it to say that as long as I work with single VOB file projects this buffer size issue is not concerning me at all? Is it affecting only multiple VOB files projects?

[Guest]

Is it to say that as long as I work with single VOB file projects this buffer size issue is not concerning me at all? Is it affecting only multiple VOB files projects?
Yes. If you have only a single file, you can't have the bad case of multiple files in the buffer at once.

[puffpio]

i dont know what language dgindex is written in, but if the files are read as streams and then loaded into the 2k buffers, could you just wrap the filestreams in a bufferedstream? i'm sure there exists some class library for bufferedstreams in whatever language is used. in that way your code doesn't need to change much

[puffpio]

i started looking at your code

it looks like all the 2k reads are made via _donread which in turn calls the _read function to grab the 2k blocks.

instead of changing buffer sizes, perhaps _donread can be internally modified to have a 64kb or 1Mb buffer that _read's from the file.

[masta_g_86]

Same here.
But on real big files there should not be a speedup, cos OS can't catch up all the data from memory and must read from HD again.
My 2 Cents, correct me if I'm wrong.
How to flush the cache without rebooting?

Taurus seems to be on the right track here.

I've been doing some episodic rips from TV shows lately. Each episode is just over 1GB. I have 2GB of RAM, so the ripped VOB stays resident in my system's RAM after the rip until I rip another episode, run a program that uses the VOB's RAM address space, or reset my machine.

If I jump straight over to DGIndex after ripping, the whole indexing process flies by. If I reset my machine and THEN index, I get comparatively poor results.

The reason for the poor results is that since the VOB is no longer also available in RAM, it must be read exclusively from the hard drive. Also, as it's being read from the hard drive, DGIndex is writing the .d2v file, slowing the process.

Here are some methods to improve performance:

1. Index your ripped movie IMMEDIATELY after ripping it.
2. Rip your movie to a secondary hard drive and save your .d2v to another hard drive that is also on a separate channel.
3. Recode DGIndex such that it only writes the .d2v file to the user's hard drive AFTER the indexing process is complete.

These can potentially provide the biggest speed improvements in my experience. Take note though that a heavily fragmented VOB file will still slow the process if you didn't immediately index after rip/indexed after a reset. Also take note that if the VOB is larger than your available system RAM the last portion of it will generally still exist in the address space, so while the indexing process may start slow, it should finish quickly.

[puffpio]

#3 seems like a good idea

[puffpio]

i started looking at your code

it looks like all the 2k reads are made via _donread which in turn calls the _read function to grab the 2k blocks.

instead of changing buffer sizes, perhaps _donread can be internally modified to have a 64kb or 1Mb buffer that _read's from the file.
ispoke too soon..i looked more and the file pointer is constantly being moved around in other functions..

[masta_g_86]

ispoke too soon..i looked more and the file pointer is constantly being moved around in other functions..

So do you think it's possible for #3 to be implemented? I'm sure I've run across programs in the past that use that type of behavior (saving to the HDD after a read operation is complete).

[Guest]

Guys, don't worry about it. I know the internals very well and will give you performance improvements in 1.5.1.