Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > General > DVD2AVI / DGIndex

Reply
 
Thread Tools Search this Thread Display Modes
Old 3rd July 2010, 01:40   #1881  |  Link
adiabatic
Registered User
 
adiabatic's Avatar
 
Join Date: Oct 2008
Posts: 17
Quote:
Originally Posted by neuron2 View Post
Would some kind souls please test the 32-bit DGIndexNV version linked below to see if it delivers faster indexing without breaking anything. I'm getting indexing about twice as fast as the 2018 release version on initial testing. Thank you.

http://neuron2.net/misc/DGIndexNV_perftest.zip
"The requested URL /misc/DGIndexNV_perftest.zip was not found on this server"

No need for further testing? Your own results seem promising... I'm willing to participate if you still need it.

Last edited by adiabatic; 3rd July 2010 at 01:43.
adiabatic is offline   Reply With Quote
Old 3rd July 2010, 01:52   #1882  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,924
Quote:
Originally Posted by adiabatic View Post
"The requested URL /misc/DGIndexNV_perftest.zip was not found on this server"

No need for further testing? Your own results seem promising... I'm willing to participate if you still need it.
That one did not use low-level IO. I'll post the new one shortly. Thank you for your willingness to assist in the testing.
Guest is offline   Reply With Quote
Old 3rd July 2010, 02:31   #1883  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,924
Quote:
Originally Posted by neuron2 View Post
I'll post the new one shortly.
OK, it's there. Use the same link as before.
Guest is offline   Reply With Quote
Old 3rd July 2010, 03:04   #1884  |  Link
adiabatic
Registered User
 
adiabatic's Avatar
 
Join Date: Oct 2008
Posts: 17
Hmmm. It gives slightly slower performance on a 11.5 GB file. Both tests repeated twice.

My system:
Windows 7 Pro 64-bit
Intel Core i7-920
GTX275 196.21 drivers
80GB Intel X25-M G2 SSD

2018 release version - 32-bit
Stream Type: Transport [188]
Video Type: MPEG2
Profile: main@high
Coded Size: 1920x1088
Display Size: 1920x1080
Aspect Ratio: 16:9 [3]
Frame Rate: 29.970030 fps
Colorimetry: BT.709*
Frame Structure:
Frame Type:
Coded Number: 200807
Playback Number: 200810
Frame Repeats: 0
Field Repeats: 7
Bitrate: 11.926
Bitrate (Avg): 13.750
Bitrate (Max): 19.111
Audio Stream: 14: AC3 3/2 48 384
Elapsed: 0:04:13
Remain: 0:00:00
FPS:
Info: Finished!

2018 test
Stream Type: Transport [188]
Video Type: MPEG2
Profile: main@high
Coded Size: 1920x1088
Display Size: 1920x1080
Aspect Ratio: 16:9 [3]
Frame Rate: 29.970030 fps
Colorimetry: BT.709*
Frame Structure:
Frame Type:
Coded Number: 200807
Playback Number: 200810
Frame Repeats: 0
Field Repeats: 7
Bitrate: 11.926
Bitrate (Avg): 13.750
Bitrate (Max): 19.111
Audio Stream: 14: AC3 3/2 48 384
Elapsed: 0:05:03
Remain: 0:00:00
FPS:
Info: Finished!


<edit> looks like I'm behind on driver versions.... updating and repeating tests.

Last edited by adiabatic; 3rd July 2010 at 03:46.
adiabatic is offline   Reply With Quote
Old 3rd July 2010, 03:12   #1885  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,924
That's hard to believe.

I'm testing on WinXP 32-bit with mechanical drives. I'll try it on my i7-960 64-bit.

I see you have an SSD. That is probably going to be a lot different from a mechanical drive. Your read rate is probably four times what I get. That makes the actual buffer size important because the smaller the buffer the more calls are made. I could try increasing the buffer in multiples of 4096 until it starts to deteriorate performance.

If I have to choose, I will optimize for mechanical drives, at least at this point in time.

Can you test on a mechanical drive?

Last edited by Guest; 3rd July 2010 at 03:22.
Guest is offline   Reply With Quote
Old 3rd July 2010, 03:25   #1886  |  Link
adiabatic
Registered User
 
adiabatic's Avatar
 
Join Date: Oct 2008
Posts: 17
OK, I'm also getting the same results with the SSD and the latest Nvidia drivers.

I'll now re-run the tests with the conventional HD.

Last edited by adiabatic; 3rd July 2010 at 03:30.
adiabatic is offline   Reply With Quote
Old 3rd July 2010, 03:48   #1887  |  Link
adiabatic
Registered User
 
adiabatic's Avatar
 
Join Date: Oct 2008
Posts: 17
OK mechanical HD (7200rpm Western Digital SATA).

With the same test file as above.
2018 "release" - 5:09
2018 "test" - 6:23 (during this test I opened Resource Monitor and I was reading and writing at about 28 MB/sec for both)

Just to be clear so I don't lead you down a false path... your earlier comment about caching related to read caching, right? I have write caching enabled for my drives which was the default Windows setting for the drives.

I did this test with the source file and demux output on the mechanical HD. Does it matter to your testing if my pagefile and the DGIndex folder is on the SSD?

I'm heading out for an hour or so now... but am willing to test more later.

Last edited by adiabatic; 3rd July 2010 at 03:53.
adiabatic is offline   Reply With Quote
Old 3rd July 2010, 03:54   #1888  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,924
Jeez, don't have resource monitor opened for the test.

That's just incredible. I can't believe it. I get a 3.5 times speedup on my system!

Let's wait and see what happens with other people.

Are you running the X32 versions in both cases?

I would so love to get to the bottom of this.

Last edited by Guest; 3rd July 2010 at 04:13.
Guest is offline   Reply With Quote
Old 3rd July 2010, 04:15   #1889  |  Link
adiabatic
Registered User
 
adiabatic's Avatar
 
Join Date: Oct 2008
Posts: 17
Quote:
Originally Posted by neuron2 View Post
Jeez, don't have resource monitor opened for the test.

That's just ridiculous. I can't believe it. I get a 4 times speedup on my system.

Let's wait and see what happens with other people.
BTW, I just popped Resource Monitor open for a minute.

Yes, x32 (double checked in task manager (*32)). Your x64 binaries are tucked away in another folder. Feel free to ask other questions or wait for other testers. The last thing I want to do is be making an error here that wastes your time.

I agree, let's see what others get. Later tonight I'll run this test on a more plain-vanilla machine in the house (mech. HDs, Core2Duo 2.0, Nvidia 8600GT card).

Though since I can't let go of this bone... for fun I've moved my dgdecnv folder and pagefile to the mech. HD. The source and target folder are also on the same HD. The only part of this equation that should now be on the SSD are core Windows files and Nvidia driver files.

(no Resource Monitor this time!)
2018 "release" - 5:11
2018 "test" - 6:08

The only thing I'm doing on this system while the demux is running is light web-browsing with Firefox but I've been doing that constantly through all tests with all variations of testing tonight.

Now I am really leaving the house for a little while

Last edited by adiabatic; 3rd July 2010 at 04:18.
adiabatic is offline   Reply With Quote
Old 3rd July 2010, 04:22   #1890  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,924
My results:

2018 - 21:30
2018 test - 5:51

Look at the graph in here:

http://pisa.ucsd.edu/cse125/2006/Pap...ing_in_C++.pdf

It shows an approximate 3-4 times speedup from using low-level IO versus C IO, consistent with my findings.

Last edited by Guest; 3rd July 2010 at 04:25.
Guest is offline   Reply With Quote
Old 3rd July 2010, 04:35   #1891  |  Link
linyx
Un-Registered User
 
linyx's Avatar
 
Join Date: Mar 2008
Location: Audio Stream - 0x80
Posts: 341
I just tried out the test version, and am seeing errors like this one with it (lots, 5 out of 7 streams tested).
These errors do not show up with the 64-Bit or the 32-Bit of 2018. Want a stream?
linyx is offline   Reply With Quote
Old 3rd July 2010, 04:42   #1892  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,924
Quote:
Originally Posted by linyx View Post
Want a stream?
Yes, please. Also diff the DGI files. They would have to be different. My tests show no differences between release 2018 and test 2018 DGI files (except for the path at the top).

Any timing results to report?

Last edited by Guest; 3rd July 2010 at 05:58.
Guest is offline   Reply With Quote
Old 3rd July 2010, 04:51   #1893  |  Link
woah!
Registered User
 
Join Date: Oct 2003
Posts: 435
My results 24.5GB file:

2018 - Elapsed: 0:13:36
2018 test - Elapsed: 0:05:24

the test version is much better here
woah! is offline   Reply With Quote
Old 3rd July 2010, 05:04   #1894  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,924
I'm not imagining it then.
Guest is offline   Reply With Quote
Old 3rd July 2010, 05:09   #1895  |  Link
woah!
Registered User
 
Join Date: Oct 2003
Posts: 435
but vc1 files are not showing as much speedup as h264 bluray files. 8.5GB file

2018 - Elapsed: 0:02:17
2018 test - Elapsed: 0:01:59

will do some more files here for you..
woah! is offline   Reply With Quote
Old 3rd July 2010, 05:49   #1896  |  Link
adiabatic
Registered User
 
adiabatic's Avatar
 
Join Date: Oct 2008
Posts: 17
OK, now I'm on my "basic" system.

32-bit Vista Home Premium
Core 2 Duo 2.0
8600GT video card (latest drivers)
good ol' mechanical HDs

Same test file as before

2018 - 10:16
2018 test - 12:26

It must be something unique to this clip. Video source is MPEG-2 TS 1080i with AC3 audio from Motorola DCT-3416 cable box. It's been edited with VideoRedo and (I'm not 100% sure) probably run through MPEG2Repair before editing.

sample: http://www.mediafire.com/file/4nm5kn...crew_donald.ts

Last edited by adiabatic; 3rd July 2010 at 05:56.
adiabatic is offline   Reply With Quote
Old 3rd July 2010, 08:44   #1897  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,301
Great work neuron2, per usual! I'm testing now on my system.

System Spec
Core i7 Q820 (mobile, 4 cores + HT)
4GB DDR3
Quadro FX 2800M
2x 250GB 2.5" HDD RAID 0
Windows 7 x64

Source Spec
1080p24 H.264 HP @ L5
45mbps average, encoded by x264 1542
CAVLC, Deblocking, B-Frames
TS Container (remuxed from an MKV via TSMuxeR 1.10.6)
24436 frames (just under 17 mins)

1018 vanilla
93 seconds = 262.75fps = ~11x realtime = ~ 495mbps

1018 perftest
136 seconds = 179.68fps = ~7.5x realtime = ~ 337mbps

So.. maybe I'm stupid and am being taunted by Windows caching as you warned us about, neuron2... is my sample just too short to avoid this? I tried deleting the source after indexing and muxing a fresh copy, but got similar results.

Any other suggestions?

BTW.. I did all this testing over UltraVNC. No broken CUDA! Slick!

Derek

Last edited by Blue_MiSfit; 3rd July 2010 at 08:46.
Blue_MiSfit is offline   Reply With Quote
Old 3rd July 2010, 12:48   #1898  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,924
Looks like it might be a Win7 versus WinXP issue.

@woah!

Please give your system details.
Guest is offline   Reply With Quote
Old 3rd July 2010, 13:34   #1899  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,924
My results on my i7-980X Win7 64bit with VelociRaptor (same project as my WinXP results).

2018 - 3:16
2018 perf - 3:12

This supports my WinXP versus Vista/Win7 theory. I'm tending towards going with the perf version due to the big gains it gives XP users. I don't know why some of you see drops on Vista/Win7; could it be the RAID 0?

@linyx

Need your stream and your diff results, please.

Last edited by Guest; 3rd July 2010 at 14:02.
Guest is offline   Reply With Quote
Old 3rd July 2010, 17:24   #1900  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by neuron2 View Post
That's hard to believe.

I'm testing on WinXP 32-bit with mechanical drives. I'll try it on my i7-960 64-bit.

I see you have an SSD. That is probably going to be a lot different from a mechanical drive. Your read rate is probably four times what I get. That makes the actual buffer size important because the smaller the buffer the more calls are made. I could try increasing the buffer in multiples of 4096 until it starts to deteriorate performance.

If I have to choose, I will optimize for mechanical drives, at least at this point in time.

Can you test on a mechanical drive?
I have SSD also. In fact my system is basically a clone of adiabatic's

can you make the buffer size a var we can pass? i.e.Default 1=4096 internally , 2=8192 internally etc... then everyone can fine tune to their own PC.
Zep is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:10.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2017, vBulletin Solutions Inc.