DG NV tools - Page 95

adiabatic · 3rd July 2010, 01:40

Quote:

Originally Posted by neuron2

Would some kind souls please test the 32-bit DGIndexNV version linked below to see if it delivers faster indexing without breaking anything. I'm getting indexing about twice as fast as the 2018 release version on initial testing. Thank you.

http://neuron2.net/misc/DGIndexNV_perftest.zip

"The requested URL /misc/DGIndexNV_perftest.zip was not found on this server"

No need for further testing? Your own results seem promising... I'm willing to participate if you still need it.

Guest · 3rd July 2010, 01:52

Quote:

Originally Posted by adiabatic

"The requested URL /misc/DGIndexNV_perftest.zip was not found on this server"

No need for further testing? Your own results seem promising... I'm willing to participate if you still need it.

That one did not use low-level IO. I'll post the new one shortly. Thank you for your willingness to assist in the testing.

Guest · 3rd July 2010, 02:31

Quote:

Originally Posted by neuron2

I'll post the new one shortly.

OK, it's there. Use the same link as before.

adiabatic · 3rd July 2010, 03:04

Hmmm. It gives slightly slower performance on a 11.5 GB file. Both tests repeated twice.

My system:
Windows 7 Pro 64-bit
Intel Core i7-920
GTX275 196.21 drivers
80GB Intel X25-M G2 SSD

2018 release version - 32-bit
Stream Type: Transport [188]
Video Type: MPEG2
Profile: main@high
Coded Size: 1920x1088
Display Size: 1920x1080
Aspect Ratio: 16:9 [3]
Frame Rate: 29.970030 fps
Colorimetry: BT.709*
Frame Structure:
Frame Type:
Coded Number: 200807
Playback Number: 200810
Frame Repeats: 0
Field Repeats: 7
Bitrate: 11.926
Bitrate (Avg): 13.750
Bitrate (Max): 19.111
Audio Stream: 14: AC3 3/2 48 384
Elapsed: 0:04:13
Remain: 0:00:00
FPS:
Info: Finished!

2018 test
Stream Type: Transport [188]
Video Type: MPEG2
Profile: main@high
Coded Size: 1920x1088
Display Size: 1920x1080
Aspect Ratio: 16:9 [3]
Frame Rate: 29.970030 fps
Colorimetry: BT.709*
Frame Structure:
Frame Type:
Coded Number: 200807
Playback Number: 200810
Frame Repeats: 0
Field Repeats: 7
Bitrate: 11.926
Bitrate (Avg): 13.750
Bitrate (Max): 19.111
Audio Stream: 14: AC3 3/2 48 384
Elapsed: 0:05:03
Remain: 0:00:00
FPS:
Info: Finished!

<edit> looks like I'm behind on driver versions.... updating and repeating tests.

Guest · 3rd July 2010, 03:12

That's hard to believe.

I'm testing on WinXP 32-bit with mechanical drives. I'll try it on my i7-960 64-bit.

I see you have an SSD. That is probably going to be a lot different from a mechanical drive. Your read rate is probably four times what I get. That makes the actual buffer size important because the smaller the buffer the more calls are made. I could try increasing the buffer in multiples of 4096 until it starts to deteriorate performance.

If I have to choose, I will optimize for mechanical drives, at least at this point in time.

Can you test on a mechanical drive?

adiabatic · 3rd July 2010, 03:25

OK, I'm also getting the same results with the SSD and the latest Nvidia drivers.

I'll now re-run the tests with the conventional HD.

adiabatic · 3rd July 2010, 03:48

OK mechanical HD (7200rpm Western Digital SATA).

With the same test file as above.
2018 "release" - 5:09
2018 "test" - 6:23 (during this test I opened Resource Monitor and I was reading and writing at about 28 MB/sec for both)

Just to be clear so I don't lead you down a false path... your earlier comment about caching related to read caching, right? I have write caching enabled for my drives which was the default Windows setting for the drives.

I did this test with the source file and demux output on the mechanical HD. Does it matter to your testing if my pagefile and the DGIndex folder is on the SSD?

I'm heading out for an hour or so now... but am willing to test more later.

Guest · 3rd July 2010, 03:54

Jeez, don't have resource monitor opened for the test.

That's just incredible. I can't believe it. I get a 3.5 times speedup on my system!

Let's wait and see what happens with other people.

Are you running the X32 versions in both cases?

I would so love to get to the bottom of this.

adiabatic · 3rd July 2010, 04:15

Quote:

Originally Posted by neuron2

Jeez, don't have resource monitor opened for the test.

That's just ridiculous. I can't believe it. I get a 4 times speedup on my system.

Let's wait and see what happens with other people.

BTW, I just popped Resource Monitor open for a minute.

Yes, x32 (double checked in task manager (*32)). Your x64 binaries are tucked away in another folder. Feel free to ask other questions or wait for other testers. The last thing I want to do is be making an error here that wastes your time.

I agree, let's see what others get. Later tonight I'll run this test on a more plain-vanilla machine in the house (mech. HDs, Core2Duo 2.0, Nvidia 8600GT card).

Though since I can't let go of this bone... for fun I've moved my dgdecnv folder and pagefile to the mech. HD. The source and target folder are also on the same HD. The only part of this equation that should now be on the SSD are core Windows files and Nvidia driver files.

(no Resource Monitor this time!)
2018 "release" - 5:11
2018 "test" - 6:08

The only thing I'm doing on this system while the demux is running is light web-browsing with Firefox but I've been doing that constantly through all tests with all variations of testing tonight.

Now I am really leaving the house for a little while

Guest · 3rd July 2010, 04:22

My results:

2018 - 21:30
2018 test - 5:51

Look at the graph in here:

http://pisa.ucsd.edu/cse125/2006/Pap...ing_in_C++.pdf

It shows an approximate 3-4 times speedup from using low-level IO versus C IO, consistent with my findings.

linyx · 3rd July 2010, 04:35

I just tried out the test version, and am seeing errors like this one with it (lots, 5 out of 7 streams tested).
These errors do not show up with the 64-Bit or the 32-Bit of 2018. Want a stream?

Guest · 3rd July 2010, 04:42

Quote:

Originally Posted by linyx

Want a stream?

Yes, please. Also diff the DGI files. They would have to be different. My tests show no differences between release 2018 and test 2018 DGI files (except for the path at the top).

Any timing results to report?

woah! · 3rd July 2010, 04:51

My results 24.5GB file:

2018 - Elapsed: 0:13:36
2018 test - Elapsed: 0:05:24

the test version is much better here

Guest · 3rd July 2010, 05:04

I'm not imagining it then.

woah! · 3rd July 2010, 05:09

but vc1 files are not showing as much speedup as h264 bluray files. 8.5GB file

2018 - Elapsed: 0:02:17
2018 test - Elapsed: 0:01:59

will do some more files here for you..

adiabatic · 3rd July 2010, 05:49

OK, now I'm on my "basic" system.

32-bit Vista Home Premium
Core 2 Duo 2.0
8600GT video card (latest drivers)
good ol' mechanical HDs

Same test file as before

2018 - 10:16
2018 test - 12:26

It must be something unique to this clip. Video source is MPEG-2 TS 1080i with AC3 audio from Motorola DCT-3416 cable box. It's been edited with VideoRedo and (I'm not 100% sure) probably run through MPEG2Repair before editing.

sample: http://www.mediafire.com/file/4nm5kn...crew_donald.ts

Blue_MiSfit · 3rd July 2010, 08:44

Great work neuron2, per usual! I'm testing now on my system.

System Spec
Core i7 Q820 (mobile, 4 cores + HT)
4GB DDR3
Quadro FX 2800M
2x 250GB 2.5" HDD RAID 0
Windows 7 x64

Source Spec
1080p24 H.264 HP @ L5
45mbps average, encoded by x264 1542
CAVLC, Deblocking, B-Frames
TS Container (remuxed from an MKV via TSMuxeR 1.10.6)
24436 frames (just under 17 mins)

1018 vanilla
93 seconds = 262.75fps = ~11x realtime = ~ 495mbps

1018 perftest
136 seconds = 179.68fps = ~7.5x realtime = ~ 337mbps

So.. maybe I'm stupid and am being taunted by Windows caching as you warned us about, neuron2... is my sample just too short to avoid this? I tried deleting the source after indexing and muxing a fresh copy, but got similar results.

Any other suggestions?

BTW.. I did all this testing over UltraVNC. No broken CUDA! Slick!

Derek

Guest · 3rd July 2010, 12:48

Looks like it might be a Win7 versus WinXP issue.

@woah!

Please give your system details.

Guest · 3rd July 2010, 13:34

My results on my i7-980X Win7 64bit with VelociRaptor (same project as my WinXP results).

2018 - 3:16
2018 perf - 3:12

This supports my WinXP versus Vista/Win7 theory. I'm tending towards going with the perf version due to the big gains it gives XP users. I don't know why some of you see drops on Vista/Win7; could it be the RAID 0?

@linyx

Need your stream and your diff results, please.

Zep · 3rd July 2010, 17:24

Quote:

Originally Posted by neuron2

That's hard to believe.

I'm testing on WinXP 32-bit with mechanical drives. I'll try it on my i7-960 64-bit.

I see you have an SSD. That is probably going to be a lot different from a mechanical drive. Your read rate is probably four times what I get. That makes the actual buffer size important because the smaller the buffer the more calls are made. I could try increasing the buffer in multiples of 4096 until it starts to deteriorate performance.

If I have to choose, I will optimize for mechanical drives, at least at this point in time.

Can you test on a mechanical drive?

I have SSD also. In fact my system is basically a clone of adiabatic's

can you make the buffer size a var we can pass? i.e.Default 1=4096 internally , 2=8192 internally etc... then everyone can fine tune to their own PC.

3rd July 2010, 03:04	#1884 \| Link
adiabatic Registered User Join Date: Oct 2008 Posts: 17	Hmmm. It gives slightly slower performance on a 11.5 GB file. Both tests repeated twice. My system: Windows 7 Pro 64-bit Intel Core i7-920 GTX275 196.21 drivers 80GB Intel X25-M G2 SSD 2018 release version - 32-bit Stream Type: Transport [188] Video Type: MPEG2 Profile: main@high Coded Size: 1920x1088 Display Size: 1920x1080 Aspect Ratio: 16:9 [3] Frame Rate: 29.970030 fps Colorimetry: BT.709* Frame Structure: Frame Type: Coded Number: 200807 Playback Number: 200810 Frame Repeats: 0 Field Repeats: 7 Bitrate: 11.926 Bitrate (Avg): 13.750 Bitrate (Max): 19.111 Audio Stream: 14: AC3 3/2 48 384 Elapsed: 0:04:13 Remain: 0:00:00 FPS: Info: Finished! 2018 test Stream Type: Transport [188] Video Type: MPEG2 Profile: main@high Coded Size: 1920x1088 Display Size: 1920x1080 Aspect Ratio: 16:9 [3] Frame Rate: 29.970030 fps Colorimetry: BT.709* Frame Structure: Frame Type: Coded Number: 200807 Playback Number: 200810 Frame Repeats: 0 Field Repeats: 7 Bitrate: 11.926 Bitrate (Avg): 13.750 Bitrate (Max): 19.111 Audio Stream: 14: AC3 3/2 48 384 Elapsed: 0:05:03 Remain: 0:00:00 FPS: Info: Finished! <edit> looks like I'm behind on driver versions.... updating and repeating tests. Last edited by adiabatic; 3rd July 2010 at 03:46.

3rd July 2010, 03:12	#1885 \| Link
Guest Guest Join Date: Jan 2002 Posts: 21,901	That's hard to believe. I'm testing on WinXP 32-bit with mechanical drives. I'll try it on my i7-960 64-bit. I see you have an SSD. That is probably going to be a lot different from a mechanical drive. Your read rate is probably four times what I get. That makes the actual buffer size important because the smaller the buffer the more calls are made. I could try increasing the buffer in multiples of 4096 until it starts to deteriorate performance. If I have to choose, I will optimize for mechanical drives, at least at this point in time. Can you test on a mechanical drive? Last edited by Guest; 3rd July 2010 at 03:22.

3rd July 2010, 03:25	#1886 \| Link
adiabatic Registered User Join Date: Oct 2008 Posts: 17	OK, I'm also getting the same results with the SSD and the latest Nvidia drivers. I'll now re-run the tests with the conventional HD. Last edited by adiabatic; 3rd July 2010 at 03:30.

3rd July 2010, 03:48	#1887 \| Link
adiabatic Registered User Join Date: Oct 2008 Posts: 17	OK mechanical HD (7200rpm Western Digital SATA). With the same test file as above. 2018 "release" - 5:09 2018 "test" - 6:23 (during this test I opened Resource Monitor and I was reading and writing at about 28 MB/sec for both) Just to be clear so I don't lead you down a false path... your earlier comment about caching related to read caching, right? I have write caching enabled for my drives which was the default Windows setting for the drives. I did this test with the source file and demux output on the mechanical HD. Does it matter to your testing if my pagefile and the DGIndex folder is on the SSD? I'm heading out for an hour or so now... but am willing to test more later. Last edited by adiabatic; 3rd July 2010 at 03:53.

3rd July 2010, 03:54	#1888 \| Link
Guest Guest Join Date: Jan 2002 Posts: 21,901	Jeez, don't have resource monitor opened for the test. That's just incredible. I can't believe it. I get a 3.5 times speedup on my system! Let's wait and see what happens with other people. Are you running the X32 versions in both cases? I would so love to get to the bottom of this. Last edited by Guest; 3rd July 2010 at 04:13.

3rd July 2010, 04:22	#1890 \| Link
Guest Guest Join Date: Jan 2002 Posts: 21,901	My results: 2018 - 21:30 2018 test - 5:51 Look at the graph in here: http://pisa.ucsd.edu/cse125/2006/Pap...ing_in_C++.pdf It shows an approximate 3-4 times speedup from using low-level IO versus C IO, consistent with my findings. Last edited by Guest; 3rd July 2010 at 04:25.

3rd July 2010, 04:35	#1891 \| Link
linyx Un-Registered User Join Date: Mar 2008 Location: Audio Stream - 0x80 Posts: 341	I just tried out the test version, and am seeing errors like this one with it (lots, 5 out of 7 streams tested). These errors do not show up with the 64-Bit or the 32-Bit of 2018. Want a stream?

3rd July 2010, 04:51	#1893 \| Link
woah! Registered User Join Date: Oct 2003 Posts: 435	My results 24.5GB file: 2018 - Elapsed: 0:13:36 2018 test - Elapsed: 0:05:24 the test version is much better here

3rd July 2010, 05:04	#1894 \| Link
Guest Guest Join Date: Jan 2002 Posts: 21,901	I'm not imagining it then.

3rd July 2010, 05:09	#1895 \| Link
woah! Registered User Join Date: Oct 2003 Posts: 435	but vc1 files are not showing as much speedup as h264 bluray files. 8.5GB file 2018 - Elapsed: 0:02:17 2018 test - Elapsed: 0:01:59 will do some more files here for you..

3rd July 2010, 05:49	#1896 \| Link
adiabatic Registered User Join Date: Oct 2008 Posts: 17	OK, now I'm on my "basic" system. 32-bit Vista Home Premium Core 2 Duo 2.0 8600GT video card (latest drivers) good ol' mechanical HDs Same test file as before 2018 - 10:16 2018 test - 12:26 It must be something unique to this clip. Video source is MPEG-2 TS 1080i with AC3 audio from Motorola DCT-3416 cable box. It's been edited with VideoRedo and (I'm not 100% sure) probably run through MPEG2Repair before editing. sample: http://www.mediafire.com/file/4nm5kn...crew_donald.ts Last edited by adiabatic; 3rd July 2010 at 05:56.

3rd July 2010, 08:44	#1897 \| Link
Blue_MiSfit Derek Prestegard IRL Join Date: Nov 2003 Location: Los Angeles Posts: 5,989	Great work neuron2, per usual! I'm testing now on my system. System Spec Core i7 Q820 (mobile, 4 cores + HT) 4GB DDR3 Quadro FX 2800M 2x 250GB 2.5" HDD RAID 0 Windows 7 x64 Source Spec 1080p24 H.264 HP @ L5 45mbps average, encoded by x264 1542 CAVLC, Deblocking, B-Frames TS Container (remuxed from an MKV via TSMuxeR 1.10.6) 24436 frames (just under 17 mins) 1018 vanilla 93 seconds = 262.75fps = ~11x realtime = ~ 495mbps 1018 perftest 136 seconds = 179.68fps = ~7.5x realtime = ~ 337mbps So.. maybe I'm stupid and am being taunted by Windows caching as you warned us about, neuron2... is my sample just too short to avoid this? I tried deleting the source after indexing and muxing a fresh copy, but got similar results. Any other suggestions? BTW.. I did all this testing over UltraVNC. No broken CUDA! Slick! Derek __________________ These are all my personal statements, not those of my employer :) Last edited by Blue_MiSfit; 3rd July 2010 at 08:46.

3rd July 2010, 12:48	#1898 \| Link
Guest Guest Join Date: Jan 2002 Posts: 21,901	Looks like it might be a Win7 versus WinXP issue. @woah! Please give your system details.

3rd July 2010, 13:34	#1899 \| Link
Guest Guest Join Date: Jan 2002 Posts: 21,901	My results on my i7-980X Win7 64bit with VelociRaptor (same project as my WinXP results). 2018 - 3:16 2018 perf - 3:12 This supports my WinXP versus Vista/Win7 theory. I'm tending towards going with the perf version due to the big gains it gives XP users. I don't know why some of you see drops on Vista/Win7; could it be the RAID 0? @linyx Need your stream and your diff results, please. Last edited by Guest; 3rd July 2010 at 14:02.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode