DG NV tools [Archive] - Page 38

kypec

1st July 2010, 08:13

Using the graphic interface seems error prone to me.
I second that request (though it hasn't been said explicitly :p).
A simple frame counter showing something like Frame X of T or just X/T (T stands for Total frames) would be fantastic.

talen9

1st July 2010, 08:36

Neuron, a little question. There are avisynth filters that are currently not well "multithreadable". I am going to bypass the problem splitting the job in multiple ones with different parts of the video. Is there a simple way to tell DGIndex to create 3/4 DGI with frame accurate splitting in the lateral points? Using the graphic interface seems error prone to me.

Isn't it simpler to just use a Trim() instructions in each of the "splitted" scripts? ;)

tormento

1st July 2010, 08:49

Isn't it simpler to just use a Trim() instructions in each of the "splitted" scripts? ;)
I think Trim does not give direct control if the selected frame is a key, a intra or anything else.

MOS-Marauder

1st July 2010, 09:26

HM ... DGIndexNV V2017 gives me errors now. "GPU Decoder: Failed to create Video Decoder (1)".

I changed nothing on my System.. only updated from 2013 to 2017....

Chris

Edit: Oh i see.. u changed it again... Regenerating License....

talen9

1st July 2010, 10:42

I think Trim does not give direct control if the selected frame is a key, a intra or anything else.

I don't think this is actually a problem ... 'cause the first frame included in the trimmed part will be decoded correctly no matter what kind of frame it is in the source video stream. Anyway, if you select the first frame of a new scene by direct-inspecting of the stream (e.g. via AvsP / AvsPmod), it will (almost surely) be a reference frame.

Guest

1st July 2010, 11:51

Is the DGI compatible? Is it sufficient to edit the 1st line? It's compatible. You don't have to edit it.

Guest

1st July 2010, 11:53

Neuron, a little question. There are avisynth filters that are currently not well "multithreadable". I am going to bypass the problem splitting the job in multiple ones with different parts of the video. Is there a simple way to tell DGIndex to create 3/4 DGI with frame accurate splitting in the lateral points? Using the graphic interface seems error prone to me. I advise splitting in your script.

I think Trim does not give direct control if the selected frame is a key, a intra or anything else. So what?

Guest

1st July 2010, 13:47

Keeping those crackers on their toes!

* Fixed case of recovery point SEI appearing before the PPS. This fixes AVC streams cut with VideoRedo.

http://neuron2.net/dgdecnv/dgdecnv.html

hejhopp

1st July 2010, 14:44

works perfect! Thx neuron2

Guest

1st July 2010, 14:50

You're welcome. Thank you for bringing it to my attention.

Next I'm going to invest some time into performance improvements for the save project operation.

Varies

1st July 2010, 16:00

neuron2
Hi! thx for many updates :)

recently i update program files from 2013 to 2018 and generate new license (and copypast key in the license.txt), but its fully not worked (from version 2013). What i'am doing wrong?

cweb

1st July 2010, 16:04

Driver 258.69 also causes some systems to crash when playing flash videos. I would avoid it for the time being.

There are lots of problems with the 257 series of drivers (even the stable version) according to this thread:
http://forums.nvidia.com/index.php?showtopic=171530
I will avoid updating to 257 for the time being. I'm still using the 196.21 stable version driver.

Guest

1st July 2010, 16:33

recently i update program files from 2013 to 2018 and generate new license (and copypast key in the license.txt), but its fully not worked (from version 2013). What i'am doing wrong? I support individual licensing issues by email only. Send your user id and machine id and explain the problem. Thank you.

RedDwarf1

1st July 2010, 18:18

Keeping those crackers on their toes!
These people need to spend their time in more productive ways than trying to gain free use of software that costs so little. It's not like it's costing hundreds of pounds/dollars.

They must like challenges, why not send them two sheets of paper and ask them to make Pr*#ks out of them using origami.:rolleyes:

* Fixed case of recovery point SEI appearing before the PPS. This fixes AVC streams cut with VideoRedo.

http://neuron2.net/dgdecnv/dgdecnv.html

That's very welcome as I use VideoRedo TVsuite V4 myself. I don't encode directly from the source TS anymore.

tormento

1st July 2010, 19:09

So what?
Is it accurate? I'd like to join the produced videos and not to have hiccups, duplicated frames or so.

Guest

2nd July 2010, 00:08

Is it accurate? I'd like to join the produced videos and not to have hiccups, duplicated frames or so. Avisynth delivers decoded frames, so the concept of key frames does not apply. Your accuracy will be determined by the reliability of the source filter's random access. I take great pains to ensure the accuracy of DG(Multi)Decode() and so if you find any bugs there you will get a very fast response from me.

Guest

2nd July 2010, 04:16

Would some kind souls please test the 32-bit DGIndexNV version linked below to see if it delivers faster indexing without breaking anything. I'm getting indexing about twice as fast as the 2018 release version on initial testing. Thank you.

http://neuron2.net/misc/DGIndexNV_perftest.zip

Don't forget about the role of Windows caching so be sure to perform fair testing. If you don't know what I am talking about please leave the testing to people that do.

tormento

2nd July 2010, 10:35

Would some kind souls please test the 32-bit DGIndexNV version linked below

No changes for me, I think to be HDD limited more than CPU (I7-920@3.6) or GPU (9800GT).

Perhaps the hdd is the direction to be followed, i.e. better disk r/w?

This evening or tomorrow I'll do some tests on the workstation I use at home, 3*Raptor Raid0.

IMHO the real advantages should be seen on SDD only.

Guest

2nd July 2010, 11:59

Did you try on a full bluray project?

I have made disk related changes, not just CPU optimizations.

hydra3333

2nd July 2010, 13:09

Hmm, if I knew what you were talking about I would, but I don't, so I won't :)

Guest

2nd July 2010, 13:14

Just do it on a full bluray project and the caching will not be a factor.

tormento

2nd July 2010, 13:17

Usually I encode BD to DVD-DL size to be seen on notebook.

The project I tried with not optimized and optimized version is Shutter Island, full movie mpls.

To be clear enough: my work pc is HDD limited I suppose. Two mean seagate in raid 0 is not the culprit of performance.

Stay tuned, if I'll have some minutes free I'll try on home workstation. If you could compile a x64 version, it should be better too ;)

Guest

2nd July 2010, 15:11

Zep

2nd July 2010, 16:19

I get good gains with the test version.

But I also find that DGIndex is almost twice as fast as this test version of DGIndexNV and I can't attribute it to any CPU bottleneck, so the only possibility is that it uses _read() rather than fread(). So I will convert DGIndexNV to use _read().

I already tried unbuffered fread() but it did not help.

Same thing happened with DGIndex.

Anyway, I do remember when i first mentioned it years ago you looked into the internal caching so you bumped it up to 6 or 7
frames IIRC and DGindex speed went WAY up. You said that adding even more cache didn't help because we were CPU bound
but that was back then and these days 4Ghz i7 things may need to be looked at again.

Could there be an internal cache issue here as well?

It may also help random frame seeking to have a few GOPs decoded and cached ahead of time.

um3k

2nd July 2010, 16:23

Whenever I open a M2T file from my HDV camera, I get this error message:
http://www.stupidideasstudios.com/DGIndexNVError.PNG
Here is a sample of a file that causes this error:
http://www.mediafire.com/file/ktuzmfkx4go/Sample.m2t_0
It happens in every version of DGIndexNV that I've used, including 2018. It's only slightly annoying when I'm just doing one file, but when I do a batch operation, well, I can't do a batch operation because the error pops up with every new file and brings it to a halt.

Guest

2nd July 2010, 17:05

Same thing happened with DGIndex.

Anyway, I do remember when i first mentioned it years ago you looked into the internal caching so you bumped it up to 6 or 7
frames IIRC and DGindex speed went WAY up. You said that adding even more cache didn't help because we were CPU bound
but that was back then and these days 4Ghz i7 things may need to be looked at again.

Could there be an internal cache issue here as well?

It may also help random frame seeking to have a few GOPs decoded and cached ahead of time. I'm revisiting all these issues. I'll report findings as I go.

Guest

2nd July 2010, 23:36

OK, I have now exceeded the performance of DGIndex!

I implemented low-level IO and CPU optimizations. I did it for basic stream reading, TS trim (which is now blazingly fast), and audio demux. I want to do it for video demux and then release a beta for you to beat on while I do my own regression testing.

Note that DGIndex uses slow IO for the audio demuxing; that's why I was able to beat it.

The MKV support uses the slow IO too. I haven't decided whether to bother with that for a first release. MKV files tend not to be as large as bluray file sets. Still, it would be nice to finish that at some point.

For techies: My buffer size is 4096 bytes. Raising it does not improve things, but when it starts getting quite large, performance actually decreases quite substantially.

A useful side effect is that when playing in DGIndexNV, you no longer get those annoying freezes while the (previously large) buffer fills.

Guest

3rd July 2010, 00:28

I just indexed a bluray and demuxed an AC3 track in 5:51. :eek:

GearX

3rd July 2010, 01:19

The MKV support uses the slow IO too. I haven't decided whether to bother with that for a first release. MKV files tend not to be as large as bluray file sets. Still, it would be nice to finish that at some point.

FWIW, I would find that very useful. When I re-encode blu-rays, I remux them to MKV first (using eac3to) to make sure I have only the streams I want. It's probably unnecessary but I find it much "cleaner". It also takes care of multiple angles and such, although I guess that's less important now that MPLS support is in.

adiabatic

3rd July 2010, 01:40

Would some kind souls please test the 32-bit DGIndexNV version linked below to see if it delivers faster indexing without breaking anything. I'm getting indexing about twice as fast as the 2018 release version on initial testing. Thank you.

http://neuron2.net/misc/DGIndexNV_perftest.zip

"The requested URL /misc/DGIndexNV_perftest.zip was not found on this server"

No need for further testing? Your own results seem promising... I'm willing to participate if you still need it.

Guest

3rd July 2010, 01:52

"The requested URL /misc/DGIndexNV_perftest.zip was not found on this server"

No need for further testing? Your own results seem promising... I'm willing to participate if you still need it. That one did not use low-level IO. I'll post the new one shortly. Thank you for your willingness to assist in the testing.

Guest

3rd July 2010, 02:31

I'll post the new one shortly. OK, it's there. Use the same link as before.

adiabatic

3rd July 2010, 03:04

Hmmm. It gives slightly slower performance on a 11.5 GB file. Both tests repeated twice.

My system:
Windows 7 Pro 64-bit
Intel Core i7-920
GTX275 196.21 drivers
80GB Intel X25-M G2 SSD

2018 release version - 32-bit
Stream Type: Transport [188]
Video Type: MPEG2
Profile: main@high
Coded Size: 1920x1088
Display Size: 1920x1080
Aspect Ratio: 16:9 [3]
Frame Rate: 29.970030 fps
Colorimetry: BT.709*
Frame Structure:
Frame Type:
Coded Number: 200807
Playback Number: 200810
Frame Repeats: 0
Field Repeats: 7
Bitrate: 11.926
Bitrate (Avg): 13.750
Bitrate (Max): 19.111
Audio Stream: 14: AC3 3/2 48 384
Elapsed: 0:04:13
Remain: 0:00:00
FPS:
Info: Finished!

2018 test
Stream Type: Transport [188]
Video Type: MPEG2
Profile: main@high
Coded Size: 1920x1088
Display Size: 1920x1080
Aspect Ratio: 16:9 [3]
Frame Rate: 29.970030 fps
Colorimetry: BT.709*
Frame Structure:
Frame Type:
Coded Number: 200807
Playback Number: 200810
Frame Repeats: 0
Field Repeats: 7
Bitrate: 11.926
Bitrate (Avg): 13.750
Bitrate (Max): 19.111
Audio Stream: 14: AC3 3/2 48 384
Elapsed: 0:05:03
Remain: 0:00:00
FPS:
Info: Finished!

<edit> looks like I'm behind on driver versions.... updating and repeating tests.

Guest

3rd July 2010, 03:12

That's hard to believe.

I'm testing on WinXP 32-bit with mechanical drives. I'll try it on my i7-960 64-bit.

I see you have an SSD. That is probably going to be a lot different from a mechanical drive. Your read rate is probably four times what I get. That makes the actual buffer size important because the smaller the buffer the more calls are made. I could try increasing the buffer in multiples of 4096 until it starts to deteriorate performance.

If I have to choose, I will optimize for mechanical drives, at least at this point in time.

Can you test on a mechanical drive?

adiabatic

3rd July 2010, 03:25

OK, I'm also getting the same results with the SSD and the latest Nvidia drivers.

I'll now re-run the tests with the conventional HD.

adiabatic

3rd July 2010, 03:48

OK mechanical HD (7200rpm Western Digital SATA).

With the same test file as above.
2018 "release" - 5:09
2018 "test" - 6:23 (during this test I opened Resource Monitor and I was reading and writing at about 28 MB/sec for both)

Just to be clear so I don't lead you down a false path... your earlier comment about caching related to read caching, right? I have write caching enabled for my drives which was the default Windows setting for the drives.

I did this test with the source file and demux output on the mechanical HD. Does it matter to your testing if my pagefile and the DGIndex folder is on the SSD?

I'm heading out for an hour or so now... but am willing to test more later.

Guest

3rd July 2010, 03:54

Jeez, don't have resource monitor opened for the test.

That's just incredible. I can't believe it. I get a 3.5 times speedup on my system!

Let's wait and see what happens with other people.

Are you running the X32 versions in both cases?

I would so love to get to the bottom of this.

adiabatic

3rd July 2010, 04:15

Jeez, don't have resource monitor opened for the test.

That's just ridiculous. I can't believe it. I get a 4 times speedup on my system.

Let's wait and see what happens with other people.

BTW, I just popped Resource Monitor open for a minute.

Yes, x32 (double checked in task manager (*32)). Your x64 binaries are tucked away in another folder. Feel free to ask other questions or wait for other testers. The last thing I want to do is be making an error here that wastes your time.

I agree, let's see what others get. Later tonight I'll run this test on a more plain-vanilla machine in the house (mech. HDs, Core2Duo 2.0, Nvidia 8600GT card).

Though since I can't let go of this bone... for fun I've moved my dgdecnv folder and pagefile to the mech. HD. The source and target folder are also on the same HD. The only part of this equation that should now be on the SSD are core Windows files and Nvidia driver files.

(no Resource Monitor this time!)
2018 "release" - 5:11
2018 "test" - 6:08

The only thing I'm doing on this system while the demux is running is light web-browsing with Firefox but I've been doing that constantly through all tests with all variations of testing tonight.

Now I am really leaving the house for a little while :)

Guest

3rd July 2010, 04:22

My results:

2018 - 21:30
2018 test - 5:51

Look at the graph in here:

http://pisa.ucsd.edu/cse125/2006/Papers/High_Performance_Game_Programming_in_C++.pdf

It shows an approximate 3-4 times speedup from using low-level IO versus C IO, consistent with my findings.

linyx

3rd July 2010, 04:35

I just tried out the test version, and am seeing errors like this one (http://forum.doom9.org/showthread.php?p=1400145#post1400145) with it (lots, 5 out of 7 streams tested).
These errors do not show up with the 64-Bit or the 32-Bit of 2018. Want a stream?:o

Guest

3rd July 2010, 04:42

Want a stream? Yes, please. Also diff the DGI files. They would have to be different. My tests show no differences between release 2018 and test 2018 DGI files (except for the path at the top).

Any timing results to report?

woah!

3rd July 2010, 04:51

My results 24.5GB file:

2018 - Elapsed: 0:13:36
2018 test - Elapsed: 0:05:24

the test version is much better here :)

Guest

3rd July 2010, 05:04

I'm not imagining it then. :D

woah!

3rd July 2010, 05:09

but vc1 files are not showing as much speedup as h264 bluray files. 8.5GB file

2018 - Elapsed: 0:02:17
2018 test - Elapsed: 0:01:59

will do some more files here for you..

adiabatic

3rd July 2010, 05:49

OK, now I'm on my "basic" system.

32-bit Vista Home Premium
Core 2 Duo 2.0
8600GT video card (latest drivers)
good ol' mechanical HDs

Same test file as before

2018 - 10:16
2018 test - 12:26

It must be something unique to this clip. Video source is MPEG-2 TS 1080i with AC3 audio from Motorola DCT-3416 cable box. It's been edited with VideoRedo and (I'm not 100% sure) probably run through MPEG2Repair before editing.

sample: http://www.mediafire.com/file/4nm5kneimzt/pitcrew_donald.ts

Blue_MiSfit

3rd July 2010, 08:44

Great work neuron2, per usual! I'm testing now on my system.

System Spec
Core i7 Q820 (mobile, 4 cores + HT)
4GB DDR3
Quadro FX 2800M
2x 250GB 2.5" HDD RAID 0
Windows 7 x64

Source Spec
1080p24 H.264 HP @ L5
45mbps average, encoded by x264 1542
CAVLC, Deblocking, B-Frames
TS Container (remuxed from an MKV via TSMuxeR 1.10.6)
24436 frames (just under 17 mins)

1018 vanilla
93 seconds = 262.75fps = ~11x realtime = ~ 495mbps

1018 perftest
136 seconds = 179.68fps = ~7.5x realtime = ~ 337mbps

So.. maybe I'm stupid and am being taunted by Windows caching as you warned us about, neuron2... is my sample just too short to avoid this? I tried deleting the source after indexing and muxing a fresh copy, but got similar results.

Any other suggestions?

BTW.. I did all this testing over UltraVNC. No broken CUDA! Slick! :devil:

Derek

Guest

3rd July 2010, 12:48

Looks like it might be a Win7 versus WinXP issue.

@woah!

Please give your system details.

Guest

3rd July 2010, 13:34

My results on my i7-980X Win7 64bit with VelociRaptor (same project as my WinXP results).

2018 - 3:16
2018 perf - 3:12

This supports my WinXP versus Vista/Win7 theory. I'm tending towards going with the perf version due to the big gains it gives XP users. I don't know why some of you see drops on Vista/Win7; could it be the RAID 0?

@linyx

Need your stream and your diff results, please.

Zep

3rd July 2010, 17:24