DG NV tools

[Guest]

I have just run DGIndexNV built for x64 for the first time! It's working great. And it's very fast on my i7-980X overclocked to 4.2 GHz. :)

The major challenge was the color conversion and display routines. In DGIndexNV x32 I use a simple C loop to unpack CUVID's NV12 to planar YV12 and then compiler intrinsics for an asm implementation of YV12 to packed RGB for display via Windows GDI (as implemented in the CUVID Client). But those intrinsics are not available on x64. (The code in DGIndex [the open source one] of course uses 32-bit asm files. Those would have to be recoded but can't be used in any case due to GPL.)

So I wrote a high-performance pure C implementation of YV12 to RGB using lookup tables. This conversion is actually a little faster than the one in the current x32 version.

I spent all day making the needed revisions to the cropping and save BMP utilities to make them work with the new conversion code.

The source filters should be easy as there is no asm code in them.

So who will tell me where to get the latest and most stable x64 version of Avisynth? Can it coexist with the x32 version?

[Audionut]

http://forum.doom9.org/showthread.php?t=152800 Set's version. Runs fine with 32bit version installed, and actually needs the 32bit installed for the registry pointers.

Oh, and I'll happily beta test with you.

[lych_necross]

Unfortunately, I don't believe there is an official x64 build of Avisynth. In addition to JoshyD's build (listed above), Squid_80 has a x64 build of Avisynth as well as precompiled plugins here (http://www.members.optusnet.com.au/squid_80/). As to which one is more stable, I would say neither are as stable as the official x86 build; however, if you forced me to pick one, I would probably recommend JoshyD's build. Hopefully IanB will get inspired and come up with an official x64 build sometime this year.

[Audionut]

Squids build is about 3 years old iirc.

[tormento]

Neuron, would you implement an auto/manual crop interface in DGIndex? Staxrip and other gui's sometimes lose a lot of time in cropping procedures. It could be glad to have the possibility to get it before.

(nice english today, I must have temperature)

[Guest]

Please provide specs for the functionality you want. I ask because it's not obvious to me. For example, say I have a letterboxed movie and cropping away just the black leaves a certain non-mod (4, 8, 16) vertical size. Should I overcrop to enforce mod? What frame(s) should be used for the calculations? Etc. I have never used autocrop so if you want this you have to be really clear about what you want.

[Selur]

I normally do something a long the lines of:

1. allow user to specify:
- threshold 0/16/32 (= color value that will still count as black)
- mod the output should have after cropping (2/4/8/16)

2. analyse 500 frames from the source:
- e.g. take every framecount/500 frames a frame
- if source is < 500 frames analyse all
- if seeking is an issue skip the first framecount/10 frames then analyse 5 frames skip 100 frames until you got 500 (if source is too short for this, lower frames to skip)
- drop results where width or height more than halves
- from the results take the crop values from the frame with the larges non black area

3. adjust crop values to mod
- overcrop to archive the output mod the user wanted

Cu Selur

[Guest]

Thanks, Selur.

My first test runs into problems. I have a 16 level black bar but as it approaches the video it gets lighter (due to encoding). It's hard to pick a threshold because some of the pixels are 46 and higher but still in the bar. Do you just get close and then hope the mod operation finishes the job? If so, it seems very unrobust.

[onesloth]

Could you adjust the threshold based on the average luma level of the non-black areas? i.e. bright image -> higher threshold, dark image -> lower threshhold.

[Guest]

That's too complicated, but thanks for the suggestion. With a decent fixed threshold (32) and enforcing the mod (and fixing the bug I had) I get acceptable results.

I've added the feature for the next release with a threshold of 32 and mod 4/8/16. It operates on the frame that is displayed when the cropping dialog is entered. I'll tweak it further as needed.

[Blue_MiSfit]

Hi Neuron2!

Congrats on getting that badass 980x CPU :D It will make whatever you had before seem like a toy!

I'm also very excited by the prospect of DGIndexNV x64. The folks currently deep in AviSynth x64 development will be delighted to add your tools to their arsenal. I think you'll find things in good shape, as many plugins have been ported over, and many see large performance increases (especially MVTools).

If only I didn't have to deal with ProRes, then I could live in a pure x64 environment for all my encoding needs...

Derek

[SideStream84]

I've noticed a quality issue when using the nvidia deinterlacer and have done some troubleshooting / investigating. At first I thought it was a 64bit issue since it was happening in XPx64 and 7x64, but I formatted one of the computers to XPx32 and it persists.

Specs:
1. Win 7x64, GeForce 9800 GT x2 (1gb ram each), driver version 197.45.
2. Win XPx32/x64, GeForce 9800M GS (512 mb ram), driver version 197.16.
3. Win XPx32, GeForce 9650M GT, driver version 197.16

Things tried (none of which changed the results):
-SLI on/off
-DGNV 2007, 2008, 2009, 2010
-Lower versions of the nvidia drivers
-multisource and CUVID server
-Win 7x64, WinXPx32, WinXPx64

On the two computers with 9800s I get "artifacts" as shown below with a few of them marked (as the scene pans they stay the same in the same locations, it's not like variable "noise"):
http://i113.photobucket.com/albums/n234/sidestream84/9800.png

On the computer with the 9650M GT, it looks as one would expect:
http://i113.photobucket.com/albums/n234/sidestream84/9650.png

Do different GPUs do it better or worse than others or does anyone have any information on what might be the cause of the artifacts? (using a supported purevideo player like windows media player on win 7 results in a good looking image without artifacts on all computers tested). Is there anything else I can/should try?

[JoeH]

Unfortunately, I don't believe there is an official x64 build of Avisynth. In addition to JoshyD's build (listed above), Squid_80 has a x64 build of Avisynth as well as precompiled plugins here (http://www.members.optusnet.com.au/squid_80/). As to which one is more stable, I would say neither are as stable as the official x86 build; however, if you forced me to pick one, I would probably recommend JoshyD's build. Hopefully IanB will get inspired and come up with an official x64 build sometime this year.

Yes, JoshyD's is the best, and currently in development as well, which always helps...

[tormento]

Please provide specs for the functionality you want. I ask because it's not obvious to me. For example, say I have a letterboxed movie and cropping away just the black leaves a certain non-mod (4, 8, 16) vertical size. Should I overcrop to enforce mod? What frame(s) should be used for the calculations? Etc. I have never used autocrop so if you want this you have to be really clear about what you want.

The StaxRip behaviour should be enough, i.e. specify in preferences the mod you need, and autocrop to the majority format of N frames. In the preferences you should get the possibility to choose between under or overcrop when auto. Obviously the possibility of gui assisted manual correction is mandatory.

Yes, I know, I am a lazy man ;)

[Guest]

Do different GPUs do it better or worse than others or does anyone have any information on what might be the cause of the artifacts? (using a supported purevideo player like windows media player on win 7 results in a good looking image without artifacts on all computers tested). Is there anything else I can/should try? Please give me the source stream so I can investigate.

[MOS-Marauder]

Hi Neuron..

I found a strange behaviour of DGindexNV 2010 on ArteHD (Gerrmany) Streams.

When i use DGIndexNV to Cut and create DGI/MP2 Audiofiles it creates 4 instead of 2 Audiofiles.
(The Stream contains only 2 Audio Streams ... German and French)

Also the Audiofiles now have a super High Delay added.

It looks like this:

klassenzimmer.dgi
klassenzimmer PID 184d L2 2ch 48 256 DELAY -1060ms.mp2 (49.836.770 bytes)
klassenzimmer PID 184d L2 2ch 48 256 DELAY -1412ms.mp2 (94.xxx bytes)
klassenzimmer PID 184e L2 2ch 48 256 DELAY -1070ms.mp2 (49.837.098 bytes)
klassenzimmer PID 184e L2 2ch 48 256 DELAY -1422ms.mp2 (94.xxx bytes)



Before i never had any Delay using DGToolsNV.
The 2 smaller Files contains audio from somewhere of the Source.

Anotehr strange behaviour is... When i open an Avisynth script with dgi inside to play @ mediaplayer it takes upto 2-3 minutes until it starts to play. Same thing if i use MeGui with DGI files.


If u need a Sample, tell me. The Recording(s) were done with DVBViewer Pro and a Dreambox 800HD.

Edit: I also found out that the Cuts are not at the Points ive made them...They are a 1-2 Seconds before....
Edit2: Only the Audio is not cutted right... Video is ok


Chris

[Guest]

If u need a Sample, tell me. Yes, I need a sample of the source stream that I can use to duplicate the problem. Please make sure that the problem occurs with the sample that you give me. Also, explain exactly how to make the problem happen with the sample.

Post the script that you claim takes 2-3 minutes to open.

[MOS-Marauder]

Here it is:

LoadPlugin("C:\Program Files (x86)\megui-Dev-031-1061\tools\dgindexnv\DGDecodeNV.dll")
DGSource("F:\000-CAPTURE-BACKUP\____BluRay-BRENN\HD-Encode\Xenius - Das schwimmende Klassenzimmer\klassenzimmer.dgi",fieldop=0)
SelectEven()
#deinterlace
#crop
#resize
#denoise


I simply started DGIndexNV 2010, loaded the Video Source... Marked Start and End and saved Project.


Ive also tried w/o Cuvid Server with DGMultiDecode but MeGui crashed then. (same error as i wouldnt have Cuvid Server started... Couldnt decode..)

Shall i cut the Sample with a special Tool? (found it..DGSplit...)


Edit: The Time for Opening differs. Long Video File...Long Time to open with DGTools... short one ..eg 25 Minutes only takes 5-10 seconds. The 4 Hrs Eurovision Show took that long (3-4 minutes)

Not that u misunderstand me... Opening in DGIndexNV works instantly... But loading/using the DGI in anyway causes this.

Edit2: The Version Number of MeGui u see in the Filename above isnt correct... its the actual Developer Version.

Btw. congrats to Your new Rig.


Here is the File.
If u only load it and save Project i only have the DGI and 2x mp2 like expected. (also with Delays of aprox 1000).
Buuuuuuut if i set cut points in and out... it creates 4 mp2 Files like described above...



http://www2.zippyshare.com/v/88814789/file.html


Chris

[Guest]

Not that u misunderstand me... Opening in DGIndexNV works instantly... But loading/using the DGI in anyway causes this. What happens if you open the script in VirtualDub?

[MOS-Marauder]

Same thing as in any Application... it opens after a while. It looks like it would check/scan the Video Source before opening. If i have the first Frame at the Display everything is like before.. normal behaviour... Only the first Opening of the Script takes a long (longer depending on Source Filesize) time.


Chris

[Guest]

@MOS-Marauder

1. Delays of ~1000ms are not wrong and not uncommon.

2. Using your sample I get two audio streams both of which play fine in MPC. There are not four as you claim.

3. I do not experience any slow opening in VirtualDub with your sample. I will try with my own longer file at home this evening, but the DGI file is read, that is all, there is no scanning of the video file, so the process should be fast.

[Guest]

The file open does do a sort of the POCs of the last GOP. If the POCs are spanning the full integer range then my simple bubble sort may take a long time. I wasn't expecting to have to sort extremely large GOPs.

To see if that is the case please give me the complete DGI file for the entire stream. I think I know what is happening but I need the DGI to confirm it.

EDIT: Ha. That sorting is not needed anymore! That was needed for previous versions when the FRM entries in the DGI file were not sorted by DGIndexNV. Since they now appear in the DGI file already sorted that sorting code in DGSource() can be removed and will very likely solve the problem.

[MOS-Marauder]

@MOS-Marauder

1. Delays of ~1000ms are not wrong and not uncommon.

2. Using your sample I get two audio streams both of which play fine in MPC. There are not four as you claim.

3. I do not experience any slow opening in VirtualDub with your sample. I will try with my own longer file at home this evening, but the DGI file is read, that is all, there is no scanning of the video file, so the process should be fast.

To 1:
Maybe but in the past i always had 0 Delay in DGIndexNV.

To 2:
Sure if i play the Sample in a player there are 2 Audio files, but that wasnt the problem. The Problem is .. if i use DGIndexNV 2010 and mark any start and end point (cut points) in it and save the project then i get 4 Audios. as i posted somewhere above.

To 3:
Yep with this small Sample its allmost impossible to see it but if u use a 21 GB Source File u will notice it ;) (I can do a screencast for u in realtime if u want it ;) )

And again.. if i do NOT set any Cut points and save the Project i get only the 2 Audios as it should be.


Chris

[MOS-Marauder]

The file open does do a sort of the POCs of the last GOP. If the POCs are spanning the full integer range then my simple bubble sort may take a long time. I wasn't expecting to have to sort extremely large GOPs.

To see if that is the case please give me the complete DGI file for the entire stream. I think I know what is happening but I need the DGI to confirm it.

EDIT: Ha. That sorting is not needed anymore! That was needed for previous versions when the FRM entries in the DGI file were not sorted by DGIndexNV. Since they now appear in the DGI file already sorted that sorting code in DGSource() can be removed and will very likely solve the problem.


DGSource("F:\000-CAPTURE-BACKUP\____BluRay-BRENN\HD-Encode\Xenius - Das schwimmende Klassenzimmer\klassenzimmer.dgi",fieldop=0)


U mean that Fieldop=0 ?

U still need the DGI of that Eurovision Song Contest File?


Chris

[Guest]

U still need the DGI of that Eurovision Song Contest File? Yes, please.

1. DGIndexNV simply reports the delay in the stream.

2. I cannot duplicate this with the sample you provided.

3. I cannot help without the full DGI file.

U mean that Fieldop=0 ? No.

BTW, is your keyboard broken? "You" keeps coming out as "U".

[MOS-Marauder]

Yes, please.

Ok this will take a while because i have to copy the 21 GB file from the Reciever to the HDD xD..and Dreamboxes arent very fast on that....


Chris

[Guest]

Don't you already have the DGI file for whatever stream you are reporting this slow open problem for? There's no need to copy the full stream anywhere.

[MOS-Marauder]

Don't you already have the DGI file for whatever stream you are reporting this slow open problem for? There's no need to copy the full stream anywhere.


Lol not for the BIG one.. because i encoded my Video already. I simply forgot to delete the big TS file ...

Ok so here it is.. i tested it .. it took 2 1/2 minutes until i saw the 1st frame.


http://www15.zippyshare.com/v/47453929/file.html


In the one above i simply loaded the File and saved Project. it saved 1 DGI and 1 AC3.

I tried it again a 2nd time.. with a Cut in and Cut out point. The result is weird... DGIindexNV created now 1 DGI and 3 AC3 files.

http://www9.zippyshare.com/v/37315488/file.html

20100529 2054 - Das Erste HD - Eurovision Song Contest 2010 (Edited)-withcut.dgi
20100529 2054 - Das Erste HD - Eurovision Song Contest 2010 (Edited)-withcut PID 1786 3_2ch 48KHz 448Kbps DELAY -1174ms.ac3
20100529 2054 - Das Erste HD - Eurovision Song Contest 2010 (Edited)-withcut PID 1786 3_2ch 48KHz 448Kbps DELAY 1513ms.ac3 <-- This is the real one
20100529 2054 - Das Erste HD - Eurovision Song Contest 2010 (Edited)-withcut PID 1786 3_2ch 48KHz 448Kbps DELAY 329ms.ac3

Why it creates the 2 other Files ? Its the absolute Same source File as in the first try.


Chris

[Guest]

Your full DGI confirms my theory about what is happening. It will be fixed in the next release.

Regarding the audio demuxing, I cannot help without a stream that I can use to duplicate the issue. I am unable to do that with the sample you gave me.

[um3k]

I am having a frame accuracy problem with DG NV tools when used in conjunction with MultiDecimate.

Here is the script:

foot = DGMultiSource("L:\8mmTest\8mmNoShut-2010_06_10-15_31_37.dgi").DoubleWeave()
#~ RestoreSuper8Frames(foot)
foot.ConvertToYUY2(interlaced=true)
MultiDecimate(pass=2)
SelectOdd()
FlipHorizontal()

It works fine at first, but after about 80 (decimated) frames, it begins returning the wrong fields. If I seek to another part of the video, it works fine again for a number of frames before reverting to the bad behavior.

Any idea what the problem could be?

[Guest]

I am having a frame accuracy problem with DG NV tools when used in conjunction with MultiDecimate.

Here is the script:

foot = DGMultiSource("L:\8mmTest\8mmNoShut-2010_06_10-15_31_37.dgi").DoubleWeave()
#~ RestoreSuper8Frames(foot)
foot.ConvertToYUY2(interlaced=true)
MultiDecimate(pass=2)
SelectOdd()
FlipHorizontal()

It works fine at first, but after about 80 (decimated) frames, it begins returning the wrong fields. If I seek to another part of the video, it works fine again for a number of frames before reverting to the bad behavior.

Any idea what the problem could be? I will try to duplicate it. Can you give me the settings you used for MultiDecimate, please?

[MOS-Marauder]

Your full DGI confirms my theory about what is happening. It will be fixed in the next release.

Regarding the audio demuxing, I cannot help without a stream that I can use to duplicate the issue. I am unable to do that with the sample you gave me.

You can duplicate it with the small sample u got from me...
SImply load it into DGindex NV and mark a start point for example at 1/3 and the others shortly before the end.

I dit it @ this Range:

DGAVCIndexFileNV5
C:\Program Files (x86)\megui-Dev-031-1061\tools\dgindexnv\

C:\DVD-HDTV\MOS-NFOGen-HD\tt\01-21_17-35-01_arte HD (ger)_Xenius.ts 99614720

STREAM 1
PKTSIZ 188
VPID 6210
YUVRGB 1
LUMFILT 0 0
CLIP 0 0 0 0
RANGE 31601952 0 67877116 0
AUDIO 184d,184e

SPS 27393899
PPS 27393940
SPS 31601903
PPS 31601944
RPS 31601952
I 31601979



Chris

[Guest]

You can duplicate it with the small sample u got from me...
SImply load it into DGindex NV and mark a start point for example at 1/3 and the others shortly before the end.Chris OK, thank you, those instructions have allowed me to duplicate it. It's bizarre.

Investigating...

[MOS-Marauder]

Hope u find it... it is really weird *G*


Chris

[Guest]

I've already found and fixed it. The fix will be in the next release.

[Guest]

A stream for you. Blu-ray source indexed by latest dgdecodenv and encoded by x264 with...bframes=8 , ref=16.
...
Can't see any of the artifacts in the screenshot below while watching with mpc-hc using latest coreavc in cuda mode.
Nvidia told me the cause of this and I made a fix. Level 5.0 allows 16 refs, and so you need 17 decode surfaces, but I had set the max to 16 because D3D mode supports only 16. So I bumped it to 20 for non-D3D mode. Such streams may still break in D3D mode but no-one should need that mode anymore. The fix will be in the next release.

Thought I forgot about this, huh? :)

[Audionut]

Thought I forgot about this, huh? :)

Actually, come to think about it, I had forgotten about it, and you asked me if it occurred with with level 4.1 or lower refs, and I can't recall if I replied. :eek:

Glad you got a fix coming though. Thanks Don.

[MOS-Marauder]

I've already found and fixed it. The fix will be in the next release.

Both Problems ? Wow xD Thx for Your hard work.

Hope the x64 Version is also working that good ;)


Chris

[tormento]

Err.. should x64 works with standard avisynth calls or it's like mix apples wit bananas?

Some filters, unfortunately, are still unavailable in x86.

[MOS-Marauder]

I guess at the beginning it will be nessesary to use a Wrapper or something like this ....

Chris

[Guest]

Err.. should x64 works with standard avisynth calls or it's like mix apples wit bananas?

Some filters, unfortunately, are still unavailable in x86. AFAIK, you cannot run 32-bit filters with 64-bit Avisynth.

For the x64 architecture, Avisynth, DGIndexNV, DGDecodeNV, DGMultiDecodeNV, and CUVID Server all will run in x264 mode.

BTW, I have just successfully run DGMultiDecodeNV in x64 mode.

The Win32 and x64 versions of DGDecNV will be functionally identical.

Yes, you will need to be patient as filters are ported to x64.

[Guest]

The 64-bit port is now complete. Everything works fine.

Tomorrow I will release build 2011 with the fixes and the 64-bit binaries.

[jpsdr]

Nice !!

[Audionut]

Sweet. Can't wait to test it out.

[MOS-Marauder]

Same here ;) Very sweet !


Chris

[Guest]

* The DGI file format version was bumped to NV6. So remake your projects
and INI files.

* NV12 to RGB code in DGIndexNV was converted to pure C for x64 compatibility.

* The PC/TV Scale option was removed. In the DGIndexNV display, PC scale is always used.
This affects only the display in DGIndexNV, and has no effect on video served by
your scripts.

* The Detected SIMD option was removed as there is now no asm code.

* An autocrop function was added to the cropping dialog.

* Fixed a problem causing spurious audio tracks to be demuxed.

* Fixed a problem with timeline navigation in DGIndexNV for some MKV files.

* Fixed a problem with AVC streams with 16 refs.

* Fixed a problem with sequence headers not being processed in some cases. This could
result in the frame rate not being correctly detected.

* Inlined a lot of the parsing code to improve indexing performance.

* Fixed a problem that caused some scripts to take a long time to be opened.

* Implemented 64-bit binaries (for power users only).

http://neuron2.net/dgdecnv/dgdecnv.html

The changes were extensive, hopefully I didn't break anything. I'm sure you guys will soon let me know if that is the case. :)

[simonb1975uk]

Am testing now. Will let you know of any bugs.

[tormento]

Neuron: what do you use as encoding gui in x64 environment?

[Guest]

Neuron: what do you use as encoding gui in x64 environment? I don't use GUIs for encoding in any environment.

[MOS-Marauder]

Testing ;).... btw i use MeGUI ....

Working like a Charm. All Bugs i mentioned gone... I always Start the X64 Cuvid Server and Index with the x64 Version.

Even when using MeGui and the 32bit Decode dll...works fine... also with 32bit Avisynth.


Chris