DG NV tools

[laserfan]

Fails with RipBot264 v1.14.5
LoadPlugin("C:\Program Files\DGIndexNV\DGMultiDecodeNV.dll")
DGMultiSource("F:\Work\00999.dgv", resize_w=1280, resize_h=720)I been using RipBot solely for its accurate calculation of a target x264 2-pass bitrate, and found today that changing my .avs files from DGSource w/CUVIDServer to DGMultiSource, and opening these now causes an error DGMultiSource "Failed to create video decoder (100)."

I dunno how Ripbot processes .avs files except that something called avs2avi.exe appears to be employed cuz I saw it running in the background using Task Manager.

This is just an FYI and no problem for me at present, as I can simply use instead DGSource w/CUVIDServer and get what I want out of RipBot. Then I run x264 using DGMultiSource as usual with my own custom cmd files.

[Guest]

* Full pulldown handling (Honor Pulldown, Force Film, Ignore Pulldown, frame repeats) implemented for all video types: AVC, MPEG2, and VC1.

* Prevent crash due to stream corruption erroneously causing the decoder to try to create a frame size bigger than 1920x1088.

* Fixed a bug where if a stream has a coded size of 1088 and you respond to the crop warning with "yes", cropping is not properly set. That happened only if the display rectangle was also 1088.

* Fixed bug that added an extra extension to the project file name if one was already present.

* A more informative error message is printed if an MPEG1 program stream is loaded.

* Removed a check that caused some valid audio streams to be rejected.

http://neuron2.net/dgdecnv/dgdecnv.html

Note that my web pages for AVC, MPG, and VC1 have now been consolidated into one page.

[laserfan]

Thanks for this, especially during the holiday! ;)

Enjoy the rest of yours, you deserve it.
:thanks:

[Turtleggjp]

Neuron2

I have a question about how your indexing tools identify frames in transport streams. I am working on a tool to correct audio sync caused by errors during recording. It currently uses a .dgm index file to determine when frame or field repeats are in the stream. I came across a recording where my program lost synchronization with the index file. After some investigation, I discovered that this recording has packets marked with the Transport Error Indicator Flag, 1197 of them to be exact. Of these, 22 have the Payload Unit Start flag, and another 5 have a valid PES start code. When I adjusted my program to ignore packets with the Transport Error Indicator flag set, it still picks up one more frame than your indexer does. I know this because at the end of the scan through the file, I can see that my program is expecting a field repeat one frame early.

So my question is, what does your indexer look for to decide that a new frame is starting? The clip I'm working on is MPEG2, and I am using the NVBeta10 MPEG2 indexer. Thanks!

Matt

[Guest]

In some previous versions I used to reject packets with transport_error_indicator set. The latest version ignores it.

For frame detection:

Wait for a picture coding extension and then:

if (picture_structure == 3 ||
(top_field_first && picture_structure == 1) ||
(!top_field_first && picture_structure == 2))
{
// New frame.
}

Note that the frame lines in the index file are sorted into display order per sequence before being printed to the DGM file. To do that I have to accumulate a full sequence before outputing it.

[Turtleggjp]

In some previous versions I used to reject packets with transport_error_indicator set. The latest version ignores it.

Ok, I'll try the newer version next to see how it does.

For frame detection:

Wait for a picture coding extension and then:

if (picture_structure == 3 ||
(top_field_first && picture_structure == 1) ||
(!top_field_first && picture_structure == 2))
{
// New frame.
}

Is the Picture Coding Extension in the MPEG2 data itself? So far I've only written code to handle the data in Transport Packet Headers and PES Headers. I'm assuming that what you are referring to is not in there.


Note that the frame lines in the index file are sorted into display order per sequence before being printed to the DGM file. To do that I have to accumulate a full sequence before outputing it.

Yeah, that was a hurdle I overcame a long time ago. My program is already handling my recordings that have pulldown just fine. This one though is proving to be difficult.

One more thing, does the non-NV indexer detect frames the same way? I don't use it much anymore, but I would like my program to be able to use those indexes too someday.

[Guest]

Is the Picture Coding Extension in the MPEG2 data itself? Yes, of course. It's part of the MPEG2 video syntax. You can't reliably find frames just using the transport layer and PES syntax.

One more thing, does the non-NV indexer detect frames the same way? I don't use it much anymore, but I would like my program to be able to use those indexes too someday. I believe it is the same. The source for it is available.

If you need any help finding the cause of discrepancies between our methods, let me know. I do think you'll need to parse the video syntax, though. It's not so hard.

[SilaSurfer]

Hey neuron2 thanks for the tool it looks very good. I'm going to test it, is there any visual quality boost compared to Dgindex? I was thinking of using it with DVD sources if this is supported? I need to update my Geforce 9500Gt drivers, the latest are 195.62 version, is this ok? Thanks

[Guest]

is there any visual quality boost compared to Dgindex? No. The benefit over DGIndex is a) offloading some CPU load to the GPU for transcoding, b) resizing/deinterlacing performed on the GPU.

I was thinking of using it with DVD sources if this is supported? Yes, it is supported, although I am in the process of adding support for DTS audio.

I need to update my Geforce 9500Gt drivers, the latest are 195.62 version, is this ok? Yes, this will be fine.

[SilaSurfer]

Ok Thanks, going to try it out.

[SilaSurfer]

What about the settings in Nvidia control panel, denoising and edge enhencment are those supported in your tool can I enable those to use with decoding?

[Guest]

The capabilities of the tools are documented in their manuals. If you don't see a feature described then it is not supported. As mentioned several times, I can support only the features available in the NVCUVID API, and currently it does not support the things you mentioned.

[SilaSurfer]

Ok neuron2 thanks for your time, I will try your tool, keep up a good work.

[DonDan]

There seems to be an issue with this version when decoding interlaced VC-1 content.

First I thought I had [] when it crashed for all 5 blurays I have here, but in fact all 5 are interlaced VC-1 (don't ask me where I get all that [] from^^)

So I had a friend test some files for me: Non-interlaced VC-1 was fine with him, interlaced crashed right after starting the indexing process. (Same as for me)

I had him test this file: http://www.mediafire.com/?zjm4jyywmdl which happens to be the same you wrote the stream corruption fix for, so I suppose the bug was introduced after this fix ;)

Anything else I can provide to help you spot and annihilate the bug?

[Guest]

You better stop the profanity or you'll start getting rule 4 strikes.

Investigating your issue...

[Guest]

That's the stream with corruption. Where did it come from? And what do you mean by "don't ask me where I get all that"?

[DonDan]

Oops, sorry for the vocabulary ;)

Yes, the sample is the same because I had it handy. It happens for ALL 5 interlaced VC-1 blurays I own though.
In beta 3 I can index that sample, but it crashes while encoding. In beta 4 it crashes right after I start indexing and does so for all interlaced VC-1 as it seems.

By "don't ask me where I got all those from" I was just saying that I don't know why out of 5 blurays I have 5 interlaced VC-1 as they are not *that* common after all and have always given me trouble ;)

[Guest]

You did not answer my question about where you got that sample. Please answer.

[DonDan]

Ah, missed that one.
It's from a King of Queens Bluray that was only released in Germany so far if I'm not mistaken. When I noticed the issue I even got another copy from a rental store and tried to encode the episode from that disc with the same result. So I believe the issue should not be a corruption on my side.

[Guest]

What did you use to get that M2TS from the disk, both for ripping and demuxing? I ask because there is strange data that is causing the corruption.

The indexing crash I understand but not the corruption half way in.

[DonDan]

For ripping I used that tool from a company in St. John's in Antigua and Barbuda that you are not allowed to possess, use or talk about in most of Europe.
The sample was created in tsmuxer which was the only programm I found that was able to actually process the file without crashing in the same place DGTools did.

[Turtleggjp]

Yes, of course. It's part of the MPEG2 video syntax. You can't reliably find frames just using the transport layer and PES syntax.

I believe it is the same. The source for it is available.

If you need any help finding the cause of discrepancies between our methods, let me know. I do think you'll need to parse the video syntax, though. It's not so hard.

Thanks for your help. I think I found all the info I need to start coding for this. This should be fun...

[kuko61]

Hi neuron2
At the beginning I would like to thank you for the excellent utility :)

I am a user DgindexNV few days. I tried to index and encode some samples. With the AVC and MPEG2 streams I had no problems mentioned here. But with interlaced VC-1 I have a similar problem as DonDan.

1. DGindexNV also crash me, when I tried SaveProject. But when I open a VC-1 stream through the Open menu, close DGindexNV, start it again, and open stream from the list of recently used files, indexing run without errors.

2. Second problem occurred when encoding VC-1 interlaced sample (with x264). Encoding always freezes at approximately 95%. Without any errors, just stopped.
I tried encode a few samples (30s - 2 mins) from different parts of the BD disc and encoding is always stopped before the end (for ripping I use same tool as DonDan and samples I cut through tsMuxer). I tried Avidemux, meGui, batch file (CLI) with the same result.
DGindexNV crash and encoding freeze issues occurs only with b4 version (not b3).

3. The third problem occurred with a length of VC-1 indexed video, when I play the AVS script with dgv project in the player. At two-minute sample is the video longer about 1 second as a source. When I indexed the complete m2ts stream without cutting (lenght 1h16min), the video length is 1h18min. I tried all three Field Operation options and video from AVS script is always longer than the original m2ts stream.
In script is only DGDecodeNV filter without arguments...

Sorry for my English, the post was written with the assistance of the dictionary :cool:

[turbojet]

Thanks for the extension fix. I don't know if you noticed my edited post earlier but is the resize/scaling code in dgnv, nvdia drivers or part of the video card?

If it's part of the video card, I see PureVideo 4 (VP4) offers a new high quality scaler according to wikipedia (http://en.wikipedia.org/wiki/Purevideo#The_Fourth_Generation_PureVideo_HD). Would that make a difference?

[Guest]

@kuko61

Can you provide one of those streams for testing?

@ turbojet

It's on the GPU but I can invoke only what is made available in the NVCUVID API.

[turbojet]

Is the scaler configurable to you?

[Guest]

Is the scaler configurable to you? I can set the width and height. :devil:

[turbojet]

OK, I'll see if I can get my hands on a VP4 card in the near future and see if the scaler quality is the same or better then VP3. Currently it looks like nearest neighbor, which is tolerable but trying to find something like lanczos.

I did a little digging into dgindex running slower when it's not minimized and found out that if the information window is closed it speeds up again. I notice dwm.exe uses 1-2% cpu when the info window is open and 0% when it's closed/minimized. However it's difficult to tell when it's done when it's minimized. Is there a chance to show progress % and/or ETA in the title bar and/or windows 7 progress taskbar?

[Guest]

It's bilinear.

Will look at your performance observations when I get a chance.

[Hunstad]

I have some serious random access problems on this H.264 stream with beta 4. It works perfect in beta 3.

http://www.mediafire.com/?nydngzzwrzj

The 9 second clip is from the Scandinavian Blu-ray of Monsters vs Aliens.
Disc decrypted with AnyDVD 6.6.0.3, and the clip is cut with beta 3 Save Project and Demux Video. (Beta 4 gave the same md5sum on the cut.)

I index the file in beta 4 and load it in VirtualDub 1.8.6 using this this AviSynth 2.58 script:

LoadPlugin("C:\Video\DGDecNV200b4\DGDecodeNV.dll")
DGSource("E:\Blu-Ray\00024.200b4.dga")
Crop(0,132,0,-132) # 2.35:1

The I try to jump to Frame 76, but this frame is corrupt.
If I step forward 1 frame at a time, from frame 0, the close-up starts at frame 76.
If I step forward 1 frame at a time, from frame 50, the close-up starts at frame 52.

This is not a problem in beta 3.

Anders

[Guest]

Ah, I love clear trouble reports with everything I need to duplicate the issue. Thank you for pointing this out. You're my kind of guy. :)

Please redownload 2.0.0 beta 4 and update DGIndexNV.exe.

Should be OK now. Only AVC with recovery points was affected.

[whatever-ever]

neuron2 I've got a new problem.

If you remember, on page 30 we talked about combining files. Making an index of all files (into one dgm) did work, but now I've found another, related, problem.

I have 5 files. together they give 58minutes and 25 seconds (played in VLC/mpc/wmplayer).
The dgm (containing all 5 files) is not. It is 53min and 29 seconds.

I tried indexing all 5 files individually, and then add up their playlength, and it gave me 29:34 (correct).
I then tried an AVS like this:
DGMultiSource("1.dgm") ++ DGMultiSource("2.dgm") ++ DGMultiSource("3.dgm") ++ DGMultiSource("4.dgm") ++ DGMultiSource("5.dgm")

and it gave me 53:29 (wrong).

btw. if i leave out file 5 then i get the same length as 1-4 played in mpc. So is this some kind of dgindex buffer I'm exceeding?

PS: I did this using beta3 and beta4

[Guest]

If I give you an FTP site, can you upload all 5 source files?

It may be a pulldown issue. What is your field operation and do you see field or frame repeats when previewing any of the files in DGIndexNV?

I tried indexing all 5 files individually, and then add up their playlength, and it gave me 29:34 (correct). You said it is 58:25 in VLC, so where did this come from?

[whatever-ever]

Oops, sorry. I meant they added up individually gave 58:25. I don't know where the 29:34 came from, that was a mistype, lol.

Yes I can get you the files, or a sample if you just need to see it's structure. I've encountered differences in runtime and "indexed" runtime before, where the problem was, as you suggested, a pulldown issue (set it to hono pulldown flags and it was correct size).

THIS situation didn't benefit from honoring pulldown flags. and in fact it gave the same (incorrect) 53:29 length if honored and if ignored pulldown flags. So maybe that's a hint of the problem?

[Guest]

I'll need all 5 files. What are their sizes so I can clear space on my FTP?

[whatever-ever]

they total 4.8 gb

[turbojet]

Link in the original post is dead, seems to be only up to beta3 on that site.

[Guest]

they total 4.8 gb Well that's a deal breaker unless you want to snail mail it to me.

Did you actually play the whole thing in VLC to see the duration, or are you trusting what it says in some info dialog?

As there is a discrepancy of 5 minutes, when you serve the video into VirtualDub and play it, do you see 5 minutes of missing material?

[whatever-ever]

Did you actually play the whole thing in VLC to see the duration, or are you trusting what it says in some info dialog?

Yes, and also in wmplayer (and mpc).

As there is a discrepancy of 5 minutes, when you serve the video into VirtualDub and play it, do you see 5 minutes of missing material?

I'm watching them played back simultaniously right now to check where it differs. I'll post result in an hour.




The demuxed audio (from both dgindexNV and regular dgindex) are 58 mins.

Also, using DirectShowSource("1.mpg") ++ ... (relying on ffdshows' libavcodec) gives 58 mins too.

Neither dgindex nor dgindexnv gives any errors when indexing.

If you want the log files from dgindex(nv) then let me know.

[whatever-ever]

What the...!? After I opened the directshowsource avs (in mpc) the dgindex avs'es are 58:25 mins too!

If I close the directshowsource avs, then both dgindex avses are 53:29 again. WHAT!?

To say again: ffdshow is set to MPEG2: libavcodec, and ffdshow is used when calling DirectShowSource().
What in the world... Does this make any sense to you, neuron2?

[Guest]

If I close the directshowsource avs, then both dgindex avses are 53:29 again. WHAT!? How do you determine the length of the video served by the DGDecNV script?

Just open it in VirtualDub and look at the duration in File/Information. If that changes due to running your other script I'll be shocked.

[whatever-ever]

In Vdub both dgindex and dgindexnv return 53:29 (also with directshowsource running in mpc in the background), directshowsource returns 58:26. If dgindex(nv) is played in mpc while playing directshowsource in mpc they both run 58:26 (in mpc). See this: http://i48.tinypic.com/1zpjq6w.png

Hmmm...

[dgeise]

hey Don,

Haven't used your tools for a while. Installed beta 4 recently and ran across a few new issues.

1. There is what appears to be a new filename parsing bug. A real simple one. I copy/b my AVCHD .MTS files together and rename them to a YYYY.MM.DD.MTS format for archiving purposes. The problem is that DGIndexNV outputs files (for example) YYYY.MM PID 1100 2_0ch 48KHz 256Kbps DELAY 0ms.AC3 and YYYY.MM.DGA filenames. This is no big deal, I just rename the files and all is well. My guess is that you're parsing for the '.' (dot) file extension separator forwards thru the filename string instead of backwards. So you pick up on the first dot and not the last dot in the filename and split the name from the extension based on that. Reverse the string search and it should be fixed. Easy fix, dont you love those?

2. I ran into a 'DGSource: Invalid license' error. It turned out I had installed VirtualBox on this system recently and its tripping up your MAC address detection logic by using the virtual adapter's info instead of my real hardware's info. That's not real good in several aspects- validating to a virtual nic is a crack in the door for piracy among other things. Right now I have to disable VirtualBox's virtual network adapter to get your tools to work. I hate DRM mostly for reasons like this. I'd suggest you enumerate all NICs, maybe filter out the virtual ones if there is a way to detect that, then try to validate the license against all remaining 'real hardware' entries. That should eliminate confusion in several scenarios I can think of.

Ok, keep up all the hard work... The tool is looking good.

-Dave

[Guest]

In Vdub both dgindex and dgindexnv return 53:29 (also with directshowsource running in mpc in the background), directshowsource returns 58:26. If dgindex(nv) is played in mpc while playing directshowsource in mpc they both run 58:26 (in mpc). See this: http://i48.tinypic.com/1zpjq6w.png

Hmmm... I think 53:29 is the correct running length and the time given in MPC is wrong. Unless you want to mail me the streams, I don't know how I can do anything else for you.

[whatever-ever]

I think 53:29 is the correct running length and the time given in MPC is wrong. Unless you want to mail me the streams, I don't know how I can do anything else for you.

While that is very possible, and I hope it is so, it still remains that the files (individually indexed) added up gives 58mins, and the full demuxed audio (demuxed with dgindex(nv)) is also 58 mins :(

[Guest]

While that is very possible, and I hope it is so, it still remains that the files (individually indexed) added up gives 58mins, and the full demuxed audio (demuxed with dgindex(nv)) is also 58 mins :( Well, that is a problem.

Do you want it resolved? If so, I will PM you with my address to send me the files by mail. Or you can make them available on an FTP site for download.

One thing that could help is for you to post here the DGM files for all of them done individually and for doing them all at once.

[Guest]

1. There is what appears to be a new filename parsing bug. A real simple one. I copy/b my AVCHD .MTS files together and rename them to a YYYY.MM.DD.MTS format for archiving purposes. The problem is that DGIndexNV outputs files (for example) YYYY.MM PID 1100 2_0ch 48KHz 256Kbps DELAY 0ms.AC3 and YYYY.MM.DGA filenames. I can't duplicate that with 2.0.0 beta 4. Give me an exact filename and tell me how you make it fail.

2. I ran into a 'DGSource: Invalid license' error. It turned out I had installed VirtualBox on this system recently and its tripping up your MAC address detection logic by using the virtual adapter's info instead of my real hardware's info. Use the license generator to generate a license number for each configuration and put them all in your license file.

[Jeff Flowerday]

The new DGIndexNV seems to struggle with DTS audio tracks now, with the old V11 beta it would load up fast and just not show the DTS track. Now when I add vob files containing a DTS track, it sits there for a couple minutes "Not Responding" and still doesn't show the DTS track.

It's relative to size, a small 50MB clipped vob will load decently fast.

http://www.mediafire.com/?jtbgluijiok

[Jeff Flowerday]

PS) DGIndexNV still indexes over twice as fast minimized as it does as an open window.

[Guest]

PS) DGIndexNV still indexes over twice as fast minimized as it does as an open window. Please try closing the Info diaglog after starting the Save Project. Does that make a difference when not minimized?