DGMPGDec 1.2.0 beta 6

[Guest]

In theory, yes. But I prefer the KISS approach. :)

[darkluna]

I'm having weird problems with the later versions of DGMPGDec.

Basically I have several .TS files that I'm making the d2v file from and serving them via simple Avisynth script into Virtualdub. Moving around the files from within DGIndex is normal. However, moving around in VirtualDub, there seems to be a problem with the timeline -- scenes jump around in the wrong order -- definitely not linear. Strangely, the total show time is correct and does not change.

I've verified I'm using the right combination of DGIndex.exe and DGDecode.dll

[tritical]

This post is most likely off topic, feel free to move it wherever; however, since the discussion started here I don't know where else to put it.

First, if you've already decided to do it the other way that's perfectly fine... your the one doing it of course :). However, after my initial claim that it would not be "nice and clean", the "in theory" comment, and there being no examples anywhere of it actually being done, I sat down and made a very simple prototype filter actually showing how it can be done. Surprisingly, it is very simple. Here: TestVar.cpp (http://bengal.missouri.edu/~kes25c/TestVar.cpp). Actually, the first filter simply allocates an avsvalue array (which could contain the needed data referenced by frame number) and then env->SetVar() and env->GetVar allow you to pass a pointer to that array to other filters so they can look at it as well. Which basically solves the entire problem of passing data. Anyways, I didn't see any other examples of how to do this so figured I'd post it.

EDIT: Or instead you could be smart and just use SetVar() and GetVar() to pass an array address and avoid having to use an avsvalue array altogether... should have thought of that before :sly:. Example: TestVar2.cpp (http://bengal.missouri.edu/~kes25c/TestVar2.cpp).

[Guest]

Originally posted by darkluna
I'm having weird problems with the later versions of DGMPGDec. What version are you using? Please try 1.2.0b5 and report back. Are you doing a selection? Tell me as much as you can so I can duplicate the problem.

EDIT: Also, don't rip in file mode. You'll mix up multiangle stuff:

http://forum.doom9.org/showthread.php?s=&threadid=32761&highlight=multiangle

[Guest]

@tritical

It seems to me that managing and purging the frame array and keeping it consistent when caching is going on is the hard part.

[Guest]

Originally posted by marcellus
Edit: I checked and it seems that my upload failed (my line dropped). I will send it again. Did you do that yet? I just checked and that file also has no sequence headers. Are you using binary mode for your transfers?

[tritical]

Hm, if the main filter uses the array as a circular linked list and always inserts info from the last frame it processes into the next slot, I don't think there would be any problems. Since filters later down the chain could not be processing any frames not already processed above. As long as the array maintained enough entries to bypass all caching and any temporal filter look aheads it would be alright, and the other filters could simply search through the array to find the data that goes with the current frame number they are processing. While this method would get around not being able to use any filters that modify the first 64 bytes (smoothers, resamplers, some deinterlacers)... it would not work with filters that removed or added frames and that is not acceptable :(. So in the end, I guess it is not better for the current purpose.

[Guest]

It's a matter of "choose your evil" as long as Avisynth doesn't cleanly support carrying per frame data along.

Here's an old thread that gets into all this in agonizing detail:

http://forum.doom9.org/showthread.php?s=&threadid=46007

[len0x]

Does this all mean that currently there is no way of making ColorMatrix use info from mpeg2source() for interlaced source?

P.S. Btw, can hints be catched the same way as "debug" information? (I'm just not sure how hints are working)

[Wilbert]

Does this all mean that currently there is no way of making ColorMatrix use info from mpeg2source() for interlaced source?
We can add an interlaced mode in ColorMatrix itself.

[Guest]

Originally posted by len0x
I'm just not sure how hints are working The low order bit of the first 64 bytes of the video itself are stolen and data is written there. It is completely imperceptible. The first word of the data is a magic number. If the receiving filter sees the magic number in those bits, it knows that there will be good data in the second word.

[len0x]

IC. Interesting...
Can debug parameter be added to mpeg2source to produce the same information as info=true does?

[marcellus]

Yes, I guess if you successfuly opened the archive everything is fine. Oh, it's an archive is it? Well it doesn't open with anything I have. What is it?

EDIT: Darn, I keep hitting the edit button instead of the reply button. I need more sleep. Anyway, how do I open this archive?

[len0x]

Also I have an idea for ColorMatrix purposes. Have an auxiliary function that catches hints from mpeg2source and use them in any form feasible (may be like tritical proposed in some cases). Later ColorMatrix can use that info from auxiliary function. So this ensures that Don has to make hints thing working only once and don't change implementation even if another way of parsing frame info is found/suggested. This auxilary function will be a part of ColorMatrix plugin and can be developed totally separate from DGDecode.

[Guest]

@marcellus

How do I open a .7z file? Can't you just put the stream there directly?

@len0x

The data has to be put somewhere that the receiving filter can access.

[len0x]

I know, but implementation can be imperfect (i.e. for DVD sources it may be unlikely that different coefficients are used throughout the movie. That's why I asked if info() can be exposed in debug mode so that I can try running some automatic tests to see if we really need frame-by-frame info for ColorMatrix).

[tritical]

Hm, one more thought... with the imbedded hints method you could make a generic wrapper filter that could preserve the hints accross filters that would ordinarily destroy them. One single generic version could not work for all filters, but it would be able to work for any filter that is 1:1 input output, such as resamplers, most denoisers, etc...

[Cyberia]

Originally posted by neuron2
How do I open a .7z file?
Type "7" and then "z" into Google, and take the first link. :p

EDIT: I make no claims as to the quality of the 7z software. (Doesn't work for me)

[Guest]

So to save less than 2MB on an upload, marcellus is forcing me to install some SW I don't need or want? Sorry Charley.

[Leak]

Originally posted by neuron2
So to save less than 2MB on an upload, marcellus is forcing me to install some SW I don't need or want? Sorry Charley.

Well, both TUGZip and WinRAR have no problems uncompressing 7Zip files, so if you happen to use one of those...

7Zip is an open-source packer providing better compression than zip, which IMHO is great. Yeah, RAR's not bad as well, but it's still proprietary.

np: Ido Gouvrin - Breaking Through The Frequencies (Clicks & Cuts 4)

[Guest]

You missed my point. I don't want to install it. He can please upload the uncompressed file. I estimate that it will be 9MB instead of 7MB.

[Cyberia]

Don, I uncompressed both 7z files and uploaded them to your site. Hope that was OK.

gop_499.m2v
long_gop.m2v

[Guest]

Sure, no problem.

So his 499 picture GOP makes a line in the D2V that is 1500+ characters long. Would *you* have thought of making your sprintf() temporary buffer that long? :) I've fixed it in beta 6, to be released shortly.

Beta 6 also fixes the long-standing decode bug I mentioned. We will be much more error tolerant. It'll still report the error in the info box, because it is a real stream error it turns out, but we won't macroblock the pictures all to hell. It's at least as tolerant as VirtualDub MPEG2 now. Thanks for access to your code, fccHandler!

[fccHandler]

K :)

[Guest]

http://neuron2.net/fixd2v/dgmpgdec120b6.zip

Relative to the brain-dead 1.1.0:

1. Adds demuxing of AAC audio from transport streams.

2. Adds preliminary support for PVA streams.

3. Adds demuxing of video from VOBs (programs streams).
Transport and PVA coming.

4. Fixes crash with long GOPs.

5. Fixes error tolerance on stream video errors.

6. Fixes problem with indexing for some selection operations. (This is a killer for 1.1.0.)

7. Further improvements to the Info dialog.

[marcellus]

Hi Neuron2
I am really sorry, that me compressing with 7-zip a m2v file sounded like I want to make you install anything, I'm not so arogant. I just thought that is a commonly enough spreaded compression, especially in open source world, so it wouldn't be a problem. I was wrong and I appologize for that.

Is true that the obtained compression ratio is only 75-80% but also true is that I'm on a dial-up connection that, beside being expensive, is highly unreliable, so I tried everything to accomplish a successful transfer before anything bad happens to the line (since AFAIK there are no such thing as resumed upload - or there are? -my experience in uploading is next to 0).

I owe Cyberia my gratitude for being so nice to unzip the files. Thank you!

That being said, I downloaded beta 6 and I can confirm that it works flawlesly on my files. Thank you, Neuron2, for taking into acount this rather exotic issue and putting the effort to solve it!
Bye.

[TEB]

@neuron2: Beta6 seems fine but i still get a Video Error 1 and later on a Video Error 2 when previewing the u571.remuxed.ts file..
It seems ok so i shall not nag !:) just a tiny FYI!.

I tested some other files now and im getting a: Video error 5, 1, 10.. also from time to time.. are these error codes?

[Guest]

Those errors occur due to real errors in your MPEG2 stream. Are you capturing from a bad signal or something?

The numbers are the "Fault_Flag" value:

1: bad block coefficients
2: bad macroblock type
4: bad macroblock address increment
5: bad prediction
10: bad motion code
99: newbie detected

[TripleA]

Originally posted by neuron2
So to save less than 2MB on an upload, marcellus is forcing me to install some SW I don't need or want? Sorry Charley.

This point perhaps is not too significant, but having a wrapper would ensure the file's integrity: if the upload/download is broken for whatever unlikely reason, the extraction will fail.

Admittedly the same effect can be achieved with any compression/packaging.

But 7z is actually quite nice, to go a bit OT, in my experience. Best compression I saw, if a bit slow. And has always worked, for me. Please give it another try when you compress something next.

[TEB]

@Neuron2: No thats why im worried... Its transports from our VOD service at work.. And im testing the content i have + DVB satellite streams to make dgindex robust..
It is possible to make a loggingwindow with some summary of the errors he finds? Would be very interesting..
Specially like the 99 one ;) hits me on the spot !! haha

[Guest]

@marcellus

I'm feeling guilty for making a big deal out of the 7z thing. Don't mind me. Thanks again for pointing out the issue.

@TEB

It's possible but triage will keep it low on the list for a while. Right now, I want to finish up video demuxing, fix problems with the audio delay calculation, and add hints for a 1.2.0 release. Quite possibly, you'll get debug output in DGDecode, where I can spit out the video errors along with the per-frame info.

[TEB]

goodie!

[MLS]

Thanks for all your hard work Neuron.

/MLS

[Black Hole]

@neuron:

A couple of days ago, you wrote the features of DGIndex 1.2.0 beta 6, relative to the "brain-dead" 1.1.0 version.

Does that mean that 1.1.0 is not safe to use with normal DVD movies? Should we still use 1.0.12 until 1.2.0 is out?

Thanks.

[Guest]

@Black Hole

Your reading comprehension is excellent. :) It has problems with some selections (setting left point) but is OK if there is no selection.

I withdrew it from my web site as well. Use the 1.2.0b6. It's miles ahead.

I'm looking at delay calculations right now. It's very complex and confusing, and the PTS/DTS values I see on DVDs aren't making sense right now. It may turn out that simple empirical syncing is the most reliable way.

[darkluna]

I'm working with HDTV .ts files, so no multi-angle-ripping is involved. I'm using version 1.2.0b6 and making a .d2v from all of the files, not just a selection.

The problem occurs right at .ts file-segments.

If I manually step through the file frame-by-frame in VirtualDub where it switches from one .ts file to another, it works; however if I use the "jump to a frame" (Ctrl-G) to the same problem area, it will give me the wrong frame.

Originally posted by neuron2
What version are you using? Please try 1.2.0b5 and report back. Are you doing a selection? Tell me as much as you can so I can duplicate the problem.

EDIT: Also, don't rip in file mode. You'll mix up multiangle stuff:

http://forum.doom9.org/showthread.php?s=&threadid=32761&highlight=multiangle

[lineman]

I have one large request that would make DGMPGDec a more awesome tool. For those of us that use it to downconvert hdtv to dvd, if DGMPGDec could do trims (cut the commercials) and still retain the ac3 audio. Right now I use projectx, which is ok but you have to have lots of harddrive. You can cut the commercials in avisynth but then you cant retain the ac3 audio. There is a prog called mpg2cut2 for cutting hdtv but it cant cut ts files only program files. Just an idea...if it to way out dont worry about...oh yea thanks for all the hard work that you have done for all of us

lineman

[Guest]

@darkluna

I would need the two files. How big are they? We need to resolve this because I want to release 1.2.0. I'll try it on my own transport files, but if you can give me ones that fail in a duplicatable way, that would be best.

@lineman

Cutting is on the development list. Out of curiosity, though, why would you want to cut video but keep the corresponding audio?

[lineman]

Originally posted by neuron2

@lineman

Cutting is on the development list. Out of curiosity, though, why would you want to cut video but keep the corresponding audio?

I read my post and I see what I typed isnt what my brain was thinking. I want to be able to cut the audio and video with DGMPGDec. I am glad to see that you have that on the list...

Thanks

Lineman

[darkluna]

I found one of the smaller ones -- the first file is 308MB, the second is 986MB

Originally posted by neuron2
I would need the two files. How big are they? We need to resolve this because I want to release 1.2.0. I'll try it on my own transport files, but if you can give me ones that fail in a duplicatable way, that would be best.

[Guest]

Trim the second one to 100Meg (from the back!). Then upload them to my ftp site:

ftp.neuron2.net
user: guest@neuron2.net (enter it just like that)
pwd: guest

Edit: If they are field structure and you have not kept an even number of fields in your first file, you will be in trouble.

[darkluna]

I've uploaded the files, but I don't know if they're going to do you much good. I made a new .d2v from the files I've uploaded and I can't duplicate the problem, but it still exists with the original file-set. I'm also uploading a sample avi (check frame 3287).

I recently started using a newer version of HDTVtoMPEG (1.11b) to create the .ts files, maybe that has something to do with it.

Thanks,

Originally posted by neuron2
Trim the second one to 100Meg (from the back!). Then upload them to my ftp site:

ftp.neuron2.net
user: guest@neuron2.net (enter it just like that)
pwd: guest

Edit: If they are field structure and you have not kept an even number of fields in your first file, you will be in trouble.

[fewtch]

Originally posted by neuron2
You missed my point. I don't want to install it. He can please upload the uncompressed file. I estimate that it will be 9MB instead of 7MB.
How about the command-line version of 7-zip (7za)... no installing necessary, just drop it in a shared folder somewhere. Works fine here.

http://www.7-zip.org/download.html