DVD2AVIdg Version 1.3.0 with transport inside! [final]

[BoNz1]

neuron2, I retested with RC5 and everything works as it should. Thanks :).

[Guest]

Great, and thank you! Then, unless there are objections, I will release 1.3.0 final. Going...going...

[Guest]

.....GONE!

Version 1.3.0 final and source code is available here:

http://neuron2.net/fixd2v/decodefix.html

I'd like to sincerely thank the people in this forum for suggestions and feedback on the betas/RCs. You can't believe what an incredible resource you are to a developer! I simply stand in awe.

Look for a new thread about version 1.4.0 feature definition. The train's still chuggin'. Y'all stay on board, ya heah?

[Guest]

Originally posted by Lord_KiRon
The "problem" is that it "looks" like the whole episode is a film , the only thing I can think of is that episodes that are not detected as film were edited after telecine ... The film detection is based solely on the RFF flags in the stream. It is possible that the problematic episode was "hard telecined", that is, 3:2 pulldown was applied prior to MPEG encoding and RFF flags were not used. Or, it's possible that they had hybrid or CG scenes and didn't apply RFF to those, thereby lowering the film percentage.

[CruNcher]

@neuron2
congrats for the release :)
at which version number pva support will be introduced ?

[Zep]

Originally posted by neuron2
@Zep

If you pass a stream to MPEG2DEC3dg with incomplete or corrupted final pictures, you will get what you deserve. The only thing I can do is not crash.

EDIT: It is *possible* to suppress the last picture but I think it runs against my general philosophy of delivering all pictures in the stream. If there is a picture header, as far as I am concerned, there is a picture. :p

oh I totally agree. That MPEG2DEC3dg should pass all frames
BUT IMHO dvdtoavi should NOT index currupted frames if it spots
one.

Hmmm... ok that may be TOO extreme so

better yet at least throw an error to a log so the user knows
what frame to trim and doesn't go to encoding thinking all is
fine and can run fix and parse which may help and/or may help the
user decide if doing a trim on a bad frame in the middle of a
scene is worth it.


Anyway, there is no messed green frames when i play the file
in the HDTV player that comes with the card. now the HDTV app
COULD have code that fixes it since it created the file. who
knows.

like i said i really do not care since i always cap a few minutes
long anyway and trim it out. For me the big issue was CRASHING on
the last ***GOOD*** frame of what was there after a trim.

That appears to be fixed. I was never able to figure out WHY
it did it. I have a hunch though. when scanning for editing,
VDub via MPEG2DEC3dg/avisynth feed always chooses the same
frames on a quick slide of the preview slider. To get to frame
level editing I have to use left and right arrow keys. When I
leave the edit at a quick slide (perhaps GOP level? or even bigger
VDub/avusynth decide on the jump amount?) it never crashes.
When I edit at frame level it is a crap shoot.

[Zep]

Originally posted by neuron2
.....GONE!

Version 1.3.0 final and source code is available here:

http://neuron2.net/fixd2v/decodefix.html

I'd like to sincerely thank the people in this forum for suggestions and feedback on the betas/RCs. You can't believe what an incredible resource you are to a developer! I simply stand in awe.

Look for a new thread about version 1.4.0 feature definition. The train's still chuggin'. Y'all stay on board, ya heah?

haha i barely had time to test RC5 :D


cool I will DL it now and jump over to the new thread.

[Guest]

Originally posted by CruNcher
at which version number pva support will be introduced? It will be introduced in the first revision at which it is fully functional. Currently video is working fine but the audio is not cooperating. It should be easy and the code is short and sweet for the audio. But when I pull out the audio it has chirps on top of the real audio. I took the audio out using other tools and compared the resulting ES streams. Mine has extra audio frames that really are in the stream and I can see nothing in the spec that tells me to exclude these frames. So I'm still scratching my head about that. Any ideas?

I do have some source code for Project X and plan to look at it to see if there is anything in there that will explain this.

[Guest]

Originally posted by Zep
That appears to be fixed. I was never able to figure out WHY
it did it. It's very simple. The decoder started decoding the last incomplete picture and while processing a slice the valid bits ran out and it just started grabbing random stuff from the bit buffer and crashed.

I'm considering ignoring the last frame if it is incomplete. I think it's important to be robust in that regard.

[Nebob]

How is mpeg2dec3dg.dll built? Is there any sense in attempting further optimizations with ICL and so on? I will try it and report my results if it hasn't been done before.

[Guest]

Originally posted by Nebob
How is mpeg2dec3dg.dll built? Is there any sense in attempting further optimizations with ICL and so on? I will try it and report my results if it hasn't been done before. It's built using VC++ 6 with sp5 and processor pack. Any speed improvement ideas will be gratefully received.

[Nebob]

Originally posted by neuron2
It's built using VC++ 6 with sp5 and processor pack. Any speed improvement ideas will be gratefully received.

Ok, I'll test some builds and see what I get. You may also be interested to know that SP6 was released back in march. I know, it snuck up on me too. ;)

edit: I did some tests with ICL and observed no practical difference.

[Zep]

Originally posted by neuron2
It's very simple. The decoder started decoding the last incomplete picture and while processing a slice the valid bits ran out and it just started grabbing random stuff from the bit buffer and crashed.

I'm considering ignoring the last frame if it is incomplete. I think it's important to be robust in that regard.

please do. if it can't be encoded it should never be sent.


nothing worse than doing a 2 pass you start it,
leave for work, get home and it crashed on the last frame of
pass one and all the time you had to encode was lost.

one of those things that makes me go grrr.........


i wonder why VDub crashes AND closes. I can see it crashing
and throwing up a "do you want to send a crash bug report" but
VDub closes cleanly and even saves the video.pass file
(cache is flushed and file is complete so it seems as if
VDub/Xvid did do a close() on that pass file)

very head scratching LOL

[Guest]

Originally posted by Nebob

edit: I did some tests with ICL and observed no practical difference. That was my experience, too.

[Cyberia]

edit: I did some tests with ICL and observed no practical difference.

The actual decoder and idct algorithms have been repeatedly optimized by many fine programmers here. Now maybe some guru will come along and fix a bug no one noticed that speeds things up, or will think of a revolutionary way to speed the process up, but... I'd say it's about as fast as it's going to get (within +/-5%)

You probably won't be able to get any significant performance improvements unless you manually walk through the compiler and hand tune each function, and even then I'd be surprised if it was a quantum leap.

[Guest]

Sure, but let's not foreclose the possibility! Who knows what a guy with fresh thinking can come up with?!

[TEB]

Nuron; Ive tested it now with my SPTS testsuite and it crashes on most of my streams. PM me and ill give ya an account on the ftp so we can check this out.
We're talking DVB SPTS, PAL world SD AND HD.

best regards
teb

[Guest]

Originally posted by TEB
Neuron; Ive tested it now with my SPTS testsuite and it crashes on most of my streams. PM me and ill give ya an account on the ftp so we can check this out. We're talking DVB SPTS, PAL world SD AND HD.
Please just PM me with the account details. I have to ask for everything? :)

Thank you.

[TEB]

1.3.0

OK. PM'ed u with the details.. Heres some observations:

*I tried to get the detect pids on a 18gb 4:2:0 50mbit Iframe only PS and it crashed after a while. (just to check!! i know there arent any pids in a PS!)
*Is it possible to add speed making the code more Hyperthreading aware? And make it multithreaded/tasked so a full SMP system could be used better?

Keep up the good work!!

teb

[malkion]

Thank you Donald. I am most appreciative of your efforts.

Best wishes.

[Guest]

You're welcome, malkion. I'm happy that you find my work useful. Speak up if you have any ideas to improve things!

[Guest]

@TEB

Let's get the basic stuff going first and then discuss the rest.

Regarding the streams provided to me that cause crashes...

1. You probably know this but you have to rename the extensions to ones that DGIndex interprets as transport streams. Currently it does not include 'm2t'. I am adding that, however. Any more you care to suggest?

2. If you have the PIDs set right before opening the file (and correcting issue 1 above), the files I tried open fine. I didn't try them all yet but will do so. I found a bug in DGIndex that causes the crash when the PIDs are wrong. I fixed it and it is fine. I'll make a point release of DGMPGDec 1.0.1 later today with that fix.

3. The files I opened had MPEG audio, not AC3. I haven't yet added support for that, but it won't crash. How do you want me to handle your audio. Do you want a demuxed transport stream, a PES, or an ES?

[seewen]

It seems that there's sometimes a problem in DVD2AVI 1.1.0/1.3.0 and/or in MPEG2Dec3dg.dll, when used with CCE or TMPGEnc.

I wanted to backup "187" (an old movie with Samuel l. Jackson). It's a french edition ("TF1" edition).

Like usually, I created the *.d2v, then the *.avs, then I launched CCE.
The movie is 1h54 long. The first 1h10 minutes are perfect, but the last 30-40 minutes are "blocky". Every part of the movie which is moving (a man, a hand, a car, mouth, etc.. ) is only composed of big blocks.

I thought the problem comes from the bitrate (4350). So I changed to 1/2 DVD resolution (352x576), and launched CBR encodes.

The result was identical. And it's strange beccause when I parse the *.avs, this problem doesn't appears.

I try CCE SP 2.67.0.27, 2.50, TMPGEnc 2.5, and it was always the same. I tried with Decodefix 1.1.0 and 1.3.0, and it was always the same.
I tried to add "TomsMoComp" to the *.avs, even if the movie wasn't interlaced, but big bloks were always here.


Then I tried MPEGDecoder.dll (from Nic. I let it create the *.d2v too), and it was perfect with TMPGenc CBR (didn't try with CCE yet, but I assume it will be perfect too).

I tried other movie (Shawshank Redemption. Same "TF1" edition) with Decodefix 1.3.0 and CCE 2.67, and many more movies with 1.1.0, but this problem only occurs with "187".

It's the first time since I use CCE/TMPGEnc (for CVD/DVD) that I see such a thing.

I don't know if you can do something with what I said.. ;)

bye

[Lord_KiRon]

Have sadly to report that this version (1.3.0) runned from robot4rip does not detect "FILM" mode even when during the work percentage was 98-99% it still creats .d2v file with 29.97fps and Field_Operation=0 , also .d2v file name does not contain word "FILM" any more :(
Rechecked few times by running 1.1.0 - averything OK there ...

[ExarKun634]

I have looked through the change logs (change_*.txt) and the readme files (Readme.html). I have searched for "raw encoded frames" in this forum. I have looked through the development thread for DVD2AVIdg 1.20. I found documentation on the difference between the Field Operations "Raw Encoded Frames" and "Forced FILM", but I found no documentation on the difference between setting Field Operation to "None" and setting Field Operation to "Raw Encoded Frames". Could someone enlighten me?

[TEB]

Neuron2

Hi. Thx for testing my streams. I will not claim to be any kind of expert in anything of this but ill try to explain in my kinda way :)

Some of these streams are dumped on a linux box from a given Satellite's transponders in SPTS format (i split out the given services i need (SPTS) from the MPTS in a hardware box before hitting the linux box)
In my experience Northern European DVB SPTS Services typically contain the following pids; Video Pid, Audio Pic1,2,3 and so on, AC3 pid 1,2,3,
DVB subtitling (similar to dvd-subs),DVB Teletext, and a ton of controllstreams for various satellite boxes (i filter these out).
I would luve to get DVD2AVI to eat all of these formats by reading the PAT/PMT get the PID's automatically like Videolan does, parse the stream and give the option on what u can demux and make a .d2v file outa.
.m2t, .ts, .spts, .mpg, are typical endings on transportstreams.
The timestamps can either be read from the PTS, gop-timestamp or skipped and base it off the complete file during the .d2v generation. Since the 2 former often are erronous, just used for sync.

The other streams there are HD-mpeg2 found on the net (and this forum).

keep up the good work!

teb

[loni_blues]

@Lord_KiRon

I´ve found the same thing myself. It is due to a change in d2v structure in new Dvd2avidg versions. I believe we now have to use F5, check the film percentage and apply force film manually, since -as len0x pointed out- R4r is not going to be developed anymore.

Kind regards,
loni_blues

[Guest]

Originally posted by seewen
It seems that there's sometimes a problem in DVD2AVI 1.1.0/1.3.0 and/or in MPEG2Dec3dg.dll, when used with CCE or TMPGEnc.
[...]
I don't know if you can do something with what I said.. I can do something if you can make a small VOB fragment available that I can use to recreate the problem. Please do so.

[Guest]

Originally posted by Lord_KiRon
Have sadly to report that this version (1.3.0) runned from robot4rip does not detect "FILM" mode even when during the work percentage was 98-99% it still creats .d2v file with 29.97fps and Field_Operation=0 , also .d2v file name does not contain word "FILM" any more :(
Rechecked few times by running 1.1.0 - averything OK there ... Yes, the D2V file format changed, as loni_blues pointed out.

[Guest]

Originally posted by ExarKun634
I have looked through the change logs (change_*.txt) and the readme files (Readme.html). I have searched for "raw encoded frames" in this forum. I have looked through the development thread for DVD2AVIdg 1.20. I found documentation on the difference between the Field Operations "Raw Encoded Frames" and "Forced FILM", but I found no documentation on the difference between setting Field Operation to "None" and setting Field Operation to "Raw Encoded Frames". Could someone enlighten me? I can help you with that one. :)

When you do FF, DGIndex will add or insert frames to keep the film rate. Let's suppose you have a hybrid VOB that is mostly 3:2 pulldown but with some sections of 30fps (no RFFs there). DGIndex has to remove frames from the 30fps sections so that when the frame rate is set to 24fps, those sections don't play too slow. Other situations may require frames to be added. It all results from DGIndex wanting to output a 24fps result.

When you do raw, DGIndex does everything the same but never adds or removes frames. This lets you see exactly what the stream has in terms of encoded pictures, hence the term raw.

[Guest]

Originally posted by TEB
Some of these streams are dumped on a linux box from a given Satellite's transponders in SPTS format (i split out the given services i need (SPTS) from the MPTS in a hardware box before hitting the linux box)
In my experience Northern European DVB SPTS Services typically contain the following pids; Video Pid, Audio Pic1,2,3 and so on, AC3 pid 1,2,3,
DVB subtitling (similar to dvd-subs),DVB Teletext, and a ton of controllstreams for various satellite boxes (i filter these out).
I would luve to get DVD2AVI to eat all of these formats by reading the PAT/PMT get the PID's automatically like Videolan does, parse the stream and give the option on what u can demux and make a .d2v file outa.
.m2t, .ts, .spts, .mpg, are typical endings on transportstreams.
The timestamps can either be read from the PTS, gop-timestamp or skipped and base it off the complete file during the .d2v generation. Since the 2 former often are erronous, just used for sync.
Thank you for the information. But you didn't answer my question about the audio. How do you want to receive the MPEG audio? Demuxed transport, PES, or ES? What is your process for handling audio?

Within minutes I will release DGMPGDec 1.0.1 that fixes your crashes problem and retrieves the video from all your streams. But I need to know how to handle the audio.

EDIT: DGMPGDec 1.0.1 released at my web site. It fixes the crashing with your streams. Please advise about the audio.

[Wilbert]

Thank you for the information. But you didn't answer my question about the audio. How do you want to receive the MPEG audio? Demuxed transport, PES, or ES? What is your process for handling audio?
I guess this question holds also for other audio stream AC3 and later hopefully AAC :) I'm not sure what the difference is between PES and ES (googled around, but it's too technical for me). I would like to have the audio as a dts, ac3, aac or mpa file (and not in an empty transport file).

[Guest]

DGMPGDec already gets the AC3 audio out of transport streams into an ES. So I suppose we should do the same for the other audio types. You've pointed me to the code for AAC. Straight MPEG audio should not be any problem either.

[Guest]

Originally posted by Wilbert
I'm not sure what the difference is between PES and ES (googled around, but it's too technical for me). ES is just the plain simple compressed audio data.

If you take the ES and break it into little chunks, called packets, and you add a header to each packet, and you put useful stuff in those headers, like presentation time stamps (PTS), then you have a packetized elementary stream (PES).

If you take the PES and break *that* up into 188-byte packets, and add useful headers to those packets, including things like packet IDs (PIDs), then you have a transport stream.

This is called layering.

[TEB]

Hi. Handle mpeg audio like u do in normal program streams. Or maybe i misunderstood ure question?
I just want it to be robust and handle all kinds of transport streams and be able to read em with one chosen audio format in avisynth! :)

best regards

teb

[Guest]

Originally posted by TEB
Handle mpeg audio like u do in normal program streams. ES it is then! Thank you for your feedback.

[Fox Mulder]

Originally posted by ExarKun634
I have looked through the change logs (change_*.txt) and the readme files (Readme.html). I have searched for "raw encoded frames" in this forum. I have looked through the development thread for DVD2AVIdg 1.20. I found documentation on the difference between the Field Operations "Raw Encoded Frames" and "Forced FILM", but I found no documentation on the difference between setting Field Operation to "None" and setting Field Operation to "Raw Encoded Frames". Could someone enlighten me?
Originally posted by neuron2
I can help you with that one. :)

When you do FF, DGIndex will add or insert frames to keep the film rate. Let's suppose you have a hybrid VOB that is mostly 3:2 pulldown but with some sections of 30fps (no RFFs there). DGIndex has to remove frames from the 30fps sections so that when the frame rate is set to 24fps, those sections don't play too slow. Other situations may require frames to be added. It all results from DGIndex wanting to output a 24fps result.

When you do raw, DGIndex does everything the same but never adds or removes frames. This lets you see exactly what the stream has in terms of encoded pictures, hence the term raw.

It seems to me that ExarKun634 wanted to know if the output of Field operation set to NONE is the same as RAW ENCODED FRAMES or not; and not the difference between Forced FILM and Raw Encoded Frames.

BTW, thank you for your work in DGMPGDec, it rocks. :D

[Guest]

Oh, OK, I misread it.

The difference is that with NONE, RFF flags are honored, i.e., pulldown is performed, while with RAW, RFF flags are ignored, i.e., pulldown is not performed. So as I said, RAW is the same as FF but with no frame decimation/duplication to keep the film rate.

[Cyberia]

Don, I wondered about that myself. We need to VERY clearly spell out the differences between the Field Operations in the manual and in the FAQ. Would you like the first shot at that paragraph? or do you want me to try and you correct it?

[Guest]

Go ahead, Cyberia! The user perspective is often the preferable one.

[jarthel]

is the d2v output still compatible with 1.76? cause I can't open the output d2v in gknot or vdubmod (through avs script). Thanks

jayel

[Guest]

The D2V file format is now much different from that of 1.76.

But that doesn't mean you can't use the new versions in VirtualDubMod and GKnot. You just have to ensure that you use both DGIndex.exe and DGDecode.dll, and that you don't accidently pick up earlier versions. Usually the problem people run into is they inadvertently use an earlier version of mpeg2dec3. In your AVS script, remove the loadplugin call for mpeg2dec3.dll and replace it with a load of DGDecode.dll.

[jarthel]

thanks neuron2. :)

jayel

edit:

Just an additional question. What version of vdubmod and gknot supports the latest d2v format? I have vdubmod 1.5.10.1 build 2424 and gknot 0.28.8 both doesn't seem to work. thanks again

[LigH]

Furthermore, it seems that you have to make a copy of DGDecode.dll, rename it to "mpeg2dec3.dll", and put this renamed DLL into the GordianKnot folder, if you want to make GordianKnot open a d2v file created with DGIndex.exe - it seems that the filename "mpeg2dec3.dll" is hardcoded into GordianKnot, it does not use the filename written into the edit line on the "Options" tab.

At least, I was not able to make GordianKnot 0.28.8 use the DGDecode.dll for loading d2v files into GordianKnot just by writing "DGDecode.dll" into that edit line (and placing the file into the folder where the "GordianKnot.exe" is, of course). GordianKnot would then write

LoadPlugin("DGDecode.dll")

into the created AviSynth skript; but for its own preview window, it seems to always use an "mpeg2dec3.dll" if the checkbox "use alternative DLL" is checked.
__

VirtualDubMod is not interested at all in any version of DGEncode.dll, if you open an AviSynth skript. If the d2v project file was made with DGIndex.exe, use

LoadPlugin("DGDecode.dll")
MPEG2Source("*.d2v")

then AviSynth skripts work well.

[jarthel]

thanks for the clarification. :)

[Guest]

Yes, thank you, LigH.

[freelock7]

Sometimes, DVD2AVIv1.76 create a bad .d2v (the beginning of the movie with 2 introduction tracks).
Sometimes, delay is not correct even corrected by a DELAY CORRECTOR.
I tried the decoder fix with AVISYNTH2.5 but the problem remain...
Ex: star wars 1&2 with DVD2AVIdg/MPEG2DECdg Decoding Fix create a bad ac3 file with many desynchro.
Is there an update to fix these errors?

[Guest]

Version 1.3.0 is about 12 revisions old and this thread is dead now. Please make a new thread describing your problem and we will try to help you.