DGMPGDec 1.1.0 version definition

[Guest]

Following are the features I am targeting for version 1.1.0. Here is your chance to influence the version planning.

* Upsampling based on progressive/interlaced picture indication
* Log file generation
* Demux MPEG audio and AAC audio from transport streams
* PVA transport support
* CLI conrol of process priority

[DDogg]

Upsampling based on progressive/interlaced picture indication Flesh that out a bit more if you have time? I want to make sure I actually understand what you are saying.

[vispgraedde]

I believe that refers to the U and V planes of the video.

Upsampling that to YUY2 or RGB is different for progressive and interlaced video.

http://arbor.ee.ntu.edu.tw/~jackei/dvd2avi/interlace/
shows an example of what happens if you upsample interlaced colour data as progressive.

[Guest]

Exactly correct, vispgraedde, thank you.

I'll add that in MPEG syntax each frame carries a progressive/interlaced indication, and when a frame is upsampled it should be done appropriate to this indication. Failure to do so is the cause of the infamous "chroma upsampling bug":

http://www.hometheaterhifi.com/volume_8_2/dvd-benchmark-special-report-chroma-bug-4-2001.html

DGMPGDec will provide upsampling support that avoids this error. Currently, you can use the MPEG2DEC3dg conversion filters but you have to specify interlaced=true/false for the entire clip. There are two problems with that: 1) the poor user doesn't know what to set, and 2) it is not per-frame. DGMPGDec will allow three possible settings: 'interlaced true/false' as before, and 'follow progressive indication flag'. BTW, VFAPIdg already follows the progressive indicator.

Of course, this is all relevant only if you want to convert to YUY2 or RGB for delivery via Avisynth. If you output YV12 and convert later, it is up to the subsequent software to do things right. Without the progressive indication that is problematic. So the best advice is to output YV12 only if you plan to encode that directly without conversion. Even then there can be problems, but I've already said enough about it.

[vispgraedde]

Is there, or will there be, any hinting from DGDecode to decomb to make ivtc of hybrid encodes easier?
I'm now thinking of hybrid as partly encoded as progressive, and partly encoded hard telecine.

[Guest]

The contemplated log file will contain all the information that Decomb would need (in fact, the current D2V file contains it). All that would be required would be a revision of Decomb capable of reading and using it. Certainly it is something that I am thinking about. :D

[ssjkakaroto]

neuron2 does this mean that we'll be able to perfectly ivtc a hybrid source automatically?

[Guest]

Theoretically, if the flags are valid and there is no hard telecining, it is possible to perfectly recreate the progressive pictures. BUT AND IT IS A BIG BUT -- you still can't rationally set a single frame rate for a hybrid. Do you choose 24 or 30fps? Maybe with newer container formats we can have variable frame rates, but with the existing tools and Avisynth, no.

[ssjkakaroto]

but this would already be a superb feature! specially for anime this will make things a lot more easier!

[Wilbert]

I digged up the sources for aac demuxing for you :)

http://pbx.mine.nu/dvd2avi/

[Guest]

Originally posted by ssjkakaroto
but this would already be a superb feature! specially for anime this will make things a lot more easier! You can already do it with the "raw frames" field operation mode. But you're still stuck with having to set a frame rate.

[Guest]

Originally posted by Wilbert
I digged up the sources for aac demuxing for you :)

http://pbx.mine.nu/dvd2avi/ Oh, cool. Thank you.

This is the best part:

"This modification will only read 188-byte TS packets. And since not a single person contacted me since I put this up, I assume either nobody cares about anything except 188 byte, or they are too f*cking stupid to figure out how to email me."

[Guest]

Oh, hey Wilbert, do you know of any code for demuxing basic MPEG audio out of the transport streams? Why re-invent the wheel if it is already done? The NIC/trbarry code I started from does just AC3.

[Wilbert]

lol I also had to laugh when I saw that remark :)

I'm not sure. But, I think TSDemux is capable of doing this. TSDemux and its source can be found at

http://arbor.ee.ntu.edu.tw/~jackei/projectbsd/

[Guest]

TSDemux is a MPEG2-TS demuxer designed for BS Digital and most D-VHS players.

[INSTRUCTION]

1. LOAD MPEG2-TS stream
2. SAVE to any folder, we will see M2V and AAC/MP2
3. decode M2V file by DVD2AVI
4. decode AAC file by FAAD2 (FAAD2 *.aac *.wav) or MP2 file by TMPGEncMP2! That's the ticket. Thank you.

[len0x]

Looking at the options of GUI anf CLI I'd like to request an option to have a process priority to be set from CLI as well (as currently if started from CLI it consumes quite a lot of resources).

[Cyberia]

Originally posted by len0x
Looking at the options of GUI anf CLI I'd like to request an option to have a process priority to be set from CLI as well (as currently if started from CLI it consumes quite a lot of resources).

Added.

[Bogalvator]

How about an autocrop feature? When DGMPGDec finishes scanning through it can place the crop(...) line in the AVS template.

[Guest]

Originally posted by Bogalvator
How about an autocrop feature? When DGMPGDec finishes scanning through it can place the crop(...) line in the AVS template. I don't understand your request. There is already functional clipping in the Video menu. DGDecode reads the clip values from the D2V file and automatically invokes Crop().

[Dark-Cracker]

any chance to have the cli mode without the comma bug ?

keep up the good work.
++

[RB]

Originally posted by Bogalvator
How about an autocrop feature? When DGMPGDec finishes scanning through it can place the crop(...) line in the AVS template.
I think you are looking for this AVISynth plugin (http://www.videofringe.com/autocrop/). Doing something like this during saving the D2V sounds appealing but would not be practicable IMHO because it means every picture would have to be fully decoded (which currently isn't the case), this would slow things down a lot.

[Guest]

Originally posted by Dark-Cracker
any chance to have the cli mode without the comma bug ? It's not a bug, it's a feature.

If you can propose a workable alternate syntax, I'll change it. But please don't rehash the previous discussion on this!

[Guest]

Oh, all right, then. I'll change it from a comma to a space. Does anybody object?

[zettai]

I object, not because I use this feature but because in all the filenames I've ever made myself I've never used a comma... whereas I use spaces all the time.

[RB]

Ah, no space for the delimiter please! I'm perfectly happy with the comma. Although, best would be something that really isn't valid ín a file name. I'd vote for the "|" character in this case.

[Dark-Cracker]

i vote also for the "|" character.

[Guest]

You see my problem now, Dark-Cracker?

I forgot that space is valid in a filename, to use it I'd have to require quotes and then parse them. etc., etc.

And | gets intercepted by the DOS shell.

So, unless you come up with a good solution it's gonna stay as a comma. :(

[vispgraedde]

I vote for ascii decimal 13 :devil:

[Cyberia]

Guys... You can rename a file and take out a comma.

Without completely rebuilding the filename parsing, Don CAN'T fix this. Period. No other characters can be used. The ONLY choices are:

1) rebuild the parsing to require quoted filenames

or

2) require a text filelist


Honestly I'd vote for option 2. The filelist can be generated in one single DOS command. We could include it in a bat file.

[Dark-Cracker]

@don

lol yes now i see your problem.
i think the CLI with comma bug is not a priority, like explain cyberia there is alway the possibility to add a checkerror on the comma in the filename.
i have just post this msg not to rehash the thread on the CLI bug but just to be sure that u write this bug in your todo list even if it's at the bottom of the list :)

in any case i think the other functions are more usefull for the users and i know u will made a good job :)

++

[Guest]

Point taken, Dark-Cracker. It will be added to the list as it is clearly an issue.

[Bogalvator]

Originally posted by RB
I think you are looking for this AVISynth plugin (http://www.videofringe.com/autocrop/). Doing something like this during saving the D2V sounds appealing but would not be practicable IMHO because it means every picture would have to be fully decoded (which currently isn't the case), this would slow things down a lot.

Yes, this is what I meant. I thought that maybe it would be a luxury that would be relatively simple to include but if as you say each frame needs to be decoded first then it's a luxury one can do without.

[SILICON]

I'd like this changes.

Remove funtions and code:
- YV12toYUY2()
- YV12toYUY2(interlaced=false)
- YV12toRGB24()
- YV12toRGB24(interlaced=false)
- FlipVertical() (I don´t sure if this is a DGMPEGDEC's funtion)

Mark FD said:
"The built-in colorspace conversions may cause problems. Please use the AviSynth functions where possible."

Remove iDCT = 4 (64 bit IEEE-1180 Reference).
This IDCT don`t give more speed or more accurate and have a bug.
http://arbor.ee.ntu.edu.tw/~jackei/dvd2avi/idctref/

This IDCT don`t give more speed or more accurate. You can remove it and don´t fix the bug.

Add the new iDCT-Algorithms to DGIndex comandline
-iDCT-Algorithm -IA
1 : 32 bit MMX
2 : 32 bit SSEMMX
3 : 64 bit FPU
4 : 64 bit IEEE-1180 Reference (REMOVED???)
5 : 32 bit SSE2
6 : Skal's SSEMMX iDCT
7 : SimpleiDCT

Add the to DGIndex comandline one option for LUMAFILTER
If you´ll remove this function of DGDecoder, the users need the new swich.

Please: A log of operation

[Cyberia]

The reference iDCT should stay but should be fixed, if indeed there is a bug at all.

We will always need this as a 'control' to test against. Granted if it's buggy that's very bad, but then someone should fix it...stay tuned.

[vispgraedde]

When you add colour upsample, would it then also be possible for the user to specify its (a user is an it! ;)) own function for this? (provided the function can handle both interlaced and progressive upsampling)

Some people may want the upsampling to behave like the current YV12toYUY2 (which sucks imo) rather than ConvertToYUY2 (much better since it interpolates), and some people (like myself) may want more advanced upsampling, and I guess it would be easiest to do if there is a way to tell dgdecode which upsampler to use.

Since the more advanced upsamplers may make use of the luminance data, the upsamplers would of course need to get access to that as well.

Just some ideas from a confused mind :confused:

[Guest]

You can already use external filters by placing them after Mpeg2Source().

The problem is that we want the upsampling done on a per-frame basis using the MPEG progressive/interlaced picture indication. It is looking like I have to use real code for the conversion and not an internal filter invocation, because the latter will not allow me to operate on a per-frame basis.

Can you please elaborate on what you perceive as the shortcomings of YV12toYUY2() and why you think the Avisynth converters are better? I want a technical explanation and not a quote from the user manual. Really good would be screenshots demonstrating why you think YV12toYUY2 "sucks". My testing suggests that the differences are imperceptible.

[vispgraedde]

Originally posted by neuron2
You can already use external filters by placing them after Mpeg2Source().

The problem is that we want the upsampling done on a per-frame basis using the MPEG progressive/interlaced picture indication. It is looking like I have to use real code for the conversion and not an internal filter invocation, because the latter will not allow me to operate on a per-frame basis.


But it is for the same reason I can't use these filters after mpeg2source(). I thought maybe there could be a way to supply a user specified function for the upsampling. One that can be told if it is progressive or interlaced on a per frame basis.



Can you please elaborate on what you perceive as the shortcomings of YV12toYUY2() and why you think the Avisynth converters are better? I want a technical explanation and not a quote from the user manual. Really good would be screenshots demonstrating why you think YV12toYUY2 "sucks".


Easy to see why by comparing the U and V channel results from YV12toYUY2 and ConvertToYUY2 respectively.
With YV12toYUY2, it seems to be a nearest neighbour method, while with ConvertToYUY2 it seems to interpolate instead.
YV12toRGB24 seems to be interpolated so I don't quite understand why the conversion to YUY2 is not.

At the moment I can't provide any samples of this though. Sorry.


My testing suggests that the differences are imperceptible.


It was when I was looking at a clip that had been upsampled with YV12toYUY2 that I noticed that something looked weird. It was at places with a large amount of red surrounded by darker colours. That's when I began looking at the UV-planes to see why it was like that.

I agree that it is usually imperceptible, but in some cases you can see it easily enough.


So. If you are going to have the upsampling code in dgdecode, I suggest using different selectable methods.
Nearest neighbour like in YV12toYUY2, and some good interpolating method of your choice... Several choices would be nice, but you can leave that for the future :p

[Guest]

Just as I thought, all talk and no proof (screenshots). :)

The only way to allow external filters to act per-frame is to pass a hint hidden in the video data. The external filter would need code to read the hint and act appropriately. Is that what you are asking for?

[vispgraedde]

Originally posted by neuron2
Just as I thought, all talk and no proof (screenshots). :)

The only way to allow external filters to act per-frame is to pass a hint hidden in the video data. The external filter would need code to read the hint and act appropriately. Is that what you are asking for?

PM'd you a link to pics to show what I mean.

And a hint is pretty much what I ask for, though I thought at first that you were going to use external filters from inside dgdecode which is why I asked for user definable in that case.
Since you're not going to use external, hinting is what I seek.

[Guest]

Thank you for the PM. I will study it carefully.

If I am going to hint, then I will probably just go ahead and do that and modify YV12toXXX() to use the hints. Then they will be there as models for anyone that wishes to implement/modify any other conversion filters.

To be quite honest, I'm with Avery Lee on this stuff: you can be pretty cavalier with the UV channels and nobody's ever going to notice. Unless they separate the planes, zoom in, and increase contrast by 50 times. :)

But, of course, getting the interlaced/progressive upsampling correct *is* important to avoid obvious artifacts.

[Guest]

OK, here's the plan for the upsampling fix...

Step 1.
Rip out existing conversion filters and replace them with copies of the existing Avisynth core conversion filters (stripped down to assume YV12 input and renamed).

Step 2.
Revise DGDecode.dll to hint the interlaced/progressive flag into the video data.

Step 3.
Revise the new conversion filters to accept hinting of the progressive/interlaced flag. The filters will support 3 modes: auto (follow the progressive/interlaced flag), force interlaced, and force progressive.

I've already completed and tested Step 1, which is the hard part. So we're close to a good fix for the upsampling issue. You'll be able to say:

mpeg2source("file.d2v")
yv12toyuy2(us=auto)
#yv12torgb24(us=auto)
#yv12torgb32(us=auto) You'll get correct upsampling as well as the good quality of the core conversions, and you'll be able to implement your own hinted filters.

Is everybody happy? zettai? vispgraedde?

[zettai]

Looks good to me... my only concern is what will happen if someone uses them foolishly later in a filter chain.

It seems to me they should always be used directly after mpeg2source, which suggests they might be better as a named argument in that command such as:

mpeg2source("somefile.d2v",output="rgb32", us=auto, ipp=auto, cpu=4)

Of course, I dont mind which way it happens, so it's all good with me... just a suggestion.

[Guest]

Yeah, I hear ya. But I've heard at least two people suggest that they could use the hinting. So I thought it'd be a good idea to make the external filters as a model. I can deal with your point through good documentation.

[zettai]

Yeah, that's fine.

[Zep]

be able to copy paste PIDS from the auto detection windows
into the select pid fields.

or


check boxes next to the PID list so i can select which two i want
then when I close the window they are used without needing to open
select pid window.


wow does this mean I'm like super lazy? :D

[vispgraedde]

I'm happy with that plan for the upsampling fix.

[Wilbert]

mpeg2source("file.d2v")
yv12toyuy2(us=auto)
Your proposed solution is good provided you convert to YUY2 directly after mpeg2source.

Three questions (sorry if they are dumb questions, I still have problems to understand this stuff):

1) What happens if you want to do postprocessing in YV12? AFAIU this works fine if mpeg2source delivers progressive YV12. Could you add a mixed/interlace YV12 to progressive YV12 conversion?

2) Suppose you want to do a IVTC with your filters, do they always expect interlaced YV12?

3) If you open an YV12 stream in VirtualdubMod, and want to encode to DivX/XviD. Does it expect progressive or interlaced YV12? Or doesn't it matter?

[Guest]

Originally posted by Wilbert
[...]Could you add a mixed/interlace YV12 to progressive YV12 conversion? I don't understand your request at all. Can you be more be explicit?

2) Suppose you want to do a IVTC with your filters, do they always expect interlaced YV12? Telecide() doesn't expect anything. He just assembles the candidate matches with what he is given. I haven't thought it through very completely, but it seems to me that any degradation for the chroma matching that is caused by progressive sampling of interlaced frames will affect all the candidate matches equally and shouldn't change the matching result.

3) If you open an YV12 stream in VirtualdubMod, and want to encode to DivX/XviD. Does it expect progressive or interlaced YV12? Or doesn't it matter? If you use fast recompress the YV12 is passed as is. If you have full processing, then the codec (usually DivX or Xvid) is used to upsample to RGB for VirtualDub. Then it is downsampled again by the codec for output. So it depends on how the codec behaves.

[Guest]

Originally posted by Zep
wow does this mean I'm like super lazy? Yes.

If I had infinite time available, I could spend some of it on stuff like this. Sadly, I do not.

[vispgraedde]

Originally posted by neuron2
Telecide() doesn't expect anything. He just assembles the candidate matches with what he is given. I haven't thought it through very completely, but it seems to me that any degradation for the chroma matching that is caused by progressive sampling of interlaced frames will affect all the candidate matches equally and shouldn't change the matching result.


I hope I understand this right if I think that if/when you change decomb to pass through progressive frames and only process interlaced frames (as talked about in the beginning of this thread), it will eliminate any possible degradation like this.

(I'll be away for a few days, so if I need to explain further what I talk about, I'll do that when I get back)