MPEG2Dec3 v0.9 for Avisynth 2.5

[Marc FD]

a lot of work for this pre release of the stable version.
and i've spend a lot of time on documentation, i'll try to clean the code more in the next days to be able to package it. (in v1.0)

dl it on my site (link below, in my signature)

i hope you'll enjoy the old&new features.
i'm sorry, defaults parameters aren't implemented but i'm thinking of it.
it's piece of cake now, because the harder stuff on the Avisynth API (~ 1000 lines) is done ^^.
BTW, i think i may add a fixed quant YV12 postprocessor (would act like if source was at a fixed quant. less efficent, but can still be usefull)

i dunno if i should add fastest colorspace convertions. say me ^^

going to sleep now XD

Regards,
MarcFD

PS @Sh0dan : do you know if dividee is going to make a YV12 version of compare ? i've a new quality mesure (coded for YV12) that Mfa gived me a long time ago, i could add it and do the YV12 convertion if you want.

[ACClarke]

Thank's for your great job.
I will test mpeg2dec3 soon.
but before I have to sleep, like you :D

SeparateFieldsYV12 is really interesting.
I often use it as you know.

merci....

[MaTTeR]

Thanks so much for the new build Marc. Documentation is much better:D

The custom CPU settings looks great, cant wait to test it!

Edit- For some reason lumoff=-1 is being reported as an invalid argument sometimes. I got it to work twice but it's not consistent:confused: I'm using sh0dan's 07112002 2.5 build and the script is-
LoadPlugin("C:\Program Files\AviSynth2\plugins\MPEG2DEC3_YV12.dll")
LoadPlugin("C:\Program Files\AviSynth2\plugins\undot.dll")
mpeg2source("D:\test\breed.d2v",iDCT=2,lumoff=-1,cpu2="xoooxo")
crop(4,4,712,472)
UnDot()
LanczosResize(640,352)
Limiter()

[HarryM]

@MarcFD:

Thanks. Much thanks for Avisynth2.5's adapted Mpeg2Dec! You are fanatical devoted :)

[Marc FD]

> Edit- For some reason lumoff=-1 is being reported as an invalid argument
> sometimes. I got it to work twice but it's not consistent I'm using
> sh0dan's 07112002 2.5 build and the script is-

please read the doc. due to the YV12 colorspace compatibility, LumaFilter and colorspace convertions are indepedant filters. this way, it's much more user friendly, code get cleaner, and you can use it WITHOUT MPEG2Dec3 decoding. you just need YV12 colospace.

LumaFilter() will do lumoff=-2 lumagain=1 by default (i thinked at iago and users who use is combo)

dunno if a sharpen2 YV12 could help to temporary replace Unfilter in iago's combo ? maybe it's worth to check ?

>SeparateFieldsYV12 is really interesting.
>I often use it as you know.

i did it thinking of you ^^

[iago]

@Marc

Thanks a lot for the new version! I'd been waiting for it for days to start some real YV12 tests/encodes. Also, the documentation looks really sharp too! ;)

regards,
iago

[Marc FD]

i've finished 0.91 (with bugfixes & default settings), but can't release it ^^
sorry, i need to go right now. i would release later ^_^

[sh0dan]

Originally posted by Marc FD
PS @Sh0dan : do you know if dividee is going to make a YV12 version of compare ? i've a new quality mesure (coded for YV12) that Mfa gived me a long time ago, i could add it and do the YV12 convertion if you want.
Dividee is very tied up at the moment, as far as I understand, but I'll see if I can do an YV12 version soon. This and Blankclip is probably top of the list right now. Layer would be nice, but it seems like a big task (haven't looked at the code at all).

[ookzDVD]

@Marc FD,

Thank you for your latest MPEGDEC3 v0.9,
with latest AVIsynth 2.5 (build 7/11) & Koepi's XviD build 9/11,
1-pass quantizer @ 2, without b-frame,
I got ~30fps :) on my Athlon XP 1700+, That's fast! ;)

[Marc FD]

Okay, v0.91 is out.

Changes :
- cleaned a bit more the source
- added MPEG2Dec3.def default settings loading (like don's filters)
- bff mode in SeparateFieldsYV12

(and bugfixes)

dl on my page, as usual.


>Dividee is very tied up at the moment, as far as I understand, but I'll
>see if I can do an YV12 version soon.

if you really want to do it, okay. but i know very well the source, and i've improvements to do, so i though i could handle it. just say me.

>This and Blankclip is probably top of the list right now.

Blankclip should be easy to do in YV12 ^^

>Layer would be nice, but it seems like a big task (haven't looked at
>the code at all).

layers would be so easy to convert to YV12, no ? i can do it if there's any request for it ^^

BTW, i'm gonna make a YV12 version of subtract (with MMX code for all colorspaces) because i often use them and i don't want to use Avisynth 2.06 anymore. i can e-mail you the code when it's finished, ok ?

[sh0dan]

Feel free to do all this - it would be nice if I could focus on brushing up other stuff now.

The best way would really be for you to have CVS access. Version control would be 10x easier than mailing changes back and forth. You should contact Richard about access.

[Marc FD]

okay i'm gonna mail him ^^

[wotef]

i'd love to have layer, thanks marc

[Bulletproof]

Hey I tried this decoder today and in the readme it says to use:

MPEG2Dec3("filename.d2v")

But it says it's an invalid command. Using MPEG2Source still works though, but is that the proper way to do it?

[N_F]

Originally posted by Bulletproof
Hey I tried this decoder today and in the readme it says to use:

MPEG2Dec3("filename.d2v")

But it says it's an invalid command. Using MPEG2Source still works though, but is that the proper way to do it?

I noticed the same thing last night when I tried it. I got a speed increase (though only 10 %) with it, so I guess YV12 was at least used in some way.

[ACClarke]

it's just a mistake
Marc knows it, he 'll probably correct it in the next release.

mpeg2source is the right keyword

[dattrax]

stupid question, but I went to your web site and there is only beta 0.6 for download and not 0.9 as advertised.

am I going daft?

Jim

[cult]

@dattrax
its there.The build is 0.91 and downloaded a couple of days ago.try to clear your browser cache

[dattrax]

Just done that, and it's still reporting version beta 6 as the latest. The odd thing is that I get the same page at work as I do at home.

Jim

[N_F]

Originally posted by dattrax
Just done that, and it's still reporting version beta 6 as the latest. The odd thing is that I get the same page at work as I do at home.

Jim

You're scrolling down too far. Check under avisynth 2.5 filter, not 2.06.

[Bulletproof]

I think I found a bug with the latest release. I just tried it on and alot of frames come out green, some frames just show chroma, some frames show luma, some show luma and chroma mixed from different frames.

It is not my setup, I have tested 3 other videos which do decode fine with MPEG2DEC3. The video decodes fine under DVD2AVI, and will show up fine in TMPGENC. I'm not using any special options.

I also noticed that the MPEG2DEC3 Beta 6 for Avisynth 2.06 does not seem to work for the YV12 beta (Maybe it wasn't supposed to), but I wasn't using that version previously anyway, I just got it to see if it might fix this problem I had. I was using 0.91.

The video seems like it may be Hybrid material, and it's quite short too, 11 minutes. I'm not sure if that plays any role though.

[MaTTeR]

Marc,

Since your documentation still says your taking feature request...

I've been thinking about how worth while multi-threading MPEG2DEC3 would be:) I'm not sure how you process each plane but in my thinking multi-threading would help speed up functions such as "Pre-Processing modes". I'd appreciate your thoughts if you get a moment.

[Marc FD]

@Bulletproof

sorry i don't see exactly. can you describe better ?
did you tried with a regular MPEG2Dec2 ??

> Since your documentation still says your taking feature request...

of course ^^

> I've been thinking about how worth while multi-threading MPEG2DEC3 would
> be I'm not sure how you process each plane but in my thinking
> multi-threading would help speed up functions such as "Pre-Processing
> modes". I'd appreciate your thoughts if you get a moment.

Post Processing is done on a frame basis, and _after_ decompression.
decoding is very sequential (especially MPEG decoding) so multithreading is useless.
so you've got a point : you can do PP absolutely when we want, even in non-sequential order !

so if you use Post Processing, multi-threading will help. but you would hardly be able to decode MPEG-2 faster than 80-90 fps with a 2 Gz CPU (and with 2x2 Ghz)

i think the easier way to make muti-threading is to decode with a CPU, to filter with CPU(s) (max 1 CPU/filter) and to encode with one CPU.
it's the easiest and fastest way of doing it, and you can use YV12,YUY2 or even RGB32 is you want.

i think VDub is doing something like that.

the problem is, i don't have a SMP computer, so i can only speak in theory.

@All

0.92 is on the way, with a x-mas surprise.
personnaly, i really like it. i think i will never transcode old encodes without it now.

Yeah, you've all guessed : it's decoder independent Post Processing ^^

i thought without the decoder info, it would do the job bad. in fact, it's of course less accurate, but really amazing if you have some crappy, old and blocky sources.

[lighty]

Originally posted by Bulletproof
I think I found a bug with the latest release. I just tried it on and a lot of frames come out green, some frames just show chroma, some frames show luma, some show luma and chroma mixed from different frames.

I noticed same problems when using mpeg2dec3 v0.91 YV12. It only happens in preview mode when I use slider in VirtualDub to browse frame by frame through video. It happens with all the movies, all formats and regardless of AVS script used. The only common things is using mpeg2dec3 and the funny thing is that encodes comes out fine it only happens when browsing through video frame by frame.

I attached images than more than enough explains encountered problems.

Soft used: Avisynth YV12 alpha 11/11, XviD 16/11, mpeg2dec3 0.91 YV12, VirtualDubMod 1.4.11.

[Bulletproof]

I will make screenshots for you later on, I couldn't try with the old Mpeg2dec, it says something like "Unexceptional error". I think it's because I use YV12 AVIsynth now. But using DVD2AVi decoder does work fine.

[cult]

@lighty
I have noticed it too.But no problems at all with the encoded movies.

[Marc FD]

okay guys, i'll take a look. i now i saw it when i rewind.

@all

MPEG2Dec3 v0.92 is out. i think it's almost final now, but sh0dan asked not to release final plugins before a stable Avisynth 2.5, so it's likely the last version before 1.00

i hope you'll enjoy my job on this plugin ^_^

[MaTTeR]

Marc FD,

Thx for the response and the new build.

In regards to multi-threading I was thinking along the same lines as you. Decoding wouldn't really gain anything but PP most likely would. What if MPEG2DEC3 detected a 2nd CPU and then fed more than 1 frame to the encoder in a non-sequential order? Or possibly even in a sequential order? I guess I would think of it as almost like a Read Ahead cache:D Two or more frames processing at once sounds alot better than one to me.

I use PP in almost every movie(except SuperBit) these days since you gave us the wonderful option of controling the thresholds with moderate_h and moderate_v. Now that we have the custom CPU settings I might even be using it alot more. Thx again for the great effort.

BTW- Your new 0.92 build contains the TVscale parameter. Does this make the external Limiter() or LegalClip() filters obsolete now?

[Marc FD]

> What if MPEG2DEC3 detected a 2nd CPU and then fed more than 1 frame to
> the encoder in a non-sequential order? Or possibly even in a sequential
> order? I guess I would think of it as almost like a Read Ahead cache Two
> or more frames processing at once sounds alot better than one to me.

it's possible, but i've never coded in SMP, and i can't debug with my system, si i would't do it. sorry.

>BTW- Your new 0.92 build contains the TVscale parameter.

from 0.9x, there's DVD2AVI YV12->RGB24 convertion can be used a a filter. TVscale/PCscale is a YV12->RGB24 feature.

>Does this make
>the external Limiter() or LegalClip() filters obsolete now?

i don't know what these filters are for. can you explain please ?

[dartbillie]

To MARC FC
I have a problem with MPEG2Dec3 for avisynth with yv12 support.
The decoder gets stuck and wobbles back and forth on some frames, after some time of decoding.
My source material is a captured mpeg2(dvb)(interlaced pal) file.
I noticed the same problem with MPEG2Dec3 beta 5 when i set the parameter Fast=True , decoding with Fast=False gave no problem.
I guess the problem are my files, but I just thought i`ll give you some feedback.
If you are interested i`ll send you a file.
Hope to here from you.
By the way ,thanks for your great programs.

[Marc FD]

strange, i'll see if i come on this kind of problem.

>If you are interested i`ll send you a file.

sorry my bandwith is too limited

[Marc FD]

BTW, for people interessed by MPEG2Dec's3 interlace/progressive detection, you can check what he thinks of each frame with DebugView. (pf==0 mean interlaced i think)

[[Toff]]

Originally posted by Bulletproof
I think I found a bug with the latest release. I just tried it on and alot of frames come out green, some frames just show chroma, some frames show luma, some show luma and chroma mixed from different frames.


I can see the same problem here (v0.92), but only when i go backward in vdub.

[MaTTeR]

Originally posted by Marc FD
you can check what he thinks of each frame with DebugView. (pf==0 mean interlaced i think) Confirmed, testing some interlaced material now using iPP=true.

[MaTTeR]

I just found a 4:3 source that MPEG2DEC3 0.91 and 0.92 totally chokes on as well. My problem looks very similar to lighty and BulletProof's. I'm attaching a very small JPG which displays the color problem. FWIW, Beta 6 does not give me this problem at all but I can reproduce the problem time after time with v0.9x builds and this particular source. I also tried the last 3 AVS 2.5 builds but that didn't affect it at all.

Let me know if you need any other information.

[Marc FD]

there is a bug when you rewind. it happends with MPEG2Dec3 (in YV12 mode, versions 0.9x)
when you rewind, the UV planes are not in synch with the Y planes.

so it's a MPEG2Dec3 bug ??
NO !! the bug is not in MPEG2Dec3 ^^ !!

for exemple, i've done this little experiment (among others) :

MPEG2Dec3 say to Avisynth it is outputing YUY2 data.
because a YUY2 frame is bigger than a YV12 frame, i can fill the frame with the YV12 buffers (1/2 Y, 1/8 U, 1/8 V)

when i decode the result, it's assuming YUY2, so i can see the three separated channels. (it's a bit ugly, it's only a hack)

and i can rewind a much as i want, i works very good. the 3 channels are in synch.

other experiments shows that MPEG2Dec3 works very good.

the very strange thing is, i can't reproduce the bug with avi files.

i assumed there could be problem in Avisynth 2.5 regarding YV12 buffering, but why avi files works ??

i wish i could help. sorry.

[MaTTeR]

Marc,

Thx for the comment. My problem happens without rewinding though. It's actually encoding like this for a period in the movie then it switches back to normal encoding and everything looks fine. It seems like maybe a certain combination of colors or scenes causes the problem but I can't be sure. Just to clarify though, I'm not seeking through my AVI file at all; just normal playback in YV12. Maybe sh0dan will see these posts and comment.

[trbarry]

I don't know if any other YV12 filters do it yet but TomsMoComp effecively has to do a rewind to look at the previous frame. Hopefully that frame will always already be in cache but I have seen these effects in encodings, not just preview mode.

- Tom

[Marc FD]

i'm 80% sure it's avisynth cache now : when you use SetMemoryMax(1), the problem occurs on a 10 frames and with SetMemoryMax(128), avisynth is caching hundreds frames (so when i rewind 100 frames and go ahead, the bug is still here).

@tom

i always cache myself in my filters, it's much faster ^^

[MaTTeR]

I'm not sure why but the pattern in some of these threads seems to point at interlaced and/or hybrid material. In my test though, I disabled TomsMoComp and all other filters and the color problem still existed. So for this interlaced source it looks like I'll have to revert back to 2.07 for the time being. So at least from my perspective it's narrowed down to MPEG2Dec3 or AVS 2.5 but it sounds like Marc thinks an AVS issue exists. Beyond any of this I'm afraid my skills aren't going to be any help.

FWIW, I've never seen artifacts rewinding or seeking ahead in preview mode using the MPEG2Dec3 0.9x builds. Very strange.

[Bulletproof]

I was working on a different movie for a while which is why I havent been able to reply recently. The other movie I was working on had no mess ups at all when rewinding or seeking in Virtualdub (YV12). I have gotten back to the movie I had problems with, and made sure everything I had was updated, the problem is still there. When seeking I see messed up frames, sometimes fully GREEN frames every few seconds in a strobing effect. Some people are saying that they don't see it when they encode the file, I do see it when I encode the file, regardless of the codec used. I tried using XviD and HuffYUV for a test, and the frame errors come out in the encoded file.

I have tried with regular MPEG2DEC and with Avisynth v2.07 (non-YV12 version) and it works then.

I tried messing with SetMemoryMax to see if it would change anything.

SetMemoryMax(128) does not make any difference to the problems.

SetMemoryMax(1) takes out the strobing green frames, but the mixed chroma/luma bad frames are still there and still end up in the encoded file. The position/frame #'s where the mixed chroma/luma bad frames occur does NOT change although the green strobing is gone.

[Marc FD]

strobing green frames = uninitialised UV planes (U=0,V=0 is green)
i saw that too.

>I have tried with regular MPEG2DEC and with Avisynth v2.07 (non-YV12
>version) and it works then.

MPEG2Dec3 YUY2 works well too. it's due to YV12 (avisynth caching bug or bad memory allocation ?)

it's alpha because we knowed we'ld need to solve this kind of problems ^^

i would like to play with avisynth, but i'm on a big project currently, and i really want to see results, so i can't code now on avisynth.

>SetMemoryMax(128) does not make any difference to the problems.

it's even worse if you have less than 378 Mb RAM ^^

maybe the bug is in MPEG2Dec3. I hope not, and my test showed me it doesn't seems to be related to MPEG2Dec3, but i can just assume things, i'm sure of nothing.

[lighty]

Ummm- I think that mpeg2dec3 is the "weak link". I concluded this because this "strobe" efect is encountered (at least in my case) exclusively when encoding "strange" mixed materials like NTSC combined progressive+interlaced sources.
When encoding PAL materials I never encountered "strobe" efects.

So if Avisynth cache is to blame then why doesn't it produce such strange effects in the PAL materials also? Why only NTSC progressive+interlaced sources are affected?:sly:

[Marc FD]

IMHO, the problem is connected to cache.
Storing 30 fps at 720x576, is different than a ripped avi in 640x360 @ 24 fps ^_^

i can make a pseudo YUY2 (no convertion, YV12 outputted) for you all to test on all sources. anyone interresed ?

[sh0dan]

Originally posted by Marc FD
strobing green frames = uninitialised UV planes (U=0,V=0 is green)
i saw that too.

>I have tried with regular MPEG2DEC and with Avisynth v2.07 (non-YV12
>version) and it works then.

MPEG2Dec3 YUY2 works well too. it's due to YV12 (avisynth caching bug or bad memory allocation ?)

Uninitialized framebuffers are not cleared in AviSynth - they are reused, without clearing the imagedata. So when you request a NewVideoFrame, you might get data from an old one.
I strongly suggest, that you look through the code again. I don't have access to it, but you should investigate, if there is separate code handling progressive frames, that doesn't work properly. Telecined movies will most likely contain progressive encoded frames, whereas PAL won't.

And considering the fact, that it seems to depend on the source material, it seems to point in the same direction.

[Marc FD]

>Uninitialized framebuffers are not cleared in AviSynth - they are >reused, without clearing the imagedata. So when you request a >NewVideoFrame, you might get data from an old one.

i know, thanks ^__^

>I strongly suggest, that you look through the code again.

i'll try to find some time.

>I don't have access to it

you have, i released MPEG2Dec3 v0.92 source code ^__^

> but you should investigate, if there is separate code handling
> progressive frames, that doesn't work properly. Telecined movies will
> most likely contain progressive encoded frames, whereas PAL won't.
> And considering the fact, that it seems to depend on the source
> material, it seems to point in the same direction.

it doesn't work in interlaced PAL when i rewind, i don't think it's related to the source material, i think it's just a buffer size chaotic side effect. but i may be wrong of course :D

i think a pseudo YUY2 builde might help to do light in dark here.

BTW, there is some SIMD code i added in a interlaced decoding copy function, and disabling it doesn't resolved anything.

[sh0dan]

Great, you put up the source!

ok, a problem seems like this:

void CMPEG2Decoder::assembleFrame(unsigned char *src[], int pf, unsigned char *dst, int pitch)

Is only called sometimes. If you put a breakpoint there, it only seems to get triggered around I-frames (it might be at the I-frame, or just before an I-frame), when seeking backwards. I don't know if this is supposed to be like this - it _seems_ like it's there you copy your data.

[Marc FD]

strange. i'll re-check that.
the weird think is, it's working in YUY2...

[Marc FD]

quick look... yes, MPEG2Dec3 (in fact YV12) seems to be the source of the problem. i'm going to kill that :devil:

[haazheel]

Hi,

I'm a new user of MPEG2DEC3, and I realized that I can't use that module and the line SelectRangeEvery(280,14) in order to run a compressibility check, whereas with the "normal" module MPEG2DEC, well it's working...

Did I do a mistake?