MPEG2Dec3 v1.10

[Sigmatador]

[i]ps
Im almost tempted to use 7 (simpleidct) myself...even though tests have shown that iDCT accuracy doesn't seem to make any real kind of a difference to the eye when mpeg-2 decoding, maybe it will make a difference to the encoder (even if very very tiny)...Hmmm, I may have to check.

i hoped you officially recommend that ^^

[killingspree]

thanks for the clarification.
i'm probably going to do some speed test soon, right now i don't have any time for it though.

thanks for your great work
steVe

[DDogg]

NIc, could you verify from reading the code that:

1> Manual entry of idct in mpeg2source line always overrides a Def file idct entry.
2> If multiple DEF files exist, such as in working dir, plugin dir and/or a dir in which a loadplugin is directed to, which DEF file will be used? [I assume the loadplugin dir]

I am learning to never assume the obvious, hence the question. As always, a big thanks for this work to you and Tom.

[CruNcher]

@ Nic Trbarry and Shodan
absolute great work you do :)

here are my Speed and Size results PSNR results based against idct2 coming soon

nullspeed in XviD was used XviD itself used SimpleIDCT
Test Sequence is a 1501 frame 640x480 capture @ 25 fps DVB
for size test encoded @ q2 no other features Simple Profile

System: Still my good old P4 1.8 Williamete :)


idct1=compression time 00:01:55.756 (12.97fps) 38,2 MB (40.080.896 bytes)
idct2=compression time 00:01:55.748 (12.97fps) 38,2 MB (40.080.896 bytes)
idct3=compression time 00:01:58.076 (12.71fps) 38,3 MB (40.178.176 bytes)
idct4=compression time 00:02:14.946 (11.12fps) 38,3 MB (40.197.120 bytes)
idct5=compression time 00:01:52.496 (13.34fps) 38,2 MB (40.080.896 bytes)
idct6=compression time 00:01:55.920 (12.95fps) 38,2 MB (40.129.536 bytes)
idct7=compression time 00:01:55.783 (12.96fps) 38,1 MB (39.989.760 bytes)


Ok this test is somewhere flawed cause i estimate that idct2 is the best idct here but that has not to be true quality wise


idct1=106.0153
idct2=106.0153 same as (infinite) no difference
idct3=58.5690
idct4=58.4507
idct5=106.0153
idct6=59.1600
idct7=58.4504


idct1,2 and 5 are equal and only differ in speed as shown above
idct3,4,6 and 7 show major differences where idct6 is the nearest to idct1 and idct7 is the farthest

if you do some fuzzy logic you could say that the one with the biggest size is the best quality wise and the one with the smallest the badest but this must not be the true have to do more research to compare them in a more mathematical way could maybe somebody help here ?

[Blight]

How about making the decoder more fault-tolerant toward streams that contain drops (HDTV) ?

[CruNcher]

@ Blight
i know for sure that MarcFD did that for DVB Streams so it should also work for HDTV i think ?

[trbarry]

How about making the decoder more fault-tolerant toward streams that contain drops (HDTV) ?

I put conditional code in MPEG2DEC2 (but not MPEG2DEC3) to set an error trap before decoding each frame and just return GIGO if it failed on that frame. But this would only run when compiled for debug for some reason (thus slower) and could crash vdub when you exited after encoding. So I only used it for problem cases.

- Tom

edit: What I should have done if not so lazy is on errors return a copy of the previously decoded good frame.

[Nic]

@CruNcher:
Thanks loads for those tests :) Dont understand why skal's was slightly slower...Oh well, good to have for completeness.
Looks like SimpleiDCT might be useful-ish though :)
Very interesting second results, idct=7 should be more accurate than idct=2 (idct=2 is almost exactly what XviD used to use (intel one with peter gubanovs changes)). Therefore it appears the difference between there output is quite large ? Or am I misreading the results?

@Tom: SimpleiDCT uses permutated data as its input, (its all in the new file idctnew.cpp (or newidct.cpp, I forget which). Meaning it goes through a little "for" loop to begin with? Is that easy to speed up with assembly or should the compiler already be doing a pretty good job?

@Blight: If you have a stream thats corrupt when using mpeg2dec3.dll and can upload a little bit, I can try and work on it. Otherwise it might be quite difficult for me to fix anything.

@DDogg: In my best homer simpson voice: "I don't know". ;) Ill have to look that up myself. I didn't really like all the override code in there so I tried to ignore it. But ill check and post back.

Cheers,
-Nic

[trbarry]

@Tom: SimpleiDCT uses permutated data as its input, (its all in the new file idctnew.cpp (or newidct.cpp, I forget which). Meaning it goes through a little "for" loop to begin with? Is that easy to speed up with assembly or should the compiler already be doing a pretty good job?

Yep. It would almost certainly be faster in assembler even if you just unrolled the whole loop and moved 1 word randomly at a time since there would be no subscript calculations or loop control.

I'm not sure yet if there are mmx games that would go faster yet. But there probably are.

- Tom

edit: (after further head scratching) ...

It would probably be faster yet if the wrapper was not called at all. Instead modify Simple_idct to rearrange the parms onto the stack. Also the idct could probably be changed to just output the values into the correct place so you didn't have to copy them again on the way back. It looks like there is a simple output pointer in edx that could be changed about half way through and the values do not have to be rearranged on output.

But I won't be able to do this soon myself.

[Kyo]

a Noob question....

If I have an Athlon XP (so, no sse2) and choose the iDCT = 5 (for P4) what iDCT is used?... this will crash? (I write this because this not crash my cpu)

When I run DebugView it only say "Overiding iDCT With: 5" and the encoding process is ok, strange I think :o


BTW Nic,Tbarry,sh0dan Thanks for the new Toys

[JohnMK]

I used idct=5 by mistake on my Athlon XP. Worked fine. I suspect it does a CPUID and when it fails to see SSE2, doesn't use SSE2 featurettes. It may revert to whatever your CPU is capable of, e.g. idct=2.

UNLESS the override code is incorrect and not working. In which case, we're probably all using idct=2 without knowing it.

By the way, where can I find the override definition file? That's my preferred method of overriding; I don't like to tweak the commandline because then it's one more thing my frontend can't do for me with absolutely zero invervention. :)

[Nic]

Override will be set to 5....but then be re-adjusted down to 2. Because SSE2 is not supported on your chip. I thought this was better than MPEG2Dec3 just crashing ;)

-Nic

[JohnMK]

But Nic, you said up there you don't know if override is working.

[Nic]

Where did I say that? And it depends what you mean by override. If you mean the idct you set in the mpeg2source line then that definitely works...but as for the .def file, I honestly cant be sure because I havent tested that bit, nor did I change any of that code, so I just "assume" it works ;)

-Nic

[trbarry]

I'm pretty sure it crashed in an early version of MPEG2DEC2 and I changed it to do pretty much what Nic said. It should drop back to SSEMMX and then to just MMX.

- Tom

[Kyo]

@devs
thx for the answer, this explain the behavior.

[yaz]

dear devels!

would anyone take care of the description of the plugin? there's one included but (imho) it's outta date. e.g. it says def.idct=0(=read from d2v).
would any of u list the available parms & their default values? i'm esp interested in moderate_h/v as it seems to be always on & i'm getting more&more convinced that it's responsible for some strange artifacting on decoding. i've been unable to figure out (from the readme) how to switch it off safely.

thx
y

[Nic]

"description of the plugin" ???
Do you mean the .def file?

-Nic

[yaz]

nic :-)

Originally posted by Nic
"description of the plugin" ???
ok, let's make it clear. there's an mpeg2dec3.html included, created (afaik) by marcfd, describing what&how is going on. that's what i call 'description', but we can call it however u like :-) imho, the infos in there should be actualized. that's all.

Originally posted by Nic
Do you mean the .def file?
say so. it was incld in the former versions but not in the latest one.

anyway, right now i'd be sufficed with the trick of swiching off moderate_h/v safely. what's the way of doing it? btw, is it operable at all?

thx
y

ps. i use this filter with pleasure&success (thx for your efforts!) but at a moment it's occured to me whether i know what am i doing xactly :-)

[Nic]

The .html is upto date and correct. I didn't like the .def file mainly because people would include it with the .dll and not realise that any changes in there would always override the d2v file etc.
(And it just confused matters...you can just use an old .def file if you wish. But I do not support its use)

The moderate_v & moderate_h aren't turned on unless you turn on post-processing. They are the variables that control the strength of post-processing (exactly the same as they are in my XviD DShow filter).

-Nic

[yaz]

Originally posted by Nic
The moderate_v & moderate_h aren't turned on unless you turn on post-processing. They are the variables that control the strength of post-processing (exactly the same as they are in my XviD DShow filter).
-Nic

whoops, i should've got it on my own :-)

thx alot
y

[DDogg]

Nic said, I didn't like the .def file mainly because people would include it with the .dll and not realise that any changes in there would always override the d2v file etc. Hmm, that nearly implies you got a chance to look at the code and the priority of the DEF file? Or did you mean, that it might be possible that it overides but you are not sure? Sorry to beat this subject to death, but you must admit it is a rather confusing area.

[Nic]

It is confusing...and the code for it is messy.

::ill go look now::

Ok, it appears that it will use the .def file (if it can find it, its hard to tell with the current implementation where it will look for the file and if its found it...)

i.e. its just opening mpeg2dec3.def in the "current directory" but when you have the .avs in your c:\, avisynth.dll in your c:\winnt\system32 and mpeg2dec3.dll in your c:\dvd2avi_nic\system directory where is the "current directory" when loading with VDub in your c:\virtualdub directory?

Confusing huh? ;)

But if it does find it, it uses it as the default values for the settings. But you can then override them with the commandline parameters (i.e. mpeg2source(..., idct=2) overrides the .def file.

Hope that clears things up,

-Nic

[DDogg]

Yes, that does clear it up pretty well. I get the feeling that reading that part of the code is as unleasant as going in to the jungle with swarms of big mosquitoes biting you. Thanks for taking the time to make the trip and I hope it will not INVOKE any nasty bites :)

[SILICON]

I made one speed test.

For the test I used
- Film: The Godfather
- Compressor: CCE 2.66.0.10
- MPEG2DEC3 v1.06
- CPU: AMD K7 2000 XP
- AVS SCRIP:

LoadPlugin("C:\DVD2CVCD\PLUGINS\MPEG2DEC3.dll")
Mpeg2Source("D:\test\film.D2V", IDCT=X)
# Trim the begin and the end of film
TRIM(25000,45000)
# Huge resize for minimize process time of encoder.
BilinearResize(16,16)
ConvertToYUY2()


The results are:

iDCT=1 Speed=3.04
iDCT=2 Speed=3.08
iDCT=3 Speed=2.12
iDCT=4 Speed=1.93
iDCT=5 Speed=3.08
iDCT=6 Speed=3.10
iDCT=7 Speed=2.64


I repeat the test two times. The speed are the same the two times.

[Nic]

I guess idct 6 isnt faster and sometimes slower then...oh well, it was faster on the Celeron 1000 I tested on initially. Best to ignore it. idct=5 (SSE2 P4 optimisations) wouldnt have worked on yours so thats why you get the same result as idct=2 (as that is what it would have fallen back on)

-Nic

[Asmodian]

Does anyone know any reason not to use idct 7? This seems like a very good option if it is really more accurate as it doesn't seem much slower then the fastest idct :)

[N_F]

@Nic - In case you don't already know, your web page seems to be down at the moment.

[seewen]

I made a test too.

- 1st Chapter of LOTR extended ED.
- VirtualDubMod 1.5.1.1 (1156)
- Avisynth 2.5.1 (21.5.03 RC4)
- DVD2AVI 1.77.3 ( don't know if there's a speed diff. with 1.76 )
- Mpeg2Dec3.dll v1.06 & MPEGDecoder 2.03
- XviD 14.5.03 ( 1-pass Quant 2, 2 B-frames, ChromaME, VHQ=1, h.263)
- CPU : Athlon XP 2000
- Script :

MPEG2Source("F:\LOTR_SEE_D1\LOTR.d2v",idct=X)
#MPEGSource("F:\LOTR_SEE_D1\LOTR.d2v")

Crop(0,76,-0,-76)


Results :

IDCT - Duration - Size in o.
----------------------------

IDCT 1 - 11:30 - 100'435'968
IDCT 2 - 11:25 - 100'435'968
IDCT 3 - 12:01 - 100'444'160
IDCT 4 - 12:14 - 100'577'280
IDCT 5 --------------
IDCT 6 - 11:25 - 100'771'840
IDCT 7 - 11:41 - 100'278'272

MPEGDecoder - 11:21 - 100'336'336



Last thing I can say is that "IDCT 1" & "IDCT 2" produced identical clips ( I compared them with "Subtract.level" (http://www.avisynth.org/index.php?page=Subtract) ).
"IDCT 3" & "IDCT 4" produced clips which are about 20-50% identical ( depend of frames )
But all other are "completely" differents.

[Lobuz]

Does IDCT 3 has higher quality then IDCT 1,2?
Could it be possible to find the cause of the bug in IDCT 4 which should be a reference IDCT? It's described with sample at DVD2AVI page.
Is there any other IDCT which has the proper(the best) reference quality? If implemented, could be useful to make tests and compares.

Regards
Lobuz

[Nic]

I do really want to test their accuracy properly. It is quite interesting the different results they are producing, id always asssumed they were mostly the same.

@N_F: Had noticed, trying to track Swede down ;)

@Asmodian: Im starting to think that too... :)

-Nic

[Blight]

Integer math will never be 1:1 against float math, but I would expect the error ratio to be close to 0.3%, if you can actually see a difference between float IDCT and integer, then something is off.

BTW, There's a small typo in the html file within the archive, it should read "idct : 1 to 7" (now it says "idct : 1 to 5").

Is iDCT 3 and 4 the same accuracy? And is the IEEE-1180 Reference the most accurate you can get?

[Nic]

Well ill test the iDCT's soon.
http://arbor.ee.ntu.edu.tw/~jackei/dvd2avi/idctref/

Explains that perhaps the reference idct isnt the best at all because of a bug that don't know was ever fixed.
SimpleiDCT is what xvid/ffmpeg rely on to do accurate QPel...So thats what im assuming to be the most accurate (because I know it well). But the floating point one _should_ be the most accurate I guess, but I wont know till I test.

I honestly didnt expect any difference between the idct's apart from speed, so maybe there is a problem or the idcts are just too optimised at the expense of accuracy.
(The slightest difference in accuracy made a huge difference to xvid's qpel encoding, without simpleidct, xvid used SSEMMX idct which caused very bad smearing...as some will remember)

-Nic

[trbarry]

Where did Simple IDCT come from? And has anyone ever made an SSE2 version of it?

- Tom

[Cyberia]

Aren't Athlons supposed to have MUCH faster floating point performance than P4? How about some optimizations of this iDCT for us Athlon owners. I'm not complaining, but there *have* been alot of SSE2 discussions for the P4.

Let the different modes target the full strengths of the different processors. Maybe it already it, I dunno. Just a suggestion.

[Nic]

SimpleiDCT came from XviD but that intern got it from the ffmpeg project, and there is quite possibly a SSE2 version of it, but I dont know of one.
(edit: actually looking at it there isn't a sse2 version of it:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/ffmpeg/ffmpeg/libavcodec/i386/simple_idct_mmx.c?rev=1.10&content-type=text/vnd.viewcvs-markup )


@Cyberia: Well we pretty much all use integer idct for decoding...so the athlons supposed improved float performance, wouldnt be that useful.

-Nic

[Sigmatador]

A remark:

If i put a lumafilter(0,1.6) in my script, it increase the luminance (all is normal for the moment ^^). But if i put a second lumafilter like lumafilter(0,0.6) for example, it increase the luminance again.

the same problem with the inverse: lumafilter(0,0.4).lumafilter(0,1.6) doubly decrease the luminance

it's only me or it's a real bug ?

[Nic]

Hmmm..I had a quick look at the code and didnt spot why that would be immediately, ill look properly soon :)

-Nic

[sh0dan]

Isn't that how lumafilter is supposed to work?

Applying it two times simply multiplies the effect - or is there something about the parameters I don't realize?

[Sigmatador]

@shodan:
For a script, i want to test if a lower luminance could help a filter. But i want to "restore" the luminance after filtering (dunno if it's stupid, i just want to test ^^)

now a stupid test ^^:
Lumafilter(0,1.6) --> increase the luminance
Lumafilter(0,0.4) --> decrease the luminance
right, all it's normal :D

Lumafilter(0,1.6).Lumafilter(0,0.4) doubleincrease the luminance
LuMafilter(0,0.4).Lumafilter(0,1.6) doubledecrease the luminance
instead "cancelling" (not true, i know ^^) the lumafiltering

i think there's a bug, or it's me that didn't realize something about the parameters :D

[ARDA]

@Nic

LumOffsetMask,LumGainMask are all static and they should be const;
Change static for const and it works ok. with several calls.

Arda

[sh0dan]

Well spotted (again) ARDA!!

[Sigmatador]

(mardfd bug spotted ^^ ) Yop good job ARDA, it works fine now ^^ thx ;)

[sh0dan]

Another bug: found in the gknot forum (http://forum.doom9.org/showthread.php?s=&threadid=52799).

The crash is 100% reproducible with the sample clip (at least on my machine) - it seems to crash somewhere in motion compensation. I have no idea how to fix it - somehow I even get a corrupted stack in the crash, so it's hard to backtrace.

If either of you have the time to look at it, it would be great!

[trbarry]

The stuff that blows up on motion compensation/estimation is usually bad data. (yes I know some other codecs play it) I think I already discussed this above somewhere and what I did for it in MPEG2DEC2. It might be interesting to try that clip with a 2.5 version of MPEG2DEC2. I think Neuron2 posted one somewhere.

I still can't compile anything on my system. As part of recovering my torqued XP system back to Win/Me I foolishly Xcopy'd a bunch of my source with a startup disk and ended up with all 8.3 dos filenames I now have to deal with for some reason. :(

- Tom

edit: as a workaround in vdub you can usually note the frame number where it dies and delete about a dozen records before and after to make it work

[Nic]

@ARDA: Sweet, well found, and easily corrected. :)

@sh0dan: Thanks for the heads up, ill fix that today (well, I dont know how easy it will be to fix correctly, but ill stop it from crashing at least ;) )

-Nic

edit: Just so you know, mpeg2dec2.dll crashes in the same place, the error is, as sh0dan states, in the motion code. it happens in the ppppf_motion call from form_prediction
(ppppf_motion is pointing to MC_put_xy16_mmx at the time and crashes on a movq)
(same crash happens differently if using SSEMMX version)
(oh, and its the 125879 iteration of form_prediction that it happens on)

[sh0dan]

@Nic: I get the same crash place, but the pointer always seem off, by a few lines - and I cannot see where the MMX/ISSE functions are actually being called and with which parameters. Could be something I missed.

btw - the same file seems to be decodeable by DVD2AVI - so it suggests some problem introduced at some time.

[Nic]

Hmmm, Well I put back the old code for form_prediction and form_prediction_component from dvd2avi and that didnt fix the problem, so the problem must be higher up...

ill keep looking :)

-Nic

ps
Ok, obviously to stop the crash, you make sure that when doing:
unsigned char *s = src + lx * (y + (dy>>1)) + x + (dx>>1);
unsigned char *d = dst + lx * y + x;

that (y + (dy>>1)) doesnt end up being negative and therefore s becomes less that src. Now just got to find out why that happens ;)

edit2: the above is using the dvd2avi code, in mpeg2dec3 its:
unsigned char *s = (src[0]+(sfield?lx2>>1:0)) + lx * (y + (dy>>1)) + x + (dx>>1);
(i.e. putting a:
if ( s < src[0] )
return;
after that line will stop the crash, but that's no solution)

edit3:
Sorry to keep filling out this, but if we're all trying to do the same thing it might help:
dvd2avi has the same problem! It just doesn't care though? ;)
(y + (dy>>1)) ends up being negative (exactly the same as mpeg2dec3) and therefore the pointer s is below src, but it can write to that memory without causing any problems???

[sh0dan]

Great work - what about simply min/maxing the values - does it produce corrupt images?

[Nic]

Just checking for negative values works fine, so adding:
if ( s < src[?] )
s = src[?];

for ? = 0, 1 and 2 in that function then that doesnt cause a crash and no noticeable corruption as the output. So that is at least a temporary solution. Feel bad about adding three compares to a function thats called thousands of times though :(

-Nic