MPEG2Dec3 v1.10

[trbarry]

(edit2: The bug's in the Add_Block code, but havent found it yet, the intra/non-intra code is fine)

Nic -

I probably did something stupid there but I should be able to find it with a compare. Or I guess you can just not use the new Add_Block portion for now.


Edit3: Forgot Marc broke the Crop support, (using crop inside dvd2avi when creating the d2v file crashes the mpeg2dec...nice. ) Ill try and fix that or at least make it so it ignores the crop params from the d2v file.

Yep, I forgot about that again too. ;)

IIRC, the crop was also broken when I first started working on MPEG2DEC, but I fixed it. The problem originally was that it was implemented inside one of the color conversion functions of Store.cpp and wasn't adjusting for the 2:1 size difference for chroma planes.

I don't know where it should be implemented now, for YV12, since those conversion functions are hopefully not even being used. Probably just as an adjustment to the CopyAll parms.

Is it really true that this is slower than Marc FD's last version for some reason?

- Tom

[sh0dan]

Wow - that's a big percentwise change, alx gets.

Nics version might be bacuse of ICL - trbarry's version could be because it has inlined assembler within a rather CPU intensive section. MSVC has a tendency to disable optimizations in a C-block, if it contains inline assembler. Calling other functions containing the assembler doesn't have this impact.

@Nic: Look for an Off by one bug. :D

(sorry Tom - couldn't help myself) ;)

[Nic]

The version ive been testing recently is definitely faster, but ive got alot of machines to test on before the next "release" :)

Crop could just be implemented at the end of ::GetFrame or anywhere really, I tried to use env->subframe, but it didnt work out.

-Nic

ps
@sh0dan: Thanks for the hint, but feel free to post the fix ;)

[WarpEnterprises]

sorry for interrupting your thoughts but can I summarize that you (Nic, trbarry) are going to create a common "stable" mpeg2dec-version?
(which is really overdue)

[sh0dan]

Using subframe is a bit tricky here, since MPEG2DEC writes full resolution to the AviSynth PVideoFrame.

You would have to create a separate full-resolution videoframe, and returning the cropped one to AviSynth in the constructor.

The full resolution VideoFrame should be used for the env->NewVideoFrame(vi), that constructs the output frame for MPEG2DEC. Then the subframe function can be used to return the cropped version, that correspons with the VideoInfo returned by the constructor.

transform.cpp / Crop function can of course be used as a reference here.

Oh - This is actually the hard way of doing this. The easiest way is probably to invoke the internal crop filter:

in AVISynthAPI.cpp:

AVSValue mpegS = new MPEG2Source(
args[0].AsString(d2v),
args[1].AsInt(cpu),
args[2].AsInt(idct),
args[3].AsBool(iPP),
args[4].AsInt(moderate_h),
args[5].AsInt(moderate_v),
args[6].AsBool(showQ),
args[7].AsBool(fastMC),
args[8].AsString(cpu2),
env);

AVSValue CropArgs[5] = {mpegS.AsClip(),10,10,-10,-10};

return env->Invoke("crop",AVSValue(CropArgs,5));


Fill in crop args as you please. I haven't had time to test it, but it compiles and should work fine. Leaving my previous ramblings for you, so you can see why SubFrame gives you problems.

[trbarry]

The reason I was first enthused about making crop work in MPEG2DEC2 was partially for convenience but mostly for speed. Some of my 1920x1080 HDTV upconverts (like Buffy, no longer an issue after tonight :( ) had to be cropped to 4:3 taking off 256 pixels on each side.

That was 512x1080 pixels on each frame that did not have to be converted to YUY2, something noticeable in performance.

It is probably less important now when the data just has to be copied for YV12, but who knows.

The other possibility I've discussed here before is to not crop in MPEG2DEC3 at all but just pass both the crop & resize parms back to Avisynth, as global variables that scripts could refer to by some known names, later on. Especially with resize this would allow the convenience of specifying the values interactively in DVD2AVI but still allow the resize to occur after deinterlacing, where it belongs.

- Tom

[Nic]

@sh0dan:
Damn, didn't think of doing it that way and its very obvious too. doh. Oh and as for subframe I realised that problem, and I thought I compensated for it, but must have made a mistake somewhere (trying to code two things at once (while I was at work ;) )

@trbarry: Its a nice idea, but I wouldn't want a special thing in avisynth just to deal with the cropping and resizing, sh0dan's method seems a good solution...beem meaning to look into "invoke" for a while...as DDogg will testify ;)

Your Intra/Non-intra code does cause mpeg2dec3 to go slower too Tom, don't know why, might be for the reason's sh0dan said.

Anyway ill put all this stuff together and test it out :)
I honestly think, unless one of us gets a brainwave, its going to be hard to squeeze much more out of mpeg2dec3 speed wise, the structure of it is more the problem than anything. I might go back do what I should have done in the first place with mpegdecoder, and use libmpeg2 to make a mpeg2dec clone.

-Nic

[trbarry]

Your Intra/Non-intra code does cause mpeg2dec3 to go slower too Tom, don't know why, might be for the reason's sh0dan said.

Strange, it was faster for me, unlike my first try. But I only tested it on a P4 so maybe I just fell into some P4 specific optimization.

But if the Add_Block stuff is busted and the other stuff is slower then I guess we might as well just delete my test release 1.05.

- Tom

[Nic]

@Sh0dan: That crop code worked great :) Thanks for that, should have been able to come up with that idea myself though :(

@Tom: Well im using an AMD 1800 XP ill leave the code in and test more. Have you spotted the bug in Add_Block? Can you fix it? :)

Ive been playing round with another iDCT (yes, I know, another one). Which seems very accurate and that little bit faster, but im still profiling it.

May as well release a latest version to update the 1.04 soon... :)

-Nic

[sh0dan]

I'm also starting to enjoy filter invokation, and have used it in my latest filters (MipSmooth), and in a more advanced version in ConditionalFilter.


Regarding the bitblitting, you could simply assign AvsEnv 0 in the mpeg2decoder constructor, and do an

if (AvsEnv) {
bitblt
} else {
use existing
}

Edit: Not sure if it is safe to always use env from the constructor - setting it each frame will not be any performance problem.

[Nic]

I thought as much..exactly what im doing. I have to check the AVSEnv in copyall/etc because of when the DLL is being used as standalone.

But I was wondering whether this "if" statement will negate any improvment we get from using BitBlt... ?

The invokation (like that word ;) ) could also be used to take the resize parameters from dvd2avi, but I dont think ill implement that (yet).

-Nic

[sh0dan]

Even in worst case it will cost 20-30 cycles (on P4 - less on K7) - but considering the amount that's being copied this is nothing.

Besides, the processor will be able to brach predict this 100% after 3 runs, since it never changes.

[trbarry]

@Tom: Well im using an AMD 1800 XP ill leave the code in and test more. Have you spotted the bug in Add_Block? Can you fix it?

Nic -

Could you set a break point and confirm it is even going through the new _SSE versions of those functions on your Athlon?

I'll still take a look at the Add_Block. It's possible I really can't do 8 bit arithmatic there without loss of precision but it's probably just a silly bug.

- Tom

[sh0dan]

I can see no reason at all, why Toms Add_Block should be slower - everything tells me it should be faster. Loop unrolling will have a larger effect on P4 compared to K7, but still it should be faster.

There are fewer instructions in the loop - no brach mispredicts.

It might be connected to the non-linear memory access. Is it possible to do a version, that accesses memory more linear (and doesn't look up eax, followed by eax+edx) - it does however seem unavoidable to me.

Perhaps doing:
// make rfp qwords 0, 1
prefetchnta [eax+edx*4]
movq mm2, [eax] // get rfp val
movq mm3, [eax+edx] // "
movq mm0, [ebx+0*16]
movq mm1, [ebx+1*16]
packsswb mm0, [ebx+0*16+8] // pack with SIGNED saturate (unlike old way)
packsswb mm1, [ebx+1*16+8] // pack with SIGNED saturate


Might be a bit faster (moved eax loopups to top, to avoid stalling the ebx lookups, and prefetching further down). Remove the prefetch, if ISSE is not allowed here.

[trbarry]

Sh0dan -

From what Nic said the Add_Block wasn't the slow part. It was the broken part. :confused:

And I didn't want to add a separate section there for SSEMMX, hence the lack of prefetch. That part is not in a tight loop anyway.

But right now I first have to figure out how to even get the right answer.

- Tom

[Nic]

Yup Add_Block was causing the errors for me. They were only slight ;) (you could see random blocks appear). If you need a test clip to re-produce it ill put one up along with an accompanying d2v file

As for my minor progress, the crop stuff is working and tested, the iDCT is now done using a function pointer, Ive added two more iDCTs: one is Skal's from his MPEG-4 project (which is the fastest ive ever come across, he's given me permission to put into mpeg2dec) and the other is SimpleiDCT from XviD, which is known to have very high precision (although a tad slower, thought it might be useful ?).

Ive added Sh0dan's suggestion of using BitBlt and all the external code (i.e. using MPEG2Dec3.dll without avisynth) seems to be working fine. (ive written a little commandline example to go with the source i.e. GetPic d2vfile frame output.bmp -> for capturing bitmaps)

The speed is now definitely faster on all machines ive tested, but trbarry's intra/non-intra code is still slowing it down on my Athlon ? But I cant think why, ill test more.

Hope that all sounds ok :)

-Nic

[JohnMK]

Can you post your source or a binary? :) I'm just learning how to use Visual Studio .Net Professional + ICL 7.1 and I'd love to experiment. :D

[Nic]

Use 1.04 source (first post of this thread) for now. ill post it when its ready for release, which will hopefully be later today :) (I dont have it on me now, hopefully Tom/Sh0dan will fix the new Add_Block in that time)

Ive got ICL 7.1 as well (well ive got the evaluation version, until they send the full license). Doesn't make any real difference about 1-2fps faster also had a problem with some of the 3DNow assembler If I remember correctly :)

Cheers,
-Nic

[trbarry]

The speed is now definitely faster on all machines ive tested, but trbarry's intra/non-intra code is still slowing it down on my Athlon ? But I cant think why, ill test more.

Nic -

Did you ever check if the following block of code (in Getpic) is even being executed? Mine's the only place in Getpic that checks that cpu.ssemmx flag and I can't test that myself on an Athlon.

- Tom

/* decode blocks */
// separate rtn for ssemmx now - trbarry 5/2003
if (cpu.ssemmx)
{
for (comp=0; comp<block_count; comp++)
{
if (coded_block_pattern & (1<<(block_count-1-comp)))
{
if (*macroblock_type & MACROBLOCK_INTRA)
Decode_MPEG2_Intra_Block_SSE(comp, dc_dct_pred);
else
Decode_MPEG2_Non_Intra_Block_SSE(comp);
if (Fault_Flag) {
#ifdef PROFILING
// stop_decMB_timer();
#endif
return 0; // trigger: go to next slice
}
}
}
}
else

[Nic]

Yup I did check...It does get called :) (it would do, the Athlon XP has all extended instructions apart from SSE2).

I honestly dont know why its slower ? But it does appear to be (but only slightly). Did you manage to fix add_block?

Cheers,
-Nic

[Sigmatador]

Someone knows this compilateur ?
http://www.codeplay.com/vectorc/bench.html

[Acaila]

I tried Codeplay out a while ago, and this is what I think of it:

- Despite that the demos look really promising, it is specifically made for vectorizing code (2D/3D modelling, rotations like the demos), and isn't very spectacular on normal stuff.
- I could hardly find any code that compiled on it at all. Almost everything gave an error of some sort.
- The latest version supports C++, but that support is so limited that you're better off using it only for plain C.
- In my opinion it was very expensive. $100 for a nutured version, $800 for the full version.
- The nutured version isn't capable of Athlon optimizations (although at first they said it would be P4 that wouldn't be supported). And since I have an Athlon I very much didn't like that :).

[sh0dan]

vectorc is primarily aimed at float point optimizations. MPEG2DEC doesn't really contain any float point code, so I very much doubt it will be of much use here.

However we really cannot know this until it is tested.

[Sigmatador]

i talk about vectorc, not specially for mpeg2dec3, but for general c/c++/asm compiling (and increasing filter speed, for people, like me who are able to write asm code slower than their pure c one ^^ )

[trbarry]

Did you manage to fix add_block?

Nic -

I found it but haven't corrected it yet. There are 2 sections of mmx code in Add_Block and the first one can not be done in 8 bit arithmatic without overflowing. But I can still optimize it a bit. Hopefully I'll get a replacement out today.

Should I still base it on v 1.04?

- Tom

[sh0dan]

Uh - conflicts :(

IMO you should use 1.05.. Maybe you should just check for SSE2 instead - that way it will only run on P4 boxes for now.

[Nic]

Use any version you like trbarry, or just post the fixed code snippets and ill add it into the version ive made up.

Ill just take your code and fit into my version, test it and release it. And then that will be that for a little while I think :)

Cheers,
-Nic

ps
BTW: I just bunged up quick the latest source at:
http://nic.dnsalias.com/src.zip
Just in case you want to see what ive done so far :)

[trbarry]

Nic -

Oops. Too late to get the new one first.

I just posted a fixed Add_Block function snippet at:
www.trbarry.com/Add_Block.txt .

And maybe give some thought to Sh0dans comments about SSE2 only for the performance functions. Though it should run faster on P3's too. Maybe Athlon has a slower bsr instruction (emulated?). I don't often use that but don't see a fast way around it.

- Tom

[sh0dan]

Originally posted by trbarry
Maybe Athlon has a slower bsr instruction (emulated?). I don't often use that but don't see a fast way around it.

- Tom

Yes - BSR is VectorPath, and executes in at least 10 cycles. Furthermore the code isn't pairable at all, leaving 2 of the three pipes unused.

[Nic]

Thanks Tom :) Im off to see the Matrix Reloaded tonight, so I cant test now. But Ive got a P4 and an old Athlon 800 to test on at home, so ill try them and check :)

Cheers,
-Nic

[trbarry]

Yes - BSR is VectorPath, and executes in at least 10 cycles. Furthermore the code isn't pairable at all, leaving 2 of the three pipes unused.

Sh0dan -

I don't know what VectorPath is, but I assume it's costly from the rest of it.

What do you think of requiring SSE instead of SSEMMX for those 2 functions of mine that need bsr? That should eliminate the Athlon's but accept P3 & P4. And it's a one line change.

- Tom

[sh0dan]

VectorPath is AMD's word for complex "emulated" instructions, that usually requires several cycles to decode and execute.

Nic's Athlon XP has SSE, so it will be used there. :(
It is however impressive that the "if" code can execute faster than the assembler.

[trbarry]

It is however impressive that the "if" code can execute faster than the assembler.

It's more than impressive. It's totally frelling mind boggling. But it is true that the if/else logic usually falls out after the first 1 or 2 tests on normal data, so it's nowhere near as bad as it looks.

- Tom

[Nic]

Ok, I released 1.06 for now. (releasing it as 1.05 might have caused confusion)

@Tom: Has your block decoding optimisations for sse2 only for now, hope thats ok.

All seem good to you?

-Nic

[DDogg]

mpeg2source("D:\mymovie.D2V",idct=6) returns:

MPEG2Source:IDCT invalid(1:mmx,2:SSEMMX,3:FPU,4:REF;5:SSE2)

[AmiRage]

Originally posted by DDogg
mpeg2source("D:\mymovie.D2V",idct=6) returns:

MPEG2Source:IDCT invalid(1:mmx,2:SSEMMX,3:FPU,4:REF;5:SSE2)

Same here for idct=7 also.

[cult]

where is it?I cant see any attachment

[ARDA]

First post of this thread

Arda

[Nic]

Sorry didn't check marcfd's override param, I always change it in the d2v...ill go ammend that now..

-Nic

edit:
Ok, that bugs fixed, please re-download:
(updated links always at start of thread)
http://nic.dnsalias.com/MPEG2Dec3v106.zip

SourceCode:
http://nic.dnsalias.com/MPEG2Dec3v106_src.zip

pps
BTW: Skal's iDCT is faster, but only slightly, you may not notice a difference, but its the fastest iDCT ive ever seen, so its included for completeness

[HarryM]

Originally posted by Nic
pps
BTW: Skal's iDCT is faster, but only slightly, you may not notice a difference, but its the fastest iDCT ive ever seen, so its included for completeness [/B]

@Nic: Can you add Skal's code into your MpegDecoder?

[lighty]

Originally posted by HarryM
@Nic: Can you add Skal's code into your MpegDecoder?
AFAIK- he already did include it as he said in the previous post.:p

[Ctrl-Alt-Suppr]

Skal's iDCT is faster
Are you sure ?

DVD zone2 : BLADE ( 3 min ) ~ Asus P4G8x ~ P4 2.53°2.89 ~ 512mo
AviSynth 2.51 RC4 ~ Xvid's Koepi 14/05/2003 ~ VDubMOD 1.4.13 v2 (for debug .log)

MPEG2Source("D:\...\.d2v",idct=X ).Crop(4,72,-8,-76).Trim(99500,104000)
BicubicResize(608,256,0,0.5)

idct=1 ___ 83 sec ___ 54.22 fps ___ 34.9 mo
idct=2 ___ 82 sec ___ 54.88 fps ___ 34.9 mo
idct=3 ___ 89 sec ___ 50.56 fps ___ 34.9 mo
idct=4 ___ 133 sec ___ 33.83 fps ____ 35 mo
idct=5 ___ 80 sec ___ 56.25 fps ___ 34.9 mo
idct=6 ___ 82 sec ___ 54.88 fps ___ 34.9 mo
idct=7 ___ 86 sec ___ 52.33 fps ___ 34.9 mo

[HarryM]

Originally posted by Ctrl-Alt-Suppr
Skal's iDCT is faster
Are you sure ?

DVD zone2 : BLADE ( 3 min ) ~ Asus P4G8x ~ P4 2.53°2.89 ~ 512mo
AviSynth 2.51 RC4 ~ Xvid's Koepi 14/05/2003 ~ VDubMOD 1.4.13 v2 (for debug .log)

MPEG2Source("D:\...\.d2v",idct=X ).Crop(4,72,-8,-76).Trim(99500,104000)
BicubicResize(608,256,0,0.5)

idct=1 ___ 83 sec ___ 54.22 fps ___ 34.9 mo
idct=2 ___ 82 sec ___ 54.88 fps ___ 34.9 mo
idct=3 ___ 89 sec ___ 50.56 fps ___ 34.9 mo
idct=4 ___ 133 sec ___ 33.83 fps ____ 35 mo
idct=5 ___ 80 sec ___ 56.25 fps ___ 34.9 mo
idct=6 ___ 82 sec ___ 54.88 fps ___ 34.9 mo
idct=7 ___ 86 sec ___ 52.33 fps ___ 34.9 mo

You have SSE2.
I have AthlonXP and idct=6 (Skal's SSEMMX) is about +2% really faster than idct=2 (SSEMMX) for me.
Of course - without Crop, Trim, etc. and with 'null - test speed' in xvid.

[HarryM]

@Nic:

Skal's SSEMMX idct is more accurate than FPU 64bit idct?

[Sigmatador]

we need someone with a 64bits processor :D

[JohnMK]

Is there a way to force an idct by placement of some kind of overriding text file with your preferred idct in the same dir as mpeg2dec3? I don't like the idea of tinkering with my .d2v, I'd rather have an automated solution for the future. :p The older versions used to have this feature . . . .

[HarryM]

Originally posted by JohnMK
Is there a way to force an idct by placement of some kind of overriding text file with your preferred idct in the same dir as mpeg2dec3? I don't like the idea of tinkering with my .d2v, I'd rather have an automated solution for the future. :p The older versions used to have this feature . . . .


Mpeg2Source("...", idct=6)

[Nic]

@ctrl-alt-supr: "Are you sure?"
I did write for a reason:
"Skal's iDCT is faster, but only slightly, you may not notice a difference, but its the fastest iDCT ive ever seen, so its included for completenes."

And yes I am sure.

-Nic

[killingspree]

wow... you got me confused now (:
what's supposed to be faster on a PIV now? idct=5 or 6?

regards
steVe

[Nic]

@killingspree: Hmmm, 5 (SSE2) probably will be, for now I wouldn't start using 6. The speed difference is small and I havent tested it fully yet.

(when I tested it, I just left it doing random iDCTs and comparing it to other iDCTs, and it is faster, but it seems in MPEG-2 Decoding it doesnt make any real difference, still its there for completeness, ill test its accuracy properly soon)

-Nic

ps
Im almost tempted to use 7 (simpleidct) myself...even though tests have shown that iDCT accuracy doesn't seem to make any real kind of a difference to the eye when mpeg-2 decoding, maybe it will make a difference to the encoder (even if very very tiny)...Hmmm, I may have to check.