DGDecode multithreaded chance?

[Zep]

Hi DG,

Just wondering if it was possible to multi thread the mpeg 2 decoding
so that the top half of the frame is decoded on one core and the bottom half
on another core. (perhaps 1 core on current frame and the other core
on next frame then just change the pointer to point to that area
in memory?)

I ask because decoding compared to say elecard SMP decoder
is about half the decode speed. I get a HUGE Xvid encoding speed
boost when i use the elecard decoder BUT DirectShowSource
gets the time/length wrong (IanB is working on a debug version
of DSS so we can figure out why that is) Anyway when using
DGDecode my Dual core AMD i sit at 68% CPU with my normal
set up. The Xvid thread in VDub has no problem keeping up.

After many tests it appears simply that DGDcode just can't feed
frames fast enough so everything else is sitting around waiting.
With the elecard SMP decoder which does use both cores I max at
100% CPU usage and get about 100% increase in frames fed to VDUB
on my direct stream copy or using AssumeFPS(1000) and doing
previews in VDub with and without showing the video.

Anyway, i would rather use DGindex (and do) as it is more precise and
seems to index about 0.5% more frames than the elecard decoders end up
sending to DSS (unless this is a DSS problem of course which IanB
is looking into. i feel the elecard decoders are dropping some frames
due to quality messages in the stream but i can't be sure)

I tested avisynth-MT but it doesn't help with this filter.
(it does speed up some other filters a ton though)


Thoughts?


Thanks,


Zep

[Revgen]

Try TSP's SetMTmode filter from the avisynth-MT like this:

SetMTmode(2)
mpeg2source("your.d2v")

It's important that you keep the SetMTmode line above the any "source" filters or parameters, or else the multithreading won't work. I have an AMD 4600+, and it works fine for me.

I get abot 6 FPS with MT off with my HDTV transport stream, and 12 FPS with it on.

[Zep]

Try TSP's SetMTmode filter from the avisynth-MT like this:

SetMTmode(2)
mpeg2source("your.d2v")

It's important that you keep the SetMTmode line above the any "source" filters or parameters, or else the multithreading won't work. I have an AMD 4600+, and it works fine for me.

I get abot 6 FPS with MT off with my HDTV transport stream, and 12 FPS with it on.

i did. That is how i always have used it. it slows down DGdecode
for me and makes the frame rate roller coaster also.

My guess is you get an increase because your FPS is so low.
I'm at 40 FPS at 624 x 352 max Xvid settings. When i use the
elecard decoder I get around 75 FPS with same settings.
This is pulling from .ts HDTV 1080i. I get a little faster
rates on 720p.

I have a 4400+ OC to 2.63GHz with PC4000 ram at
263Mhz (526Mhz effective rate) X 10 multi.


I'm curious as to why your FPS is so low? that is with x264 or
some other codec? not Xvid right? cause surely something
is wrong if that is Xvid rates :)

NOTE: even at full rez encode i get about 20 FPS with Xvid
however i do max both cores then and there is no need for
DGdecode to go faster since more time is spent in Xvid thread
than the avs thread in VDub.

[Guest]

It's not an exciting project for me, but if someone wants to try it, that would be a good thing.

[Revgen]

@Zep

Like I mentioned in my last post, it's an 1080i HDTV MPEG2 transport stream.

[NaN]

I thought of it, however more about multithreading on a per macroblock basis. However I don't have a SMP machine and for hyperthreading alone the work maybe will not pay off.

What I'm really interested in would be how the elecard decoder works. I cannot really believe that the decoding speed matters for xvid encodings, since the decoder eats below 10% of the overall cpu-time - even if you reduce dgdecode's cputime down to zero, your encodings are these <10% faster.

Further I want to see their idct and how accurate it is. And how accurate their motion compensation is.

More Speed means usually less calculations (less quality) that's the reason, why speeding up plugins is a time consuming task (after the usual SSE-optimizations are done).

Cheers, NaN

PS: to get real numbers, try a profiler!

PPS: you can of course get more speed using fast motion compensation (fastMC) and a high speed idct (usually Skals or Dmitry's SSE2).

[Zep]

It's not an exciting project for me, but if someone wants to try it, that would be a good thing.


LOL i understand DG :D


It is just your tools IMHO are by far the best and now with dual core
finally kicking into high gear on the PC side (I have been a mac user since
the 1986 also so I have gotten used to everything being written for dual
CPU/core from the get go) Well.... I would love to see your great tools
get even faster :)

Thanks for the hard work you put into this


Zep

[Zep]

I thought of it, however more about multithreading on a per macroblock basis. However I don't have a SMP machine and for hyperthreading alone the work maybe will not pay off.

Ahhh now this is gonna be fun to reply to :D

i guess some depends on how the chipset pre fetches and how each
core's cache is getting filled etc... that would be interesting tests to
run compared the simple split the frame at a marcoblock border and
have each core work on half the frame. Or maybe a combo
of both. Have each core working on a macroblock line one core
takes even lines and the other odd then work your way down the frame.

fun fun fun tests lol


What I'm really interested in would be how the elecard decoder works. I cannot really believe that the decoding speed matters for xvid encodings, since the decoder eats below 10% of the overall cpu-time - even if you reduce dgdecode's cputime down to zero, your encodings are these <10% faster.

It appears there is a wall DGDecode runs into around 40 FPS. It could be
simply DGDecode does more since it has to read the d2v file then go to
decode that frame. However my guess is the way DGDecode reads the
stream from hard drive is the bottleneck at high rates. I know DG had a
HUGE speed increase when i found a bug like a year ago in the way it read
the m2v when using selecteven(). It went from 4PFS to 29FPS after
DG changed the way it read the data in.

remember i sit at around 68% with DGDecode and 100% cpu load
with elecard that shows you right there the extra CPU used is being
used wisely as the decode rate shoots way up.


Further I want to see their idct and how accurate it is. And how accurate their motion compensation is.

More Speed means usually less calculations (less quality) that's the reason, why speeding up plugins is a time consuming task (after the usual SSE-optimizations are done).

Cheers, NaN


Output quality i can't tell the diff with my eyes.
i use Skals idct. And more speed may not mean less calcs because
in THIS case a lot MORE cpu is being used so we can't assume that unti
tests are done. In fact more calcs may be done since i know for a fact
elecard obeys quality messages and why it drops some frames where
as DGDecode as far as i know does not. Of course this means though
the upfront may have more calcs but the backend decode stage dosn't
have to decode that frame so time is gotten back. However, we are
talking out of 60,000 frames elecard only dropping around 100 frames
sometimes more sometimes less compared to what DGindex said
to send out.


PS: to get real numbers, try a profiler!

PPS: you can of course get more speed using fast motion compensation (fastMC) and a high speed idct (usually Skals or Dmitry's SSE2).

profiler not gonna help much unless you can isolate well.
You really would have to turn off 1 core and go function level
to get any usable stats and even then you are comparing apples
to oranges since one code base is written ground up for HT/SMP/DC
and the other is not, so even turning off 1 core (in this example) means
elecard may be slower than DGDcode since its code base is written
with HT/SMP/DC in mind and most likely there is some code overhead for
HT/SMP/DC and thread sync etc...

Now you could profile against elecard's NON HT/SMP/DC decoder
which is a free download and see how it stacks up against DGDecode
and that may at least tell us who has the BASICS down better. :)

but this still leaves us with the quality issue which can be hard
to factor in.


I use Skals idct.


wow that was a fun post to reply to :D


thanks

Zep

[NaN]

Sorry for replying so late!

You're right! I would love to see Elecards sources...However 2 things: dropped frames are a shame, even it is that fast. But that's only my opinion of course!

2nd I cannot see this 40fps wall you talk about. I see up to 70fps (3 times realtime) here, the open() function does not take significant time on my machine. Unfortunately I'm not a windows-api freak, so I've no own win32-experiences and cannot tell wether this could be a problem or how to fix that.

Your 68% really would need a closer look, since I think something happens on your machine that I simply do not see here.

Cheers, NaN

[Zep]

Sorry for replying so late!

You're right! I would love to see Elecards sources...However 2 things: dropped frames are a shame, even it is that fast. But that's only my opinion of course!

2nd I cannot see this 40fps wall you talk about. I see up to 70fps (3 times realtime) here, the open() function does not take significant time on my machine. Unfortunately I'm not a windows-api freak, so I've no own win32-experiences and cannot tell wether this could be a problem or how to fix that.

Your 68% really would need a closer look, since I think something happens on your machine that I simply do not see here.

Cheers, NaN

yeah the drop frames (though only like 100 in 60,000) does suck
but elecard is obeying the quality messages so elecard really is doing
the right thing even though i do not like that it does it since the frames
look ok to me.

yup not sure what to tell you. even a simple script that ONLY
is reading and passing on the stream (nothing else) can't feed
it fast like the elecard decoder via DSS call can. It sucks seeing 68%
knowing so much CPU is not being used.

I did another interesting test. I ran 2 avs test scripts with 2 VDubs
at the same time and i got 35 FPS in each and 100% cpu.

things that make you go hmmmmmm :)

[Doom9]

@Zep: I suggest you have a look at your posts on a high res display.. your quotes span the 30" diagonal on my dell (2560x1600) nicely, your posts span about 20% of my screen. I know it's completely OT but having owned large screens for a while, I feel a duty to remind people that just because something looks good at a low res, there's really no reason to hit enter over and over again if you have software that is perfectly capable of adding line breaks in function of resolution ..

[Zep]

@Zep: I suggest you have a look at your posts on a high res display.. your quotes span the 30" diagonal on my dell (2560x1600) nicely, your posts span about 20% of my screen. I know it's completely OT but having owned large screens for a while, I feel a duty to remind people that just because something looks good at a low res, there's really no reason to hit enter over and over again if you have software that is perfectly capable of adding line breaks in function of resolution ..

It has nothing to do with hi rez display. I am 1920 x 1200 on my PC and
2560 x 1600 on my Mac. I'm Old school and go by 80 character rule and
I believe high rez screen or not, nothing should ever be 1 line of 80+
characters because that is hard to read. No return characters so that a
whole paragraph sits on 1 long line make text look yucky. Sorta like your
text above which is just 1 long line and a third on my screen :D

Anyway, it is how i type. I just feel the 80 character limit and hit return
because i have been doing it since the late 70's. LOL Yup there is a reason
why most forum software input area is static to 80 no matter how wide you
grow the window. They are trying to keep true to the 80 character rule :)
Which now has me wondering why you do not override it since this is your
board and you seem to like long lines?

[Doom9]

Yup there is a reason
why most forum software input area is static to 80 no matter how wide you
grow the window. They are trying to keep true to the 80 character rule
Which now has me wondering why you do not override it since this is your
board and you seem to like long lines?The day I figure out how to kill that darned small input box I will go for it BFG blazing.. but vbb doesn't make it easy.
But you're wasting my screen space and I feel very negative about it.. so negative that I'm already thinking about php functions that would get rid of excessive linewraps.

[Zep]

The day I figure out how to kill that darned small input box I will go for it BFG blazing.. but vbb doesn't make it easy.
But you're wasting my screen space and I feel very negative about it.. so negative that I'm already thinking about php functions that would get rid of excessive linewraps.
Ok i will write this one with no wasted space for you lol Anyway, a find and replace using the source vbb outputs for the input area was easy for me to override it on a friends board. It was awhile back and maybe they updated the source a lot since but still it should be not too bad to find the code that vbb gens for the input area. Of course on my board i left it at 80. What you feel is wasting space i feel is making reading easier. Just think if books or newspapers went across pages just because they could. Or used all the white space and never gave your eyes a chance to rest. Line breaks separate thoughts and ideas also and it is IMHO very important to format first and have plenty of white space versus using all the space just because you can. As for getting rid of line wraps that would hurt formatting of code and stuff that needs formatting way beyond the run on text i am typing now that normally i would have broken into multiple paragraphs but hey it is your board so go for it! :D

[NaN]

@Doom9:

Rule 3: Keep the focus! This forum is about video not about line wraps. Thank you.

Cheers, NaN

PS: hehehe sorry, I couldn't resist. :D

[Doom9]

@Nan: too bad that ultimately if a case gets up the latter, it lands on my plate ;) And you can always use the browser window size to force content to a size you like.. I don't even use the full screen width most of the time.

[NaN]

Hey, that was just a joke :D - I never thought you would take me serious!

I've really no problem that you big-ass display guys talk about resolutions that exceed SVGA spec and their problems.

Cheers, NaN

[Inventive Software]

Lucky so-and-so's. I have a 17" CRT, and that does the job just fine, though I so crave for one of Dell's massive widescreen beauties, but don't have the dosh to get one. :(

[len0x]

I wonder if anyone came close to trying the topic during last half a year? :)

[devaster]

Sorry for replying so late!

You're right! I would love to see Elecards sources...

last decoders fom elecard use HW decoding ... there is the speed