MT 0.7(+custom avisynth)a filter to run filters multithreaded. Usefull for SMP or HT

[jordisound]

what about new MVtools and MVdegrain3?
It works with MT? can anybody show me an script that works fine?

[KML]

Guys is it possible to use MT 0.7 with "PentiumD 3.00GHz"?

[Ranguvar]

@both above: Try'n'see.

[scharfis_brain]

I suggest not to use mvtools with mt.
the problem is the splitting of the frame.
a camera pan always will show lesser denoising and more artifacting at the borders where the frame has gotten split up for mt.

I hope some time we have a native mt-support within mvtools.dll

[KML]

Originally Posted by scharfis_brain

I suggest not to use mvtools with mt.
the problem is the splitting of the frame.
a camera pan always will show lesser denoising and more artifacting at the borders where the frame has gotten split up for mt.

I hope some time we have a native mt-support within mvtools.dll

Thanks..

I should find a way to make fast mv-tools

[scharfis_brain]

Is it possible to assign each filter to a selectable thread eg.:

avisource("blah.avi")
thread("blur(1)",1) # assign blur(1) to cpu 1
thread("temporalsoften(3,12,13)",2) to cpu 2

this should make it possible to use multithreading with filters that cannot be used with split up frames or temporal order of the frames (eg. motion compensated filters)

I altered the frame of the documentation a bit:

[thetoof]

This would be AWESOME! Great idea

[cweb]

I'm really liking this. Please let us know if there is any update, or the mainstream avisynth becomes multithreaded enough to replace this.

[bestsoft666]

Perhaps by year end most of us will be able to afford a dual core though. Nice work once again tsp.

[Konrad Klar]

Sorry if this question was answered earlier.

LoadPlugin("J:\Program Files\AviSynth 2.5\plugins\exinpaint.dll")
Logo=ImageSource("K:\afd\WRHH1logoTL.bmp"). ConvertToYV12(matrix="pc.601")

AviSource("test.avi")

SetMTMode(2,2)
ExInpaint(last,Logo,color=$FF8080, Radius=100)

SetMTMode(2,2) and (1,2), (3,2), (4,2) does not give any speed advantage with ExInpaint(). CPU usage is 48-51% on my C2D (in MPC, VirtualDub, and HC22.1).
Is there any method to make ExInpaint working with (and benefiting from) SetMTMode?

[Boulder]

Move SetMTMode so that it is the first line of your script.

[Konrad Klar]

Originally Posted by Boulder

Move SetMTMode so that it is the first line of your script.

Many thanks, Boulder! Now it is working as intended.

Have you any suggestion, how improve following script:

SetMTMode(2,2)

LoadPlugin("J:\Program Files\AviSynth 2.5\plugins\exinpaint.dll")
Logo1=ImageSource("K:\afd\WRHH1logoTR.bmp"). ConvertToYV12(matrix="pc.601")
Logo2=ImageSource("K:\afd\WRHH1logoTL.bmp"). ConvertToYV12(matrix="pc.601")

Video=AviSource("example.avi", audio=false)

A=Trim(Video,0,115)
Exinpaint(A,Logo1,color=$FF8080, Radius=100)
LanczosResize(720,540)
A=last

B=Trim(Video,116,0)
Exinpaint(B,Logo2,color=$FF8080, Radius=100)
LanczosResize(720,540)
B=last

AlignedSplice(A,B)


SetMTMode(5,2)
FFT3DGPU()

SetMTMode(2,2)
AddBorders(0,18,0,18)

?

Source video has two types of logo at begining and in middle, hence AlignedSplice(A,B).

This script is rendered by VirtualDub at 99% CPU usage, however HCEnc 22.1 only utilize 49-52% while encoding it.

[Zep]

I just purchased 4 gigs of ram just so I could increase my thread count but I have run into what appears to be a memory wall. I am encoding 1080p HDTV to x.264 and everything works ok with SetMTMode(5,5) and it get an increase in FPS and CPU from 40% to 60% compared to not using it but I have a quad and 5 threads is not enough but when I try to use more than 5 threads I get a

"Microsoft visual C++ library: Run time error. this application asked runtime to terminate in an unusual way"

I have 3.2 gigs really since XP 32 bit even so I should have more than enough but and there is just over 1.1 gig of real ram (of the 3.2) still unused when I run my avs with 5 threads. If I try 6 more like I said above I get that error. It appears MT or avisynth or VDub will not use more than 2 gigs of real ram which surprised the hell out me



Now if I add a SetMemoryMax(64) to my avs I can do a SetMTMode(5,10) even since the total ram used stays under 2 gigs but it is super slow as now each thread does not have enough ram to do its thing. So I know it is not a thread COUNT problem Per Se' but appears to be a ram usage issue.


Has anyone run into this problem before? is MT limited to using 2 gigs? Is avisynth limited? Any thoughts on a work around?

thanks

[Konrad Klar]

It is rather off topic, but anyway:
Normally single process (not thread) can allocate up to 2GB. This is Windows 32bit limitation.
If Windows is launched with switch /3G some programs can allocate up to 3GB.
Some programs that uses AWE (Address Windowing Extensions) can allocate more memory that it has virtual address space.
http://en.wikipedia.org/wiki/Address...ing_Extensions

[foxyshadis]

Originally Posted by Konrad Klar

This script is rendered by VirtualDub at 99% CPU usage, however HCEnc 22.1 only utilize 49-52% while encoding it.

HC uses custom avisynth routines, you have to add Distributor() to the end of the script. It's come up a couple of times in the thread already.

[thetoof]

@ Zep

Originally Posted by Didée

Win32 has a 2GB process limit ...

[Zep]

Originally Posted by Konrad Klar

It is rather off topic, but anyway:
Normally single process (not thread) can allocate up to 2GB. This is Windows 32bit limitation.
If Windows is launched with switch /3G some programs can allocate up to 3GB.
Some programs that uses AWE (Address Windowing Extensions) can allocate more memory that it has virtual address space.
http://en.wikipedia.org/wiki/Address...ing_Extensions

ouch 32 bit pointer is 4 gig so talk about a limiting apps grrrrrr guess I should have read up on XP before I trusted it lol

BTW - I feel it is on topic since not only is MT() the trigger but not using MT() would never come close to hitting that limit and besides knowing this limit is very important when using MT() now that HDTV and Bluray encodes with lots of cores are becoming common and we want more than than 40% CPU

thanks!

[Zep]

Originally Posted by thetoof

@ Zep

like I said OUCH!!! Yes I wish I read this whole thread lol


talk about a total bummer I go get 4 gigs just for this and find out XP (32) can only use 3.2 and now I find out apps can't even use that but are limited to 2 gigs. So now I guess I start searching about XP 64 and if avisynth can use it


thanks

[Zep]

ok found a work around and it appears to work fine so far in my 1 test run with setMTmode(5,8).

*************************************************
The /3GB switch allocates 3 GB of virtual address space to an application that uses IMAGE_FILE_LARGE_ADDRESS_AWARE in the process header. This switch allows applications to address 1 GB of additional virtual address space above 2 GB.
************************************************

just put that switch in your boot.ini like this

/FASTDETECT /3GB



There is another switch you can add also I see on the Microsoft site but I have not tried that yet.

read it all here

http://www.microsoft.com/whdc/system...AE/PAEdrv.mspx

and here

https://www.microsoft.com/whdc/syste...AE/PAEmem.mspx


thanks

[TSchniede]

I just tried for several days to get the best performance (while maintaining correct output) out of MVTools. I even tried to MT MVAnalyse with OpenMP. (It only got slower - I suppose the memory overhead and crating and closing Thread several times for each frame is way too high)

I can tell, that the MT method can never produce the same output , even for very large overlaps. That's because MVAnalyse gets its best predictors (the location where the search is started from) from the block surrounding it AND which have been fully computed on this hierarchy plane. So splitting will always miss data.

SetMTmode(2,0) as suggested both in this forum thread as in the documentation doesn't work.
-MT 0.7
-VirtualDub 1.7.8
-MVTools 1.9.2/3 (I'm testing MVDegrain3)
(And no, I am NOT overclocking any component of my C2Q9300, although it would run flawlessly at 3GHz - tested with prime95 for more than 12hours at default fan control, now I set it to better cooling)

I am still testing for safe settings or old versions which work to find the cause.

I can tell right now that real 100% CPU usage and many threads clearly increases errors (two clips encoded at the same time with SetMTmode(2,0) caused faulty frames (chroma green/purple frame from somewhere else with minor errors or "rainbow" with mostly garbage in the luma plane) about once every 100-2000 frames (strange - only the middle frame has visible changes but minor pixel errors appear around it, the faulty fames seem to be wrong before the sad is determined).
only one instance of Virtualdub which results in approximatively 50% CPU usage, produces errors up to 40000 frames apart.
The codec for the resulting file seems to be irrelevant (tried Virtualdubs uncompressed and Lagarith).

I am still verifying errors i got, so this is only a first analysis of the whole issue.