Run 32-bit DLLs in 64-bit AviSnyth [Archive]

Vitality

12th January 2019, 03:02

poisondeathray

12th January 2019, 03:28

You can run QTGMC , srestore natively in 64bit with grunt x64 . But I don't think srestore is safe to run MT

MP_Pipeline adds a lot of overhead , very slow .

StainlessS

12th January 2019, 10:33

But I don't think srestore is safe to run MT

Yep, tis my understanding that it requires linear access to stay in-sync.

Stereodude

12th January 2019, 14:55

I use this code:
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\MP_Pipeline-0.18\x64\MP_Pipeline.dll")
MP_Pipeline("""
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\Plugins\AVS\GRunT\GRunT.dll")
Import("J:\StaxRip-x64-2.0.0.0\Apps\Srestore.avsi")
QTGMC(Preset = "Medium", FPSDivisor=2)
Srestore(frate=23.976)
### prefetch: 16, 0
### ###
""")

But it says MP_Pipeline is unable to create slave process because GRunT.dll is 32-bit. I thought MP_Pipline allows 32-bit DLL in 64-bit AviSynth.
Well, you have to use it correctly. Try this:
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\MP_Pipeline-0.18\x64\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win32
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\Plugins\AVS\GRunT\GRunT.dll")
Import("J:\StaxRip-x64-2.0.0.0\Apps\Srestore.avsi")
QTGMC(Preset = "Medium", FPSDivisor=2)
Srestore(frate=23.976)
### prefetch: 16, 0
### ###
""")

MP_Pipeline adds a lot of overhead , very slow .
:confused: No idea what you're talking about. Correct use of it speeds things up by allowing you to run different CPU intensive things in different independent thread pools.

Vitality

12th January 2019, 16:23

Well, you have to use it correctly. Try this:
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\MP_Pipeline-0.18\x64\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win32
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\Plugins\AVS\GRunT\GRunT.dll")
Import("J:\StaxRip-x64-2.0.0.0\Apps\Srestore.avsi")
QTGMC(Preset = "Medium", FPSDivisor=2)
Srestore(frate=23.976)
### prefetch: 16, 0
### ###
""")

It says QTGMC is not a function, but when called outside MP it works.
https://pastebin.com/raw/yavC5T8a

StainlessS

12th January 2019, 16:34

Does your QTGMC use 64 bit plugs ?, does it work in only 32 bit mode (load script calling QTGMC only into Vdub 32 bit).

poisondeathray

12th January 2019, 16:37

:confused: No idea what you're talking about. Correct use of it speeds things up by allowing you to run different CPU intensive things in different independent thread pools.

It's faster to run natively at x64 entirely, than to use mp_pipeline to insert some x86 components . Lots of overhead with mp_pipeline . It's a last resort

Stereodude

12th January 2019, 16:37

Does your QTGMC use 64 bit plugs ?, does it work in only 32 bit mode (load QTGMC only, script into Vdub 32 bit).
FWIW, I found it quite challenging to get QTGMC to run under 64-bit AVIsynth many years ago. I've pretty much only use it under 32-bit ever since as a result. I guess things are different now.

Vitality

12th January 2019, 16:40

Does your QTGMC use 64 bit plugs ?, does it work in only 32 bit mode (load script calling QTGMC only into Vdub 32 bit).

I'll try 32-bit VB

poisondeathray

12th January 2019, 16:42

QTGMC does work under avs+ x64 MT , you need the updated version with updated pinterf plugins for mvtools , masktools

But - srestore is problematic to run in MT mode under any conditions . Many people get mixed up frames. Those that claim they don't probably didn't look closely

Vitality

12th January 2019, 16:44

https://i.imgur.com/SXnt8wN.png

Stereodude

12th January 2019, 16:45

It's faster to run natively at x64 entirely, than to use mp_pipeline to insert some x86 components . Lots of overhead with mp_pipeline
What overhead are you talking about? Not to mention we don't have x64 versions of every plugin.

LoadPlugin("C:\HDTV Tools\MP_Pipeline\x86\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win64
LoadPlugin("C:\HDTV Tools\DGDecNV\DGDecodeNV.dll")
DGSource("TIM_p_1.2_BD.dgi")
trim(0,4999)
### prefetch: 20, 10
### ###
""")
Results:
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 137.9 | 570.2 | 482.6
Memory usage (phys | virt): 26 | 13 MiB
Thread count: 20
CPU usage (average): 1%

Time (elapsed): 00:00:10.360
LoadPlugin("C:\HDTV Tools\MP_Pipeline\x64\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win64
LoadPlugin("C:\HDTV Tools\DGDecNV\DGDecodeNV.dll")
DGSource("TIM_p_1.2_BD.dgi")
trim(0,4999)
### prefetch: 20, 10
### ###
""")
Results:
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 189.0 | 589.9 | 484.8
Memory usage (phys | virt): 20 | 8 MiB
Thread count: 20
CPU usage (average): 1%

Time (elapsed): 00:00:10.314
LoadPlugin("C:\HDTV Tools\DGDecNV\DGDecodeNV.dll")
DGSource("TIM_p_1.2_BD.dgi")
trim(0,4999)
Results:
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 154.1 | 635.2 | 481.1
Memory usage (phys | virt): 210 | 556 MiB
Thread count: 27
CPU usage (average): 5%

Time (elapsed): 00:00:10.393

poisondeathray

12th January 2019, 16:50

What overhead are you talking about? Not to mention we don't have x64 versions of every plugin.

The overhead of inserting additional filters, not source filters.

That's entirely my point - we don't have x64 versions of everything, so you use it as a last resort. You don't use this for fun because it adds overhead and is slower

Repeat the test with some typical scripts that have both x64 and x86 filters .

QTGMC and Srestore are available in x64 natively. So there is no reason to run in x86 for some parts

Stereodude

12th January 2019, 16:57

That's entirely my point - we don't have x64 versions of everything, so you use it as a last resort. You don't use this for fun because it adds overhead and is slower
You keep asserting it adds overhead. Please define "overhead".

And what do you mean by slower? I've just easily demonstrated it's not slower to use MP_Pipeline than not using it for the same script. There's no throughput loss in MP_Pipeline or in it piping from x86 to x64 filter chains. Are you trying to say that x86 versions of plugins are slower than their x64 counterparts?

poisondeathray

12th January 2019, 16:59

You keep asserting it adds overhead. Please define "overhead".

And what do you mean by slower? I've just easily demonstrated it's not slower to use MP_Pipeline than not using it for the same script. Are you trying to say that x86 versions of plugins are slower than their x64 counterparts?

Yes x86 versions are typically slower - that's the main reason

@Vitality - which versions of QTGMC, Srestore do you have ?

Stereodude

12th January 2019, 17:06

Yes x86 versions are typically slower - that's the main reason
:rolleyes: Which has nothing to do with MP_Pipeline. So you're blaming the wrong thing and making factually incorrect and misleading statements about MP_Pipeline. The use of it doesn't slow down a script. Taking an existing 64-bit script and wrapping MP_Pipeline around it doesn't slow the script down.

In contrast, it can speed up some scripts (at the expense of using more total system memory). Like using QTGMC followed by MCTD. Separating the two of them with MP_Pipeline into their own sections is faster than running them together in a single conventional AVS script.

Vitality

12th January 2019, 17:08

Yes x86 versions are typically slower - that's the main reason

@Vitality - which versions of QTGMC, Srestore do you have ?

QTGMC 3.358
srestore v2.792

poisondeathray

12th January 2019, 17:32

QTGMC 3.358
srestore v2.792

QTGMC_3.361s
https://forum.doom9.org/showthread.php?p=1836519=510

Srestore2.792

:rolleyes: Which has nothing to do with MP_Pipeline. So you're blaming the wrong thing and making factually incorrect and misleading statements about MP_Pipeline. The use of it doesn't slow down a script. Taking an existing 64-bit script and wrapping MP_Pipeline around it doesn't slow the script down.

Yes it does .

Compare running 1) native x64, 2) native x86, both with some filters , you would expect (1) be faster, right ?

Ok. For fun, replace the intermediate filter with an x86 version using mp_pipeline (3) . You would expect it to be faster than (2), but slower than (1) . In reality, it's the slowest, slower than native x86 . The more filters, the larger the delta

script0.avs

colorbarshd(1920,1080)
trim(0,4999)
spline36resize(720,480)
spline36resize(1920,1080)

script1.avs

MP_Pipeline("""

### platform: win64
colorbarshd(1920,1080)
trim(0,4999)
### ###

### platform: win32
spline36resize(720,480)
### ###

### platform: win64
spline36resize(1920,1080)
### ###

""")

Try this
(1) avsmeter64 script0.avs
(2) avsmeter script0.avs
(3) avsmeter64 script1.avs => you can try different prefetch values , it will still be slower native x86

In contrast, it can speed up some scripts (at the expense of using more total system memory). Like using QTGMC followed by MCTD. Separating the two of them with MP_Pipeline into their own sections is faster than running them together in a single conventional AVS script.

This is true in some cases.

In other cases, running x64 natively is faster

(Also, the threading model is much better in vapoursynth . When you have multiple filters , it tends to bog down avisynth+ mt )

Vitality

12th January 2019, 17:33

:rolleyes: Which has nothing to do with MP_Pipeline. So you're blaming the wrong thing and making factually incorrect and misleading statements about MP_Pipeline. The use of it doesn't slow down a script. Taking an existing 64-bit script and wrapping MP_Pipeline around it doesn't slow the script down.

In contrast, it can speed up some scripts (at the expense of using more total system memory). Like using QTGMC followed by MCTD. Separating the two of them with MP_Pipeline into their own sections is faster than running them together in a single conventional AVS script.
What's the point of using MP_Pipeline to run 1 x86 plugin, when you can't run the other x64 plugins? I'm trying to run QTGMC but that's x64 but SRestore is x86.

Stereodude

12th January 2019, 17:42

QTGMC_3.361s
[url]Yes it does .

Compare running 1) native x64, 2) native x86, both with some filters , you would expect (1) be faster, right ?

Ok. For fun, replace the intermediate filter with an x86 version using mp_pipeline (3) . You would expect it to be faster than (2), but slower than (1) . In reality, it's the slowest, slower than native x86 . Actually, the more filters, the slower in relation.

script0.avs

colorbarshd(1920,1080)
trim(0,4999)
spline36resize(720,480)
spline36resize(1920,1080)

script1.avs

MP_Pipeline("""

### platform: win64
colorbarshd(1920,1080)
trim(0,4999)
### ###

### platform: win32
spline36resize(720,480)
### ###

### platform: win64
spline36resize(1920,1080)
### ###

""")

Try this
avsmeter64 script0.avs
avsmeter script0.avs
avsmeter64 script1.avs
You're the one making stuff up, you provide the results.

Stereodude

12th January 2019, 17:45

What's the point of using MP_Pipeline to run 1 x86 plugin, when you can't run the other x64 plugins? I'm trying to run QTGMC but that's x64 but SRestore is x86.
Try this:
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\MP_Pipeline-0.18\x64\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win64
QTGMC(Preset = "Medium", FPSDivisor=2)
### prefetch: 16, 8
### ###
### platform: win32
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\Plugins\AVS\GRunT\GRunT.dll")
Import("J:\StaxRip-x64-2.0.0.0\Apps\Srestore.avsi")
Srestore(frate=23.976)
### prefetch: 16, 8
### ###
""")
Edit: Are you sure that FPSDivisor=2 should be there in QTGMC? I think you can't have that there if you want Srestore to work correctly.

poisondeathray

12th January 2019, 17:56

You're the one making stuff up, you provide the results.

(1) avsmeter64 script0.avs => 110fps
(2) avsmeter script0.avs => 77fps
(3) avsmeter64 script1.avs => 67fps

So mp_pipeline is slower than native x86 . The delta increases, the more processes or filters you use

In a linear filter chain, you cannot run a later process simultaneously, before you get the input from a prior dependent process

poisondeathray

12th January 2019, 17:59

What's the point of using MP_Pipeline to run 1 x86 plugin, when you can't run the other x64 plugins? I'm trying to run QTGMC but that's x64 but SRestore is x86.

All x64

Srestore can run x64

Grunt x64
https://forum.doom9.org/attachment.php?attachmentid=15400&d=1463511496

I don't recommend running srestore in MT mode

Vitality

12th January 2019, 18:04

You're the one making stuff up, you provide the results.

Do you know where I can find x86 QTGMC?

poisondeathray

12th January 2019, 18:07

Do you know where I can find x86 QTGMC?

Could you not get full x64 running ? It's definitely faster

The original x86 QTGMC is here . But you'd need to use the matching older plugins and requisites

https://forum.doom9.org/showthread.php?t=156028

Vitality

12th January 2019, 18:11

Could you not get full x64 running ? It's definitely faster

The original x86 QTGMC is here . But you'd need to use the matching older plugins and requisites

https://forum.doom9.org/showthread.php?t=156028

I need SRestore, which requires a x86 DLL.

I tried running SRestore without QTGMC:
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\MP_Pipeline-0.18\x64\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win32
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\Plugins\AVS\GRunT\GRunT.dll")
Import("J:\StaxRip-x64-2.0.0.0\Apps\Srestore.avsi")
SRestore(frate=23.976)
### prefetch: 16, 0
### ###
""")

https://i.imgur.com/BkbYLRF.png

poisondeathray

12th January 2019, 18:13

I need SRestore, which requires a x86 DLL.

It does not .

Srestore can run x64 . There is a grunt x64
https://forum.doom9.org/attachment.php?attachmentid=15400&d=1463511496

LoadPlugin("PATH\grunt-x64.dll")

Vitality

12th January 2019, 18:39

Try this:
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\MP_Pipeline-0.18\x64\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win64
QTGMC(Preset = "Medium", FPSDivisor=2)
### prefetch: 16, 8
### ###
### platform: win32
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\Plugins\AVS\GRunT\GRunT.dll")
Import("J:\StaxRip-x64-2.0.0.0\Apps\Srestore.avsi")
Srestore(frate=23.976)
### prefetch: 16, 8
### ###
""")
Edit: Are you sure that FPSDivisor=2 should be there in QTGMC? I think you can't have that there if you want Srestore to work correctly.
I'm dealing with a telecined source (23.976 to 29.970), Wouldn't I need to deinterlace to 29.97002997p to get 23.976023976p instead of deinterlacing to 59.9400599401p to get 47.952047952p?

poisondeathray

12th January 2019, 18:44

I'm dealing with a telecined source (23.976 to 29.970), Wouldn't I need to deinterlace to 29.97002997p to get 23.976023976p instead of deinterlacing to 59.9400599401p to get 47.952047952p?

For field blended NTSC source, you would typically need 60000/1001 , then srestore back to 24000/1001 (ie. do not use the fpsdivisor)

The reason is, there are 59.94 fields per second. Some of them have unique content . Srestore needs access to all of them in order to choose correctly . But throwing away half of them before srestore, you will get the worse results . SO when you bob deinterlace, essentially those fields become frames . 59.94 frames per second . Then srestore essentially "chooses" based on metrics which are less blended to keep

Stereodude

12th January 2019, 18:46

I'm dealing with a telecined source (23.976 to 29.970), Wouldn't I need to deinterlace to 29.97002997p to get 23.976023976p instead of deinterlacing to 59.9400599401p to get 47.952047952p?
1) If you have a telecined (24/1.001p) source that was turned into "i60" you should use TIVTC.

2) If you really want to use Srestore it needs the 60/1.001p to spit out 24/1.001p correctly.

Vitality

12th January 2019, 19:44

1) If you have a telecined (24/1.001p) source that was turned into "i60" you should use TIVTC.

2) If you really want to use Srestore it needs the 60/1.001p to spit out 24/1.001p correctly.

Using your code:
Import("J:\StaxRip-x64-2.0.0.0\Apps\Plugins\avs\QTGMC\QTGMC.avsi")
MPEG2Source("VTS_03_1.d2v")
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\MP_Pipeline-0.18\x64\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win64
QTGMC(Preset = "Medium")
### prefetch: 16, 8
### ###
### platform: win32
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\Plugins\AVS\GRunT\GRunT.dll")
Import("J:\StaxRip-x64-2.0.0.0\Apps\Srestore.avsi")
Srestore(frate=23.976)
### prefetch: 16, 8
### ###
""")
https://i.imgur.com/EvQHVnA.png
https://i.imgur.com/BhMoweC.png

Stereodude

13th January 2019, 01:07

It seems like you don't know how to use AVIsynth very well. It should be something like this:
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\MP_Pipeline-0.18\x64\MP_Pipeline.dll")
MP_Pipeline("""
### platform: win64
Import("J:\StaxRip-x64-2.0.0.0\Apps\Plugins\avs\QTGMC\QTGMC.avsi")
MPEG2Source("VTS_03_1.d2v")
QTGMC(Preset = "Medium")
### prefetch: 16, 8
### ###
### platform: win32
LoadPlugin("J:\StaxRip-x64-2.0.0.0\Apps\Plugins\AVS\GRunT\GRunT.dll")
Import("J:\StaxRip-x64-2.0.0.0\Apps\Srestore.avsi")
Srestore(frate=23.976)
### prefetch: 16, 8
### ###
""")

Stereodude

13th January 2019, 01:22

(1) avsmeter64 script0.avs => 110fps
(2) avsmeter script0.avs => 77fps
(3) avsmeter64 script1.avs => 67fps

So mp_pipeline is slower than native x86 . The delta increases, the more processes or filters you use

In a linear filter chain, you cannot run a later process simultaneously, before you get the input from a prior dependent process
Except by making 1 tweak to your script1 to use MP_Pipeline properly I get these results:

(1) avsmeter64 script0.avs => 72.87fps
(2) avsmeter script0.avs => 71.93fps
(3) avsmeter64 script1.avs => 100.2fps
(4) avsmeter script1.avs => 103.5fps

MP_Pipeline("""
### platform: win64
colorbarshd(1920,1080)
trim(0,4999)
### prefetch: 16, 8
### ###

### platform: win32
spline36resize(720,480)
### prefetch: 16, 8
### ###

### platform: win64
spline36resize(1920,1080)
### prefetch: 16, 8
### ###
""")

I can make a further tweak and get these results:
avsmeter64 script1.avs => 269.3fps
avsmeter script1.avs => 263.8fps

MP_Pipeline("""
### platform: win64
colorbarshd(1920,1080)
trim(0,4999)
### prefetch: 16, 8
### branch: 4, 3
### ###

### platform: win32
spline36resize(720,480)
### prefetch: 16, 8
### branch: 4, 3
### ###

### platform: win64
spline36resize(1920,1080)
### prefetch: 16, 8
### branch: 4, 3
### ###
""")
But yeah, clearly MP_Pipeline is a slow, overhead heavy, waste of time... :rolleyes:

poisondeathray

13th January 2019, 03:07

But yeah, clearly MP_Pipeline is a slow, overhead heavy, waste of time... :rolleyes:

It is slower, apples to apples . Those were single threaded results . Clearly slower

A proper comparison would tweak the base script . Prefetch(whatever) for MT . I used prefetch(8)

1) avsmeter64 script0mt.avs => 438fps
2) avsmeter script0mt.avs => 390fps
3a) avsmeter64 stereodude0.avs => 125fps
3b) avsmeter64 stereodude1.avs => 180fps

Stereodude

13th January 2019, 05:06

It is slower, apples to apples . Those were single threaded results . Clearly slower

A proper comparison would tweak the base script . Prefetch(whatever) for MT . I used prefetch(8)

1) avsmeter64 script0mt.avs => 438fps
2) avsmeter script0mt.avs => 390fps
3a) avsmeter64 stereodude0.avs => 125fps
3b) avsmeter64 stereodude1.avs => 180fps
It's not slower (without MT). It's faster. Even by your own results once you stopped trying to artificially choke it to make it look bad. stereodude0.avs is still single threaded. 125fps is more than 110, 77, or 67. You claimed it was slower than x86 (or x64), and it's clearly not.

Now you're trying to move the goal posts...

poisondeathray

13th January 2019, 05:13

It's not slower (without MT). It's faster. Even by your own results once you stopped trying to artificially choke it to make it look bad. 125fps is more than 110, 77, or 67. You claimed it was slower than x86 (or x64), and it's clearly not.

Now you're trying to move the goal posts...

It is slower. stereodude0.avs , stereodude1.avs are your multithreaded scripts .

You're comparing single threaded vs. MT ? It's not very fair

You're allowed to tweak mp_pipeline, but not the base case??

By your "logic" - Why would you artificially "choke" the base case but not MP_Pipeline ?

It's clearly slower in every "apples to apples" case . Even the "best case" . 438fps > 180fps !!

In scripts that you can run parallel , or if you have a memory bottleneck because of an x86 host process sure, those are situations where it can be faster .

Stereodude

13th January 2019, 05:18

You're comparing single threaded vs. MT ? It's not very fair

You're allowed to tweak mp_pipeline, but not the base case??

By your "logic" - Why would you "choke" the base case but not MP_Pipeline ?

It's clearly slower 438fps > 180fps

What are you smoking dude ? Seriously ?
Why can't you just admit that you're wrong about MP_Pipeline? You don't even understand how MP_Pipeline works. I am comparing "single threaded". stereodude0.avs is still "single threaded". 125FPS is greater 110, 77, or 67.

Stereodude

13th January 2019, 05:20

It is slower. stereodude0.avs , stereodude1.avs are your multithreaded scripts .
stereodude0.avs is not multithreaded. Why don't you go learn how MP_Pipeline works and then get back to the rest of us instead of spreading your misinformation all over the forum?

poisondeathray

13th January 2019, 05:39

stereodude0.avs is not multithreaded. Why don't you go learn how MP_Pipeline works and then get back to the rest of us instead of spreading your misinformation all over the forum?

It is a type of multithreading; prefetch: x,y means there are "x" frames in the process, with "y" frames in the cache . stereodude1.avs with the added branch means you're running parallel processes .

Or how about you properly "multithread" it so it's faster than the base multithreaded script ? Since apparently I don' t know how to use it . How fast can you make it go , Scotty? Since there is apparently no overhead according to you, it should be faster than the x86 MT script . If you do that, I'll admit I'm wrong.

Stereodude

13th January 2019, 12:35

It is a type of multithreading; prefetch: x,y means there are "x" frames in the process, with "y" frames in the cache . stereodude1.avs with the added branch means you're running parallel processes .

Or how about you properly "multithread" it so it's faster than the base multithreaded script ? Since apparently I don' t know how to use it . How fast can you make it go , Scotty? Since there is apparently no overhead according to you, it should be faster than the x86 MT script . If you do that, I'll admit I'm wrong.
Using prefetch is not multithreading. It's not a form of multithreading as you claim. It's a cache / buffer between the different processes. You just can't bring yourself to be honest about MP_Pipeline. You creatively edit the explanation in the documentation to try to support your false claims.
### prefetch: 5, 2
# Prefetch max. 5 frames in this process, 2 frames behind currently requested frame will be kept to allow backward seeking
# All cache frames will be purged if we seek more than 2 frames backward
and...
### prefetch: <max frames in cache per clip>, <frames to keep behind current frame>

Enable prefetching. Please see example above for details.

Note: Don't enable prefetching in processes that require random access (such as processes that are upstream of branched process)
unless you know what you are doing, otherwise performance may be significantly degraded.
stereodude0.avs is not anymore multithreaded than the base MP_Pipeline script you posted. And, it's faster than not using it with the baseline x64 or x86 case! Your data shows it. My data shows it.

Groucho2004

13th January 2019, 13:24

Multi-processing, which MP_pipeline employs, does have more overhead than (properly implemented) multi-threading. The author of MP_pipeline himself points this out in his thread:
https://forum.doom9.org/showthread.php?p=1543488#post1543488

poisondeathray

13th January 2019, 15:20

@stereodude - you're right about how mp_pipeline's prefetch works. I was wrong about the way the cache works . I had assumed incorrectly that it was using additional threads for the cache. Looking more closely at each process with a process viewer, it's the same as without prefetch. So I'm big enough to admit it and will eat crow

You mentioned "moving the goal lines", but any game has two halves. You need to look at everything.

So any way to get it working faster with MT ? I could be wrong about this too, but there seems to be significant overhead here . I tried many combinations , settings , even a long time ago with set's avisynth 2.6 MT . With avs+ Mt there is a "SetFilterMtMode("MP_Pipeline",MT_SPECIAL_MT) " , and tried different combos of using regular prefetch .

real.finder

13th January 2019, 18:06

So any way to get it working faster with MT ? I could be wrong about this too, but there seems to be significant overhead here . I tried many combinations , settings , even a long time ago with set's avisynth 2.6 MT . With avs+ Mt there is a "SetFilterMtMode("MP_Pipeline",MT_SPECIAL_MT) " , and tried different combos of using regular prefetch .

in avs+ you can do mt inside mpp (MP_Pipeline) block only see here (https://forum.doom9.org/showthread.php?p=1789631&highlight=MP_pipeline#post1789631), if you do it outside it will not work and even if it work it will be slower than anything

but for old avs mt you need my special edit of mpp see here (http://forum.doom9.org/showthread.php?p=1746479#post1746479), and same as avs+ mt only inside block

poisondeathray

13th January 2019, 19:48

in avs+ you can do mt inside mpp (MP_Pipeline) block only see here (https://forum.doom9.org/showthread.php?p=1789631&highlight=MP_pipeline#post1789631), if you do it outside it will not work and even if it work it will be slower than anything

Thanks , I tried this already, and a few other combinations

avisynth+ MT's "Prefetch(some value)" outside works with the special MP_Pipeline mode . But the speed is about the same (<1/2 of "normal" x86 MT fps) when used with MP_Pipelines' prefetch+cache . Prefetch(some value) only outside is much slower. But I only played with a few values so far. There might be some combination that makes it work better

There still seems to be significant "overhead"; or I don't know what else to call it. The author uses the term "overhead", and also in the changelog . Maybe it's thread thrashing or cache thrashing or something related .

Stereodude

13th January 2019, 20:19

The sorts of processing I'm doing doesn't work in MT mode (QTGMC and/or MCTD) and MP_Pipleline's "thunking" doesn't help either. I generally use intermediate lossless AVI files and split the source into 4 equal length AVI files and generate the 4 files at the same time (with VD2) to get an effective throughput increase. I combine the files + do any final processing in an AVIsynth+ script before feeding them to the encoder. Usually I make several encodes to settle in on the right encoding settings so the intermediate files are an overall considerable processing time savings.

real.finder

13th January 2019, 23:24