Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

View Poll Results: Do you MT?
Yes, but won't say how. 1 2.38%
Yes, with SetMT() 21 50.00%
Yes, but with SoraThread() 4 9.52%
Yes, but with ThreadRequest() 2 4.76%
Yes, but with LinearRequest() 0 0%
Yes, but with MP_Pipeline 5 11.90%
Yes, but with (other). (Please specify) 0 0%
No. Single life for me. 14 33.33%
Multiple Choice Poll. Voters: 42. You may not vote on this poll

Reply
 
Thread Tools Search this Thread Display Modes
Old 9th November 2013, 13:50   #1  |  Link
Sparktank
47.952fps@71.928Hz
 
Sparktank's Avatar
 
Join Date: Mar 2011
Posts: 940
RE: "MT" vs. "sorathread()" vs. "ThreadRequest()" vs. etc.

---------- Too long, didn't read? ----------
What's your poison? And why?

EDIT1:
Those of you that use MP_Pipeline, do you....

A) Use x86 only?
B) Use x64 only?
C) use BOTH x86 + x64?

---------- ----------

So, I've been browsing this site for a long time. Like a year before I created an account.

And since then, I began to see the nature of successful feedback and queries.

I see a lot of threads with a post or two that stipulate different conditions, in which, SetMT (be it Set's MT Avisynth, or other) or Sora's Multi-Process, Multi-Threaded pluging or even the "ThreadRequest" plugin that has no official author, yet, has its own thread, supposedly maintained, by a user of the plugin.

A lot of the "MultiThreaded" functionality seems highly experimental and not to be taken seriously as single-threaded plugins.

Yet, the hypothesis remains: What do (most) people find more effective for their needs?

I expect answer based on what they use and what they are comfortable with.

I would love feedback if they tried multiple variations (SetMT, Sorathread, ThreadRequest, etc) before settling on a single plugin for their multi-core needs.

While the notion that muti-core doesn't always equal "exact multiple cores", I am aware that many machines will treat dual-core with "HyperThreading" as though if it were 4 cores.

I am not experienced in this manner of... categorization.
Or even why this type of information would be deemed worthy to know.
I'm not a developer and not find this information useful on its most basic levels, but I am meant to understand that it does yield some level of interpretation not to be overlooked, even on the most "noobish" levels.


So, I would like to know how most people handle their...

MultiThreaded needs:

SetMT? (SEt's MT dynamic library?)
Sora's MT? (sorasmserver.exe/sorasmsource.dll/sorathread.dll)
ThreadRequest? (ThreadRequest.dll)
RequestLinear? (TIVTC.dll)
Other? (please, please, specify which works best for you)


I understand different Avisynth installations work differently for people with different needs.

For instance, I work solely on the principle what IanB updates Avisynth to 2.6 Alpha (X), and that SEt, updates Avisynth.dll to be MT compatible to 2.6 Alpha (X).

Whether or not the two are exactly the same version at the time of release is not always the same matter (in reference to any future updates where SEt is not available for immediate release of any update from IanB, so forth and so forth).

However, since it is known, IanB's AVS 2.6a4 is seen as up-to-date with SEt's MT version, I would be willing to take that into consideration.

That is, if anyone with feedback is willing to stipulate the nature of their environment for the most successful "Multithreaded" environment.

I've tested: SetMT, ThreadRequest, RequestLinear, and MP_Pipeline and all have, more or less, equally produced similar results.

Neither have yielded substantial results to merit the use of one over the other.

AviSynth version: 2.5.8
then
AviSynth version: IanB's 2.6.0a5
then
AviSynth version: SEt's 2.6 MT (2013.09.28)

----

I use mostly ThreadRequest() for my MT needs.
I find no discernible difference between TR and any other mode of Multithreading.

----
There's three machines I use (2 with Intel QuckSynch, 1 with Nvidia CUDA), and, no matter the settings, each produce, relatively, the same speed performance.

-----


I know for "ThreadRequest()", there is no official thread where the developer of the plugin participates (I think, Japanese developer), but I would like to know who still uses it and if anyone has done any updates to it (if at all relevant).

The same goes for any other method for MultiThreading.

------

What's your poison? And why?

EDIT1:
------ MP_Pipeline -----

I also understand with MP_Pipeline, you can distinguish between x86 and x64 for Avisynth usage.
I only ever use x86, as after reading so many concerns with x64, I do not find the simplest functions to be that useful to experiment with x64.

Those of you that use MP_Pipeline, do you....

A) Use x86 only?
B) Use x64 only?
C) use BOTH x86 + x64?

Last edited by Sparktank; 9th November 2013 at 14:49. Reason: EDIT1: MP_Pipeline, 86/64?
Sparktank is offline   Reply With Quote
Old 9th November 2013, 14:20   #2  |  Link
innocenat
Registered User
 
innocenat's Avatar
 
Join Date: Dec 2011
Posts: 77
Do note that avisynth and most of its plugins doesn't not currently use CUDA or QuickSync at all.

Also, I find multithreading doesn't really help much; it bottleneck on encoder anyway.
innocenat is offline   Reply With Quote
Old 9th November 2013, 14:23   #3  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
Originally Posted by innocenat View Post
Also, I find multithreading doesn't really help much; it bottleneck on encoder anyway.
it helps a lot when using slow filters like eedi3 tnlmeans

Last edited by feisty2; 9th November 2013 at 14:26.
feisty2 is offline   Reply With Quote
Old 9th November 2013, 14:36   #4  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
Quote:
Originally Posted by feisty2 View Post
it helps a lot when using slow filters like eedi3 tnlmeans
eedi3 is multithreaded internally so I have some serious doubts about more threading helping it "a lot".

As for me, I simply filter multiple sources in parallel if I need to process something fast. Any kind of multithreading implemented in avisynth right now is a hack. It requires you to use ugly syntax, it's not nearly as stable as singlethreaded, and its overhead is sometimes insane. Also running multiple processes allows avoiding the memory limit (MT_Pipeline also doesn't have this problem but god, that syntax).

That said, a lot of filters these days are threaded internally. eedi3, dfttest, dither package, mvtools - all run in multiple threads. It's not as efficient as frame-level threading, but it does improve performance and CPU utilization noticeably.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth

Last edited by TurboPascal7; 9th November 2013 at 14:41.
TurboPascal7 is offline   Reply With Quote
Old 9th November 2013, 16:07   #5  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
Originally Posted by TurboPascal7 View Post
eedi3 is multithreaded internally
my mistake,all I remembered is eedi2 isn't multithreaded originally (has moded mt version tho),so I thought eedi3 is singlethreaded too, with mt but eedi3 is still very slow,someone should really do some asm optimization for eedi3
feisty2 is offline   Reply With Quote
Old 9th November 2013, 20:04   #6  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,718
Quote:
Originally Posted by innocenat View Post
Also, I find multithreading doesn't really help much; it bottleneck on encoder anyway.
The improvement with SetMTMode() really depends entirely on the plugins you use. With MVTools2, which was written to "play nice" with SetMTMode(), the performance improvements can be sustantial, often 3x or even 4x faster. I documented some of my results in the various posts in the threads about improving amateur film transfers. Here's one of those:

SetMTMode Performance Improvements

I just now did a test, and using the script I posted in that thread, with no SetMTMode() statements, the script runs at 8.4 fps. If I use SetMTMode() and if I use all 8 threads in my computer, I get 40 fps. That's almost 5x faster!!
johnmeyer is offline   Reply With Quote
Old 10th November 2013, 10:52   #7  |  Link
innocenat
Registered User
 
innocenat's Avatar
 
Join Date: Dec 2011
Posts: 77
Most of you didn't read my statement correctly. I say it "bottleneck on encoder". Unless you are doing multi-pass filtering and encode to lossless, the avisynth result will be passed to encoder to be encoded. On all of my machines, the x264 with my settings took dominant in processing time anyway (unless I use some really slow filter like eedi3).
innocenat is offline   Reply With Quote
Old 10th November 2013, 11:17   #8  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
Originally Posted by innocenat View Post
Most of you didn't read my statement correctly. I say it "bottleneck on encoder". Unless you are doing multi-pass filtering and encode to lossless, the avisynth result will be passed to encoder to be encoded. On all of my machines, the x264 with my settings took dominant in processing time anyway (unless I use some really slow filter like eedi3).
I think this thread is talking about mt on slow avs filters,there're still some important filters which are already very slow originally
feisty2 is offline   Reply With Quote
Old 10th November 2013, 22:38   #9  |  Link
turbojet
Registered User
 
Join Date: May 2008
Posts: 1,840
It depends on which filters are being used. If using none or very fast filters which is common use for me, SoraThread() results in a 5-25% speed increase, setmtmode only slows it down. If it's slow heavy filters, sorathread() doesn't do much but setmtmode can gain >100% increase in certain situations.

I voted for sorathread because that's what I mainly use, ThreadRequest has the same sort of gains but deadlocked.
__________________
PC: FX-8320 GTS250 HTPC: G1610 GTX650
PotPlayer/MPC-BE LAVFilters MadVR-Bicubic75AR/Lanczos4AR/Lanczos4AR LumaSharpen -Strength0.9-Pattern3-Clamp0.1-OffsetBias2.0
turbojet is offline   Reply With Quote
Old 10th November 2013, 22:49   #10  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,718
Quote:
Originally Posted by innocenat View Post
Most of you didn't read my statement correctly. I say it "bottleneck on encoder". Unless you are doing multi-pass filtering and encode to lossless, the avisynth result will be passed to encoder to be encoded.
I read it correctly and answered correctly. Your encoder is external to AVISynth and therefore anything happening inside of AVISynth cannot and will not have any effect on an external process. If you simply pass video through AVISynth without doing much to it, then the encoder will obviously be the bottleneck, but if that is all you are doing, then you don't even need to be reading this particular thread.

However, for most people in this forum, the whole point of using AVISynth is to do some really advanced processing in AVISynth, and some of these scripts can only process a frame a second, or less. In those cases, MT makes a huge difference, as I documented in my previous post.
johnmeyer is offline   Reply With Quote
Old 11th November 2013, 06:00   #11  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Most of my AVIsynth work is fed into x264. The encoding settings I use don't need a very high throughput from AVIsynth in order to saturate my i7-4770k with x264. In these case I just single thread it. That said, when I have a lot of processing going on (MCTD) I use MP_Pipeline since it helps speed things up and doesn't have temporal issues. I generally use x64 plugins whenever possible if I'm feeding directly into x264 to avoid piping.

To further speed things up I then simultaneously process the source in 4 equal length segments to intermediate AVI files using VirtualDub and the Lagarith codec which gets the CPU usage pretty high. Here's a sample of 1/4th of my 4x split MCTD:
Code:
LoadPlugin("C:\HDTV Tools\MP_Pipeline\x86\MP_Pipeline.dll")
MP_Pipeline("""

### platform: win64
LoadPlugin("C:\HDTV Tools\DGDecNV\DGDecodeNV.dll")
DGSource("Video_2.dgi").trim(0,45928).crop(0,130,-0,-130)

### prefetch: 16, 16
### ###
### platform: win32
LoadPlugin("C:\HDTV Tools\MVtools\mvtools2.dll")
Import("C:\HDTV Tools\MCTD\MCTemporalDenoise.v1.4.20.avsi")
SetMemoryMax(1280)
MCTemporalDenoise(settings="medium")

### prefetch: 4, 0
### ###
### platform: win64
LoadPlugin("C:\HDTV Tools\flash3kyuu_deband\flash3kyuu_deband.dll")
flash3kyuu_deband(range=15, grainY=32, grainC=32, sample_mode=2, dither_algo=3, mt=false)

### ###
""")
I then combine the 4 AVI files in AVIsynth add the black bars back and then feed it into x264 for the two passes of encoding.

If I'm doing something in between I use MP_Pipeline and feed directly to x264 like this:
Code:
LoadPlugin("C:\HDTV Tools\MP_Pipeline\x64\MP_Pipeline.dll")

MP_Pipeline("""
### platform: win64
LoadPlugin("C:\HDTV Tools\DGDecNV\DGDecodeNV.dll")
DGSource("Video_2.dgi").crop(0,140,-0,-140)
AssumeFPS(24000,1001)

### ###
### platform: win32
LoadPlugin("C:\HDTV Tools\ResampleHQ\resamplehq-x86.dll")
ResampleHQ(1280,534, "YV12", "TV.709", "TV.709", false, 0, 0, -0, -0 ,"Spline36")

### ###
### platform: win64
LoadPlugin("C:\HDTV Tools\flash3kyuu_deband\flash3kyuu_deband.dll")
flash3kyuu_deband(range=15, grainY=32, grainC=32, sample_mode=2, dither_algo=3)

### ###
""")
I've played around with AVIsynth MT. It isn't terribly stable. It doesn't scale too well with the filters I've tried it with and it has temporal issues which makes it a poor choice when using it with something like MCTD.

Last edited by Stereodude; 11th November 2013 at 06:07.
Stereodude is offline   Reply With Quote
Old 11th November 2013, 23:24   #12  |  Link
Sparktank
47.952fps@71.928Hz
 
Sparktank's Avatar
 
Join Date: Mar 2011
Posts: 940
Very interesting reads, so far.

I'll definitely take some of this into consideration when I finally get around to some of my messy blu-rays.
Some need some serious denoising while others need to go from 1080i to 1080p.
I definitely want to do some heavy testing so this will come in handy a lot.

Some interesting test results, johnmeyer. Still going through a lot of it but haven't had much time lately.

I've been delaying the messy movies for a long time but I think it's time to tackle them.

Thanks for all the input, everyone. Still keeping track if anyone has more input or tips.
__________________
Win10 (x64) build 19041
NVIDIA GeForce GTX 1060 3GB (GP106) 3071MB/GDDR5 | (r435_95-4)
NTSC | DVD: R1 | BD: A
AMD Ryzen 5 2600 @3.4GHz (6c/12th, I'm on AVX2 now!)
Sparktank is offline   Reply With Quote
Reply

Tags
avisynth, efficiency, multithread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:26.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.