Avisynth 2.6 MT

[Boulder]

If the filter handles non-sequential requests correctly – it'll work exactly as in non-MT mode, but if it doesn't – things will break (note that they will also break in non-MT mode when some filter reorders frames like SelectEven or Trim).I think in this case you could use tsp's RequestLinear before the filter, am I right?

[SEt]

In theory – yes (but after the filer, not before), in practice – RequestLinear doesn't work here.

[BugiBugBug]

Thank you both for the information. I think the source of the misunderstanding lays in the filter sequence. I've looked at the Wiki once more, and in combination with your explanation, the problem is that an AVS script is written sequentially from source to finished output, while the actual way of functioning is the other way around. This is a different perspective, and requires a change of thinking.

Hence if I've got two frames and two threads, these are requested at the end of the filter chain, and thus also split first at the end of the filter chain. Within each thread, these frames work just as they would if there was one thread, as the filter works on one frame at the time, and can simply request from the cache(MT) above it any frames it needs to work on that particular frame.

The effectiveness of the filter is thus not affected if the filter can retrieve the frame sequence it requires. The only way MT could thus negatively influence the (quality of the) execution of a temporal filter is if (1) the sequence does not provide all the necessary frames, but more likely: (2) the frames that the filter retrieves from the preceding cache are not in the right order or are not sequential/missing some frames.

Problems would only appear if the cache preceding the temporal filter does not output frames their right sequence (meaning it just gives out an alternating sequence of frames, e.g. 1,3,5,7 to thread 1 and 2,4,6,8 to thread 2 - or even more chaotic combinations).
My guess is this is what primarily constitutes the differences between the SetMTmodes.

A truly spatial filter such as a resizer should, under that logic, function regardless of the mode because it does not matter which frame it gets. Important is that the sequence is kept in tact in the cache that follows the filter execution.

Once again, thank you for clearing this up for me. I hope this will be useful for anyone else not familiar with the subject.

On a side note, as already expected, the increase in speed on my machine (a 5300 Merom) was very small (about 0,1 fps on a typical 11-13fps using AVCHD device preset with -preset-medium). Naturally this is because the AVS script is very simple.
But I did use this script to fprofile my own x264 builds, because it is the predominant way I'll be encoding.

[SEt]

If "sequence does not provide all the necessary frames" it'll likely would just throw an error, not invent missing data.
All frames are explicitly numbered and requested by number: if you asked for frame n – you can't get some other frame.

[ajp_anton]

From RGB24, ConvertToY8 results in a slight shift to the right. Going first to YV12 and then to Y8 works.

[DVDBob]

How should i configure the mt settings for a AMD Phenom II X6 1055T???

So i can use all 6 cores.

[pbristow]

How should i configure the mt settings for a AMD Phenom II X6 1055T???

So i can use all 6 cores.

AH, here it is folks: The miracle moment. The moment when the baby stares at the five funny stick things poking out of his hand and asks "how do I use these?".

The bad news, DVDBob, is that there's no simple answer to that question. But the good news is that you will be learning lots of new answers to it every day, for the rest of your life. ;)

For a basic "get you started" template script, look at the top post in the following thread:
http://forum.doom9.org/showthread.php?p=1312666#post1312666

For further info, see this page, starting about halfway down (under "ed avisynth MT 2.5.7.5 ")
http://avisynth.org/mediawiki/MT

Have fun! :)

[Rat Killer]

Need a hand. MeGUI seems to have done some updates including DGindexNV and now when attempting to index a file DGIndexNV.exe gives the following error:

The procedure entry point cuStreamDestroy_v2 could not be located in the dynamic link library nvcuda.dll

I have an Nvidia NVS 4200M running on a Dell laptop under Windows XP if it matters. Nvidia driver version 268.83 which is the latest from Dell.

Thanks for any suggestions on how to get this working again.

[golagoda]

Need a hand. MeGUI seems to have done some updates including DGindexNV and now when attempting to index a file DGIndexNV.exe gives the following error:

The procedure entry point cuStreamDestroy_v2 could not be located in the dynamic link library nvcuda.dll

I have an Nvidia NVS 4200M running on a Dell laptop under Windows XP if it matters. Nvidia driver version 268.83 which is the latest from Dell.

Thanks for any suggestions on how to get this working again.
Uninstall that driver and try this one: http://www.nvidia.com/object/quadro-notebook-win7-winvista-64bit-296.70-whql-driver.html (used the nvidia drivers search)

[Rat Killer]

Thanks!

Updating to the version 295 driver fixed it up.

[Atak_Snajpera]

Avisynth 2.6 MT installer (2012.05.16) -> http://www.mediafire.com/?prfoy9wfj4gtnnw
Installer source files -> http://www.mediafire.com/?308aa5d3r4nszwz

[SubJunk]

Thanks!

[defalopii]

I have tried this Avisynth 2.6 MT
Replacing the old avisynth.dll with Avisynth 2.6 MT
But the result make me disappointed

Please see the screenshot

http://76.imagebam.com/download/MHh5ENPPOFkDGdaS6YkNdA/19534/195339596/MT1.JPG

http://60.imagebam.com/download/w9587gmmliRRCKuyzTF3xQ/19534/195339601/MT2.JPG

* My system is Dual AMD Opteron 6272 with Total 32 Cores (My campus server, used for math calculations)
* RAM 16GB
* I am using MeGUI 2112
* x264 v2164

Avisynth Script
SetMTMode(3)
AVISource("C:\Documents and Settings\Administrator\My Documents\Downloads\Video\Sample.548673.avi", audio=false).AssumeFPS(24000,1001)
SetMTMode(2)
#deinterlace
#crop
LanczosResize(608,320) # Lanczos (Sharp)
LoadPlugin("C:\Program Files\MeGUI\tools\avisynth_plugin\Convolution3DYV12.dll")
Convolution3D("movielq") # Heavy Noise

LoadPlugin("C:\Program Files\MeGUI\tools\avisynth_plugin\VSFilter.dll")
TextSub("C:\Documents and Settings\Administrator\My Documents\Downloads\Documents\Watermark\watermark.srt", 1)

x264 profile

program --preset fast --pass 2 --bitrate 5000 --stats ".stats" --deblock -1:-1 --b-adapt 2 --ref 3 --weightp 2 --qpmin 10 --qpmax 51 --chroma-qp-offset -2 --rc-lookahead 60 --merange 32 --me umh --direct auto --subme 9 --trellis 2 --psy-rd 0.00:0 --no-fast-pskip --output "output" "input"

My Home PC with i7 2600 (Only 4 cores) even 6x more faster encoding, same encoding profile, same Avisynth script, and same video source but with No Avisynth 2.6 MT

So guys, can you explain to me what is wrong ??

[SEt]

With so many cores you sure hitting 32-bit address space limits with default number of threads. Try manually specifying different number of used threads in first SetMTMode call. Next, pipe output of Avisynth to x264 (you can also use 64 bit x264 this way) to free some more address space.
And if some other Avisynth MT version worked better at your setup - do post in case we have regression here.

[Atak_Snajpera]

also don't forget to update x264 to r2200. btw why do use only fast profile on such beast????

[defalopii]

With so many cores you sure hitting 32-bit address space limits with default number of threads. Try manually specifying different number of used threads in first SetMTMode call. Next, pipe output of Avisynth to x264 (you can also use 64 bit x264 this way) to free some more address space.
And if some other Avisynth MT version worked better at your setup - do post in case we have regression here.

Sory i dont understand, i am very new with this :confused:
Please give me the complete sample script, what should i do with SetMTMode(3) ?? change it to 5 or anything else ??
and why there is two SetMTMode, before and after the video source ??

also don't forget to update x264 to r2200. btw why do use only fast profile on such beast????

will try to use that x264 r2200
yeah its modified fast profile. i am not sure with this but the result is better than slow profile

[Atak_Snajpera]

especially with disabled psy visual optimalization. Take my advise and stick to presets created by developers. your settings are not good.

[defalopii]

Because i can not make Avisynth 2.6 MT works, so tried to split the video into 8 parts and than encode it simultaneously.

What i used:
* Dual AMD Opteron 6272 (Total 32 Cores)
* Video Source 1080p (1920x1040)
* MeGUI 2112
* x264 r2200
* Avisynth v2.5.8 rev. 2

On 8 simultaneous encoding the 1st pass only take 50% of CPU Usage, and the 2nd pass take the makximum 100% of CPU usage

Here is the screenshot
http://70.imagebam.com/download/PBKayju-TosxhqUIxEZZAQ/19546/195454179/beast.JPG

Used very simple Avisynth Script.
DirectShowSource("C:\Documents and Settings\Administrator\My Documents\Downloads\Video\Sample.548612-defa.mkv", fps=23.976, audio=false, convertfps=true).AssumeFPS(24000,1001)
#deinterlace
#crop
#resize
#denoise

LoadPlugin("C:\Program Files\MeGUI\tools\avisynth_plugin\VSFilter.dll")
TextSub("C:\Documents and Settings\Administrator\My Documents\Downloads\Documents\Watermark\watermark.srt", 1)

I used this profile. I dont know what is the best x264 Profile, please tell me..
program --preset fast --pass 2 --bitrate 5000 --stats ".stats" --threads 16 --deblock -1:-1 --b-adapt 2 --ref 3 --weightp 2 --qpmin 10 --qpmax 51 --chroma-qp-offset -2 --rc-lookahead 60 --merange 32 --me umh --direct auto --subme 9 --trellis 2 --psy-rd 0.00:0 --no-fast-pskip --output "output" "input"

So..what do You think about this ??
I hope i can make Avisynth 2.6 MT works to optimizing the all 32 cores, but i dont know how, will keep trying. Any suggestion please tell me very detail and clearly bcoz i am very new with this

[SubJunk]

Just change the first line in the script from:

SetMTMode(3)

to:

SetMTMode(3, 32)

It's normal for SetMTMode to only use 50% of CPU on some hardware unless you specify the number of cores.

[defalopii]

Just change the first line in the script from:

SetMTMode(3)

to:

SetMTMode(3, 32)

It's normal for SetMTMode to only use 50% of CPU on some hardware unless you specify the number of cores.

Still not work, there is no different between using MT or not

[SubJunk]

That's a weird problem to have. Hopefully someone can help you.

[Boulder]

Does MeGUI need the Distributor() call?

[SubJunk]

Does MeGUI need the Distributor() call?I use MeGUI a lot and have never needed Distributor

[defalopii]

For masters who want to experiment with this monster regarding to the MeGUI, Avisynth and x264 optimizing on multicore system You can PM me, i will give You access to this system

[Gavino]

With so many cores you sure hitting 32-bit address space limits with default number of threads. Try manually specifying different number of used threads in first SetMTMode call.
Just change the first line in the script from:
SetMTMode(3)
to:
SetMTMode(3, 32)
What's the point of that? - it's just the same.
Since he has 32 cores, the default is to use 32 threads anyway.

SEt's advice was to use fewer threads, eg
SetMTMode(3, 8)

[SEt]

Yes, I was suggesting using smaller numbers like SetMTMode(3, 8).
Also, CPU usage is only a hint if you are doing something completely wrong or not – only speed matters in the end and best speed in certain situations doesn't come with 100% CPU usage.

[defalopii]

What's the point of that? - it's just the same.
Since he has 32 cores, the default is to use 32 threads anyway.

SEt's advice was to use fewer threads, eg
SetMTMode(3, 8)

The MT doesnt work. I've tried many configuration, from 1 to hundred value but still no different between using MT or not

The best way for me is split the video source using mkvmerge GUI into 8-20 parts than encode it simultaneously. This is awesome, i can encode a video 1080p (1920x1080) with total duration 1h 57mn only take not more than 2 hours using above x264 profile. And the CPU usage can reach from 60% to 100% on the 1st pass, more part means more CPU usage can be maximized

But splitting and re-merging that so many pieces of course annoying. Thats why i still want the developer of MeGUI, Avisynth 2.6 MT or x264 can make it more easy-without manually split the video source into many parts-to maximize the multi core system

[Keiyakusha]

defalopii
Do note that having that much threads can degrade quality of your video. Because it is achieved by starting encoding of the next frame while previous is not yet finished (or something like that) or by using slices, but this also bad if you have so many of them. The difference is usually unnoticeable, but with this much threads you probably want to check this.
UPD: also, looking on your spitted files 60% load i recall that 1st pass never was very thread-efficient by itself so u should make sure that load during 2nd pass is maximized. Not sure if something about it was changed lately, i don't use 2-pass encodes for years now, only crf.

[Boulder]

Does the amount of utilized threads by the encoder executable change in Task Manager when you change the number of threads?

[SubJunk]

Since he has 32 cores, the default is to use 32 threads anyway.
Often the MTMode functions only use half the available threads, especially on Intels, so it's common for it to use 16 threads on a 32-thread CPU.
That's not this guy's problem though, I'm just replying to the question :)

[defalopii]

defalopii
Do note that having that much threads can degrade quality of your video. Because it is achieved by starting encoding of the next frame while previous is not yet finished (or something like that) or by using slices, but this also bad if you have so many of them. The difference is usually unnoticeable, but with this much threads you probably want to check this.

There is no quality degrade for the video result if You split it first into many parts than encde it simultaneously. I dont know if using MT, it probably degrade the quality.

Does the amount of utilized threads by the encoder executable change in Task Manager when you change the number of threads?

Yes its increase along with MT value. But if too much it will error "Process exits with error code: -1073741819 "

[pbristow]

Should this discussion be moved to the usage forum, perhaps? It doesn't seem to be a 2.6 MT development issue, more a general question of how to use Avisynth MT effectively.

[Keiyakusha]

There is no quality degrade for the video result if You split it first into many parts than encde it simultaneously. I dont know if using MT, it probably degrade the quality.
Sure, this is exactly what i mean. I tried to say that your way of splitting files to 8 or more parts may be better idea after all.


Does anyone knows if it possible that with this much threads x264 just reaches some point when previous frame doesn't have enough information to start encoding new frame so new thread is not created? Then maybe higher resolution video will use more threads... Makes any sense?

[Boulder]

Yes its increase along with MT value. But if too much it will error "Process exits with error code: -1073741819 "Have you tried to do something that is more CPU-demanding than a simple resize and Convolution3D at low resolution?

[defalopii]

Have you tried to do something that is more CPU-demanding than a simple resize and Convolution3D at low resolution?

yes i've tried it on 720x400, thread increase but the FPS just a little different with no MT. Even if using too high value its become slower

[Overdrive80]

The MT doesnt work. I've tried many configuration, from 1 to hundred value but still no different between using MT or not



You can try SetMTMode(5, 8), with my cpu i have problems too using MT in mode 3. Intel core i7

[defalopii]

You can try SetMTMode(5, 8), with my cpu i have problems too using MT in mode 3. Intel core i7

Only got about 70 FPS 1st pass on 720x400 videos, still same with No MT/another MT value arround 65-70 FPS, no significant increase

[SEt]

If SetMTMode doesn't work – you are doing something wrong. I can't guarantee that 32 threads will work flawlessly (never tested such case), but something around 8 definitely works well. And I'm talking about Avisynth speed, not something like MeGUI to x264.

There is no quality degradation with SetMTMode. Either filters are MTMode-friendly and you will get exactly the same result as in non-MTMode, or you'll get crash, glitches, whatever but graceful quality degradation.

Often the MTMode functions only use half the available threads, especially on Intels
What? First time I hear such issue. How about making proper bug report?

[defalopii]

If SetMTMode doesn't work – you are doing something wrong. I can't guarantee that 32 threads will work flawlessly (never tested such case), but something around 8 definitely works well. And I'm talking about Avisynth speed, not something like MeGUI to x264.

Whats iwrong ?? how is the proper installation of Avisynth 2.6 MT ??
What i did, installed the Avisynth from here avisynth.org, than replace the avisynth.dll in the System32 folder with avisynth.dll from Avisynth 2.6 MT, thats it, is that wrong ??

Actually guys, how many percent increasing the FPS with or without Avisynth 2.6 MT ?? Is Avisynth 2.6 MT utilize the all cores up to arround 100% of CPU Usage ??

Do You ever trying like me splitting the videos into many parts and encode it simultaneously ?? than see how awesome it utilize the all core and CPU Usage and increasing the total encoding speed. Maybe Avisynth MT, or MeGUI or even x264 should try this methode, spliting the videos rather than playing with threads/frames

I dont know, am i still on my way.,,??
or sould move it to another sub-forum, there is so much, i am confused where must write this :confused:

[Keiyakusha]

There is no quality degradation with SetMTMode. Either filters are MTMode-friendly and you will get exactly the same result as in non-MTMode, or you'll get crash, glitches, whatever but graceful quality degradation.
It degrades on x264 level, not avisynth. Unless for some reason you use 32 threads for avisynth but only like 2 threads for x264.
As for me, I don't see the reason to use many threads in avisynth. Most of the time the actual encoding is a more CPU-intensive task than pre-processing. Of course if we use only good optimized filters, and our source is not some old self-recorded VHS that needs a lot of processing. But many "heavy" filters can have their own threads anyway.

[VideoFanatic]

I can't seem to get a speed improvement on my dual core PC over normal Avisynth:

I have Avisynth 2.5.8. I replaced the system32 avisynth file with the 2.6 MT file. I am using this script with Simple x264 Launcher to make an h264 file:

setmtmode(5,0)
Mpeg2Source("I:\file.d2v", CPU=6)
setmtmode(2,0)
DeGrainMedian(limitY=2, limitUV=3, mode=1, interlaced=true)
Crop(0,12,0,-12)
Tweak(Bright=0, Sat=1.0, Cont=0.9, Hue=10.0, Coring=False)
AddBorders(0, 12, 0, 12)

What am I doing wrong?

[Groucho2004]

I can't seem to get a speed improvement on my dual core PC over normal Avisynth

Have you tried to measure the speed of the script alone? You can do that with AVSMeter (http://forum.doom9.org/showthread.php?t=162155).

[VideoFanatic]

How do I use that?

[Bloax]

Drop your .avs over the .exe?

[VideoFanatic]

It reported the following:

Minimum: 9
Maximum: 66
Average: 37

with this script:

setmtmode(5,3)
Mpeg2Source("H:\New\z = Encode\Raw January 03 2000 Fixed.d2v", CPU=6)
setmtmode(2,0)
DeGrainMedian(limitY=2, limitUV=3, mode=1, interlaced=true)

It reported the following:

Minimum: 13
Maximum: 30
Average: 21

with this script:

Mpeg2Source("H:\New\z = Encode\Raw January 03 2000 Fixed.d2v", CPU=6)
DeGrainMedian(limitY=2, limitUV=3, mode=1, interlaced=true)

So is it the limit of the h264 encoding speed thats preventing me from getting faster encoding times or do I need more than a dual core PC?

[Atak_Snajpera]

is cpu usage at 100 percent?

[VideoFanatic]

Yes it is.

[Atak_Snajpera]

so here is your answer why you don't see speed improvement. encoding speed on dual core is too slow.

[SubJunk]

What? First time I hear such issue. How about making proper bug report?I don't know if it's an issue or by design because of Intel's Hyperthreading (maybe they don't want to support the hyperthreaded cores)

Avisynth 2.6 MT installer (2012.05.16) -> http://www.mediafire.com/?prfoy9wfj4gtnnw
Installer source files -> http://www.mediafire.com/?308aa5d3r4nszwzIf you make another version could you please make it set a unique version in the registry? It currently sets "2.6.0 MT" which is great but will be cool if it sets a different one next time so I can check against it (like "2.6.0 MT 2012.05.16" or something)

[SEt]

I have no idea what you are talking about. From Avisynth point of view hyperthreaded cores are as good as regular cores and there are no differences. I myself use hyperthreaded Intel CPU and never saw anything like that.
Note: as Avisynth is a dynamic library and not stand-alone process it inherits whatever CPU settings the host application set (like affinity) so such effects are possible but it's not Avisynth fault.