Avisynth 2.6 MT

[leeperry]

Nope, coz I run a x86 OS.

[upyzl]

Sorry for my node question...

How to use it?

I installed Avisynth 2.6.0 Alpha3 [2011-05-25], but I can't find "avisynth.dll" (avs input encoding succeeded)
so how should I use the avisynth.dll from 2011.07.19, please?

[Mini-Me]

Sorry for my node question...

How to use it?

I installed Avisynth 2.6.0 Alpha3 [2011-05-25], but I can't find "avisynth.dll" (avs input encoding succeeded)
so how should I use the avisynth.dll from 2011.07.19, please?

Assuming you have a typical Windows setup, you should find Avisynth.dll in C:\Windows\system32\. Just replace the old file with the new one, and you should be good to go.

[Chikuzen]

Assuming you have a typical Windows setup, you should find Avisynth.dll in C:\Windows\system32\. Just replace the old file with the new one, and you should be good to go.

He is a 64bit Windows7 user.
so, the place of avisynth.dll is C:\Windows\SysWOW64\.

[upyzl]

Thank you :)

[naoan]

so, which MT should I use this latest build with? none seems to be working...

[SEt]

I'm not going to maintain MT plugin. Use MTMode instead.

[RedDwarf1]

Try re-encoding the same "stuff" you did on friday if you want to compare frame rates because you can't really compare different sources and expect the exact same frame rate.

Are the scripts identical? Try using AVSMeter if the scripts are different to compare the speed.

Shutting down windows could of made windows change a system file, a windows update possibly which is slowing things down. Just a guess.

If you have other software running, including browsers, in the background they can affect encoding speed.

[jeremy33]

I'm not going to maintain MT plugin. Use MTMode instead.
SetMTMode produce audio/video desync (env. 240ms) with ffdshow (and LimitedSharpenFaster). Is there a fix for that ?

With MT.dll I don't have any audio/video desync but I have to stay with avisynth 2.58. :(

[Boulder]

SEt,

do you know whether SetMTMode(2) is safe with SRestore, which is a function that uses global variables quite extensively.

[SEt]

Boulder, as I remember it's dependent on sequential frame requests, so it won't work in current MTMode regardless or global variables.

jeremy33, MTMode works perfectly fine with ffdshow here even with complex scripts like QTGMC, just make sure you have enough "Buffer ahead" frames.

[leeperry]

it does IME, but MTMode is async AFAIK, so it's a major bummer for realtime use.

[jeremy33]

Set, leeperry, it works without desync with 10/10 buffer back/ahead :)

Set, how many buffer back/ahead you recommends with this kind of script
SetMemoryMax(512)
SetMTMode(3,4)
ffdshow_source()
SetMTMode(2)
coef=1.5
newWidth=round((float(last.width)*coef)/2)*2
newHeight=round((float(last.height)*coef)/2)*2
spline64resize(newWidth,newHeight)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=45)

Thank you

[leeperry]

I get more fps w/ MT() and seeking is sluggish in SetMTMode() due to the fact that it's async.

[jeremy33]

I do not have these problems. Seeking is fast, the CPU load is the same and there is no desync.

It works like a charm for me :)

[fabje]

Ok I really just don't get it anymore.

This is my system:
Intel i7 970
Gigabyte EX58-UD3R
Kingston 3x 2GB DDR3
OCZ Vertex 2 60GB OS
2x WD 1TB Green in RAID0

And a friend of mine has the same system.
We are doing the same Windows 7 installation including the drivers and our BIOS settings are the same.
Also our Staxrip/xvid/Avisynth & plugins are the same.

But still my friend is getting 400fps in pass 1 and 250 in pass2, but I only get about 250fps in pass 1 and 200 in pass2.

And we are using the same script:
SetMTMode(5,3)
MPEG2Source("E:\Recordings\test.d2v")
Crop(0,0, -Width % 8,-Height % 8)
ConvertToYV12()
SetMTMode(2)
LeakKernelDeint(order=1,threshold=8)
Crop(2,2,-2,-2)
BicubicResize(656,368,0,0.5)
Trim(14497,73911)

Anyone here that has a clue what is going on here?

[Boulder]

Can you check the CPU usage and number of utilized threads of both encodes?

[Zep]

well first off start by just doing this and nothing else (no MT)

MPEG2Source("E:\Recordings\test.d2v")

Get a baseline.


same hard drives and same fragmentation and source files same location on drives. I doubt it. Green drives are slow and there is NO RPM guarantee. you may get one that is 5400 RPM or you may get one that is higher. It is total luck. Always get black drives they are much faster and all 7200 RPM. Make a ram disk and test a small clip from that so the drives are not in the equation.

Same ram settings? Make sure cause ram speed plays a HUGE part in encode speed when doing FPS that high.

Same temperature? If your case is hotter you may be throttled back. Force the CPU speeds in bios do not let them float.

i could go on and on but start there lol

[fabje]

Can you check the CPU usage and number of utilized threads of both encodes?
I will do this as soon as the person who has the same system is back from his vacation.

well first off start by just doing this and nothing else (no MT)

MPEG2Source("E:\Recordings\test.d2v")

Get a baseline.


same hard drives and same fragmentation and source files same location on drives. I doubt it. Green drives are slow and there is NO RPM guarantee. you may get one that is 5400 RPM or you may get one that is higher. It is total luck. Always get black drives they are much faster and all 7200 RPM. Make a ram disk and test a small clip from that so the drives are not in the equation.

Same ram settings? Make sure cause ram speed plays a HUGE part in encode speed when doing FPS that high.

Same temperature? If your case is hotter you may be throttled back. Force the CPU speeds in bios do not let them float.

i could go on and on but start there lol
Well the two WD hardrives are in a RAID0 setup and when I do a benchmark I get these results:
http://www.fabje.eu/wd.jpg

I also have a SSD drive and with a benchmark I get the following results:
http://www.fabje.eu/ssd.jpg

When I run an encoding on both drives I get about the same speeds, so it isn't the storage drive.

Yes the ram settings are the same.

My temperature is fine during the encoding the temperature is between 40 en 50 celcuis degress depending on the core.

Also a wierd thing is that when I do a reencode with the x264 codec of a 1920 x 1080 recording to a 1280 x 720 file I get the same speeds as my friend.

[jeremy33]

jeremy33, MTMode works perfectly fine with ffdshow here even with complex scripts like QTGMC, just make sure you have enough "Buffer ahead" frames.
Actually Avisynth 2.60 and MTMode doesn't work with some kind of files, Avisynth crash.

Sample
http://www.demo-world.eu/trailers/redirect-high-definition.php?file=hd_dolby_truehd_channel_check_lossless.part1.rar
http://www.demo-world.eu/trailers/redirect-high-definition.php?file=hd_dolby_truehd_channel_check_lossless.part2.rar

[Zep]

I will do this as soon as the person who has the same system is back from his vacation.


indeed :)


You have SSD... does he? (SSD drivers are IMHO beta still and may be causing a conflict on your box. disable driver and see)

are the CPU/RAM speeds forced in the BIOS or not?



Also a weird thing is that when I do a re-encode with the x264 codec of a 1920 x 1080 recording to a 1280 x 720 file I get the same speeds as my friend.

well what does that tell you? It tells me you have an I/O bottleneck that he does not since x264 is so CPU heavy any I/O throughput problem is masked. This why i want the baseline with just

MPEG2Source("E:\Recordings\test.d2v")
(No encoding just that to test throughput)

because any decent box can read 40 frames a second in but when you get up to 400 FPS well.... that is a lot of data.


You may have a driver installed he does not that is slowing you down. A good example is most all anti virus. The default is for them to scan a file on OPEN READ WRITE and CLOSE. this will slow you way down and you should only need to scan once on OPEN and once on CLOSE. if you let it scan during read/write your I/O will be cut in half or more.

no need to wait for your friend if you make a ram disk and test from that. (note if you can't get at least 1 gig second read from ramdisk your anti virus or some other driver is slowing it down)


anyway just more testing needed and we will know enough... good luck

[fabje]

indeed :)


You have SSD... does he? (SSD drivers are IMHO beta still and may be causing a conflict on your box. disable driver and see)

are the CPU/RAM speeds forced in the BIOS or not?




well what does that tell you? It tells me you have an I/O bottleneck that he does not since x264 is so CPU heavy any I/O throughput problem is masked. This why i want the baseline with just

MPEG2Source("E:\Recordings\test.d2v")
(No encoding just that to test throughput)

because any decent box can read 40 frames a second in but when you get up to 400 FPS well.... that is a lot of data.


You may have a driver installed he does not that is slowing you down. A good example is most all anti virus. The default is for them to scan a file on OPEN READ WRITE and CLOSE. this will slow you way down and you should only need to scan once on OPEN and once on CLOSE. if you let it scan during read/write your I/O will be cut in half or more.

no need to wait for your friend if you make a ram disk and test from that. (note if you can't get at least 1 gig second read from ramdisk your anti virus or some other driver is slowing it down)


anyway just more testing needed and we will know enough... good luck
Yes he has a SSD but he doesn't use that for encoding.

And a little correction my friend is using 2 WD Blacks in RAID0.

My CPU/RAM speeds are indeed forced in the BIOS. So all that energy saving crap is disabled.

When I'm running the baseline I also get around 250 fps.
My anti virus software isn't the problem because I had this problem already when I just had a clean install of Windows 7 with only the drivers installed for my motherboard and graphic card. And I used the same drivers as my friend did.

[Zep]

Yes he has a SSD but he doesn't use that for encoding.

And a little correction my friend is using 2 WD Blacks in RAID0.




well that little correction is huge. I'm not saying it is the only problem but from what you said so far it is the main difference i see. Black drives get 40% more sustained throughput on reads than greens.

he gets about 400 you get about 250 well.... there you go lol


I say again. test from a ram disk to make sure. :)

[fabje]

well that little correction is huge. I'm not saying it is the only problem but from what you said so far it is the main difference i see. Black drives get 40% more sustained throughput on reads than greens.

he gets about 400 you get about 250 well.... there you go lol


I say again. test from a ram disk to make sure. :)
And how can I test from a RAM disk?

[Zep]

And how can I test from a RAM disk?

I use this and it works great (yes it is free if only 4 gig ram disk or smaller)

http://memory.dataram.com/products-and-services/software/ramdisk

then move a .ts or .mpg onto the ram disk and run your script so that it points to the source file on that ram disk. if it's an HD throughput wall you will know real quick. (assuming you are no where near 100% CPU etc... I do not recall if you mentioned that or not)

[fabje]

I use this and it works great (yes it is free if only 4 gig ram disk or smaller)

http://memory.dataram.com/products-and-services/software/ramdisk

then move a .ts or .mpg onto the ram disk and run your script so that it points to the source file on that ram disk. if it's an HD throughput wall you will know real quick. (assuming you are no where near 100% CPU etc... I do not recall if you mentioned that or not)
Well I did the test and I got the same results.
So seems like my HDD isn't the problem.

My CPU has an usage of about 50%.
I also did a check of how many threads my pc and the pc of my friend have and we are both around 880 threads.

And I did a quick HDD benchmark and this are the results:
Transfer Rate:
Minimum: 114.7MB/sec
Maximum: 156.3MB/sec
Average: 149.8MB/sec

Access Rate: 10.9ms
Burst Rate: 1295.9 MB/sec

And yes his average is higher but if I check my SSD, the SSD is faster but still this didn't changed a bit.

[Zep]

My CPU has an usage of about 50%.




so when your friend runs just MPEG2Source("E:\Recordings\test.d2v")

no encoding just a pass through FPS test he gets 400 and you get 250? You both are using same versions of software? Same bios version? same drivers versions? etc.... get process explorer and look at the avisynth threads. Compare your boxes and you should be able to see where the bottleneck is. you can even click on a single thread and see the I/O rates and much more. yeah it is time to go lower level lol


good luck

[fabje]

so when your friend runs just MPEG2Source("E:\Recordings\test.d2v")

no encoding just a pass through FPS test he gets 400 and you get 250? You both are using same versions of software? Same bios version? same drivers versions? etc.... get process explorer and look at the avisynth threads. Compare your boxes and you should be able to see where the bottleneck is. you can even click on a single thread and see the I/O rates and much more. yeah it is time to go lower level lol


good luck
No my friend gets 400fps when he is using this script:
SetMTMode(5,3)
MPEG2Source("E:\Recordings\recording.d2v")
Crop(0,0, -Width % 8,-Height % 8)
ConvertToYV12()
SetMTMode(2)
LeakKernelDeint(order=1,threshold=8)
Crop(2,2,-2,-2)
BicubicResize(656,368,0,0.5)
Trim(9992,46915)

And yes our software version etc is the same.
The only thing that is really different is that my cpu is overclocked and his pc isn't. So I thought maybe it was an setting in my bios, but even with everything in there on default. My speeds are getting slower then what I have right now.

[IanB]

... The only thing that is really different is that my cpu is overclocked and his pc isn't. ....
Overclocking is a very black art. Try running the tests with the exact same settings as your friend.

One possible pitfall is memory access pattern, it is possible to only get down to half the effective memory speed when comparing two CPU speeds that are almost the same This happens when the slightly faster speed requires stepping down to a much crappier memory access pattern.

Also some memory controllers drop to a lower speed as you add more than a certain number of memory modules (bus loading problems). We had 2 supposedly identical servers shipped, 1 full of 1GB memory sticks the other with nice 2GB memory sticks .....

[fabje]

Overclocking is a very black art. Try running the tests with the exact same settings as your friend.

One possible pitfall is memory access pattern, it is possible to only get down to half the effective memory speed when comparing two CPU speeds that are almost the same This happens when the slightly faster speed requires stepping down to a much crappier memory access pattern.

Also some memory controllers drop to a lower speed as you add more than a certain number of memory modules (bus loading problems). We had 2 supposedly identical servers shipped, 1 full of 1GB memory sticks the other with nice 2GB memory sticks .....
I did try it with the exact same setting as my friend.
The only thing I could try is to test it with the same memory settings.

I also did a process test and this are the results when i'm encoding something on my pc.
http://www.fabje.eu/process.jpg
http://www.fabje.eu/process2.jpg

And this is a screenshot of the process explorer on the pc of my friend.
http://www.fabje.eu/process3.jpg

[matfra]

I download th last version of Avs 2.6MT. Where do I put the file diff.7z ??? I paste Avisynth.dll infor SysWow64. But I have no clue what to do with the other file. Help me please !

[-Vit-]

The diff file just contains the source code changes from the official Avisynth 2.6. Only developers need this file, you can ignore it.

[SubJunk]

Been getting occasional crashes with the new version

[mbcd]

Yes, me too

2.5.8 runs like a charm
2.5.8 MT runs at 100% Processorusage (at all cores) but crashes with nearly (95%) every job.
2.6.0 MT the same as 2.5.8

Stability is a big problem ...

[SubJunk]

Sorry, I should have been more specific with my previous post.
I meant the new version of 2.6 compared to the older 2.6; 2011.07.19 has the occasional crashes while 2009.09.19 works great for me.

[SEt]

Update, two more MT bugfixes.

I run into crash with colorspace conversion and so disabled USE_DYNAMIC_COMPILER. No crashes but slower. If speed of it is concern for you - post here, if there is enough demand I can investigate the issue. Solved, see below.

[SubJunk]

Thanks for the update, will test

[IanB]

@SEt,

In convert_planer.cpp, unpckbuf and packbuf are class local this is not thread safe.

Uncomment the _aligned_malloc and _aligned_free in the two GetFrame calls. Probably should test for malloc failures as well.

[jeremy33]

Thanks for the update Set.

All optimizations are good to take. That said I understand that it takes a long time and I prefer to have a stable soft rather than a faster soft unstable :)

:thanks:

[SEt]

Another update, redownload if you just got the build earlier today.

IanB, it should not be a problem in MTMode 2, so I left it as is for now. Found the actual crash reason: I was playing with default calling convention and missed unspecified one on dynamically generated code call. I think conventions should be explicitly written where they are important.

[asarian]

Some 64bits pluggins can have sometime speed increase of 30%.
I've noticed, on QTMG for example, an increase speed of around 15% under 64bits.

My personnal thought, at the actual stage is :
The way MT is implemented on avisynth seems not to be very stable, so, the best way actualy to have a stable multithreading would be to not use MT version of avisynth, split your work and start several task. For exemple, launching 6 Vdub task on 6 files.
Using not MT version of avisynth keep garanty stability, and running several tasks make all your core work.
I think, it could be more interesting to focus on single threading 64bits port, because good 64bits port can provide good improvement in speed.
I've noticed it with the unfornutaly dead project of the 64 bits version begun by JoshyD.

Perhaps in abundance, but I'd like to reiterate that the advantage of the often hailed speed-improvements of a 64-bit build pales in comparison to the gained memory. Especially for HD material, memory is a constant limiting factor. For instance, when using MCTemporalDenoise, I can't even run 32-bit Avisynth, unless I use a process separation trick to pipe output to x264, using avs2yuv. And even then there are times I have to split up a movie (vertically), in two parts, in order to bypass memory issues. So a good 64-bit version of AviSynth 2.6 would be very welcome.

[IanB]

IanB, ... Found the actual crash reason: I was playing with default calling convention and missed unspecified one on dynamically generated code call. I think conventions should be explicitly written where they are important.Please explain more, I don't understand where you could break this. :confused:

[SEt]

In DynamicAssembledCode::Call
return ((int (*)(const void* *))entry)(&arg1);
should be
return ((int (__cdecl*)(const void* *))entry)(&arg1);

There are plenty of other places where convention should be specified - try changing default convention to __fastcall and compile. Good thing compiler throws errors there so it's easy to spot the places.

[IanB]

Ah! You were trying to trick me. ;) Nothing to do with any convert\convert_*.cpp's it was in file core\softwire_helpers.cpp, fair enough.

[JEEB]

Couldn't get the current trunk to build, fix (https://github.com/jeeb/avisynth/commit/647a37821ce3b9aa09f64c829d47119fa3596a95). Didn't know MSVC6 compiled stuff like this :) .

[DarkT]

What has less head-ache involved, using 2.5.8 mt or 2.6? Like, I want to use up my i7-950 as best as I can, with the current x64 filters, which would be a better choice? 2.6? 2.5.8?

[TheFluff]

What has less head-ache involved, using 2.5.8 mt or 2.6? Like, I want to use up my i7-950 as best as I can, with the current x64 filters, which would be a better choice? 2.6? 2.5.8?

32-bit single-threaded 2.5.8 is by far the best choice if the purpose is to avoid headaches.

[DarkT]

32-bit single-threaded 2.5.8 is by far the best choice if the purpose is to avoid headaches.

Seems like what I wrote wasn't clear enought(perhaps not only for you) - I will restate: Least headache, while using as much of the CPU power as possible.

I do want to use multi threading. But I wonder whether 2.5.8 MT is better than 2.6. Or is it perhaps better to just run 2 or 4 simultaneous encoding proccesses? I do not know. I was hoping somebody would help me with making that decision, instead of gruelly trying it out myself.

[Groucho2004]

32-bit single-threaded 2.5.8 is by far the best choice if the purpose is to avoid headaches.

I have not had any headaches with the current 2.6 Alpha 3. :D

[Groucho2004]

I will restate: Least headache, while using as much of the CPU power as possible.
I do want to use multi threading. But I wonder whether 2.5.8 MT is better than 2.6.

That's like choosing between a migraine and a serious hangover.

Or is it perhaps better to just run 2 or 4 simultaneous encoding proccesses?
Yes.