Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 9th August 2005, 16:06   #61  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
I don't think the to things are connected. I will investigate it when I get back from my vaccation in 14 days. Also try this script and see if it works and if it is faster than your current script:
Code:
SetMTMode(2)
mpeg2source("E:\VTS\Test\menue\d2v\sw4.d2v")
LeakKernelDeInt(order=1, threshold=5)
c=ConvertToYUY2
SetMTMode(5)
c.Distributor()
PixieDust(4)
SetMTMode(2)
VagueDenoiser(threshold=2,method=1,nsteps=6,chromaT=0.8, wiener=true, wratio=2)

Last edited by tsp; 9th August 2005 at 16:14.
tsp is offline   Reply With Quote
Old 5th September 2005, 00:12   #62  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
This is a great plugin!

I just tried it out using a 40 sec 3 Stooges test clip and here's the results using X264 2-pass with MeGUI 0.2.2.2

X264 - 1 thread - No MT: 7.23FPS Average

X264 - 2 threads - No MT: 11.92FPS Average. A 65% improvement!

X264 - 2 threads - MT On: 12.68FPS. A 75% improvement over a single thread!

Here's my script in case anybody is interested:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
SetMTmode(1,2)
Trim(4482,5454)
Greyscale()
RemoveGrain(mode=7, modeU= -1).RemoveDirt(grey= true)
Greyscale()

EDIT:
I use an Athlon X2 4600+ in case anybody is curious.

EDIT2:

I just noticed that using SetMTmode(1,2) gives very occasional graphical glitches when RemoveGrain is being used. So I've now retested it using SetMTmode(2,2).

X264 - 2 Threads - MT On: 12.59 FPS. Still 74% faster than a single thread. Not much peformance loss at all.

Last edited by Revgen; 5th September 2005 at 00:55.
Revgen is offline   Reply With Quote
Old 5th September 2005, 22:17   #63  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
looks good. Just curious. How fast is it with X264 - 1 thread and MT on?
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 5th September 2005, 22:47   #64  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
Quote:
Originally Posted by tsp
looks good. Just curious. How fast is it with X264 - 1 thread and MT on?
Well since you asked...

7.6 FPS. About a 5% increase over 1 thread W/ No MT.

It seems to me like MT works better when it's paired with a multithreading encoder.
Revgen is offline   Reply With Quote
Old 6th September 2005, 22:06   #65  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
I somewhat expected that the increase would be greater. But then again if avisynth uses 1 core and the X264 encoder uses the other when there are not much cpupower left for the second thread in avisynthMT. Strange that it's faster with MT compaired to no MT. If you just encode to MJPEG or HuffYUV how much is the increase then?

[Edit]
Hmm maybe the avisynth thread is using lets say only 20% of the first core because it's the encoder that's slowing things down. X264 uses 100% of the second core if running only singlethreaded and if running with 2 threads the second thread in X264 uses the rest of the first core. Although this doesn't explain why enabling multithreading in avisynth increases the performance.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/

Last edited by tsp; 6th September 2005 at 22:25.
tsp is offline   Reply With Quote
Old 7th September 2005, 22:41   #66  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
...More Results

I've now decided to try different codecs and filters to get a better idea of what MT can do.


My Setup:

MJPEG= FFDshow MJPEG encoder. Encoded at 1-pass 1500kbps constant bitrate.

NOTE: This MJPEG codec also has an option for 2 threads, but when I tried it gave me an error message. Hopefully it gets fixed in the future.

Huffyuv=FFDshow Huffyuv encoder in YV12 colorspace, Median predictor, and NO Adaptive Huffman Tables (for more speed).

X264= 1490Kbps 2-pass using Me-GUI.

Xvid= 1490Kbps 2-pass.



Avisynth Scripts:

RemoveGrain.RemoveDirt Filter:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
RemoveGrain(mode=7,modeU= -1).RemoveDirt(grey=true)
Greyscale()

FFT3D Filter:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
fft3dfilter(sigma=4,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,ratio=2)

Limited Sharpen Function:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
LimitedSharpen()


Results:

MJPEG (Fast Codec) W/ RemoveGrain.RemoveDirt Filter (Fast Filter)

97.3 FPS Same Performance w/ MT On and Off.


MJPEG (Fast Codec) W/FFT3D filter (Slow filter)

23.73FPS MT OFF

24.95FPS MT ON

About an 5% increase in performance.


MJPEG (Fast Codec) W/LimitedSharpen Function (Very Slow Function)

9.45FPS MT OFF

16.49FPS MT ON

About a 74% increase in performance.


HuffYUV (Fast Codec) W/RemoveGrain.RemoveDirt Filter (Fast Filter)

About 97.3 FPS (again). Also same perf w/ MT ON and OFF.


HuffYUV (Fast Codec) W/FFT3D Filter (Slow filter)

23.73FPS w/MT OFF: Same as MJPEG

24.95FPS w/MT ON: Same as MJPEG

About a 5% peformance increase.


HuffYUV (Fast Codec) W/ LimitedSharpen Function (Very Slow function)

9.36 FPS MT OFF

16.22 FPS MT ON

About a 73 % peformance increase.



X264(Slow Codec) W/ FFT3D Filter(Slow Filter)

5.83FPS 1 Thread MT OFF

7.44FPS 1 Thread MT ON: About a 28% increase.

8.51FPS 2 Threads MT OFF: About a 46% increase.

9.64FPS 2 Threads MT ON: About a 65% increase. This is a very different result than the RemoveGrain filter. It seems that MT is less effective with multithreading ON this time around. It seems to excel in single threading more with this filter.


X264(Slow Codec) W/ LimitedSharpen Function (Very Slow Function)

4.19 FPS 1 Thread MT OFF

6.85 FPS 1 Thread MT ON: About a 63% increase over a single thread.

5.45 FPS 2 Threads MT OFF: Only a 30% increase in performance. Very Interesting. It seems that LimitedSharpen affects the encoding speed of X264 more than the other filters. It could be because sharpening makes it harder for the encoder to do it's job while denoisers make it easier.

7.96 FPS 2 Threads MT ON: About a 90% increase over a single thread. Very impressive! It looks like MT is most responsible for this result.



Xvid(Medium-Slow codec) W/RemoveGrain.RemoveDirt Filter(Fast Filter)

8.69FPS MT OFF

8.69FPS MT ON: No difference in performance.


Xvid (Medium-Slow codec) W/ FFT3D Filter (Slow Filter)

8.92FPS MT OFF:

8.46FPS MT ON: 5% slower. Very strange result. Not sure why this is occuring.


Xvid(Medium-Slow codec) W/ Limited Sharpen Function (Very Slow Function)

8.32FPS MT OFF

7.91FPS MT ON: About 5% slower again. It looks like MT and Xvid don't work well together for some reason.




I 've now decided to try Xvid without any filters using the script below:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)

Results:

Xvid recorded results of 8.76FPS both with MT ON and OFF.



Summary:

MT seems to help performance in most situations, doesn't help in some, and in the case of Xvid, can even hurt performance.

I'm not sure why it doesn't work well with Xvid.

Which has gotten me thinking...What codec was Hartford using when he reported that MT was slower than NO MT?

If it was Xvid then therein may be the problem.

I for the most part, use X264 for encoding. My Dual-Core CPU makes it worth my while compared to Xvid. But, Xvid is still a popular codec and this might be a problem for others.

Other than the Xvid issue, MT in general seems to work very effectively when combined with filters that take advantage of the extra power.

I hope this helps.

Last edited by Revgen; 8th September 2005 at 02:47.
Revgen is offline   Reply With Quote
Old 8th September 2005, 05:39   #67  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
VirtualDub supports multithreading natively so that might explain the XviD thing.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 8th September 2005, 06:35   #68  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
Quote:
Originally Posted by Boulder
VirtualDub supports multithreading natively so that might explain the XviD thing.
Which Version?

I used VirtualDubMod 1.5.10.

How would it adversely affect Xvid's performance?
Revgen is offline   Reply With Quote
Old 8th September 2005, 07:41   #69  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
all versions of virtualdub does that. 1 thread runs the avs-script the other tnread does the encoding. version 1.7 of virtualdub will support SMP more than that.
Will comment on the result later.
[edit]
well that doesn't make a lot of sense. First I would have expected that fft3dfilter combined with a less demanding codec would give more than 5% speed increase. But then again I think I know why it is so slow. It might be because fft3dfilter has an internal cache of the fast fourier transformed frames (I used that with fft3dGPU so I think Fizick does the same) and when using MT mode 2 each thread has a seperate instance of fft3dfilter that doesn't share the internal cache so that it is because of increased internal cache misses that causes the slowdown. If this is true this might be faster(if it works that is):
Code:
SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
SetMTmode(3)
fft3dfilter(sigma=4,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,ratio=2)
I don't know what happends with the xvid stuff. It might be because the 2 avisynth threads somehow slows the single xvid threads down. Try increasing the virtualdub priority. Also try increasing the stream data pipelining under options->performance options in virtualdub and see if it makes any difference. Another good way to look at how well avisynth and virtual dub interacts is to look at the cpu utilization.
One last thing. Try replacing pixiedust in limitedsharpen because that filter interacts very badly with multithreading in avisynth
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/

Last edited by tsp; 8th September 2005 at 20:56.
tsp is offline   Reply With Quote
Old 8th September 2005, 22:39   #70  |  Link
ChrisBensch
Registered User
 
Join Date: Oct 2001
Posts: 169
Interesting Plugin

First off, excellent work. Now that dual-cpu/dual-core are the norm for new PCs, it's nice to see more plugins/people/apps taking advantage. I've just come across this thread and am very interested. Most of my encodes are HDTV -> DivX. My sources are always clean and don't really need any processing. Here is my normal script for film source

MPEG2Source("Harry Potter and the Prisoner of Azkaban-HDTV.d2v")

Crop(2,0,-4,-6)

LanczosResize(1280,720)

My questions are: Given the script above, do you seen any possibility of speeding things up?

how do I tell MT that I'm using the special Avisynth for MT? I read the doc where it says to set the value to true, but that was using mt("filter here"). How can I tell it to work using the scripts I've seen in this thread?
For ex:

SetMTmode(2,2)
MPEG2Source("Harry Potter and the Prisoner of Azkaban-HDTV.d2v")
Crop(2,0,-4,-6)
LanczosResize(1280,720)

where woudl I put a notifier to MT that I'm using it's special avisynth.dll?

Last edited by ChrisBensch; 8th September 2005 at 23:12.
ChrisBensch is offline   Reply With Quote
Old 8th September 2005, 22:51   #71  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
The SetMTmode(3) setting causes glitches in my video, and it also makes the Xvid video encode even slower at 7.05FPS. These glitches also apear when playing back the .AVS file, so it's not Xvid's fault.

So I decided to switch back to SetMTmode(2) and keep the the Vdub settings you recommended.

The result and FPS turned out to be the same as the previous FFT3D post.


Also, according to Didee Limited Sharpen only uses Masktools and the Warpsharp package. PixieDust shouldn't be the problem.


On the other hand I found a bug in the filter. It's similar to the WMP 6.4 memory leak bug that I found in the FFTGPU filter earlier. Every time I play an .AVS file with WMP when MT is used WMP 6.4 stays resident in memory and doesn't shut down automatically. I have to go into the taskbar to shut it down instead. This bug seems to only affect WMP though. Encoding apps close down properly.


Also, just a reminder, DON'T GO CRAZY!

MT is still an excellent filter that works well with most codecs. You should be proud!

Last edited by Revgen; 8th September 2005 at 23:03.
Revgen is offline   Reply With Quote
Old 8th September 2005, 23:26   #72  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
Quote:
Originally Posted by ChrisBensch
My questions are: Given the script above, do you seen any possibility of speeding things up?
Try it and see what you get. Make sure to read about tsp's special version of avisynth and download the file at the bottom of the post.

Replace the avisynth.dll in the C:\Windows\Sytem32\ directory with the new one. Make sure you rename the old .dll to something like "avisynth.dll.old" and keep it in case you want to use that one later.

Based on your script I believe you should be able to use the SetMTmode(2,2) script at the bottom of your page just fine.
Revgen is offline   Reply With Quote
Old 8th September 2005, 23:49   #73  |  Link
ChrisBensch
Registered User
 
Join Date: Oct 2001
Posts: 169
I did exactly that, I get about a 1fps increase. Not bad I guess. I'm doing full 1920x1080 -> 1280x720 DivX 6 encodes. I get about 4.75fps using AVS2AVI, I'll see what VDub gets me.
ChrisBensch is offline   Reply With Quote
Old 9th September 2005, 07:41   #74  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
Some Good News.

I've got some good news.

I've decided to try encoding my clip using AVS2AVI instead of Vdub to see if it changes anything:

Results:

HuffYUV W/FFT3D Filter:

23.73FPS MT OFF

26.29FPS MT ON

Interesting. The MT OFF result is the same as the Vdub result. Yet MT ON results in an 11% increase in performance. More than double the Vdub result of 5%.

I've go to go to sleep now, I'll test the other codecs and filters tommorow.
Revgen is offline   Reply With Quote
Old 9th September 2005, 13:15   #75  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
Revgen: krieger2005 also noted the problem with windows media player classic. I couldn't reproduce the error on my single processor computer but I will try it on my dual processor later. Shouldn't be that hard to fix. Proberly something like the fft3dGPU error. Also you could try using my original MT.dll with fft3dfilter and my version of avisynth.dll like this:
Code:
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
MT("fft3dfilter(sigma=4,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,ratio=2)",2,8,false,true)
also I mixed limitedsharpen(that doesn't use pixiedust) and iip (that uses pixiedust) up. Both not exactly the fastest scripts in avisynth. Maybe virtualdub 1.6 is faster than virtualdubmod?
Oh and thanks for the kind words
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 9th September 2005, 18:25   #76  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
@tsp

First of all lets start off with the bad news first this time. Xvid's results are still the same with all 3 apps despite whether I use the older MT filter or the newer SetMTmode setting.

Now for the good news using your suggested script:

HuffYUV W/FFT3D Filter--Encoded with AVS2AVI

33.55FPS. Thats about a 41% increase in performance over the previous 23.73FPS result W/MT OFF. Using this older filter seems to provide better benefits.

HuffYUV W/FFT3D Filter--Encoded with VirtualDubMod 1.5.10

31.39FPS. Better then with the SetMTmode setting, but still slower than AVS2AVI. This time it's slower than AVS2AVI by 7%.

HuffYUV W/FFT3D Filter--Encoded with VirtualDub 1.6.10

30.4FPS. This version of Vdub is even slower than Vdubmod(by about 3%) and about 10% slower than AVS2AVI.

MJPEG W/FFT3D Filter

All 3 encoding apps recorded the same results as HuffYUV.

I hope this helps.
Revgen is offline   Reply With Quote
Old 9th September 2005, 21:47   #77  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
I wonder what's causing the problem with xvid?
Good to see that the old version is working well with fft3dfilter. The old MT and the avisynth.dll version works in two different ways. MT.dll lets each thread run on a small piece of a frame and combines the result while avisynth.dll SetMTMode lets each thread work on a different frame. the MT.dll aproach is good when the filter doesn't relies on information from the whole frame so it's good when used with filters like fft3dfilter and simple sharpen and blur filters while SetMTmode is good with filters like HDRAGC and some of the deinterlacers that needs information from the entire frame.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 9th September 2005, 22:08   #78  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
Quote:
Originally Posted by tsp
...SetMTmode is good with filters like HDRAGC and some of the deinterlacers that needs information from the entire frame.
Do you think that Masktools benefits from this mode? Maybe thats why LimitedSharpen works so well in this mode. There are many Masktools functions out there that may also reap benefits.

EDIT:

Also, is there a way to turn off SetMTmode() switch once you turn it on? I'd like to be able to switch between the old and new MT in the same script so that I can use certain filters without conflicts.

IE:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
SetMTmode(OFF)
MT("fft3dfilter(sigma=4,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,ratio=2)",2,8,false,true)
SetMTmode(2,2)
LimitedSharpen()

If not, then it's no big deal. I can wait.

Last edited by Revgen; 10th September 2005 at 03:11.
Revgen is offline   Reply With Quote
Old 10th September 2005, 10:05   #79  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
Quote:
Do you think that Masktools benefits from this mode? Maybe thats why LimitedSharpen works so well in this mode. There are many Masktools functions out there that may also reap benefits.
There are an extra overhead when using mt.dll compaired to SetMTMode() so I think they would benefit the most from SetMTMode because they don't have any internal cache that I'm aware off. But nearly all the functions in MaskTools onkly needs information from the pixels that surround the current pixel that are being processed so MT.dll should work fine with them (compaired to a deinterlace where only half the frame would be deinterlaced if only 1 thread thought the frame was interlaced).
Quote:
Also, is there a way to turn off SetMTmode() switch once you turn it on? I'd like to be able to switch between the old and new MT in the same script so that I can use certain filters without conflicts.
SetMTmode(5) turns the multithreading off. The problem with it currently is that it also turns it of for all filters before SetMTMode(5) although you can enable it by inserting Distributor() just after SetMTMode(5) something like this:
[/code]
SetMTmode(2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
g=Greyscale()
SetMTmode(5)
g.Distributor()
MT("fft3dfilter(sigma=4,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,ratio=2)",2,8,false,true)
SetMTmode(2)
LimitedSharpen()
[/code]
It is one of the things I still need to add to the SetMTMode().
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 11th September 2005, 18:11   #80  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
The script works well.

I got 7.05FPS with MT OFF and 11.9FPS with MT ON using HuffYUV.

About a 69% increase.


Also,

I hope Shodan eventually puts your code into the upcoming version of AVS 2.6.

He told you "I'll do it ASAP" in this thread, but didn't give a clear response confirming whether he did or not.

You might want to try reminding him again.
Revgen is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:16.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.