Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 8th August 2008, 11:46   #941  |  Link
yup
Registered User
 
Join Date: Feb 2003
Location: Russia, Moscow
Posts: 854
josey_wells!
Good work. I find even MVTools.dll from folder MSVC08 work little faster, but I can not run IC10 version. At home I have old CPU
Number of processors in system: 1
Current processor: #1
Cores per processor: 1
Disabled cores per processor: 0
Processor Name: Intel(R) Pentium(R) 4 CPU 2.53GHz
Type: 0
Family: F
Model: 2
Stepping: 7
Revision: 37
Maximum CPUID Level: 2
L1 Trace Cache: 12 Kµops
L1 Data Cache: 8 KБ
L2 Cache: 512 KБ
Packaging: FC-PGA2
EIST: No
MMX(TM): Yes
SSE: Yes
SSE2: Yes
SSE3: No
SSE4: No
Enhanced Halt State: No
Execute Disable Bit: No
Hyper-Threading Technology: No
Intel(R) 64 Architecture: No
Intel(R) Virtualization Technology: No
Expected Processor Frequency: 2.53 ГГц
Reported Processor Frequency: 2.53 ГГц
Expected System Bus Frequency: 533 МГц
Reported System Bus Frequency: 533 МГц
My CPU support SSE2 but when I open avs file in VirtualDub, VirtualDub close without error report. Please advice.
yup.
yup is offline   Reply With Quote
Old 8th August 2008, 11:50   #942  |  Link
josey_wells
Registered User
 
Join Date: Dec 2007
Location: VA, USA
Posts: 116
Tschniede is basically correct.

In short, MVDegrain1, MVDegrain2, & MVDegrain3 were all changed to be multithreaded for the LumaY, ChromaU, and ChromaV processing where the threads to do the processing are created in the constructor and destroyed in the destructor. However, all frame getting operations are done on the mainline thread so in general little speed improvement will be achieved as the MVAnalyse calls will all be run on the main line thread only.

A new function MVAnalyseMulti was added which creates a seperate thread to do the analysing for each forward or backward frame. The threads are only created in the constructor and destroyed in the desctructor. The output is packed into an arrayed output only compatible at this time with MVDegrainMulti but other functions can easily be adopted to handle this arrayed format. MVAnalyseMulti is the same as MVAnalyse with the following changes:

1. isb and delta were removed
2. new parameters refframes and deltamult were added
refframes - is the number of forward and backward reference frames to analyse and can be set between 1-32. Default is 1. 1 would be 2 frames, 2 would be 4 frames, etc.
deltamult - is the multiplier used for the frame stepping. If deltamult is 1 then the offsets are 1, 2, 3. If deltamult is 2 then the offsets are 2, 4, 6, etc. Default is 1 and can be set between 1-32.

For example:
Code:
backward_vec2 = source.MVAnalyse(isb = true, lambda = 1000, delta = 2)
backward_vec1 = source.MVAnalyse(isb = true, lambda = 1000, delta = 1)
forward_vec1 = source.MVAnalyse(isb = false, lambda = 1000, delta = 1)
forward_vec2 = source.MVAnalyse(isb = false, lambda = 1000, delta = 2)
can be replaced by
Code:
ForBackVec = source.MVAnalyseMulti(refframes=2, lambda = 1000)
Some options still open for debate is whether it is worthwhile to add a sweep direction flag to enable forward only, backward only, or the standard forward and backward.

A new MVDegrainMulti was added which still does the multithreaded Luma, ChromaU, and ChromaV processing and has the following syntax:

Code:
MVDegrainMulti(clip, clip "MVForBack", int "refframes", int "thSAD", int "thSADC", int "plane", int "limit", clip "pelclip", int "idx")
MVForBack - is the arrayed analysis frames from MVAnalyseMulti which replaces all the seperate forward and backward clips
refframes - the number of reference frames used in MVAnalyseMulti to produce MVForBack. Default is 1.

These two calls allow all MVDegrain calls equivalent to MVDegrain1 up to MVDegrain32.

For example a MVDegrain16 would be

Code:
MVForBack = MVAnalyseMulti(refframes=16, pel = 2, overlap=4, sharp=1, idx = 1)
last.MVDegrainMulti(MVForBack, refframes=16, thSAD=400,idx=1)
As for the rest of the code, I made changes to the critical section logic used to use autoscope variables. I modified MVClip to act as normal or handle arrayed types. I formatted the text of the files to remove tabs and repalce with spaces so that display is uniform between text editors.

I fixed a memory leak in the desctructor of MVMask and MVShow.

Many other numerous small changes.

These calls can be used with the latest AviSynth V2.5.8 RC3 to due multithreaded processing or they can be combined with some of the MT or SetMTMode variants if desired.
josey_wells is offline   Reply With Quote
Old 8th August 2008, 12:18   #943  |  Link
josey_wells
Registered User
 
Join Date: Dec 2007
Location: VA, USA
Posts: 116
Quote:
Good work. I find even MVTools.dll from folder MSVC08 work little faster, but I can not run IC10 version. At home I have old CPU
I looked at project settings and was requiring SSE3 also.

Here are two versions one built for P4 with SSE2 and one built for Core2 Duo with SSE2 & SSE3

http://rapidshare.com/files/13578584...uilds.rar.html

FYI: Do not use the Core2 Duo IC10 build. The intel compiler has an issue when I tested this build. Use the P4-SSE3 build which was the original IC10 build posted with the final, or the P4-SSE2 build posted here. Or use the MSVC08 build which should work for anyone.

Which is fastest is trial and error

Last edited by josey_wells; 8th August 2008 at 14:14. Reason: Warning
josey_wells is offline   Reply With Quote
Old 8th August 2008, 12:43   #944  |  Link
yup
Registered User
 
Join Date: Feb 2003
Location: Russia, Moscow
Posts: 854
josey_wells!
Now all work from folder IC10-P4-SSE2.
One more
yup.
yup is offline   Reply With Quote
Old 8th August 2008, 13:23   #945  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
I've just finished running a test encode (Planet Earth) and the whole thing went through without a problem. I used the IC10 version, and got roughly ~1.36 fps using two computers, hah. This is far better than what I got on one computer and more modest x264 settings for that matter.
__________________
You can't call your encoding speed slow until you start measuring in seconds per frame.
Sagekilla is offline   Reply With Quote
Old 8th August 2008, 17:02   #946  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,412
Quote:
Originally Posted by josey_wells View Post
A new MVDegrainMulti was added which ... has the following syntax:
Code:
MVDegrainMulti(clip, clip "MVForBack", int "refframes", int "thSAD",
 int "thSADC", int "plane", int "limit", clip "pelclip", int "idx")
MVForBack - is the arrayed analysis frames from MVAnalyseMulti which replaces all the seperate forward and backward clips
refframes - the number of reference frames used in MVAnalyseMulti to produce MVForBack. Default is 1.
Would it be possible to store the value of refframes along with the data inside "MVForBack", so that it need not be passed as a parameter to MVDegrainMulti? One less thing for the user to keep track of (and possibly get wrong).
Gavino is offline   Reply With Quote
Old 8th August 2008, 17:12   #947  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 5,045
Quote:
Originally Posted by Gavino View Post
Would it be possible to store the value of refframes along with the data inside "MVForBack", so that it need not be passed as a parameter to MVDegrainMulti? One less thing for the user to keep track of (and possibly get wrong).
That doesn't work in situations where you use the same vectorset for two MVDegrainMultis with different temporal radii.. One solution would be to use the value used in MVAnalyseMulti unless specified otherwise.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 8th August 2008, 17:33   #948  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,412
Quote:
Originally Posted by Boulder View Post
That doesn't work in situations where you use the same vectorset for two MVDegrainMultis with different temporal radii.. One solution would be to use the value used in MVAnalyseMulti unless specified otherwise.
I wasn't sure if this was allowed (different radii). Presumably MVDegrainMulti complains if you ask for a greater radius than the one used in MVAnalyseMulti?
Gavino is offline   Reply With Quote
Old 8th August 2008, 17:38   #949  |  Link
josey_wells
Registered User
 
Join Date: Dec 2007
Location: VA, USA
Posts: 116
Actually both of you bring up a good point. I do not check now for a differing refframes than the original set created from MVAnalyseMulti.

I could see where one would want to create an MVAnalyseMulti set with say refframes=3 but only use temporal radius 2, i.e. refframes=2 in MVDegrainMulti.

I will have to make these fixes.

On second though I could default refframes to 32 and if it is larger than the set passed then use the entire set from MVAnalyseMulti. If a smaller refframes was specified then the subset would be utilized.

Last edited by josey_wells; 8th August 2008 at 17:53. Reason: added comment
josey_wells is offline   Reply With Quote
Old 8th August 2008, 17:56   #950  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 5,045
Quote:
Originally Posted by josey_wells View Post
Actually both of you bring up a good point. I do not check now for a differing refframes than the original set created from MVAnalyseMulti.

I could see where one would want to create an MVAnalyseMulti set with say refframes=3 but only use temporal radius 2, i.e. refframes=2 in MVDegrainMulti.

I will have to make these fixes.

On second though I could default refframes to 32 and if it is larger than the set passed then use the entire set from MVAnalyseMulti. If a smaller refframes was specified then the subset would be utilized.
So the current version forces for example 3 refframes in MVDegrainMulti if that is set in MVAnalyseMulti?
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 8th August 2008, 18:00   #951  |  Link
josey_wells
Registered User
 
Join Date: Dec 2007
Location: VA, USA
Posts: 116
Quote:
So the current version forces for example 3 refframes in MVDegrainMulti if that is set in MVAnalyseMulti?
In the current version refframes must be specified in both MVAnalyseMulti and MVDegrainMulti and must have the same value in each.

I propose to change so that refframes is specified in MVAnalyseMulti and if unspecified in MVDegrainMulti then will be the same as used MVAnalyseMulti, i.e. entire set. If specified and smaller thatn refframes used in MVAnalyseMulti then the subset of smaller temporal radius would be used.
josey_wells is offline   Reply With Quote
Old 8th August 2008, 18:05   #952  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 5,045
Yes, that sounds like a good idea
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 8th August 2008, 19:55   #953  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
josey_wells,
You make a real progress with new functions MVDegrainMulti and MVAnalyseMulti! It is really faster!
With new Multi syntax, the mvtools lose some ...universality (do not know right word). But speed is more important! However, the clear doc is very important too.

If your version is a final, what is state of other functions?

is it possible to mix multi and non-multi mvanalyse with other functions, is it possible to have several MVDegrainMulti in script, may idx be same.
is some functions compatible with SetMTMode or MT()?

As I understand, ...Multi functions is now free from thead locking. Other (non-multi) functions (MVFlowFPS, etc) is still not, right ?
In this case, we should use for these (other) functions the v.1.9.5.7 threading implementation (it works fine for most people besides Mutant_Fred).
Is it possible to revert all your threading changes for old functions and combine it with new MVDegrainMulti and MVAnalyseMulti?
IMO, it would be the optimal way of MVTools evolution.
If it is possible, then your version is not final.
If it is not possible, then your version is not final too

as for version number, v1.10 is seems most appropriate.
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 9th August 2008, 00:27   #954  |  Link
Spuds
dumber every day
 
Spuds's Avatar
 
Join Date: Dec 2006
Location: Planet Earth
Posts: 154
@josey_wells,

First awesome work, thanks for sticking with it! I have not been able to get it to crash or freeze

Now of course I have a question.

Are the idx values between MVAnalyseMulti and MVAnaylse completely separate or do they refer to the same pool. For example if I did a

forback=input.MVAnalyseMulti(refframes=2, idx=1)

would a

for=input.mvanalyse(isb=false,delta=1,idx=1)

be able to retrieve the vector frame that MVAnalyseMulti already computed or does it recompute the frame since its in a different function? If different is there a way to get specific vector frames out of the MVAnalyseMultii array?
Spuds is offline   Reply With Quote
Old 9th August 2008, 10:40   #955  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
Quote:
Originally Posted by josey_wells View Post

For example:
Code:
backward_vec2 = source.MVAnalyse(isb = true, lambda = 1000, delta = 2)
backward_vec1 = source.MVAnalyse(isb = true, lambda = 1000, delta = 1)
forward_vec1 = source.MVAnalyse(isb = false, lambda = 1000, delta = 1)
forward_vec2 = source.MVAnalyse(isb = false, lambda = 1000, delta = 2)
can be replaced by
Code:
ForBackVec = source.MVAnalyseMulti(refframes=2, lambda = 1000)
Is there a way to use the ForBacVec thing in MVFlowInter?

I'm not exactly skilled with MVtools and I'm trying to modify MCBob to work with this. So far the obstacle is trying get MVFlowInter to use this.
__________________
Pirate: Now how would you like to die? Would you like to have your head chopped off or be burned at the stake?

Curly: Burned at the stake!

Moe: Why?

Curly: A hot steak is always better than a cold chop.
Revgen is offline   Reply With Quote
Old 9th August 2008, 11:45   #956  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 5,045
Quote:
Originally Posted by Revgen View Post
Is there a way to use the ForBacVec thing in MVFlowInter?

I'm not exactly skilled with MVtools and I'm trying to modify MCBob to work with this. So far the obstacle is trying get MVFlowInter to use this.
AFAIK, at the moment it can only be used in MVDegrainMulti.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 9th August 2008, 13:32   #957  |  Link
halsboss
likes to tinker
 
Join Date: Jan 2004
Location: girt by sea
Posts: 635
Quote:
Originally Posted by Revgen View Post
Is there a way to use the ForBacVec thing in MVFlowInter?

... I'm trying to modify MCBob to work with this.
You beaut. Is this http://forum.doom9.org/showthread.ph...63#post1055263 the version/thread you're hoping to modify (MCBob+NNEDI) ? I did what you said in that thread and added "threads=4" to all the NNEDI calls in MCBob and it went from 0.3 fps to 0.8 fps (1080i source, Q9450 quad-core CPU) ... any MCBob speedup would be a magnificent achievement.
halsboss is offline   Reply With Quote
Old 9th August 2008, 14:05   #958  |  Link
josey_wells
Registered User
 
Join Date: Dec 2007
Location: VA, USA
Posts: 116
Wow, lots of questions. Let me address the questions in order and then wrap up with talk about a new version that I believe will make this applicable to everyone until more functions can be modified.

Quote:
If your version is a final, what is state of other functions?
All non multi functions work as they used to except for the MVDegrain1, 2, 3 which still have the multithreaded LumaY, ChromaU, and Chroma V threaded processing.

Quote:
is it possible to mix multi and non-multi mvanalyse with other functions, is it possible to have several MVDegrainMulti in script, may idx be same.
is some functions compatible with SetMTMode or MT()?
1. It used not to be possible to mix mulit mvanalyse with other functions but this is no longer the case and will be explained below.
2. It is possible to have multiple MVDegrainMulti and multiple MVAnalyseMulti in the same script. The idx values may be different or the same and work as before.
3. All new functions should be just as compativle with SetMTMode or MT() and stability using these functions would be the same as before. For higher stability using the newer AviSynth without SetMTMode or MT is recommended.

Quote:
As I understand, ...Multi functions is now free from thead locking. Other (non-multi) functions (MVFlowFPS, etc) is still not, right ?
In this case, we should use for these (other) functions the v.1.9.5.7 threading implementation (it works fine for most people besides Mutant_Fred).
Is it possible to revert all your threading changes for old functions and combine it with new MVDegrainMulti and MVAnalyseMulti?
I have removed all thread locking from all the functions so that they are the same as before. The new functions are also not thread locked. All frame getting is done on the main line thread and processing is done in spun threads which guarantees independence and does not require thread locking.

Quote:
as for version number, v1.10 is seems most appropriate.
IMO I'm still partial to V2.0 when all the issues are resolved.

I spent most of the day yesterday tracing an excessive memory usage condition and making some more enhancements. The memory usage problem turned out to be that if you pass pointers which increment a refcount to a thread which doesn't die between frame calls then the thread will hold references to frames no longer needed between calls and substantially increase memory usage. I have corrected by freeing pointers in the threads.

Now for the good stuff.

First it is now no longer necessary to add refframes to MVDegrainMulti. It will automatically know how many temporal frames were passed to it and perform the appropriate denoising. For example if MVAnalyseMulti had refframes=3 then MVDegrainMulti would be equivalent to MVDegrain3. If a smaller refframes is passed then a reduced temporal denoising can be done. If refframes=2 was specified in MVDegrainMulti then a MVDegrain2 will be performed even though MVAnalyseMulti was refframes=3.

Code:
MVMulti=c.MVAnalyseMulti(refframes=iLevel, pel=iPel, blksize=iBlksize, overlap=iOverlap, chroma=bChroma, truemotion=bTrueMotion, search=iSearch, sharp=iSharp, dct=iDct, idx=iIdx1)
last.MVDegrainMulti(MVMulti, thSAD=iTHSAD, idx=iIdx1)
Now before I go on with the next feature a little explanation is in order. MVAnalyseMulti will always produce forward and backward frames the following way.

MVAnalyseMulti with refframes 3 will produce the following frames corresponding to the array elements.
0 - Backward 3, 1 - Backward 2, 2 - Backward 1, 3 - Forward 1, 4 - Forward 2, 5 - Forward 3
And so on for higher refframes

I have added a MVMultiExtract Function which allows one to extract the regular MVAnalyse frames from a multi function call to be used in other MVTools functions until they can be modified to accept MVAnalyse directly. Thus everyone can take advantage of MVAnalyseMulti which seems to take the most processing power.

For example.

Code:
MVMulti=c.MVAnalyseMulti(refframes=1, pel=iPel, blksize=iBlksize, overlap=iOverlap, chroma=bChroma, truemotion=bTrueMotion, search=iSearch, sharp=iSharp, dct=iDct, idx=iIdx1)
back1=MVMulti.MVMultiExtract(0)
for1=MVMulti.MVMultiExtract(1)
last.MVDegrain1(back1, for1, thSAD=iTHSAD, idx=iIdx1)
This shows how the individual frames can be extracted and used in the old MVDegrain1. This can be carried to other MVTools functions.

So finally here it is V1.9.6.1:

http://rapidshare.com/files/13604247...9.6.1.rar.html

There are several versions in here use the one most appropriate for your system.

MSVC08 - Everyone
IC10-P4-SSE2 - Pentium 4 or above SSE2
IC10-P4-SSE3 - Pentium 4 or above with SSE3
IC10-Core2-SSE3 - Core 2 duo or above with SSE3

Now that the oylmpics are on I need a break, plus my DVD collection is backing up.

Have fun!

Last edited by josey_wells; 9th August 2008 at 15:22.
josey_wells is offline   Reply With Quote
Old 9th August 2008, 14:37   #959  |  Link
krieger2005
Registered User
 
krieger2005's Avatar
 
Join Date: Oct 2003
Location: Germany
Posts: 377
You made big improvements to the MVTools. For future development maybe it is intresting to note the development in avisynth:
http://forum.doom9.org/showthread.php?t=139629

So if i understand the result of the idea of IanB right it should make just the same as you have realized with the MVAnalyseMulti-Function. So if this will be reality in near future such functions should't be usefull then?
krieger2005 is offline   Reply With Quote
Old 9th August 2008, 15:00   #960  |  Link
josey_wells
Registered User
 
Join Date: Dec 2007
Location: VA, USA
Posts: 116
Quote:
Originally Posted by krieger2005 View Post
So if i understand the result of the idea of IanB right it should make just the same as you have realized with the MVAnalyseMulti-Function. So if this will be reality in near future such functions should't be usefull then?
This isn't necessarily true. What I have done is multithread a single line flow through a script.

The new AviSynth is looking to multithread individual frames, i.e. running the script in parallel for different frame numbers. There exists lots of issues with this with scripts that depend on previous/forward frames. AviSynth is the place to perform this since it has control of the frame getting mechanism which is why I had to go the way I did.

When AviSynth makes these modifications as well as pipeline improvements both modifications can exist together and benifit.
josey_wells is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 03:47.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.