Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 28th February 2008, 22:48   #281  |  Link
Nikos
Registered User
 
Join Date: Jun 2002
Location: Greece
Posts: 236
To help a little.
( x - y ) / abs( x - y ) = sign(x-y)

In masktools Round(1.5)=? and Round(-1.5)=?
I suppose 2 and -1, but i am not sure.
__________________
Greece PAL User...
Nikos is offline   Reply With Quote
Old 28th February 2008, 23:22   #282  |  Link
Terranigma
*Space Reserved*
 
Terranigma's Avatar
 
Join Date: May 2006
Posts: 953
I think you guys may want to split this thread into a new one, because it's now getting way offtopic
Terranigma is offline   Reply With Quote
Old 3rd March 2008, 10:22   #283  |  Link
NathanX
Registered User
 
Join Date: Apr 2005
Posts: 13
Quote:
Originally Posted by Sagekilla View Post
Please don't use my FastDegrain -- It's currently b0rked. Also, Sasovics please don't cross post.
Your FastDegrain scripts sounds interesting to me, is it still in development?
Thank you!
NathanX is offline   Reply With Quote
Old 3rd March 2008, 15:55   #284  |  Link
ChrisW77
Registered User
 
Join Date: Sep 2006
Posts: 72
Sagekilla, Thats one amazing script that works extremely well on SD material, and heck, even on old VHS it does wonders.

So thanks to both you and Didee

I'm currently using Temporal Degrain V1.18 (Feb 14, 2008), but cannot seem to get any joy from MT, CPU doesn't go above 60% usage, no matter where I put SetMTmode(2) in a simple script.

For instance

Code:
SetMemoryMax(512)
LoadPlugin("C:\Program Files\AviSynth\plugins\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth\plugins\mpasource.dll")
Import("C:\Program Files\AviSynth\plugins\TemporalDegrain2.avs")

SetMTmode(2)
video = mpeg2source("D:\DVB-T\test1.d2v", cpu=4, iPP=true, idct=3)
audio = MPASource("D:\DVB-T\test1 T01 DELAY 0ms.mpa", normalize = false)
AudioDub(video, audio)
ConvertToYV12(interlaced=true)

AssumeTFF()
SeparateFields()
TemporalDegrain()
Weave()
Crop(8,8,-8,-8,align=true).Addborders(8,8,8,8)
This is just a simple DVB-T capture test, keeping interlaced full frame, and on my Core2duo E6300 I get around 3 fps with or without SetMTmode(2).

One thing I have tried, as I have a powerful GPU (8800GT), that gained me 13fps from 3fps to 16fps with out any perceived loss in quality, is to modify your function as follows

Change all FFT3D entries to FFT3dGPU, changing BT=5 to BT=4, as the GPU version doesn't support 5, and blksize from 8 to 16. Huge speedup, and I couldn't see any difference in quality.

Many thanks for this function, it's amazing.
ChrisW77 is offline   Reply With Quote
Old 3rd March 2008, 19:09   #285  |  Link
archaeo
scribe
 
archaeo's Avatar
 
Join Date: Apr 2004
Location: canyons
Posts: 530
Quote:
Originally Posted by ChrisW77 View Post
I'm currently using Temporal Degrain V1.18 (Feb 14, 2008), but cannot seem to get any joy from MT, CPU doesn't go above 60% usage, no matter where I put SetMTmode(2) in a simple script.

Try MT("TemporalDegrain()").
It worked well for me - I am seeing 90% plus usage on my Core2
archaeo is offline   Reply With Quote
Old 3rd March 2008, 19:20   #286  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,815
I wonder if MVAnalyse/MVDegrain plays nice with MT..nobody ever tested it IIRC.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 3rd March 2008, 20:17   #287  |  Link
ChrisW77
Registered User
 
Join Date: Sep 2006
Posts: 72
Quote:
Try MT("TemporalDegrain()").
It worked well for me - I am seeing 90% plus usage on my Core2
Thanks, I'll give that a try.
ChrisW77 is offline   Reply With Quote
Old 3rd March 2008, 20:21   #288  |  Link
Terranigma
*Space Reserved*
 
Terranigma's Avatar
 
Join Date: May 2006
Posts: 953
Quote:
Originally Posted by Boulder View Post
I wonder if MVAnalyse/MVDegrain plays nice with MT..nobody ever tested it IIRC.
I don't know what this is exactly, but I stumbled across it in the MT thread. :P
Terranigma is offline   Reply With Quote
Old 3rd March 2008, 20:44   #289  |  Link
maxisvk
Registered User
 
Join Date: Aug 2007
Location: Rimini, Italy
Posts: 4
Quote:
Originally Posted by ChrisW77 View Post
Thanks, I'll give that a try.
This is my test on Core 2 Duo with TemporalDegrain()
SetMTMode(2,0) , 50% Cpu
SetMTMode(2,3) , 65% Cpu
SetMTMode(2,4) , 90/100% Cpu

on Q6600 is the same, only if i use SetMTMode(2,8) it work at 100%
maxisvk is offline   Reply With Quote
Old 3rd March 2008, 21:01   #290  |  Link
ChrisW77
Registered User
 
Join Date: Sep 2006
Posts: 72
Quote:
SetMTMode(2,4) , 90/100% Cpu
Yes, yes, now that worked well, getting 95% continuous. Cheers Never thought of using 4 threads.

A strange thing I found with MVTools was this

Code:
SetMemoryMax(512)
LoadPlugin("C:\Program Files\AviSynth\plugins\RemoveGrainSSE2.dll")
LoadPlugin("C:\Program Files\AviSynth\plugins\MVTools.dll")
LoadPlugin("C:\Program Files\AviSynth\plugins\DeGrainMedian.dll")
LoadPlugin("C:\Program Files\AviSynth\plugins\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth\plugins\mpasource.dll")

SetMTmode(2)
video = mpeg2source("D:\VHS\90sadverts.d2v", cpu=6, iPP=true, idct=3)
audio = MPASource("D:\VHS\90sadverts T01 DELAY 0ms.mpa.d2a", normalize = false)
AudioDub(video, audio)

source=ConvertToYV12(interlaced=true).Crop(10,4,-10,-10,align=true).Addborders(10,7,10,7)
fields=source.AssumeTFF().SeparateFields().DeGrainMedian(mode=0,interlaced=false).RemoveGrain(mode=2)

lbda=512
blk=16
ol=8
sch=2
pel=1

backward_vec2 = fields.MVAnalyse(isb=true, search=sch, truemotion=true, lambda=lbda, delta=2, pel=pel, blksize=blk, overlap=ol, sharp=1, idx=1)
backward_vec1 = fields.MVAnalyse(isb=true, search=sch, truemotion=true, lambda=lbda, delta=1, pel=pel, blksize=blk, overlap=ol, sharp=1, idx=1)
forward_vec1 = fields.MVAnalyse(isb=false, search=sch, truemotion=true, lambda=lbda, delta=1, pel=pel, blksize=blk, overlap=ol, sharp=1, idx=1)
forward_vec2 = fields.MVAnalyse(isb=false, search=sch, truemotion=true, lambda=lbda, delta=2, pel=pel, blksize=blk, overlap=ol, sharp=1, idx=1)
fields.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=800,idx=1).MVDenoise(backward_vec2,backward_vec1,forward_vec1,forward_vec2,tht=10,thSAD=800)
Weave()
That script ran well and fast, constantly at 100%, until it finished. Yet, add anything after weave, such as LimitedSharpenFaster, or even Tweak, and it kind of broke MT, and you were left with a script running 55%.

Cheers to maxisvk, that worked well.
Next, to look at a cheap quad core upgrade
ChrisW77 is offline   Reply With Quote
Old 3rd March 2008, 21:13   #291  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,394
Quote:
Originally Posted by ChrisW77 View Post
That script ran well and fast, constantly at 100%, ...
... and the result was 100% garbage.

Quote:
Originally Posted by ChrisW77
Code:
source=ConvertToYV12(interlaced=true).Crop(...).Addborders(10,7,10,7)
Progressive YV12 shouldn't be cropped or add-border'ed by odd numbers vertically .
For interlaced YV12, that is MUST NOT. Never, never, never ever. If you do, you're changing chroma phase, i.e. luma and chroma are no more temporally aligned.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 3rd March 2008, 21:41   #292  |  Link
ChrisW77
Registered User
 
Join Date: Sep 2006
Posts: 72
Quote:
... and the result was 100% garbage.
Looked good enough to MY eyes

Quote:
Progressive YV12 shouldn't be cropped or add-border'ed by odd numbers vertically .
For interlaced YV12, that is MUST NOT. Never, never, never ever. If you do, you're changing chroma phase, i.e. luma and chroma are no more temporally aligned.
Then suggest something then, rather than just spouting off one.
How would one, by Didee's standards, crop off the shite on a Interlaced VHS cap.
I'm listening, o Great one

Last edited by ChrisW77; 3rd March 2008 at 21:42. Reason: add interlace
ChrisW77 is offline   Reply With Quote
Old 4th March 2008, 08:10   #293  |  Link
thetoof
Sleepy overworked fellow
 
Join Date: Feb 2008
Location: Maple syrup's homeland
Posts: 933
Basic rule of MT : Look at the time it takes, not the CPU usage!
Also, I think setmtmode(2) for MVtools would be better since:
1- In the guide, there is this info: # Note: SetMTMode(2) mode of multithreaded AviSynth is also supported since MVTools v.1.8.4.1 (beta testing)
2-MT("filter()") splits the frames in multiple parts (depending on the # of threads)... and unless you use a HUGE overlap, the motion vectors won't be as accurate as if you had used the complete frame (which is done by setmtmode(2))

If you didn't know, setmtmode(2) separates the frames temporally (so it's always the complete frame that is processed), while MT() separates them spatially.
thetoof is offline   Reply With Quote
Old 4th March 2008, 13:43   #294  |  Link
maxisvk
Registered User
 
Join Date: Aug 2007
Location: Rimini, Italy
Posts: 4
Quote:
Originally Posted by thetoof View Post
Basic rule of MT : Look at the time it takes, not the CPU usage!
ok, obviously, but in my test:

SetMTMode(2,0) , 50% Cpu
-> encoded 120 frames, 1.61 fps, 855.64 kb/s

SetMTMode(2,3) , 65% Cpu
-> encoded 120 frames, 1.64 fps, 855.64 kb/s

SetMTMode(2,4) , 90/100% Cpu
-> encoded 120 frames, 2.15 fps, 855.64 kb/s

i also noticed that using only Mvdegrain and not TemporalDegrain best result are stuck with SetMTMode(2,3), Why?..., I really did not know!
maxisvk is offline   Reply With Quote
Old 4th March 2008, 22:35   #295  |  Link
thetoof
Sleepy overworked fellow
 
Join Date: Feb 2008
Location: Maple syrup's homeland
Posts: 933
Because Temporaldegrain uses a predenoised reference clip for motion vector search... so it takes more computer ressources and it's very possible your hard drive disk can't handle the high number of reads and writes. That's why there's a peak with speed. You can "solve" that by doing the reference clip yourself with FFT3Dfilter.

If you only want to denoise, use this:

1 - Create a simple script to create your reference clip, it doesn't really matter if you lose some details and it looks a tad overdenoise since it's only a clip used for motion vector search:

whateversource("yoursource")
FFT3DFilter(your favorite parameters... check the documentation for more info)

2 - Process through Direct Stream Copy in virtualdubmod. If you have another HDD, save it on this one. The file'll be HUGE.

3 - create another script with temporaldegrain using the source clip that is on the other HDD

source = whateversource("youroriginalclip")
denoised = avisource("thereferenceclip")
temporaldegrain(source, denoised)

About your speedtest... it's OBVIOUS that using more threads will increase the speed. SetMTmode(2) is always the best choice since it'll set the number of threads to the # of available processors.

When I say to look at speed instead of CPU usage, it's with the same # of CPUs. The thing is, when you have a more complex script, MT may not help (I have a script that runs @ 1fpm with 750MB of RAM and 25% CPU usage (100% of 1 processor on my Quad-core) and it's a LOT slower with MT and uses 3GB of RAM).
thetoof is offline   Reply With Quote
Old 5th March 2008, 07:32   #296  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,487
Eh, no matter how slow a script is, writing part of it to disk and reading it back will always be slower. Whether you're going to make another lossless encode from the final output, or use it in a one-pass encoding, by the time you're done you'll have put much more effort into it and it will have taken longer overall. Memory overflow and disk swapping is about the only way that partial scripting can be faster. TemporalDegrain can't even remotely cause a hard drive bottleneck anyway, even with uncompressed sources it takes about a hundred times as long to filter as read.

On the other hand, if you have a second computer handy, running fft3dfilter on that and using TCPSource can really speed things up, as long as the second computer's fast enough to keep up.

I'd also say that your conclusion that SetMTmode(2) is best is mistaken; after all, you just said more threads was faster, and speed is all that matters, as long as the final encoder's thread is included rather than just the avisynth script measured. In some scripts 4 threads will be faster, others 1, though I suppose yours might be swapping like crazy once it gets there. Always measure though, never take a forum poster's word.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order.
foxyshadis is offline   Reply With Quote
Old 5th March 2008, 08:39   #297  |  Link
thetoof
Sleepy overworked fellow
 
Join Date: Feb 2008
Location: Maple syrup's homeland
Posts: 933
Oh well, seems like I was lucky on my first try using the method I described before... I ran some other tests to show how effective it was, but everything I got looked like crap.
Dunno why it worked once (and there were actually great speed improvement by outputting the reference clip on another HDD and then call it in the script using temporalgrain as "denoise", from 53 mins total (complete script) to 13 mins total (split script) for the same amount of frames) and I'm sorry for posting it before running more tests... but heh, I'm learning!

Also, about my statement regarding setmtmode(2), I had in mind that it was regarding MVtools usage (and from what I understood from its documentation, it's the most appropriate mode). My bad for not being clear.

Last edited by thetoof; 5th March 2008 at 09:08.
thetoof is offline   Reply With Quote
Old 6th March 2008, 20:46   #298  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by foxyshadis View Post
Eh, no matter how slow a script is, writing part of it to disk and reading it back will always be slower. Whether you're going to make another lossless encode from the final output, or use it in a one-pass encoding, by the time you're done you'll have put much more effort into it and it will have taken longer overall. Memory overflow and disk swapping is about the only way that partial scripting can be faster. TemporalDegrain can't even remotely cause a hard drive bottleneck anyway, even with uncompressed sources it takes about a hundred times as long to filter as read.
yup 1 pass encodes it would not help but 2 pass encoding is much faster when I do twriteavi() on the first pass of slow encode due to scripts like the stuff in this thread.

If you are only getting 2 FPS like me you for sure want to save out a lossless during the first pass else you are stuck doing it all again on the second pass. resize, decimation, MVtools, fft3dfilter, etc... I go from 2 FPS on the first pass to 14 FPS on the second pass and this is now with the real encoding being done also. (it would be only about .75 FPS on pass 2 if I didn't do this) huge time savings on 2 pass encodes.

no I would not do it like thetoof did it since you want the lossless (I use VBLE cause it is fast and thus little CPU overhead) made the same time as first pass stats are being made
Zep is offline   Reply With Quote
Old 7th March 2008, 07:49   #299  |  Link
g-force
Guest
 
Posts: n/a
My very first Function

TemporalDegrain looks really nice, but my PC is way to slow to run this thing practically. So I used Didee's limiting approach and wrote something that required less MV operations with very good results IMHO. Let me know what you think!

Quote:
# GTemporalDegrainFaster ver.1.00 6MAR08
# Function by G-force,"Limited" Concept by Didee
# Requires FFT3DFilter.dll, mt_masktools, MVtools.dll, RemoveGrain.dll

function GTemporalDegrainFaster (clip input,int "threads",int "sigma")
{
source = input
ncpu = default(threads,1) #max number of CPU threads to use in FFT calculation (int>0, default=1)
sigma = default(sigma, 16) #turn down for more limited denoising
s2 = floor (sigma * 0.625)
s3 = floor (sigma * 0.375)
s4 = floor (sigma * 0.250)

filter = source.fft3dfilter(ncpu=ncpu,sigma=sigma,sigma2=s2,sigma3=s3,sigma4=s4,bt=5,bw=16,bh=16,ow=8,oh=8)
filterD = mt_makediff(source,filter)

backward_vec1= filter.MVAnalyse(isb=true, delta=1,pel=2,overlap=4,sharp=1,idx=1)
forward_vec1 = filter.MVAnalyse(isb=false,delta=1,pel=2,overlap=4,sharp=1,idx=1)
bw1 = source.MVCompensate(backward_vec1,idx = 2) #idx = 2 since working on a different clip than MVAnalyse
fw1 = source.MVCompensate(forward_vec1, idx = 2)

#nice thing about this approach is that it doesn't necessarily use any pixels from current frame
#this gets rid of a lot more dirt, scratches etc. (things that last only one frame)
nr = Interleave(bw1,source,fw1).Clense().SelectEvery(3,1)

# Limit "nr" to not do more than what "filter" would do. -Didee
nrD = mt_makediff(source,nr)
DD = mt_lutxy(filterD,nrD,"x 128 - abs y 128 - abs < x y ?")
source.mt_makediff(DD,U=2,V=2)

output = last
return(output)
}
-G

Last edited by g-force; 7th March 2008 at 07:57.
  Reply With Quote
Old 8th March 2008, 20:43   #300  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,394
Quote:
Originally Posted by ChrisW77 View Post
Quote:
... and the result was 100% garbage.
Looked good enough to MY eyes
I don't know your eyes, so I can't tell what seems good enough to them and what not.

With your sequence of crop+addborders, this is what happens to an interlaced YV12 source:




In contrast, with correct treatment you get:




Quote:
Then suggest something then, rather than just spouting off one.
How would one, by Didee's standards, crop off the shite on a Interlaced VHS cap.
I'm listening, o Great one
It's a basic technical requirement that the top border of interlaced YV12i may only be altered in mod4 steps.
This has nothing-at-all to do with "my standards".

-----

Generating script: (yeah, it's not elegant)

Code:
base = blankclip(width=704,height=576,pixel_type="YV12",fps=50.000,color=$408080)

base = base.subtitle("This is some scrolling text ...",size=40,x=128,y=480)

interleave( base, base.crop(0,16*1,-0,-0).addborders(0,0,0,16*1,color=$408080), 
 \                base.crop(0,16*2,-0,-0).addborders(0,0,0,16*2,color=$408080), 
 \                base.crop(0,16*3,-0,-0).addborders(0,0,0,16*3,color=$408080), 
 \                base.crop(0,16*4,-0,-0).addborders(0,0,0,16*4,color=$408080), 
 \                base.crop(0,16*5,-0,-0).addborders(0,0,0,16*5,color=$408080), 
 \                base.crop(0,16*6,-0,-0).addborders(0,0,0,16*6,color=$408080), 
 \                base.crop(0,16*7,-0,-0).addborders(0,0,0,16*7,color=$408080), 
 \                base.crop(0,16*8,-0,-0).addborders(0,0,0,16*8,color=$408080), 
 \                base.crop(0,16*9,-0,-0).addborders(0,0,0,16*9,color=$408080), 
 \                base.crop(0,16*10,-0,-0).addborders(0,0,0,16*10,color=$408080), 
 \                base.crop(0,16*11,-0,-0).addborders(0,0,0,16*11,color=$408080), 
 \                base.crop(0,16*12,-0,-0).addborders(0,0,0,16*12,color=$408080), 
 \                base.crop(0,16*13,-0,-0).addborders(0,0,0,16*13,color=$408080), 
 \                base.crop(0,16*14,-0,-0).addborders(0,0,0,16*14,color=$408080), 
 \                base.crop(0,16*15,-0,-0).addborders(0,0,0,16*15,color=$408080)  )

vid_50p = last

vid_50i = assumeTFF().SeparateFields().SelectEvery(4,0,3).Weave()


# the wrong way:
#---------------
  vid_50i
  crop(0,4,-0,-10)
  addborders(0,7,0,7)
  
  wrong = last
  
# the better way:
#---------------
  vid_50i
  crop(0,4,-0,-12)
  addborders(0,8,0,8)
  
  better = last
  

result_wrong = stackhorizontal( wrong.subtitle("interlaced after non-MOD4 cropping or addborders",y=20),
 \                              stackvertical( wrong.SeparateFields().selecteven().subtitle("--> top field",y=20),
 \                                             wrong.SeparateFields().SelectOdd().Subtitle("--> bottom field",y=20) ) )

result_better = stackhorizontal( better.subtitle("interlaced after correct (MOD4) cropping or addborders",y=20),
 \                               stackvertical(better.SeparateFields().selecteven().subtitle("--> top field",y=20),
 \                                             better.SeparateFields().SelectOdd().Subtitle("--> bottom field",y=20) ) )


result_wrong
#result_better

return( last )
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 04:11.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.