Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 15th July 2011, 04:31   #201  |  Link
mswaino2
Guest
 
Posts: n/a
Im liking the results from your script, I just dislike how GradFun3(smode=1) is so slow. I went from a 4 hour pass on a movie to a 12 hour pass. Is this supposed to be like that? because regular GradFun3() runs MUCH faster it just doesnt give quite as good of results as smode1.
  Reply With Quote
Old 15th July 2011, 07:27   #202  |  Link
cretindesalpes
͡҉҉ ̵̡̢̛̗̘̙̜̝̞̟̠͇̊̋̌̍̎̏̿̿
 
cretindesalpes's Avatar
 
Join Date: Feb 2009
Location: No support in PM
Posts: 610
Quote:
Originally Posted by mswaino2 View Post
I just dislike how GradFun3(smode=1) is so slow. I went from a 4 hour pass on a movie to a 12 hour pass. Is this supposed to be like that?
smode=1 is indeed slower than smode=0, but not that slower. Something must be wrong. Are you sure your figures are right? Anyway, if you do a 2-pass encoding, render first to a lossless file then encode it, you'll save one avisynth pass.
__________________
dither 1.27.2 for AviSynth | avstp 1.0.3 for AviSynth development | fmtconv r19 for Vapoursynth | trimx264opt segmented encoding
cretindesalpes is offline   Reply With Quote
Old 15th July 2011, 17:34   #203  |  Link
mswaino2
Guest
 
Posts: n/a
Quote:
Originally Posted by cretindesalpes View Post
smode=1 is indeed slower than smode=0, but not that slower. Something must be wrong. Are you sure your figures are right? Anyway, if you do a 2-pass encoding, render first to a lossless file then encode it, you'll save one avisynth pass.
I dunno, maybe its just Im using it in the script wrong or something. Could it be because Im using a denoiser and sharpener along with it?

I been doing a 2 pass encode with this for a script

Quote:
LoadPlugin("C:\Program Files\megui\tools\dgindex\DGDecode.dll")

Import("C:\Program Files\AviSynth 2.5\plugins\LimitedSharpenFaster.avsi")

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools-25.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\dither.avsi")
Import("C:\Program Files\AviSynth 2.5\plugins\GradFun2DBmod.avsi")
Import ("C:\Program Files\AviSynth 2.5\plugins\mt_xxpand_multi.avsi")
DGDecode_mpeg2source("C:\Users\Owner\Desktop\Case Closed - The Last Wizard Of The Century\VTS_06_1.d2v", info=3)
LoadPlugin("C:\Program Files\Megui\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)
#deinterlace
crop( 0, 0, -2, 0)
LanczosResize(852,480) # Lanczos (Sharp)
LoadPlugin("C:\Program Files\megui\tools\avisynth_plugin\UnDot.dll")
Undot() # Minimal Noise
deen()
LimitedSharpenFaster(ss_x=2.0, ss_y=2.0, strength=150, overshoot=0, undershoot=0, soft=0, edgemode=0)
GradFun3(smode=1)
__film = last
__t0 = __film.trim(0, 142520)
__t0

the only thing I have changed is the script and its taking a very long time for some reason. I was getting 11-12 FPS now Im at 3 FPS after adding it.

Last edited by mswaino2; 15th July 2011 at 22:24. Reason: added detail
  Reply With Quote
Old 16th July 2011, 10:50   #204  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 949
cretindesalpes: Is it possible to add lsb_in feature in mvtools2. It would be very useful for certain sources with blocking+noise, where I would run a spatial dfttest pass then mdegrain. I read that normally spatial denoisers should go before temporal.
Dogway is offline   Reply With Quote
Old 16th July 2011, 22:21   #205  |  Link
mandarinka
Registered User
 
mandarinka's Avatar
 
Join Date: Jan 2007
Posts: 678
You can use a prefilter for the motion vector search (which I assume is completely fine to do in 8bit), to partially achieve that.


Something like this I think (hopefuly I got it right).
Code:
prefilt = FFT3Dgpu(sigma=2,bw=32,bh=32,bt=3,ow=16,oh=16,plane=4)
 
superfilt = prefilt.MSuper(pel=2, sharp=1)
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(superfilt, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(superfilt, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(superfilt, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(superfilt, isb = false, delta = 2, overlap=4)
 
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=300,lsb=true)
mandarinka is offline   Reply With Quote
Old 17th July 2011, 08:37   #206  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 949
Hello mandarinka. Thanks for the input, I meant to denoise, not feed a better suited (prefiltered) clip for motion estimation. I know that lsb_in in mvtools is going to be quite annoying for cretindesalpes to make since he would probably need to adapt all the functions (manalyse,mmask,mcompensate,etc) for lsb_in in mdegrain. But I think it makes sense because spatial+temporal denoisers normally operate internally first spatial then temporally (dfttest,tnlmeans,etc)

By the way I found probably a bug in Dither_resize16 when processing a .png file. I tested and the error comes from the fh and fv parameters when resize dimensions are small:
This is my code, pitty I couldn't take advantage of high bitdepth rgb to yuv conversion : P
Code:
converttoyv12(matrix="PC.601")
MT("""Dither1Pre(flt="tnlmeans()",stacked=true)""",2,2,true)
dither_resize16(224,316,kernel="spline64",fh=1.1,fv=1.1)
Dither_convert_yuv_to_rgb(lsb_in=true,tv_range=false,output="rgb32",mode=6,ampn=0.5)

Last edited by Dogway; 17th July 2011 at 09:09.
Dogway is offline   Reply With Quote
Old 17th July 2011, 12:25   #207  |  Link
cretindesalpes
͡҉҉ ̵̡̢̛̗̘̙̜̝̞̟̠͇̊̋̌̍̎̏̿̿
 
cretindesalpes's Avatar
 
Join Date: Feb 2009
Location: No support in PM
Posts: 610
Quote:
Originally Posted by Dogway View Post
I know that lsb_in in mvtools is going to be quite annoying for cretindesalpes
Yes it would... but I could just leave the analysis part in 8 bits and focus on MSuper/MDegrain. Still, it's a lot of work, so I don't plan to do this in the near future.

Quote:
By the way I found probably a bug in Dither_resize16 when processing a .png file.
I can't reproduce it here, everything looks OK. What is the bug, exactly? What are the dimensions of the original picture?
__________________
dither 1.27.2 for AviSynth | avstp 1.0.3 for AviSynth development | fmtconv r19 for Vapoursynth | trimx264opt segmented encoding
cretindesalpes is offline   Reply With Quote
Old 17th July 2011, 12:45   #208  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 949
Quote:
Originally Posted by cretindesalpes View Post
Still, it's a lot of work, so I don't plan to do this in the near future.
No problem, I knew it was. You hard work on Dither is already impressive for a one man.
Quote:
Originally Posted by cretindesalpes View Post
I can't reproduce it here, everything looks OK. What is the bug, exactly? What are the dimensions of the original picture?
Dimensions are 610x1114. I think its relevant because the other images work.
Dogway is offline   Reply With Quote
Old 17th July 2011, 20:03   #209  |  Link
cretindesalpes
͡҉҉ ̵̡̢̛̗̘̙̜̝̞̟̠͇̊̋̌̍̎̏̿̿
 
cretindesalpes's Avatar
 
Join Date: Feb 2009
Location: No support in PM
Posts: 610
Quote:
Originally Posted by Dogway View Post
Dimensions are 610x1114. I think its relevant because the other images work.
OK I spotted the problem, a tile size calculation error again, resulting in a buffer overflow.

Fixed (I hope so) in Dither 1.9.4.
__________________
dither 1.27.2 for AviSynth | avstp 1.0.3 for AviSynth development | fmtconv r19 for Vapoursynth | trimx264opt segmented encoding
cretindesalpes is offline   Reply With Quote
Old 18th July 2011, 21:02   #210  |  Link
TheProfosist
Registered User
 
TheProfosist's Avatar
 
Join Date: Aug 2009
Posts: 122
Quote:
Originally Posted by cretindesalpes View Post
smode=1 is indeed slower than smode=0, but not that slower. Something must be wrong. Are you sure your figures are right? Anyway, if you do a 2-pass encoding, render first to a lossless file then encode it, you'll save one avisynth pass.
im getting crazy slower speed with smode=1 as well

smode=0 1.87FPS
smode=1 0.12FPS
this is with 1080p

Last edited by TheProfosist; 19th July 2011 at 03:58.
TheProfosist is offline   Reply With Quote
Old 19th July 2011, 07:46   #211  |  Link
cretindesalpes
͡҉҉ ̵̡̢̛̗̘̙̜̝̞̟̠͇̊̋̌̍̎̏̿̿
 
cretindesalpes's Avatar
 
Join Date: Feb 2009
Location: No support in PM
Posts: 610
Quote:
Originally Posted by TheProfosist View Post
im getting crazy slower speed with smode=1 as well
Weird.

I ran a few speed tests with 1280x720 progressive input, in single-threading and multi-threading modes. I added a naked dfttest() because it's the core of GradFun3(smode=1) and I would suspect this is the one slowing down the whole thing. Also added GradFun2DBMod for reference.

Code:
#SetMTMode (5, 4)
FFVideoSource ("random 720p source")
#SetMTMode (2)
Trim (0, -1000)

# Select one of these lines
#NOP ()
#GradFun3 (smode=0, mask=0)
#GradFun3 (smode=0)
#GradFun3 (smode=1, mask=0)
#GradFun3 (smode=1)
#GradFun3 (smode=2, mask=0)
#GradFun3 (smode=2)
#dfttest (sigma=35, tbsize=1, sbsize=36, sosize=27, lsb=true)
#GradFun2DBmod ()

PointResize (32, 32)
# Encoded with: x264 --preset ultrafast --output NUL "speed.avs"
Results (Phenom II X4 965 3.4 GHz):

Code:
                             |  Single-threaded  | Multi-threaded x4
                             |     fps    CPU    |     fps    CPU
-----------------------------+-------------------+------------------
NOP ()                       |   453.93   33 %   |   444.44   33 %
GradFun3 (smode=0, mask=0)   |    30.72   27 %   |    89.63   90 %
GradFun3 (smode=0)           |     7.38   25 %   |    26.74   99 %
GradFun3 (smode=1, mask=0)   |     7.96   78 %   |     9.53   97 %
GradFun3 (smode=1)           |     4.38   55 %   |     7.65   97 %
GradFun3 (smode=2, mask=0)   |    10.03   26 %   |    32.80   98 %
GradFun3 (smode=2)           |     4.93   25 %   |    17.25   99 %
dfttest (...)                |     8.87   85 %   |    10.11   98 %
GradFun2DBmod ()             |     5.19   25 %   |    18.95   96 %
So, GradFun3(smode=1) is 3.5 times slower than GradFun3(smode=0) in 4x MT mode, and just 1.6 times slower in single-threaded mode.

I'll check later if I get very different figures for 1080p input.
__________________
dither 1.27.2 for AviSynth | avstp 1.0.3 for AviSynth development | fmtconv r19 for Vapoursynth | trimx264opt segmented encoding

Last edited by cretindesalpes; 19th July 2011 at 08:05.
cretindesalpes is offline   Reply With Quote
Old 19th July 2011, 14:02   #212  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 949
Another error: 1434x1476 dimensions for Dither_resize16 resizing to half. Hope you get this kind of problems sorted, sorry I can't help more.
Dogway is offline   Reply With Quote
Old 19th July 2011, 21:28   #213  |  Link
cretindesalpes
͡҉҉ ̵̡̢̛̗̘̙̜̝̞̟̠͇̊̋̌̍̎̏̿̿
 
cretindesalpes's Avatar
 
Join Date: Feb 2009
Location: No support in PM
Posts: 610
Quote:
Originally Posted by Dogway View Post
Another error: 1434x1476 dimensions for Dither_resize16 resizing to half. Hope you get this kind of problems sorted, sorry I can't help more.
I can't reproduce it. Is 1434x1476 the input size? Halving 1434 gives 717 which is odd. Could you please post the script?
__________________
dither 1.27.2 for AviSynth | avstp 1.0.3 for AviSynth development | fmtconv r19 for Vapoursynth | trimx264opt segmented encoding
cretindesalpes is offline   Reply With Quote
Old 19th July 2011, 23:14   #214  |  Link
Yellow_
Registered User
 
Join Date: Sep 2009
Posts: 378
I'm using Dither to go from 8bit h264AVC to 16bit OpenEXR image sequences but really those should be linear not gamma encoded, the compositing application I'm importing them into assumes linear. So does the Dither_y_gamma_to_linear function serve this purpose?

Is it only necessary to linearize the luma, more accurate and technically correct to do it that way before the conversion to RGB, rather than a typical 0.45 reverse gamma on all channels in RGB data after conversion?

Also as the source was encoded with a BT709 transfer curve not sRGB 2.2, to undo that ie: linearize YCbCr, I should be assuming something like the reciprocal of 2.35? If I understand correctly this helps prevent compressing shadow detail that can occur applying 0.45 to BT709 source?
Yellow_ is offline   Reply With Quote
Old 20th July 2011, 03:22   #215  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 949
This is the code I'm testing with:
Code:
Interleave(showred("yv12"), showgreen("yv12"), showblue("yv12")).Dither_convert_8_to_16
Dither_resize16(1434/2,1476/2,kernel="spline36",y=3,u=1,v=1)
The error is very ugly, I thought Dither_resize did some kind of rounding. Anyway in the LinearResize function I changed target dimensions to (w%2+w,h%2+h). Nice to know it wasn't THAT error again.

As a last question, Im forcing to proces Dither_y_gamma_to_linear in PC range to have more data to work with. Do you think this is more detrimental than beneficial?
Dogway is offline   Reply With Quote
Old 21st July 2011, 08:29   #216  |  Link
cretindesalpes
͡҉҉ ̵̡̢̛̗̘̙̜̝̞̟̠͇̊̋̌̍̎̏̿̿
 
cretindesalpes's Avatar
 
Join Date: Feb 2009
Location: No support in PM
Posts: 610
Dither 1.9.5 released:
  • Bug fixed in Dither_resize16(), displaying green bars when SSE2 optimisations are disabled.
  • Better check of stack16 clip dimensions.

Quote:
Originally Posted by Dogway View Post
As a last question, Im forcing to proces Dither_y_gamma_to_linear in PC range to have more data to work with. Do you think this is more detrimental than beneficial?
TV range is all about keeping a headroom for the ringing causing by some filters (resizers, sharpeners…), so with PC range this headroom is lost. But this is not a big difference, and the bottom headroom is cleared by the gamma/linear conversion anyway.

Quote:
Originally Posted by Yellow_ View Post
the compositing application I'm importing them into assumes linear. So does the Dither_y_gamma_to_linear function serve this purpose?
Yes.

Quote:
Is it only necessary to linearize the luma, more accurate and technically correct to do it that way before the conversion to RGB, rather than a typical 0.45 reverse gamma on all channels in RGB data after conversion?
GammaYUV -> linearYUV -> linearRGB will give you wrong colors. Inverse colorspace conversions should be done in the same gamma/linearity as the forward conversion. For gamma YUV to linear RGB conversion you can use:

Code:
Dither_convert_yuv_to_rgb (output="rgb48y")
Dither_y_gamma_to_linear (tv_range_in=false, tv_range_out=false)
Dither_convey_rgb48_on_yv12 (
\   SelectEvery (3, 0),
\   SelectEvery (3, 1),
\   SelectEvery (3, 2) )
Quote:
Also as the source was encoded with a BT709 transfer curve not sRGB 2.2, to undo that ie: linearize YCbCr, I should be assuming something like the reciprocal of 2.35? If I understand correctly this helps prevent compressing shadow detail that can occur applying 0.45 to BT709 source?
You're right, it appears that BT709 doesn't use the same transfer curve as sRGB, the slope at 0 in sRGB is much steeper than in BT709. If you want to linearize the R'G'B' components converted from the BT709 Y'CbCr, apply on each one : R = R'/4.5 if R' < 0.081 or R = ((R' + 0.099) / 1.099) ^ (1/0.45) if R' ≥ 0.081. The formula are the same as in sRGB, but with different values. I will add the BT709 mode to Dither conversion functions in a future release.
__________________
dither 1.27.2 for AviSynth | avstp 1.0.3 for AviSynth development | fmtconv r19 for Vapoursynth | trimx264opt segmented encoding
cretindesalpes is offline   Reply With Quote
Old 21st July 2011, 19:40   #217  |  Link
SilaSurfer
Registered User
 
SilaSurfer's Avatar
 
Join Date: Oct 2009
Posts: 212
Hey cretindesalpes. Just wanted to stop by and say outstanding work. Finished off an encode using your ordered dithering in my filter chain. Awasome results. Thanks again for all of your hard work.
SilaSurfer is offline   Reply With Quote
Old 21st July 2011, 23:48   #218  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 949
hey I just needed to use addborders inside stacked 16b. Thought on sharing the function (primitive but useful). Thanks for the new version too!
Code:
Function Dither_addborders16 (clip src, int "left", int "top",
\	int "right", int "bottom")
{
	left   = Default (left,   0)
	top    = Default (top,    0)
	right  = Default (right,  0)
	bottom = Default (bottom, 0)
	
 src
	msb = crop(0,0,width,height/2).addborders (left, top, right, bottom)
	lsb = crop(0,height/2,width,height/2).addborders (left, top, right, bottom)

	StackVertical (msb, lsb)}
btw Im also looking forward the v&c implementation

Last edited by Dogway; 22nd July 2011 at 00:10.
Dogway is offline   Reply With Quote
Old 22nd July 2011, 00:08   #219  |  Link
mandarinka
Registered User
 
mandarinka's Avatar
 
Join Date: Jan 2007
Posts: 678
Quote:
Originally Posted by SilaSurfer View Post
Hey cretindesalpes. Just wanted to stop by and say outstanding work. Finished off an encode using your ordered dithering in my filter chain. Awasome results. Thanks again for all of your hard work.
Indeed. Dither brought quite some fresh momentum into the abilities of avisynth, single-handedly. Thanks from me too.
mandarinka is offline   Reply With Quote
Old 22nd July 2011, 21:49   #220  |  Link
Alek93j
Registered User
 
Join Date: Apr 2011
Posts: 9
Code:
Function zzz_denoise (clip src, float "sigma", int "thr", bool "mask", int "sad")
{
	sigma = Default (sigma,    16)
	thr   = Default (thr,       5)
	mask  = Default (mask,  False)
	sad   = Default (sad,     200)

	w = src.Width ()
	h = src.Height ()

	# Motion analysis
	super   = MSuper (src)
	super_a = MSuper (src.TTempSmooth ().RemoveGrain (12))

	fwd_vect_3 = super_a.MAnalyse (isb=false, delta=3, overlap=4)
	fwd_vect_2 = super_a.MAnalyse (isb=false, delta=2, overlap=4)
	fwd_vect_1 = super_a.MAnalyse (isb=false, delta=1, overlap=4)
	bck_vect_1 = super_a.MAnalyse (isb=true,  delta=1, overlap=4)
	bck_vect_2 = super_a.MAnalyse (isb=true,  delta=2, overlap=4)
	bck_vect_3 = super_a.MAnalyse (isb=true,  delta=3, overlap=4)

	fwd_comp_2 = src.MCompensate (super, fwd_vect_2, thSAD=sad)
	fwd_comp_1 = src.MCompensate (super, fwd_vect_1, thSAD=sad)
	bck_comp_1 = src.MCompensate (super, bck_vect_1, thSAD=sad)
	bck_comp_2 = src.MCompensate (super, bck_vect_2, thSAD=sad)

	# Spatio-temporal denoising using modified dfttest
	c_dft = Interleave (fwd_comp_2, fwd_comp_1, src, bck_comp_1, bck_comp_2)
	c_dft = c_dft.dfttest (sigma=sigma, lsb=true)	# Double height
	c_dft = c_dft.SelectEvery (5, 2)

	# Temporal-only denoising using modified MDegrain
	c_deg = src.MDegrain3 (super, bck_vect_1, fwd_vect_1, bck_vect_2, fwd_vect_2, bck_vect_3, fwd_vect_3, thSAD=sad, lsb=true)	# Double height

	# Spatio-temporal denoising smoothes too much the details,
	# therefore we use pure temporal denoising on edges or detailed areas.
	edge_src = c_deg.Crop (0, 0, w, h)
	edge_mask = edge_src.mt_edge (mode="prewitt", thY1=thr, thY2=thr)
	edge_mask = edge_mask.mt_expand ()
	edge_mask = StackVertical (edge_mask, edge_mask)	# Double height
	c_hyb = mt_merge (c_dft, c_deg, edge_mask, luma=true, y=3, u=3, v=3)

	return (mask ? edge_mask.GreyScale () : c_hyb)
}
Maybe i'm gonna say something stupid, but this line:
Code:
super = MSuper (src)
It's used only by MCompensate and MDegrain3, which don't need to use levels=0 (if I understood right), so, it would be:
Code:
super = MSuper (src,levels=1)
to get the same result with a little increase of the velocity?
Alek93j is offline   Reply With Quote
Reply

Tags
color banding, deblocking, noise reduction

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 03:38.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.