Degraining script assist?

[Bruce/Fl]

Hello all,

I am an extreme avisynth noob, and hoping that I can get an assist here. Here is my situation.

I am in the process of having some old family 8mm films transferred. I am having it done by cinepost, so the results should be very good, although pricey. And they are very good, especially when I compare them to an earlier VHS transfer from about 25 years ago. In fact they are so good that I find myself actually wishing I could smooth out a bit of the now very easy to see and inherent grain from the source. I saw an example of Freds work online, and was really impressed. Since then I saw some of John Meyers work as well and was really impressed once again. That is what led me here.

As far as video editing programs go, I have been using Vegas 13 Pro. I use it to edit my present day Canon Vixia HD tape files. (ripped with HDVSplit)

I have gone to avisynth.org, and went through some of their tutorials, and so far so good. I am sloooowly learning to write code. (I am not always understanding exactly why I am writing what I write, but it is working). I have messaged John Meyer some and he sent me just the degraining segment of his code, but all of the other peripheral code that is needed to get it to run, well, at this stage of my learning I have no idea how to put it all together.

Additionally, I am working with .mov files, so I believe this is what I need to use, per John Meyer.

LoadVFAPIPlugin("C:\Program Files\AviSynth 2.5\plugins\QTReader.vfp", "QTReader")
QTReader("C:\xxxxxxxx\xxxxxx\xxxxx.mov")

As I mentioned, I have had no luck getting a degraining script to run. Anything beyond the very basic avisynth commands has foiled me. I *was* able to run Freds complete script, by downloading it and all of it's peripherals. It ran for me, and I got the split screen results. It looked horrible, because it was much more tweaking than I needed, but hey, it works. I *can* do this, if I have a complete script.

Anyways, as I was saying, I am getting many rolls of film transferred. Cinepost did the first 100' off of one of my 7" reels as a test run to give me a sample, to see if I wanted to go forward. As I mentioned, it'll be pricey, but I want to do it. I'm hoping to tweak the results some with a degraining script to make them even more impressive, and then hopefully my sister will see the value in it and I'll get some help from her on paying for it, (we'll split the cost). It will be over $1K initially, and then I'll have a few more reels on top of that, a little later. I'm having Cinepost hold my first 7" reel, and they are waiting for my go ahead to finish it off. Then, after being able to show the results to my sister and knowing she's financially in with me on this project, I'll send the rest of the first batch of reels.

All of which is the reason for my post. Could anyone here show me a complete script that will do some degraining for me? Something that all I will need to do is insert the path to my original file to and then run? I have avisynth downloaded, virtualdub, and a few other assorted programs and some dll's. I have a frame server loaded into vegas and have seen it work as advertised. Let me know what I'll need if I don't have it already and I'll get it and install it.

I hope to eventually get familiar with this script writing to be able to understand this better, but if I waited until I could do this all without a helping hand it would take forever. I'm hoping to be able to, once I tweak my sample file and impress my sister, give the go ahead to proceed in the very near future. I hope to make it so stunning of a difference it is a no brainer for her.

All that said, the look of the film after reducing the graininess some may not end up being what I am hoping for, and in that case, I'll just leave it as is. But having seen what is possible with avisynth, I am very hopeful that it will look even better.

Can anyone assist with a complete script? Ideally something that allows me to experiment with the amount of degraining being done, by changing a number here and there in the script.

Thank you

[videoFred]

Hi Bruce,

Originally Posted by Bruce/Fl

Can anyone assist with a complete script? Ideally something that allows me to experiment with the amount of degraining being done, by changing a number here and there in the script.

Thank you

This is what my script and John's script already does

But I assume you want something more simple, to start with Avisynth and to learn everything. Is it possible to upload an example clip? Then we can see what you need.

Fred.

[Bruce/Fl]

hi Fred,

PM sent, inquiring about exactly what you'd like.

[Reel.Deel]

@Bruce/Fl

Post a sample for everyone.

Originally Posted by Bruce/Fl

Additionally, I am working with .mov files, so I believe this is what I need to use, per John Meyer.

LoadVFAPIPlugin("C:\Program Files\AviSynth 2.5\plugins\QTReader.vfp", "QTReader")
QTReader("C:\xxxxxxxx\xxxxxx\xxxxx.mov")

QTReader is really old, I would recommend LSMASHSource.

LSMASHVideoSource("xxxxx.mov")

[johnmeyer]

I received six separate PMs from the OP over at Videohelp.com, where he is also pursuing this same topic. I responded, at great length, to each PM. Eventually I realized that he is having problems with the basics, like how to read an MOV file into AVISynth. I eventually had to give up because I wasn't able to help him. Before I gave up, I did send him the basic degraining code that is in my version of Fred's script, and which is taken, more or less intact, from the MVTools2 documentation.

Here's what I sent:

  prefiltered = RemoveGrain(stabcrop,2)
  superfilt =   MSuper(prefiltered, hpad=32, vpad=32,pel=2)
  super=        MSuper(stabcrop, hpad=32, vpad=32,pel=2)

  halfblksize= (block_size>4)   ? block_size/2 : 4
  halfoverlap= (block_over>2)   ? block_over/2 : 2

  bvec1 =  MAnalyse(superfilt, isb = true,  delta = 1, blksize=block_size, overlap=block_over,dct=0)
  bvec1 =  MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100) 

  fvec1 =  MAnalyse(super, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
  fvec1 =  MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

  bvec2 =  MAnalyse(super, isb = true,  delta = 2, blksize=block_size, overlap=block_over,dct=0)
  bvec2 =  MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

  fvec2 =  MAnalyse(super, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
  fvec2 =  MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

  denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)

However, since this used variables from earlier in the script, he didn't know how to change the variables. If someone wants to help this person, you need to give him a script, absolutely complete, that will read an MOV file, and which will degrain film grain. That was more than I was able or willing to do, but that is what is needed.

Perhaps I should have simply copy/pasted that MVTools2 sample code. Heck, here it is:

AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

There, that will degrain "test.avi". All you have to do is replace that first line with one that will read your MOV file, and you should get some degraining results. Increase or decrease the thSAD number (currently 400) to change the amount of degraining.

[feisty2]

thSAD=denoising_strength

thSAD ain't no "denoising_strength"... like, really not, way not
I know cuz I just read the code of mvtools and wrote my own ultra high precision (32bits float) branch
thSAD is like, more of the tolerance of imperfect motion estimation, larger thSAD = messed up motion vectors with lots of errors would still be, like, okay, acceptable and join the process
the real strength of MDegrain is "tr", larger tr = higher strength

[Bruce/Fl]

Originally Posted by johnmeyer

I received six separate PMs from the OP over at Videohelp.com, where he is also pursuing this same topic. I responded, at great length, to each PM. Eventually I realized that he is having problems with the basics, like how to read an MOV file into AVISynth. I eventually had to give up because I wasn't able to help him. Before I gave up, I did send him the basic degraining code that is in my version of Fred's script, and which is taken, more or less intact, from the MVTools2 documentation.

Here's what I sent:

  prefiltered = RemoveGrain(stabcrop,2)
  superfilt =   MSuper(prefiltered, hpad=32, vpad=32,pel=2)
  super=        MSuper(stabcrop, hpad=32, vpad=32,pel=2)

  halfblksize= (block_size>4)   ? block_size/2 : 4
  halfoverlap= (block_over>2)   ? block_over/2 : 2

  bvec1 =  MAnalyse(superfilt, isb = true,  delta = 1, blksize=block_size, overlap=block_over,dct=0)
  bvec1 =  MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100) 

  fvec1 =  MAnalyse(super, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
  fvec1 =  MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

  bvec2 =  MAnalyse(super, isb = true,  delta = 2, blksize=block_size, overlap=block_over,dct=0)
  bvec2 =  MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

  fvec2 =  MAnalyse(super, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
  fvec2 =  MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

  denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)

However, since this used variables from earlier in the script, he didn't know how to change the variables. If someone wants to help this person, you need to give him a script, absolutely complete, that will read an MOV file, and which will degrain film grain. That was more than I was able or willing to do, but that is what is needed.

Perhaps I should have simply copy/pasted that MVTools2 sample code. Heck, here it is:

AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

There, that will degrain "test.avi". All you have to do is replace that first line with one that will read your MOV file, and you should get some degraining results. Increase or decrease the thSAD number (currently 400) to change the amount of degraining.

Hello John, Everyone,

A little background. just so everyone knows the story.

The night I initially contacted you, after first seeing Freds and then your work, I checked out various online resources looking for more info on this wonderful avisynth scripting. I was sent to doom9 first, where I started my researching. But I was not a member here yet, so I looked other places as well. I am a member over at Videohelp, and when I popped in there I saw your name over there as well, and that is why I reached out to you there. It wasn't that I was "pursuing this same topic" over there on their boards, but rather my recognizing your name over there, so I PM'd you using that forum. I don't want it to appear that I was over there sucking them dry of knowledge, and when done tossing them aside and coming here, in fact I never brought up the subject over there at all, I simply PM'd you using their messaging.

And your responses to my questions were very detailed, and I will once again thank you for the time and effort you put into trying to educate me. I am trying to learn about this scripting, but it is totally foreign to me, which you saw from my questions I am sure. I am still going through tutorials and working with more complicated scripting, but it is a very slow process. But regardless, after a few back and forth messages I gathered a sense that what I was trying to accomplish in contacting you, was most likely irritating you to some degree, so I stopped, and it looks like my instincts were correct, as you said above, what I was hoping for "... was more than I was able or willing to do". But once again, thank you very much for your efforts, and maybe someday soon I will be able to "talk shop" with you and others here when I actually have more of a clue as to what I am talking about. Because in my messages with you, and also here on my initial post, I tried to be very clear that I am very, very new to this and my knowledge is very, very limited in this area. I'm hoping I was clear on that as well here, I'm not trying to anything sneaky, I just need help.

But at the forefront of all of this is that I am hoping someone will help me jump ahead quickly in this scripting, on solely the degraining aspect, for the purpose of hopefully improving what Cinepost sent me so I can show my Sister what is possible, and that she will feel better about making a monetary contribution. All of the rest of the capabilites avisynth gives you, stabilization, adjusting levels, sharpening, etc., I am fine with learning as I go. It is only the degraining aspect of it that has, at least for me, a time constraint.

But John is right, what I really need is a complete script that is sort of plug in play, something I'll just need to put in the location of my sample clip into, and/maybe add some more plug-ins into, etc. to get it to do its thing. I hope my initial post here made that clear.

[Bruce/Fl]

Originally Posted by Reel.Deel

@Bruce/Fl

Post a sample for everyone.

I'll work on doing just that tonight.

How much (length) of a sample would be best? Taking file size into consideration. The sample scan they sent me was approximately 10 minutes. I would assume you would only need 10, 15 seconds worth?

Thank you

Bruce

[StainlessS]

Originally Posted by johnmeyer

  halfblksize= (block_size>4)   ? block_size/2 : 4
  halfoverlap= (block_over>2)   ? block_over/2 : 2

John, would below be more like what you require ?

  halfblksize= (block_size/2>4)   ? block_size/2 : block_size
  halfoverlap= (block_over/2>2)   ? block_over/2 : block_over

If no difference from original sizes, would MRecalculate be necessary ?

EDIT: Oops, maybe below better

  halfblksize= (block_size/2 >= 4)   ? block_size/2 : block_size
  halfoverlap= (block_over/2 >= 2)   ? block_over/2 : block_over

[feisty2]

All same stuff
John's version
Blksize > 4 (blksize=8, 16, 32), then halve it, if not (blksize =4), keep it 4
Your version
1: buggy, say blksize =8, and you know what will happen
2: = John's version blksize/2>=4 ---> blksize>=8---->blksize>4, If not, then obviously blksize=4, and ":blksize"=":4"

[StainlessS]

Cant remember off hand if block sizes have to be a power of 2.

What about MRecalculate thing feisty ?

EDIT: yes 1. was buggy.

[feisty2]

Recalculate is meant for, not just different, actually smaller block size, the vector clip contains 3 things of each vector, vx, vy and thsad, recalculate reads the thsad from one vector, and if it's too high, then the block is kind of "high energy" (complicated) or whatever and the motion estimation is bad and recalculate will divide the large block into 4 smaller blocks and try to match again, so you'll always have the most precise match

[StainlessS]

Yeh but, if sizes aint smaller, then is there any point at all in calling MRecalculate, is it a NOP, or just a time waster ?

(Or does it actually still do something useful)

[feisty2]

If the recalculate blksize >= analyze blksize ------> waste of time

[StainlessS]

Thank you F, then my initial post was perhaps not also a total waste of time.

[johnmeyer]

We're off on a tangent here. The block size and MRecalculate were an attempt to get a slight improvement in speed and quality, and that's exactly what I got: a slight improvement. The code I posted was just lifted directly from the MVTools2 documentation.

And, speaking of documentation, I'm still trying to figure out this statement:

Originally Posted by feisty2

thSAD ain't no "denoising_strength"... like, really not, way not
I know cuz I just read the code of mvtools and wrote my own ultra high precision (32bits float) branch
thSAD is like, more of the tolerance of imperfect motion estimation, larger thSAD = messed up motion vectors with lots of errors would still be, like, okay, acceptable and join the process
the real strength of MDegrain is "tr", larger tr = higher strength

First of all, "tr" was not a part of MVTools2 until version 2.6.x, and I'm still using 2.5.x.

Second, in reading the 2.6.x documentation, "tr" is not used in their sample degraining code, so the authors did not think it important. Here is that code, copied directly from the documentation:

To denoise interlaced source with function MDegrain2i2:

function MDegrain2i2(clip source, int "overlap", int "dct")
{
overlap=default(overlap,0) # overlap value (0 to 4 for blksize=8)
dct=default(dct,0) # use dct=1 for clip with light flicker
fields=source.SeparateFields() # separate by fields
super = fields.MSuper()
backward_vec2 = super.MAnalyse(isb = true, delta = 2, overlap=overlap, dct=dct)
forward_vec2 = super.MAnalyse(isb = false, delta = 2, overlap=overlap, dct=dct)
backward_vec4 = super.MAnalyse(isb = true, delta = 4, overlap=overlap, dct=dct)
forward_vec4 = super.MAnalyse(isb = false, delta = 4, overlap=overlap, dct=dct)
fields.MDegrain2(super, backward_vec2,forward_vec2,backward_vec4,forward_vec4,thSAD=400)
Weave()
}

AVISource("video.avi")
mdegrain2i2(4,0)

Finally, "tr" really is not a very practical denoising variable, since it can only take on a few discrete values (0 through 5 is the extent of the practical range). Most desnoising scripts, like this one:

http://forum.doom9.org/showpost.php?...54&postcount=2

use the tr variable as more of a switch than a denoising setting. So, yes, it can be included as a way to change the nature of the denoising, but I certainly wouldn't call it a "strength" variable.

thSAD isn't exactly a strength variable either, but it does continuously (rather than discretely) change the nature of the denoising, and is the best setting this function offers for changing the amount of denoising.


Note to the OP: the code above is what you might want to use if you are using the latest version of MVTools2. Just change the AVISource statement to whatever you use to read your MOV video, and then see what the result looks like.

[feisty2]

What..? tr and thsad are 2 100% different and irrelevant things, you can't just say, like, one is continuous and one is discrete
Man, you can't even compare these two
They are just like... Apples and oranges...

[johnmeyer]

Originally Posted by feisty2

What..? tr and thsad are 2 100% different and irrelevant things, you can't just say, like, one is continuous and one is discrete
Man, you can't even compare these two
They are just like... Apples and oranges...

Irrelevant to what?? thSAD is completely relevant to any discussion of using MDegrain2 for denoising.

As for your other statement, I totally agree that tr and thSAD are completely different. But I was not, in any way, "comparing" them.

So, I don't understand at all what you are trying to say. What is your point?? Also, how are you trying to help the OP (and anyone else) achieve better degraining results?

[feisty2]

I was just trying to get that misunderstanding about "thsad" as denoising strength straight
Thsad is not the strength of denoising
It's the strength of error tolerance of motion estimation

[johnmeyer]

Originally Posted by feisty2

I was just trying to get that misunderstanding about "thsad" as denoising strength straight
Thsad is not the strength of denoising
It's the strength of error tolerance of motion estimation

I agree. I also agree that it does not provide ideal control over denoising. Unfortunately, the way the code is written, and given the variables that are exposed to the user, it is pretty much all we have.