about MVtools...

[morsa]

Sorry to post this but I couldn't find any thread dedicated eclusively to MVTOOLS.....

@Manao (and anybody else interested)

May be I'm crazy but, you aren't using any kind of border detector for MVconvertFPS, right?
That came to my mind after testing it with a simple scene of a girl with a white T-shirt against a black background..
Lots of half-black blocks got inside the T-shirt near its borders..
If it weren't for that, MVconvertFPS would be really useful :(.
It is really impressive to look at its results.
I think it is far better than MotionPerfect on several situations and very poor on many others..
Is there any way to use detected borders to not allow blocks to cross them?
May be some image segmentation algo could help?(I have source code for that ;) )
Also there is an open source Morphing library at Sourceforge in case it could help...
Also some type of soft border on the target image blocks needs to be implemented... :)



Second part:
After trying to get something decent for a 24 to 48 fps convertion, I came to this:

This ugly mess gives me improved results (it is my one-track-mind way of doing an image segmentation)

vec_back = mvanalyse(last.VagueDenoiser(threshold=4,method=1,nsteps=6,chromaT=0.8,percent=100).convertToYUY2._2DCleanYUY2(0,6,3,3,0,4,4).sharpen(1.0,1.0)._2DCleanYUY2(0,6,3,3,0,4,4).convertToYV12, isb=true, pel=1)

vec_forw = mvanalyse(last.VagueDenoiser(threshold=4,method=1,nsteps=6,chromaT=0.8,percent=100).convertToYUY2._2DCleanYUY2(0,6,3,3,0,4,4).sharpen(1.0,1.0)._2DCleanYUY2(0,6,3,3,0,4,4).convertToYV12, isb=false, pel=1)

MVConvertFPS(last, vec_back, vec_forw, nb=1,fps=50.0, fbw=2,bvt=35000, thSCD1=50)
:)

[Fizick]

I will read your post later, :)
but now i can direct you to
http://forum.doom9.org/showthread.php?s=&threadid=76041

[Manao]

morsa : the motion estimation is as simple as XviD's ME. It tries to find the block in the previous frame that is the closest to a block in the current frame, and it also tries to have an homogeneous field of vectors.

Anyway, the ME wasn't meant to be used for doing such difficult things as interpolating new frames. I just wanted to make a good denoiser ( which I haven't yet done ), and a tool for those who could find a use to it.

The algorithm behind MVConvertFPS is far to simple to give good results ( basically, I take forward and backward vectors between the current and the previous frame, and between the current and the next one, and I move blocks along the vectors. Since there still was blocks, I made an overlapped motion compensation ( i think you call it soft borders ), but it blurs too much imho, and there still are remaining blocks ).

Yet, I don't think a segmentation would be useful. A segmentation in blocks has already being done, and it would interfere with any other kind of segmentation.

Now, to adress more specifically the issue you're encountering, it would be better if you could make available a sample of the video you're working on, or at least a screenshot of the video which shows the motion vectors ( and also the previous and next frame ). From what you're discribing, I find the issue rather strange ( since black on white should make a SAD very high if the vector is wrong )

BTW, which version are you using ?

[morsa]

Well, I downloaded the version which has no number MVTOOLS.DLL (August 22)
Is it wrong??
Image segmentation is not in blocks.Look for it on Google...
Image segmentation is more similar to what all that heavy filtering I use in ME looks like..

Can I send you the clip to your e-mail? PM me...

update:After I added the parameter Lambda=2000 things look a little better.Anyway I still need to use the heavy filtering before ME.

[Manao]

IIRC, the dll without version number is rather experimental. Blocks on the border of the picture may be wrong ( I implemented unrestricted ME in MVAnalyse, which means vectors can point outside the picture, but I didn't modify interpolation filters ( such as MVConvertFPS ) ). That version was made for FuPP, in order him to test the denoiser.Image segmentation is not in blocksI know, however, the frame is (implicitely) segmented in blocks, because the ME is made using a block matching algorithm. Hence, I have only vectors for a block. If I want to make another segmentation of the picture, both segmentation will interfere.

Can I send you the clip to your e-mail?Indeed : manao at melix dot net.

[morsa]

So I should think more towards a post-processing way to control the blocks.
Are you meaning that?
I thought that applying the ME a second time on a borders only image would give you the border interpolation required to keep track of final image borders.
It seems I was really misunderstanding the whole thing.
thank you :)

[Manao]

So I should think more towards a post-processing way to control the blocks.Not necessarily. A good postprocessing would certainly help, but without a good use of the motion vectors inside the filter itself, it would not solve the issue. So somebody still needs to work on the filter itself to improve things.

I look at the sample you send me. Vectors found seems quite erractic. To the end of the clip, it's rather normal, since fading are a killing for any sad based algorithm. But on the black background, there should be zero, and there are not. That's partially what is causing your issue. The filter should have been able to cope with such situation, but it may have a bug hidden somewhere...

To come back to fadings, there won't be any improvement if a weighted motion compensation isn't implemented ( as in H264 ).I thought that applying the ME a second time on a borders only image would give you the border interpolation required to keep track of final image borders.I don't understand what you mean here ?

[morsa]

I´m not worried about the fadings.That is a normal situation and I don´t expect MVtools to manage them correctly ;)
If you could notice the blocking on Girl´s skin, that is why I said that thing about blocks with soft borders...
If you´ve tested my script published at the super resolution thread you could see a big improvement..
The second thing I'm talking about is, if you could implement some kind of border interpolation inside the filter that would give you control of the image borders and would give a great improvement to visual quality of the interpolated frames..
I mean inside MVanalizy there would be two clips to analizy: the usual one, and a second which would be the result of a canny, laplacian,etc border detector.So it would be just the contour of the images inside a frame (clearer now?).
So I guess it would be easier to match correctly that kind of image, and the resulting interpolated border image could be used as a map to correct those ugly blocks misalignements on image borders..

[Fizick]

Fading can be well processed by intensity stabilizer (deflicker filter), for motion vectors only.

But I have another question to Manao.
Why MVconpensate does not work in version 0.9.4 and 0.9.5 ?
I have black screen output for script:

avisource("h:\smena0.avi")
loadplugin("mvtools.dll")
gi=ConvertToYV12(interlaced=false)
bv=MVAnalyse(gi,isb=true)
MVCompensate(gi,bv)


Version 0.9.3 work well.
Avisynth 2.5.5.
:confused:

[Manao]

I tried 0.9.5 and the latest version ( without version number ), both are working correctly for me. I can't see what's going wrong in the script you've provided :confused:

[Fizick]

Anybody else can check it?
May be i need in some DLL ? :confused:

[scharfis_brain]

another idea (maybe stupid) for getting rid of those false blocks:

calculate the full compensated frame for each fwd & bwd.

then compare the full compensated frame with the original prev/next frame.
Every area that isn't matched correctly has to be a weighted blend in mvconvertfps().
This way, it should reduce the amount of blocking.

also a global motioncompensation, that fits the largest object by near to 100% before the block-MC would be cool.

[Fizick]

Yes, I want try Global MC, but MVTools are broken for me. I have not pure black screen, but luma=0 with normal chroma. Tested on 3 computers :confused:
Anybody can post worked script and config?

[morsa]

what kind of script?
Sorry I don't understand...

[Fizick]

At last I understand why my script did not work.
My clip is bad (as always :( ), and motion vertors is bad too,
and MVCOMPENSATE at default settings consider all clip as scenechange .
I need to increase parameter thSCD2 up to 200-256, and now I have a picture.
But I think, that logic is not right: at scenechange, we must not get not black output frame, but copy of previous (non-compesated) frame,
to use its data some way.

Also for MVMask, we must have not black frame at scenechange, but pure white (=255). It is consistant with quantity of inter-frame motion (for masked temporal filtering, etc).

Manao, may I ask you to change these functions algo this way in your plugin ?
Probably restore as in v.0.93
Or please make special version for me :)

[Manao]

I agree for the compensation behavior, your proposal is better. However, I disagree for the mask : imagine someone using the mask to blur areas which are moving a lot. If a scenechange gives a white mask, it will blur scenechange, which is bad.

However, a scenechange flag will be added to MVMask, in order to activate that behavior. It will work the same way for MVCompensate. Stay tuned for a late release this night.

[scharfis_brain]

what about my idea, that wants to detect, if the MC was succesful? is it worth a try?

[Manao]

scharfi : It already works like that in MVConvertFPS ( more precisely, it does overlapped motion compensation for backward and forward, depending on the SAD, then fill the gap with others blocks moved by other vectors ( in order to fill gaps let by a divergent field of vectors ), and finally blend the frames for the remaining gaps, if any ).

Fizick : here (http://jourdan.madism.org/~manao/MVTools-v0.9.5.2.zip) it is. Changelog :

* MVMask has two new parameters : showsad (bool) which allows to build the mask of the SAD values instead of the mask of the vectors' length. Ysc allows you to set the value taken by the mask on scene changes
* MVCompensate : behavior modification on scene changes. Now, the current frame is left untouched if a scene change was detected.
* New filter : MVSCDetection, with one parameter, Ysc, which works as previously.

* MVInterpolate, MVConvertFPS and MVBlur are enabled, but may be buggy ( though I wasn't able to make MVConvertFPS crash )

Sources are included, dll is only for testing purposes ( though MVDenoise, MVMask, MVShow and MVSCDetection should be safe ).

[Fizick]

1. New MVCompensate does not work ( read memory access bug). :(
2. Now, the current frame is left untouched if a scene change was detected. But I proposed a previuos frame.

P.S. Now I can compile it with MS Toolkit 2003 compiler within Visual Studio 6 env!

I look to MVCompensate source once more. There are some bugs with prev and current frame pointers and pitches. I tried correct, but error still present.

[Fizick]

Manao,
Now I understang this bug.
Some new x and y coordinates are negative or > width,height.
It seems, you use padding for vector search, but not use it for compensation here (still).
We will wait new version. :)

BTW, why did you use padding? It is experimenal ? I never see this method in literature.

[Manao]

Indeed I use padding during the search, and I still don't use it in the filters. I'll correct MVCompensate tonight ( I didn't experience a crash while testing it, I've been lucky )

The padding is used in almost every motion compensation based codec ( at least, MPEG-4, H264 ), because it allows a far better behavior of the motion vectors on the frame boundary.

[Manao]

Alright, new version is up : http://jourdan.madism.org/~manao/MVTools-v0.9.5.3.zip

Mainly a bugfixe ( several filters were affected by a silly bug :rolleyes:, and MVCompensate now do padded motion compensation, and compensate also the chroma ( though it rounds the vector to odd coordinates to do so )

A new feature, as asked by Fizick, for mvcompensate : "scbehavior", a boolean set to true by default, will allow you to keep the previous frame over a scenechange if you set it to false.

Enjoy ( and report, because chroma MC means more bugs :) )

[Fizick]

Manao, it works! Thanks.

Now i want ask you about future deblocking.
Do yon know about "Overlapped block motion compensation".
The base article is:
MT Orchard and GJ Sullivan, "Overlapped block motion compensation: An estimation-theoretic approach", IEEE Transactions on Image Processing, vol. 3, no. 5.

There are many citations of it (I have no its content).
http://citeseer.ist.psu.edu/context/43092/0


"...OBMC is derived in [13] as a linear estimator of each pixel intensity, given that the only motion information available to the decoder is a set of block based vectors. OBMC predicts the frame by overlapping shifted blocks of pixels from the reference frame, each weighted by an appropriate window...."
Overlapped Block Motion Compensation (OBMC) Every pixel in the final prediction region for a block is obtained as a weighted sum of three values [10]. These values are obtained using the motion vector of the current block and two out of four remote motion vectors. The remote motion vectors are the motion vectors of the block above, the block below, the block to the left and the block to the right of the current block.
Overlapped block motion compensation, which is included in the advanced prediction mode of H. 263....

Is it realy solution?

[Manao]

Yes, OBMC is implemented in MVConvertFPS, MVBlur and MVInterpolate, but not in MVCompensate, since it wasn't the same purpose. However, it seems that I do OBMC in a different way ( I was moving for example a 12x12 block instead of a 8x8 )

MVCompensate is still buggy, I'll release a version which should work during the day ( hopefully, it will come with some other goodies ;) )

I'll see if OBMC is useful for plain motion compensation, but other things have to be added before.

[Didée]

Originally posted by Manao
MVCompensate is still buggy, I'll release a version which should work during the day
Looking forward to that. Currently, MVCompensate is giving me very blocky results, where MVConvertFPS & Co. are rather smooth. But I need exactly MVCompensate to test some denoising things ... oh, :o

[Manao]

New version is up : MVTools 0.9.6.1 (http://jourdan.madism.org/~manao/MVTools-v0.9.6.1.zip)

Lot of bugfixes for the existing filters. No OBMC for MVCompensate, but I'll keep the idea in mind. So, for the filters already existing, those who should work are MVMask, MVShow, MVCompensate, MVDenoise and MVSCDetection, and MVAnalyse. The three others ( the same as always ) still doesn't check whether the MV goes out of the picture and so may crash unexpectedly. Though it shouldn't happen to often, since the king of OBMC implemented in them somehow check that ( but not properly, hence the warning ).

Now, for the three new filters. Two have nothing to do with motion compensation, but I didn't want to put them in separate binaries, since they'll mainly be used with filters from this package. The third one uses vectors, and integrates somehow the two others. But let's cut to the point

* QDeQuant(clip c, int quant, int level) : takes a clip and quantizes it, using an approximation of the H264 DCT. It filters the three planes ( 4x4 blocks for each of them, so the chroma isn't processed as in H264 ). It's not exactly the H264 DCT because at q1, it's lossless, and a q51 it's not that bad, but you can raise quant over 51. Level is the reference level of the picture. By default it's zero, but it can be set, for example, to 128. The picture is then treates as if pixels were ranging from -128 to 127, hence avoiding errors around 128.

* Deblock(clip c, int quant, int aOffset, int bOffset ) : takes a clip, and deblock it using H264 deblocking, as if the picture was made only of inter blocks. This time, quant ranges from 0 to 51 as in H264, and has the same impact. aOffset and bOffset allow to raise / lower the quant when deciding for some internal thresholds. They are set by default to 0. Be warned that the filter should do nothing at quant < 16, if aOffset and bOffset are both zero. It's a wanted behavior ( thus it respect the partially the norm ).

* EncDenoise(clip c, clip vectors, bool scbehavior, int quant, int aOffset, int bOffset, int thSCD1, int thSCD2) : it merges Deblock, QDeQuant and MVCompensate, taking from them the name and behavior of their parameters. It basically does a h264 encode as if all blocks were 8x8 inter blocks. Reference frame is the previous frame output by the filter ( if it is the correct one, else it's the previous frame of the source ), mvs are those given by mvanalyse on the source. The reference frame is compensated by the vectors, then the residual difference is quantized / dequantized and added to the result of the motion compensation. Finally, the frame is deblocked, and serves as reference for the next one.

It's only because in loop filtering could not be done with avisynth that I made this filter, else the three others taken alone would have done the job.

So, of course, it's bound to be buggy ( don't set quant to zero, you'll end up with a division by zero for example :) ), but this time, it does a good job a denoising.

Have fun...

[Didée]

Hey, where are the ME-denoising-people shouting hooray? This is at least worth a comment, isn't it?

Big thanks for those interesting new tools, Manao! Won't come to testing before today evening, though.

For now, let me throw in some thoughts & questions:

- your above direct link doesn't work for me - seems to need the referer from that site itself
no, a typo in the URL - Leak prooved me being blind ;)

- I think I noticed in v0.9.5.3, that chroma was compensated incorrectly [edit] by MVCompensate , giving a sort of colored borders around objects. Is that fixed, or was it another error on my side?

- Sorry, but I've to take the steam a little down. Even if this method of quant/dequanting the residual error after motion compensating yields good results at first glance, it is conceptually wrong IMO. (You know, there's still a promise pending ;) )

- Is it somehow possible to get a short-but-more-indepth explanation about the motion estimation parameters? I'm a little lost about "one-step-search" vs. "n-step-search" vs. "diamond search", and how the step size affects each of these. Could imagine I'm not alone with that... (a good link would do it as well)

[Leak]

Originally posted by Didée
For now, let me throw in some thoughts & questions:

- your above direct link doesn't work for me - seems to need the referer from that site itself

Nope, it's just that the filename in the link needs an "-" instead of the first "."... :D

Just a simple tyop and not referrer-checking... ;)

np: Mikkel Metal - Nepal (Kompakt Total 5)

[Manao]

@All : there seems to be a crash with EncDenoise, perhaps because of scenechanges. I'll try to correct that tonight.

@Didée : As Leak said, I switched a - by a dot. For the chroma behavior, it's one of the bugs which should have been corrected in MVCompensate.

I agree when you say quant / dequant for denoising isn't theorically good. However, if it's especially true for MPEG-4's dct, the h264 behaves better in this regards, because it doesn't add ringing. Moreover, the quant / dequant isn't used alone, there is also the deblocking ( if you want, you can try without deblocking : just set aOffset and bOffset to (10 - quant) )

For the ME settings, especially the search type : the algorithm used is EPZ. Its principle is rather simple : We scan the image horizontally. For a block, we test 5 different vectors, called predictors. These 5 vectors are found like that :

- vector of the left block on the current frame ( it has already being computed, since we scan the frame left to right, top to bottom )
- vector of the top block, current frame
- vector of the top right block, current frame
- vector median ( ie each of its coordinates are the median of the coordinate of first 3 vectors )
- vector computed by the hierarchal analysis : the ME is made first on block 128x128, then 64x64 then 32x32 and so on. From the vectors for the blocks 128x128, we can interpolate a predictor for the blocks 64x64 ( a simple bilinear interpolation is enough ).

Once the best vector in this set of predictor is found ( here, best means the one which minimizes the cost : sad + lambda * length ), we'll use it as the start point of a refinement algorithm. That algorithm is defined by the parameter SearchType.

You'll find a description of the different algorithms available in this article (http://www.cmlab.csie.ntu.edu.tw/cml/dsp/prof-wu/course/itdc/slice/Data%20Compression%20-%20Motion%20Estimation%20for%20Video%20Coding%20Standards.pdf). XviD and EPZ use a diamond ( ie logarithmic ), and I also advise this one. I'll try to make the documentation clearer on this aspect, or to find a better article to link to.

[Didée]

:)

Thanks for the quick run-through. (Well, that are the very basics, aren't they :o )

[Mug Funky]

weeeee!!! thanks for the new filters!

i tried hacking in the encoder-MVdenoise using DCTfilter and MVcompensate, but the results weren't as good as i got with 4 offset Xvid encodes.

@ didee: explain why the quantization approach isn't that good for denoising? i know simple transcoding is a dumb idea, but my experiments with using several encodes overlayed (with different offsets) didn't give blocking, kept a good image and made compressibility (not surprisingly) a huge amount better. it didn't strike me as all that elegant, but i'm warming to the idea.

i'd love to see frequency based noise reduction in images for removing harmonics like those in TV interference (there's WNR for virtualdub, but that tends to take out way too much).

[Manao]

If the reference frame you're using is noisy, and the frame you want to denoise is also noisy, the quantization of the difference between the frame and the reference frame motion compensated will suppress noise from that difference. But noise will come back when you add the result od the quantization to the motion compensated frame which is noisy.

That's why the reference frame has to be denoised. Then the difference will contain only noise from current frame, and quantization / dequantization will partly remove it.

[Didée]

Manao explained one half of the story. Without integrated denoising, some noise will escape the process - therefore denoising is an important aspect of this technique. But then, through denoising also some detail will be lost, which brings the danger up of unintentional altering some detail. Most probably only weak detail, but still.

But I was looking at it from another side:
Even the best imaginable block-based ME can only work on orthogonal squares or rectangles. But, the presented textures mostly are not fully identical from frame to frame. Imagine an object that, as it is travelling its way through the frame, also is i.e. slowly rotating over any axis in space. This leads to small texture changes between the frames, and these small changes are likely to appear just as noise after compensation, and are in danger of being filtered away together with the noise as well. (I imagine something like not-constant texture smearing, or flickering, on moving objects.)
(On top of that, the whole thing definetly needs to work with at least halfpel precision, since all texture information actually appears in an anti-aliased representation of a higher resolution source. Therefore, each moving texture actually IS shimmering, if we look at it on a pixel level.)

Now, I don't say that this "new" method is just bad. In contrary, I can imagine it yields quite pleasing results if everything is done right. (but hey, wait for me ... first I need to test it, after all ;) )

I only want to say that the method is not fully optimal. What I call an "optimal denoiser" is ... the human eye, or better said, the brain: We have little problems in detecting even very weak detail way below the noise level, if there is any at all.
And I think the only way to get as close to this as possible, is good old temporal averaging - because that's mostly what our brains are doing with noisy input. Of course, it needs some more adjustments, opposed to "simply" doing MVCompensate().TemporalSoften().
I'm trying since some months to find the time to elaborate a little on that :|

[Manao]

I found a bug ( which was plaguing both MVCompensate and Encdenoise ). It should solve at last Mug Funky's issue. They will be a release tonight, hopefully with some speed improvements, because right now it's sloooow ( plain c code for everything )

[Fizick]

BTW, the are not - block matched estimation-compemsation,
but flow method (pel based).
But it will be another plugin...

[Manao]

Alright, silent update with bufixes for MVCompensate and EncDenoise ( no change of version because the only whange was a 'width' becoming 'height' as it should have been in the first place ).

No speed optimizations yet, because my favorite bug hunter ( aka FuPP ) found something rather disturbing ( huge slowdown ) with masktools + deen. But anyway, since I'm in the same office as fenrir which made x264, I think optimizations for fdct / quant / dequant / idct will come soon enough.

I forgot to correct the link on the previous post, it's done now.

And finally, EncDenoise is a temporal denoiser, far more temporal than TemporalSoften, because it uses frames up to the previous keyframe.

[marcellus]

Hi,
Is possible with one of MVTools filters or with a combination to obtain a P frame or a B frame (or a close appearance of one of those)? To make myself more clear: it is in Xvid section a thread about obtaining a good quantization matrix using DCTune on several frames of the movie and averaging the results. One problem with that aproach was that for inter matrix it used also unprocessed frames so the result couldnt be accurate because that is not how inter frames look (more precise delta frames). Now, if we could obtain such a delta frame after motion estimation was performed (or a close representation of it) to run DCTune on it would make that method more reliable.
Bye

[Manao]

source = mpeg2source("foo.d2v")

mv = source.mvanalyse(isb = false, lambda = 2000)

compensation = source.mvcompensate(mv)

return yv12lutxy(source, compensation, yexpr = "x y - 2 / 128 +", uexpr = "x y - 2 / 128 +", vexpr = "x y - 2 / 128 +", y=3, u=3, v=3)However, be aware of several issues :
* the fdct / idct on a p-frame is made on the difference between motion compensation and source frame. Here, you get (difference / 2 + 128), because you can't output frame with 'negative' pixels, and with pixels over 255.
* keyframe are ( should be ) output totally grey.
* vectors won't be exactly the same as those XviD will use.

[marcellus]

Cool, many thanks!
Here, you get (difference / 2 + 128), because you can't output frame with 'negative' pixels, and with pixels over 255.
I am aware of that. By the way, for my information, how does the encoder store it, as far my knowledge let me imagine it would need 9 bit to represent a range of values between -255 and 255.
* vectors won't be exactly the same as those XviD will use.
Yeah, but it still serves my purpose, I think. I'm not even using Xvid for my encodes, I only use ffdshow to encode mpeg2 (for standalone compatibility purposes), but at this time this is as close I can get. That being said, is there any MVAnalyze or MVCompensate settings that would emulate as close as possible ffdshow (I mean libavcodec) motion estimation algo? I use SAD for real time captures and SATD for lenghty and quality important encodes with size 3 adaptive EPZS diamond size for both cases. Maximum I frames interval of 15 and 2 B frames between P frames.
Many thanks again!

[Manao]

The information is stored on 16bit, because it will be fed directly to the fdct which also works on 16 bits. And halving the coefficient after idct is a way of quantizing.

The default settings are close enough of libavcoded behavior ( diamond, sad, epz ). However, you won't be able to put an I every 15 frames with my filters, you'll have to do it with avisynth ( discarding one out of 15 frames ). Bframe will be hard to modelize ( it could be done with forward and backward motion compensation, averaging, and differenciating ).

[johnmeyer]

The latest versions of MVTools all throw this error:

AVISynth open failure:
Evaluate: Unrecognized exception!

The version that causes the error is MVTools-v0.9.6.1.zip. The only version that seems to work with this script is: MVTools-v0.9.2.2.zip, and it works fine. I have tried the versions between these two, and since about the June releases, none work. I am using AVISynth 2.55.

[Manao]

You put inside the plugin directory of avisynth a filter that is meesing up with the other filters. Empty your plugin directory and try again.

[scharfis_brain]

how can I avoid the weird scene-change effects, mvconvertfps produces?

I didn't found any usable parameters for preserved scene changes.

I am always getting garbage scene-changes....

are there any valuable paramter combinations?

(I am trying to set up a NTSC -> PAL conversion)

[Manao]

thSCD1 and thSCD2 should allow you to configure scene change detection. However, I haven't looked into mvconvertfps for a long time now, and I don't remember if these parameters are correctly used.

Anyway, I'm currently cleaning the code of the MVTools, and I should soon clean this filter, so stay tuned for the next release.

[FuPP]

I stay tuned too :p

FuPP

[Boulder]

Is there any way to make the MVConvert60ito24p() function faster? I was thinking that maybe some tweaking is possible without affecting the image quality much. Currently it runs at 1-2fps when saving to direct stream HuffYUV in VDub which slows down my work quite a bit;) The source material is rather soft and blurry and is originally 50i.

Scharfi's initial Convert60ito24p() runs very fast but I guess it's old tech:)

[scharfis_brain]

does that make any sense using mvconvert60ito24p with 50i video?
this function is meant for NTSC-Video.

feeding 50i video to that function will result in 20p!

also convert60ito24p is not old tech, it is a very common way converting 60i to 24p for cinema.
also is it that fast, cause it only uses patterned blending.
and with 1/60 shutter the results aren't that bad!

[Boulder]

I was a bit unclear.. :)

As you told me in that other thread, there's currently no way to reverse the 50i->60i transfer in Avisynth. So I thought that to get to watch the DVD on my PAL system (which unfortunately doesn't understand NTSC), I'd go from 60i to 24p and speed up the video and audio to 25p.

Or is there a better way?

[scharfis_brain]

sure it is.
but it is not implemented jet.

50i to 60i looks like this:

fields:

PAL 0 1 2 3 4 5 6 7 8 9
. | 2:8 4:6 6:4 8:2 | 2:8 4:6 6:4 8:2
NTSC 0 1 2 3 4 5 6 7 8 9 10


the line between PAL & NTSC shows the appr. weights. | is a 100% copywithout blending.

you have to:
identify the non blended fields in your PAL -> NTSC converted video. (one in six)
take this NTSCfield x (here: 0) and the NTSCfield x+6 (here 6) as reference.
NTSCfield 0 = PALfield 0
NTSCfield 6 = PALfield 5

to get PAL-field 1 subtract NTSCfield 0 * 20% from NTSCfield 1.
to get PAL-field 4 subtract NTSCfield 6 * 20% from NTSCfield 5.

to get PALfield 2 subtract PALfield 1 *40% from NTSCfield 2.
to get PALfield 3 subtract PALfield 4 *40% from NTSCfield 4.

feel free to implement this.
you would become a king!

but let me see, what I can do lees complicated...

the way of
50i -> 60i -> 24p -> 25p seems to be a bit crappy to me...

[Boulder]

Do you need a sample? I've got one VOB (~15MB) which I could put online. The rest of the DVDs are currently at my parents' house.

Hehe, I find treating simple TV captures sometimes challenging enough:D

[morsa]

@Manao:
Could any of this be usefull for MVinterpolate??

http://www.utdallas.edu/~aria/mcl/motion.html

[Manao]

This interdependence necessitates a costly iterative approach to the computation of motion. [...] This is not a trivial task, since a straight forward insertion of BMA motion vectors in the mesh model leads to unpredictable and erratic results. Yes, it could be useful, but no, I won't implement it, because results will be too slow and too complicated to use.

[morsa]

Ohh, sorry didn't know it or understood it. :(

[Fizick]

Manao, thanks for your hard work!
1. As i understand, you still have no plan to include overlapped blocks motion estimation and compensation?
(for deblocking)

2. So, may i consider your vectors format as stable?
I want try use vectors from MVTools for global motion estimation for Depan as alternative method.
Or MVtools have some internal global motion estimation?

[Fizick]

Manao, i still will post here about development :)

I developed first beta (alpha) version of MVtools to Depan interface, as MVDepan function of MVTools.
The global motion extraction algorithm is still not very clever and robust in this version, but it works as idea.

I used v.0.9.8 as base.
I also add MVDepan function description to updated doc.
http://bag.hotmail.ru/mvtools/mvtoolsdepan.zip
I add 2 files (MVdepan.cpp and MVdepan.h) and add some lines to interface.cpp.

May we use this method of MVTools-Depan interface?
Or it is inconvenient for development and using?

I also try use v 0.9.8.1, but got compiler (VC ++ Toolkit 2003) error:
...MVCore.h(9) : fatal error C1083: Cannot open include file: 'boost/shared_ptr.hpp': No such file or directory

[Manao]

Sorry about the "boost/shared_ptr", i forgot to delete it before releasing the filter ( i was experimenting a little )

I've release MVTools v0.9.8.2 (http://manao4.free.fr/MVTools-v0.9.8.2.zip) containing MVDepan. If you make modifications, make them from that package ( portability.h changed a little, it won't suffer from the ugly bugs that were in 0.9.8 ( it's a miracle that it was working ... ), and boost won't bother you again )

It will be the last release of the MVTools for some time.

[Fizick]

Thanks, it is compiled and works now!



BTW, do you know about DIRAC project?
http://dirac.sourceforge.net/
It has some motion vectors code.

[scharfis_brain]

I want to place a complete different question here.
I wanted to write some AVS-stuff that is able to handle the mismatches of mvtools.

my intention is, to detect the deviations of full compensated frames vs. their uncompensated originals.

so, one is able to replace extreme mismatches in a compensated frame with its original areas.

the current thing I have written in AVS is only able to do this with one compensated clip.

but if I want to use two or three compensated clips
(depanned, fwd-compensated, bwd-compensated)
I really don't know how to handle this with AVS.

Is there a possibility to get such thing written as plugin for AVS?

possible syntax:

corrector(clip original, clip compensated1, clip "compensated2", clip "compensated3", clip "compensated4", int threshold)

where the first compensated clip has the most priority above the following ones.

if all clips are matching the original, the clips are getting blended together.
if only one is not matching, all clips exept the non-matching one are mixed together.
if only one clip is correct, only this correct one will be returned.
if no clip matches, the original is given to the output.

(of course, not framewise, all is meant to be pixelwise.)

I hope, this wasn't too confusing

[Manao]

I've got some free time, so here it is :

http://manao4.free.fr/MVTools-v0.9.8.3.zip

Filter's name is Corrector. It can take any number of clips as input. Threshold's name is simply 'th'.

[scharfis_brain]

That's great!
I didn't expected such fast reaction!

but unfortunately I won't be able to test this until next weekend :(

but maybe another idea:
only return the clip with the lowest deviation.
(or give this clip the most weight in the blend)

only if all clips reach the thresold (too high deviation), the original gets returned.

I intend to use this corrector in mvbob() from mugfunky.
this should speed up things while improving quality.

but as I said: wait for next weekend..

[Mug Funky]

maybe it's just my whacked-out computer, but mvcompensate doesn't seem to be compensating anymore.

try:
pel = default(pel,1)
blocksize = default(blocksize,8)

vfw = c.mvanalyse(pel=pel, isb=false, blksize=blocksize)
vbw = c.mvanalyse(pel=pel, isb=true, blksize=blocksize)

interleave(c.mvcompensate(vfw),c,c.mvcompensate(vbw))


it's just returning the current frame, so i get 3 dupes instead of 1 frame and 2 compensations.

[edit]

what on Earth's going on here? if i precede this with "reduceby2" all's well. but if i leave it out it does a NOP.

full script:

function MCdespot (clip c, int "limit", int "pel", int "blocksize")
{
limit = default(limit,6)
pel = default(pel,1)
blocksize = default(blocksize,8)

vfw = c.mvanalyse(pel=pel, isb=false, blksize=blocksize)
vbw = c.mvanalyse(pel=pel, isb=true, blksize=blocksize)

interleave(c.mvcompensate(vfw),c,c.mvcompensate(vbw))
#yv12lutxy(last,last.clense(),yexpr="x y - abs")
#degrainmedian(mode=1,limitY=limit,limitUV=limit).selectevery(3,1)
#selectevery(3,1)

}



#ASYNTHER AVISource
AVISource("C:\Ack\test2.avi")

tdeint(1,1,tryweave=false)
#reduceby2()
converttoyv12()
reduceby2()
#MCdespot()

mcdespot()


test clip is huffy. you can find it here:
http://forum.doom9.org/showthread.php?s=&threadid=86163

it's cheops' capture, not mine :)


[edit 2]

seems to be a size limit. 720x576 didn't work, nor 640x480, or 512x384. but 384x288 and 384x384 worked. i wonder why it never did this before?

[Manao]

Just a tip : MVCompensate doesn't compensate on what he thinks is a scene change....

[scharfis_brain]

Hi Manao.
I forgot to mention one really important thing for the corrector. Sorry about that!

It should be like this:


Corrector_double( \
clip original_analysis, clip compensated_1_analysis, [...], clip compensated_n_analysis, \
clip original_output, clip compensated_1_output, [...], clip compensated_n_output, \
int threshold, int mode)


its analysis behaviour should be exactly like the current corrector (with both modes! many thanks for that!).

the analysis_clips are building a mask (or whatever) to let the output of Corrector_double() being built using those masks from the analysis-clip and the output clips.

only with this implemtation of corrector, mvbob will benefit from it, cause I do the analysis on vertical blurred clips. and the correction inherently is done on some other crip clips.

(If I would to the analysis and correction on the crisp clips, all line-jaggyness will sneak through the corrector :( )

[Mug Funky]

Originally posted by Manao
Just a tip : MVCompensate doesn't compensate on what he thinks is a scene change....

has the syntax changed? thSCD1 is throwing an error for me.

i think i'll dig up and older version of MVtools for now.

[Manao]

It works on my computer. Strange. What error message do you get ?

[Mug Funky]

ah, never mind. i was putting it in the wrong place.

[tsp]

I found a memory leak that I think originates from MVTools when used with Invoke.

It's present in my filters ml3dex an medianblurt wtih motion compensation enabled. See this thread (http://forum.doom9.org/showthread.php?s=&threadid=84636) to get the filter including source.
At the moment it only works with MVTools 0.9.8.1 until I figure out how to use a named argumentlist with Invoke. The following script will cause a memory leak:

ml3dex()

while won't:

c=last
vfw = c.mvanalyse(pel=1, isb=false)
vbw = c.mvanalyse(pel=1, isb=true)
interleave(c.mvcompensate(vfw, thSCD1=300),c,c.mvcompensate(vbw, thSCD1=1000)).ml3dex(false).selectevery(3,1)


the code in ml3dex that invokes MVTools look like this:
[/code]
PVideoFrame srca[3];
if(MC){
PClip vectors;
AVSValue MVAnalyseArgs[12]={child,blksize,pel,0,2,1,false,lambda,1,true,true,12};
AVSValue MVCompensateArgs[6]={child,vectors,true,0,thSCD1,thSCD2};

vectors = env->Invoke("MVAnalyse",AVSValue(MVAnalyseArgs,12)).AsClip();
MVCompensateArgs[1]=vectors;
PClip compen=env->Invoke("MVCompensate",AVSValue(MVCompensateArgs,6)).AsClip();
srca[0]=compen->GetFrame(n,env);
srca[1]=src;
MVAnalyseArgs[6]=true;

vectors = env->Invoke("MVAnalyse",AVSValue(MVAnalyseArgs,12)).AsClip();
MVCompensateArgs[1]=vectors;
srca[2]=env->Invoke("MVCompensate",AVSValue(MVCompensateArgs,6)).AsClip()->GetFrame(n,env);
if(markscenechange)
if(srca[0]->GetReadPtr(PLANAR_Y)==srca[1]->GetReadPtr(PLANAR_Y)||srca[2]->GetReadPtr(PLANAR_Y)==srca[1]->GetReadPtr(PLANAR_Y))
scenechange=true;
else
scenechange=false;
}
[/code]

and I can't see that it should cause a memory leak.

[Manao]

Where exactly do you use that code ? I'm afraid you're using it in the GetFrame method of your filter, which is highly uneficcient, because there's a lot of memory assignment when initializing MVAnalyse.

Now, for the 'memory leak' explanation : since MVTools 0.9.7, there's a static object, MVCore::mvCore, which handle the memory allocation & stockage for every frame that is interpolated. The releasing of this memory is done when the object is destroyed. It basically means that with the way you're using the MVTools, mvCore is stocking every frames in memory, hence the memory leak.

Of course, the issue will disappear once you use MVanalyse in the constructor of your filter.

On a sidenote, a much more clean way to use it, imho, would be to provide with your filter an avsi script wrapping your filter an invoking MVAnalyse & MVCompensate. You'd then give to your filter only results of MVanalyse and MVCompensate, making it easier to manage if i (once again) change the dyntax of the mvtools. And it will let the 'power users' use their own custom settings for MVanalyse if they want to.

[tsp]

Thanks for the explanation. The code was placed in GetFrame after I moved it to the constructor the leak disappeared as you predicted and the speed increased 30%. The problem with placing the code in an avsi file is that at least with medianblurt the temporal radius is variable and until there is a loop construction in avisynth it's not possible to make such a script. Also the power user can always disable the motion compensation and do it them self and for the beginner it's easier to use.
It is a problem what I'm quite dependent on you not changing the syntax to much. It will not be such a big problem after I figured out how to use Invoke with named args. So if you don't change the parameter names to often it will be fine :) .

Another way to cause a leak with MVTools is this script:


scriptclip(s,"ml3dex1(s,10)")

function nop(clip c,int sc){
vfw = c.mvanalyse(pel=1, isb=false, blksize=8)
vbw = c.mvanalyse(pel=1, isb=true, blksize=8)
interleave(c.mvcompensate(vfw, thSCD1=sc),c,c.mvcompensate(vbw, thSCD1=sc)).selectevery(3,1)
}

The strange thing is that the destructor is called after every frame so it shouldn't cause a leak. Replacing sc with a constant doesn't change this :confused: .

Still MVTools is a really powerful tool for all these temporal filters even if you can't use it with scriptclip.

[Manao]

My MVCore::mvCore object is staying alive during the whole script execution, so no deallocation. If you want to avoid that constant memory allocation, use the idx parameter of MVAnalyse, and set it to a positive value : in that case, frames will be allocated only once, and when needing a new memory space, the oldest frame stored will be reused.

Now, why do you want to use script clip for ? What is the parameters you want to make vary ?

And finally, you'd get a better speed if you were using the interface i provide with the MVTools in order your filter to use them directly : you need to include a lot of files from the MVTools yet ( I didn't get time to make a proper interface yet, I'll do it today ), but deriving your class from GenericMotionFilter would allow you to get the result of the compensation without having to extract it with MVCompensate. And you'd also get the validity of the motion estimation for a particular block ( through the block's sad value )

[Fizick]

My small comments:
1.
an avsi script wrapping your filter an invoking MVAnalyse & MVCompensate. You'd then give to your filter only results of MVanalyse and MVCompensate, making it easier to manage if i (once again) change the syntax of the mvtools. And it will let the 'power users' use their own custom settings for MVanalyse if they want to.

I think it is the best method, if you do not use vectors,SAD, etc.

2.
using the interface i provide with the MVTools in order your filter to use them directly : you need to include a lot of files from the MVTools yet

If we all will make our filtes so, the MVTools soon will become the single remaining Avisynth filter :D

[Manao]

No, i'm making an interface right, so you don't have to include the filter inside the mvtools. I never done interfaces yet, so it may takes some times.

[tsp]

[qutoe]
My MVCore::mvCore object is staying alive during the whole script execution, so no deallocation. If you want to avoid that constant memory allocation, use the idx parameter of MVAnalyse, and set it to a positive value : in that case, frames will be allocated only once, and when needing a new memory space, the oldest frame stored will be reused.
[/quote]
Using idx didn't work. It only slowed the process down. I tried this script:

global s=AVISource("c:\test.avi")converttoyv12()
scriptclip(s,"nop(s,10)")

function nop(clip c,int sc){
vfw = c.mvanalyse(pel=1, isb=false, blksize=8,idx=12)
vbw = c.mvanalyse(pel=1, isb=true, blksize=8,idx=12)
interleave(c.mvcompensate(vfw, thSCD1=1000),c,c.mvcompensate(vbw, thSCD1=1000)).selectevery(3,1)
}


and virtualdubmod crashes after about 1800 frames when the allocated memory reaches 2 GB. Seems as if there is a problem reusing the old frames.


Now, why do you want to use script clip for ? What is the parameters you want to make vary ?


At the moment I don't need it. Just pointing out that if someone is using MVTools or a filter what calls MVTools inside scriptclip/conditionalfilter or frameevaluate the problem will arise.


And finally, you'd get a better speed if you were using the interface i provide with the MVTools in order your filter to use them directly : you need to include a lot of files from the MVTools yet ( I didn't get time to make a proper interface yet, I'll do it today ), but deriving your class from GenericMotionFilter would allow you to get the result of the compensation without having to extract it with MVCompensate. And you'd also get the validity of the motion estimation for a particular block ( through the block's sad value )

That would be a good idea. It will be the best solution. I will look forward to what speed improvement calling MVtools directly will bring. Maybe in avisynth 3.0 MVTools will be in the core.

The speed difference now between calling MVTools using Invoke and wrapping the filter in an avsi file is about 0.3%(in favor of Invoke).