Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 28th October 2005, 16:17   #21  |  Link
mg262
Clouded
 
mg262's Avatar
 
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
Thanks, guys... but the beauty of AVISynth (as a package) is that it does all the slow to write, reference-requiring, technology-dependent parts like opening files and rendering, so that's where most of the credit should go!

I briefly tested sub pixel (1/4 pixel) accuracy in a rough form in C, and it helps a little, but the blocking and accuracy still much worse than that of MVTools. The last time I tried MVTools was at least 18 months ago (it's pretty rare that I work with non-animated material), and I hit substantial blocking... but now, at least on the source at hand, blocking seems very rare and its fast -- this runs at 50FPS:*

mvanalyse(source)
mvcompensate(source, last)

I was pretty disappointed with the comparative performance of this method -- the papers on it certainly give the impression that as well as being super-fast and producing true motion, it gave very good results with minimal blocking. It's possible that I made some silly mistake (and I'll check through), but the vector fields look pretty sensible. My particular interest was in trying to develop this for denoising scripts, and that seems relatively pointless now. Don't take that the wrong way -- I'm quite happy to try other things with it, it's just that denoising was the target I was heading towards. So, I'm not really sure where if anywhere to go from here... thoughts?

*Motion compensated scripts (e.g. MVBob) have the reputation of being really slow... why is this?

____________________
I ran across one simple deblocking method for truemotion motion compensation which I might as well repeat here:

Obtain motion vectors, then subdivide each block into four.
For each top-left subblock:

Replace the motion vector with the median of
-- original motion vector
-- vector from block to the left
-- vector from block above

Median is computed in x and y separately. (+Optionally, you discard the vector if it doesn't much one of the three above.) Other 3/4 sub blocks work in the obvious way.

AFAICS, the effect is to take a stair step and reduce to a stair step of half the size. The method is particularly intended not to introduce new motion vectors which didn't exist before.

Last edited by mg262; 28th October 2005 at 16:21.
mg262 is offline   Reply With Quote
Old 28th October 2005, 17:01   #22  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
Main aim of true motion is motion interpolation.
Fizick is offline   Reply With Quote
Old 28th October 2005, 17:10   #23  |  Link
mg262
Clouded
 
mg262's Avatar
 
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
Quick note: I took a guess at the meaning of a word ("meandering") apparently defined in some papers I can't get hold of... whether or not I guessed right, it seems to help under some circumstances. So...

Motion, 28 October 2005

If you really want to try quarter-pixel accuracy the slow C way, add subpixel = true and useassembly = false. It only runs at 30 FPS or so on 720 x 576, so for testing purposes I would recommend cropping down your clip until you can watch it in real time. (I probably won't have time to assemblify it for a while.)

Last edited by mg262; 28th October 2005 at 17:50.
mg262 is offline   Reply With Quote
Old 29th October 2005, 22:09   #24  |  Link
mg262
Clouded
 
mg262's Avatar
 
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
Motion, 29 October 2005
Bug fix for meandering and slightly faster assembly integer search code.

I've been browsing through papers and I think that at least half-pixel accuracy is needed to do anything useful (because of the half-pixel shift between fields). Quarter-pixel accuracy is preferable -- although for the assembly, I can't at present see a way to deal with quarter-pixel (without sacrificing speed) other than by coding 16 cases... ... ... . (Joy.) So I will probably fight towards that and then think about applications.

@Mug Funky, if you're reading this, anything particular you would like to see this filter do?

Quote:
how are you thinking of handling occlusion and revealing of objects? i suppose that'll come with backward prediction. doing a subtract-from-source shows crossing objects are a problem area.
I have read more about the problem domain now (there are downsides to diving in headfirst ), so I'll try and answer this. First, it's not going to be dealt with at the motion detection stage, simply because there's no way of representing the information without complicating the datastorage format substantially. I.e. an object that has become revealed can't be sensibly represented with any motion vector. Second, depending on the application, we can try and infer this information from the motion vector field; for example we can look for discontinuities, and fallback to some simpler method (for whatever the application is).

Incidentally, I am not sure how well this kind of method will work on foreground objects in animation; the algorithm is not terribly likely to deal well with the fact that their motion is typically at a lower FPS than the motion of the main clip...

Last edited by mg262; 29th October 2005 at 23:06.
mg262 is offline   Reply With Quote
Old 30th October 2005, 14:44   #25  |  Link
psme
Registered User
 
Join Date: Mar 2005
Posts: 68
Tried both (P4 2.66 overclocked to 3.8G):

motion = findmotion(source, useassembly = true )
compensate(source, motion)
(around 20% CPU usage)

and

mvanalyse(source)
mvcompensate(source, last)
(around 55% CPU usage)

in FFDShow->Avisynth for real time DVD playback. Both methods give smooth playback. But both methods give tons of pixelization/blocking artifact in motion area.

I guess the filter is not yet usable...

regards,

Li On
psme is offline   Reply With Quote
Old 30th October 2005, 16:18   #26  |  Link
mg262
Clouded
 
mg262's Avatar
 
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
@psme,

Motion compensation filters aren't as far as I know intended for direct use in playback... they are intended to provide high-quality input for temporal processes, such as temporal denoising, frame rate conversion (as Fizick mentions) or deinterlacing.


@all,

Edit: scratch what was here... I have thought of a modification that may produce the backwards motion information without requiring backward tracking.

Last edited by mg262; 30th October 2005 at 16:32.
mg262 is offline   Reply With Quote
Old 30th October 2005, 17:27   #27  |  Link
castellandw
Registered User
 
Join Date: Sep 2005
Posts: 90
I mentioned in some other post about motion compensation using phase correlation being used by video engineers, especially for broadcast standards conversion (PAL to NTSC, NTSC to PAL, etc). Mug Funky mentioned that this could be accomplished by block overlapped motion compensation. Of course, MVTools can be used for motion compensation, but Fizick recently created a Block Overlap Plugin along with a sample script to use with MVTools to create block overlap motion compensation (I tested out Fizick's block overlap plugin with MVTools and it did produce better results). Mug Funky also mentioned that phase correllation could be achieved with the DePan plugin, but also some other tricks need to be involved. Mug Funky, do you know what other tricks need to be involved for AviSynth since it's possible to achieve block overlapped motion compensation in Avisynth (Block Overlap Plugin By Fizick with MVTools) and with phase correlation (possibly Depan plugin)?

@mg262, that message thread where my link to the pdf on Snell Converter's guide to motion compensation was posted, I posted the link to that message thread on another site with big expert video engineers, and one video engineer told me that motion compensation was revolutionary in 1994 and no other technique was surpassed it. So what I'm saying is that those papers posted on that message thread, especially papers by De Haan or true motion estimation using feature correspondences don't compare to that motion compensation technique with phase correlation according to that guy.

@all, any comments on what I mentioned about motion compensation?
castellandw is offline   Reply With Quote
Old 30th October 2005, 18:25   #28  |  Link
mg262
Clouded
 
mg262's Avatar
 
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
That sounds very interesting... although I'm not sure how to read this:
Quote:
one video engineer told me that motion compensation was revolutionary in 1994 and no other technique was surpassed it.
... most of the stuff I'm looking at is post-94 -- although maybe that's not what you mean?

I would really like to see the site you refer to if you happen to have the link at hand...
mg262 is offline   Reply With Quote
Old 30th October 2005, 20:55   #29  |  Link
Joel Cairo
Registered User
 
Join Date: Oct 2005
Posts: 18
mg262:

Unless I'm mistaken, he's referring to the Doctor Who Restoration Team website, which is staffed by the members of that team-- some of whom are BBC engineers. You can find them at:

http://www.rtforum.co.uk/

THe thread is currently near the very bottom of their page, and it's titled "Snell Converter's Motion estimation methods said to be dated". You'll have to check it quickly, though, as topics drop off their board after a couple of weeks, and I don't believe there's an archive.

As I read the discussion though, I actually felt that our discussion was somewhat mis-represented. I don't recall anyone here saying that the Snell converter's inner workings were dated...

Anyway-- that's where you can find the discussion.

-Kevin

Last edited by Joel Cairo; 30th October 2005 at 21:01.
Joel Cairo is offline   Reply With Quote
Old 30th October 2005, 22:01   #30  |  Link
mg262
Clouded
 
mg262's Avatar
 
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
Thank you for the link:

http://www.rtforum.co.uk/read.php?id=128649
Quote:
Charlie S
Snell Converter's motion estimation methods said to be dated

Guys, I'm at my wits end. Steve Bagley, if you're reading this, I posted the link to that paper on motion compensation from Snell Converters that you gave me and posted it on another website forum. The reaction I got says that the paper being from 1994 is very sad being dated because of still using blending techniques, and I think I'll post this link:

http://forum.doom9.org/showthread.php?t=100791
I find this a little confusing... it reads as if I posted it (which I didn't). Perhaps Charlie S posted a similar query on a different forum but linked here instead?

Last edited by mg262; 30th October 2005 at 22:06.
mg262 is offline   Reply With Quote
Old 30th October 2005, 22:28   #31  |  Link
Joel Cairo
Registered User
 
Join Date: Oct 2005
Posts: 18
Well, I think this is where the misunderstanding began. There's a post just below yours in that thread (by Mfa), which makes the ironic point that the document you linked was written over 10 years ago, but that people still weren't employing the techniques that are outlined in that document. Instead, they're still using blending, etc.

The way Mfa's message was written, however, it evidently was mis-interpreted by the poster as meaning that blending, etc. was part of the method outlined in that document-- which is not correct, if I'm understanding all this properly.

So I think that's how all of this confusion got started.

-Kevin
Joel Cairo is offline   Reply With Quote
Old 31st October 2005, 01:21   #32  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
Discussion is interesting, there are many ways.
Very interesting, for example, recent bi-directionanal motion estimation and interpolation.
Fizick is offline   Reply With Quote
Old 31st October 2005, 01:30   #33  |  Link
castellandw
Registered User
 
Join Date: Sep 2005
Posts: 90
Guys, it's a bit hard to follow you guys from one forum to another(and a bit of a headache) because the message I posted in the Doctor Who restoration team forum has been a few weeks old, and so I'm unaware where you guys are posting what unless you tell me. Yes, I posted the message in the Doctor Who restoration forum to get a second eye because I knew I could have misunderstood this message thread:
http://forum.doom9.org/showthread.php?t=100791

@Joel Cairo aka Kevin, I would have gathered that I misunderstood, especially when James Insell at the Doctor Who restoration team forum told me that no blending is being used for motion compensation. When I posted it, I assumed that when Mfa said stuff like blending is still being used, I thought he was referring to the Snell converter paper which I seem to be the first to post on the Doom9 forum:
http://forum.doom9.org/showthread.php?p=718712#post718712

Also, I was referring to Snell converter's methods being possibly dated compared to other papers posted as well which are papers by De Haan and true motion estimation using feature correspondences. Thus from there, James Insell told me that no other method has surpassed motion compensation. I didn't try to misrepresent the discussion here because I posted the link to that thread to get a second eye on the subject.(By the way, Kevin, since you mentioned discussion on the Doom9 forum being misrepresented, was there any other discussion on Snell converters apart from that link to that pdf file being discussed because I never managed to really find any?)
One more thing, I didn't post the Doctor Who restoration team forum link http://www.rtforum.co.uk here because when I asked about what they thought of PAL to NTSC conversion done on AviSynth, they said I should go to Video Editing forums, and so I had to limit my posts there with simpler explanations of techniques as best as I could when I referred to Avisynth software. So I knew it wouldn't be worth it to acknowledge the Doctor Who restoration team link if they might likely say to you guys elsewhere to go to other forums if you mention video editing software.

@mg262, I did post that Snell Converter paper first on the Doom9 forum apparently, and I assumed that you found that first from my post. I had no idea you found the paper first on Google because I didn't realize it was easily findable on Google. You're right, it does seem as if you posted as Charlie S, sorry about that, but I thought the reaction from your post was because of me posting the link to that Snell Converter paper first in the forum. I didn't post a link to my original post on Doom9 with the Snell Converter pdf link on the Doctor Who restoration team because the reaction to your post was more relevant. By the way, Mohan, you better post what you put on the Doctor Who restoration team over here as well because it'll make things very confusing for other posters looking too and as Kevin said, messages on the Doctor Who restoration team forum drop off..
Here's again the link where I originally posted the Snell Converter paper:

http://forum.doom9.org/showthread.php?p=718712#post718712


I see Kevin, has found this link from the Doctor Who restoration team:
http://www.rtforum.co.uk/read.php?id=128649
but did you guys manage to find another one I posted which has now dropped off the Doctor Who restoration team forum. Well, here's the link again on my original post involving the link to that pdf on Snell Converters involving motion compensation as well as discussion by Mug Funky on how it could be possible to do motion compensation using phase correllation:
http://forum.doom9.org/showthread.ph...376#post719376



Here's Fizick's post on the block overlap plugin I mentioned earlier that can be used with MVTools:
http://forum.doom9.org/showthread.php?t=101307
castellandw is offline   Reply With Quote
Old 31st October 2005, 01:48   #34  |  Link
castellandw
Registered User
 
Join Date: Sep 2005
Posts: 90
Oh yeah, a British sci-fi show about an alien that travels through time and space in a machine that's bigger than the inside than it is on the outside is stereotypically British. I can see that coming, Mfa.

Anyway, as I mentioned in the other post I just posted, MfA, I wasn't referring only to your comments because there was mention of papers by de Haan being mentioned on that Snell Converters paper, and your comments about blending can be understood in more than one way.
castellandw is offline   Reply With Quote
Old 31st October 2005, 02:16   #35  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
I'd appreciate if you never speak for me again ... I don't appreciate having to go into a strange forum to clear up a gross misrepresentation of what I said only to get deleted. It sets bad blood all around, you are pissed because you are told off. They are pissed because they think I slagged Snell Wilcox and because they think the thread went way too far off topic, and my double posting probably didnt help either, and this will reflect in their opinion on Doom9 and everyone involed. Lastly I am pissed, because it is very hard to reply in any way which wont turn this into a flamewar which will probably result in me getting a warning.
MfA is offline   Reply With Quote
Old 31st October 2005, 03:03   #36  |  Link
castellandw
Registered User
 
Join Date: Sep 2005
Posts: 90
Mfa, they didn't delete the message thread because of your post or being pissed at you. If you noticed, that message thread was the oldest one on the board, and so it just dropped off. Nothing to do at all with any slagging to Snell-Wilcox or the thread going way off-topic. As long as everybody was discussing technical video bits on the forum, it wasn't going off-topic and I gave the link to that thread so there is no way they'd think you're just slagging off Snell-Wilcox or actually care judging by how much attention was given to that message thread. Hey, lots of people double-post on that forum, so you're not the only one. By the way, I was not speaking for you, I was trying to get a second eye from those guys at the Doctor Who restoration team forum because they're the best video engineers, especially from their restoration work they've done, I could find to give a second opinion on all the replys to that message thread on motion compensation in general. It was my misunderstanding on your post, but they are not pissed off at you or anyone at Doom9 forum. Believe me. Besides, when I mentioned Avisynth, they practically asked me to ask about AviSynth in other video editing forums until I tried to limit my posts to just regular video techniques in general. Look, as I said, you did nothing whatsoever to piss them off, and the message thread outlasted its own timespan on that forum.

Last edited by castellandw; 31st October 2005 at 03:07.
castellandw is offline   Reply With Quote
Old 31st October 2005, 03:58   #37  |  Link
Mug Funky
interlace this!
 
Mug Funky's Avatar
 
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,547
Quote:
Originally Posted by clouded
@Mug Funky, if you're reading this, anything particular you would like to see this filter do?
i think that's already been covered - mesh warping would be very cool, and it seems it's been on your list from the start. i'm really just happy to see stuff being developed so actively for avisynth (particularly by you - i think you're a contender for "most new plugins written per month". i'm in awe).

[edit] actually, a "delta" option similar to that of mvtools would be great for denoising.[/edit]

i'm sure the algos you've implemented cover this to some degree, but if frame interpolation is involved, things like object acceleration and whatnot will have to be covered to some extent. i guess that would mean sampling at least 3 frames to derive the velocity/acceleration of an object anywhere in the middle. this is probably already done if i'm understanding correctly.

as far as the frame-rate issue with foreground/background in animation, don't be worried about that. not even the PhC stuff handles that properly (the paper itself says "garbage in, garbage out" when talking about something as dead-simple as telecined material... that could easily be handled in hardware by a separate hard match IVTC running before the PhC part, and i'm sure modern machines do just that). certainly i've seen plenty of stuff on TV that's gone through some kind of mocomped conversion, and it's clearly failed badly. "iron chef" is an example. fuzzy subtitles, and an ex-composite source (dot crawls) make it a difficult source. typically a medium-fast pan is handled perfectly, but any faster and it's blended. subtitles + mesh-warping aren't a good mix either. i see no shame in falling back on field-blending in lots of cases. for small areas of a frame, it's much more transparent than warpage or other artefacts.

@ castellandw:

about using depan... i said that partly in jest. though it could be done, there are severe limitations. you'd have to chop the clip into large blocks (depan will only find motion that fits within the frame), then do overlapped compensation, then split into smaller blocks, run again, then split into smaller blocks.... it'd be slow and poor quality (the overlapped blocks will not match in most cases), but would work as a proof of concept for the phase-corellation method. there's definitely many more ways to skin that cat though . i certainly wouldn't want to script a frankenstein's monster script like that though.
__________________
sucking the life out of your videos since 2004

Last edited by Mug Funky; 31st October 2005 at 05:44.
Mug Funky is offline   Reply With Quote
Old 31st October 2005, 04:54   #38  |  Link
castellandw
Registered User
 
Join Date: Sep 2005
Posts: 90
@Mug Funky, that sounds like it could use a recursive hashing algorithm although if it's poor quality particularly with overlapped blocks not matching, then I begin to wonder how video engineers can produce results with motion compensation using phase correllation. So every time it separates into smaller blocks, phase correlation can be performed before overlapped-block motion compensation. So, what exactly what is at least one example of a way that it can can be done to "skin that cat" of dealing with overlapping blocks not matching because I'm wondering how it's possible that video engineers produce decent results with standards converters?

By the way, Fizick and anyone involved in development with MVTools should look at this discussion here on motion compensation using phase correllation because it might prove extremely helpful.
castellandw is offline   Reply With Quote
Old 31st October 2005, 05:26   #39  |  Link
Mug Funky
interlace this!
 
Mug Funky's Avatar
 
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,547
well, the only reason the way i described would be bad is because in depan only half the job is done - it makes no attempt to find motion vectors, rather it performs phase corellation on a whole frame to find global motion.

in a PhC unit (from reading the paper), there is a phase corellation stage that finds several (i guess) local maxima that correspond to motion-vectors, and then there's a stage where it matches the motion vectors to actual locations on the screen. this makes it far far far superior to just matching global motion on ever smaller blocks.
__________________
sucking the life out of your videos since 2004
Mug Funky is offline   Reply With Quote
Old 31st October 2005, 11:01   #40  |  Link
mg262
Clouded
 
mg262's Avatar
 
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
@castellandw,

Now I follow. (That paper is the first hit when Googling "motion compensation".) What I posted on that forum amounts to little more than Fizick said above; it was mainly to elicit a more detailed response from James Insell. Now the thread is gone, let's leave things be.

@Mug Funky,

Quote:
actually, a "delta" option similar to that of mvtools would be great for denoising.
It's trickier in this instance because each motion estimate depends on the motion estimate for the previous frame... I could create something that mimicked SelectEven.FindMotion, but I think the found motion vector quality would degrade. On the other hand, this is perfectly legit:

#source = ...
motion = findmotion(source, useassembly = true )
source
compensate(motion)
compensate(motion)

I.e., get the motion vectors once, move blocks to compensate original frame n-1 -> n, then move blocks to compensate compensated frame n-1 -> n. So we have:

original frame n-2 -> compensated frame n-1 -> doubly-compensated frame n

Will that do? I can definitely implement something else, given specifics.

By the way, using that ^^ script will make the blocking/inaccuracy worse for obvious reasons. There are a range of things which can be done to improve the motion vector quality, but I think I'm going to try and get an application out first... it makes it easier for us to check whether a particular change is actually helping. For similar reasons, I would like to have better quality motion vectors before trying mesh warping.

From Phase Correlated Motion Estimation:
Quote:
Phase-correlation ME is very computationally efficient and it produces much smoother motion field with low entropy than the BM method does. Phase-correlation works better than the BM does in the cases of large scale translational motion, while BM is more suitable for predicting regular and small scale motion and multiple-object movement.
(Bear in mind that that compares phase correlation against full/logarithmic search block matching... actually I think it's some student report, so doesn't carry that much weight in itself -- but the conclusion looks very sensible to me.)
mg262 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 07:02.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2017, vBulletin Solutions Inc.