Need help cleaning up some footage. (pic included)

[mimage]

I think what they call this problem is dot crawl. I've gotten rid of the problem before like a year ago in Avisynth or Vdub but I don't exactly remember how I did. There was a site I went to that had descriptions and examples of the problem and there own filters to use to get rid of them.. sites like this.

http://www.compression.ru/video/index_en.htm

I'd like to get rid of the problem in Avisynth.. so if you know any good settings/filters for fixing this let me know. Here's an example of the problem..

http://img182.imageshack.us/img182/347/screenhunter0025xp.jpg
http://img241.imageshack.us/img241/2387/cap25ba.gif

Notice the dot's in the image? If you know any good way of fixing this, please let me know.

[Guest]

Use search: "dot crawl"
:search:

[mimage]

Well you see the problem.. no? Just wanted some suggestions. I've cleaned it up with some dotcrawl filters but I know there's probably quite a few out there I don't know much about. I've used Undot, dedot, Guava Comb, and maybe some others that do the same sort of thing. I was just looking for suggestions. I've already been to a lot of sites that talk about them.

http://www.michaeldvd.com.au/Articles/VideoArtefacts/VideoArtefactsDotCrawl.html

So yeah.. I guess I get the basic idea of what it is.. still don't see the harm in asking people for suggestions on fixing it. :(

[Guest]

Why duplicate the existing threads? Everything we know about it is there in existing threads. If you had some unique new problem or perspective, a new thread would be justified, but this is just typical dot crawl.

[mimage]

alright. sorry then.

[Didée]

Everything we know about it is there in existing threads. Surely everything?

"Well, not entirely... One small village of indomitable Gauls still holds out ..." ;)

Can't remember that deep temporal filtering, by means of motion compensation to also catch moving image parts, was ever mentioned. Always wondered why ... it's an obvious approach to combat the problem, and once I used it sucessfully on a music clip that suffered from really *strong* dot crawl.

[Guest]

That qualifies as a new perspective, then.

So, lay it on us.

[Didée]

LTSMC probably will do it more or less natively - even the old raw version already does so. But the planned Xmas present is another one ... one step after the other. ;)

[Mug Funky]

out of curiosity, what DVD is this from - title and region, if possible. also, format would help (NTSC or PAL?). i see what looks like deinterlace artefacts in the very top line of that screenshot you've given (exactly the sort that low-threshold leakkernelbob gives when fed an analog source with dot crawls).

Didee - that sounds very interesting, if a little slow :)

one terminal problem with dot crawls (and DeDot) is that they're a real pain to remove after standards-conversion has taken place.

also, if something's gone through 2 or more generations as composite video (ie tapes dubbed through composite from an ex-composite source is quite common), the dot crawls don't match and make it impossible for DeDot to remove.

i could show you a source like this. it'll make your eyes bleed out your wide-open mouth :) huge dot-crawls, rainbows and surprisingly high-level analog noise. if not for the sharpness, i'd say it's ex-VHS. the sad thing is i've had to run this source as-is through a standards converter, adding blended fields to the mix. even worse is it's digital anime - the only loss it should have suffered at all is the 2:1 compression on the digibeta it came on.

[mimage]

It's from the DVD of Bible Black: New Testament Region 1

NTSC

[Mug Funky]

aaah... i always wanted to see that (now i know why you didn't post a VOB sample).

if it's NTSC then throw DeDot at it before IVTC and degrainmedian after IVTC.

try:

mpeg2source("zomg_perverted_hentai_lol.d2v")

dedot()

your_favourite_IVTC_filters() #don't forget to decimate too if you need to:)

degrainmedian()
fft3dfilter(sigma=3,bt=3,sharpen=1.0)


you might also want to crop the crap off the top, bottom and sides, but dedot will take care of the sides for you.

[mimage]

Wow! That really cleaned it up, though I wasn't able to get fft3dfilter to work. Oh and I didn't crop it and now the top looks clean.

http://img397.imageshack.us/img397/1944/screenhunter0017ts.jpg

Thanks a lot for the help Mug Funky! Really appreciate it! ^_^

[Chainmax]

Aww, and here I was going it would be some sexy hentai :(. Bible Black is the most twisted thing I ever saw (only 2-3mins on a couple of episodes). Nothing more to contribute here since Mug Funky already suggested DeDot, although he didn't provide the download link (http://nullinfo.s21.xrea.com/#DeDot_YV12).

[Mug Funky]

hmm. looking a little too soft after the degrainmedian... (which is why i suggested fft3dfilter).

if fft3d isn't working for you, it's probably because you don't have the fftw libraries installed. but i could be wrong there (it would tell you this if it was the problem).

try limitedsharpen then (and maybe fastlinedarken if it's the kind of anime that allows it - it should be).

hehehe. twisted, huh? i'll have to ask sly from work if he's got this one (he's got an encyclopaedic anime knowledge, but i only ever see him with H stuff for some reason).

[edit]

btw, avisynth is lots more fun if you use virtualdubmod for script editing... it'll help your screenshots too, because i'm not sure if some of the softness and vertical jagginess isn't from some kind of resize being used on playback.

[mimage]

Alright thanks. I'll try some of them out.

Instead of starting another thread.. I thought I'd post some more pics from another series I'm having trouble with that is Pure Interlaced. I have a lot of trouble with Pure Interlaced footage. I have made it so you don't see much blended frames or interlacing lines showing up everywhere.. but it did add another problem to deal with and I'm unsure what it is or how to approach it. sorry I didn't make a clip. If you look at the collar in the pictures.. you'll see a lot of moving black dots moving by and on the outline of the collar. I don't know what to do about it, I'm sure there is some filter out there that might help in cleaning that up.

http://img208.imageshack.us/img208/3715/screenhunter0039ts.jpg
http://img208.imageshack.us/img208/421/screenhunter0042hh.jpg
http://img208.imageshack.us/img208/5492/screenhunter0053rf.jpg
http://img208.imageshack.us/img208/2356/screenhunter0063oo.jpg
http://img228.imageshack.us/my.php?image=screenhunter0071uk.jpg
http://img228.imageshack.us/img228/6889/screenhunter0086pv.jpg

Any help is appreciated. Thanks! ^_^

Edit: OK, I decided to upload a sample.

http://stormmetropolis.com/test/test.avi

[Chainmax]

I'm on it :).

Mug Funky: FFT3DFilter as a dedicated sharpening filter? Is it that good for it? I was thinking of using it in conjunction to LimitedSharpen and aWarpSharp on a VHS capture...

[Chainmax]

That's not a good sample, it's too short and it's a static scene, so there's no way we can discern wether it's really interlaced, telecined or whatever else. It also makes it difficult to spot artifacting. Please post a 5+ seconds sample in a scene with motion.

[mimage]

I know the trying to deinterlace is what caused it. I wanted to use a scene where there isn't suppose to be anything moving in the picture so I gave an example of that clip of that guy just standing there for that reason. It's very small detail I thought there may be some sort of smoothing out technique to fix it. But I'll cook up a sample with motion in a bit, though may be quite a few hours, maybe later today. I had a problem like this before a long time ago with the animated sequence in breath of fire 4.. but instead it was moving lines instead of moving dots.. I forget how I got rid of it but I did save the processing settings but it's on my other computer that my bro is borrowing. Oh and thanks for trying to help.

[mimage]

OK first two tests is The Guyver with no filters.

http://stormmetropolis.com/test/test2.avi
http://stormmetropolis.com/test/test3.avi

and these are with my filters.

http://stormmetropolis.com/test/test4.avi
http://stormmetropolis.com/test/test5.avi

I was gettting hecktic with cleaning the footage.. sorry the script looks so messy. One extra question is are the loadplugins suppose to all be at the top? I don't know if it really matters so much. :) neway here's my slopy script. :o

loadplugin("C:\Programs\Programs\dgmpgdec140\DGDecode.dll")
mpeg2source("H:\The_Guyver_Vol_1\The_Guyver_Vol_1.d2v")
loadplugin("C:\Programs\Programs\AviSynth Plugins\DeDot_YV12_0002\De.dll")
loadplugin("C:\Programs\Programs\AviSynth Plugins\degrainmedian08\degrainmedian.dll")
Dedot()
SeparateFields()
Weave()
TomsMoComp(0,15,1)
loadplugin("C:\Programs\Programs\AviSynth Plugins\TDeintv1b4\TDeint\TDeint.dll")
Lanczos4Resize(720,528)
Crop(6,2,-6,-0)
loadplugin("C:\Programs\Programs\AviSynth Plugins\degrainmedian08\degrainmedian.dll")
degrainmedian()
loadplugin("C:\Programs\Programs\AviSynth Plugins\fft3dfilter184\fft3dfilter.dll")
Lanczos4Resize(720,528)
Crop(6,0,-6,-0)
ConvertToRGB32()

[Chainmax]

If the source is a DVD, why is the sample an AVI and not simply a VOB chunk? What was it compressed with?

A script with just AVISource and BOB() shows what seem to be blended fields, but this is a 29.97fps source and I really don't know how to differentiate between field blending and interpolated frames that belong to the source. Can someone (Mug Funky, maybe?) clarify this for me?

[mimage]

If the source is a DVD, why is the sample an AVI and not simply a VOB chunk? What was it compressed with?
I haven't made a VOB chunk before. I'll look that up or if you have a link, please list.

I compressed in vdub to xvid format.

[Chainmax]

Bad choice. Compressing to a lossy format introduces artifacting that's not present in the source. For instance, test2 showed some blocking that's probably due to compression. Also, if you didn't enabled Interlaced Encoding in Xvid, that might introduce more artifacts. Use a program like ChopperXP (http://www.inchwest.com/chopperxp.html) in order to extract a vob sample.

[mimage]

That's a cool program. :thanks:

http://stormmetropolis.com/test/test.vob

Thanks for all the help. I'm very thankful.

[Chainmax]

Now we're talking. Let's see what can be done about this :).

[Chainmax]

Ok, before going any further I need confirmation from a specialist about wether this:

http://img389.imageshack.us/img389/4158/fbornot0qx.png (http://imageshack.us)

Is field blending or not, as I really don't know how to differentiate between field blending and interpolated frames that belong to the source.

[DryFire]

Here's what I tried this morning at about 2am. I'm pretty sure there's a better way to filter it.

mpeg2source("...\guvyer.d2v")
DeDot()
tfm(pp=7)
Tdecimate(mode=0)
removegrain()
deblock(quant=30)
hqdn3d()
Fluxsmoothst(10,10)
unfilter(-5,-5)
crop(6,0,-10,0)
lanczos4resize(720,480)

Here's the result. (http://www.savefile.com/files.php?fid=8414690)
I compressed it with x264 at 3mbps.

Chainmax that looks like it blongs to the source, but I'm nowhere near an expert.

[Chainmax]

I don't know, how about this (just mpeg2source):

http://img403.imageshack.us/img403/4927/mmm4xf.png (http://imageshack.us)

?

[mimage]

Thanks for all the help guys. :)

[Chainmax]

Hold on, I have a couple of ideas, but I need to know first wether this is telecined, interlaced or field blended.

[mimage]

I'm just saying thanks for the process of helping me out. Very appreciated. :)

I thought it was pure interlaced. 6 seconds into it if you move frame by frame you'll see horizontal lines going through it on each frame.

[dude051]

@Chainmax
My guess is classic field blending. A good reference is AMG OVA. Interpolated frames come about when you interpolate frames (duh! :scared: ) which comes usually from a bobber. This ususally only happens when you change the frame rate... found in NTSC to PAL or vice verse conversion. Interpolation is the term for creating something that wasnt there. In video, it simply means creating extra non-existing frames based off of previous and next ones (which is exactly what a dumb bobber does). Although its possible, this doesnt look like thats the problem. I recommend DeBlend, which looks at previous and next frames and based on many threshold settings it takes out dup blended frames and replaces with a static one.

Edit: Forgot to address...

I thought it was pure interlaced. 6 seconds into it if you move frame by frame you'll see horizontal lines going through it on each frame.

Well it appears pure interlaced because its blended to hell. I would venture to say the horizontal lines are from blending of interlaced frames :( but im not really sure. It's hard to say how they manage to do this stuff back then.

No expert here, just throwin out my 2 cents...

[Mug Funky]

actually, a bobber interpolates missing lines, not frames.

frame rate conversion interpolated frames, of course.

[Chainmax]

So, is this a case for Restore24? If not, what should be done about it?

[Chainmax]

What do you say, Mug Funky?

[Mug Funky]

i'm guessing this is a ye-olde style telecine. ie 24fps directly transferred to 59.94i (no slowdown to 23.976 first, so the numbers don't match 3:2). they usually did this with a tube camera (top-to-bottom scan, rather than full field at a time like with CCDs) and a spinning prism which gives the frames a nice blend.

restore24 might work for this, but be sure to feed it the right framerates or it'll give you jerky. there will probably be blends left over too, for 2 reasons - tube cameras (if one was used) have a bit of a persistence to them - a bright images can stay on the picture for a few frames. also, a temporal soften may have been applied, which would cause more blending. your clip may not be this bad (haven't looked at it, sorry :))

so yeah, give restore24 a try and see what happens, don't worry too much about denoising as it's already quite smooth, and have fun compressing this beast :). your worst case scenario is just having to encode interlaced, which isn't a bad thing at all, it just needs more bitrate and some special treatment.

[Chainmax]

What do you mean by "feed it the right framerates"? Also, could C_Restore (http://forum.doom9.org/showthread.php?t=95924) be a better option sine this is animated content?

[Chainmax]

Since Restore24 is quite complicated, I tried Crestore_pal first with the default settings:

ord = last.getparity() ? 1 : 0
a = last.tdeint(mode=1)
b = last.leakkernelbob(order=ord, sharp=false, twoway=true, threshold=6).BilinearResize(640,352)
Crestore_pal(bob=a, bclip=b)

And here's the result:

http://img203.imageshack.us/img203/2058/crestorepalresults0am.png (http://imageshack.us)

Will Restore24 give better results? This is the only thing that I'm missing, I figured out the rest of the filtering (it needed deblocking, by the way).

[Mug Funky]

i keep forgetting the restore24 syntax (is it as simple as restore24(2400,5994) or am i missing something?)

but anyway, you're not restoring a PAL framerate because the source was (probably:)) never PAL. you'll want to restore 24 fps (exactly, no 23.976) from a 59.94 clip. i'm not sure if crestore can handle that, but i dare say it should :)

as i've said before, i prefer to be adding the blends than removing them :)

[Chainmax]

I don't think CRestore can do what you are referring to, the two usages in its readme yield either PAL or FILM framerates. I guess I'll have to try Restore24, does it still have that time limit over which it gives trouble?

[mg262]

I would try expressing the decimation fraction in the simplest terms possible, as it might give the decimatrss an easier time. 2400/5994 = 400/999, I think.

[dude051]

Maybe a test video is in order..... Here (http://www.exiled-destiny.com/Test.mkv)

[mimage]

I have mkv files on this computer and they play fine but for some reason I can't get your files to play dude051. :(

[DryFire]

It seems kind of jerky.

mimage - are all of your splitters up to date?

[mimage]

OK, I fixed it. When I installed Matroska Pack, I deselected all components except MKVMuxer and that fixed it. ^_^

Maybe a test video is in order..... Here (http://www.exiled-destiny.com/Test.mkv)
Well whatever you did, I like the outcome. :)

[dude051]

ya that seems the biggest downfall, by removing the blends the way i did, it resulted in jerky play back. I kinda did that in about half an hour, i could get it looking alot better with a bit more work... it was just a test of one way of removing blends like that :sly:

Thanx btw, all i used was TIVTC and unblend really. But i need to tweak it more.

[Chainmax]

I'll keep trying with C_Restore until the new version of R24 is released. Hopefully MOmonster can give me some tips.

[mimage]

Thanks for all the help guys. :)

Whenever this is figured out I could post the script in the "User contributed AviSynth processing suggestions" (http://www.amvwiki.org/index.php/User_contributed_AviSynth_processing_suggestions) page over at AMVWiki.org (http://www.amvwiki.org/index.php/Main_Page). Or not just an idea.

[Chainmax]

MOmonster, are you around?

[MOmonster]

@Chainmax
Sorry for my late answer. It´s a little bit complicated with internet (I don´t have internet, only at the university).
Ok, I´ll have a look at your source this evening. Crestore isn´t any longer recommed. The new restore24 version is somewhere in the forum and just this day I release the new Cdeblend version (maybe worth a try). Tomorrow I can say more. :rolleyes:

[Chainmax]

Actually Didée told me that the new R24 is still under heavy development. I'll try your new CDeBlend and report back.