Advice Needed on Image Cleanup (edge corruption, dot crawl)

[osgZach]

Quick Sample (http://www.oldskoolgames.com/videos/dpf/ep2sample.avi): Taken straight from Src d2v+vob
HuffYV12, 17MB
720x480 / AR:1.5 / 29.97Fps
Pre-filtered in TMPGenc Xpress, IVTC + reinforced anti-stripes, Noise Reduction 2d/3d, Contour, Smart Sharpness.

I was getting ready to do some post through Avisynth, but I'm not sure what filters I should try for the particular noise I am seeing at the edge of the borders. It seems to me that this noise was caused by some sort of physical damage or other physical manipulation of the source?

There should be a segment in the file where you see the picture lose focus and a few lines of data (like that particular frame was moved out of alignment?) and generally I would just crop some of this stuff off, but my concern is that when I start frame stepping I occasionally come across spots (not sure if they are seen in this sample) where the damage to the image extends a good 20 - 30 pixels into the image. (guestimating, but it would be a nasty crop to get rid of it). It's not as serious as the edge corruption, so I am hoping it could be cleaned up.

Ultimately I'd leave that kind of damage in versus cutting more than 10 - 16 pixels, but I'm not sure what type of filter is best for this kind of stuff.

Other than that, I know I know to Degrain / Remove Dirt, and there is also some pretty evident dotcrawl (I didn't know what it was until I saw it and then it dawned on me so I am guessing most people will notice it). In particular I have played with Decrawl, and I can't seem to find settings that work well enough, especially on motion/high-motion scenes that won't blur the image to hell so much that even after I gently re-sharpen it, it still looks like crap.

I have to go to work, but I'll be back in about 5 hours or so and I will post a longer sample if its really needed. I just thought I would get a quick sample up to show my major concern with the edge noise.. It seems to ONLY happen upon scene changes 99.5% of the time as well. Any ideas why? Again, gonna crop off what I can within reason, but take frame 40 there is clearly junk extended up from the main corruption line, kind of putting a stain on the pixels above it. That's the kind of stuff I want to clean.

[osgZach]

Ok so I've been playing around in Vdub, and while I was able to test out what I wanted to do, I still think I probably need a different way of doing it.

On some of the scenes, where the corruption/1 frame squish/blur thing is happening, there also exist clean copies of the damaged frame.
I need a simple way to delete that frame, and replace it with a copy of the previous frame, as obviously I don't want to just delete the frame and screw up my framerate.

However I either am not using Virtualdub properly, or its just too many steps to repeatedly do it.. Should I be looking at a Non Linear Editor instead? Or is there a really simply way I can do this in Avisynth? I was thinking of splitting the file, leaving the bad frame out.. But then how would I make a copy of the very last frame from the first half of the split, and rejoin the clip?

For examples sake.

Total Frames 100
Frame 37 is bad, so split the file in half 0-36 and 38-100. but how would I dupe frame 36 to be the new frame 37 before rejoining the split ? I'm aware I can load different frame ranges into named variables, but I am clueless what to look up in regards to copying a frame and adding it to the frame count of a clip

I'm thinking there is a less complicated way someone might know of.

[Gavino]

Frame 37 is bad, so split the file in half 0-36 and 38-100. but how would I dupe frame 36 to be the new frame 37 before rejoining the split ?
No need even to split - just use
FreezeFrame (http://avisynth.org/mediawiki/FreezeFrame)(37, 37, 36)

[osgZach]

Thanks a lot, that really made the job easier!

Now we'll see how many bad frames I can nix and whats left that may need filtering. Excessive crop is a pet-peeve of mine..

One more question.. For frames where they are almost identical, but not quite, and I just want to replace a defined area of the picture, is that possible too?

For example.

I have a relatively static scene of conversation between a man and his computer (the computer is shown as a terminal monitor with animation on the monitor). I have no dupes of the bad frame, but aside from the image on the monitor, nothing is really changing. The frame proceeding the bad frame has the monitor as blank, then there is an updated animation on the monitor in the bad frame.

Can I copy the bottom half of the good frame, to replace the corrupted bottom segment in the next frame??

edit: Jesus christ, I'm 3 minutes into this 26 minute video and I've already done 33 swaps.. This thing is gonna read like a timecodes file by the time I'm done with it! and there are 21 episodes across 3 OVAs... All this just to preserve my DVD collection and have everything at my fingertips on my WDTV Live? I'm afraid to even look at my Loddoss War DVD's, this is a flat out appalling level of production for a DVD release. No wonder a lot of Anime publishers have folded in the US, when you put out crap products.... And I thought my Macross II DVD was bad the last half of the feature sporting a lovely blue tint -_-

[osgZach]

Hi,

I was largely left up to my own devices yesterday to solve some stuff (which is good) I was dealing with on account of the forum IP change, but I noticed something I don't think I've noticed before.

Encoding the same file with the exact same script, has lead to two different file sizes and frame counts, although the FPS for both does come out the same.. So I was wondering if that is unusual or not. I am using TFM/TDecimate for this project.

Basically I got output that confused me and I thought it was combed so I modified my script and neglected to back up the stats from my encode, so I had to do it all over again with the same settings to get my timecodes back. However later I realized it was probably alising and confirmed it with a minor AA option, so now that I've reverted to the original scripts I used, I was just curious because I had gotten two different outputs from the same script.

It's not really an issue, more of me being curious if that sometimes happens because the filters decide to change a decision they had made the previous time (I am guessing it was one of those things where it could flop either way?)

But to the topic on hand.
I have a 2-pass script setup I really like that produces great output. However I still cannot figure out how to get rid of that specific random noise from dirt spots/scratches. I briefly tried to hunt down some filters but things I wanted to try I couldn't find proper links for, or were linked to the forum (which I couldn't get to).

Here are my current 1st and 2nd pass scripts. I may or may not get chided for whats before TFM in the 1st pass (unneeded for an analysis), but its the only way I could get TFM to shutup and stop halting on the 2nd pass.

Analyze Pass

mpeg2source(d2vpath)
DeGrainMedian(limitY=4, limitUV=6, mode=1, interlaced=true)
DeGrainMedian(limitY=4, limitUV=6, mode=1, interlaced=true)
tfm(d2v=d2vpath,output="ep1-tfm.txt")
tdecimate(mode=4,output="ep1-tdec.txt")


Encode Pass

mpeg2source(d2vpath)
DeGrainMedian(limitY=4, limitUV=6, mode=1, interlaced=true)
DeGrainMedian(limitY=4, limitUV=6, mode=1, interlaced=true)
tfm(d2v=d2vpath,input="ep1-tfm.txt")

tdecimate(batch=true,mode=5,hybrid=2,input="ep1-tdec.txt",tfmIn="ep1-tfm.txt",mkvOut="ep1-timecodes.txt")
Stab()
FluxSmoothST(temporal_threshold=7, spatial_threshold=7)
FFT3DGPU(sigma=2.3, plane=4, bw=48, bh=48, bt=0, ow=16, oh=16, sharpen=0.5, interlaced=false)


DeGrainMedian has not completely eliminated the dot crawl, however when everything comes together it is very passable and I am satisfied.. But regarding the random spot/dirt/scratch damage.. Is there a filter that will handle all 3 or will I need a combination? I was going to look into remove dirt filter, for example.. I think that's what its called

A lot of the spots seem to be either white or black, and then scratches or some other kind of weird stuff that isn't really opaque or transparent, but more solid, are there at random as well. I've actually considered leaving them in, but I'd like to see what I can do first.
I am thinking the problem is more dirt type anomalies than scratches though.

[osgZach]

Ok now I have a really weird problem...

It's some kind of residual image / ghosting I think??

I don't know if this is considered a chroma artifact or not?
Basically remnants of some text from the previous frame are showing up on the next frame, and it looks like a weird transparent outline of the letters, and possibly some rainbowing on that transparent stuff..

I'll try and get a screenshot. But I'm confused about how to deal with it. It only ever shows up after I start denoising. The filter that affects it the most is DeGrainMedian, as it starts clearing the noise, the artifact becomes more and more clear within the picture..

I tried disabling various combinations of my filters while examing it back and forth between the same frames in Vdub.

I would stop on the frame its most present in, then turn off degrain and the noise would of course come back, but the artifact would fade significantly - then I could start stepping back a few frames, then go forward to that frame again and it was a tiny bit more identifiable.

I tried changing the mode, calling it only once, changing the luma and chroma values, etc. I also tried disabling TFM and TDecimate and those ALSO seemed to affect how present it was, until I did the backstep/forward step to the frame again and it was slightly more visible once again.

I've looked at the source, and it does not appear to be present? Or at least not detectable.. I have noticed some frames (which I would be replacing anyway) had problems with color phase shifts etc.. Could this be related? Overall the video looks normal and it would be a pain in the ass to study every frame for color phase problems.

I am kind of unsure what to do.. I do not believe when I use TMPGenc and use its noise filter and a few others, that this problem is present at all..

Originally I wanted to use TMPGenc for my initial output anyway, and then just run TDecimate to get rid of some of the dupes, but then I started using DegrainMedian/TFM to deal with the dot crawl problem I had noticed..

Perhaps I'll have to go back and run another test with TMPGenc and its filters :(

Then I'll see what happens when I apply DeGrainMedian again. It's the only filter I've tried that really managed to tackle the dot crawl at all..

I am very sure its dot crawl that I am filtering, as people mention checkerboarding, and weird moving patterns around text labels, etc.. When zoomed in to 200% I can clearly see a checkerboard pattern that looks EXACTLY like the transparent checkerboard pattern in Image editing programs, when you ser the background as transparent.

This is dot-crawl... right?

Original Source no filters copied from Vdub (http://www.oldskoolgames.com/videos/dpf/dpflash1.png)
Same image resized to 300% BiCubic (http://www.oldskoolgames.com/videos/dpf/dpflash2.png)

[Guest]

Post an unprocessed sample cut directly from the VOB.

[osgZach]

Sorry, I thought a screen would be sufficient. Here is the sample.


11 seconds, demuxed straight from DGIndex (http://www.oldskoolgames.com/videos/dpf/dpf_sample_1.demuxed.m2v)

1 or 2 calls to DeGrainMedian as in my script above, even without the other stuff, should be sufficient to expose it.
The frame of interest comes directly after the starting screen of the video with the characters profile, goes combed between the last screen of that scene and the first screen of the new scene with the man, and then the next frame is the man, and where I am concentrating on to highlight the problem. I do believe the pattern will move during playback (not the whole thing, but appearing and dissapearing like its morphing, if you watch the guys arm in the bottom right corner)

Could it be because of the comb (blend?) on the scene change?

Here is an additional sample from about 7 minutes in. It is a full 30 seconds, in particular note the pattern showing up in the headlights and other areas. At default size it almost looks like the kind of thing you see from a magazine page that has been scanned at high resolution without being resized. But zooming in there is clearly a checkerboard pattern as well. I don't believe there is any kind of ghosting in this scene though, but I haven't studied it very much, except to tweak filter settings.

Sample 2 - 30s demuxed from DGindex (http://www.oldskoolgames.com/videos/dpf/dpf_sample_2.demuxed.m2v)

Also.. This is what I get out of TMPGenc (Deinterlace Always / IVTC + reinforce antistripes - 29.97p)

Original Size - Filtered (http://www.oldskoolgames.com/videos/dpf/dpfTmpgnc1.png)
Same as above but 300% (http://www.oldskoolgames.com/videos/dpf/dpfTmpgnc2.png)

[Guest]

What is the title and region of the DVD that you took that from?

[osgZach]

Dirty Pair Flash Perfect Collection, Region 1 as far as I know. Got it at best buy a number of years ago.
I'm assuming you want to verify its not downloading or some junk, so I'm trying to find some batteries for my camera

Set 1 (http://www.oldskoolgames.com/videos/dpf/dpdvd1.jpg)
Set 2 + Date/Time (Atomic Clock) (http://www.oldskoolgames.com/videos/dpf/dpdvd2.jpg)

[Guest]

I'm trying to find some batteries for my camera No need, your reply is satisfactory. Thank you.

[osgZach]

Too late? :p

It's clearly a [poor] source.. I have written some not nice things about Anime distributers on other forums recently as a result. This is unacceptable quality for a retail product, even if most of the problems are in the overscan area. I'm still stuck having to clean them up since I want to digitize my collection and have it all available on my WDTV Live... I really seem to be able to pick them.. Gall Force, DP:Flash, Macross II with its second half of the feature having a mysterious blue tint.. I'm afraid to even take my Loddoss War DVD's out of the case and look at the VOBs :(

[Guest]

Too late? The messages one receives on clicking your links could get you a rule 6 strike. I'd change that if I were you.

[osgZach]

Are you referring to my actual web site?

I don't provide access to anything. I run a Magazine that reviews games. That's it.

I added the forum IP to my htaccess though so unless something is fudged somwhere you shouldn't get redirected..

Although I will go examine my error pages, I know my site got defaced a while ago, but I didn't think those pages were compromised.

I will PM you to try to sort out what the problem specifically is.

[Guest]

I meant rule 4, sorry.

I'm talking about the links in post #10 (not your sig) which have now apparently been changed from their earlier insulting nature.

[osgZach]

Yes, that should not happen, but I didn't change the links. The issues with the forum calling from the IP and not the domain caused the issue but it shouldn't happen in the future, and if it does I'll take the appropriate action.

[osgZach]

I am currently still trying to get suitable output without augmenting these artifacts I've been having a problem with and I had an idea but I don't know if it will work?

What I'm interested in doing is doing an analyze pass with both TFM and TDecimate, Then running another pass with TFM only and taking the 29.97p output and filtering it through TMPGenc Xpress instead which as far as I know shouldn't modify the number of frames in the file and will only be cleaning them up.

Then I want to run that through Tdecimate again to get a VFR file, by using the TFM/TDecimate stats files from the analysis pass.

I've already tried simply running TDecimate on a file I've filtered and IVTC'd with TMPGenc, but wasn't able to get proper results (my knowledge of Tdecimate is a limited understanding even after reviewing the documentation).

I was hoping this would work in a similar way to the Merge Hints method, in that the file isn't actually changing except to clean up the image, and there won't be any internal hints placed, but from the 1st pass stats instead. Unless I misunderstand what those stats are for?

[osgZach]

edit: I'm an idiot. F-lock got turned off... nvm