anime: scene change - bad frame replacement

[grumpygamer]

Hello gurus,
I am attempting to find a solution to my strange problem.
It seems there has never been such discussion on the forum or I didn't find it although I did a looong search.

What I'm talking about is the bad montage (keyword i searched for with no result) or if you prefer bad editing or again bad frame pasting.
Actually I don't know what it's called but I find it so often that it's starting to annoy me, so I was wondering weather there is a way to eliminate this problem.

It seems that in old anime on every scene change there is a bad frame (1 or 2 at max) that results in an annoying frame jump and I was wondering if there was an automatic way to detect scene changes and replace the frame accordingly.

Example:
- detect scene change on frame 200
- replace frame at 200 with frame at 201

anyway here are a couple of screenshots so you can better understand what i mean:

http://img259.imageshack.us/img259/672/snapshot20080520114917bhk3.jpg

http://img164.imageshack.us/img164/8643/snapshot20080520114940bh5.jpg

You can see the distortion (or whatever it's called) at the top in the frame.

Thanks in advance for any help
-gg

edit:

appearently the keyword was "scene switch"
and freezeframe but the only solution is performed manually with yatta which is a bit time consuming...

edit:

still, only one topic... so in doom9 gurus I trust

[roozhou]

Try temporal denoiser

[grumpygamer]

Are you sure a temporal denoiser would solve my problem?
It seems very unlikely to me, but on the other hand...

[lamer_de]

You want something like Improvesceneswitch. It was once part of the removedirt filter:
http://forum.doom9.org/showthread.php?t=120555&highlight=improvesceneswitch
http://forum.doom9.org/showthread.php?t=37856&highlight=improvesceneswitch

Sclav by akupenguin should also help. No clue where you can find it atm, use forum search/google.

[grumpygamer]

thanks, seems nothing automated is available in 2008...

I could try and code it myself if I had the skills.
As already stated it is quite simple.

If (keyframe detected) and (scenehaschanged) then framereplace(thisFrame-1, thisFrame-2)

assuming the detection occurs on the second first frame of the scenechange and framereplace args are framereplace (thisframe, with thatframe)
meh...

kind'a hard

[thetoof]

Following = big guess because it's the first time I've ever seen this.
Maybe this it is the result of some bad ivtc... it looks like it's missing a line from one of the fields.

Could you upload a small unprocessed sample containing such a scene change?

[[P]ako]

Sclav is part of YATTA's plugin pack.

While it seems quite simple to you, an automated approach to what you want it may leave you with broken pans. I know it's time consuming, but I think it is manual freezeframing with YATTA.

I recall somebody having the same issue as you and a user sugguested him to use this: http://avisynth.org/stickboy/RemapFrames.zip

[dbzgundam]

These problems arise from older cel animation having their film literally cut together.

What you're seeing is glue and/or tape. :)

The easy fix is freezeframing, and this actually great, except it is not useful on panning shots or shots in which the very next frame is different... Also, this reduces jitter to an almost unnatural point (I know we encoders often try to reduce frame jitter, but let's face it, it's technically natural to the source), so it may look awkward to have a frame or two of absolute steadiness.

The hard solution is painstakingly photoshopping these frames as some newer anime remasters do. They digitally remove the splices of these frames, but by no means is this automated...

EDIT: Here's an example of *Terrible* case of a scene change, and the restored version on the right. (2000 R1 vs. 2007 R2)
http://img292.imageshack.us/img292/4954/gwcomparison3my2.th.png (http://img292.imageshack.us/my.php?image=gwcomparison3my2.png)

[grumpygamer]

I understand your idea of photoshopping the bad frames, but believe me at the time being I don't have THAT amount of time!
Also replacing them manually is a not feasible at the moment. So i guess I'll try Remapframes and hope for the best.

@thetoof
hi! come to help me over here as well?? (now where is that donate button) :P
you can see the same artifacts in scene changes also on the other thread's samples

http://forum.doom9.org/showthread.php?t=137893

if you already have them samples downloaded you can just check at the bottom of the frame before a scene change, they occur at the bottom there but the concept is the same: glue or cellar tape, damn them, what cheap stakes!

Anyway scene change detection is still a problem, manual freezeframe is not a viable solution for me so I guess I'll just have to wait for a remaster or some other guru that might come up with a ultra-sophisticated function!

[Comatose]

I noticed the frame movement after scene changes as well, in Flanders no Inu.
It's better there... but the colors change a bit as well.

first frame of scene | second frame of scene
sample 1 - 1 (http://xs127.xs.to/xs127/08213/blah1484.jpg) | 2 (http://xs127.xs.to/xs127/08213/blah2624.jpg)
sample 2 - 1 (http://xs127.xs.to/xs127/08213/blah3558.jpg) | 2 (http://xs127.xs.to/xs127/08213/blah4341.jpg)

sample 3 (color/brightness change is more visible here) - 1 (http://xs127.xs.to/xs127/08213/colors1448.jpg) | 2 (http://xs127.xs.to/xs127/08213/colors2972.jpg)

I remember there was one scene where the change was horrible. It was a dark scene, and it looked like somebody did Tweak(bright=10) on the first frame. Very blergh.

Is there something I can use to detect scene changes?
This happens on *every* scene change with no exception, so using that script to freezeframe the first frame of each scene would be sufficient.
Later, I could play only those frames with interleave to make sure I didn't lose any motion.

[grumpygamer]

... so using that script to freezeframe the first frame of each scene would be sufficient.
Later, I could play only those frames with interleave to make sure I didn't lose any motion.

Which script do you mean?

[TheRyuu]

ako;1140038']Sclav is part of YATTA's plugin.

While it seems quite simple to you, an automated approach to what you want it may leave you with broken pans. I know it's time consuming, but I think it is manual freezeframing with YATTA.

I recall somebody having the same issue as you and a user sugguested him to use this: http://avisynth.org/stickboy/RemapFrames.zip

Scxvid is the newer one, sclavc is old and screws up too much. If you're going to use it please use scxvid. you can find them all at ivtc.org.

[LaTo]

Try this:

function BadSCReplace(clip i, int "thr") {
thr = default(thr, 50)
test = i.YDifferenceFromPrevious()
change = ( test > thr ) ? i.trim(0,1) : i
return change }

BadSCReplace(last,50)
--> ajust the "thr" as you like the result (low = accurate, high = useless)

[Didée]

LaTo, that ain't work at all. Functions of Avisynth's conditional environment can only be used - you guess it - in Avisynth's conditional environment. ;)

Besides, a simple thresholded difference to the previous frame isn't smart enough. That diff might be rather low e.g. in dark scenes, and might be rather high e.g. in pannings, explosion scenes and such, yielding to frame freezes in places where you definetly don't want them.
This method has been used already yeeaars ago. The result is it works in some places and in some not. High thresholds will miss many scenechanges, low thresholds give many false detections. In between, you get a mix of both.

I'd rather try try examining the 2nd derivative ... example: look at the diff-to-previous over three frames. If the middle one is reasonably bigger then the surrounding two ones, then a scenechange is likely. To be sure, take a 5-frame window, and evaluate the proportion of the diff-of-differences at the start & the end of that window to the one in the middle.

[LaTo]

LaTo, that ain't work at all. Functions of Avisynth's conditional environment can only be used - you guess it - in Avisynth's conditional environment. ;)
Oups... :rolleyes:
Any way to do this outside of Avisynth's conditional environment ???

[Didée]

Why outside? It's not that difficult. In the Avisynth doc's, we read:


# This will replace the last frame before a scenechange
# with the first frame after the scenechange:
ConditionalFilter(last, last, last.trim(1,0), "YDifferenceToNext()", ">", "10", true)

[LaTo]

Why outside?
Outside because I would use "the value returned by YPlaneMax()" as a parameter for an another function... No way?

[Didée]

No, no way. The Conditional Filters give runtime results on a frame-per-frame basis. Function parameters are static, their input parameters are determined when the script gets initialized.

Depending on what exactly you want to to, you probably can include the content of your "other function" into the Conditional Environment, so that it can evaluate the results of those runtime functions.

(Not that it were a shiny example of clean coding, but - ) you might have a look into the script "Restore24". There, there is lots of stuff going on in the conditional environment, a good bunch of userdefined functions are running completely within the Cond.Env., feeding each other with their results etc.

[grumpygamer]

erm...
love your debate but what is your suggestion?
-gg

[LaTo]

(Not that it were a shiny example of clean coding, but - ) you might have a look into the script "Restore24". There, there is lots of stuff going on in the conditional environment, a good bunch of userdefined functions are running completely within the Cond.Env., feeding each other with their results etc.

Thanks... I will have a look :thanks:

[Didée]

erm...
love your debate but what is your suggestion?
-gg

Get RemoveDirt (http://www.removedirt.de.tf/), use
prev = source.selectevery(1,-1)
next = source.selectevery(1,1)
source.SCSelect(next,prev,source,dfactor=2.0)
, try dfactor = 1.0 ~ 5.0

[Comatose]

Awesome :3

The only problem is I'm not sure how to use SCSelect to actually do something with this =/ how can I tell when a scene begin/end was detected?

Thanks

[grumpygamer]

thanks i'll try and see
-gg

[grumpygamer]

Incredible! IT WORKS! YOU ARE A LIFESAVER!
Sorry for Capsing but I really wanted to say that out loud!
Thanks, thanks thanks!!!!!!!!!!!!
This solution should be more visible for other people too: couldn't you copy paste the code somewhere else too??

Anyway thanks again!! Awesome!!
-gg

[GMJCZP]

I have the same problem in the bottom of the image (sometimes in the top). As already mentioned, this is due to old video editing. It comes in makeovers before and after.

I tried the suggestion Didée and does not work. Any ideas?

[Didée]

Short of seeing a sample to examine why it does not work: you can of course edit all scenechanges manually. :)

[GMJCZP]

Hi Didée:

Unfortunately it is difficult for me to upload any video now.
It's like the pictures of the first post, but mostly the defect appears on the bottom of the image in the scene change.
What I can say is that I used freezeframe as you have no idea, lol.
I could upload a photo, I do not know if that will do.

[Emulgator]

What you see for instance in the first post is just the film edges of a negative cut.
I suggest manual work in Photoshop (and if jumps occur: manual frame alignment of the cut too), this is what I did on most of my restoration work in such case...

[Didée]

Perhaps you need the equivalent of an "exclusion band", or something.

prev = source.selectevery(1,-1)
next = source.selectevery(1,1)
source.LetterBox(32,64).SCSelect(next,prev,source,dfactor=2.0) # not using top-32/bottom-64 pixels for the decisions

... or maybe it's something entirely else. The motor mechanic can't tell you what's broken when he can't look at the engine.

[GMJCZP]

What you see for instance in the first post is just the film edges of a negative cut.
I suggest manual work in Photoshop (and if jumps occur: manual frame alignment of the cut too), this is what I did on most of my restoration work in such case...

Sip, So I use freezeframe and when not resolved in this way turn to Photo-Paint and ImageSplice. But out of curiosity wanted to know if there was a "magic formula".

Perhaps you need the equivalent of an "exclusion band", or something.

prev = source.selectevery(1,-1)
next = source.selectevery(1,1)
source.LetterBox(32,64).SCSelect(next,prev,source,dfactor=2.0) # not using top-32/bottom-64 pixels for the decisions

... or maybe it's something entirely else. The motor mechanic can't tell you what's broken when he can't look at the engine.

The key would be to concentrate the attention of a script or plugin in a given area of the screen and detect a drastic change in its structure, but only in that portion in comparison with the rest of the image.
Maybe I'm asking too much of Avisynth and you, Didée. I will try your home remedy.

[GMJCZP]

Didée, the improvement is incredible!

With dfractor = 4.0, in the sample I used, eliminates most of the defects. If is increased to 5.0 removes more but slightly destabilizes the image, in fact, is increased slightly video size with respect to dfractor = 4.0. Interestingly in one case, the defect rather than being eliminated, for example, the frame 770 became 769, the new frame 770 is corrected, apparently both is swaped.
Also noted of 1064 frames that the sample had happened to be 1063, a frame was lost.
Adding your remedy (and there are other arguments originally, of course) the time processor for the sample of 1063 frames (really was 1064) became 2:55 seconds to 4:55 seconds (more or less).

What about results?

[GMJCZP]

Any comments or suggestions?

[lansing]

i would say don't put too much hope in it. All that script does is to first find the scene change, and then replace the scene change frame with its previous or next frame, depending on the calculated difference, so there will be chances of failure where the calculated result doesn't go along with the actual source, and you'll have to live with that.

[Chyrka]

I think, using of FreezeFrame is very reliable decision. Of course, manual defining of frame numbers for each scenechange is too slow, but in simple cases with two bad frames directly on the scenechange can be automatized with using any script for scenechange detection.

[mandarinka]

It's not so bad if you use yatta. You need xvid 1st pass log (load it in tools, "s from file") or a run of ymc with scxvid. Then you can just jump to next section, freeze using hotkeys and repeat.

Doing it automatically can lead to errors - xvid's scene detection misdetects a lot (object passing through in front of camera, flashes). Moreover, it might be better to not freeze scenechanges if there is motion/panning/zooming...

[GMJCZP]

Sip, freezeframe + image editor is the most solid.

Hopefully Didée or another guru to give us some light in this regard. :helpful:

[lansing]

Sip, freezeframe + image editor is the most solid.

Hopefully Didée or another guru to give us some light in this regard. :helpful:

don't even bother with the freezeframe manual fix by yourself alone. I tried once, took me 2 hours to gone through 10 minutes of video time, and then i just give up.

[GMJCZP]

Lol.

I was reading the manual version of MVTools2 of Firesledge, and noticed that there is a function with a name like SCSelect of RemoveGrain, do not know if there is to do something, I think his name MSCSelect or something.

[GMJCZP]

Well, now I can now say that there is something:

source = last
prev = BackwardClense (source, cache = -1)
next = ForwardClense (source, cache = -1)
source.SCSelect (next, prev, source, dfactor = 6.0) # dfactor> = 6.0, to avoid artifacts

Using now ForwardClense and BackwardClense resolve the problem of lost frame, moreover, it is faster. Should be used before any Denoiser, especially if Mo-Comp.
This method is taken of RemoveDirt (see the manual RemoveDirt 0.9).

However, the detection rate of known defect in the bottom of the screen is good but does not detect all cases, so I present for consideration an alternative method or beta:

source=last
super = MSuper(source)
vectors = MAnalyse(super, isb = false)
compensation = source.MCompensate(super,vectors)
# prepare blurred frame with some strong blur or deblock function:
blurred = compensation.Deblock(quant=51) # Use DeBlock function here
badmask = MSCDetection(source,vectors)
mt_merge(compensation,blurred,badmask)

Again, using a method based on a manual, in this case taken by Firesledge MVTools 2.6.0.

The downside of this method is that it produces serious artifacts (how about it?), but in return, if you play with the values ​​of thSCD1, thSCD2, thSAD and thSAD2 for MCompensate (I'm a newbie), the detection rate is higher.

I think the key is to provide a better motion estimate toSCSelect , or combine both ideas into one, or create a mask to focus attention on the top and bottom of the screen, like the idea of Didée, LetterBox.

[GMJCZP]

Formalizing the idea:

###
# Function DeCelluloid (adapted by GMJCZP).
###

# Eliminate garbage of top and bottom image of old videos.
# Low Gfactor values => larger detection, more probability appear artifacts
# High Gfactor values => minor detection, more safe.
# Thanks a Didée for the idea.
# https://forum.doom9.org/showthread.php?p=1567932#post1567932
# Requirements: RGTools, RemoveDirt.
# Version 1.0, first attempt.

function DeCelluloid(clip source, float "Gfactor", bool "_grey" )
{
Gfactor=default(Gfactor, 5.0) # Garbage factor
_grey = default(_grey, false) # Source in Black and White
prev = BackwardClense(source, grey=_grey)
next = ForwardClense(source, grey=_grey)
h = height(source)
h_top = int(h*0.20)
h_top = (h_top % 2) == 0 ? h_top : h_top + 1
h_bottom = int(h*0.375)
h_bottom = (h_bottom % 2) == 0 ? h_bottom : h_bottom + 1
source.LetterBox(h_top,h_bottom).SCSelect(next,prev,source,dfactor=Gfactor)
}

[henryperu77]

Is this possible to use this on interlaced material? or i need to deinterlace and IVTC it first?

[GMJCZP]

Hi henryperu77:

For my tests (for now) I have found it best to use DeCelluloid after IVTC (I have only tested for that case, anime content).
Otherwise one should use, I suppose, something like this:

AssumeTFF ()
SeparateFields ()
DeCelluloid ()
Weave ()
Somebody correct me if not for interlaced content.

In any case you should preview in VirtualDub how you remaining in both cases and make processing test with small pieces of video.

[Space Hopper]

[EDIT FIXED]
Odd, I CANNOT get RemoveGrain to work anymore.
It used to work, and I can load the plugin with either LoadPlugin or just autoloading from the plugin folder - no apparent problems there.
However, a call to DeCelluloid complains about missing functions - like BackwardClense. It like RemoveGrainS.dll doesn't get loaded at all.

I need to fix some glue (or celluloid) in an old Anime.

This is on Windows 7 x64.
Anyone know what the problem could be?

Ps. Tried reinstalling Avisynth obviously. No luck.

[WHAT WAS WRONG]
I'm missing some DLLs, and I can't figure out which.
The wiki page (http://avisynth.org/mediawiki/Removegrain) contains broken links, and the info is pretty sparse (the RemoveGrain docs are no better).
I fixed it by getting the entire removegrain package, and only using the statically linked files (those ending on "S").

Note: I miss out on all the SSE3 optimizations though, which is a shame.

[EDITx]
Finally figured it out... was missing AvsRecursion.dll
I really really wish this was mentioned somewhere. Spent the better half of 12 hours looking trying to get it to work.
Oh well, all is good now.

[Space Hopper]

Formalizing the idea:

###
# Function DeCelluloid
###

# Eliminate garbage of top and bottom image of old videos.
# Low Gfactor values => larger detection, more probability appear artifacts
# High Gfactor values => minor detection, more safe.
# Thanks a Didée for the idea.
# Requirements: RemoveDirt 0.9
# Version 1.0, first attempt.

function DeCelluloid(clip source, float "Gfactor", bool "_grey" )
{
Gfactor=default(Gfactor, 5.0) # Garbage factor
_grey = default(_grey, false) # Source in Black and White
prev = BackwardClense(source, grey=_grey, cache=1)
next = ForwardClense(source, grey=_grey, cache=1)
h = height(source)
h_top = int(h*0.20)
h_top = (h_top % 2) == 0 ? h_top : h_top + 1
h_bottom = int(h*0.375)
h_bottom = (h_bottom % 2) == 0 ? h_bottom : h_bottom + 1
source.LetterBox(h_top,h_bottom).SCSelect(next,prev,source,dfactor=Gfactor)
}

Gave this a test on some old Anime, and it kinda introduces more artifacts that it fixes. This is on the default setting of 5. Also it misses a lot of scene changes.

I like the idea though. But I guess one has to face the fact that this is hard to fix automatically.

[GMJCZP]

Gave this a test on some old Anime, and it kinda introduces more artifacts that it fixes. This is on the default setting of 5. Also it misses a lot of scene changes.

Each video source has its peculiarities, so that the appropriate value for GFactor may vary for each case.

I like the idea though. But I guess one has to face the fact that this is hard to fix automatically.

This script is a start, there is still no magic formula for dealing with these cases. :(

[bryrabknowles]

Just thought I'd throw out my concept for how to fix this. Needs someone more experienced than me to implement.

First, use scene change detection to define scenes - focus on the whole rather than just the separation point.

Ignoring the first 4 and last 4 frames of the shot, Average the remaining frames.

Compare the difference between the resulting "Average" and the Middle frame of the shot.

Set a maximum difference value (even better if you can find a way to only average static portions).

If the difference value is low enough, the shot is static. Otherwise, assume a pan.

On static shots, use the average found earlier. Working backwards, check the difference of frame 4 and 5. If 4 is a bad frame, use image registration to align. Otherwise skip to 3, etc.

When all 4 frames are checked and aligned, a small clip is created, essentially In frame order for that shot, 8,7,6,5,4,3,2,1,2,3,4,5,6,7,8 (provides continual motion to better calculate vectors.

Use this clip to generate the mask to be applied to the bad frames.

For Pans, the previous clip will be examined to determine whether it is static. If so, the final two frames of the previous shot will be run through QTGMC to generate 2 motion compensated extra frames.

If both shots are pans, the previous longer step will be applied to both.

[GMJCZP]

Here's the detail: turn the theory into practice. :(

[GMJCZP]

Hi, I am having trouble with my DeCelluloid (https://forum.doom9.org/showthread.php?p=1567932#post1567932) script (see post #40), some greenish frames appear, here is a sample:

Sample2 (http://www.filedropper.com/sample2)

The culprit is RemoveDirt 0.9.3 and SCSelect, I had to reuse the old RemoveDirt SSE2 1.0