Infra Red film scan as mask for spotless

[VideoMilk78]

Hello, this is just an idea and probably won't amount to much but I just wanted to see if this was possible and if it is could someone try and make it a thing?
I was reading an article about SilverFast software using an IR channel to pic up physical defects with the film and use that as a mask of sorts...

I was wondering if anyone could try and make a script with spotless that could do something similar?

thanks in advance...

[johnmeyer]

Originally Posted by VideoMilk78

Hello, this is just an idea and probably won't amount to much but I just wanted to see if this was possible and if it is could someone try and make it a thing?
I was reading an article about SilverFast software using an IR channel to pic up physical defects with the film and use that as a mask of sorts...

I was wondering if anyone could try and make a script with spotless that could do something similar?

thanks in advance...

That technology was created by Applied Science Fiction 20+ years ago and licensed to film scanner companies like Nikon and Minolta. It works extremely well on color negative film; somewhat on Ektachrome and other E-6 color positive film; and almost not at all on Kodachrome or B&W film.

You have to be able to get a scan which includes the infrared channel. Such a thing doesn't just grow on trees, and it obviously will not be available unless you do the film scanning yourself.

[VideoMilk78]

Originally Posted by johnmeyer

That technology was created by Applied Science Fiction 20+ years ago and licensed to film scanner companies like Nikon and Minolta. It works extremely well on color negative film; somewhat on Ektachrome and other E-6 color positive film; and almost not at all on Kodachrome or B&W film.

You have to be able to get a scan which includes the infrared channel. Such a thing doesn't just grow on trees, and it obviously will not be available unless you do the film scanning yourself.

Naturally, what I mean is a function that can utilize an IR scan as a mask for spotless that might have a syntax like this:

IRspotless(clip RGB, clip IR, int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "OLap",bool "tm",Bool "glob", Float "bBlur")

Or maybe use an IR scan as the dc clip...

[poisondeathray]

You would need to provide examples of both scans , and information on the variability of the IR scan on different types - negatives and positives - otherwise you'd need to customize the mask each time - it would mean a lot of manual tweaking for every case

eg. how well are defects covered, what degree of opacity , etc... Or is it good enough to use a binarized mask ? What degree of mask expansion ?

[VideoMilk78]

Originally Posted by poisondeathray

You would need to provide examples of both scans , and information on the variability of the IR scan on different types - negatives and positives - otherwise you'd need to customize the mask each time - it would mean a lot of manual tweaking for every case

eg. how well are defects covered, what degree of opacity , etc... Or is it good enough to use a binarized mask ? What degree of mask expansion ?

I won't have examples for quite some time unfortunately. I wouldn't have any problem doing manual tweaking though. I just don't know how to go about making the script to tweak.

[johnmeyer]

If you are able to get a digitized version of a still or movie, you could use something like Vuescan to remove the dirt.

[VideoMilk78]

Originally Posted by johnmeyer

If you are able to get a digitized version of a still or movie, you could use something like Vuescan to remove the dirt.

Is it just not possible in Avisynth?

[poisondeathray]

Originally Posted by VideoMilk78

Is it just not possible in Avisynth?

Is what not possible?

Dirt removal is possible in avisynth

Using masks is possible in avisynth to improve script results.

Masks can limit the side effects or "collateral damage" of some filters . e.g strong cleaning settings might remove wanted objects like eyeballs, specular highlights etc.. . If you had a perfect mask that indicated only the areas to filter, then you wouldn't cause unwanted damage to those other areas . You can combine multiple filtered layers with masks You can get as simple or complex as you want


It might be as simple as

Overlay(baselayer, filteredlayer, mask=masklayer)

But there are probably dozens of lines you have to adjust, depending on the relationship of the mask and filters used and the scans

[VideoMilk78]

Originally Posted by poisondeathray

Is what not possible?

Dirt removal is possible in avisynth

Using masks is possible in avisynth to improve script results.

Masks can limit the side effects or "collateral damage" of some filters . e.g strong cleaning settings might remove wanted objects like eyeballs, specular highlights etc.. . If you had a perfect mask that indicated only the areas to filter, then you wouldn't cause unwanted damage to those other areas . You can combine multiple filtered layers with masks You can get as simple or complex as you want


It might be as simple as

Overlay(baselayer, filteredlayer, mask=masklayer)

But there are probably dozens of lines you have to adjust, depending on the relationship of the mask and filters used and the scans

I'm well aware dirt removal and masks are possible in avisynth in fact they are better in Avisynth than expensive software in my experience.. However I'm just now sure how I take a mask and remove dirt based on the physical imperfections on the mask...

[joka]

The masks are never perfect. I use RestoreMotionBlocks (from RemoveDirt script) but skip the internal motion detection. The restoring is triggered by a white area on a black ground.

If rd is the result of spottless and rg is the Result of e.g RemoveGrain(20) than a white mask (mostly representing "motion" areas, the white area can be much smaller than the area to be restored) is required on the position where things unwanted removed by Spottless.

Function mRD_Restore (clip rd, clip rg, clip rm, int "limit") {

# rd temporal filtered (e.g. median)
# rg replacement clip (e.g RemoveGrain)
# rm restore mask

limit = default(limit, 6)

r1 = Interleave(rd, rd, rd)
r2 = Interleave(rm.mt_lut(y=0, chroma="copy"), rg, rm)

Return RestoreMotionBlocks(r1, r2, noise=limit).SelectEvery(3, 1)
}


It is also possible to do the opposit. rd .. RemoveGrain, rg ..Spottless. A white mask (in this case representing "dirt") will remove the dirt.

[poisondeathray]

Originally Posted by VideoMilk78

I'm well aware dirt removal and masks are possible in avisynth in fact they are better in Avisynth than expensive software in my experience.. However I'm just now sure how I take a mask and remove dirt based on the physical imperfections on the mask...

The mask is independent of the dirt removal filter(s). The mask(s) do not affect avisynth filters directly - ie. it does not alter things such as motion vectors or filter calculations directly. It's essentially an alpha channel compositing mask, similar to in photoshop/gimp etc... Inclusion and exclusion areas based on "brightness", or Y , greyscale levels. 100% white = 100% affected. 50% = 50% affected. 0% = 0% affected , etc... Areas passed through would be 0% white or complete black. The imperfections would be 100% white if you want 100% application of the filter(s).

But IR scans NEVER have 100% white defects, and 0% white non-defects. They have variable shades of grey, and are usually inverted - the defects are usually darker. So if you apply a filtered avs layer through a typical inverted IR scan mask as-is, the defects would partially remain AND the side effects would partially remain - ie. it would be WORSE than applying the filters not using the mask. This means you have to modify the mask to work properly for this method. Also - beware there is lots of variability among IR scan type characteristics - some would need one type of manipulation, but others might require dozens to be useful as an avisynth filter mask

A basic generic template might look like

#Load your original sequence
original = ImageSource(...) 

#Fully filtered sequence, without mask
original
Spotless(some settings)
#Some other dirt filter, etc...
filtered=last

#Load your mask sequence and apply appropriate filters to make the mask more ideal
ImageSource(...)
#Apply some filters to make the mask ideal
IRScan_mask = last

Overlay(original, filtered, mask=IRscan_mask)

When you get the IR scans, try the bundled software first, because they usually have dedicated tools for IR dirt/dust removal . If you don't have the software - there are ways to use some types of IR scans directly - using blend modes such as "divide" in photoshop/gimp might get you 70-90% of the way there automatically without the side effects of avisynth temporal filters (and avisynth has extended layer blend operations with 3rd party scripts/plugins such as overlayplus)

[VideoMilk78]

Originally Posted by poisondeathray

The mask is independent of the dirt removal filter(s). The mask(s) do not affect avisynth filters directly - ie. it does not alter things such as motion vectors or filter calculations directly. It's essentially an alpha channel compositing mask, similar to in photoshop/gimp etc... Inclusion and exclusion areas based on "brightness", or Y , greyscale levels. 100% white = 100% affected. 50% = 50% affected. 0% = 0% affected , etc... Areas passed through would be 0% white or complete black. The imperfections would be 100% white if you want 100% application of the filter(s).

But IR scans NEVER have 100% white defects, and 0% white non-defects. They have variable shades of grey, and are usually inverted - the defects are usually darker. So if you apply a filtered avs layer through a typical inverted IR scan mask as-is, the defects would partially remain AND the side effects would partially remain - ie. it would be WORSE than applying the filters not using the mask. This means you have to modify the mask to work properly for this method. Also - beware there is lots of variability among IR scan type characteristics - some would need one type of manipulation, but others might require dozens to be useful as an avisynth filter mask

A basic generic template might look like

#Load your original sequence
original = ImageSource(...) 

#Fully filtered sequence, without mask
original
Spotless(some settings)
#Some other dirt filter, etc...
filtered=last

#Load your mask sequence and apply appropriate filters to make the mask more ideal
ImageSource(...)
#Apply some filters to make the mask ideal
IRScan_mask = last

Overlay(original, filtered, mask=IRscan_mask)

When you get the IR scans, try the bundled software first, because they usually have dedicated tools for IR dirt/dust removal . If you don't have the software - there are ways to use some types of IR scans directly - using blend modes such as "divide" in photoshop/gimp might get you 70-90% of the way there automatically without the side effects of avisynth temporal filters (and avisynth has extended layer blend operations with 3rd party scripts/plugins such as overlayplus)

Appreciate it Poison! Will definitely give your suggestions a try!

[johnmeyer]

Originally Posted by poisondeathray

When you get the IR scans, try the bundled software first, because they usually have dedicated tools for IR dirt/dust removal . If you don't have the software - there are ways to use some types of IR scans directly - using blend modes such as "divide" in photoshop/gimp might get you 70-90% of the way there automatically without the side effects of avisynth temporal filters (and avisynth has extended layer blend operations with 3rd party scripts/plugins such as overlayplus)

This is why I made the recommendation above to use Vuescan. I think you will find that the dust removal made with the IR channel involves more than a simple masking operation. There were quite a few patents involved in the process because it was complex.

[edit]I strongly suspect that there are some very tricky calculations. Remember, the IR scan is still going to show an image. I suspect that the dust spots don't stick out like a sore thumb, so the dust removal algorithm is probably going to involve some very clever difference calculations. Remember, as I already posted, this dust removal does NOT work with some film emulsions. This means the you'll need to take into account things like light scatter, emulsion density, and more.

[poisondeathray]

Originally Posted by johnmeyer

This is why I made the recommendation above to use Vuescan. I think you will find that the dust removal made with the IR channel involves more than a simple masking operation. There were quite a few patents involved in the process because it was complex.

[edit]I strongly suspect that there are some very tricky calculations. Remember, the IR scan is still going to show an image. I suspect that the dust spots don't stick out like a sore thumb, so the dust removal algorithm is probably going to involve some very clever difference calculations. Remember, as I already posted, this dust removal does NOT work with some film emulsions. This means the you'll need to take into account things like light scatter, emulsion density, and more.

Yes, IR does not work for all types of film. The various software also have non IR cleaning modes

Yes, there are more "tricky" calculations for IR cleaning - but the "heavy lifting" is indeed "divide" for some types of IR scans - as I mentioned above. Go download some examples, and the defects do stick out in many cases (but not good enough for avisynth mask as-is) - and divide works

People have posted user developed scripts for use in other software such as photoshop batch actions , content aware fill. Some of them are very specific for a particular scanner model. Usually divide is applied , and/or some prefilter is made to the IR (to provide contrast to the defects if required - that is the "clever" part, probably what the commercial software does too) before divide is applied. Sometimes they actually work better than vuescan or silverfast for cleaning.

IR cleaning doesn't work well for everything either. Vuescan or Silversoft can fail miserably too. There are many factors affecting the results including scanner type, film characteristics like you mentioned, IR contamination. Sometimes the IR scans are just not very good - they are blurry , poor separation, or slightly off - and that negatively affects the cleaning.

[DTL]

Spotless is a simple motion-compensated temporal median. If you understand how median weighting (selection) works when making decisions on output value you can easily add IR channel data to median weighting calculation and get more or less benefit with output quality.

Typically to make low temporal errors at temporal median processing it is better to use as low frames as possible (typically to make a simple majority voting - the 3 frames are required and temporal radius of 1). With additional IR channel for each frame it is possible to work with even 2 frames - simply put IR channel data as weight to each frame sample (like opaque dirt in IR channel create black sample code value and use it as multiplier for other weight calculation - so more dirt will make lower weight for this frame) and pick for output sample with highest weight.

The simplest solution is IR-channel direct masking with 1 current and 1 motion compensated frame. Though it is somehow lower in quality because it does not use more temporal data.

General idea - use IR channel data as one more 'hint' channel to mark some samples in a current tr-pool as 'bad' or at least 'worse'. It also works with any temporal radius and may help to clean more damaged films where several frames have damaged areas at the same sample position. Standard non-IR-hinted temporal median will output a bad distorted sample because it has more 'statistical weight' and no more hints available to estimate samples 'quality'. Also because the IR channel is not 1 bit data - it will help to make more 'soft hinting'.

" I just don't know how to go about making the script to tweak."

If you can not do C-language programming to tweak existing temporal median plugins - you can try to simulate the required algorithm with Expr() AVS core function. See Dogway ex_tools scripts how it can be used. I am not completely sure if it is possible with Expr() but it may be an easy way to start for non-C programmers. If you can provide some real film scans with IR channel data someone may try to add such more to plugins like vsTTempSmooth (pmode=1 that work like temporal median using dissimilarity metric table and additional hinting data may be easily added - simply multiply each SAD with inverted IR channel data - at the line https://github.com/Asd-g/AviSynth-vs...mooth.cpp#L349 ).

"dust removal does NOT work with some film emulsions. "

IR channel scan ideas work only on dye-based colour films and dye-based B&W films - they use dyes transparent in IR. And dirt typically opaque in IR also. The silver films (classic B&W films) do not have a difference in visible and near IR so the idea of getting dust masks in IR scan does not work.

"slightly off "

Yes - IR channel must be perfectly aligned with RGB samples data for best results. So better to pre-process RGB and IR scans data with some auto-aligning software for best results. Some scanners uses dual pass for RGB and IR passes and it can cause some shift because of not perfectly aligned mechanics. Also the lens of scanner can have different distortions (like chroma) between RGB and IR and it can (will) cause general size mismatch errors. So for sample-perfect reqults it is highly recommended to use some aligning software (though lens fixed IR-RGB size mismatch errors can be somehow fixed with scaling).

Addition: In more general form this can be added to temporal median plugins as 'sample quality metric' input. As 1 channel clip of integer format (8..16 bit) where 0 sample code value mean lowest quality and 'max_range' mean highest quality. So script designers can use this input if some more data present about expected quality of samples in a frame (like some motion/luma/chroma etc masks) and it can offset internal median selection algorithm. As a use case - the IR channel scan 'dirt-map' can be used. So the samples marked as 'mostly probably dirt' will get lower 'weight' in similarity analysis and lower probability to be selected for output.

[VideoMilk78]

Originally Posted by johnmeyer

This is why I made the recommendation above to use Vuescan. I think you will find that the dust removal made with the IR channel involves more than a simple masking operation. There were quite a few patents involved in the process because it was complex.

[edit]I strongly suspect that there are some very tricky calculations. Remember, the IR scan is still going to show an image. I suspect that the dust spots don't stick out like a sore thumb, so the dust removal algorithm is probably going to involve some very clever difference calculations. Remember, as I already posted, this dust removal does NOT work with some film emulsions. This means the you'll need to take into account things like light scatter, emulsion density, and more.

Gotcha, even if I do end up using Vuescan I really want to give it a try in Avisynth...