RgbAmplifier - Forensic Tool - v1.04 - 28 Nov 2018

[Reel.Deel]

A bit similar.
There is a page somewhere with quite a few before/after samples,
I hacked RgbAmplifier at one point to try to simulate same effect and to some degree it worked.
However the other thing requires completely still subject for it to work, contrary to RgbAmplifier which requires just fixed camera.
Eulerian Video Magnification worked better (at what it does) than the hacked version RgbAmp, but I was using their mp4 'before' samples which were probably somewhat more noisy than the actual source that they had available.


I think I remember seeing that page with lots of sample (I also can't find it anymore), but here's the video collection from that site: vidmagSIGGRAPH2012.zip (https://web.archive.org/web/20140802235420/http://people.csail.mit.edu/mrub/vidmag/vidmagSIGGRAPH2012.zip)

[StainlessS]

I think I downloaded about 20 samples from somewhere and only picked about half of those available, dont seem to have link.

Here original posts in RT_Stats thread that I was thinking about, I think there was also some PM traffic which I no longer have.
http://forum.doom9.org/showthread.php?p=1643252#post1643252

and another thread on same with a few links (have not viewed yet).
http://forum.doom9.org/showthread.php?p=1577941&highlight=Eulerian+Video+Magnification#post1577941

EDIT: I viewed the samples that you posted, that 'after shot' of the swing looks bloody dangerous :)

[Wilbert]

An unintended side effect is ghosting. Wouldn't you get less ghosting when using a weighted average (following, say, a gaussian curve, or just bilinear) instead of a normal average? Would be interesting to see the difference ...

[StainlessS]

Not sure how to do that efficiently (EDIT: for up to 198 source frames).
As it stands, it uses an accumulator with three int sized vars for each pixel.
As it moves on to the next frame, it subtracts pixel values of the oldest frame from the accumulator.
I think I could (EDIT: might be able to) manage linear weighting [EDIT: forwards and backwards with two accumulators],
but no idea about the gaussian curve thing.

Perhaps David can come up with something.

EDIT: Perhaps any weighted average would actually stop the filter from doing its (forensic) job, the ghosting is caused
by excessive movement [forensically useless anyway unless radius reduced for that ghosted frame].

[StainlessS]

Further to previous edit.
Perhaps best bet would be to use ClipClop to select reduced radius RgbAmplifier frames where there is significant ghosting/movement.

EDIT:
David, from your post in 'New Plugins Sticky'

It performs temporal smoothing, then (optionally) enhances the difference between the windowed average and the current frame. In theory, this should let you do things like make pulses visible in videos of human faces or wrists, but I suspect you need a very low noise video to pick that up.


That is more like Eulerian Video Magnification than RgbAmplifier.

Forensic, I tried RGBAmplifier on the Face.mp4 with radius between about 7 to 25, it does actually reveal the
same facial flush of the vidmag link, it does however reveal compression artifacts and all minute differences, and does not
look anywhere near as good as vidmag. Not really surprising though as vidmag has a different purpose.
I also did a quick hack and tried magnifying difference between current frame and the average of
(current-off)+(current+off) with off of about 13 frames, even that worked quite well but again was quite noisy.

EVM works something like this (sort of what I did in my hack version).
Imagine clip of a persons wrist which is as stationary as the wrist owner can hold it, you could twin with a sinusoidal wave to match the wrist pulse, where +ve peaks are the pulse with coursing blood. These +ve peaks produce a rhythmical 'beat'.
What is required is to average +ve peaks together and then amplifiy the difference to the averaged -ve peaks.
You might be able to simulate that to some degree by selecting a 'window' that averages 540 degrees of our sinusoidal wave and amplifies
difference between current frame and that average. 540 degrees would be selecting eg 1 +ve arc and 2 -ve arcs of our sinusoidal wave.
So to use RgbAmplifier or your Amp, you need to know the beat frequency of the 'thing' you are trying to amplify and set radius/window to match. The "(current-off)+(current+off)" mentioned in my quote above is not too different to the 540 degrees thing, and as I chose 'off' of 13 frames, it would suggest that the pulse rate would be 13+1+13 ie 27 frames duration (-ve peak to next -ve peak [360 degrees]).
Hope some of that made at least some sense, just in case you would like to pursue the EVM path.
The EVM docs will describe it MUCH better, but I did not really read much of that and never did look at the published source.
The above is a gross over simplification of EVM.

[wonkey_monkey]

An unintended side effect is ghosting.

Unintended but also unavoidable, I think - in fact in some ways ghosting isn't a side effect at all, but a natural and required consequence of the method.

That is more like Eulerian Video Magnification than RgbAmplifier.

[...]

You might be able to simulate that to some degree by selecting a 'window' that averages 540 degrees of our sinusoidal wave and amplifies

I attempted it with an approximate 360 degree window (+/- 12 frames), but 540... interesting...

You need to specify an approximate target frequency with EVM too. The difference is that EVM can be made to amplify only the target frequency, whereas RGBamplifier/amp amplify all frequencies above the baseline (and that includes the highest frequencies of all, i.e. noise). But I have an idea...

David

[StainlessS]

The (current-off)+(current+off) [EDIT: average] plug hack was an attempt to do a crude amp of only target frequency.
But I have an idea...
I smell somethin' good in the oven.

EDIT: by the way, the 540 degree thing is untested, cant even remember if I thought of it some time ago when doing the
RGBAmp hack or just recently when trying to remember what I had done then.

[wonkey_monkey]

Just in case I don't get around to putting my idea into practice - I tried to get a rough idea of how well it might work by running amp twice in succession, without much interesting happening - the idea is to separate the higher frequency noise out first, before trying to enhance the level you're interested in.

In its simplest implementation, you could try specifying two windows, one for noise reduction and a wider one for enhancement. So instead of frame=frame+(frame-smooth)*multiplier, you'd do noise=frame-smooth1; frame=smooth1+(smooth1-smooth2)*multiplier+noise;

More generally (and more accurately, if EVM really is a temporal analogue of my fusion plugin), you'd do repeated iterations of a 1 2 1 temporal kernel, subtracting it from the level above to separate out that particular frequency:


smooth1=video.temporal_blur(1,2,1)
level1=video-smooth1
smooth2=smooth1.temporal_blur(1,2,1)
level2=smooth1-smooth2
...


Then you multiply up the level you're interested in, add all the levels back together, and you're done. Well, that's not quite right, because you also have to drop every other frame at each level, and interpolate them back up before you add again...

I offer you these ramblings in hope of inspiring... something.

[Forensic]

Yes RD, I came across that page + more about 6 months ago, I followed a few of his cases and such, with court verdicts and the like. Forensic is a minor celebrity it seems.

EDIT: Was wondering if he was OK as not been around for some time. Seems he was last active at some forensic shindig
in mid November 2014, so he is up and about. Am still though wondering what happened to Martin53, he went away on
safari and aint been heard of since (although, as well as 'safari' he also said I think 'going to the East' [in PM] so maybe India
or somewhere, not though heard the word 'safari' connected with India, usually Africa).
#StainlessS. I am alive and well, but have been heavily involved in some major endeavours and rarely check these posts anymore. I gave up on writing AVIsynth plug-ins, but still write a lot of scripts. To date, RgbAmplifier has identified the point of origin to several high-damage fires (the trick is first stabilizing the video using object tracking), and has proved who had the right-of-way in numerous vehicle accidents where the entire traffic signal occupies a single pixel of a building's exterior security footage. Thank you for your invaluable help with this project. Your efforts have helped to exonerate the innocent and identify the guilty. I hosted a workshop with RgbAmplifier (and dozens of other AVIsynth filters) at the world's largest forensic's conference last year, and will do so again at UC Santa Barbara in August. You can always reach me by email (you have it). P.S. I never get memory errors or crashes, but that is likely due to forensic videos being low resolution.

[StainlessS]

@forensic, great to hear from you again. Mobile right now and will likely not be around for at least a few days.
The out of men probs were related to the script version, so is nothing to worry about.

I spotted martin53 about 10 days ago sneaking about, so he is well too.

[StainlessS]

RgbAmplifier v1.04, new version, see 1st post.


v1.04, VS2008, Added version resource, and x64.

[StainlessS]

From post #55:- https://forum.doom9.org/showthread.php?p=1703475#post1703475


@StainlessS
Is RgbAmplifier similar to "Eulerian Video Magnification (http://people.csail.mit.edu/mrub/vidmag/)"? Very interesting stuff, I found a web implementation here: https://github.com/antimatter15/evm


EVM works something like this (sort of what I did in my hack version).
Imagine clip of a persons wrist which is as stationary as the wrist owner can hold it, you could twin with a sinusoidal wave to match the wrist pulse, where +ve peaks are the pulse with coursing blood. These +ve peaks produce a rhythmical 'beat'.
What is required is to average +ve peaks together and then amplifiy the difference to the averaged -ve peaks.
You might be able to simulate that to some degree by selecting a 'window' that averages 540 degrees of our sinusoidal wave and amplifies
difference between current frame and that average. 540 degrees would be selecting eg 1 +ve arc and 2 -ve arcs of our sinusoidal wave.
So to use RgbAmplifier or your Amp, you need to know the beat frequency of the 'thing' you are trying to amplify and set radius/window to match. The "(current-off)+(current+off)" mentioned in my quote above is not too different to the 540 degrees thing, and as I chose 'off' of 13 frames, it would suggest that the pulse rate would be 13+1+13 ie 27 frames duration (-ve peak to next -ve peak [360 degrees]).
Hope some of that made at least some sense, just in case you would like to pursue the EVM path.
The EVM docs will describe it MUCH better, but I did not really read much of that and never did look at the published source.
The above is a gross over simplification of EVM.

###############


David, from your post in 'New Plugins Sticky'

It performs temporal smoothing, then (optionally) enhances the difference between the windowed average and the current frame. In theory, this should let you do things like make pulses visible in videos of human faces or wrists, but I suspect you need a very low noise video to pick that up.

That is more like Eulerian Video Magnification than RgbAmplifier.

Forensic, I tried RGBAmplifier on the Face.mp4 with radius between about 7 to 25, it does actually reveal the
same facial flush of the vidmag link, it does however reveal compression artifacts and all minute differences, and does not
look anywhere near as good as vidmag. Not really surprising though as vidmag has a different purpose.
I also did a quick hack and tried magnifying difference between current frame and the average of
(current-off)+(current+off) with off of about 13 frames, even that worked quite well but again was quite noisy.



Looks like Intel is doing something similar, ‘FakeCatcher’ can detect deepfakes in real-time with 96% accuracy
https://metro.co.uk/2022/11/22/intels-fakecatcher-can-detect-deepfakes-with-96-accuracy-17803528/

EDIT: I guess that deep fakers are just gonna havta get used to using 'live video' (with pulsing blood) to avoid detection.
(they can often use multiple stills)

EDIT: "they can often use multiple stills", well that was a few years back (quite often on 'pr0n sites'), apparently they use AI now,
I guess that their AI is gonna have to implement subtle pulsating flushes, not too difficult me is guessin', then back to square one.

EDIT: And in the latest news, China brings in SmellyVision:- https://metro.co.uk/2022/11/22/public-toilet-smell-sensors-warn-visitors-if-cubicle-is-too-stinky-17805663/