RgbAmplifier - Forensic Tool - v1.04 - 28 Nov 2018

[Reel.Deel]

Just out of curiosity, for this method to work (RgbAmplifier) does it have to be RGB or can it be implemented for YUV also? Not requesting anything, just wondering.

[StainlessS]

Dont think it would yield anything useful (even at PC levels) in YUV. (Think there was mention of that early in thread)
See edit to prev post.

EDIT: I usually do implement for all colorspaces unless there is a reason not to, it aint usually that much extra bother
(EDIT: Unless non-full frame Interlaced PLANAR is involved, can be a lot of extra bother).

[wonkey_monkey]

I had no luck getting RgbAmplifier to work - it just ran out of memory - so I implemented my understanding of the process in a plugin:

http://forum.doom9.org/showthread.ph...32#post1703332

[Wilbert]

I had no luck getting RgbAmplifier to work - it just ran out of memory - so I implemented my understanding of the process in a plugin:

Pity it is closed source again ;( Guess I have to implement it myself.

Do you think we will get 'better' results if do it in HSV space?

[wonkey_monkey]

Ah, well, for once I've written a plugin that doesn't rely on my convoluted image library, so sharing the source is easy - I've added it to the zip file. I didn't put a licence in there but let's just agree it's GPLv2

Apologies if the code is a bit in opaque in places - I tend to favour conciseness over clarity.

No idea what space would give best results - avoiding conversions seemed a good idea though.

David

[StainlessS]

David, can you please give spec on you system.
Already had probs reported on clipclop, perhaps related.
I may have given false info to reelreel, perhaps yuv was not so impossible,
Maybe I confused two diff plugs.
Anyways is a concern that you had OM probs, sounds very much like clipclop
Prob, perhaps a compiler thing, don't know.
Gonna have a few more seasonally approved beers and take another peek.
Dh, post your soure or at least pm to me, there should I hope have been no problem and is no problem on 32 bit m/c,s, I think.
Running out of battery, bye.

[wonkey_monkey]

On trying to load a script using RgbAmplifier, over the course of about 30 seconds VirtualDub's memory usage creeps up to 2Gb, then I get:

Avisynth open failure:
GetFrameBuffer: Returned a VFB with a 0 data pointer!
size=2073664, max=536870912, used=1067512512
I think we have run out of memory folks!
([GScript], line 15)
([GScript], line 27)
([GScript], line 42)
([GScript], line 43)
([GScript], line 45)

I copied and pasted the function from the 2nd post in this thread, and it was the first thing in my script, so those line numbers should correspond with line numbers in the function as posted.

The source was a 1920x1080 MTS, but the same thing happened with a half-sized Ut codec version, and also when I added another reduceby2 (making it 480x270).

I've added amp's source to the .zip file linked above.

RunniRunning out of battery, bye.

Aww. It's like Hal singing Daisy, Daisy at the end of 2001.

[Reel.Deel]

@davidhorman
Have you tried the current RgbAmplifier plugin?

----

Edit:
@StainlessS
Is RgbAmplifier similar to "Eulerian Video Magnification"? Very interesting stuff, I found a web implementation here: https://github.com/antimatter15/evm

[wonkey_monkey]

Huh, I must have missed that one. It gives reassuringly identical results to mine, but is twice as fast. Mine works in YV12 and YUY2, though.

[StainlessS]

Oh thank goodness, I thought that you meant the plugin had Out Of Memory problems.
(The original script had OM problems due to OverLay which is very greedy on memory).

Originally Posted by Reel.Deel

Is RgbAmplifier similar to Eulerian Video Magnification

A bit similar.
There is a page somewhere with quite a few before/after samples,
I hacked RgbAmplifier at one point to try to simulate same effect and to some degree it worked.
However the other thing requires completely still subject for it to work, contrary to RgbAmplifier which requires just fixed camera.
Eulerian Video Magnification worked better (at what it does) than the hacked version RgbAmp, but I was using their mp4 'before' samples which were probably somewhat more noisy than the actual source that they had available.
There were a number of exchanges between me and Forensic on that subject, but not sure if in RT_Stats or some other thread, or might even have been via PM.

(My PM box got full some months ago and so I wiped it).

[Reel.Deel]

Originally Posted by StainlessS

A bit similar.
There is a page somewhere with quite a few before/after samples,
I hacked RgbAmplifier at one point to try to simulate same effect and to some degree it worked.
However the other thing requires completely still subject for it to work, contrary to RgbAmplifier which requires just fixed camera.
Eulerian Video Magnification worked better (at what it does) than the hacked version RgbAmp, but I was using their mp4 'before' samples which were probably somewhat more noisy than the actual source that they had available.

I think I remember seeing that page with lots of sample (I also can't find it anymore), but here's the video collection from that site: vidmagSIGGRAPH2012.zip

[StainlessS]

I think I downloaded about 20 samples from somewhere and only picked about half of those available, dont seem to have link.

Here original posts in RT_Stats thread that I was thinking about, I think there was also some PM traffic which I no longer have.
http://forum.doom9.org/showthread.ph...52#post1643252

and another thread on same with a few links (have not viewed yet).
http://forum.doom9.org/showthread.ph...on#post1577941

EDIT: I viewed the samples that you posted, that 'after shot' of the swing looks bloody dangerous

[Wilbert]

An unintended side effect is ghosting. Wouldn't you get less ghosting when using a weighted average (following, say, a gaussian curve, or just bilinear) instead of a normal average? Would be interesting to see the difference ...

[StainlessS]

Not sure how to do that efficiently (EDIT: for up to 198 source frames).
As it stands, it uses an accumulator with three int sized vars for each pixel.
As it moves on to the next frame, it subtracts pixel values of the oldest frame from the accumulator.
I think I could (EDIT: might be able to) manage linear weighting [EDIT: forwards and backwards with two accumulators],
but no idea about the gaussian curve thing.

Perhaps David can come up with something.

EDIT: Perhaps any weighted average would actually stop the filter from doing its (forensic) job, the ghosting is caused
by excessive movement [forensically useless anyway unless radius reduced for that ghosted frame].

[StainlessS]

Further to previous edit.
Perhaps best bet would be to use ClipClop to select reduced radius RgbAmplifier frames where there is significant ghosting/movement.

EDIT:
David, from your post in 'New Plugins Sticky'

Originally Posted by davidhorman

It performs temporal smoothing, then (optionally) enhances the difference between the windowed average and the current frame. In theory, this should let you do things like make pulses visible in videos of human faces or wrists, but I suspect you need a very low noise video to pick that up.

That is more like Eulerian Video Magnification than RgbAmplifier.

Originally Posted by StainlessS

Forensic, I tried RGBAmplifier on the Face.mp4 with radius between about 7 to 25, it does actually reveal the
same facial flush of the vidmag link, it does however reveal compression artifacts and all minute differences, and does not
look anywhere near as good as vidmag. Not really surprising though as vidmag has a different purpose.
I also did a quick hack and tried magnifying difference between current frame and the average of
(current-off)+(current+off) with off of about 13 frames, even that worked quite well but again was quite noisy.

EVM works something like this (sort of what I did in my hack version).
Imagine clip of a persons wrist which is as stationary as the wrist owner can hold it, you could twin with a sinusoidal wave to match the wrist pulse, where +ve peaks are the pulse with coursing blood. These +ve peaks produce a rhythmical 'beat'.
What is required is to average +ve peaks together and then amplifiy the difference to the averaged -ve peaks.
You might be able to simulate that to some degree by selecting a 'window' that averages 540 degrees of our sinusoidal wave and amplifies
difference between current frame and that average. 540 degrees would be selecting eg 1 +ve arc and 2 -ve arcs of our sinusoidal wave.
So to use RgbAmplifier or your Amp, you need to know the beat frequency of the 'thing' you are trying to amplify and set radius/window to match. The "(current-off)+(current+off)" mentioned in my quote above is not too different to the 540 degrees thing, and as I chose 'off' of 13 frames, it would suggest that the pulse rate would be 13+1+13 ie 27 frames duration (-ve peak to next -ve peak [360 degrees]).
Hope some of that made at least some sense, just in case you would like to pursue the EVM path.
The EVM docs will describe it MUCH better, but I did not really read much of that and never did look at the published source.
The above is a gross over simplification of EVM.

[wonkey_monkey]

Originally Posted by Wilbert

An unintended side effect is ghosting.

Unintended but also unavoidable, I think - in fact in some ways ghosting isn't a side effect at all, but a natural and required consequence of the method.

That is more like Eulerian Video Magnification than RgbAmplifier.

[...]

You might be able to simulate that to some degree by selecting a 'window' that averages 540 degrees of our sinusoidal wave and amplifies

I attempted it with an approximate 360 degree window (+/- 12 frames), but 540... interesting...

You need to specify an approximate target frequency with EVM too. The difference is that EVM can be made to amplify only the target frequency, whereas RGBamplifier/amp amplify all frequencies above the baseline (and that includes the highest frequencies of all, i.e. noise). But I have an idea...

David

[StainlessS]

The (current-off)+(current+off) [EDIT: average] plug hack was an attempt to do a crude amp of only target frequency.

But I have an idea...

I smell somethin' good in the oven.

EDIT: by the way, the 540 degree thing is untested, cant even remember if I thought of it some time ago when doing the
RGBAmp hack or just recently when trying to remember what I had done then.

[wonkey_monkey]

Just in case I don't get around to putting my idea into practice - I tried to get a rough idea of how well it might work by running amp twice in succession, without much interesting happening - the idea is to separate the higher frequency noise out first, before trying to enhance the level you're interested in.

In its simplest implementation, you could try specifying two windows, one for noise reduction and a wider one for enhancement. So instead of frame=frame+(frame-smooth)*multiplier, you'd do noise=frame-smooth1; frame=smooth1+(smooth1-smooth2)*multiplier+noise;

More generally (and more accurately, if EVM really is a temporal analogue of my fusion plugin), you'd do repeated iterations of a 1 2 1 temporal kernel, subtracting it from the level above to separate out that particular frequency:

smooth1=video.temporal_blur(1,2,1)
level1=video-smooth1
smooth2=smooth1.temporal_blur(1,2,1)
level2=smooth1-smooth2
...

Then you multiply up the level you're interested in, add all the levels back together, and you're done. Well, that's not quite right, because you also have to drop every other frame at each level, and interpolate them back up before you add again...

I offer you these ramblings in hope of inspiring... something.

[Forensic]

Originally Posted by StainlessS

Yes RD, I came across that page + more about 6 months ago, I followed a few of his cases and such, with court verdicts and the like. Forensic is a minor celebrity it seems.

EDIT: Was wondering if he was OK as not been around for some time. Seems he was last active at some forensic shindig
in mid November 2014, so he is up and about. Am still though wondering what happened to Martin53, he went away on
safari and aint been heard of since (although, as well as 'safari' he also said I think 'going to the East' [in PM] so maybe India
or somewhere, not though heard the word 'safari' connected with India, usually Africa).

#StainlessS. I am alive and well, but have been heavily involved in some major endeavours and rarely check these posts anymore. I gave up on writing AVIsynth plug-ins, but still write a lot of scripts. To date, RgbAmplifier has identified the point of origin to several high-damage fires (the trick is first stabilizing the video using object tracking), and has proved who had the right-of-way in numerous vehicle accidents where the entire traffic signal occupies a single pixel of a building's exterior security footage. Thank you for your invaluable help with this project. Your efforts have helped to exonerate the innocent and identify the guilty. I hosted a workshop with RgbAmplifier (and dozens of other AVIsynth filters) at the world's largest forensic's conference last year, and will do so again at UC Santa Barbara in August. You can always reach me by email (you have it). P.S. I never get memory errors or crashes, but that is likely due to forensic videos being low resolution.

[StainlessS]

@forensic, great to hear from you again. Mobile right now and will likely not be around for at least a few days.
The out of men probs were related to the script version, so is nothing to worry about.

I spotted martin53 about 10 days ago sneaking about, so he is well too.