Extracting frame(s) from video with stacking and denoise

[tormento]

It's years there are programs/techniques that stack multiple photos to have increased light and/or less denoised one (mosly in low light).

Is it possible to "stack" multiple frames from a video, correcting pan and zoom (sometimes rotation), and denoise to extract a specific frame or multiple frames as images?

[johnmeyer]

With photos, perhaps you are talking about HDR (High Dynamic Range) where you combine identical photos, taken on a tripod a few seconds apart, but each taken with a different shutter speed. This is now done automatically, in a fraction of a second, with the push of a single button, in some cameras. It lets you capture pictures that are perfectly exposed in both the shadows and the highlights, even when there is extreme exposure difference. Example: a woman in a white wedding dress in direct sun standing next to a cave, with the rest of the wedding party ten feet back in the cave. With this technique they will all look perfectly exposed, even without the addition of any additional light. I suppose you might get some noise reduction as part of this averaging process, but most of the time it is used to achieve absurd dynamic exposure range. The photos often look a little strange.

I don't know of any video cameras that offer this.

Averaging multiple frames to reduce noise is how temporal denoisers work. These denoisers first take the adjacent frames and attempt to move and rotate them so they match, as well as can be done, the current frame. In theory, if you can match them perfectly, when you subtract them, all you are left with is the noise, and you can then delete that noise. You need both of the adjacent frames (i.e., three frames total) to do this because otherwise you don't know if the noise is in the adjacent frame or in the current frame.

Play around with MDegrain2 (part of MVTools2) if you want to see how it works.There are several other of these temporal denoisers as well.

[Groucho2004]

Originally Posted by johnmeyer

With photos, perhaps you are talking about HDR (High Dynamic Range) where you combine identical photos, taken on a tripod a few seconds apart, but each taken with a different shutter speed.

Alternatively, AutoStakkert and/or Registax are excellent tools.

I use them for processing video footage of the moon/planets I shoot through my telescope. Here is an example of the footage from my camera (Nikon DSLR + Maksutov 1400/100 telescope + 2x focal extender) and here is the processed photo after AutoStakkert (used for stacking) / Registax (used for wavelet sharpening) / Photoshop (adjusting levels, etc.). As you can see, the video footage has a lot of atmospheric disturbance which stacking eliminates nicely.

These programs are mostly used for astronomy imaging but I used them for terrestrial purposes as well.

[wonkey_monkey]

I don't know of any video cameras that offer this.

I can think of one example I've seen in the past of a camera which recorded alternate frames at different exposures, and I'm sure there was another setup which used an optical splitter of some kind to record two different exposure videos through the same lens.

[StainlessS]

I don't know of any video cameras that offer this.

Mobile phones have been offering HDR video recording for a few years, my xperia Z2 (~2014) has this, but cant say that I've ever used it. : https://www.gsmarena.com/sony_xperia_z2-6144.php

EDIT: Currently, Chinese manufacturers seem to be making loads of 10 core tablets and phones, likely lots of new capability coming.
EDIT: My mates more recent xperia XZ has some kind of laser whatsit, and can take 3D video which can be fed into a 3D printer
(tricky to slowly walk around item to capture, easier if you have some kind of rotating table with static camera position).

[tormento]

Nope. I am not talking about HDR.

I am talking about getting multiple shots of same thing, same exposure and stack to "average" noise and/or increasing brightness, such as this.

My final idea is for underwater based shooting, when sometimes light is never enough, or astronomy.

[johnmeyer]

Originally Posted by tormento

Nope. I am not talking about HDR.

I am talking about getting multiple shots of same thing, same exposure and stack to "average" noise and/or increasing brightness, such as this.

My final idea is for underwater based shooting, when sometimes light is never enough, or astronomy.

I used to do this for video and wrote about it ages ago in this thread:

my-ultimate-vhs-tape-restoration-recipe

I can't remember now, but I think I posted the "recipe" for how you have to combine the images. If you do it wrong, you end up with exposure changes.

Also, I was capturing the same tape multiple times so you would think that aligning the multiple images would not be an issue.

Wrong.

Because of timing issues and also because of field alignment. Analog video doesn't store fields and sothere is no "top" and "bottom" and therefore one or more of the captures often started with a different field.

In your case, alignment is also going to be an issue if there is any variation between the captures. I now fully understand that you are not doing HDR, but the issues you will face are identical. Any movement in the scenery (trees moving, people walking, etc.) will screw up the result. Any movement in the camera will reduce the effectiveness. For underwater photography, both the camera and almost everything in the scene is constantly moving, so even if you take all your pictures in 1/4 of a second, which would be pretty fast, there will be lots of changes between frames.

While everyone has their own budget, I would suggest that if your still photos are particularly noisy, you might want to consider one of the high-end Canon or Nikon DSLRs. I have seen some of the results from extremely high ISO (2000-10000) and the noise is unbelievably low. At normal ISOs and normal exposures, it is non-existent.

In addition, another suggestion is to invest in Adobe Lightroom, shoot in RAW, and then use Lightroom's amazing tools to improve your shots. I got this advice from someone who has since become a major-league professional photographer, and it dramatically improved the results I was able to get.

BTW, if your main interest is underwater photography, there is an expatriate American, living in Thailand, who used to help me a lot over in the Vegas forum, when I was active there. Check out not only his web site, but his extensive writings on the tools and techniques he uses to achieve his world-class results:

Nick Hope Underwater Photography

[StainlessS]

At a guess, I think that tormento may be considering [at least in part] 're-creating' the missing red part of the spectrum which is absorbed by water when at significant depth.
(guess based on past thread, and this quote "sometimes light is never enough")

[tormento]

Originally Posted by StainlessS

At a guess, I think that tormento may be considering [at least in part] 're-creating' the missing red part of the spectrum which is absorbed by water when at significant depth.
(guess based on past thread, and this quote "sometimes light is never enough")

That would be gold but I think almost impossible

It could be simulated only applying a reverse water wavelenght absorption formula and only at a given depth and given distance from object. The first part is partly done by devices such as Paralens camera, The distance estimation is currently an impossible to have data (unless parallax blah blah..).

What I'd like to do is to extract multiple frames from a video, motion compensate them and stack such as done in astrophotography, to enhance brightness and dynamic. Please notice that in diving, such as other type of video shooting, panning is really limited and you sometimes have multiple seconds of video on same subject.

With a limited budget, what amateur divers do is to have a low end camera with an expensive diving frame. Sometimes a Gopro, that goes down to 60 meters.

That is my current equipment and, in certain situation, is really hard to switch from video to photo mode with thick gloves or case mounted over head.

The Protune material you can get from latest iteration of Gopro is really hìgh quality material but the small sensor limit the amount of collected light.

You can shoot at 4k/30 fps or in good light situation even at 60 fps but lowering ISO to more than 800 is too noisy.

Thus my idea of trying to apply the photo stacking idea to video.

But this could also apply to video surveillance (where video is almost still), night shooting and other fields.

And, please, remember that what I want as output is one photo shoot from a small frame sequence. I think that as a first step would be a great goal.

If any red channel information could be recovered, much better. But sometimes you have NO red information.

[StainlessS]

T,
Maybe give Introspected (not on-line since 4 Feb) a poke, see if he has any suggestions, his thread "AutoOverlay" here:- https://forum.doom9.org/showthread.php?t=175247
[its in the devs forum]

[lansing]

Originally Posted by tormento

If any red channel information could be recovered, much better. But sometimes you have NO red information.

You may want to check out 3D LUT Creator for this:
https://youtu.be/gHx0a4TKQbo?t=27