Req: clean-up ideas for 60->30 fps

[FredThompson]

I've always run noise removal/stabilization scripts as if every source frame will be kept. They would include mosquito noise removal, luminence stabilization (camcorder source and artificial light can create some problems), and chroma stabilization - motion-compensation is preferred, as is a little frame stabilization. Sometimes I'll use DaVinci Resolve to mask and adjust brightness such as when the video is a person speaking and their skin is a little overly bright or I forgot to turn off automatic white/brightness in the camcorder.

If the end result was to be 30 fps, every other frame was thrown away after the filtering.

What if the filters could be more efficient/effective/fast by processing only every other frame but using all frames for data used for averaging/noise removal. Maybe there could also be an increase in perceived detail.

Does anyone have experience with this idea? If so, please share your successes/failures (if they are relevant.)

My source is 1080p 60 fps AVC.

[johnmeyer]

If you first decimate to 30 fps the filters will run faster. If the noise is truly random, the quality of noise removal shouldn't be any different. However, any of the algorithms which rely on adjacent frames will not be able to do as good a job because the temporal gap will be larger. As a thought experiment, if you decimated to 1 fps, you would be left with a series of almost-unrelated still images, and things like motion stabilization would have no clue what to do.

So, if it were me, I'd continue to do all processing on the original 60 fps. In fact, if it were me, I'd keep it as 60 fps. Why do you want to degrade your video down to 30 fps??

[wonkey_monkey]

What if the filters could be more efficient/effective/fast by processing only every other frame but using all frames for data used for averaging/noise removal.

That should be what happens anyway - if you're chucking away the odd frames at the end, for example (with selecteven), frame 83 isn't going to be denoised only to be thrown away. Frame 84 will be processed, and it will use (if your filter looks one frame to either side) frames 83 and 85 to do so.

If you first decimate to 30 fps the filters will run faster.

Unless you're saving to an intermediate format after doing so, I'd be surprised if that turned out to be universally (or even mostly) true (compared to decimating at the end of a script).

[FredThompson]

Aw, geez...

It's so obvious now, either selecteven or selectodd to choose the frames to process and assumefps...

30 fps because the source is public speaking. 60 fps doesn't add anything compared to 30 fps.

[hello_hello]

That should be what happens anyway - if you're chucking away the odd frames at the end, for example (with selecteven), frame 83 isn't going to be denoised only to be thrown away. Frame 84 will be processed, and it will use (if your filter looks one frame to either side) frames 83 and 85 to do so.

Doesn't it depend on the order you're doing things? At least for Avisynth? For instance if you follow a noise filter with SelectEven() how does the noise filter know to ignore every second frame?
Unless I'm misunderstanding what you said.

I tried a quick test with these two scripts. Just a low resolution 25fps source I had handy. (I use QTGMC in progressive mode for noise filtering quite a bit) and admittedly the resulting bitrates only differed by around 1kbps, but the first script encoded at nearly twice the speed, indicating QTGMC had less work to do.

SelectEven()
QTGMC(InputType=1, Preset="Medium", EzDenoise=1)
Trim(0,1000)

Vs

QTGMC(InputType=1, Preset="Medium", EzDenoise=1)
SelectEven()
Trim(0,1000)

Potentially I think even these scripts could produce slightly different results because the first prevents QTGMC from accessing frames outside the Trim() range.

Trim(100,200)
QTGMC(InputType=1, Preset="Medium", EzDenoise=1)

Vs

QTGMC(InputType=1, Preset="Medium", EzDenoise=1)
Trim(100,200)

The way I understand it Avisynth doesn't produce unneeded frames only to throw them away, but I'm not sure that amounts to the same thing as not allowing filters to use them for processing if required.

[hello_hello]

I'd imagine denoising before removing every second frame would probably produce a different result to removing the frames first, in the same way denoising a 1080p source before downscaling to 720p would probably produce a different result to downscaling first, at least in theory, but for me it'd depend on the filtering and how much difference it'd make to both the result and the processing speed, along with my care level for the source in question. If the difference was quite small, or virtually indistinguishable, I'd probably go for speed, but I'd run some small test encodes first, or compare the scripts on my TV to choose the method for encoding.

[wonkey_monkey]

Doesn't it depend on the order you're doing things? At least for Avisynth? For instance if you follow a noise filter with SelectEven() how does the noise filter know to ignore every second frame?

It doesn't, but selecteven() knows to only request every other frame from the denoising filter.

QTGMC is rather complicated, to say the least, so who knows why you get those results. If you do something simple like:

version.pointresize(1920,1080)
selecteven
blur(1.58)


vs

version.pointresize(1920,1080)
blur(1.58)
selecteven


it runs at the same speed.

[raffriff42]

>selecteven() knows to only request every other frame from the denoising filter.
Aaah, but the denoising filter will request adjacent frames on its own, even if they are never seen at the output directly!

Select(frame 100) -> denoise (reads frames 99, 100, 101) -> denoised frame 100.
Select(frame 102) -> denoise (reads frames 101, 102, 103) -> denoised frame 102.

Defining a new 30fps clip with SelectEven, and denoising that, should avoid that problem (not tested)

[StainlessS]

To reiterate,

SelectEven last in script:-
The Non-Selected frames are used in the denoising of the requested Selected Frames, and the non Selected Frames are not requested
for output and so are not denoised (and so are not denoised using the adjacent Selected Frames).

SelectEven first in script:
If using SelectEven first, any denoising using the same filter would still request adjacent frames, its just that they will be the adjacent SelectEven
frames and not the adjacent non-SelectEven frames.

So in the end, the same number of frame requests are processed, but results will likely be better where SelectEven called last in script.

EDIT: Below would send requested frames debug stuff to DebugView (Untested, in middle of doing OS setup).

Avisource(...)

S="""
RT_DebugF("%d] Requested",current_frame)
Return Last
"""

ScriptClip(S)

###
# Additional SelectEven/Denoisng stuff after here
###

[raffriff42]

ColorBars(width=256, height=180)
ShowFrameNumber(scroll=true)
EvalShow (http://avisynth.nl/images/Utils-r41.avsi)("""
SelectEven
Merge3frames
""", size=18)
return Last

function Merge3frames(clip C)
{
gamma=0.87
cnt=3.0
C
Overlay(Trim( 0,0),
\ Overlay(Trim( 1,0),
\ Overlay(Trim( 2,0), Last,
\ opacity=Pow( 1.0/cnt, gamma)),
\ opacity=Pow( 2.0/cnt, gamma)),
\ opacity=Pow( 3.0/cnt, gamma))
return Last
}

https://www.dropbox.com/s/9xd47ddjhhz840h/merge-select-01.png?raw=1 https://www.dropbox.com/s/t0592zpcyttw0tw/merge-select-02.png?raw=1

[StainlessS]

Yep, that shows exactly what I meant Raff, nice demo.
I forgot entirely about the ShowFrameNumber(Scroll=true) thing, dont think it occurred to me how it could be used,
thanx teacher :)

EDIT:
EvalShow(), what is that, AVS+, ?
{dont remember that as builtin, no access to Avs docs at the mo, and no hits on google "EvalShow Avisynth"}.

[hello_hello]

It doesn't, but selecteven() knows to only request every other frame from the denoising filter.

That's no doubt effectively how it works, but QTGMC wouldn't know what SelectEven is going to request in advance, so it'd just process each frame as usual.
Put SelectEven before QTGMC and only every second frame is passed to QTGMC for processing.

Another example might be field matching and decimation.
I assume TFM would do exactly the same amount of work in each of these examples:

TFM()
TDecimate(mode=7, rate=24)

TFM()
TDecimate(mode=7, rate=12)

TDecimate can't know which frames to drop until they're all field matched and it can look at them, so you could maybe think of TDecimate as being a smarter SelectEven() in that example.

QTGMC is rather complicated, to say the least, so who knows why you get those results.

Yeah but blur() is no doubt quite fast. Does it work over multiple frames? ie Will it blur a single frame the same way regardless of the previous or following frame? I'd imagine encoding speed would be the bottleneck there, but if you measured just the speed at which Avisynth could output frames there'd probably be a difference.

[StainlessS]

Yeah but blur() is no doubt quite fast. Does it work over multiple frames? ie Will it blur a single frame the same way regardless of the previous or following frame?


Builtin 'Blur()' is Spatial only (not Temporal).

EDIT: To blaze077 below,
Nope, did not notice the link, thanx. Nice collection of script functions.

[blaze077]

EvalShow(), what is that, AVS+, ?
{dont remember that as builtin, no access to Avs docs at the mo, and no hits on google "EvalShow Avisynth"}.
It's linked in his post within the code - though you probably already figured it out. :) Link just in case. (http://avisynth.nl/images/Utils-r41.avsi)

[wonkey_monkey]

>selecteven() knows to only request every other frame from the denoising filter.
Aaah, but the denoising filter will request adjacent frames on its own, even if they are never seen at the output directly!

Select(frame 100) -> denoise (reads frames 99, 100, 101) -> denoised frame 100.
Select(frame 102) -> denoise (reads frames 101, 102, 103) -> denoised frame 102.

Defining a new 30fps clip with SelectEven, and denoising that, should avoid that problem (not tested)

I don't see a problem... it seems quite reasonable, if not preferred, to use those frames for denoising even if they're not going to be in the output.

Each requested frame leads to 3 input frames being requested, whether you do selecteven before or after. Unless you're using a keyframe-only codec, this will likely result in all frames being decoded at some point anyway, and the cache should be enough to avoid too much repetition of decoding.

With selecteven first:

Output frame 100 -> denoise (reads orignal frames 198, 200, 202)
Output frame 101 -> denoise (reads original frames 200, 202, 204)

With selecteven last:

Output frame 100 -> denoise (reads original frames 199, 200, 201)
Output frame 101 -> denoise (reads original frames 201, 202, 203)

I would expect the former to be slower in most cases, since internally a non-keyframe codec will most of the time be decoding five frames instead of three. Though with caching it should work out equal.

[FredThompson]

I hadn't thought about seek speed. That's a nice insight.

For this source material, there is usually a person at a podium and curtains or a wall behind them. One of the challenges is the small sensor in a camcorder. I've used DaVinci Resolve to help lessen the glare off the person's skin where possible. Thought about using edge detection to make a mask for every frame but haven't taken it beyond thought.

The goal is to reduce the chroma "swim" that comes from a small sensor and long distances. Maybe it would make sense to separate the chroma and luma and use something akin to chroma smoothing such as is sometimes used to stabilize chroma with traditional animation video. Just had that idea. A frame or two in succession in which the chroma is a little off (a couple of pixels) won't be visible during playback. That might be an acceptable tradeoff.

Thanks for the comments, guys. Quite helpful.