Weird bottlenecking caused by SelectEvery()

[Chabb]

So here is the problem. Script looks like this
(maybe some typos here, but it works):

AVCSource("00020.dga")
AssumeTFF()
SeparateFields()
SelectEvery(10, 0,1,2,3,4,5,7,6) # maybe not exact numbers
Weave()
Levels(16,1.5,235,0,255,coring=false)
FFT3DGPU(planes=4, sigma=1, sigma2=1, sigma3=2.0, sigma4=2.0)
LanczosResize(1280,720,0,0,-4,0)

Source is blu-ray interlaced video (3:2 pulldown).
The script goes directly to x264
and shows VERY low fps ~ 2,
while when SelectEvery row is commented
the speed goes up to 8-9 fps

Is this situation normal?

[Guest]

You have to give us the exact numbers on the SelectEvery() line.

If AVCSource() is forced into seeking mode by SelectEvery(), then things can get slow.

If you have decoded frame n and you request frame n+m, then if m > 7 (IIRC) then the filter uses random seek to get there, rather than just playing m frames linearly and returning the last one.

If you give me the exact numbers I can try to duplicate the issue and see what is happening.

[Chabb]

OK, but this'll happen on Monday

I don't think this random frame request is SO random (up to 7)
In this case of pulldown removal the maximum "offset" is I think 5
I'm taking 5 frames, split them into 10 fields and select correct ones
in such order, that doubtfully leads to requesting some frames with offset > 2 frames forward
The bad thing is I don't know how AVCSource interacts with SelectEvery requests.
Does SelectEvery has some sort of cache?

You may try to figure out the correct numbers,
cause in my case there is normal pulldown with pattern like PPPII
so only two last numbers are unknown.

[Guest]

I don't want to guess. I need you to give me the complete scenario, script, and source sample so I can duplicate it.

SelectEvery() does not have a cache. It simply maps requested frame numbers to other frame numbers. E.g., if you have SelectEvery(2,0), then if you request frame 1 (0-based), it is mapped to 2 before being passed on to the next filter. So SelectEvery() is really trivial and that's why I am thinking seeking slowdown could be involved.

[Gavino]

Just to clarify this a bit. Although SelectEvery() itself has no cache, Avisynth inserts a cache between each filter in the script, so in that sense there will be caching. However, the cache only comes into play if the same frame is fetched more than once, so I don't think it's relevant here.

Having said that, the existence of the various caches needs to be taken into account in predicting the frame access pattern at each point in the chain.

[Guest]

Here's another way seeking could get invoked:

SelectEvery(4,2,0,3,1)

Because frame 0 is requested after frame 2, there is a backward request, which forces seeking to get frame 0. Same for frame 1, because it is requested after frame 3.

If you are doing IVTC you have something like this:

separatefields()
selectevery(10,0,1,4,3,6,7,8,9)
weave()

Note that frame 4 is accessed out of order and so causes seeking.

A workaround would be to save the output of separatefields() in a lossless intermediate file.

[Gavino]

separatefields()
selectevery(10,0,1,4,3,6,7,8,9)
weave()

Note that frame 4 is accessed out of order and so causes seeking.
I think the selectevery could be rewritten as
selectevery(10, 0,1, 3,4, 6,7, 8,9)
to give a more efficient frame fetching order.
It looks 'wrong', but because Avisynth keeps track of the parity for each (separated) frame, the Weave will still put them back in the right place, as long as each pair of fields is correct.

[Guest]

Have you tested that? I didn't think Weave() works per-frame like that.

[Gavino]

Yes, I was surprised to discover this myself just recently. It works because parity (top or bottom) is a per-field property, not a clip property as such.
Try this:
# create dummy field-based (separated) clip
red=BlankClip(100, 400, 150, color=color_red)
blue=BlankClip(red, color=color_blue)
Interleave(red, blue)
AssumeFieldBased()

SelectEvery(4, 0,1, 3,2)
Weave()
You will see visually that each frame has bottom field red and top field blue, even though after SelectEvery, the field order is (red, blue, blue, red). Weave doesn't care about the order, it just looks for a top field and a bottom field for each pair.

Caveat: If you follow this with SeparateFields(), you will get the wrong order back again, so it's best to put Assume[BT]FF() after the Weave() to 'forget' the inherited per-frame parity and make the clip uniform.

[Guest]

Interesting! Then the OP has his needed solution.

[IanB]

Many source formats do not perform well with random access compared to strictly linear access, because they need to always start at the previous I or key frame and decode forwards to the position required.

You can abuse """ChangeFPS (clip1, clip2, bool "linear")""" with no rate change as a small range linear access filter to avoid this problem. It has a 10 frame window.

AVCSource("00020.dga")
ChangeFPS (Last, Last, True) # Cunning trick! Ensure linear access
AssumeTFF()
SeparateFields()
SelectEvery(10, 0,1,2,3,4,5,7,6)
...

[Chabb]

As promised
here it is:

LoadPlugin("H:\Soft\dgavcdec109\DGAVCDecode.dll")
LoadPlugin("c:\Program Files\AviSynth 2.5\plugins\FFT3dGPU.dll")
AVCSource("00020.dga")
AssumeTFF()
SeparateFields()
SelectEvery(10,0,1,2,3,4,5,8,7)
Weave()
Levels(16,1.75,235,0,255,coring=false)
FFT3dGPU(plane=4, sigma=1.0, sigma2=1.0, sigma3=2.0, sigma4=2.0)
Lanczosresize(1280,720,0,0,-4,0)

In order to fully replicate my conditions
the source is Iron Man BD, cutscene "Dubai party"

[Gavino]

SelectEvery(10,0,1,2,3,4,5,8,7)
As neuron2 suspected, the (8,7) is the root of the problem.

Have you tried my suggestion of replacing this by (7,8) or (I think better) IanB's fix using ChangeFPS()?

[Chabb]

the numbers' order is not random
don't know what would happen if I'll change them
cause these are fields, so your advise is to swap fields?
so frame should be spoiled - it's not the result I want visually :)

I'll try IanB's fix later, thanks.

[Gavino]

so your advise is to swap fields?
so frame should be spoiled - it's not the result I want visually :)
My fix does not affect the visual result, just the order in which the frames are fetched from the source. Within each pair, Avisynth knows which field is 'top' and which is 'bottom' so Weave() puts them in the right place - see above for the details.