How to force a GetFrame()

[flossy_cake]

Is it possible to force Avisynth to internally evaluate a clip even if that clip isn't returned 99.9% of the time?

It seems Preroll() kind of works, but only if I set it to a very large value like 100. Values like 50 weren't working. Ultimately I'm not satisfied.

Another thing that seemed to work was making a dummy call to Text that prints nothing, something like ScriptClip(last, """Text("")""", after_frame=true, local=false)

I don't care if it eats CPU I just want to make sure Avisynth is always evaluating every frame of one of my clips! This includes whatever frame properties that clip happens to sets in a ScriptClip of my choosing.

I am trying to avoid Avisynth from doing something like "oh I see you haven't output any frames from clip x in the last 5 minutes, so let's not evaluate x so often"



:thanks:

[StainlessS]

Related stuff,


RT_Stats: https://forum.doom9.org/showthread.php?t=165479

RT_YankChain(clip,int "n"=current_frame,int "delta"=0)
Compile/runtime clip function.
n (default = current_frame) frame number.
delta (default = 0), frame number offset.
Forcibly process the filter graph chain for frame n + delta. Makes intention more explicit than eg RT_AverageLuma(n=i,w=1,h=1).
See also RT_GraphLink().
eg, Below, TmpA plays no part in output clip and so bitmap would not be written. Uncomment the RT_YankChain line and it will.
a=Colorbars.ShowFrameNumber
TmpA = a.subtitle("TestA").ImageWriter("TestA_", type="BMP")
#RT_YankChain(TmpA,n=42) # Uncomment to forcibly write frame 42 of TmpA to bitmap
return MessageClip("All Done")

###

RT_ForceProcess(clip, bool "Video"=True,bool "Audio"=false,Bool "Debug"=True)
If Video (default true), Force read every frame of clip from 0 to FrameCount-1.
If Audio (default false), Force read every sample of clip from 0 to NumberOfSamples-1.
Useful where a clip outputs some kind of file for use in a second filter, this function would in such a case
forcibly read and therefore write the file, so that it may be available to other filters or to a second pass of the filter that
initially wrote the file.
Debug, default true. Write Progress to DebugView.
Returns only after completed force reading of video frames and/or audio samples.
Returns 0.

###

RT_GraphLink(clip source, clip c1, ... , clip cn, bool b1, ... , bool bn)
Standard filter function. Roughly equivalent to "Echo()" in 2.6
This function is a standard filter graph function, not a runtime/compile time function.
It takes a compulsory source clip which it will return unchanged.
The clips, c1 to cn (one or more, at least one of them) are by default forcibly linked into
the filter graph. The bools b1 to bn are optional (zero or more) and would default to True
if not supplied. These bools will if false, switch OFF the forcible linking into the filter graph
for the corresponding clip c1 to cn.
Avisynth does not normally process any filter chains that do not contribute to the output clip,
this filter allows you to select graph chains that you wish to forcibly process.
The usual way to force process an unused chain is to do a StackHorizontal/Vertical and then crop it off again.
The forced clips should all be same length as the source clip (but no error if not), only frame numbers contained
in both source and forced clips will be forcibly accessed when frame n of source clip is accessed.
All args un-named. See also RT_YankChain.
Example: [will only render the TestA" and TestC images]
a=ImageSource("d:\avs\1.jpg",end=0) # Single frame
TmpA = a.subtitle("TestA").ImageWriter("d:\avs\TestA_", type="png") # Written
TmpB = a.subtitle("TestB").ImageWriter("d:\avs\TestB_", type="png") # Skipped
TmpC = a.subtitle("TestC").ImageWriter("d:\avs\TestC_", type="png") # Written
RT_GraphLink(a,TmpA,TmpB,TmpC,True,false) # 3rd bool defaults true
return Last

Without the above RT_GraphLink, the image would be displayed and none of the images would be written.

###
TWriteAVI: https://forum.doom9.org/showthread.php?t=172837

ForceProcessAVI(clip c,bool "Debug"=True)
Force Process clip c, ie read from first to last frame, for TWriteAVI writes the AVI file (Video + Audio) without having to play clip.
Debug, default True, sends some progress info to DebugView (google).
[EDIT: similar to RT_ForceProcess(), can use for eg 2 pass scripts, pass1 create deshaker type log, pass2, use deshaker log]

###

Avisynth+ builtin [EDIT: v2.6 builtin:- http://avisynth.nl/index.php/Echo ]
Echo: https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/echo.html

Echo forces getframe accesses to all the input clips (up to the number of frames of the first clip). The results from clip2 and up are discarded. The result from clip1 is returned.

This may be useful forcing alternate script paths with ImageWriter and WriteFile.
Syntax and Parameters


Echo (clip1, clip2 [, ...])


EDIT: Echo is actually a v2.6 plug.

[flossy_cake]

Related stuff,
EDIT: Echo is actually a v2.6 plug.


I couldn't get Echo to work no matter how I called it - it just outputs last from before the Echo was called, even though Echo(some_other_clip) is supposed to be output.

Tried RT_ForceProcess - seems like I have to wait for it to process every frame before it will output anything? In other words must wait a very long time before playback - have I got that right?

Tried RT_GraphLink(last,last) - is that valid usage? I just want to output last and force it to be evaluated every frame. If that is correct usage then it didn't solve my issue either but probably wasnt designed for the funky way I'm using multiple ScriptClips upstream from the RT_GRaphLink.

Preroll()ing about 5 seconds worth of frames seems to be doing the trick though, I might just stick with that.

The Text trick I guess is similar to what you were talking about with doing a StackHorizontal and then cropping. Assumign Text(" ") actually renders something rather than just seeing it's an empty string and bypassing the function altogether.

[StainlessS]

I still dont know what it is that you are trying to do.

Echo does nearly exactly the same as RT_Graphlink, it was implemented shortly after I implemented RT_Graphlink, which was implemented to avoid the Stack:Horizontal/Vertical,Crop hack thingy.
RT_YankChain also does similar but forces fetch of a specific_frame/current_frame + optional offset.

EDIT: See example from v2.6 docs on echo:- http://avisynth.nl/index.php/Echo

Echo
Jump to: navigation, search

AviSynth+
Up-to-date documentation: https://avisynthplus.readthedocs.io


Echo(clip clip1, clip clip2 [, ...])

Echo forces getframe accesses to all the input clips (up to the number of frames of the first clip). The results from clip2 and up are discarded. The result from clip1 is returned.

This may be useful forcing alternate script paths with ImageWriter and WriteFile.

Examples

clip1 = ColorBars.Trim(0,99).ShowFrameNumber() # numbered clip (640x480)
clip2 = clip1.SelectEvery(10, 0) # select frame 0, 10, 20, ...
clip2 = clip2.BilinearResize(160, 120) # create thumbnails of frame 0, 10, 20, ...
clip2 = clip2.ImageWriter(file="D:\AviSynth\Plugins\file", type="jpg")
Echo(clip1, clip2) # returns clip1, exports clip2 (the thumbnails)

[DTL]

"if that clip isn't returned 99.9% of the time?"

What is the clip example ?

If you want all frames of a clip (output clip of a filtergraph ?) to be computed - you simply need to run total environment with some every frame consuming (requesting) software. Like open in VirtualDub and hit 'run' button to play of some player (media player classic ?). Also a clip is enough multi-instance object in filtergraph - each filter can have different clip at its input and output (different in fps and frame count/size/format etc). What clip to you mean you want all frames ?

[StainlessS]

@Flossy,

If you just mean scan entire source clip, similar to VDub2 "Run_Video_Analysis_Pass", then either of the RT_Stats::RT_ForceProcess or TwriteAVI::ForceProcessAVI functions will do.
[They are functions not filters and will only return when entire clip has been scanned, I show examples of only ForceProcessAVI from TWriteAVI thread, RT_ForceProcess could also be used]

ForceProcessAVI(clip c,bool "Debug"=True) # EDIT: Debug outputs % progress to DebugView, Google DebugView, a free Micro$oft utility (originally by SystemInternals).


TWriteAVI v2.0, can write lossless whatever can be played to AVI (or WAV), and using the ForceProcessAVI/WAV functions, can write the
files without playing them (and faster than eg VDub Video Analysis Pass [EDIT: about 70% of VDub time]).

Two Pass:

WhateverSource(...)
A_Two_Pass_Function(Pass=1) # Create some kind of deshaker log or whatever.
ForceProcessAVI() # Force Pass 1, creating log (A function From TWriteAVI plug)
A_Two_Pass_Function(Pass=2) # Use created log
return Last

The ForceProcessAVI function forcibly processes both video and audio streams (ie reads each frame/sample in sequence),
so that any eg log files produced by either video and/or audio processing filter will be forcibly written to disk.
ForceProcessAVI/WAV are runtime functions not filters and so only return on completion.

Simultaneous play and save AVI

WhateverSource(...)
FN="Output.AVI"
FOURCC="ULY0" # Ut_Video
TWriteAVI(FN,Overwrite=true,fourcc=FOURCC).
return Last


Do some kind of processing and save AVI, Then play [EDIT: saved] Lossless file.

WhateverSource(...)
FN="Output.AVI"
FOURCC="ULY0" # Ut_Video
Sharpen(0.3) # Whatever
TWriteAVI(FN,Overwrite=true,fourcc=FOURCC). # Write Sharp AVI whilst clip is played
ForceProcessAVI() # Force Write above TWriteAVI file without playing clip.
AVISource(FN) # Load Saved AVI
return Last # Play Sharpened AVI lossless clip

[flossy_cake]

I still dont know what it is that you are trying to do.

"if that clip isn't returned 99.9% of the time?"
What is the clip example ?


I tried to make a short synthetic test for you to reproduce on your end, but couldn't reproduce it. It's only when I use other more complicated 5000 line script with multiple scriptclips all talking to eachother via frameprops with multithreading enabled and such that the problem occurs. Disabling multithreading didn't solve the issue either but I know it's dependent on CPU load since inserting QTGMC into the pipeline would also randomly resolve it for some unknown reason.

From what I can tell Avisynth doesn't guarantee things get evaluated in linear order even when single threaded so now I'm thinking it's probably related to that rather than a getframe issue. Probably all the clips I want to be evaluated are in fact being evaluated but clips are going out of sync with eachother so when I set a frame prop in one clip, other clips downstream from that can't be guaranteed to get the same prop for that same frame number. The fact that I still got a frame prop at all from the upstream clip seems to imply that upstream clip did infact do a getframe but had finished processing a different frame at the time the prop was being retrieved downstream.

I know scriptclip has that "after_frame=true" but it's not 100% reliable imo. There needs to be some other parameter we can set to tell Avisynth "do not under any circumstances evaluate this frame until the following upstream filters are finished" but that would probably hurt performance a lot and I need performance cause I use it for realtime HTPC stuff

So in the end I'm satisfied with using preroll or that text("") bodge

Sorry I couldn't give you anything to work with

[flossy_cake]

I guess it would help me sleep at night if I knew that behind the scenes when I am inside a scriptclip and getting a frameprop from a clip , that this forces a getframe on that clip. I reckon it does, but would be great if I could get confirmation on that.

The props i'm getting from that clip could be a result of what happened 5 minutes ago in that clip, so it really does need to be evaluating each and every frame of that clip.

[flossy_cake]

I still dont know what it is that you are trying to do.

"if that clip isn't returned 99.9% of the time?"
What is the clip example ?

So I guess what I'm wanting to know is if in the following example, clip x will be evaluated every frame behind the scenes?

global x = some_clip

ScriptClip(last, "Function(last, current_frame)", after_frame=true, local=false)

function Function(clip c, int current_frame){

if (propGetAny(x, "some_prop") == 1){ return x }
else { return c }
}



Because in my script, Function() will be returning clip c (last) 99.99% of the time, and my concern was that Avisynth is maybe not actually evaluating clip x every frame since it almost never gets output

But looking at CPU utilization % I think it actually is evaluating x every frame regardless of whether x gets output or not

So I guess the ultimate question is whether a propGet() triggers a getFrame()?

[flossy_cake]

There needs to be some other parameter we can set to tell Avisynth "do not under any circumstances evaluate this frame until the following upstream filters are finished" but that would probably hurt performance a lot and I need performance cause I use it for realtime HTPC stuff

Actually now that I think about it , that's probably the issue that Preroll() solves. Because instead of being forced to evaluate all frames in a clip (thus hurting performance) preroll can let us set the window of how far back to go in time when it's that clip's turn to start being output

I think it coincidentally solves my issue of desyncing of frameprops between upstream/downstream filters since running the clock back 3 seconds worth of frames gives the frame cache time to settle down and "synchronise" or whatever it's doing.

[StainlessS]

When you first posted, I thought that there was another filter a bit like Preroll(), but could not remember what it was, I just remembered, TIVTC::RequestLinear().

TIVTC::RequestLinear

RequestLinear v1.4 - HELP FILE (20201020)


GENERAL INFO:


* RequestLinear supports all colorspaces

A little diagram to show downstream vs upstream in a script:

UpstreamFilter()
RequestLinear()
DownstreamFilter()

RequestLinear makes sure that all requests for frames from the filter
downstream are turned into linear requests for frames from the filter
upstream of it. This can be useful when you want to use a filter that
requests frames non-linearly in combination with a filter that requires
linear requests.

For example, if the filter downstream requests frame 0 and then requests
frame 5, RequestLinear would get the request for frame 5 and then request
frames 1, 2, 3, 4, and then 5 from the filter upstream.

It is possible to set a maximum number of frames for the gap that it
will fill and to specify how many frames it should cache.


syntax=>

RequestLinear(int rlim, int clim, int elim, bool rall, bool debug)



PARAMETERS:


rlim -

If the current frame request if greater than the last delivered frame
and the difference between the two is less than or equal to rlim, then
all frames between the last request and the current request will be
requested. If the current request is less than or equal to the
last request then the distance is measured from 0 to the current
request. rlim must be >= 0.

Default: 50 (int)


clim -

Sets the number of frames that the filter should cache. Frames that
are in the cache can be delivered without having to request any frames
from upstream. clim must be >= 0.

Default: 10 (int)


elim -

If the current frame ends up being requested without the previous frames
being requested (due to rlim not being large enough, clim not being large
enough, etc...), elim sets the number of frames prior to the current frame
to request in linear order prior to requesting the current frame.

Default: 5 (int)


rall -

If true, it is the same as if rlim is set to infinity. If false,
it does nothing.

Default: false (bool)


debug -

Will show which frames are being requested from upstream and
downstream. Use the utility "Debugview" from sysinternals to
view the output.

Default: false (bool)



EXAMPLE SCENARIOS:


1.) Downstream filter has requested every frame from 0 to 20 and then
requests frame 25.

if rlim >= 5 or rall = true, then RequestLinear will request frames
21 through 25 from the upstream filter

else if elim > 0, then all frames from 25-elim to 25 will be requested

else, frame 25 is requested by itself


2.) Downstream filter has requested every frame from 0 to 50 and then
requests frame 10.

if clim >= 40, then frame 10 will be returned from the cache w/o
making any upstream requests

else if rlim >= 10 or rall = true, then frames 0 through 10 will be
requested from the upstream filter

else if elim > 0, then all frames from 10-elim to 10 will be requested

else, frame 10 will be requested by itself


3.) Given RequestLinear(50,10,5) and last requested frame = 300 (assume linear
access so far)

current request: frame 320
-> linear request of all frames 301, ..., 320 (due to rlim)

current request: frame 295
-> read from cache (due to clim)

current request: frame 400
-> linear request 395,396,397,398,399,400 (due to elim)

current request: frame 230
-> linear request 225,226,227,228,229,230 (due to elim)



CHANGE LIST:

v1.4 (20201020)

- fix: initial large frame number difference out of order frame requests
e.g. 24, 25, 26, 6 (caused by heavy multithreading) could result in
internal error - frame not in cache


v1.3 (20200611)

- filter reports MT_SERIALIZED mt mode for Avisynth+

v1.2 (11/21/2007)

- added elim parameter


v1.1 - (4/09/2006)

- fixed not testing if n <= rlim if the current request was <=
to the last request (it was only testing if rall=true)

v1.0 - (3/23/2006)

- initial release



contact: forum.doom9.org nick = tritical or email: kes25c@mizzou.edu


TIVTC:- http://avisynth.nl/index.php/TIVTC

NOTE, there are cache's inserted between filters, and if a frame has preveiously been fetched (recently) it may be returned from the cache and not trigger another GetFrame.

EDIT: Maybe of use,
Passing values between runtime scripts:- https://forum.doom9.org/showthread.php?t=170768
Because there is a minimum value, you cannot disable the cache completely using SetMemoryMax().
However, in v2.60, you can effectively disable it by redefining the Cache() function (in your script) as follows:
function Cache(clip c) { return c }

EDIT: Related:: How to refresh output/clear the cache:- https://forum.doom9.org/showthread.php?p=1687180

[StainlessS]

Forgot that I'de done this,
CacheTest(), debug tool:- https://forum.doom9.org/showthread.php?t=184920


CacheTest(), by StainlessS @ Doom9.

CacheTest(clip c, String "text"="", String DvChar="", Bool "Always"=False, Bool "Silent"=False)

CacheTest() Sends text message to DebugView when requested frame is "Out Of Sync", ie not previous frame + 1.
Insert between filters to see if there are any repeat requests for frame, or out of sync frame requests.

Requires DebugView, [ DebugView v4.90, @ Micro$oft:- https://learn.microsoft.com/en-us/sysinternals/downloads/debugview ]
DebugView captures debug messages on your local machine (or network) and shows them in a window, can also save to file.
These messages may be output by the system, or eg graphics card driver, or some other program on a machine.
[really annoying when nVidia floods debug logs in some supposedly non beta drivers].
You can add a DebugView Filter via DebugView Edit menu, to only capture debug message lines containing some substring eg ":",
so that you only see the lines containing that substring, but you would then need to ensure that your debug messages also contain ":".
NOTE, RT_Stats plugin has RT_Debug() and RT_DebugF() functions for outputting script debug Info in real time during Filter Graph creation
stage, or during frameserving from within the Runtime environment (eg ScriptClip).
All RT_Debug/RT_DebugF messages contain a COLON character.

Args:

Text, Default "".
Allows add some kind of text label to the Title string that is output to DebugView.
Where CacheTest filter Instance is 3, and Text="PorridgeOats", and DvChar="", then
the Title output would be "CacheTest_0003_PorridgeOats:". (Ends with SINGLE COLON, ':')

DvChar, Default "".
Where CacheTest filter Instance is 3, and Text="PorridgeOats", and DvChar=":", then
the Title output would be "CacheTest_0003_PorridgeOats::". (Ends with DOUBLE COLON, ':')
The DvChar is inserted after the always present 1st ":" character, it allows to set
a more specific DebugView filter string, so as to only see CacheTest debug messages in DebugView.
Some messages sent to DebugView may not involve a COLON, some do use a COLON, you can eg set
DvChar to ":" so that only DebugView messages containing "::" will be captured, or can set some other character following ':'.
(You could even set DebugView filter to eg "CacheTest" if required, or even "PorridgeOats").

Always, Default False.
True = Output all frame requests debugview.

Silent, Default False.
True = switch off ALL messages (overrides ALWAYS).
Leaves an additional Cache between filters, without writing messages to DebugView.

Shows messages as in
"CacheTest_0005_Node_A1: 1001] *** REPEAT REQUEST {RepeatCnt=7, OSyncCnt=21}"
where there is a repeat request for CacheTest instance 0005 where Text="Node_A1",
@ frame 1001, and total Repeat Count = 7, and Out Of Sync Count = 21.
OR,
"CacheTest_0003_Marmalade: 873] *** OUT OF SYNC REQUEST, Previous=884 {RepeatCnt=7, OSyncCnt=21}"
where there is an Out Of Sync request for CacheTest instance 0003 where Text="Marmalade",
@ frame 873, where previous request was for frame 884, and total Repeat Count = 7, and Out Of Sync Count = 21.

############################################

Example:
Return Colorbars(Pixel_type="YV12").KillAudio.CacheTest()


CashTest_Test.avs

TEXT = "Marmalade"
DVCHAR = "" # Or maybe try eg ":"
ALWAYS = False # True = Output frame requests at every frame.
SILENT = False # NOT NORMALLY USED, NEARLY ALWAYS FALSE.

#Colorbars(Pixel_type="YV12").KillAudio
AVISource("D:\TEST.avi")

ORG=Last

CacheTest(Text=TEXT, DvChar=DVCHAR, Always=ALWAYS, Silent=SILENT) # OR Just CacheTest()

return Last


Some results for manually jumping about a bit [ Just using CacheTest() ]

00001691 0.22455139 [10788] CacheTest_0001: 0] *** Initialized to frame 0 {RepeatCnt=0, oSyncCnt=0}
00001692 1.09279370 [10788] CacheTest_0001: 0] *** REPEAT REQUEST (ReSeek to 0, Player starts to play ???), {RepeatCnt=0, oSyncCnt=0}
00001693 4.92930794 [10788] CacheTest_0001: 89] *** OUT OF SYNC REQUEST, Previous=122 {RepeatCnt=0, oSyncCnt=1}
00001694 4.93385696 [10788] CacheTest_0001: 2582] *** OUT OF SYNC REQUEST, Previous=89 {RepeatCnt=0, oSyncCnt=2}
00001695 10.90573597 [10788] CacheTest_0001: 2722] *** OUT OF SYNC REQUEST, Previous=2755 {RepeatCnt=0, oSyncCnt=3}
00001696 13.87600136 [10788] CacheTest_0001: 14400] *** OUT OF SYNC REQUEST, Previous=2722 {RepeatCnt=0, oSyncCnt=4}
00001697 16.37423706 [10788] CacheTest_0001: 0] *** Rewind to frame 0, Previous=14400, Counts RESET {RepeatCnt=0, oSyncCnt=0}
00001698 46.82066727 [10788] CacheTest_0001: 674] *** OUT OF SYNC REQUEST, Previous=707 {RepeatCnt=0, oSyncCnt=1}
00001699 47.55070114 [10788] CacheTest_0001: 708] *** OUT OF SYNC REQUEST, Previous=705 {RepeatCnt=0, oSyncCnt=2}


If you use Always=True, and message does not appear in DebugView when moving to a frame, its because that already visited frame
has been gotten from cache (CacheTest GetFrame was not processed because the frame was obtained from a downstream cache <nearer to the output>).


EDIT: A 2nd monitor is ideal for DebugView use, even a small 4:3 VGA monitor is extremely handy.
EDIT: [I have 28" DisplayPort 4K 10bpc monitor, and 19" 1280x1024 VGA 4:3 8bpc <EDIT: via DP->VGA adapter>, and nVidia GTX 1070 driving both, the monitors work together just fine, no real probs that I can think of]
Above 2 edits written some time ago, I now have discarded the VGA monitor and have a bigger 2nd monitor, but the VGA monitor cost me only £1.50, and even that was a really useful item.
Decades ago on the Atari ST and Commodore Amiga, I used to send serial debug info to a Sinclair QL computer, connected to an industrial 9 inch green screen {composite} monitor for debug info,
QL + green screen combo acted like DebugView [I still regret giving my QL + green screen to a charity shop].

[flossy_cake]

Great info there, thank you.