AviSynth+ thread Vol.2

[GMJCZP]

I clarify that by mistake I deleted the AssumeFps line from the previous results of the scripts but now I include it.

[Script]

LWLibavVideoSource("video.mp4")
Prefetch(1)
AssumeFPS("ntsc_film")
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(2)

[Runtime info]
Frames processed: 1501 (0 - 1500)
FPS (min | max | average): 30.67 | 97.94 | 51.43
Process memory usage (max): 79 MiB
Thread count: 9
CPU usage (average): 98.0%

Time (elapsed): 00:00:29.183


[Script]

LWLibavVideoSource("video.mp4")
AssumeFPS("ntsc_film")
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Prefetch(1)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(2)


[Runtime info]
Frames processed: 1501 (0 - 1500)
FPS (min | max | average): 7.330 | 389652 | 50.67
Process memory usage (max): 83 MiB
Thread count: 9
CPU usage (average): 96.4%

Time (elapsed): 00:00:29.625

[GMJCZP]

The issue has little to do with HDD speed, at least in my case. The file is small enough to fit in the Windows read cache and the file is located on an SSD. To me it seems that all source filters suffer from it, just tested encoding my analysis clip into a lossless HEVC file and used DGSource to decode it. GPU usage was ~100% all the time, in a normal situation it's only a few percent. Besides, the Vapoursynth Zopti works much better with multithreading with the exact same file and setup.

I see, that could be, a source filter problem.

[Boulder]

Out of interest, how does it work if you put something CPU intensive there, like SMDegrain with some heavier settings? Is the multithreaded version still very slow compared to a non-multithreaded run?

[GMJCZP]

Out of interest, how does it work if you put something CPU intensive there, like SMDegrain with some heavier settings? Is the multithreaded version still very slow compared to a non-multithreaded run?

What I have noticed is that the difference between Prefetch 1 and 2 is even more noticeable when the script is simpler, as you can see in the post #982. (https://forum.doom9.org/showthread.php?p=1943778#post1943778)

[Boulder]

What I have noticed is that the difference between Prefetch 1 and 2 is even more noticeable when the script is simpler, as you can see in the post #982. (https://forum.doom9.org/showthread.php?p=1943778#post1943778)

Just as I assumed. I've only seen the behaviour if the script is rather lightweight. Some MVTools-based stuff etc. already makes the multithreaded version faster, probably because the source filter will not get as frequent requests in general.

[FranceBB]

here is an actual snapshot build:
Avisynth+ 3.7.1 - 20210610 (https://drive.google.com/uc?export=download&id=1OLfGmsQro-fdrzMEi8AyfbHHLq3DkHsg) including XP and CUDA-aware builds

Wow!
This is like the fastest feature-request -> feature-implementation ever! :D
Thanks! ;)


x86 doesn't work on XP, though... :(

https://i.imgur.com/NfqrizJ.png

Here's Dependency Walker: https://i.imgur.com/ZyiXtkb.png

[real.finder]

maybe the time of winxp is over for new avs+, the avs+ rival vs even dropped win7 months ago

[pinterf]

What I have noticed is that the difference between Prefetch 1 and 2 is even more noticeable when the script is simpler, as you can see in the post #982. (https://forum.doom9.org/showthread.php?p=1943778#post1943778)
AssumeFPS does nothing with the clip, zero frame request or processing occurs, it just sets the frame rate in the VideoInfo structure.

With such a single script you are basically trying to test whether forcing multithreading to an intentionally single-threaded (MT_SERIALIZED) source filter works or not. No, it won't work, surely you will have zero speed gain even in the best case. But because of the internal timing conditions sometimes it will stuck in such a state that we'll encounter out-of-sequence frame request. This is when the linearized internal "Prefetch" is overtook by the consumer process's frame requests.

An example of this:
128 (P), 129 (P), 131 (C), 132 (C), 133(C), 130 (P)
where P=frame req by internal prefetch, C=by the external consumer (avsmeter, x264, etc))

Since the out-of-order frame generation is extremely slow, it took half _second_ on my i7 (!) and it was only around the 100th frame number, it will stuck in this state, it will never get out of this sequence (it is trying to escape but then falls back again periodically)

A possible solution can be that when such source filter detects serialized frame requests (usually this is true) and detects a jump (e.g. by two) it generates and caches inside that omitted frame.
E.g. in a frame number 5, 6, 7, 8, 10 it will still generate frame 9 and store it for future use.

MT_SERIALIZED mt mode is not something like MT_LINEARIZED (which does not exist)

I wonder what happens when you are using "Reverse" on this clip.

[pinterf]

Wow!
This is like the fastest feature-request -> feature-implementation ever! :D
Thanks! ;)


x86 doesn't work on XP, though... :(

https://i.imgur.com/NfqrizJ.png

Here's Dependency Walker: https://i.imgur.com/ZyiXtkb.png
I'm quite sure I built it with xp-on settings, anyway I'll check it for you later. This is all I can do, next Visual Studio will completely drop the feature.

[Boulder]

AssumeFPS does nothing with the clip, zero frame request or processing occurs, it just sets the frame rate in the VideoInfo structure.

With such a single script you are basically trying to test whether forcing multithreading to an intentionally single-threaded (MT_SERIALIZED) source filter works or not. No, it won't work, surely you will have zero speed gain even in the best case. But because of the internal timing conditions sometimes it will stuck in such a state that we'll encounter out-of-sequence frame request. This is when the linearized internal "Prefetch" is overtook by the consumer process's frame requests.

An example of this:
128 (P), 129 (P), 131 (C), 132 (C), 133(C), 130 (P)
where P=frame req by internal prefetch, C=by the external consumer (avsmeter, x264, etc))

Since the out-of-order frame generation is extremely slow, it took half _second_ on my i7 (!) and it was only around the 100th frame number, it will stuck in this state, it will never get out of this sequence (it is trying to escape but then falls back again periodically)

A possible solution can be that when such source filter detects serialized frame requests (usually this is true) and detects a jump (e.g. by two) it generates and caches inside that omitted frame.
E.g. in a frame number 5, 6, 7, 8, 10 it will still generate frame 9 and store it for future use.

MT_SERIALIZED mt mode is not something like MT_LINEARIZED (which does not exist)

I wonder what happens when you are using "Reverse" on this clip.

Is this behaviour something that comes from the basic Avisynth architecture, since it seems to affect all decoders? I somehow thought that Prefetch meant a read-ahead, so that the next x frames would be already decoded and in the cache whenever they are requested by the filters, and out of order requests would not be a problem.

[pinterf]

Yes, they are in the cache - optimally. Which is usually true, until it gets slowing down. When the request overtakes the prefetch process there is nothing in the cache.

[pinterf]

Avisynth is adaptive, it can detect the request pattern, it can assume and lock on a 2-4-6-8-10 pattern (delta = 2), even detects a 9-8-7-6-5 (delta = -1) negative pattern and is using the prefetch delta accordingly.

[Boulder]

Yes, they are in the cache - optimally. Which is usually true, until it gets slowing down. When the request overtakes the prefetch process there is nothing in the cache.

So a solution could be just increasing the number of frames in Prefetch? The faster the script is, the more frames you probably need to prefetch to avoid the issue?

[pinterf]

So a solution could be just increasing the number of frames in Prefetch? The faster the script is, the more frames you probably need to prefetch to avoid the issue?
This is where you have to test the effect of multiple Prefetches.
Prefetch(1,bignumber) after the source filter, and an ordinary Prefetch at the end of the script? These are only ideas if someone would like to test it.
I guess the actual benefits depend on the script. But I would not test the issue-originating simple script vs. Prefetch any further because it is just a zero-process shortcut between the consumer and an MT_SERIALIZED source.

[Boulder]

This is where you have to test the effect of multiple Prefetches.
Prefetch(1,bignumber) after the source filter, and an ordinary Prefetch at the end of the script? These are only ideas if someone would like to test it.
I guess the actual benefits depend on the script. But I would not test the issue-originating simple script vs. Prefetch any further because it is just a zero-process shortcut between the consumer and an MT_SERIALIZED source.

Thank you, I will try to find some time to test it (thanks to the football EC, not much time :D)
In my case, the script is slightly more complicated and does resizing + Expr stuff in the metrics calculation so it should be a nice case to try it on.

[pinterf]

Thank you, I will try to find some time to test it (thanks to the football EC, not much time :D)
OFF Yep, my son bought tickets for three matches, and yesterday when I did my usual run on an island in the city and saw the lurking fans and the police around the hotel when C.Ronaldo and the Portugals have their HQ during the EC.

[GMJCZP]

This is where you have to test the effect of multiple Prefetches.
Prefetch(1,bignumber) after the source filter, and an ordinary Prefetch at the end of the script? These are only ideas if someone would like to test it.
I guess the actual benefits depend on the script. But I would not test the issue-originating simple script vs. Prefetch any further because it is just a zero-process shortcut between the consumer and an MT_SERIALIZED source.

Continuing testing from post #982 (https://forum.doom9.org/showthread.php?p=1943778#post1943778), it is now with Prefetch (1,100):

Log file created with: AVSMeter 3.0.9.0 (x86)
Script file: Prueba.avs
Command line switches: -log

[OS/Hardware info]
Operating system: Windows 7 (x86) Service Pack 1.0 (Build 7601)

CPU: Pentium(R) Dual-Core CPU E5800 @ 3.20GHz / Wolfdale (Core 2 Duo) 2M
MMX, SSE, SSE2, SSE3, SSSE3
2 physical cores / 2 logical cores


[Avisynth info]
VersionString: AviSynth+ 3.7.0 (r3382, 3.7, i386)
VersionNumber: 2.60
File / Product version: 3.7.0.0 / 3.7.0.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\Windows\system32\avisynth.dll
Avisynth.dll time stamp: 2021-01-11, 20:46:40 (UTC)
PluginDir2_5 (HKLM, x86): C:\Program Files\AviSynth+\plugins
PluginDir+ (HKLM, x86): C:\Program Files\AviSynth+\plugins+


[Clip info]
Number of frames: 162
Length (hh:mm:ss.ms): 00:00:06.757
Frame width: 640
Frame height: 480
Framerate: 23.976 (24000/1001)
Colorspace: YUV420P12
Audio channels: n/a
Audio bits/sample: n/a
Audio sample rate: n/a
Audio samples: n/a


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.114 | 283384 | 9.439
Process memory usage (max): 174 MiB
Thread count: 9
CPU usage (average): 89.1%

Time (elapsed): 00:00:17.163


[Script]

LWLibavVideoSource("Sample2.mp4")
Prefetch(1,100)
AssumeFPS("ntsc_film")
Prefetch(2)

[FranceBB]

my son bought tickets for three matches, and yesterday when I did my usual run on an island in the city and saw the lurking fans and the police around the hotel when C.Ronaldo and the Portugals have their HQ during the EC.

Well, you might not be an overpaid football superstar, but you're one of the reason many of us are able to air stories and short documentaries about those, so you're some kind of superstar yourself eheheh
Keep up the good work and thank you for what you're doing for Avisynth (which I'm pretty sure will be used everywhere around the world to work on Euro2020 contents as well). ;)

This should cheer you up: https://i.imgur.com/XG8Axie.png

[Boulder]

OK, I tested running AVSMeter with this script. The analysis clip contains 200 frames, hence the first Prefetch call for the whole range.

With the first Prefetch enabled, everything ran smoothly until frame 196 when the processing seemed to pause. It finished by itself after a while but the average framerate went down considerably because of this.

With the first Prefetch call commented out, everything went the same way until frame 196, and then processing got completely stuck.

It would seem that at that frame 196, the deadlock kind of situation that you described happens.

orig = DGSource("c:\zopti\universe_s01e01.dgi").Prefetch(threads=1, frames=200)

b = -75/100.0 # optimize b = _n_/100.0 | -150..50 | b
c = 15/100.0 # optimize c = _n_/100.0 | -100..100 | c

downscaled_width = 1280
downscaled_height = 720

alternate = BicubicResize(orig, downscaled_width, downscaled_height, b=b, c=c).Lanczos4Resize(orig.width(),orig.height())

GMSD(alternate, orig, show=true)

# per frame logging (gmsd, time)
global delimiter = "; "
global resultFile = "perFrameResults.txt" # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write "stop" at the last frame to tell the optimizer that the script has finished
global frame_count = FrameCount()

WriteFileIf(resultFile, function() {
current_frame == frame_count-1
}, function() {
gmsd = 0.0
str = ""
for (i = 0, frame_count-1) {
value = propGetFloat("_PlaneGMSD", offset = -i)
gmsd = gmsd + value
if (i>0) { str = str + e"\n" }
str = str + string(current_frame - i) + delimiter + string(value) + delimiter + string(avstimer)
}
return str + e"\nstop " + string(gmsd)
}, append=false)

Prefetch(threads=24, frames=10)

[pinterf]

Wow!
This is like the fastest feature-request -> feature-implementation ever! :D
Thanks! ;)
x86 doesn't work on XP, though... :(

Please redownload, there was a glitch in cmake build settings. I hope it works now.

[kedautinh12]

Thanks

[GMJCZP]

OK, I tested running AVSMeter with this script. The analysis clip contains 200 frames, hence the first Prefetch call for the whole range.

With the first Prefetch enabled, everything ran smoothly until frame 196 when the processing seemed to pause. It finished by itself after a while but the average framerate went down considerably because of this.

With the first Prefetch call commented out, everything went the same way until frame 196, and then processing got completely stuck.

It would seem that at that frame 196, the deadlock kind of situation that you described happens.

orig = DGSource("c:\zopti\universe_s01e01.dgi").Prefetch(threads=1, frames=200)

b = -75/100.0 # optimize b = _n_/100.0 | -150..50 | b
c = 15/100.0 # optimize c = _n_/100.0 | -100..100 | c

downscaled_width = 1280
downscaled_height = 720

alternate = BicubicResize(orig, downscaled_width, downscaled_height, b=b, c=c).Lanczos4Resize(orig.width(),orig.height())

GMSD(alternate, orig, show=true)

# per frame logging (gmsd, time)
global delimiter = "; "
global resultFile = "perFrameResults.txt" # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write "stop" at the last frame to tell the optimizer that the script has finished
global frame_count = FrameCount()

WriteFileIf(resultFile, function() {
current_frame == frame_count-1
}, function() {
gmsd = 0.0
str = ""
for (i = 0, frame_count-1) {
value = propGetFloat("_PlaneGMSD", offset = -i)
gmsd = gmsd + value
if (i>0) { str = str + e"\n" }
str = str + string(current_frame - i) + delimiter + string(value) + delimiter + string(avstimer)
}
return str + e"\nstop " + string(gmsd)
}, append=false)

Prefetch(threads=24, frames=10)

Out of curiosity, can you recreate the test I did in post #982?

[kedautinh12]

Please redownload, there was a glitch in cmake build settings. I hope it works now.

File redownloaded, time modified same with old ver :confused: :confused: :confused:

[pinterf]

File redownloaded, time modified same with old ver :confused: :confused: :confused:
Files are different inside. Have you tried them on XP and they did not work? On VirtualBox or on a real PC?

[Boulder]

Out of curiosity, can you recreate the test I did in post #982?

The sample file seems to be gone. If you reupload it, I can give it a go.

[GMJCZP]

Corrected the link, Boulder.

[Boulder]

With Prefetch(1), "Script runtime is too short for meaningful measurements" :D
With Prefetch(2), definitely slower but that was expected per pinterf's explanation.


Frames processed: 162 (0 - 161)
FPS (min | max | average): 4.002 | 303033 | 152.2
Process memory usage (max): 88 MiB
Thread count: 46
CPU usage (average): 5.6%
Time (elapsed): 00:00:01.064

[GMJCZP]

With Prefetch(1), "Script runtime is too short for meaningful measurements" :D
With Prefetch(2), definitely slower but that was expected per pinterf's explanation.


Frames processed: 162 (0 - 161)
FPS (min | max | average): 4.002 | 303033 | 152.2
Process memory usage (max): 88 MiB
Thread count: 46
CPU usage (average): 5.6%
Time (elapsed): 00:00:01.064

In my case the difference is brutal. Will this have a solution?

[Boulder]

With a simple script like that, Avisynth multithreading is useless.

[GMJCZP]

With a simple script like that, Avisynth multithreading is useless.

I think this question becomes imperative, will it be possible to do this test in an equivalent way in Vapoursynth?

[Boulder]

Yes, it should be possible to test it in VS. The amount of threads can be set in the script.
I think the question is: what are you looking to multithread? With that script, you are just reading and decoding the source so multithreading it doesn't make any sense - setting the FPS just changes the clip properties. Some decoders can do internal multithreading, I think FFMS2 does at least in some cases.

[zorr]

Some decoders can do internal multithreading, I think FFMS2 does at least in some cases.

You can set the number of threads with parameter threads of FFVideoSource. On some formats that can be very useful, for example for some reason FFV1 decoding takes a huge number of threads for optimal performance.

https://i.postimg.cc/VNTxgWdw/ffv1-performance.png

You can see my ticket about it here (https://trac.ffmpeg.org/ticket/8694).

[GMJCZP]

All this is as if it takes 2 men to lift a large stone and they succeed, but if later we take the same men to move a pebble they both start arguing which of the two will do it, because four hands will get in the way to grab some of it. so small size, and all these the pebble is still on the ground waiting for the conclusion of the discussion.

[real.finder]

indeed you can't make the decode speed of source call faster with avs+ Prefetch, but the problem is MT_SERIALIZED not suitable for source call as said here https://forum.doom9.org/showthread.php?p=1944724#post1944724 maybe we should wait to see if pinterf add MT_LINEARIZED for this case

[GMJCZP]

I wonder what happens when you are using "Reverse" on this clip.

I have done new tests including Reverse(). I have left Performance data because at the beginning of the test with Prefetch 1 it was like stuck:

PREFETCH 1

[Script]

LWLibavVideoSource("Sample2.mp4")
AssumeFPS("ntsc_film")
Reverse()
Prefetch(1)


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.838 | 25.19 | 1.555
Process memory usage (max): 35 MiB
Thread count: 10
CPU usage (average): 89.3%

Time (elapsed): 00:01:44.162


[Performance data]
Frame Frames/sec Time/frame(ms) CPU(%) Threads Memory(MiB)
1 0.868 1152.605727 92.6 7 32
2 0.860 1163.214254 91.3 7 33
3 0.859 1163.807085 93.2 7 34
4 0.871 1147.891894 93.9 7 35
5 0.869 1150.160631 94.6 10 35
6 0.866 1154.832738 91.2 10 35
7 0.854 1170.400746 89.3 10 35
8 0.866 1154.755398 92.6 10 35
9 0.871 1148.168415 91.8 10 35
10 0.901 1109.587501 91.7 10 35
11 0.889 1124.645850 90.3 10 35
12 0.905 1104.950360 91.4 10 35
13 0.900 1111.412557 91.7 10 35
14 0.911 1097.943521 92.9 10 35
15 0.907 1102.785271 91.5 10 35
16 0.922 1084.399055 94.2 10 35
17 0.918 1089.068591 91.4 10 35
18 0.838 1193.176204 82.5 10 35
19 0.868 1152.698459 86.3 10 35
20 0.887 1127.143238 88.4 10 35
21 0.903 1107.059022 86.6 10 35
22 0.891 1121.785248 88.2 10 35
23 0.904 1106.133519 86.6 10 35
24 0.956 1046.068052 94.8 10 35
25 0.968 1033.054167 90.9 10 35
26 0.944 1059.561132 89.7 10 35
27 0.964 1037.391365 92.4 10 35
28 0.958 1043.598816 88.8 10 35
29 0.968 1033.550756 90.2 10 35
30 1.005 994.713861 89.1 10 35
31 0.984 1016.456976 92.3 10 35
32 0.986 1014.562660 86.2 10 35
33 0.992 1008.083794 89.2 10 35
34 1.019 980.900602 92.9 10 35
35 1.011 989.204094 91.3 10 35
36 1.028 972.877384 93.7 10 35
37 1.034 966.910498 93.5 10 35
38 1.019 981.714723 90.3 10 35
39 1.053 949.681402 91.0 10 35
40 1.032 969.054089 91.9 10 35
41 1.093 914.576683 93.2 10 35
42 1.057 946.248242 91.8 10 35
43 1.075 930.512359 91.5 10 35
44 1.082 924.285279 94.2 10 35
45 1.094 913.881815 93.1 10 35
46 1.090 917.400964 89.8 10 35
47 1.086 920.875541 89.0 10 35
48 1.098 910.738663 93.1 10 35
49 1.104 905.574080 92.4 10 35
50 1.128 886.395440 87.5 10 35
51 1.131 884.395587 93.9 10 35
52 1.158 863.869565 91.8 10 35
53 1.164 859.396950 89.3 10 35
54 1.136 880.626169 87.5 10 35
55 1.166 857.511328 91.8 10 35
56 1.189 841.396264 89.8 10 35
57 1.173 852.761928 91.8 10 35
58 1.227 815.170003 94.2 10 35
59 1.226 815.406132 87.5 10 35
60 1.238 807.885656 91.3 10 35
61 1.210 826.757565 86.8 10 35
62 1.248 801.225171 91.2 10 35
63 1.241 805.734323 91.3 10 35
64 1.288 776.559598 89.0 10 35
65 1.296 771.628287 92.9 10 35
66 1.316 759.677832 93.9 10 35
67 1.293 773.108110 90.8 10 35
68 1.287 776.797334 87.0 10 35
69 1.342 744.928782 91.7 10 35
70 1.394 717.433096 88.0 10 35
71 1.332 750.892134 87.5 10 35
72 1.292 774.140130 92.0 10 35
73 1.245 803.196160 82.4 10 35
74 1.386 721.294921 89.1 10 35
75 1.452 688.873649 93.3 10 35
76 1.470 680.095939 89.5 10 35
77 1.442 693.472318 92.2 10 35
78 1.479 676.164253 89.5 10 35
79 1.491 670.869132 91.9 10 35
80 1.524 656.118791 94.0 10 35
81 1.474 678.650105 89.5 8 35
82 1.624 615.826444 91.3 8 35
83 1.574 635.319155 92.7 8 35
84 1.612 620.326939 91.0 8 35
85 1.592 628.238126 92.7 8 35
86 1.658 602.955663 92.1 8 35
87 1.596 626.448099 86.6 8 35
88 1.703 587.105911 93.2 8 35
89 1.598 625.863260 86.3 8 35
90 1.809 552.660775 95.8 10 35
91 1.716 582.876526 93.2 10 35
92 1.783 560.723722 91.7 10 35
93 1.761 567.995272 94.4 10 35
94 1.777 562.802850 86.1 10 35
95 1.874 533.554610 92.9 10 35
96 1.911 523.241909 92.4 10 35
97 1.868 535.429382 91.4 10 35
98 1.901 526.158922 92.4 10 35
99 1.955 511.549740 92.4 10 35
100 1.983 504.195737 92.4 10 35
101 1.993 501.814429 91.5 10 35
102 2.061 485.252528 91.5 10 35
103 1.964 509.126098 86.5 10 35
104 2.116 472.494031 86.5 10 35
105 2.093 477.865189 92.6 10 35
106 2.223 449.822234 92.6 10 35
107 2.118 472.199231 89.0 10 35
108 2.132 469.079773 89.0 10 35
109 2.222 450.029769 86.4 10 35
110 2.256 443.280814 86.4 10 35
111 2.281 438.488376 86.0 10 35
112 2.394 417.628084 86.0 10 35
113 2.546 392.777096 92.3 10 35
114 2.466 405.587221 92.3 10 35
115 2.537 394.110014 92.2 10 35
116 2.553 391.752462 92.2 10 35
117 2.646 377.912209 89.8 10 35
118 2.704 369.879070 89.8 10 35
119 2.652 377.055371 91.7 10 35
120 2.836 352.659240 91.7 10 35
121 2.833 352.973294 93.3 10 35
122 2.744 364.379317 93.3 10 35
123 2.911 343.555287 87.0 10 35
124 3.111 321.448051 87.0 10 35
125 3.207 311.833707 90.0 10 35
126 3.220 310.557877 90.0 10 35
127 2.868 348.652453 84.9 7 35
128 3.472 288.004737 84.9 7 35
129 3.448 290.050482 85.1 7 35
130 3.486 286.857882 85.1 7 35
131 3.589 278.644158 87.5 7 35
132 3.765 265.623233 87.5 7 35
133 3.804 262.877840 91.2 7 35
134 3.994 250.397150 91.2 7 35
135 4.037 247.717520 91.2 7 35
136 4.243 235.660926 85.1 7 35
137 4.165 240.086035 85.1 7 35
138 4.397 227.417023 85.1 7 35
139 5.102 195.998610 86.9 7 35
140 5.007 199.715714 86.9 7 35
141 4.558 219.410535 86.9 7 35
142 5.479 182.525698 88.5 7 34
143 5.442 183.759807 88.5 7 34
144 6.010 166.388856 88.5 7 34
145 6.149 162.637108 86.4 7 35
146 5.470 182.807666 86.4 7 35
147 6.705 149.146230 86.4 7 35
148 6.671 149.894013 86.4 7 35
149 7.305 136.895232 80.0 7 34
150 8.430 118.623153 80.0 7 34
151 7.554 132.380941 80.0 7 34
152 8.378 119.355226 80.0 7 34
153 9.741 102.661154 80.0 7 34
154 10.705 93.416690 81.9 10 34
155 10.149 98.527661 81.9 10 34
156 11.597 86.229827 81.9 10 34
157 12.426 80.473730 81.9 10 34
158 13.111 76.273830 81.9 10 34
159 19.560 51.125105 81.9 10 34
160 12.225 81.800877 81.9 10 34
161 25.191 39.697282 75.8 10 34
162 24.674 40.528793 75.8 10 34

[GMJCZP]

Prefetch 2:

PREFETCH 2

[Script]

LWLibavVideoSource("Sample2.mp4")
AssumeFPS("ntsc_film")
Reverse()
Prefetch(2)


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.286 | 148439 | 1.959
Process memory usage (max): 40 MiB
Thread count: 10
CPU usage (average): 88.4%

Time (elapsed): 00:01:22.710


[Performance data]
Frame Frames/sec Time/frame(ms) CPU(%) Threads Memory(MiB)
1 0.861 1161.010346 88.7 8 32
2 0.867 1153.310189 95.9 8 34
3 816.667 1.224490 95.9 8 34
4 0.864 1156.834527 92.6 8 35
5 0.286 3490.868294 92.4 11 38
6 122.764 8.145730 92.4 11 38
7 714.958 1.398684 92.4 11 38
8 617.760 1.618752 92.4 11 38
9 0.892 1120.871662 91.7 11 38
10 0.296 3376.873175 90.3 11 39
11 736.060 1.358584 90.3 11 39
12 94461.086 0.010586 90.3 11 39
13 141691.641 0.007058 90.3 11 39
14 0.911 1097.751996 92.1 11 39
15 0.305 3275.588608 90.7 11 39
16 183.075 5.462246 90.7 11 39
17 97413.000 0.010266 90.7 11 39
18 119892.922 0.008341 90.7 11 39
19 0.938 1066.195277 94.2 11 39
20 0.312 3206.402249 90.2 11 39
21 143.399 6.973530 90.2 11 39
22 97413.000 0.010266 90.2 11 39
23 141691.641 0.007058 90.2 11 39
24 0.963 1038.887916 93.3 11 39
25 0.320 3121.036215 89.5 11 39
26 338.754 2.951993 89.5 11 39
27 107490.203 0.009303 89.5 11 39
28 100555.359 0.009945 89.5 11 39
29 0.974 1026.398543 92.4 9 39
30 0.981 1018.973359 93.2 9 39
31 0.493 2027.965302 88.1 9 39
32 635.000 1.574803 88.1 9 39
33 124688.633 0.008020 88.1 9 39
34 0.992 1007.803423 93.0 9 39
35 0.343 2919.189075 91.2 8 39
36 97.209 10.287064 91.2 8 39
37 13732.229 0.072821 91.2 8 39
38 111329.148 0.008982 91.2 8 39
39 1.074 931.320740 92.5 8 39
40 0.348 2869.875234 92.1 8 39
41 491.055 2.036433 92.1 8 39
42 103907.195 0.009624 92.1 8 39
43 129884.000 0.007699 92.1 8 39
44 1.061 942.834198 92.5 8 39
45 0.364 2748.311957 92.9 11 39
46 192.766 5.187642 92.9 11 39
47 21952.225 0.045553 92.9 11 39
48 103907.203 0.009624 92.9 11 39
49 1.106 904.053450 93.1 11 39
50 0.375 2663.956327 90.4 11 39
51 772.161 1.295066 90.4 11 39
52 107490.203 0.009303 90.4 11 39
53 141691.641 0.007058 90.4 11 39
54 1.149 870.302840 91.1 11 39
55 1.180 847.475730 90.7 11 39
56 0.399 2505.824395 91.6 11 39
57 81.946 12.203197 91.6 11 39
58 19361.590 0.051649 91.6 11 39
59 100555.359 0.009945 91.6 11 39
60 1.219 820.317202 92.5 11 39
61 1.254 797.715360 86.3 11 39
62 1.266 789.965491 94.1 11 39
63 0.420 2380.945022 88.2 11 40
64 195.584 5.112895 88.2 11 40
65 100555.352 0.009945 88.2 11 40
66 141691.641 0.007058 88.2 11 40
67 1.325 754.595430 92.9 11 39
68 0.443 2258.927492 89.7 11 39
69 90.059 11.103818 89.7 11 39
70 3853.172 0.259526 89.7 11 39
71 82032.000 0.012190 89.7 11 39
72 1.393 717.841176 88.3 11 39
73 0.468 2136.660046 89.8 11 39
74 188.374 5.308583 89.8 11 39
75 19007.414 0.052611 89.8 11 39
76 115452.445 0.008662 89.8 11 39
77 1.482 674.579508 95.3 11 39
78 1.516 659.717849 92.9 11 39
79 1.508 663.160949 91.9 11 39
80 764.398 1.308219 91.9 11 39
81 1.515 659.948187 89.3 11 39
82 0.789 1267.128407 86.6 11 39
83 745.389 1.341582 86.6 11 39
84 115452.445 0.008662 86.6 11 39
85 1.595 626.986087 88.8 11 39
86 0.563 1777.691316 89.0 11 39
87 191.605 5.219080 89.0 11 39
88 3471.287 0.288078 89.0 11 39
89 97413.000 0.010266 89.0 11 39
90 1.725 579.650571 89.2 11 39
91 0.588 1700.497554 88.1 11 39
92 76.892 13.005194 88.1 11 39
93 10976.112 0.091107 88.1 11 39
94 86589.336 0.011549 88.1 11 39
95 1.829 546.765780 80.6 11 39
96 0.637 1568.717683 87.6 11 39
97 835.267 1.197222 87.6 11 39
98 27344.000 0.036571 87.6 11 39
99 119892.922 0.008341 87.6 11 39
100 1.924 519.848494 84.8 11 39
101 0.680 1470.004651 86.2 8 39
102 109.885 9.100428 86.2 8 39
103 69271.469 0.014436 86.2 8 39
104 148438.859 0.006737 86.2 8 39
105 2.095 477.433728 86.2 8 39
106 1.073 931.584788 85.7 8 39
107 316.436 3.160192 85.7 8 39
108 103907.203 0.009624 85.7 8 39
109 2.186 457.430626 85.7 8 39
110 0.784 1276.154071 88.4 8 39
111 75.646 13.219488 88.4 8 39
112 16669.604 0.059989 88.4 8 39
113 5101.827 0.196008 88.4 8 39
114 2.484 402.645518 88.4 8 39
115 2.645 378.039907 86.0 8 39
116 2.525 395.996926 86.0 8 39
117 904.327 1.105794 86.0 8 39
118 2.667 374.976890 85.0 8 39
119 0.907 1101.967568 86.6 11 39
120 74.709 13.385341 86.6 11 39
121 1101.490 0.907861 86.6 11 39
122 403.889 2.475927 86.6 11 39
123 3.240 308.685383 86.6 11 39
124 0.993 1007.434213 85.9 11 39
125 56.075 17.833220 85.9 11 39
126 12723.331 0.078596 85.9 11 39
127 129884.000 0.007699 85.9 11 39
128 3.696 270.590814 85.9 11 39
129 3.558 281.051418 91.9 11 39
130 329.655 3.033476 91.9 11 39
131 1.262 792.377555 89.2 11 39
132 503.345 1.986709 89.2 11 39
133 49479.617 0.020210 89.2 11 39
134 76029.656 0.013153 89.2 11 39
135 4.311 231.941248 89.2 11 39
136 1.431 698.827100 84.7 11 39
137 555.951 1.798720 84.7 11 39
138 103907.203 0.009624 84.7 11 39
139 119892.914 0.008341 84.7 11 39
140 5.395 185.341348 84.7 11 39
141 1.746 572.697864 88.8 11 39
142 534.502 1.870900 88.8 11 39
143 84249.078 0.011870 88.8 11 39
144 129884.000 0.007699 88.8 11 39
145 5.965 167.638050 88.8 11 39
146 6.192 161.508859 88.8 11 39
147 683.600 1.462844 88.8 11 39
148 6.627 150.895544 88.8 11 39
149 2.640 378.835796 79.1 10 39
150 84.395 11.849034 79.1 10 39
151 70845.820 0.014115 79.1 10 39
152 124688.641 0.008020 79.1 10 39
153 9.759 102.472852 79.1 10 39
154 10.885 91.868513 79.1 10 39
155 804.443 1.243096 79.1 10 39
156 10.730 93.195340 79.1 10 39
157 4.468 223.838515 77.9 10 38
158 586.053 1.706330 77.9 10 38
159 241.383 4.142799 77.9 10 38
160 27832.285 0.035929 77.9 10 38
161 24.541 40.748541 77.9 10 38
162 28.543 35.035429 77.9 10 38

[pinterf]

I have done new tests including Reverse(). I have left Performance data because at the beginning of the test with Prefetch 1 it was like stuck:

PREFETCH 1

[Script]

LWLibavVideoSource("Sample2.mp4")
AssumeFPS("ntsc_film")
Reverse()
Prefetch(1)


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.838 | 25.19 | 1.555
Process memory usage (max): 35 MiB
Thread count: 10
CPU usage (average): 89.3%

Time (elapsed): 00:01:44.162


[Performance data]
Frame Frames/sec Time/frame(ms) CPU(%) Threads Memory(MiB)
1 0.868 1152.605727 92.6 7 32
2 0.860 1163.214254 91.3 7 33
3 0.859 1163.807085 93.2 7 34
4 0.871 1147.891894 93.9 7 35
5 0.869 1150.160631 94.6 10 35
6 0.866 1154.832738 91.2 10 35
7 0.854 1170.400746 89.3 10 35
8 0.866 1154.755398 92.6 10 35
9 0.871 1148.168415 91.8 10 35
10 0.901 1109.587501 91.7 10 35
11 0.889 1124.645850 90.3 10 35
12 0.905 1104.950360 91.4 10 35
13 0.900 1111.412557 91.7 10 35
14 0.911 1097.943521 92.9 10 35
15 0.907 1102.785271 91.5 10 35
16 0.922 1084.399055 94.2 10 35
17 0.918 1089.068591 91.4 10 35
18 0.838 1193.176204 82.5 10 35
19 0.868 1152.698459 86.3 10 35
20 0.887 1127.143238 88.4 10 35
21 0.903 1107.059022 86.6 10 35
22 0.891 1121.785248 88.2 10 35
23 0.904 1106.133519 86.6 10 35
24 0.956 1046.068052 94.8 10 35
25 0.968 1033.054167 90.9 10 35
26 0.944 1059.561132 89.7 10 35
27 0.964 1037.391365 92.4 10 35
28 0.958 1043.598816 88.8 10 35
29 0.968 1033.550756 90.2 10 35
30 1.005 994.713861 89.1 10 35
31 0.984 1016.456976 92.3 10 35
32 0.986 1014.562660 86.2 10 35
33 0.992 1008.083794 89.2 10 35
34 1.019 980.900602 92.9 10 35
35 1.011 989.204094 91.3 10 35
36 1.028 972.877384 93.7 10 35
37 1.034 966.910498 93.5 10 35
38 1.019 981.714723 90.3 10 35
39 1.053 949.681402 91.0 10 35
40 1.032 969.054089 91.9 10 35
41 1.093 914.576683 93.2 10 35
42 1.057 946.248242 91.8 10 35
43 1.075 930.512359 91.5 10 35
44 1.082 924.285279 94.2 10 35
45 1.094 913.881815 93.1 10 35
46 1.090 917.400964 89.8 10 35
47 1.086 920.875541 89.0 10 35
48 1.098 910.738663 93.1 10 35
49 1.104 905.574080 92.4 10 35
50 1.128 886.395440 87.5 10 35
51 1.131 884.395587 93.9 10 35
52 1.158 863.869565 91.8 10 35
53 1.164 859.396950 89.3 10 35
54 1.136 880.626169 87.5 10 35
55 1.166 857.511328 91.8 10 35
56 1.189 841.396264 89.8 10 35
57 1.173 852.761928 91.8 10 35
58 1.227 815.170003 94.2 10 35
59 1.226 815.406132 87.5 10 35
60 1.238 807.885656 91.3 10 35
61 1.210 826.757565 86.8 10 35
62 1.248 801.225171 91.2 10 35
63 1.241 805.734323 91.3 10 35
64 1.288 776.559598 89.0 10 35
65 1.296 771.628287 92.9 10 35
66 1.316 759.677832 93.9 10 35
67 1.293 773.108110 90.8 10 35
68 1.287 776.797334 87.0 10 35
69 1.342 744.928782 91.7 10 35
70 1.394 717.433096 88.0 10 35
71 1.332 750.892134 87.5 10 35
72 1.292 774.140130 92.0 10 35
73 1.245 803.196160 82.4 10 35
74 1.386 721.294921 89.1 10 35
75 1.452 688.873649 93.3 10 35
76 1.470 680.095939 89.5 10 35
77 1.442 693.472318 92.2 10 35
78 1.479 676.164253 89.5 10 35
79 1.491 670.869132 91.9 10 35
80 1.524 656.118791 94.0 10 35
81 1.474 678.650105 89.5 8 35
82 1.624 615.826444 91.3 8 35
83 1.574 635.319155 92.7 8 35
84 1.612 620.326939 91.0 8 35
85 1.592 628.238126 92.7 8 35
86 1.658 602.955663 92.1 8 35
87 1.596 626.448099 86.6 8 35
88 1.703 587.105911 93.2 8 35
89 1.598 625.863260 86.3 8 35
90 1.809 552.660775 95.8 10 35
91 1.716 582.876526 93.2 10 35
92 1.783 560.723722 91.7 10 35
93 1.761 567.995272 94.4 10 35
94 1.777 562.802850 86.1 10 35
95 1.874 533.554610 92.9 10 35
96 1.911 523.241909 92.4 10 35
97 1.868 535.429382 91.4 10 35
98 1.901 526.158922 92.4 10 35
99 1.955 511.549740 92.4 10 35
100 1.983 504.195737 92.4 10 35
101 1.993 501.814429 91.5 10 35
102 2.061 485.252528 91.5 10 35
103 1.964 509.126098 86.5 10 35
104 2.116 472.494031 86.5 10 35
105 2.093 477.865189 92.6 10 35
106 2.223 449.822234 92.6 10 35
107 2.118 472.199231 89.0 10 35
108 2.132 469.079773 89.0 10 35
109 2.222 450.029769 86.4 10 35
110 2.256 443.280814 86.4 10 35
111 2.281 438.488376 86.0 10 35
112 2.394 417.628084 86.0 10 35
113 2.546 392.777096 92.3 10 35
114 2.466 405.587221 92.3 10 35
115 2.537 394.110014 92.2 10 35
116 2.553 391.752462 92.2 10 35
117 2.646 377.912209 89.8 10 35
118 2.704 369.879070 89.8 10 35
119 2.652 377.055371 91.7 10 35
120 2.836 352.659240 91.7 10 35
121 2.833 352.973294 93.3 10 35
122 2.744 364.379317 93.3 10 35
123 2.911 343.555287 87.0 10 35
124 3.111 321.448051 87.0 10 35
125 3.207 311.833707 90.0 10 35
126 3.220 310.557877 90.0 10 35
127 2.868 348.652453 84.9 7 35
128 3.472 288.004737 84.9 7 35
129 3.448 290.050482 85.1 7 35
130 3.486 286.857882 85.1 7 35
131 3.589 278.644158 87.5 7 35
132 3.765 265.623233 87.5 7 35
133 3.804 262.877840 91.2 7 35
134 3.994 250.397150 91.2 7 35
135 4.037 247.717520 91.2 7 35
136 4.243 235.660926 85.1 7 35
137 4.165 240.086035 85.1 7 35
138 4.397 227.417023 85.1 7 35
139 5.102 195.998610 86.9 7 35
140 5.007 199.715714 86.9 7 35
141 4.558 219.410535 86.9 7 35
142 5.479 182.525698 88.5 7 34
143 5.442 183.759807 88.5 7 34
144 6.010 166.388856 88.5 7 34
145 6.149 162.637108 86.4 7 35
146 5.470 182.807666 86.4 7 35
147 6.705 149.146230 86.4 7 35
148 6.671 149.894013 86.4 7 35
149 7.305 136.895232 80.0 7 34
150 8.430 118.623153 80.0 7 34
151 7.554 132.380941 80.0 7 34
152 8.378 119.355226 80.0 7 34
153 9.741 102.661154 80.0 7 34
154 10.705 93.416690 81.9 10 34
155 10.149 98.527661 81.9 10 34
156 11.597 86.229827 81.9 10 34
157 12.426 80.473730 81.9 10 34
158 13.111 76.273830 81.9 10 34
159 19.560 51.125105 81.9 10 34
160 12.225 81.800877 81.9 10 34
161 25.191 39.697282 75.8 10 34
162 24.674 40.528793 75.8 10 34

As expected. In order the get the Nth frame it seems that all previous (N-1) frames must be decoded again from the beginning. As it is getting closer to the beginning of the video it gets quicker and quicker. What about testing with a video file where _all_ frames are I frame? (Did not investigate your specific test file how it is encoded)

[GMJCZP]

Here is the Mediainfo text:

General
Complete name : video.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : iso4 (iso4/hvc1/iso6)
File size : 223 MiB
Duration : 21 min 2 s
Overall bit rate mode : Variable
Overall bit rate : 1 484 kb/s
Encoded date : UTC 2016-07-18 23:03:09
Tagged date : UTC 2016-07-18 23:03:09

VÃ*deo
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Format Range@L3@Main
Codec ID : hvc1
Codec ID/Info : High Efficiency Video Coding
Duration : 21 min 2 s
Bit rate : 1 384 kb/s
Maximum bit rate : 3 199 kb/s
Width : 640 pÃ*xeles
Height : 480 pÃ*xeles
Display aspect ratio : 4:3
Frame rate mode : Constante
Frame rate : 25,000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 12 bits
Bits/(Pixel*Frame) : 0.180
Stream size : 208 MiB (93%)
Writing library : x265 2.0+4-43ca544799c2:[Windows][MSVC 1900][64 bit] 12bit
Encoding settings : wpp / ctu=64 / min-cu-size=8 / max-tu-size=32 / tu-intra-depth=1 / tu-inter-depth=1 / me=1 / subme=2 / merange=57 / no-rect / no-amp / max-merge=2 / temporal-mvp / no-early-skip / rskip / rdpenalty=0 / no-tskip / no-tskip-fast / strong-intra-smoothing / no-lossless / no-cu-lossless / no-constrained-intra / no-fast-intra / open-gop / no-temporal-layers / interlace=0 / keyint=250 / min-keyint=25 / scenecut=40 / rc-lookahead=20 / lookahead-slices=0 / bframes=4 / bframe-bias=0 / b-adapt=2 / ref=3 / limit-refs=3 / no-limit-modes / weightp / no-weightb / aq-mode=1 / qg-size=32 / aq-strength=1.00 / cbqpoffs=0 / crqpoffs=0 / rd=3 / psy-rd=2.00 / rdoq-level=0 / psy-rdoq=0.00 / no-rd-refine / signhide / deblock=0:0 / sao / no-sao-non-deblock / b-pyramid / cutree / no-intra-refresh / rc=crf / crf=21.0 / qcomp=0.60 / qpmin=0 / qpmax=51 / qpstep=4 / ipratio=1.40 / pbratio=1.30
Encoded date : UTC 2016-07-18 23:03:09
Tagged date : UTC 2016-07-18 23:03:14
Codec configuration box : hvcC

[Dogway]

I found something strange. Using the next expression in 16-bit to convert to TV levels yields a byte white level of 234:
"x ymax ymin - range_max / * ymin +"
That would correspond to (n*256):
"x 60160 4096 - 65535 / * 4096 +"

This is fixed if I multiply 235*257 (= 60395) instead to get the correct ymax 16-bit value.
Shouldn't this be fixed internally? I can set scale_inputs to "allf" and it fixes, but I'm not sure that's the original intention of the option (scale expression to 8-bit). Also by doing so I lose the option to use "FloatUV" for 32-bit chroma, since there's no "FloatUVf".

[GMJCZP]

pinterf, due to technical problems I will not be able to continue, at least for now, in Doom9, so I ask you please, continue looking for a solution to the use of Prefetch. Thank you and I hope to return soon, greetings to all.

[zorr]

This page (http://avisynth.nl/index.php/Internal_functions#Functions_for_frame_properties) mentions that frame property value can be a clip reference (PClip). When I try that with

final = propSet(final, "test", function[store_this_clip](clip c) { return store_this_clip} )

I get message "propAdd: Clip frame properties not yet supported". Is that true or am I doing something wrong? I also tried other variations, storing clips into an array etc.

Having clips as frame properties would be very useful, you could for example return debug clips along with the main clip. This would be an elegant solution for returning a mask visualization clip from Delta Restore (https://forum.doom9.org/showthread.php?p=1944267#post1944267).

[pinterf]

pinterf, due to technical problems I will not be able to continue, at least for now, in Doom9, so I ask you please, continue looking for a solution to the use of Prefetch. Thank you and I hope to return soon, greetings to all.
Source filter problem, which has a 10x 100x 1000x time penalty if requested out of order frames. Or do not use Prefetch directly for a serialized filter, or use its internal multithreading switch, if any. MT_SERIALIZED was never meant to guarantee linear access.

[pinterf]

This page (http://avisynth.nl/index.php/Internal_functions#Functions_for_frame_properties) mentions that frame property value can be a clip reference (PClip). When I try that with

final = propSet(final, "test", function[store_this_clip](clip c) { return store_this_clip} )

I get message "propAdd: Clip frame properties not yet supported". Is that true or am I doing something wrong? I also tried other variations, storing clips into an array etc.

Having clips as frame properties would be very useful, you could for example return debug clips along with the main clip. This would be an elegant solution for returning a mask visualization clip from Delta Restore (https://forum.doom9.org/showthread.php?p=1944267#post1944267).
I never tried and debugged how clips behave inside frame properties, so it is disabled at the moment. Though I can check it how must task is enabling it as-is. Worst case: they do not work out of box.

[real.finder]

I found something strange. Using the next expression in 16-bit to convert to TV levels yields a byte white level of 234:
"x ymax ymin - range_max / * ymin +"
That would correspond to (n*256):
"x 60160 4096 - 65535 / * 4096 +"

This is fixed if I multiply 235*257 (= 60395) instead to get the correct ymax 16-bit value.
Shouldn't this be fixed internally? I can set scale_inputs to "allf" and it fixes, but I'm not sure that's the original intention of the option (scale expression to 8-bit). Also by doing so I lose the option to use "FloatUV" for 32-bit chroma, since there's no "FloatUVf".

16bit sources will be limited to 4096 - 60160 luma and 4096 - 61440 chroma as said here https://forum.doom9.org/showthread.php?t=181857

convertbits(16)
ScriptClip("""
Subtitle(String(expr("ymax").AverageLuma()))
""")

show 60160 as it should

[pinterf]

New build, primarily for zorr, propAdd allows Clip type. Not tested at all, I had time only for adding the possibility.
Avisynth 3.7.1 20210618 test5
https://drive.google.com/uc?export=download&id=1epFQMNjU3B7rRY-btSK8oN-xFsvje-G_

[Dogway]

16bit sources will be limited to 4096 - 60160 luma and 4096 - 61440 chroma as said here https://forum.doom9.org/showthread.php?t=181857

convertbits(16)
ScriptClip("""
Subtitle(String(expr("ymax").AverageLuma()))
""")

show 60160 as it should

For conversions you need to full scale stretch if working in PC levels. Read here (https://forum.doom9.org/showthread.php?p=1685118#post1685118).

In any case, I don't care anymore what avisynth+ does, using auto-scaling is very slow, "allf" is about 30% slower than "none" because the constants (ymax, range_max, etc) are computed at runtime. So I'm bulding a look up table which will speed things up quite a bit. Thought pinterf wanted to fix it but he doesn't seem very interested on the topic from lack of feedback.

# 8-bit 10-bit 12-bit 14-bit 16-bit 32-bit
range_min = Select (bits, [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0])
ymin = Select (bits, [ 16, 16], [ 64, 80], [ 256, 272], [ 1024, 1040], [ 4096, 4112], [ 16/255., 16/255.])
cmin = Select (bits, [ 16, 16], [ 64, 80], [ 256, 272], [ 1024, 1040], [ 4096, 4112], [-112/255.,-112/255.])
range_half = Select (bits, [128,128], [ 512, 640], [2048,2176], [ 8192, 8320], [32768,32896], [ 128/255., 128/255.])
yrange = Select (bits, [219,219], [ 876,1095], [3504,3723], [14016,14235], [56064,56283], [ 219/255., 219/255.])
crange = Select (bits, [224,224], [ 896,1120], [3584,3808], [14336,14560], [57344,57568], [ 224/255., 224/255.])
ymax = Select (bits, [235,235], [ 940,1175], [3760,3995], [15040,15275], [60160,60395], [ 235/255., 235/255.])
cmax = Select (bits, [240,240], [ 960,1200], [3840,4080], [15360,15600], [61440,61680], [ 107/255., 107/255.])
range_max = Select (bits, [255,255], [1020,1023], [4080,4095], [16320,16383], [65280,65535], [1.0,1.0])
range_size = Select (bits, [256,256], [1023,1023], [4096,4096], [16384,16384], [65536,65536], [1.0,1.0])

[pinterf]

Thought pinterf wanted to fix it but he doesn't seem very interested on the topic from lack of feedback.

Feedback
I wonder why do you think I'm behind the computer and forums 24/7 and answer _all_ questions immediately at most in a couple of days? Not having time for this hobby (even reading forums) for a couple of _weeks_ is perfectly acceptable I think. Patience, please, I'm not a live L3 support line.

[tormento]

Thought pinterf wanted to fix it but he doesn't seem very interested on the topic from lack of feedback.
FranceBB is the uber color space master :)

[real.finder]

For conversions you need to full scale stretch if working in PC levels. Read here (https://forum.doom9.org/showthread.php?p=1685118#post1685118).

In any case, I don't care anymore what avisynth+ does, using auto-scaling is very slow, "allf" is about 30% slower than "none" because the constants (ymax, range_max, etc) are computed at runtime. So I'm bulding a look up table which will speed things up quite a bit. Thought pinterf wanted to fix it but he doesn't seem very interested on the topic from lack of feedback.

# 8-bit 10-bit 12-bit 14-bit 16-bit 32-bit
range_min = Select (bits, [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0])
ymin = Select (bits, [ 16, 16], [ 64, 80], [ 256, 272], [ 1024, 1040], [ 4096, 4112], [ 16/255., 16/255.])
cmin = Select (bits, [ 16, 16], [ 64, 80], [ 256, 272], [ 1024, 1040], [ 4096, 4112], [-112/255.,-112/255.])
range_half = Select (bits, [128,128], [ 512, 640], [2048,2176], [ 8192, 8320], [32768,32896], [ 128/255., 128/255.])
yrange = Select (bits, [219,219], [ 876,1095], [3504,3723], [14016,14235], [56064,56283], [ 219/255., 219/255.])
crange = Select (bits, [224,224], [ 896,1120], [3584,3808], [14336,14560], [57344,57568], [ 224/255., 224/255.])
ymax = Select (bits, [235,235], [ 940,1175], [3760,3995], [15040,15275], [60160,60395], [ 235/255., 235/255.])
cmax = Select (bits, [240,240], [ 960,1200], [3840,4080], [15360,15600], [61440,61680], [ 107/255., 107/255.])
range_max = Select (bits, [255,255], [1020,1023], [4080,4095], [16320,16383], [65280,65535], [1.0,1.0])
range_size = Select (bits, [256,256], [1023,1023], [4096,4096], [16384,16384], [65536,65536], [1.0,1.0])

how about "x 235 scalef 16 scalef - range_max / * 16 scalef +" ?

maybe ymax problem with full range can be fixed with new "nonef"

edit: btw:-

"allf" is about 30% slower than "none" because the constants (ymax, range_max, etc) are computed at runtime

this is not how it works, when using scale_inputs (auto-scaling) the expression will be in 8bit (unless you use http://avisynth.nl/index.php/Expr#Keywords_for_modifying_base_bit_depth) so ymax will be 235.0 in this case, scale_inputs as it name said it do scale (the clip/s) to 8bit (or http://avisynth.nl/index.php/Expr#Keywords_for_modifying_base_bit_depth) range but with float then the output will be rescale to original bitdepth

[Dogway]

Yes, "{n} scalef" is almost as fast as "none". The problem is we are still using 8-bit constants which is a bit misleading. I thought ymax, ymin... constants were there to replace 8-bit expression syntax and make it cleaner.


Here's the function if you want to play with it, if everything goes according I will add it to ExTools soon, forget about 24-bit for the moment.

function ex_dlut(string "str", int "bits", bool "tv_range") {

str = Default(str, "")
bits = Default(bits, 8)
tv = Default(tv_range, true)

bitd =
\ (bits == 8 ) ? 0
\ : (bits == 10 ) ? 1
\ : (bits == 12 ) ? 2
\ : (bits == 14 ) ? 3
\ : (bits == 16 ) ? 4
\ : (bits == 24 ) ? 5
\ : (bits == 32 ) ? 6
\ : Assert (false, "Unsupported bit depth.")


# 8-bit UINT 10-bit UINT 12-bit UINT 14-bit UINT 16-bit UINT 24-bit UINT 32-bit float
range_min = Select (bitd, [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0., 0.])
ymin = Select (bitd, [ 16, 16], [ 64, 64], [ 256, 257], [ 1024, 1028], [ 4096, 4112], [ 1048576, 1052672], [ 16/255., 16/255.])
cmin = Select (bitd, [ 16, 16], [ 64, 64], [ 256, 257], [ 1024, 1028], [ 4096, 4112], [ 1048576, 1052672], [-112/255.,-112/255.])
range_half = Select (bitd, [128,128], [ 512, 514], [2048,2056], [ 8192, 8224], [32768,32896], [ 8388608, 8421376], [ 128/255., 128/255.])
yrange = Select (bitd, [219,219], [ 876, 879], [3504,3517], [14016,14070], [56064,56283], [14352384,14408448], [ 219/255., 219/255.])
crange = Select (bitd, [224,224], [ 896, 899], [3584,3597], [14336,14391], [57344,57568], [14680064,14737408], [ 224/255., 224/255.])
ymax = Select (bitd, [235,235], [ 940, 943], [3760,3774], [15040,15098], [60160,60395], [15400960,15461120], [ 235/255., 235/255.])
cmax = Select (bitd, [240,240], [ 960, 963], [3840,3854], [15360,15419], [61440,61680], [15728640,15790080], [ 107/255., 107/255.])
range_max = Select (bitd, [255,255], [1020,1023], [4080,4095], [16320,16383], [65280,65535], [16711680,16776960], [ 1., 1.])
range_size = Select (bitd, [256,256], [1024,1024], [4096,4096], [16384,16384], [65536,65536], [16777216,16777216], [ 1., 1.])

tv = tv ? 0 : 1
str = ReplaceStr(str, "ymax ymin -", string(yrange[tv]))
str = ReplaceStr(str, "cmax cmin -", string(crange[tv]))
str = ReplaceStr(str, "range_min", string(range_min[tv]))
str = ReplaceStr(str, "ymin", string(ymin[tv]))
str = ReplaceStr(str, "cmin", string(cmin[tv]))
str = ReplaceStr(str, "range_half", string(range_half[tv]))
str = ReplaceStr(str, "ymax", string(ymax[tv]))
str = ReplaceStr(str, "cmax", string(cmax[tv]))
str = ReplaceStr(str, "range_max", string(range_max[tv]))
str = ReplaceStr(str, "range_size", string(range_size[tv]))
str = ReplaceStr(str, "}", "} "+string(pow(2,bits-8))+" *")

return str }

FranceBB is the uber color space master :)
@tormento: wait (https://www.artstation.com/artwork/KrzrQR)and (https://forums.libretro.com/t/dogways-grading-shader-slang/27148)see (https://github.com/Dogway/Avisynth-Scripts/blob/master/TransformsPack.v1.0.RC17.avsi):rolleyes:

@real.finder: so what do you propose (correct method), to use scalef/scaleb all along the inputs? not only the code is dirtier but also slower*. What was the constants use case for then...

417fps (*about same if using 219 scalef)
Expr("x 235 scalef 16 scalef - range_max / * 16 scalef +",scale_inputs="none")
432fps
Expr(ex_dlut("x ymax ymin - range_max / * ymin +",16,false))