AviSynth+ thread Vol.2

[jpsdr]

Unfortunately i don't have time right now, can quick test could be this :

static void Sobel_16(const unsigned char *psrc,unsigned char *pdst,const int32_t src_pitch, const int32_t dst_pitch,
const int32_t src_height,int32_t dst_row_size, int32_t thresh,uint8_t bit_pixel)
{
const int32_t i = (dst_row_size + 3) >> 2;
const int32_t i0 = (dst_row_size + 3-2) >> 2;

dst_row_size >>= 1;
thresh <<= (bit_pixel-8);

if (aWarpSharp_Enable_AVX)
{
for (int32_t y=0; y<src_height; y++)
{
uint16_t *dst=(uint16_t *)pdst;

if (y==0) JPSDR_Sobel_16_AVX(psrc+2,pdst+2,src_pitch,y,src_height,i0,thresh);
else
{
if (y==src_height-1) JPSDR_Sobel_16_AVX(psrc,pdst,src_pitch,y,src_height,i0,thresh);
else JPSDR_Sobel_16_AVX(psrc,pdst,src_pitch,y,src_height,i,thresh);
}
dst[0]=dst[1];
dst[dst_row_size-1]=dst[dst_row_size-2];

psrc += src_pitch;
pdst += dst_pitch;
}
}
else
{
for (int32_t y=0; y<src_height; y++)
{
uint16_t *dst=(uint16_t *)pdst;

if (y==0) JPSDR_Sobel_16_SSE2(psrc+2,pdst+2,src_pitch,y,src_height,i0,thresh);
else
{
if (y==src_height-1) JPSDR_Sobel_16_SSE2(psrc,pdst,src_pitch,y,src_height,i0,thresh);
else JPSDR_Sobel_16_SSE2(psrc,pdst,src_pitch,y,src_height,i,thresh);
}
dst[0]=dst[1];
dst[dst_row_size-1]=dst[dst_row_size-2];

psrc += src_pitch;
pdst += dst_pitch;
}
}
}


You have to put threads=1 in aWarpSharp2 call.

The

dst[0]=dst[1];
dst[dst_row_size-1]=dst[dst_row_size-2];

may fill the missing pixels.

[GMJCZP]

Thanks pinterf.

With frames = 2 it improved noticeably but still doesn't even match Prefetch 1:

Log file created with: AVSMeter 3.0.9.0 (x86)
Script file: Prueba.avs
Command line switches: -log

[OS/Hardware info]
Operating system: Windows 7 (x86) Service Pack 1.0 (Build 7601)

CPU: Pentium(R) Dual-Core CPU E5800 @ 3.20GHz / Wolfdale (Core 2 Duo) 2M
MMX, SSE, SSE2, SSE3, SSSE3
2 physical cores / 2 logical cores


[Avisynth info]
VersionString: AviSynth+ 3.7.0 (r3382, 3.7, i386)
VersionNumber: 2.60
File / Product version: 3.7.0.0 / 3.7.0.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\Windows\system32\avisynth.dll
Avisynth.dll time stamp: 2021-01-11, 20:46:40 (UTC)
PluginDir2_5 (HKLM, x86): C:\Program Files\AviSynth+\plugins
PluginDir+ (HKLM, x86): C:\Program Files\AviSynth+\plugins+


[Clip info]
Number of frames: 162
Length (hh:mm:ss.ms): 00:00:06.757
Frame width: 640
Frame height: 480
Framerate: 23.976 (24000/1001)
Colorspace: YUV420P12
Audio channels: n/a
Audio bits/sample: n/a
Audio sample rate: n/a
Audio samples: n/a


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.971 | 107491 | 29.03
Process memory usage (max): 37 MiB
Thread count: 8
CPU usage (average): 64.9%

Time (elapsed): 00:00:05.581


[Script]

LWLibavVideoSource("Sample2.mp4")
AssumeFPS("ntsc_film")
Prefetch(2,frames=2)

I include the test with FFMS2:

Log file created with: AVSMeter 3.0.9.0 (x86)
Script file: Prueba.avs
Command line switches: -log

[OS/Hardware info]
Operating system: Windows 7 (x86) Service Pack 1.0 (Build 7601)

CPU: Pentium(R) Dual-Core CPU E5800 @ 3.20GHz / Wolfdale (Core 2 Duo) 2M
MMX, SSE, SSE2, SSE3, SSSE3
2 physical cores / 2 logical cores


[Avisynth info]
VersionString: AviSynth+ 3.7.0 (r3382, 3.7, i386)
VersionNumber: 2.60
File / Product version: 3.7.0.0 / 3.7.0.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\Windows\system32\avisynth.dll
Avisynth.dll time stamp: 2021-01-11, 20:46:40 (UTC)
PluginDir2_5 (HKLM, x86): C:\Program Files\AviSynth+\plugins
PluginDir+ (HKLM, x86): C:\Program Files\AviSynth+\plugins+


[Clip info]
Number of frames: 162
Length (hh:mm:ss.ms): 00:00:06.757
Frame width: 640
Frame height: 480
Framerate: 23.976 (24000/1001)
Colorspace: YUV420P16
Audio channels: n/a
Audio bits/sample: n/a
Audio sample rate: n/a
Audio samples: n/a


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.756 | 239787 | 5.697
Process memory usage (max): 35 MiB
Thread count: 7
CPU usage (average): 81.5%

Time (elapsed): 00:00:28.437


[Script]
FFVideoSource("Sample2.mp4")
AssumeFPS("ntsc_film")
Prefetch(2)

My Avs+ system is suffering of "mono-nucleosis".
I need your help.

[StainlessS]

@Wonkey,

Avisynth is a bit laxly specified, you can even have a variable with same name as a function, its not until
it tries to evaluate an expression that it can figure out what it is, and whether the result of that expression is
assigned to some variable or used in some way as an argument in another expression. It may not be used
for anyhting at all, eg just plonk a "123456" on a line by itself somewhere, with or without the double quotes.

I doubt whether avisynth script lnaguage could be properly described in Backus–Naur [used in Kernighan & Ritchie,
"The C Programming Language", at the back of the book to describe Std/ISO C]. Or in those language describing tools
derived from Unix "Lex", and the like.

Making "\" line continuation optional, is just making it even more lax than it already is, and is bound to come with a bundle of trip wires, safer to forget any changes at this stage in the life of AVS.
I guess Ben implemented the language to the point that it could get the job done, and no further.

Backus–Naur form:- https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form
The C Programming Language:- https://en.wikipedia.org/wiki/The_C_Programming_Language
Lex:- https://en.wikipedia.org/wiki/Lex_(software)
Lexx [The bestest Sci-Fi ever created, weird mix of Canadian-German humour]:- https://en.wikipedia.org/wiki/Lexx

EDIT:
plonk a "123456" on a line
Except the last line. [the entire script must evaluate to a clip]

EDIT: Fixed the Lex link, the trailing ")" of the url is wrongly appended as simple text after the remaining part of the link by the vBulletin url insertion thingy.

[jpsdr]

Ok, i've tested the following change :

static void Sobel_8(const unsigned char *psrc,unsigned char *pdst,const int32_t src_pitch, const int32_t dst_pitch,
const int32_t src_height,const int32_t dst_row_size, int32_t thresh)
{
const int32_t i = (dst_row_size-2 + 3) >> 2;

if (aWarpSharp_Enable_AVX)
{
for (int32_t y=0; y<src_height; y++)
{
JPSDR_Sobel_8_AVX(psrc+1,pdst+1,src_pitch,y,src_height,i,thresh);
pdst[0] = pdst[1];
pdst[dst_row_size-1] = pdst[dst_row_size-2];
psrc += src_pitch;
pdst += dst_pitch;
}
}
else
{
for (int32_t y=0; y<src_height; y++)
{
JPSDR_Sobel_8_SSE2(psrc+1,pdst+1,src_pitch,y,src_height,i,thresh);
pdst[0] = pdst[1];
pdst[dst_row_size-1] = pdst[dst_row_size-2];
psrc += src_pitch;
pdst += dst_pitch;
}
}
}

tested with :

a=AVISource("SP_IVTC.avi",False,"YV12").SetPlanarLegacyAlignment(True)
b=aWarpSharp2(a,chroma=3,threads=1)
c=aWarpSharp2(a,chroma=3,threads=0)

Subtract(b,c).Levels(127, 1, 129, 0, 255)

As i was hopping for, it seems it produces the same result.
So can you test if still crashing with previous change, and following change :

static void Sobel_16(const unsigned char *psrc,unsigned char *pdst,const int32_t src_pitch, const int32_t dst_pitch,
const int32_t src_height,int32_t dst_row_size, int32_t thresh,uint8_t bit_pixel)
{
const int32_t i = (dst_row_size-4 + 3) >> 2;

dst_row_size >>= 1;
thresh <<= (bit_pixel-8);

if (aWarpSharp_Enable_AVX)
{
for (int32_t y=0; y<src_height; y++)
{
uint16_t *dst=(uint16_t *)pdst;

JPSDR_Sobel_16_AVX(psrc+2,pdst+2,src_pitch,y,src_height,i,thresh);
dst[0]=dst[1];
dst[dst_row_size-1]=dst[dst_row_size-2];

psrc += src_pitch;
pdst += dst_pitch;
}
}
else
{
for (int32_t y=0; y<src_height; y++)
{
uint16_t *dst=(uint16_t *)pdst;

JPSDR_Sobel_16_SSE2(psrc+2,pdst+2,src_pitch,y,src_height,i,thresh);
dst[0]=dst[1];
dst[dst_row_size-1]=dst[dst_row_size-2];

psrc += src_pitch;
pdst += dst_pitch;
}
}
}

with the following :

aWarpSharp2(a,chroma=3,threads=1)


Edit
.... I realised too late, should have put this on my aWarpsharp thread... :(

[pinterf]

Ok, i've tested the following change :
...
with the following :

aWarpSharp2(a,chroma=3,threads=1)


Thanks, neither Sobel_8 not Sobel_16 is working (I don't have AVX in this test machine), because there is a new crash at e.g. in JPSDR_Sobel_8_SSE2
movntdq XMMWORD ptr[rsi+rdi],xmm2

I think changing the destination start by +1 (8 bit) or +2 (16 bit) will make the storage unaligned :(

Unfortunately one have to treat the left and rightmost loads specially when we want to keep the aligned access in the middle.

[jpsdr]

Argh... Forgot this. For purpose testing right not, does replacing "movntdq" with "movdqu" make it work ?
If yes, will create a specific asm function with "movdqu" for 1rst and last line.

Euh.... Why Sobel_8 worked for me on my Windows7 x86 avs2.60... .... Ah... picture was grey as expected in VDub and probably didn't notice an error message displayed in the bottom information line. If there is no pop-up crash, displayed picture was grey as expected, and i didn't realise... :(

I'll redo the test.

Edit
Indeed, didn't notice the error message in the bottom line of VDub...

Edit2
Replacing with "movdqu" solved, but if it seems to produce the same result for the 1 line pixel, it seems not for the last line pixel.
Can you confirm that with "movdqu" there is also no crash with CUDA ?

[pinterf]

I'll look into it tomorrow, this bug however is not Cuda aware build specific; pure luck if it did not cause troubles. Movnt is the streaming version of mov mnemonic and requires aligned access. Replacing it with .u (unaligned) will surely solve the problem.

[GMJCZP]

Friends, I am extremely worried, I made a great effort to buy my "new" card with a dual core processor and I have not been able to take advantage of its two cores, I feel like I have a Celeron, if someone gave a light, I am trying to update and improve all the scripts that I have posted but this situation is out of my hands.

[tormento]

Let me dream about at least MVTools2 on CUDA. :)

[Gavino]

Making "" line continuation optional, is just making it even more lax than it already is, and is bound to come with a bundle of trip wires, safer to forget any changes at this stage in the life of AVS.

In the AviSynth language, newline is a statement terminator (except when using the \ escape), but the parser can also recognise it has reached the end of a statement when the next symbol is not a valid continuation of what it already has. Therefore in most cases, the newline is not strictly necessary (though recommended for readability).
However, there are some cases where a newline is required (see this post and this one), so it is not possible to have the parser ignore them altogether.
I doubt whether avisynth script lnaguage could be properly described in Backus–Naur
Well, we do have the formal Avisynth grammar (http://avisynth.nl/index.php/Formal_AviSynth_grammar) in Extended Backus-Naur Form (EBNF) (contributed a long time ago by gzardakas and possibly slightly out-of-date).

[DJATOM]

So true...
https://i.imgur.com/S2eXJek.png

[pinterf]

So true...
https://i.imgur.com/S2eXJek.png
Nekopanda extracted and rewrote some stuff what was needed for his project. Actually KTGMC filter is the beginning of what you are searching for.
https://github.com/pinterf/AviSynthCUDAFilters/blob/master/KTGMC/MV.cpp#L5416

[DJATOM]

Interesting. To actually use that, I have to build Avs+ WIP or 3.7 will work?

[kedautinh12]

Interesting. To actually use that, I have to build Avs+ WIP or 3.7 will work?

Here had built x64 CUDA
https://drive.google.com/uc?export=download&id=1CpFdkqbNRDwtuHCtWiQ4x49ZCt8W2jkS

[DJATOM]

I already did my own build, now building boost libs since it's the only one dependency that I didn't yet resolved.

[DJATOM]

Apparently it doesn't work.
ClearAutoloadDirs()
AddAutoloadDir("C:\avsCuda\scripts")
AddAutoloadDir("C:\avsCuda\plugins")
DGSource("NCOP.dgi").onCPU()
KTGMC()
gives me Error: [KMasktoolFilterBase] CUDAГtГМБ[ГАВЁУ№Ч═В╡В─ВнВ╛В│Вв occurred while reading frame 0. while KTGMC().onCUDA() doesn't crash, but process doing nothing - it simply waiting for something.

[pinterf]

I don't remember now, have you built masktools as well? (cuda branch in my masktools2 repo)

[DJATOM]

I went through some steps of your building manual with minor changes (using cuda 11.3 and setting cuda arch 7.5 where it was set to lower version, also fixed afxres.h to windows.h in nnedi3 since it doesn't build on latest msvc).

[pinterf]

Ehh, that build manual is rather a 'log of my adventures on doing something I've never encountered before'. O.K., in a somewhat polished version, but I was happy that it worked as is for me. Anyway it can be a good start for someone who really wants to be involved in the project. (Probably not me, it requires weeks or months to have an active knowledge on it, though programming CUDA is a very interesting topic).

[pinterf]

My Avs+ system is suffering of "mono-nucleosis".
I need your help.
The problem is that you are trying to use multithreading on a simple script where is source filter is MT_SERIALIZED (just seen in the source). So it cannot be requested in a parallel way.

Such filters cannot be called again for a new frame until the previous frame is ready. There is blocking, new requests are getting into a queue. Prefetch(2) is starting threads with prefetching 4 frames in advance.

It is possible that the calls in this scenario are producing nonlinear frame access. In this case the somewhat slower execution is blocking the other frame requests, we are seeing a negative feedback.

This is not something which is debuggable easily. There is probably an 1/10000sec delay in timing conditions which results in the first out-of-sequence access to LWLibavVideoSource. Debug build is quick. But release build is getting into this state after some ten frames. When I put a simple line in the source code around the delay (cout::stdout << frame_number - really not a time consuming operation) which is writing the actual frame number to the standard output - the problem disappears.
Pretty much an the observer effect: the disturbance of an observed system by the act of observation.

I'd like to understand how it begins and how this chaotic internal state can be healed but it is not easy at all.

[tormento]

Would a moderator/contributor please start a new thread with tested CUDA filter builds?

(And perhaps AVS+ CUDA builds on top)

[real.finder]

Friends, I am extremely worried, I made a great effort to buy my "new" card with a dual core processor and I have not been able to take advantage of its two cores, I feel like I have a Celeron, if someone gave a light, I am trying to update and improve all the scripts that I have posted but this situation is out of my hands.

LWLibavVideoSource("Sample2.mp4")
Prefetch(1)
AssumeFPS("ntsc_film")
Prefetch(2) #or Prefetch(4,frames = 2)

this seems do it

edit: you can also try RequestLinear()

LWLibavVideoSource("Sample2.mp4")
RequestLinear(clim=100)
#~ Prefetch(1) you can also uncomment it
AssumeFPS("ntsc_film")
Prefetch(8)

[GMJCZP]

Thanks real.finder and pinterf. I have tried all the cocktails that have been recommended to me and I cannot beat Prefetch 1. In fact, I complicated the script and in AvsMeter with RequestLinear activated I got the following error message:

RequestLinear: Internal error (Frame not cached!).

[real.finder]

Thanks real.finder and pinterf. I have tried all the cocktails that have been recommended to me and I cannot beat Prefetch 1. In fact, I complicated the script and in AvsMeter with RequestLinear activated I got the following error message:

RequestLinear: Internal error (Frame not cached!).

did you have uptodate https://github.com/pinterf/TIVTC/releases ? it should be fixed https://github.com/pinterf/TIVTC/issues/20

[pinterf]

did you have uptodate https://github.com/pinterf/TIVTC/releases ? it should be fixed https://github.com/pinterf/TIVTC/issues/20
It is not 100%, just a trivial case was fixed. So the likelyhood you are seeing this error was decreased. There still can be circumstances in the frame request pattern when you get this error.

[GMJCZP]

I have TIVTC 1.0.26.

[pinterf]

Spent some hours on the topic. Finally here are some statistics.

Out of order frames have extreme penalty for this source filter / video file. The bigger the frame number to longer it takes.

Source filters are running in MT_SERIALIZED mode.

LWLibavVideoSource("Sample2.mp4", threads = 1, seek_threshold = 10 )
Prefetch(2)

FrameNo : time for GetFrame.
124 : 0.0127242 sec. Locktime: 1.77e-05 sec
126 : 0.0034897 sec. Locktime: 7.2e-06 sec
127 : 0.0062913 sec. Locktime: 6.5e-06 sec
128 : 0.0038575 sec. Locktime: 6.8e-06 sec
125 : 0.569717 sec. Locktime: 0.0231566 sec
129 : 0.0145048 sec. Locktime: 1.64e-05 sec
131 : 0.0042856 sec. Locktime: 8.1e-06 sec
132 : 0.0065563 sec. Locktime: 7.1e-06 sec
133 : 0.0035983 sec. Locktime: 6.6e-06 sec
130 : 0.572035 sec. Locktime: 0.0242634 sec

Lock time is a small overhead but it can be more if there are other slow GetFrames in queue

[GMJCZP]

I have also tried another video with Avisource and I notice that without Prefetch or Prefetch 1 is better than Prefetch 2 (Prefetch (4,2) is just slightly better). IMHO I still think that certain features of the E5800 collide with the Avs+ MT.

[Dogway]

Is there a way to run a for loop with floats? I tried converting to int but I think there's a limited set of supported int values.

Workaround (bisection method).

a = 0.000001
b = 0.1

a = int(a*pow(10,6))
b = int(b*pow(10,6))

function poly_beta (float a) { (10*a-10/pow(a,0.45-1)+1-4.5*a)*pow(10,6) }

a_n = a
b_n = b

for (i=a, b, 1) {
m_n = (a_n + b_n)/2
f_m_n = poly_beta(m_n)
if (poly_beta(a_n)*f_m_n < 0) {
a_n = a_n
b_n = m_n }
else if (poly_beta(b_n)*f_m_n < 0) {
a_n = m_n
b_n = b_n }
else if (f_m_n == 0) {
s_n = m_n }
else { Assert (false, "Failed") } }
s_n = ((a_n + b_n)/2 ) / pow(10,6)

[StainlessS]

For(i=1,10) {
f=i/10.0
...
}

Somethinkg like above, only way.
Mobile:

[FranceBB]

I know that it's probably not something totally important, but today I was working with a BT2020 content in PQ, I had to apply a LUT to convert to HLG that was working in Studio RGB (aka Limited Range RGB as output) and I realized that in Convert the matrix PC.2020 is missing...

So this works:

ConverttoYUV422(matrix="Rec2020")

but this doesn't:

ConverttoYUV422(matrix="PC.2020")

Of course I used AVSResize and everything went through correctly so that instead of getting the Limited range conversion twice (https://i.imgur.com/8lsww1a.png) I got the result I wanted (https://i.imgur.com/CZ6laKK.png), but still, I think it's something we should add in the core...

[pinterf]

.. realized that in Convert the matrix PC.2020 is missing...
Cruel world :)
Anyway, here is an actual snapshot build:
Avisynth+ 3.7.1 - 20210610 (https://drive.google.com/uc?export=download&id=1SRnFC53zrctCcHCbLF8hQ5AlEKud7-Sx) including XP and CUDA-aware builds
You can also find the link in the first post.
20210610 WIP
------------
- Add "PC.2020" to YUV-RGB conversion matrix set
- ColorBarsHD: use BT.709-2 for +I (Pattern 2), not BT.601
These are from the SMPTE RP 219-1:2014, but those are also on Wikipedia now: https://en.wikipedia.org/wiki/SMPTE_color_bars
Former values used BT.601 matrix coeff., which is wrong.
Also fixed Pattern 1 Green.Y to conform to SMPTE RP 219-1:2014 (133, not 134).
ColorBars: fixed studio RGB values for -I and +Q for rgb pixel types
- Speedup: Overlay mode "multiply": overlay clip is not converted to 4:4:4 internally when 420 or 422 subsampled format
(since only Y is used from that clip)
- Speedup: Overlay mode "multiply": SSE4.1 and AVX2 code (was: C only), Proper rounding in internal calculations
- Fix: ConvertAudio integer 32-to-8 bits C code garbage (regression in 3.7)
- ConvertAudio: Add direct Float from/to 8/16 conversions (C,SSE2,AVX2)
- Fix: ConvertAudio: float to 32 bit integer conversion max value glitch (regression in 3.7)
- Fix: Crash in ColorBars very first frame when followed by ResampleAudio
- Fix: frame property access from C interface (for more info see readme.txt)
- Fix: StackVertical and packed RGB formats: get audio and parity from the first and not the last clip
- RGBAdjust: analyse=true 32 bit float support
- experimental! Fix CUDA support on specific builds (apply lost-during-merge differences from Nekopanda branch), add CMake support for the option.
- Fixes for building the core as a static library

[tormento]

Avisynth+ 3.7.1 - 20210610
In dll properties I see 3.7.0 while in the previous CUDA enable version (3.7.1·3396) was 3.7.1. :)

[kedautinh12]

Thank pinterf

[GMJCZP]

pinterf, will there be good news in this snapshot regarding problems with my e5800?

[pinterf]

pinterf, will there be good news in this snapshot regarding problems with my e5800?
No, it is not specific to your e5800. I was able to reproduce it on my i7 as well, since I posted here exact timing data. It's like the source filter behaves ultra-slow when encountering out-of-sequence frame request. Cannot help with it.
I don't know if this is also related to the actual video file encoding (e.g. decoding the 100th frame requires to decode all the preceeding frames) because it seems that it becomes slooower and slooooower when the frame number increases.

[pinterf]

In dll properties I see 3.7.0 while in the previous CUDA enable version (3.7.1·3396) was 3.7.1. :)
Arrgh. Thanks. I'll replace them soon.
EDIT: the file behind the link has been replaced.

[kedautinh12]

Thanks

[tormento]

Thank pinterf
Thanks

Much better the Forum admin (if any still alive) could implement a Thanks! button. :p

Most of the times I don't thank on the Forum but in my mind only because I know the cluttering it should result.

[GMJCZP]

No, it is not specific to your e5800. I was able to reproduce it on my i7 as well, since I posted here exact timing data. It's like the source filter behaves ultra-slow when encountering out-of-sequence frame request. Cannot help with it.
I don't know if this is also related to the actual video file encoding (e.g. decoding the 100th frame requires to decode all the preceeding frames) because it seems that it becomes slooower and slooooower when the frame number increases.

See please the post 1028 (https://forum.doom9.org/showthread.php?p=1944278#post1944278).

I quote this because, Imho, it is not a source problem, but at least my e5800 is not alone in this world.

Edit: once using BeHappy I was modifying the option "Parallel jobs" to encode more than one song at a time and I noticed that some tracks were cut off, as if the program was choking on so many songs. This problem with Prefetch reminded me of it.

[kedautinh12]

See please the post 1028 (https://forum.doom9.org/showthread.php?p=1944278#post1944278).

I quote this because, Imho, it is not a source problem, but at least my e5800 is not alone in this world.

I think you need buy new cpus for modern 😂😂😂

[real.finder]

See please the post 1028 (https://forum.doom9.org/showthread.php?p=1944278#post1944278).

I quote this because, Imho, it is not a source problem, but at least my e5800 is not alone in this world.

Edit: once using BeHappy I was modifying the option "Parallel jobs" to encode more than one song at a time and I noticed that some tracks were cut off, as if the program was choking on so many songs. This problem with Prefetch reminded me of it.

I think this HDD/SSD limit

I did some tests with lossless avi

Video
ID : 0
Format : YUV
Codec ID : YUY2
Codec ID/Info : YUV 4:2:2 as for UYVY but with different component ordering within the u_int32 macropixel
Duration : 2 min 54 s
Bit rate : 165 Mb/s
Width : 720 pixels
Height : 480 pixels
Display aspect ratio : 3:2
Frame rate : 29.970 (30000/1001) FPS
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:2
Compression mode : Lossless
Bits/(Pixel*Frame) : 15.936
Stream size : 3.36 GiB (99%)

Audio
ID : 1
Format : PCM
Format settings : Little / Signed
Codec ID : 1
Duration : 2 min 54 s
Bit rate mode : Constant
Bit rate : 1 536 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Bit depth : 16 bits
Stream size : 32.0 MiB (1%)
Alignment : Aligned on interleaves
Interleave, duration : 10 ms (0.30 video frame)
Interleave, preload duratio : 49 ms

AVISource("output.avi")
AssumeFPS("ntsc_film")
about 135 fps


AVISource("output.avi")
RequestLinear(clim=100)
AssumeFPS("ntsc_film")
Prefetch(8)


and


AVISource("output.avi")
Prefetch(1)
AssumeFPS("ntsc_film")
Prefetch(2)


both about same as 1st above (about 135 fps)

AVISource("output.avi")
AssumeFPS("ntsc_film")
Prefetch(8)

about 99 fps

AVISource("output.avi")
Prefetch(1)
AssumeFPS("ntsc_film")
Prefetch(8)

about 105 fps

AVISource("output.avi")
AssumeFPS("ntsc_film")
Prefetch(2)

about 132 fps

[Boulder]

I think this issue is the same I ran into with the Avisynth version of Zopti some time ago.
https://forum.doom9.org/showthread.php?p=1940286#post1940286

[GMJCZP]

real.finder:

As I have two HDDs operating I changed the location of the video source (from a WD1600AAJS, the system one, to a WD10EZEX) to check what you say and here are the results of some tests:

[Script]

LWLibavVideoSource("video.mp4")
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(1)

[Runtime info]
Frames processed: 1501 (0 - 1500)
FPS (min | max | average): 10.19 | 222658 | 40.73
Process memory usage (max): 73 MiB
Thread count: 7
CPU usage (average): 77.1%

Time (elapsed): 00:00:36.849


With Prefetch(2):

[Runtime info]
Frames processed: 1501 (0 - 1500)
FPS (min | max | average): 12.77 | 101.3 | 50.47
Process memory usage (max): 80 MiB
Thread count: 8
CPU usage (average): 95.5%

Time (elapsed): 00:00:29.743


With Prefetch(4,2):

[Runtime info]
Frames processed: 1501 (0 - 1500)
FPS (min | max | average): 33.49 | 91.06 | 51.89
Process memory usage (max): 76 MiB
Thread count: 10
CPU usage (average): 94.9%

Time (elapsed): 00:00:28.925

It could also be that this HDD, being the latter faster, has helped in the improvement. But if you comment that there is a limit, how could this be solved? According to Crystal DiskInfo both HDDs are in good condition.

Edit: To clarify more, the 160 GB HDD has two partitions, C and D, in D is the video source and I copied it to the other HDD for testing.

[GMJCZP]

I think this issue is the same I ran into with the Avisynth version of Zopti some time ago.
https://forum.doom9.org/showthread.php?p=1940286#post1940286

Thanks for the information.
That is why I have emphasized post #1028.

[real.finder]

It could also be that this HDD, being the latter faster, has helped in the improvement. But if you comment that there is a limit, how could this be solved? According to Crystal DiskInfo both HDDs are in good condition.

It depends on many things like where is the file stored on the hard disk media (near the edge or near the center) and also fragment

anyway, I think Prefetch need add some Frame Cache method for source call filters to avoid this problem (Depending on threads)

[real.finder]

[Script]

LWLibavVideoSource("video.mp4")
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(1)



can you test with

LWLibavVideoSource("video.mp4")
Prefetch(1)
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(2)

or better

LWLibavVideoSource("video.mp4")
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Prefetch(1)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(2)

since Prefetch(4,2) is less random access than Prefetch(2) because Prefetch(2) = Prefetch(2,4) (http://avisynth.nl/index.php/SetFilterMTMode#Prefetch)

[GMJCZP]

Before doing the tests that you suggest I inform you that I obtained the same results with the video source in partition D, for Prefetch 1 and 2, so that, with respect to the HDDs, we return to the starting point, anyway I have defragmented C and D.

[real.finder]

Before doing the tests that you suggest I inform you that I obtained the same results with the video source in partition D, for Prefetch 1 and 2, so that, with respect to the HDDs, we return to the starting point, anyway I have defragmented C and D.

yes, in your compressed in hevc mp4 HDD speed didn't matter much, Actually, you raised two problems, the mp4 problem resulting from nature of hevc compressing and LWLibavVideoSource as said here https://forum.doom9.org/showpost.php?p=1944004&postcount=991

and the avi problem that have a close relationship with HDD speed, but both cases have the same reason (Prefetch case random access that case slowdown in them) and both can have one fix

[Boulder]

The issue has little to do with HDD speed, at least in my case. The file is small enough to fit in the Windows read cache and the file is located on an SSD. To me it seems that all source filters suffer from it, just tested encoding my analysis clip into a lossless HEVC file and used DGSource to decode it. GPU usage was ~100% all the time, in a normal situation it's only a few percent. Besides, the Vapoursynth Zopti works much better with multithreading with the exact same file and setup.